Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
Authors: Shuo He
Lang Feng
Qi Wei
Xin Cheng
Lei Feng
Bo An
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
Policy optimization
Long-horizon tasks
Hierarchical groups
Primary area: reinforcement learning

Abstract: Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.

Published as a conference paper at ICLR 2026
HIERARCHY-OF-GROUR POLICY OPTIMIZATION FOR LONG- HORIZON AGENTIC TASKS
Shuo He', Lang Feng', Qi Wei', Xin Cheng', Lei Feng"; Bo An Nanyang Technological University', Southeast University? 1enoohe23@@maill com, fenglei Lenglei@seu. edu. en
ABSTRACT
Group-based reinforcement learning (RL), such as GRPO, has advanced the capa- bilities of large language models on long-horizon agentic tasks. To enable more fine-gres aue-ggrined policy updates, recent research has increasingly shifted toward step- wise group-based policy optimization, which treats each step in rollout trajectory independently while using a memory module to retain historical context. How- ever, we find key issue in estimating stepwise relative advantages, namely con- text inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead severely biased advan- tage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step. HGPO computes dis- tinct advantages within each group and aggregates them with an adaptive weight- ing scheme. In this way, HGPO can achieve favorable bias-variance trade- off in stepwise advantage estimation, without extra models or rollouts. Evalua- tions two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5- I.SB-Instruct and Qwen2.5-1 BB---nntttt show that HGPO significantly outper- forms existing agentic RL methods under the same computational constraints. Code is available at httt:: thb.. www.m//aggeggg//////oooooo verl -ageet/ Lree/mmstee//eecpp//tggg
saaarccy-of-Grouppe Policy Optimization
INTRODUCTION
Versatile agents powered by Large Language Models (LLMs) can perceive, reason, and act in complex, open-ended environments (Achiam et al. 2023 Team et al. 2023 Yang al., [024]] Liu et al.] [2024. Representative applications include embodied assistants navigating simulated homes (Shridhar et al. 2021 et al.) 2024), web navigators completing browsing tasks (Furuta et al. 2024 Zheng al. 2024 Gou et al.) 2025), and autonomous explorers in interactive computer games (Wang et al. 2024a bp. Beyond language and vision understanding, such agents are expected to perform long-horizon planning and robust decision-makin
Deep reinforcement learning (RL) (Sutton & Barto 2018 has emerged as a key paradigm for en- hancing agent performance in the post-training stage OpenAl 2024 Guo et al.) (025). In particular,
croup-based RL methods such as RLOO (Kool et al.) 2019: Ahmadian et al. 2024). GRPO (Sha
et al. 2024), DAPO (Yu et al., 2025ch, Clip-Cor (Cui et al. 2025), and GSPO (Zheng et al.) (025) have demonstrated strong performance in large-scale RL training while requiring fewer computa- tional resources. These methods have proven effective in single-turn tasks such mathematical reasoning (Liu et al. 2025) Yu et al. 2025c) and code generation (Wei et al. 2025a). To extend this paradigm multi-ture settings, approaches such RAGEN (Wang et al. 2025d) and Search- RI (Jin et al. 2025a) adopt trajectory-wise policy optimization Tramework, which concatenates environment states and model outputs across turns enable multi-turn rollouts. However, this framework suffers from major limitation: the effective context length grows rapidly with the num- ber of interaction turns, leading to severe context explosion.
"oorreponndngg author

---

Published as a conference paper at ICLR 2026
Trajectory-wise Policy Optimization
T1
[[46 (00)
(41 (1 (e2 =0)"))
[(9g
00
130
2
42
3
4g
Gg
Mask
Mask
Mask
Reward
Step-wise Policy Optimization
Trajectory-level Advantage Fstimation 'Trajectoryllevel
Step-level Aegpaatt
Advantuge Estimation
8t. dt
Trajectory-Jevel advanfagg
Inconsistent
Step-level
Fine-grainel Blased advantage
Ø
81
@1
Step-1
proup
Aisturic couttext
group
A (T.)
133
A (
h2
Step-2
Memory
(7
13)
A (x
Ø
h3
F'
Step33
he+11
Coarse-gruined biased advantage
(a) Traj. & Stepwise Policy Optimization
(b) Inconsistent historic context results in biased advantage estimation
Figure 1: Figure (a) compares trajectory- wwiee and stepwise policy optimization frameworks. Given two example group trajectories, Figure (b) illustrates trajectory-levar and step-level grouping with their corresponding advantage estimations. Best viewed in color.
Oraele
Adv. Oracle Adv.
GRPO
GIGPO
1Oaaee
0.5
20
60
80
100
1441
641
5U
140
10B
160
20
641
(a) Advantage Difference in GRPO
(b) Advantage Difference GIGPO
Averaage Group
Step
Figure 2: Statistics of GRPO and GiGPO. Figures (a) and (b) present the advantage differences
relative to Oracle advantages for GRPO and GiGPO, respectively. Figures (c) and (d) report the average group size and the proportion of Oracle steps, respectively.
To address this issue, recent research has shifted toward the stepwise policy optimization frame-
work (Fng et al. 2025b; Luo et al. 2025c; Chen et al. 2025b) Team 2025) Yu et al. 2025b Wang al. 2025e). which treats each step within a rollout trajectory independently while leveraging memory module to retain historical context. This design allows for flexible context management and highly scalable RL training. A comparison of the two frameworks is illustrated Figure (a). Building on the stepwise framework, group-based RL methods such as GRPO (Shao et al., 2024) can be adapted into stepwise group-based variants for long-horizon agentic tasks. Furthermore, to enable finer-grained credit assignment, GiGPO (Feng et al., 2025b) extends GRPO by estimating additional step-level advantages within groups where all steps share the same current state.
However, we identify key issue in estimating stepwise group relative advantages: historical context inconsistency. This issue occurs when rollout steps share the same current state but differ in their historical contexts. As illustrated in Figure (b), given two group trajectories T and 72, the step- level group at state (purple) contains steps with inconsistent historical contexts. Intuitively, the relative advantage of the selected actions should be computed under the same current state and the same historical context. When historical contexts vary, the estimated relative advantage can become biased, which in turn may degrade policy optimization.
To further explore this issue, we conduct pilot empirical analysis. We introduce Oracle groups, in which all steps share not only the same current state but also identical historical contexts. During GRPO and GiGPO training. we track the group sizes, step counts, and estimated advantages of these Oracle groups. and compare them with trajectory-level and step-level advantage estimates. As shown in Figure 2
(a) and (b), both trajectory-level and step-level advantages exhibit notable estimation bias, with the bias at the trajectory level being substantially larger. These results indicate that historical context inconsistency can severely distort advantage estimation. A straightforward solution is to use only Oracle steps for policy optimization. However, as shown in Figures 2 (c) and (d), Oracle steps are generally scarce within trajectories (i.e., their ratio is low), making this approach inefficient. Moreover, the average group size of Oracle steps is small, which increases the variance of estimated advantages and undermines the stability of RL training.
To address the above challenges, in this paper, we propose Hierarchy-of ofrehh--o-Grroonee aaforr Policy Optimiza- tion (HGPO), novel RL training algorithm that introduces better advantage estimator capable
2

---

Published as a conference paper at ICLR 2026
of low-bias and balanced-varrance. Specifically, HGPO is built on two key components: context- aware hierarchical grouping and adaptive weighting advantage estimation. First, within each rollout, HGPO groups steps that share the same current state and further assigns them to multiple hierarchi- cal groups according to their historical contexts. This hierarchical structure captures advantages at different context depths, improving data utilization and reducing variance. Second, HGPO ag- gregates the group advantages using an adaptive weighting scheme: groups with more consistent historical contexts are assigned larger weights, thereby lowering estimation bias. In this way, HGPO produces more reliable stepwise advantage estimates for policy optimization. We evaluate HGPO on two challenging agentic benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B Instruct and Owen2.5--BBIInstruee Results show that HGPO consistently outperforms existing baselines while maintaining the same GPU memory usage, using identical LLM rollouts, and incurring minimal additional time cost. Our main contributions are summarized as follows:
Revealing historical context inconsistency. We reveal the issue of context inconsistency in step- wise group-based RI. and empirically demonstrate that it introduces significant bias in advantage estimation, thereby degrading policy optimization.
Proposing novel policy optimization algorithm. We introduce Hierarchy-of-Grove Policy Optimization, which constructs hierarchical groups for each step based on historical context and adaptively aggregates their advantages.
Achieving strong empirical performance. HGPO achieves ofte-offtheeart results on two chal- lenging agentic benchmarks under the same computational constraints.
2 RELATED WORK
LLM-based decision-making agents. Large language models (LLMs) have been widely adopted
as autonomous agents across diverse domains, including device control (Zhang & Zhang 2024]
Hong et al., 2024 Gur et al. 2024 Hu et al. 2024), code generation (Zhang et al. 2024b),
game
interaction (Wang et al. 2024a Tan et al. 2025), and robotics (Zitkovich et al. 2023). Early approaches often relied fixed pre-trained models guided by structured prompting, such Re- Act (Yao et al., 2023) and Reflexion (Shinn et al. 2024). augmented with memory and retrieval mechanisms (Wang al. 20244 Tan al. (024) or tool integration (Schick et al. 2023 Xie al.)
2024 Zhang et al. 2024a). While such methods are simple and require no additional training, they remain limited in applicability domain-
tasks, largely due to the lack of specialized knowledge in the pre-training of the base models.
Reinforcement learning for LLM- agents. Reinforcement learning (RL) (Sutton Barto 2018) has been central to adapting large language model (LLM) agents to dynamic and open- ended environments. Early work applied classic algorithms such as DQN (Mnih et a 2015) to text games (Narasimhan et al. 2015), followed by value-based methods like PPO (Schulman et al.)
2017 and AWR (Peng et al.) 2019) in interactive domains including mobile control (Rawles et al.
2024 embodied tasks in ALFWorld (Shridhar et al. (02I). and card games (Brockman 2016
More recent research has extended RL to web and application environments (Qian et al.
2025
Sun al. 2025), with methods such ArCHer (Zhou et al. 2024b), AgentQ (Putta et al.)
2024)
CoSo (Feng et al., 2025a), and LOOP (Chen et al.] 2025a)) In parallel, RL has also become inte-
gral to LLM training itself, with RLHF (Ziegler al. 2019 Stiennon al.) 2020; Ouyang et al.)
2022 Rafailov et al.) 2024) aligning models with human preferences, and group-based RL algo-
rithms emerging as scalable and efficient alternatives to PPO. Approaches such GRPO (Shao et al. 2024). Dr. GRPO Luu et al. 2025). Clip-Cov (Cui et al. 2025). GSPO (Zheng et al. 2025) and DAPO Yu al., 2025c) avoid value networks by estimating advantages over groups of sam- ples. However, most of these methods are designed for single-turn interactions and thus struggle with context consistency in long-horizon agentic tasks.
Long-horizon agentic reinforcement learning. Long-horizon agentic RL. (Laban et al., 2025 Zhang et al., 20255] Zhou et al., 2025a Luo et al.) 2025d] Wang et al., 2025a) extends LLMS from single-turn generation to multi-turn decision-making where RL equips them with plan- ning (Hao et al.) 2023) Zhou et al. 2024a Song et al. 2024), reasoning (Chu et al.] 2025), and memory (Jin et al. 2024: Chhikara al. 2025: Zhou et al. 2025b) capabilities for sus- tained interaction in dynamic environments. Applications span code generation (Jiang et al.]
2024 Gehring et al. 2025 Jain et al. 2025 Chen et al. 2025c Jin al. 2025b), software
3

---

Published as a conference paper at ICLR 2026
gineering (Wei et al. 2025b Luo et al., 2025), and GUI interaction Wei et al.
2025a Shen et al.
2025 Wang al. 2024c Lin et al.
Qin et al. 2025). .. Recent advances include long- mong-ooizon policy optimization frameworks (Wang al. 2025d Jin et al., 2025a) that optimize over multi-turn rollouts, and stepwise policy optimization meth- ods (Feng et al. 2025b Luo et al., 20225c Chen et al., 22025b] Team 2025) that treat each step inde- pendently while retaining history through memory modules. Yet, stepwise methods often suffer from context inconsistency across long horizons, limiting their effectiveness in complex agentic tasks.
2025c Lu et al.
2025 Luo et al., 2025b
PRELIMINARIES
Problem setup long-horizon agentic tasks. Unlike single-turn tasks, long-horizon agentic tasks require LLM agent interact with the environment across multiple turns to accomplish a goal. Formally, given a task example æ E p(X), which typically includes a fixed task-related description, an LLM-based agent нæ parameterized by 9 observes environment state 8, S at each turn f and generates a textual action a, E V*, where denotes the token vocabulary and n is the maximum generation length. Here t = (1, ... T). with T being the maximum number of in- teraction turns. In this paper, we focus on the sparse delayed reward setting, where the environment provides a scalar reward r. E R only at the final step of trajectory т {(81, ...,..., (ST, aT)).
Trajectory-wise vs. stepwise policy optimization. Conventional trajectory-wise policy optimiza- tion frameworks (Wang al., 2025d Jín al., 2025a Wang al. 2025b Yu et al. (025a)) typ- ically concatenate the full interaction history of rollout trajectory t for policy optimization, i.e., æ). However, as the number of turns T grows, the context length increases rapidly, which limits the scalability and feasibility of long-horizon RL training. In contrast, stepwise policy opti- mization frameworks (Feng et al., 2025b] [uo al. [025c] [Cenn et al., 2025b) Team (2025) decou- ple the trajectory into individual steps while leveraging memory module that maintains K < T historical contexts. This memory module is updated with the latest interactions, keeping the prompt length relatively stable and enabling more scalable RL training.
Group-based reinforcement learning. Unlike PPO (Schulman et al. 2017), which estimates ad- vantages using additional value function, group-based reinforcement learning (RL) algorithms such as GRPO (Shao et al., 2024) compute advantages directly from the statistics a sampled group of trajectories G,. Specifically, GRPO originally designed for single-turn tasks under trajectory-wist policy optimization framework. To extend it to long- hong-horizon tasks, we adapt it to the stepwise setting and calculate the trajectory-level advantage as:
= ( (B)))=((()-1/GG ESea. R((T)) /oG.,
1/1G,/
where oc, denotes the standard deviation rewards within the group G,. This trajectory-Ievel computation assigns the same advantage value to every step in trajectory т., thereby overlooking the finer credit assignment required within a trajectory. To address this limitation, one can instead adopt step-level group relative advantage estimator (Feng al. (025b). Here, steps with identical current states & across all group trajectories are clustered into step-level groups Ga,, and their advantages computed
A [((s)) -
1/|Ga, Σ ((6)) /oG..
Compared to Eq. (T), the step-level estimator in Eq. (2 provides more fine-grained and effective credit assignment across steps within the same trajectory.
4 TRAINING AGENTS WITH HGPO FOR LONG-HORIZON AGENTIC TASKS
4.1 THE ISSUE OF HISTORICAL CONTEXT INCONSISTENCY
Originally, group relative advantage estimation (Sao et al. 2024) compares the relative advantages of different group responses generated from the timization setting shown in Figure even for a fixed task in the environment, rollout steps within a step-level anchor group (i.e., steps that share the same current state) may still have distinct historical
prompt. However, in the stepwise policy op-
4

---

Published as a conference paper at ICLR 2026
Rollout Group Trajectory (K=2)
Context-aware Hierarchical Grouping
Adaptive Weighting Advantage Estimation
(3)
F1
(5)
grewp
Bias-variance maac-ooff in advantage estimartion
group
181
A
A
(3)
(3)
18
04
Uitter centest
2-contest
Adapiive weighting based on histuric context
M
01 A (8)
Figure 3: Overview of HGPO. The LLM-based agent interacts with of environments initialized from the same state Su. producing four group trajectories (states with the same color are identical), HGPO comprises two key components: context-aware hierarchical grouping and adaptive weighted advantage computation. For illustration, consider the state $2 (purple). First, HGPO assigns $2 into three hierarchical groups according to its historical contexts. Then, it computes the final advantage estimate by adaptively aggregating the weighted advantages from these groups.
contexts in their memory modules. As result, the effective prompts for these step-level groups can differ, which may bias the estimated advantages. Ideally, Oracle steps, i.e., those sharing the same prompt, including both the current state and the historical context, would yield the most accurate advantage estimates for policy optimization. A straightforward approach is therefore to perform policy optimization using only Oracle steps. In practice, however, Figure 2 shows that Oracle steps are rare in rollouts, and their group sizes are typically small, which can destabilize training. Mo- tivated by these challenges, we propose leveraging hierarchy-of-gree off-guupsss structure obtain more accurate advantage estimates, reducing bias while keeping variance low.
4.2 HIERARCHY-OF-GROUG POLICY OPTIMIZATION
In this subsection, we introduce HGPO as shown Figure consisting of context-aware hierarchi- cal grouping and adaptive weighting advantage estimation.
Context-aware hierarchical grouping. We begin by introducing context-aware hierarchical grouping, which organizes steps into multi-level groups according to their historical contexts. The key intuition is that the advantage each step should be evaluated relative to different historical contexts to obtain more accurate estimates. Specifically, same current state, and then, within each group, we construct multiple hierarchical groups based on the consistency of their historical contexts. Steps with longer common histories are assigned to higher-level hierarchical groups. This hierarchy-of-grouy structure enables more fine- fine-grained com- parisons and brings two main benefits: (i) it improves step utilization for advantage estimation, and (ii) it reduces the variance of estimated advantages.
first group together steps that share the
Formally, let the i-th trajectory be T; [(={ a ( a (................. the maximum context length. We define k-step context operator for the t-th step as:
(57 aT }. and let K denote
[
(1) 8
{()), 1 k;
(a(s( (1)
(3)
) t k,
where A E [0, K]. This operator returns the io historical states preceding the current state. Based on this operator, we define the k-th hierarchical group for the t-th step as:
G (s) (j..) E I Cx(s) )) where the index set I - ((i, | 1 S N, <ts T). Considering all hierarchical groups, the resulting hierarchy-of-grooe
(4)
structure satisfies:
(5) When K = 0, the hierarchy-of -r-uus degenerates to the step-level grouping G (s used in (Feng et al. 2025b). Importantly, the entire context-aware hierarchical grouping procedure op- erates fully offline: requires only hashmap lookups over existing rollouts, without relying on additional models extra data collection.
Gl (s, (8) ) 2 (s 2 ... G
G (s. (s. ... (x
5

---

Published as a conference paper at ICLR 2026
Adaptive weighting advantage estimation. Intuitively, higher-level hierarchical groups yield more accurate advantage comparisons since they incorporate richer historical context. Building on this insight, introduce an adaptive weighting scheme that integrates information across all hierarchical groups with appropriately assigned weights, thereby enabling stable and efficient esti- mation of group-relative advantages. Formally, the advantage estimation for the k-th hierarchical group is defined as:
= - E
D Gn R( /
(6)
Finally, the advantage aggregated from K hierarchical groups is denoted by:
A H (s)
(1)
Wk. A (s())
k=0
where the adaptive weight wy (++11 tage information along the hierarchy-of-groug in Eq. (5): higher-level groups are preferred due
(k+1))
(a 2 0). is worth noting that Eq. ( fuses advan-
stronger context consistency. Besides, for each step (s a, (a) we compute its stepwise reward (4) Σ ( (Feng et al. 2025b), where E (0, 1] is the discount factor. In this way, we can obtain a stepwise reward for each step in the trajectory.
rt
The objective for policy optimization. The policy optimization objective HGPO is:
min (s(("))))))))))))))))))))))))( clip(po(a, +) (s("))
JHGPO
NT
9DI (To(: x)) Fear(- x)),
(8)
where pa(at (1)
zo(a [e") [e) new "e the importance sampling ratio, 3 controls the strength of the
KL penalty. The pseudo-code shown in Algorithm of Appendix A
Proposition 4.1 (Bias-variance trade-off in HGPO) Let by and es denote the bias and variance
of the estimated advantage A within the k-th group G Based on the following conditions: (1)
Bias satisfies, i.e., Br 2 by 2 (b..... bk-1) 2 by 2 0: (2) Variance satisfies, i.e. to S
(v1,-.. UK-1) S UK- Vr. the bias and variance of the estimator A are
Σx
K
K
Aias[A"] Bias
W
wybks
Var[A*]= Var
Σ
K A
Wk
w VarlA
-Ex
k= we
k=o
k=o
Furthermore, the bias and variance of the advantage estimator in HGPO satisfy that
by < Bias[ A 5.0 bo S BT,
K +1 < | S UK S VT,
U0
where Br, bo. bx. and Vr, Up. denote the bias and variance of the trajectory-level, step-level, and Oracle advantage, respectively. Overall, the bias and variance of the HGPO advantage estimator interpolates between the step-level (k 0) and Oracle (k K) estimators, thereby achieving better trade-off. Proof and more details are provided in Appendix
EXPERIMENTS
EXPERIMENT SETUP
Agentic benchmarks. We train the LLM agents two challenging benchmarks: ALF- World (Shridhar et al.] (202T) and WebShop (Yao et al. 2022). which are designed to assess the ability of L.M agents to perform multi-step mecision-making. The details are shown in Appendix C.2

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on ALFWorld and WebShop. For ALFWorld, we report the over- all success rate (f) for both in-distribution (In-Success) and out-of-dsstrbuute tasks (Out-Success). For WebShop, we report the average task score (1) and the average task success rate (t). Most results are averaged over 3 random seeds during testing. The best results are highlighted in bold.
ALFWorld
WebShop
Model
Type
Method
In-Success
Out-Success
Task Scores
Task Success Rates
Prompting
GPT-4o
48.0
46.0
31.8
23.7
Closed
Prompting
Gemini-2.5
60.3
50.5
42.5
35.9
Prompting
Qwen2.5
4.1
23.1
5.2
Qwen2.5-1.55 1..52.5-1..--nstteeee Instuutruct
Prompting
ReAct
12.8
40.1
11.3
Prompting
Reflexion
21.8
55.8
21.9
RL Training
PPO (with critic)
54.423.1
73.843.0
51.5229
RL Training
RLOO
69.7+2.5
68.7+10.7
73.945.6
52.1
RL Training
GRPO
72.8a36
70.1
75.8,3.5
56.8,3.8
RL Training
GiGPO (K=2)
10.16
84.7622.83
84.95a2.57
66.532192
RL Training
HGPO (K=2)
92.77
90.1640.78
85,5622.86
71.54_4.00
RL Training
GiGPO (K=4)
93.2921.07
91 91.53_1.99
86.80
73.24,2:25
RL Training
HGPO (K=4)
94.85
92.1241.63
90.6421.05
78.12. 78.12-2.06
Prompting
Qwen2.5
14.8
26.4
7.8
Qwen2.5-77 7B-BBB Instrrrcct
Prompting
ReAct
31.2
46.2
19.5
Prompting
Reflexion
42.7
58.
28.8
RL Training
PPO (with critic)
77.08a1.12
76.23=1.46
81.4g3.1
68.7+5.1
RL Training
RLOO
77 77.86=0.03
73.95
80.3
65.7+4.
RL Training
GRPO
78.64
76.82+144
79.3+2.8
66.1
RL Training
GiGPO (K=2)
93.29
92.18 18..39
88, 18.9341.4
77.60+1.8
RL Training
HGPO (K=2)
95.44 95.44.0.62
92.05+0
88.96-1.04
78.51
RL Training
GiGPO (K=4)
95.6 95.63.0.49
98.1820.98
89.51
77.79
RL Training
HGPO (K=4)
95.96.0.49
94.87+1.4
88.49.0.4
79.29_1.1
Comparing methods. We compare HGPO with many competitive baselines: (1) Closed-source
LLMs: GPT-4o (Achiam et al.) 2023) and Gemini-22 P.5-Pro TTam et al. (003)). (2) Prompting agents:
ReAct Yao et al. 2023) and Reflexion (Shinn al. 2024). (3) RL training methods:
PPO (Schul-
man et al. 2017). RLOO Kool et al. 2 et al. 2024), GRPO (Shao et al.) 2024), and
GiGPO (Feng al. 2025b). The details are shown in Appendix C.1
Yang et al. 2024) as our base models. For fairness, all RL training methods share the same hyperparameter figurations. Specifically, the rollout group size N in group-basee RL methods is set to 8. Each LLM agent is prompted first generate chain-of-thought (Wei al., (2022) enclosed within <think> </think> tags, followed by the action enclosed within <action> </action> tags. For HGPO, the weighting coefficient in Eq. (7) is set to and we omit groups with zero advantage Eq. 7 for adaptive weighting. For evaluation, we set three different random seeds and report the mean and standard deviation of the performance. The max step is set to 50 and for ALFWorld and Web- Shop, respectively. Full training setups and hyperparameter details are provided in Appendix
Implementation details. We adopt Qwen2.5-1.35 1----.5--Innstreee and
5.2 EXPERIMENTAL RESULTS
HGPO achieves the best overall results. Table I shows a pronounced gap between prompting and RL training methods. On both ALFWorld and WehShop, all RL.-trained methods substantially outperform the prompting baselines (Qwen2.5, ReAct, Reflexion), confirming that RL training is crucial for these long-horizoo tasks. With Qwen2.5-1... 1.55..55-.5B-Instruuc HGPO consistently improves over GiGPO by average of 4.01 % on ALFWorld (K = 2). 1.08% on ALFWorld (K 4), 2.81 % on WebShop (K = 2). and 4.36% on WebShop (K = 4). With Qwen2.5-7B 5-77-Inntruct HGPO re- mains superior, achieving up to 95.96% in-distribution success rate ALFWorld and 79.29% task success rate on WebShop. Moreover, all baseline methods experience significant performance degra- dation on out-of-distributiio tasks in AoFWorld. Notably, HGPO maintains superior performance
7

---

Published as a conference paper at ICLR 2026
1-Contest
2-Conteat
43
0.1
Stane
Group Stoee
Group Stoee
0-Cantect Webshop
1-Contest Webchop
2-Coatest Webshop
033
6.1
Group Stzwa
Group Stees
Group Stees
à Centext Group Allworld
Context Group Webshop
4-oostett Group Webshop
4.
0.3
Group
Group
Figure 4: Distributions of hierarchical group sizes on ALFWorld and WebShop using Qwen2.5- I.St-Instruct. "0/1/2/3/4-Comext" indicates different hierarchical groups. The first two rows corre- spond to K 2, and the last row corresponds to K = 4. The y-axis denotes the proportion.
with less degradation compared to GiGPO. This observation suggests that context inconsistency can severely impair policy generalization, while HGPO's hierarchical grouping mechanism provides bust and stable advantage estimation, enabling improved generalization to unseen tasks. Overall, HGPO achieves the strongest performance across settings.
HGPO brings larger gains on small models. Table T shows that HGPO achieves larger improve- ment with Qwen2.5-1.5B Instruct than with Qwen2.557B-Instrucc, with an average gain of 3.41% vs. 0.74% for K = 2 and 2.72% vs. 0.13% for K = 4. We explain that Qwen2.5-1.5B-Imnn tends to generate longer and more redundant steps during rollout due to limited agentic capabil- ity, which can introduce larger bias in advantage estimation. In this case, using hierarchical groups becomes more important for refining the advantage estimate and stabilizing policy optimization. By contrast, Qwen2.5-7B --- often solves tasks with fewer but more accurate steps, leading to smaller advantage-estimit
bias. Consequently, the additional benefit from hierarchical grouping more modest (though still cost-effective). Overall, these results suggest that HGPO is particularly well-suited to challenging long-horizon horizon agentic tasks where rollouts are lengthy, and advantage es- timates substantially biased.
HGPO consistently achieves superior performance with different K. We also observe that in- creasing K improves the performance of both GiGPO and HGPO. As K grows, the memory mod- ule can retain richer historical context, allowing the policy to make better use of past information when selecting actions. HGPO benefits from this larger memory well, demonstrating that it scales effectively with the memory size.
5.3 FURTHER ANALYSIS
Distribution hierarchical group sizes. Figure presents the distribution of hierarchical group sizes on ALFWorld and WebShop with D-wen..5-1.5--Instte at the 160th epoch. "0/1/2/3/4- Context" denotes steps that share 0/1/2/3/4 identical historical contexts. We find that 0-context groups have larger fraction of large groups than 1-context and 2-context groups, since they ignore history. As K increases, the mass shifts away from large groups toward smaller ones, and small groups become more common. This pattern suggests that Oracle steps sharing the same histori-
8

---

Published as a conference paper at ICLR 2026
Table 2: Step utilization ratio on ALFWorld and WebShop using Dwen2.5-1.5B-Instmee "Con- text" is abbreviated as "C", indicating different hierarchical groups.
Table 3: Time cost (s) and Peak memory (MB)
for hashing lookups at varying training epochs.
Metric
Epoch
40
120
160
Common
297.8
288.6
284.7
282.5
Dataset
K[
0-C
1-C
2-C
3-C
4-C
Time
GRPO
0.048
0.027
0.018
0.013
2
0.97
0.75
0,52
GiGPO
0.126
0.080
0.055
0.034
ALFWorld
4
0.98
0.77
0.54
0.34
0.19
HGPO
0.693
0.601
0,456
0.246
2
0.92
0.64
0.44
Mom.
GiGPO
0.1035
0.0967
0,0605
0.0391
WebShup
4
0.90
0.59
0.40
0.21
0.09
HGPO
0.1393
0.1229
0.1078
0.0406
cal context typically form small groups, which can increase the variance of advantage estimation. Additional results are provided in Appendix D.3
Step utilization ratio. rrepot[rrporrts the average proportion of steps allocated different context groups per rollout in ALFWorld and WebShop using Qwen2.5-1.55 1.55..--.1BB The results show that nearly all steps fall into O-context groups, except for a small fraction corresponding to unique states (appearing only once in group). As the number of historical contexts increases, the utilization ratio steadily decreases, since fewer steps can be aggregated into higher-level groups. This finding highlights the challenge posed by the scarcity of Oracle steps.
The computational budget analysis of HGPO. We show a detailed computational budget anal- ysis of HGPO. Specifically, HGPO shares the same core architecture as GRPO and GiGPO. The common computational components include multi-turn rollouts, computation of old and reference probabilities, and clipped policy updates. All methods are critic-free and operate with single actor LLM,, resulting in identical GPU memory usage and LLM rollout costs. The primary addi- tion of HGPO lies in advantage estimation. To evaluate its computational cost, we measured the per-iteration training time using Qwen2.5-1.55 -.5..trrBtI on ALFWorid. From Table B] we can sum- marize three key observations. (i) As the number of training epochs increases, both the time cost and peak memory usage consistently decrease, since the rollout steps become fewer when the agent learns to accomplish the tasks with fewer steps. (ii) HGPO introduces an average additional time cost of approximately 0.425 and 0.472 s compared with GRPO and GiGPO, respectively, which corresponds to less than 0.001 % of the total execution time. These results demonstrate that HGPO maintains computational efficiency comparable to that of GRPO and GiGPO. (iii) HGPO only causes a slight increase in the peak memory usage due to the additional hashing lookups. Overall, HGPO preserves the high computational and memory efficiency of GRPO and GiGPO.
5.4 PARAMETER ANALYSIS
Here, we study the effect of different values of in Eq. (7). Notably, controls how sharp the weight distribution is, i.c., a larger a puts more weight high-level groups. The experimental results are shown in Table g We can see that when K = 2, HGPO with comparatively better performance on both ALF- World and WebShop, while when K = 4, the performance of HGPO with a = 0 decreases. In contrast, the performance of HGPO with a and = 2 both increases when K increases from to 4. These observations indicate that as K increases, up-weighting the advantage es- timate from higher-level groups improves the performance. This is because, as K increases, higeer-level groups can provide more accurate advantage estimation. On the other hand, HGPO with 2 slightly drops the performance on both ALFWorld and WebShop, compared with that with a = 1. This is because, although the bias of advantage estimation in higher-level groups is relatively low, the variance could be high due the small sample size in the group. Hence,
Table 4: Parameter analysis on the effects of dif- ferent values of a on ALFWorld and WebShop using Qwen2.5-125 1-55--IIsssuuu Instruct.
0 achieves the
ALFWorld
WebShop
K
In-Suc.
Out-Suc.
Task Sco.
Task Sue.
94.33+1.08
90.03.2.58
88 28.420..2
73.95
92.12,
89.58
87. 87.96,2.48
73.82.306
92.77+1.08
90.16
85. 85.56,286
71.54.4.0
1
94.85
92. 12
90.6441.08
78.12.2.06
90.36.096
86.91
88, 22.29.2..
72.00.3m
93.55
86.78
88. 28,222
75.39

---

Published as a conference paper at ICLR 2026
cessively emphasizing the advantage estimation in higher-level groups is not always optimal, and bias-variance trade-off exists in hierarchical groups. Overall, we use a = 1 as the default. In the future, it is also interesting to develop a better adaptive weighting scheme based the uncertainty of advantage estimation in hierarchical groups.
5.5 ABLATION STUDY
In this section, we conduct an ablation study to evaluate the effectiveness of each component in HGPO. As shown in Table 5 "w/o G de- notes the setting where hierarchical grouping removed, and only the original step-level group G used to compute relative advantages for policy optimization. This configuration results in the remarkable performance degradation on ALFWorld when K 2/4, i.c., about 2.8% of in-distribution success rates and 5.8% of out- of-distribution success rates. This observation directly validates that directly using the current state of steps for grouping could lead to bi- ased advantage estimation and degrade the pol- icy optimization, and our proposed hierarchy-of- groups structures effectively refine the advantage estimation. A similar result can also be observed on WebShop, except for K = 2, which results in a little performance improvement (when a = 1). Second, "w/o Ada. wx," refers replacing adaptive weighting with uniform weights, i.e., = 0 in Eq. 7p. We can see that the performance slightly increases on both ALFWorld and WebShop when K = and decreases when K 4. This is because when K 2, the relative advantage bias of lower-level hierarchical groups is lower than when K = 4, and thus the average weighting can also be beneficial for advantage estimation. Once K increases, the advantage bias of lower-level hierar- chical groups could be increased and degrade the policy optimization. Hence, the adaptive weighting mechanism is important for advantage estimation, and requires complex hyperparameter tuning and remains scalable across different lengths of historical contexts. Third, "w Eq. O indicates using the additional trajectory-level advantage in Eq. (I) for the final advantage estimation in Eq. Z. We can see that the performance of almost all decreases on ALFWorld and WebShop, except for k on WebShop. This may be because the trajectory-leve advantage generally highly biased and low-varied, which could not provide effective information for policy optimization in most cases. Besides, we also conduct experiments that only use Oracle steps for policy optimization, but failed. Overall, we validate the effectiveness of each component in HGPO.
Table 5: Ablation study ALFWorld and Web- Shop using Qwen2.5-1.5B. Instructt
ALFWorld
WebShop
Ablation
K
In-Suc.
Out-Sue.
Task Sco. Task Suc.
2 92. 22.77 10.16 85.5622.9 71. 71.54_100
HGPO
4
94.852a92 92.122163 90.64g1.s 78. 122206
w/o Gff
2 90.88.1..
85.15
87.89.0.6 73.24.1.4
4
91.14.107
88.80.321
74.73 88.83
2 1
90.03,25s
88.40
23.95
w/o Ada. ut
4
92. 22.40 89.582100
87.96
73.8223.26
w Eq. 11
2
188.21.195 84.76,2ns
89.3641.2
76.82_2.5
4 191.27.111 90.55,a29 87.63.0.0 73.50a1.3
6 CONCLUSION
In this paper, propose HGPO, a novel group-based reinforcement learning algorithm designed to mitigate historical context inconsistency in long-horizon agentic tasks. Specifically, HGPO in- troduces context-aware hierarchical grouping and adaptive weighting advantage estimation, which enables a better advantage estimate for policy optimization. Empirical results on two complex envi- ronments, ALFWorld and WebShop, show that HGPO substantially outperforms both prompt-based agents and prior RL approaches. In the future, an interesting direction is to explore the structure of hierarchy-of-groupp on advanced agents with summarized memory. HGPO directly divides the hierarchical structure based on the raw, divisible historical contexts according to the historical steps. Currently, many advanced agents tend to summarize the historical contexts into the memory module. In this case, the straightforward division of hierarchical groups is intractable, and thus it is neces- sary to explore other ways for hierarchical grouping, e.g., the embedding similarity of the memory. Besides, it is also interesting to explore totally adaptive weighting scheme for advantage aggre- gation from hierarchical groups by considering the uncertainty of the advantage estimate in each hierarchical group.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. ArXiv preprint (xXi::2333008775 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learn- ing from human feedback in LLMs. In Proceedings of the Annual Meeting of the Association for Computational Linguisties. pp. 12248- 12267, 2024.
G Brockman. OpenAl Gym. ArXiv preprint
2016.
Kevin Chen, Marco Cusumano- Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive Ilm agents. ArXiv preprint arXiv:2502..0 060000 2025a.
Wentse Chen, Jiayu Chen, Hao Zhu, and Jeff Schneider. Context-lite multi-turn reinforeement learning for LLM agents. In Proceedings of the International Conference on Machine Learning Workshop, 2025b. httlsttss://// /tttp::://penteeeeeewe net wtt.:ee///oooummmd66EE2...
Zdw
Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang, Yang Zhang, and Chuchu Fan. R1 R1-code-interpreter: Training llms to reason with code via supervised and reinforcement learning. ArXiv preprint arXiv:2505. 2025c.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. ArXiv preprint arXiv:2504. 194.:2500...4113.
2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. ArXiv preprint 250. 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, The entropy mechanism of reinforcement learning for reasoning language models. ArXiv preprint arXiv:2505.22617. 2025.
Lang Feng, Weihao Tan, Zhiyi Lyu, Longtao Zheng, Haiyang Xu, Ming Yan, Fei Huang, and Bo An. Towards efficient online tuning of VLM agents via counterfactual soft reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2025a.
Lang Feng. Zhenghai Xue, Tingcong Liu, and Bo An. Group-in group policy optimization for Ilm agent training. ArXiv preprint arXiv:2505. 10978, 2025b.
-ngguuu
Hiroki Furuta, Kuang-Hus Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane foundation models.
Gu, and Izzeddin Gur. Multimodal web navigation with In Proceedings of the Twelfth International Conference on Learning Representations, 2024. URL
htttp:://operrewe http:/ewwwweeeeeotreeeeeeeeee Fowummiddeeeanneeeee
net
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code Ilms execution feedback with reinforcement learning. ArXiv preprint arXiv:2410.0000 2025.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and
Yu Su. Navigating the digital world humans do: Universal visual grounding for GUI agents. In Proceedings of the International Conference on Learning Representations, 2025. URL hetos:
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-R1: Incentivizing reasoning capability in LLMss via reinforcement learning. ArXiv preprint arXiv:2501. 12948, 2025.

---

Published as a conference paper at ICLR 2026
Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding. and pro- gram synthesis. In Proceedings of the International Conference Learning Representationss 2024.
net/ f rumve
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model planning with world model. ArXiv preprint arXiv:2305. 14992, 2023.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. CogAgent: A visual language model for GUT agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290, 2024.
Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of GUI agent: A prelim- inary case study with claude 3.5 computer use. ArXiv preprint 1rXiv:2411.10322 2024.
Arnay Kumar Jain, Gonzalo Gonzalez- Purrriez, Wayne Chen, Alexander M Rush, Wenting Zhao, and Sanjiban Choudhury. Multi-turn code generation through single-step rewards. ArXiv preprint arXiv:2502.20380. 2025.
Nan Jiang, Xiaopeng Li, Shiqi Wang. Qiang Zhou, Soneya B Hossain, Baishakhi Ray. Varun Kumar, Xiaofei Ma and Anoop Deoras. Ledex: Training Ilms to better self-debug and explain code. Proceedings of the Advances in Neural Information Processing Systems, 37:35517-35543, 2024.
Bowen Jin, Hansi Zeng. Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. ArXiv preprint arXiv:2503.00 09556,,,,16,, 2025a.
Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang. Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability in large language models. ArXiv preprint 2rXiv:2411.13504, 2024.
Yiyang Jin, Kunzhao Xu, Hang Li, Xueting Han, Yanmin Zhou, Cheng Li, and Jing Bai. Reveal: Self-evolving code agents via iterative ceneration-veriiiicaton. ArXiv preprint arXiv:2506. 11442, 2025b.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! In ICLR 2019 Workshop, 2019.
Philippe Laban, Hiroaki Hayashi. Yingbo Zhou, and Jennifer Neville. Lims get lost in multi-turn conversation. ArXiv preprint arXiv:2505.06120, 2025.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking LLMs for embodied decision making. Proceedings of the Advances in Neural Information Processing 2024.
Systems, 37:100428-10000
Hongyu Lin, Yuchen Li, Haoran Luo, Kaichun Yao, Libo Zhang, Mingjie Xing. and Yanjun Wu. Os-rl: Agentic operating system kernel tuning with reinforcement learning, 2025. htts::
//arxiv.org hwtpgg/abs/250000 2508.. 12551
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeck-V3 technical report. ArXiv preprint arXiv:24/2. 19437, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. ArXiv preprint arXiv:2503.2078: 2025.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing efficient action prediction of gui agents by reinforcement learning- ArXiv preprint arXiv:2503.21620, 2025.
12

---

Published as a conference paper at ICLR 2026
Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai, Tarun Venkat, Shang Zhu, Ben Athiwaratkun, Manan Roongta, Ce Zhang, Li Erran Li, Raluca Ada Popa. Koushik Sen, and Ion Stoica. Deepswe: Training a state-of-the-ar coding agent from scratch by scaling rl, 2025a. Notion Blog.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist r1 r1-style vision-language action model for gui agents. ArXiv preprint arXiv:2504. 10458, 2025b.
Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang. Siyun Zhao, Dongsheng Li, Luna K. Qiu, and Yuqing Yang. Agent lightning: Train any ai agents with reinforcement learning, 2025c. URL
bittp:://arxie
03680
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Jumnan Li. Mcp-universe: Benchmarking large lan- guage models with real-world model context protocol servers. ArXiv preprint arXiv:2508. I4704, 2025d.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540::52953333 2015.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing, pp. 1-1,,0015
OpenAl. Introducing OpenAl o1, 2024. URL htLptttss///ppaai.ccmm com/o1
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744, 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. wevintage-weighted regression: Simple and scalable off-policy reinforcement learning. ArXiv preprint arXiv:19/0.00117,, 2019.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent Q: Advanced reasoning and learning for autonomous agents. ArXiv preprint ar arXiv:2488.077 199, 2024.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Tür, Gokhan Tur, and Heng Ji. ToolRL: Reward is all tool learning needs. ArXiv preprint arXiv:2504.1395. 13958,, 2025.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. ArXiv preprint 2511v:2501.12326 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Proceed- ings of the Advances in Neural Information Processing Systems, 36, 2024.
Christopher Rawles, Alice Li. Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. In Proceedings of the Advances in Neural Information Processing Systems, volume 36, 2024.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Proceedingy of the Advances in Neural Information Processing Systems, 36:68639-6688666 68551, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint arXiv: (00iv:/777.064444 2017.
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeckMath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint arXtv:2402.03300, 2024.
Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang- Wei Hong. Zhenfang Chen, Wei Lu, Gregory Wornell, Subhro Das, David Cox, and Chuang Gan. Satori: Reinforcement learning with chain- of-action-thought enhances IIm reasoning via autoregressive search, 2025. URL https://
arxiv.orge abs 225520025000 02508
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Proceedings of the Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Proceedings of the International Conference on Learning Representations, 2021. URL https:
Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In
//openzeview net http://oormmmmd-000000ooocom
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for I1m agents. ArXiv preprint urXiv:2403.02502, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Proceedings of the Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang. and Yan Zhang. ZeroSearch: Incentivize the search capability of ilms without searching. ArXiv preprint arXiv:2505. 25000048888 2025.
Richard S Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Gang Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, al. Cradle: Empowering foundation agents towards general computer control. In Proceedings of the Advances in Neural Information Processing Systems Workshop, 2024.
Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, al. Cradle: Empowering foundation agents towards general computer control. In Proceedings of the International Conference on Machine Learning, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: A family of highly capable multimodal models. ArXiv preprint arXiv:2312.118000 2023.
OpenManus-PP RL Team. Openmanus-rl: Open platform for generalist I1m reasoning agents with rl optimization, 2025. URL URL httttttt///ttttiie wwt. wwwpppeaaao/peeaiaeaaaeee
RI
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856.
Hongru Wang, Cheng Qian, Wanjun Zhong. Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. OTC: Optimal tool calls via reinforcement learning. ArXiv preprint PrXiv:2504.14870 2025a.
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Proceedings of the Advances in Neural Information Processing Systems, 37:2686-2716 2024b.
Renxi Wang. Rifo Ahmad Genadi, Bilal El Bouardi, Yongxin Wang, Fajri Koto, Zhengzhong Liu, Timothy Baldwin, and Haonan Li. Agentfly: Extensible and scalable reinforcement learning for Im agents. ArXiv preprint arXiv:2507. /4897, 2025b.
14

---

Published as a conference paper at ICLR 2026
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning optimization for large- scale learning: An efficient and user-friendly scaling library. arXiv preprint (rXiv2250.00622, 2025c.
Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. Rlcoder: Reinforcement learning for repository-level code completion, 2024c. URL https:
//
Zihan Wang, Kangrui Wang. Qineng Wang. Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, et al. RAGEN: Understanding self-evolution in LLM agents via multi-ture turn reinforcement learning. ArXiv preprint arXiv:2504.20073, 2025d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceed- ings of the Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837, 2022.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. SWE-RL: Advancing Ilm reasoning reinforcement learning open software evolution. ArXiv preprint arXiv:2502.18449 2025a.
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing IIm reasoning rein- forcement learning open software evolution. ArXiv preprint 250.v::2002.84444 2025b.
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, et al. Webagent-rl: Training web agents via end-to-end multi-turn rein- forcement learning. ArXiv preprini arXiv:2505. 16421, 2025c.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks real computer environments. In Proceedings of the Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. ArXiv preprint arXiv:2412.15 15 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real- world web interaction with grounded language agents. Proceedings of the Advances in Neural Information Processing Systems, 35:20744-2074 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Represenaatons,, 2023. URLhttps:
id WE vluYUL-XX
Chengyue Yu, Siyuan Lu, Chenyi Zhuang, Dong Wang, Qintong Wu, Zongyue Li, Runsheng Gan, Chunfeng Wang, Siqi Hou, Gaochi Huang, et al. Aworld: Orchestrating the training recipe for agentic ai. ArXiv preprint arXiv:2508.
2025a.
Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshaping long-context 11m with multi-conv rl-based memory agent. ArXiv preprint arXiv:2507.02255: 2025b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source L.M reinforcement learning system at scale. ArXiv preprint arXiv:2503. 10.5503..4476 14476, 2025c.
Chaoyun Zhang, Liqun Li. Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. UFO: A Ul-focused agent for windows OS interaction. ArXiv preprint a arX1v:2402.07933, 2024a.
15

---

Published as a conference paper at ICLR 2026
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang. Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji. Michael Littman, Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for Ilms: A survey, 2025. httpsts:::////
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 13643-13658, 2024b.
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Proceedings of the Findings of the Association for Computational Linguistics, pp. 3132-31499 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. GPT-4V (ision) is a generalist web agent, if grounded. ArXiv preprint arXiv:2401.01614 2024.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. ArXiv preprint arXiv:2507.18071, 2025.
Andy Zhou, Kai Yan, Michal Shlapentokh-Root Shaapentokh-Rothmmnn Haohan Wang, and fu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the International Conference on Machine Learning, pp. 62138 228-62160, 2024a.
Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. ArCHer: Training lan- guage model agents via hierarchical multi-turn In Proceedings of the International Conference on Machine Learning, pp. 62178 -72209. PMLR, 2024b.
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn 1lm agents on collaborative reasoning tasks, 2025a. URL
https://arxis www//bs/////ooooooo/ot////o.aarr
Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Mem1: Learning to synergize memory and reasoning for efficient long-horizon agents. ArXiv preprint arXiv:2506.1588 2025b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv preprint arXiv: 08553, 2019.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzuan Wahid, et al. RT-2: language--ugueeeeo models transfer web knowl- edge to robotic control. In Proceedings of the Conference on Robot Learning, pp. 2165-2183. PMLR, 2023.
16

---

Published as a conference paper at ICLR 2026
A ALGORITHM
Algorithm The pseudo-code of HGPO
1: Require: Initial policy n task distribution p(X), discount factor weighting w, clipping parameter e, KL penalty 3. group size N, the length of historical context K, parameter
2: for each training iteration do
3: Update the old policy model: fold +- 0
4: // Step rollout phase
5: Sample task p(X) and initialize N identical environments
6: for to do
7: Sample actions {a Toom (:1 ( x))
8:
Execute actions, observe rewards ( and next state [+
9:
end for
I/ Grouping phase
11:
Context-awes aware hierarchical grouping by Eq.
12:
II Advantage computation phase
13:
Compute multiple advantages within each group by Eq. 7
14:
I/ Policy update phase
Update policy by maximizing objective (HGro(0)
16:
end for
B MORE DETAILS AND PROOF FOR THEOREM
Table Overall comparison three different advantage estimators.
Type
Advantage estimation
Granularity
Variance
Trajector-leeeel
(r)) R(r)) 1 G-IEEe. R(ra) /ea.
Coarse-grained Fine-grained
By
Vr
Step-level
( [R(a) 1/(Ga. Esco, R(8))) loo,,
bo
Hierarchy-of of-ups
A"()(") ) > nn A (x())
Fine grained
> wr 4
Let A denote the advantage estimator for the k-th hierarchical group. Let by and Us denote the bias and variance of the estimated advantage A within the k-th group G The definition of bias is by Bias[A] unknown true advantage. We make the following conditions:
Here, we provide more details of Proposition
A" where A is the
(1) Bias satisfies:
Br 2 bo 2 (b1, b2. ... bb- 2 dx 2 0, by Bias(A
(2) Variance satisfies:
up 22, (U. S UK S VT, Us [ar[A
where Br and V denote the bias and variance trajectory-level advantage estimation, and by and vo represent those of step-level estimation. We now justify the assumptions. First, the number of trajectories in a group generally smaller (set to 8 our experiments) than the step-level group size, which leads to higher bias and variance in trajectory- estimation. Second, as K increases, the group size of G decreases, which can result in higher variance.
Bias and variance of HGPO. Recall the advantage aggregation in Eq. and Eq.
A
Wkk A
with group weights we 2 satisfying Σ = Wk = 1. We first analyze the bias. Define by A
][ A" and Bias] A [[X] A", where A denotes the target (unbiased) advantage. Then
K E
K
K
Bias #1
Σ
wk A
wk E A²)
Σ wyb
k=
k=
A= 0
17

---

Published as a conference paper at ICLR 2026
Since bu 2 b1 by and k= W/ 1, it follows that
K bK Σ k=0 wabk ~= wpbr Bias[A"] - Σx wybh bo S Br.
K
Hence, HGPO trades off the bias between the step-level estimator (k 0) and the oracle estimator (k: K). Correspondingly, for the variance, assume Cov(A AW) 0 for to + K. Let
en - Var[A then
K Σ we A/
Var[A") Var
Σ w
Var [A
VarlAA
Σ w Uk-
wLek-
Since 00 01 S ... S UK. then
# S VarlA
S UK Σ w
S UK-
Moreover, by Cauchy-Schwarz
K
w 1
K
+
k=0
hence we can obtain:
K+1 Var[A [1 ukk
00
In summary, HGPO interpolates between the step-level (k: 0) and oracle (k- K) estimators in bias and variance, thereby achieving a better trade-off. is also interesting to explore tighter bounds on the bias and variance in HGPO.
C EXPERIMENT DETAILS
C.I COMPARING METHODS
GPT-4o: A closed-source, large-scale LLM used as baseline for multi-turn agentic tasks (Achiam et al. 2023).
Gemini-2.55PP Another closed-source LLM, comparable in scale and capability GPT- 4o (Team et al.) 2023).
ReAct: A prompting-based agent that integrates reasoning and acting in an interleaved chain-of-thought framework (Yao et al. 2023).
Reflexion: A prompting agent that incorporates self-reflection and iterative improvement over generated outputs (Shinn al. 2024).
PPO: Proximal Policy Optimization, a classic RL algorithm for policy learning Schullan
et al.] 2017).
RLOO: Reinforcement Learning with Offline Observations, group-based RL approach
that estimates advantages without value networks (Kool al. 2019 Ahmadian et al. 2024).
GRPO: Group-based RL with trajectory- advantage estimation, designed to scale RL to multi-step tasks (Shao et al. (2024).
GiGPO:
Grouped Incremental GPO, a prior hierarchical RL method that performs group-
wise advantage estimation for LLM-based agents (Feng et al. (025b).
C.2 ENVIRONMENT DETAILS
In each episode, the agent receives text goal and must accomplish through multi-tume interaction with the environment. It includes 4,639 task instances across six categories of common household activities: Pick Place (Pick), Examine in Light (Look), Clean Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). WebShop is complex, web-based interactive environment designed to test the LLM agents in realistic online shopping scenarios. To complete the task, the agent must interact with simulated HTML- shopping website to search for, navigate to, and ultimately purchase a suitable item. contains over 1.1 million products and 12k user instructions, providing rich and diverse action space.
18

---

Published as a conference paper at ICLR 2026
C.3 DETAILS OF TRAINING
Notably, we implement GiGPO and HGPO based on the new version of Verl-agent and report the performance in T Meanwhile, we report them based on the old version of Verl-agent, as shown in Table Generally, we use the same training settings in (Feng et al.) 2025b) for fair comparison.
Hyperparameters for ALFWorld. All methods are configured with identical hyperparameters: the maximum prompt length 2048 (4096) tokens, and the maximum response length is 512 tokens. Each episode allows up 50 environment steps. The learning rate is set to 1e-6 for the actor and le-5 for the critic (used only in PPO). We adopt a rule- cule-based reward, assigning a reward of 10 for success and 0 for failure. To handle invalid actions generated by the agent, we apply reward penalty of 1. For all group-based RL methods, we use a group size of 8 and a training size, resulting in a 128 environments. In contrast, PPO uses 128 separate environments for rollouts, The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. The mini-batch size is 256, and the KL-divergence loss coefficient set to 0.01. The discount factor y set to 0.95.
total of 16 x
Prompt Template for ALFWorld
You expert agent operating in the ALFRED embodied Environment. Your task is [task description). J. Prior to this step, you have already taken (step_count) count) step(s). Be- low are the most recent [history_lenge lenghhh observations and the corresponding actions you took: jistion_istory].. You are now at step [current_stn ssepp] and your current observation is: [current_ )obervaioossr. Your admissible actions of the current situation are: [[admissi- blc actions]]]
to:
Now it's your turn to take an action. You should first reason step-hy-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you've 've finished your reasoming, you should choose admissible action for current step and present it within <action> </action> tags.
Figure 5: The prompt template of ALFWorld agents.
Prompt Template for WebShop
You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: (task description). Prior to this step, you have already taken sste__come step(s). Below are the most recent phistoryJlengtl actions you took: [action_ history). You are now step [current observation /current_obseeee ooservvtion]. J. Your admissible actions for the current situation are: [availlable [evailable_actions]] actions]].
observations and the corresponding
your current
Now it's your turn to take one action for the current step. You should first reason step-by- step about the current situation, then think carefully which admissible action hest advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you't finished your reasoning. you should choose an admissible action for current step and present within <action>> </action> tags.
Figure 6: The prompt template used for WebShop agents.
Hyperparameterss for WebShop. All methods are configured with identical hyperparameters: the maximum prompt length is 4096 tokens, and the maximum response length is 512 tokens. Each episode is limited 30 environment steps. The learning rate is le-6 for the actor and le-5 for the critic (used only in PPO). We adopt rule-based reward, assigning reward of 10 for success and for failure. Invalid actions are penalized with a reward of -0.1. As with ALFWorld, all group- based RL methods use group size of 8 and sample 16 groups per rollout, totaling 16 x 8 128 environments. PPO, on the other hand, uses 128 distinct environments for rollouts. The rollout temperature is set to 1.0, while the validation temperature is set to 0.4. The mini-batch size is 64, and the KL-divergence loss coefficient is set to 0.01. The discount factor y is set to 0.95.
Computing Details. Experiments using Owen2.5-1.5B-IIsttue are conducted two NVIDIA H100 GPUs, while those using Owen2.5--BBIInstruu are trained on four NVIDIA H100 GPUs Each experiment is trained for total of 160 training iterations. The validation data size is 512.
19

---

Published as a conference paper at ICLR 2026
C.4 TRAINING METRICS
Mean Advantages: This metric shows how much better the chosen actions are compared to the average action. A positive and stable value means the agent usually selects better actions, while large fluctuations suggest unstable training.
Policy Gradient Loss: This loss is the main signal for updating the policy. smooth and gradually decreasing value indicates stable learning. If the loss becomes too large or changes sharply, it means the updates are too aggressive and may harm training stability.
KL Divergence: KL loss measures how different the new policy is from the old one. acts constraint to prevent the policy from changing too quickly. A moderate KL value means the agent is learning steadily, while a very high value can cause divergence and a very low value may slow down learning.
Policy Gradient Clip Fraction: This metric shows the proportion of gradients that are clipped during optimization. Gradient clipping prevents extreme updates. A moderate fraction suggests stable training, but if the fraction is too high, it means many updates are unstable and are being restricted.
The mean reward reflects the average return the agent receives per episode. It is direct measure of progress: higher rewards indicate better performance. If the mean reward increases smoothly, it shows effective learning, while sudden drops suggest insta- bility.
Mean Reward:
Episode Success Rate: This metric measures the percentage of episodes in which the agent completes the task. an intuitive indicator of how well the agent achieves its goal. A rising success rate shows that the agent is improving and that training is effective.
C.5 PROMPTS
The prompts we use for LLM agents are presented Figure and Figure These prompt templates constructed using Python-style string formatting, where placeholders enclosed in curly braces (()) represent semantic slots. These placeholders, such as faas_descrrption (current observation), are dynamically populated at runtime via Python's format () function. To enrich the agent's context, we use historical information and set the history length to 2.
(step_count, , and
The
block instructs the agent to perform step-by-step reasoning, thereby promot- ing ofuin-of-thouglhe style deliberation explicitly. The faction>..<<acttoon block is used to indicate the final action decision clearly.
D MORE EXPERIMENTAL RESULTS
D.1 THE PERFORMANCE OF USING THE OLD VERSION OF VERL-AGENT
Note that the results reported in the original manuscript (Table 7) were obtained using the earlier version of Verl-agent (paper mance based on the latest version of Verl-agent in Table [ Importantly, the updates to Verl-agent (veRL) involve changes to the framework implementation only and do not modify any underlying algorithms. The experimental results remain consistent and continue to demonstrate the superiority of HGPO.
(paper_versione)) version). Following subsequent updates to veRL, we report the perfor-
D.2 TRAINING DYNAMICS
We show training dynamies of HGPO (Red), GiGPO (Yellow), and GRPO (Purple) on WehShop using Swen2.5-1.5BB-hm Instruct as shown in Figure Figures illustrate the training dynamics of GRPO, GiGPO, and HGPO across six metrics: mean advantages, policy gradient loss, KL loss, policy gradient clip fraction, mean reward, and episode success rate. Detailed definitions of these metrics are provided in Appendix C.4 Overall, our method achieves more stable and efficient policy optimization. In particular, for the policy gradient clip fraction, HGPO (red curve) maintains moderate level, suggesting stable training, whereas GiGPO and GRPO display higher fractions, reflecting instability and constraint. For the KI. loss GRPO's curve is low, indicating slow
20

---

Published as a conference paper at ICLR 2026
Table Performance comparison on ALFWorld and WebShop. Note that we used the old version of Verl-agent and updated the performance of using the new version of Verl-agent (with veRL. updating) in T For ALFWorld, we report the overall success rate (t) for both in-distribution (In-Success) and ont-ef-distribution tasks (Out-Success). For WebShop (15 steps), we report the average task score (1) and the average task success rate (t). Most results are averaged over random seeds during testing. The best results are highlighted in bold.
ALFWorld
WebShop Task Success Rates
Model
Type
Method
In-Ceccess
Out- Success
Task Scores
Q2.5-BB Q2.5-1 15
RL Training
GiGPO (2)
8,.42,1.5
80.72.1
84.52.0sx
69.79
RL Training
HGPO (K=2)
89.58
80.73,233
87753
72.66.177 7M
RL Training
GiGPO (K)
85. 15
80,982045
88 52
74. 24..88
RL Training
HGPO (K=4)
92.45
89.06_2.4
88,9020.0
75.91
RL Training
GiGPO (K=2)
89 84g2.0
82.81
86. 26.234114
75. 13
RL Training
HGPO (K=2)
91.15.115
84.89
88,93
76.43 47447
RL Training
GiGPO (K=4)
90.88
87.764n
87.25ar
76. 18
RL Training
HGPO (K==)
94.79...
93.22
87.88 10.41
77.21
KL Loss
Policy Graditnt Cltp
Policy Gredient Loss
Homo
caro
Losa Vilue
0.80
100
330
120
160
140
350
20
440
1000
330
120
Moan Advantage
Meon Reward
Spisode Success Rate
Swoces Bake
65
60
300
1.0
140
160
20
100
120
140
160
180
100
Figure 7: Training dynamics of HGPO (Red), GiGPO (Yellow), and GRPO (Purple) on ALFWorld -.ettt-t..B--nstrueet.. The details of these metrics are shown in Appendix D.2
using Qwen2.5-1.558 Instruct.
KL Loss
Polley Gradient Clip
Policy Gredient Loss
caro
Losa
0.00
1000
330
120
160
60
300
140
20
33
640
32
100
330
150
Mean Adventage
Mean
Eploode Success Rate
Rowurd Vilue
Socceaa Bena
0.00
3.00
-0.15
20
60
80
300
1.0
340
360
10
20
300
120
140
380
300
120
3.00
380
Figure 8: Training dynamics of HGPO (Red), GiGPO (Yellow), and GRPO (Blue) on WebShop using 1.em2.5-1.5B-Instrrrce Best viewed in color.
learning, while GiGPO's curve is relatively high, reflecting overly aggressive learning process. By contrast, HGPO achieves a balanced trajectory, demonstrating steady and stable policy learning. We have also made the training and evaluation Weights & Biases (W&) ()B) logs publiely available.
21

---

Published as a conference paper at ICLR 2026
2-Content
43
7
Group Stone
Group Stoee
Group Stoee
Contect Aftworid
6-Contest Webchop
I-Coatest Webshop
60
Group Staea
Group Stees
Group Stees
2:Contest Group Webshop
0.3
Group
Figure 9: The distributions of hierarchical group sizes (K 4) on ALFWorld and WebShop using Qwenn.55-1.B-Inntruut
D.3 THE DISTRIBUTION
We report the distributions of hierarchical group sizes (K 4) on ALFWorld and WebShop using Qwen2.5-1.5B-Instt Instruttt shown in Table
USE OF LLMS
We used LLMs exclusively as writing assistants to refine language. In particular, their use was restricted to grammar correction, style improvement, and phrasing adjustments for clarity and con- ciseness.
22


Reviews:
[{"review_id": "qxUHXDFOLZ", "reviewer": "Reviewer_hUcL", "summary": "To improve training efficiency in long-horizon agentic tasks, existing reinforcement learning (RL) methods typically adopt step-level policy optimization. However, when estimating the advantages of identical step groups, inconsistencies in historical context can lead to biased advantage estimation. A straightforward solution is to use oracle steps, which not only share identical current steps but also identical histories. Nevertheless, oracle steps are rare in practice, leading to the utilization of only a small fraction of the data and consequently causing high-variance advantage estimation. To balance the trade-off between bias and variance, this paper proposes Hierarchy-of-Groups Policy Optimization (HGPO), which aggregates advantages computed under different levels of historical context. This hierarchical aggregation enables HGPO to effectively balance bias and variance, achieving improved training stability and efficiency. Empirical results on ALFWorld and WebShop demonstrate the effectiveness of the proposed method.", "questions": "I did few research on LLMs but lots on RL. So, I have the following questions:\n\n1. How is the similarity between states (or histories) measured, and what specific criterion is used to group them?\n\n2. How is the memory module maintained or updated throughout training?\n\n3. In Eq. (8), is the memory utilized as context when computing the importance sampling ratio? And, which policy is updated, $\\pi(a|x,s,h)$ or $\\pi(a|x,s)$? In my view, $\\pi(a|x,s,h)$ is preferred, but in the paper, it seems to be $\\pi(a|x,s)$.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a well-motivated study that clearly illustrates the limitations of trajectory-level, traditional step-level, and oracle-step advantage computations in Figure 1 and 2, thereby making the proposed method appear both natural and intuitively justified.\n\n2. Both the high-level idea and implementation of HGPO are simple yet well-grounded, making it an effective solution to address the context-inconsistency problem.\n\n3. HGPO introduces almost no additional computational overhead compared to other group-based RL methods such as GRPO and GSPO. So, it effectively alleviates context inconsistency and maintains high efficiency.\n\n4. The empirical results demonstrate the effectiveness of HGPO. Especially when K=4, we can observe a clear improvement over naive step-level methods. \n\n5. The paper combines theoretical justification with extensive empirical evaluation, resulting in a well-rounded and complete study.", "weaknesses": "1. The context inconsistency problem is only partially addressed in this paper rather than being fundamentally resolved. As the value of \nK increases, the number of grouped samples noticeably decreases. This indicates that for longer-horizon tasks, HGPO becomes less effective, since the higher-level history groups become increasingly sparse under HGPO’s grouping mechanism.\n\n2. Therefore, HGPO is effective only when K is relatively small. In the experiments, K is set to 2 and 4, corresponding to 2–4 steps in agentic tasks, which represent relatively short-horizon settings.\n\nHowever, I would not consider the above weaknesses as major limitations, since the authors have made a meaningful step toward addressing long-horizon tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rSo1sFyiD8", "reviewer": "Reviewer_QH1q", "summary": "This paper identifies the \"context inconsistency\" problem in stepwise group-based reinforcement learning for LLM agents , arguing that grouping steps by their current state while ignoring their histories leads to severely biased advantage estimation. The proposed method, Hierarchy-of-Groups Policy Optimization (HGPO), addresses this by assigning each step to multiple hierarchical groups based on k-step historical context consistency, and applying an adaptive weighting scheme that assigns higher weights to advantages computed from more consistent groups. The authors provide a theoretical analysis of the bias-variance trade-off. Experiments on the ALFWorld and WebShop benchmarks demonstrate that HGPO significantly outperforms existing baselines, including the prior SOTA method GiGPO.", "questions": "1. How would the $C_k$-based grouping function if the agent's policy $\\pi(a_t | s_t, h_t)$ used a summarized memory state $h_t$ rather than the raw history?\n\n2. The claim of minimal additional time cost is unsubstantiated; the paper provides no quantitative comparison of wall-clock time and peak memory usage (e.g., for K=4) against the baselines.\n\n3. How should K be chosen relative to a task's average horizon T? How does the method avoid the Oracle group sparsity problem when K must be large for a very long-horizon task?\n\n4. How significant do the authors believe the context inconsistency problem remains in settings with effective process rewards, as opposed to the sparse setting studied?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's primary strength is its elegant and well-motivated mechanism for managing the bias-variance trade-off in advantage estimation. Rather than forcing a binary choice between GiGPO's high-bias/low-variance step-level group and a low-bias/high-variance Oracle group, HGPO proposes a more nuanced solution. By aggregating advantage signals across all K+1 levels of the hierarchy, it provides a principled interpolation that leverages the low-bias signal from high-consistency groups while retaining the low-variance stability from large, low-consistency groups. This mechanism is strongly supported by robust empirical results on recognized benchmarks. The ablation study is particularly convincing, as it clearly isolates the necessity of both the hierarchical grouping and the adaptive weighting components for the method's success.", "weaknesses": "1. Its validity is tightly coupled to the assumption that the agent's \"memory\" is equivalent to its \"raw historical context,\" as implemented in the paper's prompts (Appendix C.5). This raw-history-as-memory paradigm, while common, is limited. The proposed $C_k$-based grouping would not directly generalize to more advanced agents that utilize summarized memory, as the grouping basis would misalign with the agent's true decision-making state. \n\n2. The claim of minimal additional time cost is unsubstantiated. The K+1-level hierarchical hash management is necessarily more computationally expensive than the baseline, and the paper provides no quantitative wall-clock time or memory overhead analysis. \n\n3. The method's scalability with respect to K is unclear. For truly long-horizon tasks (T > 100), a small K may be insufficient, while a large K would suffer from the same sparsity and high-variance issues as Oracle groups. \n\n4. The method's necessity is predicated on a sparse reward setting; its utility would likely be diminished in settings where effective process rewards are available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2NV2r5A4i", "reviewer": "Reviewer_HR8w", "summary": "This paper identifies a critical problem in existing stepwise group-based Reinforcement Learning (RL) methods for LLM agents: \"Context Inconsistency.\", which is particularly unsolved in GiGPO.\n\nThe authors provide a strong empirical diagnosis (Figure 2) demonstrating that existing methods like GiGPO (which only groups by $s_t$) suffer from high bias, while a naive \"Oracle\" solution (using only steps with perfectly identical histories) is unusable due to high variance from data sparsity.\n\nTo solve this bias-variance dilemma, the paper proposes Hierarchy-of-Groups Policy Optimization (HGPO), with two key component:\n1. Context-aware Hierarchical Grouping\n2. Adaptive Weighting Advantage Estimation\n\nExperiments on ALFWorld and WebShop show that HGPO significantly outperforms existing RL baselines, including GRPO and GiGPO, especially on out-of-distribution tasks.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written. Figure 2 of the \"context inconsistency\" problem clearly frames the bias-variance dilemma that existing methods face (GiGPO = high-bias, Oracle = high-variance).\n\n2. HGPO is an intuitive and novel solution. \n\n3. HGPO achieves sota results, significantly outperforming its baselines.\n\n4. The ablations are comprehensive.", "weaknesses": "1. The proof of the bias-variance trade-off in Appendix B relies on the assumption $b_k \\ge b_{k+1}$ (bias decreases as context depth $k$ increases). This assumption is stated but never justified or proven. The authors must provide a formal argument for this assumption or re-frame Proposition 4.1 as a heuristic analysis.\n\n2. The paper's claim to address \"long-horizon\" tasks is not supported by the evidence. Appendix C.3 reveals the maximum episode lengths are T=50 (ALFWorld) and T=15 (WebShop). A 15-step task is not \"long-horizon.\"This is a major overclaim that misrepresents the scope of the paper's validation.\n\n3. The paper makes the qualitative claim of \"minimal additional time cost.\" This is highly questionable. GiGPO hashes one item per step ($s_t$), whereas HGPO (K=4) must construct and hash $K+1=5$ separate context sequences for every step. This is a non-trivial increase in computational overhead. The paper provides no wall-clock time or GPU-hour comparison against its baselines (GiGPO, GRPO) to substantiate its efficiency claim. \n\n4. The \"adaptive weighting\" scheme (Eq. 7) is not truly adaptive; it is a fixed heuristic weighting tuned by a single, manually-set hyperparameter, $\\alpha$. Table 4 simply shows that $\\alpha=1$ (linear weighting) works best empirically. This is parameter tuning, not an adaptive mechanism that responds to data (e.g., by adjusting weights based on group size/variance). The authors should discuss this in limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2I8tGFPYWQ", "reviewer": "Reviewer_KVy8", "summary": "This paper proposes HGPO, a herarichical extension of group-based RL designed to address context inconsistency in long-horizon LLM-based agents. Empirical results on ALFWorld and WebShop show consistent improvements over GRPO and GiGPO.", "questions": "1. Figures 2–5 mention oracle groups and context inconsistency bias. How exactly were these oracle advantages computed?\n2. Only three random seeds are used. Given the small standard deviations reported for ALFWorld, can the improvements be considered statistically reliable?\n3. HGPO is said to add “minimal extra time cost”. Can the authors report wall-clock training time or GPU hours to support that claim?\n4. In Page 8, the paper claims that Figure 4 shows stable learning. Could the authors provide a quantitative evaluation to support the claim? It is not obvious that the curves of HGPO are smoother.\n5. The weighting $\\omega_k$ is heuristic and fixed across training, and I find the $\\alpha$ also fixed for main exps. It is a good setting as it does not introduce extra tuning cost. But since variance and bias evolve during training, how does such a static schedule reflect the actual trade-off? \n6. The grouping is computed offline via hashmap lookups. As the policy evolves, group memberships change. Does the method recompute them every epoch? If not, the hierarchical structure may quickly become outdated.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The hierarchical grouping and adaptive weighting mechanism is conceptually simple and computationally efficient.\n2. Experimental results demonstrate strong and stable improvements across benchmarks.", "weaknesses": "1. The Proposition 4.1 assumes bias decreases and variance increases monotonically with context depth. In practice, this assumption may not always hold, e.g., noisy long contexts could add both bias and variance.\n2. The definition of hierarchical groups $G_k^H(s_t^{(i)})$ assimes exact equality of historical contexts. Does it hold in practice?\n3. The notion of context inconsistency is intuitive but not rigorously formalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)