Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
Authors: Heyang Gao
Zexu Sun
Erxue Min
Hengyi Cai
Shuaiqiang Wang
Dawei Yin
Xu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM-based Agents
Process Supervision
Curriculum Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce HPL, a hierarchical framework that resolves the granularity mismatch in agent alignment by optimizing preferences over semantically coherent "action groups" (sub-tasks), guided by a dual-layer curriculum.

Abstract: Large Language Models (LLMs) as autonomous agents are increasingly tasked with solving complex, long-horizon problems. 
Aligning these agents via preference-based methods like Direct Preference Optimization (DPO) is a promising direction, yet it faces a critical granularity mismatch. 
Trajectory-level DPO provides stable signals but blur where credit should be assigned within long trajectories, whereas step-level DPO offers fine-grained supervision but can be statistically noisy and data-inefficient when Monte Carlo rollouts are limited, and can be hard to fully exploit multi-step structured behaviors that only reveal their effect over several actions.
To balance this trade-off, we introduce **H**ierarchical **P**reference **L**earning (HPL), a hierarchical framework that optimizes LLM agents by leveraging preference signals at multiple, synergistic granularities. 
While HPL incorporates trajectory- and step-level DPO for global and local policy stability, its core innovation lies in group-level preference optimization guided by a dual-layer curriculum. 
Our approach first decomposes expert trajectories into semantically coherent action groups and then generates contrasting suboptimal groups to enable preference learning at a fine-grained, sub-task level. 
Then, instead of treating all preference pairs equally, HPL introduces a curriculum scheduler that organizes the learning process from simple to complex. 
This curriculum is structured along two axes: the group length, representing sub-task complexity, and the sample difficulty, defined by the reward gap between preferred and dispreferred action groups.
Experiments on three challenging agent benchmarks show that HPL outperforms existing state-of-the-art methods. 
Our analyses demonstrate that the hierarchical DPO loss effectively integrates preference signals across multiple granularities, while the dual-layer curriculum is crucial for enabling the agent to solve a wide range of tasks, from simple behaviors to complex multi-step sequences.

Published as a conference paper at ICLR 2026
SOLVING THE GRANULARITY MISMATCH: HIERARCHICAL PREFERENCE LEARNING FOR LONG- HORIZON LLM AGENTS
Heyang Gao Zexu Sun Erxue Min", Hengyi Cai*,
Shuaiqiang Wang", Dawei Yin Xu Chen 1.2tt
1 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
2 Gaoling School of Artificial Intelligence, Renmin University of China
3 Baidu Inc.
[gacheyang, xu. chen]eruee edu. en, sunzexu0826@gaiil com
ABSTRACT
Large Language Models (LLMs) as autonomous agents are increasingly tasked with solving complex, Long-horizon problems. Aligning these agents via preference-based methods like Direct Preference Optimization (DPO) is promis- ing direction, yet faces a critical granularity mismatch. Trajectory-level DPO provides stable signals but blur where credit should be assigned within long tra- jectories, whereas step-level DPO offers fine-grained supervision but can be sta- tistically noisy and data-inefficci can be hard to fully exploit multi-step structured behaviors that only reveal their effect over several actions. To balance this trade-off, introduce Hierarchical Preference Learning (HPL), a hierarchical framework that optimizes LLM agents by leveraging preference signals multiple, complementary granularities. While HPL incorporates trajectory- and step-level DPO for global and local policy sta- bility, its core innovation lies in group-level preference optimization guided by a dual-layer curriculum. HPL first decomposes expert trajectories into semantically coherent action groups and then generates contrasting suboptimal groups enable preference learning at a fine-grained sub-task level. Then, instead of treating all preference pairs equally, HPL introduces a curriculum scheduler that organizes the learning process from simple to complex. This curriculum is structured along two axes: the group length, representing sub-task complexity, and the sample diffi- culty, defined by the reward gap between preferred and dispreferred action groups. Experiments on three challenging agent benchmarks show that HPL outperforms existing state-of-the-ard methods. Our analyses demonstrate that the hierarchical DPO loss effectively integrates preference signals across multiple granularities, while the dual-layer curriculum is crucial for enabling the agent solve wide range of tasks, from simple behaviors to complex multi-step sequences.
when Monte Carlo rollouts are limited, and
INTRODUCTION
Large Language Models (LLMs) have evolved from static question-answwring systems into au- tonomous agents capable of perceiving, reasoning, and acting within complex, open-ended envi- ronments (Li et al. 2024 Gou al. 2025). This transformation has powered new generation of applications, from embodied assistants that navigate simulated homes (Shridhar et al. 2021) to web navigators that execute multi-step online tasks (Zheng al., 2024 Furuta al. 2024 Unlike single-turn tasks, these agent-environment interactions unfold in multi-turn loops over extended pe- riods (Wang et al. 2024a). This paradigm shift introduces core challenge: long-horizon planning and decision-making where the agent must execute coherent sequence of actions to succeed.
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
Tyy
tomato
cinkbanie
with sinkbossh
wett aiibbadn
Know Cese path suoeeded and but whhrre exsetly yas the mistaly?
r((0))
(a) Trajectory-level DPO
2
6¹:
r(6')
9
not single better, but which sequence of actions achieves meaningt goal, like viean tomiato".
Ckay, aitson es is wrong, why? ls actiion o, always belter in conteat?
(b) Step-level DPO
(c) Group-level DPO
Figure 1: Conceptual comparison of different DPO granularities. While (a) trajectory- DPO
provides a coarse but stable signal and (b) step-level DPO offers focused but potentially noisy super- vision, (c) our proposed Group-level DPO learns from semantically coherent action groups, which provides a structured signal, enabling the agent to reason at the sub-task level.
To equip agents for such tasks, Reinforcement Learning (RL) has become crucial recipe for post- training LLMs (Liu et al. 2024). Online RI. methods like PPO (Sutton et al.] 1998; Schulman et al.) 2017) often entail substantial computational costs, high sample inefficiency, and risky, inefficient exploration in vast action spaces. These challenges have motivated approaches that reduce reliance on online interaction by learning from static datasets collected in advance. Direct Preference Op- timization (DPO) (Rafailov et al.) (023) directly aligns agent policies using preference pairs (e.g., expert vs. suboptimal behaviors) without requiring costly environment interaction or an explicitly trained reward model.
However, applying DPO to long-horizon agent tasks reveals a fundamental challenge we term the granularity mismatch. On one hand, trajectory-level DPO, such as ETO (Song et al. 2024), com- pares entire trajectories and yields stable, low-variance feedback aligned with final outcomes, but it provides limited resolution for credit assignment, which is hard to tell which segment of a long interaction actually determined success or failure. On the other hand, step-level DPO, employed by methods such as IPR (Xiong al. 2024), attributes preferences to individual decisions by estimating the expected return from each decision point via Monte Carlo rollouts. However, this fine ane-grained focus poses practical challenges in the finite-data, limited-rollout regime. Supervision becomes fragmented across many decision points, so each step is updated from only a few noisy rollouts, and it can be hard to fully exploit multi-step structured behaviors whose contribution success only becomes apparent when several actions are considered jointly. For instance, the sub-task of "retrieving an apple from the fridge" is composed of chain of catioos-navigating. opening, and taking-whar whose collective value cannot be captured by rewarding any single action in isolation.
To resolve this dilemma, we introduce Hierarchical Preference Learning (HPL), a hierarchical framework that optimizes LLM agents by leveraging preference signals at multiple, synergistic granularities. HPL. first addresses the granularity mismatch by incorporating DPO losses at the trajectory, action, and the action-grous group levels. The group-level view provides both structural prior by focusing supervision on sub-traecctories that are more likely to encode reusable skills and a statistical benefit by aggregating the contribution of multiple actions into one decision unit, which reduces variance relative to per-step Monte Carlo estimates under a fixed rollout budget. Beyond merely combining these losses, the innovation of HPL is dual-layer curriculum learning strat- egy that guides the training process. This curriculum systematically organizes the learning path from simple to complex along two orthogonal axes: sub-task complexity, defined by the length of an ac- tion group. and sample difficulty, measured by the reward gap between preferred and dispreferred behaviors. By first mastering simple, easily distinguishable sub-tasks, the agent builds a foundation before progressing to more complex challenges.
Our main contributions are summarized as follows:
We identify and address the granularity mismatch problem in preference based agent alignment by proposing novel hierarchical framework that integrates preference signals at three distinct levels: the coarse trajectory-level, the fine-grained step-level, and intermediate action-group level.
2

---

Published as a conference paper at ICLR 2026
We introduce HPL, novel training paradigm with dual-layer curriculum learning strategy. This is the first work to apply a structured curriculum to action- group level preference optimization, dynamically scheduling samples based both task complexity and distinguishability.
We design and systematically evaluate a range of action grouping strategies, from simple heuristics to a semantic approach, to generate meaningful sub-tasks for our framework.
We demonstrate through extensive experiments on three diverse and challenging long-horizon benchmarks that HPI. significantly outperforms existing state-of-the-art methods, establishing more effective and principled paradigm for preference- training LLM agents from fixed datasets after one-shot exploration phase.
2 RELATED WORK
LLM-based Agents. The remarkable reasoning and instruction instruction-follooingg capabilities of modern LLMs have enabled their use autonomous agents capable of tackling diverse array of complex, interactive tasks (Li et al., 2024 Gou et al.] 2025 Tang et al.] 2025 Wang et al., (2025b). Initial approaches primarily leveraged the in-context learning abilities of LLMs through prompts like Re- Act (Yao et al. 2023) and Reflexion (Shinn et al. 2023) to elicit multi-step reasoning and action gen- eration. This focus on inference-tine, computation has recently evolved into the broader paradigm of test-time scaling (TTS), where allocating additional compute during generation (Zhang et al., 20256). However, to enhance the performance of open-source models beyond zero-shot prompt- ing and and test-time strategies, subsequent line of work has focused on fine-tuning agents on collected trajectory data (Chen et al. (002)). These methods range from standard Supervised Fine- Tuning (SFT) expert demonstrations (Zeng et al.) [0233] Chen et al.) (2044) to more advanced techniques that learn from preference data, such as contrasting successful and failed trajectories to optimize for final outcomes (Song et al., 2024). While effective, these fine-tuning paradigms often treat entire trajectories as monolithic data points (Wang et al. (2024b) overlooking the fine-graing one-grained procedural knowledge embedded within the interaction trajectory.
Process Supervision. To address the limitations of outcome-based rewards, particularly the chal- lenge of credit assignment in long-horizon tasks, growing body of research has explored process supervision (Luo et al. 2024) Xiong et al., 2025). The core idea is to provide agents with more gran- ular feedback at intermediate steps of task. Early efforts this area often relied on costly human annotations to label the correctness of each step (Lightman et al. 2023). To automate this, recent methods have proposed various techniques to estimate step-level rewards, such as using Monte Carlo rollouts to predict the future outcome from an intermediate state (Xiong et al., (2024 or training separate reward model to predict the value of each action (Choudhury) 2025) Wang et al., (025a). These approaches typically use the estimated step-level rewards to guide the agent via reinforcement
learning (Feng et al. 2025 Zhang et al. 2025a) or DPO at the single-action level.
3 METHODOLOGY
In this section, we present our novel agent alignment framework Hierarchical Preference Learning (HPL) depicted in Figure 2 We detail the principal phases of our method below: initial policy bootstrapping through behavior cloning, the generation of multi-gran qulti-granularity preference data, the design of our dual-layer curriculum scheduler, and finally, the hierarchical optimization objective.
3.1 PROBLEM SETTING
In this work, HPL follow the same two-stage protocol previous work (Song et al.) 2024 Xiong ct al. 2024) that combines a one-shot exploration phase with purely offline preference optimization.
Fixed exploration and labeling. A frozen reference policy nee interacts with the environment once to collect a pool of interaction traces, including both full trajectories and partial segments. We then run Monte Carlo rollouts with गe this static pool derive different level reward estimates, constructing the preference datasets. Importantly, the policy no is never updated during this phase, and no further data collection is performed once the datasets are built.
3

---

Published as a conference paper at ICLR 2026
Expert Trijeciory
Stage Hierarchical Contrastive Data Generation
Stage Granularity
Preference Optimization
and adetable
Uper 2
Sueehrige11
Behavior
Cloning
LLM Agent
Trajcctor--ivell Data
Trajectory-Ieve DPO Loss
Step-level
Group-level
DPO Loss
Group level Data
Action Segmentation /:3
Find Heurlttc
Semantie based
& Roward Extination
Schaduler
DDB 3DD
Derde me ort secerto
G00000
Group
Expart Trajectory (t.)
Uharrtinty-tuued
DC 00000
readd
Difisitty all
Figure 2: An overview of our proposed framework, HPL. Stage generates hierarchical preference data with Action Group Segmentation component. Stage 2 then optimizes the agent with a composite objective, where the training is guided by dual-layer curriculum scheduler.
Offline preference optimization. Given these fixed datasets, preference peeference-based methods train the target policy using DPO-style objectives at different levels (trajectory, step, and group), without any additional environment interaction or reward queries.
We therefore view our setting as offline preference optimization after one-shot exploration phase, which is distinct from fully online RL methods such as PPO (Schulman al., (0177) that continu- ously interleave policy updates with new environment rollouts throughout training.
3.2 BOOTSTRAPPING VIA EXPERT BEHAVIOR CLONING
To equip the base model with fundamental task-solving capabilities, we perform behavior cloning on dataset of expert trajectories Dexpert ((u,)*)()))) (Papel where the task instruction and 7* is the corresponding expert trajectory. Each trajectory is sequence of alternating states and actions 7 (81, (11, #2, 02,.... ST; aT). ). state 8, textual description of the environment, and an action a, is textual command generated by the agent. This process aims to maximize the likelihood of the expert's actions. The loss is defined as:
(Bc Dexpen) = (B(u " Deuper
log ne (a
u, =2x)
where Th, represents the history (si aj...... 47-1). This initial cloning step yields a compe- tent base agent, which serves reference policy net for the subsequent optimization stages.
3.3 HIRRARCHICAL CONTRASTIVE DATA GENERATION
After obtaining competent reference policy The via behavior cloning (Section 3.2 the next stage to generate rich, multi-layered dataset for preference optimization. This is achieved by having Net interact with the environment produce diverse of suboptimal trajectories. By contrasting these with expert trajectories, we construct three distinct preference datasets at the trajectory, action, and group granularities, illustrated in the left panel of Figure
4

---

Published as a conference paper at ICLR 2026
3.3.1 DATA GENERATION AT TRAJECTORY AND ACTION LEVELS
Trajectory-Level Data. This dataset provides a coarse, outcome-based learning signal. For each expert trajectory from Dexpert, we use Fre to generate corresponding full trajectory тІ- If the outcome reward of 7) is lower than that Tw, we form a preference pair (Tar: 7.). This process yields {(u, Tw. (7)), where is the task instruction.
Tw,,
a trajectory-level dataset Panj
Step-Level Data. To provide a finer-grained, process-oriented signal, we adopt the methodology from IPR (Xiong et al. 2024). At each step of an expert trajectory, we use the history T<t as a prompt for our reference agent Tre to generate an alternative action â; and complete the rest of the trajectory, yielding Tém This is contrasted with the expert's subsequent trajectory Ttm This cre- ates a preference pair conditioned on the shared history, resulting in step-level preference dataset Dstep = ((Ter, Tin T(mm)())
3.3.2 GROUP-LEVEL DATA GENERATION VIA ACTION GROUP SEGMENTATION
To bridge the gap between coarse trajectories and single actions, introduce the core concept of an action group. These groups serve as an intermediate unit of reasoning, ideally corresponding to semantically coherent sub-tasks for more effective credit assignment. The generation of group-level data involves two key steps: segmenting trajectories into groups and then estimating quantitative reward for each group.
First, we apply segmentation function f()) to partition expert trajectory Tu into corresponding action groups {G..,i) We design and investigate four distinct segmentation strategies:
Fixed Heuristic Strategies. As baselines, we consider two straightforward methods based on length. Fixed-N Groups divides trajectory into a fixed number N of equal-length groups. Fixed-K Size creates groups with K consecutive action steps each. While simple, these methods are agnostic to the task's semantic structure and risk making arbitrary cuts.
Uncertainty-Bant Based Segmentation. This adaptive strategy is based on the intuition that policy's s uncertainty often increases at sub-task boundaries (Guo et al. 2025). We leverage the entropy of the reference policy's 's action distribution, H (T ()))) as proxy for uncertainty. A boundary is inserted after action ac-1 if the entropy at step exceeds predefined threshold e.
Semantic Segmentation. To achieve the most meaningful partitions, employ powerful, pre- trained LLM (e.g., GPT-4o) as an off-the-shelf "semantic segmenter". We provide the full text transcript of trajectory the model and prompt it to partition the sequence into high-level sub- tasks based on their apparent goals (e.g.. "find an object", "operate an appliance"). This method is expected to yield the highest quality segmentations.
Once an expert trajectory is partitioned into a sequence of winning groups (Gm..)). J. we construct the preference pairs required for our group-level optimization. For each expert action group G which begins from a context & (i.e., the history of all preceding steps), we generate corresponding losing group Gt, This is achieved by sampling new action sequence of the same length from the reference policy Tre((-(cs) This length-constraing comparison between the expert and suboptimal behaviors. The resulting tuples (c,, Gw,is GI,;) form rich dataset of fine-grained preference pairs, which are the fundamental training units for HPL.
sampling ensures a fair, apples-to-apples
3.3.3 GROUP-LEVEL REWARD ESTIMATION
After segmenting trajectories into groups. we need quantitative reward estimate for each group to filter data and enable our curriculum learning strategy (detailed in Sectioo[..4) We define the reward of an action group G, which ends at timestep i, with history Tet,, as the expected final outcome reward of trajectories completed from that point. Given the difficulty of direct computation, we estimate this value using Monte Carlo (MC) sampling. Specifically, we use our reference policy Tee to perform M stochastic rollouts starting from the state after G; has been executed. The estimated reward for the action group G denoted f(G,). is the average of the final outcome rewards R(-) from these rollouts:
1 M M
F(G.)
Σ ((T(
where ( TMCT (T<Li
5

---

Published as a conference paper at ICLR 2026
This MC estimation is applied to every winning group Gu.s and losing group Gt, to obtain their rewards. With these components, we finalize our group-level dataset DD Dgroup {(c, Guey ())(), where each entry contains a context, a preference pair of groups, and their estimated rewards.
While step-level MC estimation provides a principled, localized estimate of the expected return at each decision point, in practice the rollout budget per state is small. Under this limited-rollout regime, per-step estimates can have high variance, which makes purely step-level preference learn- ing statistically inefficient. In contrast, our group-level rewards aggregate the outcomes of multiple actions into single supervision unit, amortizing the same rollout budget over longer sub-trajectories and better capturing their joint contribution to task success.
3.4 DUAL-LAYER CURRICULUM LEARNING
Statically mixing the group-level preference data from all difficulties for training be suboptimal, as it may expose the model to highly complex samples before it has developed foundational skills, leading to unstable or inefficient learning. To address this, we introduce the core innovation of our framework: dual-layer curriculum learning strategy. This strategy dynamically organizes the training process to mimic an efficient human learning path, from simple concepts to complex ones.
The 2D Curriculum Matrix. As illustrated in Fig- ure 3 our curriculum is conceptualized as two- dimensional difficulty matrix. We categorize each group-level preference pair (G., Gi) along two or- thogonal axes:
Group Length
(Long)
Sub-task Complexity (Y-axis): This dimension measured by the Group Length (T.). Shorter ac- tion groups (e.g., 1-3 steps) correspond to simple, fundamental skills, while longer groups represent more complex, multi-step behaviors that require longer-term planning.
L2 (Medium)
(Short)
Sample Difficutty
(Eany)
Sample Discriminability (X-axis): This dimen- sion is measured by the Sample Difficulty (AR), defined the difference between the estimated re- wards of the winning and losing groups: AR
Sequencer
Figure 3: Illustration of the dual-layer cur- riculum scheduler with group length (L) and sample difficulty (AR). The training follows three-phase schedule.
f(Gw) - f(Gi). A large AR indicates an easy- to-distinguish sample where the losing group is clearly inferior. small A R represents hard-to- distinguish sample that requires finer judgment from the agent for successful policy refinement.
Based on these two axes, we partition the group-level dataset D grmupp into a 3x3 grid of data buckets, denoted as Br.DD where L, D E (1,2,3) represent the levels of length and difficulty, respectively.
The Curriculum Schedule. Our training process not a single pass over the mixed data, but staged schedule that progressively expands the training set, guiding the model along a path of increasing difficulty. The schedule consists of three distinct phases:
1. Phase (Foundational Skills):
Initially, the model is trained exclusively the easiest data bucket, Bi,1 (short length, easy difficulty). This allows the agent to quickly and stably learn the most fundamental and unambiguous skills without being distracted by more complex scenarios.
2. Phase 2 (Expanding Complexity): After the initial phase, we expand the training data to include B1,1 U B1, U B2,1- In this stage, the agent begins to tackle harder (less distinguishable) short- horizon tasks while also being introduced simple medium-horizei skills, effectively broaden- ing its capabilities.
3. Phase (Full-Scale Tuning): Finally, the training set is expanded to include all nine buckets Un BI,D). The agent now fine-tunes its policy on the full spectrum of complexities and difficulties, mastering the most challenging and nuanced aspects of the tasks.
This staged exposure ensures a smooth learning gradient, building agent's 's expertise from the ground up and preventing it from being overwhelmed by difficult samples early in the training process.

---

Published as a conference paper at ICLR 2026
3.5
PREFERENCE OPTIMIZATION
In the final stage, we optimize the policy ne using a composite loss function that integrates signals from all three granularities. This approach ensures that the agent only learns from high-level outcomes and fine-graine sub-tasks but also stays grounded in the expert's 'ss behavior. The final loss includes sum of three components:
Prajectory-Lever DPO Loss (Cuat-DPO)- To learn from the overall outcome, we apply DPO loss on the trajectory-level dataset uasj This loss encourages the policy assign higher likelihood to the entire successful trajectory over the failed one:
(o; Deraj) ()(Tss))-DD log o (Blog
log TT(Tulu) B log Teer ( (u) )
To (T(TT)))
(3)
Tie Tree(Twilus (u)
Step-Level DPO Loss (Lste--Dro) Drawing from Xiong et al. 2024), this loss uses Dstep to provide step-level supervision by comparing the entire future from decision point:
Dro Dsuep) - -E (Facr (im) ~Dpo log log
no (o Iree) mal (re))
B log so(r (lzze)) ) (4)
Group-Level DPO Loss (Cgr) This is the core component of our framework, providing mid- level supervision. We apply the DPO loss to the group-level dataset DD action groups:
Deroup= comparing corresponding
Dgooup-DO (0: Deruu) -Bo,, logo (llg
To Fo log
Te GGlc))
)] (5)
The
Ther( (Gi
We briefly analyze the bias-varianee properties of this group-level objective in the following propo- sition, with full derivation provided in Appendix G
Proposition -Bras-variance trade-off of group-level DPO loss). Let denote the trajectory length, Y [0. 1) the discount factor, and Rmax the maximum reward. Let Luuj. Cnep. and ggoup (k) denote the empirical losses of trajectory-lever step-level, and group-level DPO with group length < T, respectively. Then there exists constant C 0 depending only Fre)) such that for every
(0, 1) the choice k(e)
1ogg,
(1-7)e R
satisfies
Bias( Ceropp (oowwp(()))))))) (k)) S min [Bias(Cra)) Bias( te,
C Cllo(L//e)
Var(C (k)) <
min Var(Cua)), Var(Cstep))
Hence, by setting (llog(I/e)) group-level DPO loss simultaneously improves the variance by factor Q(T/ (gg(1))) while incurring at most an additive bias of over the other two losses.
The final training objective combines these losses:
((s)
final LBC + Lunj-DPO + Late--DP (s) -ro
where the group-level loss c(+) DPO for curriculum stage is computed over a dynamically selected subset of data, Dimupp which is determined by our curriculum scheduler (Section 3.4.. The data
subset ((s) for each stage is constructed from the 2D eurriculum matrix buckets BI follows:
Bi,1 Bi, U B Un BI.D
if 1 (Phase 1)
(()) group
if 2 (Phase
if 3 (Phase 3)
4 EXPERIMENTS
In this section, we conduct a series of experiments to comprehensively evaluate the performance of HPL framework. Our evaluation is designed to answer the following key research questions:

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of HPL and baselines across agent benchmarks over random seeds. All methods are evaluated using Qwen2.5-1 1..52..5-1.5B-Imae Inst-utt and Owen2.5-7B-Insttee as base models. The best and second-best -bost results are highlighted in bold and with an underline, respectively.
ALFWorld
WebShop
InterCade-SQU.
Models
Average
seen
unseen
avg. reward
success rate
avg. reward
success rale
GPT-4o Gemini-2.5-Pro
36.43
32.09
55.26
18.50
28.50
28.50
33.21
55.71
49.25
49.56
19.50
68.42
66.00
51.40
Owen2.5-1.0BIIstruutt
2.14
0.00
36,09
10.50
5.50
5.50
9.95
SFT
60.95.1
57.96
56.56.0.9
26.00.a.50
56.24.n61
54,33
52.01
RFT
Yuan et al.
2023
61. 69.192030
60.95
57. 17.666145
28. 17.1..4
58. 08
56,672029
53.7920.40
ETO Song et al. 2024
65,48,3.0
66.42,2.24
56.57a0.2
28. .3.37...37
58.452101
57.67,076
55,43
IPR
Xiong al.
202
65, 65.24g230
66.67223.0
57.7641.13
27.8321.04
58.2621.75
57.172104
55,49
HPL
(P
4xed [33]
69.5221.48
74.38
60.21
30.17=1.61
58. 58.75
57.67
58.45
HPI.
(Pixed-K( (3)
70.482109
66.4242.09
58.344184
28. 28.33
59.69
57.174477
56.7440.7
HPL
(Uncertainty)
74.53
64.18
58.75
27. 27.83
1
57,33
56,95
HPL
(SemanLic)
72.86.1 wP
74.13+1
60.74.1.0w
30.00.1 oo
60.39
58,502100
59,44
Owen2.5-7BIInstruct
38.57
45.52
56,61
19.50
8.80
8,50
29.58
SFT
67,624218
73.63
60.64.1.12
31,83,1.26
66.70
65.174u.8
60.93
RFT Yuan et al.
2023
11.43
72.63
61.16.085
33.50.1.00
68,01.n89
66.33
18
ETO
Song
2024
72.622251
T7.86 77.8622.40
61.85
33.1721.44
68.322n86
67.00ga.0
63.47
IPR
Xiong al 2024
13,10
78.11
62.01a043
33.67=0.58
68.8621
67.17
63.8220.m
HPI.
(P 1PPxxx--NX30
78.334251
78.86622.00
62.1 1
34.3120.76
69.55=1
68.0021.00
65.204a0.4
HPL
(Pixed-K (3)
85.71,258
78.6
62.01
33.8321.26
69.40_a.95
68.172n.5
66,29
HPL (Uncertainty)
83,10.180
83,33
62.79.0a5
35.33-1.04
69.21
67.83
66,93
HPL (Semantic)
82.62,230
84.08
62.97
35.
70.37
68.50.1.22
67.28
RQL. Does HPL outperform strong baselines that rely on conventional learning granularities, such as the trajectory-Iever (ETO) and step-level (IPR)?
RQ2. What is the impact of different action group segmentation strategies on the final perfor- mance of the agent?
RQ3. How crucial is the dual-layer curriculum mechanism to the success of HPL, and what the contribution of each curriculum layer?
RQ4. How do trajectory-level, step-level, and group-level losses contribute to HPL?
4.1 EXPERIMENTAL SETUP
We evaluate our proposed method, HPL, on three diverse and challenging long-horizon agent bench- marks: ALFWorld (Shridhar et al. 2021). WebShop (Yao et al., 2022). and InterCode-SQL. (Yang et al. 2023). In all our benchmarks, the environment returns a single terminal outcome reward at the end of each episode, which defines a finite-horizon episodic MDP and can be modeled with y close to and suitably rescaled rewards. For all experiments,
use Owen2.5-1.5BBImm Instruct and Qwen2.5-7B-IInsrruce as the backbone language models. HPL is compared against a suite of strong baselines, including SFT, RFT (Yuan al., (023). ETO (Song al., 2024), and IPR (Xiong al., 2024). All methods are initialized from an SFT model trained on expert trajectories generated by GPT-4o teacher model. For MC estimation, we use M rollouts for per group. For Fixed-N and Fixed-K strategy, we set N = 3 and K = 3. For Uncertainty strategy, the entropy thresh- old is set the 80th percentile of all action entropies (i.e., the threshold for the top 20% highest values) computed across the training dataset. Detailed descriptions of the environments, baseline implementations, and all hyperparameters are deferred to Appendix D
4.2 MAIN RESULTS (RQ1, RQ2)
As shown in Table our HPI. framework significantly outperforms all baseline methods across both model scales, providing strong affirmative answer to our first research question. For the Qwen2.5- 7B-Instruct model, our best-performing variant HPL (Semantic) achieves an average score of 67.28, surpassing the strongest grgnulggraularity baselines ETO and IPR by substantial margins of 3.81 and 3.46 points, respectively. This trend of superiority holds true across all three diverse and chal- lenging benchmarks. The advantage of HPL's hierarchical approach is particularly pronounced in tasks requiring complex, long- long-horizon generalization. For instance, on the ALFWorld benchmark's 's unseen scenarios, HPI. (Semantic) achieves mean success rate of 84.08%, remarkable improve-
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study on our curriculum learning mechanism of HPL across three agent bench- marks.
ALFWorld
WebShop
InterCode-SQL
Models
Average
seen
unseen
avg. reward
success rate
reward
success rate
Qwen2.5-1.5B-nnstiee
HPT.
71.43
72.39
59.99
30,00
60.08
58.50
58.73
HPL Static
68.57
71.64
58. 80
29.00
59,45
58.00
57.58
HPL Length CL Only
69.29
72.39
58.06
28.50
$9.39
57.50
57.52
HPL. Difficulty CL. Only
71.43
70.71
58.83
30.00
60.50
58.00
58.25
Owen2.5-7B-Istteee
HPL
83,57
86,57
62.56
34.50
70.63
69.00
67.81
HPL Static
75.71
82.84
62,05
33.00
69.71
68.50
65.30
HPI. Length CI. Only
82, 14
85.07
62.26
34.00
69.49
67.50
66.74
HPL Difficulty CL Only
81.43
85.82
62.27
34.50
69.63
68.00
66.94
S.ven2.5-1.5B-Inttte
Pick
Clean
Heat
80
Average
90
80
70
Unseen
80
70
80
Success Rate (%)
50
40
50
40
50
Qwwn22.577B-nnttree
Cool
Examine
Pick2
90
70
70
80
70
60
40
50
30
30
Phase
Phase 2
Phase
Phase Phose Phase 3
Phase Phase Phase 3
Phase Phose 2
Phase
(b)
Figure 4: Phase-wise performance progression of HPL the ALFWorld benchmark. (a) Success rates for both 1.5B and 7B models across the three curriculum phases. (b) A detailed breakdown for the 1.5B model 6 sub-task types.
ment of nearly 6 points over the state-of-the-art IPR (78.11%). These results strongly indicate that by integrating preference signals at the trajectory, action, and group levels, HPI. effectively resolves the granularity mismatch problem and learns more robust and generalizable policy.
Our experiments also reveal the critical impact of the action group segmentation strategy, directly addressing second research question. While all HPL variants consistently outperform the base- lines, the adaptive, content-aware segmentation methods generally yield better performance than heuristic approaches. Notably, HPL (Semantic), which leverages powerful language model to par- tition trajectories into semantically coherent sub-tasks, consistently emerges as the top-perforning variant for both the 1.5B and 7B models. For the 7B model, outperforms the next-best variant, HPL (Uncertainty), by over 0.35 point in the mean average score. This suggests that the quality of the action groups paramount; providing the DPO loss with more meaningful, human-alignne aaigned sub-tasks as units of comparison leads to a more effective and powerful learning signal. Even so, the strong performance of the simpler Uncertainty and Fixed-N strategies demonstrates the inherent value of incorporating an intermediate granularity, validating the core design of our framework.
4.3 ANALYSIS OF THE CURRICULUM LEARNING MECHANISM (RQ3)
Ablation of Curriculum Components. To quantitatively assess the importance of our dual-layer curriculum, we conduct an ablation study by removing each layer individually, with results presented in Table 2 The primary finding is that the full HPL model, equipped with both curriculum layers, consistently outperforms all ablated variants. Removing the curriculum entirely (HPL Static) results in the most significant performance degradation across both model scales, confirming that employ-

---

Published as a conference paper at ICLR 2026
NPL
HPL wit IraptPno
HPL win seep-PPt
HPI with gmup-
ALFWorld (seen)
ALFWorld (unseen)
(c) WabShop
InterCode-502
85.7
60.4
52.7
F1.
Parfo Parformance S-OOT
08.07
36.37
$2.36
7661
$3.50
04.5
7A
12.0
61,95
89.0
77.43
70.7
75.8
12.8
00.5
Figure 5: Ablation study on the HPL. loss components on Qwenn.5-7B-Insse
ing a curriculum is crucial for effective learning. For the 7B model, HPL Static trails the full HPL by 2.51 average points, substantial margin that underscores the necessity of structured learning progression. Furthermore, the results indicate that both the length and difficulty- contribute positively to the final performance, with their individual removal leading to noticeable performance drops. This demonstrates the synergistic benefit of our dual-layer design, where orga- nizing the learning process first by task complexity (length) and then by solution quality (difficulty) provides a more effective path to mastering complex agent behaviors.
curricula
Phase-wise Performance Progression. To provide more fine fineinriieed view of how the curricu- visualizes the agent's performance at the end of each curriculum phase. Panel (a) shows clear improvement in the overall success rate for both the 1.5B and 7B models they progress from Phase to Phase 3. This trend holds for the unseen scenarios, indicating that the cur- riculum effectively helps the model generalize its learned skills. Panel (b) offers a deeper insight by breaking down the performance by sub-task type for the 1.5B model. We observe that while simpler tasks like Pick are learned relatively early, more complex tasks requiring longer reasoning chains, such as Clean and Pick2, show the most substantial performance gains in the later phases. This stage-wise improvement, with harder tasks being mastered in later stages, provides qualitative evi- dence that our dual-layer curriculum successfully guides the agent through an easy-to-hard learning process, progressively building its capabilities for complex, long-horizon planning.
lum works, Figure /
4.4 ABLATION ON HIERARCHICAL DPO LOSSES (RQ4)
To investigate the individual contribution of each component in our hierarchical framework, we conduct an ablation study on the three DPO losses, with results shown in Figure 5 The results reveal that while all three loss components, trajectory, action, and group. contribute positively to the final performance, the group-level DPO is demonstrably the most critical. Across all bench- marks, removing the group-level signal (HPL w/o group-DPO) induces significantly more severe performance degradation than removing either the trajectory or step-level signals. This provides compelling evidence that the action-group granularity is the primary driver of HPL's effectiveness, serving as crucial bridge between coarse trajectory feedback and action supervision and validating the synergistic benefit of our three-level approach.
5 CONCLUSION
In this work, we address the critical issue of granularity mismatch in preference-based alignment for LLM agents. We introduce Hierarchical Preference Learning (HPL), novel framework that solves this challenge by integrating preference signals across three levels of abstraction: trajectory, action, and erucial, intermediate action-group level. By partitioning trajectories into semantically coherent sub-tasks and optimizing a hierarchical DPO loss, HPL. learns to directly prefer success- ful multi-step action sequences over flawed alternatives. Extensive experiments demonstrate that HPL outperforms strong prior methods that operate primarily at the extreme granularities of entire trajectories or individual actions, across suite of complex, long-horizon agent benchmarks. In con- clusion, our work underscores the importance of learning from hierarchical signals that mirror the compositional nature of complex tasks, paving the way for more capable LLM agents.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Our experimental setup is briefty summarized Section 4.1 To facilitate the reproduction of results, Appendix [ offers detailed account of all the benchmarks, data generation pro- cess, baselines, and all hyperparameters. We also release the code and data, which are available at: https: htt.a/bb.ccmmaarrcccc100/////
ACKNOWLEDGMENTS
This work is supported in part by National Natural Science Foundation of China (No. 62472427 and No. 62422215), Major Innovation & Planning Interdisciplinary Platform for the "Double First- Class" Initiative, Renmin University of China, Public Computing Cloud. Renmin University of China, fund for building world-class universities (disciplines) of Renmin University of China, and in part by the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) under Grant No. GML-KF-24-33.
REFERENCES
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.1288 2024.
Sanjiban Choudhury. Process reward models for Ilm agents: Practical framework and directions. arXiv preprint arXiv:2502.10325, 2025.
Lang Feng. Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-grond group policy optimization for Ilm agent training. arXiv preprini arXiv:2505. 10978, 2025.
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane
Gu, and Izzeddin Gur. Multimodal web navigation with pnstruction-finetune foundation models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openzeveew.o
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and
Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https: 7/ /oopnrevview www.teettuunneeee htt//oormmm?/ wtt.:////ikk.oooissoooo
Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Ef- fective segment-level credit assignment arXiv:2505.23564, 2025.
for large language models. arXiv preprint
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking 1lms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
Liangchen Luo. Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu. Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint 2665522406.06592, 2024.

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-533744 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 16:8634-8652. 2023.
Mohit Shridhar, Xingdi Yuan. Mlex-nlex Cote. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. htt.htttp:///peeeee
meet/orrm?id-iie OIX00YeCOOI
Yifan Song. Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and emror: Exploration-basee trajectory optimization of L.LM agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7584- 7600. Association for Computational Linguistics, 2024.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Jiakai Tang. Heyang Gao, Xuchen Pan, Lei Wang. Haoran Tan, Dawei Gao. Yushuo Chen, Xu Chen, Yankai Lin, Yaliang Li, Bolin Ding, Jingren Zhou, Jun Wang, and Ji-Rong Wen. GenSim: A general social simulation platform with large language model based agents. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (System Demonstrations), pp. 143-150. Association for Computational Linguistics, 2025.
Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing Ilm agents via stepwise progress attribution. arXiv preprint arXiv:2505.20732, 2025a.
Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024a.
Lei Wang, Heyang Gao, Xiaohe Bo, Xu Chen, and Ji-Rong Wen. Yulan-onesim: Towards the next generation of social simulator with large language models. In Workshop on Scaling Enviromments for Agents, 2025b. URL /ttps://openrevier http:eewwwwwww.eeeeeeeeeeeeeeeee wttp:///eoormm/e/cCCooCDDDD
Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. Learning from failure: Integrating negative examples when fine-tuning large language models as agents. arXiv preprint arXiv:2402.11655, 2024b.
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sain- bayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning, arXiv preprint arXiv:2508. 19229, 2025.
Weimin Xiong, Yifan Song. Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! LLM agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1556-1572. Association for Computational Linguistics, 2024.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826-23854, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20077 2022.
12

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human- labeled dataset for complex and cross-domain semantic parsing and text-to-SQL. task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 3911-3921. Association for Computational Linguisties, 2018
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. XrXiv preprint Stti::30080012253 2023.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun- ing: Enabling generalized agent abilities for 1lms. arXiv preprint arXiv:2310. 12800.222.,... 2023.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang. Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for Umm: A survey, arXiv preprint arXiv:2509.022 02547, 2025a.
Qiyuan Zhang. Fuyuan Lyu, Zexu Sun, Lei Wang. Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235. 2025b.
Boyuan Zheng, Boyu Gou, Jibyung Kil, Huan Sun, and Yu Su. GPT-4V(ision) is a generalist web agent, grounded. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 61349 61385. PMLR, 21-27 Jul 2024. URL
mlr 2tfrsss//235/2een22255 html
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS
In the preparation of this manuscript and the development of our codebase, we utilized LLMs to assist in two primary capacities. We detail these uses below for full transparency:
Writing and Editing Assistance: We used LLM writing assistant to improve the grammar, phrasing, and overall clarity of the manuscript. The authors directed all core ideas and claims, and are fully responsible for the final wording, arguments, and content presented in this paper.
Code Implementation Support: During the implementation of our experimental framework, an LLM served as a coding assistant for tasks such as generating basic code and aiding in debugging. The authors designed the overall software architecture, and all LLM-generated code was carefully reviewed and adapted by the authors to ensure its correctness and functionality.
B BROADER IMPACTS AND LIMITATIONS
Broader Impacts. Our work on Hicrarchical Preference Learning (HPL) presents a more efficient and effective paradigm for training capable autonomous agents from fixed datasets after one-shot exploration phase. Technologically, this reduces the reliance on costly and often unsafe online exploration, making the development of sophisticated LLM agents more accessible and sustainable. The ability to learn from the compositional structure of tasks via action groups could lead to more reliable and predictable agents in real-world applications, from personalized assistants to complex workflow automation. On a societal level, the deployment of more competent agents can enhance productivity and assist with complex decision-making. However, as agent capabilities advance, ensuring their alignment with human values becomes paramount. The preferences learned from static datasets must be carefully curated to prevent the codification biases or unintended behaviors. Furthermore, the increasing autonomy of such agents necessitates continued research into robust safety protocols, transparency, and ethical oversight to mitigate potential misuse and ensure these powerful technologies are developed responsibly for the benefit of society.
Limitations. While our HPI. framework demonstrates significant performance gains, we acknowl- edge several limitations that offer avenues for future research. Firstly, the effectiveness of HPI., particularly the semantic variant, is contingent though our experiments show that simple heuristics are beneficial, a suboptimal segmentation could yield less meaningful sub-tasks and hinder the learning process. Secondly, our dual-layer curricu- lum, while effective, introduces a new set of hyperparameters for defining task complexity and sam- ple distinguishabiliiyy, and the optimal configuration of this curriculum may be domain-specifie and require careful tuning. Finally, our current work relies on a powerful teacher model for generating preference data, which means the agent's policy may inherit biases from the teacher. Future search could explore more robust, self-supervised segmentation techniques and investigate methods for learning curricula directly from the data.
the quality of the action group segmentation. Al-
C DISCUSSION
In this section, we discuss why group-level objectives are effective for long-horizon LLM agents. Building on both our theoretical analysis and empirical ablations, we highlight two complementary perspectives: (i) segmentation introduces useful structural prior over sub-tasks, and (ii) group- level objectives have favorable statistical properties in the finite-data, limited-rollout regime.
Segmentation as a structural prior. Group-level supervision implicitly encodes prior knowledge about Long-horizon task structure: instead of treating every individual action as equally important learning unit, focuses the objective on short sub-trajectories that are more likely to correspond to meaningful sub-tasks. Although HPL reuses exactly the same interaction data as trajectory- and step- level baselines, reorganizing this data into action groups changes where supervision is concentrated. Empirical results show that the semantic variant using stronger segmentation prior achieves the best overall performance, while simpler segmenters still improve over both trajectory and step-level DPO, despite having almost no semantic information about sub-task boundaries.
14

---

Published as a conference paper at ICLR 2026
Statistical properties in the finite-data regime. From an asymptotic perspective, sufficiently expressive step-level DPO objective could in principle represent the same long-horizon credit as- signment group-level objective. Our focus. however, is the practically relevant regime where both the dataset size and the MC rollout budget per state are limited. Under this regime, step-level MC estimates provide localized but noisy supervision: each decision point is updated from only small number of rollouts. Group-level objectives alleviate this by aggregating multiple actions into single supervised unit. With the same overall rollout budget, group-level rewards effectively average out noise over longer sub- sub-trajectorie and directly evaluate the joint contribution of several actions formalizes this intuition through bias varianeee analysis, and our
to task success. Proposition [] ablation studies further show that removing the group-level loss significantly degrades performance.
D EXPERIMENTAL DETAILS
D.1 ENVIRONMENTS AND TASKS
We evaluate our framework on three challenging benchmarks that require long-horizon, multi-step reasoning and interaction, representing a diverse set of agent tasks.
D.1.1 ALFWORLD
ALFWorld (Shridhar et al. 2021> is an embodied agent benchmark set in simulated household en- vironments, uniquely designed to align abstract, text-based interactions with visually rich, embod- ied world. Agents must parse natural language instructions and perform a sequence of high-level actions (e.g., goto, take, clean) to complete common household tasks, such as "put clean tomato on the sidetable". The benchmark is structured into six distinct and compositional sub-task categories: Pick, Clean, Heat, Cool, Exami ne, and Pick2. The dataset contains 3,553 sce- narios for training. For evaluation, the test set is divided into a seen set of 140 scenarios to assess in-distribution generalization within familiar room layouts. and more challenging unseen set of 134 scenarios with novel task instances in unobserved environments to evaluate out-of out-of-distribution generalization, For all tasks, the maximum number of interaction turns is set to 30.
D.1.2 WERSHOP
WebShop (Yao et al. 2022) is a web-based simulation environment that tasks agents with navigat- ing a realistic e-commerce website to find and purchase a product that matches a given instruction. The environment is notable for its scale and realism, featuring 1.18 million real-world products and over 12,000 crowd-sourced, natural language instructions. To succeed, an agent must interact with multiple types of web pages, including search, results, and item-detail pages, by issuing high-level actions, primarily "search [QUERY]" or "click [BUTTON]] "" This process requires a combina- tion of information retrieval skills formulate effective search queries and long-horizon planning to navigate the site, compare items, and select the correct product options. The final reward is automat- ically calculated based the purchased item and the user's instruction. The benchmark includes 200 test tasks. For all tasks, the maximum number of interaction turns is set 10.
heuristic that measures the attribute, option, and price match between
1.1.3 INTERCODE-SQL.
InterCode-SQL (Yang et al. 2023) is an interactive environment designed to benchmark agent ca- pabilities in complex data querying tasks. Within safe and reproducible Docker container, the agent interacts with a live MySQL database by iteratively writing and executing SQL queries to answer natural language questions. The environment built upon the challenging, cross-domain Spider dataset (Yu et al.] 2018), which requires the agent to understand complex database schemas and formulate queries that often involve multiple tables and JOIN operations. A key feature of this benchmark is its interactive nature; after each query execution, the agent receives real-world feed- back, such as query results or error messages, which it must use to debug and refine its subsequent actions. A final reward is calculated based on the Intersection over Union (IoU) between the agent's 'ss submitted query results and the ground groumd-truth records. The benchmark consists of 200 test tasks. For all tasks, the maximum number of interaction turs is set to 10.
15

---

Published as a conference paper at ICLR 2026
D.2 MODELS AND IMPLEMENTATION DETAILS
Our methodology relies on a dataset of expert trajectories and generating a corresponding set of suboptimal trajectories for creating preference pairs.
Expert Trajectories (Pexper). Following prior work (Xiong et al. 2024), we generate our initial set of expert trajectories by prompting a powerful teacher model (GPT-40) to solve tasks in each environment using a ReAct-style reasoning process. We then filter these generated trajectories, retaining only those that achieve high outcome reward (e.g., success score of 1.0 in ALFWorld and InterCode-SQL, or >0.8 in WebShop) to form our final expert dataset, Dexpert. This dataset is used for the initial behavior cloning stage.
Models and Training. We utilize (wee2.55-1.5B-Insstte and Qwen2.55-BB-nnttrenn as the back- bone models for all experiments. For all DPO-based methods, including our HPL, the reference policy Tee is the SFT agent trained on Dexpert- During the behavior cloning phase, models are trained for 3 epochs using the AdamW optimizer with cosine learning rate schedule, peaking at 1e-5. All experiments were conducted on 8 NVIDIA A800 80G GPUs.
D.3 BASELINES
We compare HPL against a suite of strong baselines representing different alignment strategies, ranging from standard imitation learning to satt--ofthhearr preference optimization methods.
SFT: The standard Supervised Fine-Tuning approach, where the model is trained only on the expert trajectories Dexpen- This serves as our foundational base model and represents the standard behavior cloning paradigm from which all other preference-based methods initialized.
RFT (Yuan et al. 2023): Rejection sampling Fine-Tuning is an enhanced fine-tuning method that uses rejection sampling to augment the expert dataset with newly generated successful trajectories. By enriching the training data with more diverse set of successful paths, RFT serves strong imitation learning baseline that tests the performance limits learning solely from positive demonstrations.
ETO (Song et al. 2024): A trajectory-level DPO baseline that learns by contrasting full successful and failed trajectories. This method represents the coarsest end of the preference learning spectrum, providing holistic signal based the final outcome. While powerful, this approach faces challenges in credit assignment, as it can struggle to pinpoint specific errors within long action sequences.
IPR (Xiong et al.) 2024): A state-of-ther -oe-te--rtt-ttttt process supervision method that performs step- level DPO using rewards estimated from Monte Carlo rollouts. IPR operates the finest granularity, providing precise, localized feedback on individual actions. However, this step-level focus may overlook the synergistic value of multi-step sub-tasks.
D.4 HYPERPARAMETERS
Table and Table show the hyperparameters for SFT and Group-DPO stage respectively across three agent benchmarks. All experiments were conducted on NVIDIA A800 80G GPUs.
D.5 RESOURCE COMPARISON
We report in Table 5 a resource comparison of SFT, ETO, IPR, and the HPL variants on ALFWorld with Qwen2.5-1.5 -.5.-tuuttntttt including whether external powerful LLM is used, the number of LLM calls, and the generation/training time.
During data generation, adopt the same parallel sampling implementation provided by the ETO and IPR codebases to accelerate environment interaction. The actual generation time depends pri- marily the degree of parallelism in environment rollouts, well the time required reset the
16

---

Published as a conference paper at ICLR 2026
Table Hyperparamenters for SFT stage across three agent benchmarks.
Benchmark
ALFWorld
WebShop
InterCode-SQI.
Batch size
32
Learning rate
le-5
1e-5
le-5
Optimizer
AdamW
AdamW
Adam W
LR scheduler
cosine
cosine
cosine
Warmup ratio
0.1
0.1
0.1
Max epochs
3
Max seq length
6000
6000
6000
DeepSpeed Zero stage
Gradient accumulation steps
Table 4: Hyperparamenters for Group-DPO stage across three agent benchmarks.
Benchmark
ALFWorld
WebShop
InterCode-SQL.
Batch size
32
32
32
Learning rate
3e-6
le-6
le-6
3
0.3
0.3
0.3
Optimizer
AdamW
A
AdamW
LR scheduler
cosine
cosine
cosine
Warmup ratio
0.1
0.1
0.1
Max epochs
1
1
Max seq length
6000
6000
6000
DeepSpeed Zero stage
3
3
Group length (L) threshold for curriculum
(0,3.6)
(0,2,4)
(0,2,4)
Difficulty (AR) threshold for curriculum
(1.0, 0.7, 0.4)
(1.0, 0.7, 0.4)
(1.0, 0.7, 0.4)
environment and progress it to the desired states; the LLM call latency is not the dominant factor. The reported training times are based on real using 4 NVIDIA A800 80G GPUs.
E ILLUSTRATIVE EXAMPLE OF GROUP REWARD ESTIMATION
In this section, provide concrete walkthrough of how the Monte Carlo (MC) rollout mechanism estimates the expected outcome reward for a specific action group, as defined in Equation 2
Consider example from the ALFWorld benchmark:
Task Instruction: put clean apple in fridge
Current Context (c:): The agent is located at the sinkbasin and is holding a dirty apple 1.
Candidate Action Group (G:): The policy generates coherent sequence of actions in- tended complete the "cleaning" sub-task:
G; [go to sinkbasin 1, clean apple 1 with sinkbasin 1]
To estimate the reward f(G:), we first execute G,. The simulation state transitions point where the agent is holding a clean apple at the sink. From this state, we perform M = stochastic rollouts using the reference policy Øref see the task can be successfully completed.
Rollout 1 (T(1)); The agent successfully navigates to the fridge, opens it, and places the apple inside.
Outcome Reward (1)( 1.0 (Success).
Rollout 2 (2)): The agent navigates to the fridge but attempts place the apple without opening the fridge first. It fails to recover within the step limit.
Outcome Reward (2) 0.0 (Failure).
17

---

Published as a conference paper at ICLR 2026
Table 5: Resource comparison SFT. ETO, IPR, and HPL variants on the ALFWorld benchmark with Qwen2.5-1.5B-Insttee
Method
External powerful LLM
LLM calls
Gen time
Train time
SET
18min
ETO
30,000
1h 7min
13min
IPR
X
750,700 (step-level part)
6h 13min (step-level part)
26min
HPL. (Fixed-N(3))
222.00000 (group-level part)
3h 35min (group-lew level part)
25min
(Fixed-K(3))
(group-level part)
(group-level part)
26min
HPL (Uncertainty)
194,000 (group-level part)
3h 47min (group-Jevel purt)
28min
HPL. (Semantic)
221 ,000 (group-level part)
3h 21min (group-level part)
26min
Rollout 3 (r())); The agent navigates to the fridge, opens it, and successfully places the apple.
+ Outcome Reward R()(5)) 1.0 (Success).
Finally, the estimated reward for group G, is calculated as the average of these outcomes:
F(G;)
+.0 1.0) s 0.67.
This value f(G.) quality group (e.g., one that failed to clean the apple), the difference AR determines the sample difficulty for our curriculum scheduler.
0.67 serves as the quality label for this action group. If paired with lower-
F DETAILS OF DUAL-LAYER CURRICULUM SCHEDULER
In this section, we provide the algorithmic implementation of our curriculum scheduler and a visual illustration of the phase- phase-wise training progression.
Algorithm I outlines the logic for partitioning the preference data into the 3 3 grid based on Group Length (L) and Sample Difficulty (AR), and selecting the active data subsets for the current training phase 8. We utilize the hyperparameters specified in Appendix D.4
Algorithm HPL Dual-Layer Curriculum Scheduler
1: Input: Dataset Dau, Phase E 21, Thresholds T2, Tan-
2: Output: Training subset (a)
train
3: Step 1: Data Partitioning
4: Partition Dau into 3 buckets B where 1, E 21,
5: For each sample x Dau, assign Bi,a based
LLength Level 1: Determined group length Tz (Short + Long 3).
DDifficulty Level d: Determined by reward gap Tan (Fasy Hard 3).
6: Step 2: Phase-based Selection
7: Define the set of active bucket indices z(s) for current phase s:
8: if 1 then
9: I(1) + {(1, 1)} (Phase 1: Foundational (Short & Fasy))
10: else if
then
11: ((2) - {(1, 1), (1, 2), (2,1) 1)) (Phase 2: Expansion]
12: clse
13: T(a < [[1, 1 1, 3) (Phase 3: Full Scale)
14: end if
15: return D(a) Ul Bra
G ANALYSIS
We now analyze the bias and variance of group-level DPO loss. Consider an MDP with discount factor y E [0, 1). A trajectory 7 of horizon T is denoted = (81, a1.7..1......... ST, or, rT). ). Let Ther be reference policy strictly positive on every state-action pair. For any sequence u, we define its discounted return with respect to the (unknown) optimal value function V** by
R(u)= Σ KEu
to + (Sttgt//)))
(10)
18

---

Published as a conference paper at ICLR 2026
where to is the starting time index of and |t] is its length. The true preference probability that ww is preferred to in is modelled by the Bradley-Terry law
| log " (uw)
(Te(( log x T'((w)
1)
P(u u) $log
((AA),
(11)
where a(2) I+e and B 0 is a fixed inverse- tempertteme... The population DPO loss for generic distribution j over pairs (Mu,, u4) is
L ==== En logo)
(12)
Given a dataset D of N i..dd trajectories, each method forms its own empirical distribution μl, and minimizes
C. (O) = [D.]
1
Σ
(Bgo)
(13)
where Ag log decomposition
No (u( [nr(
log
To [ West an) and E {traj, step, We adopt the standard risk
Risk = B[ - L
(14)
Bias )2 + Var(C.).
where the expectation E|:] is taken over the sampling distribution of D. and Bias [.) = [[[c] L"
(15)
Var(C.) = IE (C. - E[C,))]
Proposition 1 (Bias-variance trade-off of group-level DPO loss). Let T denote the trajectory length, E 1) the discount factor, and Rmar the maximum reward. Let Lunj. Cnep. and ggoup (k) denote the empirical losses of trajectory-lever step-level, and group-level DPO with group length k < T, respectively. Then there exists a constant C >> O depending only on (7, Fres) such that for every
e E (0,1) 1) the choice k(e) logg
(1-7/1 Rus.
satisfies
Biass(Cymup (A:) S min (Bias(Cmug) Bias( +
(16)
C (log(L/e)
Var(CenmupH)) (k:) 5
min Var(Cnaa), Var(Lsmep]:
(17)
Proof. First, we analyze the bias of the three losses. We compare the population losses induced by the three sampling schemes. Recall that the logistic loss is I-Lipschitz, that is, for any scalar difference 2,
log or (2) - log (<3)) -
(18)
Hence the bias of an empirical loss L. is controlled by
[[C.] L S SE,, 1 I]
(19)
which is governed by the error in the return difference induced by the length of the comparison unit. We now analyze the three cases: trajectory, step, and group. Trajectory-level DPO compares entire trajectories with no truncation. Hence Bias(Lrraj) = O. Step-level DPO compares suffixes from time T. Following the implementation in Xiong et al. (2024), these suffixes are not truncated either. Hence Bias( Catep) = O. For group-level DPO, the unit has fixed length k < T. For clarity, 7
define R (1) = -' -r,, According to Bellman equation,
t+k-1
(((1)
= 7t (1++))).
(20)
Consider a group pair (Gw, Gi) starting from the same state st. the true return difference should be 8imj R" (t) Rt (t), while group-level DPO uses ógroup = R(G.,) R(G)). Substituting
Equation 20 into dnaj we get
Sujj
Σ
" + +
Σ
, r; + + (1)
(21)
i=t.eEG..
(R(Gww - (R(G)) + (R (t k)-
RT (t + R))
(22)
Egruup
+ * + kk) Rj (t R))).
(23)
19

---

Published as a conference paper at ICLR 2026
Hence the error in the return difference is
2fmax
Ogroupl = (1 k) Rt+ s
(24)
1-
where the last step follows from the fact that the absolute value of any finite-horizon discounted sum is bounded by
T
T-(++R)
R
Σ
S
Rmax
Σ
(25)
i=++
Therefore, for a single preference pair, the error of group-level DPO loss satisfies
2Bmax
log (3A log o (3Smj))) S B Agouu
Arroc
S 3. 1
(26)
Taking the expectation, we get
I Bias (Curj) + 2.R
Bias Cgroup)
7
(27)
By choosing k(e) log
(1-9]e
we obtain
2
Bias Cgropp (k)) Se min (Faj)) Bias (Sstep)
(28)
Next, we analyze the variance of the three losses. We derive element-wiss bounds on the variance of the empirical loss
C.
((Ao), (E
Σ
((Ao) == - log or (33o),
(29)
D
for {traj, step, group). All samples are generated from N i.i.d. trajectories.
For trajectory-level DPO, each trajectory contributes exactly one preference pair. The total number of samples [Pugj] N. The N pairs are i.i.d., hence the covariance terms in the variance vanish:
N
Var(Crraj)
Var [((A
Eraii
(30)
N2
N
where Eiraj == Var[((Ag)] under the trajectory-level sampling distribution.
For step-level DPO, from one trajectory we extract T consecutive suffixes (u,) with de Ss,)..,,TT). The total number of samples is [D [Dstopl = NT. However, the T samples inside identical transitions.
one trajectory are highly overlapped. Sequence u, and u,+1 share T Therefore the covariance part in covariance is non-zero and large.
2Raux, we have S f(De) S Lmax = log(1 + 2.
Since Ao <
For any < S T, let (number of shared steps). The Cauchy-Schwand inequality gives Cov (.) S L2 L Summing over ordered pairs in one trajectory, we get
1-7
T-1
Cov(le, S L2
[T-
7 L2 max (1 - 7
(31)
max
ISKSST
We now consider total variance across N trajectories. Each trajectory contributes T samples, and samples from different trajectories are i.i.d. Hence,
Var(
N.T. Var(6,)+N.
Σ Cov(Lerls)
(32)
(NT)²
IStCAST
I mmax
2T2 max
I22
max
1
27
(33)
NT
NT
2)
NT
(1 1 2
20

---

Published as a conference paper at ICLR 2026
Var(Cstep) is O( NT) but with constant that does not degrade with T.
For group-level DPO, we extract M [T/k] non-overlapping groups of length The total num- ber of samples is [P group NM. Between-trajectory samples are ii.d, while within-trajectory samples are independent by construction. Therefore, the covariance terms variance are zero, and
Var Var(Cgroup(H)) (k)) - (NM) NM Var[{(Ao)] = NM group
(34)
where Egroup = Var(<(Ao) under the group-level distribution. Since a sub-trajectory has smaller variance than the full trajectory, Egroup I Etraj- Inserting M T/k 1 into Equation 34 obtain
k
Var Var(C (1)) I T N
Duraj = Var(Liraj)
(35)
T
An identical comparision with step-DPO gives
Var (aroupr S Ck Var(Lstop))
2br( (lr((sstop)
C= tr(2 tr(Zgroup)
2 1.
(36)
The constant C' depends only y and Rmaxx and is independent of T,k, and N.
With k(e)
log,
(1-7le 2.
([log(1/e)), Equation 35 and 36 yield
C'log(1/c)
Varongg (k)) <
min mnin(Var(Lunn (Var(Lmj), Var(Luatp))
(37)
T
which is the variance bound claimed the proposition.
H ADDITIONAL EXPERIMENTS
H.1 SUB-TASK PERFORMANCE ON ALFWORLD
To provide a more fine-grained analysis of our method's capabilities, we present a detailed break- down of success rates the six distinct sub-task types in ALFWorld for both seen (Table (6) and unseen (Table 7 sets. The largest performance gains are often observed in the complex sub-tasks, such Examine and Pick2, which require longer reasoning chains.
H.2 CASE STUDY
H.2.1 ALFWORLD
21

---

Published as a conference paper at ICLR 2026
Table 6: Sub-task success rate (%) comparison on the ALFWorld seen set.
Sub-task
Models
Overall
Pick
Clean
Heat
Cool
Examine
Pick2
Owen2.5-1.5B-Instree
8.57
0.00
0.00
0.00
0.00
0.00
2.14
SFT
88.57
44.44
62.50
72.00
46.15
41.67
62.14
RFT
Yuan al.
2023
88.57
40.74
62.50
72.00
46.15
41.67
61.43
ETO
(ong et al.
2024
91.43
40.74
62.50
72.00
46.15
41.67
62.14
IPR
(Xiong al.
2024
94.29
44.44
68.75
72.00
46.15
41.67
64.29
HPL
(Fixed- (3))
97.14
48.15
75.00
72.00
46.15
50.00
67.86
HPL
(Fixed-K (3))
94.29
51.85
75.00
80.00
46.15
50.00
69.29
HPL
(Uncertainty)
94.29
55.56
87.50
80.00
61.54
50.00
72.86
HPL (Semantic)
97.14
51.85
87.50
80.00
61.54
41.67
71.43
Qwen2.5-7B-Instruct
74.29
29.63
37.50
32.00
15.38
16.67
38.57
SFT
94.29
51.85
75.00
72.00
53,85
41.67
67.14
RFT
Yuan ct al.
2023
97.14
55.56
75.00
80.00
61.54
54.17
72.86
ETO Song et 2024
94.29
55.56
87.50
72.00
53.85
45.83
70.00
IPR
Xiong et al.
2024
94.29
59.26
87.50
80.00
61.54
45.83
72.86
HPL
(Fixed-N (3))
97.14
66.67
87.50
88.00
76.92
50.00
78.57
HPL
(Fixed-K(3) (3))
100.00
88.89
93.75
92.00
84.62
66.67
88.57
HPL
(Uncertainty)
97.14
74.07
87.50
88.00
92.31
50.00
81.43
HPL (Semantic)
100.00
77.78
87.50
92.00
76.92
58.33
83.57
Table 7: Sub-task success rate (%) comparison on the ALFWorld unseen set.
Sub-task
Models
Overall
Pick
Clean
Heat
Cool
Examine
Pick2
Owen2.5-1.5B-Instrrue
0.00
0.00
0.00
0.00
0.00
0.00
0.00
SFT
70,83
64.52
60.87
76.19
27.78
35,29
58.21
RFT
Yuan al.
2023
79.17
61.29
65.22
76.19
33.33
35.29
60.45
ETO
Song et al.
2024
83.33
64.52
65.22
80.95
38.89
41.18
64.18
IPR
Xiong et al.
2024
83.33
64.52
65.22
80.95
38.89
47.06
64.93
HPL
[Fixed-N (3))
87.50
80.65
73.91
85.71
44.44
52.94
73.13
HPL
(Fixed-K(3) (3))
83.33
61.29
69.57
80.95
33.33
41.18
63.43
HPL
(Uncertainty)
83.33
61.29
69.57
85.71
33.33
35.29
63,43
HPL
(Semantic)
87.50
80.65
78.26
76.19
44.44
52.94
72.39
wwen2.5-7BIInstruct
62.50
54.84
52.17
52.38
16.67
17.65
45.52
SFT
91.67
80.65
78.26
85.71
50.00
58.82
76.12
RFT
Yuan et al.
2023
79.17
74.19
78.26
76.19
44.44
52.94
69.40
ETO
Song et
2024
95.83
74.19
78.26
80.95
55.56
70.59
76.87
IPR
Xiong et al.
2024
95.83
77.42
73.91
85.71
61.11
64.71
77.61
HPL
(Fixed-N (3)
95.83
77.42
73.91
80.95
55.56
64.71
76.12
HPL
(Fixed-K(3) )3))
95.83
80.65
69.57
80.95
55.56
70.59
76.87
HPL
(Uncertainty)
100.00
87.10
82.61
80.95
66.67
76.47
83.58
HPL (Semantic)
100.00
87.10
82.61
90.48
77.78
76.47
86.57
22

---

stove- Published as a conference paper at ICLR 2026
Case Study of ALFWorld
IPR
HPL
Task Instruction:
You are in the middle of room. Looking quickly around you, you see cabinet 6, cabinet
You are in the middle of room. Looking quickly around you, you see cabinet 6, cabinet 5, cabinet 4, cabinet a countertop 3, countertop 2, countertop 3, a cabinet 2, cabinet 1, a coffeemachine 1, countertop
Task Instruction:
cabinet 4, a cabinet 3, cabinet 2, cabinet 1, coffeemachine 1.
1, drawer 3, drawer 2, drawer I, fridge garbagecan
countertop 1, a drawer 3, a drawer 2, a drawer a fridge a garbagecan 1, microwave shelf 3, shelf 2, a shelf 1, sinkbasin 1, a stoveburner stoveburner
countertop
microwave I, shelf 3, a shelf 2, shelf 1, a sinkbasin 1, stoveburner
4, a stoveburner 3, a stoveburner 2, a stove- burner I, and a toaster 1.
4, stoveburner
Your task to: put cool mug in cabinet. Thought: need to find mug to put in the cabinet. The task description doesn't it specify where the mug is, I should look around the room to locate one.
burner 1, and toaster 1.
Your task is to: put a cool mug cabinet. Thought: I need to find mug put in the cabinet. The sink basin a common place where mugs found after being used or washed.
Action: go to toaster
Observation: On the toaster 1, you see noth-
Action: go sinkbasin
ing.
Observation: On the sinkbasin 1, you see a cup 1, dishsponge 2, plate 2, and potato
Thought: The toaster is not where mugs would typically be found, but it's good check all areas systematically. There's 's noth- ing on the toaster, I'll continue searching.
1.
Thought: There no mug the sink basin, but there are other items here. I should check the shelves as they often store mugs.
Action: go to stoveburner I Observation: On the stoveburner 1, you see
Action:
go shelf
pot 1.
Observation: On the shelf 1, you see cred- itcard 3.
(search for the mug)
(search for the mug)
Thought: Cabinet has a vase, but no mug.
I'll check cabinet 4.
Thought: Cabinet 3 has a vase but no mug. I'll proceed to the next cabinet.
Action:
go to cabinet
Observation: On the cabinet 4, you see a mug 2.
Action: go cabinet
Observation: On the cabinet 4, you see mug 2.
Thought: Cabinet has a mug. This could be the mug I need to put in the cabinet. I'll take and then place in the cabinet.
Thought: found a mug cabinet 4. I need to take it can put it in the cooler in- structed.
Action: take mug 2 from cabinet
Observation: You pick up the mug from the cabinet 4.
Action: take mug 2 from cabinet 4 Observation: You pick up the mug 2 from the cabinet 4.
Thought: With the mug in hand, I need to return to cabinet to place the mug inside.
Thought: Now that have the mug. I need cool it. The fridge is the best place to do this.
Action: go to cabinet I
Observation: On the cabinet 1, you see a peppershaker 1, soapbottle and vase
Action: go fridge 1
Observation: The fridge is closed. Thought: The fridge is closed. need to open it to cool the mug.
Thought: To complete the task, need put the mug into the cabinet. Action: put mug 2 in/on cabinet 1 Observation: You put the mug in/on the cabinet 1.
Action: cool mug 2 with fridge
Observation: You cool the mug 2 using the fridge 1.
23

---

Published as a conference paper at ICLR 2026
# The agent mistakenly believes that the task has been completed.
Thought: The mug is now cool. I should return to the cabinet to complete the task.
Thought: The task is now complete, and should prepare move to the next task task set. need to ensure that I approach each task with clarity and precision, adhering the given instructions and using the appropri- tools and resources.
Action: go cabinet
Observation: On the cabinet I, you see a peppershaker 1, soapbottle 1, and vase 2. Thought: I am back at cabinet 1. need to put the cool mug in the cabinet per the task instructions.
Action: prepare for next task Observation: Nothing happens.
Action: put mug in/on cabinet Reward: 1.0
(until the steps are exhausted) Reward: 0.0
H.2.2
WEBSHOP
Case Study of WebShop
IPR
HPL
Task Instruction: am looking for a gluten free, 100% vegan plant based protein shake that is soy-free, and price lower than 40.00 dollars.
Task Instruction: I am looking for gluten free, 100% vegan plant based protein shake that is soy-free, and price lower than 40.00 dollars.
Thought: I need to find plant-based protein shake that meets specific criteria including being gluten-free, 100% vegan, soy-free, and under $40. I'll start by searching with these details.
Thought: need to find plant-based protein shake that meets specific criteria including being gluten-free, 100% vegan, soy-free, and under $40. I'll start by searching with these details.
Action: search[gluten free, 100% vegan plant based protein shake that is soy-free]
Action: search [gluten free, 100% vegan plant based protein shake that is soy-free] Observation: Page 1 (Total results: 50)
Observation: Page (Total results: 50)
[SEP] Next [SEP] B08FW5Y13G [SEP]
[SEP] Next > [SEP] BO8FW5Y13G [SEP]
Orgain Bundle Chocolate and Vanilla Bean
Orgain Bundle Chocolate and Vanilla Bean
Protein Powder
[SEP] $53.90 $53.98 [SEP]
Protein Powder
[SEP] $53.98 [SEP]
B0885Y2GI4 [SEP] OWYN 100% Ve-
B0885Y2GD4 [SEP] OWYN 100% Ve-
gan Plant-Based Protein Shake, Cookies Creamless, Count, with 20g Plant Protein, ... [SEP]] $100.0 ..... (other product details)
gan Plant-Based Protein Shake, Cookies & Creamless, Count, with 20g Plant Protein, [SEP] $100.0 ..... (other product details) Thought: The first page of results doesn't 'tt seem to have any options that fit all my cri- teria. I should check the next page for more options.
Thought: The first page of results doesn't seem to have any options that all my cri-
teria. should check the page for more options.
Action: click[next >>
Action: click[next >]
Observation: Page 2 (Total results: 50)
Observation: Page 2 (Total results: 50)
[SEP] Prev [SEP] Next [SEP]
[SEP] < Prev [SEP] Next [SEP]
B07DM8SP27 [SEP] OWYN 100% Vegan
B07DM8SP27 [SEP] OWYN 100% Vegan
Plant-Based Protein Shake, Strawberry Ba-
Plant-Based Protein Shake, Strawberry Ba-
nana, 24 Pack, 20g Plant Protein [SEP]
nana, 24 Pack, 20g Plant Protein ... [SEP]
$71.18 [SEP] B08H8VJJGZ [SEP] OWYN- 100% Vegan Plant- Based Pro Elite High Pro-
$71.18 [SEP] BOSH8VJJGZ [SEP] OWYN- 100% Vegan Plant-Baa Based Pro Elite High Pro- tein Shakes Vanilla, F2 [SEP] $4.48
tein Shakes Vanilla, 12 Fl Oz [SEP] $4.48
[SEP] ..... (other product details)
[SEP] (other product details)
24

---

Published as a conference paper at ICLR 2026
Thought: On the second page, I see product called "OWYN 100% Vegan Plant-Based Protein Shake" which seems to match most of my requirements except for the price. It's also listed soy-free and vegan, and it's well below $40. I'll take a closer look this option.
Thought: On the second page, see a product called "OWYN 100% Vegan Plant-Based Protein Shake" which seems to match my requirements. It's also well within my budget $4.48. should take closer look at this option.
Action: click(b0Sh8vjjgz)
Action: click(b07dm8spp7]
Observation: < Prev [SEP] OWYN 100% Vegan Plant-Based Pro Elite High Protein Shakes Vanilla, 12 Fl Oz Dairy- Free, Gluten-Free. Soy Free, Keto Friendly, Zero Sugar, Vegetarian [SEP] Price: $4.48 [SEP] Rating: [SEP] Description [SEP] Features [SEP] Reviews [SEP] Buy Now Thought: This product has the features
Observation: < Prev [SEP] OWYN 100% Vegan Based Protein Shake, Straw- berry Banana, 24 Pack, 20g Plant Protein with Omega-3, Prebiotic supplements, Superfoods Greens Blend, Gluten-Free, Soy-Free, Non-GMO [SEP] Price: $71.18
[SEP] Rating: [SEP] Description [SEP] Features [SEP] Reviews [SEP] Buy Now Thought: This product has a high price compared to my budget, and although it's described as vegan and soy-free, the price is above $40. Since I'm looking for something within my budget, might reconsider look for another option that fits all my criteria including the price limit.
I'm looking for.
Action: click[buy now] Reward: 1.0
Action: click[buy now]
Reward: 0.5
H.2.3 INTERCODE-SQL
Case Study of LnterCode-SQL
IPR
HPL
Task Instruction: What is the money rank of the tallest poker player?
Task Instruction: What the money rank of the tallest poker player?
Thought: I need to check what tables are
available to use.
Thought: need to check what tables are available to use.
Action: SHOW TABLES
Observation: ((peoplee..) .), ('poker-playe .)]
Action: SHOW TABLES
Thought: There are two tables available.
Observation: [('people',) "poker-player .)] Thought: The table 'poker player' might have information about the height and money rank of the players. should check the struc- of this table.
'people' and 'poker player'. Since the ques-
tion is asking about the money rank of the tallest poker player, should check there is a table related poker players.
Action: DESC poker.player
Action: DESC poker-pll saaeer
Observation:
PPPokerPPlayer ID', 'int',
Observation:
[('Poker Player ID', 'int', 'NO', 'PRI,, None, 'auto ciattoinccemeet))) ) ("Peo-
'NO', 'PRI', None, 'auto (nttoinncemen")) ), ("Peo-
ple ID', 'int', 'YES', 'MUL; None, "), ('Fi-
ple ID', 'int", 'YES', 'MUL', None, '), ('Fi- nal Table Made', 'double', 'YES', None, "), ("Best_Finish,, 'double'. 'YES' " None,
nal Table Made', 'double', 'YES", None,
"), ('Best_Finish", 'double', 'YES', ". None,
"), ''MonyyRankk' 'double', 'YES',
"), "Money_Ranks 'double', 'YES',
None, "), ('Earnings', 'double', 'YES', None, "), ("Earnings', 'double", 'YES', None, ""]]
None, ")]
25

---

Published as a conference paper at ICLR 2026
Thought: The 'paker,player' table con- tains information about poker players, including their money rank. To find the money rank of the tallest poker player, I need join the 'people' table with the "poker_ booker_player table on the 'PeopleII JD'' column. Then, I can order the results based on height and limit the result to I to get the tallest poker player's money rank.
Thought: The 'poker player' table con- tains columns for money rank and height. can write SQL query to find the money rank of the tallest poker player.
Action:
SELECT Joone__Rant FROM poker-player
ORDER BY Height DESC LIMIT
Action:
Observation: Error executing query: Un- known column 'Height' in 'order clause'. Thought: Oops, it seems like there is no column named 'Height' 'poker-played table. I should check the structure the 'people" table to see if has the height information.
SELECT Money Rank FROM people
JOIN poker.player ON
the
people.People...
poker-player. Pta.yer.PooeerPPaee...
ORDER BY height
LIMIT 1
Action: DESC people
Observation: [(141.0.)]
Observation: [('People ID', 'int', 'NO', 'PRI" None, 'auto increment'), ("National- ity', 'text', 'YES' " None, "), ('Name', 'text', 'YES', None, "), (Birth_Date" 'text', 'YES',
Thought: The query executed successfully and returned single number, which is the money rank of the tallest poker player. This should be the answer the question.
None, "), ('Height', 'double', 'YES', ", None, "'] Thought: The 'people' table does have column for height. I can join the 'poker player' and 'people' tables get the money rank of the tallest poker player.
Action: submit Reward: 0.0
Action:
SELECT p ..ooey Rank
FROM poker plaeerr p
JOIN people pe ON P.People.ID
= pe.
People.II
ORDER BY pe. Height DESC
LIMIT
Observation: [(68.0.)
Thought: The query executed successfully and returned the money rank of the tallest poker player as 68. This should be the answer to the question.
Action: submit
Reward: 1.0
I PROMPTS
I.1 ALFWORLD
Instruction Prompt for ALFWorld
Interact with household to solve task. Imagine you intelligent agent in household environment and your target is to perform actions to complete the task goal. At the beginning of your interactions, you will be given the detailed description of the current environment and your goal to accomplish.
26

---

Published as a conference paper at ICLR 2026
For each of your tum, you will be given the observation of the last turn. You should first think about the current condition and plan for your future actions, and then output your action in this turn. Your output must strictly follow this format:"Thour your thoughts. AAction: your next action"..
The available actions are:
1. go to {recep} take {obj} from (recep)
3. put {obj) in/on {recep)
open (recep)
5. close {recep}
toggle {obj} {recep}
7. clean {obj) with (recep)
heat {obj} with (recep}
9. cool {obj} with (recep)
where {obj} and (recep) correspond to objects and receptacles.
After your each turn, the environment will give you immediate feedback based on which you plan your next few steps. if the envrionment output "Nothing happened", that means the previous action is invalid and you should try more options.
Your response should use the following format:
Thought: <your thoughts>> Action: youur next action
Semantic Grouping Prompt for ALFWorld
I need you to help me divide the trajectory of an agent's 's interaction with the environment into multiple action groups based on semantic relevance.
Below is interaction trajectory, which contains the environment description received by the agent and the sequence of actions performed: [trajectory]
Please divide the action sequence in this trajectory into multiple semantically related groups, each group represents a set of actions to complete a sub-goal or sub-task. Please follow the following principles when dividing:
1. Actions in the same group should be semantically closely related and complete clear subtask together
2. When the purpose of an action changes, it should be divided into new group
For each group, briefly describe the common goal of the group of actions
Please use the following format to return the results:
<action groups>
<action.groups>>>>>
Group (action index: 0-2): Find the target item
Action 0: go to toiletpaperhanger
Action 1: go toilet 1
AAction 2: take toiletpaper from toilet 1
Group 2 (action index: 3-4): Complete the main task
Action 3: go to toiletpaperhanger 1
Action 4: put toiletpaper in/on toiletpaperhanger
/action_groups.
27

---

Published as a conference paper at ICLR 2026
1.2 WEBSHOP
Instruction Prompt for WebShop
You web shopping.
I will give you instructions about what to do.
You have follow the instructions.
Every round will give you an observation and a list of available actions, you have to respond an action based on the state and instruction.
You can use search action if search is available. You can click one of the buttons in clickables.
action should be of the following structure: search[keyworde click[value]
the action is not valid, perform nothing. Keywords in search up you, but the value in click must be value in the list of available actions. Remember that your keywords in search should be carefully designed.
Your response should use the following format:
Thought: I think...
Action: click[something]
Semantic Grouping Prompt for WebShop
I need you to divide sequence of actions into groups based on semantic relevance.
possible grouping example:
Group (action index: 0-0): Initial search phase
Action 0: search[size 5 patent-beige high heel]
Group 2 (action index: 1-1): Preliminary screening and click to view product details Action 1: clic([b0ggxnyjed]]
Group 3 (action index: 2-3): Specification confirmation and detailed screening stage
Action 2: bbick(bbige-almon toe-patent leather]
Action 3: click[5]
Group 4 (action index: 4-4): Purchase decision stage Action 4: click[buy now]
Your output then should be in the following format: [[O, 0], [1, 1J, [2. 3], [4, 4]]
Below the interaction trajectory: [trajectory]
Please group the actions by their indices. Your response MUST be valid JSON array of arrays of integers, where each inner array represents a group of action indices.
Follow these rules STRICTLY:
1. Each action must belong to exactly one group.
The indices must be contiguous and cover the entire range from to
28

---

Published as a conference paper at ICLR 2026
3. The final output MUST NOT contain any text, explanations, code blocks, markdown formatting outside of the ISON array itself. should be a raw JSON string.
4. The last number in the last group MUST be {num_actions)
Example for a trajectory with 5 actions (indices 0, 2, 4): [[O, 11. [2. 3], [4, 4]]
Another valid example: [[0, 0], [1, 2), [3, 4]]
Your output must be only the JSON, like this: [[O, 1J. [2, 3], [4, 4]]
1.3 INTRRCODE-SQL.
Instruction Prompt for InterCode-SQL
You are a helpful assistant assigned with the task of problem-solving. To achieve this, you will interact with a MySQL Database system using SQL queries answer a question. At each turn, you should first provide your step-by-step thinking for solving the task. Your thought process should start with "Thought: ", for example: Thought: should write SQL query that gets the average GNP and total population from nations whose government is US territory.
After that, you have two options:
1) Interact with a mysql programming environment and receive the corresponding output. Your code should start with "Action: and should be surrounded with 'ssql ' tag, for example:
Action:
' 'ssll
SELECT AVG (GNP), SUM SUM (population)
FROM nations
WHERE government 'US Territory';
2) Directly submit the result, for example: Action: submit.
You should use this format: "Thought: your thought's nAction: \ 'sgll \n<the mysql command>> \n N2h You will receive the corresponding output for your sql command. Your output should contain only one "Action" part.
The "Action" part should be executed with mysql interpreter propose answer. Any natural language in it should be commented out.
The SQL query and submit parts not appear in your output simutaneously.
Semantic Grouping Prompt for InterCode-SQL.
I need you to divide a sequence of actions into groups based on semantic relevance.
possible grouping example:
Group (action index: 0-1): Task initialization and data structure exploration phase
Action 0: SHOW TABLES
Action 1: DESC university
29

---

Published as a conference paper at ICLR 2026
Group 2 (action index: 2-2): Query construction and execution phase
Action 2: SELECT Enrol Iment, Primary.conference FROM university
ORDER BY Founded ASC LIMIT
Group 3 (action index: 3-3): Result confirmation and submission stage
Action 3: submit
Your output then should be in the following format: [[0, 11. [2, 2), [3, 3]]
Below the interaction trajectory: {trajectory]
Please group the actions by their indices. Your response MUST be valid JSON array of arrays of integers, where each inner array represents a group of action indices.
Follow these rules STRICTLY:
1. Each action must belong to exactly one group.
The indices must be contiguous and cover the entire range from to {num actionst -
3. The final output MUST NOT contain any text, explanations, code blocks, markdown formatting outside of the JSON array itself. should be a raw JSON string.
The last number in the last group MUST be {num actions) 1.
Example for a trajectory with 5 actions (indices 2, 4): [[O, 1]. [2, 3], [4, 4]]
Another valid example: [[O, 0], [1, 21, [3, 4]]
Your output must be only the JSON, like this: [[O, 11. [2, 31, [4, 4]]
30


Reviews:
[{"review_id": "xqaSTLeZNC", "reviewer": "Reviewer_8Yvn", "summary": "This paper proposes Hierarchical Preference Learning (HPL) to address the granularity mismatch in preference-based offline alignment for long-horizon LLM agents. Conventional Direct Preference Optimization (DPO) methods operate either at the trajectory level or the step level. HPL introduces an intermediate group-level DPO that decomposes trajectories into semantically coherent action groups, paired with a dual-layer curriculum along task complexity and sample difficulty. Experiments on ALFWorld, WebShop, and InterCode-SQL demonstrate consistent improvements over existing benchmarks.", "questions": "Please see the weakness part.\n\nAt the current stage, I tend to recommend acceptance. However, I'm open to revising my final decision after the rebuttal and further discussions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work is well motivated. The paper starts with an intuitive problem, the granularity mismatch in preference optimization, and then proposes a novel solution that bridges trajectory/step-level supervisions.\n\n2. The sample code is provided. It facilitates the reviewer to verify the numerical results independently.\n\n3. The numerical performance is sound. The proposed methods beat the existing baselines by a clear margin.", "weaknesses": "1. The presentation could be improved.  While technically rich, some methodological descriptions (e.g., the Monte Carlo approach in reward estimation and curriculum scheduler) could benefit from more intuitive explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ISINzrAEE", "reviewer": "Reviewer_o3bh", "summary": "The paper aims to address the issue of granularity mismatch in preference-based alignment for LLM agents and proposes Hierarchical Preference Learning (HPL), which integrates preference signals across three levels of abstraction. The main focus of this work is providing action-group level of abstraction, which is done by semantically partitioning the trajectories into semantically coherent sub-tasks and optimizing a hierarchical DPO loss. The paper also suggests dual-layer curriculum learning strategy, which uses the information of sample discriminability and action group length. Experiments focused on long-horizon agent benchmarks show that HPL outperforms popular baselines.", "questions": "## Questions\n1. How many additional datapoints were created by action group-level data generation in Section 3.2.2? How big is the dataset compared to step-level and trajectory-level datasets?\n2. If the value of $\\gamma$ actually used for experiment was practically 1 (no discount in the trajectories), does the analysis of Proposition 1 not become incoherent?\n3. How much money in total was spent for the API call for semantic segmentation of the action groups, per each train-test run in Table 1?\n\n## Suggestions\n1. Could you provide a table which present a comparison between HPL and the presented baselines, in terms of the use of external powerful LLM, number of new generations for the training, and wall-clock time for generation and training phase?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The overall flow of the paper is well organized, allowing readers to effectively understand the paper.\n2. Figures and experiment results are well polished.\n3. Experiments such as ablation study in Figure 5 allow the readers to understand the effectiveness of components involved in the proposed algorithm.", "weaknesses": "1. The paper does not have very strong theoretical motivation or support of the proposed method.\n2. The experiment results are not having standard deviation information, which is crucial for reinforcing the credibility of the results.\n3.  The proposed method seems to be much more complicated than the compared baselines, but the potential limitation coming from this approach is not well analyzed. HPL requires expert trajectory, generations from the reference model, reward signal, and access to a powerful model such as gpt-4o when semantic segmentation is used.\n4. Some of the crucial experimental details seem to be omitted from the paper (ex. the value of $\\gamma$ for the experiments, the number of generations $M$ used to evaluate the reward in Equation 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OvZLZCS3Ku", "reviewer": "Reviewer_Q8y5", "summary": "This paper addresses the \"granularity mismatch\" in preference-based alignment for long-horizon LLM agents—trajectory-level DPO is too coarse for credit assignment, while step-level DPO is too myopic—and proposes **Hierarchical Preference Learning (HPL)** that combines trajectory-, step-, and a crucial **action-group** DPO operating on semantically coherent sub-tasks. HPL segments expert trajectories into action groups, constructs contrastive pairs, estimates group rewards, and trains with a dual-layer curriculum scheduled by group length (complexity) and reward-gap difficulty, yielding structured, sub-task credit assignment without sacrificing global stability. Experiments on ALFWorld, WebShop, and InterCode-SQL show HPL—especially with semantic segmentation—consistently outperforms SFT, RFT, ETO, and IPR; ablations highlight the group-level loss and the curriculum as the primary performance drivers.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Principled mid-granularity learning:** The action-group DPO provides a theoretically grounded bias–variance trade-off—achieving lower variance than trajectory/step DPO while adding at most ε bias with group length (k=\\Theta(\\log(1/\\epsilon))); within-trajectory sample independence further avoids the large covariance that plagues step-level pairs.   \n\n2. **Strong and well-validated gains:** Across ALFWorld, WebShop, and InterCode-SQL, HPL (especially the semantic segmentation variant) outperforms SFT/RFT/ETO/IPR—e.g., ALFWorld-unseen 86.57%—with ablations showing the group-level loss is most critical and the dual-layer curriculum (length + difficulty) delivers additional, consistent improvements.", "weaknesses": "1. **Limited novelty:** Framing DPO at an intermediate “action-group” level feels incremental—more a structured decomposition of known step/trajectory DPO than a fundamentally new alignment principle.\n\n2. **Theory under-validated:** The bias–variance claims and independence assumptions are not directly stress-tested (e.g., no gradient-variance or signal-to-noise measurements, nor controlled sweeps that confirm the predicted (k)–(\\epsilon) trade-off).\n\n3. **Sensitivity to segmentation/curriculum:** Success appears to hinge on segmentation quality and curriculum schedules; robustness to noisy grouping, alternative segmenters, and hyperparameter perturbations is insufficiently examined.\n\n4. **Scope and scalability limits:** Benchmarks and models are narrow (scripted simulators, limited agents); reliance on MC rollouts and extra segmenters raises training cost and may hinder transfer to real-web or human-preference settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z9VJo5v1mp", "reviewer": "Reviewer_9rwf", "summary": "This paper proposes Hierarchical Preference Learning (HPL), which combines preference labeling at three different granularities to improve credit assignment in DPO-based training for LLM agents. Specifically, the method constructs preference pairs at: (1) trajectory level, (2) step level, and (3) segmented sub-trajectory (group) level. Chosen examples are sampled from demonstrations while rejected examples come from a reference policy. For group-level preferences, the authors segment trajectories and use Monte Carlo value estimation with reference policy rollouts to assign preference labels. The approach incorporates curriculum learning for group-level preference optimization. The final objective combines DPO losses from all three levels plus a behavior cloning term. Experiments on three agent benchmarks demonstrate that HPL outperforms methods using only step-level or trajectory-level preferences, with ablations showing positive contributions from each granularity level.", "questions": "1. How does your approach compare to critic learning in PPO for credit assignment? If fine-grained step-level feedback is available, what is the specific advantage of segmenting trajectories into groups rather than using all step-level information directly?\n2. Is training instability the primary motivation for the curriculum scheduler in group-level preference learning? Does the high variance in trajectory lengths contribute to this instability?\n3. My hypothesis is that group-level preference learning may work better than step-level primarily because the prior knowledge injected during trajectory segmentation improves exploration efficiency, while step-level preferences could potentially learn the targeted credit assignment given sufficient data. I would like to ask for an in-depth analysis on this.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. **Technically sound approach:** The paper presents a well-designed framework that thoughtfully integrates curriculum learning and multi-granularity preference labeling to enhance agent performance.\n2. **Clear presentation:** The paper is well-written with effective visualizations. Figure 2 clearly illustrates the HPL framework and training procedure, making the methodology easy to follow.", "weaknesses": "1. **Insufficient motivation:** The paper argues that step-level DPO is \"myopic\" and cannot capture \"synergistic value\" of sub-trajectories, but this strong claim lacks adequate support. Diverse planning tasks has been successfully achieved via PPO learning with step-level rewards. The paper needs to better position itself relative to existing literature and provide more rigorous justification for why step-level preferences are fundamentally insufficient.\n2. **Contradictory problem setting:** There is conceptual confusion about whether this is truly an offline alignment method. While the authors frame this as offline preference alignment, the group-level labeling procedure involves reference model rollouts and online reward labeling. If online reward queries are required, this fundamentally cannot be offline alignment.\n3. **Incomplete experimental comparison:** Given the online reward labeling involved in HPL (per weakness 2), the experimental setup is incomplete and potentially unfair: i) Missing online RL baselines (e.g.PPO, GRPO), as the current trajectory-level and step-level DPO are purely offline methods. ii) Missing IL and IRL baselines.\n4. **Unclear role of curriculum learning:** The motivation for incorporating curriculum learning is not well established. How does curriculum learning specifically address the granularity mismatch problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)