Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents

## Abstract

Abstract Large Language Models (LLMs) as autonomous agents are increasingly tasked with solving complex, long-horizon problems.
Aligning these agents via preference-based offline methods like Direct Preference Optimization (DPO) is a promising direction, yet it faces a critical granularity mismatch.
Trajectory-level DPO provides a signal that is too coarse for precise credit assignment, while step-level DPO is often too myopic to capture the value of multi-step behaviors.
To resolve this challenge, we introduce H ierarchical P reference L earning ( HPL ), a hierarchical framework that optimizes LLM agents by leveraging preference signals at multiple, synergistic granularities.
While HPL incorporates trajectory- and step-level DPO for global and local policy stability, its core innovation lies in group-level preference optimization guided by a dual-layer curriculum. HPL first decomposes expert trajectories into semantically coherent action groups and then generates contrasting suboptimal groups to enable preference learning at a fine-grained, sub-task level.
Then, instead of treating all preference pairs equally, HPL introduces a curriculum scheduler that organizes the learning process from simple to complex.
This curriculum is structured along two axes: the group length, representing sub-task complexity, and the sample difficulty, defined by the reward gap between preferred and dispreferred action groups.
Experiments on three challenging agent benchmarks show that HPL outperforms existing state-of-the-art methods.
Our analyses demonstrate that the hierarchical DPO loss effectively integrates preference signals across multiple granularities, while the dual-layer curriculum is crucial for enabling the agent to solve a wide range of tasks, from simple behaviors to complex multi-step sequences.

## 1 Introduction

Large Language Models (LLMs) have evolved from static question-answering systems into autonomous agents capable of perceiving, reasoning, and acting within complex, open-ended environments (Li et al., 2024; Gou et al., 2025).
This transformation has powered a new generation of applications, from embodied assistants that navigate simulated homes (Shridhar et al., 2021) to web navigators that execute multi-step online tasks (Zheng et al., 2024; Furuta et al., 2024).
Unlike single-turn tasks, these agent-environment interactions unfold in multi-turn loops over extended periods (Wang et al., 2024a).
This paradigm shift introduces a core challenge: long-horizon planning and decision-making, where the agent must execute a coherent sequence of actions to succeed.

To equip agents for such tasks, Reinforcement Learning (RL) has become a crucial recipe for post-training LLMs (Liu et al., 2024).
Online RL methods like PPO (Sutton et al., 1998; Schulman et al., 2017) often entail substantial computational costs, high sample inefficiency, and risky, inefficient exploration in vast action spaces. These challenges have motivated a growing interest in offline RL paradigms that learn from static datasets.
Direct Preference Optimization (DPO) (Rafailov et al., 2023) directly aligns agent policies using preference pairs (e.g., expert vs. suboptimal behaviors) without requiring costly environment interaction or an explicitly trained reward model.

However, applying DPO to long-horizon agent tasks reveals a fundamental challenge we term the granularity mismatch.
On one hand, trajectory-level DPO, such as ETO (Song et al., 2024), compares entire trajectories.
This approach provides a learning signal that is often too coarse and sparse; it fails to pinpoint which specific sequence of actions within a long history was responsible for the outcome.
On the other hand, step-level DPO, employed by methods such as IPR (Xiong et al., 2024), operates at the finest granularity by comparing the consequences of single actions.
However, this fine-grained focus presents a challenge: it can struggle to capture the synergistic value of actions that are only meaningful as part of a longer sequence.
For instance, the sub-task of “retrieving an apple from the fridge” is composed of a chain of actions—navigating, opening, and taking—whose collective value cannot be captured by rewarding any single action in isolation.

To resolve this dilemma, we introduce Hierarchical Preference Learning (HPL), a hierarchical framework that optimizes LLM agents by leveraging preference signals at multiple, synergistic granularities.
HPL first addresses the granularity mismatch by incorporating DPO losses at the trajectory, action, and the action-group levels.
These groups, which often correspond to functional sub-tasks, provide an ideal intermediate unit for credit assignment.
Beyond merely combining these losses, the core innovation of HPL is a dual-layer curriculum learning strategy that guides the training process.
This curriculum systematically organizes the learning path from simple to complex along two orthogonal axes: sub-task complexity, defined by the length of an action group, and sample difficulty, measured by the reward gap between preferred and dispreferred behaviors.
By first mastering simple, easily distinguishable sub-tasks, the agent builds a foundation before progressing to more complex challenges.

Figure: Figure 1: Conceptual comparison of different DPO granularities. While (a) trajectory-level DPO provides a coarse signal and (b) step-level DPO provides a myopic one, (c) our proposed Group-level DPO learns from semantically coherent action groups, which provides a structured and meaningful signal, enabling the agent to reason at the sub-task level.
Refer to caption: x1.png

Our main contributions are summarized as follows:

- •
We identify and address the granularity mismatch problem in offline agent alignment by proposing a novel hierarchical framework that integrates preference signals at three distinct levels: the coarse trajectory-level, the fine-grained step-level, and a intermediate action-group level.
- •
We introduce HPL, a novel training paradigm with a dual-layer curriculum learning strategy. This is the first work to apply a structured curriculum to action-group level preference optimization, dynamically scheduling samples based on both task complexity and distinguishability.
- •
We design and systematically evaluate a range of action grouping strategies, from simple heuristics to a semantic-based approach, to generate meaningful sub-tasks for our framework.
- •
We demonstrate through extensive experiments on three diverse and challenging long-horizon benchmarks that HPL significantly outperforms existing state-of-the-art methods, establishing a more effective and principled paradigm for offline LLM agent alignment.

## 2 Related Work

LLM-based Agents.
The remarkable reasoning and instruction-following capabilities of modern LLMs have enabled their use as autonomous agents capable of tackling complex, interactive tasks (Li et al., 2024; Gou et al., 2025). Initial approaches primarily leveraged the in-context learning abilities of LLMs through prompts like ReAct (Yao et al., 2023) and Reflexion (Shinn et al., 2023) to elicit multi-step reasoning and action generation. To enhance the performance of open-source models beyond zero-shot prompting, a subsequent line of work has focused on fine-tuning agents on collected trajectory data (Chen et al., 2023). These methods range from standard Supervised Fine-Tuning (SFT) on expert demonstrations (Zeng et al., 2023; Chen et al., 2024) to more advanced techniques that learn from preference data, such as contrasting successful and failed trajectories to optimize for final outcomes (Song et al., 2024). While effective, these fine-tuning paradigms often treat entire trajectories as monolithic data points (Wang et al., 2024b), overlooking the fine-grained procedural knowledge embedded within the interaction trajectory.

Process Supervision.
To address the limitations of outcome-based rewards, particularly the challenge of credit assignment in long-horizon tasks, a growing body of research has explored process supervision (Luo et al., 2024; Xiong et al., 2025). The core idea is to provide agents with more granular feedback at intermediate steps of a task. Early efforts in this area often relied on costly human annotations to label the correctness of each step (Lightman et al., 2023). To automate this, recent methods have proposed various techniques to estimate step-level rewards, such as using Monte Carlo rollouts to predict the future outcome from an intermediate state (Xiong et al., 2024) or training a separate reward model to predict the value of each action (Choudhury, 2025; Wang et al., 2025). These approaches typically use the estimated step-level rewards to guide the agent via reinforcement learning (Feng et al., 2025; Zhang et al., 2025) or DPO at the single-action level.

Figure: Figure 2: An overview of our proposed framework, HPL. Stage 1 generates hierarchical preference data with Action Group Segmentation component. Stage 2 then optimizes the agent with a composite objective, where the training is guided by dual-layer curriculum scheduler.
Refer to caption: x2.png

## 3 Methodology

In this section, we present our novel agent alignment framework Hierarchical Preference Learning (HPL) as depicted in Figure [2](https://arxiv.org/html/2510.03253v1#S2.F2).
We detail the principal phases of our method below: initial policy bootstrapping through behavior cloning, the generation of multi-granularity preference data, the design of our dual-layer curriculum scheduler, and finally, the hierarchical optimization objective.

### 3.1 Bootstrapping via Expert Behavior Cloning

To equip the base model with fundamental task-solving capabilities, we perform behavior cloning on a dataset of expert trajectories $\mathcal{D}_{\text{expert}}=\{(u,\tau^{*})^{(i)}\}_{i=1}^{|\mathcal{D}_{\text{expert}}|}$, where $u$ is the task instruction and $\tau^{*}$ is the corresponding expert trajectory.
Each trajectory $\tau$ is a sequence of alternating states and actions $\tau=(s_{1},a_{1},s_{2},a_{2},\dots,s_{T},a_{T})$. A state $s_{t}$ is a textual description of the environment, and an action $a_{t}$ is a textual command generated by the agent.
This process
aims to maximize the likelihood of the expert’s actions.
The loss is defined as:

$$ $\mathcal{L}_{\text{BC}}(\theta;\mathcal{D}_{\text{expert}})=-\mathbb{E}_{(u,\tau^{*})\sim\mathcal{D}_{\text{expert}}}\left[\sum_{t=1}^{|\tau^{*}|}\log\pi_{\theta}(a_{t}^{*}|s_{t}^{*},u,\tau_{<t}^{*})\right],$ (1) $$

where $\tau_{<t}^{*}$ represents the history $(s_{1}^{*},a_{1}^{*},\dots,s_{t-1}^{*},a_{t-1}^{*})$.
This initial cloning step yields a competent base agent, which serves as our reference policy $\pi_{\text{ref}}$ for the subsequent optimization stages.

### 3.2 Hierarchical Contrastive Data Generation

After obtaining a competent reference policy $\pi_{\text{ref}}$ via behavior cloning (Section [3.1](https://arxiv.org/html/2510.03253v1#S3.SS1)), the next stage is to generate a rich, multi-layered dataset for preference optimization.
This is achieved by having $\pi_{\text{ref}}$ interact with the environment to produce a diverse set of suboptimal trajectories.
By contrasting these with expert trajectories, we construct three distinct preference datasets at the trajectory, action, and group granularities, as illustrated in the left panel of Figure [2](https://arxiv.org/html/2510.03253v1#S2.F2).

#### 3.2.1 Data Generation at Trajectory and Action Levels

Trajectory-Level Data. This dataset provides a coarse, outcome-based learning signal.
For each expert trajectory $\tau_{w}$ from $\mathcal{D}_{\text{expert}}$, we use $\pi_{\text{ref}}$ to generate a corresponding full trajectory $\tau_{l}$.
If the outcome reward of $\tau_{l}$ is lower than that of $\tau_{w}$, we form a preference pair $(\tau_{w},\tau_{l})$.
This process yields a trajectory-level dataset $\mathcal{D}_{\text{traj}}=\{(u,\tau_{w},\tau_{l})^{(i)}\}$, where $u$ is the task instruction.

Step-Level Data. To provide a finer-grained, process-oriented signal, we adopt the methodology from IPR (Xiong et al., 2024).
At each step $t$ of an expert trajectory, we use the history $\tau_{<t}$ as a prompt for our reference agent $\pi_{\text{ref}}$ to generate an alternative action $\hat{a}_{t}$ and complete the rest of the trajectory, yielding $\tau_{t:m}^{l}$.
This is contrasted with the expert’s subsequent trajectory $\tau_{t:n}^{w}$.
This creates a preference pair conditioned on the shared history, resulting in a step-level preference dataset $\mathcal{D}_{\text{step}}=\{(\tau_{<t},\tau_{t:n}^{w},\tau_{t:m}^{l})^{(i)}\}$.

#### 3.2.2 Group-Level Data Generation via Action Group Segmentation

To bridge the gap between coarse trajectories and myopic single actions, we introduce the core concept of an action group.
These groups serve as an intermediate unit of reasoning, ideally corresponding to semantically coherent sub-tasks for more effective credit assignment.
The generation of group-level data involves two key steps: segmenting trajectories into groups and then estimating a quantitative reward for each group.

First, we apply a segmentation function $f(\cdot)$ to partition expert trajectory $\tau_{w}$ into corresponding action groups $\{G_{w,i}\}_{i=1}^{N}$. We design and investigate four distinct segmentation strategies:

Fixed Heuristic Strategies. As baselines, we consider two straightforward methods based on length.
Fixed-N Groups divides a trajectory into a fixed number $N$ of equal-length groups.
Fixed-K Size creates groups with $K$ consecutive action steps each.
While simple, these methods are agnostic to the task’s semantic structure and risk making arbitrary cuts.

Uncertainty-Based Segmentation. This adaptive strategy is based on the intuition that a policy’s uncertainty often increases at sub-task boundaries (Guo et al., 2025). We leverage the entropy of the reference policy’s action distribution, $H(\pi_{\text{ref}}(\cdot|o_{t}))$, as a proxy for uncertainty.
A boundary is inserted after action $a_{t-1}$ if the entropy at step $t$ exceeds a predefined threshold $\epsilon$.

Semantic Segmentation. To achieve the most meaningful partitions, we employ a powerful, pre-trained LLM (e.g., GPT-4o) as an off-the-shelf “semantic segmenter”.
We provide the full text transcript of a trajectory to the model and prompt it to partition the sequence into high-level sub-tasks based on their apparent goals (e.g., “find an object”, “operate an appliance”).
This method is expected to yield the highest quality segmentations.

Once an expert trajectory is partitioned into a sequence of winning groups $\{G_{w,i}\}$, we construct the preference pairs required for our group-level optimization.
For each expert action group $G_{w,i}$, which begins from a context $c_{i}$ (i.e., the history of all preceding steps), we generate a corresponding losing group $G_{l,i}$.
This is achieved by sampling a new action sequence of the same length from the reference policy $\pi_{\text{ref}}(\cdot|c_{i})$.
This length-constrained sampling ensures a fair, apples-to-apples comparison between the expert and suboptimal behaviors. The resulting tuples $(c_{i},G_{w,i},G_{l,i})$ form a rich dataset of fine-grained preference pairs, which are the fundamental training units for HPL.

#### 3.2.3 Group-Level Reward Estimation

After segmenting trajectories into groups, we need a quantitative reward estimate for each group to filter data and enable our curriculum learning strategy (detailed in Section [3.3](https://arxiv.org/html/2510.03253v1#S3.SS3)).
We define the reward of an action group $G_{i}$, which ends at timestep $t_{i}$ with history $\tau_{<t_{i}}$, as the expected final outcome reward of trajectories completed from that point.
Given the difficulty of direct computation, we estimate this value using Monte Carlo (MC) sampling.
Specifically, we use our reference policy $\pi_{\text{ref}}$ to perform $M$ stochastic rollouts starting from the state after $G_{i}$ has been executed.
The estimated reward for the action group $G_{i}$, denoted $\hat{r}(G_{i})$, is the average of the final outcome rewards $R(\cdot)$ from these rollouts:

$$ $\hat{r}(G_{i})=\frac{1}{M}\sum_{j=1}^{M}R(\tau_{i}^{(j)}),\quad\text{where}\ \{\tau_{i}^{(j)}\}_{j=1}^{M}=\text{MC}^{\pi_{\text{ref}}}(\tau_{<t_{i}};M).$ (2) $$

This MC estimation is applied to every winning group $G_{w,i}$ and losing group $G_{l,i}$ to obtain their rewards.
With these components, we finalize our group-level dataset $\mathcal{D}_{\text{group}}=\{(c,G_{w},G_{l})^{(i)}\}$, where each entry contains a context, a preference pair of groups, and their estimated rewards.

### 3.3 Dual-Layer Curriculum Learning

Statically mixing the group-level preference data from all difficulties for training can be suboptimal, as it may expose the model to highly complex samples before it has developed foundational skills, leading to unstable or inefficient learning.
To address this, we introduce the core innovation of our framework: a dual-layer curriculum learning strategy.
This strategy dynamically organizes the training process to mimic an efficient human learning path, from simple concepts to complex ones.

Figure: Figure 3: Illustration of the dual-layer curriculum scheduler with group length ($L$) and sample difficulty ($\Delta R$). The training follows a three-phase schedule.
Refer to caption: x3.png

The 2D Curriculum Matrix.
As illustrated in Figure [3](https://arxiv.org/html/2510.03253v1#S3.F3), our curriculum is conceptualized as a two-dimensional difficulty matrix.
We categorize each group-level preference pair $(G_{w},G_{l})$ along two orthogonal axes:

- •
Sub-task Complexity (Y-axis): This dimension is measured by the Group Length ($L$).
Shorter action groups (e.g., 1-3 steps) correspond to simple, fundamental skills, while longer groups represent more complex, multi-step behaviors that require longer-term planning.
- •
Sample Discriminability (X-axis): This dimension is measured by the Sample Difficulty ($\Delta R$), defined as the difference between the estimated rewards of the winning and losing groups: $\Delta R=\hat{r}(G_{w})-\hat{r}(G_{l})$.
A large $\Delta R$ indicates an easy-to-distinguish sample where the losing group is clearly inferior.
A small $\Delta R$ represents a hard-to-distinguish sample that requires finer judgment from the agent for successful policy refinement.

Based on these two axes, we partition the group-level dataset $\mathcal{D}_{\text{group}}$ into a 3x3 grid of data buckets, denoted as $\mathcal{B}_{L,D}$, where $L,D\in\{1,2,3\}$ represent the levels of length and difficulty, respectively.

The Curriculum Schedule.
Our training process is not a single pass over the mixed data, but a staged schedule that progressively expands the training set, guiding the model along a path of increasing difficulty. The schedule consists of three distinct phases:

- 1.
Phase 1 (Foundational Skills): Initially, the model is trained exclusively on the easiest data bucket, $\mathcal{B}_{1,1}$ (short length, easy difficulty).
This allows the agent to quickly and stably learn the most fundamental and unambiguous skills without being distracted by more complex scenarios.
- 2.
Phase 2 (Expanding Complexity): After the initial phase, we expand the training data to include $\mathcal{B}_{1,1}\cup\mathcal{B}_{1,2}\cup\mathcal{B}_{2,1}$.
In this stage, the agent begins to tackle harder (less distinguishable) short-horizon tasks while also being introduced to simple medium-horizon skills, effectively broadening its capabilities.
- 3.
Phase 3 (Full-Scale Tuning): Finally, the training set is expanded to include all nine buckets ($\bigcup_{L,D}\mathcal{B}_{L,D}$).
The agent now fine-tunes its policy on the full spectrum of complexities and difficulties, mastering the most challenging and nuanced aspects of the tasks.

This staged exposure ensures a smooth learning gradient, building agent’s expertise from the ground up and preventing it from being overwhelmed by difficult samples early in the training process.

### 3.4 Multi-Granularity Preference Optimization

In the final stage, we optimize the policy $\pi_{\theta}$ using a composite loss function that integrates signals from all three granularities.
This approach ensures that the agent not only learns from high-level outcomes and fine-grained sub-tasks but also stays grounded in the expert’s behavior.
The final loss includes a sum of three components:

Trajectory-Level DPO Loss ($\mathcal{L}_{\text{traj-DPO}}$). To learn from the overall outcome, we apply a DPO loss on the trajectory-level dataset $\mathcal{D}_{\text{traj}}$.
This loss encourages the policy to assign a higher likelihood to the entire successful trajectory over the failed one:

$$ $\mathcal{L}_{\text{traj-DPO}}(\theta;\mathcal{D}_{\text{traj}})=-\mathbb{E}_{(\tau_{w},\tau_{l})\sim\mathcal{D}_{\text{traj}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(\tau_{w}|u)}{\pi_{\text{ref}}(\tau_{w}|u)}-\beta\log\frac{\pi_{\theta}(\tau_{l}|u)}{\pi_{\text{ref}}(\tau_{l}|u)}\right)\right].$ (3) $$

Step-Level DPO Loss ($\mathcal{L}_{\text{step-DPO}}$). Drawing from Xiong et al. (2024), this loss uses $\mathcal{D}_{\text{step}}$ to provide step-level supervision by comparing the entire future from a decision point:

$$ $\mathcal{L}_{\text{step-DPO}}(\theta;\mathcal{D}_{\text{step}})=-\mathbb{E}_{(\tau_{<t},\tau_{t:n}^{w},\tau_{t:m}^{l})\sim\mathcal{D}_{\text{step}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(\tau_{t:n}^{w}|\tau_{<t})}{\pi_{\text{ref}}(\tau_{t:n}^{w}|\tau_{<t})}-\beta\log\frac{\pi_{\theta}(\tau_{t:m}^{l}|\tau_{<t})}{\pi_{\text{ref}}(\tau_{t:m}^{l}|\tau_{<t})}\right)\right].$ (4) $$

Group-Level DPO Loss ($\mathcal{L}_{\text{group-DPO}}$). This is the core component of our framework, providing mid-level supervision.
We apply the DPO loss to the group-level dataset $\mathcal{D}_{\text{group}}$, comparing corresponding action groups:

$$ $\mathcal{L}_{\text{group-DPO}}(\theta;\mathcal{D}_{\text{group}})=-\mathbb{E}_{(c,G_{w},G_{l})\sim\mathcal{D}_{\text{group}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(G_{w}|c)}{\pi_{\text{ref}}(G_{w}|c)}-\beta\log\frac{\pi_{\theta}(G_{l}|c)}{\pi_{\text{ref}}(G_{l}|c)}\right)\right].$ (5) $$

To better understand the effectiveness of the group-level objective, we briefly analyze its bias-variance properties in the following proposition, with a full derivation provided in Appendix [D](https://arxiv.org/html/2510.03253v1#A4).

###### Proposition 1 (Bias-variance trade-off of group-level DPO loss) .

Let $T$ denote the trajectory length, $\gamma\in[0,1)$ the discount factor, and $R_{\text{max}}$ the maximum reward. Let $\mathcal{L}_{\text{traj}}$, $\mathcal{L}_{\text{step}}$, and $\mathcal{L}_{\text{group}}(k)$ denote the empirical losses of trajectory-level, step-level, and group-level DPO with group length $k<T$, respectively. Then there exists a constant $C>0$ depending only on $(\gamma,\pi_{\text{ref}})$ such that for every $\epsilon\in(0,1)$ the choice $k(\epsilon)=\left\lceil\log_{\gamma}\left(\frac{(1-\gamma)\epsilon}{2\beta R_{\text{max}}}\right)\right\rceil$ satisfies

$$ $\displaystyle\mathrm{Bias}(\mathcal{L}_{\text{group}}(k))$ $\displaystyle\leq\min\{\mathrm{Bias}(\mathcal{L}_{\text{traj}}),\mathrm{Bias}(\mathcal{L}_{\text{step}})\}+\epsilon,$ (6) $\displaystyle\mathrm{Var}(\mathcal{L}_{\text{group}}(k))$ $\displaystyle\leq\frac{C\log(1/\epsilon)}{T}\min\{\mathrm{Var}(\mathcal{L}_{\text{traj}}),\mathrm{Var}(\mathcal{L}_{\text{step}})\}.$ (7) $$

Hence, by setting $k=\Theta(\log(1/\epsilon))$, group-level DPO loss simultaneously improves the variance by a factor $\Omega(T/\log(1/\epsilon))$ while incurring at most an additive bias of $\epsilon$ over the other two losses.

Thus, the final training objective of our HPL combines these losses:

$$ $\mathcal{L}_{\text{HPL}}^{(s)}=\mathcal{L}_{\text{BC}}+\mathcal{L}_{\text{traj-DPO}}+\mathcal{L}_{\text{step-DPO}}+\mathcal{L}_{\text{group-DPO}}^{(s)},$ (8) $$

where the group-level loss $\mathcal{L}_{\text{group-DPO}}^{(s)}$ for curriculum stage s is computed over a dynamically selected subset of data, $\mathcal{D}_{\text{group}}^{(s)}$, which is determined by our curriculum scheduler (Section [3.3](https://arxiv.org/html/2510.03253v1#S3.SS3)).
The data subset $\mathcal{D}_{\text{group}}^{(s)}$ for each stage is constructed from the 2D curriculum matrix buckets $\mathcal{B}_{L,D}$ as follows:

$$ $\mathcal{D}_{\text{group}}^{(s)}=\begin{cases}\mathcal{B}_{1,1}&\text{if }s=1\text{ (Phase 1)}\\ \mathcal{B}_{1,1}\cup\mathcal{B}_{1,2}\cup\mathcal{B}_{2,1}&\text{if }s=2\text{ (Phase 2)}\\ \bigcup_{L,D}\mathcal{B}_{L,D}&\text{if }s=3\text{ (Phase 3)}\end{cases}$ (9) $$

###### Proposition 1 (Bias-variance trade-off of group-level DPO loss) .

## 4 Experiments

In this section, we conduct a series of experiments to comprehensively evaluate the performance of our HPL framework. Our evaluation is designed to answer the following key research questions:

- RQ1.
Does HPL outperform strong baselines that rely on conventional learning granularities, such as the trajectory-level (ETO) and step-level (IPR)?
- RQ2.
What is the impact of different action group segmentation strategies on the final performance of the agent?
- RQ3.
How crucial is the dual-layer curriculum mechanism to the success of HPL, and what is the contribution of each curriculum layer?
- RQ4.
How do trajectory-level, step-level, and group-level losses contribute to HPL?

### 4.1 Experimental Setup

We evaluate our proposed method, HPL, on three diverse and challenging long-horizon agent benchmarks: ALFWorld (Shridhar et al., 2021), WebShop (Yao et al., 2022), and InterCode-SQL (Yang et al., 2023). For all experiments, we use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as the backbone language models. HPL is compared against a suite of strong baselines, including SFT, RFT (Yuan et al., 2023), ETO (Song et al., 2024), and IPR (Xiong et al., 2024). All methods are initialized from an SFT model trained on expert trajectories generated by a GPT-4o teacher model.
For Fixed-N and Fixed-K strategy, we set $N=3$ and $K=3$.
For Uncertainty strategy, the entropy threshold $\epsilon$ is set to the 80th percentile of all action entropies (i.e., the threshold for the top 20% highest values) computed across the training dataset.
Detailed descriptions of the environments, baseline implementations, and all hyperparameters are deferred to Appendix [C](https://arxiv.org/html/2510.03253v1#A3).

### 4.2 Main Results (RQ1, RQ2)

**Table 1: Performance comparison of HPL and strong baselines across three agent benchmarks. All methods are evaluated using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as base models. The best and second-best results are highlighted in bold and with an underline, respectively.**
| Models | ALFWorld | WebShop | InterCode-SQL | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| seen | unseen | avg. reward | success rate | avg. reward | success rate |  |  |
| GPT-4o | 36.43 | 32.09 | 55.26 | 18.50 | 28.50 | 28.50 | 33.21 |
| Gemini-2.5-Pro | 55.71 | 49.25 | 49.56 | 19.50 | 68.42 | 66.00 | 51.40 |
| Qwen2.5-1.5B-Instruct | 2.14 | 0.00 | 36.09 | 10.50 | 5.50 | 5.50 | 9.95 |
| SFT | 62.14 | 58.21 | 55.99 | 25.50 | 56.85 | 55.00 | 52.28 |
| RFT (Yuan et al., 2023) | 61.43 | 60.45 | 57.88 | 29.00 | 57.99 | 56.50 | 53.88 |
| ETO (Song et al., 2024) | 62.14 | 64.18 | 56.82 | 28.50 | 58.94 | 57.50 | 54.68 |
| IPR (Xiong et al., 2024) | 64.29 | 64.93 | 58.83 | 29.00 | 59.74 | 58.00 | 55.64 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 67.86 | 73.13 | 59.99 | 29.50 | 59.26 | 58.00 | 57.96 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 69.29 | 63.43 | 58.36 | 28.00 | 59.76 | 57.00 | 55.97 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 72.86 | 63.43 | 58.16 | 27.00 | 59.86 | 57.50 | 56.47 |
| \rowcolorMyBlue<br>HPL (Semantic) | 71.43 | 72.39 | 59.99 | 30.00 | 60.08 | 58.50 | 58.73 |
| Qwen2.5-7B-Instruct | 38.57 | 45.52 | 56.61 | 19.50 | 8.80 | 8.50 | 29.58 |
| SFT | 67.14 | 76.12 | 60.53 | 32.00 | 66.86 | 65.00 | 61.28 |
| RFT (Yuan et al., 2023) | 72.86 | 69.40 | 61.76 | 33.50 | 68.43 | 66.50 | 62.08 |
| ETO (Song et al., 2024) | 70.00 | 76.87 | 61.57 | 33.50 | 68.63 | 67.00 | 62.93 |
| IPR (Xiong et al., 2024) | 72.86 | 77.61 | 61.90 | 34.00 | 69.19 | 67.50 | 63.84 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 78.57 | 76.12 | 61.75 | 34.50 | 69.65 | 68.00 | 64.77 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 88.57 | 76.87 | 62.20 | 34.00 | 69.92 | 68.50 | 66.68 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 81.43 | 83.58 | 62.48 | 35.00 | 69.25 | 67.50 | 66.54 |
| \rowcolorMyBlue<br>HPL (Semantic) | 83.57 | 86.57 | 62.56 | 34.50 | 70.63 | 69.00 | 67.81 |

As shown in Table [1](https://arxiv.org/html/2510.03253v1#S4.T1), our HPL framework significantly outperforms all baseline methods across both model scales, providing a strong affirmative answer to our first research question. For the Qwen2.5-7B-Instruct model, our best-performing variant, HPL (Semantic), achieves an average score of 67.81, surpassing the strongest single-granularity baselines, ETO and IPR, by substantial margins of 4.88 and 3.97 points, respectively. This trend of superiority holds true across all three diverse and challenging benchmarks. The advantage of HPL’s hierarchical approach is particularly pronounced in tasks requiring complex, long-horizon generalization. For instance, on the ALFWorld benchmark’s unseen scenarios, HPL (Semantic) achieves a success rate of 86.57%, a remarkable improvement of nearly 9 points over the state-of-the-art IPR (77.61%). These results strongly indicate that by integrating preference signals at the trajectory, action, and group levels, HPL effectively resolves the granularity mismatch problem and learns a more robust and generalizable policy.

Our experiments also reveal the critical impact of the action group segmentation strategy, directly addressing our second research question. While all HPL variants consistently outperform the baselines, the adaptive, content-aware segmentation methods generally yield better performance than fixed heuristic approaches. Notably, HPL (Semantic), which leverages a powerful language model to partition trajectories into semantically coherent sub-tasks, consistently emerges as the top-performing variant for both the 1.5B and 7B models. For the 7B model, it outperforms the next-best variant, HPL (Fixed-N), by over 1 point in the average score. This suggests that the quality of the action groups is paramount; providing the DPO loss with more meaningful, human-aligned sub-tasks as units of comparison leads to a more effective and powerful learning signal. Even so, the strong performance of the simpler Uncertainty and Fixed-N strategies demonstrates the inherent value of incorporating an intermediate granularity, validating the core design of our HPL framework.

### 4.3 Analysis of the Curriculum Learning Mechanism (RQ3)

**Table 2: Ablation study on our curriculum learning mechanism of HPL across three agent benchmarks.**
| Models | ALFWorld | WebShop | InterCode-SQL | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| seen | unseen | avg. reward | success rate | avg. reward | success rate |  |  |
| Qwen2.5-1.5B-Instruct |  |  |  |  |  |  |  |
| HPL | 71.43 | 72.39 | 59.99 | 30.00 | 60.08 | 58.50 | 58.73 |
| HPL Static | 68.57 | 71.64 | 58.80 | 29.00 | 59.45 | 58.00 | 57.58 |
| HPL Length CL Only | 69.29 | 72.39 | 58.06 | 28.50 | 59.39 | 57.50 | 57.52 |
| HPL Difficulty CL Only | 71.43 | 70.71 | 58.83 | 30.00 | 60.50 | 58.00 | 58.25 |
| Qwen2.5-7B-Instruct |  |  |  |  |  |  |  |
| HPL | 83.57 | 86.57 | 62.56 | 34.50 | 70.63 | 69.00 | 67.81 |
| HPL Static | 75.71 | 82.84 | 62.05 | 33.00 | 69.71 | 68.50 | 65.30 |
| HPL Length CL Only | 82.14 | 85.07 | 62.26 | 34.00 | 69.49 | 67.50 | 66.74 |
| HPL Difficulty CL Only | 81.43 | 85.82 | 62.27 | 34.50 | 69.63 | 68.00 | 66.94 |

Figure: Figure 4: Phase-wise performance progression of HPL on the ALFWorld benchmark. (a) Success rates for both 1.5B and 7B models across the three curriculum phases. (b) A detailed breakdown for the 1.5B model on 6 sub-task types.
Refer to caption: x4.png

Ablation of Curriculum Components.
To quantitatively assess the importance of our dual-layer curriculum, we conduct an ablation study by removing each layer individually, with results presented in Table [2](https://arxiv.org/html/2510.03253v1#S4.T2). The primary finding is that the full HPL model, equipped with both curriculum layers, consistently outperforms all ablated variants. Removing the curriculum entirely (HPL Static) results in the most significant performance degradation across both model scales, confirming that employing a curriculum is crucial for effective learning.
Furthermore, the results indicate that both the length and difficulty-based curricula contribute positively to the final performance, with their individual removal leading to noticeable performance drops. This demonstrates the synergistic benefit of our dual-layer design, where organizing the learning process first by task complexity (length) and then by solution quality (difficulty) provides a more effective path to mastering complex agent behaviors.

Phase-wise Performance Progression.
To provide a more fine-grained view of how the curriculum works, Figure [4](https://arxiv.org/html/2510.03253v1#S4.F4) visualizes the agent’s performance at the end of each curriculum phase.
Panel (a) shows a clear improvement in the overall success rate for both the 1.5B and 7B models as they progress from Phase 1 to Phase 3.
This trend holds for the unseen scenarios, indicating that the curriculum effectively helps the model generalize its learned skills.
Panel (b) offers a deeper insight by breaking down the performance by sub-task type for the 1.5B model.
We observe that while simpler tasks like Pick are learned relatively early, more complex tasks requiring longer reasoning chains, such as Clean and Pick2, show the most substantial performance gains in the later phases.
This stage-wise improvement, with harder tasks being mastered in later stages, provides qualitative evidence that our dual-layer curriculum successfully guides the agent through an easy-to-hard learning process, progressively building its capabilities for complex, long-horizon planning.

### 4.4 Ablation on Hierarchical DPO Losses (RQ4)

Figure: Figure 5: Ablation study on the HPL loss components on Qwen2.5-7B-Instruct.
Refer to caption: x5.png

To investigate the individual contribution of each component in our hierarchical framework, we conduct an ablation study on the three DPO losses, with results shown in Figure [5](https://arxiv.org/html/2510.03253v1#S4.F5).
The results reveal that while all three loss components, trajectory, action, and group, contribute positively to the final performance, the group-level DPO is demonstrably the most critical.
Across all benchmarks, removing the group-level signal (HPL w/o group-DPO) induces a significantly more severe performance degradation than removing either the trajectory or step-level signals.
This provides compelling evidence that the action-group granularity is the primary driver of HPL’s effectiveness, serving as a crucial bridge between coarse trajectory feedback and myopic action supervision and validating the synergistic benefit of our three-level approach.

## 5 Conclusion

In this work, we address the critical issue of granularity mismatch in preference-based alignment for LLM agents. We introduce Hierarchical Preference Learning (HPL), a novel framework that resolves this challenge by integrating preference signals across three levels of abstraction: trajectory, action, and a crucial, intermediate action-group level. By partitioning trajectories into semantically coherent sub-tasks and optimizing a hierarchical DPO loss, HPL learns to directly prefer successful multi-step action sequences over flawed alternatives. Extensive experiments demonstrate that HPL outperforms strong prior methods that operate primarily at the extreme granularities of entire trajectories or individual actions, across a suite of complex, long-horizon agent benchmarks. In conclusion, our work underscores the importance of learning from hierarchical signals that mirror the compositional nature of complex tasks, paving the way for more capable LLM agents.

## References

- Chen et al. (2023)
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.
Fireact: Toward language agent fine-tuning.
*arXiv preprint arXiv:2310.05915*, 2023.
- Chen et al. (2024)
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao.
Agent-flan: Designing data and methods of effective agent tuning for large language models.
*arXiv preprint arXiv:2403.12881*, 2024.
- Choudhury (2025)
Sanjiban Choudhury.
Process reward models for llm agents: Practical framework and directions.
*arXiv preprint arXiv:2502.10325*, 2025.
- Feng et al. (2025)
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An.
Group-in-group policy optimization for llm agent training.
*arXiv preprint arXiv:2505.10978*, 2025.
- Furuta et al. (2024)
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur.
Multimodal web navigation with instruction-finetuned foundation models.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=efFmBWioSc](https://openreview.net/forum?id=efFmBWioSc).
- Gou et al. (2025)
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su.
Navigating the digital world as humans do: Universal visual grounding for GUI agents.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=kxnoqaisCT](https://openreview.net/forum?id=kxnoqaisCT).
- Guo et al. (2025)
Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu.
Segment policy optimization: Effective segment-level credit assignment in rl for large language models.
*arXiv preprint arXiv:2505.23564*, 2025.
- Li et al. (2024)
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al.
Embodied agent interface: Benchmarking llms for embodied decision making.
*Advances in Neural Information Processing Systems*, 37:100428–100534, 2024.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*, 2023.
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Luo et al. (2024)
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al.
Improve mathematical reasoning in language models by automated process supervision.
*arXiv preprint arXiv:2406.06592*, 2024.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36:53728–53741, 2023.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shinn et al. (2023)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
*Advances in Neural Information Processing Systems*, 36:8634–8652, 2023.
- Shridhar et al. (2021)
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
Alfworld: Aligning text and embodied environments for interactive learning.
In *International Conference on Learning Representations*, 2021.
URL [https://openreview.net/forum?id=0IOX0YcCdTn](https://openreview.net/forum?id=0IOX0YcCdTn).
- Song et al. (2024)
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin.
Trial and error: Exploration-based trajectory optimization of LLM agents.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 7584–7600. Association for Computational Linguistics, 2024.
- Sutton et al. (1998)
Richard S Sutton, Andrew G Barto, et al.
*Reinforcement learning: An introduction*, volume 1.
MIT press Cambridge, 1998.
- Wang et al. (2025)
Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li.
Spa-rl: Reinforcing llm agents via stepwise progress attribution.
*arXiv preprint arXiv:2505.20732*, 2025.
- Wang et al. (2024a)
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.
A survey on large language model based autonomous agents.
*Frontiers of Computer Science*, 18(6):186345, 2024a.
- Wang et al. (2024b)
Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin.
Learning from failure: Integrating negative examples when fine-tuning large language models as agents.
*arXiv preprint arXiv:2402.11651*, 2024b.
- Xiong et al. (2025)
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sainbayar Sukhbaatar.
Stepwiser: Stepwise generative judges for wiser reasoning.
*arXiv preprint arXiv:2508.19229*, 2025.
- Xiong et al. (2024)
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li.
Watch every step! LLM agent learning via iterative step-level process refinement.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 1556–1572. Association for Computational Linguistics, 2024.
- Yang et al. (2023)
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao.
Intercode: Standardizing and benchmarking interactive coding with execution feedback.
*Advances in Neural Information Processing Systems*, 36:23826–23854, 2023.
- Yao et al. (2022)
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents.
*Advances in Neural Information Processing Systems*, 35:20744–20757, 2022.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yu et al. (2018)
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 3911–3921. Association for Computational Linguistics, 2018.
- Yuan et al. (2023)
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou.
Scaling relationship on learning mathematical reasoning with large language models.
*arXiv preprint arXiv:2308.01825*, 2023.
- Zeng et al. (2023)
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.
Agenttuning: Enabling generalized agent abilities for llms.
*arXiv preprint arXiv:2310.12823*, 2023.
- Zhang et al. (2025)
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al.
The landscape of agentic reinforcement learning for llms: A survey.
*arXiv preprint arXiv:2509.02547*, 2025.
- Zheng et al. (2024)
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su.
GPT-4V(ision) is a generalist web agent, if grounded.
In *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pp. 61349–61385. PMLR, 21–27 Jul 2024.
URL [https://proceedings.mlr.press/v235/zheng24e.html](https://proceedings.mlr.press/v235/zheng24e.html).

## Appendix A The Use of LLMs

In the preparation of this manuscript and the development of our codebase, we utilized LLMs to assist in two primary capacities. We detail these uses below for full transparency:

- •
Writing and Editing Assistance: We used LLM as a writing assistant to improve the grammar, phrasing, and overall clarity of the manuscript. The authors directed all core ideas and claims, and are fully responsible for the final wording, arguments, and content presented in this paper.
- •
Code Implementation Support: During the implementation of our experimental framework, an LLM served as a coding assistant for tasks such as generating basic code and aiding in debugging. The authors designed the overall software architecture, and all LLM-generated code was carefully reviewed and adapted by the authors to ensure its correctness and functionality.

## Appendix B Broader Impacts and Limitations

##### Broader Impacts.

Our work on Hierarchical Preference Learning (HPL) presents a more efficient and effective paradigm for training capable autonomous agents from offline data. Technologically, this reduces the reliance on costly and often unsafe online exploration, making the development of sophisticated LLM agents more accessible and sustainable. The ability to learn from the compositional structure of tasks via action groups could lead to more reliable and predictable agents in real-world applications, from personalized assistants to complex workflow automation. On a societal level, the deployment of more competent agents can enhance productivity and assist with complex decision-making. However, as agent capabilities advance, ensuring their alignment with human values becomes paramount. The preferences learned from static datasets must be carefully curated to prevent the codification of biases or unintended behaviors. Furthermore, the increasing autonomy of such agents necessitates continued research into robust safety protocols, transparency, and ethical oversight to mitigate potential misuse and ensure these powerful technologies are developed responsibly for the benefit of society.

##### Limitations.

While our HPL framework demonstrates significant performance gains, we acknowledge several limitations that offer avenues for future research. Firstly, the effectiveness of HPL, particularly the semantic variant, is contingent on the quality of the action group segmentation. Although our experiments show that simple heuristics are beneficial, a suboptimal segmentation could yield less meaningful sub-tasks and hinder the learning process. Secondly, our dual-layer curriculum, while effective, introduces a new set of hyperparameters for defining task complexity and sample distinguishability, and the optimal configuration of this curriculum may be domain-specific and require careful tuning. Finally, our current work relies on a powerful teacher model for generating preference data, which means the agent’s policy may inherit biases from the teacher. Future research could explore more robust, self-supervised segmentation techniques and investigate methods for learning curricula directly from the data.

## Appendix C Experimental Details

### C.1 Environments and Tasks

We evaluate our framework on three challenging benchmarks that require long-horizon, multi-step reasoning and interaction, representing a diverse set of agent tasks.

#### C.1.1 ALFWorld

ALFWorld (Shridhar et al., 2021) is an embodied agent benchmark set in simulated household environments, uniquely designed to align abstract, text-based interactions with a visually rich, embodied world. Agents must parse natural language instructions and perform a sequence of high-level actions (e.g., goto, take, clean) to complete common household tasks, such as “put a clean tomato on the sidetable”.
The benchmark is structured into six distinct and compositional sub-task categories: Pick, Clean, Heat, Cool, Examine, and Pick2.
The dataset contains 3,553 scenarios for training.
For evaluation, the test set is divided into a seen set of 140 scenarios to assess in-distribution generalization within familiar room layouts, and a more challenging unseen set of 134 scenarios with novel task instances in unobserved environments to evaluate out-of-distribution generalization.
For all tasks, the maximum number of interaction turns is set to 30.

#### C.1.2 WebShop

WebShop (Yao et al., 2022) is a web-based simulation environment that tasks agents with navigating a realistic e-commerce website to find and purchase a product that matches a given instruction.
The environment is notable for its scale and realism, featuring 1.18 million real-world products and over 12,000 crowd-sourced, natural language instructions.
To succeed, an agent must interact with multiple types of web pages, including search, results, and item-detail pages, by issuing high-level actions, primarily “search[QUERY]” or “click[BUTTON]”.
This process requires a combination of information retrieval skills to formulate effective search queries and long-horizon planning to navigate the site, compare items, and select the correct product options.
The final reward is automatically calculated based on a heuristic that measures the attribute, option, and price match between the purchased item and the user’s instruction.
The benchmark includes 200 test tasks.
For all tasks, the maximum number of interaction turns is set to 10.

#### C.1.3 InterCode-SQL

InterCode-SQL (Yang et al., 2023) is an interactive environment designed to benchmark agent capabilities in complex data querying tasks.
Within a safe and reproducible Docker container, the agent interacts with a live MySQL database by iteratively writing and executing SQL queries to answer natural language questions.
The environment is built upon the challenging, cross-domain Spider dataset (Yu et al., 2018), which requires the agent to understand complex database schemas and formulate queries that often involve multiple tables and JOIN operations.
A key feature of this benchmark is its interactive nature; after each query execution, the agent receives real-world feedback, such as query results or error messages, which it must use to debug and refine its subsequent actions.
A final reward is calculated based on the Intersection over Union (IoU) between the agent’s submitted query results and the ground-truth records.
The benchmark consists of 200 test tasks.
For all tasks, the maximum number of interaction turns is set to 10.

### C.2 Models and Implementation Details

Our methodology relies on a dataset of expert trajectories and generating a corresponding set of suboptimal trajectories for creating preference pairs.

##### Expert Trajectories ( 𝒟 expert \mathcal{D}_{\text{expert}} ).

Following prior work (Xiong et al., 2024), we generate our initial set of expert trajectories by prompting a powerful teacher model (GPT-4o) to solve tasks in each environment using a ReAct-style reasoning process. We then filter these generated trajectories, retaining only those that achieve a high outcome reward (e.g., success score of 1.0 in ALFWorld and InterCode-SQL, or $>$0.8 in WebShop) to form our final expert dataset, $\mathcal{D}_{\text{expert}}$. This dataset is used for the initial behavior cloning stage.

##### Models and Training.

We utilize Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as the backbone models for all experiments. For all DPO-based methods, including our HPL, the reference policy $\pi_{\text{ref}}$ is the SFT agent trained on $\mathcal{D}_{\text{expert}}$. During the behavior cloning phase, models are trained for 3 epochs using the AdamW optimizer with a cosine learning rate schedule, peaking at 1e-5. All experiments were conducted on 8 NVIDIA A800 80G GPUs.

### C.3 Baselines

We compare HPL against a suite of strong baselines representing different alignment strategies, ranging from standard imitation learning to state-of-the-art preference optimization methods.

- •
SFT: The standard Supervised Fine-Tuning approach, where the model is trained only on the expert trajectories $\mathcal{D}_{\text{expert}}$.
This serves as our foundational base model and represents the standard behavior cloning paradigm from which all other preference-based methods are initialized.
- •
RFT (Yuan et al., 2023): Rejection sampling Fine-Tuning is an enhanced fine-tuning method that uses rejection sampling to augment the expert dataset with newly generated successful trajectories.
By enriching the training data with a more diverse set of successful paths, RFT serves as a strong imitation learning baseline that tests the performance limits of learning solely from positive demonstrations.
- •
ETO (Song et al., 2024): A trajectory-level DPO baseline that learns by contrasting full successful and failed trajectories.
This method represents the coarsest end of the preference learning spectrum, providing a holistic signal based on the final outcome.
While powerful, this approach faces challenges in credit assignment, as it can struggle to pinpoint specific errors within long action sequences.
- •
IPR (Xiong et al., 2024): A state-of-the-art process supervision method that performs step-level DPO using rewards estimated from Monte Carlo rollouts.
IPR operates at the finest granularity, providing precise, localized feedback on individual actions.
However, this myopic focus may overlook the synergistic value of multi-step sub-tasks.

### C.4 Hyperparameters

Table [3](https://arxiv.org/html/2510.03253v1#A3.T3) and Table [4](https://arxiv.org/html/2510.03253v1#A3.T4) show the hyperparameters for SFT and Group-DPO stage respectively across three agent benchmarks.
All experiments were conducted on 8 NVIDIA A800 80G GPUs.

**Table 3: Hyperparamenters for SFT stage across three agent benchmarks.**
| Benchmark | ALFWorld | WebShop | InterCode-SQL |
| --- | --- | --- | --- |
| Batch size | 32 | 32 | 32 |
| Learning rate | 1e-5 | 1e-5 | 1e-5 |
| Optimizer | AdamW | AdamW | AdamW |
| LR scheduler | cosine | cosine | cosine |
| Warmup ratio | 0.1 | 0.1 | 0.1 |
| Max epochs | 3 | 3 | 3 |
| Max seq length | 6000 | 6000 | 6000 |
| DeepSpeed Zero stage | 3 | 3 | 3 |
| Gradient accumulation steps | 2 | 2 | 2 |

**Table 4: Hyperparamenters for Group-DPO stage across three agent benchmarks.**
| Benchmark | ALFWorld | WebShop | InterCode-SQL |
| --- | --- | --- | --- |
| Batch size | 32 | 32 | 32 |
| Learning rate | 3e-6 | 1e-6 | 1e-6 |
| $\beta$ | 0.3 | 0.3 | 0.3 |
| Optimizer | AdamW | AdamW | AdamW |
| LR scheduler | cosine | cosine | cosine |
| Warmup ratio | 0.1 | 0.1 | 0.1 |
| Max epochs | 1 | 1 | 1 |
| Max seq length | 6000 | 6000 | 6000 |
| DeepSpeed Zero stage | 3 | 3 | 3 |
| Group length ($L$) threshold for curriculum | (0,3,6) | (0,2,4) | (0,2,4) |
| Difficulty ($\Delta R$) threshold for curriculum | (1.0, 0.7, 0.4) | (1.0, 0.7, 0.4) | (1.0, 0.7, 0.4) |

## Appendix D Analysis

We now analyze the bias and variance of group-level DPO loss.
Consider an MDP with discount factor $\gamma\in[0,1)$. A trajectory $\tau$ of horizon $T$ is denoted as $\tau=(s_{1},a_{1},r_{1},\dots,s_{T},a_{T},r_{T})$.
Let $\pi_{\text{ref}}$ be a reference policy strictly positive on every state-action pair.
For any sequence $u$, we define its discounted return with respect to the (unknown) optimal value function $V^{*}$ by

$$ $R(u)\coloneqq\sum_{i\in u}\gamma^{i-t_{0}}r_{i}+\gamma^{|u|}V^{*}(s_{t_{0}+|u|}),$ (10) $$

where $t_{0}$ is the starting time index of $u$ and $|u|$ is its length.
The true preference probability that $u_{w}$ is preferred to $u_{l}$ is modelled by the Bradley-Terry law

$$ $P(u_{w}\succ u_{l})=\sigma\left(\beta\left[\log\frac{\pi^{*}(u_{w})}{\pi_{\text{ref}}(u_{w})}-\log\frac{\pi^{*}(u_{l})}{\pi_{\text{ref}}(u_{l})}\right]\right)\coloneqq\sigma(\beta\Delta^{*}),$ (11) $$

where $\sigma(z)=\frac{1}{1+e^{-z}}$ and $\beta>0$ is a fixed inverse-temperature.
The population DPO loss for a generic distribution $\mu$ over pairs $(u_{w},u_{l})$ is

$$ $\mathcal{L}^{\mu}\coloneqq-\mathbb{E}_{\mu}\log\sigma(\beta\Delta^{*}).$ (12) $$

Given a dataset $\mathcal{D}$ of $N$ i.i.d. trajectories, each method forms its own empirical distribution $\mu_{\bullet}$ and minimizes

$$ $\mathcal{L}_{\bullet}(\theta;\mathcal{D}_{\bullet})\coloneqq-\frac{1}{|\mathcal{D}_{\bullet}|}\sum_{(u_{w},u_{l})\in\mathcal{D}_{\bullet}}\log\sigma(\beta\Delta_{\theta}),$ (13) $$

where $\Delta_{\theta}\coloneqq\log\frac{\pi_{\theta}(u_{w})}{\pi_{\text{ref}}(u_{w})}-\log\frac{\pi_{\theta}(u_{l})}{\pi_{\text{ref}}(u_{l})}$ and $\bullet\in\{\mathrm{traj,step,group}\}$.
We adopt the standard risk decomposition

$$ $\displaystyle\mathrm{Risk}(\mathcal{L}_{\bullet})\coloneqq$ $\displaystyle\;\mathbb{E}[(\mathcal{L}_{\bullet}-\mathcal{L}^{\mu_{\bullet}})^{2}]$ (14) $\displaystyle=$ $\displaystyle\;\mathrm{Bias}(\mathcal{L}_{\bullet})^{2}+\mathrm{Var}(\mathcal{L}_{\bullet}),$ $$

where the expectation $\mathbb{E}[\cdot]$ is taken over the sampling distribution of $\mathcal{D}_{\bullet}$ and

$$ $\displaystyle\mathrm{Bias}(\mathcal{L}_{\bullet})$ $\displaystyle\coloneqq\mathbb{E}[\mathcal{L}_{\bullet}]-\mathcal{L}^{\mu_{\bullet}},$ (15) $\displaystyle\mathrm{Var}(\mathcal{L}_{\bullet})$ $\displaystyle\coloneqq\mathbb{E}\left[\left(\mathcal{L}_{\bullet}-\mathbb{E}[\mathcal{L}_{\bullet}]\right)^{2}\right].$ $$

###### Proposition 1 (Bias-variance trade-off of group-level DPO loss) .

Let $T$ denote the trajectory length, $\gamma\in[0,1)$ the discount factor, and $R_{\text{max}}$ the maximum reward. Let $\mathcal{L}_{\text{traj}}$, $\mathcal{L}_{\text{step}}$, and $\mathcal{L}_{\text{group}}(k)$ denote the empirical losses of trajectory-level, step-level, and group-level DPO with group length $k<T$, respectively. Then there exists a constant $C>0$ depending only on $(\gamma,\pi_{\text{ref}})$ such that for every $\epsilon\in(0,1)$ the choice $k(\epsilon)=\left\lceil\log_{\gamma}\left(\frac{(1-\gamma)\epsilon}{2\beta R_{\text{max}}}\right)\right\rceil$ satisfies

$$ $\displaystyle\mathrm{Bias}(\mathcal{L}_{\text{group}}(k))$ $\displaystyle\leq\min\{\mathrm{Bias}(\mathcal{L}_{\text{traj}}),\mathrm{Bias}(\mathcal{L}_{\text{step}})\}+\epsilon,$ (16) $\displaystyle\mathrm{Var}(\mathcal{L}_{\text{group}}(k))$ $\displaystyle\leq\frac{C\log(1/\epsilon)}{T}\min\{\mathrm{Var}(\mathcal{L}_{\text{traj}}),\mathrm{Var}(\mathcal{L}_{\text{step}})\}.$ (17) $$

###### Proof.

First, we analyze the bias of the three losses.
We compare the population losses induced by the three sampling schemes.
Recall that the logistic loss is 1-Lipschitz, that is, for any scalar difference $z$,

$$ $|\log\sigma(z)-\log\sigma(z^{\prime})|\leq|z-z^{\prime}|.$ (18) $$

Hence the bias of an empirical loss $\mathcal{L}_{\bullet}$ is controlled by

$$ $|\mathbb{E}[\mathcal{L}_{\bullet}]-\mathcal{L}^{\mu_{\bullet}}|\leq\beta\mathbb{E}_{\mu_{\bullet}}[|\Delta_{\theta}-\Delta^{*}|],$ (19) $$

which is governed by the error in the return difference induced by the length of the comparison unit. We now analyze the three cases: trajectory, step, and group.
Trajectory-level DPO compares entire trajectories with no truncation. Hence $\mathrm{Bias}(\mathcal{L}_{\mathrm{traj}})=0$.
Step-level DPO compares suffixes from time $t$ to $T$. Following the implementation in Xiong et al. (2024), these suffixes are not truncated either. Hence $\mathrm{Bias}(\mathcal{L}_{\mathrm{step}})=0$. For group-level DPO, the unit has fixed length $k<T$.
For clarity, we define $R^{*}(t)\coloneqq\sum_{i=t}^{T}\gamma^{i-t}r_{i}$. According to Bellman equation,

$$ $R^{*}(t)=\sum_{i=t}^{t+k-1}\gamma^{i-t}r_{i}+\gamma^{k}R^{*}(t+k).$ (20) $$

Consider a group pair $(G_{w},G_{l})$ starting from the same state $s_{t}$, the true return difference should be $\delta_{\text{traj}}=R^{*}_{w}(t)-R^{*}_{l}(t)$, while group-level DPO uses $\delta_{\text{group}}=R(G_{w})-R(G_{l})$. Substituting Equation [20](https://arxiv.org/html/2510.03253v1#A4.E20) into $\delta_{\text{traj}}$, we get

$$ $\displaystyle\delta_{\text{traj}}$ $\displaystyle=\left[\sum_{i=t,r_{i}\in G_{w}}^{t+k-1}\gamma^{i-t}r_{i}+\gamma^{k}R^{*}_{w}(t+k)\right]-\left[\sum_{i=t,r_{i}\in G_{l}}^{t+k-1}\gamma^{i-t}r_{i}+\gamma^{k}R^{*}_{l}(t+k)\right]$ (21) $\displaystyle=(R(G_{w})-R(G_{l}))+\gamma^{k}(R^{*}_{w}(t+k)-R^{*}_{l}(t+k))$ (22) $\displaystyle=\delta_{\text{group}}+\gamma^{k}(R^{*}_{w}(t+k)-R^{*}_{l}(t+k)).$ (23) $$

Hence the error in the return difference is

$$ $|\delta_{\text{traj}}-\delta_{\text{group}}|=\gamma^{k}|R^{*}_{w}(t+k)-R^{*}_{l}(t+k)|\leq\gamma^{k}\cdot\frac{2R_{\text{max}}}{1-\gamma},$ (24) $$

where the last step follows from the fact that the absolute value of any finite-horizon discounted sum is bounded by

$$ $\sum_{i=t+k}^{T}\gamma^{i-(t+k)}|r_{i}|\leq R_{\text{max}}\sum_{j=0}^{T-(t+k)}\gamma^{j}<\frac{R_{\text{max}}}{1-\gamma}.$ (25) $$

Therefore, for a single preference pair, the error of group-level DPO loss satisfies

$$ $|\log\sigma(\beta\Delta_{\text{group}}-\log\sigma(\beta\Delta_{\text{traj}}))|\leq\beta|\Delta_{\text{group}}-\Delta_{\text{true}}|\leq\beta\cdot\frac{2R_{\text{max}}}{1-\gamma}\gamma^{k}.$ (26) $$

Taking the expectation, we get

$$ $\mathrm{Bias}(\mathcal{L}_{\text{group}})\leq\mathrm{Bias}(\mathcal{L}_{\text{traj}})+\frac{2\beta R_{\text{max}}}{1-\gamma}\gamma^{k}.$ (27) $$

By choosing $k(\epsilon)=\left\lceil\log_{\gamma}\left(\frac{(1-\gamma)\epsilon}{2\beta R_{\text{max}}}\right)\right\rceil$, we obtain

$$ $\mathrm{Bias}(\mathcal{L}_{\text{group}}(k))\leq\epsilon=\min\{\mathrm{Bias}(\mathcal{L}_{\text{traj}}),\mathrm{Bias}(\mathcal{L}_{\text{step}})\}+\epsilon.$ (28) $$

Next, we analyze the variance of the three losses.
We derive element-wise bounds on the variance of the empirical loss

$$ $\mathcal{L}_{\bullet}=\frac{1}{|\mathcal{D}_{\bullet}|}\sum_{(u_{w},u_{l})\in\mathcal{D}_{\bullet}}\ell(\Delta_{\theta}),\quad\ell(\Delta_{\theta})\coloneqq-\log\sigma(\beta\Delta_{\theta}),$ (29) $$

for $\bullet\in\{\mathrm{traj,step,group}\}$.
All samples are generated from $N$ i.i.d. trajectories.

For trajectory-level DPO, each trajectory contributes exactly one preference pair.
The total number of samples $|\mathcal{D}_{\text{traj}}|=N$.
The $N$ pairs are i.i.d., hence the covariance terms in the variance vanish:

$$ $\mathrm{Var}(\mathcal{L}_{\mathrm{traj}})=\frac{1}{N^{2}}\sum_{i=1}^{N}\mathrm{Var}\left[\ell(\Delta_{\theta}^{(i)})\right]=\frac{1}{N}\Sigma_{\mathrm{traj}},$ (30) $$

where $\Sigma_{\mathrm{traj}}\coloneqq\mathrm{Var}[\ell(\Delta_{\theta})]$ under the trajectory-level sampling distribution.

For step-level DPO, from one trajectory we extract $T$ consecutive suffixes $(u_{t})_{t=1}^{T}$ with $u_{t}=(s_{t},\dots,s_{T})$.
The total number of samples is $|\mathcal{D}_{\mathrm{step}}|=NT$.
However, the $T$ samples inside one trajectory are highly overlapped.
Sequence $u_{t}$ and $u_{t+1}$ share $T-t-1$ identical transitions.
Therefore the covariance part in covariance is non-zero and large.

Since $|\Delta_{\theta}|\leq\frac{2R_{\mathrm{max}}}{1-\gamma}$, we have $0\leq\ell(\Delta_{\theta})\leq L_{\mathrm{max}}\coloneqq\log(1+e^{\frac{2\beta R_{\mathrm{max}}}{1-\gamma}})$.
For any $t<s\leq T$, let $o=s-t$ (number of shared steps).
The Cauchy-Schwarz inequality gives $\mathrm{Cov}(\ell_{t},\ell_{s})\leq\gamma^{o}L_{\mathrm{max}}^{2}$.
Summing over ordered pairs in one trajectory, we get

$$ $\sum_{1\leq t<s\leq T}\mathrm{Cov}(\ell_{t},\ell_{s})\leq L_{\mathrm{max}}^{2}\sum_{o=1}^{T-1}(T-o)\gamma^{o}<L_{\mathrm{max}}^{2}\frac{\gamma}{(1-\gamma)^{2}}.$ (31) $$

We now consider total variance across $N$ trajectories.
Each trajectory contributes $T$ samples, and samples from different trajectories are i.i.d.
Hence,

$$ $\displaystyle\mathrm{Var}(\mathcal{L}_{\mathrm{step}})$ $\displaystyle=\frac{1}{(NT)^{2}}\left[N\cdot T\cdot\mathrm{Var}(\ell_{t})+N\cdot 2\sum_{1\leq t<s\leq T}\mathrm{Cov}(\ell_{t},\ell_{s})\right]$ (32) $\displaystyle\leq\frac{L^{2}_{\mathrm{max}}}{NT}+\frac{2L^{2}_{\mathrm{max}}\gamma}{NT(1-\gamma)^{2}}=\frac{L^{2}_{\mathrm{max}}}{NT}\left(1+\frac{2\gamma}{(1-\gamma)^{2}}\right).$ (33) $$

$\mathrm{Var}(\mathcal{L}_{\mathrm{step}})$ is $O(\frac{1}{NT})$ but with a constant that does not degrade with $T$.

For group-level DPO, we extract $M=\lfloor T/k\rfloor$ non-overlapping groups of length $k$.
The total number of samples is $|\mathcal{D}_{\mathrm{group}}|=NM$.
Between-trajectory samples are i.i.d., while within-trajectory samples are independent by construction.
Therefore, the covariance terms in variance are zero, and

$$ $\mathrm{Var}(\mathcal{L}_{\mathrm{group}}(k))=\frac{1}{(NM)^{2}}\cdot NM\cdot\mathrm{Var}\!\left[\ell(\Delta_{\theta})\right]=\frac{1}{NM}\Sigma_{\mathrm{group}},$ (34) $$

where $\Sigma_{\mathrm{group}}\coloneqq\mathrm{Var}\!\left[\ell(\Delta_{\theta})\right]$ under the group-level distribution.
Since a sub-trajectory has smaller variance than the full trajectory, $\Sigma_{\mathrm{group}}\leq\Sigma_{\mathrm{traj}}$.
Inserting $M\geq T/k-1$ into Equation [34](https://arxiv.org/html/2510.03253v1#A4.E34), we obtain

$$ $\mathrm{Var}(\mathcal{L}_{\text{group}}(k))\leq\frac{k}{T}\cdot\frac{1}{N}\Sigma_{\mathrm{traj}}=\frac{k}{T}\mathrm{Var}(\mathcal{L}_{\mathrm{traj}}).$ (35) $$

An identical comparision with step-DPO gives

$$ $\mathrm{Var}(\mathcal{L}_{\text{group}}(k))\leq\frac{Ck}{T}\mathrm{Var}(\mathcal{L}_{\mathrm{step}}),\quad C=\frac{2\mathrm{tr}(\Sigma_{\mathrm{step}})}{\mathrm{tr}(\Sigma_{\mathrm{group}})}\geq 1.$ (36) $$

The constant $C$ depends only on $\gamma$ and $R_{\text{max}}$, and is independent of $T$,$k$, and $N$.

With $k(\epsilon)=\left\lceil\log_{\gamma}\left(\frac{(1-\gamma)\epsilon}{2\beta R_{\text{max}}}\right)\right\rceil=\Theta(\log(1/\epsilon))$, Equation [35](https://arxiv.org/html/2510.03253v1#A4.E35) and [36](https://arxiv.org/html/2510.03253v1#A4.E36) yield

$$ $\mathrm{Var}(\mathcal{L}_{\text{group}}(k))\leq\frac{C\log(1/\epsilon)}{T}\min\{\mathrm{Var}(\mathcal{L}_{\text{traj}}),\mathrm{Var}(\mathcal{L}_{\text{step}})\},$ (37) $$

which is the variance bound claimed in the proposition.
∎

###### Proposition 1 (Bias-variance trade-off of group-level DPO loss) .

###### Proof.

## Appendix E Additional Experiments

### E.1 Sub-task Performance on ALFWorld

To provide a more fine-grained analysis of our method’s capabilities, we present a detailed breakdown of success rates on the six distinct sub-task types in ALFWorld for both seen (Table [5](https://arxiv.org/html/2510.03253v1#A5.T5)) and unseen (Table [6](https://arxiv.org/html/2510.03253v1#A5.T6)) sets. The largest performance gains are often observed in the complex sub-tasks, such as Examine and Pick2, which require longer reasoning chains.

**Table 5: Sub-task success rate (%) comparison on the ALFWorld seen set.**
| Models | Sub-task | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Pick | Clean | Heat | Cool | Examine | Pick2 |  |  |
| Qwen2.5-1.5B-Instruct | 8.57 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.14 |
| SFT | 88.57 | 44.44 | 62.50 | 72.00 | 46.15 | 41.67 | 62.14 |
| RFT (Yuan et al., 2023) | 88.57 | 40.74 | 62.50 | 72.00 | 46.15 | 41.67 | 61.43 |
| ETO (Song et al., 2024) | 91.43 | 40.74 | 62.50 | 72.00 | 46.15 | 41.67 | 62.14 |
| IPR (Xiong et al., 2024) | 94.29 | 44.44 | 68.75 | 72.00 | 46.15 | 41.67 | 64.29 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 97.14 | 48.15 | 75.00 | 72.00 | 46.15 | 50.00 | 67.86 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 94.29 | 51.85 | 75.00 | 80.00 | 46.15 | 50.00 | 69.29 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 94.29 | 55.56 | 87.50 | 80.00 | 61.54 | 50.00 | 72.86 |
| \rowcolorMyBlue<br>HPL (Semantic) | 97.14 | 51.85 | 87.50 | 80.00 | 61.54 | 41.67 | 71.43 |
| Qwen2.5-7B-Instruct | 74.29 | 29.63 | 37.50 | 32.00 | 15.38 | 16.67 | 38.57 |
| SFT | 94.29 | 51.85 | 75.00 | 72.00 | 53.85 | 41.67 | 67.14 |
| RFT (Yuan et al., 2023) | 97.14 | 55.56 | 75.00 | 80.00 | 61.54 | 54.17 | 72.86 |
| ETO (Song et al., 2024) | 94.29 | 55.56 | 87.50 | 72.00 | 53.85 | 45.83 | 70.00 |
| IPR (Xiong et al., 2024) | 94.29 | 59.26 | 87.50 | 80.00 | 61.54 | 45.83 | 72.86 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 97.14 | 66.67 | 87.50 | 88.00 | 76.92 | 50.00 | 78.57 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 100.00 | 88.89 | 93.75 | 92.00 | 84.62 | 66.67 | 88.57 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 97.14 | 74.07 | 87.50 | 88.00 | 92.31 | 50.00 | 81.43 |
| \rowcolorMyBlue<br>HPL (Semantic) | 100.00 | 77.78 | 87.50 | 92.00 | 76.92 | 58.33 | 83.57 |

**Table 6: Sub-task success rate (%) comparison on the ALFWorld unseen set.**
| Models | Sub-task | Overall |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Pick | Clean | Heat | Cool | Examine | Pick2 |  |  |
| Qwen2.5-1.5B-Instruct | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| SFT | 70.83 | 64.52 | 60.87 | 76.19 | 27.78 | 35.29 | 58.21 |
| RFT (Yuan et al., 2023) | 79.17 | 61.29 | 65.22 | 76.19 | 33.33 | 35.29 | 60.45 |
| ETO (Song et al., 2024) | 83.33 | 64.52 | 65.22 | 80.95 | 38.89 | 41.18 | 64.18 |
| IPR (Xiong et al., 2024) | 83.33 | 64.52 | 65.22 | 80.95 | 38.89 | 47.06 | 64.93 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 87.50 | 80.65 | 73.91 | 85.71 | 44.44 | 52.94 | 73.13 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 83.33 | 61.29 | 69.57 | 80.95 | 33.33 | 41.18 | 63.43 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 83.33 | 61.29 | 69.57 | 85.71 | 33.33 | 35.29 | 63.43 |
| \rowcolorMyBlue<br>HPL (Semantic) | 87.50 | 80.65 | 78.26 | 76.19 | 44.44 | 52.94 | 72.39 |
| Qwen2.5-7B-Instruct | 62.50 | 54.84 | 52.17 | 52.38 | 16.67 | 17.65 | 45.52 |
| SFT | 91.67 | 80.65 | 78.26 | 85.71 | 50.00 | 58.82 | 76.12 |
| RFT (Yuan et al., 2023) | 79.17 | 74.19 | 78.26 | 76.19 | 44.44 | 52.94 | 69.40 |
| ETO (Song et al., 2024) | 95.83 | 74.19 | 78.26 | 80.95 | 55.56 | 70.59 | 76.87 |
| IPR (Xiong et al., 2024) | 95.83 | 77.42 | 73.91 | 85.71 | 61.11 | 64.71 | 77.61 |
| \rowcolorMyBlue<br>HPL (Fixed-N(3)) | 95.83 | 77.42 | 73.91 | 80.95 | 55.56 | 64.71 | 76.12 |
| \rowcolorMyBlue<br>HPL (Fixed-K(3)) | 95.83 | 80.65 | 69.57 | 80.95 | 55.56 | 70.59 | 76.87 |
| \rowcolorMyBlue<br>HPL (Uncertainty) | 100.00 | 87.10 | 82.61 | 80.95 | 66.67 | 76.47 | 83.58 |
| \rowcolorMyBlue<br>HPL (Semantic) | 100.00 | 87.10 | 82.61 | 90.48 | 77.78 | 76.47 | 86.57 |

### E.2 Case Study

#### E.2.1 ALFWorld

#### E.2.2 WebShop

#### E.2.3 InterCode-SQL

## Appendix F Prompts

### F.1 ALFWorld

### F.2 WebShop

### F.3 InterCode-SQL

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: Please see the weakness part.

At the current stage, I tend to recommend acceptance. However, I'm open to revising my final decision after the rebuttal and further discussions.
soundness: 2
strengths: 1. This work is well motivated. The paper starts with an intuitive problem, the granularity mismatch in preference optimization, and then proposes a novel solution that bridges trajectory/step-level supervisions.

2. The sample code is provided. It facilitates the reviewer to verify the numerical results independently.

3. The numerical performance is sound. The proposed methods beat the existing baselines by a clear margin.
summary: This paper proposes Hierarchical Preference Learning (HPL) to address the granularity mismatch in preference-based offline alignment for long-horizon LLM agents. Conventional Direct Preference Optimization (DPO) methods operate either at the trajectory level or the step level. HPL introduces an intermediate group-level DPO that decomposes trajectories into semantically coherent action groups, paired with a dual-layer curriculum along task complexity and sample difficulty. Experiments on ALFWorld, WebShop, and InterCode-SQL demonstrate consistent improvements over existing benchmarks.
weaknesses: 1. The presentation could be improved.  While technically rich, some methodological descriptions (e.g., the Monte Carlo approach in reward estimation and curriculum scheduler) could benefit from more intuitive explanations.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: ## Questions
1. How many additional datapoints were created by action group-level data generation in Section 3.2.2? How big is the dataset compared to step-level and trajectory-level datasets?
2. If the value of $\gamma$ actually used for experiment was practically 1 (no discount in the trajectories), does the analysis of Proposition 1 not become incoherent?
3. How much money in total was spent for the API call for semantic segmentation of the action groups, per each train-test run in Table 1?

## Suggestions
1. Could you provide a table which present a comparison between HPL and the presented baselines, in terms of the use of external powerful LLM, number of new generations for the training, and wall-clock time for generation and training phase?
soundness: 2
strengths: 1. The overall flow of the paper is well organized, allowing readers to effectively understand the paper.
2. Figures and experiment results are well polished.
3. Experiments such as ablation study in Figure 5 allow the readers to understand the effectiveness of components involved in the proposed algorithm.
summary: The paper aims to address the issue of granularity mismatch in preference-based alignment for LLM agents and proposes Hierarchical Preference Learning (HPL), which integrates preference signals across three levels of abstraction. The main focus of this work is providing action-group level of abstraction, which is done by semantically partitioning the trajectories into semantically coherent sub-tasks and optimizing a hierarchical DPO loss. The paper also suggests dual-layer curriculum learning strategy, which uses the information of sample discriminability and action group length. Experiments focused on long-horizon agent benchmarks show that HPL outperforms popular baselines.
weaknesses: 1. The paper does not have very strong theoretical motivation or support of the proposed method.
2. The experiment results are not having standard deviation information, which is crucial for reinforcing the credibility of the results.
3.  The proposed method seems to be much more complicated than the compared baselines, but the potential limitation coming from this approach is not well analyzed. HPL requires expert trajectory, generations from the reference model, reward signal, and access to a powerful model such as gpt-4o when semantic segmentation is used.
4. Some of the crucial experimental details seem to be omitted from the paper (ex. the value of $\gamma$ for the experiments, the number of generations $M$ used to evaluate the reward in Equation 2.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. **Principled mid-granularity learning:** The action-group DPO provides a theoretically grounded bias–variance trade-off—achieving lower variance than trajectory/step DPO while adding at most ε bias with group length (k=\Theta(\log(1/\epsilon))); within-trajectory sample independence further avoids the large covariance that plagues step-level pairs.   

2. **Strong and well-validated gains:** Across ALFWorld, WebShop, and InterCode-SQL, HPL (especially the semantic segmentation variant) outperforms SFT/RFT/ETO/IPR—e.g., ALFWorld-unseen 86.57%—with ablations showing the group-level loss is most critical and the dual-layer curriculum (length + difficulty) delivers additional, consistent improvements.
summary: This paper addresses the "granularity mismatch" in preference-based alignment for long-horizon LLM agents—trajectory-level DPO is too coarse for credit assignment, while step-level DPO is too myopic—and proposes **Hierarchical Preference Learning (HPL)** that combines trajectory-, step-, and a crucial **action-group** DPO operating on semantically coherent sub-tasks. HPL segments expert trajectories into action groups, constructs contrastive pairs, estimates group rewards, and trains with a dual-layer curriculum scheduled by group length (complexity) and reward-gap difficulty, yielding structured, sub-task credit assignment without sacrificing global stability. Experiments on ALFWorld, WebShop, and InterCode-SQL show HPL—especially with semantic segmentation—consistently outperforms SFT, RFT, ETO, and IPR; ablations highlight the group-level loss and the curriculum as the primary performance drivers.
weaknesses: 1. **Limited novelty:** Framing DPO at an intermediate “action-group” level feels incremental—more a structured decomposition of known step/trajectory DPO than a fundamentally new alignment principle.

2. **Theory under-validated:** The bias–variance claims and independence assumptions are not directly stress-tested (e.g., no gradient-variance or signal-to-noise measurements, nor controlled sweeps that confirm the predicted (k)–(\epsilon) trade-off).

3. **Sensitivity to segmentation/curriculum:** Success appears to hinge on segmentation quality and curriculum schedules; robustness to noisy grouping, alternative segmenters, and hyperparameter perturbations is insufficiently examined.

4. **Scope and scalability limits:** Benchmarks and models are narrow (scripted simulators, limited agents); reliance on MC rollouts and extra segmenters raises training cost and may hinder transfer to real-web or human-preference settings.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: 1. How does your approach compare to critic learning in PPO for credit assignment? If fine-grained step-level feedback is available, what is the specific advantage of segmenting trajectories into groups rather than using all step-level information directly?
2. Is training instability the primary motivation for the curriculum scheduler in group-level preference learning? Does the high variance in trajectory lengths contribute to this instability?
3. My hypothesis is that group-level preference learning may work better than step-level primarily because the prior knowledge injected during trajectory segmentation improves exploration efficiency, while step-level preferences could potentially learn the targeted credit assignment given sufficient data. I would like to ask for an in-depth analysis on this.
soundness: 2
strengths: 1. **Technically sound approach:** The paper presents a well-designed framework that thoughtfully integrates curriculum learning and multi-granularity preference labeling to enhance agent performance.
2. **Clear presentation:** The paper is well-written with effective visualizations. Figure 2 clearly illustrates the HPL framework and training procedure, making the methodology easy to follow.
summary: This paper proposes Hierarchical Preference Learning (HPL), which combines preference labeling at three different granularities to improve credit assignment in DPO-based training for LLM agents. Specifically, the method constructs preference pairs at: (1) trajectory level, (2) step level, and (3) segmented sub-trajectory (group) level. Chosen examples are sampled from demonstrations while rejected examples come from a reference policy. For group-level preferences, the authors segment trajectories and use Monte Carlo value estimation with reference policy rollouts to assign preference labels. The approach incorporates curriculum learning for group-level preference optimization. The final objective combines DPO losses from all three levels plus a behavior cloning term. Experiments on three agent benchmarks demonstrate that HPL outperforms methods using only step-level or trajectory-level preferences, with ablations showing positive contributions from each granularity level.
weaknesses: 1. **Insufficient motivation:** The paper argues that step-level DPO is "myopic" and cannot capture "synergistic value" of sub-trajectories, but this strong claim lacks adequate support. Diverse planning tasks has been successfully achieved via PPO learning with step-level rewards. The paper needs to better position itself relative to existing literature and provide more rigorous justification for why step-level preferences are fundamentally insufficient.
2. **Contradictory problem setting:** There is conceptual confusion about whether this is truly an offline alignment method. While the authors frame this as offline preference alignment, the group-level labeling procedure involves reference model rollouts and online reward labeling. If online reward queries are required, this fundamentally cannot be offline alignment.
3. **Incomplete experimental comparison:** Given the online reward labeling involved in HPL (per weakness 2), the experimental setup is incomplete and potentially unfair: i) Missing online RL baselines (e.g.PPO, GRPO), as the current trajectory-level and step-level DPO are purely offline methods. ii) Missing IL and IRL baselines.
4. **Unclear role of curriculum learning:** The motivation for incorporating curriculum learning is not well established. How does curriculum learning specifically address the granularity mismatch problem?


Decision: Accept (Poster)