Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach
Authors: Utsav Singh
Souradip Chakraborty
Wesley A. Suttle
Brian M. Sadler
Derrik E. Asher
Anit Kumar Sahu
Mubarak Shah
Vinay P. Namboodiri
Amrit Singh Bedi
Venue: ICLR 2026 Poster (2026)
Keywords: hierarchical reinforcement learning
preference based learning
Primary area: reinforcement learning
TL;DR: We present DIPPER, a preference-based learning approach to hierarchical reinforcement learning that mitigates the issues of non-stationary rewards and infeasible subgoal prediction.

Abstract: Hierarchical reinforcement learning (HRL) enables agents to solve complex, long-horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundamental challenges: (i) non-stationarity caused by the evolving lower-level policy during training, which destabilizes higher-level learning, and (ii) the generation of infeasible subgoals that lower-level policies cannot achieve. To address these challenges, we introduce DIPPER, a novel HRL framework that formulates goal-conditioned HRL as a bi-level optimization problem and leverages direct preference optimization (DPO) to train the higher-level policy. By learning from preference comparisons over subgoal sequences rather than rewards that depend on the evolving lower-level policy, DIPPER mitigates the impact of non-stationarity on higher-level learning. To address infeasible subgoals, DIPPER incorporates lower-level value function regularization that encourages the higher-level policy to propose achievable subgoals. We introduce two novel metrics to quantitatively verify that DIPPER mitigates non-stationarity and infeasible subgoal generation issues in HRL. Empirical evaluation on challenging robotic navigation and manipulation benchmarks shows that DIPPER achieves upto 40% improvements over state-of-the-art baselines on challenging sparse-reward scenarios, highlighting the potential of preference-based learning for addressing longstanding HRL limitations.

Published as a conference paper at ICLR 2026
DIRECT PREFERENCE OPTIMIZATION FOR PRIMITIVE- ENABLED HIERARCHICAL RL: A BILEVEL APPROACH
Utsav Singh " Souradip Chakraborty', Wesley A. Suttle", Brian M. Sadler', Derrik E. Asher, Anit Kumar Sahu, Mubarak Shah, Vinay P. Namboodiri," Amrit Singh Bedi
IIT Kanpur, India, Unniversity of Maryland, College Park, MD, USA
U.S. Army Research Laboratory, Adelphi, MD, USA, *University of Texas, Austin, TX, USA Oracle, University of Central Florida, Orlando, FL, USA, Unniversity of Bath, Bath, UK WWork done as a visiting research scholar at the University of Central Florida ut.save i atk.ac.i.
ABSTRACT
Hierarchical reinforcement learning (HRL) enables agents solve complex. long- horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundamental challenges: (i) non-stationarity caused by the evoly- ing lower-level policy during training, which destabilizes higher-level learning, and (ii) the generation of infeasible subgoals that lower-level policies cannot achieve. To address these challenges, we introduce DIPPER, novel HRL framework that formulates coal-conditioned HRL as bi-level optimization problem and leverages direct preference optimization (DPO) to train the higher-level policy. By learning from stationary preference comparisons over subgoal sequences rather than rewards that depend on the evolving lower-level policy, DIPPER mitigates the impact of non-stationarity on hierarchical learning. To address infeasible subgoals, DIPPER incorporates lower-level value function regularization that encourages the higher- level policy to propose achievable subgoals. We also introduce two novel metrics to quantitatively verify that DIPPER mitigates non-stationarity and infeasible subgoal generation issues HRL. We perform empirical evaluations on challenging robotic navigation and manipulation benchmarks and show that DIPPER achieves upto 40% improvements over state-of-the-ar baselines, demonstrating that preference-basee methods can effectively alleviate persistent challenges in hierarchical learning.
INTRODUCTION
Hierarchical Reinforcement Learning (HRL) offers an effective framework for tackling complex, long- horizon tasks by decomposing them into manageable sub-tasks (Sutton al., 1999; Harb et al., 2018). In goal-conditioned HRL (Dayan & Hinton, 1992; Vezhnevets et al., 2017), higher-level policy sets subgoals for lower-level policy (henceforth called the primitive policy), which executes lower level primitive actions to achieve these subgoals. This decomposition enables temporal abstraction and improves exploration efficiency (Nachum et al., 2019).
HRL methods face two persistent challenges, especially in sparse reward settings: (i) Von-stationarity- The higher-level policy's 's learning process becomes unstable due to the evolving nature of the lower-level policy (Levy et al., 2019; Nachum et al., 2018). As the lower-level policy updates, the higher-level reward function and transition dynamics shift, leading to non-stationary environment that hinders learning. (ii) Infeasible subgoal generation- The higher- level policy might generate subgoals that are beyond the current capabilities of the lower-level policy, resulting in suboptimal performance (Chane-Sanc et al., 2021). These challenges stem from the intertwined dependencies between the hierarchical levels. The higher-level policy's 's rewards and transitions depend the lower-level policy's behavior, while subgoals from the higher level simultancously shape the lower-level policy's ' actions. This bidirectional dependency creates a complex optimization landscape that traditional HRI. approaches fail to solve.
Challenges. However,

---

Published as a conference paper at ICLR 2026
Primitive- regularized
Higher Policy
Higher Level Replay Butfer
DPO
Higher
ft
Preference feedback
Maze
Place
RL
Lower primitie
Seplay Bulfer
Lower primitive
RL
Lower Reploy Butfer
(4
Ox
Environment
Environment
Kitchen
Vanilla HRL
DIPPER: Bi-level Formulation
Environments
Figure 1: DIPPER Overview: (left) In vanilla HRL,, the higher level predicts subgoals gt and gets the environment reward that depend on the lower primitive behavior, which causes non-stationarity in HRL. Also, the higher level may predict infeasible subgoals that too hard for lower primitive. (middle) In DIPPER, the lower level value function V leveraged condition higher level policy into predicting feasible subgoals. direct preference optimization (DPO) used to optimize higher level policy. Since this preference- learning approach does not depend on lower primitive, this mitigates non-stationarity. Note that since the current estimation of value function is used to regularize the higher policy, it does not cause non-stationarity. (right) Training environments: (i) maze navigation, (ii) pick and place, (tii) push, and (iv) franka kitchen.
Lack of principled formulation. We posit that key reason for these persistent challenges the absence of a mathematically precise formulation that fully captures the inter-dependencies between hierarchical policies. To address this, model HRI. as bi-level optimization problem, where the higher-level policy optimization constitutes the upper-level problem and the lower-level policy optimization forms the lower-level problem. This unified framework enables the joint optimization of both policies while explicitly modeling their inter-dependencis. Thus, we propose bi-level reformulation tailored to goal-conditioned HRL (Section 4.1).
A bi-level view clarifies the true sources of instability. While our bi-level formulation models dependencies berween hierarchical levels, fundamental challenges still remain: the higher-level policy's reward depends on the evolving lower-level policy, resulting in unstable and non-stationary learning signal. To address these limitations, we introduce DIPPER, a HRL method that leverages direct preference optimization (DPO) (Rafailov al., 2023) to train the higher-level policy preference datasets where trajectory pairs are ranked by stationary preference labels that, unlike environment rewards, do not depend on changing lower-level policy behavior. By optimizing the higher-level policy with DPO on this stationary dataset, we decouple higher-level learning from the non-stationary lower-level policy behavior, thus mitigating non-stationarity due higher-level rewards and stabilizing hierarchical training. Further, our bi-level formulation enables us to address the infeasible subgoal generation problem at the higher level, by incorporating lower-level value function regularization; this is direct consequence of our bi-level formulation and grounds subgoal proposals in the lower level's 's value function, ensuring that the higher-level policy generates feasible subgoals.
Our main contributions are as follows:
Bi-level optimization framework for HRL: We provide a mathematical formulation of HRL as a bi-level optimization problem, capturing the interdependencies between hierarchical policies and enabling principled algorithmic development (Section 4).
Mitigation of non-stationarity and infeasible subgoal generation: By adopting a bi-level approach, DIPPER leverages DPO reduce the effects of non-stationarity and infeasible subgoal generation, as demonstrated through ablation studies, analysis and novel metrics (Section Figure 3).
3. Improved performance in complex robotics tasks: Extensive experiments across diverse navigation and manipulation environments show that DIPPER achieves upto 40% improvement over state-of-the-a thrtt--rtttttttt baselines complex pick and place, push and franka kitchen tasks where other methods fail to make any meaningful progress (Section 5).
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Hierarchical Reinforcement Learning (HRL). HRL offers the benefits of temporal abstraction and improved exploration (Nachum al., 2019), enabling agents solve complex, long-horizon tasks by decomposing them into sub-tasks (Sutton et al., 1999; Barto & Mahadevan, 2003; Parr & Russell, 1998). Despite these advantages, HRL methods are fundamentally challenged by non- stationarity (Nachum et al., 2018; Levy et al., 2019) and the generation of infeasible subgoals. Prior works have sought to mitigate non-stationarity by simulating optimal lower-level primitive behav- ior (Levy et al., 2019), relabeling transitions in the replay buffer (Nachum et al., 2018; Singh & Namboodiri, 2023b;a), or assuming access to privileged information such as expert demonstrations or preferences (Singh et al., 2024; Singh & Namboodiri, 2023a:b). However, existing methods lack principled mathematical framework to explicitly model the bidirectional dependencies between hier- archical policies, where the higher-level policy's 's subgoal predictions influence lower-level behavior, while the evolving lower-level policy simultaneously destabilizes higher- learning. To address this, reformulate HRL as bi-level optimization problem, explicitly decoupling and coordinating the interdependent objectives of hierarchical policies through mathematical constraints.
Preference-based Learning (PbL), PbL applies reinforcement learning human preference data (Knox & Stone, 2009; Pilarski et al., 2011; Wilson et al., 2012; Daniel et al., 2015), pro- viding mechanism for guiding policy learning in the absence of explicit reward signals. Prior ap- proaches (Christiano et al., 2017; Lee et al., 2021) first train a reward model from human preferences and then optimize a policy based on this reward. Prior work (Singh et al., 2024) attempts address HRL -on-stationartty by leveraging reinforcement learning from human feedback (RLHF) (Christiano et al., 2017), by learning a reward function for the higher-level policy, avoiding direct dependence on the non-stationary environment rewards.
More recently, direct preference optimization (DPO) methods (Rafailov et al., 2023; 2024; Hejna et al., 2023) have emerged, which directly optimize the policy using KL-regularized maximum likelihood objective, bypassing the need for an explicit reward model. Our work builds on advances in maximum entropy RL (Ziebart, 2010) and DPO, deriving a DPO objective regularized by the lower-level policy's value function to address both non-stationarit, and infeasible subgoal generation issues in HRL. For comprehensive review of related work, see Appendix A.3.
PROBLEM FORMULATION
3.1 HIERARCHICAL REINFORCEMEN LEARNING (HRL)
Hierarchical Setup: The hierarchical formulation consists of two levels: a higher-level policy and lower-level policy. Let I. represent the overall task horizon, which is factorized L. = T x K, where T and K denote the horizons of the higher-level and lower-level policies, respectively. The higher-level policy generates subgoals every K timesteps, while the lower-level policy executes primitive actions to achieve these subgoals within the K-timestep window. Let E [1, T) and E [1, K] denote the timesteps for the higher-level and lower-level policies, respectively. We denote the timestep indexes for higher and lower levels separately.
Lower Level MDP: The lower level MDP is defined as (S, p4t,pprr4 where is the state space, and p² S x AL - A(S) denotes the transition dynamics. The lower level action space is denoted as AL. The lower-level policy x²
x S + A(A²) generates primitive actions subgoals gt A provided by the higher-level policy, and
ag IT (- 814k E is the current state. The lower-level policy is sparsely rewarded when it achieves the with 1c as an indicator function returning if the condition C holds, indicating that the subgoal 9r is achieved. In the lower-level replay buffer, transition is defined as (st-k, St, ax, r (sr+k, ag Gr), 81+4+1). We adopt maximum entropy
conditioned
subgoal 9r: p'((st&e, ak, 9t)
RL setting, where H(T²²) denotes the entropy of the lower-level policy = To learn optimal lower level policy, we maximize the expected lower level cumulative reward, formally defined as Th := arg max, L V VL((T") ( where
)) E Ey Harro
Σ (St4R) au 96) (+)+
3

---

Published as a conference paper at ICLR 2026
Here St T (-|s,, g") is the subgoal selected by the upper level for step t. and expectation is over the randomness induced by the environment transitions, lower level policy, and higher level policy. For each step t, we have ag - 9r), and for the current state S the next state is 81+++1 ~ p- (8t+k: ak, 9t) where p determines the environment dynamics, and is hence stationary.
Higher-Level MDP: The higher-level MDP is defined GAA p , where S is the state space, and G is the goal space. AA is the higher-level action space, which we set as AA = g (subgoals are drawn from the goal space). The environment reward for the higher level, y S x AA x G + R, encourages progress toward the final goal g" . A higher-level transition is stored in the replay buffer as 15..9", -9t, "e.."",e,""" -e,r" (st. 9t- 3²),)++)). We adopt maximum entropy RL setting, where H(x")) denotes the entropy of the higher-level policy n The higher-level objective is Te = arg max, (n ( H) )), where
Jn x² ()) =
Σ rss 9*) + XH(T"))
Here, À is the entropy weight-pannn parameer.. This objective can be expressed using the higher-level value function V g": ; nT") which estimates the expected cumulative reward starting from state s toward goal g², following policy x" at the higher level and x² at the lower level. The higher-level Q-function Qli (st,g", 9r; n x² estimates the expected cumulative reward for taking subgoal gr in state St to achieve the final goal
Non-stationarity in HRL. For every subgoal 9r ~ x (-|s,, predicted by the higher-level policy, the lower level policy n- is allowed to execute for k timesteps, after which the policy reaches state 814-k- The transition kernel p (St, 9t) models the distribution over next states 8r+* ~ n (st, (8t,9t) depends explicitly on the lower-level policy nh (as the lower-level policy determines the sequence of primitive actions executed to pursue It over & steps). Since n² is updated throughout training, both the higher-level reward nd (which depends the achieved state 8t44) and transition kernel pf. become non-stationary. This causes training instability in learning " Further, off-policy RL, since the lower level policy changes during training, prior transitions in the replay buffer become obsolete, further exacerbating the non-stationarity issue.
We now enlist the challenges of standard HRL-based approaches.
Challenges of HRL:
While HRL offers advantages over RL, such higher sample efficiency than flat RL through temporal abstraction and enhanced exploration (Nachum et al., 2019), it faces two fundamental challenges:
C1: Non-stationarity. Vanilla off- off-policy HRL suffers from non-stationarity due to changing behavior of the lower-level policy (Nachum et al., 2018; Levy et al., 2019), due to which the higher level reward function and transition dynamics become non-stationary. thus causing HRL non-stationarity.
C2: Infeasible subgoal generation. Since the sub-optimality in the lower-level policy affects its ability to reach given subgoal, it consequently impacts the higher-level credit assignment during subgoal generation. This causes the higher level to produce infeasible subgoals for the lower level policy (Chane-Sane et al., 2021). Thus, despite its theoretical advantages, HRL often underperforms in practice (Nachum et al., 2018).
Given these challenges, preference-based learning (PbL) methods emerge as a promising alternative by incorporating stationary human feedback to guide policy optimization without direct dependence on shifting rewards or transitions. PbL approaches, such as Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017 7) and Direct Preference Optimization (DPO) (Rafailov et al., 2023), rank trajectories via pairwise preferences to train models in complex, reward-sparse tasks. In what follows, we outline PbL fundamentals and the limitations of directly applying Pbl. approaches.
3.2 PREFERENCE BASED LEARNING (PeL)
Preference-basee learning (PbL) methods such as RLHF (Christiano et al., 2017: Ibarz et al., 2018; Lee et al., 2021) and DPO (Rafailov et al., 2023) leverage preference data to solve complex tasks.
RL from human feedback (RLHF): In this setting, the agent behavior is represented as a T Tenghh trajectory denoted of states and actions: = ((t),g)) (St+1, (5441,941)) (++1) (S ((-)-1)). 1)). The
4

---

Published as a conference paper at ICLR 2026
learned reward model to be learned is denoted by r: S - R, with parameters ø. The preferences between two trajectories, 5² and T², can be expressed through the Bradley-Tery model (Bradley & exp r(st-0im")
Terry, 1952) P& [T' - -2] Ele(i.a) exp Σ. "(t.9::55')))) where r¹ > T² implies that 7 is preferred over r². The preference dataset 'D has entries of the form (T 72, y), where = (1, 0) when r! is preferred over 7², y (0, 1) when 7² is preferred over 71, and = (0.5, 0.5) in case of no preference. In RLLE,, we first learn the reward function r (Christiano et al., 2017) using cross-entropy loss along with Madley--erry model to yield the formulation:
- log Σ r (t.
(ot )
-
(3)
t=0
where expectation is over (r', (,,2)) DD.
Direct Preference Optimization (DPO): Although RLHF provides a framework for learning policies from preferences. it involves RL training step which is often computationally expensive and unstable in practice. contrast, DPO (Rafailov et al., 2023) circumvents the need for RL step by using closed-form solution for the optimal policy of the KL-regularized RL problem (Levine, 2018): r" (a|s) = Z(s) Tref (a) (s))");) where Tref is the reference policy, x² is the optimal policy, and (s)) a normalizing partition function. DPO directly optimizes policies from preferences, bypassing explicit reward modeling, offering stable convergence (detailed in Section 5).
Challenges of directly applying PbL to HRL:
Directly using RLHF: Prior approaches leverage the advancements in PbL to mitigate HRL non- stationarity (Singh et al., 2024) by utilizing the reward model " learned using the reward model (corresponding to the preference dataset) as higher level rewards instead of environment rewards r used in vanilla HRL approaches, which depend on the sub-optimal lower primitive. However, such approaches may lead to degenerate solutions by generating infeasible subgoals for the lower-level primitive. Additionally, such approaches require RL as an intermediate step, which might cause training instability (Rafailov et al., 2023).
Directly using DPO: In temporally extended task environments like robotics, directly extending DPO to the HRT. framework is non-trivial due to three reasons: (i) such scenarios deal with multi-step trajectories involving stochastic transition models, (ti) efficient pre-trained reference policies are typically unavailable in robotics, (iii) similar to RLHF, such approaches may produce degenerate solutions when higher level policy subgoal predictions are infeasible.
4 PROPOSED APPROACH
To address the dual challenges of non-stationarity (CI) and infeasible subgoal generation (C2) in HRL, introduce DIPPER: DIrect Preference Optimization for Shimitive-Enables Hierarchical Reinforcement Learning. We first formulate HRL as a bi-level optimization problem to develop a principled framework that fully captures the bi-directional dependence between hierarchical policies (Section 4. 1). Subsequently, we introduce DIPPER, which decouples higher-level policy optimization from the non-stationary lower-level policy behavior by leveraging DPO (Rafailov et al., 2023) train the higher level policy and RL to train the lower-level policy, thus mitigating HRL non-stationarity (C1) (Section 4.2). Additionally, our principled bi-level formulation naturally yields a lower-level value function based regularization which ensures that the generated subgoals remain feasible for the lower-level policy, thus addressing the infeasible subgoal generation issue (C2) in HRL. Finally, we derive DIPPER objective, analyze its gradient, and provide the final practical algorithm.
4.1 HRL: BI-LEVEL FORMULATION
We present our livel formulation by using equation 2 and representing it as a constrained optimiza- tion problem assuming the lower level policy be optimal:
max J(x m5 ( s.t. n arg max V4 (m
(4)
5

---

Published as a conference paper at ICLR 2026
where [(T", (T²") is the higher level objective and (*xx") is the lower level value function, conditioned on higher level policy subgoals. Utilizing the recent advancements in the optimization literature (Liu et al., 2022), represent equation 4 by equivalent constrained optimization problem:
(TH,,T4 s.t. (=(**) (1* 0. n. ,
max
(5)
where, (x = mäXxF (. Notably, since the left-hand side of the inequality constraint S the constraint is satisfied only when ("(=")= (x")). Although the constraint in equation 5 holds for all states and subgoals 9. however, enforeing globally would make the problem intractable. Therefore, we relax the constraint by only considering the (st. pairs traversed by the higher-level policy. Leveraging this relaxed constraint and replacing J H. xx²) from equation 2, we propose the following approximate Lagrangian objective
always non nonnpossiivee due to the fact that (" v="(
E max 2
Σ (r(s) 9..9") + XH(TT) + A(V"($..9)) (.
(6)
We can use equation 6 to solve the HRL policies for both higher and lower level, where (t) the higher level policy learns to achieve the final goal and predict feasible subgoals to the lower level policy, and (fi) the lower level policy learns to achieve the predicted subgoals. However, directly optimizing equation 6 requires knowledge of higher level reward function r# which is a function of nh, which is non-stationary. Further, using RL optimize the objective may lead to unstable learning (Rafailov et al., 2023).
4.2 DIPPER
To overcome these challenges, we propose DIPPER, our hierarchical approach that leverages primitive-regularized DPO objective to optimize the hierarchical policies. Using the objective in equa- tion 6 to derive the sward-polics equivalence equation and replacing it in equation 3, we get the following DIPPER objective:
Lo -E,,,,, log [ log -.!!! (gHls) - 33 log म. (g) 9².
-X((V= (s) v (s) (52. (s..
where (7,,72,0) y) ~ ID represents a preference pair sampled from the preference dataset D, represents the sigmoid function, X is regularization weight parameter, and / is entropy weight parameter. The complete derivation of the DIPPER objective is provided in the Appendix A. This DIPPER objective optimizes the higher-level policy using primitive regularized DPO, thus decoupling the higher-level learning from non-stationary lower-level policy rewards.
Analyzing DIPPER gradient: We further analyze the DIPPER objective by interpreting the gradient with respect to higher level policy n.." denoted as:
vLo - BT (2
Σ
o (f
higher weight for incorrect preference
v Vlog " (gl (
V Vlg " (of (0f1
(8)
incrense likelihood of T1
decrense likelihood of T3
where f(*t. (e) = log x (gelse, (1)stg")) XsV 9t) - V (t), which acts as an implicit reward model determined by the higher-level policy and the lower-level value function. This objective increases the likelihood of preferred trajectories while decreasing the likelihood of dis ones. Based on the strength of the KL constraint, the examples are weighted based how inaccurately the implicit reward model f(s,, gt. 9²) ranks the trajectories. This implicit reward acts as a value function regularizer that conditions the higher-level policy generate feasible subgoals.

---

Published as a conference paper at ICLR 2026
Practical algorithm: The DIPPER objective in Eqn. requires calculation of optimal lower- level value function (st. 9t), which is computationally expensive. We accordingly consider an approximation 9t) to replace V. (t), where update the value function V (st. gradient m times for every policy update, to get the following objective:
T-1 (glog -. g - Bog -.. ( (g]ls]
Co -B( log o
(9)
gt) (V- v
4.3 HIGHLER-LEEEL PREFERENCE DATASET COLLECTION
In the vanilla PbL framework, preferences are elicited from human feedback (Christiano et al., 2017). However, collecting large amounts of human preference feedback for PbL is computationally expensive. To avoid this, we follow the caimiive-in-the-beee 2024) and collect preference feedhack using implicit sparse rewards ((*t, 9" "0"0""""00 9t.
(PiL) approach in PIPER (Singh et al.,
determine labels y between trajectories r² and 72_ Let us assume that a higher-level trajectory from the preference dataset is represented as {(Sc.k= 9:) 9t)) )= For collected trajectory
{(St=*: (: )== and final goal 9", we compute per-step reward f(Stok, ," = [[l= - [ where (t+(k+1) is the fixed state reached after & timesteps toward Jr from Stok- Notably, this per-step reward depends on the reached states St=(k+1) and not the lower level policy behavior. The total score for
trajectory T is computed as f((T)) = Er=o f(Seaki "aoikgg".... 9t). For pairs (²), label prefers 7 > 7² if its total score R(r") exceeds R(r2). Since the preference labels of these trajectory pairs do not depend on the lower-level policy behavior, the collected preference dataset remains stationary. This stationary preference dataset is used train the higher-level policy via DPO, thus mitigating HRL non-stationarity. The pseudo-code for D DIPPER is provided in the Appendix A.2.
5 EXPERIMENTS
In our empirical analysis, ask the following questions:
(1) How well does DIPPER perform against baselines? How well does DIPPER perform in complex robotics control tasks against prior hierarchical and non oon-ierarchhcal baselines?
(2) Does DIPPER mitigate HRL limitations? How well does DIPPER mitigate the issues of non-stationarity (C1) and infeasible subgoal generation (C2) in HRL?
(3) What the impact of our design decisions on the overall performance? Can we concretely justify our design choices through extensive ablation analysis?
Task details:
We assess DIPPER on four robotic navigation and manipulation environments: (i) maze navigation, (ii) pick and place (Andrychowicz et al., 2017), (iii) push, and (ie) franka kitchen environment (Gupta et al., 2019). These are formulated as sparse reward scenarios, where the agent only rewarded when comes within a distance of the goal. Due this, these environments are hard where the agent must extensively explore the environment before coming across any rewards. As an example: in franka kitchen task, the agent only receives sparse reward after achieving the final goal (e.g. successfully opening the microwave and then turning on the gas knob).
Environment details: We provide the implementation and environment details Appendix A.6 and A.7, and the implementation code in the supplementary. We provide the preference data analysis in Appendix A.4 The main objective our empirical analysis to evaluate our approach on sparse- reward hong-horizon tasks. In the franka kitchen task, the agent is sparsely rewarded only when it completes the final task, e.g open the microwave and turn on the gas knob. These nuances prohibit prior baselines from performing well in these tasks, which makes these test beds appropriate for empirical evaluations. Finally, for harder tasks such as pick and place, push and franka kitchen, we assume access one human demonstration and incorporate imitation learning objective at the lower level to accelerate learning. However, we apply the same assumption consistently across all baselines to ensure fairness.
7

---

Published as a conference paper at ICLR 2026
1.0
0.4
0.8
Success Rate
Success Rate
Success Rate
0.8
0.3
0.3
0.6
Success
0.4
0.2
0.2
0.4
0.2
0.1
0.1
0.2
0.0
200k
600K
1.0M
200k 600k
100k 300k 500k
100k 300k 500k
Timesteps
Timesteps
Timesteps
Timesteps
(a) Maze navigation
(b) Pick and place
(c) Push
(d) Kitchen
DIPPER DIPPER-NO-V DPO-FLAT
SAGA PIPER
HAC RAPS
HIER
DAC FLAT
Figure 2: Success Rate plots. This figure illustrates the success rates across four sparse-reward maze navigation and robotic manipulation tasks, where the solid lines represent the
and the shaded areas denote the standard deviation across 5 different seeds. We evaluate DIPPER against several baselines. Although HAC. SAGA and RAPS outperform DI DIPPER the easier maze task, they to perform well in other challenging tasks, where PIPERR demonstrates strong performance and significantly outperforms the baselines.
We employ DPO instead over standard policy gradient methods due to stable convergence properties without explicit reward modeling or intermediate RL steps (detailed comparisons against policy gradient methods are provided in Experiment section 5). We tune the search, with ablation studies showing balanced values yield optimal performance without extreme sensitivity.
via grid
(1) How well does DIPPER perform against baselines?
In this section, we compare DIPPER against multiple hierarchical and non- non- non-hierarchical baselines. Please refer to Figure 2 for success rate comparison plots and subsequent discussion. The solid line and shaded regions represent the mean and standard deviation, averaged over seeds.
Comparison with DPO Baselines.
This is an ablation of DIPPER without primitive regularization, The primitive regularization approach in DIPPER regularizes the higher level policy predict feasible subgoals. We employ this baseline to highlight the critical role of generating feasible subgoals. As shown in Figure 2, DIPPER outperforms this baseline, demonstrating the importance of feasible subgoal generation in achieving superior performance.
DIPPER-NO-V baseline:
This is single-level DPO (Rafailov et al., 2024) implementation. Note that since we do not have access to a pre-trained model as a reference policy in robotics scenarios like uniform policy
DPO-FLAT baseline:
reference policy, which effectively translates to an additional objective of maximizing the entropy of the learnt policy. DIPPER is an hierarchical approach which benefits from temporal abstraction and improved exploration, as seen in Figure which shows that D DIPPER significantly outperforms this baseline.
generative language modeling,
Comparison with Hierarchical Baselines.
We compare DIPPER with SAGA (Wang et al.. 2023), a hierarchical approach that employs state conditioned discriminator network training to address non-stationarity, by ensuring that the high-level generates subgoals that align with the current state of the low-level policy. We find that although SAGA performs well in the maze task, fails to solve harder tasks where DIPPER significantly outperforms. This demonstrates that SAGA suffers from non-stationarity in harder long horizon tasks, whereas DIPPER is able to better mitigate non-stationarity issue such tasks.
SAGA baseline:
This baseline (Singh et al., 2024) leverages RLHF to learn higher level reward ensure fair comparison, we implement an ablation of PIPER without HER (Andrychowicz et al., 2017). DIPPER is able to outperform this PIP ER ablation on all tasks, showing that our DPO based approach avoids training instability caused by RL, and is able to better mitigate non-stationarity in HRI.
PIPER baseline:
function to address HRL non-stationarity.
RAPS baseline: Here, we consider RAPS (Dalal et al., 2021) baseline, which employs behavior priors at the lower level for solving the task. Although RAPS is framework for solving robotic tasks where behavior priors are readily available, it requires considerable effort to construct such priors and struggles perform well in their absence, especially when dealing with sparse reward scenarios.
8

---

Published as a conference paper at ICLR 2026
Distance (*10P)
Distance (*1 (=10%)
Distance (*10P)
30
Distance 100P)
30
Subgoal
20-
Subgoal
Subgoal
Subgoal
20
20
40
10
10
20
(a) Maze navigation
(b) Pick and place
(c) Push
(d) Kitchen
10.0
Average Q-Values
10
Average
V-Values
2
Average V-Vule
7.5
Average
V-Values
1.0
5.0
1
0.5
2.5
0.0
(c) Maze navigation
(f) Pick and place
(g) Push
(h) Kitchen
DIPPER - DIPPER-NO-V - HAC
RAPS
HIER
Figure 3: (Row 1) Subgoal Distance Metric. We compare DIPPER with DIPPER-NOO No-V, HAC, RAPS, HIER baselines, based on average distance between subgoals predicted by the higher level policy and subgoals achieved by the lower level primitive. DIPPER consistently generates low average distance values, which implies that in DIPPER, the higher level policy generates achievable subgoals that induce optimal lower primitive goal reaching behavior. This shows that DIPPER is able to address non-stationary HRL and generate feasible subgoals. (Row 2) Lower Q-Function Metric. We compare DIPPER with DIPPER-ME ER-NN-V HAC, RAPS, HIER baselines, based on average lower level Q function values for the subgoals predicted by the higher level policy. DIPPER consistently produces large Q-function values, thus inducing optimal lower policy behavior thus mitigating non-stationarity and predicting feasible subgnals.
Indeed we empirically find this to be the case, since although RAPS performs exceptionally well in maze navigation task, it fails perform well in other sparse complex manipulation tasks.
We also implement HAC (Levy et al., 2019) baseline that mitigates non-stationarity HRL by simulating optimal lower level primitive behavior. HIRO (Nachum et al., 2018) is another such baseline that addresses non-stationarity. however since HAC has been found to outperform HIRO, we chose compare with HAC. Although HAC performs well in maze task, struggles to perform well in harder tasks. DIPPER outperforms this baseline in 3 out of 4 tasks.
HAC baseline:
HIER baseline: We also implement HIER, vanilla HRL baseline implemented using SAC (Haarnoja et al., 2018) at both hierarchical levels, but it fails to outperform DIPP PER on any task.
Comparison with Horerrerrlll Baselines.
DAC baseline: (DAC) (Kostrikov et al., 2018) is a single-level baseline using one demonstration per task. Despite having access to privileged information, DAC still struggles to perform well.
FLAT baseline: We also implement a single-level SAC policy, but it fails to show any progress, verifying that hierarchical abstraction key to effective performance in complex tasks.
(2) Does DIPPER mitigate HRL limitations?
Prior work lacks principled metries for quantifying non-stationarity (CI) and infeasible subgoal generation (C2) in HRL. To address this gap, we introduce two novel metrics specifically designed to measure these challenges. Using these metrics, we empirically demonstrate that DIPPER effectively mitigates both non-stationarity (C1) and infeasible subgoal generation (C2) in HRI.
We compare DIPPER with DIPPER-NO-V, HAC, RAPS, HIER baselines subgoal distance metric: the average distance between subgoals predicted by the higher level policy and subgoals achieved by the lower level primitive. low average distance value implies that the predicted subgoals are feasible, thus inducing optimal lower level policy behavior (note that the optimal, lower-level policy is stationary, and thus avoids non mttss-ttiinart).. Figure 3 (Row 1) shows that DIPPER consistently generates low average distance values, thus mitigating -tatiiationar (C1). Low average distance values imply that high-level policy in D DIPPER generates achievable subgoals for the lower primitive due to primitive regularization, thus addressing (C2).
Subgoal Distance Metric.

---

Published as a conference paper at ICLR 2026
Lower Q-Function Metric. Here, compare DIPPER average lower level Q function values for the subgoals predicted by the higher policy. High Q-values imply that the lower policy expects high returns for the predicted subgoals. Such subgoals are feasible and induce optimal lower primitive behavior. Figure 3 (Row 2) shows that DIPPER consistently leads to large Qvalues, showing that produces subgoals with high predicted returns, indicating feasibility, and thus directly addressing both (C1) and (C2).
(3) What the impact of our design choices?
We perform ablations to analyze our design choices. We first analyze the effect of varying regular- ization weight À hyper-parameter in Appendix A.5 Figure 4. À controls the strength of primitive regularization: if À is too small, we lose the benefits of primitive regularization leading to infeasible subgoals prediction. Conversely, À is too large, the higher-level policy fails achieve the final goal by repeatedly predicting trivial subgoals. We also analyze the effect of varying / Appendix A.5 Figure Excessive 3 causes exp-exxporation, preventing optimal subgoal prediction; whereas insufficient 3 limits exploration, risking suboptimal predictions.
in
DISCUSSION
Limitations and future work. Our DPO-based hierarchical formulation raises an important question: since DIPPER uses DPO for training the higher-level policy, does it generalize to out-of- out-of-distribution states and actions better than reward-moeel-bee -aasd RI.? Comparing with hierarchical RLHF could provide useful insights. Moreover, applying DIPPER to high-dimensional subgoal spaces remains challenging. We leave these directions for future work.
Conclusion. In this work, we introduce DIPPER, novel hierarchical approach that employs primitiverregularizedd DPO to mitigate the issues of non-stationarity and infeasible generation in HRL. DIPP ER employs primitive-regulent regularized token-level DPO objective efficiently learn higher level policy, and RL to learn the lower level primitive policy, thereby mitigating non-stationarity in HRI. We formulate HRI. bi-level optimization objective ensure that the higher level policy generates feasible subgoals. Based on empirical results showing up to 40% improvement, DIPP ER an important step towards learning effective control policies for solving complex roboties tasks.
ETHICS STATEMENT
This work introduces DIPPER, a hierarchical reinforcement learning framework for robotic control. We acknowledge the broader societal implications of advancing autonomous robotic capabilities, particularly regarding potential impacts on automation and employment. Additionally, the training of hierarchical policies across multiple environments entails energy consumption contributing to environmental impact. However, our methodology utilizes standard robotic simulation environments preference datasets, thereby eliminating
and does not involve human subjects risks related to annotator bias or personal data privacy.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide complete mathematical formulations and proofs in Section 4.1 and Appendix A. with the full procedure in Algorithm 1. All hyperparameters, including learning rates and regularization coefficients (X, (3), are detailed Appendix Table We describe the configurations for all four robotic environments and baseline implementation details in Section and Appendix A.6. Upon publication, we will release our complete codebase, including the bi-level optimization implementation and evaluation scripts. Computational infrastructure specifications are available in Appendix A.4 to facilitate replication.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin. Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. CoRR, abs/1707.01423 2017. URL hhtt:://arx: wrgg abs///////// /070. 01 955
10

---

Published as a conference paper at ICLR 2026
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13:341-379, 2003.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL https: //tpp/appppp. semanticscholar.
Srg/Corrpn ID: 125209808.
Zehong Cao, Kaichiu Wong, and Chin-Teng Lin. Human preference scaling with demonstrations for deep reinforcement learning. arXiv preprint arXiv:200..12004 2020.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference Machine Learning, pp. 1430-1440. PMLR, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Systems, 34:21847-21859 2021.
Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 39:389 2015.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decompo- sition. CoRR, cs. cs/999995,,4,, 1999. URL https: //apiiiv///xxivoo// atxiv.iiiii...io
org/ abs/ / cs 0999999051444 4.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 0028-4039, 2021.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint MrXiv:1910.11956. 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801 10111101011220000 .012990 2018. URL http: / aarxiv.. org/abs
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with a deliberation cost. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.136:3 2023.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations atari, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. cetpimiinttoractoo-ooitoo Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv: 180.002999 2018.

---

Published as a conference paper at ICLR 2026
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training, 2021.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review,
arXiv preprini 1805. 00909, 2018.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In International Conference Learning Representations, 2019.
Alex Lewandowski. Calarina Muslimani, Dale Schuurmans, Matthew E Taylor, and Jun Luo. Rein- forcement teaching. arXiv preprint arXiv:2204. 2022.
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. arXiv preprint NRKi::1900.088000 2019.
Bo Liu, Mao Ye, Stephen Wright, Peter Stone, and Qiang Liu. Bome! bilevel optimization made easy: A simple first-order approach. Advances in neural information processing systems, 35: 17248-17262, 2022.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint 1rXi:: 10618, 2019.
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. arXiv preprint arXiv:18/2.002835 2018
Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. CoRR, abs/2110.036555 2021. URL https: /arxiv. org/ org/abs// / 2111 10. 03655.
abs/
Ronald Parr and Stuart Russell, Reinforcement learning with hierarchies of machines. In M. Jordan, M. Kearns, and S. Solla (eds.), Advances in Neural Information Processing Systems, volume 10. MIT Press, 1998.
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard Sutton. Online human training of myoelectric prosthesis controller via actor-critic reinforcement learning. In 2011 IEEE international conference on rehabilitation robotics, pp. 1-7. IEEE, 2011.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q** Your language model is secretly a q-function. arXiv preprint arXiv:2404. 12358, 2024.
Utsav Singh and Vinay P Namboodiri. Crisp: Curriculum inducing primitive informed subgoal prediction. arXiv preprint arXiv:2304.03555 2023a.
Utsav Singh and Vinay P Namboodiri. Pear: Primitive enabled adaptive relabeling for boosting hierarchical reinforcement learning. arXiv preprint TrXiiv2306.0639. 2023b.
Utsav Singh, Wesley A Suttle, Brian M Sadler, Vinay P Namboodiri, and Amrit Singh Bedi. Piper: Primitive-informed preference-based hierarchical reinforcement learning via hindsight relabeling. arXiv preprint arXiv:2404.13423 2024.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 (1:1122::8112211 1999.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference Machine Learning, pp. 3540-3549. PMLR, 2017.
12

---

Published as a conference paper at ICLR 2026
Vivienne Huiling Wang. Joni Pajarinen, Tinghuai Wang, and Joni-Kristian Kämäräinen. State- conditioned adversarial subgoal generation. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 10184-10191, 2023.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. arXiv preprint arXiv:2101.06521, 2021.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
3 Problem Formulation
3.1
Hierarchical Reinforcement Learning (HRL)
3.2
Preference Based Learning (PbL)
4
Proposed Approach
4.1
HRL: Bi-Level Formulation
4.2
DIPPER
4.3
Higher-level preference dataset collection
5
Experiments
Discussion
10
Appendix
15
A.1 Derivation of the DIPPER Objective
15
A.2 DIPPER Algorithm
16
A.3
Related Work
17
A.4
Preference Data Usage
18
A.5
Additional Ablation Experiments
18
A.6
Implementation details
19
A.6.1 Additional hyper-parameters
19
A.7 Environment details
20
A.7.1 Maze navigation task
20
A.7.2 Pick and place and Push Environments
20
A.8
Environment visualizations.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DERIVATION OF THE DIPPER OBJECTIVE
Here, we provide the complete derivation of thee DIPPER objective.
1. Bi-level Objective:
Following Eqn (7) in the main text, the approximate Lagrangian for the bi-level HRL is represented
as:
max E
Σ (r(st,90,9") + XH(x=) + X(V 9:) v.
(10)
This equation is direct consequence of the proposed bi-level formulation of the HRL problem (Section 4. Here r(t,, 9t,"") 96) is the high level reward, H(* denotes the entropy with respect to the higher level policy, (st,, gt) is the lower level value function, (st 91) represents the optimal lower level value function, and À is the weight hyper- parameter.
Let R2 9t.9") A(V ge) (se 9t)), then we can rewrite equation 10 as:
Σ R. + x
E # 1
(11)
2. Bellman Equation for High-Level Q-function: Using equation 11, and as originally explored in Garg et al. (2021), the relationship between the optimal future return and the current timestep return for the higher level policy is captured by the following bellman equation:
{
R. + V.# (s++1,9") if 8841 isn't terminal, Rs
Qtd (+t.9" ;9))==
(12)
if 8141 is terminal.
In non-terminal cases, using the last equation write:
R₂ + V." (s++1.9") Q. (st.g", .c))
(13)
We can reformulate he items in this equation to represent the reward as:
R º (st.g" 9t) (5(+1199'"")
(14)
Thus, there a bijection between the reward function R. and the corresponding optimal Q-function Q (st,9t.9").
Trajectory-wise Expansion:
Inspired from Rafailov et al. (2024), we consider the problem in token-level MDP setting (for trajectory T). We consider equation and take a summation over t E [0, T - 1] to derive the following:
T
R
Σ
1Q" ("..g",,))) v." (s++1.9"))
(15)
(=0
1=
Now, we add and substract V.H (so, on the RHS:
Σ (s.
(s..9" .g:)
v. (s++1.9")) + V.H (so.g") v." (so.g").
(16)
This can be re-arranged as:
T-1
R V. (so. 9¹)
Σ
Q (st,g" .9) V.H (st. g))"
(17)
1=0
1=:
15

---

Published as a conference paper at ICLR 2026
Now, by definition, the advantage function A (st, 9-, (t) is defined as: A (st. 90, 9t) Q. ("t.g" :9) V. (st.g)) We can re-place the term Q g",,"''0000 .t)) V. (st, 9¹) in the last equation by A (st, --, 96) to yield:
T-1
T-1
Σ
R = V (so. g¹) + Σ
(A. (st.
(s g())..
(18)
Now, based on result (Ziebart, 2010) for maximum entropy RL setting: (sc. g". 9t) 3 log (gelst, g²)),")),,> we replace the ad advantage term in last equation yield:
T Σ
RR = V. (50.g") +
(3 log n. (grlst.g)))
(19)
Recall that we had defined R₂ r(s.,gt. + X(VT (s,, gt)) Replacing R₂ on the LHS in the last equation yields:
Σ (eot,,n.g)) + (st,((t, - (sr, = V (50. g²) + Σ ( log n. (geloe.g")) (20)
Re-arranging this term by bringing the À weighted term from LHS to RHS, we get:
T-1
Σ rstt. gt. ) V.H(so. (so. g²)
(3 Bog " (gelse.g") (se. V.L (21)
4. Primitive-Regulariiee DPO Objective:
Finally, we use the LHS ( T-1 r(st, 9t,9")) derived in equation 21, and substitute it in equation from the main paper, to yield our primitive regularized DPO objective Lo as:
Lo -RGr ?2 log Σ (og th (o!lst,g") - B log -H (02
- - V. at)
( v -V
(22)
Note that terms (. )²) is the same for both trajectories and hence it cancels. This DIPPER objective optimizes the higher-level policy using primitive regularized DPO,
A.2 DIPPER ALGORITHM
Here, provide the DIPPER pseudo-code.
16

---

Published as a conference paper at ICLR 2026
Algorithm DIPPER
{}. 2: Initialize lower level replay buffer R' = (}.
1: Initialize preference dataset D
3: for i -.... do
4: // Collect higher level trajectories using nH and lower level trajectories using x,,
5: and store the trajectories in D and R respectively.
6: // After every m timesteps, relabel ' using preference feedback
7:
Lower level value function update for each gradient step in t=0 to k do Optimize lower level value function V to get VL
8:
9:
10:
// Higher level policy update using DIPPER for each gradient step do
12:
// Sample higher level behavior trajectories.
(T'. y)
14:
Optimize higher level policy x" using equation 9.
15:
Lower primitive policy update using RL for each gradient step do Sample p from R²4,
16:
18:
Optimize lower policy n² using SAC.
A.3 RELATED WORK
Hierarchical Reinforcement Learning. HRL is elegant framework that promises the intuitive benefits temporal abstraction and improved exploration (Nachum et al., 2019). Prior research work has focused on developing efficient methods that leverage hierarchical learning efficiently solve complex tasks (Sutton et al., 1999; Barto & Mahadevan, 2003; Parr & Russell, 1998; Dietterich, 1999). foal-conditioned HRL is important framework in which higher-level policy assigns subgoals to lower-level policy (Dayan & Hinton, 1992; Vezhnevets et al., 2017), which executes primitive actions on the environment, Despite its advantages, HRL faces challenges owing to its hierarchical structure, as goal-conditioned RL based approaches suffer from non-stationarity in off-policy settings where multiple levels are trained concurrently (Nachum al., 2018: Levy al., 2019). These issues arise because the lower level policy behavior is sub-optimal and unstable. Prior works deal with these issues by either simulating optimal lower primitive behavior (Levy et al., 2019), relabeling replay buffer transitions (Nachum et al., 2018: 8; Singh & Namboodiri, 2023b;a), or assuming access to privileged information like expert demonstrations or preferences (Singh et al., 2024; Singh & Namboodiri, 2023a;b). Recent efforts explore intrinsic options mitigate non-stationarity and infeasible subgoals through self-supervised discovery, such as HIDIO (Zhang et al., 2021), which learss task- aggostic options via entropy minimization sub-trajectories for diverse exploration in sparse-reward robotics, achieving improved sample efficiency over flat RL. Similarly, HIPPO (Li al., 2019) unifies low and high-level optimization by enabling joint training reduce inter-level dependencies, yet it remains reward- reward-dependent and prone to non non-stationarity in off- off-policy updates. In contrast, we propose novel bi-level formulation to mitigate non-stationarity and regularize the higher-level policy to generate feasible subgoals for the lower-level policy.
Related efforts in automatic curriculum learning employ bi-level optimization generate adaptive goals or task sequences for lower-level policies, such as (Narvekar & Stone, 2018) who learn curriculum policies in bi-level MDP to sequence tasks for efficient transfer to a target domain, and (Lewandowwkii et al., 2022) who use bi-level optimization for multi-objective goal generation in continuous navigation environments to balance exploration, rewards, and con-stationarity through currieulum-driven subgoal refinement. These methods improve sample efficiency in HRL but often rely on heuristics, predefined task graphs, or additional outer-loop RL solvers for curriculum design.
Behavior Priors. Some prior work relies hand-crafted actions or behavior priors to accelerate learning (Nasiriany et al., 2021; Dalal et al., 2021). While these methods can simplify hierarchical learning, their performance heavily depends the quality of the priors; sub-optimal priors may lead to sub-optimal performance. In contrast, ours is an end-to-end approach that does not require prior specification, thereby avoiding significant expert human effort.
17

---

Published as a conference paper at ICLR 2026
Preference-based Learning. A variety of methods have been developed in this area to apply inforcement learning (RL) to human preference data (Knox & Stone, 2009; Pilarski et al., 2011; Wilson et al., 2012; Daniel et al., 2015), that typically involve collecting preference data from human annotators, which is then used to guide downstream learning. Prior works this area (Christiano al., 2017; Lee al., 2021) first train reward model based on the preference data, and subsequently employ RL to derive an optimal policy for that reward model. More recent approaches have focused on improving sample efficiency using off-policy policy gradient methods (Haarnoja et al., 2018) to learn the policy. Recently, direct preference optimization based approaches have emerged (Rafailov al., 2023; 2024; Hejna al., 2023), which bypass the need learn reward model and subsequent RL step, by directly optimizing the policy with a KL-regularized maximum likelihood objective corresponding to pre-trained model. this work, we build on the foundational knowledge maxi- mum entropy RL (Ziebart, 2010), and derive token-level direct preference optimization (Rafailov et al., 2023; 2024) objective regularized by lower -level primitive, resulting in an efficient hicrarchical framework capable of solving complex robotic tasks.
A.4 PREFERENCE DATA USAGE
Here, we quantify the amount of pair-wise comparisons.
Preference Data Volume:
In our experiments, we collected approximately 10,000 pairwise trajectory comparisons per environ- ment, depending task complexity and horizon length. We ensured that these pairwise comparisons represented a diverse array of trajectories, including both near-optimal and suboptimal behaviors, provide meaningful supervision for the high-level policy. We present the amount of pairwise comparisons (Pairs per Million Env Steps) as follows:
Environment
Pairwise Comparisons
Env Steps (M)
Pairs per M Steps
Maze
10,000
1.35
7,407
Pick & Place
10,000
0.9
11,111
Push
10,000
0.775
12,903
Kitchen
10,000
0.45
22,222
Table Pairwise comparisons and efficiency across environments.
A.5 ADDITIONAL ABLATION EXPERIMENTS
Here, we provide additional ablations to analyze the effect of varying regularization weight hyper-parameter and / hyper-parameter.
0.3
Success Rate
Success Rate
Success Rate
Success Rate
0.7
0.3
0.3
0.2
0.5
0.2
0.3
0.1
0.1
0.1
200k 600k 1.0M Timesteps
100k 300k
500k
100k 300k 500k 700k Timesteps
100k 300k 500k Timesteps
Timesteps
(a) Maze navigation
(b) Pick and place
(c) Push
(d) Kitchen
X-1E3 5-54-
X-5E-3 A-522
X-1E-4
A-1E-2
X=1E-5
Figure 4: Regularization weight ablation. This figure depicts the success rate performance for varying values of the primitive regularization weight A. When À is too small, we loose the benefits of primitive-infomma regularization resulting poor performance, whereas too large À values can lead to degenerate solutions. Hence, selecting appropriate value is essential for accurate subgoal prediction and enhancing overall performance.
18

---

Published as a conference paper at ICLR 2026
Success Rate
0.3
Success Rate
0.3
Success Rate
Success Rate
0.3
0.6
0.2
0.2
0.1
0.1
0.1
0.2
200k 600k 1.0M
100k 300k 500K 700k
100k 300k 500k
100k 300k 500k Timesteps
Timesteps
Timesteps
Timesteps
(a) Maze navigation
(b) Pick and place
(c) Push
(d) Kitchen
0.05
0.1
0.2
0.3
0.4
0.5
Figure 5: Max-ent parameter ablation. This figure illustrates the success rate performance for different values of the max-ent parameter & hyper- parameter. This parameter controls the exploration in maximum entropy formulation. If 8 too large, the higher-level policy may perform extensive exploration but stay away from optimal subgoal prediction, whereas /3 too small, the higher-level might not explore and predict sub-optimal subgoals. Hence, selecting an appropriate / value is essential for enhancing overall performance.
A.6 IMPLEMENTATION DETAILS
We conducted experiments on two systems, each equipped with Intel Core processors, 48GB of RAM, and Nvidia Geforce GTX 1080 GPUs. The experiments included the corresponding timesteps taken for each run. For the environments (i) (iv), the maximum task horizon T is set to 225, 50, 50, and 225 timesteps, respectively, with the lower-level primitive allowed execute for 15, 7, and 15 timesteps. We used off oof-po Soft Actor Critic (SAC)Haarioja et al., 2018) to optimize & Ba, 2014). Both the actor and critic
the RL objective, leveraging the Adam optimizer(Kingnnn networks consist of three fully connected layers with 512 neurons per layer. The total timesteps for experiments in environments (i) (iv) are 1.35e6, 9e5, 1.3E6, and 6.3e5, respectively.
For the maze navigation task, 7-degree-or -aa---offffffeeee freedom (7-DoF) robotic arm navigates four-room maze with its gripper fixed at table height and closed, maneuvering to reach goal position. In the pick-and-place task, the 7-DoF robotic arm gripper locates, picks up, and delivers square block to the target location. In the push task, the arm's gripper must push the square block toward the goal. For the kitchen task, 9-DoF Franka robot is tasked with opening microwave door as part of predefined complex sequence to reach the final goal. We compare our approach with the Discriminator Actor-Critic (Kostrikov al., 2018), which uses single expert demonstration. Although this study doesn't explore it, combining preference- based learning with demonstrations presents an exciting direction for future research (Cao al., 2020).
To ensure fair comparisons, we maintain uniformity across all baselines by keeping parameters such neural network layer width, number of layers, choice of optimizer, SAC implementation settings, and others consistent wherever applicable. In RAPS, the lower-level behaviors are structured as follows: For maze navigation, we design a single primitive, reach, where the lower-level primitive moves directly toward the subgoal predicted by the higher level. For the tasks, we develop three primitives: gripper-reach, where the gripper moves to a designated position (æ;, Vi, =;); gripper-open, which opens the gripper; and gripper-close, which closes the gripper. In kitchen task, we use the action primitives implemented in RAPS (Dalal et al., 2021).
and push
A.6.1 ADDITIONAL HYPER-PARAMETERR
Here, we enlist the additional used in DIPPER:
19

---

Published as a conference paper at ICLR 2026
Table 2: Hyperparameter Configuration
Parameter
Value tanh
Description
activation
activation for reward model
layers
3
number of layers in the critic/actor networks
hidden
512
number of neurons in each hidden layer
Q_
0.001
critic learning rate actor learning rate for experience replay
Ir
0.001
buffer size
IE7
clip_obs
200
clip observation
n_cycles
per epoch
batches
10
training batches per cycle
batch_size
1024
batch size hyper-parameter reward batch size for DPO-FLAT
swward_bttch_ng size
50
random_epp _eps
0.2
percentage of time random action is taken
alpha
0.05
weightage parameter for SAC
noise_eps
0.05
std of gaussian noise added to not-completely-random actions
norm_eps
0.01
epsilon used for observation normalization
norm_clip
5
normalized observations are cropped to this value beta 1 for Adam optimizer
adam_betal
0.9
adam_beta2
0.999
beta 2 for Adam optimizer
A.7 ENVIRONMENT DETAILS
A.7.1 MAZE NAVIGATION TASK
In this environment, a 7-DOF robotic arm gripper must navigate through randomly generated four- room mazes. The gripper remains closed, and both the walls and gates are randomly placed. The table is divided into rectangular W x И grid, with vertical and horizontal wall positions Wp and Hp selected randomly from the ranges (1, W - 2) and (1, H - 2), respectively. In this four-room also randomly chosen from (1, Wp 1), (Wp + 1, W - 2), (1, Hp 1), and p + 1, H 2). The gripper's height remains fixed at table height, and it must move through the maze to reach the goal, marked by a red sphere.
setup, gate positions
For both higher and lower-level policies, unless Stated otherwise, the environment consists of continuous State and action spaces. The State is encoded as a vector [p. MV] where p represents the gripper's current position, and M is the sparse maze representation. The input to the higher-level policy is a concatenated vector [p, M, g). with g representing the goal position, while the lower-level policy input is [p. My,,), sy), where Sy is the subgoal provided by the higher-level policy. The current position of the gripper is treated as the current achieved goal. The sparse maze array M is 2D one-hot vector, where walls are denoted by value of I and open spaces by O.
In our experiments, the sizes of p and M are set to and 110, respectively. The higher-level policy predicts the subgoal .g. so its action space aligns with the goal space of the lower-level primitive. The lower-level primitive's 's action, a, executed in the environment, is a 4-dimensional vector, where each dimension a; E [0, 1). The first three dimensions adjust the gripper's position, while the fourth controls the gripper itself: indicates fully closed, 0.5 means half-closed, and means fully open.
A.7.2 PICK AND PLACE AND PUSH ENVIRONMENTS
In the pick-and-place environment, 7-DOF robotic arm gripper is tasked with picking up square block and placing it at a designated goal position slightly above the table surface. This complex task involves navigating the gripper to the block, closing it to grasp the block, and then transporting the block to the target goal. In the push environment, the gripper must push a square block towards the goal position. The State is represented by the vector jp, o, q, e). where p is the gripper's current position, is the block's 's position on the table, is the relative position of the block to the gripper, and contains the linear and angular velocities of both the gripper and the block.
The higher-level policy input is the concatenated vector [p, o, q. e, g], where g denotes the target goal position, while the lower-level policy input is [p, o, q, e, sg), with 8y being the subgoal provided by
20

---

Published as a conference paper at ICLR 2026
the higher-level policy. The current position of the block is treated the achieved goal. In experiments, the dimensions for p, O, 9. and are set to 3, 3. 3. and 11, respectively. The higher-level policy predicts the subgoal sy. the action and goal space dimensions align. The lower-level action a is a 4-dimensional vector, where each dimension a, falls within the range [0. 1]. The first three dimensions adjust the gripper's position, and the fourth controls the gripper itself (0 for closed, 1 for open). During training, the block and goal positions are randomly generated, with the block always starting on the table and the goal placed above the table at fixed height.
A.8 ENVIRONMENT VISUALIZATIONS
Here, provide some visualizations of the agent successfully performing the task.
Figure Maze navigation task visualization: The visualization is a successful attempt at performing maze navigation task
Figure 7: Pick and place task visualization: This figure provides visualization of a successful attempt at performing pick and place task
AARALA
Figure 8: Push task visualization: The visualization is successful attempt at performing push task
Figure 9: Kitchen task visualization: The visualization is successful attempt at performing kitchen task
21


Reviews:
[{"review_id": "v3AJExzp2d", "reviewer": "Reviewer_W15h", "summary": "This introduces a hierarchical RL framework (DIPPER) aimed to tackles two HRL issues: 1. non-stationarity of the higher-level environment and 2. infeasible subgoal generation. The framework is composed of a high level goal-conditioned HRL with Direct Preference Optimization (DPO) and standard RL at the low level. This work formalize HRL as a constrained bi-level optimization problem. \n\nExperiments on four challenging sparse-reward navigation and manipulation tasks (random mazes, pick-and-place, push, and Franka kitchen) show that DIPPER substantially outperforms both standar and hierarchical baselines, including prior preference-based HRL methods. This work also uses subgoal distance and low-level Q value to empirically support claims about reduced non-stationarity and improved subgoal feasibility.", "questions": "1. Are preferences generated from ground-truth reward (e.g., via Bradley–Terry over cumulative environment reward), or from some kind of synthetic labeling procedure? How many preference queries are used per environment, and how does performance scale with the number of preferences?\n\n2. The work approximates V^{L*} by updating V^L_m for m gradient steps between policy updates. How sensitive is DIPPER to m? Intuitively, if V^L_m is poor in the earlier portion of training, the regularizer might be actively pushing the high-level toward bad subgoals. Is there any safeguard (e.g., annealing λ) or empirical evidence that this does not happen?\n\n3. Some suggestions on the presentation: \nThe work could benefit from having a more clear overview diagram that explicitly shows the bi-level view and how it leads to the primitive-regularized DPO block, with arrows from lower-level value function to the high-level DPO loss.\nIn the experiments, it would help to explicitly list the baselines and their key differences in a table, including whether they use preferences, whether they use primitive-informed regularization, and whether they are hierarchical or flat.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The paper gives a clean bi-level optimization formulation of goal-conditioned HRL, rewriting the hierarchy as an upper-level problem with a constraint that the lower-level policy is (locally) optimal. In contrast to PIPER, which learns a reward model and then uses RL/RLHF on top of it, this work directly trains the high-level policy via DPO on a preference dataset, which allows the framework to avoid modeling explicit rewards and avoids having a second RL loop. \n\n2. Significance: This paper provides a preference-based, DPO-driven recipe with a clear regularizer derived from a bi-level formulation that is not present in prior HRL work. The experimental results also suggest an improvement of around 40% in success rates on harder sparse-reward tasks, compared to state-of-the-art baselines that already use preferences and primitive-informed regularization. \n\n3. Clarity: The authors provide pseudo-code for their framework and clearly describe the alternation between preference-trajectory collection and updates of the high-level DPO objective, low-level value function, and low-level SAC policy. The introduction also clearly states the two central issues (non-stationarity and infeasible subgoals). \n\n4. Quality: The derivation from bi-level HRL to a constrained problem, then to a Lagrangian, and finally to a primitive-regularized DPO objective is detailed and internally consistent.", "weaknesses": "1. The bi-level formalization in this paper is in some sense rephrasing standard HRL coupling (optimal lower-level policy conditioned on higher-level subgoals) in the language of constrained optimization. The authors could consider adding more analysis of the resulting bi-level problem such as convergence guarantees, regret bounds, and when the relaxed constraint and approximate value function yield near-optimal behavior. \n\n2. This paper mentions that one benefit of DPO over RLHF is computational simplicity and stability. This statement is not quantified in terms of training time, memory etc for DIPPER vs RLHF-based alternatives (such as PIPER or a simple RLHF high-level baseline). ALso, sample efficiency is mainly reflected in success-rate vs timestep plots. There could be a more fine-grained analysis of preference-query complexity or number of environment transitions until convergence.\n\n3. The paper claims their framework works on “long-horizon complex robotic tasks”, however the four environments are standard MuJoCo-style navigation/manipulation tasks (with some challenge enhancements like random mazes and sparse kitchen rewards). The work could consider adding vision-based setting with more complex hierarchical structures. Prior works like CRISP/PEAR do report real-robot experiments with more challenging perception and dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AruSETHV07", "reviewer": "Reviewer_ywxX", "summary": "This paper has proposed a novel hierarchical reinforcement learning framework, which employs preference-based learning in the high-level policy to mitigate the non-stationary issue in hierarchical policy learning. The proposed framework is evaluated in a set of simulated long-horizon tasks.", "questions": "1.\tHow sensitive is DIPPER to the scale or bias of human preferences? Would synthetic or learned preferences generalize effectively?\n\n2.\tCould DIPPER be extended to fully autonomous preference learning (e.g., self-generated ranking signals) without human input?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe research problem of non-stationarity is significantly important in the hierarchical reinforcement learning domain.", "weaknesses": "1.\tThe technical contribution of this work is low, which is a direct application of preference-based learning to high-level policy optimization.\n\n2.\tExperiments are limited to simulated environments. Demonstrations in real robots or transfer scenarios would significantly strengthen the empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IOLHXKWBUJ", "reviewer": "Reviewer_DfyA", "summary": "The authors present DIPPER, a hierarchical RL method that directly optimizes environment reward and human preferences. Given human preferences, the authors derive a bi-level optimization objective that train the policy to increase the likelihood of the preferred trajectories with the high-level policy while simultaneously updating the low-level policy to try to maintain value improvement for the same objective. The specific objective is derived from DPO, and helps mitigate non-stationarity common to most HRL methods.", "questions": "Instead of a subgoal distance metric (fig 3), why not directly measure the success rate of the lower level policy’s ability to reach goals? This is the actual metric that matters for addressing the subgoal feasability problem, right?\n\nRe: annotation cost, prior work in reward learning has demonstrated that VLMs can be useful for obtaining preferences in place of humans: https://rlvlmf2024.github.io/, have the authors looked into this or tried out similar approaches at a small scale?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Comparisons:** The authors compare against a wide array of relevant baselines, demonstrating great performance against said baselines on manipulation tasks especially. \n\n**Motivation:** Non-stationarity in HRL is a big issue and this paper presents a well-motivated solution for it.\n\n**Experiments:** The experiments are performed on tasks well-suited for HRL, and the analysis on goal distance prediction against HIER and HAC demonstrates that DIPPER’s objective encourages sampling reachable goals for the lower-level policy.\n\n**Clarity:** The paper is overall quite clear and the walkthrough of how to obtain the objective was both interesting and easy to read.", "weaknesses": "**Figures**: All of the results figures have small font that make them harder to read, and are also clearly image files put into overleaf instead of vectorized PDFs. The results figures should have thicker lines for each baseline, more spacing between baselines on the legend (in fig 2), and larger font for the x and y axes labels and ticks labels.\n\n**Annotation cost:** As authors admit, there is a high annotation cost to obtaining labels with human prediction. \n\n**Experiments:** Given the fact that the authors have human annotations, it seems that more challenging tasks could’ve been demonstrated in the experiments. This is not a reason to reject the paper, however I will list this as a slight weakness of the paper.\n\n**Minor issues:**\n\n- There’s related work which also mitigates non-stationary and infeasible subgoal generation by modeling *intrinsic* options: e.g., https://arxiv.org/abs/2101.06521, some comparison against this work and any follow-ups in the related works section would be beneficial\n- There’s also related work on unifying low level and high level policy optimization: https://sites.google.com/view/hippo-rl\n- L363: “we maintain empirical consistency across all baselines to ensure fair comparisons” — what does this actually mean? be specific here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDscZ5GQIz", "reviewer": "Reviewer_fdjG", "summary": "The authors propose DIPPER, a hierarchical reinforcement learning (HRL) framework that formulates HRL as a bi-level optimization problem. In this formulation, the higher-level “teacher” policy and lower-level “student” policy are jointly optimized while explicitly modeling their interdependence. To mitigate the non-stationarity caused by the evolving lower-level policy, the authors leverage Direct Preference Optimization (DPO) to train the higher-level policy on stationary human preference data rather than environment rewards. Additionally, they introduce a lower-level value function regularization term that encourages the higher-level policy to propose feasible and achievable subgoals.", "questions": "I included my questions in the weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper provides a rigorous bi-level mathematical formulation of hierarchical reinforcement learning (HRL).\n\nThe use of real human preference data is commendable. Conducting human-in-the-loop experiments introduces significant complexity and time overhead, yet provides stronger evidence for the method’s practical relevance\n\nThe authors empirically evaluate whether DIPPER actually addresses the two challenges they identify, non-stationarity and infeasibility of subgoal, rather than assuming these issues are solved. This careful, hypothesis-driven evaluation is a strong aspect of the paper.\n\nThey explore how varying key hyperparameters (λ, β) affects performance, showing an understanding of what aspects drive DIPPER’s behavior.", "weaknesses": "**Stationarity of Human Preferences**\n\nThe paper assumes that human preferences mitigate non-stationarity, yet human preferences themselves are not necessarily stationary.\nPrior work suggests that people’s judgments can change over time [1], particularly when confidence is low. \n\nQuestion: How would DIPPER handle cases where human preferences shift if the same trajectories are presented multiple times? Would the algorithm adapt?\n\n**Missing Related Work**\n\nThe related work section omits relevant approaches in automatic curriculum learning that use bi-level optimization to generate goals for lower-level policies.\nExamples include Narvekar and Stone (2019) [2] and Muslimani et al. (2023) [3]. \nThese should be discussed to contextualize DIPPER within existing curriculum learning frameworks.\n\n**Failure Cases and Environment Suitability**\n\nDIPPER fails in the simple maze task. The paper does not explain why.\n\nQuestion: What types of environments does DIPPER work best in?\n\nIn environments where DIPPER outperforms baselines (Pick & Place and Push), success rates remain low (around 30%).\n\nQuestion: Why does performance plateau at this relatively low level? Would longer training or additional preference data improve results?\n\n\n**Baselines**\n\nThe authors remove the HER component from PIPER “for fairness,” but this substantially alters the original algorithm.\nThis change may make the comparison less meaningful.\n\nQuestion: How would DIPPER perform against the original PIPER implementation (with HER) as described in the source paper?\n\nThe authors mention tuning DIPPER’s hyperparameters using grid search, but it is unclear whether the same effort was made for the baselines.\n This introduces a potential unfair advantage if baselines are not tuned equivalently.\n\n**Evaluation Metrics and Interpretation**\n\nThe “lower Q-function metric” measures Q-values for subgoals proposed by the higher-level policy. However, this could be inflated if the higher policy proposes easy subgoals.\n\nQuestion: Do the proposed metrics actually capture whether DIPPER learns progressively more difficult subgoals over time?\n\n\n**Human Preference Data**\n\nThe paper provides limited details about the human annotators.\n\nQuestions:\n\nWho were the annotators (experts or laypeople)?\n\nWere they trained or provided with guidance?\n\nWhat were their demographic backgrounds?\n\n**Most importantly, was this data collection ethics-approved?**\n\nThese details are crucial to assess the reliability and reproducibility of the preference data.\n\n\n**Sources:**\n[1]Visser et al (2005) https://pubmed.ncbi.nlm.nih.gov/16022057/\n\n[2] Narvekar and Stone (2019) https://arxiv.org/pdf/1812.00285\n\n[3] Muslimani et al (2023) https://arxiv.org/pdf/2204.11897", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)