PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning
Authors: Utsav Singh
Vinay P. Namboodiri
Venue: ICLR 2025 Poster (2025)
Keywords: Hierarchical reinforcement learning
Learning from demonstrations
Primary area: reinforcement learning
TL;DR: We effectively leverage expert demonstrations using our adaptive relabeling based approach to deal with non-stationarity in the context of hierarchical reinforcement learning.

Abstract: Hierarchical reinforcement learning (HRL) has the potential to solve complex long horizon tasks using temporal abstraction and increased exploration. However, hierarchical agents are difficult to train due to inherent non-stationarity. We present primitive enabled adaptive relabeling (PEAR), a two-phase approach where we first perform adaptive relabeling on a few expert demonstrations to generate efficient subgoal supervision, and then jointly optimize HRL agents by employing reinforcement learning (RL) and imitation learning (IL). We perform theoretical analysis to bound the sub-optimality of our approach and derive a joint optimization framework using RL and IL. Since PEAR utilizes only a few expert demonstrations and considers minimal limiting assumptions on the task structure, it can be easily integrated with typical off-policy \RL algorithms to produce a practical HRL approach. We perform extensive experiments on challenging environments and show that PEAR is able to outperform various hierarchical and non-hierarchical baselines and achieve upto 80% success rates in complex sparse robotic control tasks where other baselines typically fail to show significant progress. We also perform ablations to thoroughly analyze the importance of our various design choices. Finally, we perform real world robotic experiments on complex tasks and demonstrate that PEAR consistently outperforms the baselines.

Published as a conference paper at ICLR 2025
PEAR: PRIMITIVE ENABLED ADAPTIVE RELABELING FOR BOOSTING HIERARCHICAL REINFORCEMENT LEARNING
Utsav Singh
Vinay P Namboodiri
CSE Deptr.
CS Deptt.
IIT Kanpur, India utsavz@iitk. ac.
University Bath, Bath, UK vpn22@bath.ac ac.uk
ABSTRACT
Hierarchical reinforcement learning (HRL) has the potential to solve complex long horizon tasks using temporal abstraction and increased exploration. However, hi- erarchical agents difficult train due to inherent non-stationarity. We present primitive enabled adaptive relabeling (PEAR), a two-phase approach where we first perform adaptive relabeling
few expert demonstrations generate effi- cient subgoal supervision, and then jointly optimize HRL agents by employing reinforcement learning (RL) and imitation learning (IL). We perform theoreti- cal analysis to bound the sub-optimality
our approach and derive a joint op- timization framework using RI. and TI.. Since PEAR utilizes only few expert demonstrations and considers minimal limiting assumptions on the task struc- ture, it can be easily integrated with typical off-policy RL algorithms to pro- duce a practical HRL approach. We perform extensive experiments on challeng- ing environments and show that PEAR is able outperform various hierarchical and non-hierarchical baselines and achieve upto 80% success rates in complex sparse robotic control tasks where other baselines typically fail to show signifi- cant progress. We also perform ablations to thoroughly analyse the importance of our various design choices. Finally, we perform real world robotic experiments on complex tasks and demonstrate that PEAR consistently outperforms the baselines.
INTRODUCTION
Reinforcement learning has been successfully applied to number of short-horizon robotic manipu- lation tasks (Rajeswaran et al., 2018; Kalashnikov et al., 2018; Gu et al., 2017: Levine et al., 2016). However, solving long horizon tasks requires long-term planning and is hard (Gupta et al., 2019) due to inherent issucs like credit assignment and ineffective exploration. Consequently, such tasks require large number of environment interactions for learning, especially in sparse reward scenar- ios (Andrychowicz et al., 2017). Hierarchical reinforcement learning (HRL) (Sutton et al., 1999; Dayan and Hinton, 1993; Vezhnevets et al., 2017: Klissarov et al., 2017: Bacon al., 2017) is an elegant framework that employs temporal abstraction and promises improved exploration (Nachum al., 2019). In goal-conditioned feudal architecture (Dayan and Hinton, 1993; Vezhnevets al., 2017), the higher policy predicts subgoals for the lower primitive, which in turn tries to achieve these subgoals by executing primitive actions directly on the environment. Unfortunately, HRL suffers from non-stationarity (Nachum et al., 2018; Levy et al., 2018) in off-policy HRL. Due to continu- ously changing policies, previously collected off-policy experience is rendered obsolete, leading to unstable higher level state transition and reward functions.
Some hierarchical approaches (Gupta et al., 2019; Fox et al., 2017: Krishnan et al., 2019) segment the expert demonstrations into subgoal transition dataset, and consequently leverage the subgoal dataset to bootstrap learning. Ideally, the segmentation process should produce subgoals that prop- erly balance the task split between hierarchical levels. One possible approach of task segmentation is to perform fixed window based relabeling (Gupta et al., 2019) on expert demonstrations. Despite be- ing simple, this is effectively brute force segmentation approach which may generate subgoals that

---

Published as a conference paper at ICLR 2025
dj
S
Expert
Padabelaad Expert Demronsrraioos
Demonsmmiions
(a) Maze
(h) Pick
[e)
(d) Hollow
(e) Rope
Figure 1: Adaptive Relabeling Overview: We segment ex- pert demonstrations by consccutively passing demonstration states subgoals the lower primitive, and finding the state where Q_ (s, ss, a.) < Quareah (here s; = ss). Since sa was the last reachable subgoal, it selected subgoal for initial state #g- The transition added to Dg. and the process continues with sg as initial state.
Figure Subgoal evolution: With training. the lower primitive
improves, the higher level subgoal predictions (blue spheres) become betler and harder, while always heing achievable by lower primitive. Row depicts initial training, Row 2 depicts mid-way through train- ing, and Row depicts end training. This generates a curriculum achievable subgoals for lower primitive (red spheres: final goal).
either too easy too hard according the current goal achieving ability of the lower primitive, thus leading to degenerate solutions.
The main motivation of this work is produce a curriculum of feasible subgoals according to the current goal achieving capability of the lower primitive. Concretely, the value function of the lower primitive is used to perform adaptive relabeling on expert demonstrations dynamically generate a curriculum of achievable subgoals for the lower primitive. This subgoal dataset is then used to train an imitation learning based regularizer, which is used to jointly optimize off-policy RI objective with II regularization. Hence, our approach ameliorates non-stationarity in HRI by using primitive enabled IL regularization, while enabling efficient exploration using RL. We call our approach: primitive enabled adaptive relabeling (PEAR) for boosting HRL.
The major contributions of this work are: (i) our adaptive relabeling based approach generates ef- ficient higher level subgoal supervision according to the current goal achieving capability of the lower primitive (Figure 2). (ii) we derive sub-optimality bounds to theoretically justify the ben- efits of periodic ce-pouulation using adaptive relabeling (Section 4.3), (iit) we perform extensive experimentation on sparse rohotic tasks: maze navigation, pick and place, bin, hollow, rope manipu- lation and franka kitchen empirically demonstrate superior performance and sample efficiency of PEAR over prior baselines (Section 5 Figure 3), and finally, (iv) we show that PEAR demonstrates impressive performance in real world tasks: pick and place, bin and rope manipulation (Figure 6).
RELATED WORK
Hierarchical reinforcement learning (HRL) (Barto and Mahadevan, 2003; Sutton et al., 1999; Parr and Russell, 1998; Dietterich, 2000) promises the advantages of temporal abstraction and increased exploration (Nachum et al., 2019). The options architecture (Sutton et al., 1999; Bacon et al., 2017; Harutyunyan al., 2018: Harb al., 2018; Harutyunyan al., 2019; Klissarov al., 2017) learns temporally extended macro actions and a termination function to propose an elegant hierarchical framework. However, such approaches may produce degenerate solutions in the absence of proper regularization. Some approaches restrict the problem search space by greedily solving for specific goals (Kaelbling, 1993; Foster and Dayan, 2002), which has also been extended to hierarchical RL (Wulfmeier et al., 2019; 2021; Ding et al., 2019). In goal-conditioned feudal learning (Dayan and Hinton, 1993; Vezhnevets et al., 2017), the higher level agent produces subgoals for the lower primitive, which in turn executes atomic actions the environment. Unfortunately, off-policy HRL approaches are cursed by stationnin issue. Prior works (Nachum et al., 2018; Levy et al., 2018) deal with the non-stationarity by relabeling previously collected transitions for training goal- conditioned policies. In contrast, our proposed approach deals with non-stationarity by leveraging adaptive relabeling for periodically producing achievable subgoals, and subsequently using im-
2

---

Published as a conference paper at ICLR 2025
itation learning based regularizer. We empirically show in Section 5 that our regularization based approach outperforms relabeling based hierarchical approaches on various long horizon tasks.
Prior methods (Rajeswaran et al., 2018; Nair et al., 2018; Hester et al., 2018) leverage expert demon- strations to improve sample efficiency and accelerate learning, where some methods use imitation learning to bootstrap learning (Shiarlis et al., 2018; 8; Krishnan et al., 2017; 2019; Kipf et al., 2019). Some approaches use fixed relabeling (Gupta et al., 2019) for performing task segmentation. How- ever, such approaches may cause unbalanced task split between hierarchical levels. In contrast, our approach sidesteps this limitation by properly balancing hierarchical levels using adaptive re- labeling. Intuitively, we enable balanced task split, thereby avoiding degenerate solutions. Recent approaches restrict subgoal space using adjacency constraints (Zhang et al., 2020), employ graph based approaches for decoupling task horizon (Leee et al., 2023), or incorporate imagined subgoals combined with Kl--contrrineed policy iteration scheme (Chane-Sane et al., 2021). However, such approaches assume additional environment constraints and only work on relatively shorter horizon tasks with limited complexity, (Kreidich et al., 2020) is inter-level cooperation based approach for generating achievable subgoals, However, the approach requires extensive exploration for selecting good subgoals, whereas approach rapidly enables effective subgoal generation using primitive enabled adaptive relabeling. In order to accelerate RL, recent works firstly learn behavior skill pri- (Pertsch al., 2020; Singh et al., 2021) from expert data or pre-train policies over related task, and then fine-tune using RL. Such approaches largely depend on policies learnt during pre-training, and are hard train when the source and target task distributions are dissimilar. Other approaches either use bottleneck option discovery (Salter et al., 2022b) or behavior priors (Salter et al., 2022a; Tirumala et al., 2022) to discover and embed behaviors from past experience, or directly hand-design action primitives (Dalal et al., 2021; Nasiriany et al., 2022). While this simplifies the higher level task, explicitly designing action primitives is tedious for hard tasks, and may lead to sub-optimal predictions. Since PEAR learns multi-level policies in parallel, the lower level policies can learn required optimal behavior, thus avoiding the issues with prior approaches.
3 BACKGROUND
Off-policy Reinforcement Learning We define our goal-conditioned off-policy RL setup as fol- lows: Universal Markov Decision Process (UMDP) (Schaul et al., 2015) is a Markov decision pro- cess augmented with the goal space G, where M = (S, A, P. R, G). Here, S is state space, A action space, P(s [s, a) is the state transition probability function, R is reward function, and y is discount factor. (als,gg) represents the goal-conditioned policy which predicts the probability of taking action a when the state is s and goal is g. The overall objective is to maximize expected future discounted reward distribution: J = (1-r))- -' "nddd (a==(aas..)).---
[r(s ac.
Hierarchical Reinforcement Learning In our goal-conditiondd HRL setup, the overall policy i is divided into multi-level policies. We consider bi-level scheme, where the higher level policy n (syls, g) predicts subgoals 8y for the lower primitive ==(a|s, sg). π generates subgoals sy after every e timesteps and =f tries to achieve 89 within e timesteps. 7 gets sparse extrinsic reward Few from the environment, whereas - gets sparse intrinsic reward r'n from म x² gets rewarded with reward 0 if the agent reaches within 8 distance of the predicted subgoal sy. and -1 otherwise: Tin -[[[s, syll2 > 84). Similarly, n! gets extrinsic reward 0 if the achieved goal is within 8 distance of the final goal g. and -1 otherwise: Ver -1 1[1|s Ulle > 8 We assume access to small number of directed expert demonstrations = (et [ /=1 where et
(s6, ag. sf, aj 17-1,a 1).
Limitations of existing approaches to HRL Off-policy HRL promises the advantages of temporal abstraction and improved exploration (Nachum et al., 2019). Unfortunately, HRL approaches suf- fer from non-stationarity due to unstable lower primitive. Consequently, HRL approaches fail to perform in complex long-horizer vation of this work is to efficiently leverage a few expert demonstrations to bootstrap RL using IL regularization, and thus devise an efficient HRL approach to mitigate non-stationarity.
tasks, especially when the rewards are sparse. The primary moti-
3

---

Published as a conference paper at ICLR 2025
4 METHODOLOGY
In this section, we explain PEAR: Primitive Enabled Adaptive Relabeling for boosting HRL, which leverages few expert demonstrations D to solve long horizon tasks. We propose two step ap- proach: (i) the current lower primitive n² is used to adaptively relabel expert demonstrations to generate efficient subgoal supervision Dg. and (ii) off-policy RL objective is jointly optimized with additional imitation learning based regularization objective using Dg. We also perform theoretical analysis to (i) bound the sub-optimality of our approach, and (ii) propose practical generalized based framework for joint optimization using RL and II, where typical off-policy RL and IL algo- rithms can be plugged in generate various joint optimization based algorithms.
Alg Algorithm 1 Adaptive Relabeling
1: Initialize Dg 0
2: // Populating Dg
3: for each e = (s si. ..... ST in do
4: Initial state index inil f- 0
Algorithm 2 PEAR
5: Subgoal transitions D 0
6: for i = 1 to T do
Initialize D, 0
7:
Find Q values for demo sub-
2:
for = 1..... N do
goals
if 1%pp
then
8:
Compute Qx ( si ai)
4:
Clear Dg
9:
where a, = = (89-1,
5:
Populate Dg via adaptive relabeling Collect experience using n and of Update lower primitive via SAC and IT. regularization with D (Eq 6 or Eq 8) Sample transitions from D, Update higher policy via SAC and IL regularization using D, (Eq or Eq 7)
10:
1/ Find first subgoal s.t. 2x2 Qeh
6:
11:
if Q+ (S sf.0) a,) < Qu then
7:
12:
for = init to i 1 do for k (énit + 1) to do 1/ Add the transition to
8:
13:
9:
14:
10:
D
11:
15:
Add 8i-,, to D Initial state index inil 4 (i (6-1) // Add selected transitions to Dg
16:
17:
Dg f D, U D
4.1
PRIMITIVE ENABLED ADAPTIVE RELABELING
PEAR performs adaptive relabeling on expert demonstration trajectories D to generate efficient higher level subgoal transition datatset Dg. by employing the current lower primitive's 's action value function Q,' (s, sf, a,). In typical goal-conditioned RL setting, Q2= (s, sf, a;) describes the ex- pected cumulative reward where the start state and subgoal are and sf. and the lower primitive takes action a, while following policy -². While parsing D, we consecutively pass the expert demonstrations states sf as subgoals, and Qa (s, sf. a;) computes the expected cumulative reward when the start state is 8, subgoal is sf and the next primitive action is ag. Intuitively, a high value of Q,1(s, sf, a.) implies that the current lower primitive considers sf be good (highly rewarding and achievable) subgoal from current state since it expects to achieve high intrinsic reward for this subgoal from the higher policy. Hence, Q (8, sf, a;) considers goal achieving capability of current lower primitive for populating D.- We depict single pass of adaptive relabeling in Figure and explain the procedure in detail below.
Adaptive Relabeling Consider the demonstration dataset D {ei JizI where each trajectory (s of.......................... sf. Let the initial state be sh. In the adaptive relabeling procedure, we incrementally provide demonstration states sf for itive's action value function Quz (5f &f aa), where a; = x'(s
subgoals to lower prim- 1.9 sf). At every step, we compare Q e (sf, si, a;) to a threshold Qthresh (Qthresh = works consistently for all experiments). If Qr s..a,) a,) state shn1. Otherwise, we consider sf s to be a good subgoal (since it was the last subgoal with Qxn (56, o5-15 a;) >= Qthresh), and use it to compute subgoal transitions for populating Dy- Sub-
T
Qthresh:
move to next expert demonstration
4

---

Published as a conference paper at ICLR 2025
sequently, we repeat the same procedure with sq-1 the new initial state, until the episode termi- nates. This is depicted in Figure and Algorithm 1.
Periodic re-population of higher level subgoal dataset HRL approaches suffer from non- stationarity due to unstable higher level station transition and reward functions. In off-po HRL, this occurs since previously collected experience is rendered obsolete due to continuously changing lower primitive. We propose to mitigate this non-stationarity by periodically re- re-populating subgoal transition dataset D, after every p timesteps according to the goal achieving capability of the cur- ren! lower primitíve. Since the lower primitive continuously improves with training and gets better at achieving harder subgoals, 2z always picks reachable subgoals of appropriate difficulty, accord- ing to the current lower primitive. This generates a natural curriculum of achievable subgoals for the lower primitive. Intuitively, Dg always contains achievable subgoals for the current lower prim- itive, which mitigates non-stationarity in HRL. The pseudocode for PEAR is given in Algorithm 2. Figure shows the qualitative evolution of subgoals with training in our experiments.
Dealing with out-of-distributi states Our adaptive relabeling procedure uses Qn² (s6, sf, ae) to select efficient subgoals when the expert state of is within the training distribution of states used to train the lower primitive. However, if the expert states are outside the training distribution, Qrz might crroncously over-estimate the values on such states, which might result in poor subgoal selec- tion. In order address this over-estimation issue, employ an additional margin classification objective(Piot et al., 2014) 4), where along with the standard QsACc objective, we also use an additional margin classification objective yield the following optimization objective Q.t, QSAC+ arg min max max( (IE (=):::) - [Q, ($6, si a:)] Begg (f.) 12 (sh sf, a;)))
Qxy
This surrogate objective prevents over-estimation of Q₂ by penalizing states that are out of the expert state distribution. We found this objective to improve performance and stabilize learning. In the next section, we explain how use adaptive relabeling to yield our joint optimization objective.
4.2
JOINT OPTIMIZATION
Here, explain our joint optimization objective that comprises of off-policy RL objective with IL based regularization, using D, generated using primitive enabled adaptive relabeling. We consider both behavior cloning (BC) and inverse reinforcement learning (IRL) regularization. Henceforth, PEAR-IRL will represent PEAR with IRL regularization and P EAR-BC will represent PEAR with BC regularization. We first explain BC regularization objective, and then explain IRL regularization objectives for both hierarchical levels.
For the BC objective, let (s", s s seett) ~ D2 represent a higher level subgoal transition from Dg
where 8 is current state, Shezt is next state, g is final goal and sy is subgoal supervision. Let
$g be the subgoal predicted by the high level policy Th (-1s%; :") with parameters On. The BC regularization objective for higher level is as follows:
min (0HH Bx
min E (s" ( (1ss (1)5s)) syl12
(1)
0g
Similarly, let (&f, af, (here) ~ Do represent lower level expert transition where sf is current state, of Sheet is next state, ge goal and a the primitive action predicted by wh, (-1s², sg) with parameters 0g. The lower level BC regularization objective is as follows:
min JbC = min E (et dr.
8%. (aff shead)DD awxy (-lat (e)...)llll all
We now consider the IRL objective, which is implemented as GAIL (Ho and Ermon, 2016) ob- jective implemented using LSGAN (Mao et al., 2016). Let D'df be the higher level discriminator with parameters еи- Let J represent higher level IRL. objective, which depends parameters (0H. e.). The higher level IRL regularization objective is as follows:
1
max min J (0 E = max min - EE (.) Do. (-l0* ge) (D (Tex (-1s-, " - 0j² 8y
2
(3)
1
2 (60
E
[1 (8%) 1]²
5

---

Published as a conference paper at ICLR 2025
Similarly, for lower level primitive, let D., be the lower level discriminator with parameters ez. Let JB represent lower level IRL objective, which depends on parameters (0L, <L)) The lower level IRI. regularization objective is as follows:
1 B (st ( ammm, (1#5 .55) [D (To, (-|s², .zg) 0]2
max min (/L...CL) max min P2
c
P el 2
(4)
E DG D. (al) - 1]²
Finally, we describe our joint optimization objective for hierarchical policies. Let the off-policy RL objective be JH and Jg for higher and lower policies. The joint optimization objectives using BC regularization for higher and lower policies are provided in Equations 5 and respectively.
max(J - + * #00))((
(5)
max (Jzc 3 * (
(6)
8z
The joint optimization objectives using IRL regularization for higher and lower policies are provided in Equations 7 and 8 respectively.
min max + *J (0n, eH))
(7)
min man_a + v Jb(0L, EL)
(8)
ez Or
Here, vp is regularization weight hyper-parameter. We describe ablations choose te in Section 5,
4.3 SUB-OPTIMALITY ANALYSIS
In this section, we perform theoretical analysis to (i) derive sub-optimality bounds for our proposed joint optimization objective and show how our periodic re-population based approach affects per- formance, and (it) propose generalized framework for joint optimization using RL and IL. Let n and nT" be unknown higher level and lower level optimal policies. Let T be our high level policy and Th, be our lower primitive policy, where fи and 0 are trainable parameters. Drv (T1, T2) de- notes total variation divergence between probability distributions _ and πΤ. Let be unknown distribution over states and actions, G be goal space, be current state, and be the final episodic goal. We use x in the importance sampling ratio later avoid sampling from the unknown optimal policy (Appendix A.1). The higher level policy predicts subgoals 8g for the lower primitive which is executed for timesteps to yield sub-trajectories r. Let 11 and 11 be some unknown higher and lower level probability distributions over policies from which we can sample policies Th and Th- Let us assume that policies = and Th represent the policies from higher and lower level datasets DH and D,, respectively. Although DEr and Dr, may represent any datasets, in discussion, use them to represent higher and lower level expert demonstration datasets. Firstly, we introduce the é--common definition (Ajay al., 2020) in goal-conditioned policies:
Definition 1. π is commmmon in 11 if EE' sonKn www.ggg.....giiiiaeoo (D g) (fTss))))))))) S dD
Now, we define the suboptimality of policy x with respect to optimal policy x² as: Subopt (0) [J(x*) J(m)
Theorem 1. Assuming optimal policy n" is @D common in II the suboptimality of higher policy .. Hay over length sub-trajectories T sampled from di can be bounded as:
[J(TT) ( )l < Aн ØD + λr E.S.n. BR BTT ClV s, | ( (r(s,g))
(10)
second
where Xy [1-))11-9 y
8" K maz
1100

---

Published as a conference paper at ICLR 2025
Similarty, the suboptimality of lower primitive Thi can be bounded as:
[J(x"*) - J( AL ØD + AL So , (er (T1s,
(11)
where Xz. [1-7]2
2
Rmaz
o00
The proofs for Equations 10 and 11 provided in Appendix A.I. We next discuss the effect of training on the two terms in RHS of Equation 10, which bound the suboptimality of "gn
Effect of adaptive relabeling sub-optimality bounds We firstly focus on the first term which is dependent on ΦD- Since we represent the generated subgoal dataset as Dg- we replace Φn with dD,- In Theorem 1, we assume the optimal policy x² to be ΦD, common in I1 Since Φn, denotes the upper bound of the expected TV divergence between x² and TB. ΦDD provides a quality measure of the subgoal dataset D, populated using adaptive relabeling. Intuitively, a lower value of ΦD, implies that the optimal policy x' is closely represented by Dg. or alternatively, the samples from D, are near optimal. As the lower primitive improves with training and able to achieve harder subgoals, and since Dy is re- re-populated using the improved lower primitive after every p timesteps, TD continually gets closer to x², which results reduced value of ΦD. This implies that due to decreasing first term, the suboptimality bound in Equation 10 gets tighter, and consequently (( gets closer to optimal JTT") objective. Hence, our periodic re-population based approach generates natural curriculum of achievable subgoals for the lower primitive, which continuously improves the performance by tightening the upper bound.
Effect of IL regularization on sub-optimality bounds Now, we focus on the second term in (T|s, g) with expectation over
Equation 10, which is TV divergence between (Tis,g g) and (8 TH
Th I 9 G. As before, D is replaced by dataset Dg- This term can be viewed as imitation learning (IL) objective between expert demonstration policy 7H, and current policy = where TV divergence is the distance measure. Due to this IL regularization objective, as policy THn gets closer to expert distribution policy TB, with training, the LHS sub-optimality bounds get tighter. Thus, our proposed periodic TI. regularization using Dg tightens the sub-optimality bounds in Equation 10 with training, thereby improving performance.
!!
Generalized framework We now derive our generalized framework for the joint optimization ob- jective, where we can plug in off-the-shelf RL and IL methods yield generally applicable prac- tical HRL algorithm. Considering sub-optimality is positive (Equation 9), we can use Equation 10 derive the following objective:
J(TT) 2 [
λ * Be (ld( (1)s (r18, du * OD
(12)
RT. term
const. wrt D₂
II. regularization term
H where (considering (T|s,, g) as TA and T8g (T|s, g) as TB). d(FAl(TB) = Drv (TB).
Notably, the second term Ay ØD in RHS of Equation 12 is constant for a given dataset Dg- Equation 12 can be perceived as minorize maximize algorithm which intuitively means: the overall objective can be optimized by (i) maximizing the objective J( via RL, and (it) minimizing the distance measure d between T and TH (IL regulariizztiinn This formulation serves as a framework where we can plug in RL algorithm of choice for off-policy RI objective J(THH ). and distance function d of choice for II. regularization, to yield various joint optimization objectives.
In our setup, we plug in entropy regularized Soft Actor Critic (Haarnoja et al., 2018a) to maximize J (= H ). Notably, different parameterizations of yield different imitation learning regularizers.
When d is formulated as Kullback-Leibler divergence, the IL regularizer takes the form of behav- ior cloning (BC) objective (Nair et al., 2018) (which results in PEAR-BC) BC)), and when d is formulated as Censen-Shannon divergence, the imitation learning objective takes the form of inverse reinforce- ment learning (IRL) objective (which results in -EAR-IRL). We consider both these objectives in Section 5 and provide empirical performance results.
7

---

Published as a conference paper at ICLR 2025
64
Thraslpp
Maze navigation
Pick and place
Bin
bap
60
03
Treeitess
Threikess
Treestess
Hollow
Rope
Franka kitchen
PEAR-RL PEAR-BC RPL HAC RAPS RAPS NIEWNEG HEER DAC BC
FLAT SAGA
performances six sparse navigution und munipulation lasks. The solid line and shaded region represent the mean and range success rales across 5 seeds. As seen, PEAR shows impressive performance and significantly ourperforms the baselines.
Figure 3: Success rafe comparison This ligure compures the
EXPERIMENTS
In this section, we empirically answer the following questions: (i) does adaptive relabeling approach outperform fixed relabeling based approaches, (ii) is PEAR able to mitigate non-stationarity. (iii) does regularization boost performance in solving complex long horizon tasks, and (iv) What is the contribution of each of our design choices? We accordingly perform experiments on six Mujoco (Todorov et al., 2012) environments: (i) maze navigation, (ii) pick and place, (iti) bin, (iv) hollow, (v) rope manipulation, and (vi) franka kitchen. Please refer to the supplementary for video depicting qualitative results, and the implementation code.
Environment and Implementation Details: We provide extensive environment and implementa- tion details, including number and procedure of collecting demonstrations in Appendix A.3. Since the environments sparsely rewarded, they complex tasks where the agent must explore the environment extensively before receiving any rewards. Unless otherwise stated, we keep the train- ing conditions consistent across all baselines to ascertain fair comparisons, and empirically tune the hyper-parameter values of our method and all other baselines.
5.1 EVALUATION AND RESULTS
In Figure 3, we depict the success rate performance of PEAR and compare it with other baselines averaged over seeds. The primary goal of these comparisons is to verify that the proposed approach indeed mitigates non-stationarity and demonstrates improved performance and training stability.
Comparing with fixed window based approach
RPL: In order to demonstrate the efficacy of adaptive relabeling, we compare our method with Relay Policy Learning (RPL) baseline. RPL (Gupta et al., 2019) uses supervised pre-training followed by relay fine tuning. In order to ascertain fair comparisons, we use an ablation of RPL. which does use supervised pre-training. PEAR outperforms this baseline, which demonstrates that adaptive relabeling outperforms fixed window based relabeling and is crucial for mitigating non-stationarity. Since PEAR and RPL both employ jointly optimizing RL and IL based learning and only differ in adaptive relabeling, it is evident that adaptive relabeling is crucial for generating feasible subgoals.
Comparing with hierarchical baselines
RAPS: RAPS (Dalal et al., 2021) uses hand designed action primitives at the lower level, where the goal of the upper level is to pick the optimal sequence of action primitives. The performance such approaches significantly depends on the quality of action primitives, which require substantial effort to hand-design. We found that except maze navigation task, RAPS unable to perform well on other tasks, which we believe is because selecting appropriate primitive sequences is hard other harder tasks. Notably, hand designing action primitives is exceptionally complex in environments like rope manipulation. Hence, we do not evaluate RAPS in rope environment.
8

---

Published as a conference paper at ICLR 2025
Tha roo to stons
Trois Roxel ens
Tagiiigg Toxelirs
Tra
Maze
(b) Pick and place
Hollow
Rope
Kitchen
PEARHRLL
HIER
Non-stationarity metric comparison This ligure compures the average distance metric between the subgoals predicted by the higher level policy and the subguals uchieved the lower level policy during training. leading to low distances between the predicted and achieved subgoals throughout the training process. This mitigutes non-stationarity in HRI..
Figure 4:
PEAR consistenlly produces efficient subgonls
3.0
200k
12%
Teestess
(n) Maze navigation
(b) Pick and place
(c) Bin
50t
2008
100k
03
12c
2008
(d) Hollow
(c) Rope
(1) Franka kitchen
PEAR
(a) Pick
(b) Bin
(c) Rope
Figure The success rate plots success performance comparison PEAR IRL (red), PEAR BC (black) and 2AR RPL (blac) ablation. PEAR IRL and PEAR-BC clearty outperform FEAR-8PL in almost all the tusks.
Figure 6: Real world experiments in pick and place, and rope environments. Row depicts initial and Row depicts goal configuration.
HAC: Hierarchical actor critic (HAC) (Levy et al., 2018) deals with non-stationarity by relabeling transitions while assuming optimal lower primitive. Although HAC shows good performance in maze navigation task, P PEAR consistently outperforms HAC on all other tasks.
We also compare PEAR with two hierarchical baselines: HIER and HIER-NEG, which are hierarchical baselines that do not leverage expert demonstrations. HIER-NEG is hierarchical baseline where the upper level is negatively rewarded if the lower primitive fails to achieve the subgoal. Since HIER, HIER NEGG and PEAR all are hierarchical ap- proaches, we these baseline comparisons to motivate that the performance improvement is just due to use of hierarchical abstraction, but instead due to adaptive relabeling and primitive- enabled regularization. This clearly evidenced by superior performance of PEAR.
HIER-NEG and HIER:
Comparing with non-hierarchical baselines
Additionally, we consider single-level Discriminator Actor Critic (DAC) (Kostrikov et al., 2019) that leverages expert demos, single-level SAC (FLAT) baseline, and Behavior Cloning (BC) baselines. However, they fail to perform well in any of the tasks.
5.2 ABLATIVE ANALYSIS
Here, we perform ablation analysis to elucidate the significance of our design choices. We choose the byper-parameter values after extensive experiments, and keep them consistent across all baselines.
We assess whether PEAR mitigates noatioatinn in HRL by comparing it with the vanilla HIER baseline, as shown in Figure 4. We compute the average distance between subgoals predicted by the higher-level pol- icy and those achieved by the lower-level primitive throughout training. lower average distance suggests that PEAR generates subgoals achievable by the lower primitive, inducing lower primitive behavior to be optimal. Our findings reveal that PEAR consistently maintains low average dis- tances, validating its effectiveness in reducing non-stationarity. Additionally, seen in Figure 4, post-training results show that PEAR achieves significantly lower distance values than the HIER baseline, highlighting its ability generate feasible subgoals through primitive regularization.
Dealing with non-stationarity and infeasible subgoal generation in HRL:

---

Published as a conference paper at ICLR 2025
0.8
6.8
00
0.
6.
Tmollass
Treestpss
Tinellps
Timedaps
Timsdage
Timedaps
(a) Maze
(h) and
(c) Bin
(d) Hollow
Rope
Franka
Figure 7: This figure illustrates the importance of margin surrogate objective by comparing PEAR-IRL and PEAR-BC (with margin objective) with PEAR-IRL-NoO Margiin and CEAARBCC---Maage Margin (without margin objective). PEAR-IRL and PEAR- BC outperform thear non margin objective counterparts almost all tasks.
Additional Ablations: We verify the importance of adaptive relabeling by replacing it in PEAR-IRL by fixed window relabeling (as in RPL (Gupta et al., 2019)). This ablation (P -AR-IRL) consistently outperforms PEAR-RPE on all tasks (Figure 5), which shows the benefit of adaptive relabeling. Further, we compare PEAR-IRI and PEAR-BC (with margin classification objectives), and
(without margin objectives) in Fig- ure 7, PEAP IRL and PEAR- BC outperform their No Margin counterparts, which shows that this objective efficiently deals with the issue of out-of-distribution states, and induces training stability.
with
Further, analyse how varying Qthresh affects performance in Appendix A.4 Figure 8. We next study the impact of varying p. Intuitively, if p is too large, it impedes generation of a good curriculum of subgoals (Appendix A.4 Figure 9). Also, low value of p may lead frequent subgoal dataset re-population and may impede stable learning. We also choose optimal window size l for RPT. ex- periments, as shown in Appendix A.4 Figure 10. We also evaluate learning rate & in Appendix A.4 Figure 11. If e is too small, PEAR is unable to utilize IL regularization, whereas conversely if &a is too large, the learned policy might overfit. We also deduce the optimal number of expert demonstra- tions required in Appendix A.4 Figure 12. Next, we compare the performance of PEAR-IRE with HER-BC, which is a single-leve implementation of HER with expert demonstrations. As seen in Appendix A.4 Figure 13, PEAR significantly outperforms this baseline, which demonstrates the ad- vantage of our hierarchical formulation. We also provide qualitative visualizations in Appendix A.5.
Real world experiments: We perform experiments real world robotic pick and place, bin and rope environments (Fig 6). We use Realsense D435 depth camera to extract the robotic arm position, block, bin, and rope cylinder positions. Computing accurate linear and angular velocities is hard in real tasks, so we assign them small hard-coded values, which shows good performance. We per- formed 5 sets of experiments with 10 trial each, and report the average success rates. PEAR-IRL achieves accuracy of 0.8, 0.6, and 0.3, whereas PEAR-BC achieves accuracy of 0.8, 0, 0.3 on pick and place, bin and rope environments. We also evaluate the performance of next best performing RPL baseline, but it fails to achieve success in any of the tasks.
6 DISCUSSION
Limitations In this work, we assume availability of directed expert demonstrations, which we plan to deal with in future. Additionally, D, is periodically re-populated, which an additional overhead and might be bottleneck in tasks where relabeling cost is high. Notably, we side-step this limitation by passing the whole expert trajectory as a mini-batch for a single forward pass through lower primitive. Nevertheless, we plan to deal with these limitations in future work.
Conclusion and future work We propose primitive enabled adaptive relabeling (PEAR), a HRL and IL based approach that performs adaptive relabeling a few expert demonstrations to solve complex long horizon tasks. We perform comparisons with a various baselines and demonstrate that PEAR shows strong results in simulation and real world robotic tasks. In future work, plan to address harder sequential decision making tasks, and plan to analyse generalization beyond expert demonstrations. We hope that PEAR encourages future research in the area of adaptive relabeling and primitive informed regularization, and leads to efficient approaches to solve long horizon tasks.
ACKNOWLEDGMENTS
This research work was partially supported by Research-I Foundation of the Department of CSE at ITT Kanpur.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofr Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In Ad- vances in Neural Information Processing Systems (NeurIPS), pages 1-12. Curran Asso- ciates Inc., 2020. URL https::/proceeddnggg hewp:/// neurips cc/paperr2020//11ee JEbbchbeisscbb11SScfBff6-------2 pdf.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Pierre- Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems, 13(4):341-379, 2003.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), page TBD, 2021.
Murtaza Dalal, Deepak Pathak, and Ruslan Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2021. URL https: / /arxivv.vv/.. waciiivorrg/aaae owg/abgg.//ooo.oooo /2110.111...... 15360.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in Neural Infor- mation Processing Systems, pages 271-278. Morgan Kaufmann Publishers Inc., 1993.
Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function de- composition. Journal of Artificial Intelligence Research, 13:227-303, 2000. URL https: / //www. jair www.r.airrroo.... orgg index.
0266.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in Neural Information Processing Systems (NeurIPS), 2019.
David Foster and Peter Dayan. Structure in the space of value functions. Machine Learning, 49 (2-3):325-22 2002.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1665- 1674. PMLR, 2017. URL httpt:ttp//// ptattp::/pproceedingg www.mmrrrrsss..... www.s/resssee..... C/ fox1 7a html.
mlr.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. In Conference Robot Learning (CoRL), pages 729-735. PMLR, 2020.
Shixiang Gu, Ethan Holly, Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 3389-3396. IEEE, 2017. doi: 10. 109/CAA. 2017.7989385. URL https: http///eeeeeeeeeeeeeeeeee/eeeeee ieee org/ document /7989385.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 1851-1860. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 1861-1870. PMLR, 2018b. URL pettss://roocee ngs ml www..p.ees......m http://vvv/aaaaajjjao/ 8b_ html.

---

Published as a conference paper at ICLR 2025
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option: Learning options with deliberation cost. In Proceedings of the 32nd AAAI Conference on Artifi- cial Intelligence (AAAI), pages 3206-3213, 2018. URL https: //ojs aaai org. index.
Anna Harutyunyan, Peter Vranex, Pierre-Luc Bacon, Doina Precup, and Ann Nowé. Learning with options that terminate off- off-policy. In Proceedings of the 32nd AAAI Conference on Artificial In- telligence (AAAI), pages 3201-3208, 2018. URL https://oje aaai, org/ iidexxixxx... .xxx.pp.. php/ MAAI/articln wwallleeviiiiie htt./vieww//////1
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Rémi Munos, and Doina Precup- The termination critic. In Proceedings of the 36th International Conference on Machine Learn- ing (ICML), pages 2645-2653, 2019. URL http: harutyunyan htttamll1.............
mww.mmmr....... ptt.s.pree//9//
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Amold, Ian Osband, John Agapiou, Joel 7. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In Proceedings of the Thirty-Second AAAI Con- ference on Artificial Intelligence (AAAI), pages 3223-3230, 2018, URL BRRLatps::// //os:////jss//// aaai. org/index: php/ http/////aaatatttaaaaaaaaeaeeeee 11794.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Ad- vances in Neural Information Processing Systems (NeurIPS), pages 4565-4573, 2016. Scc7e2b878868chi 992d1
URL https: htttp:://prcceedinng neurips wttppppppppppp////.e.eep httppps.cccc.pppp//////..... html.
Leslie Pack Kaelbling. Learning to achieve goals. In Proceedings of the International Joint Confer- ence on Artificial Intelligence (IJCAI), pages 1094-1098, 1993.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforce- ment learning for dision-based robotic manipulation. In Conference on robot learning. pages 651-673. PMLR, 2018.
Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro stette. Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3428. PMIR, 2019.
Edward Grefen-
Martin Klissarov, Pierre-Lue Bacon, Jean Harb, and Doina Precup. Learning options end-to-end for continuous action tasks. In Proceedings of the 31 Conference on Neural Information Processing Systems (NeurIPS) Hierarchical Reinforcement Learning Workshop, 2017. URL https:/
arxiv.org/ abs 111111111112 00004.
Ilya Kostríkov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- Addressing sample inefficiency and reward bias in adversarial
son. Discriminator-actor. imitation learning. In 7th International Conference on Learning Representations (ICLR), 2019.
URL httpt:tt//// /ttp:://pppnreee http://ppeeeeeeeeeeeeeeeeee net/ tttttfoumm A
Abdul Rahman Kreidieh, Samyak Parajuli, Nathan Lichtlé, Yiling You, Rayyan Nasr, and Alexan- dre M. Bayen. Inter-level cooperation in hierarchical reinforcement learning- In Proceed- ings of the 19th International Conference Autonomous Agents and MultiAgent Systems (AA- MAS), pages 2000-2002, 2020. URL https: / /dl. acm. wogg//o///ia 5555533.5553888811 3398990.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. DDCO: Discovery of deep continuous options for robot learning from demonstrations. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL), pages 418-437, 2017. URL https: /ttp:////pcceeiiggss.. press/v78
html.
Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller, Florian T. Poko- my, and Ken Goldberg. Swirl: sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards. The International Journal of Robotics Research, 38(2- 2019. doi: 10. 10.1177007833499187743333 0278 0278364918784355
URL https: : /ww.iiioooooooooooooo /ttp//iiooo///////////
12

---

Published as a conference paper at ICLR 2025
Steven M Valle. Rapidly-explorinn random trees: A new tool for path planning. Technical report, Tech. rep., 1998.
Seungjae Lee, Jigang Kim, Inkyu Jang, and H. Jin Kim. Dhrl: A graph-baser approach for long- horizon and sparse hierarchical reinforcement learning, In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), page TBD. IEEE, 2023.
Sergey Levine. Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical actor-critic. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018. URL https:
/ //operreview. wtt.://oooooomeddadsrrroroo.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, and Zhen Wang. Multi-class generative adversarial networks with the loss function. arXiv preprint (rXiv:1611.04000 5:1057-7149, 2016.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. Advances Neural Information Processing Systems, volume 31, pages 3303 3313, 2018. URL https:/ htttss://proceedingg httpp/ppppeennnnses.epp neurips caess.cc/ppppppe http://pprreec1/////eee //aash/ Abstract. .ht.ra......m...........
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019. URL https: hwtp:///g/aa///aaao//e//. abs/ aww.b//9..111.... 909. 10618. 8.
// //arxiv. warxiv.org/bbll.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292-6299. IEEE, 2018. doi: 10. 109/ (CRA.2018.8463162 URL https: /ttp:://eeeeeppppeee
ieee wwt/ http:/docuunntt//4433
Soroush Nasiriany, Huihan Liu, and Yuke Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In IEEE International Conference on Robotics and Au- tomation (ICRA), pages 7477-7484, 2022. URL https:/ http///axivv.///a/ org/ aaaabb///22 2210..11... 03655.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Advances in Neural Information Processing Systems 10. MIT Press, 1998.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), pages 944-957, 2020. URL https: // //axxiv.org/aab/.......a...../.e abs /tt..
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In European Conference on Machine Learning and Principles and Prac- tice of Knowledge Discovery in Databases (ECMI//KDD)) 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS). Robotics: Science and Systems Foundation, 2018.
Sasha Salter, Kristian Hartikainen, Walter Goodwin, and Ingmar Posner. Priors, hierarchy, and information asymmetry for skill transfer in reinforcement learning. In Proceedings of the 5th Conference on Robot Learning (CoRL), page TBD. PMLR, 2022a.
Sasha Salter, Markus Wulfmeier, Dhruva Tirumala, et al. Mo2: Model-based offline options. In Conference on Lifelong Learning Agents, pages 902-919, 2022b.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Con- ference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 2015. PMLR. URL https: S //ppoceeeding..a.ce/ee mlr. prr.ss/vs...
schaul1 15. html.
13

---

Published as a conference paper at ICLR 2025
Konstantinos Shiarlis. Markus Wulfmeeer. Shaun Salter, Shimon Whiteson, and Ingmar Posner. Taco: Learning task decomposition via temporal alignment for control. In Proceedings of the 35th International Conference on Machine Learning, pages 4654-4663. PMLR, 2018.
Avi Singh, Huihan Liu, Gaoyue Zhou, Tianhe Yu, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In 9th International Confer- ence on Learning Representations (ICLR), 2021. URL 10024.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-)):181- 211. 1999.
Dhruva Tirumala, Alexandre Galashov, al. Behavior priors for efficient reinforcement learning. Journal of Machine Learning Research, 23(221):1-68, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Alexander Sasha Vezhnevets, Simon Osindero. Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference machine learning, pages 3540-3549. PMLR, 2017.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, al. Regular- ized hierarchical policies for compositional transfer in robotics. arXiv preprint arXiv: 11228, 2019.
Markus Wulfmeier, Dushyant Rao, Roland Hafner, al. Data-efficient hindsight off-policy option learning. In International Conference Machine Learning (ICML.,, pages 11340-11350, 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. ing adjacency-constrained subgoals in hierarchical reinforcement learning. In Ad- vances in Neural Information Processing Systems, volume 33, pages 9814-9826, 2020.
Generat-
URL https:://rrocedd wew.eeeiinnnnnnnnnnneeeeeeeeeeee httpp/ccpppp////////////// BBCD88770033ebb0c07775577778 www.a.apppparaeeeeeee..eee..e...
14

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
Related Work
3
Background
4
Methodology
4.1 Primitive Enabled Adaptive Relabeling
4.2 Joint optimization
4.3
Sub-optimality analysis
5
Experiments
5.1
Evaluation and Results
5.2
Ablative analysis
Discussion
10
Appendix
16
A.1
Sub-optimality analysis
16
A.1.1
Sub-optimality proof for higher level policy
16
A.1.2
Sub-optimality proof for lower level policy
16
A.2
Generating expert demonstrations
17
A.2.1
Maze navigation task
17
A.2.2
Pick and place task
17
A.2.3
Bin task
17
A.2.4
Hollow task
18
A.2.5
Rope Manipulation Environment
18
A.3
Environment and implementation details
18
A.3.1
Maze navigation task
19
A.3.2
Pick and place, Bin and Hollow Environments
19
A.3.3
Rope Manipulation Environment
20
A.3.4
Impact Statement
20
A.4
Ablation experiments
20
A.5
Qualitative visualizations
21
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
SUB-OPTIMALITY ANALYSIS
Here, we present the proofs for Theorem for higher and lower level policies, which provide sub- optimality bounds on the optimization objectives.
A.1.1
SUB-OPTIMALITY PROOF FOR HIGHER LEVEL POLICY
The sub-optimality of upper policy Tgy. over length sub-trajectories sampled from di can be bounded as:
[J(x*) - J H Aн ΦD du * Sww.e
(13)
where AF (I->)(1-7) Rmmaa
11.00
Proof. We extend the suboptimality bound from (Ajay et al., 2020) between goal conditioned poli- cies x² and !! as follows:
W(=')- S (1 7) - 79) Rmar Ewwdy g o (F(s,))) (14)
By applying triangle inequality:
Drv (T (rls,g)) s DTv(T (1)
Drv(=((T|s.))l (r|8. (15)
Taking expectation wrt K. G and # I.
E Po-v (T|s, E Pww.aaiDlg..o oo (r|s, (rls) g)]]
E
( g| (r|s, g))
(16)
Since n² is Φn common in I can write 16
Eg GDr (T|s, 9) (T|s, < ΦD + E s - .g
(17)
g Dr (T g) | (r|s,)))
Substituting the result from align 17 in align 14, we get
[((*) - [ < An ΦD An Beva oo o
(r18,g))]] (18)
where AØ (1 (1-7)(1 7 Rema d5" 11.0
A.1.2 SUB-OPTIMALITY PROOF FOR LOWER LEVEL POLICY
Let the optimal lower level policy be n=" The suboptimality of lower primitive The can be bounded as follows:
[a(=")) - J(TTT(]))) S Ar. * &n + Ar. * E Eg B. ay (Drv (prv(T((7/s) (D s (T|s, 8g)) })]]]]
(19)
where dτ. (I-173 2 Rmax
Proof. We extend the suboptimality bound from (Ajay et al., 2020) between goal conditioned poli- cies ** and # follows:
2
[[(**) Jm )15
Rmaz Band"" "
so (Dv((m""(Tles 8y) || (r|s, 8g)]] (20)
(1 7)2
16

---

Published as a conference paper at ICLR 2025
By applying triangle inequality:
Drv (r|s, 8g) sg)ll=b (r)), Drv(a (rl(s" (rlss,,)]llb(((sssss))
(21)
Drv(s (1) 8g))
Taking expectation wrt Ki 89 Box and T 1I
E E
D (T (r sn) I (T|a, 8,)] 2
e r 6 DDv (rls s, (r)), s,))]+
(22)
E s e (rvv (T 8y) m (r18, 8g))
Since n²* is ΦD common in H. write 22 as:
E E Drv (a ( (T)s. (r|s, (1)s,,))]]
(23)
ΦD 13 g 2 w (Drv (T s,)) 11 (T1s,
s,
Substituting the result from align 23 in align 20, we get
[J(***) - J (TT0Z ) & x = oD + de * Eo t - [Dr [DTV(T s,) m (T|s, s,))]]
(24)
where d T3- Rmaa
A.2 GENERATING EXPERT DEMONSTRATIONS
For maze navigation, we use path planning RRT VLale, 1998) algorithm to generate expert demonstration trajectories. For pick and place, we hard coded an optimal trajectory generation pol- icy for generating demonstrations, although they can also be generated using Mujoco VR (Todorov et al., 2012). For kitchen task, the expert demonstrations are collected using Puppet Mujoco VR sys- tem (Fu et al., 2020). In rope manipulation task, expert demonstrations finding the closest corresponding rope elements from the current rope configuration and final goal rope configuration, and performing consecutive pokes of fixed small length the rope element in the direction of the goal configuration element. The detailed procedure are as follows:
generated by repeatedly
A.2.1 MAZE NAVIGATION TASK
We use the path planning RRT (Lale, 1998) algorithm to generate optimal paths P (Pr. Pt+ Pt+2, ..P.) from the current state the goal state. RRT has privileged information about the obstacle position which is provided to the methods through state. Using these expert paths, we generate state-action expert demonstration dataset for the lower level policy.
A.2.2 PICK AND PLACE TASK
In order to generate expert demonstrations, we can either use a human expert to perform the pick and place task in virtual reality based Mujoco simulation, hard code control policy. We hard-coded the expert demonstrations in our setup. In this task, the robot firstly picks up the block using robotic gripper, and then takes the target goal position. Using these expert trajectories, we generate expert demonstration dataset for the lower level policy.
A.2.3 BIN TASK
In order to generate expert demonstrations, we can either use human expert to perform the bin task in virtual reality based Mujoco simulation, or hard code a control policy. We hard-coded the expert demonstrations in setup. In this task, the robot firstly picks up the block using robotic gripper, and then places it in the target bin. Using these expert trajectories, we generate expert demonstration dataset for the lower level policy.
17

---

Published as a conference paper at ICLR 2025
A.2.4 HOLLOW TASK
In order to generate expert demonstrations, we can either use human expert perform the hollow task in virtual reality based Mujoco simulation, or hard code a control policy. We hard-coded the expert demonstrations in our setup. In this task, the robotic gripper has to pick up the square hollow block and place it such that a vertical structure on the table goes through the hollow block. Using these expert trajectories, we generate expert demonstration dataset for the lower level policy.
A.2.5 ROPE MANIPULATION ENVIRONMENT
We hand coded an expert policy to automatically generate expert demonstrations e (s6, sf....,,.f, .5-1); where of are demonstration states. The states of here are rope configuration vectors. The expert policy is explained below.
Let the starting and goal rope configurations be se and ge. We find the cylinder position pair (sCm, where m E [L, ni) such that sCm and gCm are farthest from each other among all other cylinder pairs. Then, we perform a poke (x, y, 0) drag sCm towards gcm- The (z, y) position of the poke is kept close to sCm, and poke direction @ is the direction from sCm towards gCm. After the poke execution, the next pair of farthest cylinder pair is again selected and another poke executed. This is repeatedly done for k pokes, until either the rope configuration of goal ge, or we reach maximum episode horizon T. Although, this policy is not the perfect policy for goal based rope manipulation, but it still is a good expert policy for collecting demonstrations D. Moreover, as our method requires states and not primitive actions (pokes), we can use these demonstrations D to collect good higher level subgoal dataset Dg using primitive parsing.
comes within ó distance
A.3 ENVIRONMENT AND IMPLEMENTATION DETAILS
Here, provide extensive environment and implementation details for various environments. We perform the experiments on two system each with Intel Core i7 processors, equipped with 48GB RAM and Nvidia GeForce GTX 1080 GPUs. We use 28 expert demos for franks kitchen task and 100 demos in all other tasks, and provide the procedures for collecting expert demos for all tasks in Appendix A.2. We empirically increased the number of demonstrations until there was no significant improvement in the performance. In our experiments, we use Soft Actor Critic (Haarnoja al., 2018b). The actor, critic and discriminator networks are formulated as 3 layer fully connected networks with 512 neurons in each layer.
When calculating p, we normalize Qx' values of a trajectory before comparing with Qthresh: ($Q sf, a.) - min value)/ /ae//maaaaaaue * 100 for = to T - 1. The experiments are run for 4. 73e5, 1.1e5, 1.32 1.8.5, 1.58e6, and 5.32e5 timesteps in maze, pick and place, bin, hollow, rope and kitchen respectively. The regularization weight hyper-parameter V is set at 0.001, p is set to be 1.1e4, 2500, Qthresh is set 10, 0, 0, O,
min__aluee
0.005, 0.005. 0.005, 0.005, and 0.005, the population
2500, 2500, 3.9e5, and 1.4c4, and distance threshold 0, and 0 for maze, pick and place, bin, hollow, rope and kitchen tasks respectively.
In maze navigation, a 7-DOF robotic arm navigates across randomly generated four room mazes, where the closed gripper (fixed at table height) has to navigate across the maze to the goal position. In pick and place task, the 7-DOF robotic arm gripper has to navigate the square block, pick up and bring it to the goal position. In bin task, the 7-DOF robotic arm gripper has to pick the square block and place the block inside the bin. In hollow task, the 7-DOF robotic arm gripper has to pick square hollow block and place it such that fixed vertical structure on the table goes through the hollow block. rope manipulation task, a deformable soft rope is kept on the table and the 7-DoF robotic arm performs pokes to nudge the rope towards the desired goal rope configuration. The rope manipulation task involves learning challenging dynamics and goes beyond prior work on navigation- like tasks where the goal space is limited.
In the kitchen task, the 9-DoF franka robot has to perform a complex multi-stage task in order to achieve the final goal. Although many such permutations can be chosen, we formulate the following task: the robot has to first open the microwave door, then switch on the specific gas knob where the kettle is placed. In maze navigation, upper level predicts a subgoal, and the lower level primitive travels in a straight line towards the predicted goal. In pick and place, bin and hollow tasks, we design three primitives, gripper-reach: where the gripper goes to given position (Iz; yis =)), gripper-
18

---

Published as a conference paper at ICLR 2025
open: opens the gripper, and gripper-close: closes the gripper. In kitchen environment, the action primitives implemented in RAPS (Dalal et al., 2021). While using RAPS baseline, we hand designed action primitives, which provide in detail in Section A.3.
A.3.1 MAZE NAVIGATION TASK
In this environment, a 7-DOF robotic arm gripper navigates across random four room mazes. The gripper arm is kept closed and the positions of walls and gates are randomly generated. The table is discretized into rectangular W * grid, and the vertical and horizontal wall positions Wp and Hp are randomly picked from (1, W 2) and (1, H - 2) respectively. In the four room environment thus constructed, the four gate positions are randomly picked from (1, Wp - 1). (Wp p + 1, W 2), H 1) and (Hp + 1, H 2). The height of gripper is kept fixed at table height, and it has to navigate across the maze to the goal position(shown as red sphere).
The following implementation details refer to both the higher and lower level polices, unless oth- erwise explicitly stated. The state and action spaces in the environment are continuous. The state is represented as the vector Mp). where p is current gripper position and M is the sparse maze array. The higher level policy input is thus a concatenated vector [p. g]. where g is the target goal position, whereas the lower level policy input is concatenated vector [p, sg), where sg is the sub-goal provided by the higher level policy. The current position of the gripper is the current achieved goal. The sparse maze array M is a discrete 2D one-hot vector array, where represents presence of wall block, and 0 absence.
In our experiments, the size of p and M are kept to be 3 and 110 respectively. The upper level predicts subgoal sg, hence the higher level policy action space dimension is the same as the dimen- sion of goal space of lower primitive. The lower primitive action a which is directly executed on the environment, is 4 dimensional vector with every dimension 0; [0. 1]. The first 3 dimensions provide offsets to be scaled and added to gripper position for moving it to the intended position. The last dimension provides gripper control(O implics a fully closed gripper, 0,5 implies half closed gripper and 1 implics a fully open gripper). We select 100 randomly generated mazes each for train- ing, testing and validation. For selecting train, test and validation mazes, we first randomly generate 300 distinct mazes, and then randomly divide them into 100 train, test and validation mazes each. We use off-policy Soft Actor Critic (Haarnoja et al., 201 8b) algorithm for optimizing RL objective in our experiments.
A.3.2 PICK AND PLACE, BIN AND HOLLOW ENVIRONMENTS
In the pick and place environment, 7-DOF robotic arm gripper has to pick a square block and bring/place it to a goal position. We set the goal position slightly higher than table height. In this complex task, the gripper has to navigate to the block, close the gripper to hold the block, and then bring the block to the desired goal position. In the bin environment, the 7-DOF robotic arm gripper has to pick square block and place it inside fixed bin. In the hollow environment, the 7-DOF robotic arm gripper has to pick a hollow plate from the table and place it on the table such that its hollow center goes through a fixed vertical pole placed the table.
In all the three environments, the state is represented as the vector [p. 0, q. e). where p is current gripper position, is the position of the block object placed on the table, q the relative position of the block with respect to the gripper, and e consists of linear and angular velocities of the gripper and the block object. The higher level policy input is thus a concatenated vector [p, o, q, e, g). where is the target goal position. The lower level policy input is concatenated vector [p. 0, syl. where sy is the sub-goal provided by the higher level policy. The current position of the block object the current achieved goal.
In our experiments, the sizes of p, O, q, e are kept to be 3, 3. 3 and 11 respectively. The upper level predicts subgoal 8g. hence the higher level policy action space and goal space have the same dimension. The lower primitive action is a 4 dimensional vector with every dimension a, E [o, 1]. The first 3 dimensions provide gripper position offsets, and the last dimension provides gripper control (0 means closed gripper and means open gripper). While training, the position of block object and goal are randomly generated (block is always initialized the table, and goal is always above the table at a fixed height). We select 100 random each for training, testing and validation. For selecting train, test and validation mazes, first randomly generate 300 distinct environments
19

---

Published as a conference paper at ICLR 2025
with different block and target goal positions, and then randomly divide them into 100 train, test and validation mazes each. We use off-policy Soft Actor Critic (Haarnoja et al., 2018b) algorithm for the RL objective in our experiments.
A.3.3 ROPE MANIPULATION ENVIRONMENT
In the robotic rope manipulation task, a deformable rope is kept on the table and the robotic arm performs pokes nudge the rope towards the desired goal rope configuration. The task horizon is fixed at 25 pokes. The deformable rope is formed from 15 constituent cylinders joined together. The following implementation details refer to both the higher and lower level polices, unless otherwise explicitly stated. The state and action spaces in the environment for the rope manipulation environment is vector formed by concatenation of the intermediate joint positions. The upper level predicts subgoal &g for the lower primitive. The action space of the poke (1r, y, n), where (xr y) is the initial position of the poke, and is the angle describing the direction of the poke. We fix the poke length be 0.08.
continuous. The state space
While training our hierarchical approach, we select 100 randomly generated initial and final rope configurations each for training, testing and validation. For selecting train, test and validation con- figurations, we first randomly generate 300 distinct configurations, and then randomly divide them into 100 train, test and validation mazes each. We use off-policy Soft Actor Critic (Haarnoja al., 2018b) algorithm for optimizing RL objective in our experiments.
A.3.4 IMPACT STATEMENT
Our proposed approach and algorithm are not anticipated to result in immediate technological advancemenss. Instead, our main contributions conceptual, targeting fundamental aspects of Hierarchical Reinforcement Learning (HRL). By introducing primitive-enabled regularization, we present novel framework that believe holds significant potential to advance HRL research and its related fields. This conceptual groundwork lays the foundation for future investigations and could drive progress in HRL and associated domains.
A.4 ABLATION EXPERIMENTS
Here, we present the ablation experiments in all six task environments. The ablation analysis in- cludes comparison with HAC-demos and HBC (Hicrarchical behavior cloning) (Figure 14), choos- ing RPL Qthreshh hyperparameter (Figure 8), p hyperparameter (Figure 9), RPL window size k hy- perparameter (Figure 10), learning weight hyperparameter é (Figure 1 11), comparisons with varying number of expert demonstrations used during relabeling and training (Figure 12), comparison with HER-BC ablation (Figure 13), effect of sub-optimal demonstrations (Figure 15).
0.8
trr 0.6
Uhrishw
0 1
6. d_theess-10
1hhhehh111 10
d_theess-30
Ihrwhh33
3,02
100c
2004
20k
10c
0
20k
DOk
120k
Tinestop
Timestess
Timestase
(a) Maze
(b) Pick and place
(c) Bin
0.8
d Ehreesod
02
0.6
d
d 2htreshwo
0.4
hu 30 0.1
d_shrestoo
Uires hed0
Brrch-77
201
50c 1ock Timesteps
150k
105k
B00k
Timesteas
Timestops
(d) Hollow
(c) Rope
(f) Franka kitchen
Figure& 8: The success rate plots show the performance of PEAR for various values of Qthresh parameter versus number of training timesteps.
20

---

Published as a conference paper at ICLR 2025
0.6
p-100
404
p=150
100k
20c
abc
123k
Timestass
(a) Maze
Pick and place
(c) Bin
p-100
p-100
0.4
p-155
p=100 p-150
p-110
p-to
poso
0.0
100c
200c
100c
200k
Timeslape
mmantapa
Timestass
(d) Hollow
(c) Rope
(f) Franka kitchen
Figure 9: The success rate plots show the performance of PEAR for various values of population number p parameter versus number of training timesteps.
window-15
sindow sendow
windou-30
0.0
window-31 window-20
50
window 30
70
window
window-
200k
400k
40k BBk
40c
UIK 2208
(a) Maze navigation
(b) Pick and place
(c) Bin
0.3
indoo-11
window-10 10
indoo 20
windowe3 window-3 5
window-17 17
wndeo 10
82
window-20
windowed
window->
window-S
100k 2006
200k
100k 2008 100k Timesteox
400k
Timesteps
(d) Hollow
(e) Rope
(I) Franka kitchen
Figure 10: The success rate plots show the performance of RPL for values of i window size param- eter versus number of training epochs.
A.5 QUALITATIVE VISUALIZATIONS
In this subsection, provide visualizations for various environments.
21

---

Published as a conference paper at ICLR 2025
0.5
Success hate
503
0.4
2006
40c
40k
2206
Timestops
(a) Maze navigation
(b) Pick and place
(c) Bin
succes Rare
0.5
pdliod o01
piloo 201
9.4
poleod 005
poleod
pol-o.000
poleo 01
2.4
pol-0 005
PS 01
pdleo
pd-- 015
3.22
PS-00.00 015
400
soc
1204
100k
200c Timazagoo
2002
(d) Hollow
(c) Rope
(f) Franka kitchen
Figure 11: The success rate plots show performance of PEAR for values of learning weight param- eter to versus number of training timesteps.
0.8
0.6
sam_demas-100 1.0
10m_demas=1100
num damos-30
sum demop=30
num. demos=30
num, demox-70
sum, demov-73 73
022
num den
50c 120c
150K 200k
23k
40k
60k
40c 50c
Timeatops
Timestopa
(a) Maze navigation
(b) Pick and place
(c) Bin
0.3
0.6
Kate
0.2
num_demosele
num Jomes--00
num damos=14
3emas-30
num demox-10
de
7em_dem
num_demas=?
9.0
0.0
Timecips
Timestape
Timesteps
(d) Hollow
(e) Rope
(f) Franka kitchen
Figure 12: The success rate plots show success rate performance plots of varying number of expert demonstrations versus number of training epochs.
06
Vtlle
BA
0.
0.4
2006
2208 Timestago
400x
20c
Timoskops
(a) Maze
(b) Pick and place
(c) Bin
0.
0.A
22
40k
23k
Timsstess
Timcusse
(d) Hollow
(e) Rope
(1) Kitchen
PEAR
HER-BC
Figure 13: Comparison with HER-BC baseline: The figure depicts success rates plots of PEAR- IRL. compared with HER-BC baseline, which is single-level implementation of Hindsight Expe- rience Replay (HER) with expert demonstrations. As can be seen, PEAR consistently outperforms this baseline, which clearly demonstrates the advantages of using our hierarchical formulation in such complex tasks.
22

---

Published as a conference paper at ICLR 2025
0.8
Sasteee Sate
Sucoage 9110
0.4
200k Timestepps
40k
Timenteps
Timestnoe
(a) Maze navigation
(b) Pick and place
(c) Bin
0.8
Sucees tate
Success Iat
0.2
04
Timesteps
Timestops
Timesstss
(d) Hollow
Rope
(1) Franka kitchen
PEARIRL
HAD-lemus
HBC
Figure 14: Success rate comparison: PEAR-IRL vs HAC-demos vs HBC This figure compares the success rate performances of PEAR-TRI with HAC-demos, and HBC six sparse maze navi- gation and manipulation tasks. HAC-demos uses hierarchical actor critic (Levy et al., 2018) as the RL objective and is jointly optimized using additional behavior cloning objective, where the lower level uses primitive expert demonstrations and the upper level uses subgoal demonstrations extracted using fixed window based approach (as in RPL (Gupta al., 2019)). HBC (Hierarchical behavior cloning) uses the same demonstrations as #AC-demos at both levels, but it trained using only behavior cloning objective (thus, does not employ RL). As in figure, although HAC-demos shows good performance in the easier maze navigation environment, both HAC- and HBC fail to solve the tasks in harder environments, and PEAR- IRL significantly outperforms both the baselines. The solid line and shaded region represent the mean and range of success rates across seeds.
0.4
Success Mate:
0.6
Success
0.4
0.2
demoo
but pemes
bed dem.ee 70
bad ceevaion
0.0
0.0
40k
80k
10k
20k
Timestaps
Timesteps
(a) Maze navigation
(b) Pick and place
bod, Jemos-0
bad
bad demos-10
bud damos=14
bod 3omos-50
bad domos-?
Timestopps
Timastop:
(c) Rope
(d) Franka kitchen
Figure 15: Ablation with sub-optimal demonstrations: The success rate plots show the perfor- mance of PEAR-IRL with varying number of sub-optimal demonstrations in the expert demon- stration dataset. As can seen, the performance suffers with increasing number of sub-optimal demonstrations.
23

---

Published as a conference paper at ICLR 2025
Figure 16: Succesful visualization: The visualization successfil altempt al performing maze navigation task
Figure 17: Successful visualization: The visualization is successful attempt at performing pick navigation task
Figure 18: Successfll visualization: The visualization successful attempt performing bin task
Figure 19; Successful visualization: The visunlization is a succesfful attempi at performing hollow task
Figure 20: Successful visualization: The visualization is successful atlempt at performing navigation task
Figure 21: Successful visualization: The visualization is uuccesffll attempt performing kitchen navigation task
24


Reviews:
[{"review_id": "FrjVcKydmH", "reviewer": "Reviewer_u7xN", "summary": "The paper presents a novel approach called Primitive Enabled Adaptive Relabeling (PEAR) aimed at enhancing Hierarchical Reinforcement Learning (HRL) for complex long-horizon tasks. The authors propose a two-phase methodology where the first phase involves adaptive relabeling of expert demonstrations to generate subgoals, followed by joint optimization of HRL agents through Reinforcement Learning (RL) and Imitation Learning (IL). The results indicate that PEAR outperforms various baselines in both synthetic and real-world robotic tasks, achieving up to 80% success rates in challenging environments.", "questions": "- How do the authors justify the reliance on expert demonstrations in the first phase of PEAR compared to HRL methods that function without such requirements?\n- Can the authors provide additional comparisons with recent HRL approaches to strengthen the positioning of PEAR within the current landscape?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Innovative Approach:** The use of adaptive relabeling to generate subgoals tailored to the capabilities of the lower primitive is a significant contribution that addresses the non-stationarity issue in HRL.\n2. **Theoretical Justification:** The authors provide theoretical analysis that bounds the sub-optimality of their approach, lending credibility to their claims.\n3. **Comprehensive Experiments:** Extensive experiments across multiple challenging tasks demonstrate the practical efficacy of PEAR, showing improved performance and sample efficiency over existing methods.\n4. **Real-World Application:** The validation of PEAR in real-world tasks enhances the relevance and applicability of the research.", "weaknesses": "1. **Expert Demonstrations Requirement:** The first phase of PEAR relies on expert demonstrations to generate subgoals, which raises concerns about the fairness of comparison with other HRL methods that do not require such demonstrations. This could affect the generalizability of the findings.\n2. **Lack of Recent Comparisons:** The paper does not include comparisons with several hierarchical reinforcement learning methods published in the last three years. This omission limits the contextual relevance of the results and could misrepresent the state of the art, such as [1], [2].\n\n[1] Kim J, Seo Y, Shin J. Landmark-guided subgoal generation in hierarchical reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 28336-28349.\n\n[2] Wang, Vivienne Huiling, et al. \"State-conditioned adversarial subgoal generation.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 8. 2023.\n\n3. **Citation Issues:** Many citations are sourced from CoRR instead of their formal conference versions. This oversight should be corrected to ensure academic integrity and accurate referencing. For example, the last two references should be cited as follows (Note that there are many other citation errors beyond these two):\n\nWulfmeier, Markus, et al. \"Data-efficient hindsight off-policy option learning.\" International Conference on Machine Learning. PMLR, 2021.\n\nZhang, Tianren, et al. \"Generating adjacency-constrained subgoals in hierarchical reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 21579-21590.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GogHcO1x2F", "reviewer": "Reviewer_Cpmf", "summary": "The authors introduce an algorithm, Primitive Enabled Adaptive Relabeling (PEAR), to address the issue of non-stationary transition and reward functions when implementing HRL.  Like Relay Policy Learning (RPL) (Gupta 2019), PEAR encourages the high level policy to output feasible subgoals using imitation learning.  The main difference from RPL is that instead of imitating the actions from a dataset that occur at fixed intervals, PEAR uses a heuristic to select the latest subgoal that the low level policy can achieve.  The authors show in a variety of experiments that PEAR can outperform several baselines.\n\nGupta et al. Relay Policy Learning. 2019", "questions": "1.  Why is the Q_{Thresh} set to 0?  If the low-level reward is -1 for not achieving the goal and 0 for achieving the goal, the Q value should be negative for any correct action that puts the agent on a path to achieve the goal.  \n2.  Is the high level policy only trained with a dataset containing expert trajectories?  Or is it also trained on its own interaction data?  If it is trained on its own interaction data, is there any relabeling done on that?\n3.  I did not understand the “margin” component of the objective.  Can you provide another explanation of this component?\n4.  Why does the paper characterize the number of demonstrations as a “handful” of demonstrations, when it uses 100 for most tasks?\n5.  Have you experimented on any domains with image observations?\n\nI am willing to raise my score if the authors can (i) provide some principled reasons why PEAR should outperform HAC+demonstrations and hierarchical behavior cloning and (ii) provide some answers to the above questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The writing was clear and easy to understand.\n- The authors included several ablation experiments that provided some important insights.\n- The paper included some real world robotic experiments.", "weaknesses": "My main concern with this approach is that the contribution seems to be marginal as I do not see a compelling reason why PEAR would consistently perform better than (a) HAC (Levy 2019) with replay buffer augmented with demonstration data and (b) hierarchical behavior cloning, in which pure imitation learning is applied to both a high and low level policies.  \n\nHAC already addresses the problem of nonstationarity through relabeling and subgoal testing.  The problem with HAC is that it does not have a built-in mechanism to explore but this can be remedied with the demonstration data that is provided to PEAR.  An advantage of HAC + demonstration data over PEAR, which would use a pure RL objective, is that if the demonstration data is suboptimal, it would not have an imitation learning regularization term forcing the agent to output suboptimal subgoals.  HAC has also demonstrated it can learn more than two levels of hierarchy.  The results of HER+BC, which I understood to be HER with a replay buffer augmented with demonstration data, was often ultimately able to match the performance of PEAR (see Figure 14), making it more likely that HAC, which is close to a hierarchical version of HER, should be able to match PEAR.  \n\nIn addition, it seems that a pure hierarchical imitation learning approach, in which both levels are trained with supervised learning should also work, at least potentially with more data.  The baseline BC may not have worked well because the tasks were too long horizon, but the addition of a high level policy trained with imitation learning should help.\n\nLevy et al. Hierarchical Actor-Crtiic. 2017", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALWWQChqsX", "reviewer": "Reviewer_di4o", "summary": "PEAR combines its key feature, adaptive goal relabeling, with IL regularization (either MSE or IRL), and other tricks (e.g. margin classification objective) to beat several prior HRL methods on a standard array of tasks using expert demonstrations.", "questions": "Have you tested on suboptimal demonstrations? Are there any interesting findings/results which may point to future avenues for research or failings of the method which should be investigated/improved upon?\n\nWere you able to find notable reasons for why the MSE-regularized learning objective would occasionally outperform the IRL-regularized version? Is there any relationship to task difficulty, data diversity, etc?\n\nIn “Algorithm 2: PEAR,” line 8, shouldn’t the lower-level policy’s IL regularization be done with $D_g$? Since we are providing state $s^f$ from the goal dataset $D_g$ and subgoal supervision $s^e_g$ to the goal-conditioned low-level policy, then the policy predicts action $a$, and we regularize this to be close to the dataset action $a^f$ (either with MSE or the IRL objective)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors carefully outline their construction of the PEAR algorithm, justifying the usage of each component (goal relabeling, the general algorithm, the joint optimization framework, etc) thoroughly and clearly. The sub optimality analysis provides further credence to their method. The method is novel, and notably outperforms previous HRL works (while, in some cases, removing the need for e.g. hand-made action primitives).\n\nI agree with the author view that the significance of the work should be gauged less by its immediate improvement over other LfD methods, and more by its conceptual groundwork. In this regard, this paper is well-written, the findings are well-presented, and the extensive ablations provide further insight into key aspects of the method.", "weaknesses": "As authors note, the method is currently reliant on expert demonstrations. However, many benchmarks exist which include other kinds of demonstrations, including human teleop. While the method may not perform well on these demonstrations just yet (as it is listed as an aim of future work), providing results on suboptimal demonstrations would help demonstrate concretely the strong and weak points of authors' method, and potentially provide insights on why it fails in these settings.\n\nFurthermore, it seems inaccurate to state that PEAR uses only “a handful” of demonstrations, when Fig. 13 shows that generally 50-70+ demonstrations are needed to solve the provided tasks (with the exception of Franka Kitchen, which provides fewer demos).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lv8bXryDGL", "reviewer": "Reviewer_GCp5", "summary": "This paper proposes a method to improve HRL, utilizing a few expert demonstrations. The key insight is that subgoals selected from the dataset can effectively guide exploration for the policy. An adaptive relabeling method is proposed to select the proper subsequent subgoal based on the Q value of the low-level policy. The relabeled data provides an imitation-based regularization for the high-level policy, encouraging it to output reachable, high-quality subgoals for the low-level policy. Experiments in diverse simulation robotic tasks demonstrate the effectiveness of the method.", "questions": "1. Many existing works [1,2,3] also use IL loss from data to regularize high-level policy learning. What differentiates the proposed regularization method from these approaches?\n2. Why can we set the threshold of Q to 0 (Section 4.1) in all experiments? I believe this hyperparameter should vary, depending on the reward function specific to each task.\n\n[1] Pertsch, et al. \"Accelerating reinforcement learning with learned skill priors.\" Conference on robot learning. PMLR, 2021.\n[2] Shi, et al. \"Skill-based model-based reinforcement learning.\" arXiv preprint arXiv:2207.07560 (2022).\n[3] Yuan, et al. \"Pre-training goal-based models for sample-efficient reinforcement learning.\" The Twelfth International Conference on Learning Representations. 2024.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Using a few expert demonstrations to improve goal-based HRL is promising. The proposed adaptive relabeling method for IL regularization is straightforward, well-motivated, and yields good results.\n2. The paper includes some theoretical analysis.\n3. The experiments cover a variety of robotic tasks, including real-world test.", "weaknesses": "1. Inconsistent definition: Section 3 states that the expert data contains only states. But in Section 4.2, the low-level regularization term uses actions from the expert data.\n2. Addressing the non-stationarity issue in HRL is a main claim of the paper. However, the proposed method does not resolve this issue. The high-level policy still faces non-stationarity, as the transitions in its replay buffer, which are determined by the low-level policy, continue to change throughout the training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)