CRISP: Curriculum inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning
Authors: Utsav Singh
Vinay P Namboodiri
Venue: Submitted to ICLR 2023 (2023)
Keywords: Hierarchical Reinforcement Learning
Inverse Reinforcement Learning
Imitation Learning
Curriculum Learning

Abstract: Hierarchical reinforcement learning is a promising approach that uses temporal abstraction to solve complex long horizon problems. However, simultaneously learning a hierarchy of policies is unstable as it is challenging to train higher-level policy when the lower-level primitive is non-stationary. In this paper, we propose to generate a curriculum of achievable subgoals for evolving lower-level primitives using reinforcement learning and imitation learning. The lower level primitive periodically performs data relabeling on a handful of expert demonstrations using our primitive informed parsing method. We derive expressions to bound the sub-optimality of our method and develop a practical algorithm for hierarchical reinforcement learning. Since our approach uses a handful of expert demonstrations, it is suitable for most real world robotic control tasks. Experimental results on complex maze navigation and robotic manipulation environments show that inducing hierarchical curriculum learning significantly improves sample efficiency, and results in better learning of goal conditioned policies in complex temporally extended tasks. 

Under review as a conference paper at ICLR 2023
CRISP: CURRICULUM INDUCING PRIMITIVE IN- FORMED SUBGOAL PREDICTION FOR HIERARCHICAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Hierarchical reinforcement learning is a promising approach that uses temporal abstraction to solve complex long horizon problems. However, simultaneously learning hierarchy of policies is unstable as it is challenging to train higher- level policy when the lower-level primitive is non-stationary. In this paper, we propose to generate a curriculum of achievable subgoals for evolving lower-level primitives using reinforcement learning and imitation learning. The lower level primitive periodically performs data relabeling tions using our primitive informed parsing. We provide expressions to bound the sub-optimality of our method and develop practical algorithm for hierarchical reinforcement learning. Since our approach uses a handful of expert demonstra- tions, is suitable for most robotic control tasks. Experimental results on complex maze navigation and robotic manipulation environments show that inducing hier- archical curriculum learning significantly improves sample efficiency, and results in better learning of goal conditioned policies in temporally extended tasks.
handful of expert demonstra-
INTRODUCTION
Reinforcement learning (RL) algorithms have made significant progress in solving continuous con- trol tasks like performing robotic arm manipulation (Levine et al., 2015: Vecerík et al. 2017) and learning dexterous manipulation (Rajeswaran et al. (017). However, the success of RL algorithms on complex long horizon continuous tasks has been limited by issues like long term credit assign- ment and inefficient exploration (Nachum et al. 2019 Kulkarni et al. 2016), especially in sparse reward scenarios (Andrychowicz al.) [ (017). Hierarchical reinforcement learning (HRL) (Dayan & Hinton 1993] Sutton et al. 1999 Parr & Russell 1998) promises the benefits of temporal abstrac- tion and efficient exploration for solving tasks that require long term planning. In goul-conditioned hierarchical framework, the high-level policy predicts subgoals for lower primitive, which in turn performs primitive actions directly on the environment (Nachum et al. 2018 Vezhnevets et al.) 2017 Levy al. 2017). However, simultaneously learning multi-level policies has been found to be challenging in practice due to non-stationary higher level state transition and reward functions.
Prior works have leveraged expert demonstrations to bootstrap learning (Nair al.) 2017 Ra-
jeswaran et al. 2017]: Hester et al. 2017). Some approaches rely on leveraging expert demon- strations via fixed parsing, and consequently bootstrapping multi-level hierarchical RL policy using imitation learning (Gupta et al. 2019). Generating an efficient subgoal transition dataset is crucial in such tasks. this work, we propose an adaptive parsing technique for leveraging expert demonstra- tions and show that it outperforms fixed parsing based approaches on tasks that require long term planning. Ideally, good subgoal should properly balance the task split between the hierarchical levels according to current goal reaching ability of the lower primitive, thus avoiding degenerate solutions. As the lower primitive improves, the subgoals provided to lower primitive should become progressively more difficult, such that (i) the subgoals are always achievable by the current lower level primitive, (ii) task split is properly balanced between hierarchical levels, and (iii) reasonable progress is made towards achieving the final goal. In this work, introduce hierarchical curricu- lum learning to deal with non con-staiionarity issue. We build upon these ideas and propose a generally applicable HRI approach: Curriculum inducing primitive informed subgoal prediction (CRISP).

---

Under review as a conference paper at ICLR 2023
CRISP parses handful of expert demonstrations using novel subgoal relabeling method: prim- itive informed parsing (PIP). In PIP. current lower primitive is used to perform data relabeling on expert demonstrations dataset generate efficent subgoal supervision for the higher level policy. Since the lower primitive performs data relabeling, this approach does not require explicit labeling segmentation of demonstrations by expert. The periodically generated higher level subgoal dataset is used with an additional imitation learning (IL) objective to provide curriculum based reg- ularization for the higher policy. For imitation learning, we devise inverse reinforcement learning regularizer (Ghasemipour et al. 2020 Kostrikov et al.) 2018] Ho & Eirmon 2016), which constraints the state marginal of the learned policy to similar to that of the expert demonstrations. The details of CRISP, PIP, and IRL objective are mentioned in Section 3 We also derive sub-optimality bounds in Section 3.2 theoretically justify the benefits of curriculum learning hierarchical framework. Finally, we provide practical approach to perform hierarchical reinforcement learning.
Since our approach uses a handful of expert demonstrations, complex long horizon tasks. We perform experiments on random maze navigation and complex robotic pick and place environments, and empirically verify that the proposed approach clearly outperforms the baseline approaches on long horizon tasks.
generally applicable on most
2 BACKGROUND
We consider Universal Markov Decision Process (UMDP) (Schaul et al. 2015) setting, which is Markov Decision process (MDP) augmented with the goal space G. UMDPs are represented as a 6-tuple (S, A, P, R, y, G). where S is the state space, is the action space, P(s [s. a) P(s++1 = |sc = s, a, = a) is the transition function that describes the probability of reach- ing state when the agent takes action a in the current state 8. The reward function R gener- ates rewards
every timestep, y is the discount factor, and G is the goal space. In the UMDP setting, a fixed goal is selected for an episode, and Taass, g) denotes the goal-conditioned pol- icy. ds((s) = (1 7) Ei=0" 7 P(s, s|m) represents the discounted future state distribution, and (s)s) = (1- 2-)
Ei= 7tc P(tte = s|x) represents the c-step future state distribution for policy m. The overall objective is to learn policy x (als,, g) which maximizes the expected future discounted reward objective J = (1 7) Eand"" (a<r(als.s): [r(Se, Qz ge.
Let be the current state and be the final goal for the current episode. In our conaitondi hi- erarchical RL setup, the overall policy æ is divided into multi-level policies. The higher level policy "" (syl*s 18, g) predicts subgoals (Dayan & Hinton 1993) Sy for the lower level primitive n' aa(als, 8g)) which in turn executes primitive actions directly on the environment. The lower primitive T² tries to achieve subgoal sy within e timesteps by maximizing intrinsic rewards T'n provided by the higher level policy. The higher level policy x gets extrinsic reward Fex from the environment, and predicts the next subgoal 8g for the lower primitive. The process is continued until either the final goal is achieved, or the episode terminates. We consider sparse reward setting where the lower primitive is sparsely rewarded intrinsic reward rin if the agent reaches within 8 distance of the predicted sub- goal sy: Tin = 1 1[]|s; salle S 8²). and the higher level policy is sparsely rewarded extrinsic reward Tex if the achieved goal is within 8 distance of the final goal g: Tez 1[][sr glle S 8 We assume access to a handful of expert demonstrations D {e* Ji=1, where e¹ (s6, $1 si..... .,, -)-)) We only assume access to demonstration states sf (and not demonstration actions) which can be obtained most robotic control tasks.
3 METHODOLOGY
In this section, we explain our hierarchical curriculum learning based approach CRISP. An overview of the method is depicted in Figure 1. First, we formulate our primitive informed parsing method PIP, which periodically performs data relabeling on expert demonstrations to populate subgoal transition dataset. Then, we explain how we use this dataset to learn high level policy using reinforcement learning and additional inverse reinforcement (earning(IRI.) based regularization objective.
2

---

Under review as a conference paper at ICLR 2023
Tranning
After terossopps
Epoch
Epoch 600
Epoch 1300
Epoch
Epoch 1400
Epoch 3000
Overview of CRISP: While training. both upper policy(x") and lower primitive(r*)) are trained using RL and [I(imitation learning). After every timesteps, Dy is re-populated using PIP.
Figure 1:
Figure 2: This figure depicts CRISP subgoals progress vs training epochs elapsed. The higher level policy generates curriculum of subgoals for lower primitive(wallsselloo, fi- nal goal:red, subgoals:blue)
3.1 PRIMITIVE INFORMED PARSING: PIP
Primitive informed parsing approach uses the current lower primitive иT to parse expert state demon- strations dataset D(PIP only requires expert state demonstrations and does not require expert actions from demonstrations, as explained later in this section.) The underlying idea is: PIP should se- lect sequences of maximally temporally separated states from demonstration trajectory These maximally temporally separated state sequences constitute the higher level subgoal dataset Dy. We explain below how PIP adaptively parses expert demonstration trajectories from D.
We start with current lower primitive F and an expert state demonstration trajectory (s6, .f..... S7-.) The environment is reset to state s6. Starting at 1 to T - 1, we incre- mentally provide states sf as subgoals to lower primitive Tg. From s6. 4y tries to achieve sf within timesteps. If I.. fails achieve the subgoal 8f from the initial state, add af the list of sub- goals. The underlying idea is that since sf_1 was the last subgoal achieved by lower-level primitive, thus makes good candidate for maximally reachable subgoal. Once we have added $ the list of subgoals, we continue the process after setting s as the new initial state until we reach the end of demonstration trajectory This subgoal transition sequence thus collected is added to Dg. The method thus populates the subgoal transition dataset with maximally temporally separated achievable subgoals.
Algorithm PIP: Primitive Informed Parsing
1: Initialize Dg (}
2: for each trajectory = (sf .f......66 ,6 in do
3: initial state -6
4: final goal <
5: Initialize list of subgoals D !
6:
for i= to T do
7:
Reset initial state Pass sf as the current goal to
8:
9:
if of is not achieved by æ in time-steps then Add (initial state, s final goal) to D
10:
11:
initial state +- 55_1
12:
Dg - D2 D
The pseudocode for PIP is given in Algorithm I PIP maximizes the utility of the lower level primitive while ensuring that it always receives achievable subgoals. However, it assumes that can reset the environment to any state in while collecting subgoal dataset. We discuss different ways to relax this assumption in 6ection
3

---

Under review as a conference paper at ICLR 2023
3.2 SUBOPTIMALITY ANALYSIS
In this section, analyze the suboptimality of method, and examine how the performance benefits from curriculum learning and imitation learning objective. Let x and x- be the unknown higher level and lower level optimal policies respectively, n be our high level CRISP policy, and THe be our lower CRISP primitive policy, where Our and Oy. are trainable parameters higher and lower level policies respectively. (T1 T2) denotes total variation divergence between proba- bility distributions T1 and 22 is the current state, is the final episodic goal, $g is the subgoal provided by upper level policy and are length sub-trajectories. Let II and Пí be the upper level probability distributions which generate datasets Dr and Dr. respectively, k is some distribu- tion over states and actions, and G' is the goal space. Firstly, we extend the definition from (Ajay al. 2020) to goal-conditioned policies:
Definition 1. π is -ommoon in 11 if B - (Tr1slll oDD
We define the suboptimality of policy æ with respect to optimal policy x² as:
Subopt (0) = [J(xx)) J(x)
(1)
Theorem 1. Assuming the optimal policy πt is ΦD common in 11 the suboptimality of upper policy TH over length sub-trajectories 7 sampled from de can be bounded as:
[JTT)) - (TTT)) λH ØD + AÆ * E Bown, - a ol 9) (F(s,)))
(2)
where AØ = (1-7)(1 -77) Rmnaa
o00
Furthermore, the suboptimality of lower primitive Tge can be bounded as:
[J(x²*) J((b))) de * ΦD + d * E. E. a e (Drv(Tb(T)s, a (r|s, sy))]]
where dr.
2 Rimax de
lcc
-T-
The proofs for Equations and 3 are provided in Appendix A.I and A.2 respectively. Equation can be rearranged to yield the following form:
J(x') 2 ( - λи ØD λи * Ba B ald( (F)s 9) (T(s,)))
(4)
where, (T|s, |=H (T|s, Drv ( 9)ll (T|s,g))
This can be solved as a minorize maximize algorithm which intuitively means: the overall ob- jective can be optimized by (i) maximizing the objective J( via RL, and (ii) minimizing TV divergence between π²" and ). We use entropy regularized RL technique Soft Actor Critic (Haarnoja et al. 2018a to maximize (mHH))
3.3 HIERARCHICAL CURRICULUM LEARNING
In Equation 2 the suboptimality bound is dependent on ΦDD,, which represents how good is the subgoal dataset Dy populated by PIP. A lower value of ΦDD, impliees that the optimal policy =² closely represented by the dataset Dg- Since we use lower primitive parse expert demonstrations, as the lower primitive gets better, æD, gets closer to n*. Hence D, improves and the value of parameter ΦD decreases, which implies that suboptimality bound in Equation] 2 gets tighter.
To implement curriculum learning while generating subgoal transition dataset Dy. the dataset Dy is cleared after every timesteps, and re- re-populated using PIP. explained in Algorithm 2. Periodi- cally re- c--poulattig buffer D, after u timesteps generates a natural curriculum for lower primitive, shown in Figure 2.
4

---

Under review as a conference paper at ICLR 2023
3.4 IMITATION LEARNING REGULARIZATION FOR HIGHER-LEVEL POLICY
Different approximations to the distance function d in Equation yield different imitation learn- ing regularizers. If d is replaced by Kullback-Leiblrr divergence, the imitation learning regularizer becomes behavior cloning objective(BC) (Nair et al.) 2017). If we replace d with Jensen-Shannon divergence, the imitation learning objective takes the form of Inverse reinforcement learning(ILL) objective. Henceforth, CRISP-IRL will denote our method CRISP with IRL regularizer, and CRISP- BC will denote our method CRISP with BC regularizer. We consider both behavior cloning and IRI. objectives in our experiments. We devise IRL objective as a GAIL (Ho & Ermon 2016) like objec- tive implemented using LSGAN (Mao et al. where 5f is a state in an expert trajectory. 9 the corresponding final goal and sy is the subgoal.
2016). Let (s", g'," ~ Dg be subgoal transition
(-Is",g") ge) and ID be the higher level Let 9g be the subgoal predicted by the high level policy = (-Is",
discriminator with parameters e. We bootstrap the learning of higher level policy by optimizing:
1
max min BB =/
(rgt"" .g) DD (sg) 1]2 + B .0 )a a or a²) 1 (To (-1s", ,g))) 0)2
(5)
This objective forces the higher policy subgoal predictions to be close to subgoal predictions of the dataset Dg. For brevity, let J and Jb represent upper and lower IRL objectives, which depend on parameters (0H {H) and E1) respectively. The discriminator DD creates a natural curriculum for regularizing higher level policy by assigning the value 1 to the predicted subgoals that are closer to the subgoals from dataset g. and 0 otherwise. The discriminator improves with training, and regularizes the higher policy to predict achievable subgoals.
3.5 POLICY OPTIMIZATION
The higher level policy is trained to produce subgoals, which when fed into the lower level primitive, maximize the sum of future discounted rewards for our task using off-policy reinforcement learning. Here, ag is the current lower primitive, is the state at time t, T is the task horizon and is the sampled goal for the current episode. For brevity, we can refer to this objective function J and
1/ for upper and lower policies. We use the IRL objective to leverage the primitive-parsed dataset Dz. Therefore, the high level policy is trained by optimizing
max + (mmin J (0H, eH) 4y
(6)
ez
Whereas, the lower level primitive trained by optimizing.
max Joe + +mmin Jh (0,0,,00) (L))
(7)
OL
Algorithm 2 CRISP
Require: Expert state demonstrations D, hyperparameter u
1: Initialize higher level subgoal transition dataset D, = 0
2: for epoch 1...N do
3: if i%u = 0 then
4:
Clear Dg Populate D, by relabeling D using PIP
5:
6:
for j=1 to T do
7:
Collect off policy experience using тH and π
8:
Update lower primitive via Soft Actor Critic (SAC) and IRL
9: Sample transitions from Dg
Update higher policy via SAC and IRL 6
The lower policy is regularized using primitive expert demonstration dataset, and the upper level is optimized using subgoal transition dataset populated using PIP. v is the regularization weight hyper- parameter for the IRL objective. When e = O, the method reduces to HRL. policy with no higher level policy regularization. When vé is too high, the method might overfit to the expert demonstration dataset. We perform ablation analysis to choose vp in our experiments in Appendix A.3] The CRISP algorithm is shown in Algorithm 2
5

---

Under review as a conference paper at ICLR 2023
4 RELATED WORK
Learning effective hierarchies of policies has garnered substantial research interest in RL (Barto
& Mahadeean 2003; Sutton al. 1999 Parr & Russell, 1998. Dietterich] 1999). Options frame- work (Suttonn et al. 1999 Bacon et al. 2016] Harutyunyan et al. 2017| et al. 2017 Harutyun-
yan et al., 2019: Klissrrov et al. 2017) learns temporally extended macro actions, and a termination function for solving long horizon tasks. However, these approaches run into degenerate solutions in absence of proper regularization, where a sub-policy either terminates after each step, or runs for the entire episode. In conditindt learning, some previous approaches restrict the search space by greedily solving for specific goals (Kacibling 1993; Foster & Dayan 2002). This approach has also been extended to hierarchical RL (Wulfmeier al.
2019 2020 Ding et al. 2019). HIRO (Nachum et al. 2018) and HRL with hindsight (Levy et al. 2017) approaches deal with the non-stationarity
issue hierarchical learning by relabeling transition data for training goal-conditioned policies, where the higher level predicts subgoals for the lower primitíve. In contrast, our method deals with non-stationarity by regularizing the higher policy with imitation learning to provide curriculum of achievable subgoals to the lower primitive. Our approach is inspired from curriculum learning (Ben-) gio et al. 2009), where the task difficulty for the lower primitive gradually increases in complexity, thereby amortizing non- non-stationarity.
Previous approaches that leverage expert demonstrations have shown impressive results (Nair al., 2017 Rajeswaran et al.) [017] Hester et al.) (017). Expert demonstrations have also been used
to bootstrap option learning (Krishnan et al. 2017b Fox et al.] 2017: Shankar & Gupta 2020
Kipf et al. 2019). Other approaches use imitation learning to bootstrap hierarchical approaches
in complex task domains (Shiarlis et al., 2018: Krishnan et al.) 2017a| 2019, Kipf et al., 2019).
Relay Policy Learning (RPL) (Gupta et al. 2019) parses uses simple fixed window based approach
for parsing expert demonstrations to generate subgoal transition dataset for training higher level policy. However, fixed parsing based approaches might predict subgoals that are either too hard for the lower level primitive, in which case the higher level is cursed with ambiguous extrinsic reward signal, or too easy subgoals, in which case the higher level is forced do most of the heavy-lifting for solving the task. In contrast, our data relabeling technique PIP segments expert demonstration trajectories into meaningful subtasks, without requiring an external expert. Our adaptive parsing approach considers the limited goal reaching ability of lower primitive, and is therefore able to produce much better subgoals.
5 EXPERIMENTS
For experimental analysis, we considered various complex tasks with continuous state and action spaces. We perform experiments on the following two robotic Mujoco [Todorov al.) 2012) envi- ronments: (i) maze navigation, and (ii) pick and place environment. These environments employ 7-DoF robotic arm to perform maze navigation and robotic manipulation. We empirically show the performance comparison of our approach with various baselines in challenging task domains.
5.1 COMPARATIVE ANALYSIS
Here, we enlist the baseline methods for comparison, and explain the rationale.
Relay Policy Learning (RPL) (Gupta et al. 2019) parses subgoals from expert state demonstrations using fixed window approach. We use this baseline highlight the ad- vantage of adaptive parsing of subgoals compared to fixed window parsing. We perform extensive search for the window size hyper- parameters RPL. for each environment, which we provide in appendix A
2-level policy (Hier) denotes hierarchical policy where the high level policy is trained using only reinforcement learning, and the lower level policy is trained using RL and IRL using primitive expert demonstrations. We use it to show the importance of curriculum based subgoal generation and consequent IRL based regularization on the higher level policy. The hierarchical 2-1evel policy where both the upper and lower levels are trained using only RL failed to show good performance compared to other methods. Hence, we do not include it in the baseline comparisons.
Hierarchical

---

Under review as a conference paper at ICLR 2023
Discriminator Actor Critic (DAC) (Kostrikov al. 2018) uses IRL to learn single-level policy using low level expert demonstrations D. Using this baseline, we demonstrate the advantage of using hierarchy, curriculum based subgoal generation and consequent IRL based regularization in our approach.
5.2
ROBOTIC MAZE NAVIGATION ENVIRONMENT
Here we provide details about the maze navigation environment, its implementation and results.
5.2.1 ENVIRONMENT SETUP
In this environment, 7-DOF robotic gripper navigates across random four room mazes. The gripper arm is kept closed and the positions of walls and gates are randomly generated. The table is discretized into rectangular W H grid, and the vertical and horizontal wall positions Wp and Hp are randomly picked from (1, W 2) and (1, H - 2) respectively. In the four room environment thus constructed, the four gate positions are randomly picked from (1, Wp - 1). (Wp + W 2), (1, H 1) and (Hp + 1, H 2). The height of gripper is kept fixed at table height, and it has to navigate across the maze to the goal position(shown as red sphere). The maximum task horizon T is kept at 225 timesteps, and the lower primitive is allowed to execute for c = 15 timesteps.
5.2.2 IMPLEMENTATION DETAILS
The following implementation details refer to both the higher and lower level polices, unless oth- erwise explicitly stated. The state and action spaces in the environment are continuous. The actor, critic and discriminator networks are formulated as 3 layer fully connected neural networks with 512 neurons each layer. The state represented as the vector [p. Ml,, where p is current gripper position and M is the sparse maze array. The higher level policy input is thus concatenated vector M,M.g) g). where g the target goal position, whereas the lower level policy input concatenated vector MP,,,s, 8g]. where 8y is the sub-goal provided by the higher level policy. The current position of the gripper the current achieved goal. The sparse maze array M is a discrete 2D one-hot vector array, where 1 represents presence of a wall block, and 0 absence. In experiments, the size of and M are kept to be 3 and 110 respectively. The upper level predicts subgoal Sy. hence the higher level policy action space dimension is the same the dimension of goal space. The lower primitive action a which is directly executed on the environment, is a 4 dimensional vector with every dimen- sion a; E [0, 1]. The first 3 dimensions provide offsets be scaled and added gripper position for moving it to the intended position. The last dimension provides gripper control() implies a fully closed gripper, 0.5 implies half closed gripper and implies fully open gripper). We select 100 randomly generated mazes each for training, testing and validation. For selecting train, test and validation mazes, we first randomly generate 300 distinct mazes, and then randomly divide them into 100 train, test and validation mazes each. Each experiment is run on 4 parallel workers. We use off-policy Soft Actor Critic (Haarnoja et al. (0188) algorithm for optimizing RL objective in our experiments. We keep the regularization weight hyperparameter as v 0.0078 in our experi- ments. We use Adam (Kingma & Ba 2014) optimizer in our experiments. The hyperparameter u which is the number of training iterations after which the replay buffer is flushed and re- re-populated set as 100. The experiments are run for 2.93e6 timesteps. The method for generating expert demonstrations is provided in Appendix A.4
5.2.3 RESULTS
In Table T we report the success rate performance of the proposed methods and other baselines in the maze navigation and pick and place environments averaged over 3 seeds. CRISP-IRL CRISP with Inverse (L(IRL) regularization objective, and CRISP-BC is CRISP with behavior cloning(BC) regularization. While training and testing, we evaluate success rates over N = 100 random episodic rollouts. Since the test mazes randomly generated, the performance also proves measure of generalization capability of our proposed approach. From the results, it is evident that the proposed approach outperforms the baselines, and demonstrates impressive generalization.

---

Under review as a conference paper at ICLR 2023
Table 1: Success rates: Maze navigation and Pick and place environment
Method
Maze Navigation
Pick and Place
CRISP-IRL
0.65 0.07
0.88 + 0.03
CRISP-BC
0.53 + 0.02
0.73 + 0.01
RPL
0.57 + 0.035
0.02 1 0.01
Hier
0.52 0.08
0.02 + 0.02
DAC 0.27 = 0,06
0.38 + 0.07
5.3 ROBOTIC PICK AND PLACE ENVIRONMENT
Here provide details about the pick and place environment, its implementation and results.
5.3.1 ENVIRONMENT SETUP
In this environment, 7-DOF robotic arm gripper has to pick square block and bring/place it to goal position. We set the goal position slightly higher than table height. The maximum task horizon T is kept 225 timesteps, and the lower primitive is allowed to execute for 15 timesteps. In this complex task, the gripper has to navigate to the block, close the gripper to hold the block, and then bring the block to the desired goal position. We provide the success results and baseline comparisons in section 5.3.3
5.3.2 IMPLEMENTATION DETAILS
In this environment, the actor, critic, and discriminator networks are formulated as 3 layer fully connected networks with 512 neurons in each layer. The state is represented as the vector [p. o, q, e]. where p is current gripper position, o is the position of the block object placed on the table, is the relative position of the block with respect to the gripper, and consists of linear and angular velocities of the gripper and the block object. The higher level policy input is thus a concatenated vector [p. o, q, e, 9), where g the target goal position. The lower level policy input is concatenated vector [p, 0, q, e, 8gl. where 8g is the sub-goal provided by the higher level policy. The current position of the block object is the current achieved goal. In our experiments, the sizes of p, o, q. are kept be 3, 3, and 11 respectively. The upper level predicts subgoal sg. hence the higher level policy action space and goal space have the same dimension. The lower primitive action is dimensional vector with every dimension a; [0, 1]. The first 3 dimensions provide gripper position offsets, and the last dimension provides gripper control(0 means closed gripper and means randomly generated(block
open gripper). While training, the position of block object and goal always initialized on the table, and goal is always above the table at a fixed height). We select 100 randomly pick and place environments each for training, testing and validation. For selecting train, test and validation mazes, we first randomly generate 300 distinct environments with different block and target goal positions, and then randomly divide them into 100 train, test and validation mazes each. Each experiment is run on 4 parallel workers. We use off-policy Soft Actor Critic (Haarnoja al. (2018b) algorithm for the RL objective our experiments. We keep the regularization weight hyperparameter as V 0.005 in our experiments, and use Adam (Kingma & Ba (2044) optimizer our experiments. The hyperparameter u which is the number of training iterations after which the replay buffer is flushed and re- ce-pouulated is set as 50. The experiments The method for generating expert demonstrations is provided in Appendix A.5
for 6. 6.75c6 timesteps.
5.3.3 RESULTS
In Table I. we report the success rate performances averaged over 3 seeds. CRISP-IRL is CRISP with Inverse RL(IRL) regularization, and CRISP-BC is CRISP with behavior cloning(BC) regu- larization. While training and testing, we evaluate success rates over N = 100 random episodic rollouts. From Table T is apparent that CRISP-BC and CRISP-IRL clearly outperform the base- lines by a large margin. This provides convincing evidence that stable hierarchical learning indeed demonstrates better performance on complex long horizon tasks.
8

---

Under review as a conference paper at ICLR 2023
Figure 3: The success rate plots show performance comparison between method and baselines in maze navigation (Column I) and pick and place (Column 2) versus number of training epochs. Columns 3 and plots compare the methods via distance metric(average distance between achieved goal and final goal in 100 episodic rollouts) in room maze navigation (Column 3) and pick and place (Column 4). As can be seen, methods show impressive performance compared to baselines.
5.4 ABLATIVE STUDIES
To elucidate the importance of various constituent design choices in our proposed approach, we show success rate comparison plots in Figure 3 columns 1 and 2. We compare proposed approach with RPL to demonstrate the advantage adaptive parsing over fixed window parsing and thus segmenting the task into meaningful subtasks using the lower primitive, while using curriculum of subgoals for evolving lower primitive. The comparison with Hier method shows the advantage of curriculum based subgoal regularization using imitation learning. Finally, DAC highlights the importance of using hierarchy and curriculum based subgoal regularization using imitation learning. CRISP shows faster convergence and stable learning when compared to other approaches. In Figure 3 columns 3 and 4, we compare the methods in terms of distance between final achieved goal and desired goal averaged over 100 episodic rollouts. This metric gives an idea of how accurately an approach solves the task. CRISP clearly demonstrates better accuracy in reaching the final goal.
The number of expert demonstrations is kept to be 100 after performing ablation experiments as shown in Figure Appendix(A.3 If the number of demonstrations is less, the policy might overfit to the demonstrations, thereby hampering overall performance, as depicted Figure 4 Although the number of available expert demonstrations is generally dependent on the task environment, we increased the expert demonstrations until there was no significant performance improvement. important to note that since CRISP uses PIP to select good subgoals from lower level expert dataset, we require "good" lower level expert demonstration trajectories. the expert trajectory is "bad", PIP is unable select good subgoals, leading to poor performance.
is
We perform ablation experiments for choosing hyperparameter u, which is the number of training iterations after which the replay buffer is flushed and re-populated. The ablation experiments provided in in Appendix A.3 For RPL. experiments, we choose the window size syper-parameter by running RPL experiments for different values of c. The experiments Appendix A.33 After performing experiments in maze navigation and pick and place environments, is set 4 and 8 respectively. We also performed ablation analysis for choosing the imitation learning weight hyperparameter A. The experiments are shown in Figure in Appendix A.3
shown in Figure 7 in
6 DISCUSSION AND FUTURE WORK
We introduce CRISP, which is our general purpose lower level primitive informed method CRISP for efficient hierarchical reinforcement learning. CRISP leverages primitive parsed expert demonstra- tions and performs data relabeling on expert demonstrations to populate subgoal transition dataset for regularizing higher level policy. CRISP employs hierarchical curriculum learning to deal with non-stationarity. We evaluate our method on complex robotic maze navigation and pick and place manipulation tasks, and demonstrate that it makes substantial gains its baselines.
However, CRISP assumes the ability to reset the environment to any state from expert demonstra- tions dataset while collecting subgoal dataset using PIP. A possible method for relaxing this assump- tion is to combine CRISP with (Eysenbach et al. 2017) that learns a backward controller that tries to reset the environment. This approach an interesting avenue for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anurag Ajay. Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofrr Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning, 2020.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel H Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NIPS, 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. abs/1609.0514 0404 2016.
The option-critic architecture.
CoRR,
Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. 13:2003, 2003.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning. ICML '09, pp. 41 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374 1553380.
URL eetpss://cc..rggg/ 1145/
1553374.15538
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in Neural Infor- mation Processing Systems 5, [NIPS Conference), pp. 271-278, San Francisco, CA, USA, 1993.
Morgan Kaufmann Publishers Inc. ISBN 1-55860-274-7.
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decom- position. CoRR, cs/9999905,14, 1999. URL http::://atxxiii. org/ abs/ cs/ 990501
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imita- tion learning- In H. Wallach, Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL httppp////ppooeee............eep
pdf
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning, 2017.
David Foster and Peter Dayan. Structure in the space of value functions. Machine Learning, 49 (2-3):325-346, 2002.
Roy Fox, Sanjay Krishnan, lon Stoica, and Ken Goldberg. Multi-level discovery of deep options, 2017.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277. PMLR, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay pol- icy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRR, abs/1910.1 1956, 2019.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 1851-1860. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs/1800.011900 01290
URL /...SS//B//tttp//aaaa...../////
Jean Harb, Pierre-Lur Bacon, Martin Klissarov, and Doina Precup. When waiting is not option Learning options with a deliberation cost. CoRR, abs/1709.0447l. 2017. URL http://
wxivv.or// wxxiworrg/ab//17000. 04571
10

---

Under review as a conference paper at ICLR 2023
Anna Harutyunyan, Peter Vranex, Pierre-Luc Bacon, Doina Precup, and Ann Nowé. Learning with options that terminate off- policy. CoRR, abs/1711.03817, 2017. URL htttlattataxi...o.ot
abs/1711. 03817
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Rémi Mumos, and Doina Precup. The termination critic. CoRR, abs/1902.00996, 2019. URL http::://aaii...or///b//////
09996
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John P. Agapiou, Joel Z. Leibo, and Au- drunas Gruslys. Learning from demonstrations for real world reinforcement learning. CoRR, abs/1704.03732, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR, abs/ abs/1606.03376, 2016. URLbttp://arxii httt::://xxxi.orrgaab/////// wwt./rrg////eeeeeeee....eeeeeeee 03476
Leslie Pack Kaelbling. Learning to achieve goals. In PROC. OF IJCAI-93, pp. 1094-1098. Morgan Kaufmann, 1993.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL cite 1.w:14442.9900ommentt Published as con-
http:// /ttt::///rxivv aariivvv owgoo///////... ference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinieius Zambaldi, Alvaro
Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference Machine Learning, pp. 3418-3428. PMLR, 2019.
Martin Klissarov, Pierre-Luc Bacon, Jean Harb, and Doina Precup. Learnings options end-to-end for continuous action tasks. CoRR, abs/1712.000044 2017. URL
1712.00004
Ilya Kostríkov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. iiseriminattr-actor--rtiii Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:/809.02990 2018.
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. DDCO:
discovery of deep continuous options forrobot learning from demonstrations. CoRR, abs/1710.054211, 2017a. URL http:
// www.i..rggabbs/1111....
Sanjay Krishnan, Roy Fox, Ion Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations, 2017b.
Sanjay Krishnan, Animesh Garg, Richard Liaw, Brijen Thananjeyan, Lauren Miller, Florian T Poko- ry, and Ken Goldberg. Swirl: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards. The International Journal of Robotics Research, 38(2-
2019. doi: 10.117770778:441188843353
URL. https:/ /oii .org oorg/10.11777 1177/
02783649187843577
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Joshua Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. CoRR, /5w/1604406057, 2016.
Steven M. Lavalle, explloii random trees: new tool for path planning. Technical report, 1998.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. CoRR, abs/1504.00702. 2015.
Andrew Levy, Robert Platt Jr., and Kate Saenko. Hicrarchical actor-critic. CoRR, abs/ abs/1712,00948, 2017.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, and Zhen Wang. Multi-class generative adversarial networks with the L2 loss function. CoRR, abx/1611.04076 2016. URL http:
7/arxiv. waa.ii..orr/bb//////1. 04076
11

---

Under review as a conference paper at ICLR 2023
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. CoRR, abs/1805.0055 185.08226, 2018.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: 10618, 2019.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration reinforcement learning with demonstrations. CoRR, abs/1 1009,, 2017.
Ronald Parr and Stuart Russell, Reinforcement learning with hierarchies of machines. M. Jordan, M. Kearns, and S. Solla (eds.), Advances in Neural Information Processing Systems, volume 10. MIT Press, 1998.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. CoRR, abs/1709. 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07-09 Jul 2015. PMLR. URL https: htttpsss/pprcceeeeiingaoo http://poooeeeeinneeeec. ml
http:cceeerrnna.......rrssss/// p.pss///377
schaul html
T. Shankar and Abhinay Gupta. Learning robot skills with temporal variational inference. In ICML, 2020.
Kyriacos Shiarlis, Markus Wulfmeier, Sasha Salter, Shimon Whiteson, and Ingmar Posner. TACO: Learning task decomposition via temporal alignment for control. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4654 4663. PMLR, 10-15 Jul 2018. URL btttss:/prroeeeinnee
press v http:/hhaaaiiiiii htm_
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 1 12(1))181-211 1999. ISSN 0004-3702. doi: http://ww.......oo..oo.coc.cctaa sciencedirect wttp:///....ccecccccneeeccc
URL ht httpp:: : www.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026-
5033. IEEE, 2012.
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demon- strations for deep reinforcement learning on robotics problems with sparse rewards. CoRR, abs/1707.08817, 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. CoRR, abs/1703.0116) 2017.
Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenherg, Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Y. Siegel, Nicolas Heess, and Martin A. Riedmiller. Regularized hierarchical policies for compositional transfer in robotics. CoRR, abs/1906.11222 2019. WRLLattp://arxee httttp:://axxi..orr/aaassss org. /196. 11228
Markus Wulfmeeerr Dushyant Rao, Roland Hafner, Thomas Lampe, Abbas Abdolmaleki. Tim Her- tweck, Michael Neunert, Dhruva Tirumala, Noah Y. Siegel, Nicolas Heess, and Martin A. Ried- miller. Data-efficien hindsight off pollcc option learning, CoRR, abs/2007.15588, 2020. URL
https://arxi:s org/ abs /200 15588
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
SUB-OPTIMALITY ANALYSIS PROOF FOR HIGHER LEVEL POLICY
The sub-optimality of upper policy Ten. over length sub-trajectories sampled from d can be bounded as:
[J(m)) ((TH)) Aω OD AH -1I (o (rls, (F(s (8)
where AØ (I-7)(1-7) Remax
10.0
Proof. We extend the suboptimality bound from (Ajay et al. 2020) between goal conditioned poli- cies n² and H. as follows:
2
(( (1 7) 70) Rmax Eswdy (T ( (r|s,g))) (9)
By applying triangle inequality:
Drv (TIs. (r|s,g)) s Div(T
(10)
Taking expectation wrt k. G and # I.
Benv.g-clDiv(( (r|s, S E (es
Bon.a g) (T|s,g))]
(11)
Since nt is ΦD common in 11 we can write as:
Genn.g-((Drv(6 (a" (r|s,g)][= m) (r|s, s ΦD + B g) (rls,g))
(12)
Substituting the result from Equation 12 in Equation 9 we get
[J(*') - [ H1 & Au ΦD λy E
B (ar (r|s,))) (13)
where A (I-クX y Remae ||
d5 * 11:0
A.2 SUB-OPTIMALITY ANALYSIS PROOF FOR LOWER LEVEL POLICY
Let the optimal lower level policy be nr" The suboptimality of lower primitive mhe can be bounded as follows:
[J(=²²) - J( S dr. * dn + dt. * E B (Drv(T ,,) a (r|8, sn))
(14)
where d = (1-7)2 Rmar de" olc
Proof. We extend the suboptimality bound from (Ajay et al. 2020 between goal conditioned poli- cies T** and Th; as follows:
-(T"") 5 (1 71- Rmaalesnty araty DDv((" (T),,, (r(8,8,))) (15)
By applying triangle inequality:
(T|s, (r(s,,,)) s ( ())((ss) sy)) + Dr (rb(TT( sg)ll (rls,s)))
(16)
13

---

Under review as a conference paper at ICLR 2023
Taking expectation wrt K, 89 Thz and x ~ 11/;
Bywnsyyrth [Pr((T (r|s, (T]s, S E snn - [Drv (t (r]ss== (T|s. 3. +g))[+
E + 11 - DD ( & (r|s, sy))
(17)
Since T² is ΦD common in IT we can write 17 as:
Bawn,syath (T 5g) | m0. (rls, S ΦD E be [Drv (T g))l (r|s, 8g))l
(18)
Substituting the result from Equation 18 in Equation
15
we get
[J(T")) - - )l S Xz. * ΦD + X * E Eanx -IT Doromt [DTV(T (T)((T)s,,))I (T|s, 8n))]]
(19)
where dL (1-7)22 Raaax d= oc
A.3 ABLATION EXPERIMENTS
Success
03
04
Epoch
Figure 4: Column 1(maze navigation environment) and column 2(pick and place environment) show success rate performance plots of varying number of expert demonstrations versus number of train- ing epochs. The number of expert demos should not be too less to cause the policy overfit. We increase the number demonstrations until there is no significant progress in policy performance. Based on the experiments, we choose 100 expert demonstrations for both environments.
-0:10
Success Rate
125
201 Epoch
752
Epoch
Figure 5: Column (maze navigation environment) and column 2(pick and place environment) show success performance plots of CRISP for different values of hyperparameter u, plotted against number of training epochs.
14

---

Under review as a conference paper at ICLR 2023
N-0
754-00012
Auud
Success Rate
9.28
Epoch
Epoch
Figure 6: The success rate plots show the performance of CRISP for various values of imitation learning weight parameter & versus number training epochs. Column shows experiments on maze navigation environment and column 2 shows experiments pick and place environment.
BALE
Success Rate
02
903
10
Epoc1
Figure 7: The success rate plots show the performance of RPL for various values of & window size parameter versus number of training epochs. Column shows experiments on maze navigation environment and column 2 should experiments pick and place environment.
A.4 GENERATING EXPERT DEMONSTRATIONS FOR MAZE NAVIGATION TASK
We use the path planning RRT Lavalle 1998) algorithm to generate optimal paths P (pt, Pt+1: 2t+2; from the current the obstacle position which is provided to the methods through state consisting sparse maze array. Using these expert paths, we generate state-action expert demonstration dataset for the lower level policy.
the goal state. RRT has privileged information about
A.S GENERATING EXPERT DEMONSTRATIONS FOR PICK AND PLACE TASK
In order to generate expert demonstrations, used human expert to perform the pick and place task in virtual reality based Mujoco simulation. In this task, an expert first picks up the block using robotic gripper, and then takes it the target goal position. Using these expert trajectories, generate state-action expert demonstration dataset for the lower level policy.
A.6 ROBOTIC ROPE MANIPULATION ENVIRONMENT
Here provide details about the rope manipulation environment, its implementation and results.
A.6.1 ENVIRONMENT SETUP
In the robotic rope manipulation task, a deformable rope is kept on the table and the robotic arm performs pokes nudge the rope towards the desired goal rope configuration. The task horizon is fixed at 25 pokes. The deformable rope formed from 15 constituent cylinders, held together by joints.
15

---

Under review as a conference paper at ICLR 2023
Table 2: Success rates:
Rope manipulation environment
Method
Rope Manipulation
CRISP-IRL
0.29 0.04
CRISP-BC
0.32 = 0.056
RPL
0.18 0.12
Hier
0.1 0.07
A.6.2 PRETRAINING TASK FOR THE LOWER LEVEL PRIMITIVE
In this complex environment, we first pretrain the lower level primitive using simpler goal rope configurations which can be achieved within few pokes (the simple goal rope configurations are chosen based on L.2 distance between the initial and goal rope configurations). We specially pre- trained the lower primitive a simpler task in rope manipulation environment since without pre- training, the methods failed to provide significant results. In order to ascertain fair comparisons, we kept this pre-training requirement consistent among all the baselines in this environment. After pre-trate pre-training the lower primitive, the lower primitive is used with our method CRISP.
A.6.3 GENERATING EXPERT DEMONSTRATIONS
In complex environments, we generally do not have access to lower level expert demonstrations. Moreover, hard coding expert policy may generate sub-optimal expert demonstrations. In rope environment we did not have access to lower level expert demonstrations. However, recall that method requires only expert state demonstrations and not expert action demonstrations. For generating expert state demonstrations, we used an interpolation based approach, where we obtain subgoals 5g. by linearly interpolating between the starting rope configuration and the final rope configuration g as:
- i (1, ... N 1
(20)
We found N = 15 to perform well empirically. After generating these interpolations, we performed simple transformations to assure that the interpolations are valid rope configurations. Note that since we do not have access to lower level expert action demonstrations, we do not use expert actions demonstrations dataset to train the lower level in CRISP and other baselines.
A.6.4 IMPLEMENTATION DETAILS
The following implementation details refer to both the higher and lower level polices, unless oth- erwise explicitly stated. The state and action spaces in the environment are continuous. The actor, critic and discriminator networks are formulated 512 neurons in each layer. The state space for the rope manipulation environment is a vector formed by concatenation of the intermediate joint positions. The upper level predicts subgoal 8g for the lower primitive. The action space of the poke is (I, V, 77,, where (æ,, y) is the initial position of the poke, and 77 is the angle describing the direction of the poke. We fix the poke length be 0.08. While training our hierarchical approach, we select 100 randomly generated initial and final rope configurations each for training, testing and validation. For selecting train, test and validation con- figurations, we first randomly generate 300 distinct configurations, and then randomly divide them into 100 train, test and validation mazes each. Each experiment run on 4 parallel workers. We use off off-policy Soft Actor Critic (Haarnoja et al., 201 18)) algorithm for optimizing RL objective in our experiments. We keep the regularization weight hyperparameter as À 0.01 in our experiments.
3 layer fully connected neural networks with
A.6.5 RESULTS
In Figure 8 we show the subgoals predicted by our method at various timesteps, thereby gen- erating curriculum of subgoals for the higher level policy. In Table 2 report the success rate performance of the proposed methods and other baselines, averaged over seeds. CRISP-IRL is CRISP with Inverse RL(IRL) regularization objective, and CRISP-BC is CRISP with behavior (loning(BC) regularization. While training and testing, we evaluate success rates over N = 100 random episodic rollouts. Since the lower level expert action demonstrations
not available,
16

---

Under review as a conference paper at ICLR 2023
Rope env
Epoch 0
Epoch 200
Epach 400
Figure 8: This figure depicts CRISP subgoals progress training epochs elapsed in rope manip- ulation environment. The higher level policy generates curriculum of subgoals for lower primi- yive(walls::ellow, final goal.rd, subgoals:blue)
140
Figure 9: The success rate plots show performance comparison between method and baselines rope manipulation environment (Column 1) versus number of training epochs. Columns plot compare the methods via distance metric(average distance between achieved goal and final goal configuration in 100 episodic rollouts). As can be seen, our methods show impressive performance compared to baselines.
we do not compute the DAC baseline. From Table 2 it is apparent that CRISP-BC and CRISP-IRE clearly outperform the baselines by large margin. This provides convincing evidence that stable hierarchical learning indeed demonstrate better performance on complex long horizon tasks. The
success rate comparison plots provided in Figure
17


Reviews:
[{"review_id": "cvAQ6pAgSx", "reviewer": "Reviewer_oc7V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Learn temporally abstracted skills from expert demonstrations. This work uses a two-level hierarchy, where the subgoals given to the lower level of the hierarchy are always achievable, and progressively increase in difficulty. This is achieved by choosing subgoals of increasing length for the agent until the agent is unable to achieve the subgoal.", "summary_of_the_review": "I propose to accept this paper because it has sufficient novelty, is well written and provides interesting insights into getting HRL to work in some simulated robotics domains.", "strength_and_weaknesses": "Strengths:\n\nThe paper is relatively easy to understand and the concept is an interesting application of hindsight in learning from demonstration. \n\nTHis paper represents a fairly sizable combination of techniques to produce results.\n\nWeaknesses:\n\nThis method is limited by the requirement of being able to reset the environment to a state mid-trajectory, which either requires a carefully designed environment or access to a simulator. This can impact the application of this method to real-world tasks.\n\nThe paper combines ideas from a wide range of areas which obfuscates where the improvements come from or if a single core idea is in particular more compelling. In particular, the primitive informed parsing could be performed without an expert and have a similar effect of curriculum, still requiring the agent to move to this information. Including the expert and thus IRL/behavior cloning, this makes it less clear why these components are inherently necessary to the core idea (or resetting hindsight labels). In addition, adding on a hierarchical meta-policy seems to only add an extra layer of complexity.\n\nIn the movement domains, there is only marginal improvement, and these domains are simple enough to be toy domains. The majority of the improvements are in the pick and place domain, where it appears that both the hierarchical and RPL methods completely fail. In this case it seems like this is because pick-and-place tasks require expert information, and a basic application of that data to policy learning would probably result in a significant improvement in performance since pick and place tasks have certainly been performed by hierarchical, RL and IRL methods in the past. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XAivYkBov9C", "reviewer": "Reviewer_WG9D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript presents CRISP, an approach for hierarchical reinforcement learning from expert trajectories. Their approach relabels states from expert demonstration as subgoals to train a higher level. States are labelled as subgoals when they are the first in a subsequence that can’t be reached by the lower level from the start of this subsequence. Thus, subgoals are chosen that are “just out of reach” for the lower level, developing a learning curriculum. The two levels are trained to optimize both offline reinforcement learning and imitation learning within one objective. The paper provides a bound for the suboptimality of the method compared to an optimal policy. The method is evaluated on two simulated tasks and outperforms two hierarchical reinforcement learning baselines.", "summary_of_the_review": "In sum, I think the idea is very interesting, but the clarity and details of the method and experiments still need improvement to be accepted. The idea seems interesting, but it’s potential and limitations could be evaluated much more thoroughly in experiments. \n\n-- post rebuttal update --\nSee the answers below. I have raised my score to 5 (and updated correctness to 3). ", "strength_and_weaknesses": "Strengths:\n+ The relabelling of the expert demonstration states based on failures to reach them is a very interesting idea.  The emergence of a learning curriculum from this simple procedure seems very promising and is nicely visualized in Figure 2.\n+ The results compared to the baselines on the two investigated tasks look impressive and highlight the strength of using a hierarchy combined with the subgoal learning curriculum.\n\nWeaknesses:\n- The method part is at different parts very hard to follow, and I found various parts confusing. Below is a list of points that I think need to be addressed:\n    • p. 3, section 3.1: The lower primitive policy $\\pi^j_L$ is introduced with superscript $j$, where $j$ denotes \"an indicator of the current subgoal reaching capability of the lower primitive\". In my opinion this sentence does not sufficiently explain $j$. The indicator $j$ is not further clarified, does not change in the pseudocode, and is dropped later (section 3.2). So why is the superscript $j$ needed?\n    • p. 4, Definition 1: What is $D_{TV}$? I'm assuming total variation, but it is not introduced.\n    • p. 4, The proof for the suboptimality of the lower primitive is missing. It can probably be derived from the proof for the upper policy, however, seeing that they differ with respect to lambda it is necessary to explain where this difference comes from.\n    • p. 5, Eq. 10: What is $\\epsilon$? This needs to be introduced.\n    • p. 5, section 3.4: Why does Eq. 10 incentivize the higher policy to make \"reasonable progress towards achieving the final goal\"? Does it not just incentivize generating subgoals that are hard to distinguish from the subgoals of dataset $D_g$?\n    • p. 5, section 3.5: It reads like the off-policy RL objectives can be referred to as $J^H_D$ and $J^L_D$. Shouldn't the off-policy objective be $J^H_{\\theta_H}$ and J^L_{\\theta_L} of Eq. 11 and Eq. 12? Because $J^H_D$ is defined above as the IRL objective (Eq. 10). $J_D^L$ is not defined before. The authors need to be more precise about the different objectives.\n    • p. 5, Eq. 11 & 12: $\\lambda$ is here introduced as a hyperparameter whereas in section 3.2, $\\lambda$ was used as a factor for the upper bound of suboptimality. To avoid confusion different names should be used.\n- From the experiments it is not clear how much the approach relies on the expert trajectories to learn to solve the tasks. How many expert trajectories were used for each of the two tasks? How were the expert trajectories generated? How does the number of expert demonstrations influence the system's performance? For example, how do fewer or more demonstrations affect the performance? Could the system deal with \"bad expert demonstrations\" that are not useful in solving the task? For example, what would happen when a couple of random rollouts are included in the demonstration dataset? I think these questions need clarification in the description of the experiments, further discussion, and maybe additional experiments.\n- What is the effect of resetting the subgoal buffer, i.e., the hyperparameter $u$? How does $u$ affect the sample efficiency of the approach? An ablative study evaluating different values of $u$ would help understand how resetting the subgoal dataset affects the method.  \n- How is CRISP-BC different to CRISP-IRL? Why does it perform better than CRISP-IRL in the Pick and Place task? Highlighting the difference between the two versions would also help understanding the approach.\n- The method requires that the simulator can be reset to an arbitrary state. This is a major restriction for applying the method. The authors acknowledge this limitation but a more detailed discussion on how this limitation could be weakened would improve the paper. For example, it prohibits applying the system on real robots. Thus, the claim of the abstract that the method “[…] is suitable for most real world robotic control tasks” is too strong.\n- Please fix the citation style. Use citet only of the citation is part sentence, grammatically (as a subject or object). Otherwise ,use \\citep. Here an example: “Learning effective hierarchies of policies has garnered substantial research interest in RL Barto & Mahadevan (2003)” should be “Learning effective hierarchies of policies has garnered substantial research interest in RL (Barto & Mahadevan, 2003)”.\nMinor corrections and suggestions:\n- typo on p. 1: \"balnced\"\n- bottom of p. 5: it should be Algorithm 2 instead of Algorithm 0\n- typo on p. 9: \"tn maze navigation\"\n- Figure 1 should be referenced in the text.\n- introduce the abbreviation IRL on p. 3 when first mentioning inverse reinforcement learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2WwBDSA_RIs", "reviewer": "Reviewer_xPx6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the problem setting of off-policy hierarchical reinforcement learning, focusing on automatic-curriculum generation (via the proposed PIP) and bootstrapping from  a handful of expert demonstrations using imitation-based regularization. The paper’s contributions include an adaptive parsing method that segments state-only expert demonstrations into skills that are at the fringe of the current capabilities of the lower-level policy. The proposed adaptive method is shown to outperform its fixed counterpart on two simulated tasks involving a robotic manipulator moving through a maze, and moving a block to a desired goal position. Furthermore, visualizations in the paper confirm that subgoals of increasing difficulties are set by the higher-level policy as training progresses (see Figure 2, Page 3 of manuscript).\n", "summary_of_the_review": "Overall, the paper presents an original and promising idea, but falls short in terms of its experimental evaluation of the idea. In addition, certain flaws related to clarity hinder the interpretability of the results (see ‘Strength And Weaknesses’). I am willing to reconsider my evaluation if weaknesses are addressed, either with new results, or clarifications to questions.", "strength_and_weaknesses": "Strengths Of The Paper:\n\n1) One way the proposed approach is appealing is due to the elimination of one form of expert annotation: using the lower-level policy to adaptively parse expert state-demonstrations into skills, rather than a fixed horizon-based parse.\n\n2) Suboptimality bounds for the higher-level policy and lower-level policy are given, which is implied to become tighter as the lower-level skills improve as training progresses.\n\n3) Implementation details for the algorithm on both tasks is discussed at length, and specific hyperparameter values are given which will help to facilitate reproducibility.\n\nWeaknesses Of The Paper:\n\n1) Experimental evaluation of the approach, while showing the proposed method in a positive light, is not thorough, and can be improved in several ways. Firstly, the method is only evaluated on two reinforcement learning tasks, both of which are based on the same position-based robotic manipulator. This environment functions similarly to a 3D pointmass, where actions are directly added to the current 3D position of the gripper. The environment is made more challenging by using a sparse reward, but the complexity of the underlying task is simple in comparison to other tasks that were studied in previous HRL papers, including locomotion-based maze navigation (from HIRO, Nachum et al. 2018), and torque-based manipulation (from HAC, Levy et al, 2018).\n\n2) While the proposed adaptive parsing method is effective in practice, some details are unclear, which I will discuss next. First, the paper states that on the maze task the replay buffer is flushed and re-populated every 200 steps, and 100 steps for the block task. How many new environment steps are taken or required during this flushing and re-population operation, and are these included in the budget reported (2.93e6 timesteps for the maze task and 6.75e6 timesteps for the block task). For a fair comparison, baselines would ideally be compared to the proposed method under the same conditions (ie, the same number of environment timesteps, including those used by PIP). It remains possible the evaluation already takes this detail into account, but is unclear from the text.\n\n3) On Page 6 of the manuscript, the authors state “our method deals with non-stationarity by regularizing the higher policy with imitation learning” and this is not validated. I agree with the author’s that it is likely the proposed approach does mitigate non-stationarity by design, but this claim is not experimentally verified. Such an experimental verification should be performed that compares to other HRL methods that mitigate non-stationarity. Building on Weakness (2), the authors should ensure all methods operate in the same conditions in regards to any additional samples that PIP requires to relabel the buffer.\n\n4) Performance is reported in a somewhat uncommon format. It would be helpful to report success rate versus the cumulative number of environment samples collected (including any additional samples that may be required during the PIP parsing step, if any).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject