Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates
Authors: Souradeep Dutta
Kaustubh Sridhar
Osbert Bastani
Edgar Dobriban
James Weimer
Insup Lee
Julia Parish-Morris
Venue: CoRL 2022 Poster (2022)
Keywords: Sample-Efficient Reinforcement Learning
Expert Intervention
Options
Planning with Primitives

Abstract: Long horizon robot learning tasks with sparse rewards pose a significant challenge for current reinforcement learning algorithms.
A key feature enabling humans to learn challenging control tasks is that they often receive expert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions.
We propose a framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider \emph{option templates}, which are specifications encoding a potential option that can be trained using reinforcement learning. We formulate expert intervention as allowing the agent to execute option templates  before learning an implementation. This enables them to use an option, before committing costly resources to learning it. 
We evaluate our approach on three challenging reinforcement learning problems, showing that it outperforms state-of-the-art approaches by two orders of magnitude. Videos of trained agents and our code can be found at: https://sites.google.com/view/stickymittens

Exploring with Sticky Mittens:
Reinforcement Learning with Expert Interventions via Option Templates
Souradeep Dutta', Kaustubh Sridhar', Osbert Bastani' Edgar Dobriban', James Weimer?,
Insup Lee', Julia Parish-Morris
University of Pennsylvania, VVanderbilt University, 'Children's Hospital of Philadelphia
Abstract: Long horizon robot learning tasks with sparse rewards pose signif- icant challenge for current reinforcement learning algorithms. A key feature en- abling humans to learn challenging control tasks is that they often receive ex- pert intervention that enables them to understand the high-level structure of the task before mastering low-level control actions. We propose framework for leveraging expert intervention to solve long-horizon reinforcement learning tasks. We consider option templates, which are specifications encoding potential op- tion that can be trained using reinforcement learning. We formulate expert in- tervention allowing the agent execute option templates before learning an implementation. This enables them to use an option, before committing costly resources to learning it. We evaluate our approach on three challenging reinforce- ment learning problems, showing that it outperforms by two orders of magnitude. Videos of trained agents and our code can be found at: https: /sites/sssess/ee google. www/viow//////eccc wtt.////siieeoe wttp//siickiiiitt ttens
oate-oo-the-asse art approaches
Keywords: Sample-Efiiet Reinforcement Learning, Expert Intervention, Op- tions, Planning with Primitives
Introduction
Reinforcement learning is an effective tool to solve difficult tasks such as robot planning and loco- motion [1] but exploration is still a challenge. Options are an RL tool to circumvent this problem [2]. Designed to achieve intermediate subgoals. For instance, in robot grasping tasks, an option might enable the robot to grasp block, which is a subgoal needed to build a tower out of blocks. The goal learn a policy mapping each state to an option, instead of concrete action take.
When learning to perform complex visual-motor skills, humans often rely on expert interventions to help them escape these challenging reward plateaus. For instance, the sticky-mitens experiment [3] considers infants who have not yet learned to grasp objects. They give a subset of these infants mittens covered with Velcro hooks and allow them to play with toys fitted with Velcro loops, making it significantly easier for them to grasp these toys. Even if the Velcro is taken away, these babies learn how grasp objects significantly faster than infants not exposed this experience. In other words, enabling infants to explore unreachable parts of the state space helps guide them towards skills that are worth learning. This is a well known phenomenon developmental psychology. which extends beyond fine motor skills.
In this paper, we design an RL algorithm based on the idea from the mtitky-mittens experiment. The agent has access to an alternative Markov Decision Process (MDP) where the agent can leap multiple states without first learning policy to do in the original MDP. We term such a jumping mechanism an option template. Option templates are described using a set of initial states and a set of final states. The idea of providing external help in the learning phase referred to as primitives or skills in literature. It offers a practical way to speed-up learning in a realistic setting. For instance,
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

using parameterized action spaces for RoboCup in [4], stitching independent behaviors in [5, 6] and providing action primitives in [7, 8, 9] to mention a few.
In the more typical RL framework the agent first uses reinforcement learning to train an option to implement the specification of each option template. The issue with this strategy is that, in RL. environments with large state spaces the options learnt need to be of a fairly generic nature, in order to be useful. Which is hard without a knowledge of the state distribution, where the options will be invoked. Using option templates we decouple the implementation of an option from its utility.
In more detail, our algorithm performs an alternating search over policies: at each iteration, it first optimizes the high-level policy over the current option templates, and then learns options to imple- ment the option templates. In our experiments, we demonstrate that by leveraging option templates, our algorithm can achieve orders of magnitude reduction in sample complexity compared to the typical strategy of learning the options upfront and then planning with these learned options.
Contributions: Our contributions are: (1) an RL algorithm that leverages option-templatee and (2) empirical comparison with state-of-the-art RL techniques on three challenging environments, demonstrating maggntt-- reduction in sample complexity.
Background
We consider the RL setting where agent interacts with an enviromment modeled an MDP [10]. Definition 2.1 (MDP). A Markov Decision Process (MDP) is a tuple E (S, P. R, 7. where S C R" is the set of states, A C rM is the set of actions, P(s'|s, a) is the probability of transitioning from state 8 to upon taking action a, [s, a) is the reward accrued in state upon taking action a E [0, 1) is the discount factor, and I is the initial state distribution.
The value of state for policy is the expected return R,,(so) starting from state 8o. while executing the policy π at every step, a (s:). The optimal policy x" maximizes the reward start- ing from the initial state distribution-i.e. x" = arg max (T). where (m) EI (R. Semi-MDPs (SMDPs) extend this framework by allowing temporally extended actions-i.. peniiiii..lllliiies policies exccuted over multiple steps-calles options [2].
Optien-iamplate
((
(((a))
Figure 1: Option vs Option Template
Definition 2.2 (Option). An option o is a tuple o := (T,
where C S are initial states, T S - [0, 1] the termination condition, and π : S - A is policy.
For now, assume that we have a set O of options available for an MDP. Intuitively, options constrain the search space over policies, thereby guiding exploration performed by the agent towards more promising action sequences. We now consider policies II : S + O that map a state E I (i.e., is a valid initial state for o) to an option o II(s). The agent uses option policy и until the termination condition holds. More precisely, after taking action T(s) and transitioning to state P(-ss, a), it stops using æ with probability T(s++1) and chooses another option of = [[ss';; otherwise, it continues using π.
Suppose that option o is invoked at time in state st, and the system follows k step trajectory (r k) = (St, Gt, Ft, 1 ..,, St4k). ). We consider policies such that for L S i < + k. the action a; depends on the entire history Tr i) from / until i + This renders the behavior non- Markovian. Denoting the set of all such state-action trajectories as H, we can allow option policies π. H + A. and termination conditions T' : H + [0. 1] to be Markovinn. Now, we recall the definition of Semi-MDPs as an extension of MDPs:
Definition 2.3 (Semi-MDP, Sutton et al. [2). A semi-MDP is tuple (8, R, I), where S is a set of states, O is a set of cemi-Markovian options, P is the transition probability between states, R. is the reward function, 7 [0. 1) is the discount factor, and I is the initial state distribution.
The state-prediction part of the model for taking an option o, in state I and transitioning to s' is given by Sutton et al. [2]: P(s'|s, (, o) = Σx= "pp(s', s. k), where p(s', s, k) is the probability that the option terminates in 8' after steps from the time the option is invoked. The option- value form of standard value iteration is, for E S and o E O, [2, eq. 12). [11] (8, R(s, 0) + Ev's P(&'|s, o) max (s', , o'), where s' is the state the system reaches after exe-
cuting option starting from state SMDP value learning updates the Q-vauues at the end of each

---

Q (s. + () o)-oa))--a--) max EO o')s''" o]], for similar definitions of k, 8, s', with r. being the cumulative discounted reward over the time hori- zon, and a E (0, the learning rate. Choosing a function approximator parameterized by 0 to represent (ss, o; 0). with loss given by: L(O.) = [r + 7 max ('(s', d'; 0.) allows one use standard Q-learning [12] methods to train with options.
option termination as [2. 195]:
Q' o; 0.)
3 Learning with Option Templates
3.1 Option Templates
Motivated by the sticky-mittens experiment, we consider an environment where in certain states, the agent can call "help switch", called an option template, that immediately transitions it from a state s, to different state st+k; this transition captures the desired result of executing an (unimplemented) option. We denote a trajectory of this MDP starting from state 8t under policy by Tr" (s:).
Definition 3.1 (Option template). Option template a tuple (I.T,)) P), where S S is the initial states, T : S > [0, is the termination condition, and P is a distribution over states 8 such that (ss) 1.
An option template similar to an option, shifts control to the expert until the termination condition is reached; this termination condition T is intended to capture the satisfaction of some sub-goal along with a timeout mechanism. Assume the termination condition time t + of an option template taken in state 9t is of the form Ts k) = F, (St+k)) V (k >> k*), where the fumction F. S - {0, 13 captures visitation of some key states, and k." an upper limit on the number of steps.
We denote the set (or subset) of option templates by Ao. By the end of training, the agent must learn policy implement each option template that it uses (i.e., the final policy cannot depend on option templates). To ensure feasiblity, we assume there is an a priori unknown policy To such that Tr" (st) satisfies T' (denoted as Tr (st) } 7) with probability at least
for some hyperparameter [0, 1). Formally, the probability of success of given policy To can be expressed as IP((T., I, } T) (1 -i..e,. the trajectory starting from state in I under the control of Tr, in environment E satisfies T with probability at least 1 8. This success probability can be estimated by empirical rollouts of m.-
We organize option-templatee in a hierarchical fashion depending on degrees of help. Option tem- plates that traverse longer sequences of transitions allow faster learning due to fewer decisions. Thus, we consider a sequence of environments E g ,"....,," _n = E, equipped with option templates of varying capabilities organized in a hierarchical fashion.
Definition 3.2 (Environment Level E¹). An environment &l learning level / is the original envi- romment & with the primitive actions A replaced by option templates A%.
The goal of the levels is to gradually increase difficulty of the learning task while guiding the agent each level in way that is similar to curriculum learning [13]. In particular, learning longer timescale option templates typically precedes learning shorter ones.
Assumption 3.3 (Realizability). For any option template E Al at level I, given by (I,T, P). there is policy Th : S - A at the following level ( such that for any E 1, Tr" = T with probability at least
That is, any option template in the current level is realizable by a policy in the following level.
3.2 Learning with Option Templates
Our algorithm for RL with option templates is presented in Algorithm 1.
3

---

Algorithm Learning with Option Templates
Algorithm Learn Option Policy
Input: Environments [ED, . Output: A set options D = for E Q} that provides implementation of each option template ag. where q E Q indexes the option templates.
Input: Environment &, Rewarr Function R, Option template a, (I,T, Set D of op- tions
Parameter: Threshold Max Episodes I. Output: Policy To as an implementation of a
Initialize =
2: for level / [L....,n} do
Initialize policy To 2: for episode = 1,..... I. do ExecutionStack =
3: Let L be the set of option templates level
4: Build reward functions {R, ag E
4: <- Top-level option from D (or d if
A--1)
D = ②)
5:
for ag E A--1 do
5: Push onto ExecutionStack
Sample initial state s ~
6:
Th =LeamOptionPolicy(("" Rgg a2 D)
7:
while not done do
7:
0 = (a., m {00}
8:
+ Pull option from Execution-
8:
Stack
9:
end for
9:
o' + +T((s)
end for
10:
if IsOption(oo' a.) then
output D
11:
(To,8)
StepAndTrrinn(E, R, m.,
12:
else if -Haslmplementtionn( D) then
Sticky Mittens Guiced RL
Hierarchical RL
13:
+ Teleport(o
Opt Templkess zt Level
14:
else
15:
Push o' to ExecutionStack end
Option Templates zt Level
17:
end while
18: Exit loop if AvgReward 2
Enviro smmntt Actions
19:
end for
Figure 2: Visualization of learning with op- tion templates.
output To
Intuitively, at each iteration, it "flattens" the option templates used on the previous level based on the ones available the current level At each level we first design reward functions R, for each option template ag A (Line 4); this reward function is later used to learn policy r implementing ag. For the base case = 1, we take the reward function R. to be the reward function for the original environment &; thus, learning the policy for the option-template level amounts to accomplishing the goals of & using the (unique) option template ag E A For subsequent iterations 1, Ry encodes the goal of achieving the termination condition ag. based on its termination condition (Line 4).
this
Given Rg. algorithm learns policy that maximizes Rg using the options available the rent level (Line 6). For / 1, we assume there is single option template A = {ag]; the corresponding policy σ aims to achieve the goal the original environment ε, so we refer to resulting option o (ag, Tz)) the top-level option. For | > 1, T implements an option template ag - at level 1 1 using the option templates available at the current level I. Importantly, this process leverages policies learned far generate initial states from which to learn xy. Once we have leamed T. we add the resulting option 0 (ag, Tz) to D.
Recall that the termination condition for option starting at state so T( k*) = (s
k*). The goal of training T is to satisfy this condition with high probability. Hence, when training policy πă for option ag. we choose the reward function Rg be F., and additionally restrict the length of each episode to E. time-steps.
Learning option policies. Next, describe LcamOptionnollccy which Algorithm 2 uses to learn a policy for option template ag. For simplicity, we denote the current environment by E, the current reward function by R. the option template by a.. and the target policy by no The goal of m. is to achieve the termination condition for a (I,T, P) from initial states & E T.

---

get[gem]
gil
upe
hure
(a) Visualization of craft envi- romment for the gef gem task: The agent (red square) uses the axe inventory to break the stone and retrieve the gem.
(b) Visualization of fetch and stack environment: Each block has to be placed at the location represented by the color-coded sphere.
(c) Visualization the Google football environment: The agents learn to use 11 players of the left team and score goals.
Figure 3: Craft, Fetch and GFootball environments.
One challenge is sampling an initial state s E from which train To. To do so, this subroutine leverages access to the previously learned options D; it uses options in D until it arrives at a state s E where o, is called. In more detail, it samples s by executing the top-level option in D until it reaches state where a is called. In general, executing an option either relies on executing its policy To in if it exists, or by executing its "teleport" functionality. Our algorithm keeps track of the execution of options for which T, exists using stack, which is initialized with the top-level option (Line 3-5). The structure is visualized in Figure Then, while executing the current option it obtains the next option o' to execute, which is processed in one of three ways:
If of is the option for a, (i.e., it has matching initial state, termination condition, and current policy a. checked by IsOption on Line 10), then takes steps to train To (Line 1 I)); as described below.
If o' does not have an implementation (checked by Haslmplementation on Line 12), then it eee. we sample 8 ~ (Line 13).
Otherwise, o' has an implementation, so pushes of onto the stack (Line 15).
Finally, once we are at a state where a, called, the subroutine StepAndTTainn takes control and trains To using a standard reinforcement learning algorithm-i.c. by collecting rewards from s and taking a policy gradient step once reaches terminal state or hits the timeout of k- steps. Then, the current episode continues from the state reached. Oncc the current episode reaches final state, then the algorithm continues to the next episode. This process continues until the average reward of To exceeds a threshold 1 - 6, until a maximum number of episodes L is reached.
4 Experiments
4.1 Experiments on Planning tasks in the Craft Environment
Description environment: The craft environment [14, 15] is a 2D world based on the Minecraft game, where agent has to complete various hierarchical tasks with sparse rewards. The envi- ronment is represented by 12 x 12 grid with cells containing raw resources (e.g.. WOOD, IRON), crafting areas (e.g., WORKBENCH), obstacles (e.g., STONE, WATER) and valuable items (GOLD or GEM). There are four move actions (up, down, left, right) and special USE action. To grab an item, the agent moves to a neighboring cell and applies the USE action. Table shows the hicrarchical arrangement of the get gem task; the details for get gold are in the Appendix.
Option template at each environment: Our option templates (in Table 1) capture the hierarchy of task-dependencies in the environment. For example, it is easier to get GEM when the agent has access to AXE (see Table 1). Thus, the topmost level the agent primitive actions, use to break stone and get GEM to realize the importance of an AXE.
"ask" for AXE, and via
Implementation: In each task of each learning level, we use vanilla actor-critic algorithm [16] with a network with one hidden layer of 100 neurons for both the actor and critic. The input to the network is feature vector consisting of one-hot encodings of the items in each cell in 5 x grid

---

Task
Policy Sketch
Task
Learning level
option templates
get wood
get iron
get gem
1
(give primitive actions)
make stick
gel wood - use anvil
make axe
2
(give stick, give iron, primitive actions]
make axe
make stick + get iron
make stick
(give wood, primitive actions]
-> use workbench
get iron
3
(primitive actions only)
get gem
make axe -> break stone
get wood
4
(primitive actions only)
Table I: Get gem hierarchical task: [LEFT] policy sketches [14] [RIGHT] option templates for each task in the hierarchy. The order of the rows represent the learning order in the two alternatives.
get gam
make stick
make ooe
1.0
0.8
avg rewwrdd
0.0
044
aug
0.4
0.2
option-temp temp
templetes
option
option value heeaaion
Reration
option value Iteration
0.0
0.0
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
calsodes
episodes
episades
Figure 4: Average reward vs episodes for solving each hierarchical sub-task. We compare option templates (ours) and option-value iteration (baseline) for the task get gem in craft environment.
around the agent along with one-hot encoding of its inventory. At the topmost level, the agent is given reward of if and only if GOLD or GEM are obtained. After training, obtain actor network for individual tasks at different learning levels.
We implement option value iteration [2] as a compari- son with standard hierarchical learning. In option value iteration, we learn options opttom-up, using options learnt at lower level to accomplish sub-tasks of options at the level above. That is the agent learns to implement lower level sub-tasks first before it learns how to use them. We plot av- erage reward as a function of episodes for each sub-task of gef gem in Figure 4; for get gold the Figure in Appendix A.2. Figure shows that option templates obtain higher average rewards than option-value iteration at all levels (detailed discussion in Appendix A.2).
Comparison with option-value iteration:
Comparison with curriculum learning [14]: Additionally, we compare our method with the curriculum learning algorithm employed in An-
Task
Episodes
Curriculum learning
Option templates
dreas et al. [14] and observe a 100 fold de-
[14]
crease in the number of episodes required to train an agent for the get gem and get gold tasks (see Table for get gem and Appendix for ger gold). For the proposed method, we report the total episodes it requires for average reward to stay consistently above 0.8. Averaged over ten runs for different random seeds. The calcula- tion of the total episodes for each task, includes all the sub-tasks in its hierarchy. For instance, the episodes for get gem completion via option templates include episodes required for complet- ing make axe, make stick, get wood and get iron. The latter two tasks are straightforward, and not included in the table.
get gem make axe
>> 3 x 10
12826.0 +: 2613.0
>> 2.7 10
11283.0 +: 2255.0
make stick
>> 1.3 10"
5026.0 +:: 2231.0
Table 2: Comparison of total episodes (and stan- dard deviations over ten random seeds) train an agent to solve the get gem task via option tem- plates and curriculum learning [14].
4.2 Experiments on Manipulation Tasks in the Fetch and Stack Environment
Description environment: This continuous action space environment, introduced in Lanier [17], consists of robotic arm and a platform with blocks placed on it in random positions; see Figure 3b. The goal for the arm to move, lift, and release the colored blocks in the location and stacking order specified by the corresponding color-coded spheres. The action consists of torques for 3 degrees-of- freedom actuation, and an on-off control input opens and closes the gripper. The environment offers sparse reward for each block placed at the goal location.

---

Task(s)
lovel
Option templates
Task(s)
level
Option templates
Fetch & Stack N blocks
(Place block i at its goal (ocation]=_1.....
Win game
1
(Attack and score goals, Defend}
Place block at its goal location
(Reach black i, Pick black & reach goal, Release block i & lift, Do nothing)i=1,.....
Attack score goals
and
2
{Maintain ball pos- session, Charge to the opponent's goal, and Shoot)
Table 3: Option templates for fetch & stack.
Table 4: Option templates for gfootball.
1.0
Three blocks
1.0
Four blocks
0.8
medium
hard
avg goal aifference
0.6
7.5
0.6
0.4
5.0
cprion templetee mwsl
a3tion tnepleane iuse
2.5
0.2
cpikasiorplalts lond
uplinnlaarpaaii knd
Anenine
0.0
Sanklina
0.0
0.0
0.5
1.0
1.5
0.0
0.5
1.0 Lime steps
1.5
2.0
Tamplates
Baselino 4.2M
Impala 200M
line sleps
1e6
1e6
2:1M
105000
(a) Average reward vs timesteps for option templates and the for option templates and the baseline.
(b) Average reward vs timesteps
(c) Comparison of time steps and corre- sponding average goal difference.
baseline.
Figure 5: Results on the Fetch & Stack and GFootball environments.
Learning levels and option templates: We introduce two learning levels with tasks and option templates described in Table 3. In this environment, the primitive actions are not continuous-space actions. Instead, at the lowest level (level 2), we expose the agent to options which are implemented with proportional feedback controllers [18]. Such simple primitives have the ability improve ex- ploration and can be easily transferred among different learning scenarios. Further, the hicrarchical structure improves the agent's learning speed as compared to a case where the agent has learn to use the options at the bottom level (2) directly (see Appendix A.3 for more details).
Implementation: We use standard DQN [12] for each level with four hidden layers of 300 neurons each. The inputs to the agent are the 3D coordinates of the different blocks, their goals, and the states of the gripper arm. The agent has 150 and 200 steps for stacking three and four blocks, respectively, in the correct order. We also supply demonstrations to speed up learning.
We consider a baseline that directly exposes the agent to all option templates, at level 2 of the hierarchy. Figure 3b shows the average rewards as a function of episodes. As can be seen, the baseline only recieves one- thirthird reward since it only succeeds in pushing the bottom block to its location. does not learn to stack even after twice the number of learning steps as our method. This demonstrates the challenge in learning longer duration tasks without a higher level guiding policy.
Baseline:
Comparison with Learning with Demonstrations [1]: In learning with demonstrations [1], the authors use demonstration traces, by combining behavioral cloning along with a Q-fitter mechanism to speed-up the learning. This allows them to evade an expensive exploration phase the early stages of the learning. This method takes upwards of 3.5 108 and 8 x 10a timesteps to learn stacking of three and four blocks respectively which three orders of magnitude larger than our average learning time of 4.5 x 10 and 6 x 10 timesteps respectively (from 5 random seeds).
4.3 Experiments on Multi- Tasks in the GFootball Environment
Description of environment: The Google football (gfootball) environment [19] is an vs. 11 game of soccer. The opponents are controlled by an inbuilt game engine with game-play at three levels of difficulty (easy, medium, and hard). An RL agent can control up to 11 players on the left team. The agent provides each player with one of 19 actions such as direction to move (e.g. top, top-right, bottom-left, etc.), type of pass (short, long or high), shoot, or toggles for dribbling and sprinting. We provide more details in Appendix A.4.
Learning levels and option templates: We create two learning levels with option templates as given in Table 4. The primitive actions in this environment are defend, maintain ball possession, charge to the opponent's goal and shoot which are options implemented with simple planers and open-loop controllers. Similarly, we implement option templates at level (see Appendix A.4).
7

---

Implementation: We train a standard DQN [12] for each level. The input to every network is 139 element vector consisting of left and right team states, ball state, score, and one-hot encodings of ball ownership and game mode. For level 1, for all levels of difficulty, we use a network with 2 hidden layers of 500 neurons each. For level 2, we use network with 5, 6, and 7 hidden layers for easy, medium and hard respectively where each hidden layer has 500 neurons.
Baseline: We consider a baseline where the agent is directly exposed all the primitive options. From Figure 5c, we find that the baseline is unable to compete with option template learning even after twice the number of steps of our method (numerical values of Figure 5e are in Appendix A.4). The eтror bars represent standard deviation over 5 random seeds.
Comparison with IMPALA and DQN from Kurach et al. [19]: We compare our agent with the agents in Kurach et al. [19]. As can he seen in Figure 5c, option template learning can achieve similar (in easy) or better (in medium and hard) performance than the IMAPALA and DQN agents in two orders of magnitude fewer steps.
5
Related Work
We present detailed discussion on related work in Appendix A.1 and summarize closely relevant literature here.
On expert help via primitives or skills: Various recent works utilize expert help in the form of
primitives or skills [4, 5, 6, 7, 8, 9, 20]. This takes place via parameterized action spaces [4], stitching together independent task schemas (or skills) [5. 6, 9, 20] or learning parameters of action primitves [7, 9, 8]. In all of these cases, learning takes place within the traditional hierarchical framework, i.e., bottom-up (see Figure 2), where a sub-task is learnt before the policy that uses it. Our method proposes a framework to learn top-down instead with aage-improvements in sample- efficiency over traditional bottom-up learning. With minimal changes in the implementation of their primitives and skills, the above diverse strategies can also benefit with shorter learning times by utilizing our framework to learn top-down.
On expert help with humans-in-the-loog Other approaches expert intervention include explicit help with gamans-in-the-lowe required throughout training [21, 22]. In contrast, in our method, human effort is only required at the beginning to create the option template hierarchies.
On exploration in high-ding -ige-dimensional tasks: An alternative approach to counter the need for ex- ploration in high dimensional tasks is through the use of demonstrations [1] for long horizon plan- ning. The intuition is that introducing a degree of behavioral cloning of expert demonstrations helps reduce the amount of exploration the agent has perform. A different approach known Hind- sight Experience Replay (HER) [23] incorporates the goal information into the state, using failed terminal state an alternative goal to reward the transitions leading it. These methods are or- thogonal to our approach. By themselves, they are unable to achieve the large degree of reduction in sample-complexity seen with our framework. Yet, alongside expert help, they further improve efnlie-ffficienyy as seen in our experiments.
Limitations and Conclusion
We have proposed approach that incorporates option templates into reinforcement learning. Our experiments show that this strategy can drastically reduce sample complexity by implementing tele- portation. This can be a potential limitation. In simulation, implementing teleportation is typically straightforward. Policies may be trained in simulation before being deployed on a real robot. For real-world environments, there are several strategies for implementing teleportation. First, for chal- lenging skills such as grasping an object, teleportation can be implemented via temporary crutch that simplifies the skill. For example, in the "sticky-mine miittens experiments (a key motivation for our work); the analog for a grasping robot would be to attach velcro to its grippers and to the objects to make them easy to pick up. Second, teleportation can be implemented via handcrafted policy that eventually achieves the goal but possibly in suboptimal way. For instance, teleportation in the Fetch and GFootball environments are implemented using handcrafted policies. While these policies used to train the RL policy, the RL policy eventually significantly outperforms them.
8

---

Acknowledgements
This work was supported in part by ARO W911NF- 20-N 20-1-0080 and AFRL and DARPA FA8750- 18-C-0090. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Air Force Research Labora- tory (AFRL), the Army Research Office (ARO), the Defense Advanced Research Projects Agency (DARPA), the Department of Defense, or the United States Government.
References
[1] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. CoRR, abs/ 10089, 2017. URL http: // ceexiv.org//bii /ttb////79... 10089.
[2] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning, Artificial Intelligence, 12((1):181-211 1999. ISSN 0004-3702. wwt.hh//woo..orrro00000000000000 sciencedirect.
URL https://www.
//doi.org/10. 111 1/desc. 13036. URL. httpa///.cc.cccaaa.aaaooo//////. 111 1111/desc. .ttssc..3333 13036.
[3] L. van den Berg and G. Gredebäck. The sticky mittens paradigm: critical appraisal of current results and explanations. Developmental Science, (e65)ee33066
September 2021. doichttps:
[4] M. Hausknecht and P. Stone. Deep reinforcement learning in parameterized action space. In Proceedings of the International Conference on Learning Representations (ICLR), May 2016.
[5] MacAlpine, M. Depinet, J. Liang, and P. Stone. UT austin villa: Robocup 2014 3d simula- tion league competition and technical challenge champions. In R. A. C. Bianchi, H. L. Akin, Ramamoorthy, and Sugiura, editors, RoboCup 2014: Robot World Cup XVIII [papers from the 18th Annual RoboCup International Symposium, João Pessoa, Brazil, July 15, vol- ume 8992 of Lecture Notes in Computer Science, pages 33-46. Springer, 2014. doi:10.1007/ 186111815-3_..
978-3-319- 8615-3 3. URL http:::tt///// http::///oooor//0//// org/10.
[6] A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, S. Mourad, D. Silver, D. Precup, et al. The option keyhoard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[7] M. Dalal, Pathak, and R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. CoRR, 1t./22100113000 2021. URL https:://axxiv..rg abs/2110. 15360.
[8] S. Nasiriany, H. Liu, and Y. Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. arXiv preprint arXiv:2110.03655, 2021.
[9] R. Chitnis, S. Tulsiani, S. Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 1149-1155. IEEE, 2020.
[10] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[11] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning. Dis- crete Event Dynamic Systems Volume 13, 13:2003, 2003.
[12] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. 2013.
[13] G. Hacohen and D. Weinshall. On the power of curriculum learning in training deep networks. CoRR,
2019. URL htttp:://arxiv.or//bbbste https/1100.0..66...1
[14] Andreas, D. Klein, and Levine. Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning, pages 166-175. PMLR, 2017.

---

[15] F. Behbahani. Craft environment.
env, 2018.
[16] V. Konda and J. Tsitsiklis. Actor-critic algorithms. Society for Industrial and Applied Mathe- maticy, 42, 04 2001.
[17] J. B. Lanier. Curiosity-driven multi-criteria hindsight experience replay. University of Cali- fornia, Irvine, 2019.
[18] K. J. Astrom and R. M. Murray. Feedback Systems: An Introduction for Scientists and Engi- neers. Princeton University Press, USA, 2008. ISBN 0691 135762.
[19] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vin- cent, M. Michalski, O. Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint 1rXXv:1907.111000 2019.
[20] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[21] Q. Li, 7. Peng, and B. Zhou. Efficient learning of safe driving policy via human-ai copilot optimization. arXiv preprint arXiv:2202. 2022.
[22] J. Spencer. S. Choudhury, M. Barnes, M. Schmittle, M. Chiang. P. Ramadge, and S. Srinivasa. Expert intervention learning. Autonomous Robots, 46(1):99-113 2022.
[23] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abheel, and W. Zaremba. Hindsight experience replay. CoRR, abs/1707.01195, 2017. URL attp://arxiv.orgg http:///gg/abbb/////// .11495.
[24] C. Watkins and P. Dayan. Technical note: Q-learning. Machine Learning, 8:279-292, 05 1992.
doi: 100:100.0007/FF09922888
[25] A. Akhmetzyanov. R. Yagfarov, S. Gafurov, M. Ostamin, and A. Klimchik. Continuous control in deep reinforcement learning with direct policy derivation from q network. In T. Ahram, R. Taiar, V. Gremeaux and K. Aminian, editors, Human Interaction, Emerging Tech- nologies and Future Applications II, pages 168-174, Cham, 2020. Springer International Pub- lishing.
[26] F.H. Hsu. Behind Deep Blue: Building the Computer That Defeated the World Chess Cham- pion. Princeton University Press, USA, 2002. ISBN 0691090655
[27] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre. G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panncershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-489) 01 2016. doi: 10. 1038/aatuee16611.
[28] A. Levy, R. P. Jr., and K. Saenko. Hierarchical actor-critic. CoRR, abs/ abs/1712.00948,, 2017. URL http: //axxi/ org abs/ 111//111110/ 00948.
[29] C. Yang, K. Yuan, Q. Zhu, W. Yu, and Z. Li. Multi-expert learning of adaptive legged locomo- tion. CoRR, 2tss/2212.08811 2020. URL https: org/ 2b12 00001....88....
[30] L.. Marzari, A. Pore, D. Dall' Alba, G. Aragon-Camarasa, A. Farinelli, and P. Fiorini. Towards hierarchical task decomposition using deep reinforcement learning for pick and place subtasks. CoRR, abs/2102.04022. 2021. URL hattps://arxae httpp//////eed.ooo////////// wttg/a//////////// 04022.
[31] J. Winder, S. Milani, M. Landen, E. Oh, S. Parr, S. Squire, M. desJardins, and C. Matuszek. Planning with abstract learned models while learning transferable subtasks. Proceedings of the AAAI Conference on Artificial Intelligence, 34(06);9992-10000, Apr. 2020. doi: 1ett1...669/// 1609/aaai. v34i06.6555. URL https:tt////// //ops:://ss..... aaai www.aaaaaaaaa..aa.a..a...a.aaaaa Beaadex.ppp/AAAee
[32] K. Jothimurugan, Bastani, and R. Alur. Abstract value iteration for hierarchical reinforce- ment learning. CoRR, abs/2010.156333 2020. URL https: /// //trx//.vvvv///////////////aaaa/ aarg/bbs/2110.. 15638.
10

---

[33] D. Abel, N. Umbanhowar, K. Khetarpal. D. Arumugam. D. Precup, and M. Littman. Value preserving actiin abstractions. In S. Chiappa and R. Calandra, editors. Proceedings of the Twenty Third International Conference of Proceedings of Machine Learning Research, pages 1639-1650, PMLR, 26-28 Aug 2020. URL. https: http///pcoceen.....////// mml. press/ catass/vvv/// v108/abel20e abel20a html.
Artificial Intelligence and Statistics, volume 108
[34] Y. Yang, J. Inala, O. Bastani, Y. Pu, A. Solar-Lezama, and M. Rinard. Program synthesis guided reinforcement learning, 02 2021.
[35] N. Gopalan, M. desJardins, M. L. Littman, J. MacGlashan. S. Squire, S. Tellex, J. Winder, and L. L. S. Wong. Planning with abstract markov decision processes. In ICAPS, 2017.
[36] K. Jothimurugan, Bansal, O. Bastani, and R. Alur. Compositional reinforcement learning from logical specifications. CoRR, 0oss/100.11306,, 2021. URL atttps://arxil org/oogg/goo /bss/
2106. 13906.
[37] C. Garrett, R. Chitnis, Holladay, Kim, T. Silver, P. Kaelbling, and Lozano-Péree Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[38] A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv: 17...11122.0.. 2017.
[39] X. Yang, Z. Ji, J. Wu, Y.-.. Lai, C. Wei, G. Liu, and R. Setchi. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 2021.


Reviews:
[{"review_id": "nCWmoEnsUd", "reviewer": "Reviewer_dp9v", "summary": "The paper provides an interesting perspective to existing Option-based hierarchical RL methods by proposing the idea of Option Templates: It introduces a novel framework to learn option templates, which enables the agent to have expert intervention as high-level option of the task when they are unsure how to continue the action from the state. Specifically, it could transit from one state to another immediately, where the transition represent the execution of an unimplemented skill primitive. It lays out different levels of option templates, and eventually learns the lower level actions in the environment. This could improve the policy performance and sample efficiency of RL and the paper shows the effectiveness of this method on tasks including robot manipulation and soccer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation for using Option Template from the idea of Sticky Mitten is interesting and strong. It provides a nice analogy between human and agent learning. \n\n- It is a very practical and interesting idea to have expert interventions as high-level instructions which largely reduces the exploration burden and provide learning signals to the agent. \n\n- The explanation on option template method and idea is very clear. There are clear comparison between the option template \n\n- The limitation section provides true and to-the-point discussion on the sources of teleportation, which should be considered in future work. \n\n- There are extensive discussions on how this approach compared with prior methods like option-value iteration and curriculum learning. \n\nWeaknesses:\n\n- The main concern of the paper is how well it could be applied to more complex robotics tasks that are 1) have longer-horizon which motivates the use of hierarchical RL 2) where the skills are not clearly defined. The robot experiments in the paper involves simple moving of objects, and does not reflect how hierarchical RL could help with more complex tasks.\n\n- There are no real robot experiments and no discussions of how well this could be scaled to real-world setting where there are more intricate manipulation of objects and more complex skills.\n\n- For more complex, long-horizon robotics tasks, how could the skills be extracted and the options to be formed in the first place? Does it assume that it already have skill segmentation done? The learning levels here seems a bit contrived and manually designed for each individual task. \n\n- While comparing the Option Template method to previous hierarchical RL method is sound, the idea of Option Template have the use of expert intervention, which relates to the line of work on learning with expert interventions. While this is a different formation and a different framing, there should be references made to these methods, listed a few in below:\n1. Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization [Li 2022]\n\n    https://arxiv.org/abs/2202.10341\n2. Expert Intervention Learning [Spencer 2020] \n\n    https://link.springer.com/article/10.1007/s10514-021-10006-9\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the Option Template is a very interesting idea that complements the existing option framework and addresses the limitations in previous hierarchical reinforcement learning methods. While it is in theory an interesting approach, the experiments in the paper haven't been able to show the effectiveness of the approach in complex robot manipulation tasks. The fact that the tasks are mainly in grid world and the robotics tasks are simply pushing and placing objects are a bit limited. The authors need to show more evidence on how this method could be used on more complex tasks like robot manipulation etc. Also, there should be more evidence of how this could be scaled to real-world setting with real robots.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "As discussed in the Weaknesses section above:\n\n1. There should be more discussion on how the expert intervention can be obtained, and the cases where it is unable to obtain a high-performing policy as the expert. \n\n2. There should be experiments on more complex tasks like more challenging manipulation of objects (e.g. grasping) instead of pushing and moving objects. Also the skills should be more realistic than simply placing one block at a certain location. If there are limitations to doing such experiments, they should be addressed in the limitation section.\n\n3. More reference and comparison need to be made to intervention-based / interactive / human-in-the-loop RL\n"}}, {"review_id": "Vxzol_Gn-1p", "reviewer": "Reviewer_zrnS", "summary": "The main contribution of this paper is a sample-efficient RL approach for learning policies for long-horizon tasks, by making use of multiple levels of abstraction, each of which is associated with a set of option templates. An option template defines an option, in terms of initial and final states, but not how to implement it. All option templates are handcrafted at the start of training, so the agent can execute an option template despite not knowing how to implement it yet. During training, the agent learns how to implement each option template in terms of the option templates at the next lower level of abstraction.\n\nThe paper evaluates this approach in three domains, all in simulation: craft (a grid world based on Minecraft), stacking with a robot arm, and 11 vs. 11 football. This top-down approach of learning options reaches higher performance than learning the options first and then combining them. It is orders of magnitude more sample-efficient than learning without options.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n* The proposed approach demonstrates orders of magnitude improvement in sample-efficiency, compared to state-of-the-art RL approaches that do not use options.\n* The approach is evaluated in complex continuous control domains.\n* Includes an interesting comparison that shows this approach of learning top-down with option templates works better than learning bottom-up (i.e., learning the options first and then connecting them together). The hypothesis is that this is because the learned options typically do not perform perfectly, whereas option templates do, which makes it easier to learn how to correctly connect options together when using the latter.\n* Clear explanation of the background, in terms of options and semi-MDPs.\n\n### Weaknesses:\n* The approach requires defining the levels of abstraction and the set of option templates in each. The effectiveness of the approach presumably depends greatly on how well the problem is decomposed into these levels of abstraction and option templates.\n* The implementation of option templates can be nontrivial, as is acknowledged in the Limitations section.\n* The description of the method could be more clear. Including a more detailed Figure 2 would help, perhaps with more concrete examples of what the options could be (e.g., from the Minecraft task), and showing how over the course of training, each of the options is learned in terms of the next level of options. Also, in the implementation of the approach, how does one restrict the policy to pick from the set of options that are valid in a particular state?\n* There is no comparison against option value iteration in the manipulation and football environments. Is it because option value iteration would take too long to learn for these tasks?\n* No evaluation on real robots. This would be a stand-out paper for CoRL if the evaluation included learning a long-horizon task with a real robot.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an exciting new approach of using options in a top-down manner, that could be highly effective at learning long-horizon tasks efficiently on real robots. This approach does require significantly more effort to apply than end-to-end RL, since the abstraction levels and sets of option templates need to be defined, as well as a handcrafted policy for each option template. But this could also be an advantage, in that it enables combining low-level controllers with higher-level learned policies. With a few improvements in clarity and evaluation, I think this would be a strong paper and very relevant for the CoRL community.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "### Suggestions / questions:\n* Points 3, 4, and 5 from Weaknesses section above\n* I'm not sure what Figure 2 is trying to communicate. The graph structure is confusing to me, since each option is executed by a _sequence_ of options at the next level, so ordering matters. I think it would be useful to make Figure 2 more concrete, as suggested above.\n* End of Section 3: What happens if the maximum number of episodes L is reached but the option has not been learned properly yet?\n* It is worth making space for Related Work in the main paper, rather than leaving it to the Appendix. I recommend mentioning Task and Motion Planning (TAMP) in Related Work, as an alternate hierarchical approach for tackling long-horizon problems in robotics. e.g. Garrett et al. Integrated Task and Motion Planning. https://arxiv.org/abs/2010.01083\n\n### Typos / wording:\n* Page 1, line 20: \"Designed to achieve intermediate subgoals.\" - not a complete sentence\n* in Algorithm 2, in line 6 shouldn't the initial state be sampled from $\\mathcal{I}$, rather than $I$? (i.e., the initial states of the semi-MDP rather than of the option)\n* Page 5, line 173: Missing a space before \"Once\"\n* Page 8, line 255: \"planers\" → \"planners\""}}, {"review_id": "9a30obQxhac", "reviewer": "Reviewer_XdN5", "summary": "The paper \"Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates\" introduces a hierarchical reinforcement learning approach that divides long time-scales in multiple \"option templates\", i.e. a group of initial and terminal states without option-specific, low-level policies. The low-level control policies then only come into play at the very end, to successively implement the previously found option templates. The approach is evaluated on multiple reinforcement learning benchmarks in simulation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n\t* In general, the research topic of the submitted manuscript is of high practical relevance and hence of interest to the community of CoRL.\n\t* It is well motivated and the background is well presented.\n\nWeaknesses:\n\n\tUnfortunately, the manuscript in its current form comes with a couple of issues.\n\n\t* Clarity: The explanation of the approach is very confusing and not easy to grasp for external readers.\n\t* Technical Novely: The general idea seems very similar to previous work, e.g. [1]. In addition to the lack of clarity, the contributions to the field remain vague and seem minor.\n\t* Evaluation: Even though the shown results in the given benchmarks are somewhat convincing, they don't really highlight the special ingredients of the proposed method. \n\n[1] Learning Multi-Level Hierarchies with Hindsight, Levy et al., ICLR 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The current manuscript neither explains the concept very well, nor does it really convince regarding real-life impact. Despite the general idea being very suitable for CoRL, this paper is no candidate for acceptance in its current form.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Clarity: Some of the mathematical definitions are very vague (definition of environment at learning level l, the definition of the reward functions, the satisfaction of the termination condition). In addition, some things remain undefined, such as \"teleporting\" or how exactly the policies are optimized. In addition, despite having a section \"Learning option policies\", from the experiments it seems that some option-policies are given (Tables 1, 3 and 4). Furthermore, the \"option templates\" shown in these tables correspond to policies rather than \"option templates\" (which should only be defined over sets of states as defined in the paper). Lastly, it remains unclear, how exactly these option templates are defined. In the paper, it says that \"For l = 1, we assume there is a single option template A0\", however, what precisely is this option template and how is the division over time for subsequent layers? To summarize, the general concept is clear, but the detailed explanations are really confusing and even seem contradictory to some degree. The respective sections really have to be improved upon clarity and precise definitions. The rather involved algorithms do not bring any additional benefit in this case. Maybe the space could be used to leave the respective sections more space.\n\n* Technical Novely: The additional benefit of the proposed approach, especially compared to [1], has to be put more in focus.\n\n* Evaluation: The proposed method claims to yield better sample complexity due to the limited horizon of each subtask by leveraging option templates in a hierarchy. However, the experiments do not really analyze the effect of this hierarchy. How does granularity (i.e. the number of layers) influence performance, which kind of division does bring the most benefit? As is, the experiments only show results for two levels (with the first being the normal task, it seems) which is not really convincing regarding the general idea.\n\n[1] Learning Multi-Level Hierarchies with Hindsight, Levy et al., ICLR 2019\n\n-------------------------------------\n\nPost-rebuttal:\nWhile I appreciate the rebuttal, the paragraph about the effect of the hierarchy is not really convincing. One key aspect of the approach, splitting a task into multiple subtasks to account for long horizons, is not well analyzed, so the experimental evaluation is really lacking some general insights beyond the chosen simulated benchmarks. On the flip side, the results themselves are impressive and hint to at least some real-life applicability -- so it really depends on the scope of CoRL whether this manuscript would be a candidate for acceptance. However, if the camera ready improves upon clarity and presentation, I rather tend to raise my score to WA."}}]

Author rebuttal:

Decision: Accept (poster)