Meta Reinforcement Learning for Fast Adaptation of Hierarchical Policies
Authors: David Kuric
Herke van Hoof
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Hierarchical Reinforcement Learning
Meta-learning
Reinforcement Learning

Abstract: Hierarchical methods have the potential to allow reinforcement learning to scale to larger environments. Decomposing a task into transferable components, however, remains a challenging problem. In this paper, we propose a meta-learning approach for learning such a decomposition within the options framework. We formulate the objective as a bi-level optimization problem in which sub-policies and their terminations should facilitate fast learning on a family of tasks. Once such a set of options is obtained, it can then be used in new tasks where only the sequencing of options needs to be chosen. Our formalism tends to result in options where fewer decisions are needed to solve such new tasks. Experimentally, we show that our method is able to learn transferable components which accelerate learning and performs better than existing methods developed for this setting in the challenging ant maze locomotion task.

Meta Reinforcement Learning for Fast Adaptation of Hierarchical Policies
Anonymous Author(s)
Affiliation Address email
Abstract
1
Hierarchical methods have the potential to allow reinforcement learning to scale to larger environments. Decomposing a task into transferable components, however, remains challenging problem. In this paper, for learning such a decomposition within the options framework. We formulate the objective as bi-level optimization problem which sub-policies terminations should facilitate fast learning on a family of tasks. Once such a set of options is obtained, it can then be used in new tasks where only the sequencing of options needs to be chosen. Our formalism tends to result in options where fewer decisions are needed solve such new tasks. Experimentally, we show that our method is able to learn transferable components which accelcrate learning and performs better than existing methods developed for this setting in the challenging ant maze locomotion task.
propose meta-learning approach
4
5
their
8
9
10
Introduction
13
Current state of the art model-free reinforcement learning methods were successfully applied to many challenging tasks [33, 41J. However, one of the main drawbacks of these methods is their 16 data-inefficiency and inability to generalize to other related tasks [12]. is often impossible use
17 the agent trained on one task to solve another related task (53) or even to use as starting point for
18 training because trained models become increasingly exploitative and thus are unable to explore in a new task. In such cases, we have gather new data and train new model which is time-consuming.
One way to mitigate this problem is by learning a policy with reusable modules which can be used in multiple tasks. For example, if assume that related tasks contain shared sub-tasks (i.e. tasks come from the same family have hicrarchical structure), we can speed up the adaptation new tasks by learning sub-poli aub-policies that solve these sub-tasks. This is because solutions to new tasks can be created hy combining known solutions to sub-tasks during adaptation. The idea of learning reusable skills in multiple environments, which dates back to at least 1995 [48], thoroughly explored within the options framework [17, 24, 25, 29, 36, 46].
23
In this framework, policy is composed of options (modules that encapsulate sub-policies), and a high-level policy that chooses among them. Options have their own termination function, and new option is only initiated when the earlier option terminates. Therefore, options define temporally extended behaviors that can form solutions to sub-tasks. Despite extensive research in this area, there is not yet consensus answers to many important questions about options: What are good options? How can find them? When should termination occur? How many options should one use? In this work our aim will to find options that allow fast adaptation to tasks from same family, We use this single principle to address all of these questions except for the number of options which we consider a hyperparameter.
34
Submitted to 35th Conference Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

36 Learning high-level policy, sub-policies and terminations at the same time is a challenging task, Recent prior work on options proposed a way to learn both, including the terminations, in an end to end manner with policy gradient methods [4, 44]. However, despite achieving good performance in single-task settings, these methods often produce options which may not he useful for transfer [21. 22]. This because such options are not explicitly trained for multi-task setting and can often terminate too often or not at all [21, 22].
To overcome this issue, Frans et al. [17] proposed to use such method a multi-task setting with 43 options that have predefined length and are optimized for performance after adaptation of the high- level policy. Although options that terminate after certain amount of steps simplify the problem and work well in some settings [17, 25J. manually setting this important hyperparameter requires
46 prior knowledge and might not work cases where options need to have different lengths [23]. For
47 example, task where this approach would not be preferable could be driving because some driving sub-tasks, such as driving on a highway, are much longer than others, such as driving out from one intersection to another in city. Consequently, capturing the length of both sub-tasks with a single hyperparameter [17] or range hyperparameters [25] can become difficult or even impossible. In 51 such cases, learned terminations are preferable.
this paper, we propose method for learning options that allows for fast adaptation multiple tasks. 53 We formalize this notion using recent ideas from gradient based meta-learning [14]. Rather than using options with fixed length (17), our algorithm learns both sub-policies and when to terminate options using single meta-learning objective. We hypothesize that this objective implicitly encourages options to terminate in way that yields reusable components. In experiments, we demonstrate the benefits of our approach in simple Taxi domain as well as in complex Mujoco [49] Ant Maze domain [17].
57
58
59
Related Work
80 Since our work builds on insights from both hierarchical reinforcement learning and meta-learning, we present related work in both domains separately, in subsections 2.1 and 2.2 respectively.
62
2.1 Hierarchical Reinforcement Learning
63 One of the aims of hierarchical reinforcement learning is to decompose a complex task or policy into simpler units. Popular approaches include learning diverse set of skills [11] or utilizing the idea of Feudal Reinforcement Learning [7, 34, 50). Another large collection of related work instead relies on the options framework [46].
Some works options rely so-called bottleneck states that be used as sub-goals [30, 31, 35] whereas others use spectral clustering create options (28). These approaches usually require prior 69 knowledge about the environment which restricts their applicability. Different from aforementioned methods, end-to-end methods such as the ones which rely on the Option-Critic architecture [4, 39] are applicable in more general settings. However, these policy gradient methods can be less efficient than concurrently introduced inference hased end-to-end methods [6, 16, 44] because they only update the 73 option that generated the action whereas inference based methods update options according to their responsibilities for each action.
75 A common problem with end-to-end methods that learn terminations in a single- task setting is option collapse [4]. This causes options to terminate after every action or to never terminate. In such cases the learning of terminations can be facilitated by augmenting the objective with entropy regularization [44] deliberation cost [21], regularizing towards termination prior [23], by
79 optimizing different objective that encourages appropriate terminations [22]. As an alternative, one can also use time-based terminations with fixed [17] or randomized length [25].
2.2 Reinfrreiifo Learning
Meta-reiinforcement learning is concerned with producing models which are able to adapt to novel tasks quickly. This sub-field includes a broad range of work such as unsupervised methods [11, 19], methods that rely on latent variables [20, 38] or methods that learn the update rule of policy
[10, 32, 51].
2

---

86 contrast with the latter, the recent gradient-baset method Model-Agnostic Cearnng (MAML) [14] assumes that policy parameters are updated with gradient descent and instcad aims to learn initial parameter values. MAML was exlended in followup works that only trained part of the network [37, 54] or showed henefits of different architcctural choices such as per-parameter learning rates [3, 26]. Several works also focused on MAML a reinforcement learning setting [2. 27, 45]. In particular, Al-Shedivat et al. [2] and Stadie et al. [45] pointed out a difference between theory and practical inplementation of MAML in automatic differentiation frameworks. This issue was further discussed and resolved in followup works [13, 15, 40].
91
Lastly, there exist methods which do not employ the techniques mentioned above and instead rely on the options framework [5, 17, 23-25, 29, 36, 52] or task-specific policies [47]. These approaches often make different assumptions about the tasks and settings which they are applied. Some require policies that solve each environment [36] whereas others need environment ID [23, 29] or cumulants that properly represents task dynamics [5]. Closest to our work are Meta Learning Shared Hierarchies (MLSH) [17] and Adaptive Skills Adaptive Partitions (ASAP) [29]. ASAP uses a policy gradient method to optimize immediate performance on multiple tasks with known environment but does not use neural networks and does not learn terminations. On the other hand, ML.SH uses hicrarchical structure with predefined options length and problem sctting with unknown environment ID. It optimizes for post-adaptation performance by using two alternating phases that either only update lege-level policy both high-level policy and sub-policies simultaneously. This approach does not use the information from the intermediate adaptation steps when calculating the gradient which can negatively affect its accuracy. Additionally, options with fixed length may be difficult to use in some settings as we' ve'v described in Section 1.
96
97
100
101
102
103
104
105
108
107
Background and Notation
108
109
this section, we will first cover the fundamentals reinforcement learning, and then focus on the options framework and gradient-base based meta-learning.
110
111
3.1
Reinforcement Learning and the Options Framework
112 We will consider environments which are episodic Markov decision processes (MDPs). An MDP Mi is a tuple (S, A. po: P. RR 7) with S being a set of states, A set of actions, Po(*)) probability distribution of initial states, P(s'|s. |s, a) a transition probability function, R(s, ss a) reward function
113
114
115
and 7 discount factor.
118
An agent with stochastic policy interacts with an environment . in the following way. At every timestep the agent receives state of the environment 8 E S and selects action a E according to conditional distribution x x(u (x).. Depending on the current state and the action performed, the environment provides the agent with a new state 81-1 ~ P($ [s, a,) and R(ss, az). This process is repeated until a so-called terminal state is reached. an ordered sequence of all states actions and rewards in single episode
117
118
119
120 a scalar reward
121
We define trajectory (Sg, ag; To. ... sr). Similarly, the history timestep consists of all states and actions preceding ar, h, = (8o, ag, ..., a.). The state value function is defined as Va( = Ex ( = a] where the
122
123
discounted return at timestep is defined as G.(T) = " ((t)
124
125
The agent's objective is to maximize the expected discounted return
Eggrie) [Ga(T))
128
We can maximize the objective with gradient descent by estimating the policy gradient
VoJ z Ep(ro)) EE Ve [s))) using Monte Carlo sampling, where A, is an advantage estimator such the generalized advantage estimator AÇAE [42].
127
128
The options framework is framework for temporal abstraction that consists of options (I", n², x²) and a policy over options T (w]ss. Each option w consists of initiation set, sub-policy and termination function. The initiation set I a set states in which an option can
130
181
132
be selected (initiated) and in our case it is the whole state space (I" = S). A sub-policy x" s), also called low-level policy, is regular policy that acts in the environment. Lastly, the termination
133
134
condition edd (s) is a function that outputs the probability of termination for the option in given state.
3

---

135
3.2
Model-Agnostic Meta-Learning and DíCE
135
Model-Agnostic Meta-Learning (MAML) [14] is meta-learning technique that trains model for maximum post-adaptation performance on a distribution of tasks. The adaptation consists of one or several inner gradient updates. If we consider an estimator fa with parameters 0 and a task- task-specific loss L.M. a supervised learning objective with single inner update can be formalized as shown in Equation 1. order to optimize this objective one only needs to take gradient of this expression.
137
138
140
141
This can be easily achieved with automatic differentiation frameworks by creating a backpropagation
142
graph for the gradient.
min EM (CM. = min
Σ CMi(fo-avoEM M,ri(M)
143
One can similarly use this approach with a reinforcement learning objective. However, the implemen- tation with an automatic differentiation framework differs because simple backpropagation through the computation graph of the gradient produces biased gradients |2, 45J. This is due additional dependency the sampling distribution on parameters that is present the supervised learning objective. To produce correct higher order gradients with automatic differentiation frameworks in a reinforcement learning setting, one can use the objective in Equation as proposed by Farquhar al. [13]. This objective utilizes the DiCB operator ( [15] which can be implemented according Equation 2 where (xx) is stop gradient operator that evaluates to æ but returns zero gradient when differentiated.
144
148
147
148
150
151
[T) Voll
(aa)) exp [log (o(()s -(log [o(ae(s)))]
VolEr~ptr10)
(2)
V J E
Σ Vo
II P==0
((a))
A GAAN (
(3)
Fast Adaptation of Modular Policies
153
154
Much of the extensive research the options framework has focused on an intuition options capturing useful sub-tasks [4, 17, 36, 46]. However, there is no consensus about capturing this intuition in an objective function or the best way to find such options. We propose conceptually simple objective: good set of options allows quick adaptation to many novel tasks. This be formulated using the MAML framework [14]. where we consider a setting which there a distribution tasks p(M) with similar (hicrarchical) structure but different reward or transition functions. Our goal is then to maximize the expected performance after I. adaptation steps of the hicrarchical policy parametrized by 0:
155
158
157
158
159
180
161
max Σ Eytmp(r (04) Mo (r5)) g3-1= = -8 + amn Vo Brs CGM (r+) (4)
Mc+p(MM
Using conventional MAML means adapting large number of parameters which can be disadvanta- geous, as was demonstrated by Zintgraf et al. [54] and Antoniou et al. [3]. By reducing the number parameters that are tuned during the adaptation phase, one can reduce the complexity of the problem during test time at the cost of a less expressive policy. We thus split the parameters into an inner group On and an outer group Oout where inner parameters are updated during the adaptation step and outer parameters are optimized in the outer objective. Note that when using such split, the initialization values of inner parameters may also be meta-lcarned [54]. We experimented with both versions and observed that fixed initialization values performed hetter. Similarly, the per-parameter inner learning rate in [3, 26] can also be meta-learned to allow for more complex inner updates. We used this approach in selting with more complex environment.
163
184
185
188
187
188
169
170
171
172
Our option model has three sets of parameters: those of the high-level policy network 0z sub-policy networks O2r and termination networks 0g. We divide these over the inner and outer parameter group. Since we assume that tasks with common sub-problems can solved using identical options, we consider the sub-policy and termination function parameters as outer parameters. On the other hand, since in each task the decision of the high-level policy to choose options would be different, its parameters constitute the inner group. By keeping sub- policies fixed during the adaptation and
173
174
175
4

---

Algorithm 1 Fast Adaptation of Modular Policies
initialize On. 0g.. Ow, Oin; Oout
set 0.n Og
set Nou repeat
Ow)
Set gradient of outer parameters Jo...
for 1 to N do
set 8/ 8/
sample a task _ p(M) for 1 1
sample & episodes Tis M using Mouc)
fit baseline V2 using data from Tisk
compute A,AA for all Tisk
compute <ag [ha] = Ba,a (a,])))
compute J/1 with A ACAE log (a:/hi (Eqs, 2, 3)
i L then
I'm = 6'n + O'in Vor, J
else
go... go... + Vona J
0 out oost + aut N M8.t until convergence
restricting the update to the high-level policy, we optimize for options that can be used to solve multiple tasks, thereby allowing the overall policy to adapt quickly with the change of high-level policy. This also allows for an expressive policy which can capture different behaviors and reduces the number of parameters and decisions an agent needs to learn and make during test time.
179
180
181
182
Formally, our final objective can be expressed as Equation with the inner update given by Equation 6. The objective is similar to the one used in MLSH [17] with some key differences. Firstly, by hackpropgg through the update step we are able to capture additional information from the
183
184
185
adaptation steps in the gradient and secondly, our objective includes the optimization of termination parameters and thus allows for options with different lengths.
186
max Puite
Σ Morp(M)
( CoM (r)
(5)
8f+1 0g amn Vog E (uppt)((..0000)))))))
(T)
(6)
187
4.1 Algorithm
188
Written in general form the objective leaves some freedom with regard to which policy gradient algorithm is used for gradient calculation. In our work we use the Inferred Option Policy Gradient (IOPG) [44] because it updates all options at the same time based on their responsibilities, i.e., the probability that the option active given the history h, of states and actions far. This can lead better data-efficiency when compared to other methods that only update a single option at time but comes at the cost of increased computation time. Another important design choice is the state value function estimator. In the MAML RL setting the policy constantly changes in every inner update. It is thus difficult to use past trajectories for fitting the value function. We therefore use lincar time-state dependent baseline [9] which works better than more complex baselines with little data and was also used in original MAML. implementation.
189
130
191
192
183
194
195
198
197
138
The resulting algorithm Fast Adaptation Modular Policies (FAMP) is outlined in Algorithm 1. Note that in order to use IOPG with DiCE we replace T(a]s;)) with x <(a, (h,) in Equation 2. An intuition about why this is possible comes from the fact that we can casily formulate a new MDP
199
200
201
M in which states & are histories h, the original MDP without otherwise altering the dynamics.
202
After L inner updates, the gradient of the objective with respect to the outer parameters is calculated.
203
In principle, we would like to optimize for performance after moderate number of gradient updates
204
L such as 10 or 20. However, with more inner updates the resulting gradient of the objective becomes
S

---

Figure 1: Left: Map of taxi environment with special states and an example task. Middle and Right: Visualization of the option usage in this task. Middle part shows states without passenger on board. Right part shows states with passenger. Arrows represent directional actions, pick-up/drop-off is shown as square. Each action is colored according to the active option.
205
noisier due the usage of Monte Carlo estimate in each inner update. Furthermore, the time complexity of gradient computation and sample complexity both scale linearly with the number inner updates. In practice we found range from to update steps to he acceptable. An important benefit of gradient-based meta-lcarning is that even though the model is optimized for performance after I. adaptation steps, it can still be improved after I. updates by performing more steps of gradient descent.
206
207
208
209
210
211
5
Experiments
212
In this section, empirically evaluate method and show its benefits when applied randomly selected tasks within and outside of the training distribution.
213
214
5.1
Taxi
215
In the first set of experiments use modified Taxi environment' [8] displayed in Figure 1. An agent acts as taxi driver who starts in one of the special (colored) locations. The goal of the driver then drive passenger from one of the special locations consists of 60 different tasks with different combination of start, goal and passenger positions. These are always initialized in special states. In 12 out of 60 easier configurations the passenger starts the episode in the The only restriction
218
217
destination. The task family
218
219
220
start, goal and passenger positions in all cases is that passenger's destination must not be the same as his initial position. Each task is an MDP which the agent can use 4 directional actions and two special actions: pick-up/drop-off and no-op. The state space is represented as a onc-hot vector with 72 entrics for every combination of possible taxi location and passenger being on board. Thus the agent does not have any information about the location of the passenger or goal state. Therefore, in order facilitate fast adaptation the (unobservable) passenger and goal locations.
221
222
223
224
226
agent must acquire options that can serve as building blocks for exploration. The reward is 2 for reaching the goal and -0.1 per step otherwise. To speed up training in the early phases, we terminate the episode if it takes longer than 1500 timesteps.
227
228
229
In this experiment, we use tabular representations implemented as a combination of lincar layer and non- oineriie for the policy over options, terminations and sub-policies such that each one-hot state has its own set of parameters. We use 48 training lasks train sub-policies and terminations with our algorithm. Learned terminations and sub-policies are then kept fixed during test time and only the
230
231
232
233
policy over options is updated. Performance is then compared on the remaining 12 test tasks (selected
234
to use combinations of special locations with similar frequency) to MLSH and two baselines. We chose MLSH because is closest hierarchical method designed for setting in which there is no
235
236
extra information about the environment available. This contrast with many other hierarchical
237
[S. 23, 29] and non-hieren non-hierarchical [38, 47] meta-reinforcement learning methods which utilize extra
238
information such as the ID of sampled environment.
239
Similarly to our method, MLSH trained on all training tasks and evaluated with fixed sub- aub-policies.
240
The multi- task baseline is an IOPG algorithu that learns a shared policy (including high-level policy)
Details are included in Appendix B
6

---

-028
-3.5
-1.00
1.35
FAMP MUSH:
Copts
HISI
-216
Eguedos
Figure 2:
Left: Average performance of different algorithms on taxi environment test tasks. Plot shows mean and standard deviation over seeds. Right: Average performance of our method with different hyperparameter values on taxi environments test tasks. Plot shows median and interquartile range over 5 seeds.
241
hy optimizing average return over tasks rather than the meta-learning objective in Equations and 6. After the training, it only adapt its high-level policy on test tasks. We expect this baseline to perform poorly in the long run because it does not optimize for post-update performance. Lastly, the single-task baseline is an IOPG algorithm that learss the test tasks from scratch without any pre-training Therefore, since it does not need to generalize to many tasks and has a policy with enough capacity, we expect that it should eventually outperform other methods after sufficiently long training. However, meta-learned policy with desirable options should find good solution much quicker. To make the single-task baseline strong as possible, we set its learning rate to the highest
242
243
244
245
246
247
248
249
value that was able solve all tasks reliably.
250
Results
251
As shown in Figure 2, our method is able to outperform both MLSH and the multi-task baseline reaching the final performance of -0.315. Furthermore, it also outperforms all other algorithms terms of adaptation speed. We additionally checked whether the single-task baseline eventually overtakes FAMP and found that after more than 200 episodes, its performance stabilizes at a final discounted return value of -0,284. This demonstrates that FAMP can learn sub-soliccee policies and termi- nations that allow for fast adaptation in similar unseen environments the cost of slightly lower asymptotic performance. An example trajectory that was produced by the agent in one of the hardest test tasks is displayed in Figure 1. In this task, the agent is able to combine three options to form an optimal solution. Plots with meta-training curves and learned options are included in Appendix C.
252
253
254
255
256
257
258
259
280
In Figure 2 (right). we show how the performance varies with changes to important hyperparameters, namely, the number of options and adaptation steps. We observe that decreasing the number of adaptation steps during training to one leads to clear drop in performance. This can be attributed to the policy not being able to switch from exploratory to exploitatory behavior a single inner update as well as the smaller amount of data ohserved before each outer update.
251
252
283
284
255
Unlike the number of adaptation steps, number of options does not seem to affect performance too much. The only noticeable exception is lower performance when using only 2 options. This exception can be explained by noticing that in some states one needs to perform 3 different actions to represent all optimal paths. As an example, consider the state two squares above the blue special state in Figure 1. To reach the blue state in the minimum number of steps the agent needs use the down action. Similarly, to go from the blue state to red or yellow one it needs to use up and right respectively. Thus the agent cannot represent the optimal policies with only 2 options. Interestingly, even in this case, the agent is still able to separate trajectories in such way that it can reach all goals albeit with slightly worse performance.
206
257
268
289
270
271
272
273
274
This outcome demonstrates another benefit of learned option lengths as the optimal option length does only depend lasks and their difficulty but also the number of options that available.
275
278
To illustrate this, consider an extreme case where there are as many options as lasks. In this case, it
277
would be sensible to have solution to a different task in each option and not terminate at all because
278
each task would be solved with only one high-level action. However, as the number of available options decreases, sharing options between tasks becomes necessary and terminations should start
7

---

Table I: Percentage of terminations in trajectories obtained from adapted policies averaged over 5 seeds. Standard deviations are in 1-2% range.
Number of options
2
3
4
5
8
16
Avg. terminations in traj 70%
63%
57%
55%
44%
28%
Ant maze tasks. The agent needs to control simulated 4-legged ant-like robot and move towards the green square.
Figure 3;
280
occur to allow for all tasks to he solved. Moreover, if the number of options is decreased even further, there may not be enough to options to capture the optimal behavior for all tasks. Consequently. it becomes
281
difficult choose the appropriate option length priori since it depend number available options. confirm this intuition, ran a followup experiment with longer time horizon in the taxi environment. While the trajectories produced by adapted policies had similar length, relative number of terminations decreased with the increase in the number of options as shown in Table 1.
293
284
285
287
5.2
Ant Maze
288 In our second experiment, we demonstrate the applicability of our method to more complex environ- ments. We use the family of ant maze tasks introduced by Frans et al. [17] shown in Figure 3. This allows
299
reproduce the results of MLSH closely possible by mirroring the setting used in the original paper. In addition MLSH, we also compare RL3, non-hierarchical meta-Icarning algorithm designed for fast-adaptation and Proximal Policy Optimization (PPO) [43], which serves
291
292
203 as strong single-task baseline.
294
In each task the agent needs to move simulated 4-legged ant-like robot through a small maze towards the goal. Both state space and action space are continuous with 29 and 8 dimensions respectively and each episode lasts 1000 timesteps. States do not contain any information about the maze layout or the location of the goal. The original implementation also resets the orientation the every 200 steps. However, we removed these resets hecause they made the MDP partially observable, introduced discontinuities and were not realistic for the robotics scenario they are supposed to imitate. Results of experiments with the original implementation are similar to the ones present. They can be found in Appendix C along with meta-training plots.
295
297
298
200
300
301
302
Both FAMP and MI.SH use the same architecture with two hidden layers of 64 nodes to represent the high-level policy, sub-policies and terminations (only applies FAMP). We used existing repositories for implementation
303
304
RI.2 [18] and PPO [1]. Hyperparameter values can be found in Appendix B. During the training phase, sub-policies (and terminations) of both hierarchical algorithms were trained on all tasks until the return averaged over all environments stopped improving. In the test frozen. Similarly, RI.2 was pre-trained on tasks and subsequently evaluated while PPO was trained from scratch.
305
308
307
phase all parameters except for the policy over options
308
The comparison of the performance and speed of adaptation
be in Figure 4 (left). Our method achieves superior performance reaching an average return 1330. We also observed a
310
311
similar trend across individual environments. Plots of these comparisons are available in Appendix C.
While the zero-shot performance of RL. is slightly better than FAMP. it often struggles to further adapt specific tasks and quickly gets outperformed both hierarchical methods. This is likely be
8

---

1397
1001
793
DAXI
200
Figure 4: Left: Average performance of algorithms on ant maze environments tasks. shows mean and standard deviation over 3 seeds. Right: Option usage visualization on ant maze tasks. Both plots were created using positions of the ant during trajectories. Each of the 3 options is represented by a different color.
314
due the objective that optimizes average return over all training episodes and post-adaptation performance directly. Lastly, PPO continuously improves but performance does not come close the meta-learning algorithms. After ahout 1000 episodes it reaches the performance of MI.H and if ran sufficiently long we would expect that it would eventually catch up to FAMP.
315
318
317
318
We visualize the option usage of FAMP on two example tasks in Figure (right). After the high-level policy is fine-tuned, we use the and positions of the ant in 3 sampled trajectories to highlight which option active at each part of the
319
320
only take out 29 dimensions into account, we are still able to gel useful insight about learned option structure. the task that is depicted in the left part of the plot, the agent uses the blue option before switching to cyan in the middle and finishing with a combination of blue and purple. On the other hand, in the right task, the agent uses combination of blue and purple to move down instead of to the right. This shows that the
space. Although
321
322
329
325
agent learned useful abstraction that allows to perform two different useful behaviors similar parts of the state space by using terminations and different options.
328
Discussion and Future Work
327
In this work, aim leam both sub-policies and terminations of options using single simple principle: options should accelerate adaptation in many tasks. proposed method for learning hierarchical policies that combines the options framework with gradient-base meta-learning and explicitly optimizes for performance after several adaptation steps. In our experiments, we have demonstrated the benefits of approach quickly learning previously unseen test tasks. Furthermore, we have shown that the proposed method outperforms the closest hierarchical and non- hin-hiirarchical meta-reinforcement learning methods designed for similar selting challenging multi-task lcarning scenario.
329
330
331
333
394
335
336
The computation limitations of our method are mostly connected to the calculation of responsibilitics in IOPG. In this calculation, many sequential matrix multiplications are required both in the forward and backward pass. The compute time for cach update thus dependent on the trajectory length because these calculations cannot be done in parallel. One direction for future work could thus be alleviating this limitation.
337
338
339
340
341
Our objective does not explicitly constrain the number of terminations as long they lead fast adaptation. Thus, there are many combinations of options with different lengths which can lead to good performance on all tasks, which do not always correspond to intuitive decompositions. One possible cause of spurious terminations lies in the continuous state space used in some tasks. When neural networks used represent termination functions, they learn generalize nearby states. tasks such as ant maze, the agent will visit many states in the same neighborhood and might thus terminate options several times in quick succession. A promising topic for future investigation whether this problem could be alleviated by using terminations that also depend on the state in which the option initiated.
342
343
345
348
347
9

---

References
350
351
[1] Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
352
[2] Maruan Al-Shedivat. Trapit Bansal, Yura Burda, liya Sutskever. Igor Mordatch, and Pieter Abbeel. Continuous Adaptation via Meta-Learning Nonstationary and Competitive Environ- ments. In International Conference on Learning Representations, 2018.
353
354
355
[3] Antreas Antoniou, Harrison Edwards. and Amos Storkey. How to train your MAML, In International Conference on Learning Representations, 2019.
358
357
[4]
Fierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. Proceedings of the AAAI Conference on Artificial Intelligence. 31(1). Feb. 2017.
358
359
[S] Andre Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The op- tion keyboard: Combining skills in reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems, volume 32. Curran Associates, Inc., 2019.
380
351
362
353
394
[6]
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3):337-357, 2016.
385
388
Peter Dayan and Geoffrey E Hinton. Feudal Reinforcement Learning. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages 271 271-278. Morgan-Kaufman
367
368
1993.
369
88]
Thomas G. Dietterich. Hierarchical Reinforeement Learning with the MAXQ Value Function Decomposition. Journal of Artificial Intelligence Research, 13(1):227-303. 2000.
370
371
[9] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. In Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1329-1338. PMLR, 2016.
372
374
375
[10]
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast Reinforcement Learning via Slow Reinforcement Learning, CoRR, abs/1611.02779, 2016.
376
377
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You Need: Learning Skills without a Reward Function. In International Conference on Learning Representations, 2019.
378
379
380
12]
Jesse Farebrother, Marlos C. Machado, and Michael Bowling, Generalization and Regularization in DQN. CoRR, abs/1 88 10.00123, 2018.
381
382
13] Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 32, pages 8151-8162. Curran Associates, Inc., 2019.
383
384
385
386
[14]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostis Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 2017.
387
388
389
390
[15] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. DICE: The Infinitely Differentiable Monte Carlo Estimator. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1538.1538. PMLR, 2018.
391
392
393
[16] Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-Level Discovery of Deep 394
385
Options. CoRR, abs/1 703.08294, 2017.
398
[17] Kevin Frans. Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta Learning Shared Hierarchies. In International Conference Learning Representations, 2018
397
10

---

398
[18]
The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https://github. http:///owooooggooo/ggaaaaaaa
399
2019.
400
[19] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised
401
Meta-Learning for Reinforcement Learning. CoRR, abs/1806,04640. 2018.
402
[00] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbecl, and Sergey Levine. Meta- Reinforcement Learning of Structured Exploration Strategies. In Advances in Neural Informa- tion Processing Systems, volume 31, pages 5302-5311, I. Curran Associates, Inc., 2018.
403
404
405
[21] Jean Harb, Pie Bacon, Martin Klissarov, and Doina Precup. When Waiting Is Not Option: Learning Options with Deliberation Cost. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
406
407
408
[22]
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The Termination Critic. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 2231-2240 PMLR, 2019.
409
410
411
[23]
Maximilian Igl, Andrew Gambardella. Jinke He, Nantas Nardelli, N Siddharth, Wendelin Boehmer, and Shimon Whiteson. Multitask Soft Option Learning. In Proceedings of the 36th Conference on Uncertainty Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pages 969-99 978. PMLR, 2020.
412
413
414
415
[24) George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement learning. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, puges 895-900, 2007.
418
417
418
[25] Alexander Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy Adaptation for Hierarchical Reinforcement Learning. In International Conference Learning Representations, 2020.
419
420
421
[26]
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few Shot Learning. CoRR, abs/1 77711007.0988 09835,985,,.2 2017.
422
423
[27] Hao Liu, Richard Socher, and Caiming Xiong. Taming MAML: Efficient unbiased meta- reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4061-4071. PMLR, 2019.
424
425
428
427
[28] Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A Laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2295- 2304, PMLR, 2017.
428
429
430
431
[29] Daniel J Mankowitz, Timothy Mann, and Shie Mannor. Adaptive skills adaptive partitions (asap). In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
432
433
434
[30] Amy McGovern and Andrew G. Barto. Automatic Discovery of Subgoals in Reinforcement Learning Using Diverse Density. In Proceedings of the 18th International Conference on Machine Learning, ICML '01, page 361-368. 2001.
435
438
[31] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-Cut-DDnamic Discovery of Sub-Goals in Reinforcement Learning. Proceedings of the 13th European Conference on Machine Learning, ECML'02, page 295-306. Springer-Verlat 2002.
438
439
440
322 Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive Meta-I.carner. In International Conference on Learning Representations, 2018
441
442
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with Deep Reinforcement Learning. CoRR, 1bs/1312.5602, 2013.
443
11

---

445
[34] Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchi- cal Reinforcement Learning. In Advances in Neural Information Processing Systems 31, pages 3303-3313. Curran Associates, Inc., 2018.
446
447
448
[35] Scott Niekum and Andrew Barto. Clustering via Dirichlet Process Mixture Models Portable Skill Discovery. In Advances in Neural Information Processing Systems, volume 24, pages
449
450
1818- 1826. Curran Associates, Inc., 2011.
451
[36] Mare Pickett and Andrew G. Barto. Policyblocks: An algorithm for creating useful maero- actions in reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 506-513, Morgan Kaufmann, 2002.
432
453
454
[37] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. In International Conference on Learning Representations, 2020.
455
458
457
[38] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient Off- Policy Meta-Reiiforcemmett Learning via Prohahilistic Context Variables. In Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5331-5340. PMLR, 2019.
458
459
480
461
[39]
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning Abstract Options. In Advances in Neural Information Processing Systems 31, pages 10434, Curran Associates, Inc., 2018.
462
463
40)
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal Meta-Policy Search. In International Conference on Learning Representations, 2019.
484
465
[41] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889-1897. PMLR, 2015.
487
468
460
[42] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Digge-iieesional Continuous Control Using Generalized Advantage Estimation. CoRR, 1bs/1550602438, 2015.
470
471
472
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, Abs//1777.0633 2017.
473
474
[44] Matthew Smith, Herke van Hoof, and Joelle Pineau. An Inference- Based Policy Gradient Method for Learning Options. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4703-4712. PMLR, 2018.
475
476
477
478
[45]
Bradly Stadic, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Picter Abbecl, and Ilya Sutskever. The importance of sampling in meta-reinforcemmnt learning. In Advances in Neural Information Processing Syxtems, volume 31, pages 9280-9290. Curran Associates, Inc., 2018
479
481
482
[46] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112((12): 181-211, 1999.
484
485
[47] Yee Teh, Victor Bapst, Wojeiech M. Czareecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, pages 4496-4506. Curran Associates, Inc., 2017.
487
488
480
[48] Sehastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. MIT Press, 1995.
490
481
432
[49] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-50060000 2026,503,,20 2012.
12

---

494
[50] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3540-3549. PMIR, 2017.
495
498
407
488
[51] Jane X. Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs//611.00 1..666611,,., 2016.
500
501
[52] Shangtong Zhang and Shimon Whiteson. DAC: The Double Actor-Critic Architecture for Learning Options. In Advances in Neural Information Processing Systems, volume 32, pages 2012-2022. Curran Associates, Inc., 2019.
502
508
504
[53] Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M. Hospedales. Investigating Generalisation in Continuous Deep Reinforcement Learning. CoRR, abs/1902.07015, 2019.
505
508
[54]
Luisa Zinigraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast Context Adaptation via Meta-Leaamig In Proceedings of the 36th International Conference on Machine Learning. volume 97 Proceedings of Machine Learning Rexearch, pages 7693-7702.
507
508
509
PMLR, 2019.
Checklist
510
511
For all authors...
512
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope'! [Yes]
514
(b) Did you deseribe the limitations of your work? [Yes] See Section 6
515
(c) Did you discuss any potential negative societal impacts of your work? [No] We propose
516
general meta-reinforcement algorithm that does not have any foresecable negative
517
social impact
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
519
520
2. If you are including theoretical results...
521
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
522
(b) Did you include complete proofs of all thcoretical results? [N/A]
523
If you ran experiments...
524
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] In supplemental material
528
527
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Some training details are given Section 5, the rest is provided in Appendix B
528
529
530
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
531
532
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? Yes] Included in Appendix B
533
534
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
535
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the codehases and works that introduced environments use in Section 5
538
S37
(b) Did you mention the license of the assets? [No] We used publicly available code
538
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We include our codebase in the supplemental material and will make public github
539
repository
541
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating" [No] We used publicly available code
13

---

543
(c) Did you discuss whether data you are usinglcurating contains personally identifiable information or offensive content? [N/A] We use data from virtual environments
544
5. If you used crowdsoureing conducted research with human subjects...
S46
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
547
548
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
549
350
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
551
14


Reviews:
[{"review_id": "gRWTAgk8c79", "reviewer": "Reviewer_NbWu", "summary": "This paper proposes an option method where the high-level agent is a meta-learning agent that can fast learn to select proper options through sampled few episodes. DiCE is used to correct the gradients. The paper demonstrates the effectiveness of the approach on Taxi and Ant Maze environments and outperforms RL^2 and PPO baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's presentation is very clear and I appreciate the idea to combine meta-learning and the options framework, but I do not think that the experiments fully demonstrate the potential of the method. First of all, the experiments in the paper focus on navigation. Ant Maze environment is not a challenging task for other hierarchical RL methods. The difficulty comes from forbidding the agent from observing the maze layout. However, since the maze size is rather small, the number of possible layouts is also small. It is possible that that the agent simply remembers the solution to all map layouts and uses the meta-learning trajectories to identify the maze layout. The experiments do not demonstrate the generalizability of the novel map layout. In this case, why not simply learn a recognition model to reconstruct the maze layout [1] and then learn a goal-conditioned policy? The approach is not significant unless it can generalize to a novel layout in a larger maze. Otherwise, I would doubt that we should use search instead of meta-learning for solving complex planning tasks. Besides, since the number of options is small, and the results are not very interpretable, if we need options is questionable to me. The paper lacks a MAML-like meta-learning baseline to demonstrate that options are necessary.  \n\nBesides, I am confused by the plots in the paper. Fig 4 shows that FAMP reaches 1200+ score after 40 episodes, but according to the pseudocode and Table 2, in each update step of the meta-agent, it needs to sample N\\times L+1\\times k episodes, which is larger than 40. Does this mean that there is no need to learn the meta-agent?\n\nIn short, the paper has good motivation, however, the experiments have flaws and are not strong enough to demonstrate the potential of their approach. I hope the authors can apply their approach to more complex domains and show its significance.\n\n[1]Multi-task Batch Reinforcement Learning with Metric Learning", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the author discussed the limitations of computation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8zJjk4dYmL9", "reviewer": "Reviewer_uSF5", "summary": "The paper presents a meta-learning approach for learning hierarchically structured policies, where the agent is trained on some distribution of similar tasks to maximize the post-adaptation performance, as well as learning useful sub-policies in different time scales that enable fast adaptation. The overall model contains a high-level policy selecting over a set of options, and each option contains a sub-policy controlling action selection and a termination probability controlling the duration of low-level policy execution.  During training, the authors argue that the update for low-level policies and termination probabilities should be slower than the update for the high-level policy, to potentially encourage learning sub-policies that can be used in multiple tasks. This is the key idea of the proposed fast adaptation of modular policies (FAMP) algorithm. The experiments on Taxi and Ant Maze environments show FAMP can fast adapt and achieve better returns than other baseline meta-learning methods on unseen test tasks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tries to address an important problem in RL to learn hierarchical policies that enable better adaptation. The proposed algorithm follows the options framework and utilizes recent optimization methods, including MAML and DiCE. The paper has good coverage of related work and is distinguished from existing work by learning both sub-policies and termination conditions. Regarding experiments, the adaptation results look promising, and the selection of baselines is reasonable because MLSH does not utilize the environment id.\n\nHowever, a few designing factors remain unjustified in the current paper. More ablation study is required to understand their effects, in addition to the comparison with MLSH. First, one key contribution, as claimed, is more flexible termination of sub-policies. However, it is unclear about its actual performance gain and how fixed termination would affect the learning sub-policies, and the adaptation performance remains to be checked. Second, the paper argues that fixing sub-policy in the inner-loop encourages better sub-policies. However, this intuition is not yet supported by any quantitative results. \n\nAlso, it seems that FAMP does not require warm-up for sub-policies, while MLSH needs warm-up. Perhaps a related discussion on this is needed. How are sub-policies and termination conditions initialized? If the initial performance of sub-policies is undesired, fixing sub-policies in the inner-loop may lead to slower meta-learning.\n\nOverall, the paper presents an interesting idea for hierarchical RL, and experiments are sufficient to prove its effectiveness in terms of better adaptation performance. However, the learned sub-policies on the Taxi task seem not easy to be interpreted. Moreover, IOPG may be computationally heavy. More analysis would strengthen the paper in understanding the effects of key designing components.\n\nTypos:   \nline 124, definition of $G_t$ should be $G_t(\\tau) = \\sum_{t’=t}^{T}\\gamma^{(t’-t)}r_{t’}$.  \nEq. 1, $\\sum_{\\mathcal M_i \\sim p(\\mathcal M)}$ is not standard, perhaps to rewrite it as $\\sum_{\\mathcal M_i} p(\\mathcal M_i) \\mathcal L_{\\mathcal M_i}(\\cdots)$   \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitation of the non-parallelizable responsibilities calculation of IOPG. I also don't foresee any negative social impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "0k9992pa9P", "reviewer": "Reviewer_JWbv", "summary": "This paper proposes to use meta reinforcement learning in order to find a higher-level policy that can quickly update to achieve higher rewards and new tasks. The method uses the options framework in this case and uses meta-learning to find and quickly update the option selection policy given only a little bit of data from the environment. The method does show some improved performance using this particular structure over two different environments, one that was used in the previous work.\n\nPros\n- The Meta-learning structure that's described in the paper does appear to be rather effective and somewhat novel. It shows some similarities to having a high-level policy that is more of a latent policy in this case in the middle learning performs a gradient step in order to adjust which options end up getting selected for a particular task. But the overall method is different enough from prior methods.\n- The described method in the paper is also not significantly complicated and seems to fit in fairly well to previous hierarchical reinforcement learning designs in particular for using an options type setup that includes a termination function.\n\n\nCons\n- While the results look promising they are applied to a limited number of environments. One environment being a discreet environment and the other being the environment from the MLSH paper. Well, potentially not necessary it could still help bolster the thoroughness of the findings in the paper if the evaluation is performed over additional environments. For example using some of the environments from meta world or other hierarchical reinforcement learning papers.\n- Well the paper does outline its novel design structure for applying the meta-adaptation step on the higher-level policy there isn't any ablation analysis in the paper to really understand the most important components of the overall method. For example, additional analysis that includes all of the levels of the hierarchy inside of the inner meta adaptation step would improve the clarity of the experiments and the solidity of the findings. Overall the paper seems to be on solid ground but the additional analysis would add to the contributions of the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Additional comments:\n\n- It is interesting that the analysis in the paper indicates that the method's performance does not significantly depend on the number of options that are available to the agent. This is usually an issue for these types of methods. It would be helpful to provide some discussion on why the method does not suffer from this common issue.\n- RL squared algorithm does not show any improvement in the evaluation environments. That meta-learning algorithm has been evaluated on similar tasks to this ant maze environment and has shown some success. It is possible this baseline is not applied correctly in the work.\n\n----- Post Author Feedback -----\nGiven the author's feedback, my rating for the paper will stay the same.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new hierarchical RL algorithm for fast adaptation. Following the options framework, the proposed method learns a set of sub-policies, a termination function, and a high-level policy via MAML with DiCE. The results on a simple grid world (Taxi) and a continuous control navigation task (Ant Maze) show that the proposed method adapts to new tasks more efficiently than a hierarchical (MLSH) and non-hierarchical baselines (PPO/RL^2). 

Although the idea is new and interesting, all of the reviewers agreed that the experiments are conducted only on simple domains. More crucially, while this paper emphasizes the importance of several ideas (e.g., learning termination function rather than using fixed-length options, fixing sub-policies during adaptation of high-level policy), there is no ablation study that justifies the claims. Thus, even though the results look encouraging, it is hard to understand the benefit of individual ideas. Therefore, this paper is not ready for publication at the moment, and I encourage the authors to conduct more comprehensive experiments for the future venue.
Decision: Reject