Meta Reinforcement Learning for Fast Adaptation of Hierarchical Policies
Year: 2021

Abstract:
Hierarchical methods have the potential to allow reinforcement learning to scale to 1 larger environments. Decomposing a task into transferable components, however, 2 remains a challenging problem. In this paper, we propose a meta-learning approach 3 for learning such a decomposition within the options framework. We formulate 4 the objective as a bi-level optimization problem in which sub-policies and their 5 terminations should facilitate fast learning on a family of tasks. Once such a set 6 of options is obtained, it can then be used in new tasks where only the sequencing 7 of options needs to be chosen. Our formalism tends to result in options where 8 fewer decisions are needed to solve such new tasks. Experimentally, we show that 9 our method is able to learn transferable components which accelerate learning and 10 performs better than existing methods developed for this setting in the challenging 11 ant maze locomotion task. 12

Hierarchical methods have the potential to allow reinforcement learning to scale to1 larger environments. Decomposing a task into transferable components, however,2 remains a challenging problem. In this paper, we propose a meta-learning approach3 for learning such a decomposition within the options framework. We formulate4 the objective as a bi-level optimization problem in which sub-policies and their5 terminations should facilitate fast learning on a family of tasks. Once such a set6 of options is obtained, it can then be used in new tasks where only the sequencing7 of options needs to be chosen. Our formalism tends to result in options where8 fewer decisions are needed to solve such new tasks. Experimentally, we show that9 our method is able to learn transferable components which accelerate learning and10 performs better than existing methods developed for this setting in the challenging11 ant maze locomotion task.12
1 Introduction13
Current state of the art model-free reinforcement learning methods were successfully applied to14 many challenging tasks [33, 41]. However, one of the main drawbacks of these methods is their15 data-inefficiency and inability to generalize to other related tasks [12]. It is often impossible to use16 the agent trained on one task to solve another related task [53] or even to use it as a starting point for17 training because trained models become increasingly exploitative and thus are unable to explore in a18 new task. In such cases, we have to gather new data and train a new model which is time-consuming.19
One way to mitigate this problem is by learning a policy with reusable modules which can be used in20 multiple tasks. For example, if we assume that related tasks contain shared sub-tasks (i.e. tasks come21 from the same family or have hierarchical structure), we can speed up the adaptation to new tasks by22 learning sub-policies that solve these sub-tasks. This is because solutions to new tasks can be created23 by combining known solutions to sub-tasks during adaptation. The idea of learning reusable skills in24 multiple environments, which dates back to at least 1995 [48], was thoroughly explored within the25 options framework [17, 24, 25, 29, 36, 46].26
In this framework, a policy is composed of options (modules that encapsulate sub-policies), and27 a high-level policy that chooses among them. Options have their own termination function, and a28 new option is only initiated when the earlier option terminates. Therefore, options define temporally29 extended behaviors that can form solutions to sub-tasks. Despite extensive research in this area, there30 is not yet a consensus on answers to many important questions about options: What are good options?31 How can we find them? When should a termination occur? How many options should one use? In32 this work our aim will be to find options that allow for fast adaptation to tasks from the same family.33 We use this single principle to address all of these questions except for the number of options which34 we consider a hyperparameter.35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
Learning high-level policy, sub-policies and terminations at the same time is a challenging task.36 Recent prior work on options proposed a way to learn both, including the terminations, in an end37 to end manner with policy gradient methods [4, 44]. However, despite achieving good performance38 in single-task settings, these methods often produce options which may not be useful for transfer39 [21, 22]. This is because such options are not explicitly trained for multi-task setting and can often40 terminate too often or not at all [21, 22].41
To overcome this issue, Frans et al. [17] proposed to use such method in a multi-task setting with42 options that have predefined length and are optimized for performance after adaptation of the high-43 level policy. Although options that terminate after certain amount of steps simplify the problem44 and work well in some settings [17, 25], manually setting this important hyperparameter requires45 prior knowledge and might not work in cases where options need to have different lengths [23]. For46 example, task where this approach would not be preferable could be driving because some driving47 sub-tasks, such as driving on a highway, are much longer than others, such as driving out from one48 intersection to another in a city. Consequently, capturing the length of both sub-tasks with a single49 hyperparameter [17] or range of hyperparameters [25] can become difficult or even impossible. In50 such cases, learned terminations are preferable.51
In this paper, we propose a method for learning options that allows for fast adaptation to multiple tasks.52 We formalize this notion using recent ideas from gradient based meta-learning [14]. Rather than using53 options with fixed length [17], our algorithm learns both sub-policies and when to terminate options54 using a single meta-learning objective. We hypothesize that this objective implicitly encourages55 options to terminate in a way that yields reusable components. In our experiments, we demonstrate56 the benefits of our approach in a simple Taxi domain as well as in a complex Mujoco [49] Ant Maze57 domain [17].58
2 Related Work59
Since our work builds on insights from both hierarchical reinforcement learning and meta-learning,60 we present related work in both domains separately, in subsections 2.1 and 2.2 respectively.61
2.1 Hierarchical Reinforcement Learning62
One of the aims of hierarchical reinforcement learning is to decompose a complex task or policy into63 simpler units. Popular approaches include learning a diverse set of skills [11] or utilizing the idea of64 Feudal Reinforcement Learning [7, 34, 50]. Another large collection of related work instead relies on65 the options framework [46].66
Some works on options rely on so-called bottleneck states that can be used as sub-goals [30, 31, 35]67 whereas others use spectral clustering to create options [28]. These approaches usually require prior68 knowledge about the environment which restricts their applicability. Different from aforementioned69 methods, end-to-end methods such as the ones which rely on the Option-Critic architecture [4, 39] are70 applicable in more general settings. However, these policy gradient methods can be less efficient than71 concurrently introduced inference based end-to-end methods [6, 16, 44] because they only update the72 option that generated the action whereas inference based methods update options according to their73 responsibilities for each action.74
A common problem with end-to-end methods that learn terminations in a single-task setting is75 option collapse [4]. This causes options to terminate after every action or to never terminate. In76 such cases the learning of terminations can be facilitated by augmenting the objective with entropy77 regularization [44] or deliberation cost [21], regularizing towards a termination prior [23], or by78 optimizing different objective that encourages appropriate terminations [22]. As an alternative, one79 can also use time-based terminations with fixed [17] or randomized length [25].80
2.2 Meta-Reinforcement Learning81
Meta-reinforcement learning is concerned with producing models which are able to adapt to novel82 tasks quickly. This sub-field includes a broad range of work such as unsupervised methods [11, 19],83 methods that rely on latent variables [20, 38] or methods that learn the update rule of a policy84 [10, 32, 51].85
In contrast with the latter, the recent gradient-based method Model-Agnostic Meta-Learning86 (MAML) [14] assumes that policy parameters are updated with gradient descent and instead aims to87 learn initial parameter values. MAML was extended in followup works that only trained a part of the88 network [37, 54] or showed benefits of different architectural choices such as per-parameter learning89 rates [3, 26]. Several works also focused on MAML in a reinforcement learning setting [2, 27, 45].90 In particular, Al-Shedivat et al. [2] and Stadie et al. [45] pointed out a difference between theory and91 practical implementation of MAML in automatic differentiation frameworks. This issue was further92 discussed and resolved in followup works [13, 15, 40].93
Lastly, there exist methods which do not employ the techniques mentioned above and instead rely94 on the options framework [5, 17, 23–25, 29, 36, 52] or task-specific policies [47]. These approaches95 often make different assumptions about the tasks and settings in which they are applied. Some require96 policies that solve each environment [36] whereas others need environment ID [23, 29] or cumulants97 that properly represents task dynamics [5]. Closest to our work are Meta Learning Shared Hierarchies98 (MLSH) [17] and Adaptive Skills Adaptive Partitions (ASAP) [29]. ASAP uses a policy gradient99 method to optimize immediate performance on multiple tasks with known environment ID but does100 not use neural networks and does not learn terminations. On the other hand, MLSH uses a hierarchical101 structure with predefined options length and a problem setting with unknown environment ID. It102 optimizes for post-adaptation performance by using two alternating phases that either only update103 high-level policy or both high-level policy and sub-policies simultaneously. This approach does not104 use the information from the intermediate adaptation steps when calculating the gradient which can105 negatively affect its accuracy. Additionally, options with fixed length may be difficult to use in some106 settings as we’ve described in Section 1.107
3 Background and Notation108
In this section, we will first cover the fundamentals of reinforcement learning, and then focus on the109 options framework and gradient-based meta-learning.110
3.1 Reinforcement Learning and the Options Framework111
We will consider environments which are episodic Markov decision processes (MDPs). An MDP112 M is a tuple �S,A, p0, P,R, γ� with S being a set of states, A a set of actions, p0(s0) a probability113 distribution of initial states, P (s�|s,a) a transition probability function, R(s,a) a reward function114 and γ a discount factor.115
An agent with a stochastic policy π interacts with an environment M in the following way. At116 every timestep t, the agent receives a state of the environment st ∈ S and selects an action117 at ∈ A according to conditional distribution π(at|st). Depending on the current state and the118 action performed, the environment provides the agent with a new state st+1 ∼ P (st+1|st,at) and119 a scalar reward rt = R(st,at). This process is repeated until a so-called terminal state is reached.120 We define a trajectory τ as an ordered sequence of all states actions and rewards in a single episode121 τ = (s0,a0, r0, ..., sT ). Similarly, the history at timestep t consists of all states and actions preceding122 at, ht = (s0,a0, ..., st). The state value function is defined as Vπ(s) = Eπ [Gt|st = s] where the123 discounted return at timestep t is defined as Gt(τ) = �T t�=t γ (t−t�)rt� .124
The agent’s objective is to maximize the expected discounted return J = Ep(τ |θ) [G0(τ)].125 We can maximize the objective with gradient descent by estimating the policy gradient126 ∇θJ ≈ Ep(τ |θ)[ �T t=0 ∇θ log πθ(at|st)At] using Monte Carlo sampling, where At is an advantage127 estimator such as the generalized advantage estimator AGAEt [42].128
The options framework is a framework for temporal abstraction that consists of options129 ω = �Iω,πω, ξω� and a policy over options πΩ(ω|s). Each option ω consists of an initiation set, a130 sub-policy and a termination function. The initiation set Iω is a set of states in which an option can131 be selected (initiated) and in our case it is the whole state space (Iω = S). A sub-policy πω(a|s),132 also called low-level policy, is a regular policy that acts in the environment. Lastly, the termination133 condition ξω(s) is a function that outputs the probability of termination for the option in a given state.134
3.2 Model-Agnostic Meta-Learning and DiCE135
Model-Agnostic Meta-Learning (MAML) [14] is a meta-learning technique that trains a model for136 maximum post-adaptation performance on a distribution of tasks. The adaptation consists of one or137 several inner gradient updates. If we consider an estimator fθ with parameters θ and a task-specific138 loss LMi , a supervised learning objective with a single inner update can be formalized as shown in139 Equation 1. In order to optimize this objective one only needs to take a gradient of this expression.140 This can be easily achieved with automatic differentiation frameworks by creating a backpropagation141 graph for the gradient.142
min θ EM [LMi(fθ�)] = min θ
�
Mi∼p(M) LMi(fθ−α∇θLMi (fθ)) (1)
One can similarly use this approach with a reinforcement learning objective. However, the implemen-143 tation with an automatic differentiation framework differs because a simple backpropagation through144 the computation graph of the gradient produces biased gradients [2, 45]. This is due to an additional145 dependency of the sampling distribution on parameters that is not present in the supervised learning146 objective. To produce correct higher order gradients with automatic differentiation frameworks in147 a reinforcement learning setting, one can use the objective in Equation 3 as proposed by Farquhar148 et al. [13]. This objective utilizes the DiCE operator [15] which can be implemented according149 to Equation 2 where ⊥(x) is a stop gradient operator that evaluates to x but returns a zero gradient150 when differentiated.151
(at) = exp [log πθ(at|st)−⊥(log πθ(at|st))] , ∇θEτ∼p(τ |θ) � GMi0 (τ) � ≈ ∇θJ (2)
152
∇θJ = Eτ∼p(τ |θ) � T�
t=0
∇θ � t�
t�=0
(at�)λ t−t�AGAEt −
t−1�
t�=0
(at�)λ t−t�AGAEt �� . (3)
4 Fast Adaptation of Modular Policies153
Much of the extensive research in the options framework has focused on an intuition of options154 capturing useful sub-tasks [4, 17, 36, 46]. However, there is no consensus about capturing this155 intuition in an objective function or the best way to find such options. We propose a conceptually156 simple objective: a good set of options allows quick adaptation to many novel tasks. This can157 be formulated using the MAML framework [14], where we consider a setting in which there is a158 distribution of tasks p(M) with similar (hierarchical) structure but different reward or transition159 functions. Our goal is then to maximize the expected performance after L adaptation steps of the160 hierarchical policy parametrized by θ:161
max θ
�
Mi∼p(M) EτL∼p(τL|θL)
� GMi0 (τ L) � , θj+1 = θj + αin∇θjEτj∼p(τj |θj) � GMi0 (τ j) � . (4)
Using conventional MAML means adapting a large number of parameters which can be disadvanta-162 geous, as was demonstrated by Zintgraf et al. [54] and Antoniou et al. [3]. By reducing the number of163 parameters that are tuned during the adaptation phase, one can reduce the complexity of the problem164 during test time at the cost of a less expressive policy. We thus split the parameters into an inner group165 θin and an outer group θout where inner parameters are updated during the adaptation step and outer166 parameters are optimized in the outer objective. Note that when using such split, the initialization167 values of inner parameters may also be meta-learned [54]. We experimented with both versions and168 observed that fixed initialization values performed better. Similarly, the per-parameter inner learning169 rate αin [3, 26] can also be meta-learned to allow for more complex inner updates. We used this170 approach in a setting with more complex environment.171
Our option model has three sets of parameters: those of the high-level policy network θΩ, sub-policy172 networks θω and termination networks θξ. We now divide these over the inner and outer parameter173 group. Since we assume that tasks with common sub-problems can be solved using identical options,174 we consider the sub-policy and termination function parameters as outer parameters. On the other175 hand, since in each task the decision of the high-level policy to choose options would be different,176 its parameters constitute the inner group. By keeping sub-policies fixed during the adaptation and177
Algorithm 1 Fast Adaptation of Modular Policies initialize θΩ, θξ, θω,αin ,αout set θin = θΩ set θout = {θξ, θω} repeat
Set gradient of outer parameters gθout = 0 for n = 1 to N do
set θ�in = θin sample a task M ∼ p(M) for l = 1 to L+ 1 do
sample k episodes τ1:k on M using π{θ�in ,θout} fit a baseline Vκ using data from τ1:k compute AGAEt for all τ1:k compute log π(at|ht) = Eω|ht [πω(at|st)] compute J with AGAEt , log π(at|ht) (Eqs. 2, 3) if l < L+ 1 then θ�in = θ � in + αin∇θ�inJ else gθout = gθout +∇θoutJ
θout = θout + αout 1 N gθout
until convergence
restricting the update to the high-level policy, we optimize for options that can be used to solve178 multiple tasks, thereby allowing the overall policy to adapt quickly with the change of high-level179 policy. This also allows for an expressive policy which can capture different behaviors and reduces180 the number of parameters and decisions an agent needs to learn and make during test time.181
Formally, our final objective can be expressed as Equation 5 with the inner update given by Equation182 6. The objective is similar to the one used in MLSH [17] with some key differences. Firstly, by183 backpropagating through the update step we are able to capture additional information from the184 adaptation steps in the gradient and secondly, our objective includes the optimization of termination185 parameters and thus allows for options with different lengths.186
max θω,θξ
�
Mi∼p(M) Eτ∼p(τ |{θω,θξ,θLΩ})
� GMi0 (τ) � (5)
θj+1Ω = θ j Ω + αin∇θjΩEτ∼p(τ |{θω,θξ,θjΩ})
� GMi0 (τ) � . (6)
4.1 Algorithm187
Written in its general form the objective leaves some freedom with regard to which policy gradient188 algorithm is used for gradient calculation. In our work we use the Inferred Option Policy Gradient189 (IOPG) [44] because it updates all options at the same time based on their responsibilities, i.e., the190 probability that the option was active given the history ht of states and actions so far. This can lead to191 better data-efficiency when compared to other methods that only update a single option at a time but192 comes at the cost of increased computation time. Another important design choice is the state value193 function estimator. In the MAML RL setting the policy constantly changes in every inner update. It is194 thus difficult to use past trajectories for fitting the value function. We therefore use a linear time-state195 dependent baseline [9] which works better than more complex baselines with little data and was also196 used in the original MAML implementation.197
The resulting algorithm for Fast Adaptation of Modular Policies (FAMP) is outlined in Algorithm 1.198 Note that in order to use IOPG with DiCE we replace π(at|st) with π(at|ht) in Equation 2. An199 intuition about why this is possible comes from the fact that we can easily formulate a new MDP200 M̃ in which states s̃t are histories ht of the original MDP without otherwise altering the dynamics.201 After L inner updates, the gradient of the objective with respect to the outer parameters is calculated.202 In principle, we would like to optimize for performance after a moderate number of gradient updates203 L such as 10 or 20. However, with more inner updates the resulting gradient of the objective becomes204
noisier due to the usage of Monte Carlo estimate in each inner update. Furthermore, the time205 complexity of gradient computation and sample complexity both scale linearly with the number of206 inner updates. In practice we found a range from 2 to 4 update steps to be acceptable. An important207 benefit of gradient-based meta-learning is that even though the model is optimized for performance208 after L adaptation steps, it can still be improved after L updates by performing more steps of gradient209 descent.210
5 Experiments211
In this section, we empirically evaluate our method and show its benefits when applied to randomly212 selected tasks within and outside of the training distribution.213
5.1 Taxi214
In the first set of experiments we use a modified Taxi environment1 [8] displayed in Figure 1. An215 agent acts as a taxi driver who starts in one of the special (colored) locations. The goal of the driver216 is then to drive a passenger from one of the special locations to his destination. The task family217 consists of 60 different tasks with different combination of start, goal and passenger positions. These218 are always initialized in special states. In 12 out of 60 easier configurations the passenger starts the219 episode in the car. The only restriction on start, goal and passenger positions in all cases is that220 passenger’s destination must not be the same as his initial position. Each task is an MDP in which the221 agent can use 4 directional actions and two special actions: pick-up/drop-off and no-op. The state222 space is represented as a one-hot vector with 72 entries for every combination of possible taxi location223 and passenger being on board. Thus the agent does not have any information about the location of224 the passenger or goal state. Therefore, in order to facilitate fast adaptation to the (unobservable)225 passenger and goal locations, the agent must acquire options that can serve as building blocks for226 exploration. The reward is 2 for reaching the goal and −0.1 per step otherwise. To speed up training227 in the early phases, we terminate the episode if it takes longer than 1500 timesteps.228
In this experiment, we use tabular representations implemented as a combination of linear layer and229 non-linearity for the policy over options, terminations and sub-policies such that each one-hot state230 has its own set of parameters. We use 48 training tasks to train sub-policies and terminations with our231 algorithm. Learned terminations and sub-policies are then kept fixed during test time and only the232 policy over options is updated. Performance is then compared on the remaining 12 test tasks (selected233 to use combinations of special locations with similar frequency) to MLSH and two baselines. We234 chose MLSH because it is a closest hierarchical method designed for our setting in which there is no235 extra information about the environment available. This is in contrast with many other hierarchical236 [5, 23, 29] and non-hierarchical [38, 47] meta-reinforcement learning methods which utilize extra237 information such as the ID of a sampled environment.238
Similarly to our method, MLSH is trained on all training tasks and evaluated with fixed sub-policies.239 The multi-task baseline is an IOPG algorithm that learns a shared policy (including high-level policy)240
1Details are included in Appendix B
by optimizing average return over tasks rather than the meta-learning objective in Equations 5 and241 6. After the training, it only adapt its high-level policy on test tasks. We expect this baseline to242 perform poorly in the long run because it does not optimize for post-update performance. Lastly,243 the single-task baseline is an IOPG algorithm that learns the test tasks from scratch without any244 pre-training. Therefore, since it does not need to generalize to many tasks and has a policy with245 enough capacity, we expect that it should eventually outperform other methods after sufficiently246 long training. However, meta-learned policy with desirable options should find good solution much247 quicker. To make the single-task baseline as strong as possible, we set its learning rate to the highest248 value that was able to solve all tasks reliably.249
Results250
As shown in Figure 2, our method is able to outperform both MLSH and the multi-task baseline251 reaching the final performance of −0.315. Furthermore, it also outperforms all other algorithms252 in terms of adaptation speed. We additionally checked whether the single-task baseline eventually253 overtakes FAMP and found that after more than 200 episodes, its performance stabilizes at a final254 discounted return value of −0.284. This demonstrates that FAMP can learn sub-policies and termi-255 nations that allow for fast adaptation in similar unseen environments at the cost of slightly lower256 asymptotic performance. An example trajectory that was produced by the agent in one of the hardest257 test tasks is displayed in Figure 1. In this task, the agent is able to combine three options to form an258 optimal solution. Plots with meta-training curves and learned options are included in Appendix C.259
In Figure 2 (right), we show how the performance varies with changes to important hyperparameters,260 namely, the number of options and adaptation steps. We observe that decreasing the number of261 adaptation steps during training to one leads to a clear drop in performance. This can be attributed to262 the policy not being able to switch from exploratory to exploitatory behavior in a single inner update263 as well as the smaller amount of data observed before each outer update.264
Unlike the number of adaptation steps, the number of options does not seem to affect the performance265 too much. The only noticeable exception is lower performance when using only 2 options. This266 exception can be explained by noticing that in some states one needs to perform 3 different actions267 to represent all optimal paths. As an example, consider the state two squares above the blue special268 state in Figure 1. To reach the blue state in the minimum number of steps the agent needs to use the269 down action. Similarly, to go from the blue state to the red or yellow one it needs to use up and right270 respectively. Thus the agent cannot represent the optimal policies with only 2 options. Interestingly,271 even in this case, the agent is still able to separate trajectories in such a way that it can reach all goals272 albeit with slightly worse performance.273
This outcome demonstrates another benefit of learned option lengths as the optimal option length274 does not only depend on tasks and their difficulty but also on the number of options that are available.275 To illustrate this, consider an extreme case where there are as many options as tasks. In this case, it276 would be sensible to have solution to a different task in each option and not terminate at all because277 each task would be solved with only one high-level action. However, as the number of available278 options decreases, sharing options between tasks becomes necessary and terminations should start to279
Table 1: Percentage of terminations in trajectories obtained from adapted policies averaged over 5 seeds. Standard deviations are in 1-2% range.
Number of options 2 3 4 5 8 16 Avg. terminations in traj 70% 63% 57% 55% 44% 28%
Figure 3: Ant maze tasks. The agent needs to control a simulated 4-legged ant-like robot and move it towards the green square.
occur to allow for all tasks to be solved. Moreover, if the number of options is decreased even further,280 there may not be enough to options to capture the optimal behavior for all tasks. Consequently, it281 becomes even more difficult to choose the appropriate option length a priori since it can depend on282 the number of available options. To confirm this intuition, we ran a followup experiment with longer283 time horizon in the taxi environment. While the trajectories produced by adapted policies had similar284 length, relative number of terminations decreased with the increase in the number of options as shown285 in Table 1.286
5.2 Ant Maze287
In our second experiment, we demonstrate the applicability of our method to more complex environ-288 ments. We use the family of ant maze tasks introduced by Frans et al. [17] shown in Figure 3. This289 allows us to reproduce the results of MLSH as closely as possible by mirroring the setting used in290 the original paper. In addition to MLSH, we also compare to RL2, a non-hierarchical meta-learning291 algorithm designed for fast-adaptation and Proximal Policy Optimization (PPO) [43], which serves292 as a strong single-task baseline.293
In each task the agent needs to move a simulated 4-legged ant-like robot through a small maze towards294 the goal. Both state space and action space are continuous with 29 and 8 dimensions respectively295 and each episode lasts 1000 timesteps. States do not contain any information about the maze layout296 or the location of the goal. The original implementation also resets the orientation of the ant every297 200 steps. However, we removed these resets because they made the MDP partially observable,298 introduced discontinuities and were not realistic for the robotics scenario they are supposed to imitate.299 Results of experiments with the original implementation are similar to the ones we present. They can300 be found in Appendix C along with meta-training plots.301
Both FAMP and MLSH use the same architecture with two hidden layers of 64 nodes to represent the302 high-level policy, sub-policies and terminations (only applies to FAMP). We used existing repositories303 for the implementation of RL2 [18] and PPO [1]. Hyperparameter values can be found in Appendix304 B. During the training phase, sub-policies (and terminations) of both hierarchical algorithms were305 trained on all tasks until the return averaged over all environments stopped improving. In the test306 phase all parameters except for the policy over options were frozen. Similarly, RL2 was pre-trained307 on all tasks and subsequently evaluated while PPO was trained from scratch.308
The comparison of the performance and speed of adaptation can be seen in Figure 4 (left). Our309 method achieves superior performance reaching an average return of 1330. We also observed a310 similar trend across individual environments. Plots of these comparisons are available in Appendix C.311 While the zero-shot performance of RL2 is slightly better than FAMP, it often struggles to further312 adapt to specific tasks and quickly gets outperformed by both hierarchical methods. This is likely be313
due to the objective that optimizes average return over all training episodes and not post-adaptation314 performance directly. Lastly, PPO continuously improves but its performance does not come close to315 the meta-learning algorithms. After about 1000 episodes it reaches the performance of MLSH and if316 ran sufficiently long , we would expect that it would eventually catch up to FAMP.317
We visualize the option usage of FAMP on two example tasks in Figure 4 (right). After the high-level318 policy is fine-tuned, we use the x and y positions of the ant in 3 sampled trajectories to highlight319 which option is active at each part of the state space. Although we only take 2 out of 29 dimensions320 into account, we are still able to get useful insight about the learned option structure. In the task that321 is depicted in the left part of the plot, the agent uses the blue option before switching to cyan in the322 middle and finishing with a combination of blue and purple. On the other hand, in the right task, the323 agent uses a combination of blue and purple to move down instead of to the right. This shows that the324 agent learned a useful abstraction that allows it to perform two different useful behaviors in similar325 parts of the state space by using terminations and different options.326
6 Discussion and Future Work327
In this work, our aim was to learn both sub-policies and terminations of options by using a single328 simple principle: options should accelerate adaptation in many tasks. We proposed a method for329 learning hierarchical policies that combines the options framework with gradient-based meta-learning330 and explicitly optimizes for performance after several adaptation steps. In our experiments, we331 have demonstrated the benefits of our approach in quickly learning previously unseen test tasks.332 Furthermore, we have shown that the proposed method outperforms the closest hierarchical and333 non-hierarchical meta-reinforcement learning methods designed for similar setting in a challenging334 multi-task learning scenario.335
The computation limitations of our method are mostly connected to the calculation of responsibilities336 in IOPG. In this calculation, many sequential matrix multiplications are required both in the forward337 and backward pass. The compute time for each update is thus dependent on the trajectory length338 because these calculations cannot be done in parallel. One direction for future work could thus be339 alleviating this limitation.340
Our objective does not explicitly constrain the number of terminations as long as they lead to fast341 adaptation. Thus, there are many combinations of options with different lengths which can lead to342 good performance on all tasks, which do not always correspond to intuitive decompositions. One343 possible cause of spurious terminations lies in the continuous state space used in some tasks. When344 neural networks are used to represent termination functions, they learn to generalize to nearby states.345 In tasks such as the ant maze, the agent will visit many states in the same neighborhood and might346 thus terminate options several times in quick succession. A promising topic for future investigation is347 whether this problem could be alleviated by using terminations that also depend on the state in which348 the option was initiated.349
References350 [1] Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.351
[2] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter352 Abbeel. Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environ-353 ments. In International Conference on Learning Representations, 2018.354
[3] Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAML. In355 International Conference on Learning Representations, 2019.356
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. Proceedings357 of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.358
[5] Andre Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel,359 Daniel Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The op-360 tion keyboard: Combining skills in reinforcement learning. In H. Wallach, H. Larochelle,361 A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Informa-362 tion Processing Systems, volume 32. Curran Associates, Inc., 2019.363
[6] Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference364 for determining options in reinforcement learning. Machine Learning, 104(2-3):337–357, 2016.365
[7] Peter Dayan and Geoffrey E Hinton. Feudal Reinforcement Learning. In S. J. Hanson, J. D.366 Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages367 271–278. Morgan-Kaufmann, 1993.368
[8] Thomas G. Dietterich. Hierarchical Reinforcement Learning with the MAXQ Value Function369 Decomposition. Journal of Artificial Intelligence Research, 13(1):227–303, 2000.370
[9] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep371 Reinforcement Learning for Continuous Control. In Proceedings of The 33rd International372 Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research,373 pages 1329–1338. PMLR, 2016.374
[10] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2:375 Fast Reinforcement Learning via Slow Reinforcement Learning. CoRR, abs/1611.02779, 2016.376
[11] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is All You377 Need: Learning Skills without a Reward Function. In International Conference on Learning378 Representations, 2019.379
[12] Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and Regularization380 in DQN. CoRR, abs/1810.00123, 2018.381
[13] Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded DiCE: Trading off Bias and382 Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning. In383 Advances in Neural Information Processing Systems, volume 32, pages 8151–8162. Curran384 Associates, Inc., 2019.385
[14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast386 Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine387 Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126–1135. PMLR,388 2017.389
[15] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and390 Shimon Whiteson. DiCE: The Infinitely Differentiable Monte Carlo Estimator. In Proceedings391 of the 35th International Conference on Machine Learning, volume 80 of Proceedings of392 Machine Learning Research, pages 1529–1538. PMLR, 2018.393
[16] Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-Level Discovery of Deep394 Options. CoRR, abs/1703.08294, 2017.395
[17] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta Learning Shared396 Hierarchies. In International Conference on Learning Representations, 2018.397
[18] The garage contributors. Garage: A toolkit for reproducible reinforcement learning research.398 https://github.com/rlworkgroup/garage, 2019.399
[19] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised400 Meta-Learning for Reinforcement Learning. CoRR, abs/1806.04640, 2018.401
[20] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-402 Reinforcement Learning of Structured Exploration Strategies. In Advances in Neural Informa-403 tion Processing Systems, volume 31, pages 5302–5311. Curran Associates, Inc., 2018.404
[21] Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When Waiting Is Not an405 Option: Learning Options with a Deliberation Cost. Proceedings of the AAAI Conference on406 Artificial Intelligence, 32(1), 2018.407
[22] Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina408 Precup. The Termination Critic. In Proceedings of Machine Learning Research, volume 89 of409 Proceedings of Machine Learning Research, pages 2231–2240. PMLR, 2019.410
[23] Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli, N Siddharth, Wendelin411 Boehmer, and Shimon Whiteson. Multitask Soft Option Learning. In Proceedings of the412 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of413 Machine Learning Research, pages 969–978. PMLR, 2020.414
[24] George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement415 learning. In Proceedings of the 20th International Joint Conference on Artificial Intelligence,416 pages 895–900, 2007.417
[25] Alexander Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy Adaptation for418 Hierarchical Reinforcement Learning. In International Conference on Learning Representations,419 2020.420
[26] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for421 Few Shot Learning. CoRR, abs/1707.09835, 2017.422
[27] Hao Liu, Richard Socher, and Caiming Xiong. Taming MAML: Efficient unbiased meta-423 reinforcement learning. In Proceedings of the 36th International Conference on Machine424 Learning, volume 97 of Proceedings of Machine Learning Research, pages 4061–4071. PMLR,425 2019.426
[28] Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. A Laplacian framework for427 option discovery in reinforcement learning. In Proceedings of the 34th International Conference428 on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2295–429 2304. PMLR, 2017.430
[29] Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Adaptive skills adaptive partitions431 (asap). In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in432 Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.433
[30] Amy McGovern and Andrew G. Barto. Automatic Discovery of Subgoals in Reinforcement434 Learning Using Diverse Density. In Proceedings of the 18th International Conference on435 Machine Learning, ICML ’01, page 361–368, 2001.436
[31] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-Cut — Dynamic Discovery of Sub-Goals437 in Reinforcement Learning. In Proceedings of the 13th European Conference on Machine438 Learning, ECML’02, page 295–306. Springer-Verlag, 2002.439
[32] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive440 Meta-Learner. In International Conference on Learning Representations, 2018.441
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan442 Wierstra, and Martin A. Riedmiller. Playing Atari with Deep Reinforcement Learning. CoRR,443 abs/1312.5602, 2013.444
[34] Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchi-445 cal Reinforcement Learning. In Advances in Neural Information Processing Systems 31, pages446 3303–3313. Curran Associates, Inc., 2018.447
[35] Scott Niekum and Andrew Barto. Clustering via Dirichlet Process Mixture Models for Portable448 Skill Discovery. In Advances in Neural Information Processing Systems, volume 24, pages449 1818–1826. Curran Associates, Inc., 2011.450
[36] Marc Pickett and Andrew G. Barto. Policyblocks: An algorithm for creating useful macro-451 actions in reinforcement learning. In Proceedings of the Nineteenth International Conference452 on Machine Learning, pages 506–513. Morgan Kaufmann, 2002.453
[37] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid Learning or Feature454 Reuse? Towards Understanding the Effectiveness of MAML. In International Conference on455 Learning Representations, 2020.456
[38] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient Off-457 Policy Meta-Reinforcement Learning via Probabilistic Context Variables. In Proceedings of the458 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine459 Learning Research, pages 5331–5340. PMLR, 2019.460
[39] Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning Abstract Options. In Advances in461 Neural Information Processing Systems 31, pages 10424–10434. Curran Associates, Inc., 2018.462
[40] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal463 Meta-Policy Search. In International Conference on Learning Representations, 2019.464
[41] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust465 Region Policy Optimization. In Proceedings of the 32nd International Conference on Machine466 Learning, volume 37 of Proceedings of Machine Learning Research, pages 1889–1897. PMLR,467 2015.468
[42] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.469 High-Dimensional Continuous Control Using Generalized Advantage Estimation. CoRR,470 abs/1506.02438, 2015.471
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal472 policy optimization algorithms. CoRR, abs/1707.06347, 2017.473
[44] Matthew Smith, Herke van Hoof, and Joelle Pineau. An Inference-Based Policy Gradient474 Method for Learning Options. In Proceedings of the 35th International Conference on Machine475 Learning, volume 80 of Proceedings of Machine Learning Research, pages 4703–4712. PMLR,476 2018.477
[45] Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel,478 and Ilya Sutskever. The importance of sampling in meta-reinforcement learning. In Advances479 in Neural Information Processing Systems, volume 31, pages 9280–9290. Curran Associates,480 Inc., 2018.481
[46] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A482 framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):483 181–211, 1999.484
[47] Yee Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell,485 Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In486 Advances in Neural Information Processing Systems, volume 30, pages 4496–4506. Curran487 Associates, Inc., 2017.488
[48] Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. In G. Tesauro,489 D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems,490 volume 7. MIT Press, 1995.491
[49] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012492 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.493
[50] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg,494 David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement495 Learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70496 of Proceedings of Machine Learning Research, pages 3540–3549. PMLR, 2017.497
[51] Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos,498 Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn.499 CoRR, abs/1611.05763, 2016.500
[52] Shangtong Zhang and Shimon Whiteson. DAC: The Double Actor-Critic Architecture for501 Learning Options. In Advances in Neural Information Processing Systems, volume 32, pages502 2012–2022. Curran Associates, Inc., 2019.503
[53] Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M. Hospedales. Investigating504 Generalisation in Continuous Deep Reinforcement Learning. CoRR, abs/1902.07015, 2019.505
[54] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast506 Context Adaptation via Meta-Learning. In Proceedings of the 36th International Conference on507 Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7693–7702.508 PMLR, 2019.509
Checklist510
1. For all authors...511
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s512 contributions and scope? [Yes]513
(b) Did you describe the limitations of your work? [Yes] See Section 6514 (c) Did you discuss any potential negative societal impacts of your work? [No] We propose515 a general meta-reinforcement algorithm that does not have any foreseeable negative516 social impact517
(d) Have you read the ethics review guidelines and ensured that your paper conforms to518 them? [Yes]519
2. If you are including theoretical results...520
(a) Did you state the full set of assumptions of all theoretical results? [N/A]521 (b) Did you include complete proofs of all theoretical results? [N/A]522
3. If you ran experiments...523
(a) Did you include the code, data, and instructions needed to reproduce the main experi-524 mental results (either in the supplemental material or as a URL)? [Yes] In supplemental525 material526
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they527 were chosen)? [Yes] Some training details are given in Section 5, the rest is provided528 in Appendix B529
(c) Did you report error bars (e.g., with respect to the random seed after running experi-530 ments multiple times)? [Yes]531
(d) Did you include the total amount of compute and the type of resources used (e.g., type532 of GPUs, internal cluster, or cloud provider)? [Yes] Included in Appendix B533
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...534
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the codebases535 and works that introduced environments we use in Section 5536
(b) Did you mention the license of the assets? [No] We used publicly available code537 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]538 We include our codebase in the supplemental material and will make a public github539 repository540
(d) Did you discuss whether and how consent was obtained from people whose data you’re541 using/curating? [No] We used publicly available code542
(e) Did you discuss whether the data you are using/curating contains personally identifiable543 information or offensive content? [N/A] We use data from virtual environments544
5. If you used crowdsourcing or conducted research with human subjects...545 (a) Did you include the full text of instructions given to participants and screenshots, if546 applicable? [N/A]547 (b) Did you describe any potential participant risks, with links to Institutional Review548 Board (IRB) approvals, if applicable? [N/A]549 (c) Did you include the estimated hourly wage paid to participants and the total amount550 spent on participant compensation? [N/A]551

Review summaries:
This paper proposes to use meta reinforcement learning in order to find a higher-level policy that can quickly update to achieve higher rewards and new tasks. The method uses the options framework in this case and uses meta-learning to find and quickly update the option selection policy given only a little bit of data from the environment. The method does show some improved performance using this particular structure over two different environments, one that was used in the previous work.  Pros - The Meta-learning structure that's described in the paper does appear to be rather effective and somewhat novel. It shows some similarities to having a high-level policy that is more of a latent policy in this case in the middle learning performs a gradient step in order to adjust which options end up getting selected for a particular task. But the overall method is different enough from prior methods. - The described method in the paper is also not significantly complicated and seems to fit in fairly well to previous hierarchical reinforcement learning designs in particular for using an options type setup that includes a termination function.   Cons - While the results look promising they are applied to a limited number of environments. One environment being a discreet environment and the other being the environment from the MLSH paper. Well, potentially not necessary it could still help bolster the thoroughness of the findings in the paper if the evaluation is performed over additional environments. For example using some of the environments from meta world or other hierarchical reinforcement learning papers. - Well the paper does outline its novel design structure for applying the meta-adaptation step on the higher-level policy there isn't any ablation analysis in the paper to really understand the most important components of the overall method. For example, additional analysis that includes all of the levels of the hierarchy inside of the inner meta adaptation step would improve the clarity of the experiments and the solidity of the findings. Overall the paper seems to be on solid ground but the additional analysis would add to the contributions of the paper.

The paper presents a meta-learning approach for learning hierarchically structured policies, where the agent is trained on some distribution of similar tasks to maximize the post-adaptation performance, as well as learning useful sub-policies in different time scales that enable fast adaptation. The overall model contains a high-level policy selecting over a set of options, and each option contains a sub-policy controlling action selection and a termination probability controlling the duration of low-level policy execution.  During training, the authors argue that the update for low-level policies and termination probabilities should be slower than the update for the high-level policy, to potentially encourage learning sub-policies that can be used in multiple tasks. This is the key idea of the proposed fast adaptation of modular policies (FAMP) algorithm. The experiments on Taxi and Ant Maze environments show FAMP can fast adapt and achieve better returns than other baseline meta-learning methods on unseen test tasks. 

This paper proposes an option method where the high-level agent is a meta-learning agent that can fast learn to select proper options through sampled few episodes. DiCE is used to correct the gradients. The paper demonstrates the effectiveness of the approach on Taxi and Ant Maze environments and outperforms RL^2 and PPO baselines.