Reusable Options through Gradient-based Meta Learning
Authors: David Kuric
Herke van Hoof
Venue: Accepted by TMLR ()

Abstract: Hierarchical methods in reinforcement learning have the potential to reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding a reusable useful temporal abstractions that facilitate fast learning remains a challenging problem. Recently, several deep learning approaches were proposed to learn such temporal abstractions in the form of options in an end-to-end manner. In this work, we point out several shortcomings of these methods and discuss their potential negative consequences. Subsequently, we formulate the desiderata for reusable options and use these to frame the problem of learning options as a gradient-based meta-learning problem. This allows us to formulate an objective that explicitly incentivizes options which allow a higher-level decision maker to adjust in few steps to different tasks. Experimentally, we show that our method is able to learn transferable components which accelerate learning and performs better than existing prior methods developed for this setting. Additionally, we perform ablations to quantify the impact of using gradient-based meta-learning as well as other proposed changes.

Published in Transactions on Machine Learning Research (03/2023)
Reusable Options through Gradient-based Meta Learning
David Kuric
d.kuric@uua.nt
AMLab, University of Amsterdam
Herke van Hoof
h. w.c.anhhoo@@uoo..
AMLab, University of Amsterdam
Reviewed on OpenReview: https: // openreview. net/ forum? id= id= gdDmazCuzu
Abstract
Hierarchical methods in reinforcement learning have the potential reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding reusable useful temporal abstractions that facilitale fast learning remains challenging problem. Recently, several deep learning approaches were proposed to learn such temporal abstractions in the form of options in an end-to-end manner. In this work, we point out several shortcomings these methods and discuss their potential negative consequences. Subsequently, we formulate the desiderata for reusable options and use these to frame the problem of learning options as gradient-based meta-learning problem. This allows us to formulate objective that explicitly incentivizes options which allow higher-level decision maker to adjust in few steps different tasks. Experimentally, is able to learn transferable components which accelcrate learning and performs better than existing prior methods developed for this setting. Additionally, we perform ablations to quantify the impact of using gradient-basd meta-learning as well as other proposed changes.
show that method
1 Introduction
In hierarchical reinforcement learning (HRT.), an agent various levels of abstraction to simplify learning in large complex environments. This is commonly achieved by utilizing multi-level policy which can be broken up into making high-level decisions (e.g,, which out of several low level controllers should be active) and low-level decisions (which actions should a low level controller take). Many such approaches utilize temporal abstraction, ic., the notion that high-level actions are only chosen at cortain time steps and then continue to influence the decision making over multiple time steps in the future. clear benefit of utilizing such high-level decisions the abstraction of the problem a simpler one and consequently, easier learning. For example, consider a navigation problem where the agent needs to travel to city on the other side of the country. In this problem, it is much casier to learn which route to take if the agent considers few high- level decisions: sequence of cities, rather than low-level actions: sequence of intersections. However, when solving this problem with high-level actions, a mapping from high-level actions to corresponding sequence of low-level actions that move the agent from one city to another needs to be available. This mapping can be provided as a form a prior knowledge, but in most settings this information is not available. Therefore, it is often necessary learn such mapping from experience and then reuse it when learning similar new tasks.
In order to study these types of problems in the context of reinforcement learning, one can utilize the options framework (Sutton et al. 1999). In this framework, hierarchical policy is composed of options (modules that encapsulate low-level sub-policies), and high-level policy that chooses among them. Options have their own termination function, and control is given back to the high-level policy only when the earlier option terminates. Therefore, options define temporally extended behaviors through their sub-policies and can be seen as a mapping from high-level actions to a sequence of low-level actions.
Recent prior work on options proposed way to learn the options in an end to end manner from experience with policy gradient methods (Bacon et al. 2017 Smith et. al.) 2018). These methods learned the options in

---

Published in Transactions on Machine Learning Research (03/2023)
the context of single task. However, options learned by single-task algorithms are not incentivized to be reusable, and thus often end up too task-specific to be used in related tasks (Harb et al. 2018
To overcome this issue, the followup work by Frans et al. (2018) proposed to learn shared options in multiple tasks by alternating between fine-tuning of the high-level policy to the sampled task and joint optimization of both high-level policy and options. While this optimization scheme worked well in practice, it does not pass any gradient signal between high-level policy and options during the training. Furthermore, even though the options produced by this method were more suited for the subsequent learning of new tasks, it used options that terminate after fixed number of low-level actions. Such adjustment simplifies the learning problem because it fixes termination function. However, the need to define a shared option length can be prohibitive because different suitable high-level actions may have very different lengths. Additionally, these lengths can also depend on the randomness in the environment. In such cases, the length of high-level actions cannot. be captured with a single hyperparameter. Moreover, the value of this hyperparameter is not learned and thus requires some prior knowledge about the tasks.
In this work, we identify the shortcomings of current methods and their ability to learn shared reusable options from multiple tasks in a setting where limited to no prior knowledge about tasks is available. We discuss the desired properties of such shared reusable options and use these properties to formulate this learning problem as a concrete meta-learning problem. This formulation then allows us to connect this problem to gradient-bassee meta-learning and to propose a method that adapts a well-known meta-learning algorithm (Finn et al.] 2017)) to learn reusable options from multiple tasks. This new method provides a more principled alternative to optimize the meta-learning objective when compared to prior work developed for this setting (Frans et al. 2018). Additionally, it allows one to use state-dependent terminations and learn multiple options from the same experience. In our experiments, based meta-learning and learned terminations contribute to learning reusable options that achieve better performance when applied on new unseen tasks from the same domain.
empirically verify that both gradient-
2 Background and Notation
We will consider environments which are episodic Markov decision processes (MDPs). An MDP M is a tuple (S, A, Po. Pe, R, 7) with S being a set of states, A a set of actions, (00) a probability distribution of initial states, pa(s'(s. s, a) a transition probability distribution, R(s, a) a reward function and yy a discount factor.
An agent with a stochastic policy я interacts with an environment the following way. At every timestep I, the agent receives a state of the environment 8, E S and selects an action a, E A according to a policy <a,]s, 0) parametrized by 0. Depending on the current state and the action performed, the en- viromment provides the agent with new state 8:41 Pa(8t+1/8t az) and scalar reward r. R[st, at). This process is repeated until a so-called terminal state is reached. We define a trajectory as an or- dered sequence of all states actions and rewards in a single episode t = (8o, ag, ro. 8r))....)..... Similarly, the history at timestep t consists of all states and actions preceding a,, h, = (sp, ag....,.)) 8t). The state value function is defined as V*(()= = Eysp(r))) [G)()) = s] where the discounted return at timestep I is de- fined Ge(T)- "T=,,('' (t'--re- The agent's objective is to maximize the expected discounted return J = Ey<p(r|>) [Go(T)]- We can maximize the objective with gradient descent by estimating the policy gra- dient VoJ s (pp En= V log π ((a 8,, (s a:)l using Monte Carlo sampling, where A is an advantage estimator such as the generalized advantage estimator AGAE (Schulman et al. 2015).
2.1 Options
The options framework (Sutton et al. 1999) is framework for temporal abstraction in reinforcement learning that consists of options w (I", Th _ and policy over options a ( 0z). Each option w has an initiation set, a sub-policy and a termination function. The initiation set Ie is a set of states in which an option can be selected (initiated) and in our case it is the whole state space (II = S). A sub-policy (alss, Ow), also called low-level policy, is a regular policy that acts the environment. Lastly, the termination function ed (s. 0&) is a function that outputs the probability of the termination for option w in a given state. We denote the parameters of policy over options, sub-policies and termination functions as Og O., and 0g respectively.
2

---

Published in Transactions on Machine Learning Research (03/2023)
In our work, we use the Inferred Option Policy Gradient (IOPG, Smith et al. (0018) ), recently introduced
policy gradient method for learning options that treats options as latent variables during gradient calculation.
This method allows for updating all options based on their responsibilities for a given action which results in better data efficioncy. The IOPG gradient can be calculated from sampled trajectories:
VoJ x ()
Σ Vo log x (ailh, () as)
(1)
Σ p(wi)sson a10.c-1], 1) (as 181, Ow))
where TI T(ar[he 0) can be decomposed as indicated in Equation 1 and we use to denote the concatenation of all parameters (On, Ow 0g). The probability p(w] ((o:t) d[o:t-1]: 0) represents the probability that option wf was active at timestep t given past actions and states and can be computed recursively:
plu/+1 1800+41) ((o4)] 0) = Ens p(wf 810.), "10:--11 11. 0) (a. (arlsr, 81, Ow) ( (1f, 81+1, O2 0c)
(2)
Ene p(wff [(or)] aloa-1), θ)T= (a, (,,
with option transition probability # (w/41 lwy s1+1, On. 05) given by:
* su+1, 6n, 0-) = E (st+1, 0g) 8:41, On) + Iw [] eu' (8)+1
(3)
2.2 Model-Agnostic Meta-Learning (MAML)
Model-Agnostic Meta-Learning Finn et al. 2017) ) is a meta-learning technique that trains a model for maximum post-adaptation performance on a distribution of tasks M ~ p(M). The adaptation consists of one or several inner gradient updates. If we consider an estimator fo with parameters 0 and a task-specific loss L a supervised learning objective with a single inner update can be formalized as shown in Equation In order to optimize this objective one only needs to take gradient of this expression. This can be easily achieved with automatic differentiation frameworks by backpropagating through the gradient update.
min EM [LM (Fo-avocM(oo)]
(4)
One can similarly use this approach with a reinforcement learning objective:
max EM
E [Brr Vor
(GM(()))
(5)
However, the implementation with an automatic differentiation framework differs because simple backprop- agalion through the computation graph of the gradient produces biased gradients (Al-Shedival et al.] 2018 Stadie et al. 2018 This is due to an additional dependency of the sampling distribution on parameters that is not present in the supervised lcarning objective. To produce correct higher order gradients with automatic differentiation frameworks we utilize an objective with DiCE (Foerster et al. 2018 Farquhar et al. 2019
3 Reusable Options
Despite extensive research on options, there is not yel a single answer to the question: What is a good oplion? Consequently, there is also no consensus on the objective for an option learning method. In this paper we take the view that good options should accelerate the learning of new tasks. In other words, we consider options good if they allow us to achieve high return on tasks fast possible. We assume that we are given a distribution of relevant training tasks p(M) with similar (hierarchical) structure but different reward or transition functions. Our goal is then to learn options that maximize expected return on this task distribution as well as the expected return achieved on unseen tasks from the same task distribution.
1 For more details about. DICE Appendix
3

---

Published in Transactions on Machine Learning Research (03/2023)
One could formulate this as maximizing MM--((M) [E)p((()(()) using a hierarchical policy with options parametrized by 0. However, maximizing immediate return with such objective directly is not beneficial because it learns a single fixed policy that solves multiple tasks. The solutions to these tasks can have conflicting requirements as shown in Figure | where some tasks require different decisions in certain states. Consequently, a single trained policy would likely be a compromise that performs reasonably well, but sub-optimally, on each individual task. Since the learning of a hierarchical policy that would lead to good zero-shot performance is not feasible, we instead aim for shared options that capture reusable sub-behaviors. Such options should achieve good performance after the policy over options adapts to the task at hand. This objective can be formulated as:
[+)] where trajectories are generated
max E MM-((M) 0, 0 arg max (r=p(T)=(0 0n)) (GM(()]
E Prmp(TIn(0..00000 03)) [ [+)]
(6)
(7)
Ba
where 0 is a high-level policy adapted to the specific task M. While one could also perform maximization over all parameters in Equations G and 7 such approach would not enforce options that capture reusable sub-behaviors. This is because the algorithm can choose to not change the high-level policy and instead make a large change in sub-policies during the adaptation. Nevertheless, we consider this variant in our experiments as an ablation.
The two-level optimization problem in Equations 6 and 7 consist of inner maximization over the parameters of the high-level policy On, which are used to select options, and outer maximization over option parameters 0g. Consequently, it is not trivial to optimize with commonly used methods such as the gradient descent because of two reasons. Firstly, in order to get the gradient of the outer objective, one needs to differentiate through the inner maximization in Equation and secondly, the inner optimization problem is not trivial to solve and may require tens or hundreds of gradient steps to optimize. However, we can nullify the latter by simplifying the problem and restricting the number of inner adaptation gradient steps to a fixed number L:
max EMp(MM) [ i [T)] 0
(8)
0 02 tamn V E Vog Bxsmp((11((0000000)))))) (G)())))
(9)
This adjustment also fixes the diffcrentiability of the inner optimization problem because Equations S and
are conceptually similar to MAML applied to reinforcement learning (Equation 5) The two differences
with the standard MAML are multiple inner updates L and limiting inner optimization to a subset of all parameters On. Such adjustments are well supported in the MAML framework. Finn et al. (2017) used multiple inner updates and as was demonstrated by Zintgraf et al. (2019) and [Antoniou et al. (2019). reducing the number of parameters that are tuned in the inner updates can be beneficial and lead to better performance. Consequently, we propose to use this MAML-like objective with a policy gradient method to learn reusable options. Note that it is also possible to learn the initial high-level policy parameters by performing the maximization over 0g in Equation to facilitate exploration during both training and test phases and to encourage more uniform option usage. Additionally, it was shown that fixed initialization of inner loop MAML parameters in supervised learning setting makes the algorithm more robust to the choice of inner learning rate (Zintgraf et al. 2019)
g However, we chose fixed uniform initial high-level policy
Another important questions from the options research are: How many options should one use? and When should termination occur?. Although these do not seem to be connected at first glance, we show that in the task of learning options from multiple environments, there is link between these two questions. This link is demonstrated by the example in Figure] | In this example, we consider four tasks with the same starting location but different goals. The optimal trajectories for each task lead from the starting location straight to the goal. We assume that learned options should be as long as possible to simplify the optimization of high-level policy when faced with a task. This assumption is based on the premise that learning problems that contain fewer decisions lhigh-level actions) should be casier to solve. Consequently, the adaptation to new tasks with longer options should also be faster as long as some of the learned sub- behaviors can be used in new tasks.
4

---

Published in Transactions on Machine Learning Research (03/2023)
G
G
G2
G
2
2
G2
$
0
Figure 1: Example in which optimal trajectories in multiple tasks cannot be represented with a single shared policy. The maximum length of options that can represent such optimal trajectories changes with number of options. Options are shown in different color. Necessary terminations are depicted as a filled square.
If we set the number of options equal to the number of tasks, each sub-policy can learn to solve task and there is no need to combine multiple options. Therefore, the longest options are options without terminations. However, once we start to decrease the number of options, parts of trajectories have to be reused to represent optimal trajectories of 4 tasks with only 3 options. This is demonstrated in the middle of Figure [ where only one option needs to terminate after several timesteps. Similarly, when using only 2 options, more lerminations are needed and the options tend to become shorter. In addition to showing the connection between the amount and positions of desirable terminations and number of options, this example also highlights why choosing a hyperparameter that represents predetermined length of options, in number of prior approaches (Frans et al. 2018 Li et al. 2020), can be challenging. This is because the appropriate hyperparancter value can change based on the number of available options and is usually not known apriori. Additionally, such an approach would most likely not work well in environments where movement actions can fail with certain probability because agent could arrive into a state in which it needs to terminate after taking different amount of steps.
done
These two examples demonstrate why using terminations that depend on current state as proposed in the original options framework work (Sutton et al.) 1999) may be preferred. To this end we propose to learn reusable options (including terminations) from multiple tasks by combining the Inferred Option Policy Gradient (IOPG) with the adapted version of MAML. Similar to prior work, we assume that mumber of options is given as a hyperparameter. The objective we propose in Equations S and g then implicilly discourages pathological options. For example, the scenario where only a single option is ever used (option collapse) is discouragod as the agent that only uses a single option to solve multiple tasks would achieve lower average return. Similarly, the scenario where options terminate every time step is discouraged because by optimixing for fast adaptation of high-level policy; the algorithm is incentivized too short (degenerate 1-step options). This is because options that are too short would not allow high-level policy to fully adapt to new tasks within several inner updates and would also lead to lower average return.
use options that are not
TOPG also allows us to learn terminations and to update all options at the same time based on their responsibilities, i.e., the probability that the option was active given the history h, of states and actions so far. This can lead to better data-efficiency when compared other methods that only update a single option at a time, such as the Option Critic (Bacon et al. 2017p, but comes at the cost of increased computation time. Additionally, IOPG is also not reliant on learned option-value function which is helpful in the meta learning setting because the option-value function changes after every inner update. This is in contrast with the Option Critic and several recent methods that extend it (Khetarpal et al. 2020; Klissarov & Precup (2021). Combining these methods with MAML would thus be difficult. Additionally, there is no straightforward way to apply these algorithms in a setting with multiple tasks when the agent does not know which environment is in.
3.1 Fast Adaptation of Modular Policies
The resulting algorithm for Fast Adaptation of Modular Policies (FAMP) is outlined in Algorithm After L inner updates, the gradient of the objective with respect to the outer parameters calculated. In principle, we would like to optimize for performance after a moderate number of gradient updates such as 10 or 20. However, with more inner updates the resulting gradient of the objective becomes noisier due to the usage
5

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm 1: Fast. Adaptation of Modular Policies (FAMP)
input A distribution of tasks p(M), number of options N, env samples per update M, adaptation steps L, episodes per update k, learning rates ainy Cout randomly initialize 0n. 0g. Aw while not converyed do ((.) =0
D zero accumulated gradients
4
for env +- 1 to M do sample task M p(M) set 0¹22 = 0g for I + to L+1 do
6
sample & episodes Tisk M using (0%) Ow. 00) fit baseline using data from Ti:k compute A GGAÆ and log (ar 0/. om, 0₂) for all Tikk compute loss J according to Equation
9
11
12
if I < L + then
13
0/ 0/= Oin Ve J
b adapt π
14
else
O0) 0.) + V Oa)
accumulate gradients of outer objective
16
end
17
end
18
end
19
(0 Ow) = (0g, + aout ow)
> update options
end
output: initial policy over options π (w|s, 0a), sub-policies T/ (a|s,, Ow), terminations (" 0g)
of Monte Carlo estimates in cach inner update. Furthermore, the time complexity of gradient computation and sample complexity both scale linearly with the number of inner updates. In practice we found a range from to 4 update steps to be acceptable. A benefit of gradient-based meta-learning is that even though the model is optimized for performance after adaptation steps, can still improved after updates by performing more steps of gradient descent. An important design choice is the state value function estimator. In the MAML RL setting the policy constantly changes in every inner update. It is thus difficult to use past trajectories for fitting the value function. Furthermore, different value function needs to be computed for linear time-state dependent baseline (Dan et al. 2016a), which works better than more complex baselines with little data.
each task. We therefore
4 Experiments
In order evaluate our method empirically and show the benefits of learned terminations as well end-to- end learning with gradient-based meta-learning, we perform experiments in Taxi and AutMaze domains D compare to the closest state of the art method and perform ablations to highlight
In these experiments, the effects of different objectives and learned terminations are interested in the final average performance on the set of test tasks. Single-task algorithms are trained on these tasks directly. On the other hand, meta-learning algorithms, including our own, are first pre-trained on a set of different but related training tasks until their performance stops improving or they reach one week of training time. We then plot the performance as the function of the number of episodes during the adaptation.
the final performance. In both domains, we
2Qur code publicly available htttp:://github..co///rrrrr

---

Published in Transactions on Machine Learning Research (03/2023)
H
Figure 2: Left: Map of a taxi environment with special states and an example task. Middle and Right: Visu- alization of the option usage in this task. Middle part shows states without passenger on board. Right part shows states with passenger. Arrows represent directional actions, pick-up/drop-off is shown as a square. Each action is colored according to the active option.
4.1 Taxi
In the first set of experiments we use modified Taxi environmen commonly used in works on hierarchical learning and options (Dietterich 2000 Igl et al. 2020) and allows one to create many different tasks with shared parts to test the reusability of learned options. Furthermore, sub- tasks and optimal trajectories have different lengths across tasks and thus may require options with different lengths. Finally, since both state and action space are discrete, the learned options can be visualized and examined. The map of the environment displayed in Figure 2 An agent acts as taxi driver who starts in one of the special (colored) locations. Its goal is to pick-up passenger from one special location and drive him to another special location. Different tasks use different combinations of start, goal and passenger locations. We use 48 combinations as training tasks and 12 as test tasks.
3 Dietterich 2000 This environment is
Each task is an MDP in which the agent can use 4 directional actions and two special actions: pick-up/ hhsp/dro--fff and no-op. The state space represented as a onc-hot vector that does not encode any information about the location of the passenger or gool state, Therefore, in order to facilitate fast adaptation to the (unobservable) passenger and goal locations, the agent must acquire options that can serve building blocks for exploration. The reward is 2 for reaching the goal and -0.1 per step otherwise. Episodes terminate if they take longer than 1500 timesteps. We use tabular representations for the policy over options, terminations and sub- policies. Our experiments are set up to answer following questions:
(1) How does FAMP compare to the closest hicrarchical method in terms of performance? We chose Meta-Learning Shared Hierarchies (MLSH, Frans et al. (2018)) as the strongest meta-learning baseline with options, because it is the closest hierarchical method designed for similar setting in which no extra information about the environment is available. This is in contrast with many other hierarchical and non-hierarchical meta reinforcement learning methods for learning options (Picket & Barto, 2002 Barreto et al. 2019), which utilize extra information such as the ID of sampled environment (Mankowilz el al. 2016 Igl et al. 2020 Veeriah et al.) 2021). However, unlike our method, MLSH uses time-based terminations and does not use gradient- meta-learning to optimize its objective. We pre-train MLSH on all training tasks to learn sub-policies. During the test-time we freeze these sub-policies and only allow the high-level policy to adapt test tasks. This setting is identical to the one used in the original work albeit with different environment. (2) How does FAMP compare to a policy that is not trained for fast adaptation and policy trained from scratch? We use the multi-task bascline to show the performance of hierarchical policy that optimizes for the immediate average return in multiple tasks rather than the return after several adaptation steps. The whole hierarchical policy of this baseline is first meta-learned and then fino-tuned on test tasks. It corresponds to using Algorithm [ with L = and (0 0g. A_)) instcad of (0g. 0.)) in lines 3, 15 and 19 during training. We also use the single-task baseline which learns to solve the test tasks from scratch without any pre-training using TOPG. Therefore, since it does not need to generalize to many tasks and has a flexible policy, we expect it to perform better than other methods after sufficiently long training. However, meta-learned policy with desirable options should find a good solution quicker. (3) Is
Details are included Appendix A

---

Published in Transactions on Machine Learning Research (03/2023)
-0.26
-0.50
-0.35
Disoousted Retum
Disoouated Retum
00.40
1.00-
-0.45
-1.25
-0.50
FAMP (Ous)
Loan all
-1.76
Our FT
Mulli-ttck
-0.55
MI.SH
No Hicrarcly
Leom High level
-200
Eptoodss
Epicodes
Figure 3: Average performance of different algorithms on Taxi test tasks. Plots show mean and 95% bootstrap confidence intervals over 9 sceds.
it important restrict the inner and outer update specific parts of the hierarchical policy? To answer this question, compare the performance of FAMP to learn high-level and learn all baselines. The former also uses hierarchical policy with learned terminations but it learns all initial parameters. This corresponds to adjusting line 15 in Algorithm | to update all parameters (0n, Qg. Ow) Similarly, the learn all baseline also learns all initial parameters but it also additionally adapts the whole hierarchical policy in the inner update step. This corresponds to adjusting both line 13 and 15 Algorithm [ to update all parameters (0n; Og. Ow.) and can be seen as simply optimizing a hierarchical policy with MAML objective. In addition to the two aforementioned ablations, we use no hierarchy baseline evaluate the performance of non-hierarchical policy with MAML objective. This be implemented in practice by using Algorithm with learn all adjustments and 1 option. (4) Are learned terminations important to perform well in this setting? In order to quantify how important learned terminations are in this setting, we use an ablation of our algorithm with fixed options length as a second baseline (Ours + FT). This baseline uses Algorithm to learn the sub-policies but does not learn terminations. Instead, it uses a termination schome of MLSH where the termination occurs every e steps where e a hyperparameter. (5) Does the value of affect the final performance? We vary the option length hyperparameter both MLSH and FAMP to determine whether its value affects the adaptability of learned options and the final performance on test tasks. (6) How does the number of options and inner updates used during training affect the performance? We compare the performance of FAMP with different number of options ranging from 2 to 16 and different number of inner update steps L E (1,2,3}.
Results
The performance comparison with baselines shown in Figure 3 Plots with meta-training curves are in- cluded in Appendix[B] Our method is able to outperform both MLSH and multi-task meta-lcarning basclines reaching final performance of -0.315. In addition to reaching good final performance, FAMP also out- performs other algorithms in terms of adaptation speed. As expected, the single-task baseline eventually overtakes FAMP after more than 200 episodes and reaches a final performance of -0.284. However, reach- ing such performance takes significantly longer than when policies with pre-trained options are used. This demonstrates that FAMP can learn sub-policies and terminations that allow for fast adaptation in similar unscen environments the cost of slightly lower asymptotic performance. An example trajectory that was produced by the FAMP agent in one of the hardest test. tasks is displayed in Figure 2 In this task, the agent is able to combine three options to form an optimal solution.
We further observe in Figure 3 that the learn all ablation, which meta-learns all policy parameters, reaches comparable albeit a bit lower performance than FAMP. This highlights the strength of using gradient-based meta-learning in this problem. However, as we have discussed in Section 3 adapting the whole policy usually does not lead reusable options bocause a lot of the adaptation is done sub-policies which may no longer be feasible in more complex tasks. We revisit this issue in the next sub-section. Trained policies of FAMP and learn all baseline available in Appendix B for comparison. Lastly, we observe that. the performance
8

---

Published in Transactions on Machine Learning Research (03/2023)
-0.4
-0.50
-0.8
Discouated Rocurn
Discousted Retum
1.00
-12
-1.25
1.4
-1.8
Ouss a FT
Ours TT
MLSEL
MLSU1
-1.75
Copte Tem2
oggs
Ours
MLSU
opts
Episodes
Epicodes
Figure 4: Left: Performance of FAMP with fixed terminations and MLSH with terminations after different amount of steps on Taxi test tasks. Right: Average performance of our method with different hyperparamcter values on Taxi Lest lasks. Plots show mean and 95% bootstrap confidence intervals over 9 seeds.
of no hierarchy and learn high-level is lower than when one uses a hicrarchical policy and meta-learns options without high-level policy.
In Figure 4 (left), we show the performance of FAMP and MLSH with different option length c. We observe that the performance of our algorithm is lower when fixed terminations used instead of learned cormsnations. This difference in performance suggests that the lermination scheme with fixed amount of steps is too restrictive for this setting. We further observe that FAMP outperforms MLSH across different values of с and both algorithms perform better with smaller option lengths.
In Figure 4 (right), we show how the performance varies with changes to important hy heprparraneeers, namely, the number of options and adaptation steps. We observe that decreasing the number of adaptation steps during training leads a clear drop in performance especially for L 1. This can be attributed to the policy not being able to switch from exploratory exploitatory behavior in single inner update. Unlike the number of adaptation steps, the number of options does not seem to affect the performance much. This highlights the advantage of using inference-based method (IOPG) for learning options since all options are also updated with experience that was generated by other options.
The only noticeable changes in performance are the extremes with or 16 options. A slight drop in perfor- mance with options can be explained by noticing that in some states, such as the state squares above the blue special state, one needs to perform 3 different actions to follow the optimal path to red, blue and yellow states. Thus the agent cannot represent the optimal policies with only 2 options. Interestingly, even in this case, the agent is still able to scparate trajectories in such way that it can reach all goals albeit with slightly worse performance. This further illustrates how choosing the appropriate option length priori can be difficult since it can depend on the number of available options.
We observed that throughout our experiments, the length options increased with the number of available options. While this is usually desirable property, increasing the number of options too much can negatively affect reusability of options and may require more episodes per adaptation step in settings with sparser rewards. For example, if the initial high-level policy that is used to select sub-policies is close to uniform and one uses too many options, 10 episodes may be insufficient to try various options before performing high-level policy update. Determining which option should be selected and appropriately adapting the high-level policy bocomes difficult such cases. Similarly, using too few options can load to sub-optimal solutions there are too few options to represent. optimal trajectories in all tasks. In such case, the drop in performance may be less or more severe depending on the task.
4.2 AntMaze
In the second experiment, we demonstrate the applicability of our method more complex environments with continuous state and action spaces and perform similar ablations. We use the ant maze domain introduced
9

---

Published in Transactions on Machine Learning Research (03/2023)
Figure 5: Ant maze tasks. The agent needs to control simulated 4-legged ant-like robot and move it from yellow square to the green square. Four rightmost tasks are tests tasks.
by Frans et al. (018)) whose lasks are shown in Figure 5 This domain allows us to perform the best possible comparison with MLSH because we can reuse the code and hyperparameter values chosen by the original authors. The only adjustment is removing the orientation resets of the ant, which occurred every 200 steps, because they introduced discontinuities and were not realistic for the robotics scenario they are supposed to imitate. The results of experiments with FAMP and ML.H the original implementation of the environment were similar and are available in Appendix B
We use 9 tasks for training and test tasks. In each task the agent needs to move simulated MuJoCo (Todorov et al. 2012) 4-logged ant-like robot through small maze towards the goal. Both state space and action space are continuous with 29 and dimensions respectively and the agent receives positive reward for moving closer to the goal location. However, states do not contain any information about the maze layout or the location of the gool. In this settings we consider questions 1-4 from previous environment and verify whether the takcaways from the discrete sctting also translate to continuous sctting. Similar to the experiments in Taxi domain, we use MLSH as closest hierarchical baseline, PPO as single-task baseline, the learn all baseline and FAMP with fixed option length (Ours + FT). Additionally, flat policy to compare to an established meta-learning algorithm without options. We used two hidden layers of 64 nodes to represent the components of hierarchical policies and the policy of PPO. For MAML, we increased the layer sizes to 128. In addition to aforementioned baselines, we have also trained an agent with RL2 (Dan et al. 2016b) on these tasks. Because its performance was comparable to MAML, which is another meta-learning algorithm without options that is directly related our method, we decided to omit RI,2 in the plots to keep them uncluttered. Its meta-training curve can be found in Appendix B For all baselines, we used existing implementations for training and evaluation. Exact hyperparameters and details can be found in Appendix A
use MAML with
Results
The comparison of the performance and the speed of adaptation on both train and test tasks can be seen in Figure 0 In both settings, FAMP outperforms the basclines. However, unlike in the previous experiment, FAMP with fixed option length has lower variance and performs slightly better
training tasks. This is likely because in this environment, the terminations do not have to occur at the exact position as was the case in Taxi. Having pre-determined option length thus does not have such adverse effect on the final performance. We have observed similar trends across individual environments (plots are available in Appendix B). Furthermore, we observe that MAML. is able to sharply increase its performance during the first few steps of adaptation but unable reach the performance of hierarchical methods that utilize options. This suggests that the adaptation of a single policy might not be sufficient to make abrupt enough changes in the agent's behavior that are required to successfully adapt to these tasks. Similarly, meta-learning hicrarchical policy without restricting the adaptation to high-level policy results in low performance. While learn all improves its performance during the adaptation, it is the worst of all meta-learning methods. This is in contrast with the simpler enviroument which its performance was similar to FAMP. Lastly, we observe that while PPO continuously improves, performance does not come close to the meta-learning algorithms after 200 episodes. It takes about 1000 episodes to reach the performance of MLSH and if ran sufficiently long, expect that it would eventually catch up with even outperform FAMP.
10

---

Published in Transactions on Machine Learning Research (03/2023)
1500
1000
750
1000
Recura
Becurs
600
250
FAMP
TAMP (Ours)
P2O
Clurs
FT
Leam All
Ouns FT
Learn All
-250
MLSI
MANI
MISSH
MAML
100 Epinodos
195
150
175
Eyinddos
Figure 6: Left: Average performance of algorithms on ant training tasks (multi-task setting). Right: Average performance of algorithms on ant test tasks (meta-learning setting). Plots show mean and 95% bootstrap confidence intervals over 9 seeds.
While it is not possible to visualize learned options duc to the high dimensionality both state and actions spaces, we can visualize which option active at each part of the stale space. After the high-level policy is fine-tuned, we use the and y positions of the ant in sampled trajectories to highlight the active option. An example option usage of FAMP in different tasks shown in Figure 7 In general, the agent uses the cyan option to move upwards and blue to move towards lower parts of the maze while the combination of options is used to fine-tune the direction. This shows that the agent learned useful abstraction ion that allows it to perform several different useful behaviors in similar parts of the state space by using meta-trained options and by combining them with a finc-tuned high-level policy.
5 Related Work
One of the aims of hierarchical reinforcement learning is to decompose a complex task or policy into simpler units. A thorough overview and classification of existing HRL methods that learn and use such policies can
be found in the recent survey by Pateria al. (2021). Prior approaches include (among others) methods
that learn set of diverse skills (Florensa. et al. 2017 Achiam et al., 2018 Eysenbach al. 2019
methods that rely on the maraee task-decompooiiion of Feudal Reinforcement Learning (Dayan & Hinton 1993 Vezhnevets et al. 2017 Nachum et al.) 2018). The goal of diversity-based methods is to learn skills that lead to different parts of the state space and behaviors in an unsupervised setting. On the other hand, Foudal Reinforcement Learning methods are guided by the reward function and aim to learn a policy in which each level (manager) provides sub-tasks the lower level (worker). However, providing sub-goals architecture leads recursively
can be difficult for some tasks (e.g. run in circle) and manager- optimal policies (Dietterich) 2000). This means that in each level the manager is optimal given its workers but the policy whole may be sub-optimal. Instead, options framework (Sutton et al. 1999).
look at the alternative paradigm of the
Several different strategies have been proposed to learn options within this framework. Some works rely on so-called bottleneck states that can be used as sub-goals (MeGovern & Barto 2001 Menache et al. 2002 Nickum & Barto 2211]] whereas others use spectral clustering (Machado et al. 2017p. These approaches are particularly suited for settings without reward function but often require prior knowledge about the environment which restricts their applicability. Different from aforementioned methods, end-to-end methods such as the ones which rely the Option-Crii Critic architecture (Bacon et al. 2017; Riemer et al.) 2018 Khetarpal et al.] 2020 Klissarov & Procup, 2021 are applicable in more general settings. However, they can be less efficient than concurrently introduced inference-based end-to-end methods (Daniel et al.) 2016 Fox et al. 2017 Smith et al. 2018) because they only update the option that generated the action whereas inference- based methods update all options according to their responsibilities for each action. The recently introduced Multi-updater Option Critic [Klissarov Precup 2021) is notable exception since it allows to update multiple options using the same experience within the Option Critic architecture.
11

---

Published in Transactions on Machine Learning Research (03/2023)
Figure Option usage visualization on ant maze tasks. Plots were created using positions of the ant in trajectories created by adapted policy. Each color represents a different option.
A common problem when using end-to-end methods that learn terminations in a single-task setting is the option collapse (Bacon et al. 2017). It causes the options to terminate after every action or to never terminate. This phenomenon occurs because a single-task can be solved without using options. Consoquently, the policy often does not have sufficient incentive to utilize terminations and learn options that are useful for transfer. Tn such cases the learning of terminations
be facilitated by augmenting the objective with entropy regularization (Smith et al.) 2018) or deliberation cost (Harb et al. 20018), regularizing towards a termination prior [Igl et al. 2020), by using a different objective for terminations (Harutyunyan et al. 2019). Preliminary results also show that the learning of interest functions (Khetarpal el al. 2020) could help to alleviate this issue. In our work, we instead incentivize the algorithm to learn appropriate terminations by training a small amount of options in a multi-task setting. Since the number of available options lower than the amount of tasks, the agent needs to learn to terminate and combine options in order to solve all tasks. While options were sometimes learned in multi-task setting in prior work (Frans al. 2018 Li al. 2020 these works utilized time-based terminations with fixed (Frans et al., 2018) randomized length (Li et al. 2020). However, we have discussed in Section [ and as our ablations have shown, this can lead lo problems and degraded performance when options with different lengths are required.
There exist methods which do not employ the techniques mentioned above but still rely on the options Tramework (Konidaris Barto 2007 Zhang & Whiteson 2019 Li el al., (220)) or lask-specific policies (Thh lt al.] 2017).. These approaches often make different assumptions about the tasks and settings in which they applied and therefore are not directly comparable to our work. Some require policies that solve each environment (Pickett & Barto 2002) whereas others need environment ID (Mankowitz et al. 2016 [gl et al. (020) or cumulants that accurately represents task dynamics (Barreto et al. 2019). Close to our work are Adaptive Skills Adaptive Partitions (ASAP) (Mankowitz et al. 20166) and Meta Learning Shared Hierarchies (MLSH) (Frans et al.] 2018). ASAP uses a policy gradient method to optimize immediate performance on multiple tasks with known environment. ID but does not use neural networks and does not learn terminations. On the other hand, MLSH uses a hicrarchical structure with predefined option lengths (Lime-based (erminations) and problem setting with unknown environment ID. It optimizes for post-adaptation performance by using two alternating phases that either only update the high-level policy or both high-level policy and sub-policies simultancously. The number of updates in each of these phases is modulated by important task-specific hyperparameters. Because MLSH does not pass the gradient signal between the master and sub-policies, the sub-policies are not explicitly optimized for fast adaptation of high-level policy. This can potentially hinder the adaptability of the final policy and its final performance.
The method we propose instead assumes that policy parameters are updated with gradient descent and aims to learn the parameter values of options. Thus, its problem formulation is closely related to the one of
12

---

Published in Transactions on Machine Learning Research (03/2023)
recent gradient-based Model-Agnostic Meta-Learning method (Finn et al. 2017:: p which uses gradient- based meta learning to learn initial parameter values in supervised learning and reinforcement learning settings. The work of Finn et al. (2017) on MAML was extended in followup works that only trained a part of the
network (Zintgraf et al.) 2019
Raghu et al. 2020) or showed benefits of per- per-parameter learning rates (Li
eet al. Antoniou et al. 2019). Furthermore, several works focused on MAML, in a reinforcement learning
setting (Al-Shodivat et al.; 2018 Stadie et al. 2018 Liu ct al. 2019). In particular, Al-Shedivat et al. (2018)
and Stadie et al. (2018) pointed out a difference between theory and practical implementation of MAML in automatic differentiation frameworks. This issue was further discussed and resolved in followup works
Foorsserr et al.; 2018 Rothfuss et al. 2019 Farquhar et al., 2019). However, the connection between the learning of reusable options and this type of gradient-based meta-learning has not been explored in prior
work.
Instead, a recent work (Veeriah et al., 2021 that utilized both options and meta-learning relied on meta- gradients (Xu et al. 2018 Zheng et al. 2018). Unlike MAML, these have been mainly applied in single-task
(single lifetime) settings to learn hyperparameters (Xu et. al.) 2018 Zahavy et al. 2020) or intrinsie rewards
work and the work of Veeriah et al. 22021] Firstly, their work meta-learns option reward networks in addition to terminations. Both of these are used to guide the learning of sub-policies and only the high-level policy is trained with a task reward. In our work, all parts of the network are trained using the task reward. Secondly, they let the agent use primitve options that always terminate after a single action and use a switching cost to prevent high-level policy from selecting primitive options too often. Lastly, they use both current state and task-specific goal encoding as an input to the high-level policy. The agent thus knows which task it is currently in. Their method is thus not applicable in sctting because we consider sctting in which this information is not available.
(Zheng et al.
2018 2020). Consequently, there are some key differences between
6 Discussion and Future Work
We considered the problem of learning a useful temporal abstraction from multiple tasks in the setting in which there is little prior knowledge available about the environments. Specifically, we were interested in learning options that can accelerate learning in new tasks that are similar to training tasks. We pointed out some of the weaknessos that prior methods for learning of options in similar settings posses. In particular, we have discussed how choosing fixed options length apriori can be restrictive and detrimental. This is because the ideal value of this hyperparameter can change based on the number of available options. Subsequently, we used a meta-learning formulation of this problem to propose a method that combines the options framework with gradient-bass based meta-learning and explicitly optimizes learned sub-policies and terminat ions for performance after several adaptation steps. In our experiments, we have shown that this method outperforms the closest hierarchical and hinrrierarl methods designed for similar meta-learning setting in one discrete and one continuous setting. Finally, have performed several ablations to better evaluate the benefit of learned terminations and gradient- based meta-learning which distinguish our approach from prior work.
The main limitation of our method is the computationally more expensive calculation of responsibilities in IOPG which cannot be computed in parallel. The computation of responsibilities scales linearly with the amount of timesteps and can become a computation bottleneck during the backpropagation. Alleviating this issue could be a potential direction for future work. Similarly, because IOPG uses policy gradient to update the policy, our method is less data efficient at train time than other methods such as MAML which use TRPO or PPO for their (outer) updates. Extension that would allow for one of the more advanced policy updates to be used in combination with our method could potentially improve the data efficiency. Lastly, while our objective implicitly encourages appropriate terminations, it does not explicitly constrain the number of option terminations. Consequently, learned options may be shorter or longer than intuitively expected as long as they lead to good performance. Future rescarch in this direction could focus on combining our objective with regularization techniques such as deliberation costs to softly encourage more intuitive option lengths.
13

---

Published in Transactions on Machine Learning Research (03/2023)
References
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational Option Discovery Algo- rithms. CoRR, 2018.
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Πlya Sutskever, Igor Mordatch, and Picter Abbeel. Con- timuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. In ICLR, 2018.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAML. In ICLR, 2019.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. AAAI, 31, 2017.
Andre Barreto, Diana Borsa, Shaobo Hou, Ghcorghe Comanici, Escr Aygün, Philippe Hamel, Daniel Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The Option Keyboard: Combining Skills in Reinforcement Learning. Tn NeurIPS, 2019.
Garage Contributors. Garage: toolkit for reproducible reinforcement learning research.
coa/rLnorkggruppgaaagg. 2019.
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104, 2016.
Peter Dayan and Geoffrey E Hinton. Feudal Reinforcement Learning. In NewrIPS, 1993.
Tristan Deleu. Model-Agnostic Meta-Learning for Reinforcement Learning in PyToorc, 2018. Available at:
Thomas G. Dicttcrich. Hicrarchical Reinforcement Learning with the MAXQ Value Function Decomposition.
Journal of Artificial Intelligence Research, 13, 2000.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Contimuous Control. In ICML, 2016a.
Yan Duan, John Schulman, Xi Chen, Peter T.. Bartlett, Tyy Sutskever, and Pieter Abbeel. RI2: Fast Reinforcement Learning via Slow Reinforcement Learning. CoRR, 2016b.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine Diversity is All You Need: Learning Skills without a Reward Function. In ICLR, 2019.
Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning. In NeurIPS, 2019.
Chelsca Finn, Picter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In ICML, 2017.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic Neural Networks for Hierarchical Reinforcement Learning, In ICLR, 2017.
Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Fric Xing, and Shimon Whiteson. DICE: The Infinitely Differentiable Monte Carlo Estimator. In JCML, 2018.
Roy Fox, Sanjay Krishnan, lon Stoica, and Ken Goldberg. Multi-Level Discovery of Deep Options. CoRR, 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta Learning Shared Hicrarchies. In ICLR, 2018
Jean Harb, Pierre-Lue Bacon, Martin Klissarov, and Doina Precup. When Waiting Is Not an Option: Learning Options with Deliberation Cost. AAAI, 2018.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The Ter- mination Critic. In ATSTATS, 2019.
Maximilian Igl, Andrew Gambardella, Jinke He, Nantas Nardelli, N Siddharth, Wendelin Boehmer, and Shimon Whiteson. Multitask Soft Option Learning. In UAI, 2020.
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier- Options of interest: Temporal abstraction with interest functions. 2020.
Puerre-Lue Bacon, and Doina Precup.
Martin Klissarov and Doina Precup. Flexible option learning. 2021.
George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforccment learning. In IJCAI, 2007.
Alexander Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy Adaptation for Hierarchical Reinforcement Learning. In ICLR, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few Shot Learning. CoRR, 2017.
Hao Liu, Richard Socher, and Caiming Xiong. Taming MAML: Efficient unbiased meta-reiiforrement learning. In ICML, 2019.
Marlos C. Machado, Marc G. Bellemare, and Michael Bowling. A Laplacian Framework for Option Discovery in Reinforcement Learning. In ICML, 2017.
Daniel J Mankowitz, Timothy Mann, and Shie Mannor. Adaptive Skills Adaptive Partitions (ASAP). NewrIPS, 2016.
Amy McGovern and Andrew G. Barto. Automatic Discovery of Subgoals in Reinforcement Learning Using Diverse Density, In JCML, 2001.
Ishai Menache, Shic Mannor, and Nahum Shimkin. Q-Cut Dynamic Discovery of Sub-Goals in Reinforce- ment Learning. In ECML, 2002.
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-Efficien Hierarchical Reinforce- ment Learning. In NeurIPS, 2018.
Scott Niekum and Andrew Barto. Clustering via Dirichlet Process Mixture Models for Portable Skill Dis- covery, In NewrIPS, 2011.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys, 2021.
Mare Pickett and Andrew G. Barto. Polccy An Algorithm for Creating Useful Macro-Actions in Reinforcement Learning. In ICML, 2002.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. In ICLR, 2020.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning Abstract Options. In NewrIPS, 2018.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal Meta- Policy Scarch. In ICLR, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- High-Dimensional Continuous Control Using Generalized Advantage Estimation. CoRR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Opti- mization Algorithms. CoRR, 2017.
15

---

Published in Transactions on Machine Learning Research (03/2023)
Matthew Smith, Herke van Hoof, and Joelle Pineau. An Inference-Based Policy Gradient Method for Learning Options. In ICML, 2018.
Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some Considerations Learning to Explore via Meta-Reinnorcement Learning. In NeurIPS, 2018.
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112, 1999.
Yee Teh, Victor Bapst, Wojeiech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In NewrIPS, 2017.
E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In IEEE/RSE RSJ IROS, 2012.
Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Jurii Kemaev, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of Options via Meta-Learned Subgoals. In NewrIPS, 2021.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In 1CML, 2017.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. 2018.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. self-tuming actor-critic algorithm. 2020.
Shangtong Zhang and Shimon Whiteson. DAC: The Double Actor-Critic Architecture for Learning Options. In NewrIPS, 2019.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. 2018.
Zeyu Zheng, Junhyuk Oh, Mattco Hessol, Zhongwen Xu, Manuel Kroiss, Hado Van Hassolt, David Silver, and Salinder Singh. What can learned intrinsic rewards capture? In ICML, 2020.
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast Context Adap- talion via Meta- Deaanne In ICML, 2019.
16


Reviews:
[{"review_id": "2PyB5tQfbP", "reviewer": "Reviewer_a2xU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1) simple, well motivated approach. One could argue it is \"just\" the combination of preexisting methods (IOPG + MAML), but this actually increases the ease of replication and the likelihood of adoption by the broader community.\n2) Clearly written, with a fairly comprehensive treatment of related works.\n3) Very small number of meta-training tasks needed. I generally thought gradient-based meta-RL needed orders of magnitude more tasks to properly generalize. I'd be curious to see a comparison of how FAMP and MAML scale with the scale of the meta-training task set.\n\n# Weaknesses\n1) Relatively simple task domains with obvious hierarchical structure. Many prior approaches have shown promising initial results that subsequently fail to scale up. I'm not convinced this method would outperform e.g. RL^2 on a pixel-based environment. Similarly, I'm worried that in tasks without such a clear separation of concerns FAMP might fail to learn anything.\n2) Qualitative evaluation could go farther. Figure 6 in particular feels lacking, as it only shows the behavior of a few options. What to all of the options do? Are the all consistently used?\n3) Why aren't the high-level controller parameters optimized in the outer-loop as well? It feels wasteful to start every task with a naive high-level policy. Perhaps this could be seen as enforcing more uniform option usage, but I'd like to see that argument made explicit (and ideally show that the alternative is less performant)\n4) Changes to the ant environment from prior work seem poorly motivated. Sure, randomly resetting the ant orientation isn't realistic, but nothing about this environment is very realistic -- might as well follow the procedures of prior work to aid in comparison. Especially when comparing to models tuned for that variant.\n5) Apart from the failure mode of num options >= num tasks, another possible failure mode might occur when the number of options is too low relative to the complexity of the environment. For example, I have a hard time believe an agent with only 2 task-agnostic options could navigate to an arbitrary 2D location. This suggests a \"sweet spot\" where you have a enough options to do everything you need to do in the environment but not so many that each ends up just corresponds to a task-specific policy. Would be nice if you address this in the paper (or convince me this 2nd failure mode doesn't exist)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a novel meta-reinforcement learning method that extends MAML with options. They do this by leveraging IOPG to learn both option policies and their terminations in parallel (i.e. even for the options not currently being executing) and in manner compatible with gradient-based meta-learning. They choose to only update the low-level policy in the outer loop, so that the option are agnostic to the current task, but are generally useful across the task distribution. The high-level policy is updated in the inner-loop, with the overall optimization process incentivizing the learning of options that allow for the rapid adaptation of a high-level controller.\n\nThe authors evaluate their method on a set of simple gridworld tasks as well as a more complex continuous control domain. These results show that FAMP is generally better than prior approaches (e.g. flat MAML) in both the meta-RL and multitask settings. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Needed for acceptance\n* Modifying the text to clarify the design decision brought up in W3, and ablating it if feasible\n* Reworking Figure 6 and/or adding additional figures to address W2\n* Rerunning on the Ant environment with the random orientation resets as discussed in W4 (to make results as compatible to prior work as possible)\n\n# Strengthen the paper\n* A pixel-based environment would highlight scalability as per W1.\n* Evaluating FAML on some of the tasks from the original MAML paper would ease concerns raised in W1\n* An experiment showing how FAMP and MAML scale with the number of training tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4tPQX7UvRz", "reviewer": "Reviewer_nzQu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Discovering temporal abstractions is one of the most important challenges in RL. This paper proposes a method that is technically reasonable in the sense that the options were discovered with a principled objective: to maximise rewards on the tasks we care about. \n- The paper provides a nice summary of relevant background and motivates the problem well. \n\n**Weaknesses**\n\n- The experiments are not comprehensive enough to support the claims of the paper.\n  - Overall, it could be much better to explicitly list a set of hypotheses (or questions) that the paper is trying to investigate through the experiments. \n  - While this paper claims that their method is better than MLSH, the domains considered in this paper are much simpler compared to the domains in the MLSH paper. While Ant Maze was taken from the MLSH paper, it is solvable using a single-task non-hierarchical policy. MLSH was in fact evaluated on much more challenging tasks such as Ant Obstacle, where the single-task policy completely fails.  \n  - Some important non-hierarchical baselines are missing. The first one is MAML which is missing in Taxi domain. Another important baseline would be a flat multi-task policy that is pre-trained on the training tasks and fine-tuned on the test tasks at test time. This would justify the importance of hierarchical RL on those domains. \n  - The discussion about the role of learned termination functions compared to options with a fixed number of steps is so important that the relevant result should be included in the paper rather than in the appendix.  \n  - What does prevent your method from collapsing to a single option or converging to trivial options such as primitive actions? While these questions have proven to be difficult to address, this paper does not theoretically or empirically verify how exactly the proposed method addressed these challenges. \n  - It seems wrong to drop the result from one of the seeds simply because it was unlucky. I suggest including all 5 seeds for fair comparison between methods or use longer episodes as the paper suggested.\n\n- The contribution and novelty of the work needs to be clarified. While the proposed method is very similar to Veeriah et al.'s work in that they also proposed to use meta-gradients to discover options with the same inner/outer optimisation objective, the departure from Veeriah et al. is not much highlighted throughout the paper. The current presentation could be misleading to the readers who are not familiar with Veeriah et al. Although the problem setting is slightly different, it would be still important to highlight the difference/similarity to Veeriah et al. at least in the method and related work section. \n\n- Clarity\n  - \"Ours + FT\" in the plots are not mentioned or explained in the text. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a hierarchical RL architecture that learns a set of option (i.e., sub-policies with their termination functions) end-to-end using meta-gradients in a multi-task setting. The empirical results show that the proposed method outperforms a hierarchical baseline (MLSH) and other non-hierarchical baselines in Taxi and Ant Maze.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Add two baselines that are mentioned in the weaknesses section.\n- Compare against MLSH on more challenging tasks that MLSH was evaluated on. \n- Provide insights how the proposed method addresses option collapse or degenerate problem that is mentioned in the weakeesses section.\n- Add one of the unlucky seeds back to the result.\n- Highlight how similar/different the proposed method is compared to Verriah et al.\n- (Recommendation) Explicitly state a list of hypotheses to verify at the start of the experiment section. This is because there are so many baselines to compare and it is not immediately clear what each comparison is showing. ", "broader_impact_concerns": "I have no broader impact concern. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ESEGZxyI4t", "reviewer": "Reviewer_shTx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Builds on well known algorithms (MAML) that perform well in practice\n- Evaluation on both a tabular domain and a more challenging continuous control task\n- Well-motivated algorithm that investigates most of its choices\n\n**Weaknesses**\n- Some of the claims do not reflect recent advances on the options framework\n- The general presentation could be improved (e.g. everything around meta gradients and MAML, or the focus on termination probabilities)\n- The empirical setup brings many questions (in terms of pertaining and number of seeds in the more challenging domain (only 3!) )\n\n\nThe empirical setting could be greatly improved. First, the fact that only 3 seeds are used on Ant and 5 on the tabular Taxi is a red flag for me. We are constantly suffering from past mistakes in deep RL where practitioners have not been rigorous in their evaluation, and that starts with having a higher number of seeds. Secondly, the amount of pretraining for each is different. In fact, FAMP (the authors method) is trained for one or two orders of magnitude longer than any baselines in the Ant experiment. This is conveniently left out to the appendix, which I find is not a good sign. This should be clearly discussed in the main paper and before substantiated for why this is the case. The main papers brings the argument that each method is trained until it doesn't improve, but is that a fair setting? What is the purpose of this setting, what are we trying to evaluate more precisely? Actually, looking at Figure 8 it even seems that MAML is still learning. \n\nThe paper's abstract mentions that several deep learning approaches present shortcomings. From the introduction, this is understood to be the line of work around the option critic (OC) and MLSH. For the former, the problem is option degeneration while for the latter it is the fact that the termination probability is uniformly fixed over states. Concerning the fixed termination probability, a lot of discussion is provided in the paper as to its importance, yet the experiments using deep networks show that there is no significant advantage in learning a termination function. This seems to contradict those paragraphs that mention the important of such a function. Moreover they show that the discussion on the termination function is focused on a case where optimal options can be recovered (i.e. in the tabular case): when options are approximated, their applicability is more flexible due to generalization and this means that we do not need to know exactly when to terminate. Concerning the work on OC, there are more recent results (e.g. Khetarpal et al. 2020) that show its relevance in the context of multi task learning (which is not the setting for the two works referred to in the paper). In fact recent results seem to show that options tend to stabilize in a transfer learning setting. I am not sure if any of the baselines presented would be an equivalent of the OC in this multi task setting.\n\nThe paper mentions that IOPG is preferred as OC is on-policy. However, there is recent work that shows a simple extension to OC for learning all options simultaneously called Multi-updates Option Critic (MOC) (Klissarov and Precup, 2021). This work would affect many discussions and claims throughout the paper. It can also potentially affect the experiments, as it is mentioned in the Appendix B that MLSH are more sample efficient since they use PPO, which is the algorithm used by MOC. It could therefore be an important baseline. Another baseline that I would to see is Veeriah et al. 2021, however it is mentioned that this baseline uses privileged information about tasks, can the authors expand on that?\n\nIn general I would appreciate either a more precise discussion of the \"multi-task\" and \"learn all\" baselines, if not a complete algorithm for each. How can multi-task be doing so badly? This seems to go against recent findings (Kurin et al., 2022)\n\nIn the appendix it is said \"MAML uses a more complex outer gradient update while FAMP relies on a simple policy gradient.\" I am not sure I understand what that means exactly, could the authors expand?\n\nFinally, it seems like a discussion about a whole line of work on meta gradients is missing, for example Xu et al 2018, Zheng et al. 2018, or Zahavy et al. 2021. At least some of these works should be cited in my opinion as they are an important part of the literature on meta gradients. \n\n\nMinor things: \n\nCitation format -> \"In particular, (Al-Shedivat et al., 2018) and (Stadie et al., 2018)\"\n\nThe notation in Equations 6,7,8,9, the notation for which MDP is sampled could perhaps be better reflected in the probability distributions.\n\nSection 2, notation inconsistencies between L_{\\mathcal{M}} and L^{\\mathca{M}}\n\n\n\nKhetarpal et al. Options of Interest: Temporal Abstraction with Interest Functions, 2020\n\nKlissarov and Precup, Flexible Option Learning, 2021\n\nKurin et al., In Defense of the Unitary Scalarization for Deep Multi-Task Learning, 2022\n\nVivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of Options via Meta-Learned Subgoals. In NeurIPS, 2021.\n\nXu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning, 2018\n\nZahavy et al., A Self-Tuning Actor-Critic Algorithm, 2021\n\nZheng, Z., Oh, J., and Singh, S. On learning intrinsic rewards for policy gradient methods, 2018\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a way to obtain re-usable options by framing the problem of option discovery through a meta learning approach. In particular, the authors suggest an objective function in which the meta-policy is updated multiple times before options are themselves updated. This is done with the goal of faster adaptation of the meta-policy. The evaluate their approach, named FAMP, on the classic Taxi domain and a challenging Ant navigation task. In these experiments, FAMP seems to outperform all baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Running the algorithms for more seeds\n\nBetter context for the approach with respect to related work in HRL and meta learning.\n\nBetter explanation on the different choices (mentioned above) for the empirical setting.\n\nClearer discussion on why some baselines are not presented.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pretty clear cut submission and discussion. There were some initial concerns about the quality of the experiments and design choices at the reviewer's and AE's requests the experiments were significantly improved. The authors also clarified several misunderstandings. Three expert reviewers in the area were all supportive of acceptance! 
Decision: Accept (poster)