Discovery of Options via Meta-Learned Subgoals
Authors: Vivek Veeriah
Tom Zahavy
Matteo Hessel
Zhongwen Xu
Junhyuk Oh
Iurii Kemaev
Hado van Hasselt
David Silver
Satinder Singh
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
meta-gradients

Abstract: Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager-worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both a set of task-independent discovered-options and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extended options in multi-task RL domains, (2) the discovered options are frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help a randomly initialised manager learn faster in completely new tasks.

Discovery of Options via Meta-Learned Subgoals
Vivek Veeriah" University of Michigan
Tom Zahavy DeepMind
Matteo Hessel DeepMind
Zhongwen Xu DeepMind
Junhyuk Oh DeepMind
Iurii Kemaev DeepMind
Hado van Hasselt DeepMind
David Silver DeepMind
Satinder Singh University of Michigan, DeepMind
Abstract
Temporal abstractions in the form of options have been shown to help reinforce- ment learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment re- mains challenge. this paper, we introduce novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager- worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both set of ieseinndeenndent ducoveredd-optonn and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extendee options in multi-task RL domains, (2) the discovered options frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help randomly initialised manager learn faster in completely tasks.
Reinforcement learning (RL) problems involve learning about emporally-extendee actions at multiple time scales. In RI., the options framework (Sutton et al. 1999) provides a well-defined formalisa- tion for the notion of temporally-extent
actions. Options that achieve specific subgoals can be usefull in reinforcement learning (RL) in at least two ways: in model-based RL, they provide faster rates of convergence through longer-backups of value functions within planning updates (Silver & Ciosek 2012 Mann & Mannor 2014 Brunskill & Li 2014), while in model-free RL, temporally-
extended actions commit agents intentional multi step behaviours, which can translate into better
exploration (Machado & Bowling 2016; Nachum et al. 2019 Osband et al. 2019).
We consider scenario where agent learns to solve distribution over tasks. In such cases having carefully designed temporal abstractions can greatly reduce the overall sample complexity of learning for RI. agent that is trying to master those tasks. The agent can produce faster learning mainly
by reusing those abstractions across multiple tasks (Sutton et al.) 1999. Solway et al.) (014). Many recent approaches have empirically validated this by demonstrating that hand-designed temporal abstractions can often lead to improved learning performance on a variety of challenging multi- task RL domains (Imazeki & Maeno 2003 Kulkarni al., 2016 Nachum al. 2018 Riedmiller et al. 2018). However, if the agent has abstractions that are not useful to a downstream task, it can significantly hurt the performance by making exploration harder (Jong al. 2008). emphasising the challenge involved in carefully hand-dest -endddeiigning abstractions that are useful in general, across many domains and tasks. Thus, the automated discovery of temporal abstractions from experience without extensive domain-specift. knowledge remains an important open problem for reducing sample complexity in RL.
"ooresponning Author. Email: vveeriah@umis ch. edu Now at Sea Al Lab
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The main contribution of this work is our hierarchical agent architecture and an associated meta- gradient algorithm to discover temporally-exteer reused across many tasks. Previously, meta-gran graddggas have been successfully used for learning hyperparameters (Xu et al. 2018 Zahavy et al. 2020), intrinsic rewards (Zheng et al.) 2018 2019; Rajendran et al.) 2019), and auxiliary tasks (Veriiah et al. (019). Our work is the first to demonstrate that they can successfully learn rich parameterisations of temporal abstractions. Our starting point is the following hypothesis: If we could discover temporal abstractions useful across many training tasks, they would capture regularities across those task environments and have a higher likelihood of being useful and reusable in new, previously unseen tasks.
actions in the form of options that can be
To discover temporal abstractions useful across tasks, our system flexibly defines task independent subgoals for options through separate discovered rewards and terminations, different for each option. We employ meta-gradients to discover the parameters of such option-rewards and terminations based on their utility across the many training tasks, so that the set of induced options is useful to a hierarchical agent trying to master all training tasks. The meta-gradient approach operates by evaluating change in the options, caused via changes to the option-rewards and terminations, w.r.to the hierarchical agent's s performance on samples drawn from many tasks; then computes and uses the gradients from this evaluation to discover the option-rewards and terminations. This differs significantly from the previous multi-task option discovery approaches (Bacon et al., 2017] Frans et al., 2017), , where all options directly optimise the same (main task) reward, which may be insufficient to discover reusable, task- inddedddeendent options.
We evaluate the proposed approach empirically in two multi-task RI. settings based on an illustrative gridworld and on 3-dimensional first- first-person task suites from DeepMind Lab (Beattie et al., 2016). For each of these, we perform three types of analysis: (1) we qualitatively demonstrate that our approach indeed produces meaningful temporally-extended options: (2) we quantitatively show that our discovered options are extensively used by the agent while learning to solve training tasks; (3) we show that our discovered options support faster learning, i.e., transfer better, test tasks, compared to options discovered by two strong hierarchical baselines (i.e., MLSH (Frans al. 2017 & Option -rritic (Harb et al. 2018))
1 MODAC: Meta-gradients for Option Discovery using Actor-Critic
Why Meta- veta-gradients? As described in the previous section, the motivation behind our work is to discover temporal abstractions in the form of options that are generally useful across many training tasks; and to leverage them to allow effective transfer of acquired skills to new tasks. This based on our hypothesis that options, if useful across many tasks, capture intrinsic properties about those tasks that could lead to better transfer to unseen tasks. For useful options to be discovered, an option-bass based agent needs to evaluate whether a change in given option is useful. Essentially, the agent needs to compute the gradient of future performance with respect to the parameterisation of each option, while the option-based behaviour is itself adapted by the conventional RL gradients; this requires the computation of a gradient through a gradient, and thus meta-gradients is used as the mechanism for driving option discovery. We make this idea concrete in our learning agent MODAC (which stands for Meta-gradients for Option Discovery using Actor-Critic).
Background on Options: Options are closed-loop behaviours over extended periods of time. An option is formally defined by specifying an initiation set (states where the option may be invoked), an option-policy (maps states to actions), and a termination function (maps states to probability of terminating execution of the option). The option-pe option-policy may be defined implicitly as the policy that maximises option-reward function describing the subgoal (or intention) for that option.
1.1 Agent Architecture
We base our agent's architecture on the standard hierarchical agent from Sutton et al. (1999), where a manager chooses among both primitive actions and temporally-extender options; and extend it to the multi-task setup. The agent (shown in Fig. T follows a call-and-return option execution-model and consists of the following four modules:
PPrimitive actions are special case of options that terminate after one step. See Sec. 3.1, pg. 194 of Sutton
al. (999).

---

actorm aptlan-poiiikee
actirn
1 π
02
P3
7
3
30
303
Manager
Option-Policies
Option-Rewards
TetionTerminations
St, 9
St
St
St
Hierarchical agent
Meta-parameters
Figure 1: MODAC: features four networks, namely: the caaaager-policye option pirionnpolicies, option- rewards, and option-terminations. Manager and option-policies are trained via (direct) RI. Option- rewards and terminations define the semantics for the option-policies and are discovered (indirectly) using meta-gradient RL. More details in Sec. 1.1
A Manager network parameterised by GM implements task-conditional policy TM It maps sequence of observations (henceforth the state s S) and task encoding g & to the union of the set of option-policies O and the set of primitive actions A, i.e. T : x - {O, A}. We denote TM (s, .). The Manager is trained to maximise the task reward.
sample from nM
The Option-police policy network represents policies for each discovered option in our hierarchical agent. It takes state as input and has multiple output head that each produce distribution over primitive actions: the jth head's output represents the option-policy T S + A). 8% denotes the parameters of nh Each option-policy learns maximise the corresponding option-rewards produced from the option-reward network described next.
The rpward network defines the subgoals/intentions for the option-policies. It takes state as input and outputs a scalar reward, one per option-policy, for each primitive action a A (the 7th option-reward head's 's output r": S RIAI). r7" denotes the parameters of r
Option-termination network defines the termination function for the option-policies and has an identical structure the option-reward network: it also takes state input and outputs scalar value 3 S - {0, 1} for each option-policy, interpreted as the probability of termination for the associated option. 77 denotes the parameters of f
Each module in the agent closely follows the options framework: the manager corresponds to the ooliey-over-options from the options framework as learns a mapping from states to options. Each option is defined by its associated option-policy and termination defined by the respective networks (initiation set for each option is softly induced by the manager's policy). Option-rewar along with their termination defines option's 's subgoal.
Note, among the four modules, only the manager gets the task goal as input while those corresponding to options (i.e., option- option-policy, reward and termination) do not. This architecturally enforces our objective of discovering fask-independent options that are useful across multiple training tasks, and also supports their transfer to new, previously unseen tasks. We emphasise that it is possible for MODAC to solve each task optimally as the manager gets task as input and can select primitive actions if need be. Also, this form of hiding task goals from the temporal abstractions has been shown to be effective for transfer in several multi-task hicrarchical RL approaches (Dayan & Hinton 1993] Heess et al.) 2016 Jaderberg et al. 2016 Nachum al. 2018). Separately, note that our assumption of access to task encodings is standard in many multi-tase RL (Beattie et al., 2016 Plappert et al. 2018) and robotics domains (Kolve et al. 2017 Deitke et al. 2020).
We motivate the choice of allowing the manager to pick between cidioon-olicies and primitive actions as opposed to option- policies alone with the following example: consider the case where options take you the doorways a building with rooms. Suppose the goal was to go to the middle of some room. Now there is no policy that maps states to options that can achieve that middle-of-the-rooe goal. But we allow the manager to choose both options and primitive actions, then could get to the doorway of the room by traveling from doorway doorway and then pick primitive actions to
option-policies

---

get to the middle of the target room. This is a fär more flexible use of options. Indeed, this is the way options were presented originally Sutton al. (T999).
(2008) also argue for using primitive actions along with options improve learning performance. Mann al. (2015) theoretically study the role of adding primitive actions and proved that using options along with primitive actions allows planning agent achieve faster convergence. [Jong al. (2008) empirieally studies this design choice of including primitive actions to a HRL agent and clearly show that the agent that can select between options and primitive actions can obtain optimal performance while the agent that only selects options fails. Furthermore, the work by Baumli et al. (202I) also explores HRL approaches that learn by selecting among options and primitive actions.
More recent works of Mann et al. (2015) and Jong et al.
In the next section, we present a discovery algorithm based on meta-gradients that is used to train each network.
1.2 MODAC's Discovery Algorithm
Algorithm Overview:
We extend the general meta- meta-gradion gradient algorithm (Xu et al., 2018) discover options within our hierarchical architecture. It consists of two nested loops: an inner-loop that updates the manager to maximise the discounted sum of task rewards and that updates the option-policies to maximise the discounted sum of its corresponding option-rewards (with discounts provided by option-ternnn setion-temminations)) and an outer-loop that evaluates the updated manager and option- option-policies on new transitions produced by the agent and then updates the option-rewards and tertinaiorms by computing meta metaa-grddiens by back- propagaaggg through the inner-loop updates.
In the inner-loop, the parameters of the manager gM and option-policy }0%} are updated with transitions produced by the agent on random sample of tasks. the outer-loop, the updated are evaluated on new transitions drawn from another random sample
manager and option-policte of tasks, and then the meta perta--rraettess of the option-reward {17²²) } and iisiiontermination (178%) networks are updated by paoopppopgg through the inner-loop updates.
Below, we instantiate the algorithm for an actor-critic architecture, leaving adaptation to other RL updates future work. A detailed derivation is presented in the Appendix.
Inner-loop: Consider a n-step trajectory {st, Ge, Pt+1, r2+1: 1 x".ggtmt fot generated by following an option-policy T" until its termination (5% +n = 1), where x² was sampled from the manager's s policy TM at while interacting with task g. For such trajectory, the inner-loop updates to the option policy and manager parameters are:
80 00 aG (s()) V log T" (aclse) ( GM M + ta - N (sto: g)) VaN log MM (ol sto 9) - RM a (Sta 9)
(2)
where KM, x² weight the value updates relative to the policy updates, and Gi and G.M are n-step returns for the option-policy and manager:
G
Σ (1- 1=1
Bit Prics + (1 Bian
(3)
G. Σ P n + + n (st+n)
(4)
where is switching cost added, option terminations, to the per-step rewards used in man- ager's update. The switching cost hyperparameter encourages the manager to pick options that are lemporally-extended (therefore aiding their discovery).
Outer-loop: the outer-loop update to the option-reward and aptiion-terminatte meta- pattta-arameers,, we +++m generated by interacting with the environment use a different trajectory (st, ar Ttt1, TI, . go 91
using the latest inner-loop parameters 0% i= l... l...K.K K. Since this trajectory is used to evaluate the change made to the manager and option-polit policy parameters, we refer to it as validation trajectory. The task g may be different from the one used in inner-loop update. On this validation trajectory, we compute the meta-update to the option-reware and option-terminatio pack-propgggting through the

---

inner-loop updates
Vi, 1/" t- 1fth to, (G - ui (st: 9) V ypo log x (ag Vi, 178" t 118" ta (Gt - - (st. 9)) V you log π (asls:)
(5)
The algorithm for training a MODAC agent is summarised in Alg. I MODAC utilises the inner-loop updates (see Eqns 2 train the parameters of the option-policies and manager, and in the outer-loop. discovers the option-reward and termination parameters via meta-gradients obtained using the updated option-policy parameters (see Eqn. (5). Furthermore, meta gradients are efficiently computed through backward-mode autodifferentiation. thus making its computational complexity similar that of the forward computation (Griewank & Walther 2008
Pseulocode:
Algorithm 1 Meta-gradicn algorithm for option discovery
Initialise parameters 0A4 (0%, ,n)" 118h) Vi 1... K Sample task g ~ G, state s ~ So(g), option-policy ~ x (s. repeat
for 1= 1,2, to do
# L. denotes number of inner-loop updates
Re-sample (s, g. o) when starting a new episode
if (s)
1 then o ~ π (s.g)
end if
Obtain transition using option-policy ~ E(s, (+))
# E denotes the environment
# Inner-loop update
Update option-pola policy 80 with Eqn.
Update manager &a on states where it samples with Eqn. end for
Obtain transitions from another task using the updated manager @ and option-policy (0o:)
parameters
Outer-loop update
Update option-reward 17""' and termination rf with Eqn. 5 Vi 1...K K
until maximum number of timesteps
1.3 Differences Between MODAC and Prior Option-Diseovery Methods
Previous dition-discovery works that have been explored in a multi-task setup, (c.f. MLSH, Option- Critic) usually train all the options to optimise the task reward for the current task generating the data, though gradients are accumulated across tasks. In contrast, in our agent, each option-policy optimises different objective, parameterised by the corresponding task-independent option-reward and termination that are discovered to be directly useful across many tasks. The MODAC architecture and the meta-gradient learning updates discover fask-independent, general-purpose. disentangled options that can help the agent not just to achieve higher rewards during training, but also to speed up learning in new tasks.
2 Empirical Results
We empirically evaluated MODAC using mix of gridworld and navigation tasks from DeepMind Lab. Our main numeric metric for the quality of the options discovered by our agent was their usefulness for transfer. In addition to this, evaluated their quality in various ways, including whether they were temporally extended and whether they are diverse in their behaviour. Our experiments all included a training phase followed by testing phase.
Training: For every episode, a training task was selected randomly from a set of training tasks that were distinct from the test tasks. In this phase, MODAC jointly leamned the manager, option-policies, option-rewards, and option-terminationn
with transitions from the episode. As noted earlier, this
'The option-poli policy parameters are function of option-reward and termination parameters, and thus, allow for computing the meta-gradients.
5

---

Oriwwers
Trana
3000 COK
Sentoing GED
20
(a)
(b)
(c)
(d)
Figure 2: (a) shows the gridworld's ' layout, the subset of goal locations (in light blue) that are used as training tasks and the disjoint set of goal locations (in orange) used test tasks. (b) shows how MODAC chose among options and primitive actions at three different points during training, as those options were being discovered. In the histogram, each of blue and red bars measure the frequency of selecting an option and primitive action respectively. From this, we can see that the agent picked options more often than primitive actions throughout the training phase. (c) shows the (transfer) performance on held-out tasks of options learned by MODAC, MLSH, Option-Critic, and Flat agent. The agent with options discovered by MODAClearned significantly faster than other baselines, demonstrating the usefulness of those options during transfer. (d) shows MODAC's average (transfer) performance, function of switching cost e used during training.
phase penalty in the form of switching cost (a negative reward) is added to the task rewards that is used in the manager's 's learning update (see Eqn. 2 It based on the intuition of incentivising the manager to make fewer decisions while trying to maximize rewards from given task. It does this by discouraging the manager from selecting primitive actions too often as it incurs a lot of switching costs, thereby decreasing its return. The switching cost is used with every update to the manager during the training phase and the role of switching cost to encourage the manager to pick and thus help discover cemporally-extendee options by penalising the selection of primitive actions too often.
Testing: We froze option-policies and option-termination functions after training, and transferred them to a new manager with randomly initialised parameters; then evaluated how fast the manager could learn on different unseen test tasks. Performance on each test task was evaluated independently, re-initialising the manager every time, and the results presented are averages across test tasks. If the manager finds the transferred options to be useful for maximising rewards from test tasks, then the manager would naturally pick those options without need of the switching cost penalty. Thus the switching cost is not used this phase.
Baselines: We compared the performance of MODAC at test time with that of Flat, a non-hierarchica. actor-critic agent, and also the performance of hierarchical agents that used options discovered using MLSH and multi-task extension of the Option-Critic, as these hierarchical approaches also use multi-task performance on rewards to drive their discovery process. We use an identical train/test setup for all the hierarchical agents. The hierarchical approaches learn to select among the union of the set of option-policies and the of primitive actions, which is identical MODAC. Details on all the agents, their hyperparameter choices and other implementation details are in the Appendix.
2.1 An Illustrative Gridworld
Consider a simple gridworld with connected rooms, whose layout is shown in Fig. 2p. The agent receives reward of on reaching the goal and 0 at every other time step. During training, at the start of each episode, a new training-goal location is randomly chosen from the set shown in blue in Fig. 2p. During testing, a goal position is chosen randomly from a disjoint set shown in orange in Fig. 2 and remains fixed for all episodes. The agent's 's observations included its current location and the grid's layout (in 2 separate image channels). The agent was also fed the goal location during training (as the 3rd channel). Each test task was evaluated separately and the overall transfer performance was computed by averaging across all test runs.
Domain Description:
Visualisation: Fig. shows the option- policies discovered in the gridworld (arrows indicate the direction of movement), at the end of training. We found the discovered options to make intuitive sense, given the distribution of training tasks. The first three options each led into one of the rooms where the training goal states (in blue) were concentrated: specifically, from any state the options shown Fig. 3 steered the agent to the upfer-left, upper-right and lower-right rooms, respectively. The option in Fig. 3d seems redundant given the options in Fig. Bp and c. We hypothesise that such

---

(d)
Figure 3: The red circle approximately marks the destination//suboall states for each discovered option-policy (arrows indicate the direction of movement). Options (a-c) each led the agent directly of the three areas where the training tasks were concentrated. Option (d) appeared to be redundant and led to either one of two such areas depending on the start state.
redundancy was due to the training task distribution only featuring goals in three rooms, thus requiring only three options capture most of the structure in the space of behaviours required by the training tasks.
Quantitative Analysis: (1) We measured how the manager selected among options and primitive actions at three points during training. Fig. [2 plots the manager's 's choices as histograms. The blue bars denote how often each of the options was picked, red bars depict how often primitive actions were chosen. The manager continued to pick both options and primitive actions throughout training. Consistent with the rediinnred identified in Sec. 2. only 3 of the 4 options were selected frequently towards the end of training.
(2) Option-policies lasted 5.46 steps average, and so were temporally extended and given how often options are picked were overwhelmingly responsible for behaviour.
(3) Fig. 2 shows the performance on test tasks, when a randomly initialised manager was provided with the (fixed) options discovered by MODAC at the end of the training phase from 6 independent runs (10M frames were used for training). The speed of learning on the test tasks was substantially faster compared to Flat agent that only used primitive actions; it also outperformed the baselines with access to the options discovered by MLSH or the Option-Critis Critic.
(4) During transfer to test tasks, the manager selected options 56.11% of the time. But recall that options last more than 5 steps and so in effect options controlled behavior more than 85% of the time. (5) Fig. Zt shows transfer performance (averaged throughout the test phase) as a function of the switching cost used in training. We found a sweet spot at a cost of 0.05, for which the discovered options were maximally useful.
2.2 Scaling Up to DeepMind Lab
We applied MODAC to DeepMind Lab (Beattic et al., 2016). challenging suite of RL tasks with consistent physics and action spaces, and a first-person view as observations to the agent (hence there is partial observability). We evaluated our approach their names), where each set corresponds to a different type of navigation problem. For instance, the task explore -ooollegoal___ lalleallllalaliillalaaaaallllale locatiooonn.. requires the agent to explore maze to visually identify and then reach a goal location identifiable by a special 3D marker. Each set includes 'simple' and 'hard' variants; in simple variants the layout includes at most handful of rooms, while harder variants require navigation in large mazes. In both cases, procedural generation is used to create a different layout for every episode. The challenge is to discover options on simple tasks that can be useful to learn faster on hard tasks. More details in Appendix.
4 different task sets (see titles of Fig. for
Visualisation:
In explore_goad locations. the state space is obviously too large to visualise entire option policies in one plot as we did for the gridworld. Instead, we generated sample trajectories of experience, and used the DEBUG information provided by DeepMind Lab environments (which is not fed to the agent) to visualise trajectories by drawing the path taken by the agent onto a top-down view of the maze. The segments corresponding primitive actions were coloured in blue, while those corresponding to the discovered options were assigned an arbitrary different colour. The agent's s start and end locations were highlighted by white and green circles, respectively.
We generated sample trajectories using trained manager with access to both discovered options and primitive actions. When an option is selected, it is executed until the aption-terminatee function is
7

---

(a)
(b)
(c)
(d)
Figure Option Execution by a Manager. The figures show distinct samples of trajectories generated by a manager with access to both discovered options (marked in non-blue colours) and primitive actions (in blue). (a) and (b) correspond to mazes in the training set. (c) and (d) are sampled from the testing set. The agent's starting and final positions are highlighted by white and green circles, respectively. In all cases, the agent successfully achieves the task's objective reaching the rewarding goal location by using mixture of primitive actions and discovered options.
(c)
Figure 5: Sampled Option-Policies. Policies. The task is to reach the rewarding goal location that always in the top left corner but not in the line of sight for the agent at the start of the episode. The agent's s starting and termmnnttnnnnni positions are highlighted by white and green circles. In (a) and (b) we show the (very different) trajectories followed by two different option-policies when initialised in the same state within maze from the training set. In (c) and (d) show two trajectories in maze from the test set, for the same pair of option-policies (again initialised in the same state).
exgiire dijert MAAT many
PX3i00e diject Jeowloee bege
PASWAVE DONE www.ttoooooooooooooo Lvrge
exptere Jecanonn
123M
150M
Framas
Trima
Frames
Tromsse
(a)
(b)
(c)
(d)
Figure 6: Transfer experiments on DeepMind Lab. Options, discovered from training tasks, were transferred to the corresponding test tasks for hierarchical agents. Figures show the (transfer) learning performance of different agents while they learned to maximise rewards in the test task. MODAC with discovered options learned better and thus was able achieve better asymptotic performance 3 of those 4 new, unseen tasks, while learning substantially faster on the 4th task.
triggered, at which point the manager chooses again. The manager made extensive use of the learned options in both the training tasks. for which sample is shown in Fig. 4h and b, and in the testing tasks, shown in Fig. 4 and d. In the simple training tasks, the manager often only required a handful of option executions reach the goal. For instance, in Fig. 4. after the goal (top-left corner) enters in the line of sight of the agent, it only takes two options to take the agent to the goal. In the hard test task, the manager still relied extensively on the learned options, but it needed to chain higher number of them during the course of an episode. In Fig. 4ec the agent was spawned in the centre of the maze at the beginning of an episode. The agent used a mix of options and primitive actions in order to identify and reach the goal location (top-left). In both the training and test tasks, we often found that the manager used options extensively to explore the maze, but it relied on primitive actions for the last mile navigation (i.c. in proximity of the goal).
8

---

Fig. 5 visualised trajectories generated by following 2 discovered cetiionpolicies on the training task set. After seeding the episode in the same way, we observed that the option-policies produced quite diverse behaviours. For instance, we found that the options in Fig. and b explored the maze in very different ways. We note that when the latter entered in line of sight of the goal in the top left corner it marched straight into it. In Fig. 55p and d, we visualised the execution of the same pair of options when triggered on the larger maze from the testing task set; while neither happened to encounter the goal marker in those two episodes, both demonstrated meaningful temporally extended behaviour that resulted in good exploration of a vast portion of the maze.
Quantitative Analysis: In each of the 4 training task-sets we discovered 5 options (using a switching cost = 0.03). and observed that the average length of the options was 12 steps. Fig. O shows the performance on testing task sets, averaged across 6 independent runs, for randomly initialised managers, given access to the pre-trained options discovered by MODAC (200M frames were used for training). Those agents learned to maximise rewards faster than the Flat agent that learned with primitive actions alone, and reached higher asymptotic performance in 3 of them. The transfer performance with MODAC-discovered options was also better than that of the MLSH and Option- Critic baselines in all 4 domains. We again measured the distribution of the manager's s choices at transfer time: options were selected 63.76% of the time, which given that options last about 12 steps implies that our discovered options were responsible for behaviour more than 95% of the time.
Additional Results: We studied the effect of MODAC's transfer performance when fewer samples are used in the training phase. Fig. shows perfor- mance on one DeepMind Lab task set when only 5M and 10M samples are used; in this case the x-axis includes the training samples for the MODAC curves. The MODAC agent learned faster than the Flat agent, showing that useful options can be discovered with small enough training samples to outperform the Flat agent in comparison that takes all samples into ac- count. In Appendix, we present results of additional studies carried out to answer the following empirical questions: (1) How are the options qualitatively differ- when the number of options (a hyperparamettr)) is varied? (2) Could MODAC discover options in Atari games from unsupervised training tasks that could be useful to maximise the game score at test time? In addition we provide more qualitative visualisations from the MODAC agent and its option-policion dis- covered from DeepMind Lab task sets.
explore_gosd Exploor__oal_locatiios locatoons_aaaatteeeoooooooo
Avg. Episodic Return
10-
6-
4-
MODAC (AM)
MOGAC 1 ('0M)
50M
100M
150M
200M
Frames
Figure 7: Shows (transfer) performance on a DeepMind Lab task set when fewer samples (5M and 10M) are used for discovering op- tions during training, compared to 200M used in our main experiments. Learning curves for MODAC are right-shifted to account for the number of samples used in training.
3 Related Work
Hierarchical Reinforcement Learning (HRL): Prior HRL work has shown that temporal abstrac- tions lead to faster learning in both single-task (Sutton et al. 1999; Dayan & Hinton 1993: Di-
etterich, 2000 Ghavamzadeh & Mahadevan 2003; Kulkarni et al. 2016) and multitaask/tansste setup (Konidaris & Barto [007] Brunskill & Li 2014 Tessler et al. 2017). The aforementioned works require deni-ne temporal abstractions; which can be challenging in scenarios where
the agent interacts with a distribution of tasks. In contrast, we aim to discover temporal abstractions purely from experience without human supervision or domain knowledge.
Discovery of Temporal Abstractions: Majority of the prior HRL work for discovering abstrac- tions are either unsupervised approaches or operate in a single-task setup. Older unsupervised approaches to discovery have exploited graph-theoree properties of the environments such as bot- tlenecks (MeGovern & Barto 2001). centrality (Şimşek al. 2005), and through clustering of states MMannrr et al.] 2004)). More recent unsupervised approaches discover options by learning proto- value functions (Machado et al., 2017), successor feature representations (Machado et al.
By unsupervised, we refer to approaches that do not require extrinsic reward signal to drive the discovery process.

---

entropy minimisation (Harutyunyan et al. 2019), maximising empowerment Grrgor et al. 2016), maximising diversity (Eysenbach al. 2018) or through probabilistic inference (Ranchod et al. 2015 Daniel et al.) 2016). Recent approaches for discovery in single-task setup learn agents with architectures that are inspired from options framework (Vezhnevets et al. 2016 [Bacon et al. 2017, Kostas et al.) 2019) and feudal RI. framework (laderberg et al. 2016). In contrast to the aforementioned approaches, our approach is motivated by the multi-task RL setup where the objective
2018),
to discover task-independent options that can be reused by the RL agent to master all of its tasks.
Among prior work that investigates discovery in multi-tast settings, Meta-Learning Shared Hierar- chies (MLSH) learns a set of option-policies by directly optimising task rewards via a joint actor actor-critic update. In contrast, our approach discovers options through the subgoals flexibly defined by option- rewards and terminations; also, our approach learns the temporal scale of each option, while most prior approaches, including MI.SH, fix the temporal scale.
4 Conclusions
We introduced MODAC, a novel hierarchical agent and meta-gradient algorithm, for option discovery in multi-task RL setup. Through visualisations from gridworld and DeepMind Lab, we showed that the options discovered capture diverse and important properties of the behaviours required by the training task distribution. Through DeepMind Lab, we show that MODAC is scalable, and thus, viable for option discovery in challenging RL domains. promising direction for future work is to investigate whether meta- veta-gradients can also be applied to the discovery of options suitable for planning in model-bare -aaee RL.
Acknowledgement
Part of this work was conducted at the University of Michigan by Vivek Veeriah where he is supported by DARPA's L2M program. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors.
References
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6732-6740, 2021.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/ ftes/1612.000 1612.03801, 2016.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML' 14, pp. 11-316-11-322 JMLR.org. 2014.
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3):337-357, 2016.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pp. 271-271. Morgan Kaufmann Publishers, 1993.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to--eal embodied ai platform. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3164-3174, 2020.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposi- tion. Journal of Artificial Intelligence Research, 13:227-303, 2000.
10

---

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint 0r2.v:1802.06070 2018.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. arXiv preprint arXiv:1710.0976; 2017.
Mohammad Ghavamzadeh and Sridhar Mahadevan. Hierarchical policy gradient algorithms. In ICML,, 2003.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an op- tion: Learning options with a deliberation cost. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critic. In The 22nd International Conference Artificial Intelligence and Stafistics, pp. 2231-2240, 2019.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.05182, 2016.
K. Imazeki and Takashi Maeno. Hierarchical control method for manipulating/graspingg tasks using multi-fingered robot hand. volume 4, pp. 3686 - 3691 vol.3, 11 2003. ISBN 0-7803-7860-1. doi: 10.1 1003.1
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05330 2016.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction reinforcement learning. In AAMAS (1), pp. 299-306. Citeseer, 2008
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 2017.
George Konidaris and Andrew G. Barto. Building portable options: Skill transfer in reinforcement learning. IJCAI, 2007.
James Kostas, Chris Nota, and Philip S Thomas. Asynchronous coagent networks: Stochas- tic networks for reinforcement learning without backpropagation or a clock. arXiv preprint 2019.
Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi, and Joshua Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv preprint arXiv:1604.06057 2016.
Marlos C Machado and Michael Bowling. Learning purposeful behaviour in the absence of rewards.
arXiv preprint arXiv: 2016.
Marlos C Machado, Mare Bellemare, and Michael Bowling. laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Searning-Volume 70, pp. 2295-2304 JMLR, org, 2017.
Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Figenoption discovery through the deep successor representation. In International Conference Learning Representations, 2018.

---

Timothy Mann and Shie Mannor. Scaling up approximate value iteration with options: Better policies with fewer iterations. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp- 127-135, Bejing, China, 22-24 Jun 2014. PMLR.
Timothy A Mann, Shie Mannor, and Doina Precup. Approximate value iteration with temporally extended actions. Journal of Artificial Intelligence Research, 53:375-4335 4338, 2015.
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein, Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the twenty-first international conference Machine learning, pp. 71, 2004.
Amy McGovern and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. 2001.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3318 2018.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well reinforcement learning? arXiv preprint arXiv: 1909. 10618, 2019.
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62, 62, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goat reinforcement learn- ing: Challenging robotics environments and request for research. arXiv preprint 2018.
Janarthanan Rajendran, Richard Lewis, Vivek Veeriah, Honglak Lee, and Satinder Singh. How should an agent practice? arXiv preprint arXiv:1912.07044 2019.
Pravesh Ranchod, Benjamin Rosman, and George Konidaris. Nonparametric bayesian reward seg- mentation for skill discovery using inverse reinforcement learning. In 2015 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 471-477. IEEE, 2015.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv: 10567, 2018.
David Silver and Kamil Ciosek. Compositional planning using optimal option models. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1267-1274, Madison, WI, USA, 2012. Omnipress. ISBN 978 97814503128
Özgür Şimşek, Alicia Wolfe, and Andrew G Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learning, pp. 8 816-823, 2005.
Alec Solway, Carlos Diuk, Natalia Córdova, Debbie Yee, Andrew G Barto, Yacl Niv, and Matthew M Botvinick. Optimal behavioral hierarchy. PLoS computational biology, 10(8), 2014.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi- semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 11(111181-21,, 1999.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. A deep hierar- chical approach lifelong learning in minecraft. In Thisty-First AAAI Conference on Artificial Intelligence, 2017.
Vivek Veeriah Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Advances in Neural Information Processing Systems, pp. 9306-9317, 2019.
12

---

Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, and koray kavukcuoglu. Strategic attentive writer for learning macro-actions. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp- 3486-3494, Curran Associates, Inc., 2016.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396-2401 2018.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado Hasselt, David Sil- ver, and Satinder Singh. Self-tuning deep reinforcement learning. arXiv preprinl 12X2v: 2020.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp- 4644 2018.
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, and Satinder Singh. What can intrinsic rewards capture? arXiv preprint arXiv:1912.0550 2019.
13


Reviews:
[{"review_id": "foj-fcc2Z0", "reviewer": "Reviewer_UfaR", "summary": "Authors propose a meta-learning algorithm for learning temporarily extended actions within the options framework. In addition to the commonly used option modules defined in the original options framework (high-level policy, terminations, sub-policies) the algorithm also uses an aditional learned option-rewards module ($\\{r^{o^i}\\}$) to reward sub-policies when they perform actions. The algorithm is designed to extract useful temporarily extended actions from multiple environments by using meta-gradients (backpropagating through gradient). Meta-gradients are taken wrt. parameters of termination functions and option-rewards whereas sub-policies and high-level policy are updated in the inner update. The intuition behind using option-rewards and terminations as meta-parameters is that they should define task-agnostic sub-goals. \n\nAuthors evaluate the performance of this new algorithm on one simple and one complex navigation tasks. The main performance metric considered is performance on a new unseen task (after pre-training on many tasks), although authors do show additional visualizations of learned options. Comparison is made with two additional algorithms for learning options  MLSH (designed for multi-task learning) and Option Critic (originally designed for single-task) and a single-task Actor-Critic baseline. In experiments, authors show that proposed algorithm outperforms all three baselines when options are transferred to previously unseen tasks. This suggests that proposed algorithm discovers options with better temporal structure.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper introduces an interesting novel method for meta-learning of options. It is clearly written and experiments show that the approach works well on both simple and challenging baseline. However, the method is mostly justified intuitively and I find the theoretical justification of the algorithm and the justification of some design choices that was provided quite weak. It is thus hard to see whether the method is theoretically sound and why it works so well when compared to prior methods. Authors also do not discuss the limitations of the approach. I would thus not recommend accepting the paper unless authors provide a clear derivation of the updates used in the algorithm from the main objective, better justify design choices and discuss limitations.  \n\nOriginality: The method builds on known techniques but includes a novel contribution (using gradient based meta-learning to learn hierarchical policies). Authors explain the differences and relation to related work although I am not completely sure if there isn't a connection missing between adding a negative reward for options switching and deliberation cost [2].\n\nQuality: Claims are mostly supported by experiments, however, I did not find theoretical analysis of the approach sufficient. From the derivations provided in the paper and the appendix I cannot tell if derived updates are sound and as far as I can tell they seem to be at least a few missing dependencies (see detailed comments). Furthermore, in my opinion, the addition of learned option-reward module needs better justification than what is provided because it is usually not part of the options framework and I do not immediately see why \"task reward may be insufficient to discover reusable, task-independent options\", why such addition would be necessary and how it would combine with the options framework. I also did not find the limitations of the approach in the paper.\n\nClarity: The writing is mostly very clear and necessary concepts are well explained. The submission is also well organized. \n\nSignificance: The submission tackles an interesting difficult task (learning hierarchical structure in multi-task learning) and according to experiments outperforms well-known baselines (also on non-trivial task). Additionally, it presents a novel idea: using gradient based meta-learning in combination with the options framework to learn such structure. I believe that this idea and its realization would be a contribution to the research community.\n\n**Detailed Comments\\Questions:**\n- The option-reward module is only shortly introduced in line 95 with the intuition (line 105) that they define an option's subgoal and implicitly defines the option-policy (line 79). However, the connection of this module with the options framework is a bit unclear. I think that authors should provide an explanation why these options-rewards define sub-goals and show what kind of option-reward functions (subgoals) are learned in four rooms environment. I think this idea might be somehow connected to the Option Keyboard [3]. \n\n- I think that addition of \"switching cost $c$\" could be connected to the Deliberation cost [2]. Doesn't this interfere with the objective/update equations?\n\n- From Eq. 3 and the learning algorithm, I understand that $r^o$ are not connected to real rewards in any way. I do not see the motivation for $G_t^o$ or why it is sensible to interpret terminations as a discount. Why does it make sense to interpret $\\beta, r^o$ and $v^o$  as discounted returns? Why isn't normal discounting used? Since $\\beta, r^o$ and $v^o$ are all meta-learned isn't $G_t^o-v^o(s_t)$ just an arbitrary \n\n- The objective and following derivation in the appendix (line 75) does not seem correct since it does not take into account the dependency of sampling distribution $p(G_t^M|\\{\\theta^o_i\\},\\theta^M, \\{\\eta^{\\beta^o_i}\\})$ on terminations $\\beta$. Following line 81 in the appendix, why is only a single $\\log\\pi^o$ inside both expectations? As far as I can tell , it should be whole $p(G_t^M|\\{\\theta^o_i\\},\\theta^M, \\{\\eta^{\\beta^o_i}\\})$ decomposed into multiple components. Furthermore, I think that is not sufficient to differentiate through ${\\theta^o}'$ since ${\\theta^M}'$ (after inner updates) also depends on termination parameters (through sampling distribution as was explained in [1]) . Updates in Eqs. 1,2,5 are also written in terms of single samples but they are actually estimates of expectations which makes a difference in meta-learning setting.  \n\n- In line 55 authors claim that all options directly optimizing the same (main task) reward may not be sufficient to discover reusable task-independent options. Why/In which settings is this a case?  \n\n- In Fig 2 c) you show average return. However in such environments where reward for reaching a goal is one, one usually optimizes for discounted return (since return is always 1 if agents finds the goal). Is there any specific reason why you are showing return instead?  Can you provide plots for discounted return?\n\n- Could you add the visualization of terminations and learned option-rewards module for four rooms task?   \n\nTypos and minor details:\n- line 32 cite\n- line 63 use citations with MLSH and OC? (they are used later instead)\n- use $\\theta'$ on the left side in Eqs 1 and 2\n- introduce L in algorithm box\n- line 166 Coagent Networks citation\n- Fig 2b text is very small\n- line 269 sight instead of site\n- line 299 as far as I remember Mankowitz et al. 2016 does not require hand-designed temporal abstractions\n\n[1] DiCE: The Infinitely Differentiable Monte Carlo Estimator (Foerster et al. 2018)  \n[2] When Waiting is not an Option : Learning Options with a Deliberation Cost (Harb et al. 2017)  \n[3] The Option Keyboard: Combining Skills in Reinforcement Learning (Barreto et al. 2019) \n\n===== Post-Rebuttal ======\nAfter the discussion with authors and other reviewers I decided to keep my score unchanged. Even though I do appreciate the effort that authors put into providing an additional baseline, I do not think that the issues related to theoretical justification of the method were properly addressed. While I realize that the work is mostly empirical, I do think that the empirical adjustments to the method should be better motivated and their effects on theoretical algorithm should be discussed in more detail especially since the performance of the algorithm is not that much better than the one of the single-task agent. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do discuss societal impact but do not discuss the limitations or drawbacks of the approach when compared to related methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "M6i8rOGK0o", "reviewer": "Reviewer_JkKg", "summary": "This paper develops an approach for option discovery using meta-gradient descent to optimize a set of option networks. These option networks are then used to define the reward, and termination functions. The option policies are learned through normal RL in a separate option-network which is used to define options for a network which learns the behaviour policy using options and the base actions. They restrict their setting to that of multi-task reinforcement learning, where they sample from distributions of tasks with shared underlying regularities. They test their approach on several environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a very natural instantiation of the meta-learning for discovery idea proposed by [Veeriah et. al. 2019] to the option discovery problem. Many of the algorithms features are shared, and are directly applicable to the option discovery framework. I believe the ideas and paper have a clear place in the literature, but I'm concerned with the empirical evaluation as is. I'm leaning towards acceptance, but would like some of my points below to be seriously considered in the final revision thus I'm setting my score to a weak reject. I will raise my score if the following are addressed: (S2, S4, Q2, Q3, Q4, S6, S7)\n\n## Introduction, Background, and Algorithm design\n\nS1. I think temporal abstractions, temporally-extended actions, and options are all used interchangeably (which they all are basically interchangeable). This leads to some clarity issues. I think you should choose a term and use it.\n\nS2. I like your hypothesis as it is stated. But I think it could be made more clear and more inline with the rest of the paper. I think the problem is in the focus on discovery in the hypothesis. Instead, maybe \"If a temporal abstraction is /useful/ across /many training tasks/, then this abstraction captures regularities shared by the training tasks leading to a higher likelihood of being useful and reusable in new, /previously unseen tasks/.\" What I like about this rephrasing is it makes clear intuitions about why options are important, why you think the multi-task setting is the right place to test new algorithms, and gives a clear indication that what we are studying is the temporal abstraction (option). The discovery of such things is not important to the hypothesis itself.\n\nAlso within this hypothesis there are several terms which are ambiguous at stating. While they are later defined, having clear definitions at the stating of the hypothesis could be beneficial. This might mean moving the statement of the hypothesis to later sections after enough framework is built up.\n\nS3. I'm not sure I would consider Coagent Networks as a technique for discovering options. I'm fine to disagree with that point, but the mention at line 166 should be cited appropriately (i.e. likely [Thomas, 2011] and [Kostas, 2019]).\n\nS4. I think a problem statement would add clarity to the presented ideas. This will also remove ambiguity on why some baselines are included over others later in the paper. \n\n## Empirical Evaluation\n\nBefore I get into the statements and questions, I would like to focus on the baselines used for the empirical evaluation. Several notable algorithms are missing: [Machado et al. 2017 & 2018], [Harutyunyan et al. 2019], [Gregor et al, 2016], and [Eysenbach et al. 2018]. Now I don't think all these algorithms should be included (especially because the amount of compute is quite large), but I'm not sure I get the reasoning behind not including an unsupervised approach. The claim is because they are motivated by single-task RL they shouldn't transfer to the multi-task setting. I'm not sure I agree with this, and it is possible because they are learning options through an unsupervised mechanism (some without knowledge of the goal states) they could discover options which are useful for many tasks. [Gregor et al, 2016] even mentions as such in section 3. I believe this is the piece of the empirical evaluation that is difficult for me to accept, and is a large flaw in the paper (and a flaw in how option algorithms seem to be evaluated in general).\n\nOne point of concern I've noticed with the results presented in the four rooms environment: the amount of samples used for training all the competitors is significant. Given other option discovery approaches such as [Machado et al. 2018] and [Harutyunyan et al. 2019] use considerably less (on the order of 1000 episodes). If we assume worst case in [Harutyunyan et al. 2019] which is 5000 steps per episode (figure 4), that is at most 5mil samples, which is 100x less than what is used to train your networks. In reality [Harutyunyan et al. 2019] use far less than this. Maybe this is a property of the multi-task setting, but it is hard to know without a comparison.\n\nQ1. What do you mean by usefulness for transfer? Do you mean how much they are used by the manager for testing tasks? Or something else? How are you measuring this?\n\nQ2. What is the motivation for the input space for the four rooms environment? It seems like a lot of information about the whole environment is being passed through the network. Something like only the tabular representation (not the full map of the environment) or the (x,y) coordinates of the agent would be a more reasonable choice.\n\nS5. The plots in figure 2 are quite difficult to read. Maybe it would be better if they were a 2x2 grid rather than 4 plots across the top.\n\nQ3. What are the shown confidence intervals?\n\nQ4. How many runs were used to tune the hyperparameters?\n\nS6. I think including details on hyperparameters in the main paper is an important addition. I think we should always strive to be able to replicate the experimental setup entirely from the details presented in the main paper. Specific values of hyperparameters, and details about frameworks can definitely be pushed to the appendix.\n\nS7. On the number of runs used for the empirical results. While your results look significant given the confidence intervals, I'm not very convinced given the insights from Deep Reinforcement Learning that Matters by Henderson. While computationally it may not be feasible to run more runs for this submission cycle, I think the paper would benefit from this. At the very least, providing confidence intervals using some of the methods discussed by Henderson et al. would be beneficial (say bootstrap). I have this issue for both the four rooms domain and the deep mind lab domain (where number of runs isn't reported).\n\n#### Post response\n\nFrom my conversation with the authors, I think I will increase my score to a 6 for now. There are still concerns over the bias introduced by the deliberation cost as it relates to the meta update, that I will have to think further on.\n\nAs of now, I view this conversation as I view most arguments between theory and empiricism. Because this paper relies much more heavily on being empirical rather than theoretical, these issues might not be problematic for this paper as it is. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the limitations of the paper are not well discussed. Specifically the sample inefficiency. I discuss this lightly in my review, but feel this should be addressed with some more comparisons.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "K7oq7Z67LHj", "reviewer": "Reviewer_zvKE", "summary": "The authors tackle the problem of learning options that would be useful across a range of similar tasks (for example similar mazes with different difficulties). The authors use a meta-gradient approach with appropriately defined parameters and meta-parameters. The paper contains quantitative and qualitative experiments on variants of the FourRooms task and on the more complicated DeepMind Lab (as well as experiments on Atari in the appendix)", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Meta-gradients have been used previously for different settings in RL but this paper seems to be the first to apply it in a HRL setting. The combination of the two could lead to more flexible options that generalize better. The paper is also generally well written and cites many important related work.\n\nThat being said, I wish some of the theoretical and experimental details were better motivated.\n\nIn particular for the experimental aspect:\n1. Although the choice of adding primitive actions to the options set is present in the original paper on options (Sutton et al.), it is not a current practice these days. Importantly, it is not clear whether OC and MLSH are using these primitive actions. It is good to know that options are used the majority of the time, but an ablation analysis looking at the role of adding primitive actions is necessary for me. The reason for this is that it has nothing to do with the meta-gradient approach and it would be enlightening for the community to understand the choice behind adding primitive actions.\n2. The way the deliberation cost is not really principled and not motivated theoretically.  The authors don't explain why the switching cost is added to the manager but not to the options themselves. Notice that in Harb et al. 2018 the derivation indicates that it should be used for all updates. Moreover, shouldn't meta gradient to learn temporally extended options, if these options were generally useful? It is not clear why it is also necessary to add a deliberation cost.\n3. The first claim of the paper is that \"our approach can discover meaningful and diverse temporally-extended options\". Personally, when I look at the qualitative results on DeepMind Lab, I am really not sure how the behaviour of each option is useful or interesting. Moreover, could we see similar options emerging from the baselines?\n4. The MODAC algorithm, which uses learned rewards, is compared to HRL approaches that don't, that is Option-Critic (OC) and MLSH. I don't find this aspect particularly surprising. I think stronger baselines would be necessary here, at least for the smaller experiments.\n\nIn particular for the theoretical aspect:\n1. Regarding Equation 3, the authors mention that when the option terminates, there is no bootstrapping. To me this seems simply wrong. Notice that in OC there is always bootstrapping and that the boostrapping is consistent with whether or not there was termination.\n2. The derivation of the meta-gradient approach, which is the essential contribution of the paper, is very succinct. Even in the appendix, there is barely any details as to the approximations being considered. As a comparison, please consider the original Meta-Gradient paper (Xu et al.) where many more details are provided (basically the whole of section 1).\n3. This leads to the question as to how the choice of the parameters vs hyper-parameters was made. Indeed, according to the original paper on options, the target for updating an option depends on the reward function, the termination function, the policy over options and even the other options' policies. What motivates which functions are considered parameters or meta-parameters? \n\n\n===== Post-Rebuttal ======\n\nI appreciate the authors' efforts to address the points raised by the reviewers. In particular, I appreciate that the author providing a more competitive baseline where reward shaping is used for learning options. I would suggest to include all the details (such as hyperparameter search) for these additional experiments. Given the authors' efforts in addressing the issues, I will be raising my score to a weak accept even though it would be better to have an updated version of the paper to be convinced.\n\nI think that the authors should be more clear in the approximations introduced in the work. This could be addressed by presenting complete and detailed derivations for their method. Many of the issues raised by the reviewers would then be apparent. As it is, the paper presents the update rules but the derivation of these updates are incomplete, and the answers provided during the rebuttal lead to more questions that remain up in the air.\n\nIn particular, both the way bootstrapping is done is not justified theoretically, even though intuitively there could be a motivation. The same goes for the deliberation cost. Why are the authors introducing actions, to then introduce a heuristic to penalize for choosing them? As we are learning options and their length can be changed, it is not necessary to include actions in the first place.\n\nThe meta-gradient method already introduces a set of approximations, and at some point it becomes hard to justify why more heuristics are necessary, especially as the empirical performance is not that much better than using a simple flat agent.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Learning options can lead to better interpretability. This could be highlighted.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "7WQRKEJD-ye", "reviewer": "Reviewer_WMS3", "summary": "The author’s propose an options-based hierarchical RL algorithm that uses meta-learned option-rewards and termination to train individual options.\nThe method is instantiated with an actor-critic RL method and experiments are conducted to test the transfer capabilities of the meta-learned options.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Very clear exposition of the method and good writing overall.\n+ Detailed appendix with hyperparameter settings and method implementation details.\n+ Strong experimental results for options with challenging DeepMind Lab environment and Atari.\n+ Visualizations of learned options.\n\nWeaknesses:\n+ It would be good to also test against a subgoal based hierarchical RL method (f.e. HIRO)\n+ shouldn’t it be $+\\kappa^o v^o$ in equation 1 since value should be pushed up if it's underestimating n-step return?\n+ Some more motivation/analysis for the benefit of using meta-learned rewards and terminations rather than sub-policies would be helpful. Right now the intuition seems to be that it is less task-dependent. In principle meta-learning could cause option-rewards to become task-dependent just like directly learned options?\n\nSummary:\nNovel approach to option discovery using meta-gradients in new way. Strong experimental results and provides a promising direction for options framework and multi-task transfer in RL.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "5pilqOGw2m", "reviewer": "Reviewer_tdpW", "summary": "The authors introduce MODAC, a method that relies on meta-gradients for option discovery in hierarchical multi-task settings. The authors extend the options framework to the multi-task setting and use meta-gradients learning to discover task-independent options that can be successfully re-used across tasks. Their architecture is based on the standard hierarchical agent from Sutton et al. (1999) where a manager, that is conditioned by the current state and the task to achieve, can choose between a set of primitive actions and temporally extended options. Adding primitive actions to the action space of the manager adds flexibility and expressiveness and matches the original hierarchical framework. MODAC also implements option-policies, option-rewards and option-terminations. The training corresponds to two nested loops. An inner loop updates the manager to maximize the sum of discounted environment rewards and the option policies to maximize the discounted sum of option rewards. An outer loop evaluates the updated manager and option-policies and updates the option-rewards and option-terminations by back-propagating through the inner loop updates. This training setting enables MODAC option policies to optimize for different objectives that are parameterized by task-independent option-reward and termination that are discovered to be directly useful across many tasks. The authors evaluate MODAC in a grid world environment and in  DeepMind lab navigation tasks. They illustrate that MODAC does not only call primitive actions but also makes use of the options and that the discovered options are diverse and useful for the task at hand. They also compare MODAC to two baselines and show that MODAC outperforms them either in terms of asymptotic performance or sample efficiency or both.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I warmly thank the authors for their work.\n\nSoundness of the claims, significance and novelty of the contribution, and relevance to the NeurIPS community:\n\n- Discovering options, in an automated manner and without any priors, that can be reused to solve distributions of tasks, has been a long-standing goal in RL and I believe this method brings a step forward in that direction.  Using meta-gradients to learn option-rewards and terminations that are themselves used to train task-independent option-policies is novel and seems to work well.\n\n- Allowing the manager to call primitive actions in addition to options is a strength of the method and the authors demonstrated the interest in doing so through examples and during the experimental study.\n\n- The experimental study is thorough and well illustrates the strengths of the method as well as validates the claims made in the paper.\n\n\n\nLimitations of this work:\n\n- In part 1, I think the authors should briefly introduce the notations relative to MDPs. For instance, the state space S and the goal space G are never properly introduced. It would also be great if the authors may give more information about the nature of the different spaces. For instance, it is never mentioned that if the action space is discrete or continuous. Also it not clear to me what is the nature of the goal space G. Is it assumed to be a sub-part of the state space or a set of possible tasks?\n\n- In part 1, the number of options K is introduced lately and almost implicitly when the authors describe the outer loop of the algorithm. I would recommend introducing it sooner in the part.\n\n- It would be interesting if the authors could discuss to which extent MODAC could be extended to continuous action spaces and assess its performance on traditional HRL benchmarks such as the ones considered in HAC.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work provides a method to discover options in multi-task HRL. Thus, it does not add limitations or potential negative societal impact to existing hierarchical reinforcement learning methods. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The reviewers thought this was an interesting paper considering how to combine meta-gradient methods with HRL, and the empirical experiments were convincing. In the internal discussions two reviewers suggested the paper would be further strengthened by a more detailed discussion on the theoretical side. The authors are encouraged to consider the reviewers’ feedback in their revision for the camera ready.
Decision: Accept (Poster)