Multitask Soft Option Learning
Authors: Maximilian Igl
Andrew Gambardella
Jinke He
Nantas Nardelli
N. Siddharth
Wendelin Böhmer
Shimon Whiteson
Venue:  (2020)
Keywords: Hierarchical Reinforcement Learning
Reinforcement Learning
Control as Inference
Options
Multitask Learning

Abstract: We present Multitask Soft Option Learning (MSOL), a hierarchical multi-task framework based on Planning-as-Inference. MSOL extends the concept of Options, using separate variational posteriors for each task, regularized by a shared prior. The learned soft-options are temporally extended, allowing a higher-level master policy to train faster on new tasks by making decisions with lower frequency. Additionally, MSOL allows fine-tuning of soft-options for new tasks without unlearning previously useful behavior, and avoids problems with local minima in multitask training. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-learning baselines in challenging multi-task environments.

Under review as a conference paper at ICLR 2020
MULTITASK SOFT OPTION LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
We present Multitask Soft Option Learning (MSOL), a hierarchical multitask framework based on Planning Inference. MSOL extends the concept of options, using separate variational posteriors for each task, regularized by a shared prior. This allows tine-tuning of options for new tasks without forgetting their learned policies, leading to faster training without reducing the expressiveness of the hi- erarchical policy. MSOL avoids several instabilities during training in multitask setting and provides a natural way to learn both intra-option policies and their ter- minations. We demonstrate empirically that MSOL significantly outperforms both hierarchical and flat transfer-leaming baselines in challenging multi-task environ- ments.
INTRODUCTION
A key challenge in Reinforcement Learning (RL) is to scale current approaches to higher complexity tasks without requiring a prohibitive number of environmental interactions. However, for many tasks, it is possible to construct or learn efficient exploration priors that allow to focus on more relevant parts of the state-action space, reducing the number of required interactions. These include, for example, reward shaping (Ng et al. 1999 Konidaris & Barto (0066), curriculum Ieriiuluummearnnnggggg (Bengio
al.
2009). ) some meta-learning algorithms (Wang et al. al.] [016 Duan et al., 2016 Gupta et al.
2018
Houthooft et al.) 2018 Xu et al. 2018), and transfer learning (Caruana
1997| Taylor & Stone 2011
Bengio 2012; Parisotto et al. 2015 Rusu et al. 2015 Teh et al. 2017).
Rusu
One promising way to capture prior knowledge is to decompose policies into a hierarchy of sub- policies (or skills) that can reused and combined novel ways to solve new tasks (Dayan & Hinton 1993] Thrun & Schwartz, 1995 Parr & Russell] 1998 [Sutton et al.) 1999 Barto & Mahade- van 2003). The idea Hierarchical RL (HRL) is also supported by findings that humans appear to
employ hierarchical mental structure when solving tasks (Botvinick et al. 2009 Collins & Frank 2016). In such hierarchical RL policy, lower-level, temporally extended skills yield directed behav- ior over multiple time steps. This has two advantages: i) it allows efficient exploration, as the target states of skills can be reached without having to explore much of the state space in between, and ii) directed behavior also reduces the variance of the future reward, which accelerates convergence of estimates thereof. On the other hand, while hierarchical approach can therefore significantly speed up exploration and training. can also severely limit the expressiveness of the final policy and lead to suboptimal performance when the temporally extended skills are not able to express the required policy for the task at hand (Mankowitz et al. 2014).
Many methods exist for constructing and/or learning skills for particular tasks (Dayan & Hinton
1993 /Stton et al.) 1999 McGoovem & Barto [000]] Menache al.) 2002; Simsek & Barto 2209 Gregor et al. 2016 Kulkarn et al. [2166] Bacon et al. 2017] Nachum et al., 201 8a). Training on multiple tasks sinultaneously one promising approach to learn skills that are both relevant and
generalise across tasks (Thrun & Schwartz 1995 Pickett & Barto 2002 Fox et al. 2016 Andreas
al. 2017 Frans et al. 2018). Ideally, the entire hicrarchy can be trained end-to-end on the obtained return, obviating the need specify proxy rewards for skills (Frans et al. 2018).
However, learning hierarchical policies end-to-end in multitask setting poses two major challenges: i) because skills optimize environmental rewards directly, correctly updating them relies already (nearly) converged master policies that use them similarly across all tasks, requiring complex train- ing schedules (Frans al. 2018), and ii) the end-to-end optimization is prone local minima in

---

Under review as a conference paper at ICLR 2020
which multiple skills have learned similar behavior. This second points is explained in more detail in Appendix
In this paper, we propose Multitask Soft Option Learning (MSOL), novel approach to learning hicrarchical policies in a multi-task setting that extends Options (Sutton et al. 1999). a common definition for skills, and casts the concept into the Planning as Inference (PAT) framework (see, e.g.,
Levine 2018 for review). MSOL brings multiple advantages: i) it stabilizes end-to-end multitask training, removing the need for complex training schedules like in [Frans al. (2018), ii) it gives rise to coordination between master policies, avoiding local minima of the type described in Appendix [A] iii) it allows fine-tuning of options, i.e. adapting them to new tasks at test-time without the risk of unlearning previously acquired useful behavior, thereby avoiding suboptimal performance due to restricted expressiveness, iv) and lastly, we show how the seft option framework gives rise to a natural solution to the challenging task of learning option-termination policies.
MSOL differentiates between a prior policy for each option, shared across all tasks, and a flexible task-specific posterior policy. The option prior can be fixed once it is fully trained, preventing un- learning of useful behavior even when the posteriors are updated. On new tasks, the option posteriors are initialized to the priors and regularized towards them, but are still adaptable to the specific task. This allows the same accelerated training as with 'hard' options, but can solve more tasks due to the adjustable posteriors. Furthermore, during option learning, we can train prior and posterior policies simultaneously (Teh et al. 2017), all without the need for complex training schedules (Frans et al., 2018: training is stabilized because only the priors shared across tasks.
Our experiments demonstrate that MSOL outperforms previous hierarchical and transfer learning algorithms during transfer tasks in multitask setting. MsOL only modifies the regularized reward and loss function, but does not require any specialized architecture. In particular, it also does not require artificial restrictions on the expressiveness of either the higher-level or intra-option policies.
2 PRELIMINARIES
An agent's task is formalized as a MDP P,,,,, consisting of the state space S, the action space A. the initial state distribution p. the transition probability P( 18, 9e) of reaching state 8141 by executing action a, in state st, and the reward r(st, an agent receives for this transition.
2.1 PLANNING AS INFERENCE
Planming as inference (PAI) Todorov 2008; Toussaint 2009 Kappen et al. 2012) frames RL as a
mobabilistic-inferemn problem (Levine 2018). 3)...)..)....... The agent learns distribution (als) s) over actions
a given states 8, i.e., a policy, parameterized by Φ. which induces a distribution over trajectories r of length T, i.e., (81, 41, .2....,07. 8T+1):
(T) = p(sn) II (arls)) P(s++1ls;,a;) 1-1
(1)
This can be seen as a structured variational approximation of the optimal trajectory distribution. Note that the true initial state probability p(s1) and transition probability P(s+11/5 ar) are used in the variational posterior, as we can only control the policy, not the environment.
An advantage of this formulation that we can incorporate information both from prior knowledge, in the form of prior policy distribution, and the task at hand through likelihood function that is defined in terms of the achieved reward. The prior policy p(a; (s)) can be specified by hand or, as in our case, learned (see Section 3). To incorporate the reward, we introduce a binary optimality variable O (Levine 2018), whose likelihood is highest along the optimal trajectory that maximizes return: p(O, S| 04) exp (r(s., a.) /B) The constraint r E (- can be relaxed without changing the inference procedure (Levine 2018). For brevity, we denote O = as O2 = (O2 = 1). If given prior policy p(a:|s) (st) explores the state-action space sufficiently, then p(r, O1:r) is the distribution of desirable trajectories. PAI aims to find a policy such that the variational posterior in (T] approximates this distribution by minimizing the Kullback-Lein
(KL) divergence
[()) = (T) p(T, Our)), with pf7, O1T)) = plxi) _ [Ip(all (s) P(x adpl ad- (2)
2

---

Under review as a conference paper at ICLR 2020
2.2 MULTI-TASK LEARNING
In multi-task setting, we have set of different tasks i E T, drawn from task distribution with probability E(i). All tasks share state space and action space A, but each task has its own initial- state distribution p., transition probability P. (s ae), and reward function r,. Our goal is to learn n tasks concurrently, distilling common information that can be leveraged to learn faster on new tasks from T. In this setting, the prior policy po (ay]ss) can be learned jointly with the task- specific posterior policies qs. (ar (Teh et al. 2017). p. To do so, we simply extend ( to
[(for},0) Eing 1ss (rrr
Bing.rng
Σ R
(3)
where 1 ri(s), - In 9e, (aclse) palax]s;) is the regularised reward. Minimizing equivalent to maximizing the regularized reward R,e Moreover,
the loss minimizing the term Eyoul In Re, (as[se) implicitly minimizes the expected KL--ivergence Esceg DKL Igo, (-ss))))
po(ay]s;)
In practise (see Appendix C.I) we will also make use of a discount
factor y [0, 1]. For details on how y arises in the PAI framework refer to Levine (2018).
2.3 OPTIONS
Options (Sutton et al. 1999) are skills that generalize primitive actions and consist of three com- ponents:
an intra-option policy p(ar|se; selecting primitive actions according to the currently active option ii) probability p(bel Se: Rt-1) 1) of terminating the previously active option žt-1, and iii) an initiation set S S, which we simply assume to be Note that by construction, the higher- level (or master-) policy p(=e/=e-1 St, by) can only select new option if the previous option 2;-1 has terminated.
METHOD
We aim to learn reusable set of options that allow for
faster training on new tasks from a given distribution. We learn both intra-option and termination policies, while preventing multiple options from learning the same be- havior.
s
TASK
TASK 2
PRIORS
1000
Pa
0000
To differentiate ourselves from classical 'hard' options, which, once learned, do not change during new tasks, we call our novel approach soft-options (this is further dis- cussed in Appendix B). Each soft-option consists of option prior, which is shared across all tasks, and a task- specific option posterior. The priors of both the intra- option policy and the termination policy capture how an option typically behaves and remain fixed once they are fully learned. At the beginning of training on new task, they are used to initialize the task-specific posterior dis- tribution. During training, the posterior is then regular- ized against the prior to prevent inadvertent unlearning. However, if maximizing the reward on certain tasks is not achievable with the prior policy, the posterior is free to deviate from it. We can thus speed up training using op- tions, while remaining flexible enough to solve any task.
1
12
1000
12
0000
4
0000
0000
1/
Figure 1: Two hierarchical posterior poli- cies (left and right) with common priors (middle). For each task i, the policy con- ditions on the current state sf and the last selected option æ!_ It samples, in order, whether to terminate the last option (b)), which option to execute next (=f) and what primitive action (a)) to execute in the envi- ronment.
Additionally, this soft option framework also allows for learning good priors in multitask setting while avoiding local minima in which several options learn the same be- havior. See Figure for an overview over the hierarchical prior-poster prioe-posterior architecture that explain further below.

---

Under review as a conference paper at ICLR 2020
3.1 HIERARCHICAL POSTERIOR POLICIES
To express options in the PAI framework, we introduce two additional variables at each time step t: option selections Ze, representing the currently selected option, and decisions b, ferminate them and allow the higher-level (master) policy to choose a new option. The agent's 's behavior depends on the currently selected option 2t. by drawing actions a from the intra-option posterior policy 4o, (0rlst,2). 2.). The selection itself is drawn from a master policy 46, 2t, = (1
be) 8(zz zt-1) + by a (z((s),, which conditions b {0, 1), drawn by the lermination posterior policy To, (belst,, 21-1). ) The master policy either continues with the previous 2t-1 or draws a new option, where we set by
at the beginning of each episode. We slightly abuse notation by referring by ö(2z 24-1) ) to the Kronecker delta 0z for discrete and the Dirac delta distribution for continuous 2, The joint posterior policy is
ldh (Ot, Re, be 88s 111 = Ueo. (bel 2-11 1) Lo, (Eel91, be) g (aelse: zc)
(4)
While 2s can be a continuous variable, we consider only 21 E {1... [1...m), where m is the number of available options. The induced distribution (T) over trajectories of task i, T = ($1, by, 21, a1, .2,...,
,.....,,TT ,T, bT, 2T, ar, sT+1), is then
I (T) = (=(s1) 24, byls,, 2 oz).
(5)
3.2 HIERARCHICAL PRIOR POLICY
Our framework transfers knowledge between tasks by shared prior pe pe(as, 21, by st, 2(-1) over all joint policies 4 : pe(ae, 24, be|st, 2t-1) = p (be 21) (1))))) p"(2= /2, 1. b,) py(as|st. 21). By choosing ph. p" and p/ correctly, we can learn useful temporally extended options. The parameterized priors P0 (be js,, zt-1) and p (a, |st, zx) are structurally equivalent to the posterior policies o% and 9/, so that they can be used as initialization for the latter. Optimizing the regularized return (see next section) w.r.t. 0 distills the common behavior into the prior policy and softly enforces similarity across posterior distributions of each option amongst all tasks
The prior p" (zel bt) (1 - b, 8(2; 2 + by 1 selects the previous option 2t-1 if 6; = 0, and otherwise draws options uniformly ensure exploration. Because the posterior master policy is different on each task, there is no need to distill common behavior into a joint prior.
3.3 OBJECTIVE
We extend the multitask objective in (3) by substituting (T, O1.r) and po, (T) with those induced by our hierarchical posterior policy in 4 and the corresponding prior. The resulting objective has the same form but with a new regularized reward that is maximized:
2 (ails1.2))
9 (b: (11. BI
feg i,t
B Il pglas(se.
(6)
pe (br 11))
As we maximize Eg [R this corresponds to maximizing the expectation over r;(s),a)) [K (9 )) DKL (1 + Da (9 (9 Ip])] ]. along the on-policy trajectories drawn from qo, (7). Term ① of the regularization encourages exploration in the space of options. It can also be seen form of deliberation cost (Harb et al. 2017) as it is only nonzero whenever we terminate an option and the master policy needs select another to execute.
Term @ softly enforces similarity between option posteriors across tasks and updates the prior to- wards the 'average' posterior. It also encourages the master policy to pick the most specialized option whose posteriors across all tasks are most similar. In other words, the master policy 9 is encouraged to pick option 2, which maximizes r;, but minimizes term @ by picking the option 2, for which prior 1 and posterior of, are the most similar. Because the prior is the average posterior, this rewards the master policy to pick the most specialized option (that still achieves high reward). As discussed in more detail in Appendix A this allows us to escape the local optimization minima that hard options face in multitask learning, while still having fully specialized options after training.
4

---

get more Under review as a conference paper at ICLR 2020
Lastly, we can use ! to also encourage temporal abstraction of options. To do so, during option learning, we fix the termination prior p to a Bernoulli distribution p" p'()) large a encourages prolonged execution of one option, but allows switching whenever necessary. This is also similar to deliberation costs (Harb et al. 2017) but with a more flexible cost model.
(b) = (1-a)% 1-b Choosing
Additionally, we can still distill termination prior pa which can be used on future tasks. Instead of learning Pg by minimizing the KL against the posterior termination policies, decisive terminations by minimizing
min Σ E, Eyng' [Du (Āpa (:18,, 21 =1)p) (-1s,, 21-1)]]
and (b, 1 21-1 - =- Go. (ze 18,, 2. by = 1) i.e., the learned termination prior dis- tills the probability that the tasks' master policies would change the active option if they had the opportunity. Details on how we optimized the MSOL objective are given in Appendix C
RELATED WORK
Most hierarchical approaches rely proxy rewards to train the lower level components and their terminations. Some of them aim to reach pre-specified subgoals (Sutton et al. 1999; Kulkarni et al, 2016), which are often found by analyzing the structure of the MDP (McGovem & Barto 1001 Menache et al. 2002; Simsek et al. 2005 Simsek & Barto 2009), previously learned policies (Goel & Huber 2003: Tessler al. 2017) predictability (Harutyunyan al., 2019). Those methods typically require knowledge, or a sufficient approximation, of the transition model, both of which are often infeasible.
Recently, several authors have proposed unsupervised training objectives for learning diverse skills based on their distinctiveness (Gregor et al. 2016 Florensa al.] 2017: Achiam al., 2018 Ey senbach et al.) 2019). However, those approaches don't 't learn termination functions and cannot guarantee that the required behavior on the downstream task included in the set of learned skills. Hausman et al. (018)) also incorporate reward information, but do not learn termination policies and are therefore restricted learning multiple solutions the provided task instead of learning a decomposition of the task solutions which can be re-composed to solve new tasks.
third usage of proxy rewards by training lower level policies to move towards goals defined by the higher levels. When those goals are set in the original state space (Nachum et al.) 2018 8a), this approach has difficulty scaling to high dimensional state spaces like images. Setting the goals in a learned embedding space (Dayan & Hinton 1993; Vezhnevets et al.) 2017; Nachum et al. 2018b) can be difficult to train, though. In both cases, the temporal extension of the learned skills are set manually. On the other hand, Goyal et al. (2019) also learn hierarchical agent, but not transfer skills, but to find decisions states based on how much information is encoded in the latent layer.
HiREPS Daniel al. (2012) also take inference motivated approach to learning options. In particular Daniel et al. (2016) propose a similarly structured hierarchical policy, albeit in a single task setting. However, they do not utilize learned prior and posterior distributions, but instead use expectation maximization to iteratively infer a hierarchical policy to explain the current reward- weighted trajectory distribution.
Several previous works try to overcome the restrictive nature of options that can lead to sub-optimal solutions by allowing the higher-level actions to modulate the behavior of the lower-level policies Schaul et al. (2015); Heess et al. (206);; ; Haarnoja et al. (2018). However, this significantly increases the required complexity of the higher-leve policy and therefore the learning time.
The multitask- and transier-learning setup used in this work is inspired by Thrun & Schwartz (9955) and |Pickett Barto (2002) who suggest extracting options by using commonalities between so- lutions to multiple tasks. Prior multitask approaches often rely on additional human supervision like policy sketches (Andreas et al. 2017) or desirable sub-goals (Tessler et al. 2017, Konidaris & Barto] [0077] Mann et al.) 2015) in order to learn skills which transfer well between tasks. In contrast, our work aims at finding good termination states without such supervision. Tirumala et al. (2019) investigate the use of different priors for the higher-level policy while we are focussing on learning transferrable option priors. Closest to our work is Meta Learning of Shared Hierarchies (MI.H) (Frans al. 2018) which, however, shares the lower-level policies across all tasks without
5

---

Under review as a conference paper at ICLR 2020
distinguishing between prior and posterior and does not learn termination policies. As discussed, this leads to local minima and insufficient diversity in the learned options. Similarly to us, Fox et al.] (2016) differentiate between prior and posterior policies on multiple tasks and utilize KL- divergence between them for training. However, they do not consider termination probabilities and instead only choose one option per task. Instead of transferring option policies between tasks, Am- mar et al. (2014) aim to share behavior through a latent embedding. Another interesting approach to multitask learning is (Mankowitz et al., 2016) which learns decision regions that are linear in the state instead of learning nonlinear master- and fermination policies.
Our approach closely related to DISTRAL (Th et al. 2017) with which we share the multitask learning of prior and posterior policies. However, DISTRAL has no hierarchical structure and applies the same prior distribution over primitive actions, independent of the task. As a necessary hierar- chical heuristic, the authors propose to also condition on the last primitive action taken. This works well when the last action indicative of future behavior; however, in Section 5 we show several failure cases where learned hierarchy is needed.
5 EXPERIMENTS
We conduct series of experiments to show: i) when learning hierarchies in multitask setting, MSOL successfully overcomes the local minimum of insufficient option diversity, as described in Appendix A ii) MSOL can learn useful termination policies; iii) MSOL is equally applicable to discrete as well as continuous domains; and iv) using soft options yields fast transfer learning while still reaching optimal performance, even on new, out-of-distribution tasks.
All architectural details and hyper-parameters can be found in the appendix. For all experiments. we first train the exploration priors and options on n tasks from the available task distribution T (training phase is plotted in Appendix E) Subsequently, we test how quickly we can learn new tasks from T (or another distribution T').
We compare the following algorithms: MSOT. is our proposed method that utilizes soft options both during option learning and transfer. MSOL (frozeen uses the soft options framework during learning to find more diverse skills, but does not allow fine-tuning the posterior sub-policies after transfer. DISTRAL (Thh et al. 2017) is a strong non-hierarchical transfer learning algorithm that also utilizes prior and posterior distributions. DISTRAL (+action)) utilizes the last action option-heuristic which works well in some tasks but fails when the last action is not sufficienlly informative. MLSH (Frans) et al. (0188) is multitask option learning algorithm like MSOL, but utilizes 'hard' options for both learning and transfer, i.e., sub-policies that are shared exactly across tasks. It also relies on fixed option durations and requires complex training schedule between master and intra-option policies to stabilize training. We use the author's MI.SH implementation. Lastly, we compare against Option Critic (OC) (Bacon et al., 2017). which takes the task-id as additional input in order to apply it to multitask setting.
5.1 MOVING BANDITS
We start with the 2D Moving Bandits environment proposed and implemented by Frans et al.) (2018), which is similar to the example in Appendix [A] In each episode, the agent receives reward of for each time step it is sufficiently close to one of two randomly sampled, distinguishable, marked positions in the environment. The agent can take actions that move it in one of the four cardinal directions. Which position is not signaled in the observation. Each episode lasts 50 time steps.
We compare against MLSH and DISTRAL to highlight challenges that arise in multitask training. We allow MLSH and MSOL to learn two options. During transfer, optimal performance can only be achieved when both options successfully learned to reach different marked locations, i.e., when they are diverse. In Figure 2(a] we can see that MSOL is able to do so but the options learned by MLSI are not sufficiently diverse, for the reason explain in Appendix [A] DISTRAL, even with the last action provided as additional input, is not able to quickly utilize the prior knowledge. The last action only conveys meaningful information when taking the goal locations into account: DISTRAL agents need to infer the intention based the last action and the relative goal positions. While this is possible, in practice the agent was not able to do so, even with a much larger network. However, longer training allows DISTRAL to perform well MSOL., since its posterior is flexible, denoted

---

Under review as a conference paper at ICLR 2020
Expected Reum
Exrpocted Romnen
Reported Rotur
30-
-2
-
20-
MSOL
(BOa())
-3
MS0L MSOL((oumm)
(istral(-astius)
MLS1I
accion)
Distrall sction) Buit
MLLSH
Datrall aacion)
1.5
Steps
x10
Steps
x10
Stegs
x10
(a) Moving Bandits
(b) Taxi
(c) Directional Taxi
150
Lart
barn rpiios preadiie apriken
125
Ear cucake
Experted Reaam
100
Ordine
Expected
75
50
XSSL
+aation
0-
0.0
0.2
1.0
0.0
1.0
X010
Steps
X10P
(d) Taxi: Generalization
(c) Taxi: Adaptation
(f) Swimmer
Figure 2: Performance during testing of the learned options and exploration priors. Each line is the median over 5 random seeds (2 for MLSH) and shaded areas indicate standard deviations. More details text.
by (+ationl(+aation)) limit". Lastly, (froeee) also outperforms (+StRAl(+action) and MLSHH, but performs worse that MSOL. This highlights the utility of making options soft, i.e. adaptable.
5.2 TAXI
Next, we use slightly modified version of the original Taxi domain (Dietterich) 1998) to show learning of termination functions as well as transfer- and generalization capabilities. To solve the task, the agent must piek up a passenger on one four possible locations by moving their location and executing special 'pickup/drop-off" action. Then, the passenger must be dropped off one of the other three locations, again using the same action exccuted at the corresponding location. grid and flag indicating
The domain has discrete state space with 30 locations arranged whether the passenger was already picked up. The observation is one-hot encoding the discrete between
state, excluding passenger- and goal location. This introduces information-asymmeett the task-specific master policy, and the shared options, allowing them to generalize well (Galashov al. 2018). Walls (see Figure 3 limit the movement of the agent and invalid actions.
We investigate two versions of Taxi. In the original (Dietterich) 1998 just called Tax)), the action space consists of one no-op, one 'pickup/drop-ole action and four actions to move all cardinal directions. In Directional Taxi, we extend this setup: the agent faces in one of the cardinal directions and the available movements are move forward or rotate either clockwise or counter-clockwise.
In both environments the set of tasks T are the 12 different combinations of pickup/drop-off loca- tions. Episodes last most 50 steps and there is reward of for delivering the passenger to its goal and penalty of -0.1 for each time step. During training, the agent is initialized to any valid state. During testing, the agent is always initialized without the passenger on board.
We allow four learnable options in MISH and MSOL.. This necessitates the options to be diverse, i.c. one option to reach each of the four pickup/drop-off' locations. Importantly, also requires the options to learn to terminate when passenger is picked up. As one can see in Figure 2(b) MLSII struggles both with aption-diversing and due to its fixed option duration which is not flexible enough for this environment. DISTRAL(+action) performs well in the original Taxi environment, in Figure 2bb) since here the last action is good indicator for the agent's 's intention. However, in the directional shown in Figure 2(c) the actions are less informative and make much harder

---

Under review as a conference paper at ICLR 2020
BY
Options learned with MSOL on taxi domain, before (top) and after pickup (bottom). The light gray area indicates walls. The left plots show the intra-option policies: arrows and colors indicated direction of most likely action, the size indicates its probability. A square indicates the pickup/dropoff action. The right plots show the termination policies: intensity and size of the circles indicate termination probability.
Figure 3:
for DISTRAL to use prior knowledge. By contrast, MSOL performs well in both taxi environments. Comparing its performance with MSOL(frozen) shows the utility of adaptable soft options during transfer.
Figure [] which visualizes the options learned by MSOL,, shows that it successfully learns useful movement primitives and termination functions. The same soft option represents different behavior depending on whether it already picked up the passenger. This expected as this behavior does not need to terminate the current option on three of the 12 tasks.
Next we show how learning information-ssymmetrii soft options can help with transfer to unseen tasks. In Figure [(d) we show learning on four tasks from T using options that were trained on the remaining eight, comparing against A2C and oC. Note that in oC, there no information- asymmetry: We share the same networks across all tasks and provide the task-id as additional input, including to the option-policies. This prevents them from generalizing well to unseen tasks. On the other hand, withholding the task-information from them would be similar to MLSH, which already showed to struggle with local minima. The strong performance of MSOL on this task shows that we need soft options to be able to train cooomatiio-asyymmenn options that generalize well.
We also investigate the utility of flexible soft options: In Figure [(e) we show learning performance on twelve changed tasks in which the pickup/dropoff locations where moved by cell while the options were trained with the original locations. As expected, hard options are not able to solve this tasks. Even with additional access to primitive actions, exploration inefficient (Jong al.) 2008). On the other hand, MSOL is able to quickly learn this new task by adapting the previously learned options, outperforming hard options and flat policies learned from scratch.
5.3 SWIMMER
Lastly, we show that MSOL. can also be applied to continuous multitask domains. In particular, we investigate the MuloCo environment 'Swimmer' (Todorov et al.) 20122] Brockman et al., 2016)_ Instead of rewarding forward movement as in the original implementation, now the rewarded move- ment direction depends on the task from T {up, down, left, right). We also include small amount of additive action noise (details in the Appendix). We show that MSOL performs competi- tive even in the absence of known failure cases of DISTRAL (see Figure ((f))
DISCUSSION
Multitask Soft Option Learning (MSOL) proposes reformulating options using the perspective of prior and posterior distributions. This offers several key advantages.
First, during transfer, it allows us to distinguish between fixed, and therefore knowledge-preseeeee preserving option priors, and flexible option posteriors that can adjust to the reward structure of the task at hand. This effects similar speed-up in learning as the original options framework, while avoiding sub-optimal performance when the available options are not perfectly aligned to the task. Second, utilizing this 'soft' version of options in multitask learning setup increases optimization stability and removes the need for complex training schedules between master and lower level policies. Fur- thermore, this framework naturally allows master policies coordinate across tasks and avoid local minima of insufficient option diversity. also allows for autonomously learning option- intiiontermination policies, very challenging task which is often avoided by fixing option durations manually.
8

---

Under review as a conference paper at ICLR 2020
Lastly, using this formulation also allows inclusion of prior information in principled manner with- out imposing too rigid a structure on the resulting hierarchy. We utilize this advantage to explicitly incorporate the bias that good options should be temporally extended. In future research, other types of information can be explored. As an example, one could investigate sets tasks which would benefit from learned master prior, like walking different types of terrain.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint IrXiv:1807.10220 2018.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In International Conference on Machine Learning, pp. 1206-1214, 2014.
policy sketches. International Conference on Machine Learning, pp. 166-175, 2017. Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI, pp. 1726- 1734, 2017.
Andrew G Barto and Sridhar Mahadevan, Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1-2)41-77, 2003.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Pro- ceedings of ICML. Workshop Unsupervised and Transfer Learning, pp. 17-36, 2012.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
Matthew M Botvinick, Yael Niv, and Andrew C Barto. Hierarchically organized behavior and its neural foundations: reinforcement learning perspective. Cognition, 113(3):262-22 2009.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 16X.v:1606.015400 2016.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Anne Gabrielle Eva Collins and Michael Joshua Frank. Neural signature of hierarchically structured expectations predicts clustering and transfer of rule sets in reinforcement learning. Cognition, 152:160-169, 2016.
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Christian Daniel, Herke Van Hoof, Jan Peters, and Gerhard Neumann. Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3)3377-57, 2016.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural infor- mation processing systems, pp. 271-278, 1993.
Thomas G Dietterich. The maxq method for hierarchical reinforcement learning. In ICML, vol- ume 98, pp. 118-126. Citeseer, 1998.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02777 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference Learning Representa- tions, 2019. URL httpp::::: http://pppeeeeveeeeeee wttp::eeneeeeettteeteeeeeeeeee 35 (RmFm
net
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv:1704.000 2017.

---

Under review as a conference paper at ICLR 2020
Roy Fox, Michal Moshkovitz, and Naftali Tishby. Principled option learning in markov decision processes. arXiv preprint arXiv:1609.05524 2016.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. META LEARNING SHARED HIERARCHIES. In International Conference on Learning Representations, 2018. wttpeewwwee/.rrrrrtt////m net forum? Sdd
URL
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nico- las Heess. Information asymmetry in kl-regularized rl. 2018.
Sandeep Goel and Manfred Huber. Subgoal discovery for hierarchical reinforcement learning using learned policies. In FLAIRS conference, pp. 346-350, 2003.
Anirudh Goyal, Riashat Islam, DJ Strouse, Zafarali Ahmed, Hugo Larochelle, Matthew Botvinick, Sergey Levine, and Yoshua Bengio. Transfer and exploration via the information bottleneck. In International Conference on Learning Representations, 2019. URL [tL[itttt//::peeeeeett
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint (X.v:1611.00500) 2016.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. arXiv preprint arXiv: (e0i::/800.022555 2018.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. arXiv preprint / 820000808 2018.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting not an option: Learning options with deliberation cost. arXiv preprint 1rXiv:1709.04571, 2017.
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup.
The termination critic, arXiv preprint arXiv:1902.09996 2019.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang. Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learn- ing Representations, 2018
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.05182. 2016.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arXiv: 1802. 0482/, 2018.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems- Volume 1, pp. 299-306. International Foundation for Autonomous Agents and Multiagent Systems, 2008
Hilbert J Kappen, Vicenç Gómez, and Manfrre Opper. Optimal control as a graphical model infer- ence problem. Machine learning, 87(2):159-18) 2012.
George Konidaris and Andrew Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 489-496. ACM, 2006.
George Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement learning. IJCAL, volume pp. 895-900, 2007.
Ilya Kostrikov. Pytorch implementationss of reinforcement learning algorithms.
https://
github.com http://www.....oo/o////////oa//.
2018.
10

---

Under review as a conference paper at ICLR 2020
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review.
arXiv preprint arXiv: (00.:0v08000000000 2018.
Daniel Mankowitz, Timothy A Mann, and Shie Mannor. Time regularized interrupting options. In Internation Conference on Machine Learning, 2014.
Daniel Mankowitz, Timothy A Mann, and Shie Mannor. Adaptive skills adaptive partitions (asap). In Advances in Neural Information Processing Systems, pp. 1588-1596, 2016.
Timothy Mann, Shie Mannor. and Doina Precup. Approximate value iteration with temporally extended actions. Journal of Artificial Intelligence Research, 13:375-438, 2015.
Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. Proceedings of the Eighteenth International Conference on Machine Learming. ICML '01, pp. 361-368, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-778-1. URL htttt:////l
acm wtt.:aa...c...r/...i.aaoeee/oo.e ? fo5..o......mm..m553333
655681
Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut-dynamic discovery of sub sub-goals in reinforcement learning. In European Conference on Machine Learning, pp. 295-306. Springer, 2002.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint weXiv:1805008290 2018a
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:1810.01257, 2018b.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic; Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.066 2015.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Advances in neural information processing systems, pp. 1043-1049, 1998.
Mare Pickett and Andrew G Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In ICML, pp. 506-513, 2002.
Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint arXiv: 2015.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pp. 1312-1320, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02455 024388 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347 2017.
Özgür Şimşek and Andrew G Barto. Skill characterization based on betweenness. In Advances in neural information processing systems, pp. 1497-1504, 2009.

---

Under review as a conference paper at ICLR 2020
Özgür Şimşek, Alicia P Wolfe, and Andrew G Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. Proceedings of the 22nd international conference on Machine learning, pp. 816-823. ACM, 2005.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211. 1999.
Matthew Taylor and Peter Stone, introduction to intertask transfer for reinforcement learning. Ai Magazine, 32(1):15, 2011.
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4499-4509, 2017.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hicrar- chical approach to lifelong learning in minecraft. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. Advances in neural information processing systems, pp. 385-392, 1995.
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint arXiv: /903.0748, 2019.
Emanuel Todorov. General duality between optimal control and estimation. In Decision and Control, 2008. CDC 2008. 47th IEEE Conference on, pp.
IEEE, 2008.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RS. International Conference on, pp. 5026- 5033. IEEE, 2012.
Mare Toussaint. Robot trajectory optimization using approximate inference. In Proceedings the 26th annual international conference on machine learming, pp. 1049-1056. ACM, 2009.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference Machine Learning, pp. 3540-3549, 2017.
Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: /61 /1/////1110.6,..11111 2016.
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning explore via meta-policy gradient. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5463-5472, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
12

---

Under review as a conference paper at ICLR 2020
A
A
A
(a)
bb)
(a) (b)
22
Z1
(a)
1
1
~ λ
1 21
21
2
22
(a)
(h)
22 ~ 2
22
B
B
B
(b) 22
(a) Local minimum
(b) Using skill priors
(c) After training
Figure 4: Hierarchical learning of two concurrent tasks (a and b) using two options (2) and 22) to reach two relevant targets (A and B). a) Local minimum when simply sharing options across tasks. b) Escaping the local minimum by using prior (2:) and posterior (z(5) policies. c) Learned options after training. Details are given in the text in Appendix A
Appendix
A LOCAL MINIMA IN MULTITASK OPTION LEARNING
Here aim is to provide intuitive explanation of why learning hard options in multitask setting can lead to local minima. In this local minimum, multiple options have learned the same behavior and are unable change it, even if doing so would ultimately lead Moving Bandits experiment schematically depicted in Figure g as example: The agent (black dot) observes two target locations A and B but does not know which one is the correct one that has be reached in order to generate a reward.
higher reward. We use the
The state- and action-spaces are continuous, requiring multiple actions to reach either A B from the starting position. Consequently, having access to two options, one for each location, can accel- erate learning. We denote the options by 21 and 22.
Assume that due to random initialization or later discovery of target B, both skills currently reach A. Furthermore, assume that we are training on multiple task simultaneously as often the case in RL. In this situation, the master policies on tasks in which the correct goal is A are indifferent between using ž and 22 and will consequently use both with high probability.
Now, changing one skill, e.g. 22, towards B in order to solve tasks in which B is the correct target, decreases the performance on all tasks that currently use 22 to reach target A because for hard options the skills are shared exactly across tasks. Due to small, stochastic gradient updates changing 22 towards B only slightly increases the probability of reaching B but significantly decreases the probability of reaching A (because now the option 22 will at first point neither A or B). Averaged across all tasks, this would at first decrease the average reward, consequently preventing any option from changing away from A, leaving B unreachable.
To "free up" 22 (or any other option) and learn a new skill, all master policies would need to coor- dinate and refrain from using it to reach A and instead use the equally useful skill 21 exclusively. Importantly, using soft options makes this possible.
In Figures 4bb and 4[c) depict this schematically: The key difference is that in MSOL we have
separate task-specific posteriors 2(() and 20x) for tasks a and b and soft options E {1,2] (for simplicity, we assume that the correct target is A for task and B for task b). This allows us, in a first step, to solve all tasks (Figure 4(b) Later, as discussed in Section 3.3 maximizing the regularized reward (6) will lead to specialized skills, i.e. option priors (Figure 4(c).
The corresponding shared priors ž, (one per option) are updated to effectively yield the 'average" (as measured by the KL-divergence) between the task-specific posteriors 2{") and 2; Allowing
the posteriors of an option, for example ((a) and (M) to deviate from the prior 2n allows that option, in first step, to solve both tasks, irrespective of the misaligned priors. This is shown in Figure (b)
13

---

Under review as a conference paper at ICLR 2020
where all posteriors of moving to A, but for 22, the posteriors different tasks and b move to different targets.
Crucially, to maximize the regularized reward (6), the KL-divergences between priors and posteriors should minimized along they trajectories, i.c. weighted by how likely they are to occur. Conse- [|2]] > 0 can be avoided on task by picking the more specialized option 21 for which [= [| O. This, in turn, "frees up" #2 to now also specialize B as shown in Figure 4(c) After this implicit coordination, 21 is always used to reach A and 22 (6) is always used to reach B, allowing both 31 and Σg to also lead to A and B respectively.
quently, the higher Du( (a)
B RELATIONSHIP BETWEEN MSOL AND CLASSICAL OPTIONS
Assume we faced with new task and given some prior knowledge in the form set of skills that we can use. Using those skills and their termination probabilities as prior policies p" and p' in the soft option framework, we can see as a temperature parameter determining how closely we are restricted to following them. For B + oo we recover the classical 'hard' option case and our posterior option policies are restricted to the prior For 3 the priors only initialize the otherwise unconstrained policy, quickly unlearning behavior that may be useful down the line. Lastly, for 0 < 3 < oo we use the prior information to guide exploration but are only softly restricted to the given skills and can also explore and use policies 'close' to them.
C MSOL TRAINING DETAILS
C.I OPTIMIZATION
Even though R depends on Φ;, its gradient w.r.t. de vanishes. Consequently, we can treat the regularized reward as a classical RI reward and use any RI algorithm to find the optimal hierarchical policy parameters Φ;. the following, we explain how to adapt A2C (Mnih et al., 2016) to soft options. The extension to PPO (Schulman al. 2017) is straightforward
The joint posterior policy in (4) depends on the current state 8, and previously selected option
Ft-1- The expected sum of regularized future rewards of task i, the value function V,, must therefore also condition on this pair:
Vi(s,, = Brog
Σ P'=l
7
81, 2-1 i.
(8)
As Vi(86, =1-1) cannot be directly observed, we approximate with parametrized model Vo, (5t, 1). The k-step advantage estimation at time of trajectory 7 is given by
As. (Te(e++))
ΣT + 7
1 Vo, (St, 2-1).
(9)
where the superscript* '_' indicates treating the term as constant. The approximate value func- tion Vo. can be optimized towards its bootstrapped k-step target by minimizing Ly (ó,, TTT)) == Ex (Ao,(Te:t++))) As per A2C, k [1... depending on the state (Mnih et al. 2016). The corresponding policy gradient loss is
La(di
Σ t=1
(T((T(()))) (aa 2n b,(st,
The gradient w.r.t. the prior parameters 0 is
VoLp(O, brrr)
Σ
Vo In (arlst, 24) Vo In (6.1s1)
(10)
However, this limiting case optimization using the regularized reward is not possible.
2f p(z) V In p(z) dr Vp(z) dr V p(x)) dæ = 0.
'However, for PAI frameworks like ours, unlike in original PPO implementation, the advantage function must he updated after cach epoch.
'Here ignore 8 as it is folded into Ap later.
14

---

Under review as a conference paper at ICLR 2020
where by = 82 (24) and z (z(lss, 2-1, b, 1). To encourage exploration in all policies of the hierarchy, we also include an entropy maximization loss:
(0s. TI:T)
In 10 2. b.) In In (at 2e) + In qz. (bels);211))) 2-)) (11)
Note that term ① in (6) already encourages maximizing Lu (dr, 7) for the master policy, since we chose uniform prior pd (zx bb, 1). As both terms serve the same purpose, we are free to drop either one of them. our experiments, we chose drop the term for qth in Rt which proved slightly more stable to optimize that the alternative.
We can optimize all parameters jointly with a combined loss over all tasks i, based on sampled trajectories 7¹ : - ~ go: and corresponding sampled values of i == birr
(( (.))) Σ (Ca(os, Av Ap B) + (
C.2 TRAINING SCHEDULE
For faster training. is important to prevent the master policies q" from converging too quickly to allow sufficient updating of all options. On the other hand, lower exploration rate leads to more clearly defined options. We consequently anneal the exploration bonus A with a linear schedule during training.
Similarly, a high value of / leads to better options but can prevent finding the extrinsic reward (s,, az) early on in training. Consequently, we increase 3 over the course of training, also using linear schedule.
D ARCHITECTURE
All policies and value functions share the same encoder network with two fully connected hidden layers of size 64 for the Moving Bandits environment and three hidden layers of sizes 512, 256, and 512 for the Taxi environments. Distral was tested with both model sizes on the Moving Bandits task to make sure that limited capacity is not the problem. Both models resulted in similar performance, the results shown in the paper are for the larger model. On swimmer the encoder model size 1024 x 256 x 512. Master-polle pestirr-oliices, as well as all prior- and posterior policies and value functions consist of only one layer which takes the latent embedding produced by the encoder as input. Furthermore, the encoder is shared across tasks, allowing for much faster training since observations can be batched together.
Options are specified an additional one-hot encoded input to the corresponding network that is passed through a single 128 dimensional fully connected layer and concatenated to the state em- bedding before the last hidden layer. We implement the single-column architecture of Distral hierarchical policy with just one option and with modified loss function that does not include terms for the master and termination policies. Our implementation builds the A2C/PPO implementa- tion by Kostrikov (2018), and we use the implementation for MLSI that is provided by the authors bittps://gitee com/ cttt:/opeaa...o aemeenaimmmmsss mm
E HYPER-PARAMETERS AND ADDITIONAL ENVIRONMENT DETAILS
We use 2)v d^ Ap = 1 in all experiments. Furthermore, we train on all tasks from the task distribution, regularly resetting individual tasks by resetting the corresponding master and re-initializing the posterior policies. Optimizing 3 for MSOL. and Distral was done over {0.01, 0.02, 0.04, 0.1, 0.2, 0.4). We use 7 = 0.95 for Moving Bandits and Taxi and y = 0.995 for Swimmer.
E. MOVING BANDITS
For MLSH, we use the original parameerssssssss (Frans et al., 2018). The duration of each option is fixed to 10. The required warm-up duration is set 9 and the training duration 1. We also
15

---

Under review as a conference paper at ICLR 2020
use 30 parallel environments split between 10 tasks. This and the training duration the main differences to the original paper. Originally, MLSH was trained on 120 parallel environments which were unable to do due hardware constraints, Training is done over 6 million frames per task.
For MSOL and Distral we use the same number of 10 tasks and 30 processes. The duration of options learned and we do not require warm-up period. We set the learning rate to 0.01 and / 0.2, a 0.95, Aи = 0.05. Training is done over 0.6 million frames per task. For Distral we use 0.04, Aн 0.05 and also 0.6 million frames per task.
E.2 TAXI
For MSOL we anneal / from 0.02 to 0.1 and λи from 0.1 to 0.05. For Distral we use 3 = 0.04. We use processes per task to collect experience for batch size of 15 per task. Training done over 1.4 million frames per task for Taxi and million frames per task for Directional Taxi. MLSH was trained on 0.6 million frames for Taxi as due to it's long runtime of several days, using more frames was infeasible. Training was already converged.
E.3 TAXI: ADAPTATION TO CHANGED TASKS
As shown in the main experiments, soft options can still be useful, even when the task distribution changes. It is unsurprising that hard options are not able solve the task. However, interestingly, providing hard options and primitive actions to the master policy performs much worse than just learning from scratch. This phenomenon was investigated previously by Jong et al. (0008) and further justifies using soft options for transfer to out-of-distribution tasks.
Whether training from scratch or re-using misspecified options that were trained on a different set of tasks is learning faster mainly depends on i) how strongly the options are misspecified and ii) how difficult the exploration problem is the environment. This tradeoff is shown in Figure [ On smaller 8x8 grid (left), learning from scratch performs competitively because exploration sufficiently simple. On the other hand, on a 10x10 grid (right, results from the main paper), explo- ration is harder and soft options allow for significantly faster learning because they can guide the exploration in a helpful way.
E.4 SWIMMER
For training Distral and MSOL we use PPO instead of A2C as it generally achieves better perfor- mance on continuous tasks. We have λи 0.0004 for both MSOL and Distral for primitive actions and X2 = 0.1 for the master- and termination policies in MSOL. We use learning rate 0.0002, GAE (Schulman et al.) 2015) with 7 0.98. We collect 2000 steps in parallel on 6 processes per task, resulting in batchsize of 12, 000 per task. Training is done over 6 million frames with a linearly scheduled increase of / from 0 to 0.04 for MSOL and 0.01 for Distral. We set 0.98.
Eipeoed BAAEE
0-
Expsted
Expeced
231
Bipeded
m5
co-
1000 Datre
MESIL
Datral
25-
Dieecall (ontee)
0.1
0.2
0.6
0.6
X
Stego
K100
Stepe
K10
(a) Moving Bandits
(b) Taxi
(c) Directional Taxi
(d) Swimmer
Figure 5: Performance during training phase. Note that MSOL. and MSOL (frozee) share the same training they only differ during testing. Further, note that the highest achievable performance for Taxi and Directional Taxi is higher during training as they can be initialized closer to the final goal
(i.e. with the passenger board).
16

---

Under review as a conference paper at ICLR 2020
sct cpto traaffe
lant cptioe lrae
HSt bntted
Soppected Settume
Sopected
0.0
0.2
0.4
0.0
30ape
2bope
X100
(a) Grid 8x8
(b) Grid of size 10x10
Figure 6: Tradeoff between exploration and misspecification: For smaller grid sizes, exploration is casier and learning from seratch performs competitively. For larger grid sizes, soft options accelerate training through faster exploration, even if they are misspecified because they were trained on different of tasks.
17


Reviews:
[{"review_id": "S1xF8IXHtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about learning hierarchical multitask policies over options.  The hierarchical prior policy is shared amongst all tasks, while the hierarchical posterior policies are allowed to adapt to specific tasks.  Once the prior is learned, it is fixed.   The parameters of the posterior policies are adjusted via an adapted version of A2C.  \n\nI liked the flow and the organization of this paper.\n\nComments and questions:\n* I see that term 1 and term 3 of equation (6) working together to ensure some kind of exploration and exploitation.  Term 2 controls how the option posterior deviates from the prior.  However, when the ratio is 1 or less than 1, the value of (6) would increase, and both cases would have made the posterior more like the prior.  There seems to be no other term that incentivizes the option posterior to deviate, and I do not see how the options are adapting to tasks.  \n\n* The term 1,2,3 in (6) are weighted equally by beta and cannot be fine-tuned to desired trade-offs.  \n\n* Should there be \\xi(i) multiplying the last term in (3)?  \n\n* What does delta(z_t - z_{t-1}) mean in section 3.1?  It was not defined anywhere.  \n\nMisspelling and typos:\n* page 5: optimized is misspelled in \"Details on how we optimiszed...\" \n\n* Appendix C.1: should there not be a superscript dash on A_{\\pi_i} since the superscript dash carries the meaning that the term is a constant.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1li36yCYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a method for learning hierarchical policies in a multi-task setting built on options and casts the concepts into the Planning as Inference framework. They claim that the method brings several advantages such as:\n-Stabilize end-to-end mtl,\n-Leads to coordination b/w master policies,\n-Allows fine-tuning of options\n-Learn termination policies naturally\n\nThe proposed approach derives the reward Eq.6 by extending the graph to options for Levine’s tutorial. Eq.6 is simply the extension of the reward of maximum entropy RL to the options framework. The ideas presented in the paper are interesting, but I have concerns about the scalability of such an approach. Please see the detailed comments below.  Additionally, please note that although I have marked a weak reject, I am open to adjusting my score if the rebuttal addresses enough issues.\n\nDetailed Comments:\nA primary weakness of this approach is that it seems like there is one network that learns the options and is shared across all task (that would be the prior) and then there is a task-specific network for all options (posterior), wouldn’t this be very difficult to scale if we want to learn reusable options over the lifetime of an agent? If there are n tasks, do you need to use n different networks? \n\nThe authors assume that all options are present everywhere i.e. I ⊆ S. I think the work could benefit from removing this assumption.\n\nThe authors mention that unlike (Frans et al., 2018), they learn both intra-option and termination policies: there is definitely more work that aims to learn both the skill and its termination. It would be more complete to cite additional references here that learn both of these or rephrase this sentence. \n\nIt does not seem clear why “term 1 of the regularization is only nonzero whenever we terminate an option and the master policy needs to select another to execute.” This doesn’t seem true as this is a ratio of the two probabilities and not just the instantiation of the random variable. \n\nThe results in moving bandits alone are very convincing. However, in Taxi (2b) distral+action seems to be as good as/even better MSOL. In directional Taxi (2c) Distral(+action) manages to reach the same final performance (if we care about that), can you please comment on this.\n\nSome parts of the experiments section does not seem clear to me, Does the proposed approach use a network per task? if yes, then it is obvious that their method could improve over learning on 12 tasks with one set of network. Please clarify. \n\nOne major concern is that the only high dimensional experiment is a swimmer and it is not immediately clear how much do we gain there. Distral is relatively closer in performance to both MSOL and MSOL frozen. I would recommend evaluation in a variety of high-dimensional domains such as other instances in Mujoco, and visual domains. In particular, the proposed ideas would make a stronger case if the baselines included other multitask hierarchical agents such as [4] for example. A discussion including some of the missing relevant related multi-task literature would also be helpful [1,2,4,5,6].\n\n[1] Mann, T. A., Mannor, S., & Precup, D. (2015). Approximate value iteration with temporally extended actions. JAIR, 53, 375-438.\n[2] Konidaris, G., & Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI,\n[3] Andreas, J., Klein, D., & Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. ICML\n[4] Tessler, Chen, et al. \"A deep hierarchical approach to lifelong learning in minecraft.\" Thirty-First AAAI Conference on Artificial Intelligence. 2017.\n[5] Ammar, Haitham Bou, et al. \"Online multi-task learning for policy gradient methods.\" International Conference on Machine Learning. 2014.\n[6] Mankowitz, Daniel J., Timothy A. Mann, and Shie Mannor. \"Adaptive skills adaptive partitions (ASAP).\" Advances in Neural Information Processing Systems. 2016.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Apologies for only receiving two reviews. R2 gave a WR and R3 gave an A. Given the lack of 3rd review and split nature of the scores, the AC has closely scrutinized the paper/reviews/comments/rebuttal. Thoughts:
 - Paper is on interesting topic.
 - AC agrees with R2's concern about the evaluation not using more complex environments like Mujoco. Without evaluation on a standard benchmark, it is difficult to know objectively if the approach works. 
 - AC agrees with authors that the DISTRAL approach forms a strong baseline. 
 - Nevertheless, the experiments aren't super compelling either.
 - AC has some concerns about scaling issues w.r.t. model size & #tasks. 

The paper is very borderline, but the AC sides with R2's concerns and unfortunately feels the paper cannot be accepted without a stronger evaluation. With this, it would make a compelling paper. 
Decision: Reject