Learning Reusable Options for Multi-Task Reinforcement Learning
Authors: Francisco M. Garcia
Chris Nota
Philip S. Thomas
Venue:  (2020)
Keywords: Reinforcement Learning
Temporal Abstraction
Options
Multi-Task RL

Abstract: Reinforcement learning (RL) has become an increasingly active area of research in recent years. Although there are many algorithms that allow an agent to solve tasks efficiently, they often ignore the possibility that prior experience related to the task at hand might be available. For many practical applications, it might be unfeasible for an agent to learn how to solve a task from scratch, given that it is generally a computationally expensive process; however, prior experience could be leveraged to make these problems tractable in practice. In this paper, we propose a framework for exploiting existing experience by learning reusable options. We show that after an agent learns policies for solving a small number of problems, we are able to use the trajectories generated from those policies to learn reusable options that allow an agent to quickly learn how to solve novel and related problems.

Under review as a conference paper at ICLR 2020
LEARNING REUSABLE OPTIONS FOR MULTI-TASK REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) has become an increasingly active area of research in recent years. Although there many algorithms that allow agent to solve tasks efficiently, they often ignore the possibility that prior experience related to the task at hand might be available. For many practical applications, it might be unfeasible for an agent to learn how to solve a task from scratch, given that it is generally computationally expensive process; however, prior experience could be leveraged to make these problems tractable in practice. In this paper, we propose a framework for exploiting existing experience by learning reusable options. We show that after an agent learns policies for solving small number of problems, we are able to use the trajectories generated from those policies to learn reusable options that allow an agent to quickly learn how to solve novel and related problems.
INTRODUCTION
Reinforcement learning (RL) techniques have experienced much of their success in simulated en- vironments, such as video games (Mnih et al.] (2015) or board games (Silver et al.) 2016) Tesauro 1995). One of the main reasons why RL has worked so well in these applications is that simulate millions of interactions with the environment in relatively short period of time, In many real world applications, however, where the agent interacts with the physical world, it might not be easy to generate such a large number of interactions. The time and cost associated with training such systems could render RL an unfeasible approach for training in large scale.
able
As a concrete example, consider training large number of humanoid robots (agents) to move quickly, as in the Robocup competition (Farchy et al., 2013). Although the agents have similar dynamics, subtle variations mean that single policy shared across all agents would not be an ef- fective solution. Furthermore, learning a policy from scratch for each agent is too data-inefficient to be practical. As shown by Farchy al. (2013), this type of problem can be addressed by leveraging the experience obtained from sol ving a related task (e.g., walking) to quickly learn a policy for each task (e.g., rumning). These situations also occurs in industry,
individual agent that is tailored to such as robots tasked with sorting items in fulfillment centers. A simple approach, like using PD controllers, would fail to adapt to the forces generated from picking up objects with different weight distributions, causing the arm to drop the objects. RL is able to mitigate this problem by learning a policy for each arm that is able make corrections quickly, which is tailored to the robot's dynam- ics. However, training new policy for each agent would be far too costly to be a practical solution. In these scenarios, it possible to use small number of policies learned subset of the agents, and then leverage the experience obtained from learning those policies to allow the remaining agents to quickly learn their corresponding policies. This approach can turn problems that are prohibitively expensive to solve into relatively simple problems.
To make use of prior experience and improve learning on new related problems in RL., several lines of work, which are complementary to each other, have been proposed and are actively being studied. Transfer learning (Taylor & Stone, 2009) refers to the problem of adapting information acquired while solving one task to another. One might consider learning mapping function that allows for
a policy leamned in one task to be used in a different task (Ammar et al. 2015) or simply learn a
mapping of the value function learned in one task to another (Taylor et al. 2007 These techniques
can be quite effective, but are also limited in that they consider mapping information from one source task to another target task. Another approach to reusing prior knowledge is through meta learning

---

Under review as a conference paper at ICLR 2020
or learning to learn (Schmidhuber 1995: Schmidhuber al. 1998). In the context of RL, the goal under this framework for an agent to be exposed to a number of tasks where it can learn some general behavior that generalizes to new tasks (Finn et al. 2017).
One last technique to leverage prior experience, and the one this paper focuses on, is through tem- porally extended actions or temporal abstractions (McGovern & Sutton 1998 Sutton et al.) 1999). While in the standard RL framework the agent has access to a set of primitive actions (i.e., actions that last for one time step), temporally extended actions allow an agent to execute actions that last for several time-steps. They introduce a bias in the behavior of the agent which, if appropriate for the problem at hand, results in dramatic improvements in how quickly the agent learns to solve new task. A popular representation for temporally extended actions is the options framework (Sutton & Precup 1998 Sutton et al. 1999) (formally introduced in the next section), which is the focus of this work. It has been shown that options learned in a specific task or set of tasks, can be reused to improve learning on new tasks (Machado et al., 2017 Bacon et al.] 2017]; however, this often requires knowledge from the user about which options or how many options are appropriate for the type of problems the agent will face.
In this paper, we propose learning reusable options for a set of related tasks with minimal infor- mation provided by the user. Throughout this paper, we refer as (near)-optimal policies to those policies that were learned to solve particular task, but not strictly speaking optimal. We con- sider the scenario where the agent must solve a large numbers of tasks and show that after learning (near)-optimal policy for small number of problems, options that facilitates learning in a remaining set of tasks. To do so, we propose learning set of options that minimize the expected number of decisions needed to represent trajectories generated from the (near)-optimal policies learned by the agent, while also maximizing the probability of gen- erating those trajectories. Unlike techniques that learn options to rach bottleneck states (McGovern & Barto] 2001) or states deemed of high value (Machado et al.) (017)) our method seeks to learn options that are able to generate trajectories known to perform well. This does not necessarily lead to learn options that reach states one might consider "interesting".
can learn an appropriate number of
2 BACKGROUND AND NOTATION
A Markov decision process (MDP) a tuple, - (,, 7, states of the environment, A is the set of possible actions that the agent can take, P(s, a, s') is the probability that the environment will transition state s' S if the agent executes action E A in state 8 E S, R(s, s') is the expected reward received after taking action in state and transitioning to state s', do is the initial state distribution, and 7 E [0. 1] is a discount factor for rewards received in the future. We use to index the time-step and write S,, A, and R2 to denote the state, action, and reward at time t. A policy, π : x A + [0, 1]. provides a conditional distribution over actions given each possible state: T(S, a) Pr(A, a[S, = s). We denote trajectory of length t as h, = (Su. ao To..... $1-1, 7t-1, St). that is, he is defined as a sequence of states, actions and rewards observed after following some policy for t time-steps. This work focuses on learning options that can be used for a set of related tasks. We consider the setting where an agent must solve set of related tasks, where each task is MDP, M = A, PM. RM, 7; ded) ); that is, each task is an MDP with its own transition function, reward function and initial state distribution, with shared state and action sets.
where S is the set of possible
An option, o = (Io, Mos 9o), is tuple in which I, C S is the set of states in which option can be executed (the initiation set), Po a policy that governs the behavior of the agent while executing o, and B. : S + [0, 1] is a termination function that determines the probability that terminates in a given state. We assume that Io- The options framework does not dictate how an agent should choose between available options or how options should be discovered. A common approach to selecting between options is to a learn a policy over options, which is defined by the probability of choosing an option in particular state. Two recent popular approaches to option discovery are cigenoptions (Machado et al. (017) and the option-critic architecture (Bacon al. 2017).
for all options o; that is, the options are available at every state.
The eigenoptions (aahado et al. 2017) of an MDP are the optimal policies for a set of implicitly defined reward functions called eigenpurposes. Eigenpurposes are defined in terms of proto-value functions (Mahadevan) 2005). which are in turn derived from the eigenvectors of a modified adja- cency matrix over states for the MDP. The intuition is that mateer the true reward function, the
2

---

Under review as a conference paper at ICLR 2020
eigenoptions allow agent to quickly traverse the transition graph, resulting in better exploration of the state space and faster learning. However, there are two major downsides: 1) the adjacency matrix is often not known a priori, and may be difficult to construct for large MDPs, and 2) for each eigenpurpose, constructing the corresponding eigenoption requires solving a new MDP. The option-critic architecture (Bacon et al.] 2017) is more direct approach learn options and a policy over options simultaneously using policy gradient methods. One issue that often arises within this framework is that the termination functions of the learned options tend to collapse "always termi- nate". In a later publication, the authors built on this work to consider the case where there is a cost associated with switching options (Harb et al. 2018). This method resulted in the agent learning to use a single option while it was appropriate and terminate when an option switch was needed, allow- ing it to discover improved policies for a particular task. The authors argue that minimizing the use of the policy over options may be desirable, as the cost of choosing an option may be greater than the cost of choosing a primitive action when using an option. Recent work by Harutyunyan et al. 2019) approaches the aforementioned termination problem by explicitly optimizing the termination function of options to focus on small regions of the state space. However, in contrast to the work presented in these paper, these methods do not explicitly take into consideration that the agent might face many related tasks in the future.
We build the idea that minimizing the number of decisions made by agent leads to the discov- ery of general reusable options, and propose an offline method where they are learned by solving a small number of tasks. The options are then leveraged to quickly solve new problems the agent will face in the future. We use the trajectories generated while learning (near)-optima policies, and learn an appropriate set of options by directly minimizing the expected number of decisions the agent makes while simultaneously maximizing the probability of generating the observed trajectories.
LEARNING REUSABLE OPTIONS FROM EXPERIENCE
In this section, introduce the objective for learning set of reusable options for set of related tasks. Our algorithm introduces one option at a time until introducing a new option does not improve the objective further. This procedure results in natural way of learning options without having to predefine it; a new option is included if it is able to improve the probability of generating optimal behavior while minimizing the number of decisions made by the agent. Our method assumes that the agent has learn a policy for a small number of tasks, and sample trajectories are obtained from these (near)-optime) policies. Notice that the propose algorithm is only concerned with being able to recreate the demonstrated trajectories, so if these were sample from a poorly performing policy the options learned are unlikely to provide any benefits.
adequate number of
3.1 PROBLEM FORMULATION
In the options framework, at each time-step, I, the agent chooses an action, Ar, based on the current option, Oz. Let Tj be Bernoulli random variable, where T₁ 1 if the previous option, O₃-1, terminated at time t. and Te = 0 otherwise. If, = 1, O2 is chosen using the policy over options, n. If T, 0, then the previous option continues, that is, O₂ O₂-1. To ensure we can represent any trajectory, we consider primitive actions to be options which always select one specific action and then terminate; that is, for an option, o, corresponding to a primitive, a, for all E S, the termination function would be given by (s)) = 1, and the policy by ulss a') 1 if = and 0 otherwise.
of options, {oi, J. where O refers to the of options corresponding to primitive actions and Oo to the set corresponding to temporal abstractions. Fur- thermore, let H be random variable denoting a trajectory of length |H] generated by a near-optimal policy, and let H, be random variable denoting the sub-trajectory of H up to the state encountered at time-step t. We seek to find a set, O (oi,..., ),). that maximizes the following objective:
Let O O Oo denote
H
J(T, O) - E Σ
Pr(T: = 0, H 0) + (19(H,00))
(I)
t=1
where 9h, Oo) is regularizer that encourages a diverse set of options, and X is scalar hyper- parameter. If we are also free to leam the parameters of म, then O' E arg max max J(z, O).
One choice for g is the average KI. divergence on given trajectory over the set of options
3

---

Under review as a conference paper at ICLR 2020
being learned: g(h, Oo) m(m-1) En.'Eoo Ei DKL ( (Po(s)) (or(s))) Intuitively, seek find options that are capable of generating near-optimal trajectories with small number of terminations. Notice that minimizing the number of terminations is the same minimizing the number of decisions made by the policy over options, each termination requires the policy to choose new option. Given set of options, policy over options, and near-optimal sample trajectory, we can calculate the joint probability for trajectory exactly, and estimate equation by averaging over of near-optimal trajectories.
3.2 OPTIMIZATION OBJECTIVE FOR LEARNING OPTIONS
Given that the agent must solve set of tasks, we can use the experienced gathered on a subset of tasks to obtain trajectories demonstrating optimal behavior. Given set, H. of trajectories generated from an initial subset of tasks, we can now estimate the expectation in equation to learn options that can be leveraged in the remaining problems. Because the probability of generating any trajectory approaches 0 as the length of the trajectory increases, we make modify the original objective for better numerical stability, and arrive to the objective J that we optimize in practice.
T. 1 H. = hs, n. O
[(T,, O,
Σ NE
PTe h 0)
X1g
h
probability geserating &.
cocourage diverse options
expected number of tecminatiins
A more detailed discussion on how we arrived to this objective is provided in Appendix A. We can express equation 2 entirely in terms of the policy over options т,, options the transition function, P (which we estimate from samples). The following theorems show how to calculate the first two terms in equation Z allowing us to maximize the proposed objective.
{oi..... . and
Theorem 1. Given a set of options, O, and policy, over options, the expected number of terminations for a trajectory h given by:
T H. he, n. o =
[6]
Σ DEO
EdeO Mo (o(St-1, Pr(O.-1 d' H1 1 n, O) Bo(st)
1r0 - = O)
Pr(O: HH = h1, 0) = ((s: T(2: () +
- (o(s
and Pr(On HHH ho, π. 0) T(s.
Proof. See Appendix B.
Theorem 2. Given set of options O and policy π over options, the probability of generating trajectory h of length [h] is given by:
|M
Pr(HI: = hyp/= 0) =d
Σ
( ()u ( 1)]
II
P(sks dks 8841)s
oEO
k=0
where f is recursive function defined as:
fii=t
( (o(s.) T
oh), =
otherwise
Proof. See Appendix
TThis term only defined when we consider more than option. Otherwise, this term
4

---

Under review as a conference paper at ICLR 2020
Given parametric representation of the op- tion policies and termination functions for each E O and for the policy over options, use Theorems 1 and 2 to differentiate the objective in equation 2 with respect their parameters and optimize with any numerical optimization technique.
Algorithm 1 Option Learning Framework Pseu- docode
1: Collect set of trajectories H.
2: Initialize option set O with primitive options
3: done = false
4: Sprev = -0
5: while done == false do
3.3 LEARNING OPTIONS INCREMENTALLY
6: Initialize new option o' = (1622 ())) ini-
One common issue in option discovery is iden- tifying how many options are needed for given problem. Oftentimes this number is pre- defined by the user based on intuition. In such a scenario, one could learn options by simply randomly initializing the parameters of num- ber of options and optimizing the proposed ob- jective in equation 2 Instead, we propose not only learning options, but also the number of options needed, by the procedure shown in Al- gorithm T This algorithm introduces one op-
tializing parameters for ø and tb.
7:
O' OUO
8:
Initialize parameters 0 of policy m for k= k= .NN do
9:
10:
Jk = J (To, O¹, H.)
8 + aJv
ae
ø + 0.k
B
13:
v = + + a dv
14:
if JN Jpree < A then
tion at time and optimizes the objective J with
done true
respect to the policy over options ny. with pa- rameters 0, and the newly introduced option, 17:
16:
else
O O'
= (16, )r), with parameters e and v. for N epochs. Optimizing both of and Ra allows us to estimate how much we can improve j given that we keep any previously introduced option fixed. After the new option is trained, we measure how much J has improved; if it fails to improve above some threshold, A, the procedure terminates. This results in natural way of obtaining an appropriate number of options, as options stop being added once a new option no longer improves the ability to represent the demonstrated behavior.
18:
Jpres JN
19:
Return new option set
4 EXPERIMENTAL RESULTS
This section describes experiments used to evaluate the proposed technique approach. We show results in the "four rooms" domain to allow us to visualize and understand the options produced by our method, and show empirically that these options produce clear improvement in learning. We use this domain to show that options generated by our method are able to generalize to tasks where the option-crite critic architecture (Bacon et al. 2017) and eigenoptions (Machado al.) 2017) would fail to do so. We then extend our experiments to evaluate our technique in few selected problems from the Atari 2600 emulator provided by OpenAl Gym (Brockman et al. 2016). These experiments demonstrate that by using the trajectories obtained from solving a small subset of tasks, our approach able to discover options that significantly improve the learning ability of the agent in the tasks it has yet to solve. For the four room experiment, assume the transition function was known in advance. In all ATARI experiments, we estimated the transition functions by fitting the parameters of linear Gaussian model to all the transitions experienced during training.
4.1 EXPERIMENTS ON FOUR ROOMS ENVIRONMENT
gridworld of size 40 40, in which the agent is placed in start state and needs to reach goal state. At each time-step. the agent executes one of four available actions: moving left, right, up or down, and receives reward of 1. Upon reaching the goal state, the agent receives reward of +10. We generated 30 different task variations by changing the goal and start locations, and collected six sample trajectories from optimal policies leamed in six tasks. We evaluated our method on the remaining 24 tasks.
We tested our approach in the four rooms domain:
Figure Ia shows the change in the average expected number of terminations and average probability of generating the observed trajectories while learning options, adapted to the sampled trajectories. Options were learned over the six sampled optimal trajectories and every 50 epochs new option was introduced. For every option, the change in probability of
options are introduced and
5

---

Under review as a conference paper at ICLR 2020
Traising mrograsionn Four Enviarment
Performanz Four Room Teeling
Opde lilt
1.20
2000
0.09
1.3
-000
1.4
0.00
315
C.ttoonn
Options
CCplicaa
(a) Visualization of loss over 200 training epochs for the four rooms domain. The decrcasing aver- age number of decisions made by и shown blue and the increasing probability of generating the sampled trajectories is shown in red.
(b) Performance comparison four rooms domain. Six tasks were used for training and 24 different for testing. The plot shows the average return (and stan- dard error) on the y-axis as a function of the episode number on the test tasks.
Figure 1: Results on four-room domain. Six tasks were used for training and 24 for testing.
generating the observed trajectories as well as the change in expected number of decisions reaches plateau after 30 or 40 training epochs. When new option is introduced, there is large jump in the loss because a new policy, т, is initialized arbitrarily to account for the new option set being evaluated. However, after training the candidate option, the overall loss improves beyond what was possible before introducing the new option.
In Figure 1b
compare the performance of Q-leariig 24 novel test tasks using options dis- covered by our method (with and without regularization using KL. divergence), eigenoptions, and option critic. We allowed each competing method to learn options from the same six training tasks and, to ensure a fair comparison, we used the original code provided by the authors. As baselines, we also compare against primitive actions and randomly initialized options. It might seem surpris- ing that both eigenoptions and the option-critic failed to reach an optimal policy when they were shown to work well in this type of problem; for that we offer the following explanation. Our imple- mentation of four rooms is defined in much larger state space than the ones where these methods were originally tested, making each individual room much larger. Since the options identified by these methods tend to lead the agent from room to room, it is possible that, once in the correct room, the agent executes an option leading to a different room before it had the opportunity to find the approach in the smaller version of the four room problem,
goal. When testing difference in performance of the competing methods. In this experiment, we set the threshold for introducing a new option to 10% of J at the previous iteration and the hyperparameter X2 = 100.0. 0.001.
found no clear
When adding KL regularization, we set
Figure 2 shows a visualization of the policy learned by the agent for specific task. The policy leads the agent navigate from specific location in the bottom-left room to location in the top-right room in a small "four-room" domain of size 10x x 15. The new task to solve is shown in the top-left figure, while the solution found is shown the top-right figure. The remaining rows of images depict the learned option policies, termination functions, and how they were used in the new task. The first row shows the learned option policies after training. the center row depict the termination functions and the bottom row shows a heat-map depicting where each option is likely to be called. The figure shows that while the options are defined over the entire state space, they are only useful in specific regions-thal is, they are specialized. These options, when used in combination in specific regions, allow the agent to learn how to solve new problems more efficiently.
4.2 EXPERIMENTS USING ATARI 2600 GAMES
We evaluated the quality of the options learned by our framework in two different Atari 2600 games: Breakout and Amidar. We trained the policy over options using A3C (Mnih et al.) 2016) with grayscale pixel input. Options were represented by a two layer convolutional neural network, and
We show smaller domain than used the experiments for ease of visualization

---

Under review as a conference paper at ICLR 2020
New Task
Solution Found
Option
Option 2
Option 3
Option 4
Frobabllity
Trobabiln
0.0
1.0
Frobabill Cosshiii
0.0
Figure 2: Visualization of our framework four rooms domain. A novel task is seen the top left, where the agent (red) has navigate goal (green). On the top right, we show the solution found by the agent. The three rows below show how the options were learned and exploited in the new task. The highlighted area the top row shows sample trajectory and the color corresponds to the probability that the option would take the demonstrated action. Notice that this trajectory was obtained on a previous tasks, so it does action that is most likely at each state. After training (first row) each option specializes in a specific skill navigation pattern). In this case, the demonstrated trajectory can generated by using option 3 and 2. The middle rows shows heat-map indicating where option is likely to terminate (close to walls), and the last row shows where each options is likely to used by the policy learned in the new task. The agent learns to use each option in specific situations; for example, option is likely to be called to make the agent move up, it is located in one of the bottom rooms.
correspond to the new task on top. The arrows show the
were given the previous two frames as input. In both experiments the task variations consisted in changing the number of frames skipped after taking an action (randomly selected between and 10), the reward function by scaling the reward with real number between 0.1 and 10.0, and initial state distribution by letting the agent execute between O and 20 actions before start it starts learning. The full implementation details for these experiments
given in Appendix E. Figures 3a and 33) show the performance the agent as a function of training time in Breakout and Amidar, respectively. The plots show that given good choices of hyperparameters, the leareed options led a clear improvement in performance during training. For both domains, we found that X 5, 000 led to reasonable trade-off between the first two term in J, and report results with three different regularization values: A 0.0., A 0.01 and X 0.1.
Note that our results do not necessarily show that the options result in a better final policy, but they improve exploration early in training and enable the agent to learn more effectively. Figure 4a depicts the behavior for one of the learned options on Breakout. The option efficiently catches the ball after it bounces off the left wall, and then terminates with high probability before the ball has to be caught again. Bear in mind that the option remains active for many time-steps, significantly reducing the number of decisions made by the policy over options. However, does not maintain control for so long that the agent is unable to respond to changing circumstances. Note that the option only useful in specific case; for example, it was not helpful in returning a ball bounced off the right wall. That is say, the option specialized in specific sub-task within the larger problem:

---

Under review as a conference paper at ICLR 2020
Loaning Poltrranc Broskoud
Learting Perkrnarion Arwdd
B.L.. 52.0
30
wo..A. A.---.0-00000
24
(a) Average returns novel tasks for Breakout.
(b) Average returns novel tasks for Amidar.
Comparison on Atari domains for primitives (blue), options before training (orange) and learned options for different values of λι and A2. Shaded regions indicate standard error.
Figure 3:
highly desirable property for generally useful options. Figure 4b shows the selection of two of the options learned for Amidar when starting new game. the beginning of the game, option is selected, which takes the agent to specific intersection before terminating. The agent then selects option 2, which chooses a direction at the intersection, follows the resulting path, and terminates at the next intersection. Note that the agent does not need to repeatedly select primitive actions in order to simply follow a previously chosen path. Having access to these types of options enables an agent easily replicate known good behaviors, allowing for faster and more meaningful exploration of the state space.
D0O
BOD
BBO 2
DOO
BOD
BOD
DB
Visualization of learned option executed until termination Breakout. The option learned to catch the ball bouncing off the left wall and terminates with high probability before the hall bounces wall again (ball size increased for visualization).
Option
Option
(b) Visualization two learned options on Amidar. agent is shown in yellow and enemies in pink. Option learned to move at the beginning of the game, and turn left until reaching intersection. Option 2 learned to tum in that intersection and move up until reaching the next
CONCLUSION AND FUTURE WORK
In this work we presented an optimization objective for learning options offline from demonstrations of near-optimal behavior on set of tasks. Optimizing the objective results in set of options that allows an agent to reproduce the behavior while minimizing the number of decisions made by the policy over options, which are able to improve the learning ability of the agent new tasks. We provided results showing how options adapt to the trajectories provided and showed, through several experiments, that the identified options are capable of significantly improving the learning ability of an agent. The resulting options encode meaningful abstractions that help the agent interact with and learn from its environment more efficiently.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E. Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAII15, pp. 2015. ISBN 0-262-51129-0. URL http://dlllennoooggaaaoiootnttnd 2886666
AAAI Press,
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. AAAI, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, 2016.
Alon Farchy, Samuel Barrett, Patrick MacAlpine, and Peter Stone. Humanoid robots learning to walk faster: From the real world simulation and back. In Proc. of 12th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS), May 2013.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1126-1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http://proceedingg.s http:///nnm....m..rrss////////mm wawr//0//nnnaaaaiiiiiin 7a.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting not an option: Learning options with deliberation cost. In AAAI, 2018.
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critic. In AISTAT, 2019.
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A Laplacian Framework for Option Discovery in Reinforcement Learning. CoRR, 2017.
Sridhar Mahadevan. Proto-value functions: Developmental reinforcement learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML-2005), pp. 553-560. ACM, 2005.
A. McGovern and R. Sutton. Macro actions in reinforcement learning: An empirical analysis. Technical report, University of Massachusetts. Amherst, Massachusetts, USA, 1998.
Amy McGovern and Andrew G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceedings the Eighteenth International Conference Machine Learning, ICML '01, pp. 361-368, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-778-1. URL
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-3 533, February 2015. ISSN 00280836. URL 12366
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1928-1937, 1937, New York, New York, USA. 20-22 Jun 2016. PMLR. URL http: Dpmneedding..m...res//////mmmmi ht.mll
Jürgen Schmidhuber, Jieyu Zhao, and Nicol N. Schraudolph. Learning to learn. chapter Reinforce- ment Learning with Self-modifying Policies, pp. 293-309, Kluwer Academic Publishers, Nor- well, MA, USA, 1998. ISBN 0-7923-8047-9, URI. http://ddlacc.oogggatttto....aaa 296658
Jürgen Schmidhuber. On learning how learn learning strategies. Technical report, 1995.

---

Under review as a conference paper at ICLR 2020
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner. Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):4844 489, 2016. ISSN 0028-0833 doi: 10. Sa//nnture16666
Richard S. Sutton and Doina Precup. Intra-option learning about temporally abstract actions. In In Proceedings of the 15th International Conference on Machine Learning (ICMI--1998), 1998.
Richard S. Sutton, Doina Precup, and Satinder P. Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. J. Mach. Learn. Res., 10: 1633-166,,, December 2009. ISSN 1532-4435, URL http://dl.ammmorr) citation. (tation.cmmm773157777777777
Matthew E. Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for tem-
poral difference learning. J. Mach. Learn. Res., 8:2125-22 2167, December 2007. ISSN 1532-4435.
URL htt.://www.rggrcaaiooonc.mdaiadd
1314566
Gerald Tesauro. Temporal difference learning and td- -d-gammon. Commun, ACM, ACM, 38())58--88 March 1995. ISSN 0001-0782. doi: 10.1 00.114/203330.203..3 203343
URL http://db.aamm.org0001114444///.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
The following list defines the notation used in all derivations:
1. Ay: random variable denoting action taken at step t.
2. Sy: random variable denoting state at step t.
3. H₁: random variable denoting history up step H, (So, Ao. S1, At)..
4. Ty: random variable denoting the event that the option used at step terminates state
S₁.
5. m: policy over options.
6. P: transition function. P(s, a, s') denotes the probability of transitioning to state s' by
taking action in state
7. O₁: random variable denoting the option selected for execution at state St.
8. o: option defined as o (14,,,0). where is the option policy for option and 3o is the termination function.
9. Assume primitives are options that perform only 1 action and last for time-step.
10. O: set of available options.
We can compute the probability of option terminating at state s, and generating trajectory he as:
Pr(T; 1, Hi, h, n. O) Pr(T; 1|H1, he 0) Pr(Hl, hel O)
(4)
To compute the proposed objective J we need to find an expression for Pr(T; 1| h4, O) and Pr h O) in terms of known quantities.
A.1 APPENDIX A DERIVATION OF j
Σー Pr(Ti 0, Hel O)]. ignoring the regularization term. Assuming
Recall J(T, O)H)
access to set H. of sample trajectories, we start by estimating J from sample averages and derive the objective J as follows:
0,)
Σ
Σ
Pr(T, 0, 0)
H
BER
IN
Σ
Σ
Pr(p, (1-Pr(TT
(1-
1| h,, 0)) Pr(H, h,|,, O)
H
bEH t=1
IN]
ΣΣ
1 E- he Ne, 0) Pr(H, - Pr(H., h,lm,
H
BEH
can easily be seen that to maximize the above expression E [TT = he, n. O should be mini- mized while Pr(H- h|/T, O) should be maximized. Given that for long trajectories the expected number of terminations increases while the probability of generating the trajectories goes to O, we normalize the number of terminations by the lenght of the trajectory, [h], and adjust a hyperpa- rameter, X2, to prevent term from dominating the other during optimization. Based on this observation we propose optimizing the following objective:
(,HH H
Σ Pr //7, hajhH
E
T.
h4, n, O
hh]
This objective allow to control trade-off, through Ag of how much care about the options reproducing the demonstrated trajectories vs. how much we want the agent to minimize the number of decisions.
11

---

Under review as a conference paper at ICLR 2020
A.2 APPENDIX B PROOF OF THEOREM
Theorem Given set of options O and policy a over options, the expected number of termina- tions for a trajectory h of length [h] is given by:
M
[A|
Polt 11 O)
E [he = 1|11, = he, n. 0] -
Σ
Σ 6EO
Bo(st)
Edee Her (8-11, Pr(O,-1 - -00(000---00-1111111 = 1e- 0)
where,
1000-1-|HHH111 = T,, { htt.b//www.......a.aaaaaaaaaaaaa
XPr(O1-2 h -
and Pr(Op- = o ho: 0) = π (s0.
Proof. Notice that E E E,, 1 h4 ग, 0] - Pr(T; 1|H, = he; T,, O) 1, if
find an expression for Pr(T; 1 h4, ग, O), we can calculate the expectation exactly. We define Pr(To 1 h1 n, 0) 1 for ease of derivation even though there is no option to
terminate at To-
,, Pr(T, = 1 = ,,, Σ = h, hn - = -h,, n, O)
nEO
Σ
Bo(s:) Pr(Or-1= o/H = h4 ग O)
0EO
Σ
(x,) Pr(O.-1 = = - -
OEO
Σ (xr) = 1
Pr(S, = 8/ H4 h
04 O)
oEO
x = = 14
= s, 1
Σ (o(ss)
Pr(S, = = 4,1,,,,00
AEO
x Pr(0,-11 = o = 4-1 -
Ba(s)) Pr(00-1==0))))) of - A, = 0 = O)
oEO
Pr(A = a:1 H 14- O, = of = 4 ,, O)
(st))
Pr(A,-1 0:- H111 11-1, O)
oEO
(ot))) (1 P 1,
Pr(A,11 0r- 1411 h, 0)
oE0
12

---

Under review as a conference paper at ICLR 2020
Ho 1 0)
(s)
Ed'Eo Pr(A,11 = a-11 0,-1 = o 1 T, O)
-11 PrO0 1
11 0)
OEO
Pr 10:4-110-1000-000 = - 1,,,,
d'EO
x - - ,,, - n, O)
(o(
Eo'E Ho (8--1, (1-1 = =d HH,1 = 14-1, O)
OEO
We are left with finding an expression in terms of known probabilities for Pr(O,-1 = o111111 1 O).
Pr(O,-1 o = h,,, 0) = - -..
+Pr(O -
[Pr(OO))
- -
x PP( h ,T, 0))
=
+ (P
= o|H,-
11-1 T 1-1 = T,
x (1 P = 1 ,,)))
= (T))--1,
= 1 1 n,, [,, 0))
+ +(Pr(Or-1 = - 0,
x ((1-Pr(T: = = -
x (Pr(O = o/ = 11,1, = 0, 0) - (6(
Given that by convention, Pr(To = 1|H4- ho n, 0)= 1.0, we now left with figuring out how to calculate Pr = H = -
Pr(O,-1 = of H, = 1 T = 0, = = 0
= 14-1, T. O)
= = O, - 0, H, = 1 n, O)
x PPr(Ot-2 = -
Pr(St-1 = 1 =
0: 0, 0, H h n. O)
x o
= 14-11, 0)
x = 1 0)
= =P(s 200---2000222
Pr(O, o H,-1 = h4-1, n, O) 21 Pr(0,)))pPP-2 = h4-2, π, O)
P( 01 21 1
o/H
where Pr(O, o Hn ho, ,, O) T 0)
13

---

Under review as a conference paper at ICLR 2020
Using the recursive function Pr(O,-1 Hi h4-1; O), the expected number of termina- tions for a given trajectory is given by:
M
No(st P - he- 1, O)
E T. 1 h4 n, 0
Σ
Σ
Bo(st)
EoEEo Hol Ho (6-1, Pr(0 (5-1)01111
H|H he-1, TT, 0)
A.3 APPENDIX C PROOF OF THEOREM
Theorem
Given set of options O and policy over options, the probability of generating trajectory h of length [h| is given by:
PM(HI hmllT, 0) do(so) Eoso (50, ao)
I
M-1
P(8k, ags 8k+1)
where f is recursive function defined as:
if
0( = Bo((s) Edeo (4
+(1 Bo(ss)))(((H)
otherwise
Proof. We define Нie be the history from time to time t, that is. Hi. (S,, Si+1, Arti..., St), where If = t, the history would contain a single state.
Pr(H, = hT/ 0) = Pr(S = ST 0) Pr(H1, = h1 Au = qnl Su O) -do(so) Pr(Ht1, (1,,, Au S. Sp, ,, 0)
=do(so)
Pr(H1, A1,, ao. O. S 80, n, 0)
nEO
(s)
Pr(Oo 0 so, 7) Pr h1 Ao = ag|So 50, Oo o, n, 0)
oEO
-do(s))
T(Su, 0) Pr(Ht1,1 = 11,4, A, ao 50, Ou = 0, 0)
=d((s))
Σ
(500 0) Pr(Ap a. SS so. Os o, O)
nEO
Pr Pr(H.
h1 S0 80, On o, Ao an, ,, O)
=do(0o
Σ
(S,, 11 ao) Pr(H1, 11 = 80, Og 0, Ag = Gn, O).
oEO
We now need to find expression to caleulate Pr(H1,1 h1 Su. Ou 0, Ag 4g. O): Consider the probability of seeing history hi given the previous state, s, the previous option, o, and the previous action,
14

---

Under review as a conference paper at ICLR 2020
= =
-Pr(S, - -0. -
Pr -
= h1t1, A; - 0:|S - 8, 0,-1 =0, A a. - 8:)
0; = A; a, a. 9:)
= =P(s, a, - =
=,(s, s1)
= A, = a, = 8:)
S = 8:)
x Pr(H = = A a, S = 8t Ti = 1)
+ +P((T: = 00:=0000--0000000000
S.S, = 8;)
x PPr(H++1,1 = - =a 0,
0, A, 0, = 8. T, 0)
= =(ss a, 8:) (Br(s))
x Pr(Hi+1,1
a 0, A a, S = 5; =
+ (1 - -B((s:)
x Pr(Hi+11 A = ai A a, S = 8; Ti 0)]
Even though the equation above might seem complicated, there are only two cases need to consider: either the current option terminates and new one must be selected (the first term), or the current option does not terminate (the second term). Let's consider each of them separately.
Case - option terminates: we terminate, we sum over new options:
=
T; 1)
-E -0 =;
dea
- 1 A -, =0)
Σ deo
a 04
= 0, = =
Σ T(,; o') Pr(Hi++, = h441, A a S s, ==)
d'
Σ d'EO
n T d') Pr(A: a,|S; 81 O, d') Pr( 1141,111 = 81, 0, d, A, a,)
Σ
T(,;, o )o (st, a.) Pr(H.+11 h1 8,, O, A =a).
deo
Note that the expanded probability has the same form Pr(Hli, h4, S1-1 s,0;11
o, A; a).
Case option does not terminate: This tells us that O, o, so we may drop the dependency on the i - 1 terms:
P A -.
= hi+1,1,
5,, T;
Pr([H = h,41,1, A, =. = 0,
=Pr(A, = S, = s;, O, 00 P h4+1, SS: = S,, - = 0, a.)
=Ho 0;) Pr(Hi++1,
SS 8;, O; A; = a.).
Plugging these cases back into earlier equation yields:
15

---

Under review as a conference paper at ICLR 2020
Pr(l = 14.1/S1-1 0,-1 =
= =,(s, 81) (3((s))
Σ
T(s,, (o) 01) Pr P 1411,1 = 81, O; = = =a;)
O'E0
+ (1-Bo(s:))po(si a4)
h1t1, O, =, a:)]
Note that each term contains an expression of the same form, Pr h ht.1/::11 8, 0,-1 A = a). We can therefore compute the probability recursively. Our recursion will terminate when we consider to be 1. Notice that for every recursive step, both inner terms will produce a P(s, 8;) term. Consider the result when we factor every recursive (ss, o, 8;) term to the front of the equation. We define the following recursive function:
f, as Ht, contains single state, and we adopt the convention of its probability
if
(o(ss) oh
0(1,,0,2)
+(1 -666))))66641...
otherwise
Notice that this is the recursive probability described above, but with the P(s, a, 8') terms factored out. We now that:
Pr(H,, =
P(sk, ab. Sk+1). 0,-1, = a0-1) f(he, 0, II
Plugging this all the back into our original equation for Pr(He = h2/T, O) gives us the desired result:
Pr(HIM) h/n (T O) = do(s0) Σ (so. (h)
II
P ags 81+1)-
A.4 APPENDIX D EMPIRICAL VALIDATION OF DERIVED EQUATIONS
To double check the derivation of the proposed objective and make sure the implementation was correct, we conducted simple empirical test to compared the calculated expected number of deci- sions in a trajectory and the probability of generating each trajectory for a set 10 trajectories on 10 MDPs. The MDPs simple chains of states with different transition functions. We randomly initialized four options and a policy over options, and estimated the probability of generating each trajectory and the expected number of terminations, for each sampled trajectory, by Montecarlo sampling 10, 000 trials. Table [] presents results for the 10 trajectories verifying empirically that correctly derived and implemented. The table compares the empirical and true probability of generating given trajectory, P-(H and Pr(Hl--.. respectively, and the empirical and true sum of expected number of decisions an agent has to make to generate those trajectories,
the equations
D E [T]- and D E (. respectively.
H
16

---

Under review as a conference paper at ICLR 2020
Pr(H O)
Pr(H O)
E [7, ,,, O
Σ E
[T
e m,
h1
0.0932
0.0957
3.060
3.178
1/2
0.0158
0.0173
4.139
4.154
h3
0.2149
0.2122
1.965
2.178
h4
0.0995
0.0957
2.979
3.178
hs
0.0962
0.0957
3.024
3.178
h6
0.1354
0.1384
2.9579
3.1596
h7
0.00040
0.00038
9.750
8.794
hs
0.1854
0.1881
2.820
3.072
lug
0.0379
0.0368
4.2612
4.4790
/110
0.1864
0.1881
2.8404
3.0723
Table
Validation of equations and implementation.
Note that the cases with largest discrepancy between the estimated and calculated number of termi- nations occur when the probability of generating a trajectory is low. This happens because, since the trajectory is unlikely to be generated, the Monte Carlo sampling is not able to produce enough samples of the trajectory.
A.5 APPENDIX E IMPLEMENTATION DETAILS FOR ATARI EXPERIMENTS
For these experiments first learned good performing policy with A3C for each game and sam- pled 12 trajectories for training. Each trajectory lasted until life was lost, not for the entire duration of the episode. Each option was represented layer, and two output layers: a softmax output layer over the four possible actions representing μ. and separate sigmoid layer representing $. We implemented our objective using PyTorch which simplifies gradient calculations. The options were represented by two-layer neural network, where the input was represented by gray scale images the last two frames. We ran training agents in parallel CPUs, the learning rate was set to 0.0001 and the discount factor y was set to 0.99.
two-layer neural network, with 32 neurons each
Because the options can only learn from the states observed in the trajectories, is possible that when using them, they will be executed in previously unseen states. When this happens, the ter- mination function may decide to never terminate, as it has not seen that region of the state space before. To address this issue, add value of 0.05 the predicted probability of termination per time-step that the option has been rumning since executed. Therefore, in our experiments an option cannot run for more than 20 time-steps in total.
17


Reviews:
[{"review_id": "SJg49tWqtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for learning options that transfer across multiple learning tasks. The method takes a number of demonstration trajectories as input and attempts to create a set of options that can recreate the trajectories with minimal terminations.\n\nI currently recommend rejection. The evaluation of the proposed method is rather weak and does not clearly demonstrate that the author’s goals have been achieved. The paper could also do a better job of situating the approach with regard to existing option learning approaches.\n\nDetailed comments:\n\n- The paper strongly emphasizes reusability of learnt options over multiple tasks as a key goal. This aspect is largely absent from the practical part of the paper, however. The proposed algorithm largely ignores the multi-task aspect beyond requiring demonstrations from different tasks - also see the next remark. In the experiments, the multi-task transfer is not emphasized. In the 4rooms domain options are learnt on training tasks and evaluated on test tasks, but the effect of task distribution or task diversity on generalisation is not investigated. Moreover, the larger scale ATARI experiments do not seem to include any multi-task aspects at all, with options immediately being learnt on the target task.\n\n- The objective in (1) omits any explicit mention of different tasks. It would be good to indicate explicitly how it depends on the distribution of tasks and what the expectation is taken over.\n\n- The authors indicate that their learning objective needs the transition function P for the MDP. This is never further discussed. Do the experiments assume known transition functions? If not, how are these functions estimated? If a model is known, does it still make sense to learn option policies from samples or would it be better to use  planning based options (see e.g.[1])?\n\n- While the paper cites a number of option learning approaches, it could do a better job of situating the research within the literature. There are a number of option induction approaches that explicitly focus on reusability of options - see e.g. [1], [4]. There have also been a large number of approaches that focus on hierarchical learning to represent a distribution over demonstration trajectories: see eg. [3],[5], [6], [7]. Some of these approaches might also be better baselines than OptionCritic which doesn’t explicitly take into account learning from demonstrations or multi-task transfer. \n\n[1] Mann, T. A., Mannor, S., & Precup, D. (2015). Approximate value iteration with temporally extended actions. JAIR, 53, 375-438.\n[2] Konidaris, G., & Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI,\n[3] Konidaris, G., Kuindersma, S., Grupen, R., & Barto, A. (2012). Robot learning from demonstration by constructing skill trees. IJRR, 31(3), 360-375.\n[4] Andreas, J., Klein, D., & Levine, S. (2017). Modular multitask reinforcement learning with policy sketches. ICML\n[5] Henderson, P., Chang, W. D., Bacon, P. L., Meger, D., Pineau, J., & Precup, D. (2018). Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. AAAI.\n[6] Co-Reyes, J. D., Liu, Y., Gupta, A., Eysenbach, B., Abbeel, P., & Levine, S. (2018). Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings.ICML\n[7] Daniel, C., Van Hoof, H., Peters, J., & Neumann, G. (2016). Probabilistic inference for determining options in reinforcement learning. Machine Learning, 104(2-3), 337-357.\n\n\nMinor comments:\n- The results on ATARI seem to have been ended before reaching final learning performance\n\n- I couldn’t find details for how the transition function in 4-rooms is changed\n\n- Does the optionCritic comparison include the deliberation cost? Since this paper aims to minimise option terminations that seems to be the most logical comparison.\n\n- Why don’t the ATARI results compare against other approaches?\n\n- The influence of the KL penalty isn’t really examined in results beyond looking at performance. How does it influence the trade-off between representing trajectories and diversity?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyetQDATYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to learn reusable options to make use of prior information and claim to do so with minimal information from the user (such as # of options needed to solve the task, which options etc). The claim is that the agent is first able to learn a near-optimal policy for a small # of problems and then is able to solve a large # of tasks by such a learned policy.  The authors build on the idea that minimizing the number of decisions made by the agent results in discovering reusable options. The options are learned offline by learning to solve a small number of tasks. Their algorithm introduces one option at a time until introducing a new option doesn’t improve the objective further. The ideas are interesting, However, the paper as it stands is lacking in thorough evaluation.\n\nDetailed comments:\nThe proposed approach offers two key contributions:\n-an objective function to minimize the decision states\n-incrementally constructing an option set that can be reused later, without the a priori specification of the # of options needed. \n\nThe introduction is well written, however, given the intuitions behind the objective function; in some sense, the idea here is to minimize the decisions or terminations intuitively relates to terminating only at critical or bottleneck states. It would be useful to provide such motivation in the introduction. \n\nIntuitively the objective criterion is interesting. With a cursory look at the proofs, they seem fine, although I have to admit I have not looked in detail into the proofs. \n\nPaper writing could be significantly improved. Several points are not clear and need further clarification:\n-The term near-optimal is mentioned several times, but it is not clear the policies are near-optimal with respect to what? The task or a set of tasks? \n-How does the proposed approach ensures that they are near-optimal? Please clarify.\n-“We can obtain the estimate for equation 1 by averaging over a set of near-optimal trajectories” The aim as states is to learn options that are capable of generating near-optimal trajectories (by using a small # of terminations). The authors then say that “given a set of options, a policy over options, a near-optimal sample trajectory, we can calculate..” Where does the near-optimal sample trajectory come from? Please provide clarifications.\n\nIn experiments: FR rooms experiments are interesting, in the visualization of the option policies, do the figures here show the flattened policy of the options? What do the termination functions look like? \n\nAtari experiments are limited in nature in that they show only two games. Moreover, It is a bit confusing as to what is multi-task in the ATARI experiments. The authors mention the training of options and then talk about the results in the plots (4) show the training curves. However, they do not mention what are “novel tasks for Breakout/Amidar” in this context. \n\nConsidering the proposed approach is closely related to the idea of selective terminations of options, it is natural to expect a comparison with Harb, 2018 and Hartyuanm 2019. The work could benefit by comparing with the aforementioned baselines. In particular, the visualization in 4b showing options learned in Amidar does not show much improvement from what was observed before in Harb, 2018.  \n\nWith the motivation of this paper, I am unable to convince myself about options being “reusable” for multi-task here. It would be very useful for the reader to clarify what “novel tasks” are here to appreciate what is learned. Looking deeper into the appendix, I understand that the authors “first learned a good performing policy with A3C for each game and sample 12 trajectories for training.” This is not at all clear in the main paper. Besides, what does it mean by a \"good\" policy? If we already have that, it is unclear what gains do we get from the proposed method.  \n\nOne obvious limitation here is that they also have a hard imposed constraint here is that the options cannot run for more than 20 time-steps in total, to make the objective function a suitable choice. \n\nOverall:\t\t\nAn interesting objective function, Learn not only option set but also the number of options needed and incrementally learn new options. \n\nPaper writing does not convey clearly what are novel tasks and could be significantly improved.\n\nSince the paper claims multi-task and mentions several lifelong learning works like [1], I was expecting rigorous baselines showing performance over multiple tasks. The experiments are lacking in that evidence except for four rooms domain, which is much simpler a domain. \n\nNear-optimal property is very much lacking the clarity to the best of my knowledge.\n\n[1]Ammar, Haitham Bou, et al. \"Online multi-task learning for policy gradient methods.\" International Conference on Machine Learning. 2014.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeGkWrhFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new option discovery method for multi-task RL to reuse the option learned in previous tasks for better generalization. The authors utilize demonstrations collected beforehand and train an option learning framework offline by minimizing the expected number of terminations while encouraging diverse options by adding a regularization term. During the offline training, they add one option at a time and move onto the next option when the current loss fails to improve over the previous loss, which enables automatically learning the number of options without manually specifying it. Experiments are conducted on the four rooms environment and Atari 2600 games and demonstrate that the proposed method leads to faster learning on new tasks.\n\nOverall, this paper gives a novel option learning framework that results in some improvement in multi-task learning. While the paper is technically sound and somewhat supported by experimental evidence, the experiments are limited to low-dimensional state space and discrete action space. I do wonder if the method can scale to high-dimensional space with continuous control.\n\nMoreover, the framework requires optimal policies to generate trajectories for offline option learning, which seems to add more supervision signals than prior work such as option-critic. I wonder how the method would perform under sub-optimal demonstrations or even random trajectories generated by some RL policy.\n\nFinally, I wonder how this method can be compared to skill embedding learning methods such as [1], which have been shown to be able to compactly represent skills in a latent space and reuse those skills in high-dimensional robotic manipulation tasks.\n\n[1] Hausman, Karol, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. \"Learning an embedding space for transferable robot skills.\" (2018).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel option discovery mechanism through incrementally learning reusasble options from a small number of policies that are usable across multiple tasks.

The primary concern with this paper was with a number of issues around the experiments. Specifically, the reviewers took issue with the definition of novel tasks in the Atari context. A more robust discussion and analysis around what tasks are considered novel would be useful. Comparisons to other option discovery papers on the Atari domains is also required.

Additionally, one reviewer had concerns on the hard limit of option execution length which remain unresolved following the discussion.

While this is really promising work, it is not ready to be accepted at this stage.
Decision: Reject