Learning Reusable Options for Multi-Task Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) techniques have experienced much of their success in simulated environments , such as video games ( Mnih et al. , 2015 ) or board games ( Silver et al. , 2016 ; Tesauro , 1995 ) . One of the main reasons why RL has worked so well in these applications is that we are able simulate millions of interactions with the environment in a relatively short period of time . In many real world applications , however , where the agent interacts with the physical world , it might not be easy to generate such a large number of interactions . The time and cost associated with training such systems could render RL an unfeasible approach for training in large scale . As a concrete example , consider training a large number of humanoid robots ( agents ) to move quickly , as in the Robocup competition ( Farchy et al. , 2013 ) . Although the agents have similar dynamics , subtle variations mean that a single policy shared across all agents would not be an effective solution . Furthermore , learning a policy from scratch for each agent is too data-inefficient to be practical . As shown by Farchy et al . ( 2013 ) , this type of problem can be addressed by leveraging the experience obtained from solving a related task ( e.g. , walking ) to quickly learn a policy for each individual agent that is tailored to a new task ( e.g. , running ) . These situations also occurs in industry , such as robots tasked with sorting items in fulfillment centers . A simple approach , like using PD controllers , would fail to adapt to the forces generated from picking up objects with different weight distributions , causing the arm to drop the objects . RL is able to mitigate this problem by learning a policy for each arm that is able to make corrections quickly , which is tailored to the robot ’ s dynamics . However , training a new policy for each agent would be far too costly to be a practical solution . In these scenarios , it is possible to use a small number of policies learned a subset of the agents , and then leverage the experience obtained from learning those policies to allow the remaining agents to quickly learn their corresponding policies . This approach can turn problems that are prohibitively expensive to solve into relatively simple problems . To make use of prior experience and improve learning on new related problems in RL , several lines of work , which are complementary to each other , have been proposed and are actively being studied . Transfer learning ( Taylor & Stone , 2009 ) refers to the problem of adapting information acquired while solving one task to another . One might consider learning a mapping function that allows for a policy learned in one task to be used in a different task ( Ammar et al. , 2015 ) or simply learn a mapping of the value function learned in one task to another ( Taylor et al. , 2007 ) . These techniques can be quite effective , but are also limited in that they consider mapping information from one source task to another target task . Another approach to reusing prior knowledge is through meta learning or learning to learn ( Schmidhuber , 1995 ; Schmidhuber et al. , 1998 ) . In the context of RL , the goal under this framework for an agent to be exposed to a number of tasks where it can learn some general behavior that generalizes to new tasks ( Finn et al. , 2017 ) . One last technique to leverage prior experience , and the one this paper focuses on , is through temporally extended actions or temporal abstractions ( McGovern & Sutton , 1998 ; Sutton et al. , 1999 ) . While in the standard RL framework the agent has access to a set of primitive actions ( i.e. , actions that last for one time step ) , temporally extended actions allow an agent to execute actions that last for several time-steps . They introduce a bias in the behavior of the agent which , if appropriate for the problem at hand , results in dramatic improvements in how quickly the agent learns to solve a new task . A popular representation for temporally extended actions is the options framework ( Sutton & Precup , 1998 ; Sutton et al. , 1999 ) ( formally introduced in the next section ) , which is the focus of this work . It has been shown that options learned in a specific task or set of tasks , can be reused to improve learning on new tasks ( Machado et al. , 2017 ; Bacon et al. , 2017 ) ; however , this often requires knowledge from the user about which options or how many options are appropriate for the type of problems the agent will face . In this paper , we propose learning reusable options for a set of related tasks with minimal information provided by the user . Throughout this paper , we refer as ( near ) -optimal policies to those policies that were learned to solve a particular task , but are not strictly speaking optimal . We consider the scenario where the agent must solve a large numbers of tasks and show that after learning a ( near ) -optimal policy for a small number of problems , we can learn an appropriate number of options that facilitates learning in a remaining set of tasks . To do so , we propose learning a set of options that minimize the expected number of decisions needed to represent trajectories generated from the ( near ) -optimal policies learned by the agent , while also maximizing the probability of generating those trajectories . Unlike techniques that learn options to rach bottleneck states ( McGovern & Barto , 2001 ) or states deemed of high value ( Machado et al. , 2017 ) , our method seeks to learn options that are able to generate trajectories known to perform well . This does not necessarily lead to learn options that reach states one might consider “ interesting ” . 2 BACKGROUND AND NOTATION . A Markov decision process ( MDP ) is a tuple , M = ( S , A , P , R , γ , d0 ) , where S is the set of possible states of the environment , A is the set of possible actions that the agent can take , P ( s , a , s′ ) is the probability that the environment will transition to state s′ ∈ S if the agent executes action a ∈ A in state s ∈ S , R ( s , a , s′ ) is the expected reward received after taking action a in state s and transitioning to state s′ , d0 is the initial state distribution , and γ ∈ [ 0 , 1 ] is a discount factor for rewards received in the future . We use t to index the time-step and write St , At , andRt to denote the state , action , and reward at time t. A policy , π : S × A → [ 0 , 1 ] , provides a conditional distribution over actions given each possible state : π ( s , a ) = Pr ( At = a|St = s ) . We denote a trajectory of length t as ht = ( s0 , a0 , r0 , . . . , st−1 , at−1 , rt−1 , st ) , that is , ht is defined as a sequence of states , actions and rewards observed after following some policy for t time-steps . This work focuses on learning options that can be used for a set of related tasks . We consider the setting where an agent must solve a set of related tasks , where each task is an MDP , M = ( S , A , PM , RM , γ , dM0 ) ; that is , each task is an MDP with its own transition function , reward function and initial state distribution , with shared state and action sets . An option , o = ( Io , µo , βo ) , is a tuple in which Io ⊆ S is the set of states in which option o can be executed ( the initiation set ) , µo is a policy that governs the behavior of the agent while executing o , and βo : S → [ 0 , 1 ] is a termination function that determines the probability that o terminates in a given state . We assume that Io = S for all options o ; that is , the options are available at every state . The options framework does not dictate how an agent should choose between available options or how options should be discovered . A common approach to selecting between options is to a learn a policy over options , which is defined by the probability of choosing an option in a particular state . Two recent popular approaches to option discovery are eigenoptions ( Machado et al. , 2017 ) and the option-critic architecture ( Bacon et al. , 2017 ) . The eigenoptions ( Machado et al. , 2017 ) of an MDP are the optimal policies for a set of implicitly defined reward functions called eigenpurposes . Eigenpurposes are defined in terms of proto-value functions ( Mahadevan , 2005 ) , which are in turn derived from the eigenvectors of a modified adjacency matrix over states for the MDP . The intuition is that no matter the true reward function , the eigenoptions allow an agent to quickly traverse the transition graph , resulting in better exploration of the state space and faster learning . However , there are two major downsides : 1 ) the adjacency matrix is often not known a priori , and may be difficult to construct for large MDPs , and 2 ) for each eigenpurpose , constructing the corresponding eigenoption requires solving a new MDP . The option-critic architecture ( Bacon et al. , 2017 ) is a more direct approach to learn options and a policy over options simultaneously using policy gradient methods . One issue that often arises within this framework is that the termination functions of the learned options tend to collapse to “ always terminate ” . In a later publication , the authors built on this work to consider the case where there is a cost associated with switching options ( Harb et al. , 2018 ) . This method resulted in the agent learning to use a single option while it was appropriate and terminate when an option switch was needed , allowing it to discover improved policies for a particular task . The authors argue that minimizing the use of the policy over options may be desirable , as the cost of choosing an option may be greater than the cost of choosing a primitive action when using an option . Recent work by Harutyunyan et al . ( 2019 ) approaches the aforementioned termination problem by explicitly optimizing the termination function of options to focus on small regions of the state space . However , in contrast to the work presented in these paper , these methods do not explicitly take into consideration that the agent might face many related tasks in the future . We build on the idea that minimizing the number of decisions made by an agent leads to the discovery of general reusable options , and propose an offline method where they are learned by solving a small number of tasks . The options are then leveraged to quickly solve new problems the agent will face in the future . We use the trajectories generated while learning ( near ) -optimal policies , and learn an appropriate set of options by directly minimizing the expected number of decisions the agent makes while simultaneously maximizing the probability of generating the observed trajectories .

Summary: This paper proposes a new option discovery method for multi-task RL to reuse the option learned in previous tasks for better generalization. The authors utilize demonstrations collected beforehand and train an option learning framework offline by minimizing the expected number of terminations while encouraging diverse options by adding a regularization term. During the offline training, they add one option at a time and move onto the next option when the current loss fails to improve over the previous loss, which enables automatically learning the number of options without manually specifying it. Experiments are conducted on the four rooms environment and Atari 2600 games and demonstrate that the proposed method leads to faster learning on new tasks.