Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies

1 INTRODUCTION . Recently , reinforcement learning ( RL ) systems have achieved super-human performance on many complex tasks ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Van Seijen et al. , 2017 ) . However , these works mostly have been focused on a single known task where the agent can be trained for a long time ( e.g. , Silver et al . ( 2016 ) ) . We argue that agent should be able to solve multiple tasks with varying sources of reward . Recent work in multi-task RL has attempted to address this ; however , they focused on the setting where the structure of task are explicitly described with natural language instructions ( Oh et al. , 2017 ; Andreas et al. , 2017 ; Yu et al. , 2017 ; Chaplot et al. , 2018 ) , programs ( Denil et al. , 2017 ) , or graph structures ( Sohn et al. , 2018 ) . However , such task descriptions may not readily be available . A more flexible solution is to have the agents infer the task by interacting with the environment . Recent work in Meta RL ( Hochreiter et al. , 2001 ; Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017 ) ( especially in few-shot learning settings ) has attempted to have the agents implicitly infer tasks and quickly adapt to them . However , they have focused on relatively simple tasks with a single goal ( e.g. , multi-armed bandit , locomotion , navigation , etc. ) . We argue that real-world tasks often have a hierarchical structure and multiple goals , which require long horizon planning or reasoning ability ( Erol , 1996 ; Xu et al. , 2017 ; Ghazanfari & Taylor , 2017 ; Sohn et al. , 2018 ) . Take , for example , the task of making a breakfast in Figure 1 . A meal can be served with different dishes and drinks ( e.g. , boiled egg and coffee ) , where each could be considered as a subtask . These can then be further decomposed into smaller substask until some base subtask ( e.g. , pickup egg ) is reached . Each subtask can provide the agent with reward ; if only few subtasks provide reward , this is considered a sparse reward problem . When the subtask dependencies are complex and reward is sparse , learning an optimal policy can require a large number of interactions with the environment . This is the problem scope we focus on in this work : learning to quickly infer and adapt to varying hierarchical tasks with multiple goals and complex subtask dependencies . To this end , we formulate and tackle a new few-shot RL problem called subtask graph inference problem , where the task is defined as a factored MDP ( Boutilier et al. , 1995 ; Jonsson & Barto , 2006 ) with hierarchical structure represented by subtask graph ( Sohn et al. , 2018 ) where the task is not known a priori . The task consists of multiple subtasks , where each subtask gives reward when completed ( see Figure 1 ) . The complex dependencies between subtasks ( i.e. , preconditions ) enforce agent to execute all the required subtasks before it can execute a certain subtask . Intuitively , the agent can efficiently solve the task by leveraging the inductive bias of underlying task structure ( Section 2.2 ) . 1The demo videos are available at https : //bit.ly/msgi-videos . Inspired by the recent works on multi-task and few-shot RL , we propose a meta reinforcement learning approach that explicitly infers the latent structure of the task ( e.g. , subtask graph ) . The agent learns its adaptation policy to collect as much information about the environment as possible in order to rapidly and accurately infer the unknown task structure . After that , the agent ’ s test policy is a contextual policy that takes the inferred subtask graph as an input and maximizes the expected return ( See Figure 1 ) . We leverage inductive logic programming ( ILP ) technique to derive an efficient task inference method based on the principle of maximum likelihood . To facilitate learning , we adopt an intrinsic reward inspired by upper confidence bound ( UCB ) that encourages efficient exploration . We evaluate our approach on various environments ranging from simple grid-world ( Sohn et al. , 2018 ) to StarCraft II ( Vinyals et al. , 2017 ) . In all cases , our method can accurately infer the latent subtask graph structure , and adapt more efficiently to unseen tasks than the baselines . The contribution of this work can be summarized as follows : • We propose a new meta-RL problem with more general and richer form of tasks compared to the recent meta-RL approaches . • We propose an efficient task inference algorithm that leverages inductive logic programming , which accurately infers the latent subtask graph from the agent ’ s experience data . • We implement a deep meta-RL agent that efficiently infers the subtask graph for faster adaptation . • We compare our method with other meta-RL agents on various domains , and show that our method adapts more efficiently to unseen tasks with complex subtask dependencies . 2 PROBLEM DEFINITION . 2.1 BACKGROUND : FEW-SHOT REINFORCEMENT LEARNING . A task is defined by an MDPMG = ( S , A , PG , RG ) parameterized by a task parameter G with a set of states S , a set of actions A , transition dynamics PG , reward functionRG . In the K-shot RL formulation ( Duan et al. , 2016 ; Finn et al. , 2017 ) , each trial under a fixed taskMG consists of an adaptation phase where the agent learns a task-specific behavior and a test phase where the adapted behavior is evaluated . For example , RNN-based meta-learners ( Duan et al. , 2016 ; Wang et al. , 2016 ) adapt to a taskMG by updating its RNN states ( or fast-parameters ) φt , where the initialization and update rule of φt is parameterized by a slow-parameter θ : φ0 = g ( θ ) , φt+1 = f ( φt ; θ ) . Gradientbased meta-learners ( Finn et al. , 2017 ; Nichol et al. , 2018 ) instead aim to learn a good initialization of the model so that it can adapt to a new task with few gradient update steps . In the test phase , the agent ’ s performance on the taskMG is measured in terms of the return : RMG ( πφH ) = EπφH , MG [ ∑H′ t=1 rt ] , ( 1 ) Algorithm 1 Adaptation policy optimization during meta-training Require : p ( G ) : distribution over subtask graph 1 : while not done do 2 : Sample batch of task parameters { Gi } Mi=1 ∼ p ( G ) 3 : for all Gi in the batch do 4 : Rollout K episodes τ = { st , ot , rt , dt } Ht=1 ∼ π adapt θ in taskMGi . adaptation phase 5 : Compute rUCBt as in Eq . ( 7 ) 6 : Ĝi = ILP ( τ ) . subtask graph inference 7 : Sample τ ′ ∼ πexe Ĝi in taskMGi . test phase 8 : Update θ ← θ + η∇θ ∑M i=1RPG+UCBMGi ( πadaptθ ) usingRPG+UCBM in Eq . ( 9 ) where πφH is the policy after K episodes ( or H update steps ) of adaptation , H ′ is the horizon of test phase , and rt is the reward at time t in the test phase . The goal is to find an optimal parameter θ that maximizes the expected return EG [ RMG ( πφH ) ] over a given distribution of tasks p ( G ) . 2.2 THE SUBTASK GRAPH INFERENCE PROBLEM . We formulate the subtask graph inference problem , an instance of few-shot RL problem where a task is parameterized by subtask graph ( Sohn et al. , 2018 ) . The details of how a subtask graph parameterizes the MDP is described in Appendix B . Our problem extends the subtask graph execution problem in ( Sohn et al. , 2018 ) by removing the assumption that a subtask graph is given to the agent ; thus , the agent must infer the subtask graph in order to perform the complex task . Following few-shot RL settings , the agent ’ s goal is to quickly adapt to the given task ( i.e. , MDP ) in the adaptation phase to maximize the return in the test phase ( see Figure 1 ) . A task consists of N subtasks and the subtask graph models a hierarchical dependency between subtasks . Subtask : A subtask Φi can be defined by a tuple ( completion set Sicomp ⊂ S , precondition Gic : S 7→ { 0 , 1 } , subtask reward function Gir : S → R ) . A subtask Φi is complete if the current state is contained in its completion set ( i.e. , st ∈ Sicomp ) , and the agent receives a reward rt ∼ Gir upon the completion of subtask Φi . A subtask Φi is eligible ( i.e. , subtask can be executed ) if its precondition Gic is satisfied ( see Figure 1 for examples ) . A subtask graph is a tuple of precondition and subtask reward of all the subtasks : G = ( Gc , Gr ) . Then , the task defined by the subtask graph is a factored MDP ( Boutilier et al. , 1995 ; Schuurmans & Patrascu , 2002 ) ; i.e. , the transition model is factored as p ( s′|s , a ) = ∏ i pGic ( s ′ i|s , a ) and the reward function is factored as R ( s , a ) = ∑ iRGri ( s , a ) ( see Appendix for the detail ) . The main benefit of factored MDP is that it allows us to model many hierarchical tasks in a principled way with a compact representation such as dynamic Bayesian network ( Dean & Kanazawa , 1989 ; Boutilier et al. , 1995 ) . For each subtask Φi , the agent can learn an option Oi ( Sutton et al. , 1999b ) that executes the subtask2 . Environment : The state input to the agent at time step t consists of st = { xt , et , stept , epit , obst } . • Completion : xt ∈ { 0 , 1 } N indicates whether each subtask is complete . • Eligibility : et ∈ { 0 , 1 } N indicates whether each subtask is eligible ( i.e. , precondition is satisfied ) . • Time budget : stept ∈ R is the remaining time steps until episode termination . • Episode budget : epit ∈ R is the remaining number of episodes in adaptation phase . • Observation : obst ∈ RH×W×C is a ( visual ) observation at time t. At time step t , we denote the option taken by the agent as ot and the binary variable that indicates whether episode is terminated as dt . 3 METHOD . We propose a Meta-learner with Subtask Graph Inference ( MSGI ) which infers the latent subtask graph G. Figure 1 overviews our approach . Our main idea is to employ two policies : adaptation policy and test policy . During the adaptation phase , an adaptation policy πadaptθ rolls out K episodes 2As in Andreas et al . ( 2017 ) ; Oh et al . ( 2017 ) ; Sohn et al . ( 2018 ) , such options are pre-learned with curriculum learning ; the policy is learned by maximizing the subtask reward , and the initiation set and termination condition are given as Ii = { s|Gic ( s ) = 1 } and βi = I ( xi = 1 ) of adaptation trajectories . From the collected adaptation trajectories , the agent infers the subtask graph Ĝ using inductive logic programming ( ILP ) technique . A test policy πtest Ĝ , conditioned on the inferred subtask graph Ĝ , rolls out episodes and maximizes the return in the test phase . Note that the performance depends on the quality of the inferred subtask graph . The adaptation policy indirectly contributes to the performance by improving the quality of inference . Intuitively , if the adaptation policy completes more diverse subtasks during adaptation , the more “ training data ” is given to the ILP module , which results in more accurate inferred subtask graph . Algorithm 1 summarizes our meta-training procedure . For meta-testing , see Algorithm 2 in Appendix D .

Summary: The authors propose a novel meta-rl problem where hierarchical tasks are characterized by a graph describing all sub-tasks and their dependencies. They propose a meta-rl approach to meta-train a policy that quickly infers the subtask graph from new task data. The approach is compared to relevant baselines from both the meta-rl and hierarchical rl literature on complex domains. In particular, the authors consider a large-scale Startcraft II experiment which proves the efficiency and scalability of the proposed methodology.