Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies

1 INTRODUCTION . Recently , reinforcement learning ( RL ) systems have achieved super-human performance on many complex tasks ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Van Seijen et al. , 2017 ) . However , these works mostly have been focused on a single known task where the agent can be trained for a long time ( e.g. , Silver et al . ( 2016 ) ) . We argue that agent should be able to solve multiple tasks with varying sources of reward . Recent work in multi-task RL has attempted to address this ; however , they focused on the setting where the structure of task are explicitly described with natural language instructions ( Oh et al. , 2017 ; Andreas et al. , 2017 ; Yu et al. , 2017 ; Chaplot et al. , 2018 ) , programs ( Denil et al. , 2017 ) , or graph structures ( Sohn et al. , 2018 ) . However , such task descriptions may not readily be available . A more flexible solution is to have the agents infer the task by interacting with the environment . Recent work in Meta RL ( Hochreiter et al. , 2001 ; Duan et al. , 2016 ; Wang et al. , 2016 ; Finn et al. , 2017 ) ( especially in few-shot learning settings ) has attempted to have the agents implicitly infer tasks and quickly adapt to them . However , they have focused on relatively simple tasks with a single goal ( e.g. , multi-armed bandit , locomotion , navigation , etc. ) . We argue that real-world tasks often have a hierarchical structure and multiple goals , which require long horizon planning or reasoning ability ( Erol , 1996 ; Xu et al. , 2017 ; Ghazanfari & Taylor , 2017 ; Sohn et al. , 2018 ) . Take , for example , the task of making a breakfast in Figure 1 . A meal can be served with different dishes and drinks ( e.g. , boiled egg and coffee ) , where each could be considered as a subtask . These can then be further decomposed into smaller substask until some base subtask ( e.g. , pickup egg ) is reached . Each subtask can provide the agent with reward ; if only few subtasks provide reward , this is considered a sparse reward problem . When the subtask dependencies are complex and reward is sparse , learning an optimal policy can require a large number of interactions with the environment . This is the problem scope we focus on in this work : learning to quickly infer and adapt to varying hierarchical tasks with multiple goals and complex subtask dependencies . To this end , we formulate and tackle a new few-shot RL problem called subtask graph inference problem , where the task is defined as a factored MDP ( Boutilier et al. , 1995 ; Jonsson & Barto , 2006 ) with hierarchical structure represented by subtask graph ( Sohn et al. , 2018 ) where the task is not known a priori . The task consists of multiple subtasks , where each subtask gives reward when completed ( see Figure 1 ) . The complex dependencies between subtasks ( i.e. , preconditions ) enforce agent to execute all the required subtasks before it can execute a certain subtask . Intuitively , the agent can efficiently solve the task by leveraging the inductive bias of underlying task structure ( Section 2.2 ) . 1The demo videos are available at https : //bit.ly/msgi-videos . Inspired by the recent works on multi-task and few-shot RL , we propose a meta reinforcement learning approach that explicitly infers the latent structure of the task ( e.g. , subtask graph ) . The agent learns its adaptation policy to collect as much information about the environment as possible in order to rapidly and accurately infer the unknown task structure . After that , the agent ’ s test policy is a contextual policy that takes the inferred subtask graph as an input and maximizes the expected return ( See Figure 1 ) . We leverage inductive logic programming ( ILP ) technique to derive an efficient task inference method based on the principle of maximum likelihood . To facilitate learning , we adopt an intrinsic reward inspired by upper confidence bound ( UCB ) that encourages efficient exploration . We evaluate our approach on various environments ranging from simple grid-world ( Sohn et al. , 2018 ) to StarCraft II ( Vinyals et al. , 2017 ) . In all cases , our method can accurately infer the latent subtask graph structure , and adapt more efficiently to unseen tasks than the baselines . The contribution of this work can be summarized as follows : • We propose a new meta-RL problem with more general and richer form of tasks compared to the recent meta-RL approaches . • We propose an efficient task inference algorithm that leverages inductive logic programming , which accurately infers the latent subtask graph from the agent ’ s experience data . • We implement a deep meta-RL agent that efficiently infers the subtask graph for faster adaptation . • We compare our method with other meta-RL agents on various domains , and show that our method adapts more efficiently to unseen tasks with complex subtask dependencies . 2 PROBLEM DEFINITION . 2.1 BACKGROUND : FEW-SHOT REINFORCEMENT LEARNING . A task is defined by an MDPMG = ( S , A , PG , RG ) parameterized by a task parameter G with a set of states S , a set of actions A , transition dynamics PG , reward functionRG . In the K-shot RL formulation ( Duan et al. , 2016 ; Finn et al. , 2017 ) , each trial under a fixed taskMG consists of an adaptation phase where the agent learns a task-specific behavior and a test phase where the adapted behavior is evaluated . For example , RNN-based meta-learners ( Duan et al. , 2016 ; Wang et al. , 2016 ) adapt to a taskMG by updating its RNN states ( or fast-parameters ) φt , where the initialization and update rule of φt is parameterized by a slow-parameter θ : φ0 = g ( θ ) , φt+1 = f ( φt ; θ ) . Gradientbased meta-learners ( Finn et al. , 2017 ; Nichol et al. , 2018 ) instead aim to learn a good initialization of the model so that it can adapt to a new task with few gradient update steps . In the test phase , the agent ’ s performance on the taskMG is measured in terms of the return : RMG ( πφH ) = EπφH , MG [ ∑H′ t=1 rt ] , ( 1 ) Algorithm 1 Adaptation policy optimization during meta-training Require : p ( G ) : distribution over subtask graph 1 : while not done do 2 : Sample batch of task parameters { Gi } Mi=1 ∼ p ( G ) 3 : for all Gi in the batch do 4 : Rollout K episodes τ = { st , ot , rt , dt } Ht=1 ∼ π adapt θ in taskMGi . adaptation phase 5 : Compute rUCBt as in Eq . ( 7 ) 6 : Ĝi = ILP ( τ ) . subtask graph inference 7 : Sample τ ′ ∼ πexe Ĝi in taskMGi . test phase 8 : Update θ ← θ + η∇θ ∑M i=1RPG+UCBMGi ( πadaptθ ) usingRPG+UCBM in Eq . ( 9 ) where πφH is the policy after K episodes ( or H update steps ) of adaptation , H ′ is the horizon of test phase , and rt is the reward at time t in the test phase . The goal is to find an optimal parameter θ that maximizes the expected return EG [ RMG ( πφH ) ] over a given distribution of tasks p ( G ) . 2.2 THE SUBTASK GRAPH INFERENCE PROBLEM . We formulate the subtask graph inference problem , an instance of few-shot RL problem where a task is parameterized by subtask graph ( Sohn et al. , 2018 ) . The details of how a subtask graph parameterizes the MDP is described in Appendix B . Our problem extends the subtask graph execution problem in ( Sohn et al. , 2018 ) by removing the assumption that a subtask graph is given to the agent ; thus , the agent must infer the subtask graph in order to perform the complex task . Following few-shot RL settings , the agent ’ s goal is to quickly adapt to the given task ( i.e. , MDP ) in the adaptation phase to maximize the return in the test phase ( see Figure 1 ) . A task consists of N subtasks and the subtask graph models a hierarchical dependency between subtasks . Subtask : A subtask Φi can be defined by a tuple ( completion set Sicomp ⊂ S , precondition Gic : S 7→ { 0 , 1 } , subtask reward function Gir : S → R ) . A subtask Φi is complete if the current state is contained in its completion set ( i.e. , st ∈ Sicomp ) , and the agent receives a reward rt ∼ Gir upon the completion of subtask Φi . A subtask Φi is eligible ( i.e. , subtask can be executed ) if its precondition Gic is satisfied ( see Figure 1 for examples ) . A subtask graph is a tuple of precondition and subtask reward of all the subtasks : G = ( Gc , Gr ) . Then , the task defined by the subtask graph is a factored MDP ( Boutilier et al. , 1995 ; Schuurmans & Patrascu , 2002 ) ; i.e. , the transition model is factored as p ( s′|s , a ) = ∏ i pGic ( s ′ i|s , a ) and the reward function is factored as R ( s , a ) = ∑ iRGri ( s , a ) ( see Appendix for the detail ) . The main benefit of factored MDP is that it allows us to model many hierarchical tasks in a principled way with a compact representation such as dynamic Bayesian network ( Dean & Kanazawa , 1989 ; Boutilier et al. , 1995 ) . For each subtask Φi , the agent can learn an option Oi ( Sutton et al. , 1999b ) that executes the subtask2 . Environment : The state input to the agent at time step t consists of st = { xt , et , stept , epit , obst } . • Completion : xt ∈ { 0 , 1 } N indicates whether each subtask is complete . • Eligibility : et ∈ { 0 , 1 } N indicates whether each subtask is eligible ( i.e. , precondition is satisfied ) . • Time budget : stept ∈ R is the remaining time steps until episode termination . • Episode budget : epit ∈ R is the remaining number of episodes in adaptation phase . • Observation : obst ∈ RH×W×C is a ( visual ) observation at time t. At time step t , we denote the option taken by the agent as ot and the binary variable that indicates whether episode is terminated as dt . 3 METHOD . We propose a Meta-learner with Subtask Graph Inference ( MSGI ) which infers the latent subtask graph G. Figure 1 overviews our approach . Our main idea is to employ two policies : adaptation policy and test policy . During the adaptation phase , an adaptation policy πadaptθ rolls out K episodes 2As in Andreas et al . ( 2017 ) ; Oh et al . ( 2017 ) ; Sohn et al . ( 2018 ) , such options are pre-learned with curriculum learning ; the policy is learned by maximizing the subtask reward , and the initiation set and termination condition are given as Ii = { s|Gic ( s ) = 1 } and βi = I ( xi = 1 ) of adaptation trajectories . From the collected adaptation trajectories , the agent infers the subtask graph Ĝ using inductive logic programming ( ILP ) technique . A test policy πtest Ĝ , conditioned on the inferred subtask graph Ĝ , rolls out episodes and maximizes the return in the test phase . Note that the performance depends on the quality of the inferred subtask graph . The adaptation policy indirectly contributes to the performance by improving the quality of inference . Intuitively , if the adaptation policy completes more diverse subtasks during adaptation , the more “ training data ” is given to the ILP module , which results in more accurate inferred subtask graph . Algorithm 1 summarizes our meta-training procedure . For meta-testing , see Algorithm 2 in Appendix D .

Summary: This paper proposes a new meta-reinforcement learning algorithm, MSGI, which focuses on the problem of adapting to unseen hierarchical tasks through interaction with the environment where the external reward is sparse. The authors make use of subtask graph inference to infer the latent subtask representation of a task through interacting with the environment using an adaptation policy and then optimize the adaptation policy based on the inferred latent subtask structure. Each task in the paper is represented as a tuple of subtask precondition and subtask reward, which are inferred via logic induction and MLE of Gaussians respectively. At meta-test time, MSGI rollouts a subtask graph execution (SGE) policy based on the graph inferred from the interactions between the environment and the adaptation policy. The authors also propose a UCB-inspired intrinsic reward to encourage exploration when optimizing the adaptation policy. Experiments are conducted on two grid-world domains as well as StarCraft II.