Program Guided Agent

1 INTRODUCTION . Humans are capable of leveraging instructions to accomplish complex tasks . A comprehensive instruction usually comprises a set of descriptions detailing a variety of situations and the corresponding subtasks that are required to be fulfilled . To accomplish a task , we can leverage instructions to estimate the progress , recognize the current state , and perform corresponding actions . For example , to make a gourmet dish , we can follow recipes and procedurally create the desired dish by recognizing what ingredients and tools are missing , what alternatives are available , and what corresponding preparations are required . With sufficient practice , we can improve our ability to perceive ( e.g . knowing when food is well-cooked ) as well as master cooking skills ( e.g . cutting food into same-sized pieces ) , and eventually accomplish difficult recipes . Can machines likewise learn to follow and exploit comprehensive instructions like humans ? Utilizing expert demonstrations to instruct agents has been widely studied in ( Finn et al. , 2017 ; Yu et al. , 2018b ; Xu et al. , 2018 ; Pathak et al. , 2018 ; Stadie et al. , 2017 ; Duan et al. , 2017 ; Wang et al. , 2017b ) . However , demonstrations could be expensive to obtain and are less flexible ( e.g . altering subtask orders in demonstrations is nontrivial ) . On the other hand , natural language instructions are flexible and expressive ( Malmaud et al. , 2014 ; Jermsurawong & Habash , 2015 ; Kiddon et al. , 2015 ; Misra et al. , 2018 ; Fried et al. , 2018 ; Kaplan et al. , 2017 ; Bahdanau et al. , 2019 ) . Yet , language has the caveat of being ambiguous even to humans , due to its lacking of structure as well as unclear coreferences and entities . Andreas et al . ( 2017a ) ; Oh et al . ( 2017 ) investigate a hierarchical approach , where the instructions consist of a set of symbolically represented subtasks . Nonetheless , those instructions are not a function of states ( i.e . describe a variety of circumstances and the corresponding desired subtasks ) , which substantially limits their expressiveness . We propose to utilize programs , written in a formal language , as a structured , expressive , and unambiguous representation to specify tasks . Specifically , we consider programs , which are composed of control flows ( e.g . if/else and loops ) , environmental conditions , as well as corresponding subtasks , as shown in Figure 1 . Not only do programs have expressiveness by describing diverse situations ( e.g . a river exists ) and the corresponding subtasks which are required to be executed ( e.g . mining wood ) , but they are also unambiguous due to their explicit scoping . To study the effectiveness of using programs as task specifications , we introduce a new problem , where we aim to develop a framework which learns to comprehend a task specified by a program as well as perceive and interact with the environment to accomplish the task . To address this problem , we propose a modular framework , program guided agent , which exploits the structural nature of programs to decompose and execute them as well as learn to ground program tokens with the environment . Specifically , our framework consists of three modules : ( 1 ) a program interpreter that leverages a grammar provided by the programming language to parse and execute a program , ( 2 ) a perception module that learns to respond to conditional queries ( e.g . is_there [ River ] ) produced by the interpreter , and ( 3 ) a policy that learns to fulfill a variety of subtasks ( e.g . mine ( Wood ) ) extracted from the program by the interpreter . To effectively instruct the policy with symbolically represented subtasks , we introduce a learned modulation mechanism that leverages a subtask to modulate the encoded state features instead of concatenating them . Our framework ( shown in Figure 3 ) utilizes a rule-based program interpreter to deal with programs as well as learning perception module and policy when it is necessary to perceive or interact with the environment . With this modularity , our framework can generalize to more complex program-specified tasks without additional learning . To evaluate the proposed framework , we consider a Minecraft-inspired 2D gridworld environment , where an agent can navigate itself across different terrains and interact with objects , similar to Andreas et al . ( 2017a ) ; Sohn et al . ( 2018 ) . A corresponding domain-specific language ( DSL ) defines the rules of constructing programs for instructing an agent to accomplish certain tasks . Our proposed framework demonstrates superior generalization ability – learning from simpler tasks while generalizing to complex tasks . We also conduct extensive analysis on various end-to-end learning models which learns from not only program instructions but also natural language descriptions . Furthermore , our proposed learned policy modulation mechanism yields a better learning efficiency compared to other commonly used methods that simply concatenate a state and goal . 2 RELATED WORK . Learning from language instructions . Prior works have investigated leveraging natural languages to specify tasks on a wide range of applications , including navigation ( Misra et al. , 2018 ; Tellex et al. , 2011 ; Fried et al. , 2018 ; Vogel & Jurafsky , 2010 ; Shimizu & Haas , 2009 ; Branavan et al. , 2009 ; 2012 ; Misra et al. , 2017 ; Tellex et al. , 2011 ) , spatial reasoning for goal reaching ( Janner et al. , 2018 ) , game playing ( Kaplan et al. , 2017 ; Fried et al. , 2017 ; Co-Reyes et al. , 2019 ) , and grounding visual concepts ( Kaplan et al. , 2017 ; Bahdanau et al. , 2019 ; Andreas et al. , 2017b ) . However , natural language descriptions can often be ambiguous even to humans . Moreover , it is not clear how end-toend learning agents trained with simpler instructions can generalize well to much more complex ones . In contrast , we propose to utilize a precise and structured representation , programs , to specify tasks . Learning from demonstrations . When a task can not be easily described in languages ( e.g . object texture or geometry ) , expert demonstrations offer an alternative way to provide instructions . Prior works have explored learning from video demonstrations ( Finn et al. , 2017 ; Yu et al. , 2018b ; Xu et al. , 2018 ; Pathak et al. , 2018 ; Stadie et al. , 2017 ; Aytar et al. , 2018 ) or expert trajectories ( Duan et al. , 2017 ; Wang et al. , 2017b ) . However , demonstrations can be expensive to obtain and are less expressive about diverging behaviors of a complex task , which are better captured by control flow in programs . Moreover , editing demonstrations such as rearranging the order of subtasks is often difficult . Program induction and synthesis . To learn acquire programmatic skills such as digit addition and string transformations and achieve better generalization , program induction methods ( Xu et al. , 2018 ; Devlin et al. , 2017a ; Neelakantan et al. , 2015 ; Graves et al. , 2014 ; Kaiser & Sutskever , 2016 ; Reed & De Freitas , 2016 ; Cai et al. , 2017 ; Xiao et al. , 2018 ) aim to implicitly induce the underlying programs to mimic the behaviors demonstrated in task specifications ( e.g . input/output pairs , demonstrations ) . On the other hand , program synthesis methods ( Bošnjak et al. , 2017 ; Parisotto et al. , 2017 ; Devlin et al. , 2017b ; Chen et al. , 2019 ; Shin et al. , 2018 ; Bunel et al. , 2018 ; Liu et al. , 2019 ; Sun et al. , 2018 ; Lin et al. , 2018 ; Liao et al. , 2019 ) explicitly synthesize the underlying programs and execute the programs to perform the tasks . Instead of trying to infer programs from task specifications , we are interested in explicitly executing programs . Also , our framework can potentially be leveraged to obtain program execution results for evaluating program synthesis frameworks when no program executor is available ( e.g . programs describe real-world activities instead of behaviors in simulation ) . Symbolic planing and programmable agent . Classical symbolic planning concerns the problem of achieving a goal state from an initial state through a series of symbolically represented executions ( Ghallab et al . ; Konidaris et al. , 2018 ) . Our work shares a similar spirit but assume a task ( i.e . a program ) is given , where the agent is required to learn to ground symbolic concepts ( Mao et al. , 2019 ; Han et al. , 2019 ) and follow the control flow . Executing programs with reinforcement learning has been studied in programmable hierarchies of abstract machines ( Parr & Russell , 1998 ; Andre & Russell , 2001 ; 2002 ) , which provide partial descriptions and subroutines of the desired task . Denil et al . ( 2017 ) ; Lázaro-Gredilla et al . ( 2018 ) train agents to execute declarative programs by grounding these well-structured languages in their learning environments . In contrast , our modular framework consists of modules for perceiving the environment and interacting with it by following an imperative program which specifies the task . An extended discussion on the related work can be found in Section C . 3 PROBLEM FORMULATION . We are interested in learning to comprehend and execute an instruction specified by a program to fulfill the desired task . In this section , we formally describe our definition of programs , the family of Markov Decision Processes ( MDPs ) , and the problem formulation . Program . The programs considered in this work are defined based on a Domain Specific Language ( DSL ) as shown in Figure 2 . The DSL is composed of perception primitives , action primitives , and control flow . A perception primitive indicates circumstances in the environment ( e.g . is_there ( River ) , and agent [ Gold ] < 3 ) that can be perceived by an agent , while an action primitive defines a subtask that describes a certain behavior ( e.g . mine ( Gold ) , and goto ( 1,1 ) ) . Control flow includes if/else statements , loops , and Boolean/logical operators to compose more sophisticated conditions . A program p is a deterministic function that outputs a desired behavior ( i.e . subtask ) given a history of states ot = p ( Hj ) , where Hj = { s1 , ... , st } is a state history with s ∈ S denoting a state of the environment , and o ∈ O denotes an instructed behavior ( subtask ) . We denote a program as p ∼ P , an infinite program set containing all executable programs given a DSL . Note that a discussion on the DSL design principle can be found in Section B. MDPs . We consider a family of finite-horizon discounted MDPs in a shared environment , specified by a tuple ( S , A , P , T , R , ρ , γ ) , where S denotes a set of states , A denotes a set of low-level actions an agent can take , P denotes a set of programs specifying instructions , T : S ×A×S → R denotes a transition probability distribution , R denotes a task-specific reward function , ρ denotes an initial state distribution , and γ denotes a discount factor . For a fixed sequence { ( s0 , a0 ) , ... , ( st , at ) } of states and actions obtained from a rollout of a given policy π , the performance of the policy is evaluated based on a discounted return ∑T t=0 γ trt , where T is the horizon of the episode . Problem Formulation . We consider developing a framework which can comprehend and fulfill an instruction specified by a program . Specifically , we consider a sampled MDP with a program describing the desired task . Addressing this task requires the ability to keep track of which parts of the program are finished and which parts are not , perceiving the environment and deciding which paths in the program to take , and performing actions interacting with the environment to fulfill subtasks .

Summary: This paper presents a reinforcement learning agent that learns to execute tasks specified in a form of programs with an architecture consisting of three modules. The (fixed) interpreter module interprets the program, by issuing queries to a (pre-trained) vision module and giving goals to a policy module that executes them in the environment. The paper also introduces a policy modulation technique, with the goal of modulating the current state with the expected (symbolic) goal. The model is evaluated on a 2D approximation of Minecraft, where it outperforms a set of baselines. In addition, the authors modified the program dataset and re-expressed it in terms of natural language and showed that the baselines perform better with programs than with the natural language instructions.