Learning to Understand Goal Specifications by Modelling Reward

1 INTRODUCTION Figure 1 : Different valid goal states for the instruction “ build an L-like shape from red blocks ” . Developing agents that can learn to follow user instructions pertaining to an environment is a longstanding goal of AI research ( Winograd , 1972 ) . Recent work has shown deep reinforcement learning ( RL ) to be a promising paradigm for learning to follow language-like instructions in both 2D and 3D worlds ( e.g . Hermann et al . ( 2017 ) ; Chaplot et al . ( 2018 ) , see Section 4 for a review ) . In each of these cases , being able to reward an agent for successfully completing a task specified by an instruction requires the implementation of a full interpreter of the instruction language . This interpreter must be able to evaluate the instruction against environment states to determine when reward must be granted to the agent , and in doing so requires full knowledge ( on the part of the designer ) of the semantics of the instruction language relative to the environment . Consider , for example , 4 arrangements of blocks presented in Figure 1 . Each of them can be interpreted as a result of successfully executing the instruction “ build an L-like shape from red blocks ” , despite the fact that these arrangements differ in the location and the orientation of the target shape , as well as in the positioning of the irrelevant blue blocks . At best ( e.g . for instructions such as the ∗Work done during an internship at DeepMind . †Now at Facebook AI Research . aforementioned one ) , implementing such an interpreter is feasible , although typically onerous in terms of engineering efforts to ensure reward can be given—for any admissible instruction in the language—in potentially complex or large environments . At worst , if we wish to scale to the full complexity of natural language , with all its ambiguity and underspecification , this requires solving fundamental problems of natural language understanding . If instruction-conditional reward functions can not conveniently or tractably be implemented , can we somehow learn them in order to then train instruction-conditional policies ? When there is a single implicit task , Inverse Reinforcement Learning ( IRL ; Ng & Russell , 2000 ; Ziebart et al. , 2008 ) methods in general , and Generative Adversarial Imitation Learning ( Ho & Ermon , 2016 ) in particular , have yielded some success in jointly learning reward functions from expert data and training policies from learned reward models . In this paper , we wish to investigate whether such mechanisms can be adapted to the more general case of jointly learning to understand language which specifies task objectives ( e.g . instructions , goal specifications , directives ) , and use such understanding to reward language-conditional policies which are trained to complete such tasks . For simplicity , we explore a facet of this general problem in this paper by focussing on the case of declarative commands that specify sets of possible goal-states ( e.g . “ arrange the red blocks in a circle. ” ) , and where expert examples need only be goal states rather than full trajectories or demonstrations , leaving such extensions for further work . We introduce a framework—Adversarial Goal-Induced Learning from Examples ( AGILE ) —for jointly training an instruction-conditional reward model using expert examples of completed instructions alongside a policy which will learn to complete instructions by maximising the thus-modelled reward . In this respect , AGILE relies on familiar RL objectives , with free choice of model architecture or training mechanisms , the only difference being that the reward comes from a learned reward model rather than from the environment . We first verify that our method works in settings where a comparison between AGILE-trained policies with policies trained from environment reward is possible , to which end we implement instructionconditional reward functions . In this setting , we show that the learning speed and performance of A3C agents trained with AGILE reward models is superior to A3C agents trained against environment reward , and comparable to that of true-reward A3C agents supplemented by auxiliary unsupervised reward prediction objectives . To simulate an instruction-learning setting in which implementing a reward function would be problematic , we construct a dataset of instructions and goal-states for the task of building colored orientation-invariant arrangements of blocks . On this task , without us ever having to implement the reward function , the agent trained within AGILE learns to construct arrangements as instructed . Finally , we study how well AGILE ’ s reward model generalises beyond the examples on which it was trained . Our experiments show it can be reused to allow the policy to adapt to changes in the environment . 2 ADVERSARIAL GOAL-INDUCED LEARNING FROM EXAMPLES . Here , we introduce AGILE ( “ Adversarial Goal-Induced Learning from Examples ” , in homage to the adversarial learning mechanisms that inspire it ) , a framework for jointly learning to model reward for instructions , and learn a policy from such a reward model . Specifically , we learn an instructionconditional policy πθ with parameters θ , from a data stream Gπθ obtained from interaction with the environment , by adjusting θ to maximise the expected total reward Rπ ( θ ) based on stepwise reward r̂t given to the policy , exactly as done in any normal Reinforcement Learning setup . The difference lies in the source of the reward : we introduce an additional discriminator network Dφ , the reward model , whose purpose is to define a meaningful reward function for training πθ . We jointly learn this reward model alongside the policy by training it to predict whether a given state s is a goal state for a given instruction c or not . Rather than obtain positive and negative examples of 〈instruction , state〉 pairs from a purely static dataset , we sample them from a policy-dependent data stream . This stream is defined as follows : positive examples are drawn from a fixed dataset D of instructions ci paired with goal states si ; negative examples are drawn from a constantly-changing buffer of states obtained from the policy acting on the environment , paired with the instruction given to the policy . Formally , the policy is trained to maximize a return Rπ ( θ ) and the reward model is trained to minimize a cross-entropy loss LD ( φ ) , the equations for which are : Rπ ( θ ) = E ( c , s1 : ∞ ) ∼Gπθ ∞∑ t=1 γt−1r̂t + αH ( πθ ) , ( 1 ) LD ( φ ) = E ( c , s ) ∼B − log ( 1−Dφ ( c , s ) ) + E ( ci , gi ) ∼D − logDφ ( ci , gi ) . ( 2 ) where r̂t = [ Dφ ( c , st ) > 0.5 ] In the equations above , the Iverson Bracket [ . . . ] maps truth to 1 and falsehood to 0 , e.g . [ x > 0 ] = 1 iff x > 0 and 0 otherwise . γ is the discount factor . With ( c , s1 : ∞ ) ∼ Gπθ , we denote a state trajectory that was obtained by sampling ( c , s0 ) ∼ G and running πθ conditioned on c starting from s0 . B denotes a replay buffer to which ( c , s ) pairs from T -step episodes are added ; i.e . it is the undiscounted occupancy measure over the first T steps . Dφ ( c , s ) is the probability of ( c , s ) having a positive label according to the reward model , and thus [ Dφ ( c , st ) > 0.5 ] indicates that a given state st is more likely to be a goal state for instruction c than not , according to D. H ( πθ ) is the policy ’ s entropy , and α is a hyperparameter . The approach is illustrated in Fig 2 . Pseudocode is available in Appendix A . We note that Equation 1 differs from a traditional RL objective only in that the modelled reward r̂t is used instead of the ground-truth reward rt . Indeed , in Section 3 , we will compare policies trained with AGILE to policies trained with traditional RL , simply by varying the reward source from the reward model to the environment . Dealing with False Negatives Let us call Γ ( c ) the objective set of goal states which satisfy instruction c ( which is typically unknown to us ) . Compared to the ideal case where all ( c , s ) would be deemed positive if-and-only-if s ∈ Γ ( c ) , the labelling of examples implied by Equation 2 has a fundamental limitation when the policy performs well . As the policy improves , by definition , a increasing share of ( c , s ) ∈ B are objective goal-states from Γ ( c ) . However , as they are treated as negative examples in Equation 2 , the discriminator accuracy drops , causing the policy to get worse . We therefore propose the following simple heuristic to rectify this fundamental limitation by approximately identifying the false negatives . We rank ( c , s ) examples in B according to the reward model ’ s output Dφ ( c , s ) and discard the top 1− ρ percent as potential false negatives . Only the other ρ percent are used as negative examples of the reward model . Formally speaking , the first term in Equation 2 becomes E ( c , s ) ∼BDφ , ρ − log ( 1 −Dφ ( c , s ) ) , where BDφ , ρ stands for the ρ percent of B selected , using Dφ , as described above . We will henceforth refer to ρ as the anticipated negative rate . Setting ρ to 100 % means using BDφ,100 = B like in Equation 2 , but our preliminary experiments have shown clearly that this inhibits the reward model ’ s capability to correctly learn a reward function . Using too small a value for ρ on the other hand may deprive the reward model of the most informative negative examples . We thus recommend to tune ρ as a hyperparameter on a task-specific basis . Reusability of the Reward Model An appealing advantage of AGILE is the fact that the reward model Dφ and the policy πθ learn two related but distinct aspects of an instruction : the reward model focuses on recognizing the goal-states ( what should be done ) , whereas the policy learns what to do in order to get to a goal-state ( how it should be done ) . The intuition motivating this design is that the knowledge about how instructions define goals should generalize more strongly than the knowledge about which behavior is needed to execute instructions . Following this intuition , we propose to reuse a reward model trained in AGILE as a reward function for training or fine-tuning policies . Relation to GAIL AGILE is strongly inspired by—and retains close relations to—Generative Adversarial Imitation Learning ( GAIL ; Ho & Ermon , 2016 ) , which likewise trains both a reward function and a policy . The former is trained to distinguish between the expert ’ s and the policy ’ s trajectories , while the latter is trained to maximize the modelled reward . GAIL differs from AGILE in a number of important respects . First , AGILE is conditioned on instructions c so a single AGILE agent can learn combinatorially many skills rather than just one . Second , in AGILE the reward model observes only states si ( either goal states from an expert , or states from the agent acting on the environment ) rather than state-action traces ( s1 , a1 ) , ( s2 , a2 ) , . . . , learning to reward the agent based on “ what ” needs to be done rather than according to “ how ” it must be done . Finally , in AGILE the policy ’ s reward is the thresholded probability [ Dφ ( c , st ) ] as opposed to the log-probability logDφ ( st , at ) used in GAIL . Our reasoning for this change is that , when adapted to the setting with goal-specifications , a GAIL-style reward logDφ ( c , st ) could take arbitrarily low values for intermediate states visited by the agent , as the reward model Dφ becomes confident that those are not goal states . Empirically , we found that dropping the logarithm from GAIL-style rewards is indeed crucial for AGILE ’ s performance , and that using the probability Dφ ( c , st ) as the reward r̂t results in a performance level similar to that of the discretized AGILE reward r̂t = [ Dφ ( c , st ) ] .

Summary: The paper presents an approach for simultaneously learning policies and reward functions for reaching goals that are described by an instruction providing spatial relations among objects. The proposed platform, called Adversarial Goal-Induced Learning from Examples (AGILE), is composed of an off-the-shelf RL module like A3C and a separate module for learning a reward function, implemented using the NMN paradigm. The RL module is trained using the reward function learned by the reward module. The reward module is trained to map a given <instruction, state> into a score between 0 and 1 depending on how well the provided state satisfies the instructions provided in the instruction. The returned score is used as a reward function. The training of the reward function is performed by using a dataset of positive examples, and using the states visited by the agent while it's learning as negative examples. To account for the fact that the agent becomes better over time and its visited states can no longer be used as negative examples, the authors proposed a heuristic where the states visited by the agent are not all used as negative examples, but only those that have the lowest scores.