From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

1 INTRODUCTION While reinforcement learning provides a powerful and flexible framework for describing and solving control tasks , it requires the practitioner to specify objectives in terms of reward functions . Engineering reward functions is often done by experienced practitioners and researchers , and even then can pose a significant challenge , such as when working with complex image-based observations . While researchers have investigated alternative means of specifying objectives , such as learning from demonstration ( Argall et al. , 2009 ) , or through binary preferences ( Christiano et al. , 2017 ) , language is often a more natural and desirable way for humans to communicate goals . A common approach to building natural language interfaces for reinforcement learning agents is to build language-conditioned policies that directly map observations and language commands to a sequence of actions that perform the desired task . However , this requires the policy to solve two challenging problems together : understanding how to plan and solve tasks in the physical world , and understanding the language command itself . The trained policy must simultaneously interpret a command and plan through ∗Work done during an internship at Google AI Research possibly complicated environment dynamics . The performance of the system then hinges entirely on its ability to generalize to new environments - if either the language interpretation or the physical control fail to generalize , the entire system will fail . We can recognize instead that the role of language in such a system is to communicate the goal , and rather than mapping language directly to policies , we propose to learn how to convert language-defined goals into reward functions . In this manner , the agent can learn how to plan and perform the task on its own via reinforcement learning , directly interacting with the environment , without relying on zero-shot transfer of policies . A simple example is shown in Figure 1 , where an agent is tasked with navigating through a house . If an agent is commanded “ go to the fruit bowl ” , a valid reward function could simply be a fruit bowl detector from first-person views of the agent . However , if we were to learn a mapping from language to actions , given the same goal description , the model would need to generate a different plan for each house . In this work , we investigate the feasibility of grounding free-form natural language commands as reward functions using inverse reinforcement learning ( IRL ) . Learning language-conditioned rewards poses unique computational problems . IRL methods generally require solving a reinforcement learning problem as an inner-loop ( Ziebart , 2010 ) , or rely on potentially unstable adversarial optimization procedures ( Finn et al. , 2016 ; Fu et al. , 2018 ) . This is compounded by the fact that we wish to train our model across multiple tasks , meaning the IRL problem itself is an inner-loop . In order to isolate the language-learning problem from the difficulties in solving reinforcement learning and adversarial learning problems , we base our method on an exact MaxEnt IRL ( Ziebart , 2010 ) procedure , which requires full knowledge of environment dynamics to train a language-conditioned reward function represented by a deep neural network . While using exact IRL procedures may seem limiting , in many cases ( such as indoor robotic navigation ) full environment dynamics are available , and this formulation allows us to remove the difficulty of using RL from the training procedure . The crucial insight is that we can use dynamic programming methods during training to learn a reward function that maps from observations , but we do not need knowledge of dynamics to use the reward function , meaning during test time we can evaluate using a reinforcement learning agent without knowledge of the underlying environment dynamics . We evaluate our method on a dataset of realistic indoor house navigation and pick-and-place tasks using the SUNCG dataset , with natural language commands . We demonstrate that our approach generalizes not only to novel tasks , but also to entirely new scenes , while directly learning a language-conditioned policy leads to poor performance and fails to generalize . 2 RELATED WORK . A popular class of approaches to language grounding in reinforcement learning is to directly train a policy that consumes language as an input . Several works adopt a behavioral cloning approach , where the model is trained using supervised learning with language-action sequences pairs ( Anderson et al. , 2018 ; Mei et al. , 2016 ; Sung et al. , 2015 ) . A second approach is to forego demonstrations but instead reward an agent whenever the desired task is completed ( Shah et al. , 2018 ; Misra et al. , 2017 ; Hermann et al. , 2017 ; Branavan et al. , 2009 ) . This approach requires reward functions ( the task completion detector ) to be hand-designed for the training tasks considered . Another related approach is semantic parsing , which has also been used to convert language into an executable form that corresponds to actions within an environment ( Forbes et al. , 2015 ; Misra et al. , 2014 ; Tellex et al. , 2011 ) . In a related task to instruction following , Das et al . ( 2018 ) consider an embodied question-answering task where an agent must produce an answer to a question , where the relevant information lies within the environment . They adopt a hybrid approach , where they pretrain with supervised learning but also give the agent reward for completing intermediate tasks . Overall , our experiments show that policy-based approaches have worse generalization performance to new environments , because the policy must rely on zero-shot generalization at test time as we show in Section 6.3 . While in this paper we argue for the performance benefits of a reward-based approach , a reason one may want to adopt a policy-based approach over a reward-based one is if one can not run RL to train a new policy in a new environment , such as for time or safety reasons . A second approach to the language grounding problem is to learn a mapping from language to reward functions . There are several other works that apply IRL or IRL-like procedures to the problem of language grounding . Perhaps most closely related to our work is MacGlashan et al . ( 2015 ) , which also aims to learn a language-conditioned reward function via IRL . However , this method requires an extensively hand-designed , symbolic reward function class , whereas we use generic , differentiable function approximators that can handle arbitrary observations , including raw images . Bahdanau et al . ( 2018 ) ; Tung et al . ( 2018 ) also learn language-conditioned reward functions , but do not perform IRL , meaning that the objective does not correspond to matching the expert ’ s trajectory distribution . Tung et al . ( 2018 ) train a task-completion classifier , but do not evaluate their reward on control problems . The strategy they use is similar to directly regressing onto a ground-truth reward function , which we include a comparison to in Section 6 to as an oracle baseline . Bahdanau et al . ( 2018 ) adopt an adversarial approach similar to GAIL ( Ho & Ermon , 2016 ) , and use the learned discriminator as the reward function . While this produces a reward function , it does not provide any guarantees that the resulting reward function can be reoptimized in new environments to yield behavior similar to the expert . We believe our work is the first to apply language-conditioned inverse reinforcement learning to environments with image observations and deep neural networks , and we show that our rewards generalize to novel tasks and environments . 3 BACKGROUND . We build off of the MaxEnt IRL model ( Ziebart et al. , 2008 ) , which considers an entropy-regularized Markov decision process ( MDP ) , defined by the tuple ( S , A , T , r , γ , ρ0 ) . S , A are the state and action spaces respectively and γ ∈ ( 0 , 1 ) is the discount factor . T ( s′|s , a ) represents the transition distribution or dynamics . We additionally consider partially-observed environments , where each state is associated with an observation within an observations space o ∈ O . The goal of ” forward ” reinforcement learning is to find the optimal policy π∗ . Let r ( τ ) =∑T t=0 γ tr ( st , at ) denote the returns of a trajectory , where τ denotes a sequence of states and actions ( s0 , a0 , ... sT , aT ) . The MaxEnt RL objective is then to find π∗ = argmaxπ Eτ∼π [ r ( τ ) +H ( τ ) ] . Inverse reinforcement learning ( IRL ) seeks to infer the reward function r ( s , a ) given a set of expert demonstrationsD = { τ1 , ... , τN } , . In IRL , we assume the demonstrations are drawn from an optimal policy π∗ ( a|s ) . We can interpret the IRL problem as solving the maximum likelihood problem : max θ Eτ∼D [ log pθ ( τ ) ] , ( 1 ) In the MaxEnt IRL framework , optimal trajectories are observed with probabilities proportional to the exponentiated returns , meaning p ( τ ) ∝ exp { r ( τ ) } ( Ziebart et al. , 2008 ) . Thus , learning a reward function rθ ( τ ) is equivalent to fitting an energy-based model pθ ( τ ) ∝ exp { rθ ( τ ) } to the maximum likelihood objective in Eqn 1 . The gradient to update the reward function is ( Ziebart , 2010 ) : ∇θE [ log pθ ( τ ) ] = ∑ s , a ( ρD ( s , a ) − ρ∗θ ( s , a ) ) ∇θrθ ( s , a ) , ( 2 ) where ρD ( s , a ) represents the state-action marginal of the demonstrations , and ρ∗θ ( s , a ) represents the state-action marginal of the optimal policy under reward rθ ( s , a ) . 4 MULTI-TASK IRL . A unique challenge of the language-conditioned IRL problem , compared to standard IRL , is that the goal is to learn a reward function that generalizes across multiple tasks . While standard IRL methods are typically trained and evaluated on the same task , we want our language-conditioned reward function to produce correct behavior when presented with new tasks . Several previous works consider a multi-task scenario , such as in a Bayesian or meta-learning setting ( Li & Burdick , 2017 ; Dimitrakakis & Rothkopf , 2012 ; Choi & Kim , 2012 ) . We adopt a similar approach adapted for the language-IRL problem , and formalize the notion of a task , denoted by ξ , as an MDP , where individual tasks may not share the same state spaces , dynamics or reward functions . Each task is associated with a context cξ which is a unique identifier ( i.e . an indicator vector ) for that task . Thus , we wish to optimize the following multi-task objective , where τξ denotes expert demonstrations for that task : max θ Eξ [ Eτξ [ log pθ ( τξ , cξ ) ] ] ( 3 ) Algorithm 1 Language-Conditioned Reward Learning ( LC-RL ) 1 : Obtain expert demonstrations and language describing the goal . 2 : Initialize reward function rθ . 3 : for step t in { 1 , . . . , N } do 4 : Sample task ξ , demonstrations dξ , and language Lξ . 5 : Compute optimal q∗ ( s , a ) using q-iteration and ρ∗ ( s , a ) using the forward algorithm . 6 : Update reward rθ with the gradient ( ρdξ ( s , a ) − ρ∗ ( s , a ) ) ∇rθ ( o , a , Lξ ) 7 : end for In order to optimize this objective , we first require that all tasks share the same observation space and action space , and the reward to be a function of the observation , rather than of the state . For example , in our experiments , all observations are in the form of 32x24 images taken from simulated houses , but the state space for each house is allowed to differ ( i.e. , the houses have different layouts ) . This means the same reward can be used across all MDPs even though the state spaces differ . Second , we share the reward function across all tasks , but substitute a language command Lξ as a proxy for the context cξ , resulting in a model pθ ( τξ , Lξ ) that takes as input language , states , and actions . For computational efficiency we run stochastic gradient descent on the objective in Eqn . 3 by sampling over the set of environments on each iteration .

Summary: Paper Summary: This paper studies the inverse reinforcement learning problem for language-based navigation. Given the panorama image as its observation, language embedding as its goal, a deep neural network architecture is proposed to predict the reward function from the input observation and goal. Maximum causal entropy IRL has been adopted to learn such language-conditioned reward function. This paper used the SUNCG environment for experiments and designed two tasks (navigation and pick-and-place) for evaluation.