Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule

1 INTRODUCTION . Vision-and-language navigation ( Anderson et al. , 2018b ) is a task in which a computational model follows an instruction and performs a sequence of actions to reach the final objective . An agent is embodied in a realistic 3D environment , such as that from the Matterport 3D Simulator ( Chang et al. , 2017 ) and asked to follow an instruction . The agent observes the surrounding environment and moves around . This embodied agent receives a textual instruction to follow before execution . The success of this task is measured by how accurately and quickly the agent could reach the destination specified in the instruction . VLN is a sequential decision making problem : the embodied agent makes a decision each step considering the current observation , transition history and the initial instruction . Previous studies address this problem of VLN by building a language grounded policy which computes a distribution over all possible actions given the current state and the language instruction . In this paper , we notice there are two ways to formulate the relationship between the action and instruction . First , the action is assumed to be generated from the instruction , similarly to most of the existing approaches ( Anderson et al. , 2018b ; Ma et al. , 2019 ; Wang et al. , 2019 ; Hu et al. , 2019 ; Huang et al. , 2019 ) . This is often called a follower model ( Fried et al. , 2018 ) . We call it a discriminative approach analogous to logistic regression in binary classification . On the other hand , the action may be assumed to generate the instruction . In this case , we build a neural network to compute the distribution over all possible instructions given an action and the transition history . With this neural network , we use Bayes ’ rule to build a language-grounded policy . We call this generative approach , similarly to naı̈ve Bayes in binary classification . The generative language-grounded policy only considers what is available at each time step and chooses one of the potential actions to generate the instruction . We then apply Bayes ’ rule to obtain the posterior distribution over actions given the instruction . Despite its similarity to the speaker model of Fried et al . ( 2018 ) , there is a stark difference that the speaker model of Fried et al . ( 2018 ) can not be used for navigation on its own due to its formulation , while our generative languagegrounded policy can be used for it by its own . The speaker model of Fried et al . ( 2018 ) takes as input the entire sequence of actions and predicts the entire instruction , which is not the case in ours . Given these discriminative and generative parameterizations of the language-grounded policy , we hypothesize that the generative parameterization works better than discriminative parameterization does , because the former benefits from richer learning signal arising from scoring the entire instruction rather than predicting a single action . Such rich learning signal arises , because the generative policy must learn to associate all salient features of a language instruction with an intended action , in order to learn the distribution over the language instructions . This is unlike the discriminative policy which may rely on only a minimal subset of salient features of the language instruction in order to model the distribution over a much smaller set of actions . Furthermore , the generative policy enables us to more readily encode our prior about the action distribution when it deemed necessary . We empirically show that indeed the proposed generative approach outperforms the discriminative approach in both the R2R and R4R datasets , especially in the unseen environments . Figure 1 illustrates the proposed generative approach on VLN . Furthermore , we show that the combination of the generative and discriminative policies results in near state-of-the art results in R2R and R4R , demonstrating that they capture two different aspects of VLN . We demonstrate that the proposed generative policy is more interpretable than the conventional discriminative policy , by introducing a token-level prediction entropy as a way to measure the influence of each token in the instruction on the policy ’ s decision . The source code is available at https : //github.com/shuheikurita/glgp . 2 DISCRIMINATIVE AND GENERATIVE PARAMETERIZATIONS OF LANGUAGE-GROUNDED POLICY . Vision-and-language navigation ( VLN ) is a sequential decision making task , where an agent performs a series of actions based on the initially-given instruction , visual features , and past actions . Given the instructionX , past and current observations s : t and past actions a : t−1 , the agent computes the distribution p ( at|X , s : t , a : t−1 ) at time t. For brevity , we write the current state that consists of the current and past scene observations , and past actions as ht = { s : t , a : t−1 } , and the next action prediction as p ( at|X , ht ) . The instruction X is a sequence of tokens X = ( w0 , w1 , ... , wk , ... ) . The relationship between these notations are also presented in Appendix B . In VLN , the goal is to model p ( at|ht , X ) so as to maximize the success rate of reaching the goal while faithfully following the instruction X . In doing so , there are two approaches : generative and discriminative , analogous to solving classification with either logistic regression or naive Bayes . In the discriminative approach , we build a neural network to directly estimate p ( at|ht , X ) . This neural network takes as input the current state ht and the language instruction X and outputs a distribution over the action set . Learning corresponds to max θ N∑ n=1 Tn∑ t=1 log p ( ant |hnt , Xn ) , ( 1 ) where N is the number of training trajectories . In the generative approach , on the other hand , we first rewrite the action distribution as p ( at|ht , X ) = p ( X|at , ht ) p′ ( at|ht ) ∑ a′t∈A p ( X|a′t , ht ) p′ ( a′t|ht ) = p ( X|at , ht ) ∑ a′t∈A p ( X|a′t , ht ) , ( 2 ) assuming p′ ( at|ht ) = 1/|A| , where A is the action set . This assumption implies that the action is independent of the state without the language instruction , which is a reasonable assumption as the goal is specified using the instructionX . p ( X|at , ht ) = Πkp ( wk|at , ht , w : k−1 ) is a language model conditioned on an action at and the current hidden state ht , and outputs the distribution over all possible sequences of vocabulary tokens . Learning is then equivalent to solving max θ N∑ n=1 Tn∑ t=1 ( log p ( Xn|ant , hnt ) − log ∑ a′nt ∈A p ( Xn|a′nt , hnt ) ) . ( 3 ) log p ( Xn|ant , hnt ) is the language model loss conditioned on the reference action ant , while the second term log ∑ a′t∈A p ( Xn|a′nt , hnt ) penalizes all the actions . Both terms of Eq . 3 are critical for learning the generative language-grounded policy . When we train the model only with the language model term log p ( Xn|ant , hnt ) of Eq . 3 , the resulting neural network may not learn how to distinguish different actions rather than simply focusing on generating the instruction from the state observation . For navigation , we use the model to capture the probability of the instruction conditioned on each action at ∈ A . The agent takes the action that maximizes the probability of generating the instruction : arg maxat p ( X|at , ht ) . In other words , the language-conditional generative policy has a language model inside and navigates the environment by choosing an action that maximizes the probability of the entire instruction . 3 RELATED WORK . While most of previous studies ( Anderson et al. , 2018b ; Ma et al. , 2019 ; Wang et al. , 2019 ; Li et al. , 2019 ; Hao et al. , 2020 ) have relied on the discriminative approach p ( at|X , ht ) , a few of previous studies ( Fried et al. , 2018 ; Tan et al. , 2019 ; Ke et al. , 2019 ) have proposed the so-called speaker model which scores the instruction against the entire trajectory . Such speaker models are mainly used for two purposes ; ( i ) data augmentation with automatically generated trajectories ( Fried et al. , 2018 ; Tan et al. , 2019 ) and ( ii ) reranking the complete trajectories in beam decoding ( Fried et al. , 2018 ; Tan et al. , 2019 ; Ke et al. , 2019 ) . They however have not been used for selecting local actions directly in either training or decoding . To the best of our knowledge , this paper is the first work that propose a standalone generative language-grounded policy for vision-and-language-navigation , that does not need the full state-action sequence nor to look ahead into the next state , before taking the action at each step . Some of the previous studies ( Thomason et al. , 2019a ; Hu et al. , 2019 ) discuss the ablation studies from the multimodal baselines . These studies suggest there are some action biases in the environments . Although it is possible to model these action biases in the action prior of Eq . 2 from the training environment , we choose not to do so in order to avoid overfitting our policy to the training environments . If we know the target environment beforehand , the engineering on the action prior is possibly effective . Inspired by the the success of the embodied navigation datasets ( Wu et al. , 2018 ; Chang et al. , 2017 ; Chen et al. , 2019 ) , new experimental settings and navigation tasks in realistic 3D modeling have been proposed , such as dialog-based navigation tasks which include vision-and-dialog navigation ( Thomason et al. , 2019b ) , vision-based navigation withlanguage-based assistanc ( Nguyen et al. , 2019 ) , and HANNA ( Nguyen & Daumé III , 2019 ) . Embodied question answering ( Das et al. , 2018 ; Wijmans et al. , 2019 ) , interactive visual question answering ( Gordon et al. , 2018 ) and ALFRED ( Shridhar et al. , 2020 ) for the navigation and object interaction are quite interesting task variants . The proposed generative language-grounded policy is applicable to these tasks where an agent solves a problem by following an instruction or having a conversation with another agent . 4 EXPERIMENTAL SETTINGS . 4.1 DATASETS . We conduct our experiments on the R2R navigation task ( Anderson et al. , 2018b ) , which is widely used for evaluating language-grounded navigation models and R4R ( Jain et al. , 2019 ) , which consists of longer and more complex paths when compared to R2R . R2R contains four splits of data : train , validation-seen , validation-unseen and test-unseen . From the 90 scenes of Matterport 3D modelings ( Chang et al. , 2017 ) , 61 scenes are pooled together and used as seen environments in both the training and validation-seen sets . Among the remaining scenes , 11 scenes form the validationunseen set and 18 scenes the test-unseen set . This setup tests the agent ’ s ability to navigate in unseen environments in the test phase . Some of previous studies make use of augmented datasets ( Fried et al. , 2018 ; Ma et al. , 2019 ; Tan et al. , 2019 ; Ke et al. , 2019 ) in R2R experiments . We use the same augmented dataset from Fried et al . ( 2018 ) which has been used in recent studies ( Ma et al. , 2019 ; Ke et al. , 2019 ) for comparison . R4R was created based on R2R . In R4R , paths are composed of two paths drawn from R2R , implying that each reference path in R4R is not necessarily the shortest path between the starting point and the goal point . R4R is more suitable for evaluating how closely the agent follows a given instruction that corresponds to a long and complex path . R4R consists of train , validation-seen and validationunseen sets , but does not contain the test-unseen set , unlike R2R . We provide more detailed statistics of R2R and R4R in Appendix C .

Summary: The paper focuses on learning a navigation policy for a vision-and-language navigation problem. In this problem, the agent are given a language instruction and are asked to follow the instruction to navigation in a simulated 3D room. Unlike baselines which maximize the probability of selecting an action given an instruction, the authors proposed to apply the Bayes rule to maximize the probability of generating the instruction given an action. The authors claim that this gives better generalization in unseen environments.