Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule
Authors: Shuhei Kurita
Kyunghyun Cho
Venue: ICLR 2021 Poster (2021)
Keywords: vision-and-language-navigation

Abstract: Vision-and-language navigation (VLN) is a task in which an agent is embodied in a realistic 3D environment and follows an instruction to reach the goal node. While most of the previous studies have built and investigated a discriminative approach, we notice that there are in fact two possible approaches to building such a VLN agent: discriminative and generative. In this paper, we design and investigate a generative language-grounded policy which uses a language model to compute the distribution over all possible instructions i.e. all possible sequences of vocabulary tokens given action and the transition history. In experiments, we show that the proposed generative approach outperforms the discriminative approach in the Room-2-Room (R2R) and Room-4-Room (R4R) datasets, especially in the unseen environments. We further show that the combination of the generative and discriminative policies achieves close to the state-of-the art results in the R2R dataset, demonstrating that the generative and discriminative policies capture the different aspects of VLN.

Published as a conference paper at ICLR 2021
GENERATIVE LANGUAGE-GROUNDER POLICY IN VISION BAIIN-AND-LANNUAEE NAVIGATION WITH BAYES' RULE
Shuhei Kurita
Kyunghyun Cho
AIP. RIKEN
Courant Institute. New York University Center for Data Science, New York University
PRESTO, JST
shuhei kuri Rauritariken
CIFAR Fellow kyunghyun. ett.yyuu..cccnuu.eeeeo.m
ABSTRACT
Vision-and-language navigation (VLN) is a task which an agent is embodied in realistic 3D environment and follows an instruction to reach the goal node. While most of the previous studies have built and investigated a discriminative approach, we notice that there are in fact two possible approaches to building such a VLN agent: discriminative and generative. In this paper, we design and investigate generative
grouunddd policy which uses a language model compute the distribution over all possible instructions i.e. all possible sequences of vocabulary tokens given action and the transition history. experiments, show that the proposed generative approach outperforms the discriminative approach in the Room-2- (R2R) and 4oom- (R4R) datasets, especially in the unseen environments. We further show that the combination of the generative and discriminative policies achieves close to the state-of-the art results in the R2R dataset, demonstrating that the generative and discriminative policies capture the different aspects of VLN.
INTRODUCTION
Vision-and- Vision-and-lannguaee navigation (Anderson et al., 2018b) is a task in which computational model follows instruction and performs sequence of actions reach the final objective. An agent embodied a realistic 3D environment, such as that from the Matterport 3D Simulator (Chang al. 2017) and asked to follow instruction. The agent observes the surrounding environment and moves around. This embodied agent receives a textual instruction to follow before execution. The success of this task is measured by how accurately and quickly the agent could reach the destination specified in the instruction. VLN is a sequential decision making problem: the embodied agent makes decision each step considering the current observation, transition history and the initial instruction.
Previous studies address this problem of VLN by building a language grounded policy which com- putes a distribution over all possible actions given the current state and the language instruction. In this paper, we notice there are two ways formulate the relationship between the action and instruction. First, the action is assumed to be generated from the instruction, similarly to most of the existing approaches (Anderson et al. 2018b; Ma et al. 2019 Wang et al. 2019 Hu et al.) 2019 Huang et al. 2019). This is often called follower model (Fried et al.) 2018). We call it discriminative approach analogous to logistic regression in binary classification.
On the other hand, the action may be assumed to generate the instruction. In this case, we build a neural network to compute the distribution over all possible instructions given an action and the transition history. With this neural network, We call this generative approach, similarly to naive Bayes in binary classification.
Bayes' rule to build a manguage-grounder policy.
The generative language- unguuage-rrondee policy only considers what is available at each time step and chooses one of the potential actions to generate the instruction. We then apply Bayes' rule to obtain the posterior distribution over actions given the instruction. Despite its similarity to the speaker

---

Published as a conference paper at ICLR 2021
Visual observations past actions
Next actions
For each action, score the instruction
Walk between the the tub and bathroom counter. Go through the opening next to the bathroom counter and into the little hallway. Wait next to the family picture hanging the left wall.
Language model
Agent
Figure 1: The generative soauuagg-grouded policy for vision-and and-languago navigation.
model of Fried et al. (2018), there is a stark difference that the speaker model of Fried et al. (2018) cannot be used for navigation on its own due to its formulation, while our generative language- grounded policy can be used for it by its own. The speaker model of Fried et al. (2018) takes as input the entire sequence of actions and predicts the entire instruction, which is not the case in ours.
Given these discriminative and generative parameterizations of the language-grounded policy, hypothesize that the generative parameterization works better than discriminative parameterization does, because the former benefits from richer learning signal arising from scoring the entire instruc- tion rather than predicting a single action. Such rich learning signal arises, because the generative policy must learn to associate all salient features of language instruction with intended action, in order to learn the distribution over the language instructions. This is unlike the discriminative policy which may rely on only a minimal subset of salient features of the language instruction or- der to model the distribution over a much smaller set of actions. Furthermore, the generative policy enables us to more readily encode our prior about the action distribution when it deemed necessary.
We empirically show that indeed the proposed generative approach outperforms the discriminative approach both the R2R and R4R datasets, especially in the unseen environments. Figure D iillus- trates the proposed generative approach on VL.. Furthermore, we show that the combination of the generative and discriminative policies results near state-of-the art results R2R and R4R, demon- strating that they capture two different aspects of VLN. We demonstrate that the proposed generative policy is more interpretable than the conventional discriminative policy, by introducing a token-level prediction entropy way to measure the influence of each token in the instruction on the policy's decision. The source code is available at /https::: / com/ shuhe kur kurita/qlgg
2 DISCRIMINATIVE AND GENERATIVE PARAMETERIZAATIONS OF SANGUAGE-GROUNDEE POLICY
Vision-and-langunn langugg navigation (VLN) is sequential decision making task, where agent per- forms a series of actions based on the mitially-giivnn instruction, visual features, and past actions. Given the instruction X, past and current observations 8,, and past actions d:t-1, the agent computes the distribution p(a,|X S:t; d at time For brevity, we write the current state that consists of the current and past scene observations, and past actions h4 {s.,, a=-1}, and the next action prediction as p(a, X, h.). The instruction is sequence of tokens X (wy, W1, ..., Wk.... The relationship between these notations are also presented in Appendix B
In VLN, the goal is to model p(a+/,,, X) so as to maximize the success rate of reaching the goal while faithfully following the instruction X. In doing so, there two approaches: generative and discriminative, analogous to solving classification with either logistic regression or naive Bayes.
In the discriminative approach, build a neural network directly estimate p(ay/he, X). This neural network takes as input the current state ht and the language instruction X and outputs a distribution over the action set. Learning corresponds to
N Te
max
log ((, (M.X")
(1)
where N is the number of training trajectories.
2

---

Published as a conference paper at ICLR 2021
In the generative approach, on the other hand, first rewrite the action distribution as
p( p(X ht)p (aa/he) p(a+ X) EatEAP(X(od, hy) (a/)h)) ExteaP(X(ai,,y)
p(X p(Xlas, ha)
(2)
assuming p' (a(g 1/1,A1, where A is the action This assumption implies that the action is independent the state without the language instruction, which is a reasonable assumption as the goal is specified using the instruction X. p(X (a, h2) H&pp(ry, la he; w 1) is language model
conditioned on an action and the current hidden state lt, and outputs the distribution over all possible sequences of vocabulary tokens.
Learning is then equivalent to solving
M+) log E ot ET
p(X"|ai"
max
is the language model loss conditioned on the reference action af', while the second term log Ex; EA EAX [a"" hr)) penalizes all the actions. Both terms of Eq. are critical for learning the generative language- grounded policy. When we train the model only with the language model term log p( Ia" h² ) of E the resulting neural network may not learn how to distinguish different actions rather than simply focusing on generating the instruction from the state observation.
logp(X"/aa" h ,
For navigation, we use the model capture the probability of the instruction conditioned on each ac- tion a, E A. The agent takes the action that maximizes the probability of generating the instruction: arg maxa, p(X Jat, ht). other words, the aanguag--conittions generative policy has a language model inside and navigates the environment by choosing action that maximizes the probability of the entire instruction.
3 RELATED WORK
While most of previous studies (Anderson et al., 2018b Ma et al. 2019 Wang et al. 2019 Li et al, 2019 Hao al., (2020) have relied on the discriminative approach p(a; X, h4), a few of previous studies (Fried et al. 2018 Tan et al. 2019; Ke et al., 2019) have proposed the so-called speaker model which scores the instruction against the entire trajectory. Such speaker models are mainly used for two purposes: (i) data augmentation with automatically generated trajectories (Fried et al.)
2018 Tan et al. 2019) and (ii) reranking the complete trajectories in beam decoding (Fried et al, 2018 Tan et al. 2019 Ke al. 2019). They however have not been used for selecting local actions directly in either training or decoding. To the best of our knowledge, this paper is the first work that propose standalone generative
policy for vision-and-land languand aaaguuagggiigationn that does not need the full state-action sequence nor to look ahead into the next state, before taking the action each step.
Some of the previous studies (Thomason et al. 2019a Hu et al. 2019) discuss the ablation studies from the multimodal baselines. These studies suggest there are some action biases the environ- ments. Although it is possible to model these action biases in the action prior of Eq. 2 from the training environment, we choose environments. If we know the target environment beforehand, the engineering on the action prior is possibly effective.
do in order to avoid overfitting our policy the training
Inspired by the the success of the embodied navigation datasets (Wu et al. 2018 Chang et al. 2017 Chen et al., 2019). new experimental settings and navigation tasks in realistic 3D modeling have been proposed, such as dialog-based navigation tasks which include vision-and-dian igation Thomaaoon et al. 2019b),, vision-based navigation withlanguage-basee assistanc (Nguyen) et al. 2019). and HANNA (Nguyen & Daumé III 2019). Embodied question answering (Das al.) 2018 Wijmans et al.] 2019). interactive visual question answering (Gordon et al. 2018) and AL- FRED (Shridhar al. 2020 for the navigation and object interaction quite interesting task grounded policy is applicable to these tasks where an agent solves problem by following instruction or having conversation with another agent.
nav-
variants. The proposed generative
3

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTAL SETTINGS
4.1 DATASETS
We conduct our experiments on the R2R navigation task (Anderson et al., 2018b), which is widely used for evaluating language-grounded navigation models and R4R (Jain al., 2019), which con- sists of longer and more complex paths when compared to R2R. R2R contains four splits of data: train, validation-sen -eenn validation- unseen and test-unseen. From the 90 scenes of Matterport 3D mod- elings (Chang et al.) 2017), 61 scenes are pooled together and used the training and validation-seen sets. Among the remaining scenes, 11 scenes form the validation- unseen and 18 scenes the test-unseen set. This setup tests the agent's ability to navigate in unseen environments in the test phase. Some of previous studies make use of augmented datasets (Fried et al. 2018 Ma et al. 2019 Tan al. 2019 Ke al. 2019) in R2R experiments. We use the same augmented dataset from Fried et al. (2018) which has been used in recent studies (Ma et al., 2019
environments in both
Ke et al., 2019) for comparison.
R4R was created based on R2R. In R4R, paths are composed of two paths drawn from R2R, implying that each reference path in R4R is not necessarily the shortest path between the starting point and the goal point. R4R is more suitable for evaluating how closely the agent follows given instruction that corresponds to long and complex path. R4R consists of train, validation-seen and validation- unseen sets, but does not contain the test-unseen set, unlike R2R. We provide more detailed statistics of R2R and R4R in Appendix
4.2 NEURAL NETWORK MODELS
We use the network architecture of the speaker from (Fried et al., 2018) to implement generative policies which include a language model ( a,, h2). We also use the follower network architec- ture by FFied al. 2018) for implementing discriminative policies. We follow Fried al. (2018)
and create the embedding of the next action by concatenating the 4-dimensional orientation feature [sin Φ; cos sin 0: cos 0) and the image feature extracted from pretrained ResNet (He al. 2016),
where e and 0 are the heading and elevation angles, respectively. Both generative and discriminative models use the panoramic view and action embedding, following Fried et al.) (2018). The generative policy scores an instruction based on the embedding of each of the next possible actions and the state representation which is also used by the discriminative policy.
Navigation
In all our experiments, a single agent navigates in an environment only once given single instruction, for each task, because it is unrealistic to have multiple agents simultaneously navigating in an indoor, hosehold environment. This implies that we do not use beam search nor pre-exploration in unseen environments. See Anderson al. (2018)) for more discussion on the condition and evaluation of the navigation task.
4.3 TRAINING
R2R We first train a language model that predict an instruction from the entire trajectory in the same way as Fried et al. (2018) from the dataset. We finetune each policy using imitation learning, where we let the policy navigate the environment and give the action that leads the shortest path at following(Anderson we start training a policy with both the augmented and original training sets, and then switches to using the original training alone.
each time step supervision, closely following Anderson et al. (2018b). Just like Fried et al. (2018),
R4R we first train a language model for the generative policy from the R4R dataset. Since there are more than 10 times more training instances in R4R than in R2R, we do not augment data. Unlike in R2R, we test both learning strategies; supervised learning and fidelity-oriented learning. In the case of supervised learning, we train both our generative and discriminative policies to maximize the log-pro bog-rrobability of the correct action from the training set (Fried et al., 2018). On the other hand, fidelity-oriente heuristics are used to determine the correct next action based on the proximity of the current state to the reference trajectory each time step. We describe fidelity-oriented learning in Appendix D.1 D.1
learning is particular instantiation of imitation learning, in which set of
4

---

Published as a conference paper at ICLR 2021
Validation (Seen)
Validation (Unseen)
Model
PLJ
NEL
SKT
SPLt
CLSt
nDTW? SDTW"
PLL
NEL
SRT
SPLt
CLST
nDTWT SDTWT
Dise.
10,69
5.40
0.519
0.482
0.619
0.588
0.445
12.88
6.52
0.380
0.335
0.488
0.458
0.304
+Aug.(A)
10.60
0.445
6.22
0.411
0.528
0.496
Gen.
11.23
5,53
0.481
0.451
0.625
0.579
0.427
12.98
6.17
0.434
0.371
0.514
0.478
0.344
Gen. +Aug. (B)
11.45
4.78
0.563
0.531
0.664
0.630
0.505
13.92
4.78
0.476
0.405
0.539
0.503
0.379
(on.+Disc.(A+B))
10.18
4.67
0.568
0.540
0.680
0.640
0,510
12.06
5.42
0 0.489
0.437
0.570
0.533
0.403
(A+++
11.30
4.58
0.575
0.541
0.678
0.636
0.509
5.19
0.518
0.439
0.564
0.397
Performance of generative policies (Gen.) and discriminative policies (Disc.) on the R2R dataset. +Aug. represents policies trained with the augmented dataset by Fried et al. (2018). We use (A) Disc.+Aug. and (B) Gug.+Aug. for the combination of the generative and discriminative policies of (A+B). * represents the use of backtracking of FAST. Bold fonts are used for the best result as a single model in major metrics.
Table 1:
4.4 TRAINING DETAILS FOR BOTH R2R AND R4R DATASETS
We use the same neural network architecture with Fried et al. 2018). We use the minibatch-size of 25. We use single NVIDIA V100 GPU for training. We use the validation-unseed dataset to select hyperparameters!
We use the mixture of supervised learning and imitation learning (Tan et al. 2019 Li et al. 2019) for both the generative and discriminative policies, which are referred as teacher-forcing and student- forcing (Anderson et al. 2018b). In particular, during training between the reference action a and a sampled action , we select the next action by
a da + (1 - 8)a"
(4)
where 8 ~ Bernoulli(7)) following Li et al. 2019). We examine 10,1/5, using the validation set and choose = 1/3.
After both generative and discriminative policies are trained separately, they are combined by arg max log h2) + (1 3)g (og
to jointly navigate in the greedy experimental setting in the R2R dataset. Here E [0, is hyperparameter, although our generative model is able to navigate on itself unlike the speaker model by Fried et al. (2018). 8 is determined after the training of both generative and discriminative policies with the same manner. In our experiment, we report the score of B = 0.5.
FAST (Ke et al. 2019) is framework of back-tracking to visited nodes. For single-agent back- tracking, (Sho()h adapts a simple heuristic to continue navigation from one of the previously visited nodes. This back-tracking triggered when the agent choose visit the same node for the second time. Simple heuristic scoring of the sum of the transition logits is used to choose the returning node. We use this mechanism of fack-tracking in the validation and test phase. We use the negative inverse of the logits to determine the node to continue from each time back mock-tracking is triggered. All movements in mack-tracking are counted in the agent trajectory and penalized in the evaluation.
4.5 EVALUATION METRICS
We use the following four metrics that have been commonly used to assess a policy in R2R: path length (PL) of the entire trajectory, navigation enror (NE), success rate (SR) and success weighted by path length (SPL). Among those evaluation metrics, we consider SR and SPL as primary ones for R2R because they derived from the number of successes trials in the navigation. We report CLS (Jain et al. 2019), nDTTW and SDTW (Ilharco et al.] 2019) in addition to the four metrics above, as these additional metrics are better suited to longer and more complex paths. These three metrics are based on the distance between the policy's 's trajectory and the reference path. Following Zhu al. (2020b), we use CLS and SDTW as primary metrics for R4R. We avoid using SPL for the R4R evaluation because it is not suitable for R4R performance comparisons (Jain et al. 2019) See Appendix E for the detailed description of each metric.
For more details of training and evaluations, we closely follow the publicly available code https: gi github. htthubb.com/rnngaaaggg.uuu speaker.eeeeerrreee follower of Fried
al. (2018).
5

---

Published as a conference paper at ICLR 2021
Validation (Seen)
Validation (Unseen)
Test (Unseen) NEL
Model
PL4
NEL
SRT
SPLT
PL4
NEJ
SRT
SPLt
PL
SRT
SPL
Random
9.58
9.45
0.16
9.77
9.23
0.16
9.93
9.77
0.13
0.12
Seq2ieq
0.39
RPA
5.56
0.43
7.65
0.25
9.15
7.53
0.25
0.23
Speaker-Follower
3.36
0.66
6.62
0.35
14.82
6.62
0.35
0.28
Self Mol--Monioring
18.04
5.67
0.48
0.35
RCM-SIL (train)
10.65
3.53
0.75
0.67
11.46
6.09
0.50
0.42
11.97
6.12
0.43
0.38
EnvDrop
3.99
0.62
0.59
10:70
5.22
0.52
0.48
11.66
5.23
0.51
0.47
FAST
21.17
4.97
0.56
0.43
22.08
5.14
0.54
0.41
PRESS
0.58
5.28
10.77
Ben.+Disc, Policy
10.18
4.67
0.57
0.54
12.06
5.42
0.49
0.44
11.90
5.52
0.51
0.46
Gen.+Disc. Policy'
11.30
4.58
0.57
0.54
14.65
5.19
0.52
0.44
14.31
5.24
0.54
0.46
Human
11.90
1.61
0.86
0.76
Table Comparison of baselines and the proposed policy under single run experimental setting on the R2R dataset. Bold fonts for the first and second best values in SR and SPL.. represents the of backtracking of FAST.
RESULTS
We use R2R as the main task to investigate the efficacy and analyze the behaviour of the proposed language-groundet generative policy and its relative performance against existing approaches, as R2R has been studied more extensively than R4R has. We thus present the result on R2R first, and then finish the section with the result on R4R.
5.1 GENERATIVE VS. DISCRIMINATIVE POLICIES
Table shows the performances of the generative language-grounded policy (Generative Policy) and discriminative policy (Discriminative Policy) in the R2R dataset. We show the result with and without data augmentation. All the policies were trained with stochastic mixture of supervised learning and imitation learning, resulting in better performance than those reported by Fried et al. 2018). even in the case of the discriminative baseline.
The first observation we make is that data augmentation has bigger effect on the validation aalidation-unseen unseen split than on the validation-seen split. This suggests that the main effect of data augmentation is to make a policy more robust to the changes in environments by discouraging the policy from overfit- ting to environments that are seen during training. This effect is observed with both discriminative and generative policies. However, we consider the discriminative policies are easy to overfit to seen environments in the training time especially without the augmentated dataset. In the validation- unseen split, the generative policy always performs better than the discriminative one in both SR and SPI..
Second, when data augmentation was used, the proposed generative policy outperforms the discrim- inative policy in both validation- and validation- unneen splits. This is particularly true with the primary metrics, SR and SPL. The path length (PL) is generally longer with the generative policy, but the difference is within meter on average.
Finally, the best performing approach is the combination of the discriminative and generative poli- cies (both trained with data augmentation). This clearly indicates that these two approaches are capturing two different aspects of visual language navigation. Back-tracking further improves this hybrid policy in terms of SR, although the improvement in SPL is minimal, back-tracking intro- duces extra transitions.
In CLS, nDTW and SDTW. the generative policy achieves higher performance than the discrimina- tive policy does, which suggests that the proposed generative policy follows the reference path more closely compared the discriminative one. We conjecture this is because the generative policy is sensitive to the language instructions by construction.
5.2 COMPARISON AGAINST BASELINES
Table 2 lists the performances in the validation-seen, validation caiddation-ussenn and test-unseen sets in R2R, collected from the public leaderboard and publications. We achieve near tate-of-the-art result only

---

Published as a conference paper at ICLR 2021
Seen
Unseen
Gen. Rec. Gold
0.8
0.8
Dar Prec Gold
Gen, Prec. hav.
0.6
0.6
Frec.
0.4
0.4
0.0
0.0
12
tim
time
Figure 2: The precision of actions by the generative (red) and discriminative (blue) models on the reference trajectory (dashed lines) and on navigation trajectories (solid lines). The horizontal axis the time step either on reference trajectory or on navigation trajectory. The vertical axis is the proportion that the agent chooses the shortest path action in each time step.
with the original training set and augmented dataset released by Fried et al. 2018). We compare
approach against the following previous baselines: Random (Anderson et al. (20T8)). Seq2Seq (An-
derson et al. 2018b), RPA (Wang al. 2018), Follower (Fried et al. 2018). Self-Monitoring (Ma
et al. 2019). RCM Wang et al. 2019). EnvDrop (Tan et al., 2011). FAST (Ke et al.) 2019) and
PRESS Li et al. 2019). They are described detail in Appendix F All of them, except for the
random agent, follow the discriminative approach, unlike our proposal.
In terms of SR, our model "Gen.+Dise. Policy"" performs comparably FAST which uses the same neural network by Fried et al. (2018), while our model is better in SPL. In terms of SPL, our model the second best only next to the EnvDrop model Our policy however ends up with better SR than EnvDrop does. Overall, the proposed approach is equivalent to or close to the existing state-of-the-ar models in both SR and SPL.
The recently proposed PREVALENT model (Hao et al. 2020) benefits from large scale cross-modal attention-bane based pretraining. They apply extensive data augmentation to create 6,482K image-text- action triples for pretraining, unlike the other approaches in Table Thanks to this extensive aug- mentation, they achieve SR of 0.54 and SPL of 0.51. On the other hand, we only use 178K aug- mented examples from Fried et al. 2018), widely used in previous studies (Ma al. 2019 Ke et al. 2019), for more direct comparison with previous studies. Although we have nevertheless achieved the comparable SR with an order of magnitude smaller augmented data, expect approach would further improve with this more aggressive augmentation strategy in the future.
5.3 ACTION PREDICTION ACCURACY
Figure 2 plots the precision of predicted actions over time on the validation- seen and validation- unseen sets in R2R for both the generative policy and the discriminative language- ground policies. We use the discriminative and generative policies notated as and in Table for this analysis. When the agents are presented with the gold trajectories, both policies predict actions more ac- curately than they would with their own trajectories, In real navigation, the action selection error accumulates, and prediction by both policies degrades over time. The generative policy, however, is more tolerant to such accumulated eтror than the discriminative policy is, achieving a higher preci- sion in later steps. This is especially the case in unseen environments. Additional analyses for the difference of policies are Appendix G
5.4 TOKEN- WISE PREDICTION ENTROPY
The proposed generative policy allows kens in an instruction often have significant influence on the agent's decision. For example. if an instruction ends with "..tthnn stop at the kitchen" and the agent between the kitchen and dinning room, the token "kitchen" decides when the agent predicts the "STOP" action. Since the generative language- grouggded policy relies on token-wisg wise scoring of the instruction given each action, we can directly measure how each token in the instruction affects the action prediction. We call this measure token-wise prediction entropy (TENT) and define it as
easily inspect how it uses the instruction. A few
S(ww) = Σ q(ae, wx) LogiAi g(a,, wx),
(5)
a.EAA
"Our reported SPL 0.4647 only marginally lower than 0.47 of EnvDrop.
7

---

Published as a conference paper at ICLR 2021
Validation Scen
Validation-Unseen
Figure 3: Token- wise prediction entropy (TENT) for two navigation instances from validation-sea (top) and validation-unseen unseen (bottom) sets in R2R. The vertical axis corresponds to the 1-TENT drawn at each time step & E [O], as ++ (1 S(w;)), where A = 0.05 so that one vertical-tick corresponds to 0.05. We draw multiple lines that correspond to different time steps colored from blue to red and green. We attach the panoramic views for some of the trial time steps.
where S(wk) E [0, A is the action set, and
p( Cep h4, W
g(a,, wp)
(6)
Ea,eA p( (de, has w.a-1)
When some tokens are influential to the action prediction, the entropy of scoring these tokens will be low. Otherwise, when S(wx) is close to 1, the corresponding token wy is deemed less influential for the next action prediction. We visualize highly influential tokens at each time step.
S(w)), to which we refer as 1-TENT, to identify
Figure visualizes how actions are related each token in each time step with two sample naviga- tion trajectories from the validation-seen and validation-unseen splits in R2R. We use the generative policy trained with data augmentation from Table [ Both trials end successfully within five and seven time steps, and plot five and seven curves of 1-TENT. In the early stage of the navigation (t < 3), initial tokens exhibit large 1-TENT, meaning the change actions yields a great difference in those token predictions. This tendency is observed in both seen and unseen environments. We conjecture this is a natural strategy learned by the policy to take when there is no navigation history.
In the navigation example, the agent is asked to navigate from the kitchen to the dinning room table. In the initial steps, the agent tries to go out from the kitchen, and phrases such as "right" and "walk out" have high 1-TENT. At L 3, the agent is out of the kitchen and needs to turn left the middle of the large room with high I-TENT on "left". Finally, the agent finds the dinning table and stops there with the high 1-TENT for the tokens indicating the stop point.
In the unseen navigation instance, the agent is asked to navigate from the hallway, cross the large bedroom and stop outside the carpet. In the trial, the agent first moves toward the goal node based on the keywords "bedroom" and "fireplace". It also exhibits high 1-TENT for "doorway", which is a clue for identifying the goal node. This agent, however, passes the node of the success for the first time at = 4. At 5, the agent has the high 1-TENT for both "doorway" and "rag" and then goes back the same place with I. 4. Finally, stops with the high 1-TENT for "before" and the slight 1-TENT for "rag" f 6. As have seen here, the agent has different 1-TENT values depending on the context even if it is in the same place.
Although the result of the 1-TENT visualization is similar to the attention maps (Bahdanau et al.) 2014 Vaswani et al. 2017). 1-TENT is much more directly related to the action prediction. The at- tention map represents the internal state of the neural network, while 1-TENT is based on the output of the neural network. This property makes the proposed 1-TENT powerful tool for investigating and understanding the generative language grruuddd policy.
8

---

Published as a conference paper at ICLR 2021
Validation (Seen)
Validation (Unseen)
Model
PLL
NEL
SRT
CLST nDTW? SDIW?
PL4
NEL
SRT
CLSt #DTW? SDIWT
RCM lidelitysee oriented
18.8
5.4
0.526
0.553
28.5
5.4
0.261
0.346
nDTW Adelity-orientte
0.354
0.304
0.126
BabyWalk Walk RL-RL
22.8
8.6
0.250
0.455
0.344
0.136
Baby Walk IL-RL+Cur.
23.8
7.9
0.296
0.478
0.381
0.181
Disc. supervised
20.1
7.0
0.386
0.622
0.512
0.305
20.0
9.8
0.172
0.446
0.305
0.101
ordelityorrcntedd
21.1
0.449
0.644
0.530
0.360
0.385
0.282
0.116
Gen.
supervised
19.8
8.8
0.316
0.563
0.442
0.246
19.7
9.8
0.193
0.479
0.325
0.121
fidelity
21.0
0.629
0.517
0.349
0.255
0.348
Table 3: Performance of generative and discriminative policies on the R4R validation-seen and unseen splits. We use bold-face to indicate the best models according to CLS and SDTW in the validation-unsee split.
5.5 R4R
Table 3 presents the results R4R with baseline model performance. Similarly servation on R2R, the proposed generative policy works as well as or better than the discriminative policy well as the other baselines, in terms of the primary metries which CLS and SDTW in this case especially in the validation mulidation-unseen unseen split. The generative policy trained with supervised learning outperforms all the baseline policies in CLS, while the generative policy trained with im- itation learning is close to Baby Walk trained with reinforcement learning (IL+RL) and curriculum learning Cur.)))Curr)
earlier ob-
SDTW. As both reinforcement learning and curriculum learning could also be applied to our approach, we expect to this gap completely close in the future.
As we have observed on R2R, without data augmentation, the discriminative policy works well as often better than the generative approach does R4R. the generative policy is however sig- nificantly better than the discriminative one in the validation-unseet split, confirming our conjecture that the discriminative policy tends to overfit to environments that were seen during training.
CONCLUSION
We have investigated two approaches, discriminative and generative, for the vision-and-language navigation task, and presented the generative unguagg-grounded policy which we empirically ob- served to perform better than the more widely used discriminative approach. We were able to com- bine the generative and discriminative policies and achieve the (near) state-of-the-art results for the Room-2 2-ooom and Room-4-Roo Room navigation datasets, despite the simplicity of both parameteriza- tion and learning relative to the existing baselines. Finally, we have demonstrated that the proposed generative approach is more interpretable than discriminative ones by designing token-wise pre- diction entropy.
The proposed generative parameterization, including 1-TENT visualization, is directly applicable reinforcement learning, such as Zhong et al.) (2020); (errman et al. (2017). which should be investigated in the future. The proposed generative parameterization further en- ables natural integration of large-scale language model pretraining, such as Radford et al. (2018;; Brown et al. (2020). for various ianguage-conditione an efficient way to approximate the posterior distribution in order to cope with a large action set, for instance, by importance sampling and amortized inference, for the proposed generative parameteri- zation to be more broadly applicable, in the future.
to
tasks. It is however important investigate
ACKNOWLEDGMENTS
SK was supported by ACT-L, JPMJPRI7US and PRESTO, JST JPMJPR20C. KC was supported by NSF Award 1922658 NRT-HDR: FUTURE Foundations, Translation. This work was done when SK visited New York University.

---

Published as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta. Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents. ArXiv, abs/1807.0673, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton den Hengel. Vision-and and--addda-aaa -an-Lagu Navigation: Interpreting visually-grounde navigation instructions in real environments. In Proceedings of the IEEE Con- ference Computer Vision and Puttern Recognition (CVPR), 2018b.
Peter Anderson, Ayush Shrivastava, Joanne Truong, A. Majumdar, D. Parikh, Dhruv Batra, and Stefan Lee. Sim-to-real transfer for vision-and- lan-nuuugge navigation. In Conference on Robot Learning (CoRL), 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URUhtttp://arx/woorrg ab1///9.b///9/..//////
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterportt3D; Learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV), 2017.
Howard Chen, Alane Suhr. Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency. Taylor Borg-Kirkpatrickk Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- models for sasion-and-aanguage navigation. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Misa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 3314-3325. Curran Associates, Inc., 2018. URL http:: http///apprr..e//// nips. cttppperrrppp////////pp/ 7592 apeaker follower models for vision and language navigation.
pdf
Tsu-Jui Fu, Xin Eric Wang. Matthew Peterson, Scott T. Grafton, Miguel Eckstein, and William Yang Wang. Counterfactual vision-and-language navigation via adversarial path sam- pler. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), European Conference on Computer Vision (ECCV), volume 12351 of Lecture Notes in Computer Science, pp. 71-86. Springer, 2020.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dicter Fox, and Ali Farhadi. IQA: Visual question answering in interactive environments. In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition. IEEE, June 2018. doi: 10.1109/cvpr.2118. 00430. BRLLhttp:://da
org. 10. 1109/ wwpp////ppp..../. 2018 2018.00430
Weituo Hao, Chunyuan Li. Xiujun Li. Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for vision-and and language navigation via pre-training. volume abs/2002. 2020.
URL htR./tttts://eete peatts://arxee orgg abs 2002. 10200000000000 10638
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.200
htR.:ttts://// //di:..//o/.../ org O. 11 10////9R/ 2016. 90
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer. David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world. CORR, abs/ 777.066661111 2017. http:/tt://aatt org/ abs /1t//7766... 06551
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. Are you looking? grounding multiple modalities in dision-and-languaae navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6551-6557, Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/V1/P19--165 URL https: / www. acl web org/ hwt.:ew..........gg.....o anthology/ P1 9-165
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhães. Jason Baldridge, and Eugene Ic. Transferable representation learning in vision-and- aaoon-nddlanguage navigation. In 2019 IEEE/CVF International Conference
Computer Vision, ICCV 2019, Seoul, Korea (South), Oc- tober 27 November 2, 2019, pp. 7403 7412. IEEE, 2019. doi: 10.1 10.1009/CC22220190000 https://dol: .rg /00 1109/ I CCV 201 9. 00750
URL
Gabriel Ilhareo, Vihan Jain, Alexander Ku, Eugene le, and Jason Baldridge. General evaluation for instruction conditioned navigation using dynamic time warping. In Visually Grounded Interaction and Language (ViGIL), NeurlPS 2019 Workshop. Vancouver, Canada, December 13, 2019, 2019.
URL http::::///v.iilwoosskkttts github. 1 staticsaciiaaao wttpsticcpppprsseeeee 33.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-languagee navigation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1862-1872, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. N0.18633/1//199-1B1 URL
https: / www. aclweb. orq/ anthhaogghhhhaaaaaee P1111
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correctinn via backtracking in vision-and- language navigation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. URL http::/
theevf.ecm com/ wowtent_tttttc via Cwwoon_________ 201 1aa.119pperrhtt..
2019 /Ke_ Tactica cal Rewind__set ftweiiddd_eff-foreeeette Co in_ _nViisi__ on And www.aaagggggeaaaiiiieeeeeeeeeee Naw__aaviiinn
Jacob Krantz, Erik Wijmans, A. Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In European Conference on Com- puter Vision (ECCV), 2020.
Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A. Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JCNLLP), pp. 1494-1499, Hong Kong, China, November 2019. Association for Computational Lingussics.. doi: 10.18653/v 1/D119 1159. Utttttss:://www http://wwww..lirrooo.o...ccoooo.
a web org httpg////tttoooooggooooeeee
Chih- Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitorin navigation agent via auxiliary progress estimation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htt.:ttts///opeeeeeeee
net. forum? d=r GAs jC5
A. Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, D. Parikh, and Dhruv Batra. Improv- ing bision-and-language navigation with image-text pairs from the web. In European Conference on Computer Vision (ECCV), 2020.
Khanh Nguyen and Hal Daumé III. Help, annal visual navigation with natural multimodal as- sistance via retrospective curiosity-encourrging imitation learning. In Proceedings of the 2019

---

Published as a conference paper at ICLR 2021
Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCLLP, pp. 684-695, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1/D19-0000
URL pRl.htts:: www. a c otg.bo.gg//tthoooooy/y//o
9- 1063
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2019, doi: 10.
1109/cvpp URLhttps: pittps://doet oww.oogg///oo 10. 1109/cvpr 2019.01281
Yuankai Qi, Zizheng Pan, S. Zhang, A. V. D. Hengel, and Qi Wu. Object-and-action aware model for visual language navigation. In European Conference on Computer Vision (ECCV), 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2018. URL httptttts////mccaapptts
p
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instruc- tions for Everyday Tasks. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020. URL http/:ttt///artt .rw.ivorrg/aaoo abs// 01734
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pp. 2610-2621, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/N19-1122 URL httpst 7/www. wwwwww. www.www..rgg/ant...o. httpb/orrgg/atooogy////e.
1268
Tadahiro Taniguchi, Takayuki Nagai, Tomoaki Nakamura, Naoto Iwahashi, Tetsuya Ogata, and Hideki Asoh. Symbol emergence in roboties: a survey, Advanced Robotics, 30(11-12);706-72,
2016. doi: 10. 10000006611844.2000 101.6.6622.11162.. 1164622.
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. Shifting the baseline: Single modality per- formance on visual navigation & QA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers). pp. 1977-1983, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. 10.18653/v1/119-111 1197. https: www. aclweb. org. http///nhhooooo/11999991 1197
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning 2019 (CoR12019), November 2019b. URL https://arxiv http:////xxxvv../////.soss////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http: /ttp://papppeepe/e www..ippp...... wttpp/ers..iss.ccppppeeeee/e/
atttattionniiin -ass-alllooo-ueee need. pdd
H. Wang, Qi Wu, and Chunhua Shen. Soft expert reward learning for vision-and-language naviga- tion. In European Conference on Computer Vision (ECCV), 2020a.
Hanqing Wang, Wenguan Wang. Tianmin Shu, W. Liang, and J. Shen. Active visual information gathering for vision-language navigation. In European Conference on Computer Vision (ECCV), 2020b.
12

---

Published as a conference paper at ICLR 2021
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridging model-free and masel-based reinforcement learning for planned-ahead vision- and-language navigation. In The European Conference Computer Vision (ECCV), September 2018. URL
http:///ccvll-vveasee. Took-befoor--you pdp.
Xin Wang, Qiuyuan Huang. Asli Çelikyilmaz. Jianfeng Gao, Dinghan Shen, Yuan- Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In IEEE Conference on Computer sion and Puttern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp- 6629-6638. Computer Vision Foundation IEEE, 2019. doi: 201110//CVR.2209100
URL http://openaccess/ thecvf com wwton//ooeettttooo coonten__CPR_ee_ 2019//0tl///t/// Reinforced Cro.s CactaaaoCoCooooaaaaatataaaaannnn atwpehhhhnnnnnnnnaaaeeeaaa Sewnndd-SeeSSSueeiiissse Learning for otg.://iee.aaacclaoooooooooooaar
Navigation CVPR 2019 1tppppppp_____ html
Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied Al research. CoRR, 1ttp/2088.12666 2020. /ttt:sts:
abo/ 12760
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied Question Answering in Photorealistic En- vironments with Point Cloud Perception. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint 1801. 2018.
Wang Xin, Jain Vihan, Ie Eugene, William Wang Yang, Kozareva Zornitsa. and Ravi Sujith. Environment-agnostic multitask learning for natural language grounded navigation. In European 2020.
Conference on Computer Vision (ECCV), pp.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2020. URL
Pittps:/cpenreviine net Eorummid
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-language navigation with self- supervised auxiliary reasoning tasks. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020a.
Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng. Vihan Jain, Eugene Ie, and Fei Sha. Babywalk: Going farther in vision-and-language navigation by taking baby steps. CoRR, abs/2005. 04625, 2020b.
/bg///// 04625
13

---

Published as a conference paper at ICLR 2021
A EMBODIED AI AND VISION-AND-LANGUAAGE NAVIGATION
In the R2R dataset (Anderson et al. 2018b). agent moves on a graph that was constructed from one of the realistic 3D models of houses and buildings based on Matteport 3D dataset (Chang al. 2017]. At the beginning of each trial, the agent is given textual instruction, is placed at the start node and attempts to reach the goal node by moving along the edges. At each node of the graph the agent observes the visual features of the surrounding environment and makes decision to which neighbour node it will move next. When the agent determines that the current node is sufficiently close to the destination node, it outputs "STOP", and the navigation trial ends. The agent is evaluated in terms of the accuracy of their final location and the trajectory length (Anderson et al. 2018h) a).
The difficulties in VLN mainly arise from the diversity of textual instructions. R2R provides mul- tiple instructions for each trajectory. These instructions are created via crowd srowd-sourcing, and their granularity and specificity highly vary (Li et al., 2019). The agent furthermore needs to generalize to unseen environments. Previous studies have reported that models with rich visual and textual features often overfit to the seen environments (Hu et al. 2019).
Recently, VLN becomes the central part of embodied navigation studies (Zhu et al.
2020a Wang
al., [2200a] Xin al. 2020 Qi et al. 2020 Wang et al. 2020b Krantz al. [020 Fu al. 2020
Majumdar et al. 2020). VLN is applicable to real-world robotic navigation even without the presel
mapping when is combined with the external SLAM and autonomous mobile modules (Anderson et al. 2020). Recent embodied AI environments (Savva et al. 20199] Weihs et al., (020) are also applicable for real-world robotic navigation. Symbol emergence Taniguchi et al. (2016) might be one clue to close the gap between machine learning-based embodied agents, human interactions,
and robotics.
In this paper, we propose the first approach to directly utilize the vision-conditioned language mod- cling for navigation. Our generative policy applies the vision-conditioned language modeling for scoring the instructions, and therefore policy implicitly maps the visual information the lan- guage information as visualized in Sec. [53] This implicit mapping from visual to language infor- mation with the language modeling may contribute to the performance of our generative policy, especially in the navigation of unseen conditions.
B RELATIONSHIPS OF NOTATIONS
In the formalism of the generative language- language-grounded policy, we denote the instruction X, past and current observations as 8;t and past actions as 0:--1 at time step t. Figure 4 illustrates the relationship between these notations. h4 includes the current and past observations and past actions.
C DETAILS OF R2R AND R4R DATASETS
R2R has in total 21,567 instructions which are 29 words long on average. The training set has 14,025 instructions, while the validation-seen and validation-unseen datascts have 1,020 and 2.349 instructions respectively. Each trajectory in the R2R dataset has three or four instructions. We use the released augmentation dataset in R2R. This augmentation dataset includes 178.3K trajectories with a single instruction for each. In the R4R dataset, training set, validation seen and validation unseen datasets contains 233k, 1k and 45k instructions respectively We don't 't use augmented datasets during the R4R training.
D TRAINING DETAILS OF LANGUAGE-GROUNDEE POLICIES
D.1 FIDELITY-BASED TRAINING FOR R4R DATASETS
To let the agent follow the instruction, even if the agent is out of the reference path during student- forcing learning (Anderson et al. 2018h), we introduce a simple heuristics determine the reference
We follow htops: //tt:////ttbb///////// com/ googl e research /eeerrcc///cceeleeeeeeereeeereee r and generate the R4R dataset from R2R.
14

---

Published as a conference paper at ICLR 2021
Time
t-2
t+1
Observation
%
Sr-1
Agent Moving
Prediction
Agent Moving
Predictiin
Agent Moving
Prediction
Action
Instruction
X
(given at t=
0
Figure 4: The relation for notations of the instruction X, visual scene 8t, action a, and h2 for the VLN agent.
actions. Given the reference path R (f',.... Pa..., as the sequence of reference places r; and the current agent trajectory P. (pt, ...y. Pv,, "..,,PP Pt) the sequence of visited places Pe time step f. (1) if the current place satisfies Pr E R. the reference action here is the action to follow the reference path including the stop action (2) the agent out of the reference path time step and was on the reference path at f', we choose the temporal goal place from the reference path as arg min, PL(x, R') where R' between the places and y. Here is the place the agent was last f at t'-th step. is also inferred as the same way with the footnote if R has multiple zy. The reference action here action to lead the agent to the temporal goal place in the shortest path length.
Titt-e). PL(x, y) the shortest path length
The key idea of this heuristic is that when the agent is out of the reference path, we choose the temporal goal place from the reference path. However, we disallow the agent to choose the temporal goal place which is far from the instruction and the visiting orders of the reference path.
E DETAILS OF EVALUATION METRICS
We use the following four metrics that commonly used in evaluation for R2R navigation:
Trajectory Length (TL) is the length of the agent trajectory in meters.
Navigation Error (NE) is the shortest path distance in meters from the point the agent stops the goal point.
Success Rate (SR) is the proportion of successes among all the trials. The task is successful when the agent stops within 3m from the goal point (Anderson et al., 2018b).
SPL is short for Success weighted by (normalized inverse) Path Length introduced in Anderson 201 SPL is variation of SR and is penalized by the trajectory length.
We analyze how well the trajectories followed by the proposed approach agree with the instructions using CLS (Jain et al. 2019), nDTW and SDTW (Ilharco et al. 2019). These three metrics are defined as:
CLS Coverage weighted by Length Score is the product of the path coverage of the reference path and the length score which penalize the longer or shorter trajectory length than the reference path length.
nDTW Normalized Dynamic Time Warping computes the fidelity of the trajectory given the refer- ence path.
SDTW Success weighted by normalized Dynamic Time Warping is equal to nDTW for task success cases and otherwise
We also consider the case that H has multiple Pt in the R4R training set. When an agent visits the place for the n-th time and At include m times of visiting, we assume this is the m -th visiting of æs on R and m' if n. <, otherwise m' We assume that the agent is at m'-th step of the reference path (De ).
15

---

Published as a conference paper at ICLR 2021
In the R2R dataset, each instruction is based the shortest path (Anderson et al. 2018b). The trajectory paths are specified only in the instructions and therefore these metrics evaluate how closely the models follow the instructions. Suppose that there two completely different routes in the navigation: the shortest path with the instruction and a different path that result in a slightly longer path length. When agent ignores the instruction and reaches the goal different route, SPI. will be close to I because of the similar path length. However, CLS and SDTW are penalized due the completely different trajectory.
We do not use SPL for the R4R evaluation metric because SPL depends on the shortest path length from the start node the goal node (Anderson et al. 2018a) and the reference path in R4R is not the shortest path in general. In R4R, the shortest path length from the start node to the goal node can become O. To consider the fidelity-based path length, we need to re-define new SPL' based on the reference path length instead of the shortest path length for R4R. However, if we do so, such SPL' incompatible with SPI. reported in the previous paper. Therefore we don't 't use SPL for R4R performance comparisons. See Jein et al. (2019) for further discussions of SPL R4R.
F DETAILS OF BASELINE MODELS
F. R2R BASELINES
We compare our approach against the following previous baselines in R2R. All of these, except for random agent, follow the discriminative approach.
Random An agent that moves one random direction for five steps (Anderson al. 2018b).
Seq2Seq An LSTM-based
model (Anderson al. 2018b).
RPA Combination of model-free and model-based reinforcement learning with look-ahead mod-
ule [Wang et al. 2018).
Follower An agent with panoramic view and trained with data augmentation (Fried et al. 2018
Self-Monitoring An agent that integrates visual and textual matching trained with progress monitor regularizer (Ma et al. 2019).
RCM An agent that enforces cross-modal grounding of language and vision features (Wang et al. 2019).
EnvDrop An agent trained with combination of imitation learning and reinforcement learning af- ter pretraining using environmental dropout and back translation for environmental data augmentation (Tan et al. 2019).
FAST An agent that exploits the fusion score of the local action selector and the progress monitor. This agent is able to back-track to visited nodes (Ke et al. 2019).
PRESS An agent with the pretrained language encoder of BERT and the capability incorporate multiple introductions for one trajectory (Li et al. 2019). We compare our model against their model trained with single instruction.
F.2 R4R BASELINES
We compare policies with models that are trained without augmented data if they are available.
RCM An RCM agent that enforces cross-modal grounding of language and vision features reported in Jain al. (2019).
nDTW An agent that with reinforcement learning with nDTW-based rewards (Ilharco al. 2019)
Baby Walk An agent that exploits the proposed BABY-STEPs follow micro-instructions. BABY- STEPs are shorter navigation tasks and trained with other learning regimes such as imitation learning (IL), reinforcement learning (RL) and curriculum reinforcement learning (Cur.).
R4R is dataset measure the agent fidelity the given instruction. Therefore choose the fidelity-oriented agents in comparison and we develop our policies with supervised learning or fidelity-oriented training.
16

---

Published as a conference paper at ICLR 2021
Gen. and Disc. Agreement Seen
Gen. and Disc. Agreement Unseen
0.8
0.8
0.6
0.0
0.4
0.0
0.0
tim
time
Figure 5: The agreement of actions between the generative and discriminative models on shortest paths (dashed lines) and navigation trials (solid lines). The horizontal axis corresponds to the time step of trials.
G AGREEMENT OF GENERATIVE AND DISCRIMINATIVE POLICIES
We present the agreement rate of action prediction between generative and discriminative policies the bottom of Figure The agreement drops over time, which implies that these policies behaves differently from each other, capturing different aspects of VLN. The agreement become lower in later time steps and we hence consider the combination of Gen and Disc can work better than either model.
H PERFORMANCE COMPARISON ON LONGER TRAJECTORIES
We present further comparisons of the discriminative and generative policies on longer trajectories with the R4R validation unseen set. prgueetJpresensss SR, CLS, and SDTW for navigation trials with the total steps in the gold trajectories and the navigation trajectories. In the left pane, the horizontal axis is the number steps to reach the goal node following reference trajectory. As seen in Figure 3 of Jain al. (2019), most of the R4R goal trajectories have to 15 steps. The CLS and SDTW graphs suggest that our generative policy closely follows the goal trajectory compared to the discriminative policy especially when the gold trajectories are long. In the right pane, the horizontal axis the number of steps when agents stop their navigation in the trials. Although the difference between the generative policy and the discriminative policy is not large in the success rates, the generative policy achieves better in CLS and SDTW with the long navigation trajectories. This suggests that the discriminative policy doesn't follow the given instruction even though it achieves similar success rates to the generative policy.
17

---

Published as a conference paper at ICLR 2021
Metrics per gold stops
per navigation steps
6
0.2
0.2
0.1
(gold)
0.5
0.3
30
1.2
15
Lobel steps (gold)
steps Inwxigation)
0.20
0.25
0.35
0.70
0.15
0.10
0.10
0.95
10
11
12
1.3
1.
1.0
15
tobal steps (gold)
total staps (navigation)
Figure 6: We present SR (top), CLS (middle) and SDTW (bottom) with the confidence interval of 20. Crimson for generative policy and cyan for discriminative policy. Both policies are trained with fidelity-oriented manner.
18


Reviews:
[{"review_id": "zq2BtaSe2f4", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a generative speaker model that selects actions at each timestep that facilitate the generation of the instruction, and demonstrates that a combination of discriminative and generative action prediction models outperform either alone, with the generative model primarily facilitating better generalization to unseen environments.\nIt would help to have more details in the paper rather than relegated to the appendix, maybe at the expense of the long TENT analysis? I didn't get too much out of that, but would have liked to see more details of the main algorithm in the paper (e.g., details of how the discriminative/generative combo was done).\n\nImprovements:\n\n- Assumption that p(a_t|h_t) is uniform is reasonable, but might be improved by using priors from the training data since there are strong action-conditioned biases in VLN R2R [ https://arxiv.org/abs/1811.00613 Figure 2 ]; some geometric form of this likely holds for the panoramic setting as well (especially for what were formerly \"forward\" actions -> continuing along headings close to previous heading).\n\n- The (A) and (B) notation in Table 1 was super confusing and it took me a while to figure out it was just an alias for Disc + Aug versus Gen + Aug since that isn't written explicitly anywhere.\n\n- That the combination of discriminative and generative policies gives best results is a major point in the paper, but the combined policy description is relegated to the appendix which feels weird.\n\nNits:\n- typo Introduction Chang et al. is \\citet but should be \\cite\n- Possible typo after Eq (3), \"penalizes all the actions\" should this be \"penalizes all actions except a_t\", or is a_t penalized as well (e.g., global optimum is zero)?\n- 5.3 typo \"Figure 5\" -> \"Figure 2\"\n- Typo 5.5 \"as good as\" -> \"as well as\"\n- Typo 6 \"parametrization\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aXJeQQwERR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Summary**\n\nThe paper addresses the problem of vision-and-language navigation (Anderson et al., 2018). The idea of the paper is to use a generative policy where a distribution over all instruction tokens given the previous actions is computed. The agent takes the action that maximizes the probability of the current instruction. The paper reports the results on R2R and R4R datasets.\n\n**Paper Strengths**\n\n- The paper shows that the proposed generative model outperforms the discriminative formulation of the approach.\n\n- The results have been compared with a number of strong baselines.\n\n**Paper Weaknesses**\n\n- The novelty is limited compared to the speaker model of Fried et al. (2018). It is mentioned in the introduction section that Fried et al. use the entire sequence of actions while this paper uses local information at each step. I do not consider this as a large change compared to the previous work.\n\n- In section 4.3, it is mentioned that the method is initially trained using the entire trajectory similar to Fried et al. (2018). This makes it even more similar to the method of Fried et al. (2018) since it does not just use the local information as mentioned in the introduction.\n\n- Figure 3 and its description in Section 5.4 are very confusing. What are different colors in Figure3? Why is the vertical axis labelled timestep? Isn't it supposed to be 1-TENT? The caption does not make it more clear either. It would be good to clarify.\n\n- Two qualitative examples in Figure 3 are not sufficient. Quantitative analysis over the entire dataset should be provided.\n\n- The paper does not respect the page limit. Section 5.3 is about a figure that appears in the appendix.\n\n**Score Justification**\n\nThe idea of the paper is not novel. The proposed method does not provide strong results either (it is outperformed by some other methods). Therefore, I cannot justify acceptance for this paper.\n\n**Post-rebuttal comments**\n\nI read the rebuttal and other reviews. I am still not convinced that there is a big difference between this method and speaker-follower of Fried et al. The speaker model cannot be used for navigation by itself but the learned distribution is used to adjust the actions taken by the follower. The only difference between these two methods is that one uses the entire trajectory and the other one uses partial trajectories up to the current time step (pre-trained on the entire trajectory though). Also, as mentioned by another reviewer, some analysis should be provided about why the generative model works better. Due to these issues, I keep my original rating. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NoW89Zw2z9h", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on learning a navigation policy for a vision-and-language navigation problem. In this problem, the agent are given a language instruction and are asked to follow the instruction to navigation in a simulated 3D room. Unlike baselines which maximize the probability of selecting an action given an instruction, the authors proposed to apply the Bayes rule to maximize the probability of generating the instruction given an action. The authors claim that this gives better generalization in unseen environments.\n\nPros:\nWhile the idea of using a critic model (to compute the probability of the instruction given the trajectory) for VLN tasks [1,2] is not new, existing works takes the whole trajectory as input to the critic model, yet this paper takes the trajectory up until time t as input. The idea is new.\n\nCons:\nThe authors didn't explain well why the proposed generative model is better than the discriminate model. The empirical results shows so, but there aren't enough discussion to explain why.\nExperiments are somewhat less satisfying, especially when the results in Table 2 are Gen.+Disc. with backtracking instead of just Gen.+Disc.\n\nDetailed Comments:\nIt is intuitive that Gen. + Disc. works well than Gen. or Disc. alone. But why Gen. performs better than Disc.? The statement that Gen. works better than Disc. is the main statement and main contribution of this paper. However, there is very little discussion on why this is true methodologically (not empirically). Also, I'd assume that it is extremely hard to learn p(X | a_t, h_t) when t is small (at the beginning of the trajectory). I don't see analysis on how well the instructions are predicted.\n\nTable 1 shows the performance of both Gen. + Disc. and Gen. + Disc. + backtracking. However, In Table 2, the performance of Gen. + Disc. on test (unseen) is not shown. This is less satisfying. Meanwhile, in test (unseen), with backtracking, Gen. + Disc. achieves the same success rate compared with FAST (which also do backtracking), so the performance gain seems small.\n\nThe paper misses implementation details. For example, how p(X|a_t, h_t) is modeled is not shown. The authors say \"We use the network architecture of the speaker from (Fried et al., 2018) to implement generative policies which include a language model p(X|at, ht).\" But the paper is not self-contained. It is better to at least describe the model in appendix. \n\nThe paper say \"We finetune each policy using imitation learning\". However, imitation learning is a very board term, not necessarily mean the student-forcing implied by the paper. It is better to use a more concrete terms.\n\n\n[1] Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation\n[2] Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JaD5EsdLtCJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the generative modeling of the task of vision and language navigation. At each timestep, for each action, the generation of the instruction is scored. This modeling achieves the-state-of-the-art or competitive results on standard benchmarks Room-to-room (R2R) and Room-for-room (R4R). The proposed generative modeling also allows us to interpret how the model process the instruction. To do so, token-wise prediction entropy (1-TENT) is introduced. The core idea is that if a token is critical for the trajectory, the entropy of scoring the token for all actions will below. Thus by visualizing the 1-TENT for each timestep we can analyze how and when a model fails or what kind of capabilities are missing in the model. Below I list my questions (Q) and suggestions (S):\n\nS1 Introduction 4th paragraph: It was rather hard to understand what's the difference between Fried et. al. and this work. Please try to simplify or add a figure or give examples\n\nS2 Introduction penultimate paragraph: Please clarify the concept of 'richer learning signal'. This paragraph is rather more cryptical compared to the rest of the paper.\n\nS3 Figure2: please add a sentence or two to clarify how this is generated and the punchline of the caption. Also, add the legend for x axis.\n\nS4 Section 5.3: I believe you are referring to Figure 2 not Figure 5 which is in the appendix.\n\nS5 Figure 3: You are trying to achieve a lot with two analyses. Please move one of them to the appendix -- which will give more space to panorama images. Also, the figure is already hard to interpret without the guidance for the level of 1-TENT scores for tokens. \n\nQ1: Section1: The assumption that the action probabilities are uniform given the state makes sense. However, in reality, it's not true. Have you done any experiments on this or any ideas on how to address that?\n\nQ2: Section 4: This might be a lot to ask but is it possible to have results on TouchDown or any other outdoor navigation datasets? Problems are structurally the same however the way people give instructions in outdoor scenes will be different. It would be great to see the generalization of the generative approach to outdoor scenes.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to take a token-level generative approach to the task of vision-language navigation (R2R/R4R). The reviewers raise a number of concerns which should be noted in the final version of this work.  The primary concern revolves around generality.  How will this approach generalize to more sophisticated generative and discriminative models?  To what extent is the model relying on the short instruction/action sequences to succeed and would not perform well on longer instructions, longer trajectories, or more abstract language.  Finally, the discussion of the uninformed prior is interesting because while "clean", reviewers note there is no realistic grounded language scenario in which an uninformative prior makes sense.
Decision: Accept (Poster)