Venue: NAACL 2019

# Learning to Navigate Unseen Environments:

Back Translation with Environmental Dropout

 Hao Tan Licheng Yu Mohit Bansal

UNC Chapel Hill

{haotan, licheng, mbansal}@cs.unc.edu

###### Abstract

A grand goal in AI is to build a robot that can accurately navigate based on natural language instructions, which requires the agent to perceive the scene, understand and ground language, and act in the real-world environment. One key challenge here is to learn to navigate in new environments that are unseen during training. Most of the existing approaches perform dramatically worse in unseen environments as compared to seen ones. In this paper, we present a generalizable navigational agent. Our agent is trained in two stages. The first stage is training via mixed imitation and reinforcement learning, combining the benefits from both off-policy and on-policy optimization. The second stage is fine-tuning via newly-introduced 'unseen' triplets (environment, path, instruction). To generate these unseen triplets, we propose a simple but effective 'environmental dropout' method to mimic unseen environments, which overcomes the problem of limited seen environment variability. Next, we apply semi-supervised learning (via back-translation) on these dropped-out environments to generate new paths and instructions. Empirically, we show that our agent is substantially better at generalizability when fine-tuned with these triplets, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.1

Footnote 1: Our code, data, and models publicly available at: [https://github.com/airsplay/R2R-EnvDrop](https://github.com/airsplay/R2R-EnvDrop)

## 1 Introduction

One of the important goals in AI is to develop a robot/agent that can understand instructions from humans and perform actions in complex environments. In order to do so, such a robot is required to perceive the surrounding scene, understand our spoken language, and act in a real-world house. Recent years have witnessed various types of embodied action based NLP tasks being proposed Correa et al. (2010); Walters et al. (2007); Hayashi et al. (2007); Zhu et al. (2017); Das et al. (2018); Anderson et al. (2018).

In this paper, we address the task of instruction-guided navigation, where the agent seeks a route from a start viewpoint to an end viewpoint based on a given natural language instruction in a given environment, as shown in Fig. 1. The navigation simulator we use is the recent Room-to-Room (R2R) simulator Anderson et al. (2018), which uses real images from the Matterport3D Chang et al. (2017) indoor home environments and collects complex navigable human-spoken instructions inside the environments, hence connecting problems in vision, language, and robotics. The instruction in Fig. 1 is _"Walk past the piano through an archway directly in front. Go through the hallway when you see the window door. Turn

Figure 1: Room-to-Room Task. The agent is given an instruction, then starts its navigation from some staring viewpoint inside the given environment. At time \(t\), the agent selects one view (highlighted as red dotted bounding boxes) from a set of its surrounding panoramic views to step into, as an action \(a_{t}\).



right to the hanged pictures..."_. At each position (viewpoint), the agent perceives panoramic views (a set of surrounding images) and selects one of them to step into. In this challenging task, the agent is required to understand each piece of the instruction and localize key views (_"piano"_, _"hallway"_, _"door"_, etc.) for making actions at each time step. Another crucial challenge is to generalize the agent's navigation understanding capability to unseen test room environments, considering that the R2R task has substantially different unseen (test) rooms as compared to seen (trained) ones. Such generalization ability is important for developing a practical navigational robot that can operate in the wild.

Recent works [14, 30, 56, 7] have shown promising progress on this R2R task, based on speaker-follower, reinforcement learning, imitation learning, cross-modal, and look-ahead models. However, the primary issue in this task is that most models perform substantially worse in unseen environments than in seen ones, due to the lack of generalizability. Hence, in our paper, we focus on improving the agent's generalizability in unseen environments. For this, we propose a two-stage training approach. The first stage is training the agent via mixed imitation learning (IL) and reinforcement learning (RL) which combines off-policy and on-policy optimization; this significantly outperforms using IL or RL alone.

The second, more important stage is semi-supervised learning with generalization-focused 'environmental dropout'. Here, the model is fine-tuned using additional training data generated via back-translation. This is usually done based on a neural speaker model [19] that synthesizes new instructions for additional routes in the _existing_ environments. However, we found that the bottleneck for this semi-supervised learning method is the limited variability of given (seen) environments. Therefore, to overcome this, we propose to generate novel and diverse environments via a simple but effective 'environmental dropout' method based on view- and viewpoint-consistent masking of the visual features. Next, the new navigational routes are collected from these new environments, and lastly the new instructions are generated by a neural speaker on these routes, and these triplets are employed to fine-tune the model training.

Overall, our fine-tuned model based on back-translation with environmental dropout substantially outperforms the previous state-of-the-art models, and achieves the most recent rank-1 on the Vision and Language Navigation (VLN) R2R challenge leaderboard's private test data, outperforming all other entries in success rate under all evaluation setups (single run, beam search, and pre-exploration).2 We also present detailed ablation and analysis studies to explain the effectiveness of our generalization method.

Footnote 2: [https://evalai.cloudcv.org/web/challenges/challenge-page/97/overview](https://evalai.cloudcv.org/web/challenges/challenge-page/97/overview)

## 2 Related Work

Embodied Vision-and-LanguageRecent years are witnessing a resurgence of active vision. For example, Levine et al. (2016) used an end-to-end learned model to predict robotic actions from raw pixel data, Gupta et al. (2017) learned to navigate via mapping and planning, Sadeghi and Levine (2017) trained an agent to fly in simulation and show its performance in the real world, and Gandhi et al. (2017) trained a self-supervised agent to fly from examples of drones crashing. Meanwhile, in the intersection of active perception and language understanding, several tasks have been proposed, including instruction-based navigation [8, 1], target-driven navigation [30, 28], embodied question answering [10], interactive question answering [11], and task planning [30]. While these tasks are driven by different goals, they all require agents that can perceive their surroundings, understand the goal (either presented visually or in language instructions), and act in a virtual environment.

Instruction-based NavigationFor instruction-based navigation task, an agent is required to navigate from start viewpoint to end viewpoint according to some given instruction in an environment. This task has been studied by many works [19, 18, 12, 13, 14] in recent years. Among them, [13] differs from the others as it introduced a photo-realistic dataset - Room-to-Room (R2R), where all images are real ones taken by Matterport3D [3] and the instructions are also natural. In R2R 

environments, the agent's ability to perceive real-world images and understanding natural language becomes even more crucial. To solve this challenging task, a lot of works [21, 34, 50, 69] have been proposed and shown some potential. The most relevant work to us is Fried et al. (2018), which proposed to use a speaker to synthesize new instructions and implement pragmatic reasoning. However, we observe there is some performance gap between seen and unseen environments. In this paper, we focus on improving the agent's generalizability in unseen environment.

Back-translationBack translation [7], a popular semi-supervised learning method, has been well studied in neural machine translation [12, 34, 51, 13]. Given paired data of source and target sentences, the model first learns two translators - a forward translator from source to target and a backward translator from target to source. Next, it generates more source sentences using the back translator on an external target-language corpus. The generated pairs are then incorporated into the training data for fine-tuning the forward translator, which proves to improve the translation performance. Recently, this approach (also known as data augmentation) was applied to the task of instruction-based navigation [20], where the source and target sentences are replaced with instructions and routes.

## 3 Method

### Problem Setup

Navigation in the Room-to-Room task [14] requires an agent to find a route \({\bf R}\) (a sequence of viewpoints) from the start viewpoint \({\bf S}\) to the target viewpoint \({\bf T}\) according to the given instruction \({\bf I}\). The agent is put in a photo-realistic environment \({\bf E}\). At each time step \(t\), the agent's observation consists of a panoramic view and navigable viewpoints. The panoramic view \(o_{t}\) is discretized into \(36\) single views \(\{o_{t,i}\}^{36}_{i=1}\). Each single view \(o_{t,i}\) is an RGB image \(v_{t,i}\) accompanied with its orientation \((\theta_{t,i},\phi_{t,i})\), where \(\theta_{t,i}\) and \(\phi_{t,i}\) are the angles of heading and elevation, respectively. The navigable viewpoints \(\{l_{t,k}\}^{N_{t}}_{k=1}\) are the \(N_{t}\) reachable and visible locations from the current viewpoint. Each navigable viewpoint \(l_{t,k}\) is represented by the orientation \((\hat{\theta}_{t,k},\hat{\phi}_{t,k})\) from current viewpoint to the next viewpoints. The agent needs to select the moving action \(a_{t}\) from the list of navigable viewpoints \(\{l_{t,k}\}\) according to the given instruction \({\bf I}\), history/current panoramic views \(\{o_{\tau}\}^{t}_{\tau=1}\), and history actions \(\{a_{\tau}\}^{t-1}_{\tau=1}\). Following Fried et al. (2018), we concatenate the ResNet [12] feature of the RGB image and the orientation as the view feature \(f_{t,i}\):

\[f_{t,i}=\left[\right. \mathrm{ResNet}(v_{t,i});\] \[\left.\left(\cos\theta_{t,i},\sin\theta_{t,i},\cos\phi_{t,i}, \sin\phi_{t,i}\right)\right. \left.\right] \tag{1}\]

The navigable viewpoint feature \(g_{t,k}\) is extracted in the same way.

### Base Agent Model

For our base instruction-to-navigation translation agent, we implement an encoder-decoder model similar to Fried et al. (2018). The encoder is a bidirectional LSTM-RNN with an embedding layer:

\[\hat{w}_{j} =\text{embedding}(w_{j}) \tag{2}\] \[u_{1},u_{2},\cdots,u_{\rm L} =\text{Bi-LSTM}(\hat{w}_{1},\cdots,\hat{w}_{\rm L}) \tag{3}\]

where \(u_{j}\) is the \(j\)-th word in the instruction with a length of \(L\). The decoder of the agent is an attentive LSTM-RNN. At each decoding step \(t\), the agent first attends to the view features \(\{f_{t,i}\}\) computing the attentive visual feature \(\tilde{f}_{t}\):

\[\alpha_{t,i} =\operatorname*{softmax}_{i}(f_{t,i}^{\intercal}W_{\textsc{F}}\, \tilde{h}_{t-1}) \tag{4}\] \[\tilde{f}_{t} =\sum\nolimits_{i}\alpha_{t,i}f_{t,i} \tag{5}\]

The input of the decoder is the concatenation of the attentive visual feature \(\tilde{f}_{t}\) and the embedding of the previous action \(\tilde{a}_{t-1}\). The hidden output \(h_{t}\) of the LSTM is combined with the attentive instruction feature \(\tilde{u}_{t}\) to form the instruction-aware hidden output \(\tilde{h}_{t}\). The probability of moving to the \(k\)-th navigable viewpoint \(p_{t}(a_{t,k})\) is calculated as softmax of the alignment between the navigable viewpoint feature \(g_{t,k}\) and the instruction-aware hidden output \(\tilde{h}_{t}\).

\[h_{t} =\mathrm{LSTM}\left([\tilde{f}_{t};\tilde{a}_{t-1}],\tilde{h}_{ t-1}\right)\] (6) \[\beta_{t,j} =\operatorname*{softmax}_{j}\left(u_{j}^{\intercal}W_{\textsc{U} }\,h_{t}\right)\] (7) \[\tilde{u}_{t} =\sum\nolimits_{j}\beta_{t,j}\,u_{j}\] (8) \[\tilde{h}_{t} =\tanh\left(W\left[\tilde{u}_{t};\,h_{t}\right]\right)\] (9) \[p_{t}(a_{t,k}) =\operatorname*{softmax}_{k}\left(g_{t,k}^{\intercal}W_{\textsc{ G}}\,\tilde{h}_{t}\right)\] (10) 

Different from Fried et al. (2018), we take the instruction-aware hidden vector \(\tilde{h}_{t-1}\) as the hidden input of the decoder instead of \(h_{t-1}\). Thus, the information about which parts of the instruction have been attended to is accessible to the agent.

### Supervised Learning: Mixture of Imitation+Reinforcement Learning

We discuss our IL+RL supervised learning method in this section.3

Footnote 3: As opposed to semi-supervised methods in Sec. 3.4, in this section we view both imitation learning and reinforcement learning as supervised learning.

Imitation Learning (IL)In IL, an agent learns to imitate the behavior of a teacher. The teacher demonstrates a teacher action \(a^{*}_{t}\) at each time step \(t\). In the task of navigation, a teacher action \(a^{*}_{t}\) selects the next navigable viewpoint which is on the shortest route from the current viewpoint to the target \(\mathbf{T}\). The off-policy4 agent learns from this weak supervision by minimizing the negative log probability of the teacher's action \(a^{*}_{t}\). The loss of IL is as follows:

Footnote 4: According to Poole and Mackworth (2010), an off-policy learner learns the agent policy independently of the agent’s navigational actions. An on-policy learner learns the policy from the agent’s behavior including the exploration steps.

\[\mathcal{L}^{\text{IL}}=\sum_{t}\mathcal{L}^{\text{IL}}_{t}=\sum_{t}\mbox{-} \log p_{t}(a^{*}_{t}) \tag{11}\]

For exploration, we follow the IL method of Behavioral Cloning Bojarski et al. (2016), where the agent moves to the viewpoint following the teacher's action \(a^{*}_{t}\) at time step \(t\).

Reinforcement Learning (RL)Although the route induced by the teacher's actions in IL is the shortest, this selected route is not guaranteed to satisfy the instruction. Thus, the agent using IL is biased towards the teacher's actions instead of finding the correct route indicated by the instruction. To overcome these misleading actions, the on-policy reinforcement learning method Advantage Actor-Critic Mnih et al. (2016) is applied, where the agent takes a sampled action from the distribution \(\{p_{t}(a_{t,k})\}\) and learns from rewards. If the agent stops within \(3m\) around the target viewpoint \(\mathbf{T}\), a positive reward \(+3\) is assigned at the final step. Otherwise, a negative reward \(-3\) is assigned. We also apply reward shaping Wu et al. (2018): the direct reward at each non-stop step \(t\) is the change of the distance to the target viewpoint.

IL+RL MixtureTo take the advantage of both off-policy and on-policy learners, we use a method to mix IL and RL. The IL and RL agents share weights, take actions separately, and navigate two independent routes (see Fig. 2). The mixed loss is the weighted sum of \(\mathcal{L}^{\text{IL}}\) and \(\mathcal{L}^{\text{RL}}\):

\[\mathcal{L}^{\text{MIX}}=\mathcal{L}^{\text{RL}}+\lambda_{\text{IL}}\mathcal{ L}^{\text{LL}} \tag{12}\]

IL can be viewed as a language model on action sequences, which regularizes the RL training.5

Footnote 5: This approach is similar to the method ML+RL in Paulus et al. (2018) for summarization. Recently, Wang et al. (2018) combines purely supervised learning and RL training however, they use a different algorithm named MIXER Ranzato et al. (2015), which computes cross entropy (XE) losses for the first \(k\) actions and RL losses for the remaining.

### Semi-Supervised Learning: Back Translation with Environmental Dropout

#### 3.4.1 Back Translation

Suppose the primary task is to learn the mapping of \(\mathbf{X}\rightarrow\mathbf{Y}\) with paired data \(\{(\mathbf{X},\mathbf{Y})\}\) and unpaired data \(\{\mathbf{Y}^{\prime}\}\). In this case, the back translation method first trains a forward model \(P_{\mathbf{X}\rightarrow\mathbf{Y}}\) and a backward model \(P_{\mathbf{Y}\rightarrow\mathbf{X}}\), using paired data \(\{(\mathbf{X},\mathbf{Y})\}\). Next, it generates additional datum \(\mathbf{X}^{\prime}\)

Figure 2: Left: IL+RL supervised learning (stage 1). Right: Semi-supervised learning with back translation and environmental dropout (stage 2).

 

from the unpaired \({\bf Y}^{\prime}\) using the backward model \(P_{{\bf Y}-{\bf X}}\). Finally, \(({\bf X}^{\prime},{\bf Y}^{\prime})\) are paired to further fine-tune the forward model \(P_{{\bf X}-{\bf Y}}\) as additional training data (also known as 'data augmentation').

Back translation was introduced to the task of navigation in Fried et al. (2018). The forward model is a navigational agent \(P_{{\bf E},{\bf I}-{\bf R}}\) (Sec. 3.2), which navigates inside an environment \({\bf E}\), trying to find the correct route \({\bf R}\) according to the given instruction \({\bf I}\). The backward model is a _speaker_\(P_{{\bf E},{\bf R}-{\bf I}}\), which generates an instruction \({\bf I}\) from a given route \({\bf R}\) inside an environment \({\bf E}\). Our speaker model (details in Sec. 3.4.3) is an enhanced version of Fried et al. (2018), where we use a stacked bidirectional LSTM-RNN encoder with attention flow.

For back translation, the Room-to-Room dataset labels around \(7\%\) routes \(\left\{{\bf R}\right\}\) in the training environments6, so the rest of the routes \(\left\{{\bf R}^{\prime}\right\}\) are unlabeled. Hence, we generate additional instructions \({\bf I}^{\prime}\) using \(P_{{\bf E},{\bf R}-{\bf I}}\left({\bf E},{\bf R}^{\prime}\right)\), so to obtain the new triplets \(\left({\bf E},{\bf R}^{\prime},{\bf I}^{\prime}\right)\). The agent is then finetuned with this new data using the IL+RL method described in Sec. 3.3. However, note that the environment \({\bf E}\) in the new triplet \(\left({\bf E},{\bf R}^{\prime},{\bf I}^{\prime}\right)\) for semi-supervised learning is still selected from the _seen_ training environments. We demonstrate that the limited amount of environments \(\left\{{\bf E}\right\}\) is actually the bottleneck of the agent performance in Sec. 7.1 and Sec. 7.2. Thus, we introduce our environmental dropout method to mimic the "new" environment \({\bf E}^{\prime}\), as described next in Sec. 3.4.2.

Footnote 6: The number of all possible routes (shortest paths) in the \(60\) existing training environments is 190K. Of these, the Room-to-Room dataset labeled around 14K routes with one navigable instruction for each, so the amount of labeled routes is around \(7\%\) of 190K.

#### 3.4.2 Environmental Dropout

Failure of Feature DropoutDifferent from dropout on neurons to regularize neural networks, we drop raw feature dimensions (see Fig. 5(a)) to mimic the removal of random objects from an RGB image (see Fig. 4(a)). This traditional feature dropout (with dropout rate \(p\)) is implemented as an element-wise multiplication of the feature \(f\) and the dropout mask \(\xi^{f}\). Each element \(\xi^{f}_{e}\) in the dropout mask \(\xi^{f}\) is a sample of a random variable which obeys an independent and identical Bernoulli distribution multiplied by \(1/(1-p)\). And for different features, the distributions of dropout masks are independent as well.

\[{\rm dropout}_{p}(f)= f\odot\xi^{f} \tag{13}\] \[\xi^{f}_{e}\sim\frac{1}{1-p}{\rm Ber}(1-p) \tag{14}\]

Because of this independence among dropout masks, the traditional feature dropout fails in augmenting the existing environments because the 'removal' is inconsistent in different views at the same viewpoint, and in different viewpoints.

Figure 4: Comparison of the two dropout methods (based on image features).

Figure 3: Comparison of the two dropout methods (based on an illustration on an RGB image).

 

To illustrate this idea, we take the four RGB views in Fig. 2(a) as an example, where the chairs are randomly dropped from the views. The removal of the left chair (marked with a red polygon) from view \(o_{t,2}\) is inconsistent because it also appears in view \(o_{t,1}\). Thus, the speaker could still refer to it and the agent is aware of the existence of the chair. Moreover, another chair (marked with a yellow polygon) is completely removed from viewpoint observation \(o_{t}\), but the views in next viewpoint \(o_{t+1}\) provides conflicting information which would confuse the speaker and the agent. Therefore, in order to make generated environments consistent, we propose our environmental dropout method, described next.

Environmental DropoutWe create a new environment \(\mathbf{E^{\prime}}\) by applying environmental dropout on an existing environment \(\mathbf{E}\).

\[\mathbf{E^{\prime}}=\mathrm{envdrop}_{p}(\mathbf{E}) \tag{15}\]

The view feature \(f^{\prime}_{t,i}\) observed from the new environment \(\mathbf{E^{\prime}}\) is calculated as an element-wise multiplication of the original feature \(f_{t,i}\) and the environmental dropout mask \(\xi^{\mathbf{E}}\) (see Fig. 3(b)):

\[f^{\prime}_{t,i} =f_{t,i}\odot\xi^{\mathbf{E}} \tag{16}\] \[\xi^{\mathbf{E}}_{e} \sim\frac{1}{1-p}\mathrm{Ber}(1-p) \tag{17}\]

To maintain the spatial structure of viewpoints, only the image feature \(\mathrm{ResNet}(v_{t,i})\) is dropped while the orientation feature \((\cos(\theta_{t,i}),\sin(\theta_{t,i}),\cos(\phi_{t,i}),\sin(\phi_{t,i}))\) is fixed. As illustrated in Fig. 2(b), the idea behind environmental dropout is to mimic new environments by removing one specific class of object (e.g., the chair). We demonstrate our idea by running environmental dropout on the ground-truth semantic views in Sec. 7.3, which is proved to be far more effective than traditional feature dropout. In practice, we perform the environmental dropout on image's visual feature where certain structures/parts are dropped instead of object instances, but the effect is similar.

We apply the environmental dropout to the back translation model as mentioned in Sec. 3.4.1. Note the environmental dropout method still preserves the connectivity of the viewpoints, thus we use the same way Fried et al. (2018) to collect extra unlabeled routes \(\{\mathbf{R}^{\prime}\}\). We take _speaker_ to generate an additional instruction \(\mathbf{I^{\prime}}\)=\(P_{\mathbf{E},\mathbf{R}-\mathbf{I}}\left(\mathbf{E^{\prime}},\mathbf{ R^{\prime}}\right)\) in the new environment \(\mathbf{E^{\prime}}\). At last, we use IL+RL (in Sec. 3.3) to fine-tune the model with this new triplet \((\mathbf{E^{\prime}},\mathbf{R^{\prime}},\mathbf{I^{\prime}})\).

#### 3.4.3 Improvements on Speaker

Our speaker model is an enhanced version of the encoder-decoder model of Fried et al. (2018), with improvements on the visual encoder: we stack two bi-directional LSTM encoders: a route encoder and a context encoder. The route encoder takes features of ground truth actions \(\{a^{*}_{t}\}_{t=1}^{T}\) from the route as inputs. Each hidden state \(r_{t}\) then attends to surrounding views \(\{f_{t,i}\}_{i=1}^{36}\) at each viewpoint. The context encoder then reads the attended features and outputs final visual encoder representations:

\[r_{1},...,r_{\text{T}} =\mathrm{Bi\mbox{-LSTM}^{\text{RTF}}}(g_{1,a^{*}_{1}},...,g_{ \text{r},a^{*}_{\text{T}}}) \tag{18}\] \[\gamma_{t,i} =\mathrm{softmax}_{i}(f^{\intercal}_{t,i}W_{k}\,r_{t})\] (19) \[\hat{f}_{t} =\sum_{i}\gamma_{t,i}f_{t,i}\] (20) \[c_{1},...,c_{\text{T}} =\mathrm{Bi\mbox{-LSTM}^{\text{CTX}}}(\hat{f}_{1},...,\hat{f}_{ \text{T}}) \tag{21}\]

The decoder is a regular attentive LSTM-RNN, which is discussed in Sec. 3.2. Empirically, our enhanced speaker model improves the BLEU-4 score by around 3 points.

## 4 Experimental Setup

Dataset and SimulatorWe evaluate our agent on the Matterport3D simulator Anderson et al. (2018). Navigation instructions in the dataset are collected via Amazon Mechanical Turk by showing them the routes in the Matterport3D environment Chang et al. (2017). The dataset is split into training set (61 environments, 14,025 instructions), seen validation set (61 environments, 1,020 instructions), unseen validation set (11 environments, 2,349 instructions), and unseen test set (18 environments, 4,173 instructions). The unseen sets only involve the environments outside the training set.

Evaluation MetricsFor evaluating our model, Success Rate (SR) is the primary metric. The execution route by the agent is considered a success when the navigation error is less than \(3\) meters. Besides success rate, we use three other metrics7 : Navigation Length (NL), Navigation Error (NE), and Success rate weighted by Path Length (SPL) Anderson et al. (2018). Navigation Error (NE) is 

the distance between target viewpoint \({\bf T}\) and agent stopping position.

Implementation DetailsSimilar to the traditional dropout method, the environmental dropout mask is computed and applied at each training iteration. Thus, the amount of unlabeled semi-supervised data used is not higher in our dropout method. We also find that sharing the environmental dropout mask in different environments inside a batch will stabilize the training. To avoid overfitting, the model is early-stopped according to the success rate on the unseen validation set. More training details in appendices.

## 5 Results

In this section, we compare our agent model with the models in previous works on the Vision and Language Navigation (VLN) leaderboard. The models on the leaderboard are evaluated on a private unseen test set which contains \(18\) new environments. We created three columns in Table 1 for different experimental setups: single run, beam search, and unseen environments pre-exploration. For the result, our model outperforms all other models in all experimental setups.

Single RunAmong all three experimental setups, single run is the most general and highly correlated to the agent performance. Thus, it is considered as the primary experimental setup. In this setup, the agent navigates the environment once and is not allowed9 to: (1) run multiple trials, (2) explore nor map the test environments before starting. Our result is \(3.5\%\) and \(9\%\) higher than the second-best in Success Rate and SPL, resp.

Footnote 9: footnotemark:

Beam SearchIn the beam search experimental setup, an agent navigates the environment, collects multiple routes, re-ranks them, and selects the route with the highest score as the prediction. Besides showing an upper bound, beam search is usable when the environment is explored and saved in the agent's memory but the agent does not have enough computational capacity to fine-tune its navigational model. We use the same beam-search algorithm, state factored Dijkstra algorithm, to navigate the unseen test environment. Success Rate of our model is \(5.9\%\) higher than the second best. SPL metric generally fails in evaluating beam-search models because of the long Navigation Length (range of SPL is \(0.01\text{--}0.02\)).

Pre-ExplorationThe agent pre-explores the test environment before navigating and updates its agent model with the extra information. When executing the instruction in the environment, the experimental setup is still "single run". The "pre-exploration" agent mimics the domestic robots (e.g., robot vacuum) which only needs to navigate the seen environment most of the time. For submitting to the leaderboard, we simply train our agent via back translation with environmental dropout on test unseen environments (see Sec.7.2). Our result is \(3.4\%\) higher than Wang et al. (2019) in Success Rate and \(2.0\%\) higher in SPL. 10

Footnote 10: The Behavioral Cloning (IL) baseline is the same as the panoramic view baseline in Fried et al. (2018) except for two differences: (1) The agent takes the teacher action instead of the sampled action from the distribution (see “imitation learning” of Sec. 3.3), (2) The hidden input of the LSTM is the instruction-aware hidden from the previous step (see Sec. 3.2). We improve our baseline result with these modifi

## 6 Ablation Studies

Supervised LearningWe first show the effectiveness of our IL+RL method by comparing it with the baselines (Table 2). We implement Behavioural Cloning11 and Advantage Actor-Critic

\begin{table}
\begin{tabular}{|l|c c c|c c c|c c c|} \hline Models & \multicolumn{6}{c|}{Test Unseen (Leader-Board)} \\  & \multicolumn{3}{c|}{Single Run} & \multicolumn{6}{c|}{Beam Search} & \multicolumn{6}{c|}{Pre-Explore} \\ \hline  & NL & SR(\(\%\)) & _SPL_ & NL & _SR(\(\%\))_ & SPL & NL & SR(\(\%\)) & _SPL_ \\ \hline Random Anderson et al. [2018b] & 9.89 & 13.2 & 0.12 & - & - & - & - & - & - \\ Seq-to-Seq Anderson et al. [2018b] & 8.13 & 20.4 & 0.18 & - & - & - & - & - & - \\ \hline Look Before You Leap Wang et al. [2018a] & 9.15 & 25.3 & 0.23 & - & - & - & - & - & - \\ Speaker-Follower Fried et al. [2018] & 14.8 & 35.0 & 0.28 & 1257 & 53.5 & 0.01 & - & - & - \\ Self-Monitoring Ma et al. [2019] & 18.0 & 48.0 & 0.35 & 373 & 61.0 & **0.02** & - & - & - \\ Reinforced Cross-Modal Wang et al. [2019] & 12.0 & 43.1 & 0.38 & 358 & 63.0 & **0.02** & 9.48 & 60.5 & 0.59 \\ \hline Ours & 11.7 & **51.5** & **0.47** & 687 & **68.9** & 0.01 & 9.79 & **63.9** & **0.61** \\ \hline \end{tabular}
\end{table}
Table 1: Leaderboard results under different experimental setups. NL, SR, and SPL are Navigation Length, Success Rate and Success rate weighted by Path Length. The primary metric for each setup is in italics. The best results are in bold font and the second best results are underlined.

 

as our imitation learning (IL) and reinforcement learning (RL) baselines, respectively. The mixture of IL+RL (see Sec. 3.3) outperforms the IL-only model and RL-only model by \(2.9\%\) and \(22.5\%\), which means that our IL+RL could overcome the misleading teacher actions in IL and significantly stabilize the training of RL.

Footnote 1: [https://github.com/face](https://github.com/face) 



### More Environments vs. More Data

In order to show that more environments are crucial for better performance of agents, in Fig. 5, we present the result of Supervised Learning (SL) with different amounts of data selected by two different data-selection methods. The first method gradually uses more #environments (see the blue line "SL with more envs") while the second method selects data from the whole training data with all 60 training environments (see the red line "SL with more data"). Note that the amounts of data in the two setups are the same for each plot point As shown in Fig. 5, the "more ens" selection method shows _higher growth rate_ in success rate than the "more data" method. We also predict the success rates (in dashed line) with the prediction method in Sun et al. (2017). The predicted result is much higher when training with more environments. The predicted result (the right end of the red line) also shows that the upper bound of Success Rate is around \(52\%\) if all the 190K routes in the training environments is labeled by human (instead of being generated by _speaker_ via back translation), which indicates the need for "new" environments.

### Back Translation on Unseen Environments

In this subsection, we show that back translation could significantly improve the performance when it uses new data triplets from testing environments -- the unseen validation environments where the agent is evaluated in. Back translation (w.o. Env Drop) on these unseen environments achieves a success rate of \(61.9\%\), while the back translation on the training environments only achieves \(46.5\%\). The large margin between the two results indicates the need of "new" environments in back translation. Moreover, our environmental dropout on testing environments could further improve the result to \(64.5\%\), which means that the amount of environments in back translation is far from enough.

### Semantic Views

To demonstrate our intuition of the success of environmental dropout (in Sec. 3.4.2), we replace the image feature \({\rm ResNet}(v_{t,i})\) with the semantic view feature. The semantic views (as shown in Fig. 6) are rendered from the Matterport3D dataset Chang et al. (2017), where different colors indicate different types of objects. Thus, dropout on the semantic view feature would remove the object from the view. With the help of this additional information (i.e., the semantic view), the success rate of IL+RL is \(49.5\%\) on the unseen validation set. Back translation (without dropout) slightly improves the result to \(50.5\%\). The result with feature dropout is \(50.2\%\) while the environmental dropout could boost the result to \(52.0\%\), which supports our claim in Sec. 3.4.2.

## 8 Conclusion

We presented a navigational agent which better generalizes to unseen environments. The agent is supervised with a mixture of imitation learning and reinforcement learning. Next, it is fine-tuned with semi-supervised learning, with speaker-generated instructions. Here, we showed that the limited variety of environments is the bottleneck of back translation and we overcome it via 'environmental dropout' to generate new unseen environments. We evaluate our model on the Room-to-Room dataset and achieve rank-1 in the Vision and Language Navigation (VLN) challenge leaderboard under all experimental setups.

## Acknowledgments

We thank the reviewers for their helpful comments. This work was supported by AROYIP Award #W911NF-18-1-0336, ONR Grant #N00014-18-1-2871, and faculty awards from Google, Facebook, Adobe, Baidu, and Salesforce. The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of the funding agency.

## References

* Anderson et al. [2018] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh

Figure 6: Comparison of semantic and raw RGB views.



Mottaghi, Manolis Savva, et al. 2018a. On evaluation of embodied navigation agents. _arXiv preprint arXiv:1807.06757_.
* Anderson et al. (2018b) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Andreas and Klein (2015) Jacob Andreas and Dan Klein. 2015. Alignment-based compositional semantics for instruction following. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1165-1174.
* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association of Computational Linguistics_, 1:49-62.
* Bojarski et al. (2016) Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. 2016. End to end learning for self-driving cars. _arXiv preprint arXiv:1604.07316_.
* Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterror3D: Learning from RGB-D data in indoor environments. _International Conference on 3D Vision (3DV)_.
* Chaplot et al. (2018) Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. 2018. Gatedattention architectures for task-oriented language grounding. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Chen and Mooney (2011) David L Chen and Raymond J Mooney. 2011. Learning to interpret natural language navigation instructions from observations. In _AAAI_, volume 2, pages 1-2.
* Correa et al. (2010) Andrew Correa, Matthew R Walter, Luke Fletcher, Jim Glass, Seth Teller, and Randall Davis. 2010. Multimodal interaction with an autonomous forklift. In _Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction_, pages 243-250. IEEE Press.
* Das et al. (2018) Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In _CVPR_.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500.
* Fried et al. (2018) Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In _Advances in Neural Information Processing Systems_, pages 3318-3329.
* Gandhi et al. (2017) Dhiraj Gandhi, Lerrel Pinto, and Abhinav Gupta. 2017. Learning to fly by crashing. _IROS_.
* Gordon et al. (2018) Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. 2018. Iqa: Visual question answering in interactive environments. In _CVPR_.
* Gupta et al. (2017) Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. 2017. Cognitive mapping and planning for visual navigation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 2616-2625.
* Hayashi et al. (2007) Kotaro Hayashi, Daisuke Sakamoto, Takayuki Kanda, Masahiro Shiomi, Satoshi Koizumi, Hiroshi Ishiguro, Tsukasa Ogasawara, and Norihiro Hagita. 2007. Humanoid robots as a passive-social medium-a field experiment at a train station. In _Human-Robot Interaction (HRI), 2007 2nd ACM/IEEE International Conference on_, pages 137-144. IEEE.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Hinton et al. (2012) Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
* Hoang et al. (2018) Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018. Iterative back-translation for neural machine translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 18-24.
* Levine et al. (2016) Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. 2016. End-to-end training of deep visuomotor policies. _JMLR_.
* Ma et al. (2019) Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. 2019. Self-monitoring navigation agent via auxiliary progress estimation.
* Mei et al. (2016) Hongyuan Mei, Mohit Bansal, and Matthew R Walter. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In _AAAI_, volume 1, page 2.
* Misra et al. (2017) Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1004-1015 

* Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In _International Conference on Machine Learning_, pages 1928-1937.
* Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In _International Conference on Learning Representations_.
* Poole and Mackworth (2010) David L Poole and Alan K Mackworth. 2010. _Artificial Intelligence: foundations of computational agents_. Cambridge University Press.
* Prabhumoye et al. (2018) Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In _Proc. ACL_.
* Ranzato et al. (2015) Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. In _ICLR_.
* Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. _International Journal of Computer Vision_, 115(3):211-252.
* Sadeghi and Levine (2017) Fereshteh Sadeghi and Sergey Levine. 2017. CAD2RL: Real single-image flight without a single real image. _RSS_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 86-96.
* Sun et al. (2017) Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting unreasonable effectiveness of data in deep learning era. In _Computer Vision (ICCV), 2017 IEEE International Conference on_, pages 843-852. IEEE.
* Tellex et al. (2011) Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth J Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In _AAAI_, volume 1, page 2.
* Walters et al. (2007) Michael L Walters, Kerstin Dautenhahn, Sarah N Woods, and Kheng Lee Koay. 2007. Robotic etiquette: results from user studies involving a fetch and carry task. In _Proceedings of the ACM/IEEE international conference on Human-robot interaction_, pages 317-324. ACM.
* Wang et al. (2019) Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
* ECCV 2018
- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI_, pages 38-55.
* Wang et al. (2018b) Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018b. Switchout: an efficient data augmentation algorithm for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 856-861.
* Wu et al. (2018) Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. 2018. Building generalizable agents with a realistic and rich 3d environment. _arXiv preprint arXiv:1801.02209_.
* Zhu et al. (2017a) Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. 2017a. Visual semantic planning using deep successor representations. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 483-492.
* Zhu et al. (2017b) Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. 2017b. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In _Robotics and Automation (ICRA), 2017 IEEE International Conference on_, pages 3357-3364. IEEE.

## Appendix A Appendices

### Implementation Details

We use ResNet-152 He et al. (2016) pretrained on the ImageNet Russakovsky et al. (2015) to extract the \(2048\)-dimensional image feature. The agent model is first trained with supervised learning via the mixture of imitation and reinforcement learning. The model is then fine-tuned by back translation with environmental dropout. To stabilize the optimization of back translation, we calculate supervised loss for half of the batch and semi-supervised loss for the other half. We find that sharing the environmental dropout mask in different environments inside the same batch will stabilize the training.

The word embedding is trained from scratch with size \(256\) and the dimension of the action embedding is \(64\). The size of the LSTM units is set to \(512\) (\(256\) for the bidirectional LSTM). In RL training, the discounted factor \(\gamma\) is \(0.9\). We use the reward shaping Wu et al. (2018): the direct 

reward \(r_{t}\) at time step \(t\) is the change of the distance \(d_{t-1}-d_{t}\), supposing \(d_{t}\) is the distance to the target position at time step \(t\). The maximum decoding action length is set to \(35\). For optimizing the loss, we use RMSprop (Hinton et al., 2012) (as suggested in Mnih et al. (2016)) with a fixed learning rate \(1e-4\) and the batch size is \(64\). We applying dropout rate \(0.4\) to the environmental dropout and \(0.5\) to the dropout layers which regularize the network. The global gradient norm is clipped by \(40\). We tuned the hyper-parameters based on the Success Rate of the unseen validation set.

When working with the semantic view, the _key_ labels (e.g., _wall_, _floor_, _ceil_) are not dropped, because they are the basic structure of the environment. Empirically, no improvement will be achieved when the _key_ labels are dropped as well.