Generalized Natural Language Grounded Navigation via Environment-agnostic Multitask Learning
Authors: Xin Wang
Vihan Jain
Eugene Ie
William Wang
Zornitsa Kozareva
Sujith Ravi
Venue:  (2020)
Keywords: Natural Language Grounded Navigation
Multitask Learning
Agnostic Learning

Abstract: Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e.g., following natural language instructions or dialog. However, existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments. In order to close the gap between seen and unseen environments, we aim at learning a generalizable navigation model from two novel perspectives:
(1) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH) tasks, which benefits from richer natural language guidance and effectively transfers knowledge across tasks;
(2) we propose to learn environment-agnostic representations for navigation policy that are invariant among environments, thus generalizing better on unseen environments.
Extensive experiments show that our environment-agnostic multitask navigation model significantly reduces the performance gap between seen and unseen environments and outperforms the baselines on unseen environments by 16% (relative measure on success rate) on VLN and 120% (goal progress) on NDH, establishing the new state of the art for NDH task. 

Under review as a conference paper at ICLR 2020
GENERALIZED NATURAL LANGUAGE GROUNDED NAVIGATION VIA ENVIRONMENT-AGNOSIE MULTI- TASK LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent research efforts enable study for natural language grounded navigation in photo-realistic environments, e.g., following natural language instructions or dia- log. However, existing methods tend overfit training data in seen environments and fail to generalize well in previously unseen environments, In order to close the gap between seen and unseen environments, we aim at learning a generaliz- able navigation model from two novel perspectives: (1) we introduce multitask navigation model that can be seamlessly trained on both Vision-Language Navi- gation (VLN) and Navigation from Dialog History (NDH) tasks, which benefits from richer natural language guidance and effectively transfers knowledge across tasks; (2) propose to learn cettoommen--ggnnosoe representations for navigation policy that are invariant among environments, thus generalizing better on unseen environments. Extensive experiments show that environment-agnostee mul- titask navigation model significantly reduces the performance gap between seen and unseen environments and outperforms the baselines on unseen environments by 16% (relative measure on success rate) on VLN and 120% (goal progress) on NDH, establishing the new state of the art for the NDH task.
INTRODUCTION
Navigation in visual environments by following natural language guidance (Hemachandra et al, 2015) is fundamental capability of intelligent robots that simulate human behaviors, because hu- mans can easily reason about the language guidance and navigate efficiently by interacting with the visual environments. Recent efforts (Anderson et al. [0188) Das et al. 2018 Thomason al. 2019) empower large-scale learning of natural language grounded navigation that is situated in photo- realistic simulation environments.
Nevertheless, the generalization problem commonly exists for these tasks, especially indoor nav- igation: the agent usually performs poorly unknown environments that have never been during training. One of the main causes for such behavior is data scarcity as it is expensive and time-consuming extend either visual environments or natural language guidance. The number of scanned houses for indoor navigation is limited due to high expense and privacy concerns. Besides, unlike vision-only navigation tasks (Mirowski al., 2018 2016 Xia al. 2018 Manolis Savva et al. 2019 Kolve et al. 2017) where episodes can be exhaustively sampled in simulation, natural language grounded navigation is supported by human demonstrated interaction and communication in natural language. It is impractical to fully collect and cover all the samples for individual tasks.
Therefore, it is essential though challenging to efficiently learn more generalized policy for nat- ural language grounded navigation tasks from existing data (Wu et al.) [018a b). In this paper, we study how to resolve the generalization and data scarcity issues from two different angles. First, previous methods are trained for one task at the time, each new task requires training brand new agent instance that can only solve the one task it was trained on. In this work, we propose a gen- eralized multitask model for natural language grounded navigation tasks such as Vision- LaaguueLe Navigation (VLN) and Navigation from Dialog History (NDH), aiming at efficiently transferring knowledge across tasks and effectively solving both tasks with one agent simultaneously.

---

Under review as a conference paper at ICLR 2020
Moreover, although there are thousands of trajectories paired with language guidance, the underlying house scans are restricted. For instance, the popular Matterport33 dataset (Chang et al.) 2017) contains only 61 unique house scans in the training set. The current models perform much better in seen environments by taking advantage of the knowledge of specific houses they have acquired over multiple task completions during training, but fail to generalize to houses not seen during training. Hence we propose an environment-agnoss invariant to specific environments but still able support navigation. Endowed with the learned environment-agnosio representations, the agent is further prevented from the overfitting issue and generalizes better on unseen environments.
learning method to learn a visual representation that is
To the best of our knowledge, we are the first to introduce natural language grounded multitask and cavironment-agnostic training regimes and validate their effectiveness on VI.N and NDH tasks. Extensive experiments demonstrate that our environment-agnostic multitask navigation model can not only efficiently execute different language guidance in indoor environments but also outperform the single-task baseline models by large margin on both tasks. Besides, the performance gap between seen and unseen environments is significantly reduced. We also set a new state of the art on NDH with over 12 120% improvement in terms of goal progress.
2 BACKGROUND
Vision-and- Srooo--and-anguurg Language Navigation. Aision-and-iee and-Language Navigation (Anderson al.
2018b
Chen et al. 2019) task requires an embodied agent to navigate in photo-realistic environments to earry out natural language instructions. The agent is spawned an initial pose Po (no; do. 0o), which includes the spatial location, heading and elevation angles. Given a natural language instruc- tion X {x1, T2, ... Jn the agent is expected to perform a sequence of actions {aj, d2,..., aT) and arrive at the target position Utar specified by the language instruction X, which describes step- by-step instructions from the starting position to the target position. In this work, we consider VLN task defined for Room-to- Koom-to-Room (R2R) (Anderson et al. (2018b) dataset which contains instruction- trajectory pairs across 90 different indoor environments (houses).
Previous VIN methods have studied various aspects to improve the navigation performance, such as planning (Wang et al.) 2018), data augmentation (Fried et al. 2018: Tan er al. 2019), cross-modal alignment (Wang et al. 2019; Huang et al. 2019b), progress estimation (Ma et al. (019a)), error correction (Ma et al. 2019b] Ke et al. 2019). interactive language assistance (Nguyen et al. 2019 Nguyen & Daumé IIII 2019) etc. This work tackles VLN via multitask learning and environment- agnostic learning, which is orthogonal to all these prior arts.
Navigation from Dialog History. Different from Visual Dialog (Das et al. (017)) which involves dialog grounded in a single image, the recently introduced Cooperative Vision-and- -nd---aaddoDgggg Navi- gation (CVDN) dataset (Thomason et al. 2019) includes interactive language assistance for indoor navigation, which consists of over 2,000 embodied, human- dialogs situated in photo-realistic home environments. The task of Navigation from Dialog History (NDH) is defined as: given a target object to and a dialog history between humans cooperating to perform the task, the embodied agent must infer navigation actions towards the goal room that contains the target object. The dialog his- tory is denoted as < to, Q1, A1, Q2, Ag, ..., Qu, A, >, including the target object to, the questions Q and answers till the turn (0 < < k. where & is the total number of Q-A turns from the beginning the goal room). The agent, located in Po: is trying move closer to the goal room by inferring from the dialog history that happened before.
Multitask Learning. The basis of Multitask (MT) learning is the notion that tasks can serve as mutual sources of inductive bias for each other (Caruana) 1993). When multiple tasks are trained jointly, MT learning causes the learner to prefer the hypothesis that explains all the tasks simulta- neously, hence leading to more generalized solutions. MT learning has been successful in natural language processing (Collobert& & Weston 20088) speech recognition (Deng et al. 2013), computer vision (Girshick] 2015), drug discovery (Ramsundar et al. 2015), and Atari games TTeh et al.) 2017).
The deep reinforcement learning methods that have become very popular for training models on nat- ural language grounded navigation tasks (Wang al.) 2019 Huang et al. 2019a by are known to be data inefficient. In this work, we introduce multitask reinforcement learning for such tasks to improve data efficiency by positive transfer across related tasks.
Tan et al.) 2019)
2

---

Under review as a conference paper at ICLR 2020
NDH Diolog or VLN Instruction
Multitask Data Sampling
Paired Demo Poth
Word Embedding
{
Lnav
Xeev
Language Encoder
CM-ATT
Trajectory Encoder
Gradient Reversal
Lnav
Action Predictor
Environment Classifier
Lenw
at
y/
Figure 1: Overview of environment-agnostic multitask learning. See Section 3.1 for more details.
Environment-agnoste Learning. A few studies agnostic learning have been proposed cently. For example, Modsl-Agnostic Meta-Learning (MAML) (Finn et al.] 2017) aims to train model variety of learning tasks and solve new task using only few training examples. Liu et al." (2018) proposes a unified feature disentangler that learns nomain-invariant representation across multiple domains for image translation. Other
techniques are also proposed for supervised (Li et al., 2018) and unsupervised domain adaption (Romijnders et al.) 2019 Peng al. 2019). In this work, pair the environment classifier with a gradient reversal layer GGain & Lempitsky 2015) to learn an enviromment-agnne
representation that can be better generalized on unseen environments zero-shot fashion where no adaptation is involved.
Distributed Actor-Learner Navigation Learning Framework. To train models for the various language grounded navigation tasks like VLN and NDH, we develop a distributed actor-learner learning infrastructure" The framework design is inspired by IMPALA (Espeholt et al. 2018) and uses its off- off-policy correction method called V-trace to efficiently scale reinforcement learning methods to thousands of machines. The framework additionally supports a variety of supervision strategies important for navigation tasks such as macher-forcing (Anderson et al., 2018b). student- forcing (Anderson et al. (018b) and mixed supervision (Thomason al.) 2019). The framework is built using TensorFlow (Abadi et al. 2016) and supports ML accelerators (GPU, TPU).
ENVIRONMENT-AGNOSTE MULTITASK LEARNING
3.1 OVERVIEW
Our environment-agnostic multitask navigation model is illustrated in Figure I First, we adapt the reinforced cross-modal matching (RCM) model (Wang et al., 2019) and make it seamlessly transfer across tasks by sharing all the learnable parameters for both NDH and VLN, including joint word embedding layer, language encoder, trajectory encoder, cross-modal attention module (CM-ATT), and action predictor. Furthermore, to learn the enviromment-agnosttc representation 26, we equip the navigation model with an environment classifier whose objective is to predict which house the agent is. But note that between trajectory encoder and environment classifier, a gradient reversal layer (Ganin Lempitsky) 2015) introduced to reverse the gradients back- propagated to the trajectory encoder, making learn representations that are environment-agnosti and thus more generalizable in unseen environments. During training, the environment classifier minimizing the environment classification loss Lene, while the trajectory encoder is maximizing Lene and minimizing the navigation loss Lnav- The other modules are optimized with the navigation loss Lrav simultaneously. Below introduce multitask reinforcement learning and environment- agnostic representation learning. A more detailed model architecture is presented in Section
'The identity not disclosed respect the anonymity of the submission.
3

---

Under review as a conference paper at ICLR 2020
3.2 MULTITASK REINFORCEMENT LEARNING
Interleaved Multitask Data Sampling. To avoid overfitting to individual tasks, we adopt an in- terleaved multitask data sampling strategy to train the model. Partieularly, each data sample within a mini-batch can be from either task, so that the VLN instrection-rraeectory pairs and NDH dialog- trajectory pairs are interleaved mini-batch though they may have different learning objectives.
Reward Shaping, Following prior art (Wang et al. 2018 2019), we first implement a discounted cumulative reward function It for the VLN and NDH tasks:
T
{d(sy, Utar) - d(v Utar) [1[((T Star) S dea
if <
R(St, 0z) = Σ 2f--tr(se ar); where r(srr ae)
if
C==1
where is the discounted factor, d(e,,, Vtar) is the distance between state and the target location Utar, and den is the maximum distance from ytar that the agent allowed to terminate for success.
Different from VLN, NDH is essentially room navigation instead of point navigation because the agent is expected reach a room that contains the target object. Suppose the goal room is occupied by a set of nodes {v, J1 we replace the distance function d( ((s Utar) in Equation with the minimum distance to the goal room (room((r: {u: ) for NDH:
droom (st, (v.))) min d(st, v.)
(2)
ISKEN
Navigation Loss. Since human demonstrations are available for both VLN and NDH tasks, we use behavior cloning to constrain the learning algorithm to model state-action spaces that are most relevant to each task. Following previous works (Wang et al. 2019), we also use reinforcement learning to aid the agent's ability to recover from erroneous actions unseen environments. During multitask navigation model training, adopt mixed training strategy of reinforcement learning and behavior cloning, so the navigation loss function is:
Lnav Ba,>x R(s,, -Ello IT (a]]s))
where we use REINFORCE policy gradients (Williams) 1992) and supervised learning gradients update the policy æ. b is the estimated baseline to reduce the variance and oz is the human demonstrated action.
3.3 ENVIRONMENT- AGNOSTIC REPRESENTATION LEARNING
To further improve the generalizability of the navigation policy, we propose to learn a latent environment-agnose representation that is invariant among seen environments. We would like to get rid the vevironment-speciff features that are irrelevant to general navigation (c.g. unique house appearances), preventing the model from overfitting to specific seen environments. We can reformulate the navigation policy as
T(alss)) p(a./2,, s()(s())))
(4)
where z, is a latent representation.
As shown in Figure p(a+ (a.lzt,,)) is modeled by the policy module (including CM-ATT and action predictor) and p( 7 St) is modeled by the trajectory encoder. In order to learn the environment- agnostic representation, we employ an environment classifier and gradient reversal layer (Ganin & Lempitsky 2015). The environment classifier is parameterized to predict the identity of the house where the agent 1s, so loss function Lene defined as
Lene - Bllog p(v y -²(=)))
(5)
where y² is the ground-truth house label. The gradient reversal layer has no parameters. It acts as but multiplies the gradient by -X and passes it to an identity transform during boock--rppggttin. Therefore. in addition to minimizing the navigation the trajectory encoder during back- loss Lrav. the trajectory encoder is also maximizing the environment classification loss Lenv. trying to increase the entropy of the classifier in am adversarial learning manner where the classifier is minimizing the classification loss conditioned on the latent representation 2,.
4

---

Under review as a conference paper at ICLR 2020
4 MODEL ARCHITECTURE
Language Encoder. The natural language guidance (instruction or dialog) is tokenized and em- bedded into n-dimensional space X { ,
22; .... .2) where the word vectors æ; are initialized
randomly. The vocabulary is restricted to tokens that occur at least five times in the training instruc- tions (The vocabulary used when jointly training VLN and NDH tasks the union of the two tasks' vocabularies.). All out-of- out-of-vocabulle vocabulary tokens are mapped to a single out-of cuttof-vvocaulayy identifier. The token sequence encoded using a bi-directional LSTM (Schuster & Paliwal 1997) create X following:
Hx = 11 11 1 h = ) h LST RZz)), = No h1t1)
(6)
where h and h are the hidden states of the forward and backward LSTM layers time step t respectively, and the a function is used to combine h x and h x into h;x
Trajectory Encoder. Similar benchmark models (Fried al. 2018 Wang al. 2019 Huang et al. 2019b), at each time step i, the agent perceives a 360-degree panoramic view at its current location. The view discretized into view angles (A: 36 in our implementation, elevations by 12 headings at 30-degree intervals). The image at view angle i. heading angle é and eleva- tion angle is represented by concatenation of the pre-trained CNN image features with the 4- dimensional orientation feature [sin Φb cos Φ; sin 0; cos 0] to form 0t.i- The visual input sequence
V = U1, 22 Um } is encoded using LSTM to create HV following:
H [n ;..;; 1
where hy V L hY-1)
v, Attention h vt,1.A)) ) is the diteetiionppoled representation of all view angles using previous agent state f as the query. We use the dot- not-product attention (Vaswani et al.) 2017) hereafter.
Policy Module. The policy module comprises of cross-modal attention (CM-ATT) unit as well as an action predictor. The agent learns a policy Ta over parameters 0 that maps the natural language instruction X and the initial visual scene U1 a sequence of actions [a1, (a1,02;-.. ..., an). The action space which is common to VLN and NDH tasks consists of navigable directions from the current the
location. The available actions at time are denoted as U4,1..1 where Ur is the representation navigable direction j from the current location obtained similarly to Ut,i. The number of available actions, I, varies per location, since graph node connectivity varies. As in [aang et al. (2019), the model predicts the probability pa of each navigable direction d using bilinear dot product:
Pd softmax (hh V: ;c, :
c. c.iiual) "W))u,.WW.""
(9)
where c: Attention (h, h ) and cxtsual Attention (ctesT v1...)). W. and W., are learnable parameters.
Environment Classifier.
The environment classifier is two-layer perceptron with SoftMax layer pe as the last layer. Given the latent representation z; (which is hy in our setting), the classifier generates a probability distribution over the house labels.
EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
Implementation Details. In the experiments, we use 2-layer bi-directional LSTM for the instruc- tion encoder where the size of LSTM cells 256 units in each direction. The inputs to the encoder are 300-dimensional embeddings initialized randomly. For the visual encoder, we use 2-layer LSTM with cell size of 512 units. The encoder inputs are image features derived as mentioned in Section 4 The cross-modal attention layer size is 128 units. The environment classifier has one hidden layer of size 128 units followed by an output layer of size equal to the number of classes. During training, some episodes in the batch are identical to available human demonstrations in the training dataset where the objective is to increase the agent's 's likelihood of choosing human actions (behavioral cloning (Bain& & Sammut 1999)). The rest of the episodes are constructed by sampling
5

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of agent performance under different training strategies. Note that the single-
task RCM model is independently trained and tested on VLN or NDH tasks.
Val Seen
Val Unseen
Model
NDH
VLN
NDH
VLN
Progress
PL
NR SR 1 SPL CLS
Progress
PL
NE. SR 7 SPL CLS
scg2seg Thomason et al. 2019
5.92
2.10
RCM Wang et al. 2019
12.08
3.25 67.60
15,00
6.02 40.60
Ours
single-task RCM
6.49
10.75
5.09
52.39
48.86
63.91
2.64
10.60
6.10
42.93
38.88
54.86
single-task RCM- + EnvAg
6.07
11.31
4.93
52.79
48.85
63.26
3.15
11.36
5.79
44.40
40.30
55.77
MT-RCM
5.28
10.63
5.09
56.42
49.67
68.28
4.36
10.23
5.31
46.20
44.19
54,99
MT-RCM EnvAg
11.60
4.83
53.30
49.39
64.10
4.65
12.05
5.41
47.22
41.80
56.22
NDH
180 + 4hl plapeey 119
Figure 2: Selected tokens from the vocabulary for VLN (left) and NDH (right) tasks which gained more than 40 additional occurrences in the training dataset due to joint-training.
from agent's 's own policy, In the experiments, unless otherwise stated, we use entire dialog history from NDH task for model training. All the reported results in subsequent studies are averages of at least 3 independent runs.
The agents are evaluated on two datasets, namely Validation Seen that con- tains new paths from the training environments and Validation Unseen that contains paths from previously unseen environments. The evaluation metrics for VLN task are as follows: Path Length (PL) measures the total length of the predieted path; Navigation Error (NE) measures the distance between the last nodes the predicted and the reference paths; Success Rate (SR) measures how often the last node in the predicted path is within some threshold distance of the last node in the reference path; Success weighted by Path Length (SPL) (Anderson et al. 2018a) measures Success Rate weighted by the normalized Path Length; and Coverage weighted by Length Score (CLS) (Jain et al., (2019) measures predicted path's 's conformity to the reference path weighted by length score. For NDH task, the agent's progress is defined as reduction (in meters) from the distance to the goal region at agent's 's first position versus at its last position (' (Thomason et al. 2019).
Evaluation Metrics.
5.2 ENVIRONMENT AGNOSTIC MULTITASK LEARNING
Table shows the results of training the navigation model using environment-agnostic learning (EnvAg) well multitask learning (MT-RCM). First, both learning methods independently help the agent learn more generalized navigation policy as is evidenced by significant reduction in agent's 's performance gap between seen and unseen environments. For instance, performance gap for agent's goal progress on NDH task drops from 3.85m to 0.92m using multitask learning and agent's 's success rate on VLN task between seen and unseen datasets drops from 9.26% 8.39% using environment- agnostic learning, Second, the two techniques are complementary the agent's performance when trained with both the techniques simultaneously improves on unseen environments compared to when trained separately. Finally, we note here that MT-RCM + EnvÁg outperforms the state-of-the- art goal progress of 2.10m (Thomason et al. 2019) on NDH validation unseen dataset by more than 120%. At the same time, it outpertorms the equivalent RCM baseline (Wang al. (019) of 40.6% success rate by more than 16% (relative measure) on VLN validation unseen dataset.
5.3 MULTITASK LEARNING
Next, we conduct studies to examine cross-task transfer using multitask learning alone. One of the main advantages of multitask learning is that under-represented tokens in each of the individual tasks
We report the performance of the equivalent RCM model without intrinsic reward the henchmark.

---

Under review as a conference paper at ICLR 2020
Table Comparison of agent performance when trained separately vs. jointly on VLN and NDH.
NDH Evaluation
VLN Evaluation
Fold
Model
Inputs for NDH
Progress
PL
NE
SR
SPL
CLS
to A; Q Au 1; Q1a-1
1
6.97
6.92
NDH-RCM
/
6.47
/
6.49
Val Scen
VLN-RCM
10.75
5.09
52.39
48.86
63.91
3.00
11.73
4.87
54.56
52.00 65.64
5.92
11.12
4.62
54.89
52.62
66.05
MT-RCM
5,43
10.94
4.59
54.23
52.06
66.93
5.28
10.63
5.09
56.42
49.67
68.28
1.25
2.69
NDH-RCM
2.69
2.64
Val Unseen
VLN-RCM
10.60
6.10
42.93
38.88
54.86
1.69
13.12
5.84
42.75
38.71
53.09
4.01
11.06
5.88
42.98
40.62
54.30
MT-RCM
3.75
11.08
5.70
44.50
39.67
54.95
4.36
10.23
5.31
46.20
44.19 54.99
Table Comparison of agent performance when language instructions are encoded by separate vs. shared encoder for VINN and NDH tasks.
Val Seen
Val Unseen
Language Encoder
NDH
VLN
NDH
VLN
Progress
PL
NE |
SR f SPL CLS f Progress 1
PL
NE
SR
SPL t CLS
Shared Separate 5.17
5.28
10.63
5.09
56.42 49.67 68.28
4.36
10.23 5,31
46.20
44.19
54.99 54.57
11.26
5.02
52.38 48.80
64.19
4.07
11.72
6.04
43.64
39.49
get a significant boost in the number training samples. Figure 2 illustrates that tokens with less than 40 occurrences end up with sometimes more than 300 occurrences during joint-training.
To examine the impact of dialog history in NDH task, we conduct studies with access to different parts of the dialog the target object for the last oracle answer A,, the prefacing navigator ques- tion Q; and the full dialog history. Table 2 shows the results of jointly training MT-RCM model on VLN and NDH tasks. MT-RCM model learns generalized policy that consistently outperforms the competing model with access to similar parts of the dialog on previously unseen environments. As noted before, multitask learning significantly reduces the gap between the agent's performance on previously seen and unseen environments for both tasks. Furthermore, we see a consistent and gradual increase in the success rate MT-RCMM on VLN task as it is trained on paths with richer di- alog history from the NDH task. This shows that the agent benefits from more complete information about the path implying the importance given by the agent to the language instructions in the task.
We also investigate the impact of parameter sharing of the language encoder for both tasks. As shown Table the model with shared language encoder for NDH and VLN tasks outperforms the model that has separate language encoders for the two tasks, hence demonstrating the importance of parameter sharing during multitask Icarning. A more detailed analysis can be found in the Appendix.
5.4 GEVIRONMENT-AGNOSTIE LEARNING
From Table I can be seen that both VLN and NDH tasks benefit from environment- aggossii learning independently. To further examine the generalization property due environment-agnosti objective, we train model with the opposite objective- to correctly predict the navigation environments by removing the gradient reversal layer (enviromment-avare learning). Interesting

---

Under review as a conference paper at ICLR 2020
Figure 3: t-SNE visualization of trajectory encoder's output (1000 random paths across 11 differ- ent color-coded environments) for models trained with environment-aware objective (left) versus environment-aggnosicc objective (right).
Table 4: Enviromment--gnostiie versus environment-ane -waar learning.
(a) Comparison on NDH.
(b) Comparison on VLN.
Val Seen
Val Unseen
Val Seen
Val Unseen
Model
Model
Progress
Progress
PL.
NE.
SR
SPL
CLS f PL NE. SR
SPL CLS f
RCM
6.49
2.64
RCM
10.75
5.09 52.39 48.86
63.91
10.60
6.10 42,93 38.88
54.86
EnvAware
8.38
1.81
EnvAware
10.30
4.36
$7.59
54.05
68.49
10.13
6.30
38.83
35.65
54.79
EnvAg
6.07
3.15
EnvAg
11.31 4.93
52.79 48.85
63.26
11.36
5.79 44.40 40.30 55.77
results are observed in Table that environment-aware learning leads to overfitting on the training dataset (performance environments seen during training consistently increases for both tasks), while cuvironment-agnostee learning leads to more generalizable policy which performs better on previously unseen environments. Figure further shows that due to environment-ave the model learns represent visual inputs from the same environment closer to each other while the representations of different environments are farther from each other resulting in a clustering learning effect. On the other hand, the environment-agnostte objective leads to more general repre- sentation across different environments which results in better performance unseen environments.
objective,
5.5 REWARD SHAPING FOR NDH TASK
As discussed in Section 3.2 we conducted stud- ies to shape the reward for NDH task. The re- sults in Table 5 indicate that incentivizing the agent to get closer to the goal room better than to the exact goal location, because it is aligned with the objective of NDH task, which to reach the room containing the goal ob- ject. Detailed ablation is presented in Appendix showing that the same holds true consistently the agent is provided access to different parts of the dialog history.
Table 5: Average agent progress towards goal
room when trained using different rewards.
Goal Progress (m)
Model
Val Seen Val Unseen
Shortest-Path Agent Random Agent
9.52
9.58
0.42
1.09
seq2seg Thomason al. 2019
5.92
2.10
Ours
NDH-RCM (dis to goal location)
5.02
2.58
NDH-RCM (dis to goal room)
6.49
2.64
CONCLUSION
In this work, we show that the model trained using euvironment-agnosie multitask learning ap- proach learns a generalized policy for the two natural language grounded navigation tasks. It closes down the gap between seen and unseen environments, learns more generalized environment repre- sentations and effectively transfers knowledge across tasks outperforming baselines on both the tasks simultaneously by significant margin. At the same time, the two approaches independently benefit the agent learning and are complementary each other. There are possible future extensions to our work the MT- HT-RCM can further be adapted to other language- -agoundddg navigation datasets, such as those using Street View (c.g., Touchdown (Chen al. 2019). The Walk (de Vries et al.] 2018)); and complementary techniques like environmental dropout (Tan et al. 2019) can be combined with environment-agnostic learning to learn general representations.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore. Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 16), pp. 265-283, 2016. URL https: http://wwww...// www. org/ system/ 1 6 abadi pdf
Files Teles/ee wtw.//ooeeeeeeecccceeeoceeooeeee cwwnfference/oddi11111
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. arXiv, 2018a. arxiv: 111.v:1000006777
[cs.AI].
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-langeen language navigation: Interpreting visually-gruudeed navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018b.
Michael Bain and Claude Sammut. A framework for behavioural cloning. Machine Intelli- gence 15, Intelligent Agents [St. Catherine's 's College, Oxford, July 1995], pp. 103-129, Ox- ford, UK, UK, 1999. Oxford University. ISBN 0- 0-19-853867-7. URL http: /ttt:://dl. acm. acm.orq/
Citatioon cfm?id=664 67666476.666 733043
Richard Caruana. Multitask learning: A knowledge baasdd source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning, pp. 41-41. Morgan Kaufmann, 1993.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1553--22547 2019.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference Machine Learning, ICML '08, pp. 160-167, New York, NY, USA, 2008 ACM. ISBN 978- 20554. doi: 145/1399999 1390177.
URL http:: acm.
1390156.13901
Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp.
2018
Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv: 1807.0307653 2018.
L. Deng, G. Hinton, and B. Kingsbury. New types of deep neural network learning for speech recognition and related applications: an overview. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 8599-8603, May 2013. doi: 10. 10.1109/ICASS. 2013.6639344.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-leamer architectures. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine
9

---

Under review as a conference paper at ICLR 2020
Learning. volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http::t:pppocceeinngggg///
mlr. www.pppwsss.eeeee http:////w//eepooottll..a..e html
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philiipe Morency. Taylor Berg- Krrr-KKikpattick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-ani -onoo--nd--aaa -ad-aaga navigation. In Neural Information Processing Systems (NeurIPS), 2018.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Pro- ceedings of the 32Nd International Conference on International Conference on Machine Learn- ing- Volume 37, ICML'15, 15, pp. 1180-1189. Jrg,. 2015. URL http://dl acm.orgg citation. cfm? i- 3045118 3045244
R. Girshick. Fast r-cnn. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 1440-1448, Dec 2015. doi: 10.1 19911C.222222
Sachithra Hemachandra, Felix Duvallet, Thomas M Howard, Nicholas Roy, Anthony Stentz, and Matthew R Walter. Learning models for following natural language directions in unknown vironments. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 5608-5615. IEEE, 2015.
Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, and Eugene le. Multi-modal dis- criminative model for vision-and-language navigation. In Proceedings of the Combined Work- shop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP), pp. 40-49, Minneapolis, Minnesota, 2019a. Association for Computational Lin- guistics. doi: 10.18653/1//119-66000 URL https:/ /www.//wcmmmmmom..ooo aclweb. org/anthologyy/
W19 W19-1605
Haoshuo Huang, Vihan Jain, Harsh Mehta, Alexander Ku, Gabriel Magalhães, Jason Baldridge, and Eugene le. Transferable representation learning in vision-and-language navigation. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision|78/ICCCV) 2019b.
Vihan Jain, Gabriel Magalhães, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL, 2019.
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and- language navigation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6741-6749,201
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Yitong Li, Timothy Baldwin, and Trevor Cohn. What's in Domain? Learning Domain-Robuse Text Representations using Adversarial Training. In NAACL-HLT, 2018
Alexander H Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. In Advances in Neural Infor- mation Processing Systems, pp. 2590-2599, 2018
Chih-Yen Yao Ma. Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caim- ing Xiong. Self-monitoring navigation agent via auxiliary progress estimation. arXiv preprint arXiv: 101100030 2019a.
Chih- Yao Ma, Zuxuan Wu, Ghassan AlRegib. Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6732-6740, 2019b.
10

---

Under review as a conference paper at ICLR 2020
Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habi- tat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), 2019.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, and Raia Hadsell. Learn- ing to navigate in cities without a map. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018. URL http: /ttp:://ppppr..a/ n ps. cc cc/paper/ in- citieittieeeii without -arooutt--mppp pdf
N.
pp. 2419-2438
7509-
Piotr W. Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. ArXiv, abs/161 1.03673, 2016.
Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective uuriosity-encouraging imitation learning. arXiv preprint arXiv:1909.0182
2019.
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 12527-12537, 2019.
Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disen- tangled representations. In Proceedings of the 36th International Conference on Machine Learn- ing, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 5102-5112, 2019.
Bharath Ramsundar, Steven M. Kearnes, Patrick Riley, Dale Webster, David E. Konerding, and Vijay S. Pande. Massively multitask networks for drug discovery. ArXiv, abs/1 502.02072, 2015.
Rob Romijnders, Panagiotis Meletis, and Gijs Dubbelman. A domain agnostic normalization layer for unsupervised adversarial domain adaptation. In 2019 JEEE Winter Conference on Applications of Computer Vision ( (WACV), pp. 1866-1875. IEEE, 2019.
Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45:2673-2681, 1997.
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back transla- tion with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 2610-2621, Minneapolis. Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.1186//11N111111
Yee Teh, Victor Bapst, Wojciech M Czarnecki John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4496-4506, 2017.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navi- gation. In Conference on Robot Learning (CoRL), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
Xin Wang. Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridg- ing model-free and model-based reinforcement learning for planned-ahe ahead vision-and and-language navigation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 37-53, 2018.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforeed cross-modal matching and self-supervised imita- tion learning for dision-languse navigation. In Proceedings of the IEÈE Conference on Computer Vision and Pattern Recognition, pp. 6629-6638 6638, 2019.

---

Under review as a conference paper at ICLR 2020
Ronald J. Williams. Simple statistical followig algorithms for connectionist reinforce-
ment learning. Machine Learming, 8(3):229-2555 May 1992. ISSN 1573-0565. doi: 10.1007/ BF00992696.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint vrXii:/801.0220% 20.
Yi Wu, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Learning and planning with a semantic model. arXiv preprint /809 10842, 2018b.
Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE, 2018.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 REWARD SHAPING FOR NDH TASK
Table presents a more detailed ablation of Table 5 using different parts of dialog history. The results prove that agents rewarded for getting closer to the goal room consistently outperform agents rewarded for getting closer to the exact goal location.
Table Average agent progress towards goal room when trained using different rewards and mixed supervision strategy.
Inputs
Goal Progress (m)
Model
to A Qi A11-1 Q14-1
Val Seen Val Unseen
Shortest-Paie Agent Random Agent
9.52
9.58
0.42
Baselines
5.71
1.29
6.04
2.05
2-q-2 Thomason et al.
2019
6.16
1.83
5.92
2.10
4.18
0.42
4.96
2.34
NDH-RCM (distance goal location)
4.60
2.25
5.02
2.58
Ours
6.97
1.25
6.92
2.69
HDH-RCM (distance to goal room)
6.47
2.69
6.49
2.64
A.2 DETAILED ABLATION ON PARAMETER SHARING OF LANGUAGE ENCODER
Table presents a more detailed analysis from 3 with access to different parts of dialog history. The models with shared language encoder consistently outperform those with separate encoders.
Table 7: Comparison of agent performance when language instructions encoded by separate shared encoder for VLN and NDH tasks. All the reported results are averages of 3 independent runs.
NDH Evaluation
VLN Evaluation
Fold
Language Encoder Inputs for NDH
Goal Progress
PL
NE
SR
SPL
CLS
to A, Q, A Q1a=1
3.00
11.73
4.87
54.56
52.00
65.64
5,92
11.12
4.62
54.89
52.62
66.05
Shared
5.43
10.94
4.59
54.23
52.06 66.93
5.28
10.63
5.09
56.42
68.28
Val Seen
2.85
11.43
4.81
54.66
51.11 65.37
4,90
11.92
4.92
53.64
49.79
61.49
Separate
5.07
11.34
4.76
55.34
51.59
65.52
5.17
11.26
5.02
52.38
48.80
64.19
1.69
13.12
5.84
42.75
38.71
53.09
4.01
11.06
5.88
42.98
40.62
54,30
Shared
3.75
11.08
5.70
44.50
39.67
54.95
4.36
10.23
5.31
46.20
44.19
54.99
Val Unseen
1.79
11.85
6.01
42.43
38.19
54.01
3.66
12.59
5.97
43.45
38.62
53.49
Separate
3.51
12.23
5.89
44.40
39.54
54.55
4.07
11.72
6.04
43.64
39.49
54.57
13

---

Under review as a conference paper at ICLR 2020
Performance on VLN
Performance on NDH
Var Seen
Val Unswon
Val Soan
Yel Unsson
9 9.26
8.39
10.22
8,08
6:49
60.2
3.85
2.92
0..92 4:56
10
4.85
9:16
Finugg
Eruug
Enwig
Enwig
Figure 4: Visualizing performance gap between seen and unseen environments for VLN and NDH tasks. For VLN, the plotted metric is agent's success rate while for NDH, the metric is agent's s progress.
A.3 PERFORMANCE GAP BETWEEN SEEN AND UNSEEN ENVIRONMENTS
As mentioned in Section 5.2 both multitask learning as well as environment-agnootie learning meth- ods reduce the agent's performance gap between seen and environments which is demon- strated in Figure 4
14


Reviews:
[{"review_id": "SkgaWnm6Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses some challenges of following natural language instructions for navigating in visual environments. The main challenge in such tasks is the scarcity of available training data, which results in generalization problems where the agent has difficulty navigating in unseen environments. Therefore, the authors propose two key ideas to tackle this issue and incorporate them in the reinforced cross-modal matching (RCM) model (Wang et al, 2019). First, they use a generalized multitask learning model to transfer knowledge across two different tasks: Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH). This results in learning features that explain both tasks simultaneously and hence generalize better. Moreover, by training on both tasks the effective size of training data is increased significantly. Second, they propose an environment-agnostic learning technique in order to learn invariant representations that are still efficient for navigation. This prevents overfitting to specific visual features of the environment and therefore helps improving generalization. The contribution of this paper is combining and incorporating these two key ideas in the RCM framework and verifying it on VLN and NDH tasks. This approach is novel compared to prior results in tackling the VLN task. Their experimental results show that the two proposed techniques improve generalization in a complementary fashion, measured by decreased performance gap between seen and unseen environments.  They demonstrate that their technique outperforms state-of-the-art methods on unseen data on some evaluation metrics.\n\nEven though the two key ideas proposed in the paper has been explored in the literature in other contexts, this paper contributes to natural language guided navigation by incorporating these ideas in a unified framework and demonstrates promising results on two datasets. Therefore, I would recommend accepting this paper if some issues in delivery and clarity are addressed.\n\nIn particular, Section 3, where the authors introduce the novelty of the paper in more detail, could be better explained and a cleaner line should be drawn between prior results from other papers and novel results proposed by this paper. In general, the new ideas would require more emphasis, since they are somewhat lost between adaptations from prior work. Most importantly, Eq. (3) is stated without sufficient motivation and would require a more detailed explanation.\n\nIn addition to these points, I would like to disclose some recommendations that might improve the paper but are not strictly part of my decision. In some cases the notation is not clear and some variables are not defined or explained. For instance, after Eq. (8) Attention(.) is used without citation or definition, in Eq. (9) Wc and Wu are not defined and some of the notation in Eq. (6)-(7) are not defined. Moreover, reading the decrease in performance gap from the presented table format is inconvenient and a better visual representation might help demonstrating the improvement in generalization better.  Lastly, I noticed that the navigation error for shared decoder is slightly higher than for separate encoders in Table 3, even though it outperforms the separate encoders in every other measures. Is there a particular explanation for this?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkepESUaFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to apply the model of Wang 2019 to the new NDH task of Thomason '19.  Both of these datasets are built on the same room-to-room environment and both are for natural language instruction following.  Thomason's work extends the R2R paradigm to include a dialogue history which is collapsed into a single instruction.  The contribution of this paper is to build a single model which alternatingly samples trajectories from each of the two datasets to train a more general actor and the authors also believe that the presence of an environment classifier assists in generalization.\n\nThe claims of the paper focus on being \"environment-agnostic\" and notions of generality.  As hinted by the authors in their future work, to properly show this would probably require two different environments or tasks (e.g. Touchdown), not training on two tasks that use the same environments and pre-computed visual features.  Am I incorrect that the only difference between the NDH and VLN formulation is the structure of the sentences?  \n\nFigure 3 is the most compelling component of the paper.  However, I am still not convinced it will generalize and all other components of the paper are largely minor tweaks to existing work.  \n\nFigure 2 mostly leads me to believe that we have a simple data-augmentation situation which makes the bump in performance somewhat predictable.  Minor: Is there any reason why in Table 1 we can't simply run the VLN models on NDH and NDH on VLN?  \n\nI commend the authors for putting this all together in the two months between the release of CVDN and the ICLR deadline.\nI think training a joint model on these two datasets is a completely natural experiment that many of us wanted to see, and so I appreciate the effort of the authors and the benefit to the community of having these numbers, but I'm not convinced there is that much meat otherwise in this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJxCjHI0KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThere have been two recent related tasks proposed in vision-langauge settings: vision-langauge navigation (VLN) where natural language turn-by-turn instructions must be decoded by an agent in an indoor environment to reach the goal location and Navigation from Dialog History (NDH) where dialog between two humans trying to reach a goal is input to an agent to try to reach a goal location. This paper uses these two tasks' data in a multi-task manner to try to generalize better between indoor environments especially unseen environments which are not in the training set of the agent. \n\nAnother proposed innovation is to use an auxiliary task of environment classification but via a gradient reversal layer such that the learnt latent representation input to the classifier should not overfit to foibles of the environment but should (hopefully) learn a representation that captures the intrinsics of the environment necessary for generalization. \n\nComments:\n\n- The paper is well-written and easy to understand! Experiments are thorough and has all the ablations one would ask for. Thanks!\n\n- Overall I like the paper but have a number of comments:\n\n1. Why not try more sophisticated methods of multi-task learning like 'MetaLearning' by Finn et al 2017 (MAML). It is common knowledge that straight up averaging across tasks is not as effective as doing the bilevel optimization in MAML. \n\n2. Why do RL at all? Already the authors are doing BC (naive form of imitation learning) but they could just do robust imitation learning like DAgger, AggreVateD, etc. The setting is already such that one has a natural oracle (which the authors are already using via BC in the objective) which is the shortest path planner during training time. Then combined with MAML one can do Meta-IL as in 'One-Shot Imitation Learning via Meta-Learning' Finn et al 2017. Note that imitation learning is exponenially more sample efficient than RL and removes all the reward-shaping complications. \n\n3. In Section 2 for error correction \"Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention\" by Nguyen et al. CVPR 2019 is directly relevant. \n\n4. Also for generalizaton performance these papers are directly relevant:\n\n\"Building Generalizable Agents with a Realistic and Rich 3D Environment\" Wu et al ICLR 2018\n\n\"Learning and Planning with a Semantic Model\" Wu et al \n\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a multitask navigation model that can be trained on both vision-language navigation (VLN) and navigation from dialog history (NDH) tasks. The authors provide experiments that demonstrate that their model can outperformance single-task baseline models.

The paper received borderline scores with two weak accept and one weak reject.  Overall, the reviewers found the paper to be well-written and easy to understand, with thorough experiments.

The reviewers had minor concerns about the following:
1. The generalizability of the work.  No results are reported on the test set, only on val.
2. The gains for val unseen are pretty small and there are other models (e.g. Ke et al, Tan et al) that have better results.  Would the proposed environment-agnostic multitask learning be able to improve those models as well?  Or is the gains limited to having a weak baseline?
3. It's unclear if the gains are due to the multitasking or just having more data available to train on.
4. There are some minor issues with the misspellings/typos.  Some examples are given:
Page 1: "Manolis Savva* et al" --> "Savva et al"
Page 5: "x_1, x2, ..., x_3" --> Should the x_3 be something like x_k where k is the length of the utterance?

The AC agrees with the reviewers that the paper is interesting and is mostly solid work.  The AC also feels that there are some valid concerns about the generalizability of the work and that the paper would benefit from a more careful consideration of the issues raised by the reviewers.  The authors are encouraged to refine the work and resubmit.
Decision: Reject