Diagnosing the Environment Bias in Vision-and-Language Navigation
Authors: Yubo Zhang
Hao Tan
Mohit Bansal
Venue:  (2020)
Keywords: vision-and-language navigation
generalization
environment bias diagnosis

Abstract: Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions, explore the given environments, and reach the desired target locations. These step-by-step navigational instructions are extremely useful in navigating new environments which the agent does not know about previously. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicating that the neural agent models are highly biased towards training environments. Although this issue is considered as one of major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment re-splitting and feature replacement, looking into possible reasons of this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results. According to this observation, we explore several kinds of semantic representations which contain less low-level visual information, hence the agent learned with these features could be better generalized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly decrease the performance gap between seen and unseen on multiple datasets (i.e., 8.6% to 0.2% on R2R, 23.9% to 0.1% on R4R, and 3.74 to 0.17 on CVDN) and achieve competitive unseen results to previous state-of-the-art models.

Under review as a conference paper at ICLR 2020
DIAGNOSING THE ENVIRONMENT BIAS IN VISION-ANE AND- NAVIGATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Vision-and-Language Navigation (VLN) requires an agent to follow natural- language instructions, explore the given environments, and reach the desired tar- get locations. These step-by-step navigational instructions are extremely useful in navigating new environments that the agent does not know about previously. Most recent works that study VLN observe a significant performance drop when tested on unseen environments (i.e., environments not used in training), indicat- ing that the neural agent models are highly biased towards training environments. Although this issue is considered one of the major challenges in VLN research, it is still under-studied and needs a clearer explanation. In this work, we design novel diagnosis experiments via environment spllitinnng and feature replacement, looking into possible reasons for this environment bias. We observe that neither the language nor the underlying navigational graph, but the low-level visual ap- pearance conveyed by ResNet features directly affects the agent model and con- tributes to this environment bias results. According explore several kinds of semantic representations which contain less low-level vi- sual information, hence the agent learned with these features could be better gen- eralized to unseen testing environments. Without modifying the baseline agent model and its training method, our explored semantic features significantly de- crease the performance gap between seen and unseen on multiple datasets (i.e.. 8.6% to 0.2% on R2R, 23.9% to 0.1% on R4R, and 3.74 to 0.17 on CVDN) and achieve competitive unseen results to previous state-of oftte-of-theaat theharrt models.
this observation, we
INTRODUCTION
Vision-and-Lange Language Navigation (VLN) tests an agent's 's ability to follow complex natural language instructions as well as explore the given environments, so as to be able to reach the desired target locations. As shown in Fig. [ the agent is put in an environment and given a detailed step-by-step navigational instruction. With these inputs, the agent needs to navigate the environment and find the correct path to the target location. In this work,
focus on the instruction guidddd navigation
(MacMahon et al. [0066] Anderson et al., 20088) 8b Misra et al. 2018 Blukis et al.] 2018 Chen et al.] 2019ch where detailed step-by-step navigational instructions
used (e.g., "Go outside the dining
room and turn left ..."), in contrast to the target-oriented navigation (Gordonn et al., 2018 Das et al.,
2018 Mirowski et al. 2018, Yu et al., 2019) where only the target is referred (e.g.. 'Go to the kitchen" or 'Tell me the color of the bedroom ). Although these step-by-step instructions are over- detailed when navigating local areas (e.g., your home), they are actively used unseen environments (e.g., your friend's 's house, new city) where the desired target is usually unknown to navigational agents. For this purpose, testing on unseen environments which are not used during agent-training is important and widely accepted by instruction-guider navigation datasets.
Recent works propose different methods to improve generalizability of agents on these unseen test- ing environments; and most of the existing works (Anderson et al. [018bb] Wang et al., 2018b] Fried] ett al. 2018 Wang et al.) 2019b Ma et al., 2019a b. Tan et al. 2019 Huang et al. 2019: Hu et al.) 2019 observe significant performance drop from seen environments (i.e., the environments used in training) to unseen environments (i.e., the environments not used in training), which indicates strong bias in the model towards the training environments. While this performance gap is em- phasized as one of the major challenges in current VLN research, the issue is still left unresolved and waits for an explicit explanation. Thus, in this paper, we aim to answer three questions this

---

Under review as a conference paper at ICLR 2020
Failed
vick loro demm
2. lavve sea ad
de triiing bort.
hateectie nom
Train
10r Triae the bedtoom
Test
Test.
Goel
Bail Beiuna Tuas
Succest
bodoues Mat.
Trinning Environments
Unseen Testing Environments
Figure 1: lawlllagnnaaiiggeeeee performance of the agent drops in unseen environments.
environment bias: 1. Where (i.e., in which component) is the bias located? 2. Why does this bias exist? 3. How to eliminate this bias?
To locate where the bias is, start by showing that natural-language navigational instructions and underlying navigational graphs are not direct reasons for this performance gap. We then investigate the effect of environments on the agent's performance. In order to conduct detailed analysis, we re- split the environment and categorize the validation data into three sets based on their visibility to the training set: wath data intersecting with the training paths, path path-unseen data using the training environments but away from the training paths, and env-unseen data using unseen environments (environments not used in training). By showing that the results gradually decrease from path-seen data to env-unseen data, characterize the environment bias at three levels: path level, region level, and environment level.
These three levels of environment biases indicate strong 'spatial localities' in the tasks of VLN, which are intuitively reasonable because environments and regions (c.g.. houses and cities) usually have their own styles when built or decorated. We next want to analyze the detailed reason why this locality would further lead to gap in seen versus unseen results. Our hypothesis is that the low-level information carried by the ResNet features (He et al., 2016) is the reason. To keep min- imal low-level visual information and promote more high-level semanntc information, we replace the ResNet features with the 1000 ImageNet classification probabilities. Although the semantic formation encoded by these features is not accurate because of the shifted domain of images and labels, the same model with ImageNet-L R4R, and CVDN Most importantly, these noisy semantic features
features performs surprisingly well on various VLN
datasets (i.e.. Room-to-Rooe effectively eliminate the performance gap between seen and unseen environments, which suggests that the environment bias is attributed to the ResNet features our hypothesis.
Following the practice in using ImageNet labels as semantic features, we further provide a discussion on how the environment bias could be eliminated. For this, we employ advanced high-level semantic features which are more rational for the VLN domain. We explore three kinds of semantic features: (1) areas of detected object labels (Ren et al. 2015); (2) ground truth semantic views (Chang al.) 2017|; and (3) learned semantic view features. We show that all of these semantic features signif- icantly reduce the environment bias in multiple datasets and also achieve strong results in testing unseen enviromments. We hope this work encourages more investigation and research into improv- ing the generalization of vision-language models unseen real-world scenarios.
RELATED WORK
Vision-and- Languuug Navigation: Vision-and- lann--uudn navigation an emerging task in the vision-and aade--nd--annnn language area. A lot of datasets have been proposed in recent years, such as Room- to o-Room (Anderson et al.) 2018b). Room-for- Room (Jain et al., 2019), TouchDown (Chen et al, 2019ch, CVDN (Thomason al., 2019b), RERERE (Qi al. 2019). House3D) (Wu et al.) 2018 and EQA (Das et al.) 2018). Recent works (Thomaoon et al. 2019a Wang et al. 2018h Fried et al.
We did not test these semantic features on touchdown (Chen et al. 2019c since the images not released.
2

---

Under review as a conference paper at ICLR 2020
2018 Wang et al. 2019b Ma et al. 2019a bj Tan et al. 2019 Hu et al. 2019: Ke et al. 2019 Anderson et al. 2019) focusing on improving the performance of navigation models, especially in unseen testing environments, have helped to increase the navigational success rate.
Domain Adaptation: The general setup of domain adaption contains two sets of data samples
(2)) and yyey from two domains X and Y. Based these samples,
could learn
domain invariant feature with adversarial training (Goodfellow et al. 2014: Zhuu et al.] 2017]] Long
al. 2018 Wang et al. 2019a Hosseini-Asl et 2019; Zhang et al. 2019: Gong et 2019
Chen et al. 2019b) 9b) or learn a transfer function f": X Y (Wang et al. 200 Sa Chen et al. 2019a
Rozantsev et al. 2018). However, samples from the target domain may not be available (e.g., the
lesting environments in navigation should not be used in training) in applications. Thus, we try to give an interpretable explanation to why performance varies in different domains and design a robust feature for it without deliberately considering the target domain. Two methods in VIN, RCM (Wang) jet al. 2019b) and EnvDrop (Tan et al., 2019), explore the possibility of domain adaptation. Both works take the testing environments in training while RCM also uses testing instructions.
Domain Generalization: In domain generalization (Blanchard et al. 2011), the goal is to predict the labels in the previous unseen domain. Similar to the test setting of VLN tasks, the testing data is unrevealed in training. Works have been proposed to Iearn the common features of the training
domain (Muandet et al. 2013: Blanchard et al. 2017 Li et al. 2017 2018 Carlucci et al. 2019 Deshmukh et al., 2019). In this paper, we focus on the domain generalization problem in VI.N task, and try find the reasons for the failures.
3 VISION-AND-LANGUAAG NAVIGATION AND ITS ENVIRONMENT BIAS
We first introduce the task of vision-and-language navigation (VLN) and briefly describe the neural agent models used in our work. We next survey previous works on multiple indoor navigation datasets to show that the environment bias is widely observed in current VLN research. Lastly, we claim that this bias also exists in the outdoor navigation tasks, if the agent tested on unseen regions.
3.1 VISION-AND-LANGUAA NAVIGATION
Tasks: As shown in Fig. | the goal of the VLN task is train agent to navigate certain type of environments [E} (e.g., indoor or outdoor environments) given the instruction I. Each environment E is an independent space, such viewpoint is represented as panoramic image and can be decomposed into separate views {o} as inputs to the neural agent models. The viewpoints and their connectivity form the navigational graph. In practice, after being placed at a particular viewpoint and given the instruction in the beginning, at each time step, the agent can observe the panoramic image of the viewpoint where is located, and choose to move along an edge of the graph to the next node (i.e., viewpoint) or stop- This navigational process produces path (i.c., a list of viewpoints). and the performance of the agent is evaluated by whether it reaches the target location that the instruction indicates in the end.
room house, and consists of set of viewpoints. Each
Neural Agent Models: Most instruction -uideed navigational agents are built based on attentive encoder-decoder models (Bahdanau et al.) 2015). The encoder reads the instructions while the de- coder outputs actions based on the encoded instructions and perceived environments. Since the main purpose of this work is to understand the environment bias in iision-and-language navigation, we use a minimal representative neural agent model that achieves comparable results to previous works. Specifically, we adopt the panoramic-view neural agent model in Fried et al. (2018) ('Fol- lower') with modifications from Tan et al. (2019) as our baseline model. We also exclude advanced training techniques (i.e., reinforcement learning and data augmentation) and only train the agent with imitation learning in all our experiments for the same purpose. More details in original papers.
3.2 ENVIRONMENT BIAS IN INDOOR NAVIGATION
In order to evaluate the generalizability of agent models, indoor vision-ande -add--nn-----aggg navigation datasets (c.g., those collected from Matterport3D (Chang et al.) 2017)) use disjoint sets of environ- ments in training and testing. Most of the datasets provide validation splits to verify the agent's
3

---

Under review as a conference paper at ICLR 2020
Table Results show the performance gap between seen ('Val Seen') and unseen ("Val Unseen') en- vironments in several VLN tasks. Room-to-Room and Sorn-for-Roon are evaluated with 'Success Rate', CVDN is evaluated with 'Goal Progress'. Touchdown is evaluated with 'Task Completion'.
Result
Task
Method
Val Seen
Val Unseen
Abs Gap |
R2R (Anderson et al. 2018b
38.6
21.8
16.8
RPA (Wang al. Kb)
42.9
24.6
18.3
S-Follower (Fred
2018)
66.4
35.5
30,9
RCM (Wang et al
2019b
66.7
42.8
23.9
Room-to- -oom-
SMNA (Ma et al.
2019a)
67
45
22
Anderson al. 2018b
Regretful (Ma
2019b
69
50
19
EnvDrop Tan
2019
62.1
52.2
9.9
ALTR (Huang ct al.
2019
55.8
46.1
9.7
RN+Obj [Flu cl al.
2019)
59.2
39.5
19.7
CG Anderson et al.
2019)
31
31
Our baseline
56.1
47.5
8.6
Our leanned-semantic
53.1
53.3
Speaker-Follower
51.9
23.8
28.1
Room for Room
RCM
55.5
28.6
(Jain et al. 2019
Our baseline
54.6
30.7
23.9
Our learned-semantic
36.2
36.1
0.1
NDH
5.92
2.10
3.82
CVDN Thomason et al. 2019b
Our baseline
5,97
2.23
3.74
Our Garned-semantee
2.60
2.43
0.17
GA (original split)
7.9 (dev)
5.5 (test)
Touchdown
RCONCAT (original split)
9.8 (dev)
10.7 (lest)
Chen et al. 2019c
baseline (original split)
15.0 (dev)
14.2 (test)
Our baseline (seen/unseen split)
17.5
5.3
12.2
performance in both sets of environments: validation seen, which takes the data from training en- vironments, and validation unseen, whose data is from new environments apart from the training environments.
In the first part Table we list most of the previous works on the Room-to-Roo toom dataset (An- derson et al.) 2018b) and report the success rate under greedy decoding (i.e., without beam-search) on validation seen and validation unseen splits. The large absolute gaps (from 30.9% to 9.7%) be- tween the results of seen and unseen environments show that current neural agent models on R2R suffer from environment biag? Besides Room-to-Room (R2R), we also analyze two newly-released indoor navigation datasets that were also collected from Matterport3D environments: Room-for- Room (R4R) (Jain et al. 2019) and Cooperative Vision-and- and---nd-DDllggg Navigation (CVDN) (Thomason) al. 2019bb. As shown in the second and third parts of Table. [ results drop significantly from seen to unseen environments (i.e., 26.9% on R4R and 3.74 on CVDN), indicating that agent models also suffer from the environment bias in these datasets. Lastly, we show the results (denoted as "ours' in Table. [ when the environment bias (reason analyzed in Sec. 5 is effectively eliminated by our learned semantic features (described in Sec. (.3). As result, the performance gaps are ef- fectively decreased on all three datasets without changing the model and learning aaper-parameter, compared to our baselines (denoted 'Our baseline") and previous works
3.3 ENVIRONMENT BIAS IN OUTDOOR NAVIGATION
Since the three indoor navigational datasets in previous sections are collected from the Matterport3D environments (Chang et al. 2017). in order to show that the environment bias is a general phe-
OOu work's 's aim is to both close the seen- unseen while also achieving competitive unseen results. Note that Anderson et al. (2019) also achieve 0% gap but at the trade-off off of low unseen results. There is also another recent work by Ke al. (2019) but they do not report val-seen results from non-heam-searde methods.
As for another major evaluation metric on the R4R dataset, Coverage weighted by Length Score (CLS), also observe similar phenomenon in performance gap; and our methods can also eliminate this gap from CLS results (34.7).
19.2 1.5 and achieve competitive state-of-the-an
4

---

Under review as a conference paper at ICLR 2020
Sean Emsen
8.323
Vall towen
ratto
Data Pan
8.055
8405
BLEU
ROUGE
(a) BLEU 4 distribution
(b) ROUGE L diistributinn
Vil Sumn
Yal Vai Unive
Success rate (%))
8.54
Spicess
8.45
8.49
0.35
BLEU-4
ROUGE_
(c) Success rate w.r.t. BLEU score
(d) Success rate w.r.t. ROUGE score
Figure 2: The language 'distance' distribution (defined by language scores) and its relationship to success rate.
nomenon also existing in other kinds of environments, we investigate the outdoor navigation task from Touchdown dataset (Chen al.) 2019c), whose environments taken from New York City. In the original data splits Touchdown, the environment is not specifically divided into seen and unseen and only involved one city. Thus the trained agent is only tested the training environments (similar to validation seen split). To reveal the environment bias in Touchdown dataset, we split the city environment according to latitude and create two nub-environments: 'training' and 'unseen' The data are then re-split into training, val-seen, and val-unseen, accordingly. We adapt our baseline R2R agent model with additional convolutional layers fit this new task. As shown in the last part of Table. ! when experimenting on the original data split, our baseline model achieves state-of-the- art results on the original 'dev' set and 'test' set, proving the validity of our model in this dataset. However, the results on our re-split data (denoted as 'Our baseline (seen/unseen split)") still show big drop from the 'training' to the 'unseen' sub uub-environment (from 17.5% 5.3%), indicating that environment bias is a broad issue.
4 WHERE: THE EFFECT OF DIFFERENT TASK COMPONENTS
In Sec. 3. we showed that current neural agent models are biased towards the training environments on multiple iision-and-language navigation (VLN) datasets. In this section, our goal is to locate the component of VINN tasks which this environment bias attributed to. As one the early-released and well-explored datasets of VLN, Room-to-Room (R2R) dataset (Anderson et al. (0188) is used as the diagnosing dataset in the experiments. We start by showing that two possible candidates, the natural language instructions and the underlying navigational graph, do not directly contribute the environment bias. Then the effect of visual environments is analyzed in detail.
4.1 THE EFFECT OF NATURAL-LANGUAGE NAVIGATIONAL INSTRUCTIONS
common hypothesis is that the navigational instructions for unseen environments (e.g., val unseen) are much different from the training environments (i.e., training and val seen) due to the different objects and layouts new environments; and this lingual difference thus leads to the performance gap. In this section, we analyze the distributions of success rate with regard to the relationship between validation data's instructions and training instructions. In order to quantitatively evaluate this relationship, we define the 'distances' from validating instruction to all training instructions as the plrase-matching metric. Suppose z is a validating datum, is the training set, and inst (x) is the instruction of the datum x, we ROUGE-L (Lin 2004) and BLEU-4 (Papineni et al. 2002) to

---

Under review as a conference paper at ICLR 2020
X-splining line
Train & Val Seen
Path seen
Val Path-unsren
Re-spliting
Figure 3: Graph split: left is original data and right is re-splitting data. Black vertices are viewpoints visited during training, red paths are val seen val path-seen; blue paths are val path-unseen.
calculate this 'distance":
disgouGE (x, T) min ROUGE-L ROUGE-L (inst(ar), .TT
(1)
disprea T) BLEU-4 (inst(x), (inst(t))
where we consider all the training instructions as references in calculating the BLEU-4 score.
We show the distributions of success rates and distances in Fig. 2 As opposed the hypothesis,
do not observe a significant difference between the distributions of 'distances' (as shown in Fig. 2 (a, b)) on seen validation and unseen validation. For the success rate distributions (in Fig. c.d)), the performance is better on instructions with smaller 'distances' (i.e., higher BLEU-4/ROUGB, scores w.rt. the training instructions) on both validation splits. However, comparing two splits, with the same 'distance' to training instructions, seen validation still significantly outperforms the unseen validation set on success rate, which implies the existence of other reasons rather than language attributed to this performance gap.
4.2 THE EFFECT OF UNDERLYING NAVIGATIONAL GRAPH
As shown in Fig. 3. an environment could be considered as its underlying navigational graph with visual information in Fig. [. In order test whether the agent model could overfit these navigational graphs (and thus be biased towards training environments), we follow the experiments Hu (2019) to train the agent without visual information. Specifically, we mask out the ResNet features with zero vectors thus the agent could only make the decision based on the instructions and the navigational graph. With our baseline model, the success rate is 38.5% on validation seen and 41.0% on validation unseen in this setting, which is consistent with the finding in Hu et al. (2019). Besides showing the relatively good performance of unseen split without visual contents (similar to Thomason et al. (20T9a) and Hu et al. (2019)), we also want to emphasize the low performance gap between seen and unseen environments (2.5% compared to the 10% gap in usual). Hence, we claim that the underlying graph is not dominant reason for the environment bias.
4.3 THE EFFECT OF VISUAL ENVIRONMENTS
To show how the visual environments affect the agent's performance, we analyze the results on unseen environments and in different spatial regions of the training environments. In order to give detailed characterization of the effect of environments, we are going to reveal the spatial localities which are related the agent's performance three different levels:
Path-level Locality: Agents are better at paths which intersect with the training paths.
Region-level Locality: Agents are better in regions which are closer to the training data.
Environment-level Locality: Agents perform better on training environments than on un- seen environments.

---

Under review as a conference paper at ICLR 2020
Table Results on our re-splitting data showing the path-level and environment-level localities.
Validation
Splitting Method
Train
Path-seen
Path-unseen
Env-unseen
R2R
61
56
0
11
Environments
X-split
61
57
16
11
Z-split
61
56
29
11
R2R
14,025
1,020
2,349
Number of Data
X-split
11,631
1,230
1,098
2,349
Z s-pplit
10,894
867
2,324
2,349
R2R
88.3
56.1
47.5
Success Rate
X-split
87.3
58.9
52.6
46.7
Z-split
94.7
62.5
47.8
42.4
And the existence of these spatial locality inspires us to find the direct cause of the problem in Sec. 3.2 However, the original split of data is not fine-grained enough to separately reveal these spatial localities. To better illustrate this, we visualize the data from one environment of the Room- to-Room dataset in Fig. 3 where the vertices are viewpoints with visual information and edges are valid connections between viewpoints. The vertices highlighted with dark-black indicate the view- points which are used in training paths, and the red edges are the connections covered by original val-seen paths. As shown in Fig- 3 nearly all viewpoints in val-seen paths (vertices connected to red lines) are used as viewpoints in training data (vertices marked by dark-black). We thus cannot categorize the path-level and region-level localities. To bypass this, we propose novel re-splitting method to create our diagnosis data splits.
Structural Data Re-splitting We employ two kinds of structural data splitting methods based on the horizontal vertical coordinates, denoted 'X-split' and 'Z-split, respectively. The 'Z- split' intuitively separates different floors in the houses and 'X-split' creates separate areas. When applying to the training environments in R2R dataset, we use side of the splitting line (see the X-splitting line' Fig. 3 as the new training 'environment', and the other side as the path-unseen "environment'. In addition this split of environments, we also re-split the original training data and val-seen data while keeping the val-unseen data the same. The data paths across the splitting line are dropped. As shown in the right part Fig. 3 we create three new data splits: training split, val-path-seen split, and split are highlighted in blue, while the color style of training split and val-path-seen split ('Black' for viewpoints in training and 'Red' for edges in val path-seen) are the same. Since the amount of original val-seen data are inadequate to fill two new validation sets (val path-seen and val path- unseen), we bring some (original) training data into our new validation splits. The overall statistics of original splits and our new splits are shown in Table 2
split. The edges covered by the new val-path-unsee
Existence of Path-level and Environment-level Localities For both splitting methods, we train our baseline model on the newly-split training set and evaluate on our three validation sets (denoted as 'X-split' or 'Z-split' rows Table [ The results of our baseline model on the original R2R (denoted as 'R2R' rows) splits are listed for comparison. As shown in Table. 2 the agent performs better on path-seen than val path-unseen, which suggests that path-level locality exists in current VLN agent models. Meanwhile, the results on val path-unseen are further higher than val env-unseen and indicates the environment-level locality which is independent of the path-level locality.
Existence of the Region-level Locality To further demonstrate region-level locality, we study how the success rate changes different regions of the environment with respect to their distances to the training data, which is similar to the analysis of language 'distance" in Sec. 4. We first calculate the point-by-poin shortest paths using the Dijkstra's 'ss algorithm (Dijkstra 1959). where the shortest distances between viewpoints and e' are denoted the graph distance discRAPu (v, v'). Based on this graph distance, we define the viewpoint distance disviewPOINT from a viewpoint v to the training
WWe only split the environments whose data contains substantial amount, thus make sure that the remaining training data is still adequate for training strong models.
7

---

Under review as a conference paper at ICLR 2020
0.585
Sucrens (4))
8.39
Success
8.46
6.41
15:77
1437
19-02
(meters)
Graph (meters)
(a) X-split
(6) Z-split
Figure 4: The success rate declines as the path moves further from training regions.
data T as v's minimal graph distance to a viewpoint o' in training data. We then define the path distance dispars from validating data the whole training data T the maximal viewpoint distance in the path of 2:
disparu T)
max diSviEwYONN (V) vεpath(a)
(3)
max <Epath(a)
min "Epath()) VIET
diSCRAPU (v, (')
(4)
We compute this path distance between paths in the env-seen validation set and training environ- ments in our re-split data. As shown Fig. 4 the success rate declines as the path moves further from the training environment sion, the closer the path to the training data, the higher the agent performance is, which suggests the existence of region-level locality.
both re-splitting methods (i.e., 'X-split" and 'Z-split'). As conclu-
WHY: WHAT INSIDE THE ENVIRONMENTS CONTRIBUTES TO THE BIAS?
In Sec. we locate the cause of performance gap visual environments by excluding other potential reasons and categorizing the spatial localities. However, there are still multiple possible aspects inside the environment which could lead to these spatial localities, c.g., the object layout convention and the room connections. The agent model could be biased towards the training environments by over-fitting or memorizing these environment-specife characteristics. In this section, we want to identify which aspect directly contributes the bias and draw the following conclusion: the environment bias attributed to low-level visual information carried by the ResNet features.
We first show experiment that effectively decreases the gap between seen and unseen environ- ments with minimal model modifications. We then clarify our conclusions based on the findings.
5.1 AN INVESTIGATION EXPERIMENT: IMAGENET LABELS AS VISUAL FEATURES
Suspecting that the over-fitting happens when the agent over-learns low-level features, we hope to find the replacement of ResNet 2048-features that contain minimal low-level information while pre- serving distinguishable visual contents. The most straightforward replacement is that instead of using mean-pooled features, inject the frozen 1000-way classifying layer in ResNet pre-training, and use the probabilities of ImageNet labels as visual features. Shown as 'ImageNet" in Table 3 the probability distribution almost closes the gap between seen and unseen. These results further con- strain the reason of environment bias to the low-level ResNet features of image views. Combining with the findings of spatial localities, we suggest that environments (i.e., houses) and regions (i.e., rooms) usually have their own 'style'. Thus the same semantic label (captured by ImageNet-1000 features) has different visual appearances (captured by ResNet features) in different environments or regions. As result, ImageNet- 1000 features, in spite of being noisy, not distracted by low-level visual appearance and could generalize to unseen environments, while ResNet features could not.
Although these ImageNet- 1000 features decrease the performance gap. it has disagreement with the VLN domain so that the validation unseen results of R4R and CVDN are slightly worse than baseline (and not much better for R2R). Hence it motivates to find better semantic representations of environmental features that can both close the scen-unseen gap while also achieving state-of-the art unseen results (which we discuss next).
8

---

Under review as a conference paper at ICLR 2020
Table Results showing that our semantic feature representations eliminate the performance gap in all three datasets.
Feature
Result
Task
Type
Name
Dim
Val Seen
Val Unseen
Abs Gap
|
Baseline
ResNet NoDrop
2, 048
54.5
38.2
16.3
Baseline
ResNet
2, 048
56.1
47.5
8.6
Invesgation
ImageNet
1, 000
47.1
48.2
1.1
Room-to-Root to-Room
Semantic
Delection
152
55.9
50.0
5.9
Semantie
Ground Truth
42
55.6
56.2
0.6
Semantic
Learned
42
53.1
53.3
0.2
Baseline
ResNet NoDrop
2, 048
52.5
25.8
26.7
Baseline
ResNet
2,048
54.6
30.7
23.9
Investigation
ImageNet
1. 000
28.7
28.9
0.2
R4R
Semantic
Detection
152
48.8
32.0
16.8
Semantic
Ground Truth
47.6
35.9
11.7
Semantic
Learned
42
36.2
36.1
0.1
Baseline
ResNet NoDrop
2, 048
5.88
2.14
3.74
Baseline
ResNet
2, 048
5.97
2.23
3.74
Invesgation
ImageNet
1, 000
3.22
2.08
1.14
CVDN
Semantic
Delection
152
3.34
2.08
1.26
Semantic
Ground Truth
3.75
2.69
1.06
Semantic
Learned
42
2.60
2.43
0.17
How: METHODOLOGY TO FIX THE ENVIRONMENT BIAS
found that the environment bias is related to the low-level visual features (i.e., 2048-dim ResNet features). Following the findings we observed in Sec. [.] build our agent the features which more correlated the VLN environmental semantics than the ImageNet label features in Sec. 5. We first demonstrate our baseline results on three VLN datasets and then explore the advanced semantic feature replacements. As shown in Table these advanced semantic features could effectively reduce the performance gap between seen and unseen environments and improve the unseen results compared our strong basclines. The effectiveness of these semantic features supports our explanation of the environment bias in Sec. 5 and also suggests that future work in VLN tasks should think about such generalization issues.
In the previous section (Sec.
6.1 BASELINE
In baseline model, following the previous works the standard ResNet features the rep- resentation of environments (Anderson et al.) 2018b: Jain et al.] 2019 Thomason et al.) 2019h).
These features come from the mean-pooled layer after the final convolutional layer of ResNet-
152 (He et al. 2016) pre-trained on ImageNet (Russakovsky et al. 2015). As shown in 'Baseline
of Table. 3 val-seen results are significantly higher than val-unseen results in all three datasets. Note that our baseline method takes the 'feature dropout' technique demonstrated in Tan et al. (2019) (without back translation): the ResNet features are randomly masked by zero before used as inputs of the agent. Without this 'feature dropout' (denoted as 'ResNet NoDrop' in Table. 3 the gaps will increase R2R and R4R, which suggests that this 'feature dropout' technique also helps eliminate the low-level visual information over-fitting we discussed in Sec. However, the performance gap is still large, which leads us to the following discussions semantic features.
6.2 DETECTED OBJECTS AREAS
During navigation, the objects in the environments crucial since their matchings with the instruc- tion often indicate the locations that can guide the agent, thus object detection results of the environ- ments can provide relevant semantic information. In our work, we utilize the detection information generated by Faster R-CNN (Ren et al. 2015)
to create the feature representations. Comparing to
'We use baseline agent model (in Sec.
3.1 for R2R and R4R. For CVDN, we take the official baseline
code in http:://gthuh.co//mmuraay

---

Under review as a conference paper at ICLR 2020
ImageNet- 1000 features (Sec. 5.1 these detection features include more environmental informa- tion since the viewing images in VLN usually contain multiple objects. Instead of directly using classification probabilities of the labels from ResNet and different from the approach in Hu et al. (2019) who utilized the embeddings of detected labels, we design our detection features DETEcT of each image view as the sum of the areas of detected objects weighted by detection confidence:
DEETECT-(aa aeas..... Gcoli
Oee2 Σ Area(obi) Conf(obi)
obj
where the G and ae, are the label and feature of each detected object, Area(*) and Conf( (x) are the area and confidence of each object. For implementation details, we use the Faster R-CNN (Ren
eet al. 2015) trained on Visual Genome (Krishna et al. (017) provided in Bottom-Up Attention (An- derson et al. 2018a). To eliminate the labels irrelevant VLN task, we calculate the total areas of each detection object among all environments and pick the labels that take up relatively large proportion of the environments, creating features of dimension 152 Denoted as 'Detection" in Ta-
3] the performance gap is diminished with these detection features compared to baselines in all three datasets, indicating that changing the features to higher semantic level has positive effect on alleviating the environment bias. Meanwhile, the improvement of unseen validation results on R2R an R4R datasets suggests the better efficiency in the VLN task than the ImageNet labels.
6.3 SEMANTIC SEGMENTATION
Although the detection features can provide adequate semantic information for the agent to achieve comparable results as the baseline model, they do not fully utilize the visual information where the content left over from detection may contain useful knowledge for navigation. A better semantic representation is the semantic segmentation, which segments each view image on the pixel level and gives the label to each segment region, allowing us utilize the semantics from the entire vironment. Matterport3D (Chang et al. 2017) dataset provides the labeled semantic segmentation information of every scene and we take the rendered images from Tan et al. (20197 A compari- son example of RGB images and semantic views is available in the Appendix. Since the semantic segmentation images are fine mne-grained and blurry in boundaries, we follow the design of detection features, using the areas of semantic classes in each image view as the semantic features (confidence excluded since semantic segmentation does not provide this value). The areas are normalized to [0, 1] by dividing the area of the whole image region. We first assume that the semantic information is provided as additional environmental information and the results of the model using the ground truth semantic areas shown in the 'ground truth' rows in Table. We next study the situation where the semantic information is not available in testing environments thus the information needs to be learned from training environments. Thus we train a separate multi-layer perceptron predict the areas of these semantic classes (details in Appendix), and the results of the model with these predicted semantics as features are shown in 'learned'. As shown in Table. 3 both 'ground truth' and 'learned' semantic representations bring the performance of seen and unseen closer comparing to the baseline model, and the smallest performance gaps come from learned semantic segmentation features in all three datasets. The highest validation unseen success rates among all the proposed feature representations are also produced by semantic segmentation features, 'learned' semantic for R4R and 'ground truth' semantic for R2R and CVDN. Overall, among all the semantic represen- tations we have explored, the semantic segmentation features are most effective in eliminating the environment bias.
7 CONCLUSION
In this paper, we focus on studying the performance gap between seen and unseen environments widely observed in saiion-and-language navigation (VLN) tasks, trying to find where and why this environment bias exists and provide possible initial solutions. By designing the diagnosis experi- ments of environment re-splitting and feature replacement, we locate the environment bias to be in the low-level visual appearance; and we discuss semantic features that decrease the performance gap in three VLN datasets and achieve state-of-the-ar results.
6 Note that this detection feature dimension 152 coincidentally is the same as the number of layers ResNet, but there is no correlation.
The rendered semantic views are downloaded from http:://ggttiia.oommmmmyyyyyy/lm
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp-
6077-6086, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-anddlanguage navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-36770 (674-3683, 2018b.
Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, and Stefan Lee. Chasing ghosts: Instruction following as bayesian state tracking. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In The International Conference on Learning Representations (ICLR), 2015.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in neural information processing systems, pp- 2178-2186, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv preprint arXiv:1711.07910 2017.
Valts Blukis, Dipendra Misra, Ross A Knepper, and Yoav Artzi. Mapping navigation instructions to continuous control actions with position-visitation prediction. Conference on Robot Learning, pp. 505-518, 2018.
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2229-2238, 2019.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 2017 International Conference on 3D Vision (3DV), pp. 667-676. IEEE, 2017.
Chao Chen, Zhihong Chen, Boyuan Jiang, and Xinyu Jin. Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9393,, 2019a.
Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong. Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment unsupervised domain adaptation. Pro- ceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 627-636. 2019b.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural language navigation and spatial reasoning in visual street environments. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 15538-12547,, 2019c.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 2054-2063 2018
Aniket Anand Deshmukh, Yunwen Lei, Srinagesh Sharma, Urun Dogan, James W Cutler, and Clay- ton Scott. A generalization error bound for culti-class domain generalization. arXiv preprint arXiv: /905. 10392, 2019.
Edsger W Dijkstra, A note on two problems in connexion with graphs. Numerische mathematik, (1):269-271, 1959.
11

---

Under review as a conference paper at ICLR 2020
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor werg-Kirkpatrick. Kate Saenko, Dan Klein, and Trevor Darrell. speaker-followee models for and-language navigation. In Advances in Neural Information Processing Sys- tems, pp. 3314-3325, 2018.
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and gener- alization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2477-2486, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- 2014.
mation processing systems, pp.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering interactive environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4089-4098, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp- 770-778, 2016.
Ehsan Hosseini-Asi, Yingbo Zhou, Caiming Xiong, and Richard Socher. Augmented cyclic adver- sarial learning for low resource domain adaptation. In The International Conference on Learning Representations (ICIR), 2019.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Kate Saenko, et al. Are you looking? grounding to multiple modalities laion-and-languuge navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Haoshuo Huang, Vihan Jain, Harsh Mchta, Alexander Ku, Gabriel Magalhaes, Jason Baldridge, and Eugene le. Transferable representation learning in rision-and-languag navigation. In IEEE International Conference on Computer Vision, 2019.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-land lan-uage navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), 2019.
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, and Siddhartha Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and- language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6741-6749, 2019..
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
12

---

Under review as a conference paper at ICLR 2020
Chih- Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self- monitoring navigation agent via auxiliary progress estimation. The International Conference on Learning Representations (ICLR), 2019a.
Chih- Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caiming Xiong, and Zsolt Kira. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6732-6740, 2019b.
Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: connecting language, knowledge, and action in route instructions. In proceedings of the 21st national conference Artificial intelligence- Volume 2. pp. 1475-1482. AAAI Press, 2006.
Piotr Mirowski. Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without a map. In Advances in Neural Information Processing Systems, pp. 2018.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2667-2678, 2018.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-31111 311-318. Association for Computational Linguistics, 2002.
Yuankai Qi, Qi Wu, Peter Anderson, Marco Liu, Chunhua Shen, and Anton van den Hengel. Rerere: Remote embodied referring expressions in real indoor environments. arXiv preprint arXiv: 10151, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Artem Rozantsev, Mathieu Salzmann, and Pascal Fua. Residual parameter transfer for deep domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4339 4348, 2018.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back transla- tion with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Papers), pp. 1610-26221) 2019.
Jesse Thomason, Daniel Gordan, and Yonatan Bisk. Shifting the baseline: Single modality perfor- mance on visual navigation & qa. In NACCL, 2019a.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navi- gation. In CoRL, 2019b.
Daixin Wang, Peng Cui, and Wenwu Zhu, Deep asymmetric transfer network for unbalanced domain adaptation. In -hirty-Second AAAI Conference Artificial Intelligence, 201 8a.
Ximei Wang. Liang Li, Weirui Ye, Mingsheng Long, and Jianmin Wang. Transferable attention for domain adaptation. In AAAI Conference on Artificial Intelligence (AAAI), 2019a.
13

---

Under review as a conference paper at ICLR 2020
Xin Wang. Wenhan Xiong. Hongmin Wang, and William Yang Wang. Look before you leap: Bridg- ing model-free and model-based reinforcement learning for planned-ahes -heed vision-an and-language navigation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 37-53, 201
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang. William Yang Wang, and Lei Zhang. Reinforeed cross cross-modal matching and sepf-supe imita- tion learning for vision-language navigation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6629-66388 2019b.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich environment. arXiv preprint arXiv:180) 02209, 2018.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi- target embodied question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6309-6318, 2019.
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5031-5040, 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A APPENDIX
RGB Images
Semantic Views
Figure 5: Comparisons between RGB images and their semantic views.
A.1 EXAMPLES OF RGB IMAGES AND SEMANTIC VIEWS
In Fig. 5 we show a rendered semantic view from Tan et al. (019) and its original RGB image. Dif- ferent colors indicate different semantic segmentation areas and 40 semantic labels considered in the Matterport3D dataset CChang et al. (2017).
A.2 DETAILS OF 'LEARNED' SEMANTIC TRAINING
We use a multi-layer perceptron over the ResNet features to generate the 'learned' semantic fea- tures. The multi-layer perceptron includes three fully fully-connected layers with ReLU activation on the outputs of the first two layers. The input is the 2048-dim ResNet feature f of each image view. The hidden sizes of the first two layers are 512 and 128. The final layer will output the 42-dim semantic feature y that represents the areas of each semantic class. After the linear layers, we use the sigmoid function a to convert the output to the ratio of areas.
Φ ReLU +b)
(6)
Zy = ReLU + bz)
(7)
y (Ag22 + bs)
(8)
14

---

Under review as a conference paper at ICLR 2020
The model is trained with ground truth semantic areas VARBa (normalized to [0, 1)) and only the views in training environments are used in training. We minimize the binary cross-entropy loss between the ground truth areas {y,} } and the predicted areas {vi), where indicate the i-th semantic class.
f= Σ (U. log + (1 v!" log (1 - -y)))
Dropout layers with a probability of 0.5 are added between fully-connected layers while training. The sigmoid function a and the cross-entropy loss are combined improve numerical stability.
After the model is fitted, we freeze the weight and use it to predict the semantic features of all seen and unseen environments (i.e., environments for training, val-seen, and val-unseen data). The predicted features are then used the input of our neural agent model for different datasets (i.e.,
R2R, R4R, and CVDN), and the neural agent models are the same except we change the input dimension from 2048 (the dimension of ResNet features) to 42 (the number of semantic classes).
15


Reviews:
[{"review_id": "r1l12CqnYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to identify the primary source of transfer error in vision&language navigation tasks in unseen environments. The authors tease apart the contributions of the out-of-distribution severity of language instructions, navigation graph (environmental structure), and visual features, and conclude that visual differences are the primary form in which unseen environments are out of distribution. They show that using ImageNet class scores as visual features results in significantly less transfer gap than using low-level visual features themselves. Experiments then show that semantic-level features dramatically reduce the transfer gap, although at a cost of absolute performance.\n\nI recommend this paper for acceptance; my decision is based on the thorough analysis of the ultimate cause of a recurring problem in this field.\n\nThese results, if shown to hold across a significant number of datasets and tasks, would significantly change the focus of research in this field toward a focus on robust high-level visual representations (as opposed to e.g. better spatial awareness or better language understanding). This work represents an important step in this direction.\n\nThe description of the 'learned' features in 6.3 could use more elaboration. Since it is the best performing approach by a large margin (as measured by transfer gap), it should probably get more than one sentence. In particular, what do the authors mean by \"train a separate multi-layer perceptron to predict the areas of these semantic labels\"? Does that mean the predicted pixel-level semantic segmentation map is used as input to the navigating agent? Or is it an auxiliary task for representation learning? etc. This should be clarified.\n\nI anticipate this paper to significantly influence future work in this area.\n\n--------\n\nAfter discussing with the reviewers about the methodological issue of the validation set, I have lowered my score to a weak accept, but I think this paper should still be published.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gbNt0nYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has two main contributions. First, the authors perform an extensive study to understand the source of what they refer to as 'environment bias', which manifests itself as a gap in performance between environments used for training and unseen environments used for validation. The authors conclude that of the three sources of information provided to the agent (the natural language instruction, the graph structure of the environment, and the RGB image), the RGB image is the primary source of the overfitting. The second contribution is to use semantic information, compact statistics derived from (1) detected objects and (2) semantic segmentation, to replace the RGB image and provide input to the system in a way that maintains state-of-the-art performance but shrinks the performance gap between the seen and unseen data.\n\nThis paper has some pretty exhaustive treatment diagnosing the source of the agent's 'environment bias' (which, as I discuss below, I believe is more accurately referred to as 'overfitting') in Sec. 4. To me, this is this highlight of the paper, and some interesting work; the investigation of the behavior of the system is interesting and informative. It provides a framework for thinking about how to diagnose this behavior and identify its source. The authors use this rather extensive study to motivate the need for new features (semantic features) to replace the RGB image that their investigation finds is where much of this 'environment bias' is located. Unfortunately, it is here that the paper falls flat. The authors proposal methods perform nominally better on the tasks being investigated, but much of the latter portion of the paper continues to focus on the 'improvement' in the metric they use to diagnose the 'bias'. As I mention below, the metric for success on these tasks is performance on the unseen data, and, though an improvement on their 'bias' metric is good anecdotal evidence their proposed methods are doing what they think, the improvements in this metric are largely due to a nontrivial decrease in performance on the training data. Ultimately, this is not a compelling reason to prefer their method. I go into more details below about where I think some of the other portions of the paper could be improved and include suggestions for improvement.\n\nHigh-level comments:\n- I am uncertain that 'bias' is the right word to describe the effect under study. In my experience, environment bias (or, more generally, dataset bias) usually implies that the training and test sets (or some subset of the data) are distinct in some way, that they are drawn from different distributions. The learning system cannot identify these differences without access to the test set, resulting in poor performance on the 'unseen' data. In the scenario presented here, the environments are selected to be in the train/test/validation sets at random. As such, the behavior described here is probably more appropriately described as 'overfitting'. The shift in terminology is not an insignificant change, because using 'bias' to describe the problem incorrectly suggests that the data collection procedure is to blame, rather than a lack of data or an overparamatrized learning strategy; I imagine that more data in the training set (if it existed) could help to reduce the gap in performance the paper is concerned with. That being said, I imagine some language changes could be done to remedy this.\n- Perhaps the biggest problem with the paper as written is that I am not convinced that the 'performance gap' between the seen and unseen data is a metric I should want to optimize. This metric is instructive for diagnosing which component of the model the overfitting is coming from, and Sec. 4 (devoted to a study of this effect) is an interesting study as a result. However, beyond this investigation, reducing the gap between these two is not a compelling objective; ultimately, it is the raw performance on the unseen data that matters most. The paper is written in a way that very heavily emphasizes the 'performance gap' metric, which gets in the way of its otherwise interesting discussion diagnosing the source of overfitting and some 'strong' results on the tasks of interest. The criteria should be used to motivate newer approaches, rather than the metric we should value for its adoption. This narrative challenge is the most important reason I cannot recommend this paper in its current state.\n- Using semantic segmentation, rather than the RBG image, as input seems like a good idea, and the authors do a good job of motivating the use of semantics (which should show better generalization performance) than a raw image. However, the implementation in Sec. 6.3 raises a few questions. First (and perhaps least important) is that 6.3 is missing some implementation details. In this section, the authors mention that 'a multilayer perceptron is used' but do not provide any training or structure details; these details should be included in an appendix. More important is the rather significant decrease in performance on the seen data (11% absolute) when switching to the learned method. Though the performance on the unseen data does not change much, it raises some concerns about the generalizability of the learning approach they have used: in an ideal world with infinite training data, the network would perfectly accurately reproduce the ground truth results, and there should be no difference between the two. Consequently, the authors should comment on the discrepancy between the two and the limits of the learned approach, which I worry may limit its efficacy if more training data were added.\n\nSmaller comments:\n- I do not fully understand why the 'Touchdown' environment was included in Table 1, since the learned-semantic agent proposed in the paper was not evaluated. The remainder of the experiments are sufficient to convince the reader that this gap exists, and I would recommend either evaluating against the proposed technique or removing this task from the paper.\n- Figure captions should be more 'self-contained'. Right now, they describe only what is shown in the figure. They should also describe what I, as a reader, should take away or learn from the figure. This is not always necessary, but in my experience improves readability, so that the reader does not need to return to the body of the text to understand.\n- The use of a multilayer perceptron for the Semantic Segmentation learned features, trained from scratch, stands out as a strange choice, when there are many open source implementations for semantic segmentation exist and could be fine-tuned for this task; a complete investigation (which may be out of scope for the rebuttal period) may require evaluating performance of one of these systems.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeuvT_0Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper provides a thorough analysis of why vision-language navigation (VLN) models fail when transferred to unseen environments. The authors enumerate potential sources of the failure--namely, the language, the semantic map, and the visual features--and show that the visual features are most clearly to blame for the failures. Specifically, they show that by removing the low-level visual features (e.g. the fc17 or similar) and replacing with various higher-level representations (e.g. the softmax layer of the pretrained CNN, or the output of a semantic segmentation system) dramatically improves generalization without a meaningful drop in absolute performance.\n\nEvaluation: The paper is easy to follow and interesting. Some results presented have been show previously (e.g. that removing visual features doesn't drastically hurt performance of VLN models) but overall, the paper presents the results in a clear and thorough manner that will be beneficial to the community. A few small questions/comments below.\n\n* I am confused by how you compute BLEU in Section 4.1. You say you compute corpus BLEU but Eq. 2 suggests you compute the BLEU for a single instruction against a set of training instructions. I think corpus BLEU is usually corpus vs. corpus (e.g. all generated sentences vs. all reference sentences) not one generated sentence against all reference sentences. Is this right? It also seems odd that your BLEU scores are distributed the way they are (Fig. 2). Can you explain why you did this the way you did?\n* nit: Sec. 5 heading. Your grammar is backwards. The question you are trying to express is \"bias is attributed to what\" not \"what is attributed to bias\". So heading should be \"to what inside the environments is bias attributed\" (which is admittedly a clunky title)\n* another nit: \"suggest a surprising conclusion: the environment bias is attributed to low-level visual information carried by the ResNet features.\" --> idk that this is that surprising, it was kind of natural given the result that removing visual features entirely doesn't hurt performance and helps generalization. So maybe rephrase this sentence.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission is a detailed and extensive examination of overfitting in vision-and-language navigation domains. The authors evaluate several methods across multiple environments, using different splits of the environment data into training, validation-seen, and validation-unseen. The authors also present an approach using semantic features which is shown to have little or no gap between training and validation performance. 

The reviewers had mixed reviews and there was substantial discussion about the merits of the paper. However, a significant issue was observed and confirmed with the authors, relating to tuning the semantic features and agent model on the unseen validation data. This is an important flaw, since the other methods were not tuned in this way, and there was no 'test' performance given in the paper. For this reason, the recommendation is to reject the paper. The authors are encouraged to fairly compare all models and resubmit their paper at another venue.
Decision: Reject