Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: This paper provides a thorough analysis of why vision-language navigation (VLN) models fail when transferred to unseen environments. The authors enumerate potential sources of the failure--namely, the language, the semantic map, and the visual features--and show that the visual features are most clearly to blame for the failures. Specifically, they show that by removing the low-level visual features (e.g. the fc17 or similar) and replacing with various higher-level representations (e.g. the softmax layer of the pretrained CNN, or the output of a semantic segmentation system) dramatically improves generalization without a meaningful drop in absolute performance.

Evaluation: The paper is easy to follow and interesting. Some results presented have been show previously (e.g. that removing visual features doesn't drastically hurt performance of VLN models) but overall, the paper presents the results in a clear and thorough manner that will be beneficial to the community. A few small questions/comments below.

* I am confused by how you compute BLEU in Section 4.1. You say you compute corpus BLEU but Eq. 2 suggests you compute the BLEU for a single instruction against a set of training instructions. I think corpus BLEU is usually corpus vs. corpus (e.g. all generated sentences vs. all reference sentences) not one generated sentence against all reference sentences. Is this right? It also seems odd that your BLEU scores are distributed the way they are (Fig. 2). Can you explain why you did this the way you did?
* nit: Sec. 5 heading. Your grammar is backwards. The question you are trying to express is "bias is attributed to what" not "what is attributed to bias". So heading should be "to what inside the environments is bias attributed" (which is admittedly a clunky title)
* another nit: "suggest a surprising conclusion: the environment bias is attributed to low-level visual information carried by the ResNet features." --> idk that this is that surprising, it was kind of natural given the result that removing visual features entirely doesn't hurt performance and helps generalization. So maybe rephrase this sentence.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper has two main contributions. First, the authors perform an extensive study to understand the source of what they refer to as 'environment bias', which manifests itself as a gap in performance between environments used for training and unseen environments used for validation. The authors conclude that of the three sources of information provided to the agent (the natural language instruction, the graph structure of the environment, and the RGB image), the RGB image is the primary source of the overfitting. The second contribution is to use semantic information, compact statistics derived from (1) detected objects and (2) semantic segmentation, to replace the RGB image and provide input to the system in a way that maintains state-of-the-art performance but shrinks the performance gap between the seen and unseen data.

This paper has some pretty exhaustive treatment diagnosing the source of the agent's 'environment bias' (which, as I discuss below, I believe is more accurately referred to as 'overfitting') in Sec. 4. To me, this is this highlight of the paper, and some interesting work; the investigation of the behavior of the system is interesting and informative. It provides a framework for thinking about how to diagnose this behavior and identify its source. The authors use this rather extensive study to motivate the need for new features (semantic features) to replace the RGB image that their investigation finds is where much of this 'environment bias' is located. Unfortunately, it is here that the paper falls flat. The authors proposal methods perform nominally better on the tasks being investigated, but much of the latter portion of the paper continues to focus on the 'improvement' in the metric they use to diagnose the 'bias'. As I mention below, the metric for success on these tasks is performance on the unseen data, and, though an improvement on their 'bias' metric is good anecdotal evidence their proposed methods are doing what they think, the improvements in this metric are largely due to a nontrivial decrease in performance on the training data. Ultimately, this is not a compelling reason to prefer their method. I go into more details below about where I think some of the other portions of the paper could be improved and include suggestions for improvement.

High-level comments:
- I am uncertain that 'bias' is the right word to describe the effect under study. In my experience, environment bias (or, more generally, dataset bias) usually implies that the training and test sets (or some subset of the data) are distinct in some way, that they are drawn from different distributions. The learning system cannot identify these differences without access to the test set, resulting in poor performance on the 'unseen' data. In the scenario presented here, the environments are selected to be in the train/test/validation sets at random. As such, the behavior described here is probably more appropriately described as 'overfitting'. The shift in terminology is not an insignificant change, because using 'bias' to describe the problem incorrectly suggests that the data collection procedure is to blame, rather than a lack of data or an overparamatrized learning strategy; I imagine that more data in the training set (if it existed) could help to reduce the gap in performance the paper is concerned with. That being said, I imagine some language changes could be done to remedy this.
- Perhaps the biggest problem with the paper as written is that I am not convinced that the 'performance gap' between the seen and unseen data is a metric I should want to optimize. This metric is instructive for diagnosing which component of the model the overfitting is coming from, and Sec. 4 (devoted to a study of this effect) is an interesting study as a result. However, beyond this investigation, reducing the gap between these two is not a compelling objective; ultimately, it is the raw performance on the unseen data that matters most. The paper is written in a way that very heavily emphasizes the 'performance gap' metric, which gets in the way of its otherwise interesting discussion diagnosing the source of overfitting and some 'strong' results on the tasks of interest. The criteria should be used to motivate newer approaches, rather than the metric we should value for its adoption. This narrative challenge is the most important reason I cannot recommend this paper in its current state.
- Using semantic segmentation, rather than the RBG image, as input seems like a good idea, and the authors do a good job of motivating the use of semantics (which should show better generalization performance) than a raw image. However, the implementation in Sec. 6.3 raises a few questions. First (and perhaps least important) is that 6.3 is missing some implementation details. In this section, the authors mention that 'a multilayer perceptron is used' but do not provide any training or structure details; these details should be included in an appendix. More important is the rather significant decrease in performance on the seen data (11% absolute) when switching to the learned method. Though the performance on the unseen data does not change much, it raises some concerns about the generalizability of the learning approach they have used: in an ideal world with infinite training data, the network would perfectly accurately reproduce the ground truth results, and there should be no difference between the two. Consequently, the authors should comment on the discrepancy between the two and the limits of the learned approach, which I worry may limit its efficacy if more training data were added.

Smaller comments:
- I do not fully understand why the 'Touchdown' environment was included in Table 1, since the learned-semantic agent proposed in the paper was not evaluated. The remainder of the experiments are sufficient to convince the reader that this gap exists, and I would recommend either evaluating against the proposed technique or removing this task from the paper.
- Figure captions should be more 'self-contained'. Right now, they describe only what is shown in the figure. They should also describe what I, as a reader, should take away or learn from the figure. This is not always necessary, but in my experience improves readability, so that the reader does not need to return to the body of the text to understand.
- The use of a multilayer perceptron for the Semantic Segmentation learned features, trained from scratch, stands out as a strange choice, when there are many open source implementations for semantic segmentation exist and could be fine-tuned for this task; a complete investigation (which may be out of scope for the rebuttal period) may require evaluating performance of one of these systems.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper aims to identify the primary source of transfer error in vision&language navigation tasks in unseen environments. The authors tease apart the contributions of the out-of-distribution severity of language instructions, navigation graph (environmental structure), and visual features, and conclude that visual differences are the primary form in which unseen environments are out of distribution. They show that using ImageNet class scores as visual features results in significantly less transfer gap than using low-level visual features themselves. Experiments then show that semantic-level features dramatically reduce the transfer gap, although at a cost of absolute performance.

I recommend this paper for acceptance; my decision is based on the thorough analysis of the ultimate cause of a recurring problem in this field.

These results, if shown to hold across a significant number of datasets and tasks, would significantly change the focus of research in this field toward a focus on robust high-level visual representations (as opposed to e.g. better spatial awareness or better language understanding). This work represents an important step in this direction.

The description of the 'learned' features in 6.3 could use more elaboration. Since it is the best performing approach by a large margin (as measured by transfer gap), it should probably get more than one sentence. In particular, what do the authors mean by "train a separate multi-layer perceptron to predict the areas of these semantic labels"? Does that mean the predicted pixel-level semantic segmentation map is used as input to the navigating agent? Or is it an auxiliary task for representation learning? etc. This should be clarified.

I anticipate this paper to significantly influence future work in this area.

--------

After discussing with the reviewers about the methodological issue of the validation set, I have lowered my score to a weak accept, but I think this paper should still be published.
review_title: Official Blind Review #1

Meta-review: The submission is a detailed and extensive examination of overfitting in vision-and-language navigation domains. The authors evaluate several methods across multiple environments, using different splits of the environment data into training, validation-seen, and validation-unseen. The authors also present an approach using semantic features which is shown to have little or no gap between training and validation performance. 

The reviewers had mixed reviews and there was substantial discussion about the merits of the paper. However, a significant issue was observed and confirmed with the authors, relating to tuning the semantic features and agent model on the unseen validation data. This is an important flaw, since the other methods were not tuned in this way, and there was no 'test' performance given in the paper. For this reason, the recommendation is to reject the paper. The authors are encouraged to fairly compare all models and resubmit their paper at another venue.
Decision: Reject