Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Overview]
In this paper, the authors proposed a unified model for combining vision, language, and action. It is aimed at controlling an agent in a virtual environment to move to a specified location in a 2D map, and answer user's questions as well. To address this problem, the authors proposed an explicit grounding way to connect the words in a sentence and spatial regions in the images. Specifically, By this way, the model could exploit the outputs of concept detection module to perform the actions and question answering as well jointly. In the experiments, the authors compared with several previous attention methods to show the effectiveness of the proposed concept detection module and demonstrated its superiority on several configurations, including in-domain and out-of-domain cases.

[Strengths]

1. I think this paper proposed interesting tasks to combine the vision, language, and actions. As we know, in a realistic environment, all three components are necessary to complete a complex tasks which need the interactions with the physical environments. The authors should release the dataset to prompt the research in this area.

2. The authors proposed a simple method to ground the language on visual input. Specifically, the authors grounded each word in a sentence to all locations of the visual map, and then perform a simple concept detection upon it. Then, the model used this intermediate representation to guide the navigation of agent in the 2D map and visual question answering as well.

3. From the experiments, it is shown that the proposed model outperforms several baseline methods in both normal tasks and out-of-domain ones. According to the visualizations, the interpreter could generate meaningful attention map given a textual query.

[Weakness]

1. The definition of explicit grounding is a bit misleading. Though the grounding or attention is performed for each word at each location of the visual map. It is a still kind of soft-attention, except that is performed for each word in a sentence. As far as I know, this has been done in several previous works, such as: (a). Hierarchical question-image co-attention for visual question answering (https://scholar.google.com/scholar?oi=bibs&cluster=15146345852176060026&btnI=1&hl=en). Lu et al. NIPS 2016. (b). Graph-Structured Representations for Visual Question Answering. Teney et al. arXiv 2016. At most recent, we have seen some more explicit way for visual grounding like: (c). Bottom-up and top-down attention for image captioning and VQA (https://arxiv.org/abs/1707.07998). Anderson et al. arXiv 2017.

2. Since the model is aimed at grounding the language on the vision based on interactions, it is worth to show how well the final model could ground the text words to each of the visual objects. Say, show the affinity matrix between the words and the objects to indicate the correlations.

[Summary]

I think this is a good paper which integrates vision, language, and actions in a virtual environment. I would foresee more and more works will be devoted to this area, considering its close connection to our daily life. To address this problem, the authors proposed a simple model to ground words on visual signals, which prove to outperform previous methods, such as CA, SAN, etc. According to the visualization, the model could attend the right region of the image for finishing a navigation and QA task. As I said, the authors should rephrase the definition of explicit grounding, to make it clearly distinguished with the previous work I listed above. Also, the authors should definitely show the grounding attention results of words and visual signal jointly, i.e., showing them together in one figure instead of separately in Figure 9 and Figure 10.

review_title: Review from AnonReviewer3

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a new task that combines elements of instruction following
and visual question answering: agents must accomplish particular tasks in an
interactive environment while providing one-word answers to questions about
features of the environment. To solve this task, the paper also presents a new
model architecture that effectively computes a low-rank attention over both
positions and feature indices in the input image. It uses this attention as a
common bottleneck for downstream predictors that select actions and answers to
questions. The paper's main claim is that this model architecture enables strong
generalization: it allows the model to succeed at the instruction following task
even when given words it has only seen in QA contexts, and vice-versa.
Experiments show that on the navigation task, the proposed approach outperforms
a variety of baselines under both a normal data condition and one requiring
strong generalization.

On the whole, I think this paper does paper does a good job of motivating the
proposed modeling decisions. The approach is likely to be useful for other
researchers working on related problems. I have a few questions about the
evaluation, but most of my comments are about presentation.

EVALUATION

Is it really the case that no results are presented for the QA task, or am I
misreading one of the charts here? Given that this paper spends a lot of time
motivating the QA task as part of the training scenario, I was surprised not to
see it evaluated. 

Additionally, when I first read the paper I thought that the ZS1 experiments
featured no QA training at all. However, your response to one of the sibling
comments suggests that it's still a "mixed" training setting where the sampled
QA and NAV instances happen to cover the full space. This should be made more
clear in the paper. It would be nice to know (1) how the various models perform
at QA in both ZS1 and ZS2 settings, and (2) what the actual performance is NAV
alone (even if the results are terrible).

MODEL PRESENTATION

I found section 2 difficult to read: in particular, the overloading of \Phi
with different subscripts for different output types, the general fact that
e.g. x and \Phi_x are used interchangeably, and the large number of different
variables. My best suggestions are to drop the \Phis altogether and consider
using text subscripts rather than coming up with a new name for every variable,
but there are probably other things that will also help.

OTHER NOTES

- This paper needs serious proofreading---just in the first few pages the errors
  I noticed were "in 2D environment" (in the title!), "such capability", "this
  characteristics", "such language generalization problem", "the agent need to",
  "some early pioneering system", "commands is". I gave up on keeping track at
  this point but there are many more.

- \phi in Fig 2 should be explained by the caption.

- Here's another good paper to cite for the end of 2.2.1:
  https://arxiv.org/pdf/1707.00683.pdf.

- The mechanism in 2.2.4 feels a little like
  http://aclweb.org/anthology/D17-1015

- I don't think the content on pages 12, 13, and 14 adds much to the
  paper---consider moving these to an appendix.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces XWORLD, a 2D virtual environment with which an agent can constantly interact via navigation commands and question answering tasks. Agents working in this setting therefore, learn the language of the "teacher" and efficiently ground words to their respective concepts in the environment. The work also propose a neat model motivated by the environment and outperform various baselines. 

Further, the paper evaluates the language acquisition aspect via two zero-shot learning tasks -- ZS1) A setting consisting of previously seen concepts in unseen configurations ZS2) Contains new words that did not appear in the training phase. 

The robustness to navigation commands in Section 4.5 is very forced and incorrect -- randomly inserting unseen words at crucial points might lead to totally different original navigation commands right? As the paper says, a difference of one word can lead to completely different goals and so, the noise robustness experiments seem to test for the biases learned by the agent in some sense (which is not desirable). Is there any justification for why this method of injecting noise was chosen ? Is it possible to use hard negatives as noisy / trick commands and evaluate against them for robustness ?  

Overall, I think the paper proposes an interesting environment and task that is of interest to the community in general. The modes and its evaluation are relevant and intuitions can be made use for evaluating other similar tasks (in 3D, say). 
review_title: interesting contribution

Meta-review: This manuscript was reviewed by 3 expert reviewers and their evaluation is generally positive. The authors have responded to the questions asked and the reviewers are satisfied with the responses. Although the 2D environments are underwhelming (compared to 3D environments such as SUNCG, Doom, Thor, etc), one thing that distinguishes this paper from other concurrent submissions on the similar topics is the demonstration that "words learned only from a VQA-style supervision condition can be successfully interpreted in an instruction-following setting." 
Decision: Accept (Poster)