Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I thank the authors for their detailed response and appreciate their hard work in bringing us this paper. 

I think that my main point is that this work relies too much on the extra information/constraints in the synthetic env. E.g., 1. since the vocab size is small, thus the feature map could be designed 'equal to the vocabulary size' 2. The bag-of-words representation is effective but it is not the case for natural language. Although the authors kindly point me to some recent works on sim2real, I am still not convinced whether this proposed method could be transferred to real setups based on the referenced papers.

However, it is a personal research taste that I always take real setup into considerations, because I have worked on both synthetic and real setup (on both lang and visn sides) for years and observed a large gap. My opinion is that methods of synthetic setups are not naturally convertible to the real ones. If AC/meta-reviewer considers the ability of vision-and-language interactions could be effectively studied through this setup with synthetic language and simulated-unrealistic images, I am OK with acceptance. I have downgraded my confidence scores (but kept my overall score) for this purpose.


-----------------------------------------------------------------------------------

Pros: 
(1) The proposed model makes sense to me, which tries to have two attention layers to extract the information related to the questions. It seems to have the ability to deal with "and"/"or" logical relationships as well. 

(2) Fig. 4 is impressive. It is clear and well-designed. 

(3) The results in Table 2 are convincing. They show that both the proposed dual-attention method and multi-task learning would contribute to the performance. 

Cons:
(1) It seems that the two main contributions are related to the language. Thus the synthetic language might not be proper to study. For example, in Eqn. 2, the first GA multiplies the BOW vector with the vision feature map, which could filter out unrelated instruction. This method could not be directly transferred to a real setup where natural language and natural images are involved.

(2) The designed attention modules is lack of generalizability. It implements a two-step attention module, while the first step selects the related visual regions w.r.t the words and the second step gathers the information regarding these attended regions. However, it might not be aware of the spatial relationships and thus be limited to simple questions. For example, if the question is "What is the object on top of the apple?". To my understanding, the current module would not explicitly handle this one-hop spatial relationship. 

Comments:
(1) According to Sec. 3, 70 instructions and 29 questions are involved in this task. Using GRU to encoder these questions seems to be redundant. A simple one-hot embedding for these instructions might already be enough to encode the information.

(2) I am not sure why the visual attention map x_S could be used as the state of the module.

(3) After Eqn. 3, the paper says that "ReLU activations ... make all elements positive, ensuring ...". I am confused about the intuition behind this argument because of the softmax activation. Softmax will projects 0 to 1. So the sum of the all-zero vector would still be non-zero after softmax. 

Typo:
- In Sec. 4, X_{BoW} \in \{0, 1\}^V.
- In Sec. 4.1, "this matrix is multiplied ..." --> this tensor.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: *Summary 

The paper describes a Dual-Attention model using Gated- and Spatial-Attention for disentanglement of attributes in feature representations for visually-grounded multitask learning. It has been shown that these models are capable of learning navigational instructions and answering questions. However, they addressed two limitations of previous works about visually-grounded embodied language learning models.  The first is the inability to transfer grounded knowledge across different
tasks, and the other is the inability to transfer to new words and concepts not seen during the training phase. To overcome the problem, a multitask model is introduced. The model can transfer knowledge across tasks via learning disentanglement of the knowledge of words and visual attributes. The paper shows that the proposed model outperforms a range of baselines in simulated 3D environments. 


*Decision and supporting arguments

I think the paper is on the borderline. The reason is as follows. 
The motivation of the study is described appropriately, and the performance is quantitatively evaluated, as shown while Table 2. 
However, the generality of the proposed method, i.e., dual attention, is still ambiguous. Though the devised module performs effectively in this specific simulation environment and specific two tasks, an explanation of the theoretical basis and generality of dual attention seem to be missing.
Even though the title has the phrase "multitask learning," what the system copes with is just two specific tasks.  If the system is designed to solve the two specific tasks simultaneously, it's better to change the title. The title seems to be misleading.
Some of the main contributions, e.g., "modularity and interpretability" and "transfer to new concepts," are not evaluated quantitatively.


*Additional feedback
In conclusion, "interpretablew" -> "interpretable"
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
The paper explores multi-task learning in embodied environments and proposes a Dual-Attention Model that disentangles the knowledge of words and visual attributes in the intermediate representations. It addresses two tasks, namely Semantic Goal Navigation (SGN) and Embodied Question Answering (EQA), using a simple synthetic environment. The paper compares against a few simple baselines and baselines adapted from models in each task.

I would recommend for acceptance, as the experimental results show that the proposed approach successfully transfers knowledge across tasks.

However, I would also like to note that the paper has a few drawbacks.

First, the paper uses a new environment to evaluate the SGN and EQA task instead of the benchmark environments for these two tasks, making it difficult to compare performance to previous work. The environment in the paper is small (compared to e.g., House3D for EQA) and has a limited variety. Also, the paper only compares to relatively out-of-date approaches on EQA and SGN, instead of the state-of-the-art approaches on them.

In addition, the paper should also discuss its connections to other multi-task learning approaches in the related work section.
review_title: Official Blind Review #1

Meta-review: This paper offers a new approach to cross-modal embodied learning that aims to overcome limited vocabulary and other issues.  Reviews are mixed.  I concur with the two reviewers who say the work is interesting but the contribution is not sufficiently clear for acceptance at this time.
Decision: Reject