Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: First of all, I should acknowledge that this is a last-minute emergency review. Second, I should also acknowledge that I haven’t actively followed the specific research topic handled in this paper, although I work quite a lot in reinforcement learning and dialogue systems. The last paper I read on the topic was Branavan et al’s “Learning to win by reading manuals in monte-carlo framework” (ACL 2011).

This paper is about language-conditioned reinforcement learning, where the agent is required to perform machine reading of documents to learn policies to solve a task in new environments. The contribution of this paper is seemingly two fold: first, a new benchmark test suite is proposed (named “read to fight monsters” funny acronym), and second, an improved neural model (named “txt2pi”) which extends FiLM proposed for related tasks. Various experiments on txt2pi are conducted to show how the model performs against baselines, as well as it behaves under different learning settings.

Overall, this paper makes a good contribution to the research on deep models for language-conditioned reinforcement learning, but I see a number of things I miss in this paper. 

The first one is about the test suite RTFM. The authors claim that it poses a new challenge that the agent has to understand the language specification of the goal, the environment dynamics, and the observations *all together* (compared to prior work that misses one or two), I am not confident that it is not challenging enough. 

- The main concern I have is that they are all machine-generated, and soon or later, this kind of test suite could be beaten-to-death. A more ideal test suite would be the texts collected by humans, aiming for more diversity, ambiguity, and sometimes incorrectness. 
- The RTFM would combinatorily generate a large number of environments, but still, they all look pretty simple. It would be interesting to have “partially specified” documents, such as “lightning monsters are known to be weak against grandmaster’s or soldier’s weapons, but it is not specifically known”. It would be interesting to have partially observable environments, where the agent has to do some information gathering in the environment to behave optimally.

The second one is about txt2pi model proposed in the paper.
- txt2pi adequately extends FiLM to handle multiple sources of information, but what is the lesson being learned? 
I find the discussions in page 15 very enlightening. Could you move the content to the main text and defer the curriculum learning details to the appendix?

In summary, this paper proposes an interesting test suite for language-conditioned reinforcement learning, but I wish it is more realistic in the language data and more complex in the RL tasks. 

--- 
I have updated my rating based on the reviewer response. Thank you for clarifying some of my concerns. I am still a bit reserved because of the use of synthetic corpus. However, at the same time, I think there could be a lot of follow-up papers based on this work.  

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper constructs a new game that requires combining visual reasoning with text understanding to win.  The authors propose a new model txt2π, based on a new layer called FiLM², which combines text and visual features in a way that allows visual features to be encoded with knowledge of the text features (as in the FiLM layer from previous work), as well as text to be encoded with knowledge of the visual features. The model is trained to play the game using IMPALA.  Ablation studies show that the FiLM² layer leads to a substantial improvement, and also shows the necessity of curriculum learning.  Performance is still below human performance suggesting this is a promising area for future work.

The work here is complicated, but the writing and explanations are very clear. A new problem domain is introduced, which I think will be interesting to many other researchers.  I am not an expert in this area, so it's hard for me to give a confident assessment of the significance of this paper, so I give this paper a Weak Accept, despite enjoying the paper and having no specific points of criticism.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This work proposes a new environment, Read to Fight Monsters (RTFM), and correspondingly a new algorithm, txt2\pi, for solving this problem. The RTFM requires the agent to read a description of the rules (x beats y, etc) and a description of the goal (to eliminate y), and perform the task correctly to win the game. The txt2\pi algorithm uses the newly proposed FiLM^2 module and consists of integrating the visual input (grid world configuration) and the text input (descriptions) to learn a policy and baseline (actor-critic). 

Pros
- The presentation is relatively clear.
- A new environment that can be impactful for the field.

Cons
- Certain design choices are not discussed properly.
- The experimental evidence is relatively weak.

(1) For RTFM, a complete list of possible elements would be helpful, at least in the supplementary. For example, the possible monsters/elements/weapons, etc.

(2) Some proposals lack motivation or explanation. 
(2.1) How is Eq.(8) a summary?
(2.2) Why the attention in (18) should base on the vis-doc embedding instead of the goal embedding, or goal-conditioned doc embedding?
(2.3) It would be helpful to provide an example of the inventory description.
(2.4) What is the difference between the Goal and Goal-doc in Fig.3?

(3) Experiment
(3.1) "no group", "no dyna" and "no nl" seem to be suggesting a simpler environment. How is no natural language templated descriptions (no nl) considered as an easier scenario than with a template? The description would be harder to parse or understand without structure (template).
(3.2) It is mentioned several times that the models are "trained on one set of dynamics and evaluated on another set of dynamics". Specifically, what are the differences?
(3.3) For the curriculum training (Table 2), adding dyna does not hurt the performance (from 84 to 85). Further explanation would be helpful.
(3.4) How are the baselines and alternative methods perform in the full environment (+group etc.)?
(3.5) The texts on Fig.5 are not evenly spaced, which makes them harder to align with the attention (also some letters have dots above them). Does this figure correspond to (15)? In Fig.5b, a0 focuses on "gleaming" and a1 focuses on "lighting", which are not present in the entities (compared to the caption). Something is wrong or missing here.
(3.6) Eventually, the txt2\pi algorithm only achieves modest improvement over random agents (65% versus 50%), which is not very impressive.

## Update ##
Thank you for the thorough explanations, which are very helpful. It would be better to include them in the paper to clarify potential misunderstandings.

One more remark is that the goal-doc attention map (i.e., lg) in Fig. 5b focuses on lynx, which does not make sense in an environment with only a beetle and a wolf. This is strange and problematic given that it is an important attention map for many subsequent modules.

I have changed my score accordingly. It also needs to be considered that I'm not very familiar with the field.
review_title: Official Blind Review #2

Meta-review: This paper proposes RTFM, a new model in the field of language-conditioned policy learning. This approach is promising and important in reinforcement learning because of the difficulty to learn policies in new environments. 

Reviewers appreciate the importance of the problem and the effective approach. After the author response which addressed some of the major concerns, reviewers feel more positive about the paper. They comment, though, that presentation could be clearer, and the limitations of using synthetic data should be discussed in depth.

I thank the authors for submitting this paper.
Decision: Accept (Poster)