Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper is studying the problem of learning to interpret manuals / textual information about the task, with the goal of faster learning and generalization in RL. Unlike recent prior work (Narasimhan et al 2018, Zhong et al 2020) where the entities in the environment are already partly grounded to text (e.g. by representing them as their textual description), the agent here needs to learn the mapping between entities and corresponding text. In order to study this problem, the authors use a new environment and dataset of natural language descriptions, as well as propose a self-attention model that matches entities to relevant sentences. The proposed model performed comparable to a partly grounded policy (as in Narasimhan et al 2018).

Reasons for score:

I find the topic interesting and relevant, and the use of human-generated descriptions (in contrast to procedurally generated manuals as in Zhong et al 2020) is welcomed. However, there is relatively little in terms of novelty both in the proposed model and the environment. I think the paper would benefit from identifying and focusing on one or two of the more specific issues and exploring them in-depth.

More specifically: 
(1) The proposed model is a fairly straightforward application of self-attention to text, which has been similarly used (albeit with BiLSTMs) in Zhong et al 2020. 
(2) While the challenge of needing to learn which entities map onto which words is greater, each sentence refers to exactly one entity, so the agent just needs to match 3 sentences to 3 entities, which is a marginally greater challenge (if at all). 
(3) With slight modifications, the environment proposed in Zhong et al 2020 should be suitable for addressing the main topic of this paper, why introduce a new environment? 

Questions and comments: 
- there should be a more detailed comparison to the two of the closest works (Narasimhan et al 2018, Zhong et al 2020), i.e. how exactly is your model and environment different?
- the MDP notation for transition and reward function is imprecise, actions are a random variable in both expressions (i.e. use r(s, a, s'))
- why is the curriculum introduced, is it necessary to solve the task?
- the use of the term _validation_ here is confusing, it seems you are both interested in zero-shot (on test games) and transfer learning performance (which is referred to as validation)?  
- in Fig 5 right, why is learning from scratch so much worse? Shouldn't learning from scratch be as easy as in Fig 5 right/middle?  

------------------------------------

UPDATE:
After reading the author's response, the reviewer's comments, and the revised version, I have increased my score. 
See my response to the authors for more comments.   
review_title: Addressing under-explored problem, but unfocused and missing clear novel contribution 

final_score: rating: 5: Marginally below acceptance threshold
review_content:  
Natural language grounding is an interesting research direction and has attracted many researchers in recent years. Previous work mainly considered grounding the text to image objects. This paper considers collaboratively to learn “entity” representations and natural language explanations with a reinforcement learning framework. Specifically, a multi-modal attention network is proposed to model the interaction between the entity representation and the text descriptions. The entire framework is trained over multiple games in a multi-task manner. Experiments are conducted on a newly designed benchmark. The proposed RL framework achieves reasonable performance in domain games (training & test are from the same games) and also has a strong zero-shot generalization to unseen games and "entities" (thanks to the parameter sharing and multi-task learning). Besides, the newly released dataset may facilitate future research in natural language grounding.

 I am not familiar with the natural language grounding literature, so this might be an educated guess, which is listed as follows,


- The writing is relatively clear, although some specific illustrations are sometimes hard to follow. For example, the term “entity” is very ambiguous. In the NLP field, an entity might relate to a phrase that has specialized semantic meaning. I read several recent papers in the natural language grounding field but still puzzled about the term (for example, the semantic meaning of “entity” in  [2] is different from this paper). 
 
- After reading this paper, I am not sure which part is the main novelty. The proposed reinforcement learning framework is relatively standard (no specialized designed state, policy, and reward). The interaction module is designed in a slightly straightforward way. Utilizing parameter sharing to encourage stronger generalization ability also seems reasonable. The novelty might be further clarified.
- The proposed approach seems to be similar to [1] (cited). The general idea is to learn a parameterized policy model that inputs the pair (entity [this paper] or visual representation [1] and text) and outputs the action of the agent in the game (correct me if I am wrong). Depending on different scenarios, the encoding network can be slightly different. It might be beneficial to have a detailed discussion of the difference between the methods proposed in the two papers.
- In Figure 5, the reward of different approaches seems to have small differences. However, the final winning rate of games varies a lot. It might be beneficial to have some discussions on this part.
- If my understanding is correct, the contribution of the paper is mainly focused on the strong generalization ability of the proposed approach across different games. However, it may need to clarify which part of the proposed approach contributes to the most significant influence. 

Other minor points:
 - I am interested in the O-Map baseline, which seems to be a very good upper bound of the proposed approach, I am surprised to see that the EMMA approach outperforms the O-Map in the training settings. Some intuitive explanations might be useful. The second question is that, though utilizing the upper bound (O-Map), the winning rate still has some room to improve. It is suggested to have some further analysis.
 - I do not quite understand why the overall training process contains two rounds of stages. Is S1 stage training a warm-up training?

I would like to hear the authors’ responses to make my decision.

[1] Zhong et al. RTFM: Generalising to New Environment Dynamics via Reading. ICLR 2020

[2] Lai et al. Contextual Grounding of Natural Language Entities in Images. NeurIPS 2019 workshop

review_title: An overall clear and straightforward paper based on an educated guess.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers the task of training an agent to act following a manual expressed in natural language. The manual describes the roles and the behaviors of the entities in the environment. Each entity can be the goal, the message or the enemy and it can also be either fleeing, chasing or not moving. The agent has to bring the message to the goal while avoiding the enemy. A model called EMMA is proposed to change entity representations based on the manual, thereby making the agent aware of the entities’ roles. It is shown that EMMA is more effective than simple baselines.

The paper is mostly clearly written.  The proposed model makes a lot of sense for the considered task. The paper can be seen as another proof-of-concept paper for acting based on a manual, but it should be noted that it is not the first one of its kind [1]. 

Some things were not entirely clear to me.

- In several places in the paper it is mentioned that the model learns “mapping between entity IDs in observation space and their symbols in text entirely through *interaction* with the environment.” I am not sure what interaction means here. Looking at the formulas, my understanding is that EMMA establishes a correspondence between the observed object symbols and the entities based on the entities’ descriptions. What role does interaction play in determining the object-role mapping? Same question about a similar sentence in Paragraph “Multi-Task Setup” of Section 5.1.
- Why is G-ID baseline performing so poorly on validation and test data? Doesn’t it have the complete access to the roles and behaviour of every object?
- Why is the paper not using a setup from the prior work, e.g. from [1]? 

While the paper is clear and seems to be very accurately and correctly executed, I have concerns (which might be considered subjective) about the limited real-world impact of this kind of work. The paper does not discuss how this work is related to any real world RL applications, such as e.g. robotics or even building RL agents for games that people actually play. It does not discuss what the manual would be in these cases. Would it look anything like the description of the objects’s roles? It appears common in the field of RL to not ask for such justifications and happily accept gridworld studies motivated by generalization to “new environments”. But I would like to flag that for an outsider this disconnect from reality may seem troublesome. 

review_title: accurately executed language-conditioned RL paper, but somewhat too toy-ish

final_score: rating: 7: Good paper, accept
review_content: ## Summary
This is a significantly improved extension of prior work in this area. The authors significantly improve upon the previous techniques: see below. I have a few questions on clarity but overall the paper was well-presented.

## Quality & Clarity
The paper is generally fairly clear. There are a lot of details in this paper and some of them did not quite feel fully developed (understandable given the space constraints), such as justification for certain modeling choices.

For example, in section 4 under “Entity representation generator”, the authors say:
>  For each entity e in the observation we place its representation xe into a tensor X at the same coordinates as the entity position in the state s to maintain full spatial information
This implies to me that tensor X is a 2D grid matching the coordinate system. The next section says that the three most recent observations are concatenated together and a 2D convolution is run over them. Which of the three potential dimensions are used in the 2D convolution and which is flattened: the x/y coordinates or the time dimension?

In Section 4: Model, Text Encoder, you say that “These [alpha and beta] weights imbue our model with the ability to focus on relevant tokens”. Can you provide an intuition for what u_k and u_v are learning?

In Section 6.1: Multi-task Performance, I was not quite able to grasp why the single-combination tasks are more difficult. Can you explain the difference between these two kinds of tasks a bit more?

Overall though, I was able to easily understand the main points of the paper and the data was clearly visualised in graphs and tables.

## Originality & Significance
There are a lot of good contributions in this paper:
- natural language crowd-sourced manuals with two layers of “turk-indirection”: not only was the text crowdsourced but first the templates for the texts were crowdsourced
- a comparison of several different techniques with different attention mechanisms
- experiments with additional evaluations of generalisation and robustness, including more difficult test scenarios, adding neutral distractor entities, fine-tuning with additional sources of punishment/reward, negating the text, and replacing entities with synonyms
- a visualization of attention to demonstrate that the attention mechanism is working properly

These are original and significant contributions that go beyond what was done prior.

**Suggestion**: While I didn’t understand the difference in difficulty between the multi-combination / single-combination games, what I did hope to see when they were presented was an evaluation of how the agent was able to transfer learning from one of these enabled transfer to the other (e.g. learning in the fantasy setting transferred to the nature setting). This would demonstrate that the kind of structure present in the tasks is learned and can be applied to different kinds of entities / a different vocabulary.
review_title: Solid extension of prior methods

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a model for entity grounding from its textual description for a text-based language game. To test their new model named EMMA (Entity Mapper with Multi-modal Attention), they also present a new toy game framework and crowdsourced data composed of 1320 games (from 3,881 entity descriptions with crowdsourcing). In this game, the agent has to associate an entity with its description. The game is described in Figure 1 and section 5.1: there are three entities to each game (messenger, enemy, goal), and each entity can be stationary, chasing, or fleeing movement. The game has a one-sentence description for each entity (if I’m interpreting it correctly). The space of action is up, down, left, right, and stay. The goal of the player is always static — bring a message to the goal while not touching the enemy. From what I understand, the game here itself is substantially simpler than that of previous work (Zhong et al 20), which involves varying goals and modifiers. 

Their architecture deviates from existing approaches (Zhong et al 20, Narasimhan et al 18) which did not require the model to learn a mapping from an entity to its description either by providing a mapping between objects and their textual descriptions or using entity names plainly. This is the main focus of the model — to learn a mapping between entities and their descriptions. The evaluation is also done in a manner that between train/validation/testing there’s no overlap between entity - role combination. 

The model architecture is pretty standard, using a bert-base encoder with attention. Figure 2 clearly explains their model, where each character takes a weighted sum of descriptions as value vectors to decide the action. 

The game feels a bit retrospectively designed to test the research idea. It would have been a lot more convincing if they use their model on the existing game environment, instead of creating a whole new game specifically designed to test their idea. The paper is clearly written and easy to follow, except for some parts (BAM parts weren’t easy to follow). 

The paper presents nice studies throughout section 6, looking into learning in a more challenging evaluation set up such as generalization and supplementary material is also thorough.

For Figure 5, graphs are on the union of single and multi-combination games? 
How much does it help to use pre-trained BERT encoder vs. training from scratch? 

I’m not sure what future work can be done in this space. The model seems to be doing well in this constrained setting (Table 2). Could the authors provide how this model can be applied to more challenging and complex game scenarios, where there's no one-to-one mapping between entity description and an entity, and the goal is more complex?
review_title: Official Review #2

Meta-review: Like the reviewers, I find this paper extremely borderline. On the one hand, it is clearly written, about a topic I find fascinating, and generally well motivated if not shockingly novel (i.e. removing some of the simplifying assumptions from Zhong et al. 2020, e.g. requiring grounding to be learned, use of real language rather than synthetically generated). On the other hand, I agree with the leitmotiv present amongst the reviews that the problem at the centre of the experimental setting is very, very simple (3 objects, 3 descriptions). I am mindful of the fact that access to computational resources is unevenly distributed, and am not expecting a paper like this to immediately scale their experiments to highly complex settings with photorealism, etc, but I can't help but feel that a more challenging task, with a deeper analysis of the problems presented by both grounding and the use of non-synthetic language, would both have been highly desirable to make this paper uncontroversially worth accepting.

As a result, the decision is to not accept the paper in its present form. Work on this topic should definitely be presented at ICLR, but it's a shame this paper did not make a stronger case for itself.
Decision: Reject