Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper considers the problem of fast mapping: in a 3d environment, learning to explore an environment to learn a (new) mapping between objects and names, and then during a new phase, learning to pick up objects by their name ('pick up a dax'). This problem is challenging because the names change every time, so a model cannot simply learn connections between a shape and a static name (that persist longer than one episode).

Strengths:
* The paper defines the task and situates it within the context of a 3d environment, which could be useful for future work to build off of.
* The paper compares several sequence models for this task, and finds that a Transformer-XL and a a 'Dual-Coding Episodic Memory (DCEM)' model both do well. The DCEM model seems novel to this reviewer at least, and it seems to function like a hybrid LSTM and Transformer. It seems that the DCEM model performs better than the transformer-XL model when the number of timesteps is limited (so the transformer-Xl model needs to rely on its recurrence mechanism to propagate information).
* The paper tries several different generalization experiments involving more objects, new objects, a different evironment, and reward ablations. These generalization experiments provide us some insight as to what models learn in this setup.

Weaknesses: 
At least to this reviewer, there are no major weaknesses (except perhaps that the dataset is a bit toy, which isn't a concern to me). However...
* It's not quite clear how much the memory bank is learning as opposed to the LSTM hidden state (which is also something mentioned in the supplemental). It would be interesting to learn a probe to measure when the model learns to assign an object to a name (if this is indeed something measurable by the memory bank alone).
* The reference to the 'DeepMind Lab Suite' is worded in a way that comes somewhat close to breaking anonymity...


-----

update: thanks for the clarification points + figure 9 :) I still recommend acceptance (score of 8).
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary:** An agent following instructions in a grounded world is a core task in AI. This paper studies agent that accomplish this using memory-based architecture. This paper presents an argument for a multi-modal memory-architecture called DCEM whose key/queries and values are dependent on language and vision modalities respectively (or vice versa). An argument is made that this will be helpful for generalizing to novel language at test-time. Results are presented in a simple 3D domain containing several objects randomly sampled each time from a set of 30 objects. Task contain two types of instructions: "pick up an object" and "place an object on another object". Interaction proceeds in episodes where each episode contains a discovery phase where the agent learns the phrase associated with each object, and an instruction phase where the agent solves a given instruction. The proposed DCEM model outperforms baselines on various metrics and ablation. Importantly, it is shown that the DCEM can generalize to novel object names. 

**Strengths:**
- The proposed architecture learns a near-optimal policy, requires few memory slots than baseline and can generalize to novel object names (fast mapping).
- It is argued that the language and observation key-value embedding can be used to compute a simple intrinsic reward that incentivizes reaching states (language+vision) whose distance in the embedding space is far from what has been encountered. Results show that this intrinsic reward removes the need for any reward shaping that is required by baselines. This claim is a bit more speculative since comparison with other exploration methods has not been provided. Distance in the observation space is also not a robust choice for defining intrinsic reward since similar observations can be actually very different in the latent state. E.g., consider observations which always contain a certain noise (say due to camera), then the intrinsic reward would never go down to 0 and the agent would keep exploring everywhere. In another example, walking down a long uniformly decorated corridor can result in a very small change in the observation but the agent would be making progress in the latent state. Such a path would not be adequately incentivized.

**Weakness:**
- The paper only investigates generalization of object names. This is kind of a low hanging fruit for generalization in language space. More difficult generalization include different textual styles, different choice of verbs, etc.
- DCEM architecture cannot generalize to more objects. 
- Experiments are a bit underwhelming. The task is quite simple involving pick and place, with very simple language capabilities and a few objects. There is a plethora of complex environments now. E.g., see 

1. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Task. Shridhar et al., 2020
2. Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. Chen et al., 2019
3. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. Anderson et al., 2018.

I understand the advantage of control that comes with synthetic tasks, but replicating some results to more realistic tasks makes the claim stronger.

**Questions**
1. Is the language description changing with time? If not, wouldn't the language embedding l_t always be the same?
2. By "slot-based external memory" you mean memory with a fixed size of key-value pairs updated in a First-In-First-Out style? Section 3 can use some more explanation or an architecture Figure. 
3. How do "independent read heads" work? Do they work similarly to "heads" in a transformer?
4. Did you try to generalize verbs? E.g., use a different word to refer to "Pick" during testing.
5. Does the background of the environment change across episodes (e.g., layout, colour of the wall, etc.)
review_title: Contains assortment of interesting results for grounded language learning with memory-based agents. However, results are more indicative than complete.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # Overall review

The authors use a 3D world to explore grounded language learning, in which an agent uses RL to combine novel word-learning with stably acquired meanings to successfully identify and manipulate objects.  They show that a novel, psychologically-inspired memory mechanism is more memory-efficient than Transformers (both of which outperform plain LSTMs) and that it exhibits surprisingly robust generalization to novel action-object pairs.  The results should be of interest to many working in grounded language / multimodal representation learning, and the experiments are thorough and well-motivated. 

Pros:
* Interesting environment for combining fast-mapping with stable language learning in a grounded task.
* Novel memory architecture, shown to improve memory-efficiency.
* Psychologically-motivated and thorough experimentation, demonstrating surprising level of generalization.

Cons:
* A figure describing the three memory architectures in addition to / instead of the plain text could have helped compare / contrast them.
* More analysis of how the agent uses its memory would be welcome.


# Minor comments

* 4.1: "Unless stated otherwise, all experiments in this section involve the DCEM+Recons agent".  I think this means with full memory size (1024) and not the smaller one (100), but it would help to clarify.

* Figure 2: while training with 3 objects and testing with more does not work, training with 5 and testing with 8 works as well as training with 8 and testing with 8.  What do the authors make of this?  Is there some kind of "threshold of diversity" beyond which the agent can generalize to more objects?  Such a threshold idea also seems consistent with the results in Figure 3.

* Fast category extension: these results show that if the agents are trained to pick up different exemplars of a category, they can do so in testing.  During training, there was one exemplar from each of three distinct categories.  I was curious if the authors experimented with relaxing that to environments where more than one exemplar of a category could be present, with the agent being rewarded for picking any of the correct exemplars.  

* The environments the authors use seem like they would allow for testing of mutual exclusivity phenomena (see Ghandi and Lake 2020 [https://arxiv.org/abs/1906.10197] and references therein), by providing instructions with a novel word in a setting with one unseen object (or category).  I would be curious to see if their memory architecture does better with these phenomena than existing ones.


# Typographic comments

* p 3, "the current visual embedding l_t": l_t should be v_t

* Appendices were referenced in the text, but not included in the uploaded PDF (even though I believe this was allowed at ICLR).  These appendices seem very helpful for complete model/experiment details.


# Update

I thank the authors for their thoughtful reply and for incorporating the feedback.  The additional information is most welcome, and so I maintain my score of 8 for the paper.
review_title: Interesting results on grounded language learning and fast-mapping in a 3D world

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents experiments for acquiring words via fast-mapping in an embodied environment. The technical contribution is interesting and solid, but the experiments fail to address some important questions that are yet scoped by the claims of the paper (namely, that learning is being done -both- fast and slow, as per the title). Notably, the paper is really well-written and readable, and the experiments on novel category + novel instance recognition are really convincing specifically for fast-mapping (4.1).

Experiments That Would Really Strengthen the Paper [I would love to hear your thoughts on these for rebuttal]:

- Fast mapping test where some object categories use the canonical name across trials, to demonstrate that the agent can still associate descriptive + instructive (not just instructive) langauge with both the fast and slow paradigm

- Experiments where the nonsense words are randomly generated characters in every episode of training and evaluation, to exactly tease out that the new word is being memorized and associated, not that the agent is memorizing that "dax" per episode changes meaning, which is itself knowledge about the word that renders it not truly "nonsense" or unseen. Currently, its meaning just becomes instead "a variable assigned to a new object per trial that I need to memorize".

Questions:

- Table 1 random object selection / LSTM perform at .33; does this mean there's a control module that decides how to do the navigation? Is the agent guided at test time to each object? This confuses me a lot! I thought it had to learn to go visit the objects and then to go get the one later, which seems unlikely to happen from an LSTM alone or random actions.


Improvements:

- I think it would be worth spending some time in the intro motivating why the agent is embodied. What does embodiment get us for the hypothesis that a network can be trained to do fast mapping? We can imagine just giving a sequence of images and associated descriptions (e.g., "this is a dax"), followed by the command "pick up the dax" together with all object images and the task of classification. What challenges does embodiment present that are missing from that paradigm? E.g., seeing the objects from non-canonical angles, having to find them, etc. There's some recent push in the NLP community in general for embodiment, so I think it's worth spending time on as motivation in this paper. [There's like a little bit? 4.1 mentions that learning from multiple views of the object is important, but this is sort of the entirety of what embodiment is buying us here versus a static image paradigm, so it's worth emphasizing, probably.]


- 4.3 is cool but doesn't -quite- do the fast-mapping test we'd like to see, which is that a word used consistently applies consistently. That would look more like including a few object categories whose names are used "correctly" across episodes, rather than one of the random names that are used inconsistently. This brings up a larger question of the consequences of using the same nonsense words for every trial, such that they can be memorized as being vacuous and in need of memorizing. Alternatively, the nonsense word could have been a random string to really test the memorization of an entirely novel token.



Nits:
- Figure 5 caption says these are accuracy curves, but the y axis is "reward"; is this task accuracy or the intrinsic reward values from surprisal?
- 4.4 why include this if there are no results in the paper for it? The constant references to the appendix are generally jarring, but this one is just sort of silly. It could be a single sentence maybe but not a whole subsection.

review_title: Well-written paper with an interesting contribution; scoping of the contribution claim is a bit wider than what experiments demonstrate [scope clarified via discussions and revisions]

Meta-review: The paper investigates the capacity for neural language models to perform fast-mapping word acquisition using a proposed multimodal external memory architecture. Much work exists that shows that neural models are capable of following instructions whose meaning persists across episodes (i.e., slow-learning), however much less attention has been paid to instruction-following in a one-shot learning context. Using a simulated 3D navigation/manipulation domain, the paper shows that the proposed multimodal memory network is capable of both slow and one-shot word learning when trained via standard RL.

The submission was reviewed by four knowledgable referees, who read the author feedback and engaged in discussion with the authors. The paper is topical---one-shot language learning for instruction-following using neural models is of significant interest of-late. The reviewers agree that the proposed multimodal memory architecture is both interesting and technically solid. The reviewers raised concerns about the experimental evaluation and the role of embodiment. The author feedback together with discussion with reviewers were helpful in resolving some of these issues. However, the authors are encouraged to ensure that the paper clearly motivates the importance of embodiment to slow learning and fast-mapping, particularly given the large body of work in language acquisition in robotics, a truly embodied domain, which is notably missing from the related work discussion.
Decision: Accept (Spotlight)