Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper intends to contribute a novel task (Cache, as realized in AI2-THOR), the architecture of a strong Cache agent which learns reusable representations which allow significant transfer performance, and novel methods for evaluating the quality of dynamic image representations. The first and third contributions are directly related to the conference topics, and the second provides additional evidence in favor of the paper’s core idea: training on interactive gameplay allows learning flexible representations (in the sense of supporting many tasks via transfer) even when images are highly correlated and synthetic.

The key strength of the paper is the very general core idea it advances and how this idea is explored via a novel task. The paper is easy to read and convincing. The partition into details needed for the paper’s core argument and details specific to the experiments in the appendix is well done. (If anything even more could have been pushed to the appendix.)

One weakness of the paper is that a specific notion of “flexible” (which is mentioned in the title and twice in the abstract but nowhere else) is not advanced or integrated with the core idea. How does gameplay relate to flexibility? Why might flexibility be harder to achieve via passive learning or reinforcement learning with fixed reward functions? Because the authors place stress on the idea of how play and interaction contribute to representation learning (rather than a new method), slightly more space should be given to developing the general idea. The idea is not specific to vision, but only vision-related representations are considered. Sketching how the idea ought to work for text or audio would be useful if the focus really is on this very general idea.

Recommendation: strong accept. The philosophical aims of the paper make it stand out amongst the mass of related work that is otherwise very engineering focused. The experiments are soundly executed in a way that ends up clearly demonstrating the core idea.

Questions for the authors:
- A step where the hider needs to retrieve the object they hid would seem appropriate. Are there certain limitations of the AI2-THOR environment that make adding this step (which would seem to expose more of the richness of the simulated world through fixed rules of the game) infeasible to add?
- Inversely, do the authors feel that it was important that the hider manipulate the object into the desired location? How much of the richness of the simulated world comes through in the task feels relevant to the core ideas of the paper, but the paper currently does not address this kind of detail in the design of the Cache game within AI2-THOR.

Section-by-section reactions: (to see how opinions change over time)

Title+Abstract:
- The notion of “flexible” seems to be at the heart of this paper’s intended contribution. Hopefully it will be defined in the body text. Uh oh, it looks like “flex” only ever appears on the first of the submission’s 36 pages. Hopefully a synonym will get defined later.

Introduction:
- Excellent motivation.
- Good that representations of interest (SIRs/DIRs) are named and distinguished. Many other papers, in the interest of highlighting end-to-end training, would forget to do this.
- Good explicit list of contributions, excellent that two are specifically centered on representation learning.
- Missed opportunity to highlight a distinct role for “flexible” representations. (I don’t quite know what it should mean beyond supporting transfer well. A representation that could easily be scaled up or down in dimensionality by stripping channels in a well defined order might be considered flexible in another sense. Likewise, one that was defined in terms of pluggable input modules to work with novel combinations of familiar input types might be considered differently flexible. What kind of flexibility do you want?)

Related Work:
- Another take on learning visual representations via interactive gameplay is seen in https://arxiv.org/abs/1812.03125 where the authors learn a SIR (trained on a proxy task of predicting videogame memory state) that supports the use of low-continuous-space exploration strategies like rapidly-exploring random trees. The representations are learned offline/passively, but they are learned as to improve the efficiency of the very exploration process that builds that dataset for offline learning.

Playing Cache in a Simulation:
- It seems notable that the hiding agent is never asked to retrieve the object they have hidden. Without this step, the hiding agent may find ways of manipulating objects in a way that makes them simply unretrievable (e.g. the object is pushed into a corner in a way that causes it to glitch out of the room, etc.). A step like this would require the hider to learn a finer grained representation of the hiding location that gives itself a clue as to how it should be retrieved (e.g. “under the couch in a place you’ll never be able to see but will be there if you actually reach for it).

Learning to Play Cache:
- Great!

Experiments:
- All well done.

Discussion:
- “We believe that it is time for a paradigm shift via a move towards experiential,
interactive, learning.” -- something similar has been said by many other researchers in many different decades, so it would be good to say what’s different about the situation in 2021. The difference now seems to be the availability of simulators with visual fidelity comparable enough to reality to demonstrate meaningful sim2real transfer. Are there other bullet points that could be added to a why-now argument?
review_title: Convincing paper about learning transferrable reps from correlated/synthetic images via competitive interaction

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the author's propose an embodied adversarial reinforcement learning agent that can play a variation of hide-and-seek called Cache. This environment is a high fidelity interactive world. The authors argue that the agents are able to learn flexible representations of their observations which encode information such as object permanence, free space and containment. 

The authors provide a well-written description of their game and provide well-designed visualizations to understand the interactions of the agent and the observations required for the learning problem. The authors present a concise and well-researched literature review which serves to distinguish their work as novel and well built on underlying prior work. 

The authors make several contributions in this paper. First, an adversarial game Cache which permits the study of representation learning in the context of interactive visual gameplay. Furthermore, they present an agent which can perform strongly on the benchmark that they create even in comparison with human players. Finally, they present a study of the static and dynamic representations learned by the agent.

The authors provide an overview of the architecture for the Cache agent. It is well researched, well-reasoned and presented in a way that is easy to understand and reproduce.

The authors present multiple experiments in their paper and the corresponding deep Appendices. Specifically, they attempt to understand how agents can learn to proficiently hide and seek objects with static image representations and dynamic image representations

Of particular interest are the dynamic tasks the authors present in Section 5: Experiments. The authors make allusions to study of human children and object permanence. The results are quite compelling and well presented in a way that makes them easy to understand. I would urge the authors to consider if there are any non-ablative baselines against which they may be able to compare their model. One baseline that the author's use is human volunteer comparisons, these are compelling and presented in the appendix.

Figure 3 is perhaps the weakest figure in the paper. While it provides a great amount of information I would argue that the results are presented in a way that makes them less legible than if the figure was broken out into methods and results. The same could be said about Figure 4. The visualizations of the agents representations of synthetic and natural images are combined into a single image with performance on corresponding tasks. While I understand that the authors made these choices for space constraints, I would urge them to reconsider the visual hierarchy of the figures to emphasize the performance of their agents. 

I feel like this is a paper that would be of great interest and benefit to the ICLR community.

review_title: Review of Learning Flexible Visual Representations via Interactive Gameplay

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary
-------

This paper examines the representations learned during adversarial gameplay, specifically a hide-and-seek game called Cache.  The hiding agent must place an object in a room such that the seeker agent cannot find it.  The authors argue that the adversarial nature of the game shapes the representations.  Inspired by psychology experiments performed on children, the authors examine both static and dynamic representations to probe whether they contain information about properties of the environment such as object permanence.   

Positives
---------

The paper addresses an interesting problem by studying representations learned by agents interacting in an environment.  These representations perform favorably compared to passive representations. 

The paper is thorough, detailed, and well written.  

The inclusion of dynamic representations is interesting, and the psychology-inspired experiments are adapted well to the Cache agent's environment.


Negatives
---------

My primary concern with the paper hinges on a trick the authors used to get their agent to learn meaningful representations.  The authors introduce Visual Dynamics Replay (VDR) to improve the agent's performance.  Essentially, this is a set of auxiliary tasks and self-supervision derived from the agent's interactions with the environment.  Although not a problem in itself, VDR was not used in the Navigation agent or Random agent used as baselines in the paper.  This leaves the conclusions of the paper much weaker.  It is unclear if these auxiliary tasks of VDR or the adversarial gameplay of the problem are the driving factor behind the learned representations.  

Because the scope of the paper is so broad, it is difficult to evaluate some of the contributions.  For example, the perspective simulation module seems useful, but there are no ablation experiments evaluating changes to the model.


Reasons for Score
-----------------

Although the paper is intersting, detailed, and thorough, I have trouble with the strong conclusions drawn from the experiments because of the possible confounds of VDR.


Minor Issues that did not affect score
--------------------------------------

Graphs in figure 5 are difficult to see and interpret, even at full zoom on a monitor.


Update after rebuttal
------------------

I am very interested to see the results of the experiment mentioned in the rebuttal.  Although you mention that gameplay dynamics and ability to interact with objects is crucial, I still feel that this isn't as cleanly demonstrated as it could be without addressing these confounds.  However, in retrospect I also realize that my initial score suffered from tunnel vision on that single issue, which is important to several claims in the paper, but is by no means the only contribution.  The paper is ambitious in scope, novel, and well written, so I have increased my score.

review_title: Initial review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes embodied game-playing with artificial agents as a method to learn better representations of their environment. They describe a game, cache, which is a variant of hide-and-seek played in a virtual environment and a method for training an agent to play the game. They present results which demonstrate that the static representations learned through game-playing perform better than other pre-training tasks within the same virtual environment, on both virtual vision and real world vision applications. They also show that the dynamic representations are useful for completing object permanence tests inspired by developmental psychology research.

This research direction (both biologically-inspired CV as a whole and specifically game playing as a pre-training task) is exciting and seems extremely promising. Assuming I understand the methods and results correctly, this seems like a clear accept. My only reservations are the complexity of the task and how the results are presented in the paper, which are relatively minor issues.

The task, Cache, seems extremely complicated to implement and train, since it involves five different stages of embodied exploration/action and adversarial reinforcement learning. While all of these steps are necessary to play the game like humans (or ravens) do, it is unclear to me whether the important learning is occurring as a result of the whole process, or of a specific stage. It seems possible that a simpler task may have comparable results, and it would be an interesting direction for future research to investigate how each of these stages are contributing to learned object permanence. I'm not an expert in developmental psychology, but I know there is some research showing that smooth-motion visual signals are the main prerequisite for object permanence in chickens (https://onlinelibrary.wiley.com/doi/abs/10.1111/desc.12796). An ablation study over game mechanics or world properties could lead to some really interesting results.

Of course, the paper you've submitted already describes a huge volume of research work and doesn't need more experiments. Unfortunately, it is a bit difficult to follow as a result. You may want move some content from Fig. 3 and 4 (those figures are difficult to parse, even with my pdf zoomed in 250%) to the appendix, if only to focus the reader's attention on specific results. While arguing that your agent successfully plays the game is important, it should take a backseat to the experiments which probe the properties of your static and dynamic image representations.
review_title: Very dense paper, interesting research direction

Meta-review: Motivated by the importance of gameplay in the development of critical skills for humans and other biological species, this work aims to explore representation learning via gameplay in a realistic, high fidelity environment. Inspired by childhood psychology, they propose a variant of hide-and-seek game called "Cache" built on top of AI2-THOR, where one agent must place an object in a room such that another agent cannot find it, and demonstrate that the adversarial nature of the game helps the agents learn useful representations of the environment. They examine the difference in representations learned via such a dynamic, interactive adversarial gameplay approach, vs other more passive approaches involving static images.

The paper is well written and motivated, and easy to follow. All reviewers agree that the paper will be a great contribution to the ICLR community. I believe this is an important work, because not only does it challenge the traditional way of training many components of our systems passively (via static image recognition models), it synthesizes ideas from various disciplines (psychology, embodiment, ML) and provides an excellent framework for future research. For these reasons I'm recommending we accept this work as an Oral presentation.
Decision: Accept (Oral)