Learning Generalizable Visual Representations via Interactive Gameplay
Authors: Luca Weihs
Aniruddha Kembhavi
Kiana Ehsani
Sarah M Pratt
Winson Han
Alvaro Herrasti
Eric Kolve
Dustin Schwenk
Roozbeh Mottaghi
Ali Farhadi
Venue: ICLR 2021 Oral (2021)
Keywords: representation learning
deep reinforcement learning
computer vision

Abstract: A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across a variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solving, decision making, and socialization. Comparatively little is known regarding the impact of embodied gameplay upon artificial agents. While recent work has produced agents proficient in abstract games, these environments are far removed the real world and thus these agents can provide little insight into the advantages of embodied play. Hiding games, such as hide-and-seek, played universally, provide a rich ground for studying the impact of embodied gameplay on representation learning in the context of perspective taking, secret keeping, and false belief understanding. Here we are the first to show that embodied adversarial reinforcement learning agents playing Cache, a variant of hide-and-seek, in a high fidelity, interactive, environment, learn generalizable representations of their observations encoding information such as object permanence, free space, and containment. Moving closer to biologically motivated learning strategies, our agents' representations, enhanced by intentionality and memory, are developed through interaction and play. These results serve as a model for studying how facets of vision develop through interaction, provide an experimental framework for assessing what is learned by artificial agents, and demonstrates the value of moving from large, static, datasets towards experiential, interactive, representation learning.

Published as a conference paper at ICLR 2021
LEARNING GENERALIZABLE VISUAL REPRESENTA- TIONS VIA INTERACTIVE GAMEPLAY
Luca Weihs', Aniruddha Kembhavi', Kiana Ehsani', Sarah M Pratt?, Winson Han', Alvaro Herrasti', Eric Kolve', Dustin Schwenk', Roozheh Mottaghi', Ali Farhadi²
Allen Institute for Artificial Intelligence
University of Washington
1 "[lucaw, anik, kianae, winsonh, alvaroh, crick, dustins, roozbchm)@allenna.ccom 2 (sprattt,, all ali] washington.edu
ABSTRACT
A growing body of research suggests that embodied gameplay, prevalent not just in human cultures but across variety of animal species including turtles and ravens, is critical in developing the neural flexibility for creative problem solv- ing, decision making, and socialization. Comparatively little is known regarding the impact of embodied gameplay upon artificial agents. While recent work has produced agents proficient in abstract games, these environments are far removed from the real world and thus these agents can provide little insight into the advan- tages of embodied play. Hiding games, such as hide-and-seek, played universally, provide rich ground for studying the impact of embodied gameplay represen- tation learning in the context of perspective taking, secret keeping, and false belief understanding. Here we are the first show that embodied adversarial reinforce- ment learning agents playing Cache, a variant of hide-and-seek, in a high fidelity, interactive, environment, learn generalizable representations of their observations encoding information such as object permanence, free space, and containment. Moving closer to biologically motivated learning strategies, our agents' represen- tations, enhanced by intentionality and memory, are developed through interaction and play. These results serve as model for studying how facets of vision de- velop through interaction, provide experimental framework for assessing what is learned by artificial agents, and demonstrates the value of moving from large, static, datasets towards experiential, interactive, representation learning.
INTRODUCTION
We interested in studying what facets of their environment artificial agents learn to represent through interaction and gameplay. We study this question within the context hide-and-seek, for which proficiency requires ability navigate around in an environment and manipulate objects as well as an understanding of visual relationships, object affordances, and perspective. Inspired by behavior observed juvenile ravens (Burghardt) 2005), we focus on variant of hide-and-seck called cache in which agents hide objects instead of themselves. Advances in deep reinforcement learning have shown that, in abstract games (e.g. Go and Chess) and visually simplistic environments (e.g. Atari and grid-worlds) with limited interaction, artificial agents exhibit surprising emergent behaviours that enable proficient gameplay (Mnih et al. 2015 Silver et al. 2017); indeed, recent work (Chen et al., 2019 Baker et al.) 2020) has shown this in the context of hiding games. Our interest, however,
understanding how agents learn to represent their visual environment, through gameplay that requires varied interaction, in high-fidelit environment grounded in the real world. This requires a fundamental shift away from existing popular environments and a rethinking of how the capabilities of artificial agents evaluated.
Our agents must first be embodied within an environment allowing for diverse interaction and pro- viding rich visual output. For this we leverage A12-THOR (Kolve et al.) 2017), interactive, simulated, 3D environment of indoor living spaces, see Fig. la Our agents are param- eterized using deep neural networks, and trained adversarially using the paradigms of reinforce-
photo-realistie

---

Published as a conference paper at ICLR 2021
Opening
Crouching
Holding
Exploration & Mapping
Pers. Simulation
Object Hiding
Manipulation
Seeking
b
Hiding Stages
Seeking Stage
Figure I: A game of cache within A12-THOR, Multiple views of single agent who, time, explores and interacts objects. [ five consecutive stages of cache. In Infetth agent must choose where hide the object at high level, using its map of the environment to make this choice, while in agent must choose where to hide the object from a first person viewpoint. In the object being manipulated is tomato.
ment (Mnih et al. 2016 and self-supervised learning. After our agents trained to play cache, we then probe how they have learned to represent their environment. To this end we distinguish two distinct categories of representations generated by our agents. The first, static image represen- tations (SIRs), correspond to the output of CNN applied to the agent's egocentric visual input. The second, dynamic image representations (DIRs), correspond the output of the agents' RNN. While SIRs are timeless, operating only on single images, DIRs have the capacity to incorporate the agent's previous actions and observations.
Representation learning within the computer vision community is largely focused on developing SIRs whose quality measured by their utility in downstream tasks (e.g. classification, depth- prediction, etc) (Zamir et al. 2018). Our first set of experiments show that our agents develop low- level visual understanding of standard tasks from the computer vision literature, these tasks include pixel-to-pixel depth (Sax-) jena et al. 2006) and surface normal (Fouhey al. 2013) prediction, from a single image. While SIRs clearly important facet of representation learning, they are also definitionally unable to represent an environment as a whole: without the ability to integrate observations through time, representation only capture single snapshot of space and time. To represent envi- ronment holistically, we require DIRs. Unlike for SIRs, we are unaware of any well-established benchmarks for DIRs. In order to investigate what has been learned by our agent's DIRs we develop a suite of experiments loosely inspired by experiments performed on infants and young children. These experiments then demonstrate our agents' ability integrate observations through time and understand spatial relationships between objects (Casasola et al. 2003), occlusion (Hespos et al. 2009), object permanence (Piaget 1954), and seriation (Piaget 1954) of free space.
individual images measured by their capacity to perform a collection
It is important to stress that this work focuses studying how play and interaction contribute to representation learning in artificial agents and not on developing new, state-of-the-an method- ology for representation learning. Nevertheless, to better situate our results in context of existing work, we provide strong baselines in our experiments, e.g. in our low-level vision experiments we compare against a fully supervised model trained on ImageNet (Deng et al. 2009). Our results pro- vide compelling evidence that: (a) on a suite of low level computer vision tasks within AI2-THOR, static representations learned by playing cache perform very competitively (and often outperform) strong unsupervised and fully supervised methods, (b) these static representations, trained using only synthetic images, obtain non-trivial transfer to downstream tasks using real-world images, (c)
2

---

Published as a conference paper at ICLR 2021
unlike representations learned from datasets of single images, agents trained via embodied game- play learn to integrate visual information through time, demonstrating an elementary understanding of free space, objects and their relationships. and (d) embodied gameplay provides natural means by which to generate rich experiences for representation learning beyond random sampling and rel- atively simpler tasks like visual navigation.
In summary, we highlight the contributions: (1) Cache - we introduce cache within the A12 A12TTHOR environment, an adversarial game which permits the study of representation learning in the context of interactive, visual, gameplay. (2) Cache agent training agents to play Cache is non-trivial. We produce strong Cache agent which integrates several methodological novelties (see, e.g., perspec- tive simulation and visual dynamics replay in Sec. 4 and even outperforms humans at hiding on training scenes (see Sec. 3 (3) Static and dynamic representation study - we provide comprehen- sive evaluations of how our Cache agent has learned to represent its environment providing insight into the advantages and current limitations of interactive sattiir--gmmppaaaaaae -based representation learning.
2 RELATED WORK
Deep reinforcement learning for games, As games provide an interactive environment that enable agents to receive observations, take actions, and receive rewards, they are a popular testbed for RL algorithms. Reinforcement learning has been studied in the context of numerous single, and multi, agent games such as Atari Breakout (Mnih al., (013), VizDoom (Lample & Chaplot 2017), Go (iilerr et al. 2017). StarCraft (Vinyals et al. 2019). Dota (Berner et al., 2019) and Hide-and -nd--nnd-Se (Baker et al. 2020). The goal of these works proficiency: create an agent which can achieve (super-) human performance with respect to the game's is success criteria. In contrast, our goal is to understand how an agent has learned to represent its environment through gameplay and to show that such an agent's representations can be employed for downstream tasks beyond gameplay.
Passive representation learning. There is a large body of recent works that address the problem of representation learning from static images or videos. Image colorization (Zhang et al. 2016), ego- motion estimation (Agrawal et al., 2015), predicting image rotation (Gidaris et al. 2018), context prediction (Doersch al.) 2015), future frame prediction (Vondrick al. 2016) and more recent contrastive learning based approaches (He et al. 2020; Chen et al. (020b) are among the successful examples of passive representation learning. We refer to them as passive since the representations are learned from a fixed set of images or videos. Our approach in contrast is interactive in that the images observed during learning are decided by the actions of the agent.
Interactive representation learning. Learning representations in dynamic and interactive environ- ments has been addressed in the literature as well. In the following we mention a few examples. Burda et al. (2019) explore curiosity-driven representation learning from a suite of games. Annnd al. (2019) address representation learning of the latent factors used for generating the interac- tive environment. ZZhan et al. (2018) learn to improve exploration in video games by predicting the memory state. Ghosh et al. (2019) learn a functional representation for decision making as opposed to representation for the observation space only. Whitney et al. (2020) simultaneously learn em- beddings of state and action sequences. JJonsckkowski & Brock (2015) learn a representation by measuring inconsistencies with a set of pre-defined priors. PPinto et al. (2016) learn visual repre- sentations by pushing and grasping objects using a robotics arm. The representations learned using these approaches are typically tested on tasks akin to tasks they were trained with (c.g., differ- ent level of the same game). In contrast, we investigate whether cognitive primitives such as depth estimation and object permanence maybe be learned via gameplay in visual dynamic environment.
3 PLAYING CACHE IN SIMULATION
We situate our agents within A12-THOR, a simulated 3D environment of indoor living spaces within which multiple agents can navigate around and interact with objects (e.g. by picking up. placing, opening, closing, cutting, switching on, etc.). In past works, A12-THOR has been leveraged to teach agents to interact with their world (Zhu et al. 2017 Hu et al.) 2018 Huang et al.) 2019 Gan et al. 2020b Wortsman et al. 2019 Gordon et al.] 2017). interact with each other (Jain et al. 2019 2020 as well as learn from these interactions (Nagrajaan & Grauman 2020 Lohmann al. 2020
A12-THOR contains a total of 150 unique scenes equally distributed into five scene types: kitchens, living rooms, bedrooms, bathrooms, and foyers. We train our cache agents subset of the kitchen
3

---

Published as a conference paper at ICLR 2021
h4-1
DIR he
512
gt Hide cup
hig
CGRU
LSTM
257x7x7
512
CNNO
CNN
Hiding Locations
128x7x7
1t.
Metric map m/
Mental rollout
CNN
supervised
M*
G1+
Visual Dynamics Replay
Figure 2: Overview of the neural network architecture for the Cache agent. The trainable components are shown in yellow, the inputs and outputs in pink, and the intermediate outputs in blue. Refer to text for details.
and living room scenes as these scenes are relatively large and often include many opportunities for interaction; but we use all scenes types across our suite of experiments. Details regarding scene splits across train, validation, and test for the different experiments can be found in Sec. A.
In a game of cache, two agents (a hider and seeker) compete, with the hiding agent attempting to place given object in the environment so that the seeking agent cannot find it. This game is zero-sum with the hiding agent winning if and only if the seeking agent cannot find the object. We partition the game of cache into five conceptually-distinct stages: exploration and mapping (E&M), perspective simulation (PS), object hiding (OH), object manipulation (OM), and seeking (S); see Figures Tb to Lf A game of cache begins with the hiding agent exploring its environment and building an internal map corresponding to the locations it has visited (E&M). The agent then chooses globally, among the many locations it has visited, location where believes it can hide the object so that the seeker will not be able to find it (PS). After moving to this location the agent makes a local decision about where the object should be placed, e.g. behind TV inside a drawer (OH). The agent then performs the low-level manipulation task of moving the object in its hand to the desired location (OM), if this manipulation fails then the agent tries another local hiding location (OH), Finally the seeking agent searches for the hidden object (S). Pragmatically, these stages are distinguished by the actions available the agent (see Table a description of all 219 actions) and their reward structures (see Sec.
C.5
For more details regarding these stages, see Sec.
B
4 LEARNING TO PLAY CACHE
In the following we provide an overview of our agents' neural network architecture as well as our training and inference pipelines. For space, comprehensive details deferred Sec. D
Our agents are parameterized using deep convolutional and recurrent neural networks. In what fol- lows, we will use 0 θ catch-all parameter representing the learnable parameters of model. Our model architecture (see Fig 2) has five primary components: (1) CNN that encodes input ego- centric images from AI2-THOR into SIRs, (2) a RNN model that transforms input observations into DIRs, (3) multi-layer perceptrons (MLPs) applied to the DIR at given timestep, to produce the actor and critic heads, (4) perspective simulation module which evaluates potential object hiding places by simulating the seeker, and (5) U-Net (Ronneberger et al.) 2015) style decoder CNN used during visual dynamics replay (VDR), detailed below.
Static representation architecture. Given images of size 324 224, we generate SIRs using 13-layer U-Net style encoder CNN, CNNg. CNN₂ downsamples input images by a factor of 32 and produces our SIR of shape 128x7x7.
Dynamic representation architecture. Unlike an SIR, DIRs combine current observations, histor- ical information, and goal specifications. Our model has three recurrent components: metric map, a l-layer convolutional-GrR
CGRU, (Cho et al. 2014: Kalchbrenner et al.) 2016). and 1-layer LSTM LSTM, (Hochreiter & Schmidhuber] 1997). At time 2 0, the environment provides the agent with observations o, including the agent's ' current egocentric image i, and goal ge (e.g. hide the cup). CNN₂ inputs i, produce the SIR 8 CNNo(i+). Next the CGRU processes existing

---

Published as a conference paper at ICLR 2021
information to produce the (grid) DIR h2 = CG ot. h4-1) of shape 257x7x7. hy is then pro- cessed by the LSTM LSTM, to produce the (flat) DIRs (hg, hg) = 1.STMg(he, 0t, ht-1) where hgg hy are the so-called output and cell states of the LSTM with length 512. Finally the agents' metric map is updated by inserting a 128-dim tensor from CNN₂ into the
representation m from time position in the map corresponding to the agent's current position and rotation, to produce m,. The historical information h2 for the next step now includes (he, hG, h2, m.)-
Actor-critic heads. Our agents use an actor-critic style architecture, producing a policy and a value at each time f. As described in Sec. [ there are five distinct stages in game of cache (E&M, PS, OH, OM, S). Each stage has unique constraints on the space of available actions and different reward structures, and thus require distinct polices and value functions obtained by distinct 2-layer MLPs taking as input the DIR hy described above.
Perspective Simulation. Evidence suggests that in humans, perspective taking, the ability to take on the viewpoint of another develops well after infancy (Trafton et al. 2006). While standard RNNs can perform sophisticated tasks, there is little evidence to suggest they can perform the multi-step if- then reasoning necessary in perspective taking. Inspired by AlphaGo's (Silver et al.) 2017)) intuition guided Monte-Carlo tree search and imagination-augmennte RL agents (Racanière et al., 2017) we
require, in the perspective simulation stage, that our hiding agent evaluate prospective hiding places by explicitly taking on the seeker's s perspective using internal mental rollouts (see Fig. 3c).
To this end, the metric map is processed by CNN to produce an internal map representation M which is used to coarsely evaluate potential hiding locations, see Fig. 3b For a subset of these locations L, M is processed (along with E) to produce new representation M Intuitively, A can be thought of as representing that agent's 's beliefs about what the environment would look like after having hidden the object position f. A "mental" seeker agent defined using an LSTM (not shown in the model figure for simplicity) then moves about M and obtains observations as slices of M see Fig. 3c. The quality of a potential hiding location is then quantified by the number of steps taken by the "mental" secking agent to find the hidden object. After scoring hiding locations in this way, the agent randomly samples position move to (beginning the OH-stage) with larger probability given to higher scoring positions.
Training and losses. Our agents are trained using the asynchronous advantage actor-critic (A3C) algorithm (Mnih et al. 2016) with generalized advantage estimation (GAE) (Schulman et al., 2015) and several (self) supervised losses. As we found it critical to producing high quality SIRs, we highlight our visual dynamics methodology below. See Sec[[for details on all losses.
Visual dynamics replay. During early experiments, we found that producing high quality SIRs using only existing reinforcement learning techniques to be infeasible as the produced gradients suffer from high variance and so signal struggles to propagate through the many convolutional layers. To correct this deficiency we introduce technique we call visual dynamics replay (VDR) inspired by experience replay (Mnih et al. 2015) and intrinsic motivation (Pathak et al.] 2017). In VDR, agent state transitions produced during training are saved to separate process where encoder- decoder architecture (using CNN, as the encoder) is trained to predict a number of targets including forward dynamics, predicting an image from the antecedent image and action, and inverse dynamics, predicting which action produced the transition from one image to another. Given triplet (ig, a, 41), where io. i are the images before and after an agent took action a, we employ several self-supervised tasks including pixel-to-image predication. See Sec. C for more details.
5 EXPERIMENTS
Our experiments are designed address three questions: (1) "has our cache agent learned to pro- ficiently hide and seek objects?"', (2) "how do the SIRs learned by playing cache compare to those learned using standard supervised approaches and when training using other interactive tasks?", and (3) "has the cache agent learned to integrate observations over time produce general DIR repre- sentations?". For training details, e.g. learning rates, discounting parameters, etc., see Sec. C
Playing cache. Figures 3a to 31 qualitatively illustrate that our agents have learned to explore. navigate, provide intuitive evaluations of hiding spaces, and manipulate objects. Figures 4 and 4b show quantitatively that hiders from later in training consistently chose hiding places that our seekers are less likely to find. Also, for hiding spots generated by human experts and by an automated brute force approach, seekers from later in training more frequently find the hidden objects. Interestingly,
5

---

Published as a conference paper at ICLR 2021
Exploration Trajectory
Evaluation Hiding Locations
Mantal Rollout
Evalustion of Hiding Locations (Local)
Object Manipulation
Initial Placeubo Localions
Contained
Rehind
Contained In
Behind
Seeking Trajectory
Hidden object location
Seaking agent
Agent opened object
oow Avllue
High
Agent pach
rollouts paths
Monipulsted object path
Figure 3: Cache agent. Actions and decisions made during game cache. Hider's trajectory during exploration. Insets show first person views before and after an open action. [b] Heatmap displaying the agent's initial beliefs of high quality hiding locations. Insets show how this hcatmap varies when the secking agent's ' start position is changed. Initial beliefs in b are refined by the agent performing explicit mental rollouts of the secking agent. Three trajectorics, stopped after 30 steps, are highlighted. Faint white lines are paths from other simulated trajectories. Insets show rollouts when varying the hiding location and agent starting location. d After choices made in bic hiding location for the object must be chosen in pixel space. Top row shows the estimated probability that the object be placed in location. The bottom averaged over the episode, of agent trying to hide the object location. @ The object's 's' trajectory through space shown as the agent places between two trash cans. Top down view of seeker.
shows the probability,
the seeker is able locate objects hidden by humans and the brute force approach with similar level of accuracy. Surprisingly, shown in Table human seekers find objects hidden by humans and by our hiding agent at similarly high rates: on training scenes human seekers find our agent's objects 77% of the time and their fellow humans hidden objects 80% of the time, on test scenes humans are more successful at finding our agent's objects (90% success rate). Finally note that the (effectively random) hg agent hides objects generally very near its starting location and such positions are very frequently found by all agents. Somewhat surprisingly, the $te6 agent appears very slightly worse at finding such positions than the seeker 8p agent. This can be explained by the $1e6 agent having learned to not expect such an audacious hiding strategy.
Static tasks. We evaluate SIRs depth estimation by freezing the SIRs and training decoder models. These evaluations are performed on synthetic (AI2-THOR) data well natural images, which can be considered out of domain.
suite of standard computer vision targets such monocular

---

Published as a conference paper at ICLR 2021
Training Scenes
Seeking Agent
9g
a/
S4 Sed
Testing Scenes
Freq. Object Found
1.0
Freq. Object Found
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
hg
his hasee
M
hg
his h4.506
E
M
Cache Agent
Exhaustive
Human
Cache Agent
Exhaustive
Human
Seeker
Hider
Split
Freq.
Found
Seeker Steps
$4.5e6
ha. Sedi
Train
0.44 = 0.01
362 + 4
$4,5e6
Human
Train
0.29 + 0.01
406 4
Human
h4 be.
Train
0.77 = 0.03
198 + 15
Human
Human
Train
0.80 = 0.03
204 16
84.5eG
ha
Test
0.64 = 0.01
265 +
$4.5e6
Human
Test
0.33 + 0.01
386 1
Human
h4
Test
0.90 = 0.02
108 +
Human
Human
Test
0.77 + 0.03
155
12
Figure 4: Evaluating the cache agent. Frequency that seekers from three points during training (after O, 1-10", and 10 training episodes) find objects hidden by: hiders from the same training points, an automated exhaustive approach (E=casy, M=medium, H=hard), and humans. Error bars denote 95% Cls. c Results of best hider (h4.5ea) and seeker (54.5ea) competing with humans. Here the mean frequency of finding the hidden object reported (with 95% Cls) along with the number of steps the seeker took find the hidden object (with 95% Cls).
Predictions with synthetic images We use SIRs to predict geometry and appearance-basee targets namely depth, surface normals, object class, object depth and object normals, as well affordance- based targets namely traversable surfaces and object openability. On the test set, our (Cache) SIR consistently outperforms multiple baselines (Fig (5) including: Auto Encoder on A12-THOR im- ages, Navigation agent within AI2-THOR, and Classifier CNN trained to classify objects in A12-THOR images. Importantly, the Cache SIRs compare favorably (statistically indistinguishable performance for 4 tasks and statistically significant improvement for the remaining 2) to SIRs pro- duced by a model trained on the 1.28 million habe-labelled images from ImageNet (Deng et al, 2009), the gold-standard for fully-supervised representation learning with CNNs. Note that SIR is trained without any explicitly human labeled examples.
Predictions with natural images As images from AI2-THOR highly correlated and synthetic, we would not expect cache SIRs to transfer well to natural images. Despite this, Fig. shows that the cache SIRs perform remarkably well: for SUN scene classif. (Xiao al., 20T0 as well the NYU V2 depth (Nathan Silberman & Fergus 2012) and walkability tasks (Mottagi et al. (2166). the cache SIRs outperform basclines and even begin approach the ImageNet SIR performance. This shows that there is strong promise in using gameplay as method for developing SIRs. Increasing the diversity and fidelity of synthetic environments may allow us further narrow this gap.
Dynamic tasks. While CNN models can perform surprisingly sophisticated reasoning indi- vidual images, these representations are fundamentally timeless and cannot hope to model visual understanding requiring memory and intentionality. Through experiments loosely inspired by those from developmental psychology we show that, memory enhanced representations emerge within our agents' DIRs when playing cache. In the following experiments, ensure that objects and scenes used to train the probing classifiers are distinct from those used during testing. For a description of all baseline models, see Sec
Developmental psychology has an extensive history of studying childrens' capacity to differentiate between object being contained in, or behind, another object (Casasola et al. 2003). Following this line of work, our hider is made to move an object through space until the object is placed within, or behind, receptacle object (see Fig. 7ak. The DIR then used predict the relationship between

---

Published as a conference paper at ICLR 2021
Auto Encoder
Navigation
* Stat significant
Walkability
Depth
Surface Normals
Cache
Classifier
Walkable
Near
Non walkabic
Depth
Input
Surface
Normals
Classification
Warse
Better
Prediction Ground Truth
Openability
Traversable
Surfaces
Object
Depth
Object
Normals
2.0
Loss Relative ImageNet Model
Figure 5: SIR evaluation synthetic images. Left: test-set losses (averaged first within each scene) of the different models relative the fully supervised baseline (ImageNet). Values less than indicate performance better than that of the ImageNet SIR (stat. significance computed a <0.01 level using a generalized linear mixed-model). Right: qualitative examples of geometric and affordance-based image predictions using SIR.
Input
Ground Truth Prediction
Depth
Sceme Class, (SUN)
Depth (NYU)
Wallabble (NYU)
Model
Top Ace.
Abs
Error
IOU
Cache
0.311
0.523
Anto. Enc.
9.23
0.408
0.366
Nav.
6.31
0.377
0.408
Walkabbiitt
Classifer
2.338
0.39
0.317
hugNet 41.1
0.27
0.605
Figure SIR evaluation on natural images. Quantitative and qualitative results for natural image tasks.
the object and the receptacle using a linear classifier. Cache DIRs dramatically outperform other methods obtaining 71.5% accuracy (next best baseline obtaining 63.4%)
Two and half month old infants begin to appreciate object permanence (Aguiar & Baillargeon 2002) and are surprised by impossible occlusion events highlighting the importance of permanence as building block of higher level cognition, and begin acting on this information at around nine months of age (Hespos al.] 2009). Inspired by these studies, we first find that (Fig. 7b) linear classifier trained on cache DIR able to infer the location of an object becomes fully occluded with substantially greater accuracy (64%) than the next best competing baseline (59.6%) Second (Fig. 7c), test agents' ability rememher and act previously seen objects. We find that our seeker's DIR can be used to predict whether it has previously seen hidden object after being foreed rotate away from it, having the object vanish, and taking four additional steps. Surprisingly we see that the seeker has statistically significantly larger odds of rotating back towards the object after seeing than control agent (mean difference log-odds of rotating back towards the object vs. rotating in the other direction between treatment and control agents 0.08, 95% CI =0.02) demonstrating its ability nence. Numerical markers denote the number of steps since the agent has seen the object.
continuous path after
on its understanding of object perma-
Piaget argues that the capacity for seriation, ordering objects based on common property, develops after age seven (Piaget, 1954). We cannot hope that our agents have learned seven year old's understanding of seriation but we hypothesize that rudimentary seriation should be possible among properties central to cache. One such property is free space, the amount of area in front of the agent that could be occupied by the agent without collision. We place our agent randomly within the environment, rotate it in 90º increments (counter-) clockwise, and save SIRs and DIRs after each rotation (see Fig. 7d). We then perform two-image seriation, namely to predict, from the representation at time whether the viewpoint the previous timestep / contained more free
8

---

Published as a conference paper at ICLR 2021
Contained or behind?
Where the object?
Did you see the object?
Which has free space?
013
2040
Inturppon
OR Icucnel
DR Randd
DRRACCNN Cushre
DR JACNN Clusal
DR O4C4NWWNog
SRiinagsNeo
SRR
Steps
Figure 7: DIR evaluation. Schematic illustrations, example images from A12-THOR and numerical results of our psychologically motivated experiments. All results are for the test set with 95% confidence intervals. See Sec. [oror detinitions of plot labels and Sec. for a description of these experiments. Note: (i) X-coordinates in our line charts are offset slightly from one another so as to improve readability. (ii) Our plots show the performance of the representations the amount of data used in training the probing linear classifier increases; but in the main text, we report only the values corresponding the largest training sizes considered.
space. Our cache DIR is able to perform this seriation task with accuracy significantly above chance (mean accuracy of 84.92%, 95% CI +2.27%) and outperforms all baselines.
6 DISCUSSION
Modern paradigms for visual representation learning with static labelled and unlabelled datasets have led to startling performance on a large suite of visual tasks. While these methods have revo- lutionized computer vision, it is difficult imagine that they alone will be sufficient to learn visual representations mimicking humans' capacity for visual understanding which is augmented by in- tent, memory, and intuitive understanding of objects and their physical properties. Without such representations it seems doubtful that we will ever deliver one of the central promises of artificial intelligence, an embodied artificial agent who achieves high competency in an array of diverse, de- manding, tasks, Given the recent advances in self-supervised and unsupervised learning approaches and progress made in developing visually rich, physics- basedd embodied Al environments, we be- lieve that it is time for a paradigm shift via a move towards experiential, interactive, learning, Our experiments, in showing that rich interactive gameplay can produce representations which encode an understanding of containment, object permanence, seriation, and free space suggest that this di- rection promising. Moreover, we have shown that experiments designed in developmental psychology literature can be fruitfully translated to understand the growing competencies of arti- ficial agents. Looking forward, real-world simulated environments like A12-THOR have focused primarily on providing high-fidelity visual outputs in physics-enabled spaces with comparatively little effort being placed on developing other sensor modalities (e.g. sound, smell, texture, etc.). Recent work has, however, introduced new sensor modalities (namely audio) into simulated envi- ronments (Chen et al. 2020a Gan et al. (2200)) and we suspect that the variety and richness of these observations will quickly grow within the coming years. The core focus of this work can eas- ily be considered in the context of these emerging sensor modalities and we hope in future work to examine how, through representation and gameplay, an artificial agent may learn combine these separate sensor streams to build a holistic, dynamic, representation of its environment.
ACKNOWLEDGMENTS
We thank Chandra Bhagavatula, Jaemin Cho, Daniel King, Kyle Lo. Nicholas Lourie, Sarah Pratt, Vivek Ra- manujan, Jordi Salvador, and Eli VanderBilt for their contributions in human evaluation. We also thank Oren Etzioni, David Forsyth, and Aude Oliva for their insightful comments on this project and Mrinal Kembhavi for inspiring this project.

---

Published as a conference paper at ICLR 2021
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning see by moving. In /CCV, 2015.
Andréa Aguiar and Renée Baillargeon. Developments in young infants' reasoning about occluded objects. Cognitive psychology. 45(2):267, 2002.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Mare- Alexandre Côté, and R. Devon Hjelm. Un- supervised state representation learning in atari. In NeurlPS, 2019.
Marcin Andrychowicz. Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In NewrlPS, 2017.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In ICLR, 2020.
Christopher Berner, G. Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, D. Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, R. Józefowicz, Scott Gray, C. Olsson, Jakub W. Pachocki, M. Petrov, Henrique Pond'e e de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Schneider, S. Sidor, llya Sutskever, Jie Tang, F. Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv. 2019.
Yuri Burda, Harri Edwards. Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros, Large-scale study of curiosity-driven learning. In ICLR, 2019.
Gordon Burghardt. The Genesis of Animal Play: Testing the Limits. 2005.
Marianella Casasola, Leslie Cohen, and Flizabeth Chiarello. Six-month-old infants' categorization of con- tainment spatial relations. Child development, 74(3)6679-93, 2003.
Boyuan Chen, Shuran Song, Hod Lipson, and Carl Vondrick. Visual Hide and Seek. arXiv, 2019.
Changan Chen, Unnat Jain, Carl Schissler. Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation 3d environments. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI, volume 12351 of Lecture Notes in Computer Science, pp. 17-36. Springer, 2020a. doi: 10. 10.1007/978-3-030--88339--
httthttss///....coo crt. 1tth:/11..111/191 1001 978 -30-55533-------
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for con- trastive lcarning of visual representations. arXiv, 2020b.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statisti- cal machine translation. In EMNLP), 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kehui Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Carl Doersch, Abhinay Gupta, and Alexci A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
David F. Fouhey, Abhinav Gupta, and Martial Hebert. Data-driven 3D primitives for single image understand-
ing. In ICCV, 2013.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Damian Mrowea, Michael Lingelbach, Aidan Curtis, Kevin T. Feigelis, Daniel M. Bear, Dan Gutfreund, David D. Cox, James J. DiCarlo, Josh H. McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threedworld: A plat- form for interactive multi-moda physical simulation. CoRR, abs/2007.0495.. 2020a. URL https: //arxi //arxiv.crg//aeeee...eeaaaoaeaaa hbtpg//bb///00/// 04954]
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boging Gong, and Joshua B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA, 2020b.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal conditioned policies. ICLR, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICL.R, 2018.
10

---

Published as a conference paper at ICLR 2021
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering interactive environments. In CVPR, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Susan Hespos, Gustaf Gredebäck, Claes von Hofsten, and Elizabeth S. Spelke. Occlusion is Hard: Comparing predictive reaching for visible and hidden objects in infants and adults. Cognitive Science, 33(8):1483- 1502, 2009.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compur., (183::173--7888 1997.
Hexiang Hu, I.. Chen, Boqing Gong, and F. Sha. Synthesize policies for transfer and adaptation across tasks and environments. In NeurlPS, 2018.
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles. Neural task graphs: Generalizing to unseen tasks from single video demonstration. In CVPR, 2019.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NeurlPS. 2015.
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G. Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task completion. In CVPR, 2019.
Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, S. Lazebnik, Aniruddha Kembhavi, and Alexander G. Schwing. A cordial syne: Going beyond marginal policies for multi-agent embodied tasks. In ECCV, 2020.
Rico Jonschkowski and Oliver Brock. Learning state representations with robotic priors. Autonomous Robots, 2015.
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. Grid long short-term memory. In ICLR, 2016.
Diederik Kingma and Jimmy Adam: method stochastic optimization. ICLR, 2015.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Al2-THOR: An Interactive 3D Environment Visual Al. arXiv, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. NeurlPS. 2012.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learning. In AAAI, 2017.
Russell Lenth. emmeans: Estimated Marginal Means, aka Least-Squares Means, 2019. R package version 1.4.
Tsung- Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. CoRR, abs/1612.03114, 2016. URL 1612. 03144
M. Lohmann, J. Salvador, Aniruddha Kembhavi, and R. Mottaghi. Learning about objects hy learning to interact with them. arXiv, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hass- abis. Human-level Control through Deep Reinforcement Learning. Nature. 2015.
Volodymyr Mnih, Adria Puigdomencch Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML, 2016.
Roozbeh Mottaghi, Hannanch Hajishirzi, and Ali Farhadi. task-oriented approach for cost-sensitive recogni- tion. In CVPR, 2016.
Tushar Nagarajan and Kristen Grauman. Learning affordance landscapes for interaction exploration in 3d environments. arXiv, 2020.
11

---

Published as a conference paper at ICLR 2021
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.
Jean Piaget. The construction of reality in the child. Basic Books, 1954.
Jose Pinheiro, Douglas Bates, Saikat DebRoy, Deepayan Sarkar, and Core Team, nime: Linear and Nonlinear Mixed Effects Models, 2019. R package version 3.1-141.
Lerrel Pinto, Dhiraj Gandhi. Yuanfeng Han, Yong-Lae Park, and Abhinav Cupta. The curious robot: Learning visual representations via physical interactions. In ECCV, 2016.
R Core Team. R: Language and Environment for Statistical Computing. R Foundation for Statistical Com- puting. Vienna, Austria, 2019.
Sébastien Racanière, Theophane Weber, David Reichert, Lars Bucsing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. imagination-augmennt agents for deep reinforcement learning. In NeurlPS, 2017.
Benjamin Recht, Christopher Rc, Stephen Wright, and Feng Niu. Hogwild: lock-free approach to paralleliz- ing stochastic gradient descent. In NeurlPS. 2011.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICT.R, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Cosputer-Assisted Intervention, 2015.
Ashutosh Saxena, Sung H. Chung, and Andrew Y. Ng. Learning depth from single monocular images. In NeurlPS, 2006.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- High-DDimensionl Con- tinuous Control Using Generalized Advantage Estimation. In ICLR, 2015.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub- pixel convolu- tional neural network. CoRR,
2016. URL ltttt::pp//xii/ org/ abs 111./11119911818
David Silver, Julian Schrittwieser, Karen Simonyan, Joannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the Game of Go without Human Knowledge. Nature, 2017.
J. Gregory Trafton, Alan C. Schultz, Dennis Perznowski, Magdalena D. Bugajska, William Adams, Nicholas L. Cassimatis, and Derek P. Brock. Children and Robots Learning to Play Hide and Seek. HRI, pp. 242, 2006.
Oriol Vinyals, L. Babuschkin, W. Czamecki, Michaöl Mathieu, A. Dudzik, Chung. D. Choi, R. Powell, Timo Ewalds. P. Georgiev, Junhyuk Oh, Dan Horgan. Manuel Kroiss, Ivo Danihelka. Aja Huang. L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, A. S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, Budden, Yury Sulsky, James Molloy, T. L. Paine, Caglar Gulechre, Ziyu Wang, Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinncy, O. Smith, T. Schaul, Lillicrap, K. Kavukcuuglu, Demis Hassabis, Chris Apps, and D. Silver. Grandmaster level in starcraft ii using multi- agent reinforcement learning. Nature, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016.
William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-aware embeddings. In ICLR, 2020.
Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to leam: aelf-adeptive visual navigation using meta- meta-learning. In CVPR, 2019.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling Task Transfer Learning. CVPR, 2018
12

---

Published as a conference paper at ICLR 2021
Zeping Zhan, Batu Aytemiz, and Adam M. Smith. Taking the scenic videogames. arXiv, 2018.
Automatic exploration for
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.
2016,
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 VIRTUAL ENVIRONMENT AND PREPROCESSING
While A12-THOR can output images of varying dimensions and quality we find that simulating images of size 224x224 with the "Very Low" quality setting provides a balance of image fidelity and computational efficiency. AIOR provides RGB images with the intensity of each channel encoded by an integer taking values between 0 and 255. Following standards used for natural images, we preprocess these images by first scaling the channels so that they are floating point numbers between 0 and I after which we normalize the R, G, and B channels with means 0,485, 0.456, 0.406, and standard deviations 0.229, 0.224, 0.225. Agents are constrained to lie on a grid with 0.25 meter increments and may face in one of four cardinal directions with 0 being north and 90º being east. The agent's 's camera is angle 30 below the horizontal, and lies at a height 1.5765 meters when standing and 0.9015 meters when crouching. Following conventions in computer graphics, the agent's position is parameterized a single (r, y, 2) coordinate triple with z and x corresponding vertical movements. Other than when standing and crouching, our agent's 's vertical position is fixed. Letting Z denote the integers, an agent's position is uniquely identified by location tuple (z,, 2, E (0.25-Z)) (0.25- 2) x f0, 90, 180, 270} x (0, 1} where 2, 2 are as above, r denotes the agent's 's rotation in degrees, and equals if and only if the agent is standing. It will be useful in the following to be able to easily describe subcomponents of location tuple. to this end we let proj I2 - I²" be the projection removing the first coordinate of r; s), and similarly for proj 20 p.roj-rj proj A12-THOR
have field of view of 90º, is fixed an
movement in the horizontal plane and with reserved for
location tuple so that proj-z s) contains a total of 150 scenes evenly grouped into five scene types, kitchens (1-30), living rooms (201-230), bedrooms (301-330), bathrooms (401 (430), and foyers (501-530). Excluding foyers, which are reserved for our dynamic image representation experiments and used nowhere else, we consider the first 20 scenes of each scene type to be train scenes, the next five of each type to be validation scenes, and the last five of each type to he test scenes. When training our cache agents we only use kitchen and living room scenes as these scenes are relatively large and often include many opportunities for interaction.
B CACHE STAGES
cache the hiding agent, or hider, attempts to place a goal object in a given A12-THOR scene such that the seeking agent, or seeker, cannot find it. This goal object is one of five standard household items, see Fig. [. being either loaf of bread, cup, knife, plunger, or tomato. Recall that we divide the game of cache into the live stages Senges (E&M, PS, OH, OM, S}. The hider participates in the first four of these stages and the seeker the last stage. Each of these stages is associated with a distinct discrete collection of possible actions, see Table G. When a stage 8 E Stiness is instantiated, so that the scene has been chosen and all goals specified, we call this instantiation an 8-episode. During an s-episade, an agent takes actions, each taking a single unit of time called a step, until the goal state has been reached or maximum number of steps, N steps has been taken. Episodes begin with time 0 and advances by 1 after every action regardless of whether not that action is successful. Note that these episodes must sequentially instantiated and we call a collection of sequentially instantiated episodes ending with an S-episode a game of cache.
In the following we deseribe these stages in detail along with the metrics we use to track the quality of the learned policies and assign rewards. In order to describe these metrics we require the following sets. First let Sysaee (S%deee) be the set of all location tuples visited by the hider (seeker) by time step of its episode. Similarly we let Supened (Sugene)) be the set of all objects the hider (seeker) has opened by time step t. Finally we let Sextrap be the of all location tuples that can, assuming the agent can move through objects, be reached by starting at some location in gvisted and moving the agent ahead by 0.25m and then either staying put, moving left by 0.25m, or moving right 0.25m.
B. EXPLORATION AND MAPPING (E&M)
game of cache begins with an E&M episode. During this episode, the agent may traverse the scene and interact with objects while building a representation of its environment for up to N stos = 200 steps. In our implementation, this representation is a metric map, see the model architectures section below, which is then available in the subsequent PS- and OH-episodes. Let grochable be the set of all feasible location tuples in given episode and, similarly, let A copeaable be total number of objects that can opened. We evaluate the quality of exploration with three metrics:
Cooverage = jproj_, Swistee /// proj (Srmhhii
Q gomenge+ =
|proj_, (H ) A proj pr ( //proj (Srentutte ), and
Q Openss |S /N openable
14

---

Published as a conference paper at ICLR 2021
Qeavenae and Q evaluate the agent's ability to traverse the entire scene while Quea'' evaluates how fre- quently the agent interacts with its environment.
B.2 PERSPECTIVE SIMULATION (PS)
After exploration the hider allowed to consult the representation it has developed of its environment and must, from that representation, pick a single location tuple to which the agent will then be moved. As we do not give the agent a global reference frame it must choose this location tuple by specifying a location relative its final position after the E&M episode. As the hiding agent is not constrained to pick valid location tuple, it may, for instance, choose location that is outside of the seene already occupied by object, such choice is considered failed action. As the quality of choice of location tuple in this episode is tied intimately with the outcome of the OH-episode we will measure the progress of training for both episodes using the same metrics described in the section below.
B.3 OBJECT HIDING (OH)
In the OH--epsode the agent has been moved to a location tuple determined in the PS-episode and is now holding the goal object. In this episode the agent is not allowed to perform any navigational actions beyond standing or crouching, but may interact with objects by opening or closing them. Most importantly, the agent must choose location current view which should hide the item. To discretize the possible hiding locations recall that the images available to agents are of size 224x224. Viewing C' = (1, ..., 224}² as a grid of pixel coordinates, let (Ct,) (55 be partitioning of C into a 7x7 7 grid so that Ci5 = [(32- i + k. 32 j +1) | S 1, k S 32} is of size 32x32. When specifying location in which to hide the object, the agent picks one of these grid locations along with one of the positional keywords in the set Sumnduliy = [ONTOP, CONTAINEDIN, BEHIND). After choosing a location we begin an OM- OM-episode, described below, during which the agent attempts to move the object through space and to the specified location. In order to reduce the complexity of specifying plausible object locations, potentially difficult when, e.g., specifying final location behind another object, we consider an object to have been successfully placed at the grid location Ctj. when using the CONTAINEDIN, BEHIND keywords, even in those cases that moved to C(-)1)) C(++)) instead. If OM-episode is successful given this relaxation, all future actions specifying object locations are considered failures. If the OM-episode is unsuccessful the hiding agent may continue to specify locations, causing the item to be picked up and a new OM-episode to begin, until manipulation is successful. If the hiding agent fails successfully place the object within N seps dropped. Otherwise, the agent may continue to take other actions, closing a cabinet into which has placed an object for instance, or specify that it is done by performing the SeadyFooSookke
15 steps the object is returned its default position in the agent's 's hand and then simply
action.
Objectively quantifying a hiding location is difficult as the quality of a location is coupled with psychology and artifacts of perception. Indeed, the phenomenon of an object being lost "in plain sight" suggests that seemingly irrational hiding strategies may he highly successful. Nevertheless, we require some automated metric with which to measure
quality of a hidden object and, for this purpose, define the following three metrics. First we let N visible from be the number of feasible location tuples from which any portion of the goal object can be seen at any distance. To complement A Aisibbl from with measure that takes the seeking agent's start location into account, we let ArBrSsuep he the number of steps it takes breadth first search (BFS) agent, starting at the seeking agent's start location, to find the goal ohject. A single step for the BFS agent corresponds to teleporting to the location tuple closest to the seeker's start position that has not yet been visited hy the BFS agent and checking if the goal object can be seen from any rotation, crouching or standing, from that location. The BFS agent is allowed to see through openable objects and the goal object is considered found if any portion be seen any distance. If the BFS agent cannot find the object then BBS seps equals the maximum number of steps, proj p (Steachuble) }),, in which case we let gBES found object equal O, otherwise gRts found abject equals 1. Finally we also define the proportional versions of two of the above metrics.
of the object
visible from%
Nvisible from
/|5 strchabble and
RFS stepess
RPS seps /1 proj (proj ( (Smaacaall)))
B.4 OBJECT MANIPULATION (OM)
In the object manipulation episode the agent set at fixed location tuple holding the goal object and given a grid location 1 < i,j < along with the positional keyword MODALITY E Smodialit During this episode the agent can interact with objects by opening them, move the object held in its hand in one of six axis-aligned directions, drop the goal object. The object manipulation episode is considered successful if, after dropping the goal object, the goal object is in the correct grid position specified by i,j and respects the constraints imposed by MODALITY. In particular, after manipulator has dropped the object, we obtain from A12- THOR the sets S SONTOP pis. (coordinates of pixels corresponding to the object that are visible from the agent's current viewpoint), S BHINND (pixel coordinates of the object that are visible only after making all other
15

---

Published as a conference paper at ICLR 2021
objects in the scene invisible). and the value 8CoNTAINEDIN E {0, 1) which equals if and only if the goal object is contained within another object. If d = 1, then we let SCoNTAnNntt pix. SONTöP pix. U S SBIIND pix otherwise S SCONTANNIINN pix. 0.
If [SoNTor | [ p < 10, the episode is considered failure. Otherwise, success depends on which
MODALITY was specified and on the quantity M = maxi,, max([Ca, MSSToo pix [Ciy SHRIND (SBuInn pi. (Cz,5 n SCONTAINEDIN 1), l), Le. maximum number of pixels in any single modality and grid location.
If MODALITY = ONTOP then the episode was successful if [Cay n n& g pix 2 0.95 M, i... if there are at least 95% as many pixels in the specified location and modality as in in the grid location and modality with the most of the object's pixels.
Similarly if MODALITY BEHIND (or MODALITY CONTAINEDIN) then the episode was successful [Cig n S BEHIND pts 2 0.95 - M (or [C,, n SONTABNEDN pix. 2 0.95 M).
To track the progress of this episode simply record successes, namely we let maget E {0, 1) equal if and only if object manipulation was successful.
B.5 SEEKING (S)
In the seeking episode the secking agent is tasked with finding the object hidden by the hiding agent. The seeking agent begins from the same location tuple as the hiding agent began the E&M-episode and given a budget of Ng stpps = 500 steps before the episode is considered a failure. During these 500 steps the seeking agent may perform any of the actions allowed in the exploration and mapping episode along with one additional action, claimvis miiiisble If the seeking agent performs the clainvisibll action successfully the sceking agent has successfully found the goal object and the episode ends.
Just as when attempting to automatically measure the quality of a hiding location, developing objective mea- sures of the quality of a sceking policy is challenging. As deseribed previously, a secking strategy that is highly successful against one hiding strategy may be fatally flawed when faced with another. To remove any dependence
hiding agent obtained through reinforcement learning brute force approach, three collections of hiding locations for each scene s and object o, Sany pos. Smmed pos. and Sheel pos. To define these collections ate collection Lo,s (fo,,, 1 S... I No..] of No,, >> possible hiding locations. For each Lo,s, we compute Uo.a,i which equals the Quisibie fromnss metric defined above for the given hiding location. Letting Vo.. = (oVo... 1 < S No.. b. consider hiding position fo.s. hard if Vo... is within the lower fifth percentile of Vo,s values, medium Vo,s. I 0.15 and Vo,,, between the fifth and 20th percentiles, and finally easy if it is neither hard nor medium difficulty. The set Say pos, and similarly for Stndd pos, and Shard pos, then randomly sampled collection of, approxinately. 20 casy locations from Lo.s. We then quantify the quality a seeking policy by computing the success rate of the seeker when the hiding locations are restricted to those from each of the above three sets.
generate, through an exhaustive,
A12-THOR to, for each goal object and scene 8, gener-
C TRAINING DETAILS AND LOSSES
We train our cache agent using eight GPUs with one GPU reserved for running A12-THOR processes, one reserved for VDR, and the other six dedicated to training with reinforcement and self supervised learning. Each of the cache stages generates its own losses, detailed below, which are minimized by HOGWILD! (Recht et al. 2011 style unsynchronized gradient descent train model parameters. Reinforcement learning in the E&M. OH, OM, and S stages is accomplished using the asynchronous advantage actor-critic (A3C) algorithm (Mnih) ct al. 2016) with generalized advantage estimation (GAF) (Schulman ct al. 2015). For the A3C loss we let the discounting parameter 0.99 (except for in the OH-stage where y 0.8), the entropy weight 8 0.01, and GAE parameter 7 1. We use the ADAM optimizer (Kingma & Ba 2015) with AMSGrad (Reddi et al.) 2018). moving average parameters 3, = 0.99, 32 0.999, learning rate of 10 for VDR, and varying learning rates for the different cache stages (10 -4 for the E&M, OH. OM, and stages, 5 10 for the PS-stage).
In the following we give an overview various training details, namely: (1) data augmentation, (2) (self) super- vised losses, (3) and reward structures for the E&M, OH, OM, and S stages necessary for computing the A3C loss.
C.I TRAINING DATA AUGMENTATION
To increase diversity of the images seen by our cache agents during training, apply two forms of data augmentation to A12-THOR.
16

---

Published as a conference paper at ICLR 2021
Random world rotation, Recall that our A12-THOR cache agents are constrained to 90" rotations and move on a 0.25mx0.255 grid. while this is beneficial that it abstracts many of the technical complexities of movement in true robotics, it has the disadvantage of reducing the variety of images that can be seen by our agent. In particular, the default orientation of the agent is "axis aligned" such that it is generally moving parallel to walls and objects and thus it will not see objects and rooms from large variety of orientations. To remedy this limitation while still maintaining grid-constrained movement, we, with a 50% probability the start of every new game of cache, randomly rotate the agent by some degree in (O...., .39).. After this initial rotation the agent moves along a 0.25mx0.22m grid as usual but, because of this rotation, it will now observe objects at a variety of different orientations.
Image augmentation. To improve generalization, it is common with the computer vision literature to apply random transformations images before giving them as input neural network models for prediction. We follow this approach and, at the beginning of every game of cache, we randomly select:
(a) A resized crop function Ci.r,, which, given an input image, crops pixels from the left, r pixels from the right, pixels from the top, and pixels from the bottom, and then resizes the cropped image back to the start dimensions with bilinear sampling. In particular we select (4, r, b) = (0, 0,0, O, 0) with 0.5 probability and, otherwise, select (l, r. t, b) uniformly random from (1,..., 10}".
(b) A color jitter function J. This function J is chosen vary the brightness, contrast, saturation, and huc of input images by small amounts.
(c) A rotation function Ra. Here Ra rotates the input image about its center by d degrees and we choose d random to equal 0 with probability 0.5 and, otherwise, sample d uniformly at random from
{ (-3,...,33 3).
With the above functions in hand, apply the composed transform a C, to every image seen during the game of cache.
C.2 VISUAL DYNAMICS REPLAY
Here we give an overview of our VDR training procedure. We direct anyone interested in exact reproduction of our VDR procedure to our code base. During training our agents save triplets (fn, a, in). where in is an antecedent image. a is an action taken (with associated success or failure signal), and in is the subsequent image resulting from taking action a when the agent's view was io. to queue that is then periodically emptied and processed by the VDR thread. The VDR thread processes the triplets by saving them into buffer, similar to an experience replay buffer (Mnih et al. 2015). To ensure that certain actions do not dominate the buffer we group and selectively subsample the triplets added to the buffer. Moreover, the buffer automatically removes triplets to ensure that triplets do not exceed some fixed age and that the buffer does not exceed its max size of approximately 20,000 triplets.
After processing incoming data and storing triplets in the dynamies buffer, the VDR thread then randomly partitions the buffer into batches of size 64 and, for each batch, computes the self-supervised losses detailed below and performs a single gradient step to minimize the sum of these losses. We call a single pass through all batches an epoch. VDR then alternates hetween processing incoming triplets, forming batches, and training for single epoch.
The losses minimized in VDR include the following.
Inverse dynamics loss: predicting from ig and i1. From the VDR model we obtain, from and i1, a vector imv dyn of length A the total number unique actions across all agents. By assigning each action a a unique index, index(a) E {1.... Nuccion ). we then compute the inverse dynamics loss as the negative cross entropy between the point mass distribution gindex((s) and softmax (". d.n..
Forward dynamics loss: predicting i from a and ip. From the VDR model we obtain, from in and a, tensor 2 Puxn E 2xxax for E {14, 28,56, 112,224] where Tix" represents the prediction of x7xx which equals in downsampled to have spatial resolution X Lett An be the, per pixel, E1 error hetween Pax" and ftx",, and let A, be its mean. We let the forward dynamics loss equal Σngg A.. practice we found it helpful to compute An as a weighted average with larger weight given to areas of the image that have changed.
Pixel difference loss: predicting which pixels in én are different from those in in from in and a. Computed similarly to the forward dynamics loss using tensor T difi. returned from the VDR model except using binary cross entropy loss to predict which pixels are different between ig and 41.
Pixel quality loss: predicting contidence of predictions of forward dynamics. From the VDR model we obtain 1 x 224 x 224 tensor Tromiidence, We compute the pixel equality loss as the mean squared error between Toeeideence and A 224-
17

---

Published as a conference paper at ICLR 2021
Action success loss: from a and io predict if the action would be successful. From the VDR model we obtain, from a and iu. Twould saccedd The action success loss is then computed as the binary cross entropy loss between 7oooud sucordd and the indicator guttion did sucoced
Note that we do not compute the forward dynamics loss and pixel quality loss on those triplets whose action a (sce table Table G. for all action types) is Rotateieft, RotateRight F Placeat, ReadyForSccker or Claimvis sill.. This is because such actions either cause very unpre- dictable changes (e.g. Rotateleft results in the agent looking at entirely separate portion of the scene) no pixel changes (e.g. ReadyFor
OpenAt, CloseOb
action simply ends episode and changes nothing).
C.3 TRAINING PERSPECTIVE SIMULATIION
this section. when we refer to a location tuple, we mean location tuple without the standing component. Recall that in the PS-stage the agent must consult its map of the environment and, from this map, choose a location tuple where from which it will begin the OHl-sage. In the following we describe our implementation of the PS -tase We begin by deseribing the two-phase process by which a location tuple is chosen and then describe the losses we use to train this process. In the below description it will be helpful define the set sOft outcomes = {fail} (10, 1 x {0, 1, x {1, ..., 7))) where fail S0 ouicomes denotes the event in which the OH-episode did not successfully choose a hiding location (so the object was simply dropped) and (s, m. i. E gOH outcomes denotes the event in which, during the OH-stage, the agent successfully hid the object and
the hiding agent was standing when it hid the object if s = and was otherwise crouching,
the hiding agent hid the object with PlaceAt [,, 1, action.
For simplicity, we will say object was hidden at (f, e) to mean that, during the OH-stage, the agent was located at and e E 5oll outeomes occurred. We now describe the two phase process for choosing location tuple.
1. Initial evaluation (recall Fig, 3b). The agent assigns score all location tuples with the goal of highlighting locations that deserve further evaluation. This score, for particular location tuple f, is computed as the weighted average
Σ eE externnc
softmax(P'). V.
where P and V are generated by the agent and are described below.
2. Mental rollout (recall Fig. 3c The agent begins by picking the five highest scoring location tuples (fi, E from the previous phase along with five randomly selected location tuples {la,.., (.o)). For each 1 S i S 10, the secking agent simulates the outcome running the OH-stage with the agent at at lo- cation & three times. Namely the agent itcratively samples three distinct elements e11, e22, ega E 5 ourcomes with the probability of sampling eij SOH ouccomes at every iteration being proportional to softmax (atimmxxxPP).)) (P'))..)-
Then, for each 1 < i < 10 and 1 S < 3, the agent simulates 50 rollout trajectories (1aza) of the seeking agent in the hypothetical setting in which the object was hidden (L,, Let jay 50 >in, #steps(t.ga) be the average number of steps taken in the simulated seeker trajectories (t= Straightforward analysis, using the probabilities defined in softmax (P*)) ), shows that there is closed-form solution for weights way so that μ4, ΣS= Weyfhy is an unbiased estimator of the expected mental rollout path length when hiding at (fs, e) and is selected from goll outcomes with prohability softmax (P*)). )e-
After computing μ = (Hi....., (10). the location tuple & where the actual OH-episode will begin chosen by sampling from a callinomial(ssotmaay
(H1...., Hn))) distribution.
We now describe the losses minimized so that obtain high quality initial evaluations of location tuples and mental rollouts of seeker trajectories.
Given a tensor M encoding the agent's 's map of the environment (see the Metric mapping section helow), the perspective simulation model produces an intermediate representation of this map M From M it generates two tensors P and VL of size 2 x 3 x x 7 well as two values Puone, Vince E R. For simplicity we will abuse notation and let p = pfane and V were the agent to choose to hide the object at location f, the event e E çoll ouicomes would occur. This tensor softmax( P*) is supervised using cross-entropy loss with the hiding location chosen during the OH-stage. The entries of the tensor V then represent an estimate of the quality of a hiding location. The collection of tensors V is supervised as follows. Let 1 < i # i' < 10 and 1 < j # j' < 3. Now let C/..3'' be the binary cross entropy loss between V.l. Verz, and sign( 12z) )- Note that minimizing Cly,eyy' encourages V.f.
of Now softmax (P)), )e is prediction of the probability that,
18

---

Published as a conference paper at ICLR 2021
be larger than Veyy if and only if jay pey,, namely in those settings where mental rollouts suggest that (e,, ety) a better hiding location than (4p, (pg).. We thus let our loss supervising the tensors V equal the average C = (2) sum(t 1 i1sis S 10, 1 1 J S 3, (t. * (f)))))).
Now we discuss losses used to guide learning to produce mental rollouts of seeker trajectories. Suppose that a full game of cache has taken place. We begin by collecting several pieces of data from the game.
The location tuple during the PS-episode and the event E 8 ouncomes that occurred during the OH-stage.
2. The trajectory (f+) (ordered set of location tuples visited) by the seeker.
3. A {0, 1) 1 vector e mould suceed with ith entry equalling if and only if the seeker would have been able successfully claim the object visible at the ith step of its trajectory.
4. The actions a, taken by the seeking agent taken at step i along with their success signal 8;.
The policy, probability distribution over actions, m, of the seeker at step i.
The hiding agent then forms, from the intermediary representation M and pair (f,e), a hiding-place- conditional representation M Using this conditional representation as "mental environment," the agent then simulates an internal seeker who followed the same trajectory (t+)²=1 and took the same actions (a.) as the true seeking agent. At every time step i the hiding agent generates a mental policy ā; for the simulated seeker along with vector of probabilities 8; where Riy denotes the predicted probability that seeking agent, the location t,, taking action with index j would succeed in doing For every the mental policy æ, is then trained, using a cross entropy loss, to mimic the true seeker policy и. Similarly at every step, the predictions S; of whether actions would, or would no succeed, is supervised using a binary cross entropy loss with & if j is the index of action ag Furthermore each 8yy corresponding to the ClaimVis sibl action is supervised binary cross entropy with would succeod
C.4 ADDITIONAL LOSSES
To improve training sample efficiency we augment the usual A3C losses, described below, by also computing several other losses, these losses are averaged across timesteps within each stage and then summed together along with any other losses, e.g. the A3C loss, and gradients are computed for this total loss before being synchronized with the shared model. As there are number of additional losses, we will not describe the full mathematical details these losses and instead describe their intent. Let E Saiges step in s-pisode,, and i be the image observed at that time step. if applicable.
be the current time
Other than the A3C loss, there are two additional losses shared among the (E&M, OH, OM, S} stages described below:
1. Encouraging understanding the effect of actions: we would like for the DIR representation learned by our agents to be self-predictive, namely we would like the DIR to encapsulate how the environment will change when acted by the agent. To this end, recall the notation from Section where 8t represented the 128 7 x SIR output from CNN₂ at time t and hi represents created generated by tertsclstms at time t. Moreover let a; be the action taken by the agent at time t. To encourage self eef-prediitiiity. we then define two learnable embedding functions E and E and define loss which is minimized when the dot product E ( E3 (12 az) is large while (s. ) E3 (42 oe) small for all # Minimizing this loss encourages he contain enough information the true next visual observation 8c+1 from all other observations seen during the episode. Moreover it encourages the SIR 8141 to contain information that will make it readily distinguishable from other the SIRs of other timesteps.
512-dimensional hidden output vector
he able distinguish
2. Localizing changes in input images: similarly to the above, we would like our agent to be able to localize where changes have occurred in input images. As described in the neural models section below, at every time step the agent generates 1 x 7 x 7 tensor W where (rftmax("*".).... he nonzero only if the pixels corresponding Ciy have changed from the previous to the current frame. To supervise Wa.! we first generate the x 7 7. {0, 1) -valled,, tensor where Di,4, equals one if and only if the pointwise difference between the encodings of in and i generated by the first convolutional block of CNNo. has 10% of the spatial locations corresponding to the i,j grid location with fo-norm greater than 10 -3 We then form the loss
should is encouraged to
C log softmax(W** )1,3 D1.s.).
Note that minimizing C. encourages the sum of the entries in Doftmax(Weit corresponding entries in D equalling one be large.
19

---

Published as a conference paper at ICLR 2021
Beyond the above losses shared among all of (E&M. OHI OM, S), also include several stage-specific losses described below.
OH-specific losses.
1. Avoiding early PlaceAt modality collapse: early in training the hiding agent makes many failed P PlaceAt actions, large number of these actions are impossible for given viewpoint, incurring substantial penalty for these repeated failures. Because this, we found that, in this stage, the agent often would often quickly learn to only use the ONTOP modality of the PlaceAt as this is the modality is, generally, the easiest for the manipulator to successfully complete. To prevent this policy collapse, we introduce a loss that encourages the agent to maintain prohability mass on P1 PlaceAt actions with CONTAINEDIN and BEHIND modalities. This is similar the standard loss in A3C encouraging higher entropy an agent's policy. This loss is removed after 5,000 episodes training.
2. Avoiding early P PlaceAt spatial collapse: similarly as above, sometimes found that the agent's policy would concentrate PlaceAt actions single grid position i.j early in training. To prevent this, we create a loss encouraging the probability mass over PlaceAt to be similar to prediction of which PlaceAt actions would be successful coming from the manipulator model (described below). As above, this loss is removed after 5,000 episodes of training.
Recalling PlaceAt actions: to encourage the agent to remember where attempted to place object and where, on those attempts, the object ended we designed two losses penalizing the agent for failing to accurate make such predictions correctly.
Closing objects: after placing an object so that the CONTAINEDIN modality we design a loss encour- aging the agent to attempl closing objects.
OM OM-specific losses.
Better sample efficiency: inspired by hindsight experience replay (Andrychowicz et al. 2017). we design a loss that rephrases failed trajectories made during an OH-episode, i.c. a trajectory that placed the object in a location that was not specified as the goal location, into successful trajectory where the goal was whichever location the object actually was placed in.
Opening objects before placing things into them: we designed a loss that encouraged the agent, during the OM-episode, to try opening an object whenever the goal location for the object was in a CONTAINEDIN modality.
Predicting if goal object location was attainable: better sample efficiency and for use during other stages, we encourage the agent to predict whether or not it will be able to manipulate the current object in such a way that it will reach the given goal object location. In particular, assume that the agent was directed place the object in location (m,,, i,.). The hiding agent then produces, at every step, a x 7 tensor phiitable 1 and, at the end of the episode, computes an average binary cross entropy loss between the entries 1 hinable and the indicator g0M-ejasode success
PS-specific losses. See the section training perspective simulation above.
S-specific losses.
1. Better sample efficiency through self superviee imitation learning: using the mental representation of the environment created by hiding agent in the PS-stage compute shortest path from the seeking agent's current location location from which, in this mental representation, the object is visible and then en- courage the seeking agent to follow this path.
2. Reducing false claim visible actions: at every step made by the seeker encourage (or discourage) the agent from taking the ClaimVis sibbe action based whether or not the goal object currently visible to the agent.
C.5 REWARD STRUCTURE FOR REINFORCEMEN LEARNING
In order to compute the A3C loss we must define, for each stage E Satagos a reward structure, that is, a function which assigns a reward r E IR to every state-action-state triplet (s, a, 8'), where s denotes possible agent state and action which, when taken from state resulted in the agent moving into state s'. When applicable and unless otherwise stated, agents obtain a penalty of Taep -0.01 long trajectories and an additional penalty Tru -0.02 whenever an action fails (for instance, when running into a wall). For clarity of exposition we will define the reward structure by defining r in the case that agent starts at a generic state at time t ' 0 and performs action resulting in a transition to state s'. E&M reward structure. During exploration and mapping, we would like to encourage the agent comprchensively explore the environment, including interacting with and opening objects if possible, with particular attention heing paid
every step to discourage
20

---

Published as a conference paper at ICLR 2021
areas that would make good hiding locations. To this end, but for rede, described below, define the E&M reward structure Algorithm [ Now for 0 S + 1, at time m, be the metrie map generated by the agent, &, be the location tuple of the agent, and let vs be the value of the position l, when evaluating m, using the perspective simulation module. Now letting unseen, = of with action as the agent opened an object, or entered a location, for the first time), we define
U1 =
V unseen, )/C unseen.)
so that V. captures an average quality of unique states the agent has visited up to and including time t. We then define the reward
TERM. hide, = 0.2 m1, max(0, u++1 ve)) unseenet1
Note that FRAM, t will be if the agent has not opened new object or entered a previously unvisited location tuple and, otherwise, will only be large if the perspective simulation module considers the current state to be substantially more valuable than previously visited states. Thus Team, aide.t provides a signal through which the exploration agent may learn to visit states that the perspective simulation module considers good hiding locations.
OH reward structure. Except for the definition of rOR, percetiles the reward structure for the OH episode is defined in Algorithm g In the below we define and motivate roll prcentkk- Suppose that the during the OH episode the hiding agent has successfully completed a P PlaccAt action, we would like the reward given to the agent after its final action reflect the quality of the hiding location chosen and thus natural reward for this final action number of steps taken by the seeker to find the object N steps sipps normalized by the maximum number of steps allotted to the seeking agent (500), i.c. the reward
staps /500.
I' staps
Unfortunately i's seps fails in two respects.
The reward I's stes suffers from very high variance. As example, consider the case in which the object hidden directly behind the seeker. If the seeker performs two atttateRi ath actions it will immediately find the object but, if it instead chooses to search the area ahcad of itself then it may be many steps before it backtracks to its starting position and finds the object. In such a case the reward I's steps may be very large or very small with high probability.
The number of steps required to find the object is not only determined by the quality of the hiding place chosen in the OH episode but also by the topology of the scene and the location tuple chosen in the per- spective simulation episode. Namely, in a large, mazze-like, room the OH episode may terminate with the object in the open from the hiding agent's perspective poor hiding place) but still require many steps from the seeker to find the object as the location tuple chosen in perspective simulation was far away from the seeker's 's starting position.
This high variance and confounding with the quality of perspective simulation results in I's seps providing a poor signal as to where an object should be hidden in the OH episode. While the high variance of the reward intrinsic to the game of cache, especially so in the case of randomly initialized agents, we can partially address the second issue above by considering a transformation of i's stes defined as follows. First we generate 100 100 sample seeker trajectories (#:) when simulating hiding the object at the location chosen during the PS- stage; see the training perspective taking section above for details as to how these trajectories are generated. Letting l, be the length of 8,, i.c. the number of steps the seeker took to find the goal object in that mental rollout, we then define
100
Σ S/z S N
FOR, percenile = -11+ 11++2
100
Assuming these mental rollouts are of reasonable quality, fOn, percmtile naturally calibrates the reward for the topology and size of the scene. This helps disentangle the final reward in the OH episode from contributions made purely during the PS episode.
OM reward structure. The OM episode runs for a maximum of 50 steps and has the reward structure for this episode is outlined in Algorithm 3 Broadly speaking, we reward the agent for dropping the goal object near the target location with proximity measured in the image space and for placing the object in a location respecting the target modality among ONTOP, CONTAINEDIN, and BEHIND. We found it especially important give the agent a large negative reward whenever placed the object outside of the agent's viewport.
S reward structure. The reward structure of the seeking episode is simple and encodes our desire for the agent to locate the object in as short as time as is possible without having large false positive rate, that is without having many instances in which the agent claims to he able to see the object when the object is not actually visible. This reward structure is given in Algorithm
21

---

Published as a conference paper at ICLR 2021
C.6 PRETRAINING THE E&M AND OM STAGES
As the ability to navigate about a room and perform object manipulation are basic competencies for the hiding agent in cache, we pretrain our hiding agent on the E&M and OM stages until it attains reasonable proficiency. The reward structure when training the E&M-stage was essentially identical to that from Algorithm [ but without the rE&Mo tide, reward, this trained for approxinately 85,000 episodes. To pretrain the OM-stages generated, across A12-THOR scenes, a collection of plausihle locations the objects could be placed when location tuples, and then trained the hiding agent to attempt placing objects these locations using the reward structure in A Aggorihhm 3 the manipulator agent was pretrained for approximately 1.8 million episodes. We ran this pretraining procedure for approximately three days.
D NEURAL MODELS
Here we give an overview of the neural network architectures we use parameterize our agents. As these architectures can be complex, we do not attempt to make this description sufficiently comprehensive to exactly reproduce our results; we instcad direct any parties interested in extract reproduction to our code base.
Our neural network architecture contains components that are shared among both the hider and seeker well components that unique to individual stages. Recall from 4 that shared components include the image processing CNN CNNs, the recurrent networks CGRU) and LSTMg and metric mapping components. Previ- ously unmentioned shared components include embeddings for actions, goal objects, and agent standing state. Unshared components include, for example, the MLPs defining actor-critic heads.
D.1 INFORMATION FLOW TO POLICY AND VALUE
Here we describe how. during the E&M-, OH-, OM- or S-episode, data flows through our agents' neural architecture to produce the agent's policy and value of the current state.
Suppose the agent has received input corresponding to timestep t 2 0 in stage E guage and must take an action. This input comprises the following.
A 3 x 224 x 224 (channels x height x width) tensor corresponding to the image of the agents' current view. This tensor is processed CNNs to produce an SIR 8t of shape 128 x 7 x 7.
A non-negative integer specifying the agent's last action (including a 'none' action when = 0) along with a 'success' or 'failure' signal indicating whether the agent's 's last action succeeded. This pair of inputs is used to index into an embedding matrix resulting in 16-dimensional embedding vector.
A non-negative integer specifying whether or not the agent is standing. This integer is also used to index into an embedding matrix resulting in a 4- dimeeniional embedding
A non-negate negative integer specifying the goal object (i.e. object to hide or object to seek). This integer
is also used to index into an embedding matrix resulting in a 16-dimensional embedding vector.
The 257 X 7 7-dimensional hidden state h corresponding the output of the convolutional GRU CGRUs from the previous time step. If 0 then h4-1 are tensors filled with zeros.
The 312-dimensional hidden states hi h corresponding to the output of the LSTM LSTM, from the previous time step. If = 0 then these tensors are identically zero.
The agents' current metric map it's environment, this includes 144-dimensional vector for every unique location tuple the agent has visited.
(OH only) the last action taken in the OH-episode was a PlaceAt, an action embedding specifying the location location the OM-episode placed the object, e.g. if the OM-episode placed the object in the Ce grid location with modality BEHIND, then the agent obtains the embedding for a successful PlaceAt 12, 1, j action. As an object can potentially land in many grid locations and modalities simultaneously, the chosen action corresponds to the location and modality containing the largest number of the object's visible pixels.
(OM only) The modality and grid location i,j where the object should be placed. This triple used to slice into from a parameter tensor size x 8 13 x 13 retrieving x 7 x 7 tensor which is concatenated to the 128 x 7 x 7 tensor of visual features.
Excluding the recurrent states and the map, each of the above embeddings are repeated (if necessary) so that they are of shape N 7 7 and concatenated together with the 128x7x7 7 SIR. For instance, the 16-dimensional object embedding becomes 16 x 7 x tensor T where T.,.. is copy of the original embedding for each j.j. To this tensor is additionally concatenated the first 256-dimension of hy after having been tiled he of
22

---

Published as a conference paper at ICLR 2021
shape 256x7x7. This concatenated tensor is then fed into CGRU, along with the hidden state hi- producing Ju.
The tensor he-1 is then split into two components. It's final channel, (256, is removed and becomes the tensor W described in Sec. C.4 The remaining x56x7xx tensor then averaged pooled create the 356-dimenssinal vector fs.
Now the network reads from the input metric map using both attention mechanism and collection of convolutional layers resulting in an embedding of size 356. This embedding is concatenated to he and fed, along with the hidden states (hg-1,hh-1). into LSTMs producing the hidden states (h,, A.).
For each stage X E {E&M, OH, OM, S} there is now a unique MLP MLP corresponding to a 512 x 512 linear layer followed by ReLU non-linearity. Additionally there unique networks ACTOR (correspond- ing a 512x (number of actions in stage X) linear layer followed by softmax nonlinearity) and VALUE corresponding to 512 x linear layer.
Now the agents' policics and values for cach stage are computed as ACTOR (MLP) (hg + h()) and VALUE (MLP3 (1:) AE)) respectively.
Finally the agents then write into their map, at a position defined by their current location tuple, 128- dimensional vector extracted from one of the final layers the CNNo (when applicd to i,) along along with the 16-dimensional object embedding.
D.2 METRIC MAPPING ARCHITECTURE
Intuitively, our metric mapping architecture is designed so that an agent can record information regarding its current state for later retrieval. This map is called metric data is written into the map so that it is spatially coherent, that where data is written in memory is uniquely defined by the agent's location tuple when it is written. Our metric map model implements designed to allow for the following.
Writing: given the current location tuple of an agent, passes given value through a Gated Recurrent Unit (GRU) (Cho et al. (0044) whose hidden state taken from the last value saved at that location (if any, otherwise a learned initial state is used) and this output of the GRU then overwrites the previously saved values.
Reading location: given a location tuple, returns the value saved at that location.
Reading with attention: given input key, uses soft-attention mechanism to return a weighted sum of saved values.
Reading as tensor: return a 4 x 2 x 144 x h x w tensor M where Mejansi is the value written into the map when the agent was most recently at location tuple
(min(z values written into map) + 0.25, max(= values written into map) - i- 0.25, s, r);
h is the number of distinct = values visited by the agent, plus optional padding, it built the map, and w is the number of distinct æ values written into the map, again possibly with optional padding. Notice that decreasing in Mr,a... results reading from locations north the current location and similarly increasing j results in reading locations more east.
Reading egocentric map: the agent and appropriately transformed so that north whichever direction the agent is currently facing.
reading map but with spatial coordinates centered the
Using the above, the agent build map while it explores its enviromment while querying that map locally and globally so as to make decisions.
D.3 VISUAL DYNAMICS REPLAY DECODER ARCHITECTURES
Recall from our training and losses section, that in VDR we are given a triplet (io. a, i1). where io, in are two images with é occurring after an agent took action a, and must, accomplish a number of self-supervised tasks. To accomplish these tasks we require several neural network architectures building upon our SIR generating architecture CNNo. These architectures are all convolutional neural networks with the largest of these networks inspired by spatial transformer networks (Jaderberg al. 2015), as well as U-Nets, and contains 13 convolu- tional layers, of which are transposed convolutions which work to upsample the input tensor, the SIR output from CNNs,
final size of 16 224 x 224 after which x convolutions are used to produce 7724 as well as 7 confideence and 7 pis. dilr. The tensor 1 would sucoedd is produced by convolutional network acting on CNN)(fo) and the emhedding of while the value wim is produced by convolutional network applied to concatenation of CNN₂'s emheddings of both in and 1j. Implementation details can he found in code hase.
23

---

Published as a conference paper at ICLR 2021
D.4 PERSPECTIVE SIMULATION ARCHITECTURE
The perspective simulation architecture operates on the tensor representation of the map, see the metric mapping architecture above, which is reshaped to be of size 1152 x h x where It and w are the height and width of the map respectively. First this map is concatenated with a grid of distances from the agent start location after which it processed by several X convolutional blocks to form the intermediary representation M of the map, An additional 1 X 1 convolution produces, for each location tuple (merging those which only differ in that the agent is, or is not, standing) after and after reshaping, two tensors P and V of size 2 x 3 x 7 x 7 as well as two values Puone: Vinee E R. For simplicity we will abuse notation and let Puone Piane and Vione = vhone- Next, given hypothetical hiding outcome
5 ouseome another convolutional neural network, acting on M along with an embedding of e indicating where M¹ the object being hidden, produces the second intermediary representation of the map M A "mental" seeker agent is then defined in this architecture using an LSTM whose observations come from slices of M This mental seeker can move about the map defined by the representation in M and, each step produces the vectors म, and 8, defined in the training in losses section by using two lincar layers acting on the mental agent's 's hidden state.
E HUMAN AND SELF EVALUATION FOR CACHE AGENTS
In Figures 4a to 4c we report the results of evaluating our hiding and seeking agents from three points during their training as they compete against one another as well against hiding places chosen by an exhaustive approach and hiding places chosen by humans. In the following we discuss how we collected hiding locations for the various agents and ran our seeking trials.
E.1 GENERATING HIDING LOCATIONS
Human agents. We recruited 16 human volunteers hide objects within AI2-THOR. Participants structed that they would be randomly positioned in simulated enviroument holding an object which they were then expected to hide so that a seeking agent, who started the same position that they started in, would find the object difficult to find. Furthermore participants were instructed to be in the mindset that they might be playing against the same seeker many times and thus should choose diverse hiding locations.
in-
Before beginning. participants were allowed to familiarize themselves with the interface until they judged themselves sufficiently competent to begin the task. Unlike our artificial agents, humans subjects were not constrained in the number of actions they were allowed to take and, after hiding their object, were allowed to pick it up again and try other hiding locations if desired.
We collected 600 human hiding locations evenly distributed among all object types and training, validation, and test scenes. To reduce the amount of data be collected used only ten (of 40) training scenes (A12-THOR floor plans 10, 13, 16, 18, 203, 209, 213, 215, and 218).
Artificial agents. We generate hiding locations from our artificial hiding agent using the agent after had trained for O, 10t and 4.5 10 episodes. We call these agents ho. Jases and ha.s.g. For each of these agents we generate 4,000 total hiding locations, 2,000 each for the train and test scenes. The train (test) hiding locations are generated such that for each (scene, object type) pair we have an equal number of hiding locations. For fair comparison, we also generated set of 1,545 additional hiding locations from haae using the same scenes, object types, and agent starting locations as those from the human trials (with multiple repeats per agent start location).
Exhaustive approach. Recall the exhaustive automated approach for generating hiding places of easy, medium, and hard difficulties detailed in the cache stages section. Using this approach we generate, for each of these difficulties, 3,999 hiding locations in the train scenes and 998 in the test scenes. Because of our auto- mated, metric-based based, approach for generating these hiding spots there were three cases (one in the train scenes and two in the test scenes) where there were not be sufficiently many hiding locations 20 desired locations per object, scene, and difficulty. In such cases we reduced the number of locations saved for that object and scene pair for each difficulty level that the total number of hiding places in each difficulty type remained equal.
E.2 RUNNING SEEKING TRIALS
In these trials a seeking agent is required to find an object hidden in a hiding location from one of the above collections. If, through automated checks, we find that a hiding place is such that the hidden object cannot be seen from any reachable position in the scene then this hiding location is never shown to any seeking agent and all seeking agents considered have failed to find the object for that trial.
24

---

Published as a conference paper at ICLR 2021
Artificial agents. We rum seeking trials using the artificial seeking agents from the same three points in training and name these secking agents analogously as Su, S1et; and 84.seti
Human agents. For sceking we recruited human participants using Amazon's Mechancical Turk platform. As in the hiding task participants were allowed to familiarize themselves with the interface until they judged themselves sufficiently competent to begin the task and were not constrained in the number actions they were allowed take.
On starting a trial, the participant was randomly assigned hiding location generated either by ha.sed or from the set of 600 human locations. To ensure fair comparison, we matched each the 600 human locations to a unique hiding location generated by he. (from the additional set of 1, 1,545) where the artificial hiding agent started the same position the human hider. After 100 seconds participants were allowed to give up. moving them to the next seeking trial, they felt the object could not be found.
At the of these trials we were left with complete seeking trials for all but two instances where invalid data was returned.
F STATIC IMAGE REPRESENTATION EXPERIMENTS
these experiments we consider SIRs produced by the model CNNs when trained:
(I) to play cache (taken our final cache agent),
(2) to do joint object class prediction using precisely the same images seen by our cache agent during training (here object is considered present in an image if A12-THOR returns that the object is visible)
(3) complete navigation task (see Sec. G.T).
(4) using an autoencoding loss (i.e. pixel-wise mean absolute error) with 35,855 A12-THOR images, or
(5) with image classification loss on the 1.28 million hand-labeled images in ImageNet (Krizhevsky et al. 2012
More details regarding these baselines can be found in Sec. We begin by describing our experiments for synthetic images before moving to our natural image experiments.
F.1 SYNTHETIC IMAGES
F.1.1 EVALUATION DATA AND TASKS
To evaluate quality our SIRs we train decoder models on these SIRs to complete number of distinct tasks described below. Except for the classification task, where the decoder model is a max-pooling layer followed by linear layer, the decoder models are designed similarly to those from U-Net (Ronneberger et al. 2015), for details please see code.
Depth prediction. In A12-THOR we generate collection of 3 224 x 224 RGB images paired with their corresponding 224 224 depth maps. Models are trained to predict the depth map given corresponding RGB image by minimizing a per-pixel mean absolute error loss.
Surface normals. In AI2-THOR we generate a collection of 3 x 224 x 224 RGB images where cach image is paired with corresponding 3 x 224 x 224 tensor T where T, represents the surface normal at pixel i. in the RGB image. Models are trained to predict this tensor of surface normals given its corresponding RGB image by minimizing a per-pixel negative cosine similarity loss.
Classification. In order to generate a dataset for classification we first select 90 types of objects available in A12-THOR and assign each of these types unique index in I.... 90. Next, in A12-THOR, we generate a collection of 3 x 224 x 224 RGB images where each image is paired with {0, 1 1]-valued vector of length 90 whose jth entry equals if and only if an object whose type has index is present in the image and whose pixels occupy at least 1% the image. Models are trained to predict this vector of 0,1 values from the input image using binary cross entropy.
Openability. In A12-THOR we generate a collection of 3 x 224 x 224 RGB images where each image is paired with a 224 x 221 (0, 1, 1,2)-vlleed tensor T with entry T. equaling 1 if pixel i, in the input image corresponds
object that can he opened, equalling 2 if the pixel corresponds and otherwise equals Models are trained to predict the (0. 1, -valued tensor from the corresponding RGB image by minimizing a per-pixel mean cross entropy loss where the loss associated with predicting a 0 has a weight of 0.2 and the other losses have a weight of 1.0.
object that can be closed,
Traversable surfaces prediction. In A12-THOR generate collection of 3 x 224 x 224 RGB images where each image paired with corresponding 224 x 224 (0, -aaludd tensor T with Ti, equalling if and
25

---

Published as a conference paper at ICLR 2021
only if the ejth pixel in the image corresponds surface onto which the agent walk. Models trained predict the {0, ( }-valued tensor given corresponding RGB image by minimizing a per-pixel binary cross entropy loss.
Object depth. In A12-THOR we generate a collection of 3 x 224 x 224 RGB images each where each image is associated with collection of 224 x 224 {0, 13-valued masks of, sufficiently large, objects in the image paired with depth maps corresponding the image in which the masked object has been removed. Models trained predict, from an input RGB image and object mask, depth map of the image where the object been removed by minimizing a pex-p weighted mean absolute error loss. In this loss, pixels corresponding to the masked object are weighted to contribute equally to those pixels where the object is not present.
Object surface normals. Similar to object depth but predicting surface normals after an object has been removed.
F.1.2 TRAINING
We train using an ADAM optimizer with (81. ) (0.99.0.999), weight decay of 2e-6, and plateau-based learning rate scheduler beginning with learning rate of le-2 and decaying by factor of 10 with patience of Appeienee that is, scheduler reduces when the loss on the validation set has failed decrease by some small absolute quantity for Npanieee subsequent epochs). When training CNN₂ to perform auto-encoding and ImageNet classification we used patience of Apiriiue = 5 and trained the models using all available training data. When training our other tasks, used patience of Nu
5 and subsampled the training data include 1,200 examples. Training was stopped the learning rate ever decreased below le-4. During training models were evaluated on the validation set after every epoch and, once training completed, the model with the best validation loss was returned as the final model.
F.1.3 EVALUATION
To evaluate our models we compute the loss of each model on each of the test set examples for its respective task. As any test set results generated within the same scene and at nearby location tuples are correlated, we generalized linear mixed effects model (GLMM) to obtain statistically meaningful comparisons hetween models. We our analysis using the R (R Core Team 2019) programming language, in particular we use the 9 mmPQI. function in n me (Pinheiro al.) 2019) package our GI MMM models and
emmeans [Lenhh] 2019) package to obtain p-values of contrasts between fixed effects. In these analyses we:
Let the per test set example loss be our endogenous variable.
Estimate fixed effects corresponding which SIR was used to produce the loss (auto-encoder, Ima- geNet, or cache).
Include mixed effects corresponding to scene nested within scene type.
Model spatial correlation, nested within scene, using spherical correlation where two test set exam- ples corresponding to different SIRs or with different rotations are considered infinitely distant.
Use a Gamma family model with an identity link.
Subsample test set to include most datapoints per scene and so that the model can be more quickly trained.
To fully reproduce our results please see our code base.
F.2 NATURAL IMAGES
We consider three distinct tasks for natural images, scene classification the SUNCG dataset Xiao al.] (2010). depth estimation the NYU V2 dataset Nathan Silberman & Fergus (2012). and walkable surface estimation also on NYU dataset. As the synthetic image experiments, we use CNN₂ as our backbone model.
F.2.1 EVALUATION DATA AND TASKS
Scene Classification. In scene classification the goal is to predict, from natural image, the type of scene the image is staged in. For instance, scene categories include beach, palace, and sauna. We evaluate our predictions using the meean-per-elass) top-1 accuracy metric.
Depth prediction. As for synthetic images, our goal is to predict pixel-to-pixel depth but for natural images. We evaluate our model using the standard absolute relative error of predictions.
26

---

hand movement action we Published as a conference paper at ICLR 2021
Walkable surface estimation. This is synonymous with traversable surfaces prediction in A12- -HO. We evaluate the quality of our predictions using the standard IOU (intersection over union) metric where we only consider the IOU with walkable segments.
F.2.2 TRAINING
Scene Classification. Our decoder architecture scene classification corresponds to 1-by-1 convolutional layer which transforms the 1 x SIR output from CNN₃ to a 64x7x7 7 tensor. Two fully connected layers then transform this tensor into a vector of size 512 followed by vector of size 397. For training we use the cross-entropy loss.
Depth estimation. To perform depth prediction our network CNN, composed with feature pyramid network (Lin et al. (016) and pixel shuffle layers Shi et al. (2016) are used for upscaling. The Huber loss minimized for training.
Walkable surface estimation. This architecture is essentially identical as for depth prediction. The architecture of this network is the same as the depth estimation network. Our training object is pixel-wise binary cross entropy.
G DYNAMIC IMAGE REPRESENTATION EXPERIMENTS
In the following deseribe our training procedure for our dynamic image experiments along with any task- specific details including definitions of our baselines, namely the labels in the legend of Fig. [7] For tractability of inference, we will always consider our training set Durin to fixed and ignore any randomness, if applicable, in the generation of Denin- Each of our considered tasks is a discrimination task and requires predicting, from a given representation, whether or not some binary condition holds. We call cases where the condition holds positive examples and others negative examples. We will accomplish these predictions by fitting logistic regression models, details are given the training section below.
G.1 BASELINES
We will now describe our baselines referenced in the labels of Fig. aad in Sec.
DIR (Cache). The is the DIR generated by our trained cache agent. In the tracking experiment (the experiment corresponding to Fig. 7b this DIR corresponds to the x7x tensor he (recall the notation from See. 4 produced by the trained CGRU, model. In all other experiments this corresponds to the 112-dimensional vector hi produced by the LSTMg architecture.
DIR (Nav.). For comparison we trained a navigation agent to perform the seeking task with object locations chosen from the sets of hiding locations, of casy and medium difficulty, generated by our exhaustive procedure. This navigation agent was trained, without VDR, for 150,000 episodes at which point performance appeared saturate. The navigation agent trained with similar architecture tracking experiment the DIR this agent was taken as
cache agent with convolution
GRU without a final LSTMs model. the 257 x tensor output from its convolutional GRU. For the other experiments, the DIR was taken as a weighted average pool of the aforementioned 257x7x tensor. Note that, unlike the cache agent, the navigation does not have any actions corresponding experiments, whenever would tell the navigation agent that its last action instead tell that last action was a Stand action.
moving the object in its hand. Because of this, in the below
DIR (Rand.) It is well-know that convolutional networks structurally encode powerful inductive biases for image processing and that even randomly initialized networks can provide surprisingly powerful features for downstream tasks. This raises the question of whether not the cache DIR performs well simply because model architecture encodes strong inductive biases and, perhaps, that we do not need to train our model all to obtain good results for our DIR tasks. address this question, DIR (Rand.) is generated identically to DIR (Cache) except that the underlying model weights are kept as their randomly initialized values.
SIR (Cache). This baseline corresponds to the features produced by CNN₂ trained by playing cache. For the entire 128x7x1 SIR produced by this CNN, in all other experiments
the tracking experiment average pool this tensor reduce it 128-dimensional vector.
SIR (ImageNet). This model is identical SIR (Cache) except the underlying CNN was trained with an image classification loss on the 1.28 million hand-labeled images in ImageNet (Krizhevsky et al. 2012).
SIR (Class.) One may wonder if a CNN trained to perform classification on A12-THOR images would obtain similar performance to our cache-trained model. To answer this question, while training our cache agent we have, parallel, trained CNN₂ model to elassify A12-THOR objects. Critically, this model was trained with
27

---

Published as a conference paper at ICLR 2021
exactly the same of images cache agent and received exactly the same number of gradient updates. Objects were considered to be present an image they were considered visible by AI2-THOR. To train this model we minimized a binary cross entropy loss after average pooling the final representation from CNN₂ to obtain a 128 -8-dimensional vector after which we applied a 128x 101 linear transformation giving us the 101 binary logits used for prediction (of the 101 applicable A12-THOR objects). As for SIR (Cache), we use the 128x7x1 7 tensor for the tracking experiment and, for all other experiments, average pool this vector be of size 128.
Forward Steps. This baseline is only included in the "Contained or Behind?" experiment. In this experiment there is non-trivial bias in the dataset, namely that an object needs to be moved further forward to be placed behind a receptacle than it does to be placed into it. To ensure that DIR (Cache) is not simply exploiting this bias, we have included the count of the number of forward steps haseline.
An straightforward strategy by which to produce DIR from SIRs is concatenation. Namely, if we have an SIR 8t, then it is we may build a representation that "integrates obser- vations through time" simply by concatenating several such SIRs together. That is, DIR representation at time t could simply be the concatenation [8 8:-2; 8t-1, St]- The three haselines DIR (4CNN-Cache), DIR (4CNN-Class), and DIR MCCNN-ImgNet each take this approach by concatenating the SIR from time t with the SIRs from the three previous timesteps. For DIR (4CNN- nated (and similarly for DIR (4CNN-C) -Cass),. and DIR (4CNN-ImgNet)). For the tracking experiments these DIRs are thus of shape 512x7x7 7 while for the the other experiments these DIRs are vectors of length 512. As we wish these DIRs to be strong baselines, in the "Contained or Behind?" experiment we also concatenate these DIRs the Forward Steps feature described above (producing 313-dimensional DIRs).
DIR Www.N-CCche/Caas/mgg...eeo
SIR (Cache) representations are concate-
G.2 TRAINING
We wish to evaluate how our considered DI- and S1- representations perform in our tasks as the amount of data available for training varies when using training procedure that follows best practices in machine learning by using dimensionality reduction and cross validation. Our procedure, based on iteratively subsampling Dunin. follows. Suppose we are interested in our representations' performance given training of size n >> 0 with n < /eail We then repeat the following R > 0 times.
1. Subsample training set Dr C Dusin of size taking to retain as close to the same proportion of negative and positive examples as possible.
2. Perform dimensionality reduction on Dn using (a variant of) principal component analysis, reducing the datasets dimensionality so that least 99.99% of variance explained by the remaining components, obtain datasct D...
3. Fit a logistic regression model on D,, using cross validation to choose an appropriate weighting of lz- regularization.
4. Apply the principal component analysis transformation learned in the second step to the test set Drest to obtain a Dies and then compute the test-accuracy on this transformed dataset.
This gives us R accuracy values anl...., ann from which we can estimate the accuracy of our model on test examples when training our model on a randomly sampled training set of size n as the mean T. = RR. Em ani- Note that randomness in the estimator a. comes both from our resampling procedure and, assuming our test set is not exhaustive, from the randomness in which examples where chosen to be included in the test set. The first source of randomness can be reduced simply by increasing R while the second can only be reduced by gather more test set examples. When appropriate, we account for both sources randomness when plotting confidence intervals.
1
G.3 CONTAINED OR BEHIND RELATIONSHIPS
For these experiments generate dataset using the first five scenes of the foyer type in A12-THOR. For each of these five scenes we generate large collection of paired examples of objects being places into, or behind, one five receptacle objects while that receptacle sits on one of five possible tables at varying offsets from the agent. The placed objects are the same as those hidden in cache and we use five different receptacle objects of varying size. In order to make our setting especially difficult, we modify the above training procedure so that, in step 1 above, we restrict our subsampled training scenes, objects, receptacles, and tables. step 4, our testing set is those examples which the scene, object, receptacle, and table are all among those excluded from the training set. This makes the test examples especially challenging as they are, visually, very dissimilar from those see during training. In this setting, only consider randomness as a result of our resampling procedure in our error estimates and so our inference is restricted to within our generated set of examples.
to exclude randomly selected combination of
28

---

Published as a conference paper at ICLR 2021
Here predictions are made using the representation the timestep after the object has been moved into its position but before it has been let go by the agent (i.e. before DropOb ject action has been taken).
G.4 OCCLUDED OBJECT TRACKING
As in our contained or behind relationship experiments we generate a dataset using the first five scenes of the foyer type in A12-THOR and five table types. Here, as we are interested in tracking completely occluded objects, we use only three of our five item items, the cup. knife, and tomato, as these are sufficiently small to be frequently occluded. For each configuration of object. table, and scene, generate collection of object trajectories the object moves behind collcction of books, recall Fig. [7b For additional variability we produce many such trajectorics with the books positioned in different distances and orientations. As the object moves behind the occluding books, we record all grid locations Ciş that the object passes through into the set P (it is considered to have passed through grid location Cij if, any point during its trajectory behind the books, at least 10 pixels of the object were Ci5 after making all other objects invisible). At every step of a trajectory where the object is occluded such that less than 10 pixels of the image correspond to the object we generate [PP data points, one for each grid location C. E P with the grid location being considered a positive example if, after making all other objects invisible, the moving object has greater than 10 pixels in that grid location and otherwise being consider negative. In the paragraph below we describe how obtain the SIR and DIR features corresponding to each such data point. Note that, as objects tend to occupy only a small portion of an image, there are many more negative examples in our dataset than positive ones. During training we always subsample negative examples so that there is an, approximately, equal number of positives and negatives. Furthermore, during testing, we implicitly balance positive negative examples by reweighting; thus the accuracy we report in this setting bulanced accuracy. As in the previous experimental design, we modify the above training procedure so that, step above, we restrict our subsampled training set to exclude a randomly selected combination of seenes, objects, and tables. In step 4, our testing set is those examples in which the scene, object, and table are all among those excluded from training. Again as above, only consider randomness result of our resampling procedure in error estimates and inference is restricted to within our generated set of examples.
Recall from our Sec. G.I all SIRs and DIRs in this experiment have shape Nx7x7 for some N 2 0. Let T be such an SIR or DIR and suppose that, at time t >> O, the object is fully occluded and X is a 7x7 {0, 1} -aalud matrix with X., equalling one only if grid position Ciş is considered positive example. To obtain, from I', the features used to predict X we simply take a slice from I' at position é. Namely the features used predict the value of X1 are i.)
G.5 OBJECT PERMANENCE WHEN ROTATING
we let our For each the hiding locations our automatically generated easy hiding places, recall Sec. seeking agent attempt to find the object for 500 steps. If, during one of these episodes, the agent would success- fully take a Cl ClaimVi sssbll action at time f, so that it has found the object, we consider the counterfactual setting in which the agent instead rotates right or left once random and the, previously seen, object vanishes. The agent then allowed to take 4 more actions following its policy. Each of the five representation generated after the rotation is saved as a positive example. To create negative examples we replay this same episode, forcing the seeking agent to take all of the same actions, but with the goal object invisible during the entire trajectory. From this replayed trajectory we save the final five resulting representations negative examples. When sampling training sets consider only training scenes and randomly exclude one object, when sampling test sets only testing scenes and examples including the previously excluded object.
G.6 FREE SPACE SERIATION
our final experiment we test ability of our DIRs and SIRs to predict free space. We construct our dataset for this task by, for every A12-THOR living room and kitchen scene 8, performing the following pracedure.
1. Let N* min (0. #unique reachable (a,, 2) positions in the scene 0.25m grid), and ran- domly select (11,,1), (I==,,xx=) locations in s.
2. For every position (z:, 21) perform the following.
a. Teleport the agent to (I,, =) with random (axis- (axis-aligned) rotation, and randomly select direction in which to rotate (left or right). The first set of rotations is habituation phase.
b. Suppose we have chosen to rotate right. Then require the agent to take seven Rot Rotat actions so that every orientation is seen exactly twice.
c. Save the DIRs, and SIRs, corresponding to the agents evaluation when looking in each orientation for the second time along with whether or not the current orientation has strictly more free space than the previous orientation. Free space is measured by the count of the grid locations that the agent could
29

---

Published as a conference paper at ICLR 2021
Algorithm 1: Exploration and mapping episode reward structure
Input: Time step 2 O, full history of E&M episode H, number of steps taken episode N Output: Reward for E&M episode step at time I.
begin
for i - ..... N do
0, - ith action taken in episode
success; Settap. E&M,
ith action successful
extrapr
new.location, + true the location of the agent time did not oceur previously in the episode.
r - -0.01
if not success, then
r r+r 0.02
else
11
// encourage the agent to explore new areas ncm.new.explored.
12
r ++2++021 (extrap; extrap:+1)33
14
// encourage the agent to attempt opening objects
15
new_opened + a, opened an object that was not previously opened in the episode new-opened then r
16
18
// encourage the agent to explore areas considered good for hiding new.openeed or locaticooon then
19
r+ FE&Midde.
if is OpenAt action then // never punish the agent heavily when taking an OpenAt
22
action
r + max - -0.001)
23
24
return
potentially occupy within the agent's visibility Cases where given orientation has strictly space are considered positive examples, the others negative.
3. Combine all positive and negative examples into our training and test datascts.
G.6.1
WHY OTHER DIRS OBTAIN HIGH PERFORMANCE?
In Fig. 7d that several 4CNN- DIRs do nearly well as DIR (Cache). Initially this may surprising. our other experiments seem suggest our cache agent's DIR significantly more capable than simple concatenation of SIR features. We argue, however, that this result is not at all surprising. used to predict the amount of free space in an image to a reasonable degree of accuracy then this task can be tackled using 4CNN-DIR by simply predicting the free space using the SIRs in the last two frames and then taking their difference. That is say, this seriation task well suited for 4CNN-DIRs when their underlying SIRs record free space.
an SIR can be
30

---

Published as a conference paper at ICLR 2021
Algorithm 2: Object hiding episode reward structure
Input: Time step è 2 0, full history of OH episode H, number of steps taken in episode N Output: Reward for OH episode step at time
begin
r r -0.01
for i +- I..., N do
a₁ f- ith action taken in episode
success;
ith action successful
af READYFORSEEKER then not success; then
else if success; and is OpenAt action then
10
// encourage the agent to attempt opening objects
if the opened object was not previously opened in the episode then
12
r + + 0.2
else if a, PlaceAt action but a PlaceAt action was previously successful then 13
r r 0.1
15
else if success; and az is PlaceAt action then
// Reward successful manipulation and encourage the agent to try hiding spots that are more difficult to manipulate
17
p < the probability the object manipulation module assigns to successfully accomplishing at
18
+ r + 0.02 + 1/2 min(max(p, 0.0001) 100)/100
19
else not success; then
r- 0.01
21
there was a successful PlaceAt action taken in the episode and ae is the last successful action that is not READYFRRSEEKER action then
22
- r 5 perentie
23
return
Figure G.1: The five objects hidden that can hidden during cache game. From left to right these are loaf of bread, cup, knife, plunger, and tomato.
31

---

Published as a conference paper at ICLR 2021
Algorithm 3: Object manipulation episode reward structure
Input: Time step 2 O, full history of OM episode H, number of steps taken in episode N, the larget location specified by the action Placeêt m,, i, j recall Table G.1 Output: Reward for OM episode step at time t. begin
ions = empty list for k f I..., N
"x +- kth action taken in episode
success; + was kth action successful
(z. v. world coordinates of agent's hand at time & rounded to 2 dec. places Append (ær, y, 2) to the end hand locations r f- -0.01
if not success; then
10
r +- 0.02
else if a, is OpenAt action then
12
// Encourage the agent to attempt opening objects if the opened object was not previously opened in the episode then
r + r 0.1
15
else if a, = Dropob ject then total number of pixels corresponding the object after making all other objects invisible
b_obj_pix
if n_obj-pixels 10 then
// Discourage the agent from placing objects off screen -+ 1.0
else
21
hit + 3x7x7 7 tensor recording the number of pixels of the object dropped object in each of the grid locations and modalities, recall the discussion in Sec. B.4 and
Fig. 3d
M Compute M as in described in Sec. B.4 hit inds + list of the multi-indices of nit where the corresponding entry of hit 2 0.95 M
// Encourage the agent to place the object close to the goal in pixel space (ignoring modality) + i (a.h.e)eiit.inds)
26
// Encourage the agent to place the object in the correct modality (only for BERIND and CONTAINEDIN modalities)
if m # and m for any (a, b. hit.inide inds then
28
r+
29
Give extra reward if the agent got everything exactly right if i, is hit-inds then r ++1+1
32
else if = N then
33
r + 1.0
34
return
32

---

Published as a conference paper at ICLR 2021
Algorithm 4: Seeking episode reward structure
Input: Time step 2 0, full history of OM episode H, number of steps taken in episode N, the goal object goal
Output: Reward for S episode step at time t.
begin
for 1 do
0; f ith action taken in episode
success; was ith action successful
new locaticiio... + action resulted in the agent entering a previously unvisited location tuple when ignoring rotation
new_openedt <- action (; resulted in the agent opening a previously unopened object
r + -0.01
not success; then if a CIAIMVISIBLE then
10
// penalize the agent for claiming the object is visible when it cannot be seen
11
the action failed as the object was not visible in the viewport and not because the agent
far away from the visible object then
r - r 0.05
13
else
r 0.02 else if new locationn then
15
16
// don't use step penalty the seeking agent entered new location
17
+ r
18
else if new.opened, then
r + r +
29
else if a; = CLAIMVISIBLE then
21
// the agent has successfully found the object <- r 1.0
22
return
33

---

Published as a conference paper at ICLR 2021
Action
Success condition
Impact upon success
Ep.
MoveAhead, MoveLeft, MoveRi ght
No obstructions 0.25m direction movement.
Agent moves 0.25m forward, left, or right respectively.
E&M,
Rotateleft, RotateRight
If the agent is holding object, it will not collide after rotation.
The agent rotates 90"
E&M.
the specified direc- tion.
Stand, Crouch
Agent must begin crouching if taking a Stanc action and must begin standing if taking a Crouch action. Additionally, the agent is holding an object, this object will not collide with anything when the agent changes height.
Agent stands or E&M crouches.
OH,
MoveHlnndhhead, MoveHandleef,, MoveHlnndRight MoveHandßBcck, MoveHandUp, MoveHlnndDow
Agent must be holding an object, agent applics a constant force on the ob- ject in the desired direction until it reaches velocity of Im/s. The agent stops total of 0.1m
the
The hand object mains in new posi- tion.
OM
the object if either from its starting position, object reaches a max distance (1.5m) from the agent, the object exits the agent's viewport, or four sec- onds have elapsed. Once the agent stops the object, the object has moved more 0.001 rotated at least 0.001 then the movement is considered failed and the ob- ject is reset to its starting position. Other- wise hand movement
successful.
Rotatelland DD, D E {-x, +X, -Y, +Y, -2, +2]
Agent must holding an object, so the agent rotates the object by 30 degrees along the axis specified by D. So if D = -Y then the object is rotated 30 degrees counter- clockwise along the y-axis. would result in the object colliding with an- other object then the action fails and the ob- ject is reset
The hand object re- mains in its rotated orientation.
OM
rotation
starting rotation.
DrepObject
The agent holding an object.
The object is dropped. The object near the location specified by i.j opens.
OM
OpenAt i,, 1SiJ 57
Recall the partitioning of an image into a 7x7 grid shown in Fig. ld There is openable object grid location.
F&M.
OH,
the center of the j,
OM,
C1 Closebbjects
There is at least one object within the agent's interactable range which open.
All open interactahle
F&M.
object are closed.
OH,
S
PlaceAt [, 1, < i,j < 7,
See the description of the object hiding episode. Here the 0,1,2 values for m cor- respond to the modalities ONTOP, Con- TAINEDIN, and BEHIND respectively.
An object manipula- tion episode begins with target specified by i.j.
OH
m 1,2]
ReadyPorSeeke:
The agent has successfully performed a FlaceAt action.
The hiding episode ends and the secking episode begins.
OH
Clainvisible
The goal object within 1.5m of agent and at least 10 pixcls the agents current view can be attributed to the object.
The seeking episode ends successfully.
Table G.1: A description of the actions available to agents along their success criteria, impact the environment, episodes which they are available.
34


Reviews:
[{"review_id": "jRSk4ylDYyk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author's propose an embodied adversarial reinforcement learning agent that can play a variation of hide-and-seek called Cache. This environment is a high fidelity interactive world. The authors argue that the agents are able to learn flexible representations of their observations which encode information such as object permanence, free space and containment. \n\nThe authors provide a well-written description of their game and provide well-designed visualizations to understand the interactions of the agent and the observations required for the learning problem. The authors present a concise and well-researched literature review which serves to distinguish their work as novel and well built on underlying prior work. \n\nThe authors make several contributions in this paper. First, an adversarial game Cache which permits the study of representation learning in the context of interactive visual gameplay. Furthermore, they present an agent which can perform strongly on the benchmark that they create even in comparison with human players. Finally, they present a study of the static and dynamic representations learned by the agent.\n\nThe authors provide an overview of the architecture for the Cache agent. It is well researched, well-reasoned and presented in a way that is easy to understand and reproduce.\n\nThe authors present multiple experiments in their paper and the corresponding deep Appendices. Specifically, they attempt to understand how agents can learn to proficiently hide and seek objects with static image representations and dynamic image representations\n\nOf particular interest are the dynamic tasks the authors present in Section 5: Experiments. The authors make allusions to study of human children and object permanence. The results are quite compelling and well presented in a way that makes them easy to understand. I would urge the authors to consider if there are any non-ablative baselines against which they may be able to compare their model. One baseline that the author's use is human volunteer comparisons, these are compelling and presented in the appendix.\n\nFigure 3 is perhaps the weakest figure in the paper. While it provides a great amount of information I would argue that the results are presented in a way that makes them less legible than if the figure was broken out into methods and results. The same could be said about Figure 4. The visualizations of the agents representations of synthetic and natural images are combined into a single image with performance on corresponding tasks. While I understand that the authors made these choices for space constraints, I would urge them to reconsider the visual hierarchy of the figures to emphasize the performance of their agents. \n\nI feel like this is a paper that would be of great interest and benefit to the ICLR community.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wrqc9ZeUN3J", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper intends to contribute a novel task (Cache, as realized in AI2-THOR), the architecture of a strong Cache agent which learns reusable representations which allow significant transfer performance, and novel methods for evaluating the quality of dynamic image representations. The first and third contributions are directly related to the conference topics, and the second provides additional evidence in favor of the paper’s core idea: training on interactive gameplay allows learning flexible representations (in the sense of supporting many tasks via transfer) even when images are highly correlated and synthetic.\n\nThe key strength of the paper is the very general core idea it advances and how this idea is explored via a novel task. The paper is easy to read and convincing. The partition into details needed for the paper’s core argument and details specific to the experiments in the appendix is well done. (If anything even more could have been pushed to the appendix.)\n\nOne weakness of the paper is that a specific notion of “flexible” (which is mentioned in the title and twice in the abstract but nowhere else) is not advanced or integrated with the core idea. How does gameplay relate to flexibility? Why might flexibility be harder to achieve via passive learning or reinforcement learning with fixed reward functions? Because the authors place stress on the idea of how play and interaction contribute to representation learning (rather than a new method), slightly more space should be given to developing the general idea. The idea is not specific to vision, but only vision-related representations are considered. Sketching how the idea ought to work for text or audio would be useful if the focus really is on this very general idea.\n\nRecommendation: strong accept. The philosophical aims of the paper make it stand out amongst the mass of related work that is otherwise very engineering focused. The experiments are soundly executed in a way that ends up clearly demonstrating the core idea.\n\nQuestions for the authors:\n- A step where the hider needs to retrieve the object they hid would seem appropriate. Are there certain limitations of the AI2-THOR environment that make adding this step (which would seem to expose more of the richness of the simulated world through fixed rules of the game) infeasible to add?\n- Inversely, do the authors feel that it was important that the hider manipulate the object into the desired location? How much of the richness of the simulated world comes through in the task feels relevant to the core ideas of the paper, but the paper currently does not address this kind of detail in the design of the Cache game within AI2-THOR.\n\nSection-by-section reactions: (to see how opinions change over time)\n\nTitle+Abstract:\n- The notion of “flexible” seems to be at the heart of this paper’s intended contribution. Hopefully it will be defined in the body text. Uh oh, it looks like “flex” only ever appears on the first of the submission’s 36 pages. Hopefully a synonym will get defined later.\n\nIntroduction:\n- Excellent motivation.\n- Good that representations of interest (SIRs/DIRs) are named and distinguished. Many other papers, in the interest of highlighting end-to-end training, would forget to do this.\n- Good explicit list of contributions, excellent that two are specifically centered on representation learning.\n- Missed opportunity to highlight a distinct role for “flexible” representations. (I don’t quite know what it should mean beyond supporting transfer well. A representation that could easily be scaled up or down in dimensionality by stripping channels in a well defined order might be considered flexible in another sense. Likewise, one that was defined in terms of pluggable input modules to work with novel combinations of familiar input types might be considered differently flexible. What kind of flexibility do you want?)\n\nRelated Work:\n- Another take on learning visual representations via interactive gameplay is seen in https://arxiv.org/abs/1812.03125 where the authors learn a SIR (trained on a proxy task of predicting videogame memory state) that supports the use of low-continuous-space exploration strategies like rapidly-exploring random trees. The representations are learned offline/passively, but they are learned as to improve the efficiency of the very exploration process that builds that dataset for offline learning.\n\nPlaying Cache in a Simulation:\n- It seems notable that the hiding agent is never asked to retrieve the object they have hidden. Without this step, the hiding agent may find ways of manipulating objects in a way that makes them simply unretrievable (e.g. the object is pushed into a corner in a way that causes it to glitch out of the room, etc.). A step like this would require the hider to learn a finer grained representation of the hiding location that gives itself a clue as to how it should be retrieved (e.g. “under the couch in a place you’ll never be able to see but will be there if you actually reach for it).\n\nLearning to Play Cache:\n- Great!\n\nExperiments:\n- All well done.\n\nDiscussion:\n- “We believe that it is time for a paradigm shift via a move towards experiential,\ninteractive, learning.” -- something similar has been said by many other researchers in many different decades, so it would be good to say what’s different about the situation in 2021. The difference now seems to be the availability of simulators with visual fidelity comparable enough to reality to demonstrate meaningful sim2real transfer. Are there other bullet points that could be added to a why-now argument?", "extra_scores": {}, "extra_text": {}}, {"review_id": "U-LzNNFC_mH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes embodied game-playing with artificial agents as a method to learn better representations of their environment. They describe a game, cache, which is a variant of hide-and-seek played in a virtual environment and a method for training an agent to play the game. They present results which demonstrate that the static representations learned through game-playing perform better than other pre-training tasks within the same virtual environment, on both virtual vision and real world vision applications. They also show that the dynamic representations are useful for completing object permanence tests inspired by developmental psychology research.\n\nThis research direction (both biologically-inspired CV as a whole and specifically game playing as a pre-training task) is exciting and seems extremely promising. Assuming I understand the methods and results correctly, this seems like a clear accept. My only reservations are the complexity of the task and how the results are presented in the paper, which are relatively minor issues.\n\nThe task, Cache, seems extremely complicated to implement and train, since it involves five different stages of embodied exploration/action and adversarial reinforcement learning. While all of these steps are necessary to play the game like humans (or ravens) do, it is unclear to me whether the important learning is occurring as a result of the whole process, or of a specific stage. It seems possible that a simpler task may have comparable results, and it would be an interesting direction for future research to investigate how each of these stages are contributing to learned object permanence. I'm not an expert in developmental psychology, but I know there is some research showing that smooth-motion visual signals are the main prerequisite for object permanence in chickens (https://onlinelibrary.wiley.com/doi/abs/10.1111/desc.12796). An ablation study over game mechanics or world properties could lead to some really interesting results.\n\nOf course, the paper you've submitted already describes a huge volume of research work and doesn't need more experiments. Unfortunately, it is a bit difficult to follow as a result. You may want move some content from Fig. 3 and 4 (those figures are difficult to parse, even with my pdf zoomed in 250%) to the appendix, if only to focus the reader's attention on specific results. While arguing that your agent successfully plays the game is important, it should take a backseat to the experiments which probe the properties of your static and dynamic image representations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LWRpScftEqa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\n\nThis paper examines the representations learned during adversarial gameplay, specifically a hide-and-seek game called Cache.  The hiding agent must place an object in a room such that the seeker agent cannot find it.  The authors argue that the adversarial nature of the game shapes the representations.  Inspired by psychology experiments performed on children, the authors examine both static and dynamic representations to probe whether they contain information about properties of the environment such as object permanence.   \n\nPositives\n---------\n\nThe paper addresses an interesting problem by studying representations learned by agents interacting in an environment.  These representations perform favorably compared to passive representations. \n\nThe paper is thorough, detailed, and well written.  \n\nThe inclusion of dynamic representations is interesting, and the psychology-inspired experiments are adapted well to the Cache agent's environment.\n\n\nNegatives\n---------\n\nMy primary concern with the paper hinges on a trick the authors used to get their agent to learn meaningful representations.  The authors introduce Visual Dynamics Replay (VDR) to improve the agent's performance.  Essentially, this is a set of auxiliary tasks and self-supervision derived from the agent's interactions with the environment.  Although not a problem in itself, VDR was not used in the Navigation agent or Random agent used as baselines in the paper.  This leaves the conclusions of the paper much weaker.  It is unclear if these auxiliary tasks of VDR or the adversarial gameplay of the problem are the driving factor behind the learned representations.  \n\nBecause the scope of the paper is so broad, it is difficult to evaluate some of the contributions.  For example, the perspective simulation module seems useful, but there are no ablation experiments evaluating changes to the model.\n\n\nReasons for Score\n-----------------\n\nAlthough the paper is intersting, detailed, and thorough, I have trouble with the strong conclusions drawn from the experiments because of the possible confounds of VDR.\n\n\nMinor Issues that did not affect score\n--------------------------------------\n\nGraphs in figure 5 are difficult to see and interpret, even at full zoom on a monitor.\n\n\nUpdate after rebuttal\n------------------\n\nI am very interested to see the results of the experiment mentioned in the rebuttal.  Although you mention that gameplay dynamics and ability to interact with objects is crucial, I still feel that this isn't as cleanly demonstrated as it could be without addressing these confounds.  However, in retrospect I also realize that my initial score suffered from tunnel vision on that single issue, which is important to several claims in the paper, but is by no means the only contribution.  The paper is ambitious in scope, novel, and well written, so I have increased my score.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Motivated by the importance of gameplay in the development of critical skills for humans and other biological species, this work aims to explore representation learning via gameplay in a realistic, high fidelity environment. Inspired by childhood psychology, they propose a variant of hide-and-seek game called "Cache" built on top of AI2-THOR, where one agent must place an object in a room such that another agent cannot find it, and demonstrate that the adversarial nature of the game helps the agents learn useful representations of the environment. They examine the difference in representations learned via such a dynamic, interactive adversarial gameplay approach, vs other more passive approaches involving static images.

The paper is well written and motivated, and easy to follow. All reviewers agree that the paper will be a great contribution to the ICLR community. I believe this is an important work, because not only does it challenge the traditional way of training many components of our systems passively (via static image recognition models), it synthesizes ideas from various disciplines (psychology, embodiment, ML) and provides an excellent framework for future research. For these reasons I'm recommending we accept this work as an Oral presentation.
Decision: Accept (Oral)