Visual Hide and Seek
Authors: Boyuan Chen
Shuran Song
Hod Lipson
Carl Vondrick
Venue:  (2020)
Keywords: Embodied Learning
Self-supervised Learning
Reinforcement Learning

Abstract: We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator. We place a variety of obstacles in the environment for the prey to hide behind, and we only give the agents partial observations of their environment using an egocentric perspective. Although we train the model to play this game from scratch without any prior knowledge of its visual world, experiments and visualizations show that a representation of other agents automatically emerges in the learned representation. Furthermore, we quantitatively analyze how agent weaknesses, such as slower speed, effect the learned policy. Our results suggest that, although agent weaknesses make the learning problem more challenging, they also cause useful features to emerge in the representation.

Under review as a conference paper at ICLR 2020
VISUAL HIDE AND SEEK
Anonymous authors Paper under double-blind review
ABSTRACT
We train embodied agents to play Visual Hide and Seek where prey must nav- igate in simulated environment in order to avoid capture from predator. We place variety of obstacles in the environment for the prey hide behind, and we only give the agents partial observations of their environment using ego- centric perspective. Although we train the model to play this game from scratch, experiments and visualizations suggest that the agent learns predict its own visibility in the environment. Furthermore, we quantitatively analyze how agent weaknesses, such as slower speed, effect the learned policy. Our results suggest that, although agent weaknesses make the learning problem more challenging, they also cause more useful features be learned.
INTRODUCTION
We introduce the task of Visual Hide and Seek where a neural network must learn to steer an em- bodied agent around its environment in order to avoid capture from predator. Our hypothesis, which our experiments suggest, that learning to play this game will cause useful representations of multi-agent dynamics emerge. We train the agent to navigate through its environment to maximize its survival time, which the model successfully learn to do. However, since we train the model from scratch to directly map pixels to action, the network can learn representation of the environment and its dynamics. Figure [ illustrates the problem setup.
We designed this game mimic the typical dynamics between predator and prey. For example, we place a variety obstacles inside the environment, which create occlusions that the agent can leverage hide behind. We also only give the agents access to the first-person perspective of their three-dimensional environment. Consequently, this task is substantial challenge for reinforcement learning because the state is both visual (pixel input) and partially observable (due occlusions). To carefully study the emerged representations and agent dynamics, we probe different environmental factors well as the agents' abilities, and our experiments quantitatively suggest that the agent automatically learns to recognize the perspective of the predator and its own visibility, which enables robust hiding behavior.
Seeker
Hider
(a) Secker's Visual Observation
(b) Simulation Environment
(c) Hider's Visual Observation
Figure| Visual Hide-and-Seek: We train models to play game of visual hide and seek and analyze the dynamics that automatically emerge. We show that, although agent weaknesses make the learning problem more challenging, they collaterally encourage the learning of rich representations for the scene dynamics.

---

Under review as a conference paper at ICLR 2020
However, what intrinsic structure in the game, if any, caused this strategy to emerge? We quantita- tively compare a spectrum of hide and seek games where we perturb the abilities of agents and the environmental complexity. Our experiments show that, although agent weaknesses, such slower speed, make the learning problem more challenging, they also cause the model to learn more useful representations of its environmental dynamics. We show there is "sweet spot" where the weakness is strong enough to cause useful strategies to emerge without derailing the learning process.
This paper makes three principal contributions to embodied agents. Firstly, we introduce the prob- lem of visual hide-and-seek where an agent receives a partial observation of its visual environment and must navigate to avoid capture, Secondly, we empirically demonstrate that this task causes representations of other agents in the scene to emerge. Thirdly, we analyze the underlying reasons why these representations emerge, and show they are due to imperfections in the agent's abilities. The rest of this paper analyzes these contributions in detail. We plan to release all software, data, environments, and models publicly to promote further progress on this problem.
RELATED WORK
Our work builds on research for embodied agents that learn to navigate and manipulate environ- ments. Embodied agents with extensive training experience are increasingly able to solve a large number of problems across manipulation, navigation, and game-playing tasks (Mnih et al. 2015
Gu et al. 2017 Zhu et al.) 2b) [Siveer et al. 2017 Kahn et al., 2018: Kalashnikov et al. 2018 Mirowski et al., 2018). Extensive work has demonstrated that, after learning with indirect supervi- sion from a reward function, rich representations for their task automatically emerge (Bansal et al.
2017 Lowe et al. 2017 Liu et al. 2019 Jaderberg al., 2019). Several recent works have created
3D embodiment simulation environment Kolve et al. (017] Brodeur et al. 2017 Savva et al. 2017
Das et al., 2018 Xia et al. 2018 Savva al. 2019) for navigation and visual question answering tasks. To train these models, visual navigation is often framed as a reinforcement learning problem
(Chen al. 2015 Giusti al.] 2015 Oh al., 2016 Abel et al. 2016; Bhatti et al. 2016 Daftry
ct al.] 2016: Mirowski et al. [016]] Brahmbbatt & Hays 2017: Zhang et al.) 2017aa Gupta et al.,
2017a] Zhu et al. [017a] Gupta et al. 2017b; Kahn al. 2018). Moreover, by incorporating mul-
tiple embodied agents into the environment, past work has explored how to learn diverse strategies
and behaviors in multi-agent visual navigation tasks (Jaderberg et al. 2019 Jain et al. 2019). For
full review of multi-agent reinforcement learning, please see (Panait & Luke 2005 Bu et al. 2008
Tuyls & Weiss 2012 Shoham et al. 2007 Hernandez-Leal et al.) 2018).
Our paper contributes to rapidly growing area to learn representations from indirect supervision.
Early work has studied how features automatically emerge in convolutional networks for image recognition (Zhou et al.) 2014 Zeiler & Fergus 2014). Since direct supervision is often expensive to collect, there has been substantial work in learning emergent representations across vision (Doersch)
al.
2015 Vondrick al. 2018), language (Kottur et al. 2017 Radford et al. 2017). sound (Owens et al. 2016 Aytar et al., (016)), and interaction (Aytar et al. 2018 Burda et al. 2018). We also study how representations emerge. However we investigate the emergent dynamics in the two-player game of visual hide and seek. We characterize why representations emerge, which we believe can refine the field's understanding of self-supervised learning dynamics.
This paper is concurrent to (Baker et al. 2019), and we urge readers to watch their impressive results on learning to play hide and seek games. However, there are a few key differences between the two papers that we wish to highlight. Firstly, in contrast to (Baker et al.) 2019), we focus on hide and seek in partially observable environments during both training and testing. Our environment is firee-dimensional and agents only receive an egocentric two-dimensional visual input, which creates situations abundant with occlusions. Secondly, the input to our model is a visual scene, and not the state of game engine. The learning problem is consequently very challenging because the model must leam perceptual representations in addition to its policy. Our experiments suggest this happens, but the richness of the visual representation depends on the impediments to the model. Finally, we focus our investigation on analyzing the underlying reasons why different behaviors emerge, which we believe will refine the field's insight into self-supervised learning approaches.
2

---

Under review as a conference paper at ICLR 2020
3 HIDE AND SEEK
We first present our environment and learning problem, then describe our interventions to understand the cause of different emergent behaviors.
3.1 ENVIRONMENT AND LEARNING
We created 3D simulation for hide-and-seck using the Unity game engine, which through- out this paper. There are two agents in this game: the hider and the seeker. Each agent receives only first-person visual observation of the environment with 120-degree field of view, and navigates around the environment by selecting actions from a discrete set (move forward, move backward, turn left, turn right, stand still). The environment is square of 14 unit 14 unit, and any real value position inside the square without being occupied any obstacle is a valid game state. The speed of the hider two units per second while the speed of the seeker is one and half units per second. Our simulation runs in 50 frame per second. The agents can turn 3.3 degree per frame. Each agent has diameter of one unit. In contrast to other multiplayer games (Sukhbaatar al.) [2016] Moravčík et al. 2017 Lample & Chaplot 2017; MacAlpine & Stone [017] et 201] et al., 2018). the egocentric perspective of our problem setup makes this task very challenging. We place obstacles throughout the environment to create opportunities for occlusion.
We use simple deterministic policy for the seeker, which is as follows. If the seeker can see the hider, move towards it. If the seeker cannot see the hider, move towards the hider's last known position. If the seeker still cannot find the hider, then will explore the environment by waypoints in round-robin fashion. The game episode concludes when the seeker successfully "catches" the hider. We define catching the hider as a collision between the two agents.
Seeker Policy:
Hider Policy: Given the environment and the seeker algorithm, we train policy for the agent to hide. Given a first-person observation of the agent's environment, we estimate the action to execute with convolutional neural network. We train the model to maximize its reward function using PPO (Schulman et al. 2017). Each time step, we provide a small reward (+0.001) for each living step, and large negative reward (-1) once captured. We call this model basic.
Starting Positions: The starting positions of the hider and the seeker are randomly sampled on a continuous grid for each episode. We also specify their initial orientations so that they always start by looking towards each other (even if there is an obstacle between them).
Our intention that, by learning hide, the model will learn features for the recognition of objects in its visual field, such as obstacles and other agents. To analyze this, we fit linear regression from the learned features to classify different objects and game states. We encode object and game states as discrete variables, and use the classification accuracy as measure of how well the features are encoding different states. Our experiments will use this approach to probe the learned features.
3.2 SYSTEMATIC INTERVENTIONS
We systematically intervene on the learning process to understand the mechanisms behind emergent features. We chose these interventions because they either give the agent an ad advantage or weakness. We summarize these variations in Table [ By training the models from scratch with different in- terventions and analyzing the learned representation, we can analyze how environmental features cause different strategies to be learned.
Speed: The first intervention we will make is manipulating the relative speed of the hider and seeker. We explore two variations. In the first variant, we slow down the hider. Likewise, in the second variant, we speed up the hider. We call these agents slowerhider and fasterhider respectively. The conditions are otherwise the same as the basic model.
Self- Visibilii; Awareness of your own visibility is a crucial feature for successfully surviving during hide and seek (Russell et al. 2012 Reinhold et al. 2019). We explicitly incorporate this into the reward function as auxiliary dense reward signal. Along with the sparse reward above, this agent receives an additional reward of 0.001 if the seeker does not see at the current time step. Likewise, the agent receives punitive reward of -0.001 if is currently visible by the seeker. We refer to this
3

---

Under review as a conference paper at ICLR 2020
agent visibilityreward. Furthermore, we also variant where the hider has faster speed, which we call visbilittyrewrrd-fatter
Environmental Complexity: We also intervene the complexity of the environment, which lets us analyze the impact of the environment on the emergent features. We use two variations. Firstly, use stochastic policy for the seeker agent (instead of deterministic). Specifically, the stochastic seeker randomly visits locations in the map until the hider is within its field of view, at which point immediately goes towards it. We name this variation stochasticseeker. Secondly, we use stochas- tic map where we randomly select a number of objects and also randomly position throughout the environment. Consequently. the difficulty of the maps will change between easy (many occlusions) to difficult (few to none occlusions). We name this variation stochasticmaps.
Hider Name
Speed
Seeker Policy
Maps Deterministic
Visibility Reward
basic fasterhider
2
Deterministic
None
A
Deterministic
Deterministic
None
slowerhider
1
Deterministic
Deterministic
None
stochasticseeker
2
Stochastic
Deterministic
None
stochasticmaps stochasticseeker
2
Stochastic
Stochastic
None
visibilityreward
2
Deterministic
Deterministic
Yes
visibilityreward + faster
A
Deterministic Deterministic
Yes
Table Interventions: We perturb the learning process in order to understand the causes of differ-
ent strategies during visual hide and seek. When the speed is "A", the agent has allowed to accelerate to reach higher speeds at a rate of two units per time period squared.
3.3 IMPLEMENTATIO DETAILS
We implement our simulation using the Unity game engine along with ML-Agents (Juliani et al. 2018) and PyTorch (Paszke et al. 2017). We plan to publicly release our framework. Given an image as input, we use a four-layer convolutional network as a backbone, which is then fed to two-layer fully connected layers for the actor and critic models. We use LeakyReLU (Maas al.] 2013) activation function throughout the network. The input pixel values are normalized to range between 0 and We train the network using Proximal Policy Optimization, which has been successful across reinforcement learning tasks (Schulman et al. 2017). We optimize the objective using Adam optimizer (Kingma Ba 2014) for 10" steps with a learning rate 3.0e 4 and maximum buffer size of 1,000. We then rolled out the policies for 100 episodes using the same random seed across all the variants. Each episode had 1, 1,000 number of steps at maximum. Following (Mnih al., (015), we 6-step repeated action, which helps the agent explore the environment. Please see the appendix for full details.
4 EXPERIMENTS
The goal of our experiments is to analyze the learned capabilities of the polices, and characterize why strategies emerge in hide and seek games.
Overall, our experiments show that the hiding agent is able to learn to avoid capture. We show that each policy learns different strategy depending on its environmental abilities. During the process of learning to hide, visible or not. We also found that with judicious weakness, the model leams to overcome its disadvantage by learning rich features of the environment. The rest of this section investigate these learning dynamics.
results suggest that the agent automatically learns to recognize whether itself
4.1 DOES THE AGENT LEARN TO SOLVE ITS TRAINING TASK?
We first directly evaluate the models on their training task, which is to avoid capture from the seeker. We quantify performance using two metrics. Firstly, use the average number of living steps across all testing episodes. Secondly, we use the success rate, which is the percentage games that the hider avoids capture throughout the entire episode. Table 2 reports performance, and shows that
4

---

Under review as a conference paper at ICLR 2020
Average of Living Steps
Success Rate
Environment
(trained / random)
(trained / random)
basic fasterhider
513 +: 10/
46.02% 2.60%
$22 3/43 +
33.27%
2.00% /
visibilityreward faster
422 24/4:1 4
33.29%
1.33% /
stochasticsecker
271 27133/422
23.75% = 1.52% /
stochasticmaps + stochasticseeker
281 12/55 #7
17.45%
1.59% 0
visibilityreward slowerhider
248 = 13/46
19.93% = 3.55% /
5
1.08% + 0.07%
Table 2: Hider Performance: We show the success rate and average number of living steps of different Hider agents and environments. The maximum number of steps for each episode is 1,000.
all agents solve their task better than random chance, suggesting the models successfully learned to play this game. As one might expect, the stronger the agent, the better the agent performed: the faster agent frequently outperforms the slower agent.
4.2 WHAT VISUAL CONCEPTS ARE LEARNED IN REPRESENTATIONS?
In this experiment, want to investigate which visual concepts encoded in the latent repre- sentation. For example, learning to recognize the predator and whether the agent is itself visible to the predator are important prerequisites for hide and seek strategies. Consequently, corresponding tasks to study the encoded visual concepts from the trained policies. The first task an- alyzes whether the learned features are predictive of the seeker (Seeker Recognition). The second analyzes whether the hider is able to infer the visibilityy of itself to the seeker (Awareness of Self- Visibility). These two tasks can be categorized
design two
binary classification downstream tasks. To do this, we extract the mid-level features from the learned policy network in different environments, and train binary logistic regression model on these two proposed tasks.
Seeker Recognition (/random init policy)
Awareness of Self Visibiisiyylity
Environment
( (/random init polcy)
basic fasterhider
91.03 0.21 79.25 + 2.42
75.84 + 0.66 63.92 + 0.50
90.01 + 0.34 / 74.84 + 1.91
77.88 + 0.05 60.30 +'t 0.13
visibilityrewaard + fasterhider
79.19 + 1.11/74.18 + 1.95
65.05 + 1.88 60.84 + 0.84
stochasticseeker
96.17 0.25 1.29
94.00 0.58//6 65.05 + 0.13
stochasticmaps stochastiesecker
95.55 + 0.38/80.75 1.67
94.25 = 0.75 70.80 + 0.38
visibilityreward slowerhider
96.21 1..40 = 0.15
95.30 0.38 63.88 + 2.30
83.96 + 0.46 / 77.79 + 0.54
81.71 + 0.63 68.59 + 0.34
Table 3: Downstream Visual Perception Tasks: the table shows the classification accuracies on two important visual perception tasks using mid-level features from all the trained policies. Regard- less the advantages of the hider agent and high performance in the hiding task, the "visibilityre- ward faster" policy has the worst performance on both of the tasks.
The extracted feature the activation from the last convolutional layer in the policy network with dimension of 512. The labels are from ground-truth game states. We emphasize that we do not use any of these labels during learning. We only rely on them for studying the representations after the training is completed. Throughout the paper, we use "S" and "-S" to represent the game state of whether the seeker visible to the hider, and "H" and "-H"" represent the game state of whether the hider is visible to the seeker. Therefore, the inputs of these two tasks the visual observa- tions of the hider agent, and the labels are "H"M/1H and "S" S"-S" respectively. We then use the classification accuracy the test images measurement how well the latent representations encode corresponding visual concepts. We rolled out each learned policy for 6,000 steps for feature extraction and use 0.8 train and test split.
Table shows the results for both tasks. The faster agent, especially "visibilityreward + faster", performs worst these perception tasks, even though has the most advantages among all the
5

---

Under review as a conference paper at ICLR 2020
variants. Since the faster agents performed better at the survival task (Table 2 this suggests that the performance in the training task does not directly cause the performance in other downstream tasks. To further demonstrate this, visualize the embedding mid-level features of "visibilityrwward" and "visibilityrewardlfaas faster" policy with respect to the labels of the above two tasks using t-SNE (Maaten & Hinton 2008) in Figure 2 The features for "visibilitrreward" policy is more separable than the features for "visibilityrewart" ityreward faster" policy is worse downstream visual perception tasks even though the hider in this case can move a lot faster.
faster". This qualitative evidence supports that the "visibil-
In the second task, we analyze whether the hider is able to infer its own visibility the seeker. The features from the stochastic variants and the "visibilityrwward" are the most predictive of the hiders own visibility. This could because the hider agent encouraged to pay more frequent attention to the seeker agent in order to get higher rewards.
(A) visibilityreward
(B) visibilityrwward fasterhider
Figure 2: t-SNE Embedding of mid-level features colorized by the labels of the two visual percep- tion tasks from Section 4.2.
4.3 WHAT CAUSES USEFUL FEATURES TO EMERGE?
To analyze the relationship between training task difficulty and emerging skills, Figure compares the classification accuracy of using the features to predict self-visibility versus the agent's survival time. Each dot in the plot represents corresponding value of each dot drawn from Table Table 3 capture at all, then the features capture, then the features are also poor! Instead, there is a concave relation that suggests agent weaknesses actually advantageous for learning rich representations. We believe this is the because by giving the model disadvantage, the learning process will compensate for the weakness, which it does by learning stronger features.
variant of the environments specified in Table [ The the agent is not able to avoid clearly poor. However, if the agent is able completely evade
4.4 How DOES THE TRAINED MODEL HIDE?
Each agent is trained with varying advantages and disadvantages. We are interested in understanding how these variations affect the behavior of learned policy and how exactly the trained model learns to hide. We quantitatively analyze the dynamics of the learned policies by probing the internal game engine state. Note that we only the game engine states for analysis, not learning.
Classification Accuracy Self- Self-Visibiltt Visibility
Figure 3: Quality of Representation Sur- vival Time: We compare the performance of models their survival time versus how well the internal representation is predictive of downstream recognition tasks. Each red dot represents one policy trained with differ- ent advantages or disadvantages. The curve is the parabolic best fit. Interestingly, improved task performance (survival time) does not ways create stronger representations. Instead, the best representations arise from the mod- els with intermediate disadvantages. When the model has a weakness, the model learned to overcome it by instead learning better features.
100%
90%
80%
70%
50%
100
200
300
400
500
Survival Time (steps)

---

Under review as a conference paper at ICLR 2020
basic
fasterhider
slowerhider
0.4
visibi
visibilityrewandaatte
0.2
stochasticmaps
stochasticsocker
-0.2
(S, H. (0) or (S, H. '00
(-S; Wr 00)
S" 'H, (0)
("Ss ".: (O)
(*S",, W. -0)
(s; H.. '0)
(Sy W.. '0)
Figure 4: Frequencies of Visual States. We show the relative frequency over chance that each visual perception states are visited. "S" stands for whether the Seeker is visible to the Hider, "H" stands for whether the Hider is visible to the Seeker, and "O" means whether there is any obstacle visible to the Hider. "-" denotes the opposite condition. This plot demonstrates quantitative way to measure different behaviors using visiting frequencies among important states. For example, we can tell that "basic" agent learns to run with its back facing the seeker as shown in the high blue bar for state ('-S,, 'H', 'O'). The "basic" hider learns always run away because its similar speed with the seeker. The "slowerhider" favors to stay at ("-S', '-H'. 'O") where the hider and the seeker cannot see each other and there at least one obstacle in front itself. This suggests the slower hider is learning to leverage the environment to compensate for its speed disadvantage. "vissbilitrreearffaseee"""" often stays either the state where they cannot see each other as suggested by the high brown bars of state ("-S', '-H', 'O") and ('-S', '-H', '-O'), or stays at the state where the it can see the seeker but the seeker cannot see itself as indicated the high brown bars at ('S', "H'', 'O') and ("S", H'' 'O'). This suggests that the "visibilitrrewaaffattee"" can use its speed to hide completely and check the status of the seeker while keeping itself safe.
Frequency of Visual States: To analyze states for each policy, we track three visual states in the hide and seek game. 1) "S" / "-S": whether the hider can see the seeker, 2) "0" / "-0": whether the hider can see any obstacle and 3) "H" """": whether the hider visible to the seeker. We rolled out each learned policy for 50,000 steps, and counted how often they enter these game states. Then we plot their relative frequencies by subtracting the frequencies of corresponding random initialized policies from the absolute visiting frequencies.
Figure 4 ccoparre the frequency of states for each model. We observe a few key differences among the agents. When the speeds and abilities of the hider and seeker the same (basic), the hider learns a policy to first turn away from the seeker then run away, as evidenced by the high blue bar for the state (-S, H. O). However, when the hiding agent slower than the seeker, the hider frequently enters a state where the two agents cannot see each other, but an obstacle is visible, as evidenced by the high green bar for the state (-S, H. O). In other words, when the hiding agent has a disadvantage, the model cannot rely on its intrinsic capabilities, and will instead learn to leverage the environment perform its task. In contrast, when the hiding agent has several advantages such as "visibllitreearrffattee"""" the policy also learns to make use of them during the training process.
shown in the high brown bars at state ('-S', '-H', 'O') and ('-S', '-H', '-O'), ), as well as ('S', "H'', 'O') and ('S", 'HH', 'O'), this hider learns to its speed and auxiliary visibility reward to hide completely from the sight of the seeker and monitor the status of the seeker while keeping itself safe at the same time.
Probability of State Transitions: While the state frequency shows the agents' most common states, does not reflect the dynamics of the decision making process. We use the state transitions to further analyze the agent behaviors. For example, a transition from "H" to "-H" indicates a successful hiding attempt. Following the previous state definitions, there are eight possible combinations
7

---

Under review as a conference paper at ICLR 2020
(-S, HH
(",,,
viehiliyecoenllleii
modeninooiishiiigrevvrtt
anchooirnccker
vilbiityrewadd
sochasicmaps
andemintssecbaatccmrg
aloerhier
futerhider
redominitasii
ectos
0.015
oaces
Road
R0ag
(A) The hider and seeker cannot see each other The hider and the seeker can see each other
( H. -0- -00000--000 "H H., 0) (" H. 0) (S. H. 00)
vistoliistewardhstrr
sochudiosskkrr
nundkmatisschhticsseke
vishiliiyrwwar
stchazicmaae
andeminttseccheeinn
sboucthicr
modomintstoweeide
fecurtider
Ieaic
ramkminilteii
Paze
oar
0012
00r4
The hider and seeker cannot see each other The hider
see the seeker now while keep itself invisible from the seeker
Figure 5: Transition Dynamics: We show the probability of transitions between states across all the variants. All of the four plots share the the probability of transiting from one state to the other. The state transitions are labeled on top of each graph, and explanation of the transition on the bottom of each graph. "randominit" prefix represents the corresponding random initialized policy for each variant.
y-axis in the table above. Each bar indicates
of these three states and (8 x - 8) = 56 transitions in total excluding identity state transitions. We summarize representative cases, and will release the full results for all combinations along with code bases.
Figure 5 quantifies different exploration strategies for each model. For example, the top (A) suggests that faster agents tend to monitor the status of the seeker while exposing itself. This likely happens because the faster agents are fast enough that they can immediately react when the seeker becomes too close. Specially, the two red bars in (A) denotes that "fasterhider" policy has the highest probability transition from the state where the hider and the seek cannot see each other to the state where the hider and the seeker can see each other. When the status of the seeker becomes clear, the hider in "fasterhider" also exposes itself to the seeker's view.
However, when the faster agent is explicitly penalized for exposing itself ("visibilityrewarrd faster- hider"), the agent tends to monitor the status of the seeker while keeping itself hidden. This suggested by the two red bars from Figure (B) (B) where "visibilltyreward + fasterhider" policy has the highest probability to transit from the state where the hider and the seeker cannot see each other to the state where the hider the agent has learned to watch the predator, but remain outside of its field of view.
the seeker, yet the secker cannot the hider. In these cases,
Distance between Agents with respect Time: We also measure the learned behavior by quantifying distances between the two agents with respect to time. In contrast to previous analysis which focused on short-term dynamics, this lets quantify long-term dynamics.
Figure plots the distance between agents versus time, alongside representative qualitative cases. When the two agents have the same capabilities (Figure (b). the agents gradually get closer each
8

---

Under review as a conference paper at ICLR 2020
Disiance
(A) basic
Time stops
Distance
BB) fasterhider
Timee slepss
(C) fasterhider + visibilityreward
Distunce
Time sleps
We use distance between agents with respect to time steps to quantify long-term dynamics of the game. Each rollout trajectory represented by constant color over time. We show multiple trajectories to reflect the general pattern for different policies. Hider agents from different policies exhibit diverse pattern in this plot. We further append qualitative demonstrations to explain what dynamics each pattern correspond to on the right side of the plots. The corresponding trajectory plotted red line.
Figure 6: Distance Versus Time:
other. However, when one of the hiding agent is faster, the distance between the hider and seeker will significantly oscillate (Figure bp.c). This is consistent with reactive strategy where faster agents are able to stand still until they are threatened, at which point they simply run away.
Next the plots, also show visual demonstrations that depicts the correlated strategies learned by different hider agents. The corresponding trajectory colored in red. One key difference is between the faster hiders with and without visibility dense reward. The "fasterhider" moves towards the seeker while facing at the same time, then it moves away when they get too close. On the other hand, the "fasterhider + visibilityreward" agent moves towards the seeker by monitoring the seeker from behind.
DISCUSSION AND FUTURE WORK
Our experiments suggest there are many diverse strategies for learning to hide from a predator. Moreover, during the learning process, the agents learn visual representation for their task, such as recognizing their own visibility. However, experiments show that this emergent representation requires a judicious disadvantage to the agent. If the weakness is too severe, then the learning is derailed. If there is weakness policy without requiring strong representations. However, with moderate disadvantages, the model learns to sufficiently overcome its weakness, which it does by learning strong features.
advantage, then the learning just discovers reactive

---

Under review as a conference paper at ICLR 2020
(a) fasterhider the Hider moved towards the Seeker tended move backward, found design flaw in the Hider caught because quickly, then got caught
(b) slowerhider the Hider
(c) slowerhider the Hider (d) stochastiemaps the
then died due
game, then stay there to did notice there forever
tion
an obstacle behind
Figure 7: Representative Failures: We show few cases where the hider is unable to avoid cap- ture. Many of these failures are due to lack of memory in the model, suggesting improved memory representations will help this task.
We believe visual hide and seek, especially from visual scenes, is promising self- self-supervised task for learning multi-agent representations. Figure X shows a few examples where the hider fails to escape from the predator. Many of these failures are due to the lack of memory in the agent, both for the memory of the map and memory previous predator locations, which invites further research on computational memory models (Milford et al. 2004; Gupta et al., [201aa] Zhang et al. 2017b
Parisotto & Salakhutdinov 2017: Khan et al. 2017 7 Wayne et al. (2018). Overall, these results sug- gests that improving the agent's ability learn to hide while simultaneously increasing the severity of the disadvantage will cause increasingly rich strategies to emerge. With more complicated en- vironment up such
layout, diverse object categories and allowance for agents and objects interaction, we believe more visual representations and different level of game dynamics can be studied in future research.
complex
Furthermore, we hope that our work can also encourage future works on specialized algorithms for visual egocentric multi-agent learning. Group of agents with different goals and capabilities can cause the environment become highly non-stationary. Better policy learning algorithms and feature extraction models are likely improve the learned strategies and representations. We hope that our work can step closer to study these future research topics and can open up interesting ideas and directions.
REFERENCES
David Abel, Alekh Agarwal, Fernando Diaz, Akshay Krishnamurthy, and Robert E Schapire. Ex- ploratory gradient boosting for reinforcement learning in complex domains. arXiv preprint arXiv:1603.04119 2016.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, pp. 892-900, 2016.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018.
Bowen Baker, Ingmar Kanitscheider. Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 1XXx:1909.07530 2019.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. arXiv preprint arXiv:1710.037488 2017.
Shehroze Bhatti, Alban Desmaison, Ondrej Miksik, Nantas Nardelli, N Siddharth, and Philip HS Torr. Playing doom with slam-augmented deep reinforcement learning. arXiv preprint arXiv: /6/2. 00380, 2016.
10

---

Under review as a conference paper at ICLR 2020
Samarth Brahmbhatt and James Hays. Deepnav: Learning navigate large cities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 193-5202, 2017.
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. Home: A household multimodal environment.
arXiv preprint arXiv:1711.1101 2017.
Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part (Applications and Reviews), 38(2):156-17) 2008.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 04355, 2018.
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2722-272 2730, 2015.
Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monoc- ular reactive mav control. In International Symposhum Experimental Robotics, pp. 3-11. Springer, 2016.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 1054-2063 2018.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Jakob Foerster, Richard Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-leaming awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 122-130. International Founda- tion for Autonomous Agents and Multiagent Systems, 2018
Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin He, Juan P Rodríguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1(2):661-60) 2015.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEBE, 2017.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive the
mapping and planning for visual navigation. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 261 6-2625, 2017a.
Saurabh Gupta, David Fouhey, Sergey Levine, and Jitendra Malik. Unifying map and landmark based representations for visual navigation. arXiv preprint 1711v:1712.08125, 2017b.
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Is multiagent deep reinforcement learn- ing the answer the question? brief survey, arXiv preprint arXiv:1810005587, 2018.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukeuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population- baasd reinforcement learning. Science, 364(6443):859 865. 2019. ISSN 0036-8075. doi: htw.1////cceecc.auu2444..4 URL 364/ 064/6443/855 859

---

Under review as a conference paper at ICLR 2020
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexan- der G Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task comple- tion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6689-6699, 2019.
Arthur Juliani, Vincent-Piert Berges, Esh Vekay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv preprint arXiv: 2018.
Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8. IEEE, 2018.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
Arbaaz Khan, Clark Zhang, Nikolay Atanasov, Konstantinos Karydis, Vijay Kumar, and Daniel D Lec. Memory augmented control networks. arXiv preprint urXiv:1709.05706 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2- thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 2017.
Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Natural language does emerge' naturally' in multi-agent dialog. arXiv preprint arXiv: 770::00000222222 2017.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learn- ing. In Thirty-Firrs AAAI Conference Artificial Intelligence, 2017.
Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Graepel. Emer- 2019.
gent coordination through competition. arXiv preprint
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiive environments. In Advances in Neural Information Processing Systems, pp. 6379-6390 2017.
Andrew Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. icml, volume 30, pp. 3, 2013.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-265) 2605, 2008
Patrick MacAlpine and Peter Stone. Ut austin villa: Robocup 2017 3d simulation league competition
and technical challenges champions. In Robot World Cup, pp. 473-485, Springer, 2017.
Michael Milford, Gordon Wyeth, and David Prasser. Ratslam: a hippocampal model for simulta- neous localization and mapping. In IEEE International Conference Robotics and Automation, 2004. Proceedings. ICRA '04. 2004, volume 1, pp. 403-408. IEEE, 2004.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint (tl:::661.06661 2016.
Piotr Mirowski, Matt Grimes, Mateusz Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, al. Learning navigate in cities without a map. In Advances in Newral Information Processing Systems, pp. 2419-2430, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
12

---

Under review as a conference paper at ICLR 2020
Matej Moravčík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limi no-limit poker. Science, 356(6337):508 -1000855111110 2017.
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprini arXiv: (005.0928,, 2016.
Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In European conference on computer vision, pp- 801-816. Springer, 2016.
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 1133)3877-43, 2005.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning, arXiv preprint vrXiv:1702.006 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W- W, 2017.
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment, arXiv preprint 1e0::01444444444 2017.
Annika Stefanie Reinhold, Juan Ignacio Sanguinetti-Scheck Konstantin Hartmann, and Michael Brecht. Behavioral and neural correlates of hide-and-seck in rats. Science, 365(6458): 180- 1183, 2019.
James Russell, Brioney Gee, and Christina Bullard. Why do young children hide by closing their eyes? self-visibility and the developing concept of self. Journal of Cognition and Development, 13(4):550-576, 2012.
Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. MI- NOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931, 2017.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, al. Habitat: A platform for embodied ai research. arXiv preprint urXiv:1904.011 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Yoav Shoham, Rob Powers, and Trond Grenager. If multi-agent learning is the answer, what is the question? Artificial Intelligence, 171(7):365-377, 2007.
David Silver, Julian Schrittweeerr, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropaga- tion. In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Karl Tuyls and Gerhard Weiss. Multiagent learning: Basics, challenges, and prospects. Ai Magazine, 33(3):4141,, 2012.
Carl Vondrick, Abhinay Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing emerges by colorizing videos. In Proceedings of the European Conference Computer Vision (ECCV), pp. 391-408, 2018.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska. Jack Rae, Piotr Mirowski. Joel Leibo, Adam Santoro, et al. Unsupervised predictive predict memory in goal-directed agent. arXiv preprint arXiv: 1803. 2018.
13

---

Under review as a conference paper at ICLR 2020
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9068-9079, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371-2378. IEEE, 2017a. 7a.
Jingwei Zhang, Lei Tai, Joschka Boedecker, Wolfram Burgard, and Ming Liu. Neural slam: Learn- ing to explore with external memory. arXiv preprint 1706,07706.0999 2017b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv: 0tfi::1411..8555555 2014.
Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In Proceedings of the IEEE International Conference Computer Vision, pp. 483-492, 2017a.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei. and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017b.
14

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
NETWORK ARCHITECTURE
Layer
Kernel Size
Num Outputs
Stride
Padding
Dilation
Activation
Conv1
32
4
0
LeakyReLU
Conv2
64
2
0
LeakyReLU
Conv3
64
1
0
LeakyReLU
FCI
N/A
512
N/A
N/A
N/A
LeakyReLU
Actor-FC2
N/A
512
N/A
N/A
N/A
LeakyReLU
Actor-FC3
N/A
N/A
N/A
N/A
LeakyReL.U
Critic-FC2
N/A
512
N/A
N/A
N/A
LeakyReLU
Critic-FC3
N/A
N/A
N/A
N/A
LeakyReLU
Table 4: Policy Network Architecture We list all the parameters used the policy network.
A.2
MORE VISUALIZATIONS FOR HIDING BEHAVIORS
fasterhider
(b) basic
(c) fasterhider
(d) visibilityreward faster- hider
(e) visibilityreward
faster-
(i) fasterhider
hider
Figure 8: More Visual Demonstrations
15


Reviews:
[{"review_id": "rJxTnqahYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Visual Hide and Seek\n\nIn this paper, the authors propose a two-agent hide and seek environment, and uses reinforcement learning to train the hider. The authors interpret the learned representation by using the learnt features to do classification.\n\nI tend to vote rejection for this paper, mostly due to my feeling that the empirical contribution is interesting, but not novel enough for this conference.\nThere is still a lot of potential improvement available for this project, which is summarized below.\n\nPros:\n- The environment itself, once open-sourced, can be quite valuable to the community.\nI think it is fair to say that the proposed environment in this project is better than the hide-and-seek environment from OpenAI in that it provides the visual input. \nBy the way, I believe OpenAI environment is also partially observable, where unseen agents’ information is masked out.\n- It is an interesting finding on how meaningful features and performance correlate with each other.\n\nCons:\n- Findings in the project are very practical and interesting, but they do not seem to provide valuable information for future research.\nFor example, to improve the quality of the paper, is it possible to utilize the features and performance correlation to improve unsupervised visual feature learning?\nIs it possible to draw a connection to Psychology? Does this “Representation vs. Survival Time” also happens in real-life, or is it just some empirical things that happen in optimization?\n\n- There are a lot more potential to improve the environments. It will greatly increase the impact of the paper by, for example, considering multi-agent training, self-play, unsupervised training.\n\nIn general, I think this project still have a lot of room for fine-tuning.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxrcIXG9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Review ICLR20, Visual Hide and Seek\n\nThis review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.\n\n\n\n## Overall\n\n**Summary**\n\nThe authors introduce a new RL environment and task, \"Visual Hide and Seek\", in which they analyze how the agent's learned visual representations are impacted by its speed, auxiliary rewards, and opponent behavior.\n\n\n**Overall Opinion**\n\nThis paper presents a thorough analysis and great visualizations of agent behaviors and representations under different conditions. I wish more papers would put this much effort into analyzing their agents. I'd highly recommend this paper get accepted since I believe the analysis carried out here and the conclusions reached are quite novel and the paper is overall well-written.\nHowever, at the same time, the work of [Baker et al., 2019][1] was published with significantly more fanfare. I hope their work does not overshadow this one since they are only related in the general task concept.\n\n[1]: https://arxiv.org/abs/1909.07528\n\nSome major issues I had with this work:\n\n- In general, please run more random seeds. Just reporting on a single random seed is not enough, as per [Henderson et al., 2018][2].\n- There are some sections of the paper where the order of paragraphs is confusing. You start the introduction by stating what you've done and letting the reader wonder \"why?\". The explanation is only given in the second paragraph. So I'd suggest rotating the second paragraph upwards before the first. Similarly, at the beginning of section 4, you just mention the results - this should either be shorter (1 sentence, as an overview of the work in this section) or moved to the end of that section.\n- You're missing a section about future work and flaws/problems of your work at the very end (the latter if which should be in \"Discussion\"), which is common to include in ICLR publications.\n\n[2]: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16669/16677\n\nHere are some minor...\n\n## Specific comments and questions\n\n### Abstract\n\nall good\n\n### Intro\n\n- Fig.1: What is that white thing that the seeker/hider have on the capsule?\n\n### Rel. Work\n\nall good\n\n### Method\n\n- What's the speed (FPS) of that Unity engine? Why didn't you use Mujuco/(Py)Bullet/Gym-Miniworld?\n- You need to add some measurements and units: the arena size doesn't have a unit, the size (diameter) of the hiders/seekers is unclear, turning left/right is unclear (how far left/right, after action-repeat)\n- You mention any real-valued position to be valid - so the agents can step into obstacles? And how about through obstacles?\n- \"Affordance Learning\" is usually not used for static environment geometry like obstacles (e.g. [Georgia-Tech course on \"Human-Robot Interaction\"][3])\n- Why 4-layer CNNs? Why not 6 or 8 or a ResNet? Would you think the features would be stronger/weaker in an 8-layer CNN?\n\n[3]: https://www.cc.gatech.edu/~athomaz/classes/CS8803-HRI-Spr08/MayaChandan/Site/Affordance_Learning.html\n\n### Experiments\n\n- 4.1 \"... learned this play game\" -> \"... learned this game\".\n- 4.2 \"mid-level features\" -> what's that? The activation of the convolutional kernels after the second layer CNN? What's the dimension? And why did you pick the 2nd layer, not any of the other 3?\n- Tab.3: This is averaged over how many frames of rollout?\n- \"... case can moves a lot faster.\" -> \"... case can move a lot faster\".\n- Fig.2 is very interesting. Well done.\n- Fig.3: the font is not consistent with other figures\n- Fig.4: remove the blueish background to increase contrast. Increase the font size of the ticks on the left. Make the legend color boxes slightly bigger. Add more space or a visual divider between the different states - especially on the right side it's hard to make out where one stops and the next begins.\n- Fig.4/5: This analysis is lovely and we need more of this in DRL.\n- 4.4 in the text, you sometimes write \"not S\" and sometimes \"¬S\". Please change the \"not s\"\n- Fig.5: (suggestion) Merge/sum the 2 columns (in both A/B merge the left and right plot into one by summing); subtract the random policy values as you did with Fig.4.\n- \"We summarize representative cases, and put the full results for all combinations in the Appendix\" - no you didn't.\n- Fig.6: What is going on in the left third of this diagram? What is this colorful mush? If this is by any chance indicating a change over time, do you maybe want to spread a single, very colorful plot of distance over time into multiple less colorful plots? At least add a legend, please. Also, I'd recommend smoothing (moving average or smoothing spline). \n- Also maybe add reward over time plots, as is common in DRL, to show that your policies converged after 8 mil. steps.\n\n### Conclusion\n\nAll good, save for the missing future work and critical analysis of your work.\n\n### Appendix\n\nall good", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJezg2rAuB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nPaper Summary: The paper studies what an embodied agent that is trained using RL learns in the context of the hide and seek game. There are two agents in a simple environment (one hider and one seeker). The strategy for the seeker is fixed, but the hider learns a policy based on ego-centric visual input. The paper attempts to analyze how the learned representation differs with different capabilities of the agents or environment structure.\n\nOriginality:\n\nThe study of the learned representations in the context of hide and seek is new. There is concurrent work on hide and seek (e.g., Baker et al., 2019), but unlike this paper, they have access to groundtruth location information of the agents and it is not based on visual input.\n\nQuality:\n\n- The conclusions of the paper are either counter-intuitive or are based on some hypotheses and guesses.\n\nFor example, it is strange that \"visibilityreward\" performs worst for \"Awareness of Self-Visibility\". It is exactly trained for that task.\n\nThere are conclusions about \"temporal events\" (e.g., the hider learns to first turn away from the seeker then run away) just based on a single frequency number (Figure 4). There are many other possibilities that can result in the same frequency. If the authors draw conclusions about temporal events, they should show how values change over time. Single frequency numbers cannot be used for such conclusions.\n\n- To check the learned representation, the common practice is to use features on a very different task. However, this paper addresses only two tasks \"Seeker Recognition\" and \"Awareness of Self-Visibility\" which are very close to the original task. One of the models even receives explicit rewards for the latter case.\n\n- No standard deviation is reported for the results. Given the random nature of RL algorithms, standard deviations should be reported.\n\nSignificance:\n\nI expected a more rigorous analysis since this is an analysis paper and there is no new methodology in the paper. Conclusions that are just based on a guess about some qualitative result or based on two state changes as in Figure 5 are groundless. More thorough analysis based on quantitative results are required to justify the claims and provide generalizable conclusions.\n\nClarity:\n\nThe paper is mostly clear, but there is some missing information:\n\n- In Figure 3, which dot corresponds to which model? It is claimed that \"When the model has a weakness, the model learned to overcome it by instead learning better features\". To justify that, we need to know what the dots are.\n\n- What do \"randominits\" in Figure 5 correspond to?\n\n- What is red, yellow, blue and green plots in Figure 6? What is the standard deviation? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a technique for training embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator. The model is trained to play this game from scratch without any prior knowledge of its visual world, and experiments and visualizations show that a representation of other agents automatically emerges in the learned representation. Results suggest that, although agent weaknesses make the learning problem more challenging, they also cause useful features to emerge in the representation.

While reviewers found the paper explores an interesting direction, concerns were raised that many claims are unjustified. For example, in the discussion phase a reviewer asked how can one infer "hider learns to first turn away from the seeker then run away" from a single transition frequency? Or, the rebuttal mentions "The agent with visibility reward does not get the chance to learn features of self-visibility because of the limited speed hence the model received samples with significantly less variation of its self-visibility, which makes learning to discriminate self-visibility difficult". What is the justification for this? There could be more details in the paper and I'd also like to know if these findings were reached purely by looking at the histograms or by combining visual analysis with the histograms.

I suggest authors address these concerns and provide quantitative results for all of the claims in an improved iteration of this paper. 
Decision: Reject