Visual Hide and Seek

We train embodied agents to play Visual Hide and Seek where a prey must navigate in a simulated environment in order to avoid capture from a predator . We place a variety of obstacles in the environment for the prey to hide behind , and we only give the agents partial observations of their environment using an egocentric perspective . Although we train the model to play this game from scratch , experiments and visualizations suggest that the agent learns to predict its own visibility in the environment . Furthermore , we quantitatively analyze how agent weaknesses , such as slower speed , effect the learned policy . Our results suggest that , although agent weaknesses make the learning problem more challenging , they also cause more useful features to be learned . 1 INTRODUCTION . We introduce the task of Visual Hide and Seek where a neural network must learn to steer an embodied agent around its environment in order to avoid capture from a predator . Our hypothesis , which our experiments suggest , is that learning to play this game will cause useful representations of multi-agent dynamics emerge . We train the agent to navigate through its environment to maximize its survival time , which the model successfully learn to do . However , since we train the model from scratch to directly map pixels to action , the network can learn a representation of the environment and its dynamics . Figure 1 illustrates the problem setup . We designed this game to mimic the typical dynamics between predator and prey . For example , we place a variety of obstacles inside the environment , which create occlusions that the agent can leverage to hide behind . We also only give the agents access to the first-person perspective of their three-dimensional environment . Consequently , this task is a substantial challenge for reinforcement learning because the state is both visual ( pixel input ) and partially observable ( due to occlusions ) . To carefully study the emerged representations and agent dynamics , we probe different environmental factors as well as the agents ’ abilities , and our experiments quantitatively suggest that the agent automatically learns to recognize the perspective of the predator and its own visibility , which enables robust hiding behavior . Hider SeekerVisual Observation Visual Observation However , what intrinsic structure in the game , if any , caused this strategy to emerge ? We quantitatively compare a spectrum of hide and seek games where we perturb the abilities of agents and the environmental complexity . Our experiments show that , although agent weaknesses , such as slower speed , make the learning problem more challenging , they also cause the model to learn more useful representations of its environmental dynamics . We show there is a “ sweet spot ” where the weakness is strong enough to cause useful strategies to emerge without derailing the learning process . This paper makes three principal contributions to embodied agents . Firstly , we introduce the problem of visual hide-and-seek where an agent receives a partial observation of its visual environment and must navigate to avoid capture . Secondly , we empirically demonstrate that this task causes representations of other agents in the scene to emerge . Thirdly , we analyze the underlying reasons why these representations emerge , and show they are due to imperfections in the agent ’ s abilities . The rest of this paper analyzes these contributions in detail . We plan to release all software , data , environments , and models publicly to promote further progress on this problem . 2 RELATED WORK . Our work builds on research for embodied agents that learn to navigate and manipulate environments . Embodied agents with extensive training experience are increasingly able to solve a large number of problems across manipulation , navigation , and game-playing tasks ( Mnih et al. , 2015 ; Gu et al. , 2017 ; Zhu et al. , 2017b ; Silver et al. , 2017 ; Kahn et al. , 2018 ; Kalashnikov et al. , 2018 ; Mirowski et al. , 2018 ) . Extensive work has demonstrated that , after learning with indirect supervision from a reward function , rich representations for their task automatically emerge ( Bansal et al. , 2017 ; Lowe et al. , 2017 ; Liu et al. , 2019 ; Jaderberg et al. , 2019 ) . Several recent works have created 3D embodiment simulation environment ( Kolve et al. , 2017 ; Brodeur et al. , 2017 ; Savva et al. , 2017 ; Das et al. , 2018 ; Xia et al. , 2018 ; Savva et al. , 2019 ) for navigation and visual question answering tasks . To train these models , visual navigation is often framed as a reinforcement learning problem ( Chen et al. , 2015 ; Giusti et al. , 2015 ; Oh et al. , 2016 ; Abel et al. , 2016 ; Bhatti et al. , 2016 ; Daftry et al. , 2016 ; Mirowski et al. , 2016 ; Brahmbhatt & Hays , 2017 ; Zhang et al. , 2017a ; Gupta et al. , 2017a ; Zhu et al. , 2017a ; Gupta et al. , 2017b ; Kahn et al. , 2018 ) . Moreover , by incorporating multiple embodied agents into the environment , past work has explored how to learn diverse strategies and behaviors in multi-agent visual navigation tasks ( Jaderberg et al. , 2019 ; Jain et al. , 2019 ) . For a full review of multi-agent reinforcement learning , please see ( Panait & Luke , 2005 ; Bu et al. , 2008 ; Tuyls & Weiss , 2012 ; Shoham et al. , 2007 ; Hernandez-Leal et al. , 2018 ) . Our paper contributes to a rapidly growing area to learn representations from indirect supervision . Early work has studied how features automatically emerge in convolutional networks for image recognition ( Zhou et al. , 2014 ; Zeiler & Fergus , 2014 ) . Since direct supervision is often expensive to collect , there has been substantial work in learning emergent representations across vision ( Doersch et al. , 2015 ; Vondrick et al. , 2018 ) , language ( Kottur et al. , 2017 ; Radford et al. , 2017 ) , sound ( Owens et al. , 2016 ; Aytar et al. , 2016 ) , and interaction ( Aytar et al. , 2018 ; Burda et al. , 2018 ) . We also study how representations emerge . However we investigate the emergent dynamics in the two-player game of visual hide and seek . We characterize why representations emerge , which we believe can refine the field ’ s understanding of self-supervised learning dynamics . This paper is concurrent to ( Baker et al. , 2019 ) , and we urge readers to watch their impressive results on learning to play hide and seek games . However , there are a few key differences between the two papers that we wish to highlight . Firstly , in contrast to ( Baker et al. , 2019 ) , we focus on hide and seek in partially observable environments during both training and testing . Our environment is three-dimensional , and agents only receive an egocentric two-dimensional visual input , which creates situations abundant with occlusions . Secondly , the input to our model is a visual scene , and not the state of a game engine . The learning problem is consequently very challenging because the model must learn perceptual representations in addition to its policy . Our experiments suggest this happens , but the richness of the visual representation depends on the impediments to the model . Finally , we focus our investigation on analyzing the underlying reasons why different behaviors emerge , which we believe will refine the field ’ s insight into self-supervised learning approaches . 3 HIDE AND SEEK . We first present our environment and learning problem , then describe our interventions to understand the cause of different emergent behaviors . 3.1 ENVIRONMENT AND LEARNING . We created a 3D simulation for hide-and-seek using the Unity game engine , which we use throughout this paper . There are two agents in this game : the hider and the seeker . Each agent receives only a first-person visual observation of the environment with 120-degree field of view , and navigates around the environment by selecting actions from a discrete set ( move forward , move backward , turn left , turn right , stand still ) . The environment is a square of 14 unit x 14 unit , and any real value position inside the square without being occupied any obstacle is a valid game state . The speed of the hider is two units per second while the speed of the seeker is one and a half units per second . Our simulation runs in 50 frame per second . The agents can turn 3.3 degree per frame . Each agent has a diameter of one unit . In contrast to other multiplayer games ( Sukhbaatar et al. , 2016 ; Moravčı́k et al. , 2017 ; Lample & Chaplot , 2017 ; MacAlpine & Stone , 2017 ; Lowe et al. , 2017 ; Foerster et al. , 2018 ) , the egocentric perspective of our problem setup makes this task very challenging . We place obstacles throughout the environment to create opportunities for occlusion . Seeker Policy : We use a simple deterministic policy for the seeker , which is as follows . If the seeker can see the hider , move towards it . If the seeker can not see the hider , move towards the hider ’ s last known position . If the seeker still can not find the hider , then it will explore the environment by waypoints in a round-robin fashion . The game episode concludes when the seeker successfully “ catches ” the hider . We define catching the hider as a collision between the two agents . Hider Policy : Given the environment and the seeker algorithm , we train a policy for the agent to hide . Given a first-person observation of the agent ’ s environment , we estimate the action to execute with a convolutional neural network . We train the model to maximize its reward function using PPO ( Schulman et al. , 2017 ) . Each time step , we provide a small reward ( +0.001 ) for each living step , and a large negative reward ( −1 ) once captured . We call this model basic . Starting Positions : The starting positions of the hider and the seeker are randomly sampled on a continuous grid for each episode . We also specify their initial orientations so that they always start by looking towards each other ( even if there is an obstacle between them ) . Our intention is that , by learning to hide , the model will learn features for the recognition of objects in its visual field , such as obstacles and other agents . To analyze this , we fit a linear regression from the learned features to classify different objects and game states . We encode object and game states as discrete variables , and use the classification accuracy as a measure of how well the features are encoding different states . Our experiments will use this approach to probe the learned features . 3.2 SYSTEMATIC INTERVENTIONS . We systematically intervene on the learning process to understand the mechanisms behind emergent features . We chose these interventions because they either give the agent an advantage or weakness . We summarize these variations in Table 1 . By training the models from scratch with different interventions and analyzing the learned representation , we can analyze how environmental features cause different strategies to be learned . Speed : The first intervention we will make is manipulating the relative speed of the hider and seeker . We explore two variations . In the first variant , we slow down the hider . Likewise , in the second variant , we speed up the hider . We call these agents slowerhider and fasterhider respectively . The conditions are otherwise the same as the basic model . Self-Visibility : Awareness of your own visibility is a crucial feature for successfully surviving during hide and seek ( Russell et al. , 2012 ; Reinhold et al. , 2019 ) . We explicitly incorporate this into the reward function as an auxiliary dense reward signal . Along with the sparse reward above , this agent receives an additional reward of 0.001 if the seeker does not see it at the current time step . Likewise , the agent receives a punitive reward of−0.001 if it is currently visible by the seeker . We refer to this agent as visibilityreward . Furthermore , we also use a variant where the hider has a faster speed , which we call visibilityreward+faster . Environmental Complexity : We also intervene on the complexity of the environment , which lets us analyze the impact of the environment on the emergent features . We use two variations . Firstly , we use a stochastic policy for the seeker agent ( instead of deterministic ) . Specifically , the stochastic seeker randomly visits locations in the map until the hider is within its field of view , at which point it immediately goes towards it . We name this variation stochasticseeker . Secondly , we use a stochastic map where we randomly select a number of objects and also randomly position throughout the environment . Consequently , the difficulty of the maps will change between easy ( many occlusions ) to difficult ( few to none occlusions ) . We name this variation stochasticmaps .

Summary: Paper Summary: The paper studies what an embodied agent that is trained using RL learns in the context of the hide and seek game. There are two agents in a simple environment (one hider and one seeker). The strategy for the seeker is fixed, but the hider learns a policy based on ego-centric visual input. The paper attempts to analyze how the learned representation differs with different capabilities of the agents or environment structure.