Learning Generalizable Visual Representations via Interactive Gameplay

1 INTRODUCTION . We are interested in studying what facets of their environment artificial agents learn to represent through interaction and gameplay . We study this question within the context of hide-and-seek , for which proficiency requires an ability to navigate around in an environment and manipulate objects as well as an understanding of visual relationships , object affordances , and perspective . Inspired by behavior observed in juvenile ravens ( Burghardt , 2005 ) , we focus on a variant of hide-and-seek called cache in which agents hide objects instead of themselves . Advances in deep reinforcement learning have shown that , in abstract games ( e.g . Go and Chess ) and visually simplistic environments ( e.g . Atari and grid-worlds ) with limited interaction , artificial agents exhibit surprising emergent behaviours that enable proficient gameplay ( Mnih et al. , 2015 ; Silver et al. , 2017 ) ; indeed , recent work ( Chen et al. , 2019 ; Baker et al. , 2020 ) has shown this in the context of hiding games . Our interest , however , is in understanding how agents learn to represent their visual environment , through gameplay that requires varied interaction , in a high-fidelity environment grounded in the real world . This requires a fundamental shift away from existing popular environments and a rethinking of how the capabilities of artificial agents are evaluated . Our agents must first be embodied within an environment allowing for diverse interaction and providing rich visual output . For this we leverage AI2-THOR ( Kolve et al. , 2017 ) , a near photo-realistic , interactive , simulated , 3D environment of indoor living spaces , see Fig . 1a . Our agents are parameterized using deep neural networks , and trained adversarially using the paradigms of reinforce- ment ( Mnih et al. , 2016 ) and self-supervised learning . After our agents are trained to play cache , we then probe how they have learned to represent their environment . To this end we distinguish two distinct categories of representations generated by our agents . The first , static image representations ( SIRs ) , correspond to the output of a CNN applied to the agent ’ s egocentric visual input . The second , dynamic image representations ( DIRs ) , correspond to the output of the agents ’ RNN . While SIRs are timeless , operating only on single images , DIRs have the capacity to incorporate the agent ’ s previous actions and observations . Representation learning within the computer vision community is largely focused on developing SIRs whose quality is measured by their utility in downstream tasks ( e.g . classification , depthprediction , etc ) ( Zamir et al. , 2018 ) . Our first set of experiments show that our agents develop lowlevel visual understanding of individual images measured by their capacity to perform a collection of standard tasks from the computer vision literature , these tasks include pixel-to-pixel depth ( Saxena et al. , 2006 ) and surface normal ( Fouhey et al. , 2013 ) prediction , from a single image . While SIRs are clearly an important facet of representation learning , they are also definitionally unable to represent an environment as a whole : without the ability to integrate observations through time , a representation can only ever capture a single snapshot of space and time . To represent an environment holistically , we require DIRs . Unlike for SIRs , we are unaware of any well-established benchmarks for DIRs . In order to investigate what has been learned by our agent ’ s DIRs we develop a suite of experiments loosely inspired by experiments performed on infants and young children . These experiments then demonstrate our agents ’ ability to integrate observations through time and understand spatial relationships between objects ( Casasola et al. , 2003 ) , occlusion ( Hespos et al. , 2009 ) , object permanence ( Piaget , 1954 ) , and seriation ( Piaget , 1954 ) of free space . It is important to stress that this work focuses on studying how play and interaction contribute to representation learning in artificial agents and not on developing a new , state-of-the-art , methodology for representation learning . Nevertheless , to better situate our results in context of existing work , we provide strong baselines in our experiments , e.g . in our low-level vision experiments we compare against a fully supervised model trained on ImageNet ( Deng et al. , 2009 ) . Our results provide compelling evidence that : ( a ) on a suite of low level computer vision tasks within AI2-THOR , static representations learned by playing cache perform very competitively ( and often outperform ) strong unsupervised and fully supervised methods , ( b ) these static representations , trained using only synthetic images , obtain non-trivial transfer to downstream tasks using real-world images , ( c ) unlike representations learned from datasets of single images , agents trained via embodied gameplay learn to integrate visual information through time , demonstrating an elementary understanding of free space , objects and their relationships , and ( d ) embodied gameplay provides a natural means by which to generate rich experiences for representation learning beyond random sampling and relatively simpler tasks like visual navigation . In summary , we highlight the contributions : ( 1 ) Cache – we introduce cache within the AI2-THOR environment , an adversarial game which permits the study of representation learning in the context of interactive , visual , gameplay . ( 2 ) Cache agent – training agents to play Cache is non-trivial . We produce a strong Cache agent which integrates several methodological novelties ( see , e.g. , perspective simulation and visual dynamics replay in Sec . 4 ) and even outperforms humans at hiding on training scenes ( see Sec . 5 ) . ( 3 ) Static and dynamic representation study – we provide comprehensive evaluations of how our Cache agent has learned to represent its environment providing insight into the advantages and current limitations of interactive-gameplay-based representation learning . 2 RELATED WORK . Deep reinforcement learning for games . As games provide an interactive environment that enable agents to receive observations , take actions , and receive rewards , they are a popular testbed for RL algorithms . Reinforcement learning has been studied in the context of numerous single , and multi , agent games such as Atari Breakout ( Mnih et al. , 2015 ) , VizDoom ( Lample & Chaplot , 2017 ) , Go ( Silver et al. , 2017 ) , StarCraft ( Vinyals et al. , 2019 ) , Dota ( Berner et al. , 2019 ) and Hide-and-Seek ( Baker et al. , 2020 ) . The goal of these works is proficiency : to create an agent which can achieve ( super- ) human performance with respect to the game ’ s success criteria . In contrast , our goal is to understand how an agent has learned to represent its environment through gameplay and to show that such an agent ’ s representations can be employed for downstream tasks beyond gameplay . Passive representation learning . There is a large body of recent works that address the problem of representation learning from static images or videos . Image colorization ( Zhang et al. , 2016 ) , egomotion estimation ( Agrawal et al. , 2015 ) , predicting image rotation ( Gidaris et al. , 2018 ) , context prediction ( Doersch et al. , 2015 ) , future frame prediction ( Vondrick et al. , 2016 ) and more recent contrastive learning based approaches ( He et al. , 2020 ; Chen et al. , 2020b ) are among the successful examples of passive representation learning . We refer to them as passive since the representations are learned from a fixed set of images or videos . Our approach in contrast is interactive in that the images observed during learning are decided by the actions of the agent . Interactive representation learning . Learning representations in dynamic and interactive environments has been addressed in the literature as well . In the following we mention a few examples . Burda et al . ( 2019 ) explore curiosity-driven representation learning from a suite of games . Anand et al . ( 2019 ) address representation learning of the latent factors used for generating the interactive environment . Zhan et al . ( 2018 ) learn to improve exploration in video games by predicting the memory state . Ghosh et al . ( 2019 ) learn a functional representation for decision making as opposed to a representation for the observation space only . Whitney et al . ( 2020 ) simultaneously learn embeddings of state and action sequences . Jonschkowski & Brock ( 2015 ) learn a representation by measuring inconsistencies with a set of pre-defined priors . Pinto et al . ( 2016 ) learn visual representations by pushing and grasping objects using a robotics arm . The representations learned using these approaches are typically tested on tasks akin to tasks they were trained with ( e.g. , a different level of the same game ) . In contrast , we investigate whether cognitive primitives such as depth estimation and object permanence maybe be learned via gameplay in a visual dynamic environment . 3 PLAYING CACHE IN SIMULATION . We situate our agents within AI2-THOR , a simulated 3D environment of indoor living spaces within which multiple agents can navigate around and interact with objects ( e.g . by picking up , placing , opening , closing , cutting , switching on , etc. ) . In past works , AI2-THOR has been leveraged to teach agents to interact with their world ( Zhu et al. , 2017 ; Hu et al. , 2018 ; Huang et al. , 2019 ; Gan et al. , 2020b ; Wortsman et al. , 2019 ; Gordon et al. , 2017 ) , interact with each other ( Jain et al. , 2019 ; 2020 ) as well as learn from these interactions ( Nagarajan & Grauman , 2020 ; Lohmann et al. , 2020 ) . AI2-THOR contains a total of 150 unique scenes equally distributed into five scene types : kitchens , living rooms , bedrooms , bathrooms , and foyers . We train our cache agents on a subset of the kitchen Published as a conference paper at ICLR 2021 Inp t L A & R c NN A . E D C . D c D A M Me c Ma Inp t L A & R c NN A . E D C . D c D A M Me c Ma Inp t L A & R c NN A . E D C . D c D A M Me c Ma Hide cup gt < latexit sha1_base64= '' +0+K1AhyGutp5i38YYIOTiq/JMQ= '' > AAAB6nicbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2ge0oUymk3ToZBJmboQS+gluXCji1i9y5984SYOo9cCFwzn3cu89Xiy4Btv+tEpLyyura+X1ysbm1vZOdXevo6NEUdamkYhUzyOaCS5ZGzgI1osVI6EnWNebXGd+94EpzSN5D9OYuSEJJPc5JWCku2AIw2rNrts58CJxClJDBVrD6sdgFNEkZBKoIFr3HTsGNyUKOBVsVhkkmsWETkjA+oZKEjLtpvmpM3xklBH2I2VKAs7VnxMpCbWehp7pDAmM9V8vE//z+gn4F27KZZwAk3S+yE8Ehghnf+MRV4yCmBpCqOLmVkzHRBEKJp1KHsJlhrPvlxdJ56TunNYbt41a86qIo4wO0CE6Rg46R010g1qojSgK0CN6Ri+WsJ6sV+tt3lqyipl99AvW+xdwcI4K < /latexit > SIR 128x7x7 st < latexit sha1_base64= '' uwrevV+/ludXMHIgESK4k17/zD4= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOz3AQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wKCuI4W < /latexit > it < latexit sha1_base64= '' 49jzbtNkZShn+iIjbBN4ZQyo2i4= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOz7AQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wJzfI4M < /latexit > ot < latexit sha1_base64= '' ws3F94vsh/kOa4LUOCliZNGhMaM= '' > AAAB6nicbVDLSgNBEJyNrxhfUY9eBoPgKWw0+LgFvXiMaB6QLGF2MpsMmZ1ZZnqFsOQTvHhQxKtf5M2/cXaziBoLGoqqbrq7/EhwA6776RSWlldW14rrpY3Nre2d8u5e26hYU9aiSijd9YlhgkvWAg6CdSPNSOgL1vEn16nfeWDacCXvYRoxLyQjyQNOCVjpTg1gUK64VTcDXiS1nFRQjuag/NEfKhqHTAIVxJhezY3AS4gGTgWblfqxYRGhEzJiPUslCZnxkuzUGT6yyhAHStuSgDP150RCQmOmoW87QwJj89dLxf+8XgzBhZdwGcXAJJ0vCmKBQeH0bzzkmlEQU0sI1dzeiumYaELBplPKQrhMcfb98iJpn1Rrp9X6bb3SuMrjKKIDdIiOUQ2dowa6QU3UQhSN0CN6Ri+OcJ6cV+dt3lpw8pl99AvO+xd8oI4S < /latexit > ht 1 < latexit sha1_base64= '' w0z1MIqjcwqqrZvbS0vyNwXmoPE= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtcf9BE+cWb9csat2BrJInJxUIEejX/7oDUIW+zxAJqnWXceO0E2oQsEkn5V6seYRZRM64l1DA+pz7SbZuTNyZJQBGYbKVIAkU39OJNTXeup7ptOnONZ/vVT8z+vGOLx0ExFEMfKAzRcNY0kwJOnvZCAUZyinhlCmhLmVsDFVlKFJqJSFcJXi/PvlRdI6rTpn1dpdrVK/zuMowgEcwjE4cAF1uIUGNIHBBB7hGV6syHqyXq23eWvBymf24Res9y8TFo+J < /latexit > DIR 257x7x7 h̃t < latexit sha1_base64= '' /FD56DYReBR0BgD9sJp/o5zbwTA= '' > AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwdpCGspms2mXbrJhdyKU0J/hxYMiXv013vw3btIgan0w8Hhvhpl5fiK4Btv+tCpLyyura9X12sbm1vZOfXfvXstUUdalUkjV94lmgsesCxwE6yeKkcgXrOdPrnO/98CU5jK+g2nCvIiMYh5ySsBI7gC4CFg2ng1hWG/YTbsAXiROSRqoRGdY/xgEkqYRi4EKorXr2Al4GVHAqWCz2iDVLCF0QkbMNTQmEdNeVpw8w0dGCXAolakYcKH+nMhIpPU08k1nRGCs/3q5+J/nphBeeBmPkxRYTOeLwlRgkDj/HwdcMQpiagihiptbMR0TRSiYlGpFCJc5zr5fXiT3J03ntNm6bTXaV2UcVXSADtExctA5aqMb1EFdRJFEj+gZvVhgPVmv1tu8tWKVM/voF6z3L9XnkcE= < /latexit > ht 1 < latexit sha1_base64= '' w0z1MIqjcwqqrZvbS0vyNwXmoPE= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtcf9BE+cWb9csat2BrJInJxUIEejX/7oDUIW+zxAJqnWXceO0E2oQsEkn5V6seYRZRM64l1DA+pz7SbZuTNyZJQBGYbKVIAkU39OJNTXeup7ptOnONZ/vVT8z+vGOLx0ExFEMfKAzRcNY0kwJOnvZCAUZyinhlCmhLmVsDFVlKFJqJSFcJXi/PvlRdI6rTpn1dpdrVK/zuMowgEcwjE4cAF1uIUGNIHBBB7hGV6syHqyXq23eWvBymf24Res9y8TFo+J < /latexit > Inp t L A & R c NN A . E D C . D c D A M Me c Ma Metric map mt < latexit sha1_base64= '' hSQoieJvwDg/ssTgCtOF9DX6PMY= '' > AAAB6nicbVDLSgNBEJyNrxhfUY9eBoPgKWw0+LgFvXiMaB6QLGF2MpsMmZldZnqFsOQTvHhQxKtf5M2/cXaziBoLGoqqbrq7/EhwA6776RSWlldW14rrpY3Nre2d8u5e24SxpqxFQxHqrk8ME1yxFnAQrBtpRqQvWMefXKd+54Fpw0N1D9OIeZKMFA84JWClOzmAQbniVt0MeJHUclJBOZqD8kd/GNJYMgVUEGN6NTcCLyEaOBVsVurHhkWETsiI9SxVRDLjJdmpM3xklSEOQm1LAc7UnxMJkcZMpW87JYGx+eul4n9eL4bgwku4imJgis4XBbHAEOL0bzzkmlEQU0sI1dzeiumYaELBplPKQrhMcfb98iJpn1Rrp9X6bb3SuMrjKKIDdIiOUQ2dowa6QU3UQhSN0CN6Ri+OcJ6cV+dt3lpw8pl99AvO+xd5lI4Q < /latexit > Inp t L A & R c NN A . E D C . D c D A M Me c Ma Mental rollout M l < latexit sha1_base64= '' Xv1qiawUvtLxc2P1nxc4r9LTkLo= '' > AAAB6nicbVDLSsNAFL2pr1pfVZduBovgqiRafOyKbtwIFe0D2lgm00k7dDIJMxOhhH6CGxeKuPWL3Pk3TtIgaj1w4XDOvdx7jxdxprRtf1qFhcWl5ZXiamltfWNzq7y901JhLAltkpCHsuNhRTkTtKmZ5rQTSYoDj9O2N75M/fYDlYqF4k5PIuoGeCiYzwjWRrq9vuf9csWu2hnQPHFyUoEcjX75ozcISRxQoQnHSnUdO9JugqVmhNNpqRcrGmEyxkPaNVTggCo3yU6dogOjDJAfSlNCo0z9OZHgQKlJ4JnOAOuR+uul4n9eN9b+mZswEcWaCjJb5Mcc6RClf6MBk5RoPjEEE8nMrYiMsMREm3RKWQjnKU6+X54nraOqc1yt3dQq9Ys8jiLswT4cggOnUIcraEATCAzhEZ7hxeLWk/Vqvc1aC1Y+swu/YL1/ATsvjec= < /latexit > hct < latexit sha1_base64= '' 377/FCzba4IWjxe7BzJkwcV+C/I= '' > AAAB7HicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwbSFNpbNdtMu3WzC7kQoob/BiwdFvPqDvPlv3KRB1Ppg4PHeDDPz/FhwDbb9aZWWlldW18rrlY3Nre2d6u5eW0eJosylkYhU1yeaCS6ZCxwE68aKkdAXrONPrjO/88CU5pG8g2nMvJCMJA84JWAkd3xPBzCo1uy6nQMvEqcgNVSgNah+9IcRTUImgQqidc+xY/BSooBTwWaVfqJZTOiEjFjPUElCpr00P3aGj4wyxEGkTEnAufpzIiWh1tPQN50hgbH+62Xif14vgeDCS7mME2CSzhcFicAQ4exzPOSKURBTQwhV3NyK6ZgoQsHkU8lDuMxw9v3yImmf1J3TeuO2UWteFXGU0QE6RMfIQeeoiW5QC7mIIo4e0TN6saT1ZL1ab/PWklXM7KNfsN6/AOMLjuA= < /latexit > hot < latexit sha1_base64= '' aAOHlQFvdNC8xtzT88UW8qr16Eo= '' > AAAB7HicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwbSFNpbNdtMu3WzC7kQoob/BiwdFvPqDvPlv3KRB1Ppg4PHeDDPz/FhwDbb9aZWWlldW18rrlY3Nre2d6u5eW0eJosylkYhU1yeaCS6ZCxwE68aKkdAXrONPrjO/88CU5pG8g2nMvJCMJA84JWAkd3wfDWBQrdl1OwdeJE5BaqhAa1D96A8jmoRMAhVE655jx+ClRAGngs0q/USzmNAJGbGeoZKETHtpfuwMHxlliINImZKAc/XnREpCraehbzpDAmP918vE/7xeAsGFl3IZJ8AknS8KEoEhwtnneMgVoyCmhhCquLkV0zFRhILJp5KHcJnh7PvlRdI+qTun9cZto9a8KuIoowN0iI6Rg85RE92gFnIRRRw9omf0YknryXq13uatJauY2Ue/YL1/AfVTjuw= < /latexit > ⇡X✓ < latexit sha1_base64= '' suTQFUFY1lP6zw363skjiclUM5s= '' > AAAB83icbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2Ac0sUymk3boZDLM3Agl9DfcuFDErT/jzr8xSYOo9cCFwzn3cu89vhLcgG1/WqWl5ZXVtfJ6ZWNza3unurvXMVGsKWvTSES65xPDBJesDRwE6ynNSOgL1vUn15nffWDa8EjewVQxLyQjyQNOCaSS6yo+cGHMgNz3BtWaXbdz4EXiFKSGCrQG1Q93GNE4ZBKoIMb0HVuBlxANnAo2q7ixYYrQCRmxfkolCZnxkvzmGT5KlSEOIp2WBJyrPycSEhozDf20MyQwNn+9TPzP68cQXHgJlyoGJul8URALDBHOAsBDrhkFMU0JoZqnt2I6JppQSGOq5CFcZjj7fnmRdE7qzmm9cduoNa+KOMroAB2iY+Sgc9REN6iF2ogihR7RM3qxYuvJerXe5q0lq5jZR79gvX8BJ3WR5g== < /latexit > vX✓ < latexit sha1_base64= '' GdMYSlPeVUO3HzqoaS+rmkTfYe0= '' > AAAB8XicbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2Ae2sUymk3boZBJmbgol9C/cuFDErX/jzr9xkgZR64ELh3Pu5d57vEhwDbb9aRWWlldW14rrpY3Nre2d8u5eS4exoqxJQxGqjkc0E1yyJnAQrBMpRgJPsLY3vk799oQpzUN5B9OIuQEZSu5zSsBI95N+D0YMyEOnX67YVTsDXiROTiooR6Nf/ugNQhoHTAIVROuuY0fgJkQBp4LNSr1Ys4jQMRmyrqGSBEy7SXbxDB8ZZYD9UJmSgDP150RCAq2ngWc6AwIj/ddLxf+8bgz+hZtwGcXAJJ0v8mOBIcTp+3jAFaMgpoYQqri5FdMRUYSCCamUhXCZ4uz75UXSOqk6p9Xaba1Sv8rjKKIDdIiOkYPOUR3doAZqIookekTP6MXS1pP1ar3NWwtWPrOPfsF6/wK0DpET < /latexit > at < latexit sha1_base64= '' YaG+GfOaZFAdbmyIZ6rpWb7fQcE= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOzLAQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wJnTI4E < /latexit > it 1 < latexit sha1_base64= '' I57o8i9M8qP7Gcb8lapa+6gjDYU= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtUU/wRNn1i9X7KqdgSwSJycVyNHolz96g5DFPg+QSap117EjdBOqUDDJZ6VerHlE2YSOeNfQgPpcu0l27owcGWVAhqEyFSDJ1J8TCfW1nvqe6fQpjvVfLxX/87oxDi/dRARRjDxg80XDWBIMSfo7GQjFGcqpIZQpYW4lbEwVZWgSKmUhXKU4/355kbROq85ZtXZXq9Sv8ziKcACHcAwOXEAdbqEBTWAwgUd4hhcrsp6sV+tt3lqw8pl9+AXr/QsUoI+K < /latexit > at 1 < latexit sha1_base64= '' 8N4WOjgCSh1etWpJRyuHRbcdus4= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtWk/wRNn1i9X7KqdgSwSJycVyNHolz96g5DFPg+QSap117EjdBOqUDDJZ6VerHlE2YSOeNfQgPpcu0l27owcGWVAhqEyFSDJ1J8TCfW1nvqe6fQpjvVfLxX/87oxDi/dRARRjDxg80XDWBIMSfo7GQjFGcqpIZQpYW4lbEwVZWgSKmUhXKU4/355kbROq85ZtXZXq9Sv8ziKcACHcAwOXEAdbqEBTWAwgUd4hhcrsp6sV+tt3lqw8pl9+AXr/QsIUI+C < /latexit > 512 512 Visual Dynamics Replay Self-supervised Tasks Hiding Locationscnn✓ < latexit sha1_base64= '' JLXS1g9bnxHVObe2qxmsdrRe0vE= '' > AAAB/HicbVBNS8NAEN3Ur1q/qj16CRbBU0m0+HErevFYwX5AE8pmO2mXbjZhdyKWUP+KFw+KePWHePPfmLRB1Ppg4PHeDDPzvEhwjZb1aRSWlldW14rrpY3Nre2d8u5eW4exYtBioQhV16MaBJfQQo4CupECGngCOt74KvM7d6A0D+UtTiJwAzqU3OeMYir1yxUH4R41S5iU076DI0DaL1etmjWDuUjsnFRJjma//OEMQhYHIJEJqnXPtiJ0E6qQMwHTkhNriCgb0yH0UippANpNZsdPzcNUGZh+qNKSaM7UnxMJDbSeBF7aGVAc6b9eJv7n9WL0z92EyyhGkGy+yI+FiaGZJWEOuAKGYpISyhRPbzXZiCrKMM2rNAvhIsPp98uLpH1cs09q9Zt6tXGZx1Ek++SAHBGbnJEGuSZN0iKMTMgjeSYvxoPxZLwab/PWgpHPVMgvGO9fwr6VoA== < /latexit > cnn✓ < latexit sha1_base64= '' JLXS1g9bnxHVObe2qxmsdrRe0vE= '' > AAAB/HicbVBNS8NAEN3Ur1q/qj16CRbBU0m0+HErevFYwX5AE8pmO2mXbjZhdyKWUP+KFw+KePWHePPfmLRB1Ppg4PHeDDPzvEhwjZb1aRSWlldW14rrpY3Nre2d8u5eW4exYtBioQhV16MaBJfQQo4CupECGngCOt74KvM7d6A0D+UtTiJwAzqU3OeMYir1yxUH4R41S5iU076DI0DaL1etmjWDuUjsnFRJjma//OEMQhYHIJEJqnXPtiJ0E6qQMwHTkhNriCgb0yH0UippANpNZsdPzcNUGZh+qNKSaM7UnxMJDbSeBF7aGVAc6b9eJv7n9WL0z92EyyhGkGy+yI+FiaGZJWEOuAKGYpISyhRPbzXZiCrKMM2rNAvhIsPp98uLpH1cs09q9Zt6tXGZx1Ek++SAHBGbnJEGuSZN0iKMTMgjeSYvxoPxZLwab/PWgpHPVMgvGO9fwr6VoA== < /latexit > LSTM < latexit sha1_base64= '' vPJoMPQDUReM0WP4fXQB9xlgLcE= '' > AAAB9HicbVDJSgNBEO2JW4xb1KOXxiB4CjMaXG5BLx4UImaDZAg9nZ6kSc9id00wDPkOLx4U8erHePNv7JkMosYHBY/3qqiq54SCKzDNTyO3sLi0vJJfLaytb2xuFbd3miqIJGUNGohAth2imOA+awAHwdqhZMRzBGs5o8vEb42ZVDzw6zAJme2Rgc9dTgloye4CewBF4+u7+s20VyyZZTMFnidWRkooQ61X/Oj2Axp5zAcqiFIdywzBjokETgWbFrqRYiGhIzJgHU194jFlx+nRU3yglT52A6nLB5yqPydi4ik18Rzd6REYqr9eIv7ndSJwz+yY+2EEzKezRW4kMAQ4SQD3uWQUxEQTQiXXt2I6JJJQ0DkV0hDOE5x8vzxPmkdl67hcua2UqhdZHHm0h/bRIbLQKaqiK1RDDUTRPXpEz+jFGBtPxqvxNmvNGdnMLvoF4/0LAIuSYw== < /latexit > CGRU < latexit sha1_base64= '' nM6xeF3pGpjXoIKKbzLIh07YJfs= '' > AAAB9HicbVDLSsNAFJ3UV62vqks3wSK4KokWH7tiF7qsYtpCG8pkOmmHTiZx5qZYQr/DjQtF3Pox7vwbJ2kQXwcuHM65l3vv8SLOFFjWh1FYWFxaXimultbWNza3yts7LRXGklCHhDyUHQ8rypmgDjDgtBNJigOP07Y3bqR+e0KlYqG4hWlE3QAPBfMZwaAltwf0HhRJGpc3zqxfrlhVK4P5l9g5qaAczX75vTcISRxQAYRjpbq2FYGbYAmMcDor9WJFI0zGeEi7mgocUOUm2dEz80ArA9MPpS4BZqZ+n0hwoNQ08HRngGGkfnup+J/XjcE/cxMmohioIPNFfsxNCM00AXPAJCXAp5pgIpm+1SQjLDEBnVMpC+E8xcnXy39J66hqH1dr17VK/SKPo4j20D46RDY6RXV0hZrIQQTdoQf0hJ6NifFovBiv89aCkc/soh8w3j4B6XySVA== < /latexit > M L P < latexit sha1_base64= '' ZuRbuE9xq4Izk61cfi3C2PUB7Ck= '' > AAAB83icbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4UKtgHNKFMppN26GQSZm7EEvobblwo4tafceffOE2DqPXAhcM593LvPX4suAbb/rQKC4tLyyvF1dLa+sbmVnl7p6WjRFHWpJGIVMcnmgkuWRM4CNaJFSOhL1jbH11O/fY9U5pH8g7GMfNCMpA84JSAkVwX2ANomt5cNya9csWu2hnwPHFyUkE5Gr3yh9uPaBIyCVQQrbuOHYOXEgWcCjYpuYlmMaEjMmBdQyUJmfbS7OYJPjBKHweRMiUBZ+rPiZSEWo9D33SGBIb6rzcV//O6CQRnXsplnACTdLYoSASGCE8DwH2uGAUxNoRQxc2tmA6JIhRMTKUshPMpTr5fnieto6pzXK3d1ir1izyOItpD++gQOegU1dEVaqAmoihGj+gZvViJ9WS9Wm+z1oKVz+yiX7DevwBS5ZIC < /latexit > CNN✓0 < latexit sha1_base64= '' xB+lnkBgE8liBuAkDOj7QM06nDQ= '' > AAACAHicbVDLSsNQEL2pr1pfURcu3ASL6KokWnzsit24KhXsA5pSbm4n7aU3D+6diCVk46+4caGIWz/DnX9j2gZR64GBwzkzzMxxQsEVmuanlltYXFpeya8W1tY3Nrf07Z2mCiLJoMECEci2QxUI7kMDOQpohxKo5whoOaPqxG/dgVQ88G9xHELXowOfu5xRTKWevmcj3KNicbVWS3qxjUNAepQUenrRLJlTGPPEykiRZKj39A+7H7DIAx+ZoEp1LDPEbkwlciYgKdiRgpCyER1AJ6U+9UB14+kDiXGYKn3DDWRaPhpT9edETD2lxp6TdnoUh+qvNxH/8zoRuhfdmPthhOCz2SI3EgYGxiQNo88lMBTjlFAmeXqrwYZUUoZpZrMQLic4+355njRPStZpqXxTLlausjjyZJ8ckGNikXNSIdekThqEkYQ8kmfyoj1oT9qr9jZrzWnZzC75Be39C6XwlpE= < /latexit > Figure 2 : Overview of the neural network architecture for the Cache agent . The trainable components are shown in yellow , the inputs and outputs in pink , and the intermediate outputs in blue . Refer to text for details . and living room scenes as these scenes are relatively large and often include many opportunities for interaction ; but we use all scenes types across our suite of experiments . Details regarding scene splits across train , validation , and test for the different experiments can be found in Sec . A.1 . In a game of cache , two agents ( a hider and a seeker ) compete , with the hiding agent attempting to place a given object in the environment so that the seeking agent can not find it . This game is zero-sum with the hiding agent winning if and only if the seeking agent can not find the object . We partition the game of cache into five conceptually-distinct stages : exploration and mapping ( E & M ) , perspective simulation ( PS ) , object hiding ( OH ) , object manipulation ( OM ) , and seeking ( S ) ; see Figures 1b to 1f . A game of cache begins with the hiding agent exploring its environment and building an internal map corresponding to the locations it has visited ( E & M ) . The agent then chooses globally , among the many locations it has visited , a location where it believes it can hide the object so that the seeker will not be able to find it ( PS ) . After moving to this location the agent makes a local decision about where the object should be placed , e.g . behind a TV or inside a drawer ( OH ) . The agent then performs the low-level manipulation task of moving the object in its hand to the desired location ( OM ) , if this manipulation fails then the agent tries another local hiding location ( OH ) . Finally the seeking agent searches for the hidden object ( S ) . Pragmatically , these stages are distinguished by the actions available to the agent ( see Table G.1 for a description of all 219 actions ) and their reward structures ( see Sec . C.5 ) . For more details regarding these stages , see Sec . B .

Summary: In this paper, the author's propose an embodied adversarial reinforcement learning agent that can play a variation of hide-and-seek called Cache. This environment is a high fidelity interactive world. The authors argue that the agents are able to learn flexible representations of their observations which encode information such as object permanence, free space and containment.