Learning Generalizable Visual Representations via Interactive Gameplay

1 INTRODUCTION . We are interested in studying what facets of their environment artificial agents learn to represent through interaction and gameplay . We study this question within the context of hide-and-seek , for which proficiency requires an ability to navigate around in an environment and manipulate objects as well as an understanding of visual relationships , object affordances , and perspective . Inspired by behavior observed in juvenile ravens ( Burghardt , 2005 ) , we focus on a variant of hide-and-seek called cache in which agents hide objects instead of themselves . Advances in deep reinforcement learning have shown that , in abstract games ( e.g . Go and Chess ) and visually simplistic environments ( e.g . Atari and grid-worlds ) with limited interaction , artificial agents exhibit surprising emergent behaviours that enable proficient gameplay ( Mnih et al. , 2015 ; Silver et al. , 2017 ) ; indeed , recent work ( Chen et al. , 2019 ; Baker et al. , 2020 ) has shown this in the context of hiding games . Our interest , however , is in understanding how agents learn to represent their visual environment , through gameplay that requires varied interaction , in a high-fidelity environment grounded in the real world . This requires a fundamental shift away from existing popular environments and a rethinking of how the capabilities of artificial agents are evaluated . Our agents must first be embodied within an environment allowing for diverse interaction and providing rich visual output . For this we leverage AI2-THOR ( Kolve et al. , 2017 ) , a near photo-realistic , interactive , simulated , 3D environment of indoor living spaces , see Fig . 1a . Our agents are parameterized using deep neural networks , and trained adversarially using the paradigms of reinforce- ment ( Mnih et al. , 2016 ) and self-supervised learning . After our agents are trained to play cache , we then probe how they have learned to represent their environment . To this end we distinguish two distinct categories of representations generated by our agents . The first , static image representations ( SIRs ) , correspond to the output of a CNN applied to the agent ’ s egocentric visual input . The second , dynamic image representations ( DIRs ) , correspond to the output of the agents ’ RNN . While SIRs are timeless , operating only on single images , DIRs have the capacity to incorporate the agent ’ s previous actions and observations . Representation learning within the computer vision community is largely focused on developing SIRs whose quality is measured by their utility in downstream tasks ( e.g . classification , depthprediction , etc ) ( Zamir et al. , 2018 ) . Our first set of experiments show that our agents develop lowlevel visual understanding of individual images measured by their capacity to perform a collection of standard tasks from the computer vision literature , these tasks include pixel-to-pixel depth ( Saxena et al. , 2006 ) and surface normal ( Fouhey et al. , 2013 ) prediction , from a single image . While SIRs are clearly an important facet of representation learning , they are also definitionally unable to represent an environment as a whole : without the ability to integrate observations through time , a representation can only ever capture a single snapshot of space and time . To represent an environment holistically , we require DIRs . Unlike for SIRs , we are unaware of any well-established benchmarks for DIRs . In order to investigate what has been learned by our agent ’ s DIRs we develop a suite of experiments loosely inspired by experiments performed on infants and young children . These experiments then demonstrate our agents ’ ability to integrate observations through time and understand spatial relationships between objects ( Casasola et al. , 2003 ) , occlusion ( Hespos et al. , 2009 ) , object permanence ( Piaget , 1954 ) , and seriation ( Piaget , 1954 ) of free space . It is important to stress that this work focuses on studying how play and interaction contribute to representation learning in artificial agents and not on developing a new , state-of-the-art , methodology for representation learning . Nevertheless , to better situate our results in context of existing work , we provide strong baselines in our experiments , e.g . in our low-level vision experiments we compare against a fully supervised model trained on ImageNet ( Deng et al. , 2009 ) . Our results provide compelling evidence that : ( a ) on a suite of low level computer vision tasks within AI2-THOR , static representations learned by playing cache perform very competitively ( and often outperform ) strong unsupervised and fully supervised methods , ( b ) these static representations , trained using only synthetic images , obtain non-trivial transfer to downstream tasks using real-world images , ( c ) unlike representations learned from datasets of single images , agents trained via embodied gameplay learn to integrate visual information through time , demonstrating an elementary understanding of free space , objects and their relationships , and ( d ) embodied gameplay provides a natural means by which to generate rich experiences for representation learning beyond random sampling and relatively simpler tasks like visual navigation . In summary , we highlight the contributions : ( 1 ) Cache – we introduce cache within the AI2-THOR environment , an adversarial game which permits the study of representation learning in the context of interactive , visual , gameplay . ( 2 ) Cache agent – training agents to play Cache is non-trivial . We produce a strong Cache agent which integrates several methodological novelties ( see , e.g. , perspective simulation and visual dynamics replay in Sec . 4 ) and even outperforms humans at hiding on training scenes ( see Sec . 5 ) . ( 3 ) Static and dynamic representation study – we provide comprehensive evaluations of how our Cache agent has learned to represent its environment providing insight into the advantages and current limitations of interactive-gameplay-based representation learning . 2 RELATED WORK . Deep reinforcement learning for games . As games provide an interactive environment that enable agents to receive observations , take actions , and receive rewards , they are a popular testbed for RL algorithms . Reinforcement learning has been studied in the context of numerous single , and multi , agent games such as Atari Breakout ( Mnih et al. , 2015 ) , VizDoom ( Lample & Chaplot , 2017 ) , Go ( Silver et al. , 2017 ) , StarCraft ( Vinyals et al. , 2019 ) , Dota ( Berner et al. , 2019 ) and Hide-and-Seek ( Baker et al. , 2020 ) . The goal of these works is proficiency : to create an agent which can achieve ( super- ) human performance with respect to the game ’ s success criteria . In contrast , our goal is to understand how an agent has learned to represent its environment through gameplay and to show that such an agent ’ s representations can be employed for downstream tasks beyond gameplay . Passive representation learning . There is a large body of recent works that address the problem of representation learning from static images or videos . Image colorization ( Zhang et al. , 2016 ) , egomotion estimation ( Agrawal et al. , 2015 ) , predicting image rotation ( Gidaris et al. , 2018 ) , context prediction ( Doersch et al. , 2015 ) , future frame prediction ( Vondrick et al. , 2016 ) and more recent contrastive learning based approaches ( He et al. , 2020 ; Chen et al. , 2020b ) are among the successful examples of passive representation learning . We refer to them as passive since the representations are learned from a fixed set of images or videos . Our approach in contrast is interactive in that the images observed during learning are decided by the actions of the agent . Interactive representation learning . Learning representations in dynamic and interactive environments has been addressed in the literature as well . In the following we mention a few examples . Burda et al . ( 2019 ) explore curiosity-driven representation learning from a suite of games . Anand et al . ( 2019 ) address representation learning of the latent factors used for generating the interactive environment . Zhan et al . ( 2018 ) learn to improve exploration in video games by predicting the memory state . Ghosh et al . ( 2019 ) learn a functional representation for decision making as opposed to a representation for the observation space only . Whitney et al . ( 2020 ) simultaneously learn embeddings of state and action sequences . Jonschkowski & Brock ( 2015 ) learn a representation by measuring inconsistencies with a set of pre-defined priors . Pinto et al . ( 2016 ) learn visual representations by pushing and grasping objects using a robotics arm . The representations learned using these approaches are typically tested on tasks akin to tasks they were trained with ( e.g. , a different level of the same game ) . In contrast , we investigate whether cognitive primitives such as depth estimation and object permanence maybe be learned via gameplay in a visual dynamic environment . 3 PLAYING CACHE IN SIMULATION . We situate our agents within AI2-THOR , a simulated 3D environment of indoor living spaces within which multiple agents can navigate around and interact with objects ( e.g . by picking up , placing , opening , closing , cutting , switching on , etc. ) . In past works , AI2-THOR has been leveraged to teach agents to interact with their world ( Zhu et al. , 2017 ; Hu et al. , 2018 ; Huang et al. , 2019 ; Gan et al. , 2020b ; Wortsman et al. , 2019 ; Gordon et al. , 2017 ) , interact with each other ( Jain et al. , 2019 ; 2020 ) as well as learn from these interactions ( Nagarajan & Grauman , 2020 ; Lohmann et al. , 2020 ) . AI2-THOR contains a total of 150 unique scenes equally distributed into five scene types : kitchens , living rooms , bedrooms , bathrooms , and foyers . We train our cache agents on a subset of the kitchen Published as a conference paper at ICLR 2021 Inp t L A & R c NN A . E D C . D c D A M Me c Ma Inp t L A & R c NN A . E D C . D c D A M Me c Ma Inp t L A & R c NN A . E D C . D c D A M Me c Ma Hide cup gt < latexit sha1_base64= '' +0+K1AhyGutp5i38YYIOTiq/JMQ= '' > AAAB6nicbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2ge0oUymk3ToZBJmboQS+gluXCji1i9y5984SYOo9cCFwzn3cu89Xiy4Btv+tEpLyyura+X1ysbm1vZOdXevo6NEUdamkYhUzyOaCS5ZGzgI1osVI6EnWNebXGd+94EpzSN5D9OYuSEJJPc5JWCku2AIw2rNrts58CJxClJDBVrD6sdgFNEkZBKoIFr3HTsGNyUKOBVsVhkkmsWETkjA+oZKEjLtpvmpM3xklBH2I2VKAs7VnxMpCbWehp7pDAmM9V8vE//z+gn4F27KZZwAk3S+yE8Ehghnf+MRV4yCmBpCqOLmVkzHRBEKJp1KHsJlhrPvlxdJ56TunNYbt41a86qIo4wO0CE6Rg46R010g1qojSgK0CN6Ri+WsJ6sV+tt3lqyipl99AvW+xdwcI4K < /latexit > SIR 128x7x7 st < latexit sha1_base64= '' uwrevV+/ludXMHIgESK4k17/zD4= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOz3AQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wKCuI4W < /latexit > it < latexit sha1_base64= '' 49jzbtNkZShn+iIjbBN4ZQyo2i4= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOz7AQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wJzfI4M < /latexit > ot < latexit sha1_base64= '' ws3F94vsh/kOa4LUOCliZNGhMaM= '' > AAAB6nicbVDLSgNBEJyNrxhfUY9eBoPgKWw0+LgFvXiMaB6QLGF2MpsMmZ1ZZnqFsOQTvHhQxKtf5M2/cXaziBoLGoqqbrq7/EhwA6776RSWlldW14rrpY3Nre2d8u5e26hYU9aiSijd9YlhgkvWAg6CdSPNSOgL1vEn16nfeWDacCXvYRoxLyQjyQNOCVjpTg1gUK64VTcDXiS1nFRQjuag/NEfKhqHTAIVxJhezY3AS4gGTgWblfqxYRGhEzJiPUslCZnxkuzUGT6yyhAHStuSgDP150RCQmOmoW87QwJj89dLxf+8XgzBhZdwGcXAJJ0vCmKBQeH0bzzkmlEQU0sI1dzeiumYaELBplPKQrhMcfb98iJpn1Rrp9X6bb3SuMrjKKIDdIiOUQ2dowa6QU3UQhSN0CN6Ri+OcJ6cV+dt3lpw8pl99AvO+xd8oI4S < /latexit > ht 1 < latexit sha1_base64= '' w0z1MIqjcwqqrZvbS0vyNwXmoPE= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtcf9BE+cWb9csat2BrJInJxUIEejX/7oDUIW+zxAJqnWXceO0E2oQsEkn5V6seYRZRM64l1DA+pz7SbZuTNyZJQBGYbKVIAkU39OJNTXeup7ptOnONZ/vVT8z+vGOLx0ExFEMfKAzRcNY0kwJOnvZCAUZyinhlCmhLmVsDFVlKFJqJSFcJXi/PvlRdI6rTpn1dpdrVK/zuMowgEcwjE4cAF1uIUGNIHBBB7hGV6syHqyXq23eWvBymf24Res9y8TFo+J < /latexit > DIR 257x7x7 h̃t < latexit sha1_base64= '' /FD56DYReBR0BgD9sJp/o5zbwTA= '' > AAAB8nicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwdpCGspms2mXbrJhdyKU0J/hxYMiXv013vw3btIgan0w8Hhvhpl5fiK4Btv+tCpLyyura9X12sbm1vZOfXfvXstUUdalUkjV94lmgsesCxwE6yeKkcgXrOdPrnO/98CU5jK+g2nCvIiMYh5ySsBI7gC4CFg2ng1hWG/YTbsAXiROSRqoRGdY/xgEkqYRi4EKorXr2Al4GVHAqWCz2iDVLCF0QkbMNTQmEdNeVpw8w0dGCXAolakYcKH+nMhIpPU08k1nRGCs/3q5+J/nphBeeBmPkxRYTOeLwlRgkDj/HwdcMQpiagihiptbMR0TRSiYlGpFCJc5zr5fXiT3J03ntNm6bTXaV2UcVXSADtExctA5aqMb1EFdRJFEj+gZvVhgPVmv1tu8tWKVM/voF6z3L9XnkcE= < /latexit > ht 1 < latexit sha1_base64= '' w0z1MIqjcwqqrZvbS0vyNwXmoPE= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtcf9BE+cWb9csat2BrJInJxUIEejX/7oDUIW+zxAJqnWXceO0E2oQsEkn5V6seYRZRM64l1DA+pz7SbZuTNyZJQBGYbKVIAkU39OJNTXeup7ptOnONZ/vVT8z+vGOLx0ExFEMfKAzRcNY0kwJOnvZCAUZyinhlCmhLmVsDFVlKFJqJSFcJXi/PvlRdI6rTpn1dpdrVK/zuMowgEcwjE4cAF1uIUGNIHBBB7hGV6syHqyXq23eWvBymf24Res9y8TFo+J < /latexit > Inp t L A & R c NN A . E D C . D c D A M Me c Ma Metric map mt < latexit sha1_base64= '' hSQoieJvwDg/ssTgCtOF9DX6PMY= '' > AAAB6nicbVDLSgNBEJyNrxhfUY9eBoPgKWw0+LgFvXiMaB6QLGF2MpsMmZldZnqFsOQTvHhQxKtf5M2/cXaziBoLGoqqbrq7/EhwA6776RSWlldW14rrpY3Nre2d8u5e24SxpqxFQxHqrk8ME1yxFnAQrBtpRqQvWMefXKd+54Fpw0N1D9OIeZKMFA84JWClOzmAQbniVt0MeJHUclJBOZqD8kd/GNJYMgVUEGN6NTcCLyEaOBVsVurHhkWETsiI9SxVRDLjJdmpM3xklSEOQm1LAc7UnxMJkcZMpW87JYGx+eul4n9eL4bgwku4imJgis4XBbHAEOL0bzzkmlEQU0sI1dzeiumYaELBplPKQrhMcfb98iJpn1Rrp9X6bb3SuMrjKKIDdIiOUQ2dowa6QU3UQhSN0CN6Ri+OcJ6cV+dt3lpw8pl99AvO+xd5lI4Q < /latexit > Inp t L A & R c NN A . E D C . D c D A M Me c Ma Mental rollout M l < latexit sha1_base64= '' Xv1qiawUvtLxc2P1nxc4r9LTkLo= '' > AAAB6nicbVDLSsNAFL2pr1pfVZduBovgqiRafOyKbtwIFe0D2lgm00k7dDIJMxOhhH6CGxeKuPWL3Pk3TtIgaj1w4XDOvdx7jxdxprRtf1qFhcWl5ZXiamltfWNzq7y901JhLAltkpCHsuNhRTkTtKmZ5rQTSYoDj9O2N75M/fYDlYqF4k5PIuoGeCiYzwjWRrq9vuf9csWu2hnQPHFyUoEcjX75ozcISRxQoQnHSnUdO9JugqVmhNNpqRcrGmEyxkPaNVTggCo3yU6dogOjDJAfSlNCo0z9OZHgQKlJ4JnOAOuR+uul4n9eN9b+mZswEcWaCjJb5Mcc6RClf6MBk5RoPjEEE8nMrYiMsMREm3RKWQjnKU6+X54nraOqc1yt3dQq9Ys8jiLswT4cggOnUIcraEATCAzhEZ7hxeLWk/Vqvc1aC1Y+swu/YL1/ATsvjec= < /latexit > hct < latexit sha1_base64= '' 377/FCzba4IWjxe7BzJkwcV+C/I= '' > AAAB7HicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwbSFNpbNdtMu3WzC7kQoob/BiwdFvPqDvPlv3KRB1Ppg4PHeDDPz/FhwDbb9aZWWlldW18rrlY3Nre2d6u5eW0eJosylkYhU1yeaCS6ZCxwE68aKkdAXrONPrjO/88CU5pG8g2nMvJCMJA84JWAkd3xPBzCo1uy6nQMvEqcgNVSgNah+9IcRTUImgQqidc+xY/BSooBTwWaVfqJZTOiEjFjPUElCpr00P3aGj4wyxEGkTEnAufpzIiWh1tPQN50hgbH+62Xif14vgeDCS7mME2CSzhcFicAQ4exzPOSKURBTQwhV3NyK6ZgoQsHkU8lDuMxw9v3yImmf1J3TeuO2UWteFXGU0QE6RMfIQeeoiW5QC7mIIo4e0TN6saT1ZL1ab/PWklXM7KNfsN6/AOMLjuA= < /latexit > hot < latexit sha1_base64= '' aAOHlQFvdNC8xtzT88UW8qr16Eo= '' > AAAB7HicbVBNS8NAEN3Ur1q/qh69LBbBU0m0+HErevFYwbSFNpbNdtMu3WzC7kQoob/BiwdFvPqDvPlv3KRB1Ppg4PHeDDPz/FhwDbb9aZWWlldW18rrlY3Nre2d6u5eW0eJosylkYhU1yeaCS6ZCxwE68aKkdAXrONPrjO/88CU5pG8g2nMvJCMJA84JWAkd3wfDWBQrdl1OwdeJE5BaqhAa1D96A8jmoRMAhVE655jx+ClRAGngs0q/USzmNAJGbGeoZKETHtpfuwMHxlliINImZKAc/XnREpCraehbzpDAmP918vE/7xeAsGFl3IZJ8AknS8KEoEhwtnneMgVoyCmhhCquLkV0zFRhILJp5KHcJnh7PvlRdI+qTun9cZto9a8KuIoowN0iI6Rg85RE92gFnIRRRw9omf0YknryXq13uatJauY2Ue/YL1/AfVTjuw= < /latexit > ⇡X✓ < latexit sha1_base64= '' suTQFUFY1lP6zw363skjiclUM5s= '' > AAAB83icbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2Ac0sUymk3boZDLM3Agl9DfcuFDErT/jzr8xSYOo9cCFwzn3cu89vhLcgG1/WqWl5ZXVtfJ6ZWNza3unurvXMVGsKWvTSES65xPDBJesDRwE6ynNSOgL1vUn15nffWDa8EjewVQxLyQjyQNOCaSS6yo+cGHMgNz3BtWaXbdz4EXiFKSGCrQG1Q93GNE4ZBKoIMb0HVuBlxANnAo2q7ixYYrQCRmxfkolCZnxkvzmGT5KlSEOIp2WBJyrPycSEhozDf20MyQwNn+9TPzP68cQXHgJlyoGJul8URALDBHOAsBDrhkFMU0JoZqnt2I6JppQSGOq5CFcZjj7fnmRdE7qzmm9cduoNa+KOMroAB2iY+Sgc9REN6iF2ogihR7RM3qxYuvJerXe5q0lq5jZR79gvX8BJ3WR5g== < /latexit > vX✓ < latexit sha1_base64= '' GdMYSlPeVUO3HzqoaS+rmkTfYe0= '' > AAAB8XicbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4r2Ae2sUymk3boZBJmbgol9C/cuFDErX/jzr9xkgZR64ELh3Pu5d57vEhwDbb9aRWWlldW14rrpY3Nre2d8u5eS4exoqxJQxGqjkc0E1yyJnAQrBMpRgJPsLY3vk799oQpzUN5B9OIuQEZSu5zSsBI95N+D0YMyEOnX67YVTsDXiROTiooR6Nf/ugNQhoHTAIVROuuY0fgJkQBp4LNSr1Ys4jQMRmyrqGSBEy7SXbxDB8ZZYD9UJmSgDP150RCAq2ngWc6AwIj/ddLxf+8bgz+hZtwGcXAJJ0v8mOBIcTp+3jAFaMgpoYQqri5FdMRUYSCCamUhXCZ4uz75UXSOqk6p9Xaba1Sv8rjKKIDdIiOkYPOUR3doAZqIookekTP6MXS1pP1ar3NWwtWPrOPfsF6/wK0DpET < /latexit > at < latexit sha1_base64= '' YaG+GfOaZFAdbmyIZ6rpWb7fQcE= '' > AAAB6nicbVBNS8NAEJ3Ur1q/qh69BIvgqaRa/LgVvXisaD+gDWWz3bRLN5uwOxFK6E/w4kERr/4ib/4bN2kQtT4YeLw3w8w8LxJco+N8WoWl5ZXVteJ6aWNza3unvLvX1mGsKGvRUISq6xHNBJeshRwF60aKkcATrONNrlO/88CU5qG8x2nE3ICMJPc5JWikOzLAQbniVJ0M9iKp5aQCOZqD8kd/GNI4YBKpIFr3ak6EbkIUcirYrNSPNYsInZAR6xkqScC0m2SnzuwjowxtP1SmJNqZ+nMiIYHW08AznQHBsf7rpeJ/Xi9G/8JNuIxiZJLOF/mxsDG007/tIVeMopgaQqji5labjokiFE06pSyEyxRn3y8vkvZJtXZard/WK42rPI4iHMAhHEMNzqEBN9CEFlAYwSM8w4slrCfr1XqbtxasfGYffsF6/wJnTI4E < /latexit > it 1 < latexit sha1_base64= '' I57o8i9M8qP7Gcb8lapa+6gjDYU= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtUU/wRNn1i9X7KqdgSwSJycVyNHolz96g5DFPg+QSap117EjdBOqUDDJZ6VerHlE2YSOeNfQgPpcu0l27owcGWVAhqEyFSDJ1J8TCfW1nvqe6fQpjvVfLxX/87oxDi/dRARRjDxg80XDWBIMSfo7GQjFGcqpIZQpYW4lbEwVZWgSKmUhXKU4/355kbROq85ZtXZXq9Sv8ziKcACHcAwOXEAdbqEBTWAwgUd4hhcrsp6sV+tt3lqw8pl9+AXr/QsUoI+K < /latexit > at 1 < latexit sha1_base64= '' 8N4WOjgCSh1etWpJRyuHRbcdus4= '' > AAAB7nicbVDJSgNBEK2JW4xb1KOXxiB4McxocLkFvXiMYBZIhtDT6SRNeha6a4Qw5CO8eFDEq9/jzb+xZzKIGh8UPN6roqqeF0mh0bY/rcLS8srqWnG9tLG5tb1T3t1r6TBWjDdZKEPV8ajmUgS8iQIl70SKU9+TvO1NblK//cCVFmFwj9OIuz4dBWIoGEUjtWk/wRNn1i9X7KqdgSwSJycVyNHolz96g5DFPg+QSap117EjdBOqUDDJZ6VerHlE2YSOeNfQgPpcu0l27owcGWVAhqEyFSDJ1J8TCfW1nvqe6fQpjvVfLxX/87oxDi/dRARRjDxg80XDWBIMSfo7GQjFGcqpIZQpYW4lbEwVZWgSKmUhXKU4/355kbROq85ZtXZXq9Sv8ziKcACHcAwOXEAdbqEBTWAwgUd4hhcrsp6sV+tt3lqw8pl9+AXr/QsIUI+C < /latexit > 512 512 Visual Dynamics Replay Self-supervised Tasks Hiding Locationscnn✓ < latexit sha1_base64= '' JLXS1g9bnxHVObe2qxmsdrRe0vE= '' > AAAB/HicbVBNS8NAEN3Ur1q/qj16CRbBU0m0+HErevFYwX5AE8pmO2mXbjZhdyKWUP+KFw+KePWHePPfmLRB1Ppg4PHeDDPzvEhwjZb1aRSWlldW14rrpY3Nre2d8u5eW4exYtBioQhV16MaBJfQQo4CupECGngCOt74KvM7d6A0D+UtTiJwAzqU3OeMYir1yxUH4R41S5iU076DI0DaL1etmjWDuUjsnFRJjma//OEMQhYHIJEJqnXPtiJ0E6qQMwHTkhNriCgb0yH0UippANpNZsdPzcNUGZh+qNKSaM7UnxMJDbSeBF7aGVAc6b9eJv7n9WL0z92EyyhGkGy+yI+FiaGZJWEOuAKGYpISyhRPbzXZiCrKMM2rNAvhIsPp98uLpH1cs09q9Zt6tXGZx1Ek++SAHBGbnJEGuSZN0iKMTMgjeSYvxoPxZLwab/PWgpHPVMgvGO9fwr6VoA== < /latexit > cnn✓ < latexit sha1_base64= '' JLXS1g9bnxHVObe2qxmsdrRe0vE= '' > AAAB/HicbVBNS8NAEN3Ur1q/qj16CRbBU0m0+HErevFYwX5AE8pmO2mXbjZhdyKWUP+KFw+KePWHePPfmLRB1Ppg4PHeDDPzvEhwjZb1aRSWlldW14rrpY3Nre2d8u5eW4exYtBioQhV16MaBJfQQo4CupECGngCOt74KvM7d6A0D+UtTiJwAzqU3OeMYir1yxUH4R41S5iU076DI0DaL1etmjWDuUjsnFRJjma//OEMQhYHIJEJqnXPtiJ0E6qQMwHTkhNriCgb0yH0UippANpNZsdPzcNUGZh+qNKSaM7UnxMJDbSeBF7aGVAc6b9eJv7n9WL0z92EyyhGkGy+yI+FiaGZJWEOuAKGYpISyhRPbzXZiCrKMM2rNAvhIsPp98uLpH1cs09q9Zt6tXGZx1Ek++SAHBGbnJEGuSZN0iKMTMgjeSYvxoPxZLwab/PWgpHPVMgvGO9fwr6VoA== < /latexit > LSTM < latexit sha1_base64= '' vPJoMPQDUReM0WP4fXQB9xlgLcE= '' > AAAB9HicbVDJSgNBEO2JW4xb1KOXxiB4CjMaXG5BLx4UImaDZAg9nZ6kSc9id00wDPkOLx4U8erHePNv7JkMosYHBY/3qqiq54SCKzDNTyO3sLi0vJJfLaytb2xuFbd3miqIJGUNGohAth2imOA+awAHwdqhZMRzBGs5o8vEb42ZVDzw6zAJme2Rgc9dTgloye4CewBF4+u7+s20VyyZZTMFnidWRkooQ61X/Oj2Axp5zAcqiFIdywzBjokETgWbFrqRYiGhIzJgHU194jFlx+nRU3yglT52A6nLB5yqPydi4ik18Rzd6REYqr9eIv7ndSJwz+yY+2EEzKezRW4kMAQ4SQD3uWQUxEQTQiXXt2I6JJJQ0DkV0hDOE5x8vzxPmkdl67hcua2UqhdZHHm0h/bRIbLQKaqiK1RDDUTRPXpEz+jFGBtPxqvxNmvNGdnMLvoF4/0LAIuSYw== < /latexit > CGRU < latexit sha1_base64= '' nM6xeF3pGpjXoIKKbzLIh07YJfs= '' > AAAB9HicbVDLSsNAFJ3UV62vqks3wSK4KokWH7tiF7qsYtpCG8pkOmmHTiZx5qZYQr/DjQtF3Pox7vwbJ2kQXwcuHM65l3vv8SLOFFjWh1FYWFxaXimultbWNza3yts7LRXGklCHhDyUHQ8rypmgDjDgtBNJigOP07Y3bqR+e0KlYqG4hWlE3QAPBfMZwaAltwf0HhRJGpc3zqxfrlhVK4P5l9g5qaAczX75vTcISRxQAYRjpbq2FYGbYAmMcDor9WJFI0zGeEi7mgocUOUm2dEz80ArA9MPpS4BZqZ+n0hwoNQ08HRngGGkfnup+J/XjcE/cxMmohioIPNFfsxNCM00AXPAJCXAp5pgIpm+1SQjLDEBnVMpC+E8xcnXy39J66hqH1dr17VK/SKPo4j20D46RDY6RXV0hZrIQQTdoQf0hJ6NifFovBiv89aCkc/soh8w3j4B6XySVA== < /latexit > M L P < latexit sha1_base64= '' ZuRbuE9xq4Izk61cfi3C2PUB7Ck= '' > AAAB83icbVDLSsNAFJ3UV62vqks3g0VwVRItPnZFNy4UKtgHNKFMppN26GQSZm7EEvobblwo4tafceffOE2DqPXAhcM593LvPX4suAbb/rQKC4tLyyvF1dLa+sbmVnl7p6WjRFHWpJGIVMcnmgkuWRM4CNaJFSOhL1jbH11O/fY9U5pH8g7GMfNCMpA84JSAkVwX2ANomt5cNya9csWu2hnwPHFyUkE5Gr3yh9uPaBIyCVQQrbuOHYOXEgWcCjYpuYlmMaEjMmBdQyUJmfbS7OYJPjBKHweRMiUBZ+rPiZSEWo9D33SGBIb6rzcV//O6CQRnXsplnACTdLYoSASGCE8DwH2uGAUxNoRQxc2tmA6JIhRMTKUshPMpTr5fnieto6pzXK3d1ir1izyOItpD++gQOegU1dEVaqAmoihGj+gZvViJ9WS9Wm+z1oKVz+yiX7DevwBS5ZIC < /latexit > CNN✓0 < latexit sha1_base64= '' xB+lnkBgE8liBuAkDOj7QM06nDQ= '' > AAACAHicbVDLSsNQEL2pr1pfURcu3ASL6KokWnzsit24KhXsA5pSbm4n7aU3D+6diCVk46+4caGIWz/DnX9j2gZR64GBwzkzzMxxQsEVmuanlltYXFpeya8W1tY3Nrf07Z2mCiLJoMECEci2QxUI7kMDOQpohxKo5whoOaPqxG/dgVQ88G9xHELXowOfu5xRTKWevmcj3KNicbVWS3qxjUNAepQUenrRLJlTGPPEykiRZKj39A+7H7DIAx+ZoEp1LDPEbkwlciYgKdiRgpCyER1AJ6U+9UB14+kDiXGYKn3DDWRaPhpT9edETD2lxp6TdnoUh+qvNxH/8zoRuhfdmPthhOCz2SI3EgYGxiQNo88lMBTjlFAmeXqrwYZUUoZpZrMQLic4+355njRPStZpqXxTLlausjjyZJ8ckGNikXNSIdekThqEkYQ8kmfyoj1oT9qr9jZrzWnZzC75Be39C6XwlpE= < /latexit > Figure 2 : Overview of the neural network architecture for the Cache agent . The trainable components are shown in yellow , the inputs and outputs in pink , and the intermediate outputs in blue . Refer to text for details . and living room scenes as these scenes are relatively large and often include many opportunities for interaction ; but we use all scenes types across our suite of experiments . Details regarding scene splits across train , validation , and test for the different experiments can be found in Sec . A.1 . In a game of cache , two agents ( a hider and a seeker ) compete , with the hiding agent attempting to place a given object in the environment so that the seeking agent can not find it . This game is zero-sum with the hiding agent winning if and only if the seeking agent can not find the object . We partition the game of cache into five conceptually-distinct stages : exploration and mapping ( E & M ) , perspective simulation ( PS ) , object hiding ( OH ) , object manipulation ( OM ) , and seeking ( S ) ; see Figures 1b to 1f . A game of cache begins with the hiding agent exploring its environment and building an internal map corresponding to the locations it has visited ( E & M ) . The agent then chooses globally , among the many locations it has visited , a location where it believes it can hide the object so that the seeker will not be able to find it ( PS ) . After moving to this location the agent makes a local decision about where the object should be placed , e.g . behind a TV or inside a drawer ( OH ) . The agent then performs the low-level manipulation task of moving the object in its hand to the desired location ( OM ) , if this manipulation fails then the agent tries another local hiding location ( OH ) . Finally the seeking agent searches for the hidden object ( S ) . Pragmatically , these stages are distinguished by the actions available to the agent ( see Table G.1 for a description of all 219 actions ) and their reward structures ( see Sec . C.5 ) . For more details regarding these stages , see Sec . B .

Summary: This paper proposes embodied game-playing with artificial agents as a method to learn better representations of their environment. They describe a game, cache, which is a variant of hide-and-seek played in a virtual environment and a method for training an agent to play the game. They present results which demonstrate that the static representations learned through game-playing perform better than other pre-training tasks within the same virtual environment, on both virtual vision and real world vision applications. They also show that the dynamic representations are useful for completing object permanence tests inspired by developmental psychology research.