Explore and Control with Adversarial Surprise

1 INTRODUCTION . Reinforcement learning methods have attained impressive results across a number of domains ( e.g. , Berner et al . ( 2019 ) ; Kober et al . ( 2013 ) ; Levine et al . ( 2016 ) ; Vinyals et al . ( 2019 ) ) . However , current RL methods typically require a large number of samples for each new task ( Dann et al. , 2018 ) . In other areas of machine learning , an effective way to mitigate high data requirements has been the use of unsupervised or self-supervised learning ( Sutskever et al. , 2014 ; Radford et al. , 2019 ) . Similarly , humans and animals seem to be able to learn rich priors from their own experience without being told what to do , and children engage in structured but unsupervised play in part as a way to acquire a functional understanding of the world ( Smith & Gasser , 2005 ) . Based on this intuition , unsupervised RL methods rely on intrinsic motivation ( IM ) : task-agnostic objectives that incentivize the agent to autonomously explore the world and learn behaviors that can be used to solve a range of downstream tasks with little supervision . A general strategy is to exactly or approximately express this task-agnostic objective in the form of a reward function that uses only environment statistics , and then optimize it using standard RL algorithms . A reasonable goal for a good unsupervised learning algorithm is to fully explore the state space of the environment , since this ensures the agent will have the experience which with to learn the optimal policy for a downstream task . Therefore , past work on IM has frequently focused on novelty-seeking agents that maximize surprise or prediction error ( Achiam & Sastry , 2017 ; Schmidhuber , 1991 ; Yamamoto & Ishikawa , 2010 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) . However , these methods are vulnerable to becoming distracted by inherently stochastic elements of the environment , such as a “ noisy TV ” ( Schmidhuber , 2010 ) . In contrast , active inference researchers inspired by biological agents have focused on developing agents that seek to control their environment and minimize surprise ( Friston , 2009 ; Friston et al. , 2009 ; 2016 ; Berseth et al. , 2021 ) . These methods suffer from the opposite issue , the “ dark room problem ” , in which a surprise-minimizing agent in a low-entropy environment does not need to learn any behaviors at all in order to satisfy its objective ( Friston et al. , 2012 ) . Yet humans seem to maintain a balance between optimizing for both novelty and familiarity . For example , a child in a play room does not just try to toss their toys on the floor in every possible pattern , or immediately put them away in the toy box , but instead tries to stack them together , find new uses for parts , or combine them in various structured ways . We argue that an effective unsupervised RL method should find the right balance between exploration and control . With this goal in mind , we introduce a new algorithm based on an adversarial game between two policies , which take turns sequentially acting for the same RL agent . The goal of the Control policy is to minimize surprise , by learning to manipulate its environment in order to return to safe and predictable states . In turn , the Explore policy is novelty-seeking , and attempts to maximize surprise for the Control policy , putting it into a diverse range of novel states . When combined , the two adversaries engage in an arms race , repeatedly putting the agent into challenging new situations , then attempting to gain control of those situations . Figure 8 shows an illustration of the method , including a sample interaction . Rather than simply adding noise to the environment , the Explore policy learns to adapt to the Control policy , and to search for increasingly challenging situations from which the Control policy must recover . Thus our method , Adversarial Surprise ( AS ) , leverages the power of multi-agent training to generate a curriculum of increasingly challenging exploration and control problems , leading to the emergence of complex , meaningful behaviours . The contributions of this paper are : i ) The Adversarial Surprise algorithm ; ii ) Theoretical results which prove that when the environment is formulated as a stochastic Block MDP ( Du et al. , 2019 ) , traditional surprise-maximizing methods will fail to fully explore the underlying state space , but Adversarial Surprise will succeed ; iii ) Empirical evidence which supports the theoretical results , and shows that AS fully explores the state space of both Block MDPs and traditional benchmarking environments like VizDoom ; iv ) Experiments that compare AS to state-of-the-art unsupervised RL baselines Random Network Distillation ( RND ) ( Burda et al. , 2018 ) , Asymmetric Self-Play ( ASP ) ( Sukhbaatar et al. , 2017 ) , Adversarially Guided Actor-Critic ( AGAC ) ( Flet-Berliac et al. , 2021 ) , and Surprise Minimizing RL ( SMiRL ) ( Berseth et al. , 2021 ) , and show that AS is able to explore more effectively , learn more meaningful behaviors , and achieve higher task reward when transferred zero-shot to common benchmarking environments Atari and VizDoom . Videos of our agents are available on the project website : https : //sites.google.com/corp/view/ adversarialsurprise/home , and show that AS is able to learn interesting , emergent behaviors in Atari , VizDoom , and MiniGrid , even without ever having trained with the game reward . 2 RELATED WORK . Novelty-seeking and exploration methods lead the agent to increase coverage of the environment . A simple way to implement novelty-seeking is to maximize the prediction error of a world model ( Achiam & Sastry , 2017 ; Schmidhuber , 1991 ; Yamamoto & Ishikawa , 2010 ; Pathak et al. , 2017 ; Burda et al. , 2018 ; Raileanu & Rocktäschel , 2020 ; Zhang et al. , 2020b ) . Random Network Distillation ( RND ) ( Burda et al. , 2018 ) is a highly effective example of one such method . However , a major problem of prediction-error-based methods is that the intrinsic reward is not useful when the envi- ronment contains aleatoric uncertainty , i.e . inherently stochastic elements . This problem is often referred to as the noisy TV problem , after Schmidhuber ( 2010 ) used the example of an agent becoming stuck staring at static on a TV screen . We show in this work that RND is indeed vulnerable to this problem , performing poorly in stochastic environments . Several works have proposed solutions to deal with aleatoric uncertainty . For example , some approximate information gain on the agent ’ s dynamics model of the environment ( Houthooft et al. , 2016 ; Still & Precup , 2012 ; Bellemare et al. , 2016 ; Pathak et al. , 2017 ; Schmidhuber , 1991 ) , using variational Bayes or ensemble disagreement ( Shyam et al. , 2019 ; Pathak et al. , 2019 ; Houthooft et al. , 2016 ) . However , implementing such Bayesian procedures is difficult , because it requires scalable and effective modeling of epistemic uncertainty , which itself is a major open problem with high-dimensional models such as neural networks ( Bhattacharya & Maiti , 2020 ) . Another method based on maximizing information gain between the agent ’ s actions and future state is known as empowerment ( Klyubin et al. , 2005 ; Salge et al. , 2014 ; Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) , but can also be difficult to approximate for high-dimensional states ( Karl et al. , 2015 ; Zhao et al. , 2020 ; de Abril & Kanai , 2018 ; Zhang et al. , 2020a ; Mohamed & Rezende , 2015 ; Gregor et al. , 2016 ; Hansen et al. , 2019 ) . Instead of attempting to directly approximate information gain , our approach maximizes state coverage via an adversarial competition over observation entropy . As we will show , this recovers an effective coverage strategy even in the presence of rich , stochastic observations , and performs well in practice . The asymptotic policy learned by standard novelty-seeking methods is not exploratory . Recent work tries to learn a policy that approximately maximizes the state marginal entropy at convergence ( Hazan et al. , 2019 ; Lee et al. , 2019 ) . The state marginal entropy is hard to compute in general , and recent work has proposed various approximations ( Seo et al. , 2021 ; Liu & Abbeel , 2021 ; Mutti et al. , 2021 ) . We prove that even with only noisy observations of the underlying state , our method asymptotically maximizes the state marginal entropy at convergence under some assumptions . Surprise minimization and active inference : The design of the Control agent in our method draws on ideas from surprise minimization and active inference ( Friston , 2009 ; Friston et al. , 2009 ; 2016 ) . The free energy principle , originating in the neuroscience community , argues that complex nicheseeking behaviors of biological systems are the result of minimizing long-term average surprise on system sensors , leading agents to stay in safe and stable states ( Friston , 2009 ) . SMiRL is an unsupervised RL method that leverages surprise minimization as a means to discover skills that stabilize an otherwise chaotic environment ( Berseth et al. , 2021 ) . However , such approaches require strong assumptions on the stochasticity of the environment . In low-entropy environments , surprise minimization will not lead to learning interesting behavior due to the dark room problem , in which the agent is not incentivized to explore the environment to find a better niche ( Friston et al. , 2012 ) . Our method does not require that the environment is stochastic , since the Explore agent itself drives the Control agent into situations from which surprise minimization is challenging . Multi-Agent competition has been shown to drive the emergence of complex behavior ( Baker et al. , 2019 ; Dennis et al. , 2020 ; Xu et al. , 2020 ; Leibo et al. , 2019 ; Schmidhuber , 1997 ; Campero et al. , 2020 ) . Asymmetric Self-Play ( ASP ) aims to learn increasingly complex skills via a competition between two policies , where one policy ( Bob ) tries to imitate or reverse the trajectory of the other policy ( Alice ) ( Sukhbaatar et al. , 2017 ; OpenAI et al. , 2021 ) . Our empirical results compare to ASP , and demonstrate that it can fail in stochastic environments , because Alice can easily produce random trajectories which are very difficult for Bob to imitate . Similar to ASP , Adversarially-Guided Actor Critic ( AGAC ) ( Flet-Berliac et al. , 2021 ) introduces an adversary agent which attempts to mimic the action distribution of the actor , while the actor tries to differentiate itself from the adversary . Like these methods , Adversarial Surprise uses a two-player game to induce exploration and emergent complexity . However , our objective is general and information-theoretic , focusing on minimization or maximization of surprise rather than reaching specific states . Unlike these methods , we provide theoretical results showing AS provides a principled approach to maximizing state coverage in stochastic environments . Finally , our method is reminiscent of recent work that learns separate exploration and exploitation policies but without competition between them ( Badia et al. , 2020b ; a ; Campos et al. , 2021 )

Summary: This paper proposes Adversarial Surprise (AS), a method for unsupervised training of RL agents based on the competition of two policies dubbed Explore and Control. These two policies control compete to maximize/minimize surprise, respectively, by taking turns to control a shared body. Authors show that under some conditions this objective leads to maximizing state entropy within an episode, and formalize the settings where other approaches may fail. Evaluation is performed on three different domains: procedurally generated MiniGrid tasks, VizDoom and four Atari games. AS explores more rooms within an episode in Minigrid, visits more x positions within an episode in VizDoom, and obtains higher zero-shot scores in 3/4 Atari games.