Explore and Control with Adversarial Surprise

1 INTRODUCTION . Reinforcement learning methods have attained impressive results across a number of domains ( e.g. , Berner et al . ( 2019 ) ; Kober et al . ( 2013 ) ; Levine et al . ( 2016 ) ; Vinyals et al . ( 2019 ) ) . However , current RL methods typically require a large number of samples for each new task ( Dann et al. , 2018 ) . In other areas of machine learning , an effective way to mitigate high data requirements has been the use of unsupervised or self-supervised learning ( Sutskever et al. , 2014 ; Radford et al. , 2019 ) . Similarly , humans and animals seem to be able to learn rich priors from their own experience without being told what to do , and children engage in structured but unsupervised play in part as a way to acquire a functional understanding of the world ( Smith & Gasser , 2005 ) . Based on this intuition , unsupervised RL methods rely on intrinsic motivation ( IM ) : task-agnostic objectives that incentivize the agent to autonomously explore the world and learn behaviors that can be used to solve a range of downstream tasks with little supervision . A general strategy is to exactly or approximately express this task-agnostic objective in the form of a reward function that uses only environment statistics , and then optimize it using standard RL algorithms . A reasonable goal for a good unsupervised learning algorithm is to fully explore the state space of the environment , since this ensures the agent will have the experience which with to learn the optimal policy for a downstream task . Therefore , past work on IM has frequently focused on novelty-seeking agents that maximize surprise or prediction error ( Achiam & Sastry , 2017 ; Schmidhuber , 1991 ; Yamamoto & Ishikawa , 2010 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) . However , these methods are vulnerable to becoming distracted by inherently stochastic elements of the environment , such as a “ noisy TV ” ( Schmidhuber , 2010 ) . In contrast , active inference researchers inspired by biological agents have focused on developing agents that seek to control their environment and minimize surprise ( Friston , 2009 ; Friston et al. , 2009 ; 2016 ; Berseth et al. , 2021 ) . These methods suffer from the opposite issue , the “ dark room problem ” , in which a surprise-minimizing agent in a low-entropy environment does not need to learn any behaviors at all in order to satisfy its objective ( Friston et al. , 2012 ) . Yet humans seem to maintain a balance between optimizing for both novelty and familiarity . For example , a child in a play room does not just try to toss their toys on the floor in every possible pattern , or immediately put them away in the toy box , but instead tries to stack them together , find new uses for parts , or combine them in various structured ways . We argue that an effective unsupervised RL method should find the right balance between exploration and control . With this goal in mind , we introduce a new algorithm based on an adversarial game between two policies , which take turns sequentially acting for the same RL agent . The goal of the Control policy is to minimize surprise , by learning to manipulate its environment in order to return to safe and predictable states . In turn , the Explore policy is novelty-seeking , and attempts to maximize surprise for the Control policy , putting it into a diverse range of novel states . When combined , the two adversaries engage in an arms race , repeatedly putting the agent into challenging new situations , then attempting to gain control of those situations . Figure 8 shows an illustration of the method , including a sample interaction . Rather than simply adding noise to the environment , the Explore policy learns to adapt to the Control policy , and to search for increasingly challenging situations from which the Control policy must recover . Thus our method , Adversarial Surprise ( AS ) , leverages the power of multi-agent training to generate a curriculum of increasingly challenging exploration and control problems , leading to the emergence of complex , meaningful behaviours . The contributions of this paper are : i ) The Adversarial Surprise algorithm ; ii ) Theoretical results which prove that when the environment is formulated as a stochastic Block MDP ( Du et al. , 2019 ) , traditional surprise-maximizing methods will fail to fully explore the underlying state space , but Adversarial Surprise will succeed ; iii ) Empirical evidence which supports the theoretical results , and shows that AS fully explores the state space of both Block MDPs and traditional benchmarking environments like VizDoom ; iv ) Experiments that compare AS to state-of-the-art unsupervised RL baselines Random Network Distillation ( RND ) ( Burda et al. , 2018 ) , Asymmetric Self-Play ( ASP ) ( Sukhbaatar et al. , 2017 ) , Adversarially Guided Actor-Critic ( AGAC ) ( Flet-Berliac et al. , 2021 ) , and Surprise Minimizing RL ( SMiRL ) ( Berseth et al. , 2021 ) , and show that AS is able to explore more effectively , learn more meaningful behaviors , and achieve higher task reward when transferred zero-shot to common benchmarking environments Atari and VizDoom . Videos of our agents are available on the project website : https : //sites.google.com/corp/view/ adversarialsurprise/home , and show that AS is able to learn interesting , emergent behaviors in Atari , VizDoom , and MiniGrid , even without ever having trained with the game reward . 2 RELATED WORK . Novelty-seeking and exploration methods lead the agent to increase coverage of the environment . A simple way to implement novelty-seeking is to maximize the prediction error of a world model ( Achiam & Sastry , 2017 ; Schmidhuber , 1991 ; Yamamoto & Ishikawa , 2010 ; Pathak et al. , 2017 ; Burda et al. , 2018 ; Raileanu & Rocktäschel , 2020 ; Zhang et al. , 2020b ) . Random Network Distillation ( RND ) ( Burda et al. , 2018 ) is a highly effective example of one such method . However , a major problem of prediction-error-based methods is that the intrinsic reward is not useful when the envi- ronment contains aleatoric uncertainty , i.e . inherently stochastic elements . This problem is often referred to as the noisy TV problem , after Schmidhuber ( 2010 ) used the example of an agent becoming stuck staring at static on a TV screen . We show in this work that RND is indeed vulnerable to this problem , performing poorly in stochastic environments . Several works have proposed solutions to deal with aleatoric uncertainty . For example , some approximate information gain on the agent ’ s dynamics model of the environment ( Houthooft et al. , 2016 ; Still & Precup , 2012 ; Bellemare et al. , 2016 ; Pathak et al. , 2017 ; Schmidhuber , 1991 ) , using variational Bayes or ensemble disagreement ( Shyam et al. , 2019 ; Pathak et al. , 2019 ; Houthooft et al. , 2016 ) . However , implementing such Bayesian procedures is difficult , because it requires scalable and effective modeling of epistemic uncertainty , which itself is a major open problem with high-dimensional models such as neural networks ( Bhattacharya & Maiti , 2020 ) . Another method based on maximizing information gain between the agent ’ s actions and future state is known as empowerment ( Klyubin et al. , 2005 ; Salge et al. , 2014 ; Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) , but can also be difficult to approximate for high-dimensional states ( Karl et al. , 2015 ; Zhao et al. , 2020 ; de Abril & Kanai , 2018 ; Zhang et al. , 2020a ; Mohamed & Rezende , 2015 ; Gregor et al. , 2016 ; Hansen et al. , 2019 ) . Instead of attempting to directly approximate information gain , our approach maximizes state coverage via an adversarial competition over observation entropy . As we will show , this recovers an effective coverage strategy even in the presence of rich , stochastic observations , and performs well in practice . The asymptotic policy learned by standard novelty-seeking methods is not exploratory . Recent work tries to learn a policy that approximately maximizes the state marginal entropy at convergence ( Hazan et al. , 2019 ; Lee et al. , 2019 ) . The state marginal entropy is hard to compute in general , and recent work has proposed various approximations ( Seo et al. , 2021 ; Liu & Abbeel , 2021 ; Mutti et al. , 2021 ) . We prove that even with only noisy observations of the underlying state , our method asymptotically maximizes the state marginal entropy at convergence under some assumptions . Surprise minimization and active inference : The design of the Control agent in our method draws on ideas from surprise minimization and active inference ( Friston , 2009 ; Friston et al. , 2009 ; 2016 ) . The free energy principle , originating in the neuroscience community , argues that complex nicheseeking behaviors of biological systems are the result of minimizing long-term average surprise on system sensors , leading agents to stay in safe and stable states ( Friston , 2009 ) . SMiRL is an unsupervised RL method that leverages surprise minimization as a means to discover skills that stabilize an otherwise chaotic environment ( Berseth et al. , 2021 ) . However , such approaches require strong assumptions on the stochasticity of the environment . In low-entropy environments , surprise minimization will not lead to learning interesting behavior due to the dark room problem , in which the agent is not incentivized to explore the environment to find a better niche ( Friston et al. , 2012 ) . Our method does not require that the environment is stochastic , since the Explore agent itself drives the Control agent into situations from which surprise minimization is challenging . Multi-Agent competition has been shown to drive the emergence of complex behavior ( Baker et al. , 2019 ; Dennis et al. , 2020 ; Xu et al. , 2020 ; Leibo et al. , 2019 ; Schmidhuber , 1997 ; Campero et al. , 2020 ) . Asymmetric Self-Play ( ASP ) aims to learn increasingly complex skills via a competition between two policies , where one policy ( Bob ) tries to imitate or reverse the trajectory of the other policy ( Alice ) ( Sukhbaatar et al. , 2017 ; OpenAI et al. , 2021 ) . Our empirical results compare to ASP , and demonstrate that it can fail in stochastic environments , because Alice can easily produce random trajectories which are very difficult for Bob to imitate . Similar to ASP , Adversarially-Guided Actor Critic ( AGAC ) ( Flet-Berliac et al. , 2021 ) introduces an adversary agent which attempts to mimic the action distribution of the actor , while the actor tries to differentiate itself from the adversary . Like these methods , Adversarial Surprise uses a two-player game to induce exploration and emergent complexity . However , our objective is general and information-theoretic , focusing on minimization or maximization of surprise rather than reaching specific states . Unlike these methods , we provide theoretical results showing AS provides a principled approach to maximizing state coverage in stochastic environments . Finally , our method is reminiscent of recent work that learns separate exploration and exploitation policies but without competition between them ( Badia et al. , 2020b ; a ; Campos et al. , 2021 )

Summary: This paper introduces Adversarial Surprise, a new approach for unsupervised reinforcement learning in stochastic BMDPs, where the goal is to explore an environment without rewards. The algorithm uses a single agent with two policies, an Explorer and a Controller, which switch during an episode with opposite rewards: to maximize and minimize "surprise". The method is supported by a theoretical argument under the assumptions of the stochastic BMDP. When these are present in an environment, the empirical results are strong. The method is also tested in Atari and Doom.