SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments

1 INTRODUCTION . Organisms can carve out environmental niches within which they can maintain relative predictability amidst the entropy around them ( Boltzmann , 1886 ; Schrödinger , 1944 ; Schneider & Kay , 1994 ; Friston , 2009 ) . For example , humans go to great lengths to shield themselves from surprise — we band together to build cities with homes , supplying water , food , gas , and electricity to control the deterioration of our bodies and living spaces amidst heat , cold , wind and storm . These activities exercise sophisticated control over the environment , which makes the environment more predictable and less “ surprising ” ( Friston , 2009 ; Friston et al. , 2009 ) . Could the motive of preserving order guide the automatic acquisition of useful behaviors in artificial agents ? We study this question in the context of unsupervised reinforcement learning , which deals with the problem of acquiring complex behaviors and skills with no supervision ( labels ) or incentives ( external rewards ) . Many previously proposed unsupervised reinforcement learning methods focus on noveltyseeking behaviors ( Schmidhuber , 1991 ; Lehman & Stanley , 2011 ; Still & Precup , 2012 ; Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . Such methods can lead to meaningful behavior in simulated environments , such as video games , where interesting and novel events mainly happen when the agent executes a specific and coherent pattern of behavior . However , we posit that in more realistic open-world environments , natural forces outside of the agent ’ s control already offer an excellent source of novelty : from other agents to unexpected natural forces , agents in these settings must contend with a constant stream of unexpected events . In such settings , rejecting perturbations and maintaining a steady equilibrium may pose a greater challenge than novelty seeking . Based on this observation , we devise an algorithm , surprise minimizing reinforcement learning ( SMiRL ) , that specifically aims to reduce the entropy of the states visited by the agent . SMiRL maintains an estimate of the distribution of visited states , pθ ( s ) , and a policy that seeks to reach likely future states under pθ ( s ) . After each action , pθ ( s ) is updated with the new state , while the policy is conditioned on the parameters of this distribution to construct a stationary MDP . We illustrate this with a diagram in Figure 1a . We empirically evaluate SMiRL in a range of domains that are characterized by naturally increasing entropy , including video game environments based on Tetris and Doom , and simulated robot tasks that require controlling a humanoid robot to balance and walk . Our experiments show that , in environments that satisfy the assumptions of our method , SMiRL automatically discovers complex and coordinated behaviors without any reward signal , learning to successfully play Tetris , shoot enemies in Doom , and balance a humanoid robot at the edge of a cliff . We also show that SMiRL can provide an effective auxiliary objective when a reward signal is provided , accelerating learning in these domains substantially more effectively than pure novelty-seeking methods . Videos of our results are available online1 2 RELATED WORK . Prior work on unsupervised learning has proposed algorithms that learn without a reward function , such as empowerment ( Klyubin et al. , 2005 ; Mohamed & Jimenez Rezende , 2015 ) or intrinsic motivation ( Chentanez et al. , 2005 ; Oudeyer & Kaplan , 2009 ; Oudeyer et al. , 2007 ) . Intrinsic motivation has typically focused on encouraging novelty-seeking behaviors by maximizing model uncertainty ( Houthooft et al. , 2016 ; Still & Precup , 2012 ; Shyam et al. , 2018 ; Pathak et al. , 2019 ) , by maximizing model prediction error or improvement ( Lopes et al. , 2012 ; Pathak et al. , 2017 ) , through state visitation counts ( Bellemare et al. , 2016 ) , via surprise maximization ( Achiam & Sastry , 2017b ; Schmidhuber , 1991 ; Sun et al. , 2011 ) , and through other novelty-based reward bonuses ( Lehman & Stanley , 2011 ; Achiam & Sastry , 2017a ; Burda et al. , 2018a ; Kim et al. , 2019 ) . We do the opposite . Inspired by the free energy principle ( Friston , 2009 ; Friston et al. , 2009 ; Ueltzhöffer , 2018 ; Faraji et al. , 2018 ; Friston et al. , 2016 ) including recent methods that train policies using RL ( Tschantz et al. , 2020a ; b ; Annabi et al. , 2020 ) that encode a prior over desired observations , we instead incentivize an agent to minimize surprise over the distribution of states generated by the policy in unstable environments , and study the resulting behaviors . In such environments it is non-trivial to achieve low entropy state distributions , which we believe are more reflective of the real world . Learning progress methods that minimize model parameter entropy ( Lopes et al. , 2012 ; Kim et al. , 2020 ) avoid the issues novelty-based methods have with noisy distractors . These methods are based on learning the parameters of the dynamics where our method is learning to control the marginal state distribution . Several works aim to maximize state entropy to encourage exploration ( Lee et al. , 2019 ; Hazan et al. , 2019 ) . Our method aims to do the opposite , minimizing state entropy . Recent work connects the free energy principle , empowerment and predictive information maximization under the same framework to understand their differences ( Biehl et al. , 2018 ) . Existing work has also studied how competitive self-play and competitive , multi-agent environments can lead to complex behaviors with minimal reward information ( Silver et al. , 2017 ; Bansal et al. , 2017 ; Sukhbaatar et al. , 2017 ; Baker et al. , 2019 ; Weihs et al. , 2019 ; Chen et al. , 2020 ) . Like these works , we also consider how complex behaviors can emerge in resource-constrained environments , but instead of multi-agent competition , we utilize surprise minimization to drive the emergence of complex skills . 1https : //sites.google.com/view/surpriseminimization 3 SURPRISE MINIMIZING AGENTS . We propose surprise minimization as a means to operationalize the idea of learning useful behaviors by seeking out low entropy state distributions . The long term effects of actions on surprise can be subtle , since actions change both ( i ) the state that the agent is in , and ( ii ) its beliefs , represented by a model pθ ( s ) , about which states are likely under its current policy . SMiRL induces the agent to modify its policy π so that it encounters states s with high pθ ( s ) , as well as to seek out states that will change the model pθ ( s ) so that future states are more likely . In this section , we will first describe what we mean by unstable environments and provide the surprise minimization problem statement , and then present our practical deep reinforcement learning algorithm for learning policies that minimize surprise . Many commonly used reinforcement learning benchmark environments are stable , in the sense the agent remains in a narrow range of starting states unless it takes coordinated and purposeful actions . In such settings , unsupervised RL algorithms that seek out novelty can discover meaningful behaviors . However , many environments – including , as we argue , those that reflect properties commonly found in the real world , – are unstable , in the sense that unexpected and disruptive events naturally lead to novelty and increased state entropy even if the agent does not carry out any particularly meaningful or purposeful behavior . In unstable environments , minimizing cumulative surprise requires taking actions to reach a stable distribution of states , and then acting continually and purposefully to stay in this distribution . An example of this is illustrated in Figure 1b : the agent ’ s environment is unstable due to varied weather . If the robot builds a shelter , it will initially experience unfamiliar states , but in the long term the observations inside the shelter are more stable and less surprising than those outside . Another example is the game of Tetris ( Figure 2 ) , where the environment spawns new blocks and drops them into random configurations , unless a skilled agent takes actions to control the board . The challenge of maintaining low entropy in unstable settings forces the SMiRL agent to acquire meaningful skills . We defer a more precise definition of unstable environments to Section 4 , where we describe several unstable environments and contrast them with the static environments that are more commonly found in RL benchmark tasks . In static environments , novelty seeking methods must discover complex behaviors to increase entropy , leading to interesting behavior , while SMiRL may trivially find low entropy policies . We show that the reverse is true for unstable environments : a novelty seeking agent is satisfied with watching the environment change around it , while a surprise minimizing agent must develop meaningful skills to lower entropy . Problem statement . To instantiate SMiRL , we design a reinforcement learning agent that receives larger rewards for experiencing more familiar states , based on the history of states it has experienced during the current episode . This translates to learning a policy with the lowest state entropy . We assume a fully-observed controlled Markov process ( CMP ) , where we use st to denote the state at time t , at to denote the agent ’ s action , p ( s0 ) to denote the initial state distribution , and T ( st+1|st , at ) to denote the transition probabilities . The agent learns a policy πφ ( a|s ) , parameterized by φ . The goal is to minimize the entropy of its state marginal distribution under its current policy πφ at each time step of the episode . We can estimate this entropy by fitting an estimate of the state marginal dπφ ( st ) at each time step t , given by pθt−1 ( st ) , using the states seen so far during the episode , τt = { s1 , . . . , st } that is stationary . The sum of the entropies of the state distributions over an episode can then be estimated as T∑ t=0 H ( st ) = − T∑ t=0 Est∼dπφ ( st ) [ log d πφ ( st ) ] ≤ − T∑ t=0 Est∼dπφ ( st ) [ log pθt−1 ( st ) ] , ( 1 ) where the inequality becomes an equality if pθt−1 ( st ) accurately models d πφ ( st ) . Minimizing the right-hand side of this equation corresponds to maximizing an RL objective with rewards : r ( st ) = log pθt−1 ( st ) . ( 2 ) However , an optimal policy for solving this problem must take changes in the distribution pθt−1 ( st ) into account when selecting actions , since this distribution changes at each step . To ensure that the underlying RL optimization corresponds to a stationary and Markovian problem , we construct an augmented MDP to instantiate SMiRL in practice , which we describe in the following section . Algorithm 1 SMiRL 1 : while not converged do 2 : β ← { } . Reset experience 3 : for episode = 0 , . . . , M do 4 : s0 ∼ p ( s0 ) ; τ0 ← { s0 } . Initialize state 5 : s̄0 ← ( s0,0 , 0 ) . Initialize aug. state 6 : for each t = 0 , . . . , T do 7 : at ∼ πφ ( at|st , θt , t ) . Get action 8 : st+1 ∼ T ( st+1|st , at ) . Step dynamics 9 : rt ← log pθt ( st+1 ) . SMiRL reward 10 : τt+1←τt ∪ { st+1 } . Record state 11 : θt+1 ← U ( τt+1 ) . Fit model 12 : s̄t+1 ← { ( st+1 , θt+1 , tt+1 ) } 13 : β ← β ∪ { ( s̄t , at , rt , s̄t+1 ) } 14 : end for 15 : end for each 16 : φ← RL ( φ , β ) . Update policy 17 : end while Training SMiRL agents . In order to instantiate SMiRL , we construct an augmented MDP out of the original CMP , where the reward in Equation ( 2 ) can be expressed entirely as a function of the state . This augmented MDP has a state space that includes the original state st , as well as the sufficient statistics of pθt ( s ) . For example , if pθt ( s ) is a normal distribution with parameters θt , then ( θt , t ) – the parameters of the distribution and the number of states seen so far – represents a sufficient statistic . Note that it is possible to use other , more complicated , methods to summarize the statistics , including reading in the entirety of τt using a recurrent model . The policy conditioned on the augmented state is then given by πφ ( at|st , θt , t ) . The parameters of the sufficient statistics are updated θt =U ( τt ) using a maximum likelihood state density estimation process θt=arg max θ ∑t n=0 log pθ ( sn ) over the experience within the episode τt . When ( θt , t ) is a sufficient statistic , the update may be written as θt = U ( st , θt−1 , t − 1 ) . Specific update functions U ( τt ) used in our experiments are described in Appendix C and at the end of the section . Since the reward is given by r ( st , θt−1 , t− 1 ) = log pθt−1 ( st ) , and θt is a function of st and ( θt−1 , t− 1 ) , the resulting RL problem is fully Markovian and stationary , and as a result standard RL algorithms will converge to locally optimal solutions . Appendix D include details on the MDP dynamics . In Figure 8 , we illustrate the evolution of pθt ( s ) during an episode of the game Tetris . The pseudocode for this algorithm is presented in Algorithm 1 . Density estimation with learned representations . SMiRL may , in principle , be used with any choice of model class for the density model pθt ( s ) . As we show in our experiments , relatively simple distribution classes , such as products of independent marginals , suffice to run SMiRL in many environments . However , it may be desirable in more complex environments to use more sophisticated density estimators , especially when learning directly from high-dimensional observations such as images . In these cases , we can use variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) to learn a non-linear state representation . A VAE is trained using the standard ELBO objective to reconstruct states s after encoding them into a latent representation z via an encoder qω ( z|s ) , with parameters ω . Thus , z can be viewed as a compressed representation of the state . When using VAE representations , we train the VAE online together with the policy . This approach necessitates two changes to the procedure described Algorithm 1 . First , training a VAE requires more data than the simpler independent models , which can easily be fitted to data from individual episodes . We propose to overcome this by not resetting the VAE parameters between training episodes , and instead training the VAE across episodes . Second , instead of passing the VAE model parameters to the policy , we only update a distribution over the VAE latent state , given by pθt ( z ) , such that pθt ( z ) replaces pθt ( s ) in the SMiRL algorithm , and is fitted to only that episode ’ s ( encoded ) state history . We represent pθt ( z ) as a normal distribution with a diagonal covariance , and fit it to the VAE encoder outputs . Thus , the mean and variance of pθt ( z ) are passed to the policy at each time step , along with t. This implements the density estimate in line 9 of Algorithm 1 . The corresponding update U ( τt ) is : z0 , . . . , zt = E [ qω ( z|s ) ] for s ∈ τt , µ = 1/t+1 t∑ j=0 zj , σ = 1/t+1 t∑ j=0 ( µ− zj ) 2 , θt = [ µ , σ ] . Training the VAE online , over all previously seen data , deviates from the recipe in the previous section , where the density model was only updated within an episode . In this case the model is updated after a collection of episodes . This makes the objective for RL somewhat non-stationary and could theoretically cause issues for convergence , however we found in practice that the increased representational capacity provides significant improvement in performance .

Summary: This work proposes an RL approach SMiRL that is able to learn effective policies in unstable environments without the need for external reward. The idea at a high-level is almost the opposite of intrinsic motivation RL approaches, which encourage novelty-seeking behaviors. The proposed method instead aims to minimize surprise or state entropy. To train the agent, rewards come from state marginal estimates, but because this distribution is changing, the authors create an augmented MDP. Through experiments on game domains and robot control tasks, the authors show that SMiRL outperforms intrinsic motivation methods. The authors also show that SMiRL can be used to do imitation and can be combined with regular reward signals.