SMiRL: Surprise Minimizing RL in Entropic Environments

1 INTRODUCTION . The general struggle for existence of animate beings is not a struggle for raw materials , nor for energy , but a struggle for negative entropy . ( Ludwig Boltzmann , 1886 ) All living organisms carve out environmental niches within which they can maintain relative predictability amidst the ever-increasing entropy around them ( Boltzmann , 1886 ; Schrödinger , 1944 ; Schneider & Kay , 1994 ; Friston , 2009 ) . Humans , for example , go to great lengths to shield themselves from surprise — we band together in millions to build cities with homes , supplying water , food , gas , and electricity to control the deterioration of our bodies and living spaces amidst heat and cold , wind and storm . The need to discover and maintain such surprise-free equilibria has driven great resourcefulness and skill in organisms across very diverse natural habitats . Motivated by this , we ask : could the motive of preserving order amidst chaos guide the automatic acquisition of useful behaviors in artificial agents ? Our method therefore addresses the unsupervised reinforcement learning problem : how might an agent in an environment acquire complex behaviors and skills with no external supervision ? This central problem in artificial intelligence has evoked several candidate solutions , largely focusing on novelty-seeking behaviors ( Schmidhuber , 1991 ; Lehman & Stanley , 2011 ; Still & Precup , 2012 ; Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . In simulated worlds , such as video games , novelty-seeking intrinsic motivation can lead to interesting and meaningful behavior . However , we argue that these sterile environments are fundamentally lacking compared to the real world . In the real world , natural forces and other agents offer bountiful novelty . The second law of thermodynamics stipulates ever-increasing entropy , and therefore perpetual novelty , without even requiring any agent intervention . Instead , the challenge in natural environments is homeostasis : discovering behaviors that enable agents to maintain an equilibrium , for example to preserve their bodies , their homes , and avoid predators and hunger . Even novelty seeking behaviors may emerge naturally as a means to maintain homeostasis : an agent that is curious and forages for food in unlikely places might better satisfy its hunger . We formalize allostasis as an objective for reinforcement learning based on surprise minimization ( SMiRL ) . In highly entropic and dynamic environments with undesirable forms of novelty , minimizing surprise ( i.e. , minimizing novelty ) causes agents to naturally seek a stable equilibrium . Natural environments with winds , earthquakes , adversaries , and other disruptions already offer a steady stream of novel stimuli , and an agent that minimizes surprise in these environments will act and explore in order to find the means to maintain a stable equilibrium in the face of these disturbances . SMiRL is simple to describe and implement : it works by maintaining a density p ( s ) of visited states and training a policy to act such that future states have high likelihood under p ( s ) . This interaction scheme is shown in Figure 1 ( right ) Across many different environments , with varied disruptive forces , and in agents with diverse embodiments and action spaces , we show that this simple approach induces useful equilibrium-seeking behaviors . We show that SMiRL agents can solve Tetris , avoid fireballs in Doom , and enable a simulated humanoid to balance and locomote , without any explicit task reward . More pragmatically , we show that SMiRL can be used together with a task reward to accelerate standard reinforcement learning in dynamic environments , and can provide a simple mechanism for imitation learning . SMiRL holds promise for a new kind of unsupervised RL method that produces behaviors that are closely tied to the prevailing disruptive forces , adversaries , and other sources of entropy in the environment . Videos of our results are available at https : //sites.google.com/view/surpriseminimization 2 SURPRISE MINIMIZING AGENTS . We propose surprise minimization as a means to operationalize the idea of learning useful behaviors by seeking to preserve order amidst chaos . In complex natural environments with disruptive forces that tend to naturally increase entropy , which we refer to as entropic environments , minimizing surprise over an agent ’ s lifetime requires taking action to reach stable states , and often requires acting continually to maintain homeostasis and avoid surprise . The long term effects of actions on the agent ’ s surprise can be complex and somewhat counterintuitive , especially when we consider that actions not only change the state that the agent is in , but also its beliefs about which states are more likely . The combination of these two processes induce the agent to not only seek states where p ( s ) is large , but to also visit states so as to alter p ( s ) , in order to receive larger rewards in the future . This “ meta ” level reasoning can result in behaviors where the agent might actually visit new states in order to make them more familiar . An example of this is shown in Figure 1 where in order to avoid the disruptions from the changing weather an agent needs to build a shelter or home to protect itself and decrease its observable surprise . The SMiRL formulation relies on disruptive forces in the environment to avoid collapse to degenerate solutions , such as staying in a single state s0 . Fortunately , natural environments typically offer no shortage of such disruption . 2.1 SURPRISE MINIMIZATION PROBLEM STATEMENT . To instantiate SMiRL , we design a reinforcement learning agent with a reward proportional to how familiar its current state is based on the history of states it has experienced during its “ life , ” which corresponds to a single episode . Formally , we assume a fully-observed controlled Markov process ( CMP ) , though extensions to partially observed settings can also be developed . We use st to denote the state at time t , and at to denote the agent ’ s action , ρ ( s0 ) to denote the initial state distribution , and T ( st+1|st , at ) to denote the transition dynamics . The agent has access to a datasetDt = { s1 , . . . , st } of all states experienced so far . By fitting a generative model pθt ( s ) with parameters θt to this dataset , the agent obtains an estimator that can be used to evaluate the negative surprise reward , given by rt ( s ) = log pθt ( s ) ( 1 ) We denote the fitting process as θt = U ( Dt ) . The goal of a SMiRL agent is to maximize the sum∑ t log pθt ( st+1 ) . Since the agent ’ s actions affect the future Dt and thus the future θt ’ s , the optimal policy does not simply visit states that have a high pθt ( s ) now , but rather those states that will change pθt ( s ) such that it provides high likelihood to the states that it sees in the future . 2.2 TRAINING SMIRL AGENTS . Algorithm 1 Training a SMiRL agent with RL 1 : Initialize policy parameters φ 2 : Initialize RL algorithm RL 3 : for each episode = 1 , 2 , . . . do 4 : s0 ∼ ρ ( s0 ) . Initial state distribution . 5 : D0 ← { s0 } . Reset state history . 6 : for each t = 0 , 1 , . . . , T do 7 : θt ← U ( Dt ) . Fit density model . 8 : at ∼ πφ ( at|st , θt , t ) . Run policy . 9 : st+1 ∼ T ( st+1|st , at ) . Transition dynamics . 10 : rt ← log pθt ( st+1 ) . Familiarity reward . 11 : Dt+1 ← Dt ∪ { st+1 } . Update state history . 12 : end for each 13 : φ← RL ( φ , s [ 0 : T ] , θ [ 0 : T ] , |D| [ 0 : T ] , a [ 0 : T ] , r [ 0 : T ] ) 14 : end for each We now present a practical reinforcement learning algorithm for surprise minimization . Recall that a critical component of SMiRL is reasoning about the effect of actions on future states that will be added to D , and their effect on future density estimates – e.g. , to understand that visiting a state that is currently unfamiliar and staying there will make that state familiar , and therefore lead to higher rewards in the long run . This means that the agent must reason not only about the unknown MDP dynamics , but also the dynamics of the density model pθ ( s ) trained on D. In our algorithm , we accomplish this via an episodic training procedure , where the agent is trained over many episodes and D is reset at the beginning of each episode to simulate a new lifetime . Through this procedure , SMiRL learns the parameters φ of the agent ’ s policy πφ for a fixed horizon . To learn this the policy must be conditioned on some sufficient statistic of Dt , since the reward rt is a function of Dt . Having trained parameterized generative models pθt as above on all states seen so far , we condition π on θt and |Dt| . This implies an assumption that θt and |Dt| represent the sufficient statistics necessary to summarize the contents of the dataset for the policy , and contain all information required to reason about how pθ will evolve in the future . Of course , we could also use any other summary statistic , or even read in the entirety of Dt using a recurrent model . In the next section , we also describe a modification that allows us to utilize a deep density model without conditioning π on a high-dimensional parameter vector . Algorithm 1 provides the pseudocode . SMiRL can be used with any reinforcement learning algorithm , which we denote RL in the pseudocode . As is standard in reinforcement learning , we alternate between sampling episodes from the policy ( lines 6-12 ) and updating the policy parameters ( line 13 ) . The details of the updates are left to the specific RL algorithm , which may be on or off-policy . During each episode , as shown in line 11 , D0 is initialized with the first state and grows as each state visited by the agent is added to the dataset . The parameters θt of the density model are fit to Dt at each timestep to both be passed to the policy and define the reward function . At the end of the episode , DT is discarded and the new D0 is initialized . 2.3 STATE DENSITY ESTIMATION WITH LEARNED REPRESENTATIONS . While SMiRL may in principle be used with any choice of model class for the generative model pθ ( s ) , this choice must be carefully made in practice . As we show in our experiments , relatively simple distribution classes , such as products of independent marginals , suffice to run SMiRL in simple environments with low-dimensional state spaces . However , it may be desirable in more complex environments to use more sophisticated density estimators , especially when learning directly from high-dimensional observations such as images . In particular , we propose to use variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) to learn a non-linear compressed state representation and facilitate estimation of pθ ( s ) for SMiRL . A VAE is trained using the standard loss to reconstruct states s after encoding them into a low-dimensional normal distribution qω ( z|s ) through the encoder q with parameters ω . A decoder pψ ( s|z , ) with parameters ψ computes s from the encoder output z . During this training process , a KL divergence loss between the prior p ( z ) and qω ( z|s ) is used to keep this distribution near the standard normal distribution . We described a VAE-based approach for estimating the SMiRL surprise reward . In our implementation , the VAE is trained online , with VAE updates interleaved with RL updates . Training a VAE requires more data than the simpler density models that can easily be fit to data from individual episodes . We propose to overcome this by not resetting the VAE parameters between training episodes . Instead , we train the VAE across episodes . Instead of passing all VAE parameters to the SMiRL policy , we track a separate episode-specific distribution pθt ( z ) , distinct from the VAE prior , over the course of each episode . pθt ( z ) replaces pθt ( s ) in the SMiRL algorithm and is fit to only that episode ’ s state history . We represent pθt ( z ) as a vector of independent normal distributions , and fit it to the VAE encoder outputs . This replaces the density estimate in line 10 of Algorithm 1 . Specifically , the corresponding update U ( Dt ) is performed as follows : z0 , . . . , zt = E [ qω ( z|s ) ] for s ∈ Dt µ = ∑t j=0 zj t+ 1 , σ = ∑t j=0 ( µ− zj ) 2 t+ 1 θt = [ µ , σ ] . Training the VAE online , over all previously seen data , deviates from the recipe in the previous section , where the density model was only updated within an episode . However , this does provide for a much richer state density model , and the within-episode updates to estimate pθt ( z ) still provide our method with meaningful surprise-seeking behavior . As we show in our experiments , this can improve the performance of SMiRL in practice .

Summary: This paper proposes Surprise Minimizing RL (SMiRL), a conceptual framework for training a reinforcement learning agent to seek out states with high likelihood under a density model trained on visited states. They qualitatively and quantitatively explore various aspects of the behaviour of these agents and argue that they exhibit a variety of favourable properties. They also compare their surprise minimizing algorithm with a variety of novelty-seeking algorithms (which can be considered somewhat the opposite) and show that in certain cases surprise minimization can result in more desirable behaviour. Finally, they show that using surprise minimization as an auxiliary reward can speed learning in certain settings.