SMiRL: Surprise Minimizing RL in Entropic Environments

1 INTRODUCTION . The general struggle for existence of animate beings is not a struggle for raw materials , nor for energy , but a struggle for negative entropy . ( Ludwig Boltzmann , 1886 ) All living organisms carve out environmental niches within which they can maintain relative predictability amidst the ever-increasing entropy around them ( Boltzmann , 1886 ; Schrödinger , 1944 ; Schneider & Kay , 1994 ; Friston , 2009 ) . Humans , for example , go to great lengths to shield themselves from surprise — we band together in millions to build cities with homes , supplying water , food , gas , and electricity to control the deterioration of our bodies and living spaces amidst heat and cold , wind and storm . The need to discover and maintain such surprise-free equilibria has driven great resourcefulness and skill in organisms across very diverse natural habitats . Motivated by this , we ask : could the motive of preserving order amidst chaos guide the automatic acquisition of useful behaviors in artificial agents ? Our method therefore addresses the unsupervised reinforcement learning problem : how might an agent in an environment acquire complex behaviors and skills with no external supervision ? This central problem in artificial intelligence has evoked several candidate solutions , largely focusing on novelty-seeking behaviors ( Schmidhuber , 1991 ; Lehman & Stanley , 2011 ; Still & Precup , 2012 ; Bellemare et al. , 2016 ; Houthooft et al. , 2016 ; Pathak et al. , 2017 ) . In simulated worlds , such as video games , novelty-seeking intrinsic motivation can lead to interesting and meaningful behavior . However , we argue that these sterile environments are fundamentally lacking compared to the real world . In the real world , natural forces and other agents offer bountiful novelty . The second law of thermodynamics stipulates ever-increasing entropy , and therefore perpetual novelty , without even requiring any agent intervention . Instead , the challenge in natural environments is homeostasis : discovering behaviors that enable agents to maintain an equilibrium , for example to preserve their bodies , their homes , and avoid predators and hunger . Even novelty seeking behaviors may emerge naturally as a means to maintain homeostasis : an agent that is curious and forages for food in unlikely places might better satisfy its hunger . We formalize allostasis as an objective for reinforcement learning based on surprise minimization ( SMiRL ) . In highly entropic and dynamic environments with undesirable forms of novelty , minimizing surprise ( i.e. , minimizing novelty ) causes agents to naturally seek a stable equilibrium . Natural environments with winds , earthquakes , adversaries , and other disruptions already offer a steady stream of novel stimuli , and an agent that minimizes surprise in these environments will act and explore in order to find the means to maintain a stable equilibrium in the face of these disturbances . SMiRL is simple to describe and implement : it works by maintaining a density p ( s ) of visited states and training a policy to act such that future states have high likelihood under p ( s ) . This interaction scheme is shown in Figure 1 ( right ) Across many different environments , with varied disruptive forces , and in agents with diverse embodiments and action spaces , we show that this simple approach induces useful equilibrium-seeking behaviors . We show that SMiRL agents can solve Tetris , avoid fireballs in Doom , and enable a simulated humanoid to balance and locomote , without any explicit task reward . More pragmatically , we show that SMiRL can be used together with a task reward to accelerate standard reinforcement learning in dynamic environments , and can provide a simple mechanism for imitation learning . SMiRL holds promise for a new kind of unsupervised RL method that produces behaviors that are closely tied to the prevailing disruptive forces , adversaries , and other sources of entropy in the environment . Videos of our results are available at https : //sites.google.com/view/surpriseminimization 2 SURPRISE MINIMIZING AGENTS . We propose surprise minimization as a means to operationalize the idea of learning useful behaviors by seeking to preserve order amidst chaos . In complex natural environments with disruptive forces that tend to naturally increase entropy , which we refer to as entropic environments , minimizing surprise over an agent ’ s lifetime requires taking action to reach stable states , and often requires acting continually to maintain homeostasis and avoid surprise . The long term effects of actions on the agent ’ s surprise can be complex and somewhat counterintuitive , especially when we consider that actions not only change the state that the agent is in , but also its beliefs about which states are more likely . The combination of these two processes induce the agent to not only seek states where p ( s ) is large , but to also visit states so as to alter p ( s ) , in order to receive larger rewards in the future . This “ meta ” level reasoning can result in behaviors where the agent might actually visit new states in order to make them more familiar . An example of this is shown in Figure 1 where in order to avoid the disruptions from the changing weather an agent needs to build a shelter or home to protect itself and decrease its observable surprise . The SMiRL formulation relies on disruptive forces in the environment to avoid collapse to degenerate solutions , such as staying in a single state s0 . Fortunately , natural environments typically offer no shortage of such disruption . 2.1 SURPRISE MINIMIZATION PROBLEM STATEMENT . To instantiate SMiRL , we design a reinforcement learning agent with a reward proportional to how familiar its current state is based on the history of states it has experienced during its “ life , ” which corresponds to a single episode . Formally , we assume a fully-observed controlled Markov process ( CMP ) , though extensions to partially observed settings can also be developed . We use st to denote the state at time t , and at to denote the agent ’ s action , ρ ( s0 ) to denote the initial state distribution , and T ( st+1|st , at ) to denote the transition dynamics . The agent has access to a datasetDt = { s1 , . . . , st } of all states experienced so far . By fitting a generative model pθt ( s ) with parameters θt to this dataset , the agent obtains an estimator that can be used to evaluate the negative surprise reward , given by rt ( s ) = log pθt ( s ) ( 1 ) We denote the fitting process as θt = U ( Dt ) . The goal of a SMiRL agent is to maximize the sum∑ t log pθt ( st+1 ) . Since the agent ’ s actions affect the future Dt and thus the future θt ’ s , the optimal policy does not simply visit states that have a high pθt ( s ) now , but rather those states that will change pθt ( s ) such that it provides high likelihood to the states that it sees in the future . 2.2 TRAINING SMIRL AGENTS . Algorithm 1 Training a SMiRL agent with RL 1 : Initialize policy parameters φ 2 : Initialize RL algorithm RL 3 : for each episode = 1 , 2 , . . . do 4 : s0 ∼ ρ ( s0 ) . Initial state distribution . 5 : D0 ← { s0 } . Reset state history . 6 : for each t = 0 , 1 , . . . , T do 7 : θt ← U ( Dt ) . Fit density model . 8 : at ∼ πφ ( at|st , θt , t ) . Run policy . 9 : st+1 ∼ T ( st+1|st , at ) . Transition dynamics . 10 : rt ← log pθt ( st+1 ) . Familiarity reward . 11 : Dt+1 ← Dt ∪ { st+1 } . Update state history . 12 : end for each 13 : φ← RL ( φ , s [ 0 : T ] , θ [ 0 : T ] , |D| [ 0 : T ] , a [ 0 : T ] , r [ 0 : T ] ) 14 : end for each We now present a practical reinforcement learning algorithm for surprise minimization . Recall that a critical component of SMiRL is reasoning about the effect of actions on future states that will be added to D , and their effect on future density estimates – e.g. , to understand that visiting a state that is currently unfamiliar and staying there will make that state familiar , and therefore lead to higher rewards in the long run . This means that the agent must reason not only about the unknown MDP dynamics , but also the dynamics of the density model pθ ( s ) trained on D. In our algorithm , we accomplish this via an episodic training procedure , where the agent is trained over many episodes and D is reset at the beginning of each episode to simulate a new lifetime . Through this procedure , SMiRL learns the parameters φ of the agent ’ s policy πφ for a fixed horizon . To learn this the policy must be conditioned on some sufficient statistic of Dt , since the reward rt is a function of Dt . Having trained parameterized generative models pθt as above on all states seen so far , we condition π on θt and |Dt| . This implies an assumption that θt and |Dt| represent the sufficient statistics necessary to summarize the contents of the dataset for the policy , and contain all information required to reason about how pθ will evolve in the future . Of course , we could also use any other summary statistic , or even read in the entirety of Dt using a recurrent model . In the next section , we also describe a modification that allows us to utilize a deep density model without conditioning π on a high-dimensional parameter vector . Algorithm 1 provides the pseudocode . SMiRL can be used with any reinforcement learning algorithm , which we denote RL in the pseudocode . As is standard in reinforcement learning , we alternate between sampling episodes from the policy ( lines 6-12 ) and updating the policy parameters ( line 13 ) . The details of the updates are left to the specific RL algorithm , which may be on or off-policy . During each episode , as shown in line 11 , D0 is initialized with the first state and grows as each state visited by the agent is added to the dataset . The parameters θt of the density model are fit to Dt at each timestep to both be passed to the policy and define the reward function . At the end of the episode , DT is discarded and the new D0 is initialized . 2.3 STATE DENSITY ESTIMATION WITH LEARNED REPRESENTATIONS . While SMiRL may in principle be used with any choice of model class for the generative model pθ ( s ) , this choice must be carefully made in practice . As we show in our experiments , relatively simple distribution classes , such as products of independent marginals , suffice to run SMiRL in simple environments with low-dimensional state spaces . However , it may be desirable in more complex environments to use more sophisticated density estimators , especially when learning directly from high-dimensional observations such as images . In particular , we propose to use variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) to learn a non-linear compressed state representation and facilitate estimation of pθ ( s ) for SMiRL . A VAE is trained using the standard loss to reconstruct states s after encoding them into a low-dimensional normal distribution qω ( z|s ) through the encoder q with parameters ω . A decoder pψ ( s|z , ) with parameters ψ computes s from the encoder output z . During this training process , a KL divergence loss between the prior p ( z ) and qω ( z|s ) is used to keep this distribution near the standard normal distribution . We described a VAE-based approach for estimating the SMiRL surprise reward . In our implementation , the VAE is trained online , with VAE updates interleaved with RL updates . Training a VAE requires more data than the simpler density models that can easily be fit to data from individual episodes . We propose to overcome this by not resetting the VAE parameters between training episodes . Instead , we train the VAE across episodes . Instead of passing all VAE parameters to the SMiRL policy , we track a separate episode-specific distribution pθt ( z ) , distinct from the VAE prior , over the course of each episode . pθt ( z ) replaces pθt ( s ) in the SMiRL algorithm and is fit to only that episode ’ s state history . We represent pθt ( z ) as a vector of independent normal distributions , and fit it to the VAE encoder outputs . This replaces the density estimate in line 10 of Algorithm 1 . Specifically , the corresponding update U ( Dt ) is performed as follows : z0 , . . . , zt = E [ qω ( z|s ) ] for s ∈ Dt µ = ∑t j=0 zj t+ 1 , σ = ∑t j=0 ( µ− zj ) 2 t+ 1 θt = [ µ , σ ] . Training the VAE online , over all previously seen data , deviates from the recipe in the previous section , where the density model was only updated within an episode . However , this does provide for a much richer state density model , and the within-episode updates to estimate pθt ( z ) still provide our method with meaningful surprise-seeking behavior . As we show in our experiments , this can improve the performance of SMiRL in practice .

Summary: This paper proposes a novel form of surprise-minimizing intrinsic reward signal that leads to interesting behavior in the absence of an external reward signal. The proposed approach encourages an agent to visit states with high probability / density under a parametric marginal state distribution that is learned as the agent interacts with its environment. The method (dubbed SMiRL) is evaluated in visual and proprioceptive high-dimensional "entropic" benchmarks (that progress without the agent doing anything in order to prevent trivial solutions such as standing and never moving), and compared against two surprise-maximizing intrinsic motivation methods (ICM and RND) as well as to a reward-maximizing oracle. The experiments demonstrate that SMiRL can lead to more sensible behavior compared to ICM and RND in the chosen environments, and eventually recover the performance of a purely reward-maximizing agent. Also, SMiRL can be used for imitation learning by pre-training the parametric state distribution with data from a teacher. Finally, SMiRL shows the potential of speeding up reinforcement learning by using intrinsic motivation as an additional reward signal added to the external task-defining reward.