SMiRL: Surprise Minimizing RL in Entropic Environments
Authors: Glen Berseth
Daniel Geng
Coline Devin
Dinesh Jayaraman
Chelsea Finn
Sergey Levine
Venue:  (2020)
Keywords: intrinsic motivation
reinforcement learning
unsurpervised RL

Abstract: All living organisms struggle against the forces of nature to carve out niches where
they can maintain relative stasis. We propose that such a search for order amidst
chaos might offer a unifying principle for the emergence of useful behaviors in
artificial agents. We formalize this idea into an unsupervised reinforcement learning
method called surprise minimizing RL (SMiRL). SMiRL trains an agent with the
objective of maximizing the probability of observed states under a model trained on
all previously seen states. The resulting agents acquire several proactive behaviors
to seek and maintain stable states such as balancing and damage avoidance, that
are closely tied to the affordances of the environment and its prevailing sources
of entropy, such as winds, earthquakes, and other agents.  We demonstrate that
our surprise minimizing agents can successfully play Tetris, Doom, and control
a  humanoid  to  avoid  falls,  without  any  task-specific  reward  supervision.   We
further show that SMiRL can be used as an unsupervised pre-training objective
that substantially accelerates subsequent reward-driven learning

Under review as a conference paper at ICLR 2020
SMIRL: SURPRISE MINIMIZING RL IN DYNAMIC ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
All living organisms struggle against the forces of nature to carve out niches where they can maintain relative stasis. We propose that such a search for order amidst chaos might offer a unifying principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing RL (SMiRL). SMiRL trains an agent with the objective of maximizing the probability of observed states under model trained on all previously seen states. The resulting agents acquire several proactive behaviors to seek and maintain stable states such as balancing and damage avoidance, that are closely tied to the environment's ' prevailing sources of entropy, such winds, earthquakes, and other agents. We demonstrate that our surprise minimizing agents successfully play Tetris, Doom, and control humanoid to avoid falls, without any task- task-specific reward supervision. We further show that SMiRL can be used together with standard task reward accelerate reward-driven learning.
INTRODUCTION
The general struggle for existence of animate beings is not struggle for raw materials, nor for energy, but a struggle for negative entropy. (Ludwig Boltzmann, 1886)
All living organisms carve out environmental niches within which they can maintain relative pre- dictability amidst the ever- ever-incread increascc entropy around them (Boltzmann 1886 Schrödinger T944 Schneider & Kay, 1994] Friston, 2009). Humans, for example, go great lengths shield themselves band together in millions to build cities with homes, supplying water, food, gas, and electricity to control the deterioration of our bodies and living spaces amidst heat and cold, wind and storm. The need to discover and maintain such surprise-free frree equilibria has driven great resourcefulness and skill in organisms across very diverse natural habitats. Motivated by this, we ask: could the motive of preserving order amidst chaos guide the automatic acquisition of useful behaviors in artificial agents?
from surprise
Our method therefore addresses the unsupervised reinforcement learning problem: how might an agent in an environment acquire complex behaviors and skills with no external supervision? This central problem in artificial intelligence has evoked several candidate solutions, largely focusing
on novelty-seeking behaviors (Schmidhuberi 1991 Lehman & Stanley 2011 Still & Precup 2012
Lehman
Bellemare et al.] 2016 Houthooft et al. [0166] Pathak al.) 2017p. In simulated worlds, such as video games, sevelty-sekking intrinsic motivation can lead to interesting and meaningful behavior.
However, we argue that these sterile environments are fundamentally lacking compared the real world. In the real world, natural forces and other agents offer bountiful novelty. The second law of thermodynamics stipulates ever-increasing entropy, and therefore perpetual novelty, without even requiring any agent intervention. Instead, the challenge in natural environments is homeostasis: discovering behaviors that enable agents to maintain an equilibrium, for example to preserve their bodies, their homes, and avoid predators and hunger. Even novelty seeking behaviors may emerge naturally as a means to maintain homeostasis: an agent that is curious and forages for food in unlikely places might better satisfy its hunger.

---

Under review as a conference paper at ICLR 2020
St
p(s)0)
Cat
91
are mare Σ log poge)(
Satals DJ
B
8
at
org mart
Figure I: In natural environments (left), an inactive agent will experience a wide variety of states. By reasoning about future surprise. a SMiRL agent can take actions that lemporarily increase surprise but reduce it in the long term. For example. building house initially results in novel states, but once it built, the house allows the agent to experience more stable and surprise-fre environment. On the right we show an interpretation of the agent interaction loop using SMiRL. When the agent observes a state, it updates it belicf p(s) over states. Then, the action policy x (s 0) is conditioned on this belief and maximizes the expected likelihood of the next state under its belicf.
We formalize allostasis an objective for reinforcement learning based surprise minimization (SMiRL). In highly entropic and dynamic environments with undesirable forms of novelty, minimizing surprise (i.e., minimizing novelty) causes agents to naturally seek stable equilibrium. Natural environments with winds, earthquakes, adversaries, and other disruptions already offer a steady stream of novel stimuli, and agent that minimizes surprise in these environments will act and explore in order to find the means to maintain a stable equilibrium in the face of these disturbances.
SMiRL is simple to describe and implement: it works by maintaining density p(s) of visited states and training a policy to act such that future states have high likelihood under p(s). This interaction scheme is shown in Figure rightt Across many different environments, with varied disruptive forces, and in agents with diverse embodiments and action spaces, we show that this simple approach induces useful equilibrium-seeking behaviors. We show that SMiRL agents can solve Tetris, avoid fireballs in Doom, and enable simulated humanoid to balance and locomote, without any explicit task reward. More pragmatically, we show that SMiRL can be used together with a task reward to accelerate standard reinforcement learning in dynamic environments, and can provide simple mechanism for imitation learning. SMiRL. holds promise for a new kind of unsupervised RI. method that produces behaviors that are closely tied to the prevailing disruptive forces, adversarics, and other sources of entropy in the environment. Videos of our results are available at https::/siiee: http://ssseee..glllllleeeeo wwt.coo//viiiiee http////wwwwrrrreeiiiiiiieeeeeee zation
2 SURPRISE MINIMIZING AGENTS
We propose surprise minimization as a means to operationalize the idea of learning useful behaviors by seeking preserve order amidst chaos. In complex natural environments with disruptive forces that tend to naturally increase entropy, which we refer to as entropic environments, minimizing surprise over an agent's lifetime requires taking action to reach stable states, and often requires acting continually maintain homeostasis and avoid surprise. The long term effects of actions the agent's 's surprise can be complex and somewhat counterintuitive, especially when we consider that actions only change the state that the agent is in, but also its beliefs about which states are more likely. The combination of these two processes induce the agent to not only seek states where p(s) large, but also visit states so as alter p(s), in order receive larger rewards in the future. This "meta" level reasoning can result in behaviors where the agent might actually visit new states in order make them more familiar. An example of this shown in Figure I where in order to avoid the disruptions from the changing weather an agent needs to build a shelter or home to protect itself and decrease its observable surprise. The SMiRL formulation relies on disruptive forces in the environment to avoid collapse to degenerate solutions, such as staying in a single state So. Fortunately, natural environments typically offer no shortage of such disruption.
2.1 SURPRISE MINIMIZATION PROBLEM STATEMENT
To instantiate SMiRL, we design reinforcement learning agent with reward proportional to how familiar its current state is based on the history of states it has experienced during its "life," which corresponds to single episode. Formally, assume fully-observed controlled Markov process
2

---

Under review as a conference paper at ICLR 2020
(CMP), though extensions to partially observed settings can also be developed. We use Sy to denote the state at time t, and ay to denote the agent's 's action, p(5b) to denote the initial state distribution, and T(s++1 1 (ss, a+) to denote the transition dynamics. The agent has access to dataset D, ........ 8,} of all states experienced so far. By fitting generative model po, (s) with parameters 0₂ to this dataset, the agent obtains estimator that can be used evaluate the negative surprise reward, given by
(r(s)
log Po, (s)
We denote the fitting process as 02 - (D,). The goal of SMiRL agent is to maximize the sum Eg log po. ($)+1). Since the agent's actions affect the future Dj and thus the future 8, 's the optimal policy does not simply visit states that have a high pe, (s) now, but rather those states that will change po, (s) such that it provides high likelihood to the states that it sees in the future.
2.2 TRAINING SMIRL AGENTS
We now present a practical reinforce- ment learning algorithm for surprise minimization. Recall that a critical component of SMiRL is reasoning about the effect of actions on future states that will be added D, and their effect on future density estimates -e.g., understand that visiting a state that is currently unfamiliar and stay- ing there will make that state familiar,
Algorithm Training SMiRL agent with RL
1: Initialize policy parameters &
2: Initialize RL algorithm RL
3: for each episode = 1, 2,2,... ... do
4: Sq ~ p(5o)
b Initial state distribution.
5:
Do <- {so}
Reset state history.
6:
for each = 0, ...,, T do
7:
02 + H(D,)
Fit density model.
8:
ag ~ n (ax St, B1,
D Run policy. Transition dynamics.
and therefore lead to higher rewards 9:
S1+1 T( S, a1)
in the long run. This means that the agent must reason not only about the unknown MDP dynamics, but also the dynamics of the density model (s(s) trained on D. In our algorithm, we ac- complish this via episodic training procedure, where the agent is trained over many episodes and 'D is reset at the beginning of each episode simulate new lifetime. Through this procedure, SMiRL learns the parameters e of the agent's policy To for fixed horizon. To learn this the policy must be conditioned on some sufficient statistic of D., since the reward Ft is function of D,,
10:
Ti + log (8++1)
b Familiarity reward.
11:
D++1 + +D, {8441)
Update state history.
12:
end for each
13:
Φ + RL(, Sto:T), 0 (0:T) [D] [o:7]
14:
end each
Having trained parameterized generative models pa, as above on all states seen so far, we condition on Og and [De] This implies an assumption that 0g and [De] represent the sufficient statistics necessary to summarize the contents of the dataset for the policy, and contain all information required to reason about how Po will evolve in the future. Of course, we could also use any other summary statistic, or even read in the entirety of 'D using recurrent model. In the next section, we also describe a modification that allows us to utilize a deep density model without conditioning x on a high-dimensional parameter vector.
Algorithm| provides the pseudocode. SMiRL can be used with any reinforcement learning algorithm, which we denote RL in the pseudocode. As is standard in reinforcement learning, we alternate between sampling episodes from the policy (lines 6-12) and updating the policy parameters (line 13). The details of the updates are left to the specific RL algorithm, which may be on off off-policy. During each episode, as shown in line 11, Do is initialized with the first state and grows as each state visited by the agent is added to the dataset. The parameters 0, of the density model are fit to D, each timestep to both be passed to the policy and define the reward function. At the end of the episode, Dr is discarded and the new Do is initialized.
2.3 STATE DENSITY ESTIMATION WITH LEARNED REPRESENTATIONS
While SMiRL may in principle be used with any choice of model class for the generative model (s).. this choice must be carefully made in practice. As show in our experiments, relatively simple distribution classes, such as products of independent marginals, suffice to run SMiRL in simple environments with dow-dimensional state spaces. However, it may be desirable in more complex
3

---

Under review as a conference paper at ICLR 2020
environments to more sophisticated density estimators, especially when learning directly from high-dimensional observations such as images.
In particular, propose to variational autoencoders (VAEs) Kingma & Welling 2014) to learn a non-linear compressed state representation and facilitate estimation of Po(s) for SMiRL. A VAE is trained using the standard loss reconstruct states after encoding them into low oow-dimensional normal distribution qu (z|s) through the encoder with parameters w. A decoder (s)z,, 22 with parameters computes from the encoder output z. During this training process, KL divergence loss between the prior p(z) and qu (z|s) is used to keep this distribution near the standard normal distribution. We described VAE-based approach for estimating the SMiRL surprise reward. In our implementation, the VAE is trained online, with VAE updates interleaved with RL updates. Training a VAE requires more data than the simpler density models that can easily be fit to data from individual episodes. We propose to overcome this by not resetting the VAE parameters between training episodes. Instead, we train the VAE across episodes. Instead of passing all VAE parameters to the SMiRL policy, we track separate episode-speciff distribution pe, (z), distinct from the VAE prior, over the course of each episode. Pe, (z) replaces Po. (s) in the SMiRL algorithm and is fit to only that episode's 's state history. We represent po, (z) vector of independent normal distributions:
and fit it to the VAE encoder outputs. This replaces the density estimate in line 10 of Algorithm Specifically, the corresponding update (D:) is performed as follows:
20: ................................ 2z
T2 gg [z|s) for Dr
Σj
2
o 23)²
11
1+1
1+1
0. [u,
Training the VAE online, over all previously seen data, deviates from the recipe in the previous section, where the density model was only updated within an episode. However, this does provide for a much richer state density model, and the within-episode updates to estimate Po, (z) still provide our method with meaningful surprise-seeking behavior. As show in experiments, this can improve the performance of SMiRL in practice.
3 ENVIRONMENTS
We evaluate SMiRL on a range of environments, from video game domains to simulated robotic control scenarios. These are rich, dynamic environments - the world evolves automatically even without agent intervention due to the presence of disruptive forees and adversaries. Note that SMiRL relies on such disruptions to produce meaningful emergent behavior, since mere inaction would otherwise suffice to achieve homeostasis. However, as we have argued above, such disruptions are also an important property of most real world environments. Current RL benchmarks neglect this, focusing largely on unrealistically sterile environments where the agent alone drives change (Bellemare et al., 2015 Brockman et al., 2016). Therefore, our choices of environments, discussed below, are not solely motivated by suitability to SMiRL; rather, we aim to evaluate unsupervised RL. approaches, ours as well as others, in these more dynamic environments.
Tetris. The classic game of Tetris offers a naturally entropic environment - the world evolves according its own rules and dynamics even in the absence of coordinated behavior of the agent, piling up pieces and filling up the board. It therefore requires active intervention to maintain homeostasis. We consider 4 10 Tetris board with tromino shapes (composed of 3 squares), as shown in Figure 2a The observation is binary image of the current board with one pixel per square, as well as an indicator for the type of shape that will appear next. Each action denotes one of the columns in which to drop the shape and one of 4 shape orientations. For evaluation, we measure how many rows the agent clears, as well as how many times the agent dies in the game by allowing the blocks to reach the top of the board, within the max episode length of 100. Since the observation is binary image, we model p(s) as independent Bernoulli. See Appendix A for details.
VizDoom. We consider two VizDoom environments from Kempka et al. (0016); TakeCover and DefendTheLine. TakeCover provides a dynamically evolving world, with enemies that appear over time and throw fireballs aimed at the player (Kempka et al.) 2016). The observation space consists of the 4 previous grayscale first-person image observations, and the action space consists of moving left or right. We evaluate the agent based on how many times it is hit by fireballs, which term
4

---

Under review as a conference paper at ICLR 2020
Terris
(b) Take
(c) DefendTheLine
(d) miniGrid
(e) Humanoid Clif'
Humanoid Treadmill
(g) Humanoid Walk
(h) Pedestal
Figure 2: Illustrations of the entropic evaluation eavironments: (a) A rendering of the Tetris environment, (b) The Cover environment, with enemies throwing fireballs in the distance. (c) VizDoom DefendTheLine environment with multiple enemies. (d) is the miniGrid environment with one agent (in red) and number of enemeies (blue). The simulated Humanoid next cliff. (f) The Humanoid treadmill, (g) a Humanoid learning to walk and (h) Pedestal.
the "damage" taken by the agent. Images from the TakeCover environment shown in Fig 2b and Fig ??.
In RefendThel.ine. additional enemies can move towards the player, and the player can shoot the enemies. The agent starts with limited ammunition. This environment provides a "survival" reward function (r for each timestep alive) and performance measured by how long the agent survives in the environment. For both environments, we model p(s) as independent Gaussian over the pixels. See Appendix for details.
miniGrid Is navigation task where the agent has partial observation of the environment shown by the lighter gray area around the red agent in Figure 2d The agent needs to navigate down the hallways to escape the enemy agents (bluc) to reach the safe room on the right the enemies can not enter, through a randomly placed door.
Simulated Humanoid robots. In the last set of environments, simulated planar Humanoid robot is placed in situations where is in danger of falling. The action consists the PD targets for each of the joints. The state space comprises the rotation of each joint and the linear velocity of each link. evaluate several versions of this task, which are shown in Figure 2 The Cliff tasks starts agent the edge of cliff, in random pose and with forward velocity of m/s. Falling off the cliff leads to highly irregular and unpredictable configurations, so a surprise minimizing agent will want learn to stay the cliff. In the Treadmill environment, the robot starts platform that is moving at m/s backwards; an agent will be carried backwards unless it learns some locomotion. The Pedestal environment is designed to show that SMiRL can learn more active balancing policy. In this environment the agent starts out on a thin pedestal and random forces are applied to the robots links and boxes of random size thrown the agent. The Walk domain used evaluate the use of the SMiRL reward as form of "stability reward" that assists the agent in learning how to walk while reducing the number of falls. This done by initializing p(s) from example walking data and adding this to the task reward, as discussed in Section 4.2. The task reward in Walk is Twzlk = exp((Na * va)= 1.5), where Va the difference between the x velocity and the desired velocity of m/s. In these environments, a fall. state classified as a fall if either the agent's links, except for the feet, are touching the ground, or if the agent is -5 meters or more below the level of the platform or cliff. Since the state is continuous, we model p(s) as independent Gaussian; see Appendix A for details.
measure performance the proportion of episodes with
4 EXPERIMENTAL RESULTS
Our experiments aim to answer the following questions: (1) Can SMiRL learn meaningful and complex emergent behaviors in the environments described in Section 3 (2) Can incorporate
5

---

Under review as a conference paper at ICLR 2020
deep generative models into SMiRL and use state densities in learned representation spaces? (3) Can SMiRL serve as a joint training objective to accelerate acquisition of seward-guidd behavior, and does it outperform prior intrinsic motivation methods in this role? We also illustrate several applications of SMiRL, showing that it can accelerate task learning, provide for exploration with fewer damaging falls, and provide for elementary imitation. Videos of learned behaviors available on the website https:/ / / ssttes.tss...//ses///eess..eete. google. com/ view//iwww//////// cetuurriiemiin zat ion /home
4.1 EMERGENT BEHAVIOR IN UNSUPERVISED LEARNING
First, we evaluate SMiRL on the Tetris, VizDoom, Cliff. and Treadmill tasks, studying its ability to generate purposeful coordinated behaviors after training using only the surprise minimizing objective, in order to answer question (1). The SMiRL agent demonstrates meaningful emergent behaviors in each of these domains. In the Tetris environment, the agent is able to learn proactive behaviors to eliminate rows and properly play the game. The agent also learns emergent game playing behaviour in the VizDoom environment, acquiring an effective policy for dodging the fireballs thrown by the enemies. In both of these environments, stochastic and chaotic events force the SMiRL agent to take a coordinated course of action to avoid unusual states, such as full Tetris boards or firehall explosions. In the Cliff environment, the agent learns policy that greatly reduces the probability of falling off of the cliff by bracing against the ground and stabilize itself at the edge, as shown in Figure 2c In the Treadmill environment, SMiRL learns more complex locomotion behavior, jumping forward to increase the time it stays on the treadmill, as shown in Figure 21 A quantitative measurement of the reduction in falls is shown in Figure 4
We also study question (2) in the TakeCover, Cliff, Treadmill and Pedestal environments, training a VAE model and estimating surprise in the latent space of the VAE. In most of these environments, the representation learned by the VAE leads to faster acquisition of the emergent behaviors in Take- Cover Figure 3 (right), Cliff Figure ( (left), and Treadmill Figure 4 (middle), leads to a substantially more successful locomotion behavior.
Comparison to intrinsic motivation. Figure 3 shows plots of the environment--pecciic rewards over time on Tetris, TakeCover, and the Humanoid domains Figure In order to compare SMiRL to more standard intrinsic motivation methods, which seek out states that maximize surprise or novelty, we also evaluated ICM (Pathak et al. (017)) and RND (Burda et al. 20188). We also plot an oracle agent that directly optimizes the task reward. On Tetris, after training for 2000 epochs. SMiRL achieves near perfect play, on par with the oracle reward optimizing agent, with no deaths, as shown in ( middle). ICM seeks novelty by creating more and more distinct patterns of blocks rather than clearing them, leading to deteriorating game scores over time. On TakeCover, SMiRL effectively learns to dodge fireballs thrown by the adversaries, as shown in (right.. Novelty-seeking ICM once again yields deteriorating rewards over time due to the method seeking novel events that correspond to damage. The baseline comparisons for the Cliff and Treadmill environments have similar outcome. The novelty seeking behaviour of ICM causes it to learn a type of irregular behaviour that causes the agent to jump off the Cliff and roll around on the Treadmill, maximizing the variety (and quantity) of falls Figure
SMiRL and curiosity are not mutually exclusive. We show that these intrinsic reward functions can be combined to achieve better results on the Treadmill environment Figure | right). The combination of methods leads to increased initial learning speed and producing a walking-type gait on that task.
To illustrate SMiRL's 's desire to explore we evaluate over an environment where the agent needs to produce long term planning behaviour. This environment is shown in Fig- ure 2d where the agent needs to navigate its way through the hallways, avoiding enemies, to reach a safe room through randomly placed door. We found that SMiRL is able to solve this task. Results from these examples are shown on the accompanying website.
Exploration for SMiRL
4.2 APPLICATIONS OF SMIRL
While the central focus of this paper is the emergent behaviors that can be obtained via SMiRL, in this section we study more pragmatic applications. We show that SMiRL can be used for joint training to accelerate reward-dri -rrvve learning of tasks, and also illustrate how SMiRL can be used to produce rudimentary form of imitation learning.

---

Under review as a conference paper at ICLR 2020
Tetrs Deaths
"etris Rows Cleaned
ViZDcom Takan Cover Total Damaga
400
350
Soee Cherred
300
200
86
28
600
18
502
3030
2000
2300
3003
12:00
2200
2000
2500
300
200
400
300
500
Epocdda
Eceandes
52 5o-cc
SHR0L Seurca
SMMRL VMG Dacm)
160
Orada
Figure 3: Results for video game environments: Comparison between SMiRL, ICM, RND, and an oracle RL algorithm with access to the true reward in Tetris on (left) number of deaths per episode (lower is better). (center) number of rows cleared per episode (higher is better), and (right) in TakeCover taken (lower is better). The SMiRL agent is able to learn how to play Tetris and avoid fireballs in Take Tover almost as well as an agent trained on the task reward. Using VAE features for the density model (SMiRL VAE) improves performance in VizDoom. Five random seeds are sampled for cach method on each plot, the mean and standard deviation shown. Videos of the policies can be found at: https: httptss///sttsseee . google. com/ view/ surpri semi nimi zation
amount of damage
ciift: teppisodes with nalls
Treadmil: seppisodes with Palls
Pededtal tasi Eptoodes W/ Falls
24 T2 oours
(sund)
88 Eirooddss we Fatas
0.8
SPEAL K.M
0.7
0.1
c.
0.5
5M RL. (ours)
0.4
Vel
SM RL VSR (ours)
KMV
SM RL KCM
0.00
0.23
0.75
1.00
1.5
150
177
Steps
0.2
3.5
2.0
1.2
200
Figure 4: Results for the Cliff. Treadmill and Pedestal environments. In all cases, the SMiRL reward function reduces the fraction of episodes that results in falls (lower is better). The use of a VAE to estimate p(s) often increases learning speed and final performance. Three random seeds are sampled for each method on each plot, the mean and standard deviation are shown.
Imitation. We can easily adapt SMiRL to perform imitation by initializing the buffer Dn with states from expert demonstrations, or even individual desired outcome states. To study this application of SMiRL, we initialize the buffer Do in Tetris with user-specified desired board states. An illustration of the Tetris imitation task is presented Figure [6] showing imitation of a box pattern (top) and a checkerboard pattern (bottom), with the leftmost frame showing the user-specified example, and the other frames showing actual states reached by the SMiRL agent. While a number of prior works have studied imitation without example actions (Liu et al.) 2018 Torabi et al., 2018a Aytar et al.) 2018; Torabi et al., [0118b Edwards et al.] 2018 Lee et al.). this capability emerges automatically in SMiRI., without any further modification to the algorithm.
SMiRL as a stability reward. In this next experiment, we study how SMiRL can accelerate acquisition of reward-driven behavior in environments that present a large number of possible actions leading to diverse but undesirable states. Such settings are common in real life: a car can crash in many different ways, robot can drop a glass on the ground causing While this is of course not the case for all tasks, many real-world tasks do require the agent to stabilize itself in a specific and relatively narrow set of conditions. Incorporating SMiRL into the learning objective in such settings can accelerate learning, and potentially improve safety during training. as the agent automatically learns to avoid anything that unfamiliar. We study this application of SMiRL in the DefendTTheLing task and the Walk task. In both cases, we use SMiRL to augment the task reward, such that the full reward is given by (sombinca(s) = Timsk (s) + aTSMRL( where a is chosen to put the two reward terms at a similar magnitude. In the Walk task, illustrated in Figure (s)s) is additionally initialized with 8 example walking trajectories (256 timesteps each), similarly to the imitation setting, to study how well SMiRL can incorporate prior knowledge into the stability reward (Reward- + SM:RL (ours). We include another version that is not initialized with expert data (Reward SMiRL (no-expert). We measure the number of falls during training, with and without
break in many ways, etc.
7

---

Under review as a conference paper at ICLR 2020
Defend The Line Sarvell Tims
Sayerd
Fcees 1910
023
8:7
Ranard SHITL (oura)
S.
Rewnrd Saiii the rxpert Anward
50
100
0.00
0.50
0.23
2.00
1.75
Led
8.0 8.50
0.75
(a) DefendTheLLine joint training
(b) Walk joint training (falls)
(c) Walk Joint training (Fask)
Figure 5: In (a) different intrinsic reward methods are combined with the survival time task reward in the VizDaom DefendTheLine task, showing that SMiRL accelerates learning compared to intrinsic motivation methods and the pure task reward. In (b) and (c) we combine the SMiRL reward with the Walk reward and initialize SMiRL with walking demonstrations and without (no-expert). This results in significantly fewer falls (b) and faster learning w.r.t the task reward (c). Five random seeds are sampled for (a) and three for (b and c), the mean and standard deviation are shown.
the SMiRL reward term. The results in Figure 5b show that adding the SMiRL reward results in significantly fewer falls during training, and less when using expert data while learning to walk well, indicating that SMiRL stabilizes the agent more quickly than the task reward alone.
In the DefendTheLine task, shown in Figure 2c we compare the performance of SMiRL as a joint training objective to the more traditional novelty-driven bonus provided by ICM (Pathak al. 2017) and RND (Burda et al. 2018b). Novelty-driven bonuses are often used to accelcrate learning in domains that present an exploration challenge. However, as shown in the results in Figure 5a the SMiRL reward, even without demonstration data, provides for substantially faster learning on this task than novelty-seeking intrinsic motivation. These results suggest that SMiRL can be a viable method for accelerating learning and reducing the amount of unsafe behavior (e.g., falling) in dynamic environments.
RELATED WORK
Prior works have sought to learn intelligent behaviors through reinforcement learning (Sutton &
Barto 2018 with respect to a provided reward function, such the score of video game (Mnih
et al. 2013) or a hand-defined cost function (Levine et al. 2016). Such rewards are often scarce or
difficult to provide in practical real world settings, motivating approaches for reward-free learning
such as empowerment (Kl (Klyubin et al. 22055] Mohamed & Jimenez Rezendel (2015) or intrinsic
motivation (Chentanez et al.) 2005 Oudeyer & Kaplan 2009;; Oudeyer al.) (007). Intrinsic
motivation has typically focused on encouraging novelty-seeking behaviors by maximizing model
uncertainty (Houthooft al. 2016, Still Precup 2012 Shyam al. 2018 Pathak al. (2019), by maximizing model prediction error or improvement (Lopes et al. [012 Pathak et al. |201]] through state visitation counts (Bellemare et al. 2016),, via surprise maximization (Achiam Sastry 2017
Schmidhuber 1991 Sun et al. [201]) 1), and through other novelty- based reward bonuses (Lehman Stanley] [01 1l Burda et al. 80 Kim al. 2019). We do the opposite. Inspired by the free energy principle (Friston, 2009| Friston et al. 2009), we instead incentivize an agent to minimize surprise and study the resulting behaviors in dynamic, entropy-increasing environments. In such environments, which we believe are more reflective of the real-world, we find that prior novelty- seeking environments perform poorly.
Prior works have also studied how competitive self-play and competitive, multi-agent environments can lead to complex behaviors with minimal reward information (Silver et al., 2017 Bansal et al. 2017: Sukhbaatar et al., 2017 Baker et al.) 2019). Like these works, we also consider how complex behaviors can emerge in resource constrained environments. However, our approach can also be applied in non-competitive environments.
8

---

Under review as a conference paper at ICLR 2020
6 DISCUSSION
We presented an unsupervised reinforcement learning method based on minimization of surprise We show that surprise minimization can used to learn a variety behaviors that maintain "homeostasis, putting the agent into stable and sustainable limit cycles in its environment. Across range of tasks, these stable limit cycles correspond to useful, semantically meaningful, and complex behaviors: clearing rows in Tetris, avoiding fireballs in VizDoom, and learning to balance and hop forward with a bipedal robot. The key insight utilized by our method is that, in contrast to simple simulated domains, realistic environments exhibit dynamic phenomena that gradually increase entropy over time. An agent that resists this growth in entropy must take active and coordinated actions, thus learning increasingly complex behaviors. This stands in stark contrast commonly proposed intrinsic exploration methods based on novelty, which instead seek to visit novel states and increase entropy.
Besides fully unsupervised reinforcement learning, where show that our method can give rise to intelligent and complex policies, we also illustrate several more pragmatic applications of our approach. We show that surprise minimization can provide general-purpose risk aversion reward that, when combined with task rewards, can improve learning in environments where avoiding catastrophic (and surprising) outcomes desirable. We also show that SMiRI, can be adapted to perform rudimentary form of imitation.
Our investigation of surprise minimization suggests number of directions for future work. The particular behavior of a surprise minimizing agent is strongly influenced by the particular choice of state representation: by including or excluding particular observation modalities, the agent will be more or less surprised. Thus, tasks may potentially be designed by choosing appropriate state observation representations. Exploring this direction may lead to new ways of specifying behaviors for RL agents without explicit reward design. Other pragmatic applications of surprise minimization may also be explored in future work, including its effects for mitigating reward misspecification, by disincentivizing any unusual behavior that likely deviates from what the reward designer intended. Finally, we believe that a promising direction for future research is to study how lifelong surprise minimization can result in intelligent and sophisticated behavior that maintains homeostasis by acquiring increasingly complex behaviors. This may be particularly relevant in complex real-world environments populated by other intelligent agents, where maintaining homeostasis may require constant adaptation and exploration.
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv: 2017.
Yusuf Aytar. Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 1xXiv:1909.073 2019.
Trapit Bansal, Jakub pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. arXiv preprint arXiv:/710.037788, 2017.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. In Proceedings of the 24th International Conference on Artificial Intelligence, 15, pp. 4148-4152 AAAI Press, 2015. ISBN 978-1- 57735-738-4. URL htt.: //dl acm. wwwp:/ooggcctiiiiooo htt.m//.wrrcccccccooooooooooeee.
Ludwig Boltzmann. The second law of thermodynamics. 1886.

---

Under review as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Damell, and Alexei A. Efros.
Large-Scale Study of Curiosity-Drive Driven Learning. 2018a. URL httt:://axxi..... wwwpxiiv.rg//aaoooo /bss
1808 04355
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. ICLR, 2018b.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 1281-1288, 2005.
Ashley D Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. arXiv preprint arXiv:1805.07914, 2018.
Karl Friston. The free-energy principle: a rough guide to the brain? Trends in cognitive sciences, 13 (7):293-301, 2009.
Karl J. Friston, Jean Daunizeau, and Stefan J. Kiebel. Reinforcement learning or active inference?
PLOS ONE, 1(7) 07 2009. doi: 10. / 0006441.. URL https://doi
or org/10. 13710.1371 371 journal www.enaalppp......mm pone. 0006421
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turek, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. 2016. URL httt:://axxv..rgg//////
1605. 09674
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Vizdoom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference Computational Intelligncce and Games (CIG), pp. 1-8. IEEE, 2016.
Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity-botllenckk Exploration by distilling task-specific novelty. In International Conference Machine Learning, pp. 1378933888, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Alexander S. Klyubin, Daniel Polani, and Chrystopher L. Nehaniv. All else being equal be empow- ered. In Mathieu S. Capcarrère, Alex A. Freitas, Peter J. Bentley, Colin G. Johnson, and Jon Timmis (eds.), Advances in Artificial Life, pp. 744-753, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31816-3,
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Ruslan Salakhutdinov, and Sergey Levine. State marginal matching with mixtures of policies.
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189 220, 2011.
S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(1), January 2016.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1118-1125. IEEE, 2018
Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Yver Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems, pp. 206-214, 2012.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.5023,, 2013.
10

---

Under review as a conference paper at ICLR 2020
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In C. Cortes, N. D. Lawrence, D. D. Lee,
M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28. pp. 2125-2133 Curran Associates, Inc., 2015. URL http:/ httst:://paper..... n nips. httppprrss.ipssccpppprp.rrr en-ma mb sat iontiioo-aae t
ielb-variaiion. iona pdf
vatedrreinforcemee
Pierre- Oudeyer and Frederic Kaplan. What is intrinsic motivation? ttpologg of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre-Y Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-266. 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven Exploration by Self-supervised Prediction. 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-Supervised Exploration via Disagreement. 2019.
Jürgen Schmidhuber. Curious mouel-building control systems. Proc. international joint conference neural networks, pp. 1458-1463, 1991.
Eric D Schneider and James J Kay. Life manifestation of the second law of thermodynamics. Mathematical and computer modelling, 19(6-8):25 48, 1994.
Erwin Schrödinger. What is life? The physical aspect of the living cell and mind. Cambridge University Press Cambridge, 1944.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Pranav Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-based active exploration. arXiv preprint 18000 12162, 2018
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint 1rXiv:1712.01811 2017.
Susanne Still and Doina Precup. An information--heerrtiic approach to curiosity-driven reinforcement learning. Theory in Biosciences, 131(3):13--148, 2012.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint 1733::1003.05407. 2017.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning be surprised: Optimal bayesian ex- ploration in dynamic environments. In International Conference on Artificial General Intelligence, pp. 41-51. Springer, 2011.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv: 1eXivv1805.011 01944, 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807:0887706/55 2018b.

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
SMiRL on Tetris. In Terris, since the state is a binary image, we model p(s) as a product of independent Bernoulli distributions for each board location. The SMiRL reward log (s)) from (I) becomes:
'SMILL (s) Σ S, log 0g + (1 (1-s;) log(1 0,).
where s is a single state, 02 is the sample mean calculated from D, indicating the proportion of datapoints where location i has been occupied by a block, and s; is a binary variable indicating the presence of block at location i. If the blocks stack to the top, the game board resets, but the episode continues and the dataset D, continues to accumulate states.
SMiRL on VizDoom and Humanoid. In these environments the observations placed in the buffer downsampled 10 13 single-frame observations for VizDoom environments and the full state for the Humanoid environments. We model p(s) as an independent Gaussian distribution for each dimension in the observation. Then, the SMiRL reward can be computed as:
- (oogo + (s--AM p₂)² 20
SSMRL (s) - - -
where is single state, ,4, and og calculated the sample mean and standard deviation from D and 8; is the fth observation feature of s.
SMiRL rewards We emphasize that the RL algorithm in SMiRL is provided with standard stationary MDP (except in the VAE setting, more on that below), where the state is simply augmented with the parameters of the belief over states and the timestep t. We emphasize that this MDP is indeed Markovian, and therefore it is reasonable to expect any convergent RL algorithm to converge to near-optimal solution. Consider the augmented state transition p( ((5 1. Oc+1, 1 4+ 188, Gt, Oz, t). This transition model does not change over time because the updates to 9 are deterministic when given 8t and t. The reward function R(s:, 0,,t) is also stationary: it is in fact deterministic given se and fg. Because SMiRL uses RL in MDP, we benefit from the same convergence properties as other RL methods.
However, the version of SMiRL that a representation learned from VAE is not Markovian because the VAE parameters are not added to the state, and thus the reward function changes over time.. We find that this does not hurt results, and note that many intrinsic reward methods such as ICM and RND also lack stationary reward functions. This process described in Algorithm D
Entropic Environments We do not use entropic to mean that state transition probabilities change over time. Rather, it means that for any state in the environment, random disruptive perturbations may applied to the state. In such settings, SMiRL seeks to visit state distributions p(s) that are easy to preserve.
VAE on-line training When using a VAE to model the surprise of new states, we evaluate the probability of the latent representations z. as described in Section 2.3 The VAE is trained at the end of each episode on all data seen so far across all episodes. This means that the encoder qu (z (bs) is changing over the course of the SMiRL algorithm, which could lead to difficulty learning good policy. In practice, the rich representations learned by the VAE help policy learning overall.
Training parameters. For the discrete action environment (Tetris and VizDoom), the RL algorithm used is deep Q-learning (Mnih et al. 2013) with target Q network. For the Humanoid domains, we use TRPO (Schulman et al.) 2015). For Tetris and the Humanoid domains, the policies are parameterized by fully connected neural networks, while VizDoom uses convolutional network. The encoders and decoders of the VAEs used for VizDoom and Humanoid experiments are implemented as fully connected networks over the same buffer observations as above. The coefficient for the KL-divergence term in the VAE loss was 0.1 and 1.0 for the VizDoom and Humanoid experiments, respectively.
12

---

Under review as a conference paper at ICLR 2020
Figure Results for imitation Tetris.
Imitation Results
13


Reviews:
[{"review_id": "Sklfr9LFKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel RL algorithm to minimize ‘surprise’, and the empirical results show the efficiency. The experiments are well-done jobs. However, there are several problems in other parts:\n1. Some terms in Section 2 are too general, such as ‘surprise’ and ‘stable states’. Please define these more precise.\n2. Please give an exact description of the MDP you consider about. It seems that the transition probabilities change by time as you mention ‘entropic environments’. So I guess it’s a finite horizon MDP?\n3. What’s the final situations of the policy and density function? Please give some theoretical results such as convergence or asymptotic properties of the policy and the density functions. At least, the results can be derived under simple MDP and density function settings.\n4. What’s the relationship between the density function you mention in your paper and state distributions in RL?\n5. The agent trying to reach stable states sounds like exploration is discouraged. How do you explain? “Provably Efficient Maximum Entropy Exploration” by Elad et al. 2018 proposes to maximize negative log-likelihood of state distributions to encourage exploration, which is opposite to your setting. How do you think?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Ske_SVOntr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a novel form of surprise-minimizing intrinsic reward signal that leads to interesting behavior in the absence of an external reward signal. The proposed approach encourages an agent to visit states with high probability / density under a parametric marginal state distribution that is learned as the agent interacts with its environment. The method (dubbed SMiRL) is evaluated in visual and proprioceptive high-dimensional \"entropic\" benchmarks (that progress without the agent doing anything in order to prevent trivial solutions such as standing and never moving), and compared against two surprise-maximizing intrinsic motivation methods (ICM and RND) as well as to a reward-maximizing oracle. The experiments demonstrate that SMiRL can lead to more sensible behavior compared to ICM and RND in the chosen environments, and eventually recover the performance of a purely reward-maximizing agent. Also, SMiRL can be used for imitation learning by pre-training the parametric state distribution with data from a teacher. Finally, SMiRL shows the potential of speeding up reinforcement learning by using intrinsic motivation as an additional reward signal added to the external task-defining reward.\n\nQuality\n\nAs a practical paper, this work needs to be judged based on the quality of the experiments. I find the number of benchmarks and baselines sufficient. One major issue, that currently prevents me from voting for acceptance, is that experiments have not been conducted with enough seeds. I couldn't find any information in the paper regarding how many repetitions there are for each experiment. Also, most figures do not indicate any uncertainty measures (standard deviation, percentiles or the like)---some do, e.g. Figure (4), but it is not mentioned what type of uncertainty is depicted. One seed is certainly not enough to support the claims made by the authors---especially not the one that SMiRL can help improve RL in Figure (5). Figure (5a) clearly  does not draw a clear picture under one seed, and (5b) and (5c) require additional expert demonstrations. If experiments are repeated with more seeds and still support the claims, I am happy to increase my score to acceptance (presupposing that the discussion phase does not prevent acceptance for other reasons).\n\nClarity\n\nThe paper is clearly written and easy to follow. However, I do not like one aspect in the way the authors motivate their approach. The problem formulation starts with an MDP formulation. MDPs rely on a stationary reward signal and RL agents aim to optimize for future cumulative rewards based on these stationary reward signals. The authors propose to optimize for a non-stationary signal since the parametric state distribution changes over time. This itself is not uncommon and nothing controversial from a practical perspective. However, the statement that SMiRL agents seek to visit states that will change the parametric state distribution to obtain higher intrinsic reward in the future is controversial (see e.g. at the end of Section 2.1), because optimizing a non-stationary signal is outside the scope of the problem formulation. The reinforcement learning problem of maximizing intrinsic rewards does not know how the intrinsic reward signal is altered in the course of the future, i.e. how the parametric state distribution is updated. These statements should therefore be either adjusted accordingly, or the claims should be backed up theoretically rather than intuitively. On a minor note, I don't think that Figure (1) is necessary and the quote at the beginning of the paper might be better suited for a book chapter (but that is just my personal opinion).\n\nOriginality\n\nI find the simple idea presented by the authors to minimize rather than maximize surprise quite original. However, I do not have much experience in the domain of intrinsic motivation and leave the judgement of originality to the other reviewers and the area chair. Also, some references might be missing regarding learning with intrinsic reward (e.g. empowerment).\n\nSignificance\n\nThe fact that a simple intrinsic reward, as presented by the authors, can lead to interesting behavior, as demonstrated by the experiments, is quite significant. Unfortunately, the experiments are not significant from a statistical perspective which is why I do not recommend acceptance at this stage (as mentioned above, depending on how the authors address this issue and the discussion period, I might change to acceptance).\n\nUpdate\n\nThe authors have addressed my main concern regarding missing seeds. I therefore change to weak accept. But I am not happy how the authors responded to my concern regarding their motivation:\n1.) The formulation that tries to justify their reasoning as given in the rebuttal was absent in the first version of this paper, but something along this line would have been necessary since the argument relies on a non-standard MDP formulation.\n2.) I don't think the argumentation given in the rebuttal is correct. Imagine hypothetically the agent is in exactly the same augmented state s_t, a_t, \\theta_t at two different points in time, e.g. in the first episode and after multiple episodes. In both cases, the collected states seen so far are going to be different, hence the optimization objective for learning a marginal state distribution is different, hence the parameter updates are different, hence the transitions are not stationary.\nI do therefore encourage the authors to attenuate their wording.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgOkLqitS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Surprise Minimizing RL (SMiRL), a conceptual framework for training a reinforcement learning agent to seek out states with high likelihood under a density model trained on visited states. They qualitatively and quantitatively explore various aspects of the behaviour of these agents and argue that they exhibit a variety of favourable properties. They also compare their surprise minimizing algorithm with a variety of novelty-seeking algorithms (which can be considered somewhat the opposite) and show that in certain cases surprise minimization can result in more desirable behaviour. Finally, they show that using surprise minimization as an auxiliary reward can speed learning in certain settings.\n\nMinimizing surprise with RL presents challenges because, as the authors point out, exploring a surprising state in the present might minimize surprise in the future if the surprising state can be maintained (and therefore made unsurprising). To formulate this notion of surprise as a meaningful RL problem it is necessary to include some representation of the state likelihood in the agent's state representation; so that the agent can learn how it's actions affect not only the environment state but it's own potential for future surprise. This paper takes a pragmatic approach to this by training a density model on the full set of visited states which are reset at the start of each of a series of training episodes. The agent's policy is then conditioned on the parameters of the density model and the current size of the dataset.\n\nI lean toward accepting this paper as an interesting initial step in applying the idea of surprise minimization to reinforcement learning. The proposed approach is simplistic in how they treat surprise, and therefore illustrative, but probably not practical. The experiments are also illustrative, and while it is clear that surprise minimization won't always generate useful behaviour, the paper doesn't overclaim in this regard. I found the motivation for being useful in natural environments, where maintaining some kind of homeostasis is often key, to be well presented. Given that a significant body of recent work focuses on novelty seeking as a means to guide exploration, I think it is a point worth making that there are many reasonable environments where the opposite behaviour is desirable.\n\nI also found the paper to be quite well written and enjoyable to read overall.\n\nGeneral Comments:\nSection 3, VizDoom: \"The observation space for consists...\"->\"The observation space consists...\"\nSection 2.3: \"...as a multivariate normal distribution...\" it looks like covariance is not accounted for so wouldn't it more accurately be representing each component as an independent normal distribution?\nSection 4.1: \"...fireball explorations.\"->\"...fireball explosions.\"?\nSection 4.1: \"...as shown in Figure Figure...\"->\"...as shown in Figure...\"\n\nQuestions for the authors:\n-Out of curiousity, what do the results look like when using SMiRL as a stability reward but without example trajectories?\n-SMiRL is largely pitched as an alternative to novelty-seeking methods. But it seems to me novelty-seeking could be usefully combined since as you point out SMiRL may still have to explore to find stable states. Do you see such a combination as feasible or are the two methods fundamentally opposed?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes augmentation of the state exploration strategy that is interesting and has a potential to lead to improvement. However, the current presentation makes it difficult to properly assess that. In particular, the way the authors convey both the underlying intuition and its implementation is fairly vague and does not build confidence in the grounding of the underlying methodology.
Decision: Reject