Information is Power: Intrinsic Control via Information Capture
Authors: Nicholas Rhinehart
Jenny Wang
Glen Berseth
John D Co-Reyes
Danijar Hafner
Chelsea Finn
Sergey Levine
Venue: NeurIPS 2021 Poster (2021)
Keywords: intrinsic motivation
latent state-space models
deep reinforcement learning
intrinsic control
world models

Abstract: Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpose objective for an agent? We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. We instantiate this approach as a deep reinforcement learning agent equipped with a deep variational Bayes filter. We find that our agent learns to discover, represent, and exercise control of dynamic objects in a variety of partially-observed environments sensed with visual observations without extrinsic reward.

Information is Power:
Intrinsic Control via Information Capture
Nicholas Rhinehart UC Berkeley
Jenny Wang UC Berkeley
Glen Berseth UC Berkeley
John Co-Reyes UC Berkeley
Danijar Hafner University of Toronto Google Research, Brain Team
Chelsea Finn Stanford University
Sergey Levine
UC Berkeley
Abstract
Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpon objective for an agent? We study this question in dynamic bartially-obseree environments, and argue that compact and general learning objective to minimize the entropy of the agent's state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. instantiate this approach as a deep reinforcement learning agent equipped with deep variational Bayes filter. We find that our agent learns discover, represent, and exercise control of dynamic objects in variety of partially- observed environments sensed with visual observations without extrinsic reward.
Introduction
Reinforcement learning offers framework for learning control policies that maximize given measure of reward ideally, rewards that incentivize simple high-level goals, such as survival, accumulating particular resource, or accomplishing some long-term objective. However, extrinsic rewards may be insufficiently informative to encourage an agent to explore and understand its environment, particularly in partially observed settings where the agent has limited view of its environment. A generalist agent should instead acquire an understanding of its environment before specific objective or reward is provided. This goal motivates the study of self-supervised or unsupervised reinforcement learning: algorithms that provide the agent with an intrinsically groundddd drive to acquire understanding and control of its environment in the absence of an extrinsic reward signal. Agents trained with intrinsic reward signals might accomplish tasks specified via simple and sparse rewards more quickly, may acquire broadly useful skills that could be adapted specific task objectives. Our aim is to design an embodied agent and a general-purpose intrinsic reward signal that leads to the agent controlling partially robserel environments when equipped only with high-dimensional sensor (camera) and no prior knowledge.
A large body of prior methods for self-supervised reinforcement learning focus attaining coverage, typically through evvelty-sekking or skill-discovery. As argued in prior work [18, 16, 14, 6], a com- pelling alternative coverage suited complex and dynamic environments is to minimize surprise, which incentivizes an agent to control aspects of its environment to achieve homeostasis within - ie. constructing and maintaining niche where it can reliably remain despite external perturbations. We generally expect agents that succeed at minimizing surprise in complex environments to develop
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Time
The agent uses a latent state space model to represent beliefss about the world, including dynamic objects like the goat. The blue window represents the agent's field-of-view, which defines the extent of the current observation. Each observation informs the agent's 's belief's about the world. In the first panel, the agent sees a tree, a goat, and clouds. In the second panel, the agent chopped down tree and the goat moved out of view, causing the agent's belief about the goat's ss position to become uncertain. If the agent reduces the long- morizoo average entropy of its beliefs, it will first seek out information (e.g. find the goat), and then modify the environment to reduce changes in belief (e.g. huild fence around the goat). Finally, the agent builds house create area with few changes in weather.
Figure I:
similarly complex behaviors; such acquired complex behaviors may be repurposed for other tasks [6]. However, these frameworks do not explicitly address the difficulty of controlling partially-observer environments: if an otherwise complex and chaotic environment contains "dark room" (small reliable niche), an agent could minimize surprise simply by hiding in this room and refusing to make meaningful observations, thereby failing to explore and control the wider surrounding environment.
Consider Fig. 1, which depicts a partially-observed outdoor enviromment with trees, goat, weather, and an agent. We will discuss three different intrinsic incentives an agent might adopt in this environment. If the agent's incentive (i) minimize the entropy its next observation, will seek the regions with minimal unpredictable variations in flora, fauna, and weather. This is unsatisfying because merely requires avoidance, rather than interaction. Let us assume the agent will maintain a model of its belief about a learned latent state the agent cannot observe the true state, instead learns a state representation. Further, let us assume the agent maintains separate model of the visitation of its latent state we will refer to this distribution as its latent visitation. If the agent's s incentive is to (ii) minimize the entropy of belief (either at every step or at some final step), the agent will gather information and take actions to make the environment predictable: find and observe the changes in flora, fauna, and weather that are predictable and avoid those that aren't. tt However, once has taken actions to make the world predictable, this agent is agnostic to future change it will not resist predictable changes in the environment. Finally, if the agent's 's incentive is to (iil) minimize the entropy of its latent visitation, this will result in categorically different behavior: the agent will seek both to make the world predictable by gathering information about it and prevent it from changing. While both the belief and latent visitation entropy minimization objectives worthwhile intrinsic motivation objectives to study, we speculate that an agent that is adept at preventing its environment from changing will generally learn more complex behaviors.
We present a concise and effective objective for self-supervised reinforcement learning in dynamic partially-observer environments: minimize the entropy of the agent's latent visitation under latent state-space model Icarned from exploration. Our method, which we call Intrinsic Control by Information Capture (IC2), causes agent to learn to seek out and control factors of variation outside of its immediate observations. We instantiate this framework by learning a state-space model as a deep variational Bayes filter along with a policy that employs the model's 's beliefs. Our experiments show that our method learns to discover, represent, and control dynamic entities in partially visual environments with no extrinsic reward signal, including in several 3D environments.
Maxwell's Demon and belief' entropy The main concept behind our approach to self-supervised reinforcement learning is that incentivizing an agent to minimize the entropy of its belief' about the world is sufficient to lead it to both gather information about the world and learn to control aspects of its world. Our approach is partly inspired by a well-known connection between in- formation theory and thermodynamics, which can illustrated informally by a version of the Maxwell's Demon thought experiment [46, 41]. Imagine a container separated into compartments, as shown in Fig. 2. Both compartments contain gas molecules that bounce off of the walls and each other in a somewhat unpredictable fashion, though short-term motion of these molecules

---

(between collisions) is predictable. The compartments are separated by massless door, and the agent (the eponymous "demon") can open or close the door at will to sort the particles. By sorting the particles onto one side, the demon appears to reduce the disorder of the system, as mea- sured by the thermodynamic entropy, S, which in- creases the energy, F available to do work, as per Helmholtz's 's free energy relationship. === TTS. The ability to do work affords the agent control over the environment. This apparent violation of the ond law of thermodynamics is resolved by account- ing for the agent's 's information processing needed to make decisions [5]. By concentrating the particles into smaller region, the number of states each par- ticle visits is reduced. This illustrates an example environment in which reducing the entropy of the visitation distribution results in an agent gaining the ability do work. In the same way that Maxwell's demon accumulates free energy via information- gathering and manipulating its environment, we expect self-supervised agents guided by belief visitation entropy minimization accumulate the equivalent of ing sequential decision processes, which leads them to gain control over their environments.
Figure 2: A "demon" gathering information sort particles, reducing the entropy of the particle configuration.
potential energy in their correspond-
Preliminaries Our goal in this work will be to design self-supervised reinforcement learning methods in partially observed settings to acquire complex behaviors that both gather information and gain control over their environment. To this end, we will formulate the learning problem in the context of discrete-time partially-observe hidden Markov process (CHMP), which corresponds to a POMDP without a reward function. The CHMP is defined by a state space S with states s S, action space A with actions E A, transition dynamics (( |st, a.). observation space with observations o E Ω and emission distribution (s.).. The agent a policy : (Ta,) (og)). Note that it does observe any states
controlled Markov process, also known as controlled
We denote the undiscounted finite-horizon state visitation as d' (s) = 1/2 TT- Pr, (s, = s), where Pr, (s; = s) is the probability that s, = after executing π for steps. Using d' (s). we can quantify the average disorder of the environment with the Shannon entropy. H (dd (s)). Prior work proposes surprise minimization (min, log p(s)) as an intrinsic control objective [15, 70, 6]; in Berseth et al. [6] (SMiRL), the agent models the state visitation distribution, d (s) with ((s). which it computes by assuming access to s. In environments in which there are natural sources of variation outside of the agent, this incentivizes the SMiRL agent, fully aware of these variations observed through s, to take action to control them. In a partially-observed setting, it is matter of interpretation of what the most natural extension of SMiRL is whether (1) the observation should be treated as the state, or (2) an LSSM's belief should be treated interpretations, the latter constituting one of our methods. In the former, SMIRL's model becomes p(o), which generally will enable the agent to ignore variations that it can prevent from appearing in its observations. We observe this phenomenon in our experiments.
estimate of the latent state. Thus,
experiments include both
2 Control and Information Gathering via Belief Entropy Minimization
The main question that we tackle in the design of our algorithm is: how can we formulate a general and concise objective function that enable an RL agent to gain control over its partially-observed environment, in the absence of any user- -roviddovided task reward? Consider the following partially- observed "TwoRoom" environment, depieted in Fig. 3. The environment has two rooms: an empty ("dark") room on the left, and a "busy" room on the right, the latter containing two moving particles that move around until the agent "tags" them, which stops their motion. Intuitively, an agent that aims to gather information and gain control of its environment should search for the moving particles to find out where they are. However, it is difficult observe both particles at the same time. A more effective strategy is to "tag" the particles then, their position remains fixed, and the agent will know where they are at all times, even when they are not observed. This task, remininscent of the
In Maxwell's original example, the demon sorts the particles into the two chambers based on velocity. Our example is actually more closely related to Szilard's engine, which based on similar principle [68, 45j.
3

---

Time
IC2
(ours)
SMiRL
Infogain
Figure 3: Comparison intrinsic control objectives on the TwoRoom environment. The agent, white, can view limited area around it, in grey, and can stop particles within its view and darken their color. The vertical wall, in brown, separates particles (blue and green) in the "busy room" (on right) from the "dark room" (on left). Top: Our approach seeks out the particles and stops them. Middle: The observational surprise minimization method in Berseth al. [6] leads the agent frequently hide in the dark room, leaving the particles unstopped.
Bottom: Latent-state infogain leads the agent find and observe the particles, not stop them.
previously
mreviousyyddicuussed discussed Maxwell's s Demon thought experiment, can be seen as a simple analogy for more complex settings that can occur in natural environments.
Representing variability with latent state-space models. In order for the agent represent the dynamic components of the environment observed from images, our method involves learning a latent state-space model (LSSM) [72, 38, 29, 44, 22, 47, 73, 71, 40, 10, 23, 48. 58]. We intermittently refer to these dynamic components as "factors of variation" to distinguish the model's representation of variability in the environment (latent state) from the true variability (true state). At timestep i, the LSSM represents the agent's s current belief as 4o (z agr-1). )) where 2, is the model's latent state. We defer the description of the LSSM learning and architecture to Section and now motivate how we will use the LSSM for constructing intrinsic control objective.
Belief entropy and latent visitation entropy. Consider a policy that takes actions to minimize the entropy of the belief H (gg 1)). This corresponds to the agent performing active state es- timation [12, 74, 37J, and is equivalent to taking actions to maximize expected latent-state information gain I(0,, zo/0<1, a,) [2]. However, active state estimation is satisfied by a policy that simply collects informative observations, as does not further incentivize actions to "stabilize" " the environment by preventing the latent state from changing. Analogous to the standard definition of undiscounted state visitation, consider the undiscounted latent visitation: d (z) = 1/r D 1 Pr, (z; = z), where Pr, = z) Ex q (Z, O.gr, agt-1) (the expected belief after executing π for t timesteps). Our goal minimize H H(d" [z)), as this corresponds to stabilizing the agent's beliefs, which incentivizes both reducing uncertainty in each 9a (z, oci, ag(-1), as well as constructing a niche such that each 1)) concentrates probability on the same latent states. While in principle we could approximate this entropy and minimize it directly, later we discuss our more practical approach.
Discovering factors of variation. In order for belief entropy minimization to incentivize the agent to control entities in the environment, the LSSM's belief must represent the underlying state variables in some way and model their uncertain evolution until either observed or controlled. For example, the demon in the thought experiment in Section would have no incentive to gather the particles if did not know that they existed. If the agent unable to encounter or represent dynamic objects in its LSSM, it could choose not to move. However, if the LSSM properly captures the existence and motion of dynamic objects (even out of view), then the agent has an incentive to exert control over them, rather than avoid them by going parts of the environment where it does not encounter them. While sufficient random exploration may result in a good-enougl! LSSM, making this approach generally practical requires suitable exploration strategy to collect the experience necessary train an LSSM that represents all of the underlying factors of variation.
To this end, we learn separate exploratory policy to maximize expected model information gain, similar to Schmidhuber [60], Houthooft et al. [26), Gheshlaghi Azar et al. [20], Sekar al. [62]. The exploration policy is employed as a way to encounter dynamic objects and collect data which the model performs poorly. The exploration policy does not need to learn accurate estimates of the latent state and its uncertainty it is merely incentivized to visit where the current model is inaccurate.

---

(2)
(,(2)
(2(2)
IC2
be combined Niche Creation
Certainty
9(2,)
State Infogain
0
0
6
r=E, [log d-log q]
X-E, (nlggg)
(a) Latent state-space and visitation models.
(b) Intrinsic rewards.
Figure 4: Figure of latent-state space model and rewards, Left: The model observes images, o, to inform beliefs about latent states, (2, ogg, ag,,, )). and observes actions to make one-step predictions p(z++1 [2,, ac). Each belief is used to update the latent visitation, Ør (z). Right: The belief's and latent visitations into various reward functions. directions of belief expansion and contraction incentivized by rewards; the dotted arrows denote directions of belief translation incentivized by rewards.
denote our main method's ' reward function "IC2". The solid arrows denote
Expected information gain about model parameters 0 is relative to set of prior experience 'D and partial trajectory h,-1, given as T(o;; - D) E,, (L( o, h. p( D)). Note that model information gain is distinet from the information an agent may gather to reduce its belief entropy H (g agt-1)) within the current episode. Computing the full model parameter prior p(0) h, D) and posterior p(0)or, h D) is generally computationally expensive, and also requires evaluating an expectation over observations instead, we approximate this expected information gain following method similar Sekar et al. [62]: we an ensemble of latent }= to compute the variance of latent states estimated by ts (z, (o ag(-1). We build the ensemble throughout training using the method of Izmailov et al. [27]. Thus, the exploration reward given as: Te Var (e.) [log po (Ze 2t- at-1) 1)) Ze ~ 9a}-
dynamics models, & = {pe, Ze a i==
The IC2 Algorithm
Now describe how implement intrinsic control algorithm. The main components the latent-state space model, the latent visitation model, and the exploration and control policies.
Latent state-space model. In our CHMP setting, the agent only has access to partial observations of the true state In order to estimate representation of states and beliefs, we employ sequence-based Variational Autoencoder to learn latent variable belief, dynamics, and emission models. We formulate the variational posterior to be qu (z ( aI.T) = II (z agi-1)
and the generative model to be p(Z1:T, 01:7 (ai.r) I (o (2)) Po))po(2 v at-1). Denoting h = (Oge, 1), the log-evidence of the model and its lower-bound are:
log p(oi.TlaTT-1 = log r plzzrrlz II= logr(or/ze) 2 L(o, 0: (00,0:01,;,
-ET (z.(a.) [pe 2z)] Ego [KL((g(Z; h.) pe a. (1)
Given dataset D m 1/1D) E, L(d,, 0:0i.T- -).r-)) The focus of our work is not to further develop the per- formance of LSSMs, but improvements in the particular L.SSM employed would translate to im- provements in our method. In practice, we implemented an LSSM in PyTorch [55] similar to the categorical LSSM architecture described in [24]. In this case, both the belief prior and belief pos- terior are distributions formed from products of K₁ categorical distributions over K2 categories: g(:; v) = II
((O1TT aI:T): Eq. is used train the model via
Zey I Vxn.K2 where the vector of Ki K₂ parameters, is predicted by neural networks: Vposterior fa(Ocs, ac) for the posterior, and Vprior fe(oct, ar) for the prior. We found effective to construct fø and fe following the distribution-sharing decomposition described in [29, 30, 10), in which the posterior is produced by transforming the parameters of the prior with measurement update. We implemented the prior as an RNN, and the posterior as an MLP, and defer further details to the appendix.
Latent visitation model. Our agent cannot, in general, evaluate dx) or H(d" (x))) at best, can approximate them. We do maintaining within-episode estimate of d' (z) by constructing
5

---

Algorithm Intrinsic Control via Information Capture (IC2)
1: procedure ICV(Env; K, M, M,,NN1 L)
2: Initialize ee. ee 96. (6.)P(.
0.
3:
for episode ...... M do
4: De f Collect( (N, :n), De + Collect(NN, Env, (.W.T.);D++ DUDD UD
5: Update qs. Pe using SGD on Eq. with data D for rounds.
6: Update Te using SGD on PPO's s objective with rewards re with data De
7: Update Te using SGD on PPO's 's objective with the IC2 reward (Eq. (2)) with data De
mixture across the belief history samples qr (z) Ei=o To (Ze Pell agt-1). This corresponds to mixture (across time) of single-sample estimates of each E, 90 (z, (z,, agr-1). Given (z), we have an estimate of the visitation the policy, and we can use this as part or all of the reward signal of the agent. To implement q: (z), we found that simply recording each belief sufficed. While we adopt a within-episode estimate of dº (z), our approach could be extended to multi-episode estimation, at the expense of extra computational burden.
3.1 The IC2 reward
We now describe our main reward function, "IC2". In what follows, we denote qs (Ze) aci-)) qe (z)) for brevity. The IC2 reward the KL from the belief the belief visitation (Eq. (2)). is visualized in Fig. 4. This incentivizes the agent reduce belief visitation entropy by bringing the current belief towards the current latent visitation distribution, and to expand the current belief to encourage exploration, and thus potentially find broader niche.
rt - (I-((( (la) (z)) = (z.) (log ge (z) - log qe (z.)).
The LSSM and visitation models used in our method allow for easy constructions of other intrinsic reward functions. We construct and experiment with several other intrinsic rewards in order to provide more context for the effectiveness of our main method.
Niche Creation. The expected negative surprise of the current belief under the latent visitation distribution measures how unsurprising the current belief is relative to the agent's experience the episode so far. One interpretation of this reward is that it is SMiRL [6] applied to samples of latent beliefs estimated from LSSM in a CHMP. Minimizing this reduces the number of visited belief states and increases the agent's certainty. The niche creation reward is:
r H(q (z:). qr (z)) = E, (r.)) [log ge (z)).
Certainty. The entropy of the current belief measures the agent's certainty of the latent state, and by extension, about the aspects of the environment captured by the latent. Minimizing the belief entropy increases the agent's 's certainty. It is agnostic to predictable changes, and penalizes unpredictable changes. We define the certainty reward the negative belief entropy:
rf -(((((z))) = (x. [log ge
(4)
State Infogain. The latent state information gain (not the model information gain in Section 2) measures how much more certain the belief is compared to its temporal prior. Gaining information does not always coincide with being certain, because an infogain agent may cause chaotic events in the environment with outcomes that it only understands partially. As a result, it has gained more information than standing still but has also become less certain. We define the infoguin reward
E U: (2 KL (z (z;) (p( 2 1)) Egg (x. [log (x (x))//((2zee 12. 12-1 a.--))) (5)
3.2 Algorithm summary
Conceptual pseudocode for for Intrinsic Control via Information Capture (IC2) is presented in Alg. 1. The algorithm begins by initializing the LSSM, 9o and Pe, well as two separate policies: one trained to collect difficult-tu to tr-ddit data with the exploration objective with rewards defined by r.. Te, and one trained to maximize intrinsic control objectives defined by of Eqs. (2) (5).
We represent each policy π (a, (posterio,.:) as a two-layer fully-connected MI.P with 128 units. Recall that yposterior is the vector of posterior parameters, which enables the policy to use the memory

---

represented by the LSSM. We do not back-propagee propagate the policies' losses to the LSSM for simplicity of implementation, although prior work does so the case of a single policy [40]. Our method is agnostic to the subroutine used to improve the policies. In our implementation, we employ PPO [61].
4 Experiments
Our experiments are designed to answer the following questions: Q1: Intrinsic control capability: Does our latent visitation-based self-supervised reward signal cause the agent to stabilize partially- observed visual environments with dynamic entities more effectively than prior self-supervised stabilization objectives? Q2: Properties of the IC2 reward and alternative intrinsic rewards: What types of emergent behaviors does each belief-basee objective described in Section 3.1 evoke?
In order to answer these questions, we identified environments with the following properties (i): partial observability, (ii): dynamic entities that the agent can affect, and (iii): high-dimensional observations. Because many standard RL benchmarks do not contain the significant prevalent in the real world, it is challenging to answer Q1 or Q2 with them. Instead, we create several environments, and employ several existing enviromments we identified have these properties. In what follows, we give an overview of the experimental settings and conclusions. We defer comprehensive details to the appendix.
cartial-observability that is
TwoRoom Environment. As previously described, this environment has two rooms: empty ("dark") room the left, and "busy" room on the right, the latter containing moving particles (colored dots) that move unless the agent "tags" them, which permanently stops their motion, shown in Fig. 5. The agent (white) observes a small area around it (grey), which it receives as an image. In this environment, control corresponds to finding and stopping the particles. The action space is A = {left, right, up, down, tag, no-op). and the observation space is normalized RGB-images: 5 [0, 1] |xx30x30 An agent that has significant control over this environment should tag particles to reduce the uncertainty over future states. To evaluate policies, we use the average fraction of total particles locked, the average fraction of particles visible, and the discrete true-state visitation entropy of the positions of the particles, (dd (sz)). We employed two versions of this environment, with details provided in the appendix. In the large environment, the agent observes 5x5 area inside 15x15 area, and the busy room contains particles.
Figure 5: TwoRoom
Large Environment.
VizDoom DefendThheCnnee environment The VizDoom DefendTheeCnn- ter environment shown in Fig. is circular arena in which a stationary agent, equipped with a partial field-of-view of the arena and a weapon, can rotate and shoot encroaching monsters [31]. The action space is A [turn left, turn right, shoot], and the observation space is normalized RGB- images: Ω [0, 1j3x64x64, In this environment, control corresponds reducing the number of monsters by finding and shooting them. We use the average original environment return, (which no policy has access to during training), the average number of killed monsters at the end of an episode, and the average number of visible monsters to measure the agent's 's control.
Figure 6: Vizdoom Defend The Center.
OneRoomCapturre3) environment. The MiniWorld framework is a cus- tomizable 3D environment simulator in which agent perceives the world through a perspective camera [9]. We used this framework to build the environment Fig. which an agent and bouncing box both inhabit a large room; the agent can lock the box to stop it from moving if it is nearby, as well as constrain the motion of the box by standing nearby it. In this environment, control corresponds to finding the box and ei- ther trapping (turn left 20", turn right 20", move forward, move backward, tag), and obser- vation space is normalized RGB-images: Ω = [o, 1 113x61x61 We use the
near a wall, or tagging
The action space
Figure 7: One Room Capture 3D. average fraction of time the box is captured, the average time the box is visible, and differential entropy (Gauusia of the true-state visitation of the box's position Hdd (sa)) to measure the agent's ability to control the environment.
7

---

TwoRoom
OneRoomCapturee30
DefendTheCenter
TwoRoomLarge
(a) Obj. Locking (t)
(b) Obj. Capture (1)
(c) Return (1)
(d) Obj. Locking (1)
Hdd (s2) (1)
Hdd (sa))
g Kills (t)
(h) H (d (sa)) (L)
Objects Visibility
Objects Visibility
(k) Monster Visibility
(l) Object Visibility
$1.+ N.C.
Niche Creatior
Certainty
State Infogain
SMRRL
Empoweement
Random
Oracke
Figure 8: Policy evaluation. Means and their standard errors are reported. The IC2 and Niche Creation+Infogan objectives Icad the agent to stabilize the dynamic objects more effectively than other methods.
4.1 Comparisons
In order answer Q1, we compare to SMiRL [6] and recent empowerment method [77] that estimates the current empowerment of the agent from an image. We use the empowerment estimate reward signal for training policy. In order answer Q2, ensured each environment was instrumented with the aforementioned metrics of visibility and control, and deployed Algorithm separately with each of Eqs. (2) to (5), as well a with simple sum of Eq. (3) and Eq. (5). We also compare to a "random policy" that chooses actions by sampling from a uniform distribution over the action space, as well as an "oracle policy" that has access to privileged information about the environment state in each environment. We perform evaluation at 5e6 environment steps with 50 policy rollouts per random seed, with 3 random seeds for each method (150 rollouts total). Exploration: Note that while our method employs an exploration policy, it is used only to improve the LSSM the data collected by the exploration policy is not used by the on-policy learning of the control policy (although it could be used, in principle, by an off-policy learning procedure). No method, including ours, is provided with an explicit separate mechanism augment data collection for policy optimization. All control policies employed in our methods and the baselines follow the same exploration procedure that occurs as part of PPO optimization.
Oracle policies. The oracle policies are designed to maximize the control performance in each environment as much as practically possible to put the control performance of our intrinsic objectives in context. Their purpose is to approximate the highest possible performance in each euvironment, for which access to privileged information can convenient. the TwoRoom and OneRoomCapturel] environments, the oracle policies are scripted, using access to the environment state, to seek out the nearest unfrozen object and freeze it, thereby achieving high performance on the lock, capture, and entropy control metrics. In the DefendTheCenter environment, the oracle policy is neural network

---

Figure 9: Visualization of a sequence the VizDoom DefendTheLine environment. Row The image provided to the agent. Row 2: The agent's 's reconstruction a sample from q. Row 3: The agent's 's one-step image forecast. Right: The state infogain signal, E,y log log pl. Each colored rectangle identifies a keyframe that corresponds to colored circle the infogain plot. The infogain signal measures how much more certain the helief is compared to its temporal prior: when stochastic events happen (monster appears nearby), the signal is high; when the next image is predictable (monster disappears when shot). the signal low.
trained using PPO to shoot as many monsters as possible by accessing the true extrinsic task reward of the environment.
Results summary. Our primary results are presented in Fig. 8. We observe the IC2 and Niche Creation++Infoginn (NCT) rewards to yield policies that exhibit a high degree of control over each environment finding and stopping the moving objects, and finding and shooting the monsters, in the complete absence of any extrinsic reward signal. The results show that employing the NC objective accompanied by within-episode bonus term resulting in either the IC2 objective or the NCI objective- - achieves the best control performance across the environments tested. The Infogain- based agent generally seeks out and observes, but does not interfere with, the dynamic objects; the high Visibility metric and low control metrics (Lock, Capture, Kill) in each environment illustrates this. Qualitative results of this phenomenon are illustrated in Fig. 9, in which the infogain signal is high when docchssticallymovvin the method of Berseth et al. [6] tends to learn policies that hide from the dynamic objects in the environment, as indicated by the low control and visibility values of the final policy. Furthermore, we observe the method of Zhao et al. [77] not to exhibit controlling behavior in these partially-observe environments, instead it views the dynamic objects in the TwoRoom and OneRoomCapture3D Environments somewhat more frequently than the random policy. We present videos on our project page: https://sites. wttp:/ts..oooll..ooo/oooooooeeee wwt.g/ll..oovvvieeeeee wwtpw////wwwwww////......
monsters are visible, and low when they are not. We observe
Detailed DefendTheCenter results. We observe the infogain- agent to rotate the camera until a close monster is visible and then oscillate the camera centered the monster. The infogain leads to the agent maintaining a large visual stimulus that is difficult for the model to predict due to the stochastic motion of the monster, an interpretation supported by Fig. 8k, where Infogain and NCI agents have the highest number of visible monsters in their field of view. We also observe that the NCI policy exhibits similar behavior to Infogain, which we speculate may be due to the infogain component of the objective dominating the NC component. We observe the NC agent to minimize visibility of the monsters in the following way: the agent tends to shoot the immediately visible monsters upon first spawning and then toggle between turning left and right. Since there is no no-op action, this allows the agent to see as few monsters as possible, since its view is centered on a region of the map it just cleared. We observe the IC2 agent to typically rotate in one direction and reliably shoot monsters. Our interpretation of this is that the IC2 reward is causing the agent to seek out and reduce the number of surprising monsters more often than the NC agent due to the belief entropy bonus term that IC2 contains, which encourages more exploration compared to NC. The Certainty agent has similar behavior to the NC agent but we observe that it is not as successful and has fewer monster kills and more visible monsters than NC.
Detailed OneRoom Capture33 results. Figs. 8b and 8f show that the IC2 scores lower (better) than NCI in the box entropy metric yet slightly worse in the Captured metric due to (1) differences in how each agent captures the box and (2) higher variance in the NCI policy optimization. (1) The IC2 agent tends to moves towards the box and execute the "freeze" action to stop the box from moving, whereas the NCI agent tends to move toward the box and trap it next to the wall without executing the freeze action, which causes the box to bounce back and forth between the wall and the agent every timestep, which is measured as Captured. We hypothesize NCI does this because allowing the

---

box to retain some movement gives the agent more Infogain reward. (2) Higher variance in the policy optimization of the NCI agent resulted in some runs that performed worse, which impacted both the entropy and capture metrics, but had a stronger effect on the box capturing metric.
Related Work
Much of the previous work learning without extrinsic rewards has been based either on (i)
exploration [8, 53, 52], or (ii) some notion of intrinsic control, such as empowerment [34, 49, 30]. Exploration approaches include those that maximize model prediction error or improvement [59, 43, 65, 56], maximize model uncertainty [26, 66, 64, 57, 20], maximize state visitation [4, 19, 69, 25], maximize surprise [59, 67]. and employ other novelty-based exploration bonuses [42, 7, 32, 33]. Our method can be combined with prior exploration techniques to aid in optimizing our proposed objective, and in that sense our work is largely orthogonal to prior exploration methods.
Prior works on intrinsic control include empowerment maximization [34, 35, 50), observational surprise minimization [15, 17, 70, 6, 54], and skill discovery [3, 36, 21, 11, 63, 75]. Observational surprise minimization seeks policies that make observations predictable and controllable, and is closely connected to entropy minimization, Friston et al. [18], the notion of Free Energy Minimization corresponds to minimizing observational entropy, and states that the entropy of hidden states in the environment is bounded by the entropy of sensory observations. However, the proof assumes a diffcomorphism to hold between states and observations, which is explicitly violated in CHMPs and any real-world setting, agents cannot perceive the state of anything outside their egocentric sensory observations. Similarly, empowerment [35, 28, 30, 77] is a measure of the degree of control agent could have over future observations, whereas state visitation entropy is a measure of the degree of the control an agent has over the underlying environment state. Our approach seeks to infer and gain control over representation of the environment's 's state, as opposed to the agent's 's observations. We demonstrate environments where minimizing observational surprise and maximizing empowerment leads to degenerate solutions that ignore important factors of variation, whereas our approach identifies and controls them. Representation learning methods have been explored in a variety of prior work, including, but not limited to, [39, 72, 29, 13, 51, 76, 22, 40]. Our approach employs a representation learning method build latent state-space model [72, 38, 29, 22, 47. 73, 71, 40, 10, 23, 48. 58].
entropy is defined to be the expected surprise. In
Discussion
We presented IC2, method for intrinsically motivating an agent to discover, represent, and exercise control of dynamic objects in a partially-observed environments sensed with visual observations. We found that our method approached expert-level performance on several environments and substantially surpassed prior work in its unsupervised control capability. While our experiments represent proof- of-concept that illustrates how latent state belief entropy minimization can incentivize an agent to both gather information and gain control over its environment, there are a number of exciting future directions. First, our method is inspired by a connection between thermodynamics and information theory, but the treatment of this connection is informal. Formalizing this connection could lead to an improved theoretical understanding of how IC2 and other intrinsic motivation methods can lead to desirable behavior, and perhaps allow deriving conditions environments under which such desirable behaviors would emerge; such a characterization might invoke the fact that the Certainty reward degenerates to activate state estimation unless the environment contain sources of uncertainty outside of the robot's state. Second, a precise characterization of environments in which it may not be desirable to deploy IC2 would be useful, for example, (i) if the Niche-based approaches were deployed simultancously on multiple robots in the same environment, each robot's goal would likely learn to trap or disable the other robots, in order to collapse its uncertainty about them, which may be undesirable, or (ii) if the Niche-based approaches were deployed without safeguards in an environment with living things, similarly, the agent may learn trap or destroy them, unless the agent estimated it could better avoid future uncertainty by cooperating with them. Finally, since IC2 relies on the LSSM to estimate beliefs, it is limited by the capacity of LSSMs to properly model beliefs - an exciting future direction and an active area of research is extending the capabilities of L.SMss which would enable study of the behavior that emerges when IC2 deployed more complex environments.
10

---

Acknowledgements This research was supported by ARL DCIST CRA I the Office of Naval Research, and an NSERC Vanier Canada Graduate Scholarship.
References
[1] J. Achiam and S. Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv: 000177073.322... 2017.
[2] F. Aoki, A. Bagchi, P. Mandal, and Y. Boers. theoretical look at information-driven sensor manage- ment criteria. In 14th International Conference on Information Fusion, pages 1-8. IEEF, 2011.
[3] A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages 112-19. Piscataway, NJ, 2004.
[4] Bellemare, S. Srinivasan, G. Ostrovski, Schaul, D. Saxton, and Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
[5] C. H. Bennett. Physics, 21(12):905-94 1982.
The thermodynamics of computation review. International Journal of Theoretical
G. Berseth, D. Geng, C. Devin, Rhinchart, Finn, Jayaraman, and Levine. (SM)irl: Surprise minimizing reinforeement learning in unstable environments, International Conference on Learning Representations, 2021. URL atttps://pperrrr////
net fort///mmm? wwwpmmmmiiimiee wit.idddd=dd.ox......
[7] Y. Burda, H. Edwards, D. Pathak, Storkey, T. Darrell, and A. Efros. Large-Scale Study Curiosity- Driven Learning. 2018. URL http: / /tt////axvvvv/////////eeeeeaaav/ org oogooogg//aa..ooooo..o 1808. 04355.
[8] N. Chentanez, A. G. Barto, and P. Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 1281-1288. 2005.
[9] M. Chevalier-Boisvert, gym- gym-miniworld environment for openai gym. https:///ithhu: htt.:hhbwb..co//maaimme///////// sym-miniworld, 2018.
[10] N. Das, M. Karl, P. Becker-Ehmek, and P. van der Smagt. Beta dvbf: Learning state-space models for control from high dimensional observations. arXiv preprint 1rXiv:1911.0077: 2019.
[11] B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Leaming skills without a reward function. arXiv preprint 1rXiv:1802 000000 2018.
[12] H. Feder, J. J. Leonard, and C. M. Smith. Adaptive mobile robot navigation and mapping. The International Journal of Robotics Research, 18(7):550-668, 1999.
[13] C. Finn, X. Tan, Y. Duan, Darrell, Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512 519. IEEF, 2016.
[14] Z. Fountas, N. Sajid, P. A. Mediano, and Friston. Deep active inference agents using conte-carlo methods. arXiv preprins arXiv:2006. 04176, 2020.
[15] K. Friston. The free-energy principle: rough guide the brain? Trends in cognitive sciences, 13(7):
293-301, 2009.
[16] K. Friston. Life as know it. Journal the Royal Society Interface, 10(86):00130775 2013.
[17] K. J. Friston, J. Daunizeau, and S. J. Kiebel. Reinforcement learning or active inference? PLOS ONE, 4 (7):1-13, 07 2009. doi: fone.0006421.
URL httpps://///o.... org/ 1etg/1..137....
[18] K. J. Friston, J. Daunizeau, J. Kilner, and S. J. Kiebel. Action and behavior: free-energy formulation. Biological cybernetics, 102())227-2666 2010.
[19] Fu, Co-Reyes, and Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. In Advances in neural information processing systems, pages 2577-2587, 2017.
[20] Gheshlaghi Azar, Piot, B. Avila Pires, J.-B. Grill, F. Altché, and R. Munos. World discovery models. arXiv e-prints. pages arXiv-1902. 1902, 2019.
[21] K. Gregor, D. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint 1r.70276...0750.. 2016.
[22] D. Hafner, T. Lillicrap. I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.0455: 2018.
[23] D. Hafner, T. Lillicrap. J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.0163. 2019 2019.

---

[24] D. Hafner, T. Lillicrap. M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprint urXiv:2010.02193, 2020.
[25] E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
[26] R. Houthooft, X. Chen, Duan, J. Schulman, De Turck, and P. Abheel. VIME: Variational Information Maximizing Exploration. 2016. URL http://arxiv.. wwt.aa////aa/o/aoooe/oo/eee/eee/ wwwp///0..0a..e0
[27] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and G. Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 1803:05407 2018.
[28] M. Karl, J. Bayer, and P. van der Smagt. Efficient empowerment. arXiv preprint arXiv:1509.0845: 2015.
[29] M. Karl, M. Saelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data, arXiv preprint arXiv: arXiv: (1660500632,, 2016.
[30] M. Karl, M. Soelch, P. Becker-Fhmete D. Benbouzid, P. van der Smagt, and J. Bayer. Unsupervised real-time control through variational empowerment. arXiv preprint arXiv:
2017.
[31] Kempka, M. Wydmuch, G. Rune, Toczek, and W. Jaškowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pages 1 8. IEEE, 2016.
[32] H. Kim, J. Kim, Jeong. S. Levine, and H. O. Song. Emi: Exploration with mutual information. arXiv 2018.
preprint arXiv:
[33] Y. Kim. W. Nam, H. Kim, J.-H. Kim, and G. Kim. Suriossttybotttleeeck Exploration by distilling task-specific novelty. International Conference on Machine Learning, pages 3379-3388, 2019.
[34] A. S. Klyubin, D. Polani, and C. L.. Nehaniv. All else being equal be empowered. In M. S. Capcarrère, A. A. Freitas, J. Bentley, C. G. Johnson, and J. Timmis, editors, Advances in Artificial Life, pages 744-753, Berlin, Heidelherg. 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31816-1
[35] A. S. Klyubin, D. Polani, and C. I.. Nehaniv. Empowernent: A universal agent-centric measure of control. In 2005 IEFE Congress Evolutionary Computation, volume 1, pages 128-135. IEFE, 2005.
[36] G. Konidaris and A. G. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in neural information processing systems, pages 1015-1023, 2009.
[37] C. Kreucher, K. Kastella, and A. O. Hero lii. Sensor management using an active sensing approach. Signal Processing, 85(3):607- 624, 2005.
[38] R. G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint arXiv:1511.05122 2015.
[39] S. Lange and M. Riedmiller. Deep auto-eneoder neural networks in reinforcement learning. In The 2010 International Joint Conference on Neural Networks (ICCN), pages 1-8. IEEE, 2010.
[40] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv: 2019.
[41] H. S. Leff' and A. F. Rex. Maxwell's demon: entropy, information, computing, Princeton University Press, 2014.
[42] Lehman and K. O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.
[43] M. Lopes. T. Lang, M. Toussaint, and P.-Y. Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. Advances neural information processing systems, pages 206-214, 066-214,72 2012.
[44] C. J. Maddison, D. Lawson, G. Tucker, N. Heess, M. Norouzi, A. Mnih, A. Doucet, and Y. W. Teh, Filtering variational objectives. arviv preprint / 2017.
[45] M. Magnasco. Szilard's heat engine. EPL (Eumophyscc Letters), 33(8):583, 1996.
[46] J. C. Maxwell and P. Pesic. Theory of heat. Courier Corporation, 2001.
[47] A. Mirchev, B. Kayalibay, M. Saelch, P. van der Smagt, and J. Bayer. Approximate bayesian inference in spatial environments. arXiv preprint 1rXivv1805.077
2018.
[48] A. Mirchev, B. Kayalibay, P. van der Smagt, and J. Bayer. Variational state-space models for localisation and dense 3d mapping in dof. arXiv preprint wwXiv:2006.10/70 2020.
[49] S. Mohamed and D. Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2125-2133. Curran Associates, Inc., 2015.
[50] S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated rein- forcement learning. In Advances in neural information processing systems, pages 2125-2133, 2015.
12

---

[51] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforvement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018.
[52] P.-Y. Oudeyer and F. Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neuroroboties, 1:6, 2009.
[53] P.-Y. Oudeyer. F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental develop-
ment. IEEE Iransactions on evolaionary computation, 11(2):265-22) 2007.
[54] T. Parr and K. J. Friston. Generalised free energy and active inference. Biological Cybernetics, 113(5):
495-513, Dee 2019, ISSN 1432-0770.
doi: 10. (0.10007:00022--19-000000
URL /tL.ttt:::/doo..or//
1007 0007/s00422-000 0002--099-000 0019-00805w.
[55] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
I.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 2019.
wwXiv:192.00/
[56] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven Exploration by Sepf-super Prodiction. 2017.
[57] D. Pathak, D. Gandhi, and A. Gupta. Self-Supervised Exploration via Disagreement. 2019.
[58] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent. space models. Learning for Decision Making and Control (LADC), 2021.
[59] J. Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
[60] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions Autonomous Mental Development, 2(3):230-247, 2010.
[61] Schulman, F. Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.
arXiv preprint 1707037777.03477,2 2017.
[62] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via self- supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
[63] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 2019.
[64] P. Shyam, W. Jaškowski, and F. Gomez. Model-based active exploration. arXiv preprint arXiv: 2018.
[65] B. C. Stadic, S. Levine, and P. Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:150.000814, 2015.
[66] S. Still and D. Precup. An information theorettc approach to curiosity- reinforcement learning. Theory in Biosciences,
2012.
167] Y. Sun, F. Gomez, and J. Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International Conference on Arrificial General Intelligence, pages 41-51. Springer, 2011 I.
[68] I. Szilard. Uher die entropieverminderung in einem thermodynamischen system hei eingriffen intelligenter wesen. Zeitschrift für Physik, 53(1 ):840-856, 1929.
[69] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. X. Chen, Duan, J. Schulman, F. DeTurek, and P. Abbeel. # exploration: A study of count- based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753-2762, 2017.
[70] K. Ueltzhöffer. Deep active inference. Biological Cybernetics, 112(6):547-573, 2018
[71] G. Vezzani, A. Gupta, I.. Natale, and P. Abbeel. Learning latent state representation for speeding up exploration. arXiv preprint a905:: 12621, 2019.
[72] M. Watter, J. T. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint arXiv:1506.07365 2015.
[73] G. Wayne, C.-C. Hung, D. Amos, M. Mirza, A. Ahuja, A. Grabska-Barwinska. J. Rae, Mirowski,
J. Z. Leibo, A. Santoro, et al. Unsupervised predictive memory in goal-directed agent, arXiv preprint arXiv: arXiv:1803. 10760, 2018.
[74] L. Williams. Information Theoretic Sensor Management. PhD thesis, Massachusetts Institute of Technology, 2007.
[75] K. Xu, S. Verma, C. Finn, and S. Levine. Continual lcarning of control primitives: Skill discovery via reset-games, arXiv preprint arXiv:201 152286, 2020.
[76] M. Zhang. S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine. Solar: Deep structured latent representations for model-bat based reinforcement learning. arXiv preprint 198 2018.
13

---

[77] R. Zhao, K. Lu, P. A.beel,, and Tromkin. Efficient empowerment estimation for unsupervised stabilization.
In International Conference Learning Representations, 2021. URL https://openrevier. net/ forum//id-u210pPcQ11.g
14


Reviews:
[{"review_id": "ydXMktJGQn", "reviewer": "Reviewer_48zc", "summary": "This paper studies the problem of designing intrinsic rewards that encourages active exploration in partially observable dynamic environments. The authors observe that by minimizing the entropy of the latent visitation frequency, the agent is able to achieve both the active information gathering and prevent the environment from changing. Specifically, the authors introduce Believer, a novel framework based on variational latent Bayes filters with discrete latent states, and designed several rewards. In the experiments, Believer has achieved good performance on 3 different tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is an interesting paper with good insights. Reinforcement learning in partially observable environments is always a challenging task, and a good intrinsic reward can significantly improve the exploration and the overall performance of the system. Believer has made a good observation of using the latent visitation frequency for encouraging both the active information gathering and interacting with the environment. Furthermore, the discrete latent states make evaluating the visitation frequency possible.\n\nOverall, I think the paper has made enough contributions. I have some minor suggestions about the presentation and the experiments. Please find detailed comments below.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1/ I think the current experiments are all conducted in relatively simple domains. It would be good to see Believer applied to more complex domains, e.g., robot navigation in REALISTIC environments. The current experiments have small spaces, and it would be good to see if the proposed intrinsic reward is still good for much more complex environments. In addition, a minor point is that it is a bit unclear if the proposed LSSM is still sufficient for modeling the complex environments.\n\n2/ Is it possible to visualize the learned latent states under different intrinsic rewards? I think it would be nice to have some visualizations of how the beliefs are being updated through time.\n\n3/ How does the size of the latent belief affect the performance? Will a larger belief size keep improving the performance? Or is it possible that with a slightly bigger belief size, other variants also achieve comparable performance with Believer?\n\n4/ Some notations are undefined. For example, in line 107 and 108, \\hat{p}(o) and \\hat{p}(s) are undefined.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "oYX7GDrtLj5", "reviewer": "Reviewer_6Sqq", "summary": "This paper introduces a new intrinsic reward signal for partially observable environments with dynamic objects that can potentially be controlled by an agent. This reward seeks to minimize the entropy of the agent's state visitation as estimated by a latent-space model.\nTo do so it makes use of a latent state estimation approach using a VAE and an ensemble of dynamics models, and specifies 4 reward functions that can be seen as optimizing the above objective through different means.\n\nThese proposed reward functions are evaluated on three domains with partial observability and dynamic objects that the agent's actions can somehow affect and potentially control. The experiments show that one or the other of the proposed reward functions lead to the agent controlling these dynamic objects, whereas previously proposed rewards fail to do so. Of note in the VizDoom domain, the paper shows that one of the proposed rewards leads to maximization of the extrinsic reward without learning to do so.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Technical Positives:\n* Taking the problem formulation used for intrinsic control to the harder domain of partial observability with dynamic objects is a step forward towards more complex domains.\n* This paper identifies what intrinsic control would mean in such an environment and contrasts it with approaches that have been used in the fully observable setting.\n* It then proposes reward functions that would lead to such control being exerted by the agent, by manipulating the entropy of its state belief and visitation.\n* These rewards seem to be validated in the three domains the paper evaluates on, though in practice it seems like the Niche expansion and Niche Creation + Infogain objectives are the ones that show results that are different and improvements on the baselines.\n\n## Questions, Remarks, and Drawbacks:\n* Since the proposed objective seeks to reduce the entropy of its state visitation, if left to its own devices it seems like the approach would also learn to go to a part of the environment where it does not encounter any dynamic objects and not move. The control aspect seems to arise from a well trained latent state model whose components include the encoding from observations to the latent state (which is explained appropriately in the paper) but also an exploration policy that learns both an accurate estimate of the latent state as well as an accurate estimate of the uncertainty of the dynamic objects. This exploration policy seems to not have been emphasized and explained enough in the paper.\n* Is the same exploration policy used to collect data for the baseline methods? Would those methods benefit from a similar exploration?\n* SMiRL is proposed as an intrinsic reward in CMPs that proposes rewards on the states encountered. This paper argues that the equivalent in the CHMP would be rewards computed based on observations. However I would argue that a reasonable equivalent would be to compute the SMiRL reward on the state estimated by the latent state encoder. Why is considering the observations a more appropriate analog for SMiRL? If the state used for estimating the reward in SMiRL is the latent state, then do the differences between the proposed rewards and SMiRL become more nebulous?\n* In Figure 4 (b), it would be helpful if the reward functions depicted are further labeled with the names in Section 5.1, so that correspondence can be drawn more readily.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussed in supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "lPH3jg4sMC", "reviewer": "Reviewer_xeVZ", "summary": "This paper presents a set of novel objectives for intrinsic motivation that aim to encourage an agent to learn to exercise control over its environment, despite only having partial observability. The key motivating idea is that the agent be encouraged to reduce \"Belief Entropy\" by limiting the ability of the environment to change, particularly when that part of the environment is not being observed. Four different objectives are presented that encourage the agent to learn policies that achieve this goal and results are demonstrated on a handful of toy environments meant to illustrate the central claims of the paper. Among the results is good performance on the DefendTheCenter VisDoom task, in which the agent learns to shoot enemies without specific instruction to do so, as this prevents the enemies from continuing to exist and \"surprising\" the agent in the future.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*High-level Comments*\n\nOverall the paper is clearly written and motivates an interesting problem and proposes a handful of intrinsic objectives that are shown to encourage the desired behavior. Yet while the core narrative of the paper is mostly solid (excepting some of the more minor comments below), the results are lacking and somewhat unconvincing. In particular, the results section is too short, and the results themselves do not support the authors' narrative in a way that leads to clear recommendations about when their approach (or which of their approaches) will perform better than others and when. The discussion of the results (aside from the tables, figures and captions themselves) exists across only a single paragraph in , problematically short considering that a number of the results do not \"speak for themselves\". There are a few such instances:\n\n- Across all the different environments shown, the Niche Creation + Infogain and the Niche Expansion objectives compete for the best performance, yet /why/ each objectives performs better or worse or how their behaviors differ in a way that could lead to a recommendation across environment types is not explored. In particular, the number of monster kills for the DefendTheCenter environment is unintuitively low compared to the Random Policy (and *zero* for the NC+I objective) despite that killing monsters is described as behavior that we might expect across all policies. This behavior should be discussed.\n- Second, the performance on the TwoRoom-Large environment includes only the Niche Creation + Infogain results. This is strange, seeing as it is the only environment for which this is that case.\n- Relatedly, the text highlights that the Niche Expansion performs well for the DefendTheCenter but it is unclear why this is the case or what the reader should hope to understand, important details to include if the goal is to prescribe how this approach can be used across environments.\n\nExpanding on the results section to address these points and to better understand when and why each objective is well-suited to its task would greatly strengthen the paper.\n\n*Smaller comments and suggestions*\n\n- To the best of my knowledge, the nature of the \"oracle policy\" is not elaborated upon. It is unclear how an agent with \"access to privileged about the environment state\" would achieve good behavior on the objectives of interest, which seek to maintain control over the belief. This would help to better clarify the results and the relation of the proposed techniques against this oracle.\n- The addition of the exploratory data and policy intuitively seems an important part of the ability of the chosen approach to succeed. Yet it appears that the approaches the authors compare against (SMiRL and Empowerment) do not include such a mechanism to augment data collection. It may help understanding if the authors comment on how this may influence the comparison between the various approaches.\n- The limitations of the approach should not exist only in the supplementary material. In particular, some discussion should be included in the paper regarding in what environments the proposed approach would be expected to either perform poorly or result in undesirable behavior would be a welcome addition.\n\nEdit: The authors have provided significant additional detail in response to my questions and I am satisfied by the depth of their investigation. While I still feel as if the approach has some limitations (and the authors have agreed to expand the discussion of these in the relevant section of their paper) I do not think these limitations diminish the quality of the work. I was also persuaded by the other reviews that I was perhaps too harsh on these grounds. I have updated my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Though the limitations and societal impact are touched upon in the supplementary materials, it is worth expanding on the impacts in a bit more detail. In particular, it would be meaningful to briefly expand on some example environments in which the proposed method would be likely to result in undesired or dangerous behavior.\n\nEdit: The authors have agreed to address this point in their updated draft, and I am satisfy by their reply in this regard.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "TAUvarCmHHk", "reviewer": "Reviewer_UHCB", "summary": "This paper introduces a novel latent state-space method for unsupervised reinforcement learning in dynamic partially-observed visual environments. The key insight of the paper is that minimizing the entropy of the belief over the latent state does not incentivize agents to construct a \"niche\", i.e. modify the environment such that it becomes more easily controlled. As a remedy, the authors propose to instead minimise the entropy of the belief over latent state visitation. The authors discuss a variety of reward incentivised constructed from the latent-space model and introduce a separate exploration policy. The authors evaluate on three separate single-agent environments, including one with continuous action spaces. It is found that the novel latent-visitation based reward incentives outperform existing methods on a variety of metrics.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper's fundamental idea about minimizing the entropy of the belief order latent-state visitation is original, and clearly marks a step change over previous approaches. The ideas are subsequently studied in appropriate detail and evaluated on a broad set of toy environments. The quality of both theoretical and empirical developments is high.\nThe paper is written very clearly. \n\nOverall, I believe this paper is very solid and deserves acceptance.\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In Table 1 (OneRoomCapture3D), how is it possible that Niche Expansion score significantly lower in entropy, while Niche Creation+Infogain wins wrt both obj capture and obj visible metrics?\n\nThe analogy with Maxwell's demon may deserve further detailed explanation: It is not clear to me what exactly the \"potential energy\"-analogy constitutes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Learning agents need to be able to explore and control their environment. The authors introduce a novel objective, the entropy of the state visitation, which combines these two desiderata and is tightly connected to information gain for controlling environments. 

The authors show promising results on a number of benchmarks created to highlight partial observability and clearly ablate the different components of their algorithm on these environments. 

The discussion focussed on a few key points: The authors provided a more detailed comparison to SMiRL, added additional experimental data as requested by the reviewers and clarified technical points. Overall, the reviewers felt that their concerns had been addressed appropriately by the authors.

Given the positive reviews, the discussion and the overall contributions of the paper, I recommend this paper to be accepted.

Decision: Accept (Poster)