Information is Power: Intrinsic Control via Information Capture

1 Introduction . Reinforcement learning offers a framework for learning control policies that maximize a given measure of reward – ideally , rewards that incentivize simple high-level goals , such as survival , accumulating a particular resource , or accomplishing some long-term objective . However , extrinsic rewards may be insufficiently informative to encourage an agent to explore and understand its environment , particularly in partially observed settings where the agent has a limited view of its environment . A generalist agent should instead acquire an understanding of its environment before a specific objective or reward is provided . This goal motivates the study of self-supervised or unsupervised reinforcement learning : algorithms that provide the agent with an intrinsically-grounded drive to acquire understanding and control of its environment in the absence of an extrinsic reward signal . Agents trained with intrinsic reward signals might accomplish tasks specified via simple and sparse rewards more quickly , or may acquire broadly useful skills that could be adapted to specific task objectives . Our aim is to design an embodied agent and a general-purpose intrinsic reward signal that leads to the agent controlling partially-observed environments when equipped only with a high-dimensional sensor ( camera ) and no prior knowledge . A large body of prior methods for self-supervised reinforcement learning focus on attaining coverage , typically through novelty-seeking or skill-discovery . As argued in prior work [ 18 , 16 , 14 , 6 ] , a compelling alternative to coverage suited to complex and dynamic environments is to minimize surprise , which incentivizes an agent to control aspects of its environment to achieve homeostasis within it – i.e . constructing and maintaining a niche where it can reliably remain despite external perturbations . We generally expect agents that succeed at minimizing surprise in complex environments to develop 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . similarly complex behaviors ; such acquired complex behaviors may be repurposed for other tasks [ 6 ] . However , these frameworks do not explicitly address the difficulty of controlling partially-observed environments : if an otherwise complex and chaotic environment contains a “ dark room ” ( small reliable niche ) , an agent could minimize surprise simply by hiding in this room and refusing to make meaningful observations , thereby failing to explore and control the wider surrounding environment . Consider Fig . 1 , which depicts a partially-observed outdoor environment with trees , a goat , weather , and an agent . We will discuss three different intrinsic incentives an agent might adopt in this environment . If the agent ’ s incentive is to ( i ) minimize the entropy of its next observation , it will seek the regions with minimal unpredictable variations in flora , fauna , and weather . This is unsatisfying because it merely requires avoidance , rather than interaction . Let us assume the agent will maintain a model of its belief about a learned latent state – the agent can not observe the true state , instead it learns a state representation . Further , let us assume the agent maintains a separate model of the visitation of its latent state – we will refer to this distribution as its latent visitation . If the agent ’ s incentive is to ( ii ) minimize the entropy of belief ( either at every step or at some final step ) , the agent will gather information and take actions to make the environment predictable : find and observe the changes in flora , fauna , and weather that are predictable and avoid those that aren ’ t . However , once it has taken actions to make the world predictable , this agent is agnostic to future change – it will not resist predictable changes in the environment . Finally , if the agent ’ s incentive is to ( iii ) minimize the entropy of its latent visitation , this will result in categorically different behavior : the agent will seek both to make the world predictable by gathering information about it and prevent it from changing . While both the belief and latent visitation entropy minimization objectives are worthwhile intrinsic motivation objectives to study , we speculate that an agent that is adept at preventing its environment from changing will generally learn more complex behaviors . We present a concise and effective objective for self-supervised reinforcement learning in dynamic partially-observed environments : minimize the entropy of the agent ’ s latent visitation under a latent state-space model learned from exploration . Our method , which we call Intrinsic Control by Information Capture ( IC2 ) , causes an agent to learn to seek out and control factors of variation outside of its immediate observations . We instantiate this framework by learning a state-space model as a deep variational Bayes filter along with a policy that employs the model ’ s beliefs . Our experiments show that our method learns to discover , represent , and control dynamic entities in partially-observed visual environments with no extrinsic reward signal , including in several 3D environments . Maxwell ’ s Demon and belief entropy The main concept behind our approach to self-supervised reinforcement learning is that incentivizing an agent to minimize the entropy of its beliefs about the world is sufficient to lead it to both gather information about the world and learn to control aspects of its world . Our approach is partly inspired by a well-known connection between information theory and thermodynamics , which can be illustrated informally by a version of the Maxwell ’ s Demon thought experiment [ 46 , 41 ] . Imagine a container separated into compartments , as shown in Fig . 2 . Both compartments contain gas molecules that bounce off of the walls and each other in a somewhat unpredictable fashion , though short-term motion of these molecules ( between collisions ) is predictable . The compartments are separated by a massless door , and the agent ( the eponymous “ demon ” ) can open or close the door at will to sort the particles.1 By sorting the particles onto one side , the demon appears to reduce the disorder of the system , as measured by the thermodynamic entropy , S , which increases the energy , F available to do work , as per Helmholtz ’ s free energy relationship , F = U − TS . The ability to do work affords the agent control over the environment . This apparent violation of the second law of thermodynamics is resolved by accounting for the agent ’ s information processing needed to make decisions [ 5 ] . By concentrating the particles into a smaller region , the number of states each particle visits is reduced . This illustrates an example environment in which reducing the entropy of the visitation distribution results in an agent gaining the ability to do work . In the same way that Maxwell ’ s demon accumulates free energy via informationgathering and manipulating its environment , we expect self-supervised agents guided by belief visitation entropy minimization to accumulate the equivalent of potential energy in their corresponding sequential decision processes , which leads them to gain control over their environments . Preliminaries Our goal in this work will be to design self-supervised reinforcement learning methods in partially observed settings to acquire complex behaviors that both gather information and gain control over their environment . To this end , we will formulate the learning problem in the context of a discrete-time partially-observed controlled Markov process , also known as a controlled hidden Markov process ( CHMP ) , which corresponds to a POMDP without a reward function . The CHMP is defined by a state space S with states s ∈ S , action space A with actions a ∈ A , transition dynamics P ( st+1|st , at ) , observation space Ω with observations o ∈ Ω , and emission distribution O ( ot|st ) . The agent is a policy π ( at|o≤t ) . Note that it does not observe any states s. We denote the undiscounted finite-horizon state visitation as dπ ( s ) .= 1/T ∑T−1 t=0 Prπ ( st = s ) , where Prπ ( st = s ) is the probability that st = s after executing π for t steps . Using dπ ( s ) , we can quantify the average disorder of the environment with the Shannon entropy , H ( dπ ( s ) ) . Prior work proposes surprise minimization ( minπ − log p̂ ( s ) ) as an intrinsic control objective [ 15 , 70 , 6 ] ; in Berseth et al . [ 6 ] ( SMiRL ) , the agent models the state visitation distribution , dπ ( s ) with p̂ ( s ) , which it computes by assuming access to s. In environments in which there are natural sources of variation outside of the agent , this incentivizes the SMiRL agent , fully aware of these variations observed through s , to take action to control them . In a partially-observed setting , it is a matter of interpretation of what the most natural extension of SMiRL is – whether ( 1 ) the observation should be treated as the state , or ( 2 ) an LSSM ’ s belief should be treated as an estimate of the latent state . Thus , our experiments include both interpretations , the latter constituting one of our methods . In the former , SMIRL ’ s model becomes p̂ ( o ) , which generally will enable the agent to ignore variations that it can prevent from appearing in its observations . We observe this phenomenon in our experiments . 2 Control and Information Gathering via Belief Entropy Minimization . The main question that we tackle in the design of our algorithm is : how can we formulate a general and concise objective function that can enable an RL agent to gain control over its partially-observed environment , in the absence of any user-provided task reward ? Consider the following partiallyobserved “ TwoRoom ” environment , depicted in Fig . 3 . The environment has two rooms : an empty ( “ dark ” ) room on the left , and a “ busy ” room on the right , the latter containing two moving particles that move around until the agent “ tags ” them , which stops their motion . Intuitively , an agent that aims to gather information and gain control of its environment should search for the moving particles to find out where they are . However , it is difficult to observe both particles at the same time . A more effective strategy is to “ tag ” the particles – then , their position remains fixed , and the agent will know where they are at all times , even when they are not observed . This task , remininscent of the 1In Maxwell ’ s original example , the demon sorts the particles into the two chambers based on velocity . Our example is actually more closely related to Szilard ’ s engine , which is based on a similar principle [ 68 , 45 ] . previously-discussed Maxwell ’ s Demon thought experiment , can be seen as a simple analogy for more complex settings that can occur in natural environments . Representing variability with latent state-space models . In order for the agent to represent the dynamic components of the environment observed from images , our method involves learning a latent state-space model ( LSSM ) [ 72 , 38 , 29 , 44 , 22 , 47 , 73 , 71 , 40 , 10 , 23 , 48 , 58 ] . We intermittently refer to these dynamic components as “ factors of variation ” to distinguish the model ’ s representation of variability in the environment ( latent state ) from the true variability ( true state ) . At timestep t , the LSSM represents the agent ’ s current belief as qφ ( zt|o≤t , a≤t−1 ) , where zt is the model ’ s latent state . We defer the description of the LSSM learning and architecture to Section 3 , and now motivate how we will use the LSSM for constructing an intrinsic control objective . Belief entropy and latent visitation entropy . Consider a policy that takes actions to minimize the entropy of the belief H ( qφ ( zt|o≤t , a≤t−1 ) ) . This corresponds to the agent performing active state estimation [ 12 , 74 , 37 ] , and is equivalent to taking actions to maximize expected latent-state information gain I ( ot , zt|o < t , at ) [ 2 ] . However , active state estimation is satisfied by a policy that simply collects informative observations , as it does not further incentivize actions to “ stabilize ” the environment by preventing the latent state from changing . Analogous to the standard definition of undiscounted state visitation , consider the undiscounted latent visitation : dπ ( z ) .= 1/T ∑T−1 t=0 Prπ ( zt = z ) , where Prπ ( zt = z ) = Eπqφ ( zt|o≤t , a≤t−1 ) ( the expected belief after executing π for t timesteps ) . Our goal is to minimizeH ( dπ ( z ) ) , as this corresponds to stabilizing the agent ’ s beliefs , which incentivizes both reducing uncertainty in each qφ ( zt|o≤t , a≤t−1 ) , as well as constructing a niche such that each qφ ( zt|o≤t , a≤t−1 ) concentrates probability on the same latent states . While in principle we could approximate this entropy and minimize it directly , later we discuss our more practical approach . Discovering factors of variation . In order for belief entropy minimization to incentivize the agent to control entities in the environment , the LSSM ’ s belief must represent the underlying state variables in some way and model their uncertain evolution until either observed or controlled . For example , the demon in the thought experiment in Section 1 would have no incentive to gather the particles if it did not know that they existed . If the agent is unable to encounter or represent dynamic objects in its LSSM , it could choose not to move . However , if the LSSM properly captures the existence and motion of dynamic objects ( even out of view ) , then the agent has an incentive to exert control over them , rather than avoid them by going to parts of the environment where it does not encounter them . While sufficient random exploration may result in a good-enough LSSM , making this approach generally practical requires a suitable exploration strategy to collect the experience necessary to train an LSSM that represents all of the underlying factors of variation . To this end , we learn a separate exploratory policy to maximize expected model information gain , similar to Schmidhuber [ 60 ] , Houthooft et al . [ 26 ] , Gheshlaghi Azar et al . [ 20 ] , Sekar et al . [ 62 ] . The exploration policy is employed as a way to encounter dynamic objects and collect data on which the model performs poorly . The exploration policy does not need to learn accurate estimates of the latent state and its uncertainty – it is merely incentivized to visit where the current model is inaccurate . Expected information gain about model parameters θ is relative to a set of prior experience D and a partial trajectory ht−1 , given as I ( ot ; θ|ht−1 , D ) = EotKL ( p ( θ|ot , ht−1 , D ) || p ( θ|ht−1 , D ) ) . Note that model information gain is distinct from the information an agent may gather to reduce its belief entropy H ( qφ ( zt|o≤t , a≤t−1 ) ) within the current episode . Computing the full model parameter prior p ( θ|ht−1 , D ) and posterior p ( θ|ot , ht−1 , D ) is generally computationally expensive , and also requires evaluating an expectation over observations – instead , we approximate this expected information gain following a method similar to Sekar et al . [ 62 ] : we use an ensemble of latent dynamics models , E = { pθi ( zt|zt−1 , at−1 ) } Ki=1 to compute the variance of latent states estimated by qφ ( zt|o≤t , a≤t−1 ) . We build the ensemble throughout training using the method of Izmailov et al . [ 27 ] . Thus , the exploration reward is given as : re = Var { θi } [ log pθ ( zt|zt−1 , at−1 ) |zt ∼ qφ ] .

Summary: This paper introduces a new intrinsic reward signal for partially observable environments with dynamic objects that can potentially be controlled by an agent. This reward seeks to minimize the entropy of the agent's state visitation as estimated by a latent-space model. To do so it makes use of a latent state estimation approach using a VAE and an ensemble of dynamics models, and specifies 4 reward functions that can be seen as optimizing the above objective through different means.  These proposed reward functions are evaluated on three domains with partial observability and dynamic objects that the agent's actions can somehow affect and potentially control. The experiments show that one or the other of the proposed reward functions lead to the agent controlling these dynamic objects, whereas previously proposed rewards fail to do so. Of note in the VizDoom domain, the paper shows that one of the proposed rewards leads to maximization of the extrinsic reward without learning to do so.