Action and Perception as Divergence Minimization
Authors: Danijar Hafner
Pedro A Ortega
Jimmy Ba
Thomas Parr
Karl Friston
Nicolas Heess
Venue:  (2021)
Keywords: objective functions
reinforcement learning
information theory
probabilistic modeling
control as inference
exploration
intrinsic motivation
world models

Abstract: We introduce a unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions to align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's representations and inputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as a latent variable model suggests powerful world models as a path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common language for comparing a wide range of objectives, advances the understanding of latent variables for decision making, and offers a recipe for designing novel objectives. We recommend deriving future agent objectives the joint divergence to facilitate comparison, to point out the agent's target distribution, and to identify the intrinsic objective terms needed to reach that distribution.

Under review as a conference paper at ICLR 2021
ACTION AND PERCEPTION AS DIVERGENCE MINIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce unified objective for action and perception of intelligent agents. Extending representation learning and control, we minimize the joint divergence between the combined system of agent and environment and a target distribution. Intuitively, such agents use perception to align their beliefs with the world, and use actions align the world with their beliefs. Minimizing the joint divergence to an expressive target maximizes the mutual information between the agent's s representations and imputs, thus inferring representations that are informative of past inputs and exploring future inputs that are informative of the representations. This lets us explain intrinsic objectives, such as representation learning, information gain, empowerment, and skill discovery from minimal assumptions. Moreover, interpreting the target distribution as latent variable model suggests powerful world models as path toward highly adaptive agents that seek large niches in their environments, rendering task rewards optional. The framework provides a common language for comparing wide range of objectives, advances the understanding latent variables for decision making, and offers a recipe for designing novel objectives. We recommend deriving future agent objectives the joint divergence facilitate comparison, to point out the agent's 's target distribution, and to identify the intrinsic objective terms needed to reach that distribution.
Missing Data
Controllable Future
Latent Representations
Low Entropy Preferences
Variational Inference
Information Gain
Input Density Exploration
Expected Reward
Filtering
Encoder
Latent Skills
Point Estimate
Latent Actions
Factorized Target
Maximum
Amortized
Skill Discovery
Empower- ment
MaxEnt
Likelihood
Inference
Reward
Perception
Action
Both
Figure 1: Overview of methods connected by the introduced framework of action and perception divergence minimization. Each latent variable leads to mutual information term between said variable and the data. The mutual information with past inputs explains representation learning. The mutual information with future inputs explains information gain, empowerment, and skill discovery. By leveraging multiple latent variables for the decision making process, agents can naturally combine multiple of the objectives. This figure shows the methods that drive from the rell-established KL divergence and analogous method trees can be derived by choosing different divergence measures.

---

Under review as a conference paper at ICLR 2021
INTRODUCTION
To achieve goals in complex environments, intelligent agents need to perceive their environments and choose effective actions. These two processes, perception and action, are often studied in isolation. Despite the many objectives that have been proposed in the fields of representation learning and reinforcement learning, it remains unclear how the objectives relate to each other and which fundamentally new objectives remain yet to be discovered. Based on the KI. divergence (Kullback and Leibler. 1951), we propose unified framework for action and perception that connects wide range of objectives to facilitate our understanding of them while providing recipe for designing novel agent objectives. Our findings are conceptual in nature and this paper includes no empirical study. Instead, we offer unified picture of wide range of methods that have been shown to be successful in practice in prior work. The contributions of this paper are described as follows.
Unified objective function for perception and action We propose joint KL minimization as principled framework for designing and comparing agent objectives. KL minimization was proposed separately for perception as variational inference (Jordan et al., 1999; Alemi and Fischer, 2018) and for actions as KL control (Todorov. 2008; Kappen et al., 2009). Based on this insight, we formulate action and perception as jointly minimizing the KL from the world to unified target distribution. The target serves both as the model to infer representations and as reward for actions. This extends variational inference controllable imputs, while extending KL control to latent representations. We show a novel decomposition of joint KL divergence that cxplains several representation learning and exploration objectives. Divergence minimization additionally connects deep reinforcement learning to the free energy principle (Friston, 2010; 2019), while simplifying and overcoming limitations of its active inference implementations (Friston et al., 2017) that we discuss in Appendix B.
Understanding latent variables for decision making Divergence minimization with an expres- sive target maximizes the mutual information between inputs and latents. Agents thus infer repre- sentations that are informative of past inputs and explore future inputs that are informative of the representations. For the past, this yields reconstruction (Hinton al., 2006; Kingma and Welling, 2013) or contrastive learning (Gutmann and Hyvärinen, 2010; Oord et al., 2018). For the future, it yields information gain exploration (Lindley et al., 1956). Stochastic skills and actions are realized over time, so their past terms are constant. For the future, they lead to empowerment (Klyubin al., 2005) and skill discovery (Gregor al., 2016). RL as inference (Rawlik et al., 2010) does not maximize mutual information because its target is factorized. To optimize a consistent objective across past and future, latent representations should be accompanied by information gain exploration.
Expressive world models for large ecological niches The more flexible an agent's 's target or model, the better the agent can adapt to its environment. Minimizing the divergence between the world and the model, the agent converges to natural equilibrium or niche where it can accurately predict its inputs and that it can inhabit despite external perturbations (Schrödiinger, 1944; Wiener, 1948; Haken, 1981; Friston, 2013; Berseth et al., 2019). While surprise minimization can lead to trivial solutions, divergence minimization encourages the niche to match the agent's model class, thus visiting all inputs proportionally to how well they can be understood. This suggests designing expressive world models of sensory inputs (Ebert et al., 2017; Hafner et al., 2018; Gregor et al., 2019) as a path toward building highly adaptive agents, while rendering task rewards optional.
2
FRAMEWORK
This section introduces the framework of action and perception as divergence minimization (APD). To unify action and perception, we formulate the two processes as joint KL minimization with a shared target distribution. The target distribution expresses the agent's 's preferences over system configurations and is also the probabilistic model under which the agent infers its representations. Using an expressive model as the target maximizes the mutual information between the latent variables and the sequence of sensory inputs, thus inferring latent representations that are informative of past inputs and exploring future inputs that are informative of the representations. We assume knowledge of basic concepts from probability and information theory that are reviewed in Appendix D.
2.1 JOINT KL MINIMIZATION
Consider stochastic system described by joint probability distribution over random variables. For example, the random variables for supervised learning are the inputs and labels and for an agent they are the sequence of sensory inputs, internal representations, and actions. More generally, we combine
2

---

Under review as a conference paper at ICLR 2021
Formulation
Preferences
Latent Entropy
Input Entropy
Divergence Minimization
Active Inference
X
Expected Reward
Table High-level comparison of different agent objectives. All objectives express preferences over system configurations scalar value. Active inference additionally encourages entropic latents. Divergence minimization additionally encourages entropic inputs. Active inference makes additional choices about the optimization, detailed in Appendix and the motivation for our work is in part to offer a simpler alternative to active inference. We show that when using expressive models as preferences, the entropy terms result in wide range of task-agnostic agent objectives.
all input variables into x and the remaining variables that we term latents into z. We will see that different latents correspond to different representation learning and exploration objectives.
The random variables are distributed according to their generative process or actual distribution Ps. Parts of the actual distribution can be unknown, such as the data distribution, and parts can be influenced by varying the parameter vector o, such as the distribution stochastic representations or actions. As counterpart to the actual distribution, we define the desired target distribution over the same support. describes our preferences over system configurations and can be unnormalized,
Actual distribution: 2, PO(I,
Target distribution: T(Z, =).
(1)
We formulate the problem of joint KL minimization changing the parameters to bring the actual distribution of all random variables as close as possible to the target distribution, as measured by the KL divergence (Kullback and Leibler, 1951; Li et al., 2017; Alemi and Fischer, 2018).
(2) All expectations and KLs throughout the paper are integrals under the actual distribution, so they can be estimated from samples of the system and depend d. Equation 2 is the reverse KI information projection used in variational inference (Csiszár and Matus, 2003).
min KL [P x)
|| =]]
Examples For representation learning, Pe is the joint of data and belief distributions and t is latent variable model. Note that we use Ps to denote not the model under which we infer beliefs but the generative process of inputs and their representations. For control, Ps the trajectory distribution under the current policy and corresponds to the utility of the trajectory. The parameters o include everything the optimizer can change directly, such as sufficient statistics of representations, model parameters, and policy parameters.
Target parameters There are two ways to denote deterministic values within our framework, also known as MAP estimates in the probabilistic modeling literature (Bishop, 2006). We can either use fixed target distribution and use latent variable that follows point mass distribution (Dirac, 1958), or we explicitly parameterize the target using deterministie parameter as Tg. In either case, refers to the fixed model class. The two approaches are equivalent because in both cases the target receives a deterministic value that has no entropy regularizer. For more details, see Appendix A.I.
Assumptions Divergence minimization uses only two inductive biases, namely that the agent optimizes an objective and that it uses random variables to represent uncertainty. Choosing the well- established KL the divergence measure an additional assumption. It corresponds to maximizing the expected log probability under the target while encouraging high entropy for all variables in the system avoid overconfidence, as detailed in Appendix C. Common objectives with different degrees of entropy regularization are summarized in Table 1.
Generality Alternative divergence measures would lead to different optimization dynamics, dif- ferent solutions if the target cannot be reached, and potentially novel objectives for representation learning and exploration. Nonetheless, the KL can describe any converged system, trivially by choosing its actual distribution the target, and thus offers simple and complete mathematical perspective for comparing a wide range of specific objectives that correspond to different latent variables and target distributions.
3

---

Under review as a conference paper at ICLR 2021
action
perception
explanation
prediction
action
min KL
po(2 m) po(a)
r(æ, =)
.
Q
beliefs
inputs
target
Σ>
perception
world
rewards
(a) Action perception cycle
(b) Actual distribution Pe (c) Target distribution T4.
Figure 2: Action and perception minimize the joint KI. divergence to a unified target distribution that can be interpreted aligns the agent's s belief's with past inputs while actions align future inputs with its belief'. There are many ways to specify the target, for example as a latent variable model that explains past inputs and predicts future inputs and an optional reward factor that is shown as a filled square.
learning probabilistic model of the system. Given the target, perception
2.2 INFORMATION BOUNDS
We show that for expressive targets that capture dependencies between the variables in the system, minimizing the joint KL increases both the preferences and the mutual information between inputs and latents This property allows divergence minimization explain wide range of existing representation learning and exploration objectives. We use the term representation learning for inferring deterministic or stochastic variables from inputs, which includes local representations of individual inputs and global representations such as model parameters.
Latent preferences The joint KL can be decomposed in multiple ways, for example into marginal KL plus a conditional KL or by grouping marginal with conditional terms. To reveal the mutual information maximization, information seeking term. The decomposition can be done either with the information term expressed over imputs and the preferences expressed over latents or the other way around.
decompose the joint KL into a preference seeking term and an
KT. [2 =) || (xx, =)]
F. KT. [Po( [PT(( | x)) || [()]]
F. T =) - In (x)]
joint divergence
realizing latent preferences
information bound
All expectations throughout the paper are over all variables, under the actual distribution, and thus depend on the parameters Φ The first term on the right side of Equation is a KL regularizer that keeps the belief (=(= I) over latent variables close to the marginal latent preferences T(z). The second term a variational bound on the mutual information I The bound is expressed in input space. Maximizing the conditional In [(2 222 seeks latent variables that accurately predict inputs while minimizing the marginal In Po (x) seeks diverse inputs.
| (Barber and Agakov, 2003).
Variational free energy When the agent cannot influence its inputs, such when learning from a fixed dataset, the input entropy E In pz(x)] is not parameterized and can be dropped from Equation 3. This yields the free energy or ELBO objective used by variational inference to infer approximate posterior beliefs in latent variable models (Hinton and Van Camp, 1993; Jordan et al., 1999). The free energy regularizes the belief p=(= æ) to stay close to the prior T(2) while reconstructing inputs via r(x z). However, in reinforcement and active learning, inputs can be influenced and thus the input entropy should be kept, which makes the information bound explicit.
Input preferences Analogously, we decompose the joint KL the other way around. The first term on the right side of Equation 4 is KL regularizer that keeps the conditional input distribution Po(x 2) close to the marginal input preferences Txx). This term is analogous to the objective in KL control (Todorov, 2008; Kappen et al., 2009), except that the inputs now depend upon latent variables via the policy. The second term is again variational bound on the mutual information T [<:; =]. this time expressed in latent space. Intuitively, the bound compares the belief 7(2 xx) after observing the inputs and the belief (2) before observing any inputs to measure the gained information,
KL (ps,,,, =) (xx,))
E KKLL[pp> (I 2) [(x)] E [n x) In p+]
(4)
joint divergence
realizing input preferences
information bound
The information bounds are tighter the better the target conditional approximates the actual condi- tional, meaning that the agent becomes better at maximizing mutual information as it learns more about the relation between the two variables. This requires an expressive target that captures correla- tions between inputs and latents, such a latent variable model or deep neural network. Maximizing the mutual information accounts for both learning latent representations that are informative of inputs well exploring inputs that are informative of the latent representations.
4

---

Under review as a conference paper at ICLR 2021
2.3 MODELS AS PREFERENCES
The target distribution defines our preferences over system configurations. However, we can also interpret it as probabilistic model, or mergy-based model if unnormalized (LeCun et al., 2006). This because minimizing the joint KL infers beliefs over latent variables that approximate the posteriors under the model, as shown in Section 2.2. Because the target is not parameterized, it corresponds to the fixed model class, with parameters being inferred as latent variables, optionally using point mass distributions. As the agent brings the actual distribution closer to the target, the target also becomes a better predictor of the actual distribution. Divergence minimization thus emphasizes that the model class simply expresses preferences over latent representations and inputs and lets us interpret inference as bringing the joint of data and belief distributions toward the model joint.
Input preferences Minimizing the joint divergence also minimizes the divergence between the agent's 's input distribution p²(x)) and the marginal input distribution under its target or model 7(x). The marginal input distribution of the model is thus the agent's 's preferred input distribution, that the agent aims to sample from in the environment. Because T(x) marginalizes out all latent variables and parameters, it describes how well an input sequence x can possibly be described by the model class, as used in the Bayes factor (Jeffreys, 1935; Kass and Raftery, 1995). Divergence minimizing agents thus seek out inputs proportionally to how their models can learn to predict them through inference, while avoiding inputs that are inherently unpredictable given their model class. Because the target can be unnormalized, we can combine latent variable model with reward factor of the form exp(r(x)) to create a target that incorporates task rewards. The reward factor adds preferences for certain inputs without affecting the remaining variables in the model. We describe examples of such reward factors this in Appendix A.4 and Section 3.1.
Action perception cycle Interpreting the target as a model shows that divergence minimization is consistent with the idea of perception as inference suggested by Helmholtz (Helmholtz, 1866; Gregory, 1980). Expressing preferences as models is inspired by the free energy principle and active inference (Friston, 2010; Friston et al., 2012; 2017), which we compare to in Appendix B. Divergence minimization inherits an interpretation of action and perception from active inference that we visualize in Figure 2a. While action and perception both minimize the same joint KL, they affect different variables. Perception is based on inputs and affects the beliefs over representations, while actions based on the representations and affect inputs. Given unified target, perception thus aligns the agent's 's beliefs with the world while actions align the world with its beliefs.
Niche seeking The information bounds responsible for representation learning and exploration are tighter under expressive targets, as shown in Section 2.2. What happens when we move beyond task rewards and simply define the target as a flexible model? The more flexible the target and belief family, the better the agent can minimize the joint KL. Eventually, the agent will converge natural equilibrium or ecological niche where it can predict its inputs well and that it can inhabit despite external perturbations (Wiener, 1948; Ashby, 1961). Niche seeking connects to surprise minimization (Schrödinger, 1944; Friston, 2013; Berseth et al., 2019), which aims to maximize the marginal likelihood of imputs under a model. In environments without external perturbations, this can lead to trivial solutions once they are explored. Divergence minimization instead aims to match the marginal input distribution of the model. This encourages large niches that cover all inputs that the agent can learn to predict. Moreover, it suggests that expressive world models lead to autonomous agents that understand and inhabit large niches, rendering task rewards optional.
2.4 PAST AND FUTURE
Representations are computed from past inputs and exploration targets future inputs. To identify the two processes, we thus need to consider how an agent optimizes the joint KL after observing past inputs I< and before observing future inputs z;, as discussed in Figure 2b. For example, past inputs can be stored in an experience dataset and future inputs can be approximated by planning with a learned world model, on-policy trajectories, joint KL on past inputs, we first split the information bound in Equation into two smaller bounds on the past mutual information I [x<2>
replay of past inputs (Sutton, 1991). To condition the
=] and additional future mutual information I [ 2 s=)].
E In [(2 2) - In px)]
= E In (2 x) - In p (z x<) + In po x2)) - [n)
information bound
2 E
In (2 x) - In (= x<)) + In (x)) - pn) ]. (5)
future information bound
pasi information bound
5

---

Under review as a conference paper at ICLR 2021
Latent
Target
Past Term
Future Term
Agents
Actions
Factorized
Action entropy
A3C, SQL, SAC VIM, ACIE, EPC
Actions
Expressive
Empowerment
Skills
Expressive
Skill discovery
VIC, SNN, DIAYN, VALOR NDIGO, DVBF-LM
States
Expressive
Repr. learning
Information gain
Parameters
Expressive
Model learning
Information gain VIME, MAX, Plan2Explore
Table 2: Divergence minimization accounts for wide range of agent objectives. Each latent variable used by the agent contributes a future objective term. Moreover, latent variables that are not observed over time, such latent representations and model parameters, additionally each contribute past objective term. Combining multiple latent variables combines their objective terms. Refer to Section 3 for detailed derivations of these individual examples and citations of the listed agents.
Equation 5 splits the belief update from the prior Pv(z) to the posterior 7(2 x)) into two updates via the intermediate belief p xz) and then applies the variational bound from Barber and Agakov (2003) to allow both updates to be approximate. Splitting the information bound lets us separate past and future terms in the joint KL, or even separate individual time steps. It also lets us separately choose express terms in input or latent space. This decomposition is one of our main contributions and shows how the joint KI. divergence accounts for both representation learning and exploration,
KL (po( 2) 7 (Ix,
I E. [po(= x<)) [=)]]
E In (3>< 2) - In p& (3+x
realizing pasi latent preferences
representation learning
+ EKL Po( x<. 2)
r(0>> x<)]
E I z z) In po xx2)
realizing future input preferences
exploration
Conditioning past inputs V< removes their expectation and renders px(x=) constant. While some latent variables in the set 2 are never realized, such as latent state estimates or model parameters, other latent variables become observed over time, such stochastic actions or skills. Because the agent selects the values of these variables, we have to condition the objective terms on them as causal interventions (Pearl, 1995; Ortega and Braun, 2010). In practice, this means replacing all occurrences of = by the unobserved latents Σy and conditioning those terms on the observed latents do(zz). To keep notation simple, we omit this step in our notation.
To build an intuition about Equation 6, we discuss the four terms on the right-hand side. The first two terms involve the past while the last two terms involve the future. The first term keeps the agent's belief (=(2| xz)) close to the prior r(z) to incorporate inductive biases. The second term encourages the belief to be informative of the past inputs so that the inputs are reconstructed by 7( | 2), where is (x(x2) is a constant because && are observed. The third term is the control objective that steers toward future inputs that match the preferred input distribution T xx)). The fourth term is an information bound that seeks out future inputs that are informative of the latent representations in and encourages actions or skills in z that maximally influence future inputs.
The decomposition shows that the joint KL accounts for both learning informative representations of past inputs and exploring informative future inputs as two sides of the same coin. From this, we derive several representation and exploration objectives by including different latent variables in the set z. These objectives are summarized in Table 2 and derived with detailed examples in Section
Representation learning Because past inputs are observed, the past information bound only affects the latents. Expressed as Equation 3, it leads to reconstruction (Hinton al., 2006), and as Equation 4, it leads to contrastive learning (Gutmann and Hyvärinen, 2010; Oord et al., 2018). This accounts for local representations of individual inputs, as well as global representations, such as latent parameters. Moreover, representations can be inferred online amortized using an encoder (Kingma and Welling, 2013). Latents with point estimates are equivalent to target parameters and thus are optimized jointly to tighten the variational bounds. Because past actions and skills realized, their mutual information with realized past inputs is constant and thus contributes no past objective terms.
Exploration Under a flexible target, latents in z result in information-maxt maximizingiiiii exploration. For latent representations, this is known as expected information gain and encourages informative future inputs that convey the most bits about the latent variable, such world model parameters, policy parameters, or state estimates (Lindley et al., 1956; Sun et al., 2011). For stochastic actions, fully factorized target leads maximum entropy RI. An expressive target yields empowerment,

---

Under review as a conference paper at ICLR 2021
maximizing the agent's influence over the world (Klyubin et al., 2005). For skills, it yields skill discovery or diversity that learns distinct modes behavior that together cover many different trajectories (Gregor al., 2016; Florensa et al., 2017; Eysenbach et al., 2018; Achiam et al., 2018).
EXAMPLES
We use the framework of action and perception as divergence minimization presented in Section 2 to derive a wide range of concrete objective functions that have been proposed in the literature, shown in Figure 1. For this, we analyze the cases of different latent variables and factorizations of the actual and target distributions. These derivations serve as practical examples for producing new objective functions within framework. We start by describing maximum entropy RL because of its popularity in the literature. Due to space constraints, we refer to Appendix A for the remaining examples, which include variational inference, amortized inference, filtering, KL control, empowerment, skill discovery, and information gain.
Designing novel objectives In practice, agent is determined by its target distribution, belief family, and optimization algorithm. Our framework thus suggests to break down the implementation of agent into the same three components that are typically considered in probabilistic modeling. As Section 2 showed, the target distribution is also the model under which the agent infers its beliefs about the world. We also saw that more expressive models allow agents to increase the mutual information between their inputs and latents more. To design an agent that learns a lot about the world, we should thus design expressive world model and use them as the target distribution. For example, these could include latent state estimates, latent parameters, latent skills, hierarchical latents, or temporal abstraction. Each world model corresponds to a new agent objective.
3.1 MAXIMUM ENTROPY RL
Actual p
Target T
Maximum entropy RL (Williams and Peng. 1991;
Kappen et al., 2009; Rawlik et al., 2010; Tishby and Polani, 2011; Fox et al., 2015; Schulman et al., 2017; Haarnoja et al., 2018) chooses stochastic actions to maximize task reward while remaining close an action prior. The action prior is typically independent of the inputs, corresponding The objective thus does not contain a mutual informa- tion term. Despite factorized targets being common in practice, we suggest that expressive targets, such as world models, are preferable in the longer term.
a
a>
ar
a&
a factorized target.
Figure 3: Maximum Entropy RL
Figure shows the actual and target distributions for maximum entropy RL. The input sequence is {ar). the graphical model, these are grouped into past actions and inputs azz, future actions as. and future inputs x> The actual distribution consists of the fixed environment dynamics and a stochastic policy. The target consists of a reward factor, an action prior that is often the same for all time steps, and the environment dynamics,
x {ar} } and the action sequence
Actual:
Po(x, p(æ, (===-1 (a Fi:t. Q1:c-1),
environment
policy
Target:
a(x,a) & II. exp(p((x) ((z)) p(za
F1 01:-11 (ax).
roward
enviroament
action prior
Minimizing the joint KL results in a complexity regularizer action space and the expected reward. Including the environment dynamics in the target cancels out the curiosity term in the expected reward case in Appendix A.4, leaving maximum entropy RL to explore only in action space. Moreover, including the environment dynamics in the target gives up direct control over the agent's input preferences, as they depend not just on the reward but also the environment dynamics marginal. Because the target distribution is factorized and does not capture dependencies between r and a, maximum entropy RL does not maximize their mutual information,
KL pp
r] Σ,
EKL (p æi 01:-11) T(a))] E [r())))
complexity
expected reward
The action complexity KL can be simplified into an entropy regularizer by choosing a uniform action prior as in SQL. (Haarnoja al., 2017) and SAC (Haarnoja et al., 2018). The action prior
7

---

Under review as a conference paper at ICLR 2021
can also depend the past inputs and incorporate knowledge from previous tasks in Distral (Teh et al., 2017) and work by Tirumala et al. (2019) and Galashov et al. (2019). Divergence minimization motivates combining maximum entropy RL with input density exploration by removing the environment dynamics from the target distribution. The resulting agent aims to converge to the input distribution that is proportional to the exponentiated task reward. Moreover, divergence minimization shows that the difference between maximum entropy RL and empowerment, that we describe in Appendix A.5, is the target factorizes actions and inputs or captures their dependencies.
4 RELATED WORK
Divergence minimization Various problems have been formulated as minimizing divergence between two distributions. TherML (Alemi and Fischer, 2018) studies representation learning as KL minimization. We follow their interpretation of the data and belief actual distribution, although their target is only defined by its factorization. ALICE (Li et al., 2017) describes ad adversarial learning as joint distribution matching, while Kirsch et al. (2020) unify information- -based objectives. Ghasemipour et al. (2019) describe imitation learning as minimizing divergences between the inputs of learned and expert behavior. None of these works consider combined representation learning and control. Thompson sampling minimizes the forward KL to explain action and perception as exact inference (Ortega and Braun, 2010). In comparison, we optimize the backward KL to support intractable models and connect to wide range of practical objectives.
Active inference The presented framework is inspired by the free energy principle, which studies the dynamics of agent and environment as stationary SDEs (Friston, 2010; 2019). We inherit the interpretations of active inference, which implements agents based on the free energy principle (Friston al., 2017). While divergence minimization matches the input distribution under the model, active inference maximizes the probability of inputs under it, resulting in smaller niches. Moreover, active inference optimizes the exploration terms only with respect to actions, which requires specific action prior. Finally, typical implementations of active inference involve an expensive Bayesian model average over possible action sequences, limiting its applications to date (Friston et al., 2015; 2020). We compare to active inference in detail in Appendix B. Generalized free energy (Parr and Friston, 2019) studies a unified objective similar to ours, although its entropy terms are defined heuristically rather than derived from a general principle.
is well known that RL can be formulated as KL minimization over inputs and actions (Todorov, 2008; Kappen et al., 2009; Rawlik al., 2010; Ortega and Braun, 2011: 1; Levine, 2018), as well as skills (Hausman et al., 201 8; Tirumala et al., 2019; Galashov et al., 2019). We build upon this literature and extend agents with latent representations, leading to variational inference on past inputs and information seeking exploration for future inputs. Divergence minimization relates the above methods and motivates an additional entropy regularizer for inputs (Todorov, 2008; Lee et al., 201 9b; Xin et al., 2020). SLAC (Lee et al., 2019a) combines representation learning and control but does not consider the future mutual information, so their objective changes over time. In comparison, we derive the terms from general principle and point out the information gain that results in an objective that is consistent over time. The information gain term may also address concerns about maximum entropy RL raised by O' D'Donoghue et al. (2020).
Control as inference
CONCLUSION
We introduce a general objective for action and perception of intelligent agents, based on minimizing the KL divergence. To unify the two processes, we formulate them joint KL minimization with shared target distribution. This target distribution is the probabilistic model under which the agent infers its representations and expresses the agent's preferences over system configurations. We summarize the key takeaways as follows:
Unified objective for action and perception Divergence minimization with an expressive target maximizes the mutual information between latents and inputs. This leads to inferring representations that are informative of past inputs and exploration of future inputs that informative of the representations. To optimize a consistent objective that does not change over time, any latent representation should be accompanied by a corresponding exploration term.
Understanding of latent variables for decision making Different latents lead to different objective terms. Latent representations are never observed, leading to both representation learning
8

---

Under review as a conference paper at ICLR 2021
and information gain exploration. Actions and skills become observed over time and thus do not encourage representation learning but lead to generalized empowerment and skill discovery.
Adaptive agents through expressive world models Divergence minimization agents with an expressive target find niches where they can accurately predict their inputs and that they can inhabit despite external perturbations. The niches correspond to the inputs that the agent can learn to understand, which is facilitated by the exploration terms. This suggests designing powerful world models as a path toward building autonomous agents, without the need for task rewards.
General recipe for designing novel objectives When introducing new agent objectives, recommend deriving them from the joint KL by choosing a latent structure and target. For information maximizing agents, the target an expressive model, leaving différent latent structures to be explored. Deriving novel objectives from the joint KL facilitates comparison, renders explicit the target distribution, and highlights the intrinsic objective terms needed reach that distribution.
Discovering new families of agent objectives Our work shows that family representation learning and exploration objectives can be derived from minimizing joint KL between the system and a target distribution. Different divergence measures give rise to new families of such agent objectives that could be easier to optimize converge to better optima for infeasible targets. We leave exploring those objective families and comparing them empirically as future work.
Without constraining the class of targets, our framework is general and can describe any system. This by itself offers framework for comparing many existing methods. However, interpreting the target as a model further suggests that intelligent agents may use especially expressive models as targets. This hypothesis should be investigated in future work by examining artificial agents with expressive world models or by modeling the behavior of natural agents as divergence minimization. Acknowledgements Hidden for review.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. arXiv preprint 18X. 2018.
A. A. Alemi and I. Fischer. TherML: Thermodynamies of machine learning. arXiv preprint arXiv:1807.04162, 2018.
S. Amari. A theory of adaptive pattern classifiers. IEEE Transactions Electronic Computers, pages 299-307, 1967.
P. Ao, C. Tian-Qi, and S. Jiang-Hong. Dynamical decomposition of markov processes without detailed balance. Chinese Physics Letters, 30(7):070201, 2013.
W. R. Ashby. An introduction to cybernetics. Chapman & Hall Ltd, 1961.
M. G. Azar, B. Piot, B. A. Pires, Grill, F. Altché, and R. Munos. World discovery models. arXiv preprint arXiv:1902.07660 2019.
K. Azizzadenesheli, E. Brunskill, and A. Anandkumar, Efficient exploration through bayesian deep Q-Networks. 2018 Information Theory and Applications Workshop (ITA), pages 1-9, IEEE, 2018.
D. Barber and F. V. Agakov. The IM algorithm: variational approach to information maximization. In Advances in neural information processing systems, 2003.
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
G. Berseth, D. Geng, Devin, Finn, D. Jayaraman, and S. Levine. Smirl: Surprise minimizing in dynamic environments. arXiv preprint
2019.
C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
C. Blundell, Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty neural networks. arXiv preprint arXiv: 150505540 2015.
S. R. Bowman, L.. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences 2015.
from continuous space. arXiv preprint
L. D. Brown. A complete class theorem for statistical problems with finite sample spaces. The Annals of Statistics, pages 1309, 1981.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv 2018.
preprint
O. Chang, Y. Yao, D. Williams- Williams-King, and Lipson. Ensemble model patching: A parameter-efficient variational bayesian neural network. arXiv preprint arXiv: 1905. 09453, 2019.
Chen, Kornblith, M. Norouzi, and Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
I. Csiszár and F. Matus. Information projections revisited. IEEE Transactions Information Theory, 1976):147,,149,, 2003.
L. Da Costa, T. Parr, N. Sajid, S. Veselic, V. Neacsu, and K. Friston. Active inference discrete state-spaces: a synthesis. arXiv preprint arXiv:2001.07200 2020.
P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel. The Helmholtz machine. Neural computation, 7(5):889-900) 1995.
I. M. de Abril and R. Kanai. A unified strategy for implementing curiosity and empowerment driven reinforcement learning. arXiv preprint arXiv: (oX.v:/806606610 2018.
10

---

Under review as a conference paper at ICLR 2021
J. Denker, D. Schwartz, B. Wittner, S. Solla, R. Howard, L. Jackel, and J. Hopfield. Large automatic learning, rule extraction, and generalization. Complex Systems, 1(5):877-92) 1987.
A. M. Dirac. The principles of quantum mechanics. Oxford university press, 1958.
M. W. Dusenberry, G. Jerfel, Y. Wen, Y-a. Ma, Snock, K. Heller, B. Lakshminarayanan, and D. Tran. Efficient and scalable bayesian neural nets with rank-1 factors. arXiv preprint arXiv:2005.07186. 2020.
Ebert, C. Finn, A. X. Lee, and S. Levine sulf-sue visual planning with temporal skip connections. arXiv preprint arXiv:1710.05228, 2017.
F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprini arXiv:/812.00568, 2018,
S. A. Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garell,, A. Ruderman, A. A. Rusu, Danihelka, K. Gregor, et Neural scene representation and rendering, Science, 360(6394): 1204-1210, 2018.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: learning skills without reward function. arXiv preprint arXiv:1802.06070, 2018.
I. Fischer. The conditional entropy bottleneck. arXiv preprint arXiv:2002.05373 2020.
C. Florensa, Y. Duan, and Abbeel. Stochastic neural networks for hierarchical reinforcement learning, arXiv preprint (704::/770400122 2017.
R. Fox, A. Pakman, and N. Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint
2015.
K. Friston. The free-energy principle: unified brain theory? Nature reviews neuroscience, 11(2): 127-138, 2010.
K. Friston. Life as we know it. Journal of the Royal Society Interface, 10(86):2013045,, 2013.
K. Friston. A free energy principle for particular physics. arXiv preprint
10184, 2019.
K. Friston, R. Adams, and R. Montague. What is aalte-aaccumlltter reward or evidence? Frontiers in neurorobotics, 6:11, 2012.
K. Friston, Rigoli, Ognibene, C. Mathys, T. Fitzgerald, and G. Pezzulo. Active inference and epistemic value. Cognitive neuroscience, 6(4):187-214 2015.
K. Friston, T. FitzGerald, F. Rigoli, P. Schwartenbeck, and G. Pezzulo. Active inference: a process theory. Neural computation, 29(1): 1-49, 2017.
K. Friston, L. Da Costa, D. Hafner, C. Hesp, and T. Parr. Sophisticated inference. arXiv preprint arXiv:2006. 04120, 2020.
A. Galashov, S. M. Jayakumar, L. Hasenclever, D. Tirumala, J. Schwarz, G. Desjardins, W. M. Czarnecki, Y. W. Teh, R. Pascanu, and N. Heess. Information asymmetry kl-regularized rl. arXiv preprint arXiv: 19050099 2019.
Z. Ghahramani and M. Jordan. Learning from incomplete data, 1995.
S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. arXiv preprint arXiv:1911.02256. 2019.
Ghosh and Doshi Velez. Model selection bayesian neural networks via horseshoe priors. arXiv preprint arXiv: 1705. 2017.
K. Gregor, D. J. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint 16tv:1611.000 1.07507 2016.
K. Gregor, D. J. Rezende, Besse, Y. Wu, H. Merzic, and A. v. Oord. Shaping belief states with generative environment models for rl. arXiv preprint arXiv: 1906 09237, 2019.
11

---

Under review as a conference paper at ICLR 2021
R. L. Gregory. Perceptions hypotheses. Philosophical Transactions of the Royal Society of London.
B, Biological Sciences, 290 (0008))):11-113 1980.
Z. D. Guo, M. G. Azar, B. Piot, B. A. Pires, T. Pohlen, and R. Munos. Neural predictive belief representations. arXiv preprint arXiv: 2018.
M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
estimation principle for
D. Ha and J. Schmidhuber. World models. arXiv preprint arXtv:1803.10122. 2018.
2018.
D. Ha, A. Dai, and Q V. Le. Hypemetworks. arXiv preprint
2016.
T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1352-1361. JMLR. org. 2017.
T. Haarojj, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprini urXiv: 2018.
D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 18/0:/5511.4,,, 2018.
D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint
2019a.
D. Hafner, Tran, A. Irpan, T. Lillicrap, and J. Davidson. Reliable uncertainty estimates in deep neural networks using noise contrastive priors. In Conference on Uncertainty in Artificial Intelligence, 2019b.
H. Haken. The science of structure: Synergetics. Van Nostrand Reinhold, 1981.
K. Hausman, T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
H. Helmholtz. Concerning the perceptions in general. Treatise on physiological optics, 1866.
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint 1200011 2015.
G. E. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
G. E. Hinton, S. Osindero, and Y.-W. Teh. A fast learning algorithm for deep belief nets. Neural computation, 1877):1527-1554 2006.
R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 109-11117, 2016.
R. A. Howard and E. Matheson, Risk-sensitive markov decision processes. Management science, 18(7):56-369, 1972.
D. A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings the IRE, 4099): 1952.
A. Immer, M. Korzepa, and M. Bauer. Improving predictions of bayesian neural networks via local linearization. arXiv preprint arXiv:2008..840 2020.
Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 1803.05000 2018.
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1116611105550 1.05997, 2016.
12

---

Under review as a conference paper at ICLR 2021
E. T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
W. H. Jefferys and O. Berger. Ockham's razor and bayesian analysis. American Scientist, 80(1):
64 64-72, 1992.
H. Jeffreys. Some tests of significance, treated by the theory of probability. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 31, pages 203-222. Cambridge University Press, 1935.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. Saul, An introduction to variational methods for graphical models. Machine learning, 37(2):183 1999.
R. E. Kalman. A new approach to linear filtering and prediction problems. Journal of basic Engiieeriigg 82(1):35-45, 1960.
H. J. Kappen, V. Gómez, and M. Opper. Optimal control graphical model inference problem. Machine learning, 8792):159-182, 2009.
M. Karl, M. Soelch, J. Bayer, and P. van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv: 1605006400 2016.
M. Karl, M. Soelch, P. Becker-Ehmck, D. Benbouzid, P. van der Smagt, and Bayer. Unsupervised real-time control through variational empowerment. arXiv preprint arXiv: 17Xvv:1710.05101.201 2017.
R. E. Kass and A. E. Raftery, Bayes factors. Journal of the american statistical association, 90(400): 773-795, 1995.
Y. Kim, S. Wiseman, A. C. Miller, D. Sontag, and A. M. Rush. Semi-amortized variational autoen- coders. arXiv preprint arXiv:1802.02550. 2018.
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
A. Kirsch, C. Lyle, and Gal. Unpacking information bottlenecks: Unifying ciltommation-hhhoreett objectives in deep learning. arXiv preprint arXiv:2003. /2537, 2020.
A. Klyubin, D. Polani, and C. Nehaniv. Empowerment: universal agent-centric measure of control. In 2005 IEFE Congress on Evolutionary Computation, volume 1, pages 128-135. IEEE, 2005.
R. G. Krishnan, U. Shalit, and D. Sontag. Deep kalman filters. arXiv preprint 1rXiv:1511.05) 2015.
Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
Lange and M. Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In The 2010 International Joint Conference on Neural Networks (IJCN), pages 1-8. IEEE, 2010.
Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial energy-based learning. Predicting structured data, 1(0), 2006.
A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv:1907.0m 000500 2019a.
L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching, arXiv preprint arXiv:
2019b.
Leibfried, S. Pascual-Diss and Grau-Moya. unified bellman optimality principle combining reward maximization and empowerment. In Advances in Neural Information Processing Systems, pages 7869-7880, 2019.
Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint /005.00900 2018.
13

---

Under review as a conference paper at ICLR 2021
C. Li, H. Liu, C. Chen, Y. Pu, L. Chen, R. Henao, and L. Carin, Alice: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems, pages 5495-5503, 2017.
D. V. Lindley et al. On a measure of the information provided by an experiment. The Annals of Mathematical Statisttics, 27(4):986-1005, 1956.
C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference Machine Learning, pages 1708-1716, 2016.
C. Louizos and M. Welling. Multiplicative normalizing flows for variational bayesian neural networks. arXiv preprinl arXiv: 7703.0196, 2017.
Y.-A. Ma, T. Chen, and Fox. complete recipe for stochastic gradient meme. In Advances in Neural Information Processing Systems, pages 2917-2925, 2015.
D. MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3):448-44 472, 1992a.
D. MacKay. Information-basee objective functions for active data selection. Neural computation. 4(4):590 -004, 1992b.
D. MacKay, Information theory, inference and learning algorithms. Cambridge university press, 2003.
A. Mirchev, Kayalibay, M. Soelch, P. van der Smagt, and J. Bayer. Approximate bayesian inference in spatial environments. arXiv preprint arXiv:1805.07200 2018.
S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-2133, 2015.
O. Morgenstern and Von Neumann. Theory of games and economic behavior. Princeton university press, 1953.
Moutarlier and R. Chatila. Stochastic multisensory data fusion for mobile robot location and environment modelling. 5th int. In Symposium on Robotics Research, page 207, 1989.
K. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
B. O' O'Donoghue, I. Osband, and C. lonescu. Making sense of reinforcement learning and probabilistic inference. arXiv preprint arXiv:2001.00800 2020.
M. Okada, N. Kosaka, and T. Taniguchi. Planet of the bayesians: Reconsidering and improving deep planning network by incorporating bayesian inference. arXiv preprint arXiv:2003.000 00370, 2020.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1007:3807.0337 1807.03748. 2018
D. A. Ortega and P. A. Braun. Information, utility and bounded rationality. In International Conference on Artificial General Intelligence, pages 269-274. Springer, 2011.
P. A. Ortega and D. A. Braun. A minimum relative entropy principle for learning and acting, Journal of Artificial Intelligence Research, 38:475-511. 2010.
I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In Advances in neural information processing systems, pages 4026-4034, 2016.
P.YY Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions evolutionary computation, 11(2):265-286, 2007.
T. Parr and K. J. Friston. Generalised free energy and active inference. Biological cybernetics, 113 455-):493-533, 2019.
14

---

Under review as a conference paper at ICLR 2021
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. puriosity-driver exploration by self- self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.
Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995.
C. Peterson. A mean field theory learning algorithm for neural networks. Complex systems, 995-1019, 1987.
V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint arXivv/903..0
2019.
B. Poole, S. Ozair, A. v. d. Oord, A. A. Alemi, and G. Tucker. On variational bounds of mutual information. arXiv preprint arXiv:1905.06922 2019.
W. Pratt. Risk aversion in the small and the large. Econometrica, 32(1/2): 1964.
K. Rawlik, M. Toussaint, and S. Vijayakumar. Approximate inference and stochastic optimal control. arXiv preprint arXiv: 1009.3998, 2010.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1401.4000 2014.
R. Rudin. Complex analysis. 1966.
C. Salge, C. Glackin, and D. Polani. Approximation of empowerment in the continuous domain. Advances in Complex Systems, 16(02n03): 2013.
C. Salge, C. Glackin, and D. Polani. Changing the environment based on empowerment as intrinsic motivation. Entropy, 16(5):2789 28189 2014.
N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly. Episodic curiosity through reachability. arXiv preprint arXiv: 1.X0.02//0 2018.
Schmidhuber. Curious model-buildngg control systems. In (Proceedings) 1991 IEEE International Joint Conference on Neural Networks, pages 1458-1463. IEEE, 1991.
E. Schrödinger. What is life? The physical aspect of the living cell and mind. Cambridge University Press Cambridge, 1944.
Schulman, X. Chen, and P. Abbeel. Equivalence between policy gradients and soft q-learning. 2017.
arXiv preprini arXiv:1704.000
R. Sekar, O. Rybkin, Daniilidis, P. Abbeel, Hafner, and D. Pathak. Planning explore via self-supervised world models. arXiv preprint 2rXiv:2005.05960, 2020.
T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. arXiv preprint arXiv:2006. 16232, 2020.
C. F. Shannon. mathematical theory of communication. Bell system technical journal, 27(3): 379-423, 1948.
A. Sharma, S. Gu, S. Levine, V. Kumar, and Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1rXiv:/907.01633 2019.
P. Shyam, W. Jaskowski, and F. Gomez. 10X.v:1810.12162. 2018.
Model-based active exploration. arXiv preprint
R. Stratonovich. Markov's conditional processes. Teoriya Veroyatn. Primen, 5:172-195, 1960.
S. Sun, C. Chen, and I.. Carin. Learning structured weight uncertainty in bayesian neural networks. In Artificial Intelligence and Statistics, pages 1283-1292, 2017.
S. Sun, G. Zhang, J. Shi, and R. Grosse. Functional variational bayesian neural networks, arXiv preprint arXiv: arXiv: /903.09333503 005797 2019.
15

---

Under review as a conference paper at ICLR 2021
Y. Sun, F. Gomez, and J. Schmidhuber. Planning be surprised: Optimal bayesian exploration in dynamic environments. In International Conference on Artificial General Intelligence, pages 41-51, Springer, 2011.
R. Sutton. Dyna, integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4):160-163; 1991.
R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 12(1 12:1111, 1999.
Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pages 4499-4509, 2017.
D. Tirumala, H. Noh, A. Galashov, L. Hasenclever, A. Ahuja, G. Wayne, R. Pascanu, Y. W. Teh, and N. Heess. Exploiting hierarchy for learning and transfer kl-regularized rl. arXiv preprint arXiv: 1903.00383 2019.
N. Tishby and D. Polani. Information theory of decisions and actions. In Perception-action cycle, pages 601-636. Springer, 2011.
E. Todorov. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control, pages 4286-4292, IEEE, 2008
D. Tran, M. Dusenberry, M. van der Wilk, and Hafner. Bayesian layers: module for neural network uncertainty. In Advances in Neural Information Processing Systems, pages 14660-14672, 2019.
M. Tschannen, Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucie. On mutual information maximization for representation learning. arXiv preprint arXiv: 13625, 2019.
A. Wald. An essentially complete class of admissible decision functions. The Annals of Mathematical Statistics, pages 549-555, 1947.
Y. Wen, P. Vicol, J. Ba, D. Tran, and R. Grosse. Flipout: Efficient pseudo-independent weight perturbations on mini-batches. arXiv preprint IrXiv:1803.0455 2018.
N. Wiener. Cybernetics or Control and Communication in the Animal and the Machine. MIT press, 1948.
R. J. Williams and J. Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
B. Xin, H. Yu, Y. Qin, Q. Tang. and Z. Zhu. Exploration entropy for reinforcement learning. Mathematical Problems Engineering, 2020, 2020.
D. Yarats, A. Zhang, I. Kostrikov, B. Amos, Pineau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint 191.:1111 2019.
G. Zhang, S. Sun, D. Duvenaud, and R. Grosse. Noisy natural gradient variational inference. In International Conference on Machine Learning, pages 5852-5861, 2018.
M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. Johnson, and S. Levine. Solar: deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, 2019.
R. Zhao, P. Abbeel, and S. Tiomkin. Efficient online estimation of empowerment for reinforcement learning. arXiv preprint arXiv:2007.07356, 2020.
16

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL EXAMPLES
This section leverages the presented framework to explain a wide range of objectives in a unifying review, as outlined in Figure 1. For this, include different variables in the actual distribution, choose different target distributions, and then rewrite the joint KL to recover familiar objectives. We start with perception, the case with latent representations but uncontrollable inputs and then turn to action, the case without latent representations but with controllable inputs. We then turn to combined action and perception. The derivations follow the general recipe described Section The same steps can be followed for new latent structures and target distributions to yield novel agent objectives.
A.1 VARIATIONAL INFERENCE
Actual p
Target T
Following Helmholtz, we describe perception as in- ference under model (Helmholtz, 1866; Gregory, 1980; Dayan et al., 1995). Inference computes a pos- terior over representations by conditioning the model on inputs. Because this has no closed form in general, variational inference optimizes parameterized belief to approximate the posterior (Peterson, 1987; Hinton and Van Camp, 1993; Jordan et al., 1999).
w
y
y
Figure 4 shows variational inference for the example of supervised learning using a BNN (Denker et al., 1987: MacKay, 1992a; Blundell et al., 2015). The inputs images = {.r and their classes } and we infer the latent parameters w as a global representation of the data set (Alemi and Fischer, 2018). The parameters depend on the inputs only through the optimization process that produces ø. The target consists of a parameter prior and a conditional likelihood that uses the parameters predict classes from images,
Figure 4: Variational Inference
Actual:
Po(x, y, (o(w)l II, p(r;, ys).
belief
data
Target:
T(x, y, w) ((w) II, 7
71/s 24, w)
prior
likelihood
Applying the framework, we minimize the KL. between the actual and target joints. Because the data distribution is fixed here, the input marginal p(x, y) is constant. In this case, the KL famously results in the free energy or ELBO objective (Hinton and Van Camp, 1993; Jordan et al., 1999) that trades off remaining close to the prior and enabling accurate predictions. The objective interpreted as the deseription length of the data set under entropy coding (Huffman, 1952; MacKay, 2003) because measures the nats needed for storing both parameter belief and prediction residuals,
be
KL pp |l r] = KL[Po(w) [w)]
In x. w)] + E [In p(x,
(10)
complexity
accuracy
constant
Variational methods for BNNs (Peterson, 1987; Hinton and Van Camp, 1993; Blundell et al., 2015) differ in their choices of prior and belief distributions and inference algorithm. This includes hierarchical priors (Louizos and Welling, 2016; Ghosh and Doshi Velez, 2017), data priors (Louizos and Welling, 2016; Hafner et al., 2019h; Sun et al., 2019), flexible posteriors (Louizos and Welling, 2016; Sun al., 2017: Louizos and Welling, 2017: Zhang al., 2018: Chang al., 2019), low rank posteriors (Izmailov et al., 2018; Dusenberry et al., 2020), and improved inference algorithms (Wen al., 2018: Immer et al., 2020). BNNs have been leveraged for RL for robustness (Okada al., 2020: Tran et al., 2019) and exploration (Houthooft et al., 2016; Azizzadenesheli et al., 2018).
Target parameters While expressive beliefs over model parameters lead global search for their values, provide uncertainty estimates for predictions, and enable directed exploration in the RL setting, they can be computationally expensive. When these properties are not needed, we can choose point mass distribution Ps (w) + dg(w) - [1 if & else 0} to simplify the expectations and avoid the entropy and mutual information terms that are zero for this variable (Dirac, 1958),
KL (p)(w)
r(w)]
E In T(y ax, w)]
In (
In T x. Φ)
In T (v) x))
(11)
complexity
nocuracy
complexity
accuracy
paramnterrized Larget
Point mass beliefs result in MAP or maximum likelihood estimates (Bishop, 2006: Murphy, 2012) that are equivalent parameterizing the target Tạ- Parameterizing the target is thus notational
17

---

Under review as a conference paper at ICLR 2021
choice for random variables with point mass beliefs. Technically, also require the prior target parameters to be integrable but this true in practice where only finite parameter spaces exist.
A.2 AMORTIZED INFERENCE
Actual p
Target T
Local representations represent individual inputs. They can summarize inputs more compactly, enable interpolation between inputs, and facilitate general- ization to unseen inputs. In this case, we can use amortized inference (Kingma and Welling, 2013; Rezende et al., 2014; Ha et al., 2016) to learn an encoder that maps each input its corresponding belief. The encoder is shared among inputs to reuse computation. It can also compute beliefs for new inputs without further optimization, although opti- mization can refine the belief (Kim al., 2018).
2
S
Figure 5: Amortized Inference
Figure shows amortized inference on the example of a VAE (Kingma and Welling, 2013; Rezende al., 2014). The inputs are images = {x; and we infer their latent codes {zz}. The actual distribution consists of the unknown and fixed data distribution and the parameterized encoder Pz (z)). The target probabilistic model defined as the prior over codes and the decoder that computes the conditional likelihood of each image given its code. We parameterize the target here, but one could also introduce an additional latent variable infer a distribution over decoder parameters in Appendix A.1,
Actual:
Pe(x, 2) = II, p(x.) po(zi xz).
data
encoder
(12)
Target:
Tx( 2) = II To (T; 2.) (2x)
decoder
prior
Because the data distribution is still fixed, minimizing the joint KI. again results in the variational free energy or ELBO objective that trades of prediction accuracy and belief simplicity. However, by including the constant input marginal, we highlight that the prediction term is variational bound on the mutual information that encourages the representations to be informative of their inputs,
KL po || T
E. [pp(2 2r) || 7 r=)] E In ( =) - In p(x))
(13)
complexity
information bound
In input space, the information bound leads reconstruction in DBNs (Hinton al., 2006), VAEs (Kingma and Welling, 2013; Rezende et al., 2014), and latent dynamics (Krishnan et al., 2015; 5; Karl al., 2016). latent space, it leads to contrastive learning as NCE (Gutmann and Hyvärinen, 2010), CPC (Oord et al., 2018; Guo et al., 2018), CEB (Fischer, 2020), and SimCLR (Chen et al., 2020). To maximize their mutual information, x and 2 should be strongly correlated under the target distribution, which explains the empirical benefits of ramping up the decoder variance throughout learning (Bowman et al., 2015; Eslami al., 2018) or scaling the temperature of the contrastive loss (Chen et al., 2020). The target defines the variational family and includes inductive biases (Tschannen al., 2019). Both forms have enabled learning world models for planning (Ebert et al., 2018; Ha and Schmidhuber, 2018; Zhang et al., 2019; Hafner et al., 2018; 2019a) and accelerated RL (Lange and Riedmiller, 2010; Jaderberg et al., 2016; Lee et al., 2019a; Yarats et al., 2019; Gregor et al., 2019).
A.3 FUTURE INPUTS
Actual p
Target T
Before moving actions, discuss perception with unobserved future inputs that are outside of our con- trol (Ghahramani and Jordan, 1995). This is typical in supervised learning where the test set is unavailable during training (Bishop. 2006), in online learning where training inputs become available over time (Amari, 1967), and in filtering where only inputs up to the current time are available (Kalman, 1960).
z
Φ
x<
Σ>
I<
Figure 6 shows missing inputs the example of filtering with an HMM (Stratonovich, I 1960; Kalman, 1960; Karl al., 2016), although the same graphical model applies to supervied learning with BNN
Figure 6: Future Inputs
18

---

Under review as a conference paper at ICLR 2021
representation learning with VAE given train and test data sets. The imputs 2: = {z<, } consist of past images Z< and future images V that follow an unknown and fixed data distribution. We represent the input sequence using chain of corresponding compact latent states. However, the representations are computed only based on x< because Ø> is not yet available, as expressed in the factorization of the actual distribution,
Actual:
Po(a,2) = p(2>, Po xx).
data
belief
(14)
Target:
T0( = Ta( 2)
To TT 1=) T(2)
likelihood
prediction
prior
Bayesian assumption Bayesian reasoning operates within the model class and makes the assump- tion that the model class is correct. Under this assumption, the future inputs 2> ~ p(r> (x<.2) =
p(z>> x<) follow the target distribution Ty (a>> I<,,) 2) = ( 2). This renders the diver- gence of future inputs given the other variables zero, so that yy does not need to be considered for optimization, recovering standard variational inference from Appendix A.1,
KL [L To
KL (o(x<, 2))
EKL p(>>> | x<) [T( 2)]
(15)
variational inference
uncontrolled future
Assuming that future inputs follow the model distribution appropriate when the model accurately reflects knowledge about future inputs. However, the assumption does not always hold, for example for data augmentation or distillation (Hinton et al., 2015) that generate data from another distribution improve the model. Importantly assuming that future inputs already follow the target is not appropriate when they can be influenced, because there would be no need to intervene.
A.4 CONTROL
Actual p
Target T
We describe behavior an optimal control problem where the agent chooses actions to move its distri- bution of sensory inputs toward preference distri- bution over inputs that can be specified via rewards (Morgenstern and Von Neumann, 1953; Lee al., 2019b). We first cover deterministic actions that lead KL control (Kappen al., 2009; Todorov, 2008) and input density exploration (Schmidhuber, 1991; Bellemare al., 2016; Pathak et al., 2017).
H<
Σ>
x>
Figure 7: Control
Figure shows deterministic control with the input sequence = {ar) that the agent deterministic policy, control rule, or plan. In the graphical model, we group the input sequence into internal latent variables. The target describes the preferences over input sequences that can be unnormalized,
partially influence by varying the parameters 6 of the
past inputs xx and future inputs x> There
Actual:
(z( = II. Po( 21:111,, contrilld dynamics
(16)
Target:
+xx) = Ile [3,, 11
preferences
Minimizing the KL between the actual and target joints maximizes log preferences and the input entropy. Maximizing the input entropy is a simple form of exploration known as input density exploration that encourages rare inputs and aims for uniform distribution over inputs (Schmidhuber, 1991; Oudeyer et al., 2007). This differs from the action entropy of maximum entropy RI. in Section 3.1 and information gain in Appendix A.7 that takes inherent stochasticity into account,
KL. Po r] E. E [In 7(x, 11)-1)] +
[P 0 1)]
(17)
expected preferences
curiosity
Task reward Motivated by risk-sensitivity (Pratt, 1964; Howard and Matheson, 1972). KL control
(Kappen et al., 2009) defines the preferences as exponential task rewards 7 (xx, x1a--1 ) exp(r(x:)). KL-regularized control (Todorov, 2008) defines the preferences with an additional passive dynamics 21:-1). Expected reward (Sutton and Barto, 2018)
term T(x, 21:-1) & exp(r(z;))r'nx) corresponds to the preferences (x (1:(-1) & exp(r(2)))pp(z)
F14-1) that include the controlled dynamics. This cancels out the curiosity term in the joint KL, leading to a simpler objective that does not encourage rare inputs, which might limit exploration of the environment.
19

---

Under review as a conference paper at ICLR 2021
Input density exploration Under divergence minimization, maximizing the input entropy is not an exploration heuristic but an inherent part of the control objective. In practice, the input entropy is often estimated by learning density model of individual inputs in pseudo-counts (Bellemare et al., 2016), latent variable models as in SkewFit (Pong et al., 2019), unnormalized models as in RND (Burda al., 2018), and con-parmmeteric models as in reachability (Savinov et al., 2018). More accurately, it can be estimated by sequence model of inputs as in ICM (Pathak et al., 2017). The expectation over inputs estimated by sampling episodes from either the actual environment, replay buffer, or learned model of the environment (Sutton, 1991).
A.5 EMPOWERMENT
Actual p
Target T
Remaining in the stochastic control setting of Sec- tion 3.1, consider different target distribution that predicts actions from inputs. This corresponds an exploration objective that term generalized empowerment, which maximizes the mutual infor- mation between the sequence of future inputs and future actions. It encourages the agent to influence its environment in as many ways as possible while avoiding actions that have no predictable effect.
a>
Ø
Ø
ax<
I>
QT
Figure 8: Empowerment
Figure shows stochastic control with an expressive target that captures correlations between inputs and actions. The input sequence is x = {xx, and the action sequence is = {ax). In the graphical model, these are grouped into past actions and inputs aze, future actions a>, and future inputs T>- The actual distribution consists of the environment and the stochastic policy. The target predicts actions from the inputs before and after them using a reverse predictor. We use uniform input preferences here, but the target can also include an additional reward factor as in Section 3.1,
Actual:
Pa a) II. p(æ, #1:-1, 01.1 (o 11: 01:1-1),
environment
policy
(18)
Target:
To(x, a) & Il. (a(a F1.r, (1:r,)
reverse predictor
Minimizing the joint KL reveals an information boudn between future actions and inputs and a control term that maximizes input entropy and, specified, task rewards. Empowerment (Klyubin al., 2005) was originally introduced as potential empowerment to "keep your options open" and was later studied realized empowerment "use your options" (Salge et al., 2014). Realized empow- erment maximizes the mutual information I [ axat+h I1s,, 11 Divergence minimization generalizes this to the mutual information [ du: 1iit, 41 between the sequences of future actions and future inputs. The k-step variant is recovered by target that conditions the reverse predictor on fewer inputs. Realized empowerment measures agent's influence on its environment and can be interpreted maximizing information throughput with the action marginal (a aa-1) as source, the environment as noisy channel, and the reverse predictor as decoder,
T KL p
EKL a) [(x)]
E In (a x) In pa (a)].
generalized empowerment
(19)
E (a x) - In
Σ In (at
|x, x1, -
In (a (1=-1)
generalized empowerment
decoder
source
Empowerment has been studied for continuous state spaces (Salge et al., 2013), for image inputs (Mohamed and Rezende, 2015), optimized using a variational bound (Karl al., 2017), combined with input density exploration (de Abril and Kanai, 2018) and task rewards (Leibfried et al., 2019), and used for task-agnostic exploration of locomotion behaviors (Zhao et al., 2020). Divergence minimization suggests generalizing empowerment from the input steps ahead to the sequence of all future inputs. This can be seen as combining empowerment terms of different horizons. Moreover, offer principled motivation for combining empowerment with input density exploration. In comparison to maximum entropy RL in Section 3.1, empowerment captures correlations between x and in its target distribution and thus leads information maximization. Moreover, it encourages the agent to converge to the input distribution that is proportional to the exponentiated reward.
A.6 SKILL DISCOVERY
20

---

Under review as a conference paper at ICLR 2021
Actual p
Target T
Many complex tasks be broken down into se- quences of simpler steps. To leverage this idea, we condition a policy temporally abstract options or skills (Sutton et al., 1999). Skill discovery aims to learn useful skills, either for specific task without rewards to solve downstream tasks later on. Where empowerment maximizes the mutual informa- tion between inputs and actions, skill discovery can be formulated as maximizing the mutual information between inputs and skills (Gregor et al., 2016).
2>
27
Ø
I>
20 <
Σ>
Figure Skill Discovery
Figure shows skill discovery with the input sequence x [ action sequence a {ar), , and the sequence of temporally abstract skills = {- The graphical model groups the sequences into past and future variables. The actual distribution consists of the fixed environment, an abstract policy that selects skills by sampling from fixed distribution shown here the low-level policy that selects actions based on past inputs and the current skill. The target consists of action prior and reverse predictor for the skills and could further include reward factor,
function of past inputs, and
Actual:
Pa(a, a, 2)
TTVK abstract policy
(=(()) I Pa
F1:s, (1/K1,
p(a, 111-1,0014-11
policy
environment
Target:
To(x, I1
T/K
(z reverse prodictor
I T(ae).
(20)
action prior
Minimizing the joint KL results in control term in Appendix A.5, complexity regularizer for actions as in Section 3.1, and variational bound on the mutual information between the sequences of inputs and skills. The information bound a generalization of skill discovery (Gregor et al., 2016; Florensa et al., 2017). Conditioning the reverse predictor only inputs that align with the duration of the skill recovers skill discovery. Maximizing the mutual information between skills and inputs encourages the agent to learn skills that together realize as many different input sequences as possible while avoiding overlap between the sequences realized by different skills,
KL pe
Ta]
EKL p(
a) || rz)] + E KL (a(a) [L[(aaa x, 2) r(a)] E In To x) In p=(2)
complexity
skill discovery
(21)
VIC (Gregor et al., 2016) introduced information- based skill discovery as an extension of empower- ment, motivating line of work including SNN (Florensa et al., 2017), DIAYN (Fysenbach et al., 2018), work by Hausman al. (2018), VALOR (Achiam et al., 2018), and work by Tirumala et al. (2019) and (Shankar and Gupta, 2020). DADS (Sharma al., 2019) estimates the mutual information in input space by combining a forward predictor of skills with a contrastive bound. Divergence minimization suggests generalization of skill discovery where actions should not just consider the current skill but also seek out regions of the environment where many skills are applicable.
A.7 INFORMATION GAIN
Actual p
Target T
Agents need to explore initially unknown environ- ments to achieve goals. Learning about the world is beneficial even when it does not serve maximiz- ing the currently known reward signal, because the knowledge might become useful later during this or later tasks. Reducing uncertainty requires repre- senting uncertainty about aspects we want explore, such as dynamics parameters, policy parameters, or state representations. To efficiently reduce uncer- tainty, the agent should select actions that maximize the expected information gain (Lindley et al., 1956).
x<
Σ>
D
Figure 10: Information Gain
Figure 10 shows information gain exploration deterministic actions. The inputs are a sequence x (x+) } and the latent parameters are global representation The graphical model separates inputs into past inputs V~ and future inputs Z>. The actual distribution consists of the controlled dynamics and the parameter belief. Amortized latent state representations would include link from & z. Latent policy parameters would include
the example of latent model parameters and
21

---

Under review as a conference paper at ICLR 2021
link from w to .>. The target distribution is latent variable model that explains past inputs and predicts future inputs, as in Appendix A.3. The target could further include a reward factor,
Actual:
Po(z, = px(w) II Po 11:0-1),
helief
controlled dynamics
(22)
Target:
[(z, w) = ( II. [ 11 W)
prior
likeiboodd
Minimizing the KL between the two joints reveals a control term as in previous sections and the information bound between inputs and the latent representation, as derived in Section 2.2. In contrast to Appendix A.3, we can now influence future inputs. This leads to learning representations that are informative of past inputs and exploring future inputs that are informative of the representations. The mutual information between the representation and future inputs is the expected information gain (Lindley et al., 1956; MacKay, 1992b) that encourages inputs that are expected to convey the most bits about the representation to maximally reduce uncertainty in the belief,
KL Pe
To]
s EEKL
[ x<) [(w]]
E In To w)
In p (Ix)
simplicity
representation learning
EKL [KL[pp(0s Iee w)
To (
[x)]] E [In xx) - In (w xxc ]]
contml
information gain
E In (ow x) - In pow x<e)
2
E In Tww (x:ee) - In po 0114-1)] 1)] (23)
information gain
t'St
intrinsic reward
Information gain can be estimated by planning (Sun et al., 2011) or from past environment interaction (Schmidhuber, 1991). State representations lead to agents that disambiguate unobserved environment states, for example by opening doors to see objects behind them, such as in active inference (Da Costa et al., 2020), INDIGO (Azar et al., 2019), and DVBF-LM (Mirchev et al., 2018). Model parameters lead to agents that discover the rules of their environment, such as in active inference (Friston et al., 2015), VIME (Houthooft et al., 2016), MAX (Shyam et al., 2018) 18), and Plan2Explore (Sekar et al., 2020). SLAM resolves uncertainty over both states and dynamics (Moutarlier and Chatila, 1989). Policy parameters lead to agents that explore to find the best behavior, such as bootstrapped DQN (Osband et al., 2016) and Bayesian DQN (Azizzadenesheli et al., 2018).
One might think exploration should seek inputs with large error, but reconstruction and exploration optimize the same objective. Maximizing information gain minimizes the reconstruction error at future time steps by steering toward diverse but predictable inputs. Divergence minimization shows that every latent representation should be accompanied with expected information gain term, so that the agent optimizes a consistent objective for past and future time steps. Moreover, it shows that representations should be optimized jointly with the policy support both reconstruction and action choice (Lange and Riedmiller, 2010; Jaderberg et al., 2016; Lee et al., 2019a; Yarats et al., 2019).
B ACTIVE INFERENCE
Divergence minimization is motivated by the free energy principle (Friston, 2010; 2019) and its implementation active inference (Friston et al., 2017). Both approaches share the interpretation of models as preferences (Wald, 1947: Brown, 1981; Friston al., 2012) and account for variety of intrinsic objectives (Friston et al., 2020). However, typical implementations of active inference have been limited to simple tasks of today, problem that divergence minimization overcomes. Active inference differs from divergence minimization in the three aspects discussed below.
Maximizing the input probability Divergence minimization aims to match the distribution of the system to the target distribution. Therefore, the agent aims to receive inputs that follow the marginal distribution of inputs under the model. In contrast, active inference aims to maximize the probability of inputs under the model. This is often described as minimizing Bayesian surprise. Therefore, the agent aims to receive inputs that are the most probable under its model. Mathematically, this difference stems from the conditional input entropy of the actual system that distinguishes the joint KL divergence in Equation 2 from the expected free energy used in active inference,
KT. (2 2) (t,, =)]
E In (x
=)] + K. (p [T.
(x)
(()) F. Iu [4()] (24)
juint divergence
expected free energy
input entropy
Both formulations include the entropy of latent variables and thus the information gain that encourages the agent explore informative future inputs. Moreover, in complex environments, it is unlikely
22

---

Under review as a conference paper at ICLR 2021
that the agent ever learns everything that its beliefs concentrate and it stops exploring. However, in this hypothetical scenario, active inference converges to the input that is most probable under its model. In contrast, divergence minimization aims to converge to sampling from the marginal input distribution under the model, resulting in a larger niche. That said, it is possible to construct a target distribution that includes the input entropy of the actual system and thus overcome this difference.
Expected free energy action prior Divergence minimization optimizes the same objective with respect to representations and actions. Therefore, actions optimize the expected information gain and representations optimize not just past accuracy but also change to support actions in maximizing the expected information gain. In contrast, active inference first optimizes the expected free energy to compute a prior over policies. After that, it optimizes the free energy with respect to both representations and actions. This means active inference optimizes the information gain only with respect to actions, without the representations changing to support better action choice based on future objective terms.
Bayesian model average over policies Typical implementations of active inference compute the action prior using Bayesian model average. This involves computing the expected free energy for every possible policy or action sequence that is available to the agent. The action prior is then computed as the softmax over the computed values. Enumerating all policies is intractable for larger action spaces or longer planning horizons, thus limiting the applicability of active inference implementations. In contrast, divergence minimization absorbs the objective terms for action and perception into single variational optimization thereby finessing the computational complexity of computing separate action prior. This leads simple framework, allowing us draw close connections to the deep RL literature and to scale to challenging tasks, as evidenced by the many established methods that are explained under the divergence minimization framework.
C KL INTERPRETATION
Minimizing the KL divergence has variety of interpretations. In simple terms, it says "optimize function but don't ot be too confident." Decomposing Equation 2 shows that we maximize the expected log target while encouraging high entropy of all the random variables. Both terms expectations under Ps and thus depend on the parameter vector d,
KL (1,
7(x, =)] E
In 7xx,
H x. 2)
(25)
energy
entropy
The energy term expresses which system configurations we prefer. It is also known as the cross entropy loss, expected log loss, (Bishop, 2006; Murphy, 2012). energy function when unnormalized (LeCun et al., 2006), and agent preferences in control (Morgenstern and Von Neumann, 1953).
The entropy term prevents all random variables in the system from becoming deterministic, encourag- ing a global search over their possible values. It implements the maximum entropy principle to avoid overconfidence (Jaynes, 1957), Occam's 's razor to prevent overfitting (Jefferys and Berger, 1992), bounded rationality to halt optimization before reaching the point solution (Ortega and Braun, 2011) 1), and risk- lisk-sensttivity account for model misspecification (Pratt, 1964; Howard and Matheson, 1972).
Expected utility The entropy distinguishes the KL from the expected utility objective that is typical in RL (Sutton and Barto, 2018). Using distribution the optimization target is more general, as every system has a distribution but not every system has a utility function it is optimal for. Moreover, the dynamics of any stochastic system maximize only its log stationary distribution (Ao et al., 2013; Friston, 2013; Ma et al., 2015). This motivates using the desired distribution as the optimization target. Expected utility is recovered in the limit of sharp target that outweighs the entropy.
D BACKGROUND
This section introduces notation, defines basic information thhorretc quantities, and briefly reviews KL control and variational inference for latent variable models.
Expectation A random variable v represents an unknown variable that could take on one of multiple values F. each with an associated probability mass or density p(æ = z). Applying a function to a random variable yields a new random variable =xx).. The expectation of a random variable is the weighted average of the values it could take on, weighted by their probability,
B[f(x)] = f(x)p(a) dr.
(26)
23

---

Under review as a conference paper at ICLR 2021
We use integrals here, as used for random variables that take on continuous values. For discrete variables, the integrals simplify to sums.
Information The information of an event E measures the number of bits it contains (Shannon, 1948). Intuitively, rare events contain more information. The information is defined as the code length of the event under an optimal encoding for x ~ p(x),
In
In
(27)
P
The logarithm base 2 measures information in bits and the natural base in the unit nats.
Entropy The entropy of random variable is the expected information of its events. It quantifies the randomness or uncertainty of the random variable. Similarly, the conditional entropy measures the uncertainty of x that we expect remain after observing another variable y,
H ø E In p())]]
[(x y) = =[n-np(( 3)]
(28)
Note that the conditional entropy uses an expectation over both variables. A deterministic distribution reaches the minimum entropy of zero. The uniform distribution reaches the maximum entropy, the logarithm of the number of possible events.
KL divergence The Kullback-Leiin directed similarity of one distribution to another distribution. The KL divergence is defined as the expectation under p of the log difference between the two distributions p and T.
divergence (Kullback and Leibler, 1951), measures the
KL [p(x) || r(x)] E [Inp(x) - In)( = E[ In Tx)) - H[x).
(29)
The KL divergence is non-negative and reaches zero if and only if
Also known relative entropy, it is the expected number of additional bits needed to describe x when using the code for different distribution to encode events from p(x). This is shown by the decomposition as cross-entropy minus entropy shown above. Analogously to the conditional entropy, the conditional KL divergence is expectation over both variables under the first distribution.
Mutual information The mutual information, or simply information, between two random vari- ables r and y measures how many bits the value of carries about the unobserved value of y. It is defined as the entropy of one variable minus its conditional entropy given the other variable,
(30) The mutual information is symmetric in its arguments and non-negative. It reaches zero if and only if x and y are independent so that p(z, y) = p(x))(v). Intuitively, is higher the better we can predict one variable from the other and the more random the variable is by itself. It also be written as KL divergence between the joint and product of marginals.
I[x:Y] Y] = H[ H [ Y] = E In p(x v) - In p(z)] KL [L[p(z, v) p(=)p(v)).
Variational bound Computing the exact mutual information requires access to both the conditional and marginal distributions. When the conditional is unknown, replacing it with another distribution bounds the mutual information from below (Barber and Agakov, 2003; Poole et al., 2019).
[ æ] Z I [s; 2] E KLL[pa [p(2 =) || ( =]] = E [In ra( - Inp()))-
(31)
Maximizing the bound with respect to the parameters tightens the bound, thus bringing (a 22)
closer to p(x 2). Improving the bound through optimization gives the name variational bound. The more flexible the family of (z(x) (I: z)), the more accurate the bound can become.
Dirac distribution The Dirac distribution (Dirac, 1958), also known as point mass, represents a random variable x with certain event T. We show an intuitive definition here; for rigorous definition using measure theory see Rudin (1966).
[1 z 0 else.
(x())
(32)
The expectation under Dirac distribution is simply the inner expression evaluated the certain event, Ex5(x) [x)] = f()). The entropy of Dirac distributed random variable is therefore H In 8g (z) = and its mutual information with another random variables is also zero.
KL control control (Todorov, 2008; Kappen et al., 2009) minimizes the KI. divergence between the trajectory (2) of inputs x = {21, (2,1,22,-. .2. .TT) and target distribution 7 (1x) & exp(r(a:)) defined with a reward r(x).
KT. (1(
T(x) larget
E [[ expected reward
H[z) entropy
(33)
trajectory
24

---

Under review as a conference paper at ICLR 2021
The KL between the two distributions is minimized with respect to the control rule action sequence 6, revealing the expected reward and an entropy regularizer. Because the expectations are terms of the trajectory they are integrals under its distribution Pa-
Variational inference Latent variable models explain inputs a using latent variables 2. They define prior 7(2) and an observation model ((x: 2). To infer the posterior 7(2 ..) that represents given input F. we need to condition the model on the input. However, this requires inverting the observation model using Bayes rule and has no closed form in general. To overcome this intractabiliiyy, variational inference (Hinton and Van Camp. 1993; Jordan et al., 1999) optimizes a parameterized belief (=(2 I) to approximate the posterior by minimizing the KL.
KT. KT. (ps( Φ) r(2 {)] + In (T)
KI. p 3) || [()) E In [ 2)]
(34)
complexity
accuracy
Adding the marginal (xx) that does not depend on Φ completes the intractable posterior to the joint that can factorized into the available parts r(2) and T(x 2). This reveals a complexity regularizer that keeps the belief close to the prior and an accuracy term that encourages the belief to be representative of the input. This objective known as the variational free energy or ELBO.
25


Reviews:
[{"review_id": "vcakzIUU_Qx", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper propose a unified optimisation objective for (sequential) decision-making (i.e., _action_) and representation learning (i.e., _perception_), built on joint (KL) divergence minimisation. As also mentioned by the authors, this is a concept paper and it includes no empirical study.\n\nIn particular, the authors demonstrate how existing ideas and approaches to (sequential) decision-making and representation learning can be expressed as a joint KL minimisation problem between a target and \"actual\" distribution. Such examples are (a) MaxEnt RL, (b) VI, (c) amortised VI, (d) KL control, (e) skill discovery and (f) empowerment, which are all cases of the KL minimisation between a target and an ``actual'' distributions.\n\n**Concerns**:\n1. Although the proposed perspective and language is rich and expressive, I question the novelty of the proposed framework, since the information-theoretic view of decision-making and perception is a rather established and old idea, even the term/idea of perception-action cycle is already defined [1]!\n2. The power of latent variables for decision-making and their interpretation is also a known idea [1].\n\n**References**\n\n[1] Tishby, N. and Polani, D., 2011. Information theory of decisions and actions. In Perception-action cycle (pp. 601-636). Springer, New York, NY.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nDlpy1miu1R", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed to use the joint KL divergence between the generative joint distribution and the target distribution (containing latent variables which could correspond to latent parts we wanted to model (e.g. beliefs). It was illustrative to discuss decomposing the joint KL into different ways and thus forming information bounds in different scenarios. The decomposition of past and future in Eq.6 also provided a unified perspective for looking at the most currently used objectives.\n\nThe examples shown in the paper and appendix give a good illustration of how people can make assumptions or design the terms to convert prevalent objectives into objectives that follow from this joint KL divergence framework. This is, in my mind, one of their key contributions for connecting the past progress in a general and unified way.\n\nHowever, one concern about this paper is that the proposal of such a unified KL minimization framework is in fact a bit too general and abstract. In fact, many methods mentioned in this work shared a similar insight of deriving objectives from a KL-minimization perspective, but some factors are omitted to better fit the corresponding tasks. The general decomposition discussed in this paper provides little hint on how new objectives could be derived for problems. The general framework does somehow serve as the guideline, but my worry is that its effect will be limited as we still need to design the mapping for the terms in the general objective accordingly in different tasks.\n\nGiven the pros and cons of this paper, I'm putting a borderline decision for now. The authors should clear any of my misunderstandings and perhaps show the potential for this general framework as a source for new objectives.\n\n=====================================================================================================\n\nAfter reading the authors rebuttal, my major concerns are fully addressed and I decide to keep my decision as weak accept", "extra_scores": {}, "extra_text": {}}, {"review_id": "d22odngcxYP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors formulate a general framework that unifies inference, action/perception, control, and several other tasks. The framework is based on minimizing the KL divergence between a parameterized \"actual\" distribution and a \"target\" distribution. The authors argue that this formulation unifies a wide range of previously proposed objectives. They also argue that it has some advantages when compared to Friston's \"free energy principle\" framework, with which it shares many similarities, in particular that probability matching is preferred to surprise minimization.\n\nThe paper is clearly-written and provides a very thorough literature review. However, generally I question the scientific value of such all-encompassing unifying frameworks, and this paper in particular offers no concrete formal or empirical results, while promising a lot. At the end of the day, the divergence minimization objective is nothing more than MaxEnt, decorated with various interpretations and decompositions. Without empirical support, I do not find the interpretations and decompositions very convincing -- as one example, does divergence minimization *really* mean that \"expressive world models lead to autonomous agents that understand and inhabit large niches\"? \n\nOne of the issues is that the paper appears to treat the \"heart of the matter\" (i.e., the source of interesting solutions) as if it lay in the elegant and generic objective. In my opinion, however, the real heart of the matter will be encoded in (1) the structure of the target distribution, (2) the structure/parameterization of the actual distribution, and (3) the optimization algorithm that can actually minimize the (typically) high-dimensional  objective. The quality of resulting solutions depend on 1-3 -- all of which need to be exogenously specified --- because divergence minimization cannot on its own produce interesting behavior. \n\nAt the end of the day, I do think there is some value in providing a unifying framework, and developing information-theoretic decompositions and interpretation. However, I think the paper would be *much* stronger if it was considerably longer and had more room to breathe (which it doesn't have right now -- given all the connections it tries to make), and if qualitative statements (of the type discussed above) were accompanied by empirical results (even if simulations with simple toy models).", "extra_scores": {}, "extra_text": {}}, {"review_id": "-3T1MGtPeOm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n\nIn this manuscript, the authors propose a unifying framework for a large class of inference and reinforcement learning objectives, which have been studied in prior works by various authors. They demonstrate that approaches and central ideas from many different fields in the ML/AI community can be derived as limiting cases of their framework.\n\n##########################################################################\nReasons for score: \n\nOverall, I vote for acceptance (7). Like many, I have employed various variational approaches in the past and see its merit. While I agree with the main idea, this work is not without problems. This is especially problematic for such a broadly applicable work that will most likely influence plenty of future research. My main problems with this submission are:\n1.\tPresentation. While the paper is, for the most part, well written and well organized, there are some gaps/ jumps that render understanding difficult. Two examples:\nA)\tThe parameters \\phi. The authors start by introducing parameters \\phi as abstract placeholders for (i) parameters of the true joint distribution of data and latents of the underlying system and (ii) a set of actions an agent can perform to interact with this world. The agent's target distribution has no explicit parameter dependence. So far, so good. Then, one is redirected to the appendix A.1 and A.2. Section A.1 is already a bit confusing because suddenly, additional latents w are introduced that were not mentioned before. Then, suddenly in A.2. the target \\tau is suddenly dependent on the parameters \\phi, which were initially parameters of the underlying system's true joint distribution. This also happens in Figure 2. C), which is also never referenced in the text. I find this strange mixing of parameters of agent and system very confusing. It also sheds some doubt about the generality of the framework.\nB)\tI have read the paper carefully and still do not understand Figure 1 completely. This may also be due to the reason that it is only referenced in the appendix. Related: Why does Information gain play such a central role if all derived objectives only contain (upper) bounds for it appear?\n\n2.  \t(Unsupported) Claims: In the abstract, the authors promise to offer \"a recipe for designing novel objectives\". As much as I can see, they only come back to this promise in the conclusion, where they say that one could look at other divergence measures to arrive at new objectives, and they will leave it for future work. \nI would not call this a recipe, but an outlook at most. \n\n\n3.\tToo many ideas: It is hard, if not impossible, to explain a broad framework well in a conference proceeding. This work contains so many ideas and establishes many connections that following this work, and understanding them in detail becomes very hard. I would suggest sacrificing some connections in favor of a more concise presentation.\n4.\tFixation on KL-divergence: This is more of a suggestion. I understand that many works use the (non-symmetric) KL due to its favorable analytic properties. Thus, I agree that it makes sense to focus this framework on this measure. However, I believe this work's main idea still holds if one would exchange the KL with some other measure of similarity between distributions. Maybe it would make sense to first introduce and discuss the abstract idea of aligning target and belief before fixation on a particular measure. This would also go well with resolving my concern 2.. \n    \n\n##########################################################################\nPros: \n1.\tUnifying framework of many inference, and RL objectives.\n2.\tWell written.\n3.\tWill be impactful to a lot of future research.\n\n##########################################################################\nCons: \n1.\tSee my Reasons for score.\n\n##########################################################################\nQuestions during the rebuttal period: \n\nPlease address and clarify the cons above. \n\n#########################################################################\n\nMinor: \n· Please consider citing \nToussaint, M., & Storkey, A. (2006). Probabilistic inference for solving discrete and continuous state Markov Decision Processes. International Conference on Machine Learning (ICML), 945–952. https://doi.org/10.1145/1143844.1143963\nin the \"control as inference\" section. To my knowledge, it is one of the first to establish the connection between planning and inference. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an KL-divergence minimisation approach to the action–perception loop, and thus presents a unifying view on concepts such as Empowerment, entropy-based RL, optimal control, etc.  The paper does two things here: it serves as a survey paper, but on top of that puts these in a unifying theory.  While the direct merit of that may not be obvious, it does serve as a good basis to combine the fields more formally.

Unfortunately, the paper suffers from the length restrictions.  With more than half of the paper in the appendix, it should be published at a journal or directly at arXiv.   Not having a page limit would improve the readability much.  ICLR may not be the best venue for review papers.
Decision: Reject