Action and Perception as Divergence Minimization

1 INTRODUCTION . To achieve goals in complex environments , intelligent agents need to perceive their environments and choose effective actions . These two processes , perception and action , are often studied in isolation . Despite the many objectives that have been proposed in the fields of representation learning and reinforcement learning , it remains unclear how the objectives relate to each other and which fundamentally new objectives remain yet to be discovered . Based on the KL divergence ( Kullback and Leibler , 1951 ) , we propose a unified framework for action and perception that connects a wide range of objectives to facilitate our understanding of them while providing a recipe for designing novel agent objectives . Our findings are conceptual in nature and this paper includes no empirical study . Instead , we offer a unified picture of a wide range of methods that have been shown to be successful in practice in prior work . The contributions of this paper are described as follows . Unified objective function for perception and action We propose joint KL minimization as a principled framework for designing and comparing agent objectives . KL minimization was proposed separately for perception as variational inference ( Jordan et al. , 1999 ; Alemi and Fischer , 2018 ) and for actions as KL control ( Todorov , 2008 ; Kappen et al. , 2009 ) . Based on this insight , we formulate action and perception as jointly minimizing the KL from the world to a unified target distribution . The target serves both as the model to infer representations and as reward for actions . This extends variational inference to controllable inputs , while extending KL control to latent representations . We show a novel decomposition of joint KL divergence that explains several representation learning and exploration objectives . Divergence minimization additionally connects deep reinforcement learning to the free energy principle ( Friston , 2010 ; 2019 ) , while simplifying and overcoming limitations of its active inference implementations ( Friston et al. , 2017 ) that we discuss in Appendix B . Understanding latent variables for decision making Divergence minimization with an expressive target maximizes the mutual information between inputs and latents . Agents thus infer representations that are informative of past inputs and explore future inputs that are informative of the representations . For the past , this yields reconstruction ( Hinton et al. , 2006 ; Kingma and Welling , 2013 ) or contrastive learning ( Gutmann and Hyvärinen , 2010 ; Oord et al. , 2018 ) . For the future , it yields information gain exploration ( Lindley et al. , 1956 ) . Stochastic skills and actions are realized over time , so their past terms are constant . For the future , they lead to empowerment ( Klyubin et al. , 2005 ) and skill discovery ( Gregor et al. , 2016 ) . RL as inference ( Rawlik et al. , 2010 ) does not maximize mutual information because its target is factorized . To optimize a consistent objective across past and future , latent representations should be accompanied by information gain exploration . Expressive world models for large ecological niches The more flexible an agent ’ s target or model , the better the agent can adapt to its environment . Minimizing the divergence between the world and the model , the agent converges to a natural equilibrium or niche where it can accurately predict its inputs and that it can inhabit despite external perturbations ( Schrödinger , 1944 ; Wiener , 1948 ; Haken , 1981 ; Friston , 2013 ; Berseth et al. , 2019 ) . While surprise minimization can lead to trivial solutions , divergence minimization encourages the niche to match the agent ’ s model class , thus visiting all inputs proportionally to how well they can be understood . This suggests designing expressive world models of sensory inputs ( Ebert et al. , 2017 ; Hafner et al. , 2018 ; Gregor et al. , 2019 ) as a path toward building highly adaptive agents , while rendering task rewards optional . 2 FRAMEWORK . This section introduces the framework of action and perception as divergence minimization ( APD ) . To unify action and perception , we formulate the two processes as joint KL minimization with a shared target distribution . The target distribution expresses the agent ’ s preferences over system configurations and is also the probabilistic model under which the agent infers its representations . Using an expressive model as the target maximizes the mutual information between the latent variables and the sequence of sensory inputs , thus inferring latent representations that are informative of past inputs and exploring future inputs that are informative of the representations . We assume knowledge of basic concepts from probability and information theory that are reviewed in Appendix D . 2.1 JOINT KL MINIMIZATION . Consider a stochastic system described by a joint probability distribution over random variables . For example , the random variables for supervised learning are the inputs and labels and for an agent they are the sequence of sensory inputs , internal representations , and actions . More generally , we combine all input variables into x and the remaining variables that we term latents into z . We will see that different latents correspond to different representation learning and exploration objectives . The random variables are distributed according to their generative process or actual distribution pφ . Parts of the actual distribution can be unknown , such as the data distribution , and parts can be influenced by varying the parameter vector φ , such as the distribution of stochastic representations or actions . As a counterpart to the actual distribution , we define the desired target distribution τ over the same support . It describes our preferences over system configurations and can be unnormalized , Actual distribution : x , z ∼ pφ ( x , z ) Target distribution : τ ( x , z ) . ( 1 ) We formulate the problem of joint KL minimization as changing the parameters φ to bring the actual distribution of all random variables as close as possible to the target distribution , as measured by the KL divergence ( Kullback and Leibler , 1951 ; Li et al. , 2017 ; Alemi and Fischer , 2018 ) , min φ KL [ pφ ( x , z ) ∥∥ τ ( x , z ) ] . ( 2 ) All expectations and KLs throughout the paper are integrals under the actual distribution , so they can be estimated from samples of the system and depend on φ . Equation 2 is the reverse KL or information projection used in variational inference ( Csiszár and Matus , 2003 ) . Examples For representation learning , pφ is the joint of data and belief distributions and τ is a latent variable model . Note that we use pφ to denote not the model under which we infer beliefs but the generative process of inputs and their representations . For control , pφ is the trajectory distribution under the current policy and τ corresponds to the utility of the trajectory . The parameters φ include everything the optimizer can change directly , such as sufficient statistics of representations , model parameters , and policy parameters . Target parameters There are two ways to denote deterministic values within our framework , also known as MAP estimates in the probabilistic modeling literature ( Bishop , 2006 ) . We can either use a fixed target distribution and use a latent variable that follows a point mass distribution ( Dirac , 1958 ) , or we explicitly parameterize the target using a deterministic parameter as τφ . In either case , τ refers to the fixed model class . The two approaches are equivalent because in both cases the target receives a deterministic value that has no entropy regularizer . For more details , see Appendix A.1 . Assumptions Divergence minimization uses only two inductive biases , namely that the agent optimizes an objective and that it uses random variables to represent uncertainty . Choosing the wellestablished KL as the divergence measure is an additional assumption . It corresponds to maximizing the expected log probability under the target while encouraging high entropy for all variables in the system to avoid overconfidence , as detailed in Appendix C. Common objectives with different degrees of entropy regularization are summarized in Table 1 . Generality Alternative divergence measures would lead to different optimization dynamics , different solutions if the target can not be reached , and potentially novel objectives for representation learning and exploration . Nonetheless , the KL can describe any converged system , trivially by choosing its actual distribution as the target , and thus offers a simple and complete mathematical perspective for comparing a wide range of specific objectives that correspond to different latent variables and target distributions . 2.2 INFORMATION BOUNDS . We show that for expressive targets that capture dependencies between the variables in the system , minimizing the joint KL increases both the preferences and the mutual information between inputs x and latents z . This property allows divergence minimization to explain a wide range of existing representation learning and exploration objectives . We use the term representation learning for inferring deterministic or stochastic variables from inputs , which includes local representations of individual inputs and global representations such as model parameters . Latent preferences The joint KL can be decomposed in multiple ways , for example into a marginal KL plus a conditional KL or by grouping marginal with conditional terms . To reveal the mutual information maximization , we decompose the joint KL into a preference seeking term and an information seeking term . The decomposition can be done either with the information term expressed over inputs and the preferences expressed over latents or the other way around , KL [ pφ ( x , z ) ∥∥ τ ( x , z ) ] joint divergence = E KL [ pφ ( z | x ) ∥∥ τ ( z ) ] realizing latent preferences − E [ ln τ ( x ∣∣ z ) − ln pφ ( x ) ] information bound . ( 3 ) All expectations throughout the paper are over all variables , under the actual distribution , and thus depend on the parameters φ . The first term on the right side of Equation 3 is a KL regularizer that keeps the belief pφ ( z | x ) over latent variables close to the marginal latent preferences τ ( z ) . The second term is a variational bound on the mutual information I [ x ; z ] ( Barber and Agakov , 2003 ) . The bound is expressed in input space . Maximizing the conditional ln τ ( x | z ) seeks latent variables that accurately predict inputs while minimizing the marginal ln pφ ( x ) seeks diverse inputs . Variational free energy When the agent can not influence its inputs , such as when learning from a fixed dataset , the input entropy E [ − ln pφ ( x ) ] is not parameterized and can be dropped from Equation 3 . This yields the free energy or ELBO objective used by variational inference to infer approximate posterior beliefs in latent variable models ( Hinton and Van Camp , 1993 ; Jordan et al. , 1999 ) . The free energy regularizes the belief pφ ( z | x ) to stay close to the prior τ ( z ) while reconstructing inputs via τ ( x | z ) . However , in reinforcement and active learning , inputs can be influenced and thus the input entropy should be kept , which makes the information bound explicit . Input preferences Analogously , we decompose the joint KL the other way around . The first term on the right side of Equation 4 is a KL regularizer that keeps the conditional input distribution pφ ( x | z ) close to the marginal input preferences τ ( x ) . This term is analogous to the objective in KL control ( Todorov , 2008 ; Kappen et al. , 2009 ) , except that the inputs now depend upon latent variables via the policy . The second term is again a variational bound on the mutual information I [ x ; z ] , this time expressed in latent space . Intuitively , the bound compares the belief τ ( z | x ) after observing the inputs and the belief pφ ( z ) before observing any inputs to measure the gained information , KL [ pφ ( x , z ) ∥∥ τ ( x , z ) ] joint divergence = E KL [ pφ ( x | z ) ∥∥ τ ( x ) ] realizing input preferences − E [ ln τ ( z ∣∣ x ) − ln pφ ( z ) ] information bound . ( 4 ) The information bounds are tighter the better the target conditional approximates the actual conditional , meaning that the agent becomes better at maximizing mutual information as it learns more about the relation between the two variables . This requires an expressive target that captures correlations between inputs and latents , such as a latent variable model or deep neural network . Maximizing the mutual information accounts for both learning latent representations that are informative of inputs as well as exploring inputs that are informative of the latent representations .

Summary: The authors proposed to use the joint KL divergence between the generative joint distribution and the target distribution (containing latent variables which could correspond to latent parts we wanted to model (e.g. beliefs). It was illustrative to discuss decomposing the joint KL into different ways and thus forming information bounds in different scenarios. The decomposition of past and future in Eq.6 also provided a unified perspective for looking at the most currently used objectives.