Efficient Empowerment Estimation for Unsupervised Stabilization

1 INTRODUCTION . Intrinsic motivation allows artificial and biological agents to acquire useful behaviours without external knowledge ( Barto et al . ( 2004 ) ; Chentanez et al . ( 2005 ) ; Schmidhuber ( 2010 ) ; Barto ( 2013 ) ; Oudeyer et al . ( 2016 ) ) . In the framework of reinforcement learning ( RL ) , this external knowledge is usually provided by an expert through a task-specific reward , which is optimized by an artificial agent towards a desired behavior ( Mnih et al . ( 2013 ) ; Schulman et al . ( 2017 ) ) . In contrast , an intrinsic reward can arise solely from the interaction between the agent and environment , which eliminates the need for domain knowledge and reward engineering in some settings ( Mohamed & Rezende ( 2015 ) ; Houthooft et al . ( 2017 ) ; Pathak et al . ( 2017 ) ) . Previously , it was shown that maximizing mutual information ( Cover & Thomas ( 2012 ) ) between an agent ’ s actuators and sensors can guide the agent towards states in the environment with higher potential to achieve a larger number of eventual goals ( Klyubin et al . ( 2005 ) ; Wissner-Gross & Freer ( 2013 ) ) . Maximizing this kind of mutual information is known as the empowerment principle ( Klyubin et al . ( 2005 ) ; Salge et al . ( 2014 ) ) . Previously , it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems ( Jung et al . ( 2011 ) ; Salge et al . ( 2013 ) ; Karl et al . ( 2019 ) ) . In this application , ascending the gradient of the empowerment function coincides with the objective of optimal control for stabilization at an unstable equilibrium ( Strogatz ( 2018 ) ) , which is an important task for both engineering ( Todorov ( 2006 ) ) and 1Project page : https : //sites.google.com/view/latent-gce biological2 systems ; we refer to this as the essential property of empowerment . It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property . Empowerment has been found to be useful for a broad spectrum of applications , including : unsupervised skill discovery ( Sharma et al . ( 2020 ) ; Eysenbach et al . ( 2019 ) ; Gregor et al . ( 2017 ) ; Karl et al . ( 2019 ) ; Campos et al . ( 2020 ) ; human-agent coordination ( Salge & Polani ( 2017 ) ; Guckelsberger et al . ( 2016 ) ) ; assistance ( Du et al . ( 2021 ) ) ; and stabilization ( Tiomkin et al . ( 2017 ) ) . Past work has utilized variational lower bounds ( VLBs ) ( Poole et al . ( 2019 ) ; Alemi et al . ( 2017 ) ; Tschannen et al . ( 2020 ) ; Mohamed & Rezende ( 2015 ) ) to achieve an estimate of the empowerment . However , VLB approaches to empowerment in dynamical control systems ( Sharma et al . ( 2020 ) ; Achiam et al . ( 2018 ) ) have high sample complexity , are often unstable in training , and may be biased . Moreover , it was not previously studied if empowerment estimators learned via VLBs possess the essential properties of empowerment . In this work , we introduce a new method , Latent Gaussian Channel Empowerment ( Latent-GCE ) , for empowerment estimation and utilize the above-mentioned property as an “ indicator ” for the quality of the estimation . Specifically , we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories . This representation admits an efficient estimation of empowerment by convex optimization ( Cover & Thomas ( 2012 ) ) , both from raw state and from images . We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such as Proximal Policy Optimization ( Schulman et al . ( 2017 ) ) , and Soft Actor-Critic ( Haarnoja et al . ( 2018 ) ) . We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward , showing our estimator exhibits essential properties of the empowerment function . We demonstrate the advantages of our method through comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the OpenAI Gym simulator ( Brockman et al . ( 2016 ) ) . We find that our method ( i ) has a lower sample complexity , ( ii ) is more stable in training , ( iii ) possesses the essential properties of the empowerment function , and ( iv ) allows us to accurately estimate empowerment from images . We hope such a review of the existing methods for empowerment estimation will help push this research direction . 2 PRELIMINARIES . In this section , we review the necessary background for our method , consisting of the reinforcement learning setting , various empowerment estimators , and the Gaussian channel capacity . We also review the underlying components in relevant prior work which we use for comparison to our method . 2.1 REINFORCEMENT LEARNING . The reinforcement learning ( RL ) setting is modeled as an infinite-horizon Markov Decision Process ( MDP ) defined by : the state space S , the action space A , the transition probabilities p ( s′|s , a ) , the initial state distribution p0 ( s ) , the reward function r ( s , a ) ∈ R , and the discount factor γ . The goal of RL is to find an optimal control policy π ( a|s ) that maximizes the expected return , i.e . maxπ Es0∼p0 , at∼π , st+1∼p [ ∑∞ t=0 γ tr ( st , at ) ] ( Sutton & Barto ( 2018 ) ) . 2.2 EMPOWERMENT . Interaction between an agent and its environment is plausibly described by the perception-action cycle ( PAC ) , ( Tishby & Polani ( 2011 ) ) , where the agent observes the state of the environment via its sensors and responds with its actuators . The maximal information rate from actuators to sensors is an inherent property of PAC which characterizes the empowerment of the agent , formally defined below . Empowerment ( Klyubin et al . ( 2005 ) ) is defined by the maximal mutual information rate ( Cover & Thomas ( 2012 ) ) between the agent ’ s sensor observations o ∈ O and actuators a ∈ A given the current state s ∈ S . It is fully specified by a fixed probability distribution of sensor observations conditioned 2Broadly speaking , an increase in the rate of information flow in the perception-action loop ( Tishby & Polani ( 2011 ) ) could be an impetus for the development of homosapiens , as hypothesized in ( Yuval ( 2014 ) ) . on the actuators and the state , p ( O | A , s ) , denoted by “ channel ” , and by a free probability of the actuator signals conditioned on the state , ω ( A | s ) , denoted by “ source ” . The empowerment E is the capacity of the channel , which is found via optimization of the source distribution ω ( A | s ) : E ( s ) = max ω ( A|s ) I [ O ; A | s ] = max ω ( A|s ) ∑ O , A p ( O , A | s ) log ( p ( A | O , s ) ω ( A | s ) ) ( 1 ) where I [ O ; A | s ] is the mutual information , reflecting the difference between the entropy of the sensor inputs , H ( O | s ) and the corresponding conditional entropy , H ( O | A , s ) 3 . In this work , we follow the original empowerment formulation ( Klyubin et al . ( 2005 ) ) , where A is an action sequence of length T starting from the state s and O is the resulting observation sequence . However , our method is applicable to other choices of observation and control sequences . Remark 1 The source distribution in Equation 1 , ω∗ ( A | s ) , is used only for the estimation of empowerment . Another policy , π ( a | s ) , is calculated using empowerment as an intrinsic reward by either reinforcement learning ( Karl et al . ( 2019 ) ) or a 1-step greedy algorithm ( Jung et al . ( 2011 ) ; Salge et al . ( 2013 ) ) to reach the states with maximal empowerment . Strictly speaking , the source distribution ω maximizes the mutual information , ( i.e . estimates the empowerment ) , while the policy distribution π maximizes expected accumulated empowerment along trajectories . In general , the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel , as reviewed below . As a result , past works have largely relied on variational lower bounds instead . 2.2.1 EMPOWERMENT BY VARIATIONAL LOWER BOUNDS . The mutual information in Equation 1 can be bounded from below as follows ( Mohamed & Rezende ( 2015 ) ; Gregor et al . ( 2017 ) ; Karl et al . ( 2019 ) ; Poole et al . ( 2019 ) ) : ∑ O , A p ( O , A | s ) log ( p ( A | O , s ) ω ( A | s ) ) = ∑ O , A p ( O , A | s ) log ( q ( A | O , s ) ω ( A | s ) ) +DKL [ p ( A | O , s ) || q ( A | O , s ) ] ≤ ∑ O , A p ( O , A | s ) log ( q ( A | O , s ) ω ( A | s ) ) ( 2 ) where q ( A | O , s ) and ω ( A | s ) are represented by neural networks with parameters φ and ψ respectively . The lower bound in Equation 2 can be estimated from samples using reinforcement learning with intrinsic reward r ( st , at ) = ∑T t log qφ ( at | ot , s ) − logωψ ( at | st ) , as detailed in ( Gregor et al . ( 2017 ) ; Karl et al . ( 2019 ) ) . To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions : qφ ( at | ot , s ) and ωψ ( at | st ) for the lower bound in Equation 2 , and π ( a | s ) as the control policy ( see Remark 1 ) . Developing the mutual information in Equation 1 in the opposite direction , one gets another lower bound on empowerment , an approximation of which was used in ( Sharma et al . ( 2020 ) ) : ∑ O , A p ( O , A | s ) log ( p ( O | A , s ) ω ( O | s ) ) = ∑ O , A p ( O , A | s ) log ( q ( O | A , s ) ω ( O | s ) ) +DKL [ p ( O | A , s ) || q ( O | A , s ) ] ≤ ∑ O , A p ( O , A | s ) log ( q ( O | A , s ) ω ( O | s ) ) . ( 3 ) For sake of the clarity of the exposition , the above-explained variational empowerment is schematically depicted in Figure 1 . The controller policy , πθ ( a | s ) , collects transitions in environment , 3when the agent does not have an access to the true state , s , empowerment is calculated as the maximal mutual information between actuators and sensors given an estimated state . ( s , a , s′ ) . The source policy ωψ ( a | s ) generates the action a at every state s visited by the controller policy , and the environment transits to the next state , s′ ( yellow block ) . The inverse channel policy qφ ( a | s , s′ ) is regressed to a given s and and s′ , generating the intrinsic reward , C = logωψ ( a | s ) − log qφ ( a | s , s′ ) , which is used as the reward to improve the controller policy via reinforcement learning . This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment .

Summary: The paper studies reward-free reinforcement learning (RL) methods based on empowerment. The authors propose a technique for empowerment estimation under the assumption that a state after H timesteps can be factorized as a product of the current action and a matrix G(s) that depends on the current state. In contrast to the existing methods that rely on the optimization of variational lower bounds on mutual information for empowerment estimation, the technique allows having an almost closed-form expression for empowerment. The authors qualitatively demonstrate the convergence of their method to the true empowerment function on simple environments such as 2D ball-in-box as well on image-based Pendulum environment.