EFFICIENT EMPOWERMENT ESTIMATION FOR UNSUPERVISED STABILIZATION
Year: 2021

Abstract:
Intrinsically motivated artificial agents learn advantageous behavior without externally-provided rewards. Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking. This follows from the coincidence between the objective of stabilization and the objective of empowerment. Unfortunately, sample-based estimation of this kind of mutual information is challenging. Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, unstable in training, and have high sample complexity. In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efficiently calculate an unbiased estimator of empowerment by convex optimization. We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches. Specifically, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images. Consequently, our method opens a path to wider and easier adoption of empowerment for various applications. 1

heading: 1 INTRODUCTION
text: Intrinsic motivation allows artificial and biological agents to acquire useful behaviours without external knowledge (Barto et al. (2004); Chentanez et al. (2005); Schmidhuber (2010); Barto (2013); Oudeyer et al. (2016)). In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task-specific reward, which is optimized by an artificial agent towards a desired behavior (Mnih et al. (2013); Schulman et al. (2017)).
In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates the need for domain knowledge and reward engineering in some settings (Mohamed & Rezende (2015); Houthooft et al. (2017); Pathak et al. (2017)). Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent’s actuators and sensors can guide the agent towards states in the environment with higher potential to achieve a larger number of eventual goals (Klyubin et al. (2005); Wissner-Gross & Freer (2013)).
Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al. (2005); Salge et al. (2014)). Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems (Jung et al. (2011); Salge et al. (2013); Karl et al. (2019)). In this application, ascending the gradient of the empowerment function coincides with the objective of optimal control for stabilization at an unstable equilibrium (Strogatz (2018)), which is an important task for both engineering (Todorov (2006)) and
1Project page: https://sites.google.com/view/latent-gce
biological2 systems; we refer to this as the essential property of empowerment. It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property.
Empowerment has been found to be useful for a broad spectrum of applications, including: unsupervised skill discovery (Sharma et al. (2020); Eysenbach et al. (2019); Gregor et al. (2017); Karl et al. (2019); Campos et al. (2020); human-agent coordination (Salge & Polani (2017); Guckelsberger et al. (2016)); assistance (Du et al. (2021)); and stabilization (Tiomkin et al. (2017)). Past work has utilized variational lower bounds (VLBs) (Poole et al. (2019); Alemi et al. (2017); Tschannen et al. (2020); Mohamed & Rezende (2015)) to achieve an estimate of the empowerment. However, VLB approaches to empowerment in dynamical control systems (Sharma et al. (2020); Achiam et al. (2018)) have high sample complexity, are often unstable in training, and may be biased. Moreover, it was not previously studied if empowerment estimators learned via VLBs possess the essential properties of empowerment.
In this work, we introduce a new method, Latent Gaussian Channel Empowerment (Latent-GCE), for empowerment estimation and utilize the above-mentioned property as an “indicator” for the quality of the estimation. Specifically, we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories. This representation admits an efficient estimation of empowerment by convex optimization (Cover & Thomas (2012)), both from raw state and from images. We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such as Proximal Policy Optimization (Schulman et al. (2017)), and Soft Actor-Critic (Haarnoja et al. (2018)). We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.
We demonstrate the advantages of our method through comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the OpenAI Gym simulator (Brockman et al. (2016)). We find that our method (i) has a lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows us to accurately estimate empowerment from images. We hope such a review of the existing methods for empowerment estimation will help push this research direction.

heading: 2 PRELIMINARIES
text: In this section, we review the necessary background for our method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity. We also review the underlying components in relevant prior work which we use for comparison to our method.

heading: 2.1 REINFORCEMENT LEARNING
text: The reinforcement learning (RL) setting is modeled as an infinite-horizon Markov Decision Process (MDP) defined by: the state space S, the action space A, the transition probabilities p(s′|s, a), the initial state distribution p0(s), the reward function r(s, a) ∈ R, and the discount factor γ. The goal of RL is to find an optimal control policy π(a|s) that maximizes the expected return, i.e. maxπ Es0∼p0,at∼π,st+1∼p [∑∞ t=0 γ tr(st, at) ] (Sutton & Barto (2018)).

heading: 2.2 EMPOWERMENT
text: Interaction between an agent and its environment is plausibly described by the perception-action cycle (PAC), (Tishby & Polani (2011)), where the agent observes the state of the environment via its sensors and responds with its actuators. The maximal information rate from actuators to sensors is an inherent property of PAC which characterizes the empowerment of the agent, formally defined below.
Empowerment (Klyubin et al. (2005)) is defined by the maximal mutual information rate (Cover & Thomas (2012)) between the agent’s sensor observations o ∈ O and actuators a ∈ A given the current state s ∈ S . It is fully specified by a fixed probability distribution of sensor observations conditioned
2Broadly speaking, an increase in the rate of information flow in the perception-action loop (Tishby & Polani (2011)) could be an impetus for the development of homosapiens, as hypothesized in (Yuval (2014)).
on the actuators and the state, p(O | A, s), denoted by “channel”, and by a free probability of the actuator signals conditioned on the state, ω(A | s), denoted by “source”. The empowerment E is the capacity of the channel, which is found via optimization of the source distribution ω(A | s):
E(s) = max ω(A|s) I[O;A | s] = max ω(A|s) ∑ O,A p(O,A | s) log ( p(A | O, s) ω(A | s) ) (1)
where I[O;A | s] is the mutual information, reflecting the difference between the entropy of the sensor inputs,H(O | s) and the corresponding conditional entropy,H(O | A, s)3. In this work, we follow the original empowerment formulation (Klyubin et al. (2005)), where A is an action sequence of length T starting from the state s and O is the resulting observation sequence. However, our method is applicable to other choices of observation and control sequences.
Remark 1 The source distribution in Equation 1, ω∗(A | s), is used only for the estimation of empowerment. Another policy, π(a | s), is calculated using empowerment as an intrinsic reward by either reinforcement learning (Karl et al. (2019)) or a 1-step greedy algorithm (Jung et al. (2011); Salge et al. (2013)) to reach the states with maximal empowerment. Strictly speaking, the source distribution ω maximizes the mutual information, (i.e. estimates the empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.
In general, the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel, as reviewed below. As a result, past works have largely relied on variational lower bounds instead.

heading: 2.2.1 EMPOWERMENT BY VARIATIONAL LOWER BOUNDS
text: The mutual information in Equation 1 can be bounded from below as follows (Mohamed & Rezende (2015); Gregor et al. (2017); Karl et al. (2019); Poole et al. (2019)):
∑ O,A p(O,A | s) log (p(A | O, s) ω(A | s) ) = ∑ O,A p(O,A | s) log (q(A | O, s) ω(A | s) ) +DKL [p(A | O, s) || q(A | O, s)]
≤ ∑ O,A p(O,A | s) log (q(A | O, s) ω(A | s) ) (2)
where q(A | O, s) and ω(A | s) are represented by neural networks with parameters φ and ψ respectively. The lower bound in Equation 2 can be estimated from samples using reinforcement learning with intrinsic reward r(st, at) = ∑T t log qφ(at | ot, s) − logωψ(at | st), as detailed in (Gregor et al. (2017); Karl et al. (2019)). To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions: qφ(at | ot, s) and ωψ(at | st) for the lower bound in Equation 2, and π(a | s) as the control policy (see Remark 1). Developing the mutual information in Equation 1 in the opposite direction, one gets another lower bound on empowerment, an approximation of which was used in (Sharma et al. (2020)):
∑ O,A p(O,A | s) log (p(O | A, s) ω(O | s) ) = ∑ O,A p(O,A | s) log (q(O | A, s) ω(O | s) ) +DKL [p(O | A, s) || q(O | A, s)]
≤ ∑ O,A p(O,A | s) log (q(O | A, s) ω(O | s) ) . (3)
For sake of the clarity of the exposition, the above-explained variational empowerment is schematically depicted in Figure 1. The controller policy, πθ(a | s), collects transitions in environment,
3when the agent does not have an access to the true state, s, empowerment is calculated as the maximal mutual information between actuators and sensors given an estimated state.
(s, a, s′). The source policy ωψ(a | s) generates the action a at every state s visited by the controller policy, and the environment transits to the next state, s′ (yellow block). The inverse channel policy qφ(a | s, s′) is regressed to a given s and and s′, generating the intrinsic reward, C = logωψ(a | s)− log qφ(a | s, s′), which is used as the reward to improve the controller policy via reinforcement learning. This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.

heading: 2.2.2 EMPOWERMENT FOR GAUSSIAN CHANNELS
text: The Gaussian linear channel in our case is given by:
O = G(s) ·A+ η, (4) where O and A are the observation and actuator sequences with dimensions dO × 1 and dA × 1, respectively; G(s) is a matrix with dimensions dO × dA; and η is multivariate Gaussian noise with dimension dO×1. Following Salge et al. (2013), we assume the noise is distributed as a Gaussian. For completeness, Appendix A provides the analytical derivation of G(s) in the case of known dynamics for pendulum for the action sequence A with length 3. In contrast, the current work proposes a method how to estimate G(s) for arbitrary dynamics and arbitrary length of the action sequence from samples. The details of the calculation of Gaussian channel capacity are included in Appendix B.
The key observation in our method is that we can learn G(s) from samples, which allows us to estimate empowerment efficiently for arbitrary dynamics, including for high-dimensional dynamical systems or from images (pixels). As a result, it is not required to resort to lower bounds as done in prior work. In Section 4, we describe how to learn G(s) from samples using deep neural networks.

heading: 3 RELATED WORK
text: In this section, we review the relevant prior work and state of the art in empowerment estimation, emphasizing the main differences between these existing works and our proposed approach.
Estimation via Gaussian channel. The representation given by Equation 4 was previously considered in (Salge et al. (2013)), where the channel p(O | A, s) was assumed to be known. In our current work, we remove this assumption and extend the applicability of the efficient channel capacity solution given by Equation 11 to arbitrary observations, including images.
Variational approaches to estimation. The lower bound in Equation 2 is used in (Mohamed & Rezende (2015); Gregor et al. (2017); Karl et al. (2019)), where different additional assumptions are made. Specifically, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor et al. (2017); Karl et al. (2019) provide a solution for closed-loop empowerment, and Karl et al. (2019) assumes differentiability of the dynamics. In our work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efficient, and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained in the Introduction. The works by (Mohamed & Rezende (2015); Gregor et al. (2017)) precede that by (Karl et al. (2019)), and we also experimentally find the latter to be more stable. Consequently, in Section 5 we provide comparisons to Karl et al. (2019), which represents the current start of the art in this line of research.
Skill discovery. An approximation of the lower bound in Equation 3 is successfully used by Sharma et al. (2020) in the context of learning a set of skills. Another relevant and important work is Eysenbach et al. (2019). Citing the authors, their work is about “maximizing the empowerment of a hierarchical agent whose action space is the set of skills”. Section 5 provides comparisons between our empowerment estimator and these methods with regards to stability in training, sample efficiency, and correctness of the empowerment landscape generated by both methods. Although they aim to achieve a different goal from our work – unsupervised skill discovery vs unsupervised stabilization – their objectives are the same type us ours: maximizing mutual information between actuators and sensors. In Section 5 we show that the empowerment landscape derived by these methods converge to the true empowerment landscape in simple cases, but fail to converge to a known empowerment landscape in more complicated dynamical systems.
These works represent the state of the art in empowerment estimation. We believe a thorough comparison between our method and these works, as provided in Section 5, clearly demonstrate the main properties of our method and clarify our contribution to the improvement of sample-based empowerment estimation in dynamical systems. As mentioned in the introduction, empowerment is useful for a broad spectrum of different applications. In this section, we highlighted the most relevant and updated methods, which we hope are helpful to see the merits and advances of our approach.
Our improved empowerment estimation possesses the essential properties of the empowerment function: in particular, it converges to the correct empowerment landscape. The training has improved stability. Furthermore, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5, where we show purely intrinsically motivated (empowerment-based) stabilization of inverted pendulum from images. Accurate estimation of empowerment from images significantly expands the possible breadth of applications for empowerment.

heading: 4 PROPOSED METHOD
text: The desiderata for our method as follows:
• to learn the channel matrix (G(s) in Equation 4 from samples for arbitrary dynamics. • to utilize general RL optimizers to reliably train intrinsically motivated policies. • to enable empowerment maximization from arbitrary dynamics using latent representations.
These are desirable as: the ability to learn G(s) from samples would allow us to estimate empowerment by the efficient line search algorithm in Equation 12, recent advances in RL algorithms can alleviate challenges in stability and sample efficiency, and latent representations would allow our method to be applicable even when the observation or underlying dynamics do not satisfy the Gaussian Linear Channel assumption.
In this work, we present Latent Gaussian Channel Empowerment, or Latent-GCE in short. Schematically, our method is depicted in Figure 2. The controller policy πθ(a | s) collects trajectories in the environment, (s, a, s′), where s is either a state or image observation. The channel matrix Gχ(st) predicts the final state of a trajectory corresponding to the action sequence aT−1t , starting from st. After performing singular value decomposition onGχ(st), the empowerment value E(st) is estimated by Equation 11 and Equation 12. The empowerment along trajectories provide intrinsic rewards, denoted by r′(τ), for the update of the controller policy, πθ, using off-the-shelf RL algorithms. Our method is summarized in high level in Algorithm 1.

heading: 4.1 EMBEDDING OF OBSERVATIONS AND ACTION SEQUENCES
text: While the Gaussian Linear Channel defined in Equation 4 provides a good estimate of simple dynamic systems, we want our empowerment estimator to generalize to arbitrary dynamics, including highdimensional systems, and environments where the observations are images. Using autoencoders, we embed the observations and action sequences such that their latent vector representations satisfy the Gaussian Linear Channel property. We use CNN encoder and decoder for image observations, or fully-connected networks for state observations. The encoder-decoder pair for action sequences are also parameterized by fully-connected networks. Let fχ, f−1χ , gχ, g −1 χ denote the encoder-decoder pairs for the observations and action sequences, fχ maps the observations into df dimensional latent
Algorithm 1 Latent Gaussian Channel Empowerment Maximization 1: Input: πθ0(a | s) - control policy, H - prediction horizon. 2: for i = 0 to N do 3: {τk}Kk=1 ← run policy πθi(a | s) {data collection in environment} 4: extract tuples (st, aH−1t , st+H) from {τk}Kk=1 {data arrangement for Gaussian channel} 5: for j = 1 to M do 6: χ∗ ← arg min
χ
∑ tuples ∥∥Gχ(st) · aHt − st+H∥∥22 {projection to Equation 4} 7: end for 8: {σ∗i (st)}ki=1 ← SVD(Gχ∗(st)) {an alternative explained in Appendix C} 9: E(st)← 12 ∑k i=1 log(1 + σ ∗ i (s)p ∗ i ) {Equation 11 and Equation 12}
10: reinforce πθi(a | s) with r′(τ) = ∑T t=1 γ
t−1E(st). {policy update} 11: end for
states, and gχ maps the H-step action sequences into dg dimensional latent actions. Our Latent Gaussian Linear Channel is formulated as follows:
fχ(st+H) = Gχ(fχ(st)) · gχ(aHt ) + η (5)

heading: 4.2 LEARNING CHANNEL MATRIX G
text: The channel matrixGχ(fχ(st)) has dimensions df×dg , where df and dg are the dimensions of latent states and actions as defined in section 4.1. In the Latent Gaussian Linear Channel, as formulated in Equation 5, the ending latent state is approximated by a matrix-vector multiplication between the channel matrix Gχ(fχ(st)) and the latent action gχ(aHt ). The mapping from a latent state to its channel matrix is parameterized by a 3-layer fully-connected neural network. The output layer of the network contains df · dg neurons, which is then reshaped into the df × dg matrix.

heading: 4.3 OVERALL OPTIMIZATION OBJECTIVES
text: The overall training objective for our Latent-GCE estimation consists of two parts: reconstruction loss and latent prediction loss. The reconstruction loss LR is the average L2 distance between the decoded latent representations and the original encoder inputs. Minimizing LR encourages the autoencoders to learn meaningful latent representations. The prediction loss LP is the average L2 distance between the matrix-vector product and the true latent ending state. Minimizing LP forces the latent space to maintain the Gaussian Linear Channel structure:
L(χ) = LR + LP = Es ∥∥f−1χ (fχ(s))− s∥∥22 + EaHt ∥∥g−1χ (gχ(aHt ))− aHt ∥∥22 +
E(st,aHt ,st+H) ∥∥Gχ(fχ(st)) · gχ(aHt )− fχ(st+H)∥∥22 (6)
Overall, our Latent-GCE Maximization scheme (Figure 2) trains two sets of parameters, (θ, χ), while VLB methods (Figure 1) generally needs three sets, (θ, φ, ψ). The reduced trainable components
suggest that our method might be superior in terms of better stability and quicker convergence, which we confirm in our experiment section.

heading: 5 EXPERIMENTS

heading: 5.1 UNSUPERVISED STABILIZATION OF CLASSIC DYNAMIC SYSTEMS
text: As discussed in Section 1, a control policy that maximizes agent empowerment should lead to agent stabilization, even at an unstable equilibrium. To evaluate this, we compare our method with two prior works by Mohamed & Rezende (2015) and Karl et al. (2019) on their ability to stabilize simple dynamic systems. Specifically, in the 2D ball-in-box environment, empowerment is highest at the center of the box, where the ball is free to move in all directions; in the pendulum environment, empowerment is highest at the up-right position, where gravity assists in reaching other states. We compare average squared distance away from the stable position during policy training. As shown in Figure 3, all three methods are successful on the simple ball-in-box environment, whereas only our method consistently stabilizes the pendulum at an unstable equilibrium. We refer readers to Appendix D for detailed visualization of pendulum empowerment/policy learning using our method, and Appendix E for results on cart-pole.

heading: 5.2 QUALITATIVE CONVERGENCE OF DIFFERENT ESTIMATORS
text: We now qualitatively compare the convergence of different empowerment estimators to the true empowerment function. Not only do we include solutions by VLB methods that approximate the classic empowerment definition (Mohamed & Rezende (2015); Karl et al. (2019)), but we also compare with recent works that employ empowerment-like objectives for skill discovery (Eysenbach et al. (2019); Sharma et al. (2020)). From Figure 4, we observe that in simple environments such as ball-in-box, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, corresponding to previous work (Klyubin et al. (2005)). For the pendulum environment, it is clear that while both the method by Karl et al. (2019) and ours succeed in creating results similar to the analytical solution (the ground truth), ours outputs an empowerment landscape that is smoother and more discerning. Detailed derivation of the analytical solution is included in Appendix A.

heading: 5.3 STABILITY OF EMPOWERMENT ESTIMATIONS
text: Besides the convergence to correct empowerment values, the stability and sample efficiency are also key aspects to determine whether the empowerment estimators are suitable as reliable intrinsic reward signals. We compute the variance of average empowerment values over 10 seeds as a measurement of algorithm stability. Since the outputs by existing methods are on different scales, we instead compare the relative standard deviation. As shown in Figure 5, our method outperforms the existing sample-based methods in that it has lower variance in training and faster convergence to the final estimator. These properties make our method competent as a reliable unsupervised training signal.

heading: 5.4 EMPOWERMENT ESTIMATION IN HIGH-DIMENSIONAL SYSTEMS
text: We verify the applicability of Latent-GCE in high-dimensional systems from two aspects: first, the latent space should be invertible - able to predict the true underlying dynamics; moreover, the empowerment maximization policy should develop stabilizing behaviors.

heading: 5.4.1 PIXEL-BASED PENDULUM
text: We use a pendulum environment that supplies 64 × 64 images as observations. As input to our Latent-GCE algorithm, we concatenate two consecutive frames so as to capture the angular velocity. In Figure 6, we start the pendulum at its up-right position s0. Using the encoder fχ, we map the images into the latent representation fχ(s0). Given different action sequences aH0 , we predict the ending latent state as zt+H = Gχ(fχ(st)) · gχ(aHt ). If our learned Latent-GCE model captures the true system dynamics, we expect the reconstruction of the ending state f−1χ (zt+H) to match the ground truth st+H . This property holds true as verified in Figure 6. More importantly, as shown in Figure 7, the empowerment estimations given by our method is not only close to the analytical solution, but also trains a policy that swing the pendulum up from bottom to balance at the top. On the other hand, VIM by Mohamed & Rezende (2015) which we compare as a baseline, fails to assign a high empowerment estimation for the up-right position.

heading: 5.4.2 HOPPER
text: We evaluate the ability of Latent-GCE to stabilize the Hopper environment from OpenAI Gym. This environment is more challenging because of higher state dimension and long horizon (64) for empowerment estimation. Here we expect an empowerment maximization policy to balance the hopper at its standing position where it is able to move left and right freely. In Figure 8 we see our method successfully balances hopper whereas a random policy inevitably makes the hopper fall over.

heading: 6 SUMMARY AND DISCUSSION
text: In this work we introduce a new sample-based method for empowerment estimation in dynamical systems. The key component of our method is the representation of dynamical system by the Gaussian channel, which allows us to estimate empowerment by convex optimization. Our estimator converges to true empowerment landscape both from raw states and images. Empowerment is used in various applications and our improved estimator has a potential to improve these applications. For example in Appendix F, we show that when the agent is trained with an empowerment-aware objective, it develops distinguishable levels of attention to safety. Moreover, a natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.

heading: ACKNOWLEDGMENTS
text: This research is supported by Berkeley Deep Drive, Intel, and Amazon through providing AWS cloud computing services.

heading: B GAUSSIAN CHANNEL CAPACITY
text: The capacity of the channel in Equation 4 is given by, (Cover & Thomas (2012)):
E(s) = max pi≥0∑k i pi≤P
1
2 k∑ i=1 log(1 + σi(s)pi) (11)
where σi(s), pi, k, and P , are the i-th singular value of G(s), the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively. The optimal solution to Equation 11 is given by:
p∗i (ν) = max(1/ν − 1/σi(s), 0), k∑ i p∗i (ν) = P, (12)
where ν is found by a line search (Cover & Thomas (2012)) to satisfy the total power constraint.

heading: C SVD ALTERNATIVE
text: The singular values, σi(st), can be estimated directly without the singular values decomposition by representing the channel matrix as Gχ(st) = Uχ(st) ∗ Diagm(Σχ(st)) ∗ Vχ(st) with two additional orthonormality constraints: ∥∥Uχ(st)U†χ(st)− I{ds×ds}∥∥22 and∥∥Vχ(st)V †χ (st)− I{da(H−1)×da(H−1)}∥∥22. In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images. While we don’t implement this in our experiments, this alternative direct calculation can be applied when SVD is undesired.

heading: D TWO PHASES IN EMPOWERMENT-BASED STABILIZATION
text: Empowerment of the pendulum is highest when it is stationary at the up-right position. In this section, we show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm 1 without any external reward. In Figure 10, we observe that Algorithm 1 includes two phases: empowerment learning and policy learning. Empowerment converges from epoch 0 to 10 and then, policy converges from epoch 10 to 50.

heading: E CART-POLE SWING-UP
text: The cart-pole environment is made up of a stiff pole with one end fixed to a pivot on a cart movable along one axis. The cart can be pushed left or right within a confined region. The state space of cart-pole has 4 components: cart position, cart velocity, pole angle, pole angular velocity. We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states. To measure whether Algorithm 1 successfully brings the cart-pole to up-right position, we plot the average squared angle to the top against the number of training steps. We also include 2 different reward functions that directly aim this objective for comparison.
As shown in Figure 11, the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar final performance compared to direct reward signals. Moreover, it learns this objective at a speed on par with the dense reward.

heading: F EMPOWERMENT ESTIMATION FOR SAFETY
text: A useful application of empowerment is its implication of safety for the artificial agent. A state is intrinsically safe for an agent when the agent has a high diversity of future states, achievable by its actions. This is because in such states, the agent can take effective actions to prevent undesirable futures. In this context, the higher its empowerment value, the safer the agent is. In this experiment, we first check that our calculation of empowerment matches the specific design of the environment. Additionally, we show that empowerment augmented reward function can affect the agent’s preference between a shorter but more dangerous path and a longer but safer one.
Environment: a double tunnel environment implemented with to the OpenAI Gym API (Brockman et al., 2016). Agent (blue) is modeled as a ball of radius 1 inside a 20×20 box. The box is separated by a thick wall (gray) into top and bottom section. Two tunnels connect the top and bottom of the box. The tunnel in middle is narrower but closer to the goal compared to the one on the right.
Control: In each time step, the agent can move at most 0.5 unit length in each of x and y direction. If an action takes the agent into the walls, it will be shifted back out to the nearest surface.
Reset criterion: each episode has a maximum length of 200 steps. The environment resets when time runs out or when the agent reaches the goal.
Since the tunnel in the middle is narrower, the agent is relatively less safe there. The effectiveness of the control of the agent is damped in 2 ways:
1. In a particular time step, it’s more likely for the agent to bump into the walls. When this happens, the agent is unable to proceed as far as desired.
2. The tunnel is 10 units in length. When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon. Thus, the agent has fewer possible future states.
We trained the agent with PPO algorithm (Schulman et al., 2017) from OpenAI baselines (Dhariwal et al., 2017) using an empowerment augmented reward function. After a parameter search, we used discount factor γ = 0.95 over a total of 106 steps. The reward function that we choose is:
R(s, a) = 1goal + β × Emp(s)
where β balances the relative weight between the goal conditioned reward and the intrinsic safety reward. With high β, we expect the agent to learn a more conservative policy.
The results from the tunnel environment again support our proposed scheme. First, the empowerment landscape matches our design of the environment. Second, high quality empowerment reward successfully alters the behavior of the agent.

heading: G EXPERIMENT DETAILS
text: G.1 USE OF NEURAL ENCODERS
In our state-based classic control environments, the encoder/decoder pairs for observations and actions are set to be the identity map. In this case, the sequences of actions are simply concatenated. For the Hopper environment and pixel-based pendulum, encoders and decoders are parameterized by neural networks.
G.2 DETAILS ON NEURAL NETWORK LAYERS
G.2.1 CNN FOR IMAGE ENCODING
(h1) 2D convolution: 4 filters, stride 2, 32 channels, ReLU activation (h2) 2D convolution: 4 filters, stride 2, 64 channels, ReLU activation (h3) 2D convolution: 4 filters, stride 2, 128 channel, ReLU activations (h4) 2D convolution: 4 filters, stride 2, 256 channel, ReLU activations (out) Flatten each sample to a 1D tensor
G.2.2 DECONVOLUTIONAL NET FOR IMAGE RECONSTRUCTION
(h1) Fully connected layer with 1024 neurons, ReLU activation (h1’) Reshape to 1× 1 images with 1024 channels (h2) 2D conv-transpose: 5 filters, stride 2, 128 channels, ReLU activation (h3) 2D convolution: 5 filters, stride 2, 64 channels, ReLU activation (h4) 2D convolution: 6 filters, stride 2, 32 channel, ReLU activations (out) 2D convolution: 6 filters, stride 2, C channel
G.2.3 MLP FOR ACTION SEQUENCE ENCODING
(h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h3) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with 32 output neurons
G.2.4 MLP FOR ACTION SEQUENCE RECONSTRUCTION
(h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with T × da neurons, tanh activation then scaled to action space
G.2.5 MLP FOR TRANSITION MATRIX A
(h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with df × dg neurons
G.3 DYNAMICS-AWARE DISCOVERY OF SKILLS (DADS)
For our experiments using DADS, we use policy and discriminator architecture of two hidden layers of size 256 with ReLU activations and a latent dimension of 2. Our discount factor is .99. As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3× 10−4 and batch size of 256 are used throughout. We use a replay buffer to help stabilize training of size 20000, used only for policy updates.
G.4 HOPPER EXPERIMENT DETAILS
Each episode is set to a fixed length of 200 steps. We use PPO as the RL backbone for our control policy. The policy is updated every 1000 steps in the environment, with learning rate of 1e-4 and γ = 0.5.

Review summaries:
The paper proposes an new algorithm to simultaneously estimate and maximise empowerment for achieving unsupervised stabilization. The method relies on the formulation of a dynamic system as a linear Gaussian channel. In this formulation, empowerment can be efficiently estimated by solving a line search problem. The authors propose to learn the channel matrix G(s) from samples and exploit then linear formulation to learn a corresponding policy maximising empowerment. These two steps are done in an alternating fashion until convergence. In the experiments the method is compared to previous approaches for unsupervised control via empowerment. The authors show convergence close to the true empowerment landscape and prove the sample efficiency and low variance of their method. Additionally, their method is capable of unsupervised control based on image observations.

The paper studies reward-free reinforcement learning (RL) methods based on empowerment. The authors propose a technique for empowerment estimation under the assumption that a state after H timesteps can be factorized as a product of the current action and a matrix G(s) that depends on the current state. In contrast to the existing methods that rely on the optimization of variational lower bounds on mutual information for empowerment estimation, the technique allows having an almost closed-form expression for empowerment. The authors qualitatively demonstrate the convergence of their method to the true empowerment function on simple environments such as 2D ball-in-box as well on image-based Pendulum environment.