Efficient Empowerment Estimation for Unsupervised Stabilization
Authors: Ruihan Zhao
Kevin Lu
Pieter Abbeel
Stas Tiomkin
Venue: ICLR 2021 Poster (2021)
Keywords: unsupervised stabilization
representation of dynamical systems
neural networks
empowerment
intrinsic motivation

Abstract: Intrinsically motivated artificial agents learn advantageous behavior without externally-provided rewards. Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking. This follows from the coincidence between the objective of stabilization and the objective of empowerment. Unfortunately, sample-based estimation of this kind of mutual information is challenging. Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, unstable in training, and have high sample complexity. In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efficiently calculate an unbiased estimator of empowerment by convex optimization. We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches. Specifically, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images. Consequently, our method opens a path to wider and easier adoption of empowerment for various applications.

Published as a conference paper at ICLR 2021
EFFICIENT EMPOWERMENT ESTIMATION FOR UNSUPERVISED STABILIZATION
Ruihan Zhao, Kevin Lu, Pieter Abbeel, Stas Tiomkin Berkeley Artificial Intelligence Research Lab Electrical Engineering and Computer Sciences University of California, Berkeley, CA, USA (philipzhao, kzl, pabbeel, stas) @berkeley.ede
ABSTRACT
Intrinsically motivated artificial agents learn advantageous behavior without externally-provie rewards. Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright posi- tions, which a prototypical intrinsically motivated behavior for upright standing and walking. This follows from the coincidence between the objective of stabiliza- tion and the objective of empowerment. Unfortunately, sample-based estimation of this kind of mutual information is challenging. Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, unstable in training, and have high sample complexity. In this work, we propose an alternative solution based on trainable representation of a dynamical system as Gaussian channel, which allows us to efficiently calculate unbiased estimator of empowerment by convex optimization. We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of method by comparing it the existing VLB approaches. Specifically, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from im- ages. Consequently, our method opens a path to wider and easier adoption of empowerment for various applications.
INTRODUCTION
Intrinsic motivation allows artificial and biological agents acquire useful behaviours without external knowledge (Barto et al.) (2004): (henntanz et al. (2005); Schmidhuber (2010); Barto (2013); Oudeyer et al. (2016)). In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task task-specific reward, which is optimized by an artificial agent towards desired behavior (Mnih al. (2013); Schulman al. (2017)).
In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates the need for domain knowledge and reward engineering in some settings (Mohamed & Rezende (2015); [oothhofft et al. (2017); Pathak et al. (2017)). Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent's actuators and sensors can guide the agent towards states in the environment with higher potential to achieve a larger number of eventual goals (Klyubin al. (2005): Pissner-Grros & Freer 2013)).
Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al. (2005): [Salee al. (2014)). Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems (Jung ctt al. [201 I): Salge et al, (2013); Karl et al. (2019). this application, ascending the gradient the empowerment function coincides with the objective of optimal control for stabilization at unstable cquilibrium (Strogatz (2018), ) which is an important task for both engineering (Todorov (0066) and
Project page: httpt:://////.....
tes http://sseees.ggggl..oo 1e. www/view////eccccocccocc.t.....w htwp//atettggee

---

Published as a conference paper at ICLR 2021
biologica systems; refer this the essential property of empowerment. It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property.
Empowerment has been found to be useful for a broad spectrum of applications. including: unsu- pervised skill discovery (Sharma al. (2000); EEsenbbchh et al. (2019); Gregor et al. (2017) Karl
et al. (2019); Campos et al. (2020): human-agent coordination (Salge & Polani (2017 Guuckessberger
(0166) ); assistance (Du al. (202I); and stabilization (Tiomkin
(201 7)). Past work
has utilized variational lower bounds (VLBs) (Poole et al. A019);[Alemi et al. (2017) Tschannen
al. (2020); Mohamed Rezende (2015) achieve an estimate of the empowerment. However, VLB approaches to empowerment in dynamical control systems (Sharma et al. (2020); Achiam et al.)
(2018) ) have high sample complexity, are often unstable training, and may be biased. Moreover, was not previously studied if empowerment estimators learned via VLBS possess the essential properties of empowerment.
In this work, we introduce new method, Latent Gaussian Channel Empowerment (Latent-GCE), for empowerment estimation and utilize the above-mentioned property as an "indicator" for the quality of the estimation. Specifically, we propose particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories. This representation admits an efficient estimation of empowerment by convex optimization (Cover & Thomas (2012), both from raw state and from images. We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such Proximal Policy Optimization (Schulman et al. ),017), ) and Soft Actor-Critic (Haarnoja et al. (2018)) We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.
We demonstrate the advantages of our method through comparisons to the existing state-of-the -fettrrrrtt empowerment estimators in different dynamical systems from the OpenAl Gym simulator (Brockman (2016)). We find that our method has lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows us to accurately estimate empowerment from images. We hope such a review of the existing methods for empowerment estimation will help push this research direction.
2
PRELIMINARIES
In this section, review the necessary background for method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity. We also review the underlying components in relevant prior work which for comparison to method.
2.1 REINFORCEMENT LEARNING
The reinforcement learning (RL) setting is modeled as aifinite-horizon Markov Decision Process (MDP) defined by: the state space S, the action space A, the transition probabilities p(s'|s,a) a). the initial state distribution Po(s), the reward function r(s, a) E IR, and the discount factor y- The goal of RL is to find an optimal control policy (aass) that maximizes the expected return, i.e. max,
Ei (s(.s)))) (Sutton Barto (2018)).
2.2 EMPOWERMENT
Interaction between an agent and its environment is plausibly described by the perception-action cycle (PAC), Tishby & Polani 2011)). where the agent observes the state of the environment via its sensors and responds with its actuators. The maximal information rate from actuators to sensors an inherent property of PAC which characterizes the empowerment of the agent, formally defined below.
Empowerment (Klyubin et al. (2055) defined by the maximal mutual information rate (Covrr & Thomas (2012)) between the agent's sensor observations O and actuators E A given the current state s E S. It is fully specified by a fixed probability distribution of sensor observations conditioned
Broadly speaking, increase in the rate of information flow in the pereeption-action loop Tishby & Polani T0 could be impetus for the development of homosapiens, hypothesized in Yuval (014)).
2

---

Published as a conference paper at ICLR 2021
on the actuators and the state, p(O A, s), denoted by "channel", and by free probability of the actuator signals conditioned on the state, w(A s), denoted by "source". The empowerment the capacity of the channel, which is found via optimization of the source distribution w(A) s):
P(A 0,,) w(A ())
E()
max I w(Als)
max w(A)a)
Σ O,A
log
where I[O; A { is the mutual information, reflecting the difference between the entropy of the sensor inputs, H(O) s) and the corresponding conditional entropy, H(O
In this work, we follow the original empowerment formulation (Klyubin et al. (2005)), where A is an action sequence of length 7 starting from the state and O is the resulting observation sequence. However, our method is applicable to other choices of observation and control sequences.
Remark The source distribution in Equation ( s). is used only for the estimation of empowerment. Another policy, (a 8), is calculated using empowerment as an intrinsic reward by either reinforcement learning (Karl et al. (2019)) or a 1-step greedy algorithm (Jung et al. (2011); Salge et al. (2013)) to reach the states with maximal empowerment. Strictly speaking, the source distribution w maximizes the mutual information, (i.e. estimates the empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.
In general, the capacity of an arbitrary channel for continuous random variable is unknown except for a few special cases such as the Gaussian lincar channel, as reviewed below. As a result, past works have largely relied on variational lower bounds instead.
2.2.1 EMPOWERMENT BY VARIATIONAL LOWER BOUNDS
The mutual information in Equation can be bounded from below as follows (Mohamed & Rezende (2013): Gregor et al. (2017): Karl et al. 2019); Poole et al. (2019))
Σ E(O log 0.1
p(A [ w(A
Σ p(O, log
9(4 0,+) + DEL [p(A (,s) ll 2(A 0,
w(A]
[(AA 0, w(A] ()
(O,A| 8) log
0,4
where (A O) and w(A s) represented by neural networks with parameters 6 and ri respectively. The lower bound in Equation 2 can be estimated from samples using reinforcement
st), as detailed in (Gregor al.) (2017): Karl al. (2019). To apply this variational lower bound on empowerment to
learning with intrinsic reward r(st, 0z) = Σi log qo(a=
s) log
unsupervised stabilization of dynamical systems one needs to learn three distributions: (a, o, s)
and (ae 8x) for the lower bound in Equation 2 and π (a {s) as the control policy (see Remark I].
Developing the mutual information in Equation D in the opposite direction, one gets another lower bound on empowerment, an approximation of which was used (Sharma al. (2020)):
Σ E(OAA|)) 5) log P(O A,,) 0.A
Σ P(O,A) A log
{0 A. w(O [s)
DEL [p(O AAs) 11 (( | A)
w(O
0,A
9(O A.s) w(O 8)
Σ 0,4
p(, log
(3)
For sake of the clarity of the exposition, the above-explained variational empowerment is schemati- The controller policy, Ta(a s), collects transitions in environment,
cally depicted in Figure [
when the agent does not have access to the true state, 8, empowerment is calculated the maximal mutual information between actuators and sensors given estimated state.
3

---

Published as a conference paper at ICLR 2021
xy(a [)
Environment
log s)
g(a (s)
(s,
gota s()
(1,0,
My 8)
Variational Approach
Figure 1: Estimation of empowerment via variational lower bounds.
(s, a. s').) The source policy (y s) generates the action a every state visited by the controller policy, and the environment transits to the next state, s' (yellow block). The inverse channel policy go 8, 8') is regressed to a given 8 and and 8', generating the intrinsic reward, C' = log we (a 8) log (a s, s,s)), which is used as the reward to improve the controller policy via reinforcement learning. This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.
2.2.2 EMPOWERMENT FOR GAUSSIAN CHANNELS
The Gaussian linear channel in our case is given by:
O G(s) A +.
(4)
where and A are the observation and actuator sequences with dimensions do 1 and da respectively; G(s) is matrix with dimensions do dA and is multivariate Gaussian noise with dimension do Following Salge et al, (201)). we assume the noise distributed as Gaussian. For completeness, Appendix (A] provides the analytical derivation of G(s) in the case of known dynamics for pendulum for the action sequence with length 3. contrast, the current work proposes method how to estimate G(s) for arbitrary dynamics and arbitrary length of the action sequence from samples. The details of the calculation of Gaussian channel capacity are included in Appendix [
The key observation in method is that estimate empowerment efficiently for arbitrary dynamics, including for high-dimensional dynamical systems from images (pixels). As result, it is not required to resort to lower bounds done in prior work. In Section we describe how to learn G(s) from samples using deep neural
learn G(s) from samples, which allows to
networks.
3 RELATED WORK
In this section, we review the relevant prior work and state of the art empowerment estimation, emphasizing the main differences between these existing works and our proposed approach.
Estimation via Gaussian channel. The representation given by Equation 4 was previously con- sidered in (Salge et al. )2 where the channel p(O A, 8) was assumed to be known. In current work, we remove this assumption and extend the applicability of the efficient channel capacity solution given by Equation II to arbitrary observations, including images.
Variational approaches to estimation. The lower bound in Equation 2 is used in (Mohamed & Rezende (2015): Gregor al. (2017); Karl et al. (2019)). where different additional assumptions are made. Specifically, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor al. 2017); 7 Karl al. (2019) provide solution assumes differentiabilityy of the dynamics. In our work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efficiemt and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained the Introduction. The works by (Mohamed & Rezende (2013); Gregor et al. (2017h) precede that by (Karl et al. (2019)), and also experimentally find the latter be more stable. Consequently. in Section we provide comparisons to Karl et al. (019), which represents the current start of the art in this line of research.
closed-loop empowerment, and Karl al. (2019)
4

---

Published as a conference paper at ICLR 2021
Skill discovery. An approximation of the lower bound in Equation is successfully used by Sharma et al. (2020) in the context of learning a set of skills. Another relevant and important work is Eysenbach et al. (2019). Citing the authors, their work is about "maximizing the empowerment of hierarchical agent whose action space is the set of skills". Section provides comparisons between empowerment estimator and these methods with regards stability in training, sample efficiency, and correctness of the empowerment landscape generated by both methods. Although they aim to achieve different goal from our work unsupervised skill discovery vs unsupervised stabilization their objectives are the same type ours: maximizing mutual information between actuators and we show that the empowerment landscape derived by these methods converge to the true empowerment landscape in simple cases, but fail to converge to a known empowerment landscape in more complicated dynamical systems.
sensors. In Section we
These works represent the state of the art in empowerment estimation. We believe thorough comparison between our method and these works, as provided in Section clearly demonstrate the main properties of our method and clarify our contribution to the improvement of sample-based empowerment estimation in dynamical systems. As mentioned in the introduction, empowerment useful for broad spectrum of different applications. In this section, we highlighted the most relevant and updated methods, which we hope are helpful to see the merits and advances our approach.
Our improved empowerment estimation possesses the essential properties of the empowerment function: in particular, it converges to the correct empowerment landscape. The training has improved stability. Furthermore, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5 where we show purely intrinsically motivated (empowerment-bascel stabilization of inverted pendulum from images. Accurate estimation of empowerment from images significantly expands the possible breadth of applications for empowerment.
4 PROPOSED METHOD
The desiderata for our method as follows:
to learn the channel matrix (G(s) in Equation from samples for arbitrary dynamics.
utilize general RL optimizers to reliably train intrinsically motivated policies.
to enable empowerment maximization from arbitrary dynamics using latent representations.
These are desirable as: the ability to learn G(s) from samples would allow us to estimate empow- erment by the efficient line search algorithm in Equation [2] recent advances in RL algorithms can alleviate challenges in stability and sample efficiency, and latent representations would allow our method be applicable even when the observation or underlying dynamics do not satisfy the Gaussian Linear Channel assumption.
In this work, we present Latent Gaussian Channel Empowerment, or Latent- GCE in short. Schemati- cally, our method is depicted in Figure 2 The controller policy (a s) collects trajectories in the environment, (s, a, s'), where s either state or image observation. The channel matrix G2 (s:) predicts the final state of trajectory corresponding to the action sequence a, aT-! starting from st- After performing singular value decomposition on Gx(ss) the empowerment value E(s)) is estimated by Equation TT and Equation 12 The empowerment along trajectories provide intrinsic rewards, denoted by (T). for the update of the controller policy, mo. using off-the-shelf RL algorithms. Our method is summarized in high level in Algorithm
4.1 EMBEDDING OF OBSERVATIONS AND ACTION SEQUENCES
While the Gaussian Linear Channel defined in Equation provides good estimate simple dynamic systems, we want our empowerment estimator to generalize to arbitrary dynamics, including high- dimensional systems, and environments where the observations are images. Using autoencoders, we embed the observations and action sequences such that their latent vector representations satisfy the Gaussian Linear Channel property. We use CNN encoder and decoder for image observations, or fully-com fully-connected networks for state observations. The encoder-decoder pair for action sequences also parameterized by fully-connected networks. Let fx- f- 9xx Ix denote the encoder-decoder pairs for the observations and action sequences, fx maps the observations into dy dimensional latent
5

---

Published as a conference paper at ICLR 2021
xy(a [})
Convex Optimiztion
(s,
(,,0,
SVD
MIMO Gaussian Model
(y(s))
Approach
Figure 2: Empowerment maximization via Latent-Goe
Algorithm Latent Gaussian Channel Empowerment Maximization
1: Input: Te, (a. 8) control policy, H prediction horizon.
2: for i = 0 to do
3: {Te)em
run policy TT, s)
(data collection in environment
4: extract tuples (st, ai H-1
8141) from (Te)
(data arrangement for Gaussian channel)
5: for = 1 to do
6:
arg min Etuples [[x(s)) at -14H 11 x
prooeection to Equation 4
7: end for
8: (s) - SV
(an alternative explained in Appendix
9: E(s,) + E log(1 + t
EEqution 11 and Equation 12
10: reinforce ne, (a with ('()) = Ei= --
(policy update) )
11:
end for
states, and 9x maps the H-step action sequences into dg dimensional latent actions. Our Latent Gaussian Linear Channel is formulated as follows:
fx (5+ Gx(x((s)) (att) +
4.2 LEARNING CHIANNEL MATRIX G
The channel matrix Gy (fx (st)) has dimensions d, x dg. where d and dg are the dimensions of latent states and actions defined section In the Latent Gaussian Linear Channel, as formulated in Equation 5 the ending latent state is approximated by matrix-vector multiplication between the channel matrix Gx (sx()) and the latent action 9x (a The mapping from latent state to its channel matrix is parameterized by 3-layer fully-connected neural network. The output layer of the network contains d,- d, neurons, which then reshaped into the d, x dg matrix.
4.3 OVERALL OPTIMIZATION OBJECTIVES
The overall training objective for Latent-GCE: estimation consists of two parts: reconstruction loss and latent prediction loss. The reconstruction loss LR is the average L2 distance between the decoded latent representations and the original encoder inputs. Minimizing LR encourages the autoencoders to learn meaningful latent representations. The prediction loss Lp is the average L2 distance between the vector product and the true latent ending state. Minimizing Lp forces the latent space to maintain the Gaussian Linear Channel structure:
L(x) = LR Lp
-aff = =E a112+ E || (a
[s m)) |[x((x(*1)) (x(a (x(++H)) 2
(6)
Overall, our Latent-Gee GCE Maximization scheme (Figure 2 trains two sets of parameters, (8, X). while VLB methods (Figure generally needs three sets, (0, Φ. v/). The reduced trainable components

---

Published as a conference paper at ICLR 2021
suggest that method might be superior in terms of better stability and quicker convergence, which we confirm in our experiment section.
5 EXPERIMENTS
5.1 UNSUPERVISED STABILIZATION OF CLASSIC DYNAMIC SYSTEMS
As discussed in Section control policy that maximizes agent empowerment should lead agent stabilization, even at an unstable equilibrium. To evaluate this, we compare our method with two prior works by Mohamed & Rezende (203)) and Karl et al. (2019) on their ability stabilize simple dynamic systems. Specifically, in the 2D Ball-in-box environment, empowerment is highest at the center of the box, where the ball is free to move all directions; in the pendulum environment, empowerment is highest at the up-right position, where gravity assists in reaching other states. We compare average squared distance away from the stable position during policy training, As shown in Figure 3 all three methods are successful on the simple ball-in-bol environment, whereas only our method consistently stabilizes the pendulum at an unstable equilibrium. We refer readers to Appendix for detailed visualization of pendulum empowermen//polccy learning using method, and Appendix for results on cart-pole.
Bal-Inbbxx
Pendulum
Mean IMean Sqiared Dictance to Conter
Outs
Outs
22.5
03.9
25
anviromenn stops
1x3
owwroommen stops
Figure 3: Evaluating whether methods learn to stabilize simple dynamical systems. All methods succeed for ball-in-box, whereas only our method succeeds for pendulum.
5.2 QUALITATIVE CONVERGENCE OF DIFFERENT ESTIMATORS
We now qualitatively compare the convergence of different empowerment estimators to the true empowerment function. Not only do we include solutions by VLB methods that approximate the classic empowerment definition (Mohamed & Rezende (015); Karl al. (2019)), but we also compare with recent works that employ empowerment- al. (2019); Sharma al. (020)). From Figure 4 we observe that in simple environments such as -n---i, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, corresponding For the pendulum environment, it is clear that while both the method by Karl et al. (2019) and ours succeed in creating results similar the analytical solution (the ground truth), ours outputs an empowerment landscape that is smoother and more discerning. Detailed derivation of the analytical solution is included in Appendix A
objectives for skill discovery (Eysenbach)
previous work (Klyubin al. (2005).
5.3 STABILITY OF EMPOWERMENT ESTIMATIONS
Besides the convergence to correct empowerment values, the stability and sample efficiency are also key aspects to determine whether the empowerment estimators are suitable as reliable intrinsic reward signals. We compute the variance of average empowerment values over 10 seeds as measurement of algorithm stability. Since the outputs by existing methods are on different scales, we instead compare the relative standard deviation. As shown in Figure our method outperforms the existing sample-based methods that has lower variance in training and faster convergence to the final estimator. These properties make our method competent as reliable unsupervised training signal.

---

Published as a conference paper at ICLR 2021
Top:
Ball-in-Box
Bottom:
Pendulum
Ground truth
Ours
VIM
Karl 2019
DIAYN
DADS
Figure 4: Empowerment landscapes (normalized) for ball-in-box (top) and pendulum (bottom). In ball-in-box, central positions correspond to high empowerment. For pendulum, upright positions correspond to high empowerment: x-axis is the angle to top. y-axis is the angular velocity. From
left to right, the plots show: ours, Mohamed Rezende 2015), Karl al. (201)) Eysenbach et al. (2019). Sharma et al. 22020).
Karl
DIAYN
10
10
201
10a
10a
10a
209
10F
105
105
850 of
20-1
10
200
LO00
0
4002
420
M00
200
1000
402
400
M00
200
1000
400
400
M00
200
1000
(a) Ball-in-box
(b) Pendulum
(c) Mountain car
(d) Cartpole
Figure 5: Relative standard deviation (RSD) in average empowerment value across 10 seeds. Our method converges faster to lower RSD during training.
5.4 EMPOWERMENT ESTIMATION IN DIGH--IMENSIONAL SYSTEMS
We verify the applicability of Latent-GCE in high-dimensional systems from two aspects: first, the latent space should be invertible able to predict the true underlying dynamics; moreover, the empowerment maximization policy should develop stabilizing behaviors.
5.4.1 PIXEL-BASED PENDULUM
We use a pendulum environment that supplies 64 x 64 images observations. As input to our Latent-Ger GCE algorithm, we concatenate two consecutive frames so as capture the angular velocity. In Figure [ we start the pendulum at its up-right position images into the latent representation (so). Given different action sequences ag we predict the ending latent state as ž Gx( (a If our learned Latent-GCE model captures the true system dynamics, we expect the reconstruction of the ending state (=++ to match the ground truth 8t+ This property holds true verified in Figure ( More importantly, shown in Figure 7 the empowerment estimations given by our method is not only close to the analytical solution, but also trains policy that swing the pendulum up from bottom to balance the top. On the other hand, VIM by Mohamed & Rezende (203)) which we compare as baseline, fails to assign a high empowerment estimation for the up-right position.
Using the encoder fx- we map the
5.4.2 HOPPER
We evaluate the ability of Latent-GCE to stabilize the Hopper environment from OpenAl Gym. This enviromment is more challenging because of higher state dimension and long horizon (64) for empowerment estimation. Here expect empowerment maximization policy to balance the hopper at its standing position where it is able to move left and right freely. In Figure & we see our method successfully balances hopper whereas random policy inevitably makes the hopper fall over.
8

---

Published as a conference paper at ICLR 2021
Action Sequences
Lalent Prediction
Ground Truth
-1,-1,-1.-11 -1,-11
/
1. 0,
1,1,1
Figure 6: Reconstruction in the latent dynamics vs the ground truth. Starting from the same upright position, a different action sequence is taken in each row (action +1: counter- cloner-clockwise torque, O: no torque, -1: clockwise torque). Reconstruction matches the actual observation all cases.
Figure Empowerment estimates for pixel-based pendulum. Latent-GCH learns reasonable empowerment function from pixels. Left: Analytical (x axis shifted by from Figure 4 Middle: Latent-GCE along with the swinging-up trajectory of the final control policy, Right: VIM.
Happer
Dlurs
Averoge triight torso
Random
environment Steps
Figure 8: Average torso height over evaluation episode of length 200 for the Hopper environment. Our method stabilizes the hopper to a standing position (2 xz 1).
SUMMARY AND DISCUSSION
In this work we introduce new sample-based method for empowerment estimation in dynamical systems. The key component of our method is the representation of dynamical system by the Gaussian channel, which allows to estimate empowerment by convex optimization. Our estimator converges to true empowerment landscape both from raw states and images. Empowerment is used in various applications and improved estimator has potential to improve these applications. For example in Appendix [F] we show that when the agent is trained with an empowerment-awwrr objective, develops distinguishable levels of attention to safety. Moreover, natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.
ACKNOWLEDGMENTS
This research is supported by Berkeley Deep Drive, Intel, and Amazon through providing AWS cloud computing services.

---

Published as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms, 2018
Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations (ICLR), 2017.
Andrew G Barto. Intrinsic motvattin and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pp. 17-47. Springer, 2013.
Andrew G Barto, Satinder Singh, and Nuttapong Chentanez. Intrinsically motivated learning of hier- archical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pp. 112-19, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning (ICML), 2020.
Nuttapong Chentanez, Andrew G. Barto, and Satinder P. Singh. Intrinsically motivated reinforcement learning. In K. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems 17, pp. 1-81-111811 1288. MIT Press, 2005. URL h http:/aapppssspssss .nps wwwppcccppaa...acccccp pappap 2552- intrinst ca y-mmotttt vated reinforcementt htwprreemment--leeainnii.neeee learning..p.....
Thomas M Cover and Joy Thomas. Elements of information theory. John Wiley & Sons, 2012.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford.
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines.
https
//githab.com//pee i/baeenna//aeelleeeee nes, 2017.
Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1): 219-245, 2000.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave: Assistance via empowerment. In Conference on Neural Information Processing Systems, NeurIPS. Thirty-fourth annual conference on Neural Information Processing Systems, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019, hRttss:::///sttteeeeee httpenrrwwwwwwweeeeeeeeeeeeeeeee net/ fttr:/oooummmi-SS663333333 jRqEm
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL htttps://openrevieee
pet//orum?id-aa Skc Swie---kc---aaooo
Christian Guckelsberger, Christoph Salge, and Simon Colton. Intrinsically motivated general com- panion npes via coupled empowerment maximisation. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration, 2017.
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous enviroment systems. Adaptive Behavior, 19(1) 1961);6-339, 2011.
Maximilian Karl, Philip Becker-Ehmckk Maximilian Soelch, Djalel Benbouzid, Patrick van-der Smagt, and Justin Bayer Unsupervised real-time control through variational empowerment, International Symposium Robotics Research, 2019.
10

---

Published as a conference paper at ICLR 2021
A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume pp. 128-135 Vol.1, Sep. 2005. doi: 10. 10010//CE.2200555556666
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. All else being equal be empowered. In European Conference on Artificial Life, pp. 744-753. Springer, 2005.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
P-Y Oudeyer, Jacqueline Gottlieb, and Manuel Lopes. Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. In Progress in brain research, volume 229, pp. 257-284--24 Elsevier, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction, 2017.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information, 2019.
C. Salge and D. Polani. Empowerment replacement for the three laws of roboties. Frontiers Robotics AI, 4:25, 2017.
Christoph Salge, Comelius Glackin, and Daniel Polani. Approximation of empowerment in the continuous domain. Advances in Complex Systems, 12(52093) 2013.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment- an introduction. In Guided Self-Organization: Inception, pp. 67-114. Springer, 2014.
Jürgen Schmidhuber. Formal theory creativity, fun, and intrinsic motivation (1990-2010). JEEE Transactions on Autonomous Mental Development, 2(3):230-247, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020. httlhtttt///ppereeeett
hwtp:/eet//otttteooeeee net forum idd HJgL2R4Vv
Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering. CRC press, 2018
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 2018
Stas Tiomkin, Daniel Polani, and Naftali Tishby. Control capacity of partially observable dynamic systems in continuous time. arXiv preprint (001v:7701.0440 2017.
Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perception-action cycle, pp. 601-636. Springer, 2011.
Emanuel Todorov. Optimal control theory. Bayesian brain: probabilistic approaches neural coding, pp. 269-298, 2006.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020. URL httttttss//ppperviiiee wwweeeiieeweeeeee.ee.eee .ne 2ttpeeerrrieeee.eeoooouu.uae.m.e
Alexander D Wissner- Gross and Cameron E Freer. Causal entropic forces. Physical review letters, 110(16):168702,, 2013.
Harari Yuval, Noah. Sapiens: A Brief History of Humankind. Harvill Secker, 2014.

---

Published as a conference paper at ICLR 2021
A ANALYTICAL DERIVATION OF G(s) FOR INVERTED PENDULUM
The main steps in the derivation of G(s) appear below. The full derivation appears in Section 4.1 at Salge et al. (003)). The current state of the pendulum, St, is given by dg and 6,:
0a 0g
=
where 0 is measured from the upright position. We consider the development of the pendulum for three time steps of duration dt. The behaviour of the pendulum can be approximate linearly every time step for small enough dt. Then, the next state of the pendulum is given by:
S1+ = 8t + A,dl + BeOs.
(8)
0
where a, is the agent's 's action, A, calculate a new matrix, A, because the non non-linearity of the system causes A to depend on the 0.+de
and B,
For the next time step need to
4 sin (0n)))
similar matrix can be calculated for A++2at- An propagation of pendulum dynamics for 3 time steps allows us calculate $1+3at explicitly, which appears as:
new state. Consequently, A sin(6
00+3dt dt( gdt sin sin(dté, + 0c) + gdt sin(O.) + 0.) sin(O.) + 20.) Oc + 3dla, 3dlac4d:
+ di( gdt
and,
gdt
7 sin(dtt, (dtt, + 0,) + gdt sin( +
gdl sin(di(
gdl 1 s0.) + 2 + 0g + 2dtaa) + dla, + dia + dla++2tt
1
where a++dt is the action at the next step. Our goal is to represent 8t43dt by:
at actdi a+
81+-dt K + G(t))
which is achieved by linearisation 0,43dt and d around a A straightforward calculation shows that the matrix G(s,) is given by:
2dt
dt 0
G(st)=
dt cos(dt (dt adt sin 20.) 0.) +1 1
(10)
This calculation be continued beyond 3-steps, involving more complicated terms in the expansions. Given G(st) in Equation I0 one can represent 8143d by Equation 9 where the matrix K does not affect the mutual information Cover & Thomas (2012). Adding to Equation 9 the normal noise, m, as suggested in Salge et al. (2013). we get the representation in Equation 4 In contrast, we propose learn G(s) from samples via interaction with the environment for an arbitrary dynamics and an arbitrary number of steps (c.f. 3 steps in the above derivation). Using G(s) from Equation 10 we can calculate the analytical (ground truth empowerment landscape), which is shown at Figure
B GAUSSIAN CHANNEL CAPACITY
The capacity of the channel in Equation is given by, (Cover & Thomas 2012
{(s) = max
log(1 + (s()pp))
(II)
pt 2
Σ P.
1
12

---

Published as a conference paper at ICLR 2021
Angular Ve leliiyty
Angle e from top
Figure 9: Analytical Empowerment Landscape
where oy(s), Pt. k and P. are the i-th singular value of G(s), the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively, The optimal solution to Equation 11 given by:
p (v) mwxpmau 1/0,())))
(12)
where v is found by a line search (Cover & Thomas (012 to satisfy the total power constraint.
C SVD ALTERNATIVE
The singular values, (8t), can estimated directly without the singular values decom- position by representing the channel matrix as (x()) Ux(s)) * Diagm(Ex(e (x)))
(s()) with two additional orthonormality constraints: [[Ux(s))U((s)) - (d, 112 and (()))) - (H
In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images. While we don't ' implement this in experiments, this alternative direct calculation be applied when SVD is undesired.
D TWO PHASES IN EMPOWERMENT-BASHE STABILIZATION
Empowerment of the pendulum is highest when it is stationary at the up-right position. this section, show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm [ without any external reward. In Figure TO we observe that Algorithm inccuuess two phases: empowerment learning and policy learning. Empowerment converges from epoch to 10 and then, policy converges from epoch 10 to 50.
E CART-POLE SWING-UP
The cart-pole environment made up of stiff pole with one end fixed to a pivot on cart movable along one axis. The cart can be pushed left right within a confined region. The state space of cart-pole has components: cart position, cart velocity, pole angle, pole angular velocity. We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states. To measure whether Algorithm [ successfully brings the cart-pole to up-right position, plot the average squared angle the top against the number of training steps. We also include different reward functions that directly aim this objective for comparison.
the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar final performance compared to direct reward signals. Moreover, it learns this objective speed par with the dense reward.
As shown in Figure 11
13

---

Published as a conference paper at ICLR 2021
epoch-0
epoch-10 10
epoch-20
epoch-30
epoch-40
epoch-50
Figure 10: Snapshots of the empowerment landscape (top). the control policy (middle) and the state concentration (bottom). The x-axis is / (angle) and the y-axis is (angular velocity). The angle is measured from the upright position, (0 = Orad corresponds to the upright position). Each epoch corresponds to 10 steps. The policy corresponds to the optimal policy for the stabilization task with external reward (Doya 2000
15
Empowemeent reward
DenGo rowand
Sparse rewird
30
3.5
Figure 11: Cart-Pole Comparison of unsupervised and supervised reward functions. The "dense reward" 222 and the "sparse reward" is (( < #
F EMPOWERMENT ESTIMATION FOR SAFETY
A useful application of empowerment is its implication of safety for the artificial agent. A state is intrinsically safe for an agent when the agent has high diversity of future states, achievable by its actions. This is because in such states, the agent can take effective actions to prevent undesirable futures. this context, the higher empowerment value, the safer the agent is. In this experiment, first check that our calculation of empowerment matches the specific design of the environment. Additionally, we show that empowerment augmented reward function can affect the agent's preference between shorter but more dangerous path and longer but safer one.
Environment: a double tunnel environment implemented with to the OpenAl Gym API (Brockman et al. 2016). Agent (blue) is modeled by a thick wall (gray) into top and bottom section. Two tunnels connect the top and bottom of the box. The tunnel in middle is narrower but closer the goal compared the
ball of radius inside 20 box. The box is separated
the right.
Control: each time step, the agent can move at most 0.5 unit length in each of x and y direction. If an action takes the agent into the walls, it will be shifted back out the nearest surface.
Reset criterion: each episode has a maximum length of 200 steps. The environment resets when time when the agent reaches the goal.
14

---

Published as a conference paper at ICLR 2021
16.0
15
50
2.5
65
-25
+.0
1.5
-00.
5.0
-100
-7.5
-5.0
-2.5
3.0
2.5
5.0
7.5
30.0
Figure 12: Left sub-figure: Double tunnel environment. The goal marked in red. The agent in blue. Right sub-figure: Empowerment landscape for the tunnel environment. The values of empowerment reduce at the corner and inside the tunnel where the control of the agent less effective compared to more open locations.
Since the tunnel in the middle is narrower, the agent is relatively less safe there. The effectiveness of the control of the agent is damped in 2 ways:
1. In particular time step, it's more likely for the agent to bump into the walls. When this happens, the agent is unable proceed far desired.
2. The tunnel is 10 units in length. When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon. Thus, the agent has fewer possible future states,
We trained the agent with PPO algorithm (Schulman al. 2017) from OpenAI baselines (Dhariwal et al. 2017) using an empowerment augmented reward function. After a parameter search, we used discount factor 7 0.95 over total of 10 steps. The reward function that we choose is:
R(s, a) 1 goat + Emp(ss
where 3 balances the relative weight between the goal conditioned reward and the intrinsic safety reward. With high 3, we expect the agent to learn a more conservative policy.
(a) 3
(b) 8 1/1600
(c) 3 = 1/800
Figure 13: Trajectories trained policy. As 3 increase, the agent develops a stronger preference over safer route, sacrificing hitting time.
The results from the tunnel environment again support our proposed scheme. First, the empowerment landscape matches our design of the environment. Second, high quality empowerment reward successfully alters the behavior of the agent.
15

---

Published as a conference paper at ICLR 2021
G EXPERIMENT DETAILS
G. USE OF NEURAL ENCODERS
In our state- state-based classic control environments, the encoder/decoder pairs for observations and actions set to be the identity map. In this case, the sequences of actions are simply concatenated. For the Hopper environment and pixel-based pendulum, encoders and decoders are parameterized by neural networks.
G.2 DETAILS ON NEURAL NETWORK LAYERS
G.2.1 CNN FOR IMAGE ENCODING
(h1) convolution: 4 filters, stride 2, channels, ReLU activation
(h2) 2D convolution: 4 filters, stride 2, 64 channels, ReLU activation
(h3) 2D convolution: 4 filters, stride 2. 128 channel, ReLU activations
(h4) 2D convolution: 4 filters, stride 2, 256 channel, ReLU activations
(out) Flatten each sample to a 1D tensor
G.2.2 DECONVOI.UTIONAL NET FOR IMAGE RECONSTRUCTION
(h1) Fully connected layer with 1024 neurons, ReLU activation
(h1') Reshape to x images with 1024 channels
(h2) 2D conv-transpose:
filters, stride 2, 128 channels, ReLU activation
(h3) 2D convolution: 5 filters, stride 2, 64 channels, ReLU activation
(h4) 2D convolution: 6 filters, stride 2, channel, ReLU activations
(out) 2D convolution: 6 filters, stride 2, C' channel
G.2.3 MLP FOR ACTION SEQUENCE ENCODING
(h1) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(h3) Fully connected layer with 512 neurons, ReLU activation
(out) Fully conected layer with 32 output neurons
G.2.4 MLP FOR ACTION SEQUENCE RECONSTRUCTION
(h1) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(out) Fully conected layer with T x dg neurons, tanh activation then scaled to action space
G.2.5 MLP FOR TRANSITION MATRIX
(h1) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(out) Fully conected layer with dy d, neurons
G.3 DYNAMICS- AWARE DISCOVERY OF SKILLS (DADS)
For our experiments using DADS, we use policy and discriminator architecture two hidden layers
of size 256 with ReLU activations and latent dimension of 2. Our discount factor is .99. As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3 x 10 and batch size of 256 are used throughout. We use a replay buffer help stabilize training of size 20000, used only for policy updates.
16

---

Published as a conference paper at ICLR 2021
G.4 HOPPER EXPERIMENT DETAILS
Each episode is set to a fixed length of 200 steps. We use PPO as the RL backbone for our control policy. The policy is updated every 1000 steps in the environment, with learning rate of le-4 and = 0.5.
17


Reviews:
[{"review_id": "sNBTg0gKyN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an new algorithm to simultaneously estimate and maximise empowerment for achieving unsupervised stabilization. The method relies on the formulation of a dynamic system as a linear Gaussian channel. In this formulation, empowerment can be efficiently estimated by solving a line search problem. The authors propose to learn the channel matrix G(s) from samples and exploit then linear formulation to learn a corresponding policy maximising empowerment. These two steps are done in an alternating fashion until convergence. In the experiments the method is compared to previous approaches for unsupervised control via empowerment. The authors show convergence close to the true empowerment landscape and prove the sample efficiency and low variance of their method. Additionally, their method is capable of unsupervised control based on image observations.\n\nOverall, I am torn between accepting and rejecting the paper and I lean towards rejection. I think it represents a good step  toward intrinsically motivated agents which can learn useful behaviour solely based on interaction with their environment, however, my major concern is about the clarity and the form of the paper. I think the description of the method is missing important details (see cons below) and I would like to see additional evaluations to support the claims of the paper. Overall I would recommend to polish especially the method and the experimental section before accepting it for publication.\n\nPros:\n1. The method is well placed in the existing literature on empowerment and provides a thorough comparisons of past approaches.\n\n2. The algorithm is novel and at the same time simple and very easy to understand. It relies on previous work for empowerment of linear Gaussian channels but frames the problem in a data-driven setting by learning the system dynamics from raw data.\n\n3. The paper provides good experimental results which compare the method to existing approaches and show advantages when it comes to sample efficiency and convergence to the true empowerment landscape for the pendulum. Additionally, the method allows empowerment-based control from image observations for the first time.\n\nCons:\n1. I miss important details in the paper:\n    \n    a) How is the parametrisation of G(s) with a neural network being done? Especially in the case of image based observations I would like more details.\n\n    b) Connected to (a), please make it clear how the sequence of actions a_t^{H-1} is handled.\n\n    c) You present an alternative to the SVD, that requires constraints. How do they enter the cost function or do you perform constraint optimization?\n\n2. I think the claims of the paper are a bit bold. The method requires that the system dynamics can be formulated as a linear Gaussian channel, which requires a control affine system. Nonetheless the paper claims that the approach can handle arbitrary dynamics. Can you clarify how the approach would tackle non-affine systems (dx = f(x, u)). There is also no support for this claim in the experiments. Specifically, except for the ball-in-box, all experiments are on pendulum-like systems.\n\n3. The subfigures 3g and 3h only show that the std of average empowerment is very low for the proposed method. I am missing an evaluation of the reward (empowerment) over time itself.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nccDaTSPcoP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, the paper is well organized, provides a very clear description of the method. \n\nReproducing results\nWill code be available? It is not mentioned that code is made available.\n\nCorrectness\nAuthors mention that training with empowerment converges to an unstable equilibrium (3rd paragraph in Introduction). This is not always the case but depends on the system. \nThe agent does not maximize the MI, but maximizes the potential MI, as Empowerment is not MI but channel capacity (4th paragraph Introduction).\n\nNovelty/Relevance\nThe relevance is that they use a representation for the dynamical system (i.e. Gaussian channel), that should improve the quality of the Empowerment estimation. \nThey mention that they for the first time estimate empowerment from images. Karl et al. 2017 also use a DVBF, which also allows a transition in latent space from images observed. \n\nUnclear\nThey estimate empowerment for a state s_t, which now is used to train a policy. However, normally the reward is a value for s_t, a_t pairs, so essentially a function of s_t+1. Should empowerment also be computed for s_t+1?\n\nSmall notes\nMaximilian et al. should be Karl et al\n\nI am uncertain whether the Gaussian channel is a contribution to the research community. Is it only useful for research done on empowerment? Can it be used for other systems that use a VLB?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EmzeQgMBum", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "EDIT: the authors have removed the claims I was concerned by, as well as adding a requested baseline. The new quantitative results on stabilization also make their method much more compelling. The new hopper results are a bit noisy (would it stay stabilized if training continued?), but are still quite promising in that they show an ability to handle more complex dynamics.\n4-->7, thanks for the great revision effort!\n\nThis work tackles the generally intractable problem of calculating channel capacity by learning a dynamics model with a highly constrained latent space (linear with Gaussian noise) that permits an efficient solution. This allows for using the exact empowerment (of the approximate dynamics) as the reward function for a reinforcement learning agent. This method is demonstrated on a few simple control problems, with the most difficult between a pendulum from pixel observations, which results in unsupervised stabilization.\n\nThe fundamental idea is sound; generalizing a special-case algorithm by forcing its constraints to hold in latent space has a long line of success (e.g. \"Embed to Control\"). And the model-based approach used here might allow for completely off-policy learning of empowerment. DADS is also model-based and could be adapted to optimize empowerment, but since its not a true lower-bound the case in favor of your approach over variational methods would be quite strong.\n\nBut the experiment baselines are not what they should be. Neither DADS nor DIAYN are measuring the same quantity as this method, namely empowerment: I(A^k; s_{t+k} | s_t). DADS is (almost) a lower bound on I(z ; s' | s) and DIAYN lower bounds I(z; s)+H(a|s, z). If there were some performance metric they all aim to achieve then this might be fine, but the only comparisons are to the quality of empowerment estimation (which they aren't estimating). I'd suggest switching to VIM[1] and VIC which lower bound I(A^k; s_{t+k} | s_t) and I(z; s_{t+k} | s_t) respectively.\n\nA larger problem that is more easily remedied is the use of overstated claims:\n\n\"our method has a lower sample complexity\" -- this is not shown. Yes, the empowerment of the modelled channel can be computed without additional interaction, but is building the model more sample efficient than getting a model-free estimate using a variational lower-bound? Maybe, but this isn't demonstrated.  \n\n\"allows, for the first time, to estimate empowerment from images\" -- this can only be true in the most contrived sense. While it's true that DIAYN and DADS both used explicit state, only DADS really exploits that decision architecturally. Indeed [2] includes a pixel-based version of DIAYN as a baseline without comment, and while that version of DIAYN doesn't perform that well, the method introduced there (VISR) tackles full Atari games, which constitute a much harder perceptual challenge than a pendulum on a blank background. Now, you could argue that none of these methods actually lower-bound empowerment (indeed, I just did in the previous paragraph). But adapting them lower-bound empowerment requires trivial changes e.g. for VISR you'd just have to evaluate the reverse predictor at the end of a trajectory rather than at every state. But one can look back far earlier. VIM [1] calculated empowerment from pixels in 2015. You could argue the pendulum environment is more complicated than the toy navigation tasks they used, but its hard to argue that it wouldn't work for pendulum since you didn't try it out.\n\nI am not willing to see this work published in its current state, but I promise I'll support publication if 1) VIM is added as a baseline for all (including pendulum from pixels) experiments 2) the overstated claim are dropped.\n\n[1] Mohamed, S., & Rezende, D. J. (2015). Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems (pp. 2125-2133).\n[2] Hansen, S., Dabney, W., Barreto, A., Van de Wiele, T., Warde-Farley, D., & Mnih, V. (2019). Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.05030.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D09ZBABPwyJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper studies reward-free reinforcement learning (RL) methods based on empowerment. The authors propose a technique for empowerment estimation under the assumption that a state after H timesteps can be factorized as a product of the current action and a matrix G(s) that depends on the current state. In contrast to the existing methods that rely on the optimization of variational lower bounds on mutual information for empowerment estimation, the technique allows having an almost closed-form expression for empowerment. The authors qualitatively demonstrate the convergence of their method to the true empowerment function on simple environments such as 2D ball-in-box as well on image-based Pendulum environment.\n\nStrengths\n- Under the proposed factorization, the empowerment estimation requires only calculating SVD of the G(s) matrix and line search for satisfying constraints.\n- The plots for 2D ball-in-box environment demonstrate that the proposed method estimates empowerment more accurately than other algorithms that have empowerment estimation as a component.\n\nWeaknesses\n- The experimental results could be demonstrated on harder environments such as MuJoCo. For example, DIAYN [1], one of the methods the authors compare with, provides results on Ant and Cheetah environments.\n- The proposed factorization, which is a key component for empowerment estimation, might be too restrictive and not scalable. Moreover, the reviewer did not find the details on how exactly G(s) is calculated except that it is the output of a neural network.\n- The positioning of the paper is unclear. If the main contribution is the improvement over existing reward-free RL methods, then it is more appropriate to demonstrate that the agent that uses estimated empowerment achieves high returns in the environment or has an interesting emergent behavior. If the main contribution is the improvement in empowerment estimation, it is more appropriate to compare with methods designed for mutual information estimation.\n- The overall clarity of the paper could be significantly improved.\n\nRecommendation\n\nThe reviewer votes for rejection. The methods the authors compare with are not designed for empowerment estimation, they use empowerment only as a proxy reward e.g. for overcoming exploration or learning skills. Moreover, it is unclear whether the method will scale beyond simple environments. Addressing the outlined weaknesses might increase the final score.\n\n**\n\nPost-rebuttal update: the score is increased from 3 to 5. See the response to authors' comments.\n\n**\n\nNotes:\n1. The connection between equations (1) and (3) which both define empowerment is unclear. \n2. It would be helpful to provide more experimental details. For example, the authors state that their method requires only training only two neural networks compared to three networks for variational lower bound methods. However, the benefit of this is unclear without providing the architectures of the neural networks and the total number of parameters.\n3. The code and additional materials were not available in the Google Drive folder indicated in Supplementary Material at the moment of submitting the review. However, this did not influence the final score.\n\n[1] Eysenbach, Benjamin, Abhishek Gupta, Julian Ibarz, and Sergey Levine. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper advocates empowerment for stabilising dynamical systems, the dynamics of which is estimated with Gaussian channels. Baseline comparisons have improved and that makes the experimental section good. While initial versions of the paper were problematic, all reviewer issues have been addressed and acceptance is almost unanimous.
Decision: Accept (Poster)