Dynamical System Embedding for Efficient Intrinsically Motivated Artificial Agents
Authors: Ruihan Zhao
Stas Tiomkin
Pieter Abbeel
Venue:  (2020)
Keywords: intrinsic motivation
empowerment
latent representation
encoder

Abstract: Mutual Information between agent Actions and environment States (MIAS) quantifies the influence of agent on its environment. Recently, it was found that intrinsic motivation in artificial agents emerges from the maximization of MIAS.
For example, empowerment is an information-theoretic approach to intrinsic motivation, which has been shown to solve a broad range of standard RL benchmark problems. The estimation of empowerment for arbitrary dynamics is a challenging problem because it relies on the estimation of MIAS. Existing approaches rely on sampling, which have formal limitations, requiring exponentially many samples. In this work, we develop a novel approach for the estimation of empowerment in unknown arbitrary dynamics from visual stimulus only, without sampling for the estimation of MIAS. The core idea is to represent the relation between action sequences and future states by a stochastic dynamical system in latent space, which admits an efficient estimation of MIAS by the ``Water-Filling" algorithm from information theory. We construct this embedding with deep neural networks trained on a novel objective function and demonstrate our approach by numerical simulations of non-linear continuous-time dynamical systems. We show that the designed embedding preserves information-theoretic properties of the original dynamics, and enables us to solve the standard AI benchmark problems.

Under review as a conference paper at ICLR 2020
DYNAMICAL SYSTEM EMBEDDING FOR EFFICIENT INTRINSICALLY MOTIVATED AGENTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Mutual Information between agent Actions and environment States (MIAS) quan- tifies the influence of agent on its environment. Recently, was found that the maximization of MIAS can be used as intrinsic motivation for artificial agents. In literature, the term empowerment is used represent the maximum of MIAS certain state. While empowerment has been shown to solve broad range of rein- forcement learning problems, its calculation in arbitrary dynamics is challenging problem because it relies the estimation of mutual information. Existing ap- proaches, which rely on sampling, are limited to low dimensional spaces, because high-confidence distribution-fee lower bounds for mutual information require ponential number of samples. In this work, we develop a novel approach for the estimation of empowerment in unknown dynamics from visual observation only, without the need to sample for MIAS. The core idea is to represent the relation between action sequences and future states using a stochastic dynamic model latent space with a specific form. This allows us to efficiently compute empow- erment with the "Water-Filling" algorithm from information theory. We construct this embedding with deep neural networks trained function. Our experimental results show that the designed embedding preserves informatiinn-heeoreiie properties of the original dynamics.
sophisticated objective
INTRODUCTION
Deep Reinforcement Learning (Deep RI.) provides solid framework for learning optimal policy given a reward function, which is provided to the agent by an external expert with specific domain knowledge. This dependency on the expert domain knowledge may restrict the applicability of Deep RL techniques because, in many domains, is hard to define an ultimate reward function.
On the other hand, intrinsically-motivaie edge but rather get a reward from interacting with the environment. This motivates study of antrinically-mootiaaeed motivated AI in general, and to develop efficient sarinsically-motivate methods in particular as an alternative and/or complementary approach to the standard reinforcement learning setting.
artificial agents do not require external domain knowl-
In recent works, different intrinsic motivation and unsupervised approaches were introduced (Klyu- bin et al. 2005 Klyubin al. 2005 Wissner-Gross & Freer 2013) Salge et al.) 2013b Pathak
et al.] 201 Warde-Farley et al. 2018). Among others, the empowerment method reviewed in Salge the diversity of future states bistinguisaably-achivaab by agent
intrinsic reward. By definition, empowerment of the agent at given state is the channel capacity between the agent's 's choice of action and its resulting next state. Previously, this information theeoreti ap- proach has proved to solve broad range of the AI tasks (Salge et al.) 2014 [0133] Tiomkin et al.) 2017 Klyubin et al., 2005). Despite the empirical success, the computational burden of the estima- tion of empowerment is significant, even for known dynamics and fully observable state spaces, as estimating mutual information between high-dimensional random variables is known to be a hard problem (McAllester & Statos 2018). Concretely, high-confidence distribution-free lower bounds for mutual information require exponential number of samples. The difficulty in computation has significantly limited the applicability of empowerment in real-life scenarios, ubiquitous in biology and engineering, where an agent observes the environment through sensors (e.g. vision) and does not know exact dynamical model of the environment.
al. (2014)

---

Under review as a conference paper at ICLR 2020
In this work, present novel approach for efficient estimation of empowerment from visual observations (images) in unknown dynamic environments by learning embedding for the dynamic system. The new approach allows to avoid computationally expensive sampling in high dimen- sional state/action spaces required for the estimation of MIAS. Efficient computation erment from images opens new directions in the study of intrinsic motivation in real-life environ- ments. As a generic scheme, we structure the interaction between the agent and the environment in perception-action cycle, shown in Figure
empow-
Channel capacity
by water water-Illlig
Controllar
Environment
Intrinsic rewards
Actions
Sensor Data
Embedding
Actuations
State
Action
Figure 1: Intrinsically Motivated Perception and Action Cycle. Left-to-right: (a) the observations with the corresponding action sequences are embedded the latent space by DNNs, where the final states are constrained to be linear in the action sequences (non-linear in the initial states). (b) Gaussian channel capacity between the embedded actions and the embedded final states is computed efficiently by the 'Water-filling' algorithm. (c) intrinsic reward is derived, triggering the agent to improve its policy for better reward. (d) when the agent operates under the new policy, additional observations from the environment are perceived by the sensors, closing the loop.
While our proposed scheme is broadly applicable, we demonstrate its usage in two different do- mains: intrinsically motivated stabilization of non-linear dynamical systems and intr deep reinforcement learning. First, the experiments help verify that the embedding correctly cap- tures the dynamics in the original space. Additionally, our approach for the estimation of MIAS in the embedded space provides qualitatively similar MIAS landscape compared to the corresponding one in the original space, which allows us to construct meaningful intrinsic reward for reinforcement learning. Our study of the inverted pendulum is especially important because this non-linear model a prototype for upright stabilization in various biological and engineering systems.
The paper is organized as follows: in Section 2 compare work with previous relevant works. Especially, table provides a thorough comparison between existing methods for the estimation of empowerment, and differentiates our work from the existing state of the art. In Section we provide the necessary formal background to present our approach. In Section 4 we present the method, with the implementation described in Section 5 Section 6 we demonstrate the proposed method by numerical simulations, and compare to the analytical solution derived in the previous work (Salge) al. 2013a). Finally, in Section 7 we conclude, and propose promising continuations this work.
RELATED WORKS
In this section, we compare our approach the existing approaches for the estimation of empower-
ment. The criteria for comparison are: (a) the knowledge of the dynamical model, (b) the ability to take high dimensional inputs, (e.g. images/video) and (c) the need for sampling to estimate MIAS. The criteria of our choice are relevant because (a) the true dynamic model is often inaccessible in real-life environments.
(b) our IMPAC assumes generic sensor reading, which can be images. (c) sampling of high-dimensional data is expensive and it is desired to have alternative approach for the estimation of MIAS.
One of the contributions of this work is efficient representation scheme, achieved by deep neural networks, which does not require sampling for the estimation of mutual information. This is the first work that makes it possible to estimate empowerment from images with unknown dynamics and without sampling.
As shown at Table the proposed method makes the least assumptions, (unknown dynamics, visual input, no sampling for the estimation of MIAS) compared to existing methods, which makes applicable broader class of problems.
2

---

Under review as a conference paper at ICLR 2020
Table Comparison between existing methods for the estimation of empowerment.
Method
Unknown dynamics
Visual input
Sampling
Salge et al.
2013a
n10
no no
Mohamed Rezende (0015)
yes
yes
yes
Gregor et al. 2016)
yes
yes
yes
Karl et al. 2017
yes
no
yes
l1omkın et al. (2017
no
Our work
yes
ycs
no
PRELIMINARIES
3.1 MARKOV DECISION PROCESS
A Markov decision process (MDP) is discrete-time control model with parameters S: the state space, A: the action space, p(s' 8, a): the transition probability model, r(s,a) R: the reward function, Po: the initial state distribution, H: the horizon in number of steps and y the reward discount factor.
In reinforcement learning, we try to find the optimal policy that maximizes the expected sum of returns along the trajectory:
H
max
Σ t=
r(s (s) (T)
3.2 EMPOWERMENT
In this work, we use the established definition of empowerment a function of state (Klyubin et al. 2005). The empowerment at state is the channel capacity between the action sequence (at, 1e.1....,, 11) 1)
and the final state 8T.
Emp(s;) max I(a ; 88 (1))
k-1
max w(a R-1 (ss)
H(s 8x) (sx lat aat-1,,a s1)]
w(a [st)
where I(ss; de st) is the mutual information functional, and (a k-1 (()) is stochastic policy, described by probability distribution of action trajectories conditioned on 8t.
PROPOSED APPROACH
The key idea of our approach is to represent the relation between action sequences and corresponding final states by a linear function in the latent space and to construct a Gaussian linear channel in this space. Importantly, even though the relation between action sequences and future states is linear, the dependency of future states on initial states is non-linear. This representation of non-linear dynamical system enables to compute empowerment by an efficient "water-filling" algorithm in the latent space, as explained in Section 4.4
4.1 INTERACTION MODEI.
We formulate the interaction between the agent and the environment as an MDP (Section 3.1). Each step taken in the environment can be written as:
$1+1 = f(st, at, 7)),
(1)
where is the state at time t, ae is the action at time and m) is assume to be Gaussian process noise. The agent does not know the dynamical model f. but rather it obtains through its visual sensors, an image observation i of the current state 8t. Then, after the agent applies action, at,
3

---

Under review as a conference paper at ICLR 2020
the system moves to the next state, 81+1- The agent has access to all previous observation and action trajectories, [Tr)==n 7; = {1) at 4 where the number of trajectories and T is the length of each trajectory.
4.2 EMBEDDED SPACES
From the collected data, ] we embed tuples in the form of (V,, a; a v/++)) where of (a,, Os+1.-.. -++k-1). Observations and actions are embedded by deep neural networks to the latent spaces Z E d. and B E IRd,, respectively. Here d2 is the dimension of the embedded state, and d, is the dimension of the embedded action sequence. These two spaces are related by linear equation:
24+ A (2t
where 2y is the embedded representation of the image V4- A(ze) is d by d, state dependent matrix that relates the embedded action sequence, bi+h-1 to the future embedded state, žr- This model was suggested previously by Salge al. (2013a) in the original state space with known dynamics.
Our key contribution of this work is the embedding of generic unknown dynamic system into latent space that satisfy this model. The architecture for the embedding and the objective functions used to train the networks are provided in Section 5
4.3 INFORMATION CHANNEL IN EMBEDDED SPACE
To compute mutual information between embedded action sequences, bf bi+ and embedded states, 24+k+ assume that the noise in the system comes from Gaussian in latent state space:
N NOa xd,, Id, xd,). ) As a result, we end up with an Gaussian linear channel in embedded spaces:
2 A( k-1 +1
(3)
This particular formulation of dynamic systems in the latent space allows us to solve for the channel capacity directly and efficiently, as described in below.
4.4 CHANNEL CAPACITY IN LATENT SPACE
To compute the information capacity of the channel given in Eq. 4.3 we apply the "Water-Filling" algorithm (Cover & Thomas 2012), which is computationally efficient method for the estimation of capacity, C", of linear Gaussian channel:
C* =max Pt 2
Σ log(1 + (z,))))
under the constraint on the total power, Σi= Pt where {2)(2)) are the singular values of the channel matrix, A(z,), and P is the total power in the latent space.
5 ARCHITECTURE
The most crucial component of the IMPAC scheme is the embedding from sensor inputs (in our case, images) and action sequences, into latent state and action tensors while remaining consistent with the real environment dynamics. In this section, we organize our discussion around how single sample: {St, aq. G S+++} fits in the training of our deep neural networks, as illustrated in Figure 2 We will explain each individual components of the block diagram and wrap up by discussing the whole picture.
EMBEDDING OF IMAGE OBSERVATIONS
The left side and right side of the block diagram in Figure eembeds raw observations (images). In this work, we take images scaled down to 64 x 64 as observations. The image resolution is chosen balance of clarity and performance: visual features of the environments retained
4

---

Under review as a conference paper at ICLR 2020
2
Dynamics (MLP)
1064
x 1024
2
Z
Encoder (corv)
Decoder (decon
Encoder (MLP)
Decoder (MLP)
Encoder (COnN)
R2 R6-1
Rea1
å
S
64
64
Py-T-1
Figure 2: An illustration of data flow around single training sample.
while training batches be fit single GPU. Multiple images be concatenated capture additional information (e.g. velocity). When encoding multi-channel images into latent vectors, use an auto-encoder and decoder pair parameterized by convolutional neural networks. Given a training sample {St, a,, a1+k-1, ). both S, and S+++ are encoded into their respective latent vectors Z, and Ze- using a same CNN encoder. To ensure that essential state information is conserved after embedding, we decoded the latent tensor Ze back to images using deconvolutional network for correct reconstruction St The objective is to make the images S, and S, similar. The encoder and decoder network details are presented Appendix B
5.2 EMBEDDING OF ACTION SEQURNCES
The mid-lower section of Figure emmeed action sequences.
Standard MDPs (Section models environment dynamies in the granularity of single steps, which results in an exponentially large action space over long horizons. Another contribution of this work is that compute information-theoretiic quantities across multiple time-steps without exploding the action space. We consider k--tep dynamic model directly by embedding action sequence (a,, 4++1 0++ } into single action by in the latent space (1 step in the latent space corresponds to k steps in the original space). Actions inside the sequence are concatenated into a vector and passed into an auto-encoder get b, Again, to ensure that necessary information retained in the latent action tensor, we decode b, for reconstruction of {a,, a++1 0 In our implementation, the encoder and decoder are both fully connected MLPs. The detailed information about each layer can be found in Appendix B
5.3 DYNAMICAL MODEL LATENT SPACE
The top part of Figure fits dynamics model linear in action (non-linear in state).
As discussed Section our scheme requires a dynamic model with a particular form. Specif- ically, given current latent state Z,, the next state Ze+ is linear in the action by Mathematically, latent states have dimension d, and latent action have dimension dg, we want to find mapping : d + Rd xd, such that Ze+ Re A(Z,) x b.
The mapping from Z, to an d₂ by matrix is parameterized by fully connected MLP with d2 da output neurons. The final layer is then reshaped into a matrix. (See appendix B)
5.4 THE ENTIRE SCHEME
In the previous 3 sub-sections, we introduce 3 objectives: 1) Find an embedding of observations that retains their information. 2) Find embedding of action sequences that retain their information.. 3)
5

---

Under review as a conference paper at ICLR 2020
Find dynamics model linear in the latent action. When these three objectives trained jointly, the neural networks learn towards a latent space that extract information from the original space and learns dynamics model that is consistent with the actual environment.
L = adE [(s - S + 3E((a: a.)²] + [Z )
(5)
In practice, regularization on the latent tensors is enforced at training time to encourage a structured latent space. The details about our choice of regularization are discussed in appendix
EXPERIMENTS
6.1 INVERTED PENDULUM
Inverted pendulum a prototypical system for upright bi-pedal walking, and it often serves to check new AI algorithms. Moreover, inverted pendulum is baseline for the estimation of empowerment in dynamical systems because its landscape is well-studied in numerous previous studies (Salge
al., 2013b 2014] 2013a Karl al. 2017] Mohamed & Rezende 2015). So, we chose inverted pendulum from the Openai Gym environment for testing our new method, and comparing it to the previous methods (Brockman al. 2016).
6.1.1 CORRECTNESS OF LATENT DYNAMICS
Prior to applying the proposed model, given by Eq. to the estimation of MIAS in the latent space, we verify the correctness of our embedded dynamic model. We compare reconstruction of latent prediction with ground truth observation. In inverted pendulum, two consecutive images are used to capture the state information (angle and angular velocity). When we supply the initial two images for time along with choices of different action sequences of length k; expect the reconstructions to match the corresponding observations at time t + k. As shown in Figure 3 the model is capable of reconstructing the future images from the current images and action sequences.
Action Sequences
Latent
Ground Truth
Prediction
[-1, -1,11 -1,
0.
1, 1, 1]
Figure 3: Reconstruction through the latent dynamics in comparison with the ground truth. Starting from the same upright position, a different action sequence is taken in each row. In all 3 cases, reconstruction matches the actual observation after the actions are taken.
6.2.2 EMPOWERMENT LANDSCAPE AND PENDULUM SWING-UP
In this section, compute the empowerment values and use those as intrinsic reward to train an optimal trajectory for swinging up the pendulum from the bottom position to the upright position. The optimal trajectory is computed by the standard PPO algorithm with reward signal coming solely from empowerment. (Schulman et al.) 2017b). Figure compares the empowerment landscape computed under approach with previous result:
As shown in Figure 4 at the beginning, the pendulum swings with a small amplitude (the dense black dots around = π rad, 0 = Orads-)). Once it accumulated enough energy. the pendulum starts traversing the state space, and arrives at the top position, where it stabilises, (dense black points around 0 rad, i 0 -1).

---

Under review as a conference paper at ICLR 2020
Optimal trajectory super-imposed on our empowerment landscape. The pendulum swings up from bottom (T, 0) to balance at the top (0, 0). Right: Empowerment values calculated with knowledge of the ground truth model (Salge et al. 2013a). This figure shows that the empow- erment landscapes are qualitatively similar, which verifies the validity of our approach.
Figure 4: Left:
Results from the pendulum experiment effectively demonstrate the validity of our proposed ap- proach. Not only was our method able to create an empowerment value plot similar to that of previous analytical method, but also these empowerment values train to balance the pendulum at upright position. The pendulum experiment shows that our latent representation effectively captures the information-thcoree properties the original environment.
6.2 SAFETY OF RL AGENT
Another useful application of empowerment is its implication of safety of the artificial agent. A state is intrinsically safe for an agent when the agent has a high diversity of future states, achiev- able by its actions. This is because such states, the agent can take effective actions prevent undesirable futures. In this context, the higher its empowerment value, the safer the agent is. In this experiment, we first check that our calculation of empowerment matches the specific design of environment. Additionally, we show that empowerment augmented reward function can affect the agent's preference between a shorter but more dangerous path and a longer but safer one.
double tunnel environment implemented with to the OpenAl Gym API (Brockman ctt al. 2016). Agent (bluc) is modeled as ball of radius inside a 20x20 box. The box is separated by thick wall (gray) into top and bottom section. Two tunnels connect the top and bottom of the box. The tunnel in middle is narrower but closer to the goal compared to the one on the right.
Environment:
Control: In each time step, the agent move most 0.5 unit length in each of and direction. an action takes the agent into the walls, will be shifted back out to the nearest surface.
Reset criterion: each episode has maximum length of 200 steps. The environment resets when time runs out or when the agent reaches the goal.
Since the tunnel in the middle is narrower, the agent is relatively less safe there. The effectiveness of the control of the agent is damped in ways:
1. In particular time step, it's more likely for the agent to bump into the walls. When this happens, the agent is unable to proceed as far as desired.
2. The tunnel is 10 units in length. When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon. Thus, the agent has fewer possible future states.
We trained the agent with PPO algorithm (Schulman et al. 201 7a) from OpenAl baselines (Dhariwal al. 2017) using an empowerment augmented reward function. After a parameter search, we used discount factor 0.95 over a total of 10t steps. The reward function that we choose is:
R(s, a) goat + 3 Emp((s
7

---

Under review as a conference paper at ICLR 2020
16.0
15
50
2.5
65
-25
-.0
155
-00.
5.0
-000
-7.5
-5.0
-2.
3.0
2.5
5.0
7.5
30.0
Figure 5: Left sub-figure: Double tunnel environment. The goal is marked in red. The agent in blue. Right sub-figure: Empowerment landscape for the tunnel environment. The values of em- powerment reduce the corner and inside the tunnel where the control of the agent is less effective compared to more open locations.
where 3 balances the relative weight between the goal conditioned reward and the intrinsic safety reward. With high 8, expect the agent to learn conservative policy.
(a) 9 = 0
(b) B = 1/1600
(c) 8 = 1/800
Figure 6: Trajectories of trained policy. As B increase, the agent develops stronger preference a safer route, sacrificing hitting time.
The results from the tunnel environment again support our proposed scheme. First, the empower- ment landscape matches our design of the environment. Second, high quality empowerment reward successfully alters the behavior of the agent.
DISCUSSION
Intrinsically motivated artificial agents do not rely on external reward and thus, do not need domain knowledge for solving broad class of AI problem such as stabilization, tracking, etc. In this work, we introduce a new method for efficient estimation of certain type of information-then theoortti intrinsic motivation, known empowerment. The learnt embedding representation reliably captures not only the dynamic system but also its underlying information-theoreii properties.
With this work, also introduced the IMPAC framework, which allows for systematic study of intrinsically motivated agents broader scenarios, including real-world physical environments like robotics. Future works on physical environment will better demonstrate the usefulness and general applicability of empowerment as an intrinsic motivation.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Thomas M Cover and Joy Thomas. Elements of information theory. John Wiley & Sons, 2012.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. 2017.
https:
//github. htt.taub...comopeaaaaaasseeeeee
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Maximilian Karl, Maximilian Soelch, Philip Becker-Ehmck, Djalel Benbouzid, Patrick van der Smagt, and Justin Bayer. Unsupervised real-time control through variational empowerment, 2017.
A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135 Vol.1, Sep. 2005. doi: 10.1 11110//E...00.555444
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. All else being equal be empow- ered. In European Conference on Artificial Life, pp. 744-753. Springer, 2005.
David McAllester and Karl Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.042555 2018.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125 2133,,133,20 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- exploration by sulf-super prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 488 489, 2017.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Approximation of empowerment in the continuous domain. Advances Complex Systems, 16(02m03):125007 2013a.
Christoph Salge, Comelius Glackin, and Daniel Polani. Empowerment and state-dependent noise-an intrinsic motivation for avoiding unpredictable agents. In Artificial Life Conference Proceedings 13, pp. 118-125. MIT Press, 2013b.
Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment-aa introduction. In Guided Self-Orgunization: Inception, pp. 67-114. Springer, 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017b.
Stas Tiomkin, Daniel Polani, and Naftali Tishby. Control capacity of partially observable dynamic systems in continuous time. arXiv preprint arXiv: 2017.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non parameert discriminative rewards. CoRR, abs/1811.11359, 2018. URL http::ttt//aattti wrg./rrg//ooooo abs////b/iiae abs.//11...135...1111111.1111111
Alexander D Wissner-Gross Gross and Cameron E Freer. Causal entropic forces. Physical review letters, 110(16):16802,, 2013.

---

Under review as a conference paper at ICLR 2020
A DETAILS OF TRAINING ALGORITHM
Collect trajectories using random policy and random reset.
2. Separate out tuples (St, (t, dt+1,".. 0 1, St+R) from trajectories. For simplicity of notation, let a represent the concatenated action sequences ae, 0c+1,--. 0t+k-1
3. Use randomly initialized encoder networks to map the observations and action sequence into latent space. This gives us tuples (Z,, bf. Z(+k).
4. Use randomly initialized MLP network to get the corresponding transformation matrices A(Z)).
5. Calculate the predicted next state in latent space 21+ - A(Z
6. Use randomly initialize decoder networks reconstruct images and action sequences from latent vectors.
S, Dec(
at Dec(b))
S1+* = Dea(Zi++) Note that S1++ reconstructed from latent prediction.
7. Calculate the following loss terms:
(a) Observation reconstruction error: Lobe ||S So
(b) Action sequence reconstruction error: Lrccion = ||a
0 2
(C) Prediction error in latent space: Ltatent 11 22+1
(d) Prediction error in original space: Lorg = ||Sr+* - Sct111
8. In additional to the loss terms, we add regularization terms to prevent latent vectors from shrinking. This help us get consistent and comparable empowerment values across different trials and even different environments.
(a) Regularization latent state: Reg, = 12,
2,
|6?
1
(b) Regularization of latent action: Rege [1
de
9. Finally, we use batched gradient descent on the overall loss function train all the neural networks in the same loop.
Okobs Lobs + Qcection Laction + Slatent + Corg Lorg + areg (Reg,
areg(Reg, + +egg))
For both of experiments, chose
Globs Qorg 100
Qaction 10
Ollatent O'regg
and were able to produce desired empowerment plots.
B DETAILS ON NEURAL NETWORK LAYERS
B. CONVOLUTIONAL NET FOR IMAGE ENCODING
(h1) 2D convolution: 4 filters, stride 2, 32 channels, ReLU activation
(h2) convolution:
filters, stride 2, 64 channels, ReLU activation
(h3) 2D convolution: 4 filters, stride 2, 128 channel, ReLU activations
(h4) 2D convolution: 4 filters, stride 2, 256 channel, ReLU activations
(out) Flatten each sample to ID tensor of length 1024
B.2 DECONVOLUTIONAL. NET FOR IMAGE RECONSTRUCTION
(hl) Fully connected layer with 1024 neurons, ReLU activation
(hl') Reshape to x 1 images with 1024 channels
(h2) 2D conv-transpose: filters, stride 2, 128 channels, ReLU activation
(h3) 2D convolution: 5 filters, stride 2, 64 channels, ReLU activation
(h4) 2D convolution:
filters, stride 2, 32 channel, ReLU activations
(out) 2D convolution: 6 filters, stride 2, C' channel
10

---

Under review as a conference paper at ICLR 2020
B.3 MLP FOR ACTION SEQUENCE ENCODING
(h1) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(h3) Fully connected layer with 512 neurons, ReLU activation
(out) Fully conected layer with 32 output neurons
B.4 MLP FOR ACTION SEQUENCE RECONSTRUCTION
(h1) Fully connected layer with 512 neurons, ReLU activation
(h2) Fully connected layer with 512 neurons, ReLU activation
(out) Fully conected layer with & x da neurons, tanh activation then scaled to action space
B.5
MLP FOR TRANSITION MATRIX A
(h1) Fully connected layer with 1024 neurons, ReLU activation
(h2) Fully connected layer with 4096 neurons, ReLU activation
(h2) Fully connected layer with 8192 neurons, ReL.U activation
(out) Fully conected layer with dg dp (1024 32) neurons


Reviews:
[{"review_id": "SJlRd9oGYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a model to embed states and action sequences into latent spaces in order to enable efficient estimation of empowerment in a reinforcement learning system. \n\nThe paper shows some interesting experimental results. But overall, it is not ready to publish yet for the following reasons: \n\n1. The technical section is lack of important description/theorems/derivation/etc that are necessary to support the claims.  \n\nWhat is the detailed definition of empowerment, i.e., how to spell out the formula of mutual information I? What is the distribution of action sequences and states? What is the policy function \\pi? How is \\pi related to or different from \\omega? How is the learned empowerment used in (training) policy? \n\nThe authors proposed a parametrization in equation-3 for state transition and claimed that this parametrization yields ``an efficient estimation of MIAS as explained in the next section’’, but did not give any explanation throughout the paper. Does it mean the equation-4 is easy to solve? Does equation-4 end up that way because of the parametrization in equation-3? Why so? \n\nWhat is the water-filling algorithm? How does it associate the capacity with the empowerment? Elaboration is needed for this bit. \n\nThe authors claim that putting more weight on empowerment in reward ends up in a more conservative policy, but didn’t give any technical justification. They indeed mention that ``a state is intrinsically safe for an agent when the agent has a high diversity of future states’’ and that ``the higher its empowerment value, the safer the agent is’’. The former is a hypothesis and the latter needs technical/derivation support---e.g., why empowerment is correlated with the diversity? The interesting experimental results in figure-6 seem to support the authors’ claim, but precise technical justification is needed. \n\n2. Some technical description/argument should be more precise and accurate. \n\nThe authors claim that they ``observes the current state through its visual sensors’’---but the actual state (i.e. the exact angle and height) can’t be observed and the visual sensor data is only an approximation, so the correct claim should be something like: we observe the visual representation of the actual state. \n\nThe authors claim that the ``existence of such representation ... is one of the contributions of our work’’---the existence of something shouldn’t be a contribution of a technical paper, what can be a contribution is the proof of its existence. \n\nThe authors claim that they ``inject Gaussian noise … into the latent space.’’ This is very confusing: it sounds like (1) there wasn’t any randomness in this formulation; (2) the estimation of empowerment is difficult because of that and (3) the authors added the Gaussian noise to enable the efficient estimation. However, based on my understanding after reading multiple times, I guess what actually happened is: (1) there should be randomness and the noise can be anyway distributed and (2) the authors assumed it is Gaussian so it is simple enough to yield an efficient estimation. The authors should really clarify this. \n\n3. There are also some typos that may confuse readers. \n\nThe authors mentioned that ``a linear Gaussian channel given by Eq. 4.3’’---is it section-4.3 or equation-3?\n\nIn appendix, what is d_z and d_b? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xfo2e9tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to take advantage of a known result on the channel capacity of a linear-Gaussian channel in order to estimate the empowerment and maximize mutual information between  policies (action sequences) and final states (given  initial states). The idea is to map the raw action sequences and states to a latent space where learning would force that linear property to be appropriate.\n\nI like the general idea of the paper (as stated above) along with its objectives but I have several concerns.\n\nFirst, I need to be reassured that we are computing the right quantity. Channel capacity is the maximum mutual information (between inputs and outputs) over the input distribution, whereas I had the impression that empowerment would be this mutual information, and that we want to increase it, but not necessarily reach its maximum over all possible policies: it would usually be one of the terms in an objective function (e.g. here we have reconstruction error, and in practice there would be some task to solve in addition to the exploration reward). One way to see this problem in the given formulation is that the C* objective only depends on the matrix A (which encapsulates the conditional density of z_{t+1} given the trajectory) and it does not depend at all on the distribution of the trajectory itself! This is weird since if we are going to use this as reward the objective is to improve the trajectory. What's the catch? So either I misunderstand something (which is quite possible) or there is something seriously wrong here.\n\nI am assuming that the training objective for the encoder is a sum of the reconstruction error and of C*. But note how this does not give a reward for policies, as such. This is a bit strange if the goal is to construct an exploratory reward!\n\nA less radical comment is: have you verified that the linear relationship between z_{t+k} and the b actually holds well? In other words, is the encoder able to map the raw state and actions to a space where the linearity assumption  is correct, and thus where equation (3) is satisfied.\n\nFigure 3 has something weird,  probably one of the two sequences of -1's should be a sequence of +1's.\n\nFigure 4 is difficult to interpret, the caption should do a better job.\n\nThe experiment on the safety of the RL agent is weak. I don't see the longer path as safer, here. And the results are not very impressive, since the agent is only doing what it's told, i.e., go in areas with more options (more open areas) but there is no reason to believe that this is advantageous, here.\n\nFinally, what would make this paper much more appealing is if the whole setup led to learning better high-level representations (how to measure that is another question, but it is a standard kind of question in representation learning papers).\n\nRelated work:\n\nI don't understand why in the abstract the authors refer to sampling-based methods as requiring exponentially many samples. This is not generally the case for sampling based methods (e.g. think of VAEs). I suppose the authors refer to something in particular but it was not clear to me what.\n\nReferences: in the intro, you might want to refer to the contrastive methods and variational methods to maximize mutual information between representations of the state and representations of the actions, e.g.,\n Thomas et al, 2018, 1802.09484\n Kim et al, 2018, arXiv:1810.01176\n Warde-Farley et al, 2018, arXiv:1811.11359\n \n\nEvaluation: for now I suggest a weak reject but I am ready to modify my score if I am convinced that my main concerns were unfounded.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJeX20UYqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach based on empowerment for reinforcement learning applicable to the cases that the dynamical system is unknown. The model is estimated by a water filling algorithm and is evaluated on two RL tasks. \n\nTraining of RL agents via on empowerment and intrinsic rewards is an important alternative to conventional training algorithms. The paper is tacking an important paper. \nThe paper is weak in terms of writing and motivation. Empowerment on section 3.2 could have been explained more intuitive and more thoroughly the make the paper self-contained.\nMoreover, the paper lacks motivation of the design choices. It seems to be a combination of a few recent techniques in machine learning or statistics that are mechanically attached to each other without sufficient justification or intuition. \nThe paper keeps claiming to solve AI however what it actually experimented on RL safety and/or one synthetic environment. They are not really \"AI benchmark problems\". I'd rather the paper focuses on its contribution: direct and concise. \nMoreover, the experiments are not sufficient to support the paper or investigate how much each part is contributing to the success. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel method for embedding sequences of states and actions into a latent representation that enables efficient estimation of empowerment for an RL system. They use empowerment as intrinsic reward for safe exploration. While the reviewers agree that this paper has promise, they also agree that it is not quite ready for publication in its current state. In particular, the paper is lacking a theoretical justification for the proposed approach, the definition of empowerment used by the authors raised questions, and the manuscript would benefit from more clear and detailed description of the method. For these reasons I recommend rejection.
Decision: Reject