Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
Authors: Denis Yarats
Amy Zhang
Ilya Kostrikov
Brandon Amos
Joelle Pineau
Rob Fergus
Venue:  (2020)
Keywords: reinforcement learning
model-free
off-policy
image-based reinforcement learning
continuous control

Abstract: Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. The agent needs to learn  a latent representation together with a control policy to perform the task. Fitting a high-capacity encoder using a scarce reward signal is not only extremely sample inefficient, but also prone to suboptimal convergence. Two ways to improve sample efficiency are to learn a good feature representation and use off-policy algorithms. We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based RL. Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-art performance across both model-free and model-based algorithms on many challenging control tasks. We release our code to encourage future research on image-based RL.

Under review as a conference paper at ICLR 2020
IMPROVING SAMPLE EFFICIENCY IN MODEL-FREE REINFORCEMENT LEARNING FROM IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Training an agent to solve control tasks directly from sish-dmmensional images with model-free reinforcement learning (RL) has proven difficult. The agent needs to learn latent representation together with a control policy to perform the task. Fitting a high-capacity encoder using a scarce reward signal is not only sample in- efficient, but also prone to suboptimal convergence. Two ways to improve sample efficiency are to extract relevant features for the task and use off-policy algorithms. We dissect various approaches of learning good latent features, and conclude that the image reconstruction loss is the essential ingredient that enables efficient and stable representation learning in image-based based RL. Following these findings, we devise an off-policy actor-critic algorithm with an auxiliary decoder that trains end-to-end and matches state-of-the-e performance across both model-free and model- based algorithms on many challenging control tasks. We release our code to encourage future research on image-basee RL'.
INTRODUCTION
Cameras are a convenient and inexpensive way acquire state information, especially in complex, unstructured environments, where effective control requires access to the proprioceptive state of the underlying dynamics. Thus, having effective RI. approaches that can utilize pixels as input would potentially enable solutions for a wide range of real world problems.
The challenge to efficiently learn mapping from pixels to an appropriate representation for con- trol using only a sparse reward signal. Although deep convolutional encoders can learn good repre- sentations (upon which a policy can be trained), they require large amounts training data. As exist- ing reinforcement learning approaches already have poor sample complexity, this makes direct use of pixel-based inputs prohibitively slow. For example, model-free methods on Atari (Bellemare et al., 2013) and DeepMind Control (DMC) (Tassa et al., 2018) take tens of millions of steps (Mnih al., 2013; Barth-Maron et al., 2018), 8), which is impractical in many applications, especially robotics.
A natural solution is add auxiliary task with an unsupervised objective to improve sample effi- ciency. The simplest option is an autoencoder with a pixel reconstruction objective. Prior work has attempted learn state representations from pixels with autoencoders, utilizing a two-step training procedure, where the representation is first trained via the autoencoder, and then either with a policy learned on top of the fixed representation (Lange & Riedmiller, 2010; Munk et al., 2016; Higgins et al., 20;7b; Zhang et al., 2018; Nair et al., 2018), or with planning (Mattner et al., 2012; Finn al., 2015). This allows for additional stability in optimization by circumventing dueling training objectives but leads to suboptimal policies. Other work utilizes end-to-end model-free learning with an auxiliary reconstruction signal in an on-policy manner (Jaderberg et al., 2017).
We revisit the concept of adding an autoencoder to model-free RL approaches, but with focus on off-policy algorithms. We perform sequence careful experiments understand why previous approaches did not work well. We found that pixel reconstruction loss is vital for learning good representation, specifically when trained end-to-end. Based on these findings, we propose a simple autoencoder-bact based off off-policy method that can be trained end-to-end. Our method is the first model- free off- off-polii ooflcol algorithm to successfully train simultaneously both the latent state representation and policy in stable and sample-efficient manner.
An anonymous website with code, results, and videos: http:///iieeggooooccoovvvv/eeeee

---

Under review as a conference paper at ICLR 2020
Figure I: Image-based continuous control tasks from the DeepMind control suite (Tassa al., 2018) used in our experiments. Each task offers an unique set of challenges, including complex dynamics, sparse rewards, hard exploration, and more. Refer to Appendix for more information.
Of course, some recent state-of-the-arr model-based RL methods (Hafner al., 2018; Lee et al., 2019) have demonstrated superior sample efficiency to leading model-free approaches pixel tasks from (Tassa et al., 2018). 8). But we find that our model-free, off off-policy, autoencoder based approach is able to match their performance, closing the gap between model-based and model-free approaches in image-based RI, despite being a far simpler method that does not require a world model.
This paper makes three main contributions: (i) demonstration that adding simple auxiliary recon- struction loss to a model-free off-policy RL algorithm achieves comparable results to state-of-the-ar model-based methods the suite of continuous control tasks from Tassa et al. (2018) 8); (ii) an under- standing of the issues involved with combining autoencoders with model-free RL in the off-policy setting that guides our algorithm; and (iii) an open-source PyTorch implementation of our simple method for researchers and practitioners to use as a strong baseline that may easily be built upon.
RELATED WORK
Efficient learning from nigh-dimensional pixel observations has been a problem of paramount im- portance for model-free RL. While some impressive progress has been made applying model-free RL to domains with simple dynamics and discrete action spaces (Mnih et al., 2013), attempts to scale these approaches complex continuous control environments have largely been unsuccessful, both in simulation and the real world. A glaring issue is that the RI. signal is much sparser than in supervised learning, which leads sample inefficiency, and higher dimensional observation spaces such as pixels worsens this problem.
One approach to alleviate this problem by training with auxiliary losses. Early work (Lange & Riedmiller, 2010) explores using deep autoencoders to learn feature spaces in visual reinforcement learning, crucially Lange & Riedmiller (2010) propose to recompute features for all collected expe- riences after each update of the autoencoder, rendering this approach impractical to scale to more complicated domains. Moreover, this method has been only demonstrated on toy problems. Alter- natively, Finn et al. (2015) apply deep autoencoder pretraining to real world robots that does not require iterative re-training, improving upon computational complexity of earlier methods. How- ever, in this work the linear policy is trained separately from the autoencoder, which we find to not perform as well as end-to-end methods.
Shelhamer et al. (2016) use auxiliary losses in Atari that incorporate forward and inverse dynamics with A3C, an on-policy algorithm. They recommend a multi-task setting and learning dynamics and reward to find good representation, which relies the assumption that the dynamics in the task are easy to learn and useful for learning good policy. Jaderberg et al. (2017) propose to use unsupervised auxiliary tasks, both observation-based and reward-based based off of real world in- ductive priors, and show improvements in Atari, again in the on-policy regime, which is much more stable for learning. Unfortunately, this work also relies on inductive biases by designing internal rewards to learn good representation which is hard to scale to the real world problems. Higgins al. (2017b); Nair et (2018) use beta variational autoencoder (3-VAE) (Kingma & Welling, 2013; Higgins et al., 201 7a) and attempt to extend unsupervised representation pretraining to the off-policy setting, but find hard to perform end-to-end training, thus receding to the iterative re- training procedure (Lange & Riedmiller, 2010; Finn al., 2015).
There has been more success in using model-based methods on images, such as Hafner et al. (2018); Lee et al. (2019). These methods use world model (Ha & Schmidhuber, 2018) approach, learning a representation space using a latent dynamics loss and pixel decoder loss to ground on the original observation space. These model-based reinforcement learning methods often show improved sam- ple efficiency, but with the additional complexity of balancing various auxiliary losses, such as a dynamics loss, reward loss, and decoder loss in addition to the original policy and value optimiza-
2

---

Under review as a conference paper at ICLR 2020
tions. These proposed methods correspondingly brittle to hyperparameter settings, and difficult to reproduce, as they balance multiple training objectives.
To close the gap between model-based and model-free image- based RL in terms of sample efficiency and sidestep the issues model learning, our goal is to train model-free off-policy algorithm with auxiliary reconstruction loss in stable manner.
BACKGROUND
A fully observable Markov decision process (MDP) is described by tuple (S,A,, R, 7), where S is the state space, A is the action space, P(s St, ar) is the probability distribution over transitions, R(s ac, St-1) is the reward function, and y is the discount factor (Bellman, 1957). An agent starts in initial state s sampled from a fixed distribution p(s1), then at each timestep it takes an action a, A from state S, E S and moves receives a reward rt = R(s,, a,, 8t+1). We consider episodic environments with the length fixed to T. The goal of standard RL is to learn policy : =aar]s,) that maximize the agent's expected cumulative reward Σ'=1 a.)pp..)>pp [rx], where Prr state-action marginal distribution induced by the policy π (a (8c) and transition distribution P(s++1 |St, a.). An important modification (Ziebart et al., 2008) auguments this objective with an entropy term H-(s(-)s)) to encourage exploration and robustness to noise. The resulting maximum entropy objective is then defined as:
state 8 P(-|st, at). After each action the agent
arg max Σ Ess.aii [r. (F((s(():))).
where is temperature parameter that balances between optimizing for the reward and for the stochasticity of the policy.
We build on Soft Actor-Crite (SAC) (Haarnoja et al., 2018), an off-policy actor-critic method that
uses the maximum entropy framework derive soft policy iteration. At each iteration SAC performs a soft policy evaluation step and a soft policy improvement step. The soft policy evaluation step fits a parametric soft Q-function ((s,, a.) (critic) by minimizing the soft Bellman residual:
(Q) = E (( ( Q(5: ar) - --Ba [Q(5:41 ac+1) - a log T (a 1 (ac+ [+++
where D is the replay buffer, and Q is the target soft Q-function parametrized by a weight vector obtained using the exponentially moving average of the soft Q-function weights to stabilize training. The soft policy improvement step then attempts to learn a parametric policy T(ay (actor) by directly minimizing the KL divergence between the policy and Boltzmann distribution induced by the current soft Q-function, producing the following objective:
( EaDDDD Ear (a log((T(a)ss)) - Q(s,, a:)]
The policy π (a.(s)) is parametrized as diagonal Gaussian to handle continuous action spaces.
When learning from raw images, we deal with the problem of partial observability, which is formal- ized by partially observable MDP (POMDP). In this setting, instead of getting low-dimensional state S; E S at time f, the agent receives high-dimennsooal observation o, E O, which is a render- ing of potentially incomplete view of the corresponding state S, of the environment (Kaelbling et al., 1998). This complicates applying RL as the agent now needs to also learn a compact latent represen- tation to infer the state. Fitting& nigh-capacity encoder using only scarce reward signal is sample inefficient and prone to suboptimal convergence. Following prior work (Lange & Riedmiller, 2010: Finn al., 2015) we explore unsupervised pretraining via an image-based autoencoder. In practice, the autoencoder is represented as a convolutional encoder fene that maps an image observation o, to a low-dimensional latent vector 2,, and a deconvolutional decoder fiee that reconstructs z, back to the original image o, The optimization is done by minimizing the standard reconstruction objective:
J (AE)) = Eo,-D Ilao (Zx) o, where = =enc(O:))
(3)
3

---

Under review as a conference paper at ICLR 2020
Or in the case of 3-VAE (Kingma & Welling, 2013; Higgins et al., 2017a), where the variational distribution is parametrized as diagonal Gaussian, the objective is defined as:
J(VAE) -D (fll - 0g112 2 ( o.. - -
(4)
where 2, = (o.) and of fenc The latent vector 2, is then used by an RL algo- rithm, such as SAC, instead of the unavailable true state s,, To infer temporal statistics, such as velocity and acceleration, it is common practice to stack three consecutive frames to form sin- gle observation (Mnih et al., 2013). We emphasize that in contrast to model-based methods (Ha & Schmidhuber, 8; Hafner et al., 2018), we do not predict future states and solely focus learning representations from the current observation to stay model-free.
4 A DISSECTION OF LEARNING STATE REPRESENTATIONS WITH 3- 3-VAE
In this section we explore in systematic fashion how model-free off-policy RI. can be made to train directly from pixel observations. We start by noting dramatic performance drop when SAC is trained on pixels instead of proprioceptive state (Section 4.2) in the off- off-policy regime. This result motivates us to explore different ways of employing auxiliary supervision to speed up representation learning. While wide range of auxiliary objectives could be added to aid effective representation learning, for simplicity we focus our attention on autoencoders. We follow Lange & Riedmiller (2010); Finn et al. (2015) and in Section 4.3 try an iterative unsupervised pretraining of toencoder that reconstructs pixels and is parameterized by 3- as per Nair et al. (2018); Higgins et al. (2017a). Exploring the training procedure used in previous work shows it to be sub-optimal and points towards the need for end-to-end training of the 3-VAE with the policy network. Our investiga- tion in Section 4.4 renders this approach useless due severe instability in training, especially with larger 3 values. We resolve this by using deterministic forms of the variational autoencoder (Ghosh al., 2019) and careful learning procedure. This leads to our algorithm, which described and evaluated in Section 5.
4.1 EXPERIMENTAL SETUP
We briefly state our setup here, for more details refer to Appendix B. Throughout the paper we evaluate on 6 image-based based challenging continuous control tasks from Tassa al. (2018) depicted in Figure 1. For a concise presentation, in some places of the main paper we choose to plot results for rea reacher_easy, ball wll.inccu_ccatcctt and wa ker_wal only, while full results are available in the Appendix. An episode for each task results in maximum total reward of 1000 and lasts for exactly 1000 steps. Image observations are represented as 3 x 84 x 84 RGB renderings, where each pixel is scaled down to [0, 1] range. To infer velocity and acceleration frames following standard practice from Mnih et al. (2013). We keep the hyper parameters fixed across all tasks, except for action repeat, which we set only when learning from pixels according to Hafner et al. (2018) for a fair comparison to the baselines. If action repeat is used, the number of training observations is only fraction of the environment steps (e.g. 1000 steps episode action repeat 4 will only result in 250 training observations). The exact action repeat settings can be found in Appendix B.3. We evaluate an agent after every 10000 training observation, by computing an average total reward across 10 evaluation episodes. For reliable comparison we run 10 random seeds for each configuration and compute mean and standard deviation of the evaluation reward.
stack 3 consecutive
4.2 MODEL-FREE OFF-POLICY RL WITH NO AUXILJARY TASKS
We start with an experiment comparing model-free and off off-policy algorithm SAC (Haarnoja et al., 2018) on pixels, with two state-of-the-a model-based algorithms, PlaNet (Hafner et al., 2018) and SLAC (Lee et al., 2019), and an upper bound of SAC on proprioceptive state (Table 1). We see a large gap between the capability of SAC on pixels (SAC:pixel), versus PlaNet and SLAC, which make use of many auxiliary tasks to learn a better representation, and can achieve performance close to the upper bound of SAC proprioceptive state (SAC:state). From now, SAC:pixel will be our lower bound on performance as we gradually introduce different auxiliary reconstruction losses in order to close the performance gap.
4

---

Under review as a conference paper at ICLR 2020
Number of Episodes
Task name
SACpixel
PlaNet
SLAC
SAC:state
finger.spin
1000
645 + 37
659 + 45
900 + 39
945 19
walker.walk
1000
33 +
949 9
864 = 35
974 1
ball in cup catch
2000
593 - 84
861 = 80
932 = 14
981 + 1
hartpole.swinge reacher_easy cheetah.run
2000
758 + 58
802 = 19
860 + 8
2500
121 + 28
949 + 25
953 + 11
3000
366 + 68
701 6
830 + 32
836 + 105
Table 1: A comparison over 6 DMC tasks of SAC from pixels, PlaNet, SLAC, and an upper bound of SAC from proprioceptive state, numbers are averaged over the last episodes across 10 sceds. The large performance gap between SAC:pixel SAC:state motivates us to address the representation learning bottleneck model- free off-policy RL.. Best performance bolded.
reacher_eae _
ball in li__inn_______________ cup_catch
walker_warl walk
1000
1000
1000
average return
750
750
500
500
500
250
250
250
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.8
1.0
environment stops (*10%)
environment steps (*10")
anvironment steps (*10*)
SAC
SAC:pixel
SAC SAC-VAE,plxel (iter, m)
SAC+ pAll (iter, 100)
VAC+VAE: (iter,
Figure 2: Separate 3-VAE and policy training with shared gradients SAC+VAE:pixe (iter. N), with SAC:state shown as an upper bound. refers to frequency environment steps at which the 3-VAE updates after initial pretraining. More frequent updates are beneficial for learning better representations, but cannot fully address the gap in performance. Full results in Appendix C.
4.3 ITERATIVE REPRESENTATION LEARNING WITH 3-VAE
Following Lange& & Riedmiller (2010); Finn et al. (2015), we experiment with unsupervised repre- sentation pretraining using pixel autoencoder, which speeds up representation learning in image- based RL. Taking into account successful results from Nair et al. (2018); Higgins et al. (2017b) of using a 3-VAE (Kingma & Welling, 2013; Higgins al., 2017a) in the iterative re-training setup, we choose to employ a 3-VAE likewise. We then proceed to first learn a representation space by pretraining the feues fenc and fdee networks of the 8- VAE according to the loss ((VAE) Equa- tion (4) on data collected from random policy. We then learn control policy
top of the frozen latent representations 2: = fenc [onc(o:). (o.). We tune 3 for best performance, and find large 3 to worse, and that very small 3 [10-$, 10-6) performed best. In Figure 2 we vary the frequency N at which the representation space is updated, from N = J0, where the representation is never updated after an initial pretraining period with randomly collected data, to N 1 where the representation is positive correlation between this frequency and the
updated after every policy update. There final policy performance. We emphasize that the gradients never shared between the 3-VAE for learning the representation space, and the actor-critic learning the policy. These results suggest that if we can combine the representation pretraining via 3-VAE together with the policy learning in a stable end-to-end procedure, we would expect better performance. However, we note that prior work (Nair al., 2018; Higgins al., 2017a) has been unable to successfully demonstrate this. Regardless, we next perform such experiment to gain better understanding on what goes wrong.
reacher reay
ball cup_catch
walker walk
1000
1000
1000
average return
750
750
750
500
500
500
250
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
environment steps (*10%)
environment steps (*109)
environment steps (*109)
SAC:state
SAC:pixel
SAC+VAE:pixel liter,
VAC+vAE:
An unsuccessful attempt propagate gradients from the actor-critic down the encoder of the 3-VAE to enable end- end-to-end off- off-policy training. The learning process SAC-VAE exhibits instability together with the subpar performance comparing to the baseline SAC+ VAE:pixel (iter, 1), which does not share gradients with the actor-critic. Full results in Appendix D.
Figure 3:
5

---

Under review as a conference paper at ICLR 2020
forward
reconstruction target
VJ(AE)
VJ(Q)
VJ(x)
dhoors AGZEZA
J(K
2Q
decoder
MLP
J(@)
463ba)
((s,
encoder
observation 3x[3x84x84]
Z
MLP
J(T)
Figure 4: Our algorithm (SAC+AE) auguments SAC (Haarnoja et al., 2018) with regularized autoen- coder (Ghosh al., 2019) to achieve stable end-to-end training from images in the off-policy regime. The stability comes from switching a deterministic encoder that is carefully updated with gradients from the reconstruction J (AE) (Equation (3)) and soft Q-learning J (Q) (Equation (1)) objectives.
4.4 AN ATTEMPT FOR END-TO-END REPRESENTATION LEARNING WITH 3-VAE
Our findings and the results from Jaderberg et al. (2017) motivate us to allow gradient propagation to the encoder of the 3-VAE from the actor-critic, which in our case is SAC. We enable end-to-end learning by allowing the encoder to not only update with gradients from the J(VAE) loss (Equa- tion (4), as done in Section 4.3, but also with gradients coming from the (() and (TT) (Equa- tions (1) and (2)) losses specified in Section 3. Results in Figure 3 show that the end-to-end policy learning together with the 3-VAE in unstable the off-policy setting and prone to divergent be- haviours that hurt performance. Our conclusion supports the findings from Nair et al. (2018); Hig- gins et al. (2017a), which alleviate the problem by receding to the iterative re-training procedure. We next attempt stabilizing end-to-end training and introduce our method.
5 OUR METHOD: SAC+AE WITH END-TO-END OFF-POLICY TRAINING
We now seek to design a stable training procedure that can update the pixel autoencoder simulta- neously with policy learning. We build on top of SAC (Haarnoja et al., 2018), model-free and off-policy actor-critic algorithm. Based on our findings from Section 4, we propose a new, simple algorithm, SAC+AE, that enables end-to-end training. We notice that electing to learn deterministic latent representations, rather than stochastic in the 3-VAE case, has stabilizing effect on the end-to-end learning in the off- off-policy regime. We thus use a deterministic autoencoder in a form of the regularized autoencoder (RAE) (Ghosh et al., 2019), that has many structural similarities with 3-VAE. We also found it is important to update the convolutional weights in the target critic network faster, than the rest of the parameters. This allows faster learning while preserving the stability of the off-policy actor-critic. Finally, we share the encoder's 's convolutional weights between the actor and critic networks, but prevent the actor from updating them. Our algorithm is presented in Figure 4 for visual guidance.
5.1 PERFORMANCE ON PIXELS
We now show that our simple method, SAC+AE, achieves stable end-to-end training of an off-policy algorithm from images with an auxiliary reconstruction loss. We test our method on 6 challenging image-based based continuous control tasks (see Figure 1) from DMC (Tassa et al., 2018). The RAE consists of convolutional and deconvolutional trunk of layers of 32 filters each, with size. The actor and critic networks are 3 layer ML.s with ReLU activations and hidden size of 1024. We update the RAE and actor-critic network at each environment step with a batch of experience sampled from replay buffer. A comprehensive overview of other hyper paremeters is Appendix B.
3 kernel
We perform comparisons against several state-of-the-art model-free and model-based RL algorithms for learning from pixels. In particular: D4PG (Barth-Maron et al., 2018), an off-policy actor-critic algorithm, PlaNet (Hafner et al., 2018), a model-based method that learns a dynamics model with deterministic and stochastic latent variables and employs cross-entropy planning for control, and SLAC (Lee et al., 2019), which combines a purely stochastic latent model together with an model- free soft actor-critic. In addition, compare against SAC that learns from low-dimensional pro-

---

Under review as a conference paper at ICLR 2020
reacher_ee
ball cup_cccccccchhhhhcc
walker walk
1000
average return
750
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
finger spin
cartpole: swingup
cheetah run
1000
1000
1000
average return
1/1
750
750
750
500
500
500
250
250
0.0
0.5
1.5
2.5
environment (*10%)
environment steps (*10')
envirorment steps (*109)
SAC:state
pixll (10* steps)
Pla
SLAC
SAC:pixal
AA: (ours)
Figure 5: The main result our work. Our method demonstrates significantly improved performance over the baseline SAC:pixel, Moreover, it matches the state-of-the-ari performance of model-based algorithms, such PlaNet (Hafner et al., 2018) and SLAC (Lee al., 2019), as well as model-free algorithm D4PG (Barth- Maron et al., 2018), that also learns from images. Our algorithm exhibits stable learning across ten random seeds extremely easy to implement.
prioceptive state, as an upper bound on performance. In Figure we show that SAC+eE:iixel is able to match the state-of-the-ar model-based methods such as PlaNet and SLAC, and significantly improve performance over the baseline SAC:pixel. Note that we use 10 random seeds, as recom- mended in Henderson et (2018) whereas the PlaNet and SLAC numbers shown are only over and 2 seeds, respectively, as per the original publications.
ABLATIONS
To shed more light some properties of the latent representation space learned by our algorithm we conduct several ablation studies. In particular, we want to answer the following questions: (i) our method able extract sufficient amount of information from raw images recover cor- responding proprioceptive states readily? (ii) can our learned latent representation generalize to unseen tasks with similar image observations, but different reward objective, without reconstruction signal? Below, we answer these questions.
6.1 REPRESENTATION POWER OF THE ENCODER
Given how significantly our method outperforms variant that does not have access to the image construction signal, we hypothesize that the learned representation space encodes a sufficient amount of information about the internal state of the environment from raw images. Moreover, this infor- mation can be easily extracted from the latent state. To test this conjecture, we train SAC+AE:pixel and SAC:pixel until convergence cheetah.run, then fix their encoders. We then train two
position[1] []
pasition[2]
velacity(1]
velocity[2]
0.5
2.5
value
0.0
0.0
-2.5
-0.5
-1
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
episode steps
steps
episode
steps
SAC+AE:pixel
SACpixel
Figure 6: Linear projections of latent representation spaces learned by our method (SiC+AE:pixel) and the baseline (SAC:pixel) onto proprioceptive states. We compare ground truth value of cach propriocoptive co- ordinate against their reconstructions for aheotah_roe and conclude that method successfully encodes proprioceptive state information. For visual clarity we only plot position (out of 8) and velocity (out of 9) coordinates. Full results Appendix

---

Under review as a conference paper at ICLR 2020
walker_stanm _ttadd
walker run
1000
1000
average return
750
750
500
500
250
250
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
environment steps (*10f)
environment steps (+109)
SAC:state
SAC:pixe
SAC:pixel (pretrained on walker_walk
Figure 7: Encoder pretrained with method (EEiie))) walker. wwakk is able generalize unseen walker malker.stande stand walker.rum tasks, All three tasks share similar image observations, but have quite different reward structure. SAC with a pretrained on walker.walk encoder achieves impressive final performance. while the baseline struggles to solve the tasks.
identical linear projections map the encoders' latent embedding of image observations into the corresponding proprioceptive states. Finally, we compare ground truth proprioceptive states against their reconstructions a sample episode. Results in Figure 6 confirm our hypothesis that the coder grounded on pixel observations is powerful enough to almost perfectly restore the internals of the task, whereas SAC without the reconstruction loss cannot. Full results in Appendix F.
6.2 GENERALIZATION TO UNSEEN TASKS
To verify whether the latent representation space learned by our method is able to generalize to different tasks without additional fine-tuning with the reconstruction signal, we take three tasks walker stand, walker walk, and walker run from DMC, which share similar obser- vational appearance, but have different reward structure. We train an agent using our method (SAC+AE:pixe)) walker_wall task until convergence and extract its encoder. Consequently, we train two SAC agents without reconstruction loss on ker.standd and wa walker tasks from pixels. The encoder of the first agent is initialized with weights from the pretrained walker.wa wa encoder, while the encoder of the second agent is not. Neither of the agents use the reconstruction signal, and only backpropogate gradients from the critic the encoder (see Figure 4). Results in Figure 7 suggest that our method learns latent representations that can readily generalize to unseen tasks and help SAC agent achieve strong performance and solve the tasks.
DISCUSSION
We have presented the first end-to-end, off-policy, model-free RL algorithm for pixel observations with only reconstruction loss as an auxiliary task. is competitive with state-of-the-ar model-based methods, but much simpler, robust, and without requiring learning a dynamics model. show through ablations the superiority of end-to-end learning over previous methods that use two-step training procedure with separated gradients, the necessity of a pixel reconstruction loss over recon- struction to -iver-dimesiional "correct" representations, and demonstrations of the representation power and generalization ability of our learned representation.
We find that deterministic models outperform 3-VAEs (Higgins al., 2017a), likely due to the other introduced instabilities, such as bootstrapping, off-po data, and end-to-end training with auxiliary losses. We hypothesize that deterministic models that perform better even in stochastic environments should be chosen over stochastic ones with the potential to learn probability distri- butions, and argue that determinism has the benefit of added interpretability, through handling of simpler distributions.
In the Appendix we provide results across all experiments on the full suite of 6 tasks chosen from DMC (Appendix A), and the full set of hyperparameters used in Appendix B. There are also ad- ditional experiments autoencoder capacity (Appendix E), look at optimality of the learned latent representation (Appendix H), importance of action repeat (Appendix I), and set of benchmarks on learning from proprioceptive observation (Appendix J). Finally, we opensource our codebase for the community to spur future research in image-based RL.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv e-prints, 2016.
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference Learning Representations, 2018.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling- The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 2013.
Richard Bellman. A markovian decision process. Indiana Univ. Math. J., 1957.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Learning visual feature spaces for robotic manipulation with deep spatial autoencoders. CoRR, 2015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, ICMI, 2018, Stockholmsmiisaan Stockholm, Sweden, July 10- 15, 2018, 2018.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv e-prints, 2019.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31. Curran Associates, Inc., 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: 2.05905, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 18/ 1811.00551, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. Thirty-Second AAAI Conference On Artificial Intelli- gence (AAAI), 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017a.
Irina Higgins, Arka Pal, Andrei A. Rusu, Burgess Christopher P Matthey, Loic and, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero- shot transfer in reinforcement learning. CoRR, 2017b.
Max Jaderberg, Volodymyr Mnih, Wojciech Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. International Conference on Learning Representations, 2017.
Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions.
arXiv e-prints, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
1.Xiv:131..6114.20 2013.
Sascha Lange and Martin A. Riedmiller. Deep auto-encoder neural networks in reinforcement leamn ing. In IJCNN, 2010.

---

Under review as a conference paper at ICLR 2020
A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforce- ment learning with latent variable model. arXiv e-prints, 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 2015.
Jan Mattner, Sascha Lange, and Martin Riedmiller. Learn to swing up and balance real pole based on raw visual input data. In Neural Information Processing, 2012.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv e-prints, 2013.
J. Munk, Jens Kober, and Robert Babuska. Learning state representation for deep actor-critic control. In Proceedings 2016 IEEE 55th Conference on Decision and Control (CDC), 2016.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Bengio, Wallach, H. Larochelle, Grau- man, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 9191-9200
Curran Associates, Inc., 2018.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dy- namics of learning in deep linear neural networks. arXiv e-prints, 2013.
Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss its own reward: Self- supervision for reinforcement learning. CoRR, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint http::10800000900 2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. arXiv e-prints, 2015.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How train 10,000-layer vanilla convolutional neural networks. arXiv e-prints, 2018.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. CoRR, 2018
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference Artificial Intelligence Volume 3, 2008
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A THE DEEPMIND CONTROL SUITE
We evaluate the algorithms in the paper the DeepMind control suite (DMC) (Tassa et al., 2018) collection of continuous control tasks that offers an excellent testbed for reinforcement learning agents. The software emphasizes the importance of having standardised set of benchmarks with unified reward structure in order to measure made progress reliably.
Specifically. consider six domains (see Figure 8) that result in twelve different control tasks. Rach task (Table 2) poses a particular set challenges to a learning algorithm. The ball_in_cup.caten task only provides the agent with sparse reward when the ball is caught; the cheetah.run task offers high dimensional internal state and action spaces; the reacher harre task requires the agent explore the environment. We refer the reader the original paper find more information about the benchmarks.
Task name
dim O)
dim(.A)
Reward type
Proprioceptive
Image- based
call.inccap.catch
3 84 84
2
sparse dense
cartpole.(ball swingup) cheetah.run
5
3 x 84 x 84
1
17
3 x 84 x 84
6
dense
finger.(spin, turn.easy, turn.hard] reacher.(easy, hard) walker.(stand, walk, run)
12
3 3 84
x 84
2
dense/sparse
7
3 x
84
x 84
2
sparse
24
3
84
84
6
dense
Table 2: We list specifications of observation space O (proprioceptive and image-based), action
space A, and the reward type for each task.
U
(a) ball in.cup
(b) cartpole
(c) cheetah
(d) fi Finger
(e) reacher
ker
Figure 8: We use six domains spanning the total of twelve challenging continuous control tasks: finger_(spin, turn,eas, turn_hars), (balanc, swingup), cheetah_run, walker_[stand walk, run), bal..i..cu.cctcc..
reacher reacher_(easy. hard),
and

---

Under review as a conference paper at ICLR 2020
B HYPER PARAMETERS AND SETUP
B.1 ACTOR AND CRITIC NETWORKS
We employ double Q-learning (van Hasselt et al., 2015) for the critic, where each Q-function is parametrized 3-layer MLP with ReLU activations after each layer except of the last. The actor is also a 3-layer MLP with ReLUs that outputs mean and covariance for the diagonal Gaussian that represents the policy. The hidden dimension is set to 1024 for both the critic and actor.
B.2 ENCODER AND DECODER NETWORKS
We employ an almost identical encoder architecture as in Tassa al. (2018), with two minor differ- ences. Firstly, we add two more convolutional layers to the convnet trunk. Secondly, we use ReLU activations after each conv layer, instead of ELU. We employ kernels of size 3 with 32 channels for all the conv layers and set stride 1 everywhere, except of the first conv layer, which has stride 2. We then take the output of the convnet and feed it into single fully-connected layer normalized by LayerNorm (Ba al., 2016). Finally, we add tanh nonlinearity to the 50 dimensional output of the fully-connected layer.
The actor and critic networks both have separate encoders, although share the weights of the conv layers between them. Furthermore, only the critic optimizer is allowed to update these weights (e.g. we truncate the gradients from the actor before they propagate the shared conv layers).
The decoder consists of one fully-connected layer that is then followed by four deconv layers. We ReLU activations after each layer, except the final deconv layer that produces pixels representa- tion. Each deconv layer has kernels of size x 3 with 32 channels and stride 1, except of the last layer, where stride is 2.
We then combine the critic's encoder together with the decoder specified above into an autoencoder. Note, because we share conv weights between the critic's 's and actor's encoders, the conv layers of the actor's 's encoder will be also affected by reconstruction signal from the autoencoder.
B.3 TRAINING AND EVALUATION SETUP
We first collect 1000 seed observations using random policy. We then collect training observations by sampling actions from the current policy. We perform one training update every time we receive new observation. In cases where we use action repeat, the number of training observations is only a fraction the environment steps (e.g. 1000 steps episode at action repeat will only results into 250 training observations). The action repeat used for each environment is specified in Table 3, following those used by PlaNet and SLAC.
We evaluate agent after every 10000 environment steps by computing average episode return over 10 evaluation episodes. Instead sampling from the Gaussian policy we take its mean during evaluation.
We preserve this setup throughout all the experiments the paper.
Task name
Action repeat
cartpole.swingup reacher easy
cheetah. finger_spin balliincup.catchh walker.walk
Table 3: Action repeat parameter used per task. following PlaNet and SLAC.
12

---

Under review as a conference paper at ICLR 2020
B.4 WEIGHTS INITIALIZATION
We initialize the weight matrix of fully-connected layers with the orthogonal initialization (Saxe al., 2013) and set the bias be zero. For convolutional and deconvolutional layers use delta- orthogonal initialization (Xiao et al., 2018).
B.5 REGULARIZATION
We regularize the autoencoder network using the scheme proposed in Ghosh al. (2019). In par- ticular, we extend the standard reconstruction loss for deterministic autoencoder with a L2 penalty on the learned representation 2 and add weight decay on the decoder parameters 8
CRAE Free + >=11=112 + Xo delc
We set A 10-6 and Ag 10-?
B.6 PIXELS PREPROCESSING
We construct an observational input as an 3-stack of consecutive frames (Mnih et al., 2013), where each frame a RGB rendering of size 84 84 from the Oth camera. We then divide each pixel by 255 scale it down to [0, 1) range. For reconstruction targets we instead preprocess images by reducing bit depth to bits as Kingma & Dhariwal (2018).
B.7 OTHER HYPER PARAMETERS
We also provide comprehensive overview of all the remaining hyper parameters in Table
Parameter name
Value
Replay buffer capacity
1000000
Batch size
128
Discount y
0.99
Optimizer
Adam
Critic learning rate
10
Critic target update frequency
2
Critic Q-function soft-update rate TO
0.01
Critic encoder soft-update rate
0.05
Actor learning rate
10-3
Actor update frequency
2
Actor log stddev bounds Autoencoder learning rate Temperature learning rate Temperature Adam's 81 Init temperature
[-10
10-3
10
0.5
0.1
Table 4: A complete overview of used hyper parameters.
13

---

Under review as a conference paper at ICLR 2020
C ITERATIVE REPRESENTATION LEARNING WITH 3- BAE
Iterative pretraining suggested in Lange & Riedmiller (2010); Finn et al. (2015) allows for faster rep- resentation learning, which consequently boosts the final performance, yet it sufficient enough to fully close the gap and additional modifications, such as end-to-end training, are needed. Figure provides additional results for the experiment described in Section 4.3.
reacher easy
cup_catch
average return
750
750
750
500
500
500
250
250
250
0
0.5
1.5
2.0
finger_ste _spin
cartpole_swe swingup
cheetah_rue _ru
1000
1000
average return
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
crvvronment stops (110%)
environment stops (*10")
environment steps (*10%)
SAC:pixel
SAC=VAE:pixel (iter, =)
SAC+VAE:pixe (iter. 100)
SAC-VAE;pixel (iter,
Figure 9: Separate 3-VAE and policy training with no shared gradients SAC+VAE:pixel (iter, N), with SAC:state shown as an upper bound. N refers to frequency in environment steps at which the 3-VAE updates after initial pretraining. More frequent updates are beneficial for learning better representations, but cannot fully address the gap performance.
14

---

Under review as a conference paper at ICLR 2020
D AN ATTEMPT FOR END-TO-END REPRESENTATION LEARNING WITH B-VAE
Additional results the experiments from Section 4.4 are in Figure 10.
reacher_ese _easy
ball cat__n__uu_cachh
walker_ walk
1000
1000
average retum
250
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
finger_Ste _ppin
cartpole_swingup
cheetah_mee run
1000
1000
1000
average retur
750
750
750
500
500
500
250
250
0.2
0.0
0.5
1.0
1.5
2.0
1.0
2.5
environment steps (*10%)
environment steps (*10")
envirorment steps ('10%)
SAC:state
SAC pixel
SAC+VAE:pixel (iter,
SAC+VAE
Figure 10: An unsuccessful attempt to propagate gradients from the actor-critic down to the encoder of the B- VAE to enable mndtt--end off- off-policy training. The learning process SAC+ Vii:piee exhibits instability together with the subpar performance comparing to the baseline SAC+ VAE:pixel (iter, 1), which does not share gradients with the actor-critic.
15

---

Under review as a conference paper at ICLR 2020
E CAPACITY OF THE AUTOENCODER
We also investigate various autoencoder capacities for the different tasks. Specifically, we measure the impact of changing the capacity of the convolutional trunk of the encoder and corresponding deconvolutional trunk of the decoder. Here, we maintain the shared weights across convolutional layers between the actor and critic, but modify the number of convolutional layers and number of filters per layer in Figure 11 across several environments. We find that SAC+AE is robust to various autoencoder capacities, and all architectures tried were capable of extracting the relevant features from pixel space necessary to learn good policy. We use the same training and evaluation setup as detailed in Appendix B.3.
reacher_ess easy
ball in__iin___c________
walker_warl
average return
750
750
500
500
500
250
250
250
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.2
0.4
0.8
1.0
finger_spin
cartpole swingup
cheetah run
1000
1000
1000
average return
750
750
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
ervironment steps (*10%)
environment steps (*10")
environment steps (*10%)
SAC:state SAC:pixel
SAC+AE,pixel (2x32)
SAC+AE pixel (4x64) SAC+AE (6x64)
AAcl (4x32)
Figure 11: Different autoencoder architectures, where we vary the number of conv layers and the number of output channels in each layer both the encoder and decoder. For example, 4 x 32 specifies architecture with conv layers, each outputting 32 channels. We observe that the difference in capacity has only limited effect on final performance.
16

---

Under review as a conference paper at ICLR 2020
F REPRESENTATION POWER OF THE ENCODER
Addition results to the experiment in Section 6.1 that demonstrates encoder's power to reconstruct proprioceptive state from image-observations are shown Figure 12.
position[O]
position[1]
position[2]
0.5
0.0
0.5
0.0
0.0
0.1
-0.5
position[3]
position[4]
position[5]
-0.5
11
200
200
200
[osition[6]
position[7]
0.5
0.5
0.0
0.0
-0.5
-0.5
200
velacity[o]
velocity[1]
velocity[2]
7.5
2.5
value
5.0
0
2.5
2.5
50
100
150
200
50
100
150
200
50
100
150
200
velocity[3]
velocity[4]
velocity[5]
20
value
-10
-20
50
100
200
50
100
150
200
50
100
150
200
velocity[6]
velocityL7]
velocity[8]
20
10
value
0
-10
-10
-20
50
100
150
200
50
100
150
200
50
100
150
200
episode steps
episode steps
episode steps
Truth
SAC+AE.plke
SAC:pixel
Figure 12: Linear projections of latent representation spaces learned by our method (SAC+AE:pixel)
and the baseline (SAC:pixel) onto proprioceptive states. We compare ground truth value of each proprioceptive coordinate against their reconstructions cheetah run, and conclude that our method successfully encodes proprioceptive state information. The proprioceptive state of cheetah. run has position and velocity coordinates.
17

---

Under review as a conference paper at ICLR 2020
G DECODING TO PROPRIOCEPTIVE STATE
Learning from low-dimensional proprioceptive observations achieves better final performance with greater sample efficiency (see Figure 5 for comparison pixels and Appendix for proprioceptive baselines), therefore our intuition is to directly use these compact observations as the reconstruction targets to generate an auxiliary signal. Although, this an unrealistic setup, given that we do not have access to proprioceptive states in practice, we use it as tool to understand if such supervision beneficial for representation learning and therefore can achieve good performance. We augment the observational encoder Sene, that maps an image into a latent vector with state decoder /state_dec= that restores the corresponding state 8; from the latent vector 2;- This leads an auxililary where Z, fenc(Or), We parametrize the state decoder fstate_dec as a 3-layer MLP with 1024 hidden size and ReLU activations, and train end-to-end with the actor-critic network. Such auxiliary supervision helps less than expected, and surprisingly hurts performance in ball_ pall_in_cupp waaneuupccateeeeee happ.ctccc. ch, as seen in Figure 13. Our intuition is that such low- dimensional supervision is able to provide the rich reconstruction error needed fit the high- capacity convolutional encoder Senc. We thus seek for denser auxiliary signal and try learning latent representation spaces with pixel reconstructions.
objective ,,,, arrr ffllfstate.des (z)) -s [
reacher_ease
ball l____j____cclll___
walker_
average return
750
750
500
500
500
250
250
250
0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.2
0.8
finger_spin
cartpole swingup
cheetah_red run
1000
1000
1000
average return
750
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
environment steps (110%)
environment stops (*10")
envirorment stops (*10%)
SACstaate
SAC-pixel
SAC:pixel (state supervision)
Figure 13: An auxiliary signal is provided by reconstructing low-dimensional state from the cor- responding image observation. Perhaps surprisingly, such synthetic supervision doesn't guarantee sufficient signal to fit the high-capacity encoder, which we infer from the suboptimal performance of SAC:pixel (state supervision) compared to SAC:pixel in bal 1 cn.ccuccattcc.
18

---

Under review as a conference paper at ICLR 2020
H OPTIMALITY OF LEARNED LATENT REPRESENTATION
We define the optimality of the learned latent representation as the ability our model to extract and preserve all relevant information from the pixel observations sufficient learn good policy. For example, the proprioceptive state representation is clearly better than the pixel representation because we can learn better policy. However, the differences in performance of SAC:state and SAC+eE. can be attributed not only to the different observation spaces, but also the difference in data collected in the replay buffer. To decouple these attributes and determine how much informa- tion loss there is in moving from proprioceptive state to pixel images, we measure final task reward of policies learned from the same fixed replay buffer, where one trained on proprioceptive states and the other trained pixel observations.
first train a SAC+AE policy until convergence and save the replay buffer that we collected during training. Importantly, in the replay buffer store both the pixel observations and the corresponding proprioceptive states. Note that for two policies trained on the fixed replay buffer, we are operating in off- policy regime, and thus it is possible won't t be able to train policy that performs as well.
reacher easy
ball cup_ccu_cccee_____ catch
walker walk
average return
500
500
500
250
250
250
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
finger_spie
cartpole_swingup
cheetah_ run
10000
1000
1080
average retumn
750
750
750
500
500
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
environment stops (110%)
environment stops (*10F)
envirorment stops (*109)
AEpiiel (collector)
pixell
buffer)
Figure 14: Training curves for the policy used to collect the buffer (SAC+AE:piell (collector)). and the two policies learned on that buffer using proprioceptive (SAC:state (fixed buffer)) and pixel observations (SAC+AE:pixel) (fixed buffer)). We see that our method actually outperforms proprio- ceptive observations this setting.
Figure 14 we find, surprisingly, that our learned latent representation outperforms proprioceptive state fixed buffer. This could be because the data collected in the buffer is by policy also learned from pixel observations, and is different enough from the policy that would learned from proprioceptive states that SAC:state underperforms in this setting.
19

---

Under review as a conference paper at ICLR 2020
1 IMPORTANCE OF ACTION REPEAT
We found that repeating nominal actions several times has a significant effect on learning dynamics and final reward. Prior works (Hafner al., 2018; Lee et al., 2019) treat action repeat hyper parameter to the learning algorithm, rather than property the target environment. Effectively, action repeat decreases the control horizon of the task and makes the control dynamics more stable. Yet, action repeat can also introduce a harmful bias, that prevents the agent from learning an optimal policy due to the injected lag. This tasks a practitioner with a problem of finding an optimal value for the action repeat hyper parameter that stabilizes training without limiting control elasticity too much.
To get more insights, we perform an ablation study, where we sweep over several choices for action repeat on multiple control tasks and compare acquired results against PlaNet (Hafner et al., 2018) with the original action repeat setting, which was also tuned per environment. We use the same setup as detailed in Appendix B.3. Specifically, we average performance over 10 random seeds, and reduce the number of training observations inverse proportionally to the action repeat value. The results are shown in Figure 15. We observe that PlaNet's choice of action repeat not always optimal for our algorithm. For example, we can significantly improve performance of our agent on the ball.in.cup.c.. ch task if instead of taking the same nominal action four times, as PlaNet suggests, we take once twice. The same is true on a few other environments.
reacher_ easy
ball jailin_nco ua__catch
walker walk
1000
1000
average retur
250
250
250
PoNect
PaNeet
Plaiett
0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
0.0
0.2
0.4
0.6
0.8
1.0
finger spin
cartpole swingup
cheetah run
1000
1000
1000
average retur
750
750
750
500
500
500
250
250
Plafent (2)
PlaNter IR]
Raaeet (4)
0.2
0.0
0.5
1.0
1.5
2.0
environment steps (*10%)
environment steps (*109)
environment steps (*10%)
SAC state
SAC AAC+AE:plxl (1)
SAC AEcpicl (2)
AAC (4)
Figure 15: We study the importance of the action repeat hyper parameter on final performance. We evaluate three different settings, where the agent applies a sampled action once (SAC+AE:pixel (1)), twice (Six+AE.pixel (2)), or four times (SAC+AE:pixel (4)). As reference, we also plot the PlaNet (Hafner et al., 2018) results with the original action repeat setting. Action repeat has significant effect on learning. Moreover, we note that the PlaNet's choice of hyper parameters is not always optimal for our method (e.g. better to apply an action only once on wa 1 ker_wa k, than taking twice).
J LEARNING FROM PROPRIOCEPTIVE OBSERVATIONS
In addition to the results when an agent learns from pixels, we also provide comprehensive compar- ison of several state-of-the-ate continuous control algorithms that directly learn from proprioceptive states. Specifically, we consider four agents that implement SAC (Haarnoja et al., 2018), TD3 (Fuji- moto et al., 2018), DDPG (Lillicrap et al., 2015), and D4PG (Barth-Maron et al., 2018). We leverage open-source implementations of TD3 and DDPG from reported set of optimal hyper parameters, except of the batch size, which we increase 512, find improves performance of both the algorithms. Due to lack of publicly accessible imple- mentation of D4PG, take the final performance results after 10 environments steps reported
and use the
20

---

Under review as a conference paper at ICLR 2020
in Tassa et al. (2018). We use listed Appendix B, again we increase the batch size to 512. Importantly, we keep the same set of hyper parameters across all tasks to avoid overfitting individual tasks.
implementation of SAC together with the hyper parameters
For this evaluation we do not repeat actions and perform one training update per every environment step. We evaluate policy every 10000 steps (or every 10 episodes steps) by running 10 evaluation episodes and averaging corresponding returns. To assess the stability properties of each algorithm and produce reliable baselines we compute mean and std of evaluation performance over 10 random seeds. We test on twelve challenging continuous control tasks from DMC (Tassa al., 2018), as described Appendix A. The results are shown in Figure 16.
episode consists of 1000
ball in hauncuuu__ccc_____pp_p
cartpole balance
cartpole_ste serppoll_ssinuupp
1000
1000
1000
averoge recurn
750
750
750
500
500
500
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
walker_stanm sstand
reacher_east _esh
reacher __ard
1000
1000
1000
average return
750
750
750
500
500
500
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
cheetah_ne
walker walk
finger_spin
1000
1000
1000
average return
750
750
750
500
500
500
250
250
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
finger turn jurn_easy
finger turn hard
walker run
1000
1000
1000
average return
750
750
750
500
500
500
Mapo
250
250
250
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
3.0
0.0
0.5
2.0
2.5
3.0
environment steps (*10F)
environment steps (*10%)
environment steps (t109)
D4PG (10" steps)
DDPG
Figure 16: We benchmark SAC. TD3, DDPG, and D4PG when learning from proprioceptive states on multiple tasks of various difficulty from DMC. We run 10 random seeds for each algorithm and evaluate on 10 trajectories (except for D4PG, as its implementation then report mean and standard deviation. For D4PG we take the performance after 10 environment steps reported Tassa et al. (2018). We observe that SAC demonstrates superior performance and sample efficiency over the other methods all of the tasks.
not publicly available). We
21


Reviews:
[{"review_id": "r1ly8pPKuH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to tackle the problem of improving sample efficiency of model-free, off-policy reinforcement learning in an image-based environment. They do so by taking SAC and adding a deterministic autoencoder, trained end-to-end with the actor and critic, with the actor and critic trained on top of the learned latent space z. They call this SAC-AE. Experiments in the DeepMind control suite demonstrate that the result models train much faster than SAC directly on the pixels, in some cases reaching close to the performance of SAC on raw state. Ablation studies demonstrate their approach is most stable with deterministic autoencoders proposed by (Ghosh et al, 2019), rather than the beta-VAE autoencoder proposed in (Nair et al, 2018), end-to-end learning of the autoencoder gives improved performance, and the encoder transfers to some similar tasks.\n\nI thought the paper was written well, and its experiments were done quite carefully, but it was lacking on the novelty front. At a high level, the paper has many similarities with the UNREAL paper (Jaderberg et al, 2017), which is acknowledged in the related work. This paper says it differs from UNREAL because they use an off-policy algorithm, and that UNREAL's auxiliary tasks are based off real-world inductive priors.\n\nI don't see the off-policy distinction as very relevant, because in the end, both UNREAL and SAC-AE are actor-critic algorithms (using A3C and SAC respectively). The way that SAC is used in the paper always collects data in a near on-policy manner, and UNREAL includes experience replay from a replay buffer, which introduces some off-policy nature to UNREAL as well. Therefore this doesn't feel like a strong argument.\n\nFurthermore, although some of the auxiliary tasks in UNREAL are based off human intuition for what makes sense in those environments, they also include task-agnostic auxiliary tasks: reward prediction and pixel-level control. These do not depend on real-world inductive priors, and are shown to improve performance.\n\nOverall, this doesn't feel like a strong enough contribution for ICLR.\n\nMore specific comments:\n* Section 6.1 examines the representation power of the encoder by reconstructing proprioceptive state from the encoder. I am not sure the comparison between SAC+AE and SAC is particularly meaningful here. The predictors are learned on top of the encoder output, and in SAC+AE we would expect task information to be encoded in the learned z. But in baseline SAC, there is no reason to expect this to be true - task information is more likely to be distributed across the entire network architecture. The case for SAC+AE seems much stronger from the reward curves, rather than these plots.\n* The paper argues that their approach is stable and sample-efficient, but when looking at the reward curves, it looked about as stable as SAC. Figure 3 (where they do not train the VAE end-to-end in the red curve) has a similar story. This makes me believe that any claims of added stability are more thanks to SAC, rather than proposed methods.\n\nEdit: I would like to clarify that the rating system only provides a 3 for Weak Reject and 6 for Weak Accept. On a 1-10 scale I would rate this as a 5, I feel it is closer to Weak Accept than Weak Reject.\n\nEdit 2: I've read the other author's comments. I'm not particularly convinced by the case for novelty, but I didn't realize that UNREAL's replay buffer was only 2k transitions instead of 1 million transitions. On reflection, I believe the main contribution here is showing that deterministic autoencoders are more reliable than stochastic ones for the RL setting, and this isn't the biggest contribution, but it's enough to make me update to weak accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeFDVFtur", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a simple method for model-free RL from image observations. The key component of the method is the addition of an autoencoder that is trained jointly with the policy and value function, in contrast to previous methods which separate feature learning from policy learning. Another important modification is the use of a deterministic regularized autoencoder instead of a stochastic variational autoencoder. The method is evaluated a variety of control tasks, and shows strong performance when compared to a number of state-of-the-art model-based and model-free methods for RL with image observations.\n\nThe paper is well written and provides a very clear description of the method. The approach is fairly simple and appears to be effective for a suite of challenging tasks. RL from images remains a very challenging problem, and the approach outlined in this work could have a significant impact on the community. The experiments are also thorough and well thought out, and the release of the source code is much appreciated. While the overall novelty is a bit limited, this could be a case where details matter, and insights provided by this work can be valuable for the community. For these reasons, I would like to recommend acceptance.\n\nThere is mention of SLAC as a model-based algorithm. This is not entirely accurate. SLAC does learn a dynamics model as means of acquiring a latent state-representation, but this model is not used to train the policy or for planning at runtime. The policy in SLAC is trained in a model-free manner.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1l3eXJ8tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes an approach to make the model-free state-of-the-art soft actor-critic (SAC) algorithm for proprioceptive state spaces sample-efficient in higher-dimensional visual state spaces. To this end, an encoder-decoder structure to minimize image reconstruction loss is added to SAC's learning objectives. Importantly, the encoder is shared between the encoder-decoder architecture, the critic and the policy. Furthermore, Q-critic updates backpropagate through the encoder such that encoder weights need to trade off image reconstruction and critic learning. The approach is evaluated on six tasks from the DeepMind control suite and compared against proprioceptive SAC, pixel-based SAC, D4PG as well as to the model-based baselines PlaNet and SLAC. The proposed method seems to achieve results competitive with the model-based baselines and significantly improves over raw pixel-based SAC. Further ablation studies are presented to investigate the information capacity of the learned latent representation and generalization to unseen tasks.\n\nQuality\n\nSince this is a paper with a strong practical focus, the quality needs to be judged based on the experiments. The quality of those are good in terms of the number of environments, baselines, benchmarks and seeds. I also liked the ablation studies to investigate latent representations and generalization to new tasks.\n\nClarity\n\nThe paper is very clearly written and easy to follow.\n\nOriginality\n\nUnfortunately, the originality is very low. Combining reinforcement learning with auxiliary objectives is not novel and has been studied in the Atari domain (discrete actions) as noted by the authors, see Jaderberg et al., ICLR, 2017 and Shelhamer et al., arXiv, 2017. The conceptual idea of using a reconstruction loss for images as auxiliary objective is not novel either and has been presented in earlier work already, see Shelhamer et al. The idea of sharing parameters between RL and auxiliary components is also not novel, see Jaderberg et al. One citation that is conceptually very similar to the authors' work is missing: 'Felix Leibfried and Peter Vrancx, Model-based regularization for deep reinforcement learning with transcoder networks. In NeurIPS Deep Reinforcement Learning Workshop, 2018'. The former work combines Q-value learning with auxiliary losses for learning an environment model end to end (with a reconstruction loss for the next state) in the domain of Atari.\n\nSignificance\n\nThe significance is minor to low. The fact that the authors investigate auxiliary losses in continuous-action domains has minor significance. But all in all, the paper might be better suited for a workshop rather than the main track of ICLR.\n\nMinor Details\n\nOn page 3, first equation (not numbered), there is an average over s_{t+1} missing because of the reward definition used by the authors?\n\nUpdate\n\nI read the other reviews and the authors' response. I still feel that the novelty of the work is very limited and the authors' response to lacking novelty does not convince me. However, in light of the strong experimental analysis, I feel in hindsight that a score of 1 from my side was too harsh. I therefore increase my score to 3, but I do still believe that the paper is better suited as a workshop contribution.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates how sample efficiency of image based model-free RL can be improved  by including an image reconstruction loss as an auxiliary task and applies it to soft actor-critic. The method is demonstrated to yield a substantial improvement compared to SAC learned directly from pixels, and comparable performance to other prior works, such as SLAC and PlaNet, but with a simpler learning setup. The reviewers generally appreciate the clarity of presentation and good experimental evaluation. However, all reviewers raise concerns regarding limited novelty, as auxiliary losses for RL have been studied before, and the contribution is mainly in the design choices of the implementation. In this view, and given that the results are on a par with SOTA, the contribution of this paper seems too incremental for publishing in this venue, and I’m recommending rejection. 
Decision: Reject