Guiding Representation Learning in Deep Generative Models with Policy Gradients
Authors: Luca Lach
Timo Korthals
Malte Schilling
Helge Ritter
Venue:  (2021)
Keywords: VAE
RL
PPO

Abstract: Variational Auto Encoder (VAE) provide an efficient latent space representation of complex data distributions which is learned in an unsupervised fashion.
Using such a representation as input to Reinforcement Learning (RL) approaches may reduce learning time, enable domain transfer or improve interpretability of the model.
However, current state-of-the-art approaches that combine VAE with RL fail at learning good performing policies on certain RL domains.
Typically, the VAE is pre-trained in isolation and may omit the embedding of task-relevant features due to insufficiencies of its loss.
As a result, the RL approach can not successfully maximize the reward on these domains.
Therefore, this paper investigates the issues of joint training approaches and explores incorporation of policy gradients from RL into the VAE's latent space to find a task-specific latent space representation.
We show that using pre-trained representations can lead to policies being unable to learn any rewarding behaviour in these environments.
Subsequently, we introduce two types of models which overcome this deficiency by using policy gradients to learn the representation.
Thereby the models are able to embed features into its representation that are crucial for performance on the RL task but would not have been learned with previous methods.

Under review as a conference paper at ICLR 2021
GUIDING REPRESENTATION LEARNING IN DEEP GEN- ERATIVE MODELS WITH POLICY GRADIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Variational Auto Encoder (VAE) provide an efficient latent space representation of complex data distributions which learned an unsupervised fashion. Us- ing such a representation as input to Reinforcement Learning (RL) approaches may reduce learning time, enable domain transfer or improve interpretability of the model. However, current state-of-the-art approaches that combine VAE with RL fail at learning good performing policies on certain RL. domains. Typically, the VAE is pre-trained in isolation and may omit the embedding of task-relevant features due to insufficiencies of its loss. As a result, the RL approach can not successfully maximize the reward vestigates the issues of joint training approaches and explores incorporation policy gradients from RL into the VAE's latent space to find task-specific la- tent space representation. We show that using pre-trained representations can lead policies being unable to learn any rewarding behaviour in these environments. Subsequently, we introduce two types of models which overcome this deficiency by using policy gradients learn the representation. Thereby the models are able to embed features into its representation that are crucial for performance on the RL task but would not have been learned with previous methods.
these domains. Therefore, this paper in-
INTRODCTION
Reinforcement Learning (RL) gained much popularity in recent years by outperforming humans in games such as Atari (Mnih et al. (2015)), Go (Silver et al. (2016)) and Starcraft 2 (Vinyals et al. (017)) These results were facilitated by combining novel machine learning techniques such as deep neural networks (LeCun et al. (215)) with classical RL methods. The RL framework has shown to be quite flexible and has been applied successfully in many further domains, for example, robotics (Andrychowicz et al. (2000), resource management (Mao et al. (2016)) or physiologically accurate locomotion (Kidzinski et al. (2018)).
The goal of representation learning is to learn a suitable representation for a given application domain. Such representation should contain useful information for a particular downstream task and capture the distribution of explanatory factors (Bengio et al.) (2013)). Typically. the choice of downstream task influences the choice of method for representation learning. While Generative Adversarial Network (GAN)s are frequently used for tasks that require high-fideliyy re- constructions or generation of realistic new data, auto-encoder based methods have been more common in RL. Recently, many such approaches employed the Variational Auto Encoder (VAE) Kingmm& & Welling (2013) framework which aims to learn a smooth representation of its domain.
Most of these approaches follow the same pattern: First, they build dataset of states from the RL environment. Second, they train the VAE on this static dataset and lastly train the RL mode using the VAE's representation. While this procedure generates sufficiently good results for certain scenarios, there are some fundamental issues with this method. Such an approach assumes that it is possible to collect enough data and observe all task-relevant states in the environment without knowing how to act in it. As a consequence, when learning to act the agent will only have access to a representation that is optimized for the known and visited states. As soon the agent becomes more competent, it might experience novel states that have not been visited before and for which there is no good representation (in the sense that the experienced states are out of the original learned distribution and the mapping is not appropriate).

---

Under review as a conference paper at ICLR 2021
(a) Input frame after pre-processing
(b) Reconstruction of
Figure I: A frame from Atari Breakout. The original image was passed through pre-trained
VAE yielding the reconstruction Tb Note the missing ball in the lower right hand corner.
Another issue arises from the manner the representation is learned. Usually, the VAE is trained in isolation, decides what features are learned based on its own objective function and on what is helpful for the downstream task. Mostly, such a model is tuned for good reconstruction. Without the information from the RL model, such representation does not reflect what important for the downstream task. As a consequence, the VAE might omit learning features that are crucial for good performance on the task because they appear negligible with respect to reconstruction (Goodfellow et al. (2016), Chapter 15, Figure 15.5). For example, small objects in pixel-space are ignored as they affect a reconstruction based loss only marginally. Thus, any downstream task using such a representation will have access information about such objects. A good example for such task is Atari Breakout, a common RL benchmark. Figures Ia and 1b show an original Breakout frame and its reconstruction. While the original frame contains the ball in the lower right hand corner, this crucial feature is missing completely in the reconstruction.
We approach this issue through simultancously learning representation and RL task, that is by com- bining the training of both models. As an advantage, this abolishes the need of collecting data before knowing the environment as it combines VAE and RL objectives. In consequence the VAE has an incentive to represent features that are relevant to the RL model. The main contributions of this paper are follows: First we show that combined learning possible and that yields good performing policies. Second, we show that jointly trained representations incorporate additional, task-specific information which allows RL agent to achieve higher rewards then if was trained on static representation. This will be shown indirectly by comparing achieved rewards as well as directly through an analysis of the trained model and its representation.
RELATED WORK
Lange& Riedmiller 2010) explored Auto Encoder (AF) (Lecun 1987 Bourlard & Kamp (1888;; Hinton & Zemel (1994)) possible pre-processor for RI. algorithms. The main focus in their work was finding good representations for high dimensional state spaces that enables policy learn- ing. As input, rendered images from the commonly used grid world environment were used. The agent had to manoeuvre through discretized map using one of four discrete movement actions per timestep. It received positive reward once reaching the goal tile and negative rewards else- where. The AE bottleneck consisted only of two neurons, which corresponds to the dimensionality of the environemnt's state. Fitted Q-Iteration (FQI) (Ernst et al. 22055) Q-function, which the agent then acted e-greedy upon. Besides RL, they also used the learned rep- resentation classify the agents position given an encoding using a Multi-Layer Perceptron (MLP) (Rumelhart et al. (1855). For these experiments, they found that adapting the encoder using MLP
used estimate the
2

---

Under review as a conference paper at ICLR 2021
gradients lead to an accuracy of 99.46 %. However, they did not apply this approach to their RL task.
A compelling example for separate training of meaningful representation is provided by Higgins
ct al. (2017b) who proposed a framework called D DARLA. They trained RL agents on the encoding of ( VAE (Higgins et al. (2016): Higgins et al. (2017a) with the goal of zero-shot domain transfer. In their approach, 3- VAE and agent were trained separately on a source domain and then evaluated in a target domain. Importantl, source and target domain are similar to a certain extent and only differ in some features, e.g. a blue object in the source domain might be red in the target domain. During training of the 8- VAE, the pixel-based reconstruction loss was replaced with a loss calculated in the latent space of Denoising Auto Encoder (DAE) (Vincent et al. (2008). Thereby their approach avoids missing task relevant feature encodings at the cost of training another model. For one of their evaluation models, they allowed the RL gradients to adapt the encoder. Their results show that subsequent encoder learning improves performance of Deep Q-Learning (DQN) but decreases performance of Asynchronous Advantage Actor-Critic (A3C) (Mnih et al. (2016)).
Ha & Schmidhuber (2018) proposed a combination of VAE, Recurrent Neural Networks (RNN) (Hochreiter & Schmidhuber (1997)) and simple policy a controller. They hypothesized that by learning a good representation of the environment and having the ability to predict future states, learning the policy itself becomes a trivial task. Like in most other models, the VAE was pre-trained on data collected by a random policy. Only the RNN and the controller were trained online. The compressed representation from the VAE was passed into RNN in order to estimate probability density for the subsequent state. The controller was deliberately chosen as single linear layer and could thus be optimized with Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen (2006))
This work demonstrated how a VAE can provide a versatile representation that can be utilized in reinforcement learning. In addition, such an approach allows to predict the subsequent encoded state. While these findings encourage the usage of VAE in conjunction with RL, this only possible in environments where the state space can be explored sufficiently by random policy. However, the policy can only discover important features after acquiring minimal level of skill, sampling the state space using random policy will not yield high-performing agents. Learning such features would only be possible if the VAE is continuously improved during policy training.
Another interesting combination of VAEs and RL was recently proposed by Yang et al. 2019)) with their so called creating this model was to train transparent, interpretable policy network. Usually, the 8- B-VAEs decoder trained to reconstruct the input based on the representation the encoder produced. In this work though, the decoders objective latent space vector z combined with an vettion-mapiin-veerr zero-padding match the latent spaces dimensionality. Inspecting the decoder estimates for 8 when varying one dimension of the latent space showed, that each dimension encoded a possible subsequent state that is likely to be encountered if the corresponding action from this dimension was taken. Unfortunately, the authors did not report any rewards they achieved on Breakout, hence it was not possible for us to compare model performances.
Variational Auto-Encoder (AC-VAE). Their motivation for
predict the subsequent state St+1- As input it got the which is the action vector a, with a
COMBINATION OF REINFORCEMENT AND REPRESENTATION LEARNING OBJECTIVES
In this section, we will first revisit the fundamentals of RL and VAES and discuss their different objective functions. Then, we propose a joint objective function that allows for joint training of both models using gradient descent based learning methods.
3.1 REINFORCEMENT LEARNING WITH POLICY OPTIMIZATION
RL tries to optimize a Markov Decision Process (MDP) (Bellman (957)) that is given by the tuple (S, A, r. r.p. 7). S denotes the state space, AA the action space and p S x R x S x A - [o. 1] the environment's dynamics function that, provided state-action pair, gives the state distribution for the successor state. r S x A + R is the reward and 7 E (0, 1) the scalar discount factor. The policy ma(ss)) is stochastic function that gives probability distribution over actions for state
3

---

Under review as a conference paper at ICLR 2021
0 denotes the policy's parameter vector which is typically subject to optimization. A trajectory T = (Su, ag ...,....... ST, ar) consisting of an alternating sequence of states and actions can be sampled in the environment, where T stands for the final timestep of the trajectory and a; [o(o,(s;).
The overarching goal of RL is to find a policy that maximizes the average collected reward over all trajectories. This can be expressed the optimization problem max maxEr~p(r) [ r(s, a)], which can also be written in terms of an optimal policy parameter vector 8" arg maxg By=-p(r) r(s,a)) When trying to optimize the policy directly be searching for 0-, policy optimization algorithms like A3C, Actor-Critic with Experience Replay (ACER) Waan al. (2016a)) Trust Region Policy Optimization (TRPO) (Schulman et al.] (2015a)) or Proximal Policy Optimiztiion (PPO) (Schulman et al. (2017) are commonly used. The fundamental idea behind policy optimization techniques to calculate gradients of the RI. objective with respect to the policy parameters:
VoJ = E T-)p(T)
Vo log r())
where we defined Σn_ r(s. a) r(r) for brevity. However, most policy optimization methods introduce heavy modifications to this vanilla gradient in order to achieve more stable policy updates. Throughout our work, we have used PPO as RI. algorithm because it is quite sample efficient and usually produces stable policy updates. For an in-depth description of PPO, we refer to our A. or the original work |Schulman et al. (2017).
3.2 LEARNING REPRESENTATION USING VARIATIONAL IUTO-ENCODERS
Kingma & Welling 2013) introduced the VAE as a method to perform Variational Inference (VI) (Jordan et al. (1999) using function approximators, e.g. deep neural networks. VI tries approxi- mate a distribution over the generative factors of a dataset which would otherwise involve calculating an intractable integral. The authors present an algorithm that utilizes the auto encoder framework, an unsupervised learning method which learns data encodings by reconstructing its input. There- fore, the input first compressed until it reaches a given size and is afterwards decompressed to its original size. When using deep neural networks, these transformations can be achieved by using for example fully connected or convolutional layers. order for the VAE to approximate a distribution over generative factors, the authors used the called "reparametrization trick". It allows for gradi- ent based optimization methods to be used in searching for the distribution parameters. For training the VAE, gradient based optimizer tries minimize the following loss:
CV AE (x, db vb) -D (g)(((z))))) [1)()))
E
(x|2)
(=(2)
(2)
with = = 1(,, ,) E) and E ~~ p(E)
where DxL denotes the Bullback-Leible Divergence (KL) (Kullback & Leibler (1511) of the ap- proximated distribution over generative factors produced by the encoder 2o 2 ( and some prior distribution p(z). The expectation is often referred to as reconstruction loss that typically caleu- lated on a per-pixel basis. Lastly, 1(11. o. <) is sampling function that is differentiable w.r.t. the distribution parameters, for example z = u + ore.
3.3 JOINT OBJECTIVE FUNCTION
Combining both loss functions such that both models can be trained at the same time is rather straight-forward. Adding both individual losses and using an optimizer such as ADAM (Kingma & Ba (2014) to minimize them is sufficient to achieve joint training. During backpropagation, gradi- ents from the policy and the VAE are combined in the latent space. Due to different topologies of the networks, gradient magnitudes differ significantly. Therefore, we introduced the hyperparameter which can be used to either amplify or dampen the gradients and we arrive at the following loss:
Cioint (00 02-10 0x) + AV Φ, v. B)
where PPG is some policy gradient algorithm's 's objective function. As mentioned before, we used PPO's loss CPPO (equation 4 in the appendix).
4

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
In order to test our model with the combined objective function given by equation 3 we have used the well-known benchmark of Atari Breakout. This environment has several properties that make appealing to use: it is easily understandable by humans, used often as a RL task and the conventional pre-trained methods fail mastering it. The ball is the most important feature that is required to be encoded in order to perform well, is heavily underrepresented (approximately 0.1% of the observation space). Therefore. the VAE's incentive to encode it is very low whereas our model succeeds in encoding it. In the following, we compare the pre-trained approach to two different continuously trained models that use the loss from equation
4.1 DATA COLLECTION AND PRE- PROCESSIO
The raw RGB image data produced by the environment has dimensionality of 210 160 pixels. We employ similar pre-precessing as Mnih et al. (2015), but instead of cropping the grey- scaled frames, we simply resize them 84 to those introduced in previous works with pre-trained VAE, we needed to construct dataset containing Breakout states. We used an already trained policy to collect total of 25, 000 frames, the approximate equivalent of 50 episodes.
pixels. As we will first train models similar
4.2 PRE-TRAINING THE VARIATIONAL AUTO-ENCODER
Our first model is based on those of the previously introduced works which involve isolated pre- training the VAE process. For the first model, PPO the encoder and decoder (shown in orange and red) pre- trained before policy training. During this phase, there no influence from the RL loss. Once the VAE training is finished, the decoder shown in red in Figure 2 is discarded completely. Later during
static dataset. Figure 2 shows the individual parts of the complete training
PPO
π
h2
h1
VAE
U
X
840...n-1
Z
f n-
x
Σ
Figure 2: Model combining PPO and VAE. Depending on the model configuration, the colored parts are trained differently. X is the VAE's 's imput and X the reconstructions, PPO receives the mean vectors U input and calculates distribution over actions Note that capital letters in the VAE to emphasize that we pass n frames at the same time when a policy is trained.
5

---

Under review as a conference paper at ICLR 2021
policy training, of the last n frames as input. Stacking allows us to incorporate temporal information and for the policy to predict the ball''s trajectory. By sharing the weights, ensure that the resulting encodings originate from the same function. U then represents the concatenated encodings the sequence. This weight sharing method has proven be more time efficient than query the encoder times and concatenate the results afterwards.
instances of the same encoder with shared weights that receive sequence
Prior to policy training, trained the VAE the dataset have collected before, with hyperpa- rameters from table Once pre-training was finished, we discarded the decoder weights and used the stacked encoder as input for the policy MLP. The MLP was then trained 10M steps with hy- perparameters from table During this training, the encoder weights were not changed by gradient updates anymore but remained fixed.
The second model we introduce is called PPO which has the same structure and hyperparame- ters as the first model. For this model, we also train the VAE in isolation first, however the encoder weights are not fixed anymore during policy training. Gradients from the RL objective are back propagated through the encoder, allowing it to learn throughout policy training. We hypothesize that features that are important for policy performance can incorporated in an already learned representation.
Figure 3 compares the median rewards of three rollouts with different random seeds for all models. PPO was once able achieve a reward 10 or higher, while PPO steadily improved its performance with final rewards well over 50. The learning curve of adpp shows that the model is able to learn how act in the environment, whereas PPO does The non-zero rewards from PPO fixe are similar to those of random agents in Breakout. From these results, we can assume that training the VAE in isolation on a static dataset for Breakout results deficient representation for RL. Therefore, using policy gradients to adapt an already learned representation can be beneficial environments where the VAE fails to encode task-relevant features.
4.3 JOINTLY LEARNING REPRESENTATION AND POLICY
The last model introduce, PPO VAE combines complete VAE with policy MLP that receives U, the concatenated state encodings, as input. As opposed to the first two models, all weights are initialized randomly before policy training and the VAE is pre-trained. For this procedure an
VAO
50
PPOfided
PPOmsapp
Median Reward
30
20
2 10
x 106
6 10
8 x 10
10 10
Steps
Figure 3: Reward of the three proposed models across three random seeds each. PPO lixe is not able achieve high rewards while the other two models consistently improve their performance.

---

Under review as a conference paper at ICLR 2021
already trained agent that gathers dataset for the VAE beforehand is not necessary. The decoder is trained exactly as in the isolated setting, meaning its gradients are also only computed using the VAE's loss function. During backpropagation, the gradients coming from Z and h1 are added together and passed through the encoder. This model has the same network configuration and hy- perparameters as the first two, with the only difference that we also evaluated different values for from the joint loss equation 3 (see A.33). For the results reported here, we chose K = 20. All hyperparameters can be found table
By simultaneously training representation and policy, we expect the VAE to learn task-relevant features from the beginning of training. This assumption is supported by the learning curve shown in figure 3 which compares PPO VAE to the previous two models. The curve shows steady increase in reward over the course of training with PPO VAB achieving slightly higher rewards than PPOndapt in the beginning. This characteristic changes after less than IM steps and from that point on PPOndup consistently outperforms PPO VAE This difference in performance is likely attributed to the fact, that in PPO VAL the decoder is trained throughout the complete training. While for PPO the latent space can be changed without restrictions, the decoder of PPO VAE constantly produces gradients do not contain information about the ball. Therefore, PPO VAE has less incentive to achieve higher rewards, which is reflected by its performance.
4.4 ANALYZING THE VALUE FUNCTION GRADIENTS
So far, the results imply that PPO VAE and aPO do indeed learn encodings of the ball. One difficulty when analyzing the represen- tation is, that the decoder still has incentive to reconstruct the ball, even if it is present in the latent space. In a work that enhances DQN algorithm (Wang al. (2016)), the authors visualized the Jacobian of the value function w.r.t. the input images. These visu- alizations showed which features or regions from the input space are considered as important in terms of future reward. As we also learn a value function, we did the same and visualized what our model considered important and what not.
In figure 4 we illustrate a pre-processed frame and added the val- ues of the Jacobian to the blue channel if the were greater than the mean value of the Jacobian. Only visualizing above-mean Ja- cobian values removes some noise in the blue channel makes the images much easier to interpret and only highlights regions of high relevance. We can clearly see, that the Jacobian has high values missing blocks as well as around the ball, meaning that these re- gions are considered to have high impact on future rewards. By visualizing the Jacobian we have confirmed that the policy gradi- ents encourage the VAE embed rask-relevant features.
Figure 4: The Jacobian of PPO's value function. Highlighted areas mean high importance in terms of future rewards. Note the high Ja- cobian values around the ball and the blocks.
CONCLUSION
This paper focused on the issue of pre-training VAEs with the purpose of learning a policy for a downstream task based on the VAE's representation. In many environments, the VAE has little to no incentive to learn task-relevant features if they are small in observation space. Another issue arises if the observation of these features depends on policy performance and as result, they are underrepresented in a dataset sampled by a random agent. both cases, fixing encoder weights during policy training prevents the VAE to learn these important features and policy performance will be underwhelming.
We carried out experiments on the popular RL benchmark Atari Breakout. The goal was to an- alyze whether policy gradients guide representation learning towards incorporating performance- critic features that VAE would not learn a pre-recorded dataset. First experiments confirmed, that the common pre-trained approach did not yield well-performing policies in this environment. Allowing the policy gradients to adapt encoder weights in two different models showed significant
7

---

Under review as a conference paper at ICLR 2021
improvements in terms of rewards. With policy gradients guiding the learned representation, agents consistently outperformed those that were trained on a fixed representation.
Our work verifies that the fundamental issue with pre-trained representations still exists and shows possible solutions RL scenarios. Nonetheless, future work can still explore a variety improve- ments to models. For once, training not only the encoder but also the decoder with RL gradients can improve interpretability of the VAE and enable to be used as generator again that also gen- erates task-relevant features. Another direction during joint training of VAE and policy. The goal there would be to maintain the desired latent space characteristics of VAEs while still encoding task-relevant features.
impose further restrictions on the latent space
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-
276. 1998.
OpenAl: Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappent, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1)33-20, 2020.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pp. 679-684, 1957.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-188) 1828, 2013.
Herve Bourlard and Y Kamp. Auto-association by multilayer perceptrons and singular value de- composition. Biological cybernetics, 59:291 4, 02 1988. doi: 10.1007/BF00332911
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-556, 2005.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolu- tion. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2450- 2462. Curran Associates, Inc.. 2018. URL http: /ttp://papers.... nips cips.cc/ppppp.....
Nikolaus Hansen. The ema evolution strategy: a comparing review. In Towards new evolutionary computation, pp. 75-102. Springer, 2006.
Irina Higgins, Loic Mathhey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mo- hamed, and Alexander Lerchner. Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv: (606.0v660005557777 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 3, 2017a.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1480-1490. JMLR. org, 2017b.
Geoffrey E Hinton and Richard S Zemel. Autoencoders, minimum description length and helmholtz free energy. pp. 3-10, 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
8

---

Under review as a conference paper at ICLR 2021
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-22 -33333,, 1999.
Sham M Kakade. A natural policy gradient. pp. 1531-1538, 2002.
Eukasz Kidziński, Sharada Prasanna Mohanty, Carmichael F Ong. Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, et al. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromuseu- loskeletal environments. In The NIPS' Competition: Building Intelligent Systems, pp. 121-153.
Springer, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: wetp::/42660000 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79 86, 1951.
Sascha Lange and Martin A. Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In IJCN,, pp. 1-8, IEEE, 2010. ISBN 978-1-4244 778-1-4244-6116-11 URL hattp://bblp. uni-trier de db http:///oooootto///tttoott fiann//iiinnnnninn O. html LangeR10
Yann Lecun. PhD thesis: Modeles connexionnistes de l'apprentissage (connectionist learning mod- els). Universite et M. Curie (Paris 6), June 1987.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436, 2015.
Hongzi Mao, Mohammad Alizadeh, Ishai Menache, and Srikanth Kandula. Resource management with deep reinforcement learning. In Proceedings of the 15th ACM Workshop on Hot Topics in Networks, pp. 50-56. ACM, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. pp. 1928-1937, 2016.
OpenAl Spinning Up- PPO. Spinning up explanation of ppo, 2018. URL https:// spinningup. openai. www/
html
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
John Schulman. Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. pp. 1889-1897, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1XXiv:5506.02338. 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0663 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panncershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 42777787)4484, 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1 110,, 2008

---

Under review as a conference paper at ICLR 2021
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yco, Alireza Makhzani. Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint 17x1:1770..04882 2017.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224. 2016a.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003, 2016b.
John Yang. Gyuejeong Lee, Simyung Chang, and Nojun Kwak. Towards governing agent's 's efficacy: Action-conditionaal 3-vae for deep transparent reinforcement learning. volume 101 of Proceedings of Machine Learning Research, pp. 32-47, Nagoya, Japan, 17-19 Nov 2019. PMLR. URL U //proceedings.. , mlr press v111 / 9a. html
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1
STABLE POLICY LEARNING WITH PROXIMAL POLICY OPTIMIZATION
Most actor-critic algorithms successfully reduce the variance of the policy gradient, however they show high variance in policy performance during learning and are at the same time very sample inefficient. Natural gradient (Amari (1998)) methods such TRPO from [Shulmaan et al. (2015a) greatly increase sample efficiency and learning robustness. Unfortunately, they are relatively com- plicated to implement and are computationally expensive as the require some second order approxi- mations. PPO (Schulman et al. (2017)) is family of policy gradient methods that form pessimistic estimates of the policy performance. By clipping and therefore restricting the policy updates, PPO prohibits too large of policy change as they have been found to be harmful to policy performance practice. PPO often combined with another type of advantage estimation (Schulman et al. (2015b)) that produces high accuracy advantage function estimates.
We define the PPO-Clip objective is defined as
JPPO (0k. 0x-1 = E min ((0)) (s, a), clip clip (,,),,1c,1++ (
(4)
s.t. 6MB < Surget
Te (a|s)
where o(0)
denotes the probability ratio of two policies.
5o2-1 1
This objective is motivated by the hard KL constraint that TRPO enforces on policy updates. Should policy update result in policy that deviates too much from its predecessor, TRPO performs a line search along the policy gradient direction that decreases the gradient magnitude. If the constraint is satisfied during the line search, the policy is updated using that smaller gradient step. Otherwise the update is rejected after a certain number of steps. This method requires to calculate the second order derivative of the KI. divergence, which is computationally costly. PPO uses its clipping objective to implicitly constrain the deviation of consecutive policies. In some settings, PPO still suffers from diverging policy updates (OpenAl Spinning Up-PPO (2018)), so we included a hard KL constrained on policy updates. The constraint can be checked after each mini-batch update analytically and is therefore not very computationally demanding.
PPO extends the policy gradient objective function from Kakade (2002). With the probability ratio o0)), we utilize importance sampling order to use samples collected with any policy to update our current one. Thereby we can use samples more often than in other algorithms, making PPO more sample efficient. Using importance sampling, we still have a correct gradient estimate. Com- bining the new objective with actor-critic methods yields algorithm [ K denotes the number of optimization epoch per set of trajectories and B denotes the mini-batch size. In the original paper, combined objective function is also given with:
C Ok 0k-1 0K) E [cr PPOO (Ok,0k-1) c2L (1x)) ( H((TO))
(5)
s.t. 8MB < Stanget
where (((TT.)) denotes the policy entropy. Encouraging the policy entropy not to decrease too much prohibits the policy from specializing on one action. As discussed in OpenAl Spinning Up- PPO 2018), there are two cases for (0x, f): either the advantage function was positive or negative. In case the advantage is positive, it can be written as:
JPPO 0) E min (o(0), (1 +c)) AT (s,
(6)
ATH (s, a) > O indicates that the action yields higher reward than other actions in this state, hence we want its probability no, (als) to increase. This increase clipped (1 + c) once Te, (als) > To.-, To._,(a|s] (a|s)(1 e). Note however, that updates that would worsen policy performance are neither clipped nor bound. If the the advantage is negative, it can expressed as:
JPP (Oz. 0) = E max (o(0), (1 e)) T (s)
This equation behaves conversely to A (s. a) 0 indicates that chose suboptimal action, thus we want to decrease its probability. Once To, (als) < no 1 (al s) - e). the max bounds the magnitude by which the action's probability can be decreased.
11

---

Under review as a conference paper at ICLR 2021
Algorithm Proximal Policy Optimisation with KL constraint
1: Initialize policy parameters On and value function parameters do
2: for = 0, 2,2,... do
3: Collect set of trajectories Dx = (7i) with Te, and compute R
4: бMB 0
5: for 0, 1.2,...K do
6:
for each mini- batch of size B in {Ti} do
7:
Update the policy by maximizing the PPO-Clip objective Minimize L the mini-batch
8:
9:
end for end for
11: if' бMB > óurget then
0x+1 0k
13: end if
end for
A.2 HYPERPARAMETER TABLES
Parameter
Value
epochs
100
batch size
128
input size
(84, 84, 1)
optimizer
ADAM
learning rate
10-1
encoder
Conv2D 32 x 4 x 4 (stride 64 x 4 (stride 2)
FC 512 (ReLU)
latents
20 (linear)
decoder
FC 512 (ReLU) 64 x (stride 2) 32 x 4 X 4 (stride
2) Conv2D Transpose
Table 1: Hyperparameter table for VAE training on Breakout
Parameter
Value
Parameter
Value
timesteps
X 10?
timesteps
x 107
environments
16
environments
16
batch size
32
batch size
32
mmxx
2048
mmx
2048
K
10
K
10
1.0
Ci
1.0
0.5
C2
0.5
0.0
C3
0.0
0.99
X
0.99
0.95
0.95
network
FC 64 (tanh)
network
FC 64 (tanh)
FC 64 (tanh)
FC 64 (tanh)
optimizer
ADAM
optimizer learning rate
ADAM
learning rate
3 x 10-1
3 x 10-4
K
(1, 10, 20)
Table Policy hyperparameters of PPO
Table 3:
Policy hyperparameter table of
and adppt
PPO VAE
12

---

Under review as a conference paper at ICLR 2021
A.3 CHOOSING APPROPRIATE VALUES FOR
In equation 3 we introduced the hyperparameter k. to balance VAE and PPO gradients. We found empirically, that tuning is straight forward and requires only few trials. In order simplify the search for K, one can evaluate gradient magnitudes the different losses at the point where they are merged at U. Our experiments showed PPO's gradients to be significantly smaller, thus scaling up the loss function was appropriate. This will likely differ if the networks are configured differently. Increasing from 10 led considerably higher rewards, however the difference in performance was small when increasing x further to 20. Therefore, we chose 20 in our reported model performances.
1
10
40
20
Median Reward
30
20
10
0
2 x 106
4 x 106
x 106
8 x 10
10 x 106
Steps
Figure 5: Performance comparison PPO VAE with different values for
13


Reviews:
[{"review_id": "xsSRr2EEso", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focused on the issue of learning a policy for a given task using the learned representations a pre-trained VAE. The authors visualize that using a learned latent space of a pre-trained VAE is not good enough for learning policies and propose a solution for this problem: back-propagate gradient policies through the VAE encoder. The authors proposed two versions on this method, one with pre-training and one fully online.\n\nThis paper suffers from fundamental issues. The biggest one in my opinion is its poor literature review. There is a bold statement in the introduction which shapes up the problem that the paper is trying to address and I quote: \"Recently, many such approaches employed the VAE framework which aims to learn a smooth representation of its domain. Most of these approaches follow the same pattern: First, they build a dataset of states from the RL environment. Second, they train the VAE on this static dataset and lastly train the RL mode using the VAE’s representation.\" But the authors do not provide any reference for this. The related work section is also more focused on VAEs rather than its combination with RL. And as one would expect, there is no comparison with previous work either. Joint training of a VAE with a reward predictor or a policy network is not new. There is a massive body of research doing this. PlaNet, Dreamer by Danijar Hafner et al and their variations being the latest ones. \n\nThe experiments are also done on a single Atari game (Breakout) which is not sufficient to demonstrate the capability of the proposed method in a variety of tasks. And again, with no comparison to ANY of the previous work.\n\nOverall, the paper is suffering from multiple fundamental issues which makes it hard to be accepted as a scientific contribution in a top conference. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YtueAiI_rUN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for reinforcement learning with representations learned from a VAE. The VAE is used to encode the states (images) and the mean of the posterior is used as input to the policy. The VAE is trained using the variational lower bound and the policy is optimized using PPO. The model can be jointly trained with VAE and RL losses from scratch, or the VAE can be pre-trained and fixed, or pre-trained and finetuned.\n\nStrengths:\n\n+ This method is fairly simple and straightforward \n+ The results suggest the representations learned can capture small details such as the ball which would normally not be reconstructed under the VAE loss\n+ Pre-training the VAE seems to have promising results suggesting the pre-trained representations can improve learning\n\nConcerns:\n- This paper seems like a straightforward mash-up of VAE and PPO. It’s not clear whether the main contribution is particularly novel.\n- No comparisons are made to non-VAE methods (e.g. a standard convolutional architecture). It’s not clear how much of the method’s performance can be attributed to the representation learning part vs just running a standard RL method from scratch\n- Limited evaluation: there is one training curve on one Atari game, and there is one visualization of the jacobian. I would like to see a more thorough evaluation with comparisons to other baselines outside of this paper\n\nOverall this paper is working on an interesting research direction, however I cannot recommend to accept it in its current state. I would like to see stronger empirical results or a more novel formulation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RuR3tUMRBi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First off, I would like to thank the authors of \"Guiding Representation Learning in Deep Generative Models with Policy Gradients\" for their neat, well written submission.  \n\n\nSummary of the paper \n--------------------------------\nThe paper discusses the problem of representation learning in deep RL, with a focus on the effects of pretraing the state embedding function on agent performance. More specifically, the authors consider the case where a variational autoencoder (VAE) is to be trained to embed the state, and three training regimes are compared:\n1. The VAE is pretrained on a dataset of episodes - collected separately with a pretrained agent. It weights are then frozen, and a policy is trained online.\n2.  The VAE is pretrained on a dataset of episodes - collected separately with a pretrained agent. Its weights are then refined durign the training a policy online (gradients of the policy are backpropped into the VAE encoder).\n3. The VAE and the policy are traiend end-to-end online jointly.\n\nThe authors' main claim is not only possible, but that pretraining without refinemenet / joint optimization can potentially result in the VAE discarding essential details needed for policy/value optimization; the claim is supported by: \n1. Comparing the three training regimes on Breakout using a PPO;\n2. Providing a visualization of the PPO value function highlighting how in the regimes in 2 and 3 details of the game that intuitively are relevant for good performance are indeed picked up during training.\n\n\nAssessment\n-----------------\n\n-- The positives --\n\nThe paper is well structured and mostly well written - it was in fact a pleasure to read apart from a few confusing bits that I will point to later in the review. \nThe results are a bit thin in quantity, but to the point! The goal and the description of the experiments are clear and detailed, the figures clear and easy to understand.\n\n-- The concerns --\n\n1. The broad topic addressed in the paper (model training in RL) is interesting, but the contributions are not novel enough for this venue.\n\n* First contrbution in Section 1: \"we show that combined learning is possible and that it yields good performing policies\"\nTraining models (variational or not) alongside a policy (in this or similar settings) is commonplace, and we now they can be trained. Example of much richer (action-conditional) and larger scale models than those presented in this submission are for example MuZero (https://arxiv.org/abs/1911.08265), or belief state models (one example https://arxiv.org/pdf/1906.09237.pdf).\n\n* Second contribtion of Section 2: \"we show that jointly trained representations incorporate additional, task-specific information which allows a RL agent to achieve higher rewards then if it was trained on a static representation\" \nThis statement is quite unsurprising, as we are literally training the encoder to retain that information and provide it to the policy/value functions. Furthermore, the authors do not actually strictly verify that the fully pretrained model doesn't encode the ball location - that is, is the VAE decoder failing to reconstruct or the encoder failing to capture the ball localtion? What happens if you apply the method in Wang et al to PPO^{fixed}?  \n \n2. Whilst running on 1 environment with 1 training setup helps making the point, I can't help wondering if the comparisons between PPO^{VAE} and PPO^{adapt} could have yielded different results if multiple training tricks had been compared, e.g. something like using target networks for the VAE, using replay - which would have also made the comparison of the two method more fair to PPO^{VAE} since PPO^{adapt} has access to more compute.  \nThe problem of how to use a pretrained model is indeed very exciting and I wish they authors had expanded more this section of the paper.\n\nSuggestions \n-- \n\nTo make the paper more interesting, the authors could consider framing the paper in the broader context of unsupervised pretraining in DL (e.g. https://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf) .\n\nPlease consider clarifying: \n\n- Section 4.2 \"This weight sharing method has proven to be more time efficient than to query the encoder n times\nand concatenate the results afterwards.\" --> I find this a bit confusing - isn't the graph compiler going to optimized this out? Can you elaborate?\n\n- Section 4.3. \"While for PPOadapt, the latent space can be changed without restrictions, the decoder of PPOVAE constantly produces gradients do not contain information about the ball. \" --> Sentence is badly formed, please fix. \n\n- Section 4.3 \"Therefore, PPOVAE has less incentive to achieve higher rewards, which is reflected by its performance.\" --> I do not see this, and would need to see an experiement to show/support this claim. Please point me to the relevant sections of the paper if I missed them.\n\n- Conclusion \"Our work verifies that the fundamental issue with pre-trained representations still exists and shows possible solutions in RL scenarios.\"  --> Why should the fundamental issue should have gone away? Also, the paper doesn't seem to suggest any solutions to improving the bridge between unsupervised pretraining and online training. Please do correct me if I am mistaken.\n\n\nI look forward to the rebuttal, hoping the author will help me reassess the paper in a more positive light. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9CmBPt0tOlP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "My recommendation would be to reject this paper for reasons which I will outline below.\n\nPros:\nThe problem of learning good features for RL is an important one to tackle and has application in many important tasks with complex input spaces like vision. The authors explain why this is important and I encourage them to continue to explore this line of research albeit more thoroughly as detailed below. \n\nCons:\nOne big issue with the paper is in its lack of experimental rigour. This can be improved by considering more than just the single Breakout task (the Atari suite itself contains many example tasks ripe for research) and also by comparing against more competitive baselines.\nFor instance the authors mention the AC-VAE (Yang et al. [1]) but do not compare against their method since ‘they do not report rewards achieved on Breakout’. To me this is not a good enough reason to not have a comparison. Instead I encourage the authors to reimplement the alternative approach and compare it on the tasks they consider.\nWhile the paper lists some early work on representation learning and learning features for RL they do not mention or compare against many other works (e.g. Jaderberg et. al. [2] and Achiam et al [3] to name a few). A thorough literature survey helps place the work in the broader context of the field and allows the work to be evaluated more comprehensively. \n\nFinally there are a number of minor points in the presentation which did not affect my final decision but would be good to fix.  There is a typo where 'Introduction' is spelled 'Introdction'. Reference capitalisation is inconsistent - in some places the paper refers to 'Figure 1' but elsewhere there is 'figure 3' and 'figure 4' (on page 7). If references are to be capitalised, 'table 1' would also have to be replaced with 'Table 1'.\n\nReferences: \n[1] John Yang, Gyuejeong Lee, Simyung Chang, and Nojun Kwak. Towards governing agent’s efficacy: Action-conditional β-vae for deep transparent reinforcement learning. volume 101 of Proceedings of Machine Learning Research, pp. 32–47, Nagoya, Japan, 17–19 Nov 2019. PMLR. URL http: //proceedings.mlr.press/v101/yang19a.html.\n[2]. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016.\n[3] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Due to uniformly unfavourable reviews and lack of author engagement in the discussion period, this paper is rejected.
Decision: Reject