Model Based Reinforcement Learning for Atari
Authors: Łukasz Kaiser
Mohammad Babaeizadeh
Piotr Miłos
Błażej Osiński
Roy H Campbell
Konrad Czechowski
Dumitru Erhan
Chelsea Finn
Piotr Kozakowski
Sergey Levine
Afroz Mohiuddin
Ryan Sepassi
George Tucker
Henryk Michalewski
Venue:  (2020)
Keywords: reinforcement learning
model based rl
video prediction model
atari

Abstract: Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observations. However, this typically requires very large amounts of interaction -- substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games in low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.

Published as a conference paper at ICLR 2020
MODEL BASED REINFORCEMENT LEARNING FOR ATARI
Lukasz Kaiser' Baaiserr,,,MMannnn Mohammad Babaeizadeh Piotr Milos jBaiee Osiński Roy H. Campbell', Konrad Czechowski', Dumitru Erhan', Chelsea Finn 1.6 Piotr Kozakowski", Sergey Levine', Afroz Mohiuddin' Ryan Sepassi', George Tucker', Henryk Michalewski Google Brain, ddeepsense.ai ai, Instittte of Mathematics of the Polish Academy of Sciences,
-Baazej
Faculty of Mathematies, Informatics and Mechanics, University of Warsaw, University of Illinois Urbana-Champaign, Stannfrrd University
ABSTRACT
Model-free reinforcement learning (RL) can used to learn effective policies for complex tasks, such Atari games, even from image observations. However, this typically requires very large amounts of interaction substantially more, in fact, than human would need learn the same games. How people learn quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a comparison of several model architectures, including novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on range of Atari games low data regime of 100k interactions between the agent and the environment, which corresponds to two hours of real-time play. In most games SimPLe outperforms state-of-the-art model-free algorithms, in some games by over an order of magnitude.
INTRODUCTION
Human players learn to play Atari games in minutes (Tsividis al., (017). However, of the best model-free reinforcement learning algorithms require tens or hundreds millions time steps the equivalent of several weeks of training in real time. How is it that humans learn these games so much faster? Perhaps part of the puzzle is that humans possess an intuitive understanding of the physical processes that are represented the game: we know that planes can fly, balls can roll, and bullets can destroy aliens. We can therefore predict the outcomes of our actions. In this paper, we explore how learned video models can enable learning the Atari Learning Environment (ALE) benchmark Bellemare et al. (2015); Machado et al. (2018) with budget restricted to 100K time steps rruuhhly to two hours play time.
Although prior works have proposed training predictive models for next-frame, future-frame, as well as combined future-frame and reward predictions Atari games (Oh et al.) (2015); Chiappa al.. (2017); Leibfried et al. (2016)), prior work has successfully demonstrated model-based control via predictive models that achieve competitive results with model-free RL. Indeed, a recent survey (Section 7.2 in Machado et al. (2018)) this was formulated the following challenge: "So far, there has been no clear demonstration of successful planning with a learned model in the ALE".
Using models of environments, informally giving the agent ability to predict its future, has a fundamental appeal for reinforcement learning. The spectrum of possible applications vast, including learning policies from the model (Watter al., 2015 Finn al.) 2016 Finn & Levine 2017 Ebert et al. 2017 Hafner et al. 2019 Piergiovanni et al. 2018 Rybkin et al. 2018 Sutton & Barto
'Equal contribution, authors listed in random order. BO performed the work partially during an internship Google Brain. Correspondence osinski @mi muw.edu.pl

---

Published as a conference paper at ICLR 2020
Interaction
Oouer Qhver rertoosoo
World Model
Pollcy
Agent Eval Evalbation
Aggntt TV Training
World
actoote mote Hoaly
Observations
World Model
Traliding
L.
RL.
Pollcy
Figure I: Main loop of SimPLe. 1) the agent starts interacting with the real environment following the latest policy (initialized to random). 2) the collected observations will be used train (update) the current world model. 3) the agent updates the policy by acting inside the world model. The new policy will be evaluated measure the performance of the agent as well as collecting more data (buck 1). Note that world model truining is self-supervised for the observed states and supervised for the reward.
2017
Chapter 8). capturing important details of the scene (Ha Schmidhuber 2018 encouraging
exploration (Oh et al.) (2015), creating intrinsic motivation (Schmidhuber) (0100) counterfactual reasoning (Buesing et al.j One of the exciting benefits of model-based learning the promise to substantially improve sample efficiency of deep reinforcement learning (see Chapter 8 in Sutton &
Barto (2077))
Our work advances the state-of of-the-art in model-based reinforcement learning by introducing system that, to our knowledge, is the first to successfully handle variety of challenging games the ALE benchmark. To that end, we experiment with several stochastic video prediction techniques, including a novel model based on discrete latent variables. We present an approach, called Simulated Policy Learning (SimPLe), that utilizes these video prediction techniques and trains policy to play the game within the learned model. With several iterations of dataset aggregation, where the policy is deployed collect more data in the original game, we learn a policy that, for many games, successfully plays the game in the real environment (see videos on the project webpage https://goo gl/itykP8).
In our empirical evaluation, we find that SimPLe is significantly more sample-efficient than a highly tuned version of the state-of-the-art Rainbow algorithm (Hessel al., (018) on almost all games. In particular, in low data regime of 100k samples, on more than half of the games, our method achieves score which Rainbow requires at least twice as many samples. In the best case of Freeway. our method is more than 10x more sample-effciient see Figure 3 Since the publication of the first preprint of this work, it has been shown in van Hasselt al. (2019); Kielak (2020) that Rainbow can be tuned to have better results in low data regime. The results are par with SimPLe both of the model-free methods are better in games, while SimPLe better the other out of the total 26 games tested (note that in Section 4.2 van Hasselt et al. (2019) compares with the results of our first preprint, later improved).
RELATED WORK
Atari games gained prominence benchmark for reinforcement learning with the introduction of
the Arcade Learning Environment (ALE) [Bellemare et al. (2015). The combination of reinforcement
learning and deep models then enabled RL algorithms learn play Atari games directly from
images of the game screen, using variants of the DQN algorithm (Mnih et al. 2013 (0133|200 2015 Hessel
al. 2018) and actor-critic algorithms (Mnih et al., 2016, Schulman et al.) 2017 Babaeizadeh et al.
2017b Wu et al. 2017| Espeholt et al., 2018). The most successful methods in this domain remain
model-frre algorithms (Hessel et al. 2018 Espeholt et al. 2018). Although the sample complexity of these methods has substantially improved recently, it remains far higher than the amount of experience required for human players to learn each game (Tsividis et al., 2017). this work, we aim to learn Atari games with budget of just 100K agent steps (400K frames), corresponding to about two hours of play time. Prior methods are generally not evaluated in this regime, and we therefore optimized Rainbow (Hessel al. 2018) for optimal performance 1M steps, Appendix E for details.
2

---

Published as a conference paper at ICLR 2020
Oh et al. 2015) and Chiappa et al. (2017) show that learnigg predictive models of Atari 2600 environments is possible using appropriately chosen deep learning architectures. Impressively, in some cases the predictions maintain low L2 error over timespans of hundreds of steps. As learned simulators of Atari environments are core ingredients of our approach, in many aspects our work is motivated by Oh al. (2015) and and[Chiappa in the context of learning how to play the game well and positively verify that learned simulators can be used train a policy useful in original environments. An important step in this direction was made by Leibfried et al. 2016). which extends the work of of et al. (0155) by including reward prediction, but does not use the model to learn policies that play the games. Most of these approaches, including ours, encode knowledge of the game in implicit way. Unlike this, there are works in which modeling is more explicit, for example Ersen & Sariel (2014) uses testbed of the Incredible Machines to learn objects behaviors and their interactions. Similarly Guzdial et al. (2017) learns an engine predicting interactions of predefined set of sprites in the domain of Super Mario Bros.
al. (2017). however we focus using video prediction
Perhaps surprisingly, there is virtually no work on model-based RL in video games from images. Notable exceptions are the works of Oh et al. (2017). Sodhani et al. (2019), Ha & Schmidhuber (2018), Holland et al. (2018). Leibfried et al. (2018) and Azizzadenesheli et al. (2018). [Oh et al. (217 use a model of rewards to augment model-free -ree learning with good results on a number of Atari games. However, this method does not actually aim to model or predict future frames, and achieves clear but relatively modest gains in efficiency. [Sodhani et al. (2019) proposes learning a model consistent with RNN policy which helps to train policies that are more powerful than their model-free baseline. Ha & Schmidhuber (2018) present a way to compose a variational autoencoder with a recurrent neural network into an architecture that is successfully evaluated in the VizDoom environment and on a 2D racing game. The training procedure is similar to Algorithm| [ but only one iteration of the loop needed as the environments are simple enough to be fully explored with random exploration. Similarly, Alaniz (2018) utilizes a transition model with Monte Carlo tree search to solve a block- poaciig task in Minecraft. [Holland et al. (2018) use a variant of Dyna (Sutton) 199T) to learn model of the environment and generate experience for policy training in the context of Atari games. Using six Atari games as a benchmark Holland et al. (2018) measure the impact of planning shapes on performance of the Dyna-DQN algorithm and include ablations comparing scores obtained with perfect and imperfect models. Our method achieves around 330% of the Dyna-DQN score on Asterix, 120% Q-Bert, 150% Seaquest and 80% on Ms. Pac-Man. Azizzadenesheli iet al. (2018) propose an algorithm called Generative Adversarial Tree Search (GATS) and for five Atari games train a GAN-based world model along with a Q-function. Azizzadeneshelli et al. (008) primarily discuss various failure modes of the GATS algorithm. Our method achieves around 64
times the score of GATS on Pong and 10 times on Breakout.
Outside of games, model-based reinforcement learning has been investigated at length for applications such as robotics (Deisenroth et al. 2013). Though most of such works not use image observations,
several recent works have incorporated images into real-world (Finn et al.] 2016 Finn & Levine
2017 Babaeizadeh et al. 2017a| Ebert et al. 2017 Piergiovanni et al. 2018 Paxton et al. 2019
Rybkin et al. [2018] Ebert et al.) 2018) and simulated (Wttter et al.) [2055] Hafner et al. 2019) robotic control. Our video models of Atari environments described in Section are motivated by models
developed in the context of robotics. Another source of inspiration are discrete autoencoders proposed
by van den Oord et al. (2077) and Kaiser & Bengio (2018).
The structure of the model-based RL algorithm that we employ consists of alternating between learning a model, and then using this model to optimize a policy with model-free reinforcement learning. Variants of this basic algorithm have been proposed in number of prior works, starting from Dyna Q SSttoon| (1991) to more recent methods that incorporate deep networks Heess et al. (2015); Feinberg et al. (2018) Kalweit & Boedecker (2017) Kurutach et al. (2018).
'Copparison with Dyna-DQu DQN and GATS is based on random- normalized scores achieved at 100K interac- tions. Those are approximate. as the authors Dyna-DQQ DQN and GATS have not provided tabular results. Authors of Dyna-DQN also report scores on two games which do not consider: Beam Rider and Space Invaders. For both games the reported scores are close to random scores, as are GATS scores on Asterix.
3

---

Published as a conference paper at ICLR 2020
Piec's Enbedtiing
Decrettztion
Legendi
demar
recurrent
Riil
Next Frame
Predicted Frame
Frames
Plali Embrdding
Legis
4s4
404
404
404
AsA
4u4
4a4
464
414
4e4
464
Pui
Predicted Rewaed
muttipiccatoo
Figure 2: Architecture of the proposed stochastic model with discrete latent. The input to the model is four stacked frames {as well as the action selected by the agent) while the output is the next predicted frame and expected reward. Input pixels and action are embedded using fully connected layers, and there is per-pixel softmax (256 colors) in the output. This model has two main components. First, the bottom part of the network which consists of a skip-comnected convolutional encoder and decoder. To condition the output on the actions of the agent, the ouiput of each layer in the decoder is multiplied with the (learned) embedded action. Second part of the model is convolational inference network which approximates the posterior given the next frume, similarly to Babaeizadeh el al. (2017a). At training time, the sampled latent values from the approximated bits. keep the model differentiable, discretization following Kaiser & Bengio (2018). A third LSTM based network is trained to approximate each bit given the previous ones. At inference time, the latent bits are predicted auto outo-regressivee regressively using this network. The deterministic model has the same architecture this figure but without the inference network.
posterior will be discretized
backpropagation bypasses the
3 SIMULATED POLICY LEARNING (SIMPLE)
Reinforcement learning is formalized in Markov decision processes (MDP). An MDP is defined as a tuple (8, A, P, P.r.7). where S is state space, AA is a set of actions available to an agent, P the unknown transition kernel, r is the reward function and 7 E (0, 1) is the discount factor. In this work we refer to MDPs as environments and assume that environments do not provide direct access to the state (i.e., the RAM of Atari 2600 emulator). Instead we visual observations, typically 210 160 RGB images. A single image does not determine the state. In order to reduce environment's partial observability, we stack four consecutive frames and use it as the observation. A reinforcement learning agent interacts with the MDP by issuing actions according to a policy. Formally, policy x is a mapping from states probability distributions over A. The quality of policy is measured by the
value function Ex Itto 7'rc+1|so = s). which for starting state estimates the total discounted reward gathered by the agent.
In Atari 2600 games our goal is to find a policy which maximizes the value function from the beginning of the game. Crucially, apart from an Atari 2600 emu- lator environment env we will use a neural network simulated environment env' which we call world model and describe in detail in Section 4 The en- vironment env' shares the action space and reward space with env and produces visual observations in the same format, as it will be trained to mimic env. Our principal aim is to train policy π using a sim- ulated environment env' so that π achieves good per- formance in the original environment env. In this training process we aim to use as few interactions with env possible. The initial data to train env' comes from random rollouts of enc. As this is unlikely to capture all aspects of env, we use the iterative method presented in Algorithm
Algorithm 1: Pseudocode for SimPLe
Initialize policy π
Initialize model parameters 0 of env' Initialize empty set D while not done do
D collect observations from real env.
D + U COLLECT(eRe, .т)
> update model using collected data.
0 TRAIN SUPERVISED(cn/) D)
D update policy using world model.
< --RRIIN TRAIN_RL(TT,, RL(,, env')
end while
4

---

Published as a conference paper at ICLR 2020
4 WORLD MODELS
In search for effective world model experimented with various architectures, both new and modified versions of existing ones. This search resulted in a novel stochastic video prediction model (visualized in Figure 2 which achieved superior results compared other previously proposed models. In this section, we describe the details of this architecture and the rationale behind our design decisions. In Section 6 we compare the performance of these models.
Deterministic Model. Our basic architecture, presented as part of Figure 2 resembles the con- volutional feedforward network from Oh et al. (2015). The input X consists of four consecutive game frames and an action Stacked convolution layers process the visual input. The actions are one-hot-encodes and embedded in a vector which is multiplied channel-wis with the output of the convolutional layers. The network outputs the next frame of the game and the value of the reward.
In our experiments, we varied details of the architecture above. In most cases, we use a stack of four convolutional layers with 64 filters followed by three dense layers (the first two have 1024 neurons). The dense layers are concatenated with 64 dimensional vector with a learnable action embedding. Next, three deconvolutional layers of 64 filters follow. An additional deconvolutional layer outputs an image of the original 105 x 80 size. The number of filters is either or x 256. In the first case, the output is real-valucd approximation of pixel's 's RGB value. In the second case, filters are followed by softmax producing a probability distribution on the color space. The reward is predicted by a softmax attached to the last fully connected layer. We used dropout equal to 0.2 and layer normalization.
Loss functions. The visual output of our networks is either one float per pixel/channel or the categorical 256-dimensional softmax. In both cases, used the clipped loss max Loss, C*) for constant C. We found that clipping was crucial for improving the models (measured with the correct reward predictions per sequence metric and successful training using Algorithm D) We conjecture that clipping substantially decreases the magnitude of gradients stemming from fine-tuning of big areas of background consequently letting the optimization process concentrate on small but important areas (e.g. the ball in Pong). In our experiments, we set C = 10 for L-2 loss on pixel values and to C' = 0.03 for softmax loss. Note that this means that when the level of confidence about the correct pixel value exceeds 97% (as - In(0.97) = 0.03) we get no gradients from that pixel any longer.
Scheduled sampling. The model env' consumes its own predictions from previous steps and due to compounding errors, the model may drift out of the area of its applicability. Following [engio et al. (2015): Venkatraman et al. (2016), we mitigate this problem by randomly replacing in training some frames of the input X by the prediction from the previous step while linearly increasing the mixing probability to 100% around the middle of the first iteration of the training loop.
Stochastic Models. A stochastic model can be used to deal with limited horizon of past observed frames as well as sprites occlusion and flickering which results to higher quality predictions. Inspired by [Babaeizadeh et al. (2017)), we tried a variational autoencoder (Kingma & Welling. [014] to model the stochasticity of the environment. In this model, an additional network receives the input frames as well as the future target frame input and approximates the distribution of the posterior. At each timestep. a latent value 2 is sampled from this distribution and passed as input to the original predictive model. At test time, the latent values are sampled from an assumed prior A( I). To match the assumed prior and the approximate, we use the Kullback-Leibe divergence term as an additional loss term (Babaeizadeh et 2017a).
We noticed two major issues with the above model. First, the weight of the KL divergence loss term is game dependent, which is not practical if one wants to deal with a broad portfolio of Atari games. Second, this weight is usually very small number in the range of [10-3, 10-5) which means that the approximated posterior can diverge significantly from the assumed prior. This can result in previously unseen latent values at inference time that lead to poor predictions. We address these issues by utilizing a discrete latent variable similar to Kaiser & Bengio (2018).
As visualized in Figure 2 the proposed stochastic model with discrete latent variables discretizes the latent values into bits (zeros and ones) while training an auxiliary LSTM- Hochreiter & Schmidhuber (197)) recurrent network to predict these bits autoregressively. At inference time, the latent bits will be generated by this auxiliary network in contrast to sampling from a prior. To make the predictive model more robust to unseen latent bits, we add uniform noise to approximated latent
5

---

Published as a conference paper at ICLR 2020
values before discretization and apply dropout (Srivastava al. 2014) bits after discretization. More details about the architecture is in Appendix
5 POLICY TRAINING
We will now describe the details of SimPLe, outlined in Algorithm [ In step 6 we use the proximal policy optimization (PPO) algorithm (Schulman et al. (017)) with y = 0.95. The algorithm generates rollouts in the simulated environment env' and uses them to improve policy æ. The fundamental difficulty lays in imperfections of the model compounding over time. To mitigate this problem we use short rollouts of env'. Typically every N 50 steps we uniformly sample the starting state from the ground ground-truth buffer D and restart env' (for experiments with the value of and N see Section 6.4). Using short rollouts may have a degrading effect as the PPO algorithm does not have way to infer effects longer than the rollout length. To ease this problem, in the last step of rollout we add to the reward the evaluation of the value function. Training with multiple iterations re-starting from trajectories gathered in the real environment is new to our knowledge. It was inspired by the classical Dyna-Q Q algorithm and, notably, in the Atari domain no comparable results have been achieved.
The main loop in Algorithm T is iterated 15 times (cf. Section 6.4). The world model is trained for 45K steps in the first iteration and for 15K steps in each of the following ones. Shorter training in later iterations does not degrade the performance because the world model after first iteration captures already part of the game dynamics and only needs to be extended to novel situations.
In each of the iterations, the agent is trained inside the latest world model using PPO. In every PPO epoch we used 16 parallel agents collecting 25, 50 or 100 steps from the simulated environment env' (see Section [6.4 for ablations). The number of PPO epochs is 1000, where equals to I in all passes except last one (where = = 3) and two passes number 8 and 12 (where = = 2). This gives 800K-> interactions with the simulated environment in each of the loop passes. In the process of training the agent performs 15.2M interactions with the simulated environment enc'.
6 EXPERIMENTS
We evaluate SimPLe on suite of Atari games from Atari Learning Environment (ALE) benchmark. In our experiments, the training loop is repeated for 15 iterations, with 6400 interactions with the environment collected each iteration. We apply a standard pre-processing for Atari games: frame skip equal to 4, that is every action is repeated 4 times. The frames are down-scaled by factor of 2.
Because some data is collected before the first iteration of the loop, altogether 6400 16 = 102, 400 interactions with the Atari environment are used during training. This is equivalent to 409, 600 frames from the Atari game (114 minutes at 60 FPS). At every iteration, the latest policy trained under the learned model is used to collect data in the real environment env. The data is also directly used to train the policy with PPO. Due to vast difference between number of training data from simulated environment and real environment (15M vs 100K) the impact of the latter policy is negligible.
We evaluate our method on 26 games selected on the basis of being solvable with existing of the-art model-free deep RL algorithms] which in our comparisons Rainbow Hessel et al. (2018) and PPO |Schulman et al. (0017). For Rainbow, we used the implementation from the Dopamine package and spent considerable time tuning it for sample efficiency (see Appendix
For visualization of all experiments see Figure 3 It can be seen that our method more sample-efficient than a highly tuned Rainbow baseline on almost all games, requires less than half of the samples on more than half of the games and, on Freeway, more than 10x more sample-efficient Our method outperforms PPO by an even larger margin. We also compare our method with fixed score baselines (for different baselines) rather than counting how many steps are required to match our score, see Figure for the results. For the
and for a summary see
SSpecifically, for the final evaluation we selected games which achieved non-random results using method the Rainbow algorithm using 100K interactions.

---

Published as a conference paper at ICLR 2020
Trewey
crary c Imber
crayydt moxr
pump
sung master
porg
trostbite
seaquest
boxing
boxing
breakout
bcbertx
1Ten
anequant
chapper command
gopher
a5cau E
battie
Kangar00
kung tu masier
n.
jemeddonn
hest
down
heo
bonk Peg:
battle zone
tangarco
privtteaaye
private
10 12 15
36
22
Figure 3: Comparison with Rainbow and PPO. Each bar illustrates the number of interactions with environment required by Rainbow (left) or PPO (right) to achieve the same score as our method (SimPLe). The red line indicates the 100K interactions threshold which is used by the our method.
qualitative analysis of performance on different games see Appendix B The source code is available as part of the Tensor2Tensor library and includes instructions on how to run the experiments
6.1 SAMPLE EFFICIENCY
The primary evaluation in our experiments studies the sample efficiency of SimPLe, in comparison with state-of-the-aa art model-free deep RL methods in the literature. To that end, we compare with Rainbow (Hessel et al. 2018 Castro et al. 20188. which represents the state-of-the-art Q-learning method for Atari games, and PPO (Schulman et al. (2017), model-free policy gradient algorithm (see Appendix for details of tuning of Rainbow and PPO). The results of the comparison are presented in Figure 3 For each game, we plot the number of time steps needed for either Rainbow or PPO to reach the same score that our method reaches after 100K interaction steps. The red line indicates 100K steps: any bar larger than this indicates a game where the model-free method required more steps. SimPLe outperforms the model-free algorithms in terms of learning speed on nearly all of the games, and in the case of few games, does so by over order of magnitude. For some games, it reaches the same performance that our PPO implementation reaches at 10M steps. This indicates that model-based reinforcement learning provides an effective approach learning Atari games, at fraction of the sample complexity.
The results in these figures are generated by averaging 5 runs for each game. The model-based agent is better than a random policy for all the games except Bank Heist. Interestingly, we observed that the best of the 5 runs was often significantly better. For 6 of the games, it exceeds the average human score (as reported in Table 3 Pohlen et al. (218)). This suggests that further stabilizing SimPLe should improve its performance, indicating an important direction for future work. some cases during training we observed high variance of the results during each step of the loop. There are a number of possible reasons, such as mutual interactions of the policy training and the supervised training or domain mismatch between the model and the real environment. We present detailed numerical results, including best scores and standard deviations, in Appendix D

---

Published as a conference paper at ICLR 2020
Figure 4: Fractions of Rainbow and PPO scores at different numbers of inteructions calculated with the formula (Sim SSssPLLe_ccoree see_scoor@ui @00Kk -awddon_scom sccore) (baselinee_scol than both nominator and denominator are increased by 1. From left right, the baselines are: Rainbow 100K, Rainbow at 200K, PPO at 100K, PPO at 200K. SimPLe outperforms Rainbow and PPO even when those are given twice as many interactions.
raanraannddoaaaeeeeeee sccoree if denominator is smaller
1nC
1.0
06
6.4
numbee
00
50
0.4 number camples ae
5.4
5.5
10
Sumber samples
126
Model based (SimPI.e)
(b) Model hased model free (SimPL.c PPO)
Figure 5: Behaviour with respect to the number of used samples. We report number of frames required by PPO to reach the score of our models. Results are averaged over all games.
6.2 NUMBER OF FRAMES
We focused our work on learning games with 100K interaction steps with the environment. In this section we present additional results for settings with 20K, 50K, 200K, 500K and 1M interactions: see Figure 5 (a). Our results are poor with 20K interactions. For 50K they are already almost as good as with 100K interactions. From there the results improve until 500K samples
also the point at which they are par with model-free PPO. Detailed per game results can be found in Appendix
This demonstrates that SimPLe excels in a low data regime, but its advantage disappears with a bigger amount of data. Such behavior, with fast growth at the beginning of training, but lower asymptotic performance commonly observed when comparing model-based and model-free methods (Wang et al. (2019)) As observed in Section 6.4 assigning bigger computational budget helps in 100K setting. We suspect that gains would be even bigger for the settings with more samples.
Finally, we verified if model obtained with SimPLe using 100K is useful initialization for model- free PPO training. Based on the results depicted in Figure 5 (b) we can positively answer this conjecture. Lower asymptotic performance probably due to exploration. A policy pre-trained with SimPLe was meant to obtain the best performance on 100K, at which point its entropy is very low thus hindering further PPO training.
8

---

Published as a conference paper at ICLR 2020
6.3 ENVIRONMENT STOCHASTICITY
crucial decision in the design of world models is the inclusion of stochasticity. Although Atari is known to be a deterministic environment, it is stochastic given only a limited horizon of past observed frames (in our case 4 frames). The level of stochasticity is game dependent; however, can be observed in many Atari games. An example of such behavior be observed in the game Kung Fu Master after climinating the current set of opponents, the game screen always looks the same (it contains only player's character and the background). The game dispatches diverse sets of new opponents, which cannot be inferred from the visual observation alone (without access to the game's 's internal state) and thus cannot be predicted by a deterministic model. Similar issues have been reported in [Babaeizadeh et al. (2117)) where the output of their baseline deterministic model was a blurred superposition possible random object movements. As can be seen in Figure in the Appendix, the stochastic model learns reasonable behavior samples potential opponents and renders them sharply.
Given the stochasticity of the proposed model, Sim- PLe can be used with truly stochastic environments. To demonstrate this, full pipeline (both the world model and the policy) was trained in the presence of sticky actions, as ommended in (Machado et al. 2018 Section 5). Our
feeway
Kung_fu
experiment where the
enterte
kangaroo
sepprr_eemmadd
creay climbrr
pong
world model learned to account for the stickiness of
boxing
actions and in most cases the end results were very
road, runnor
similar to the ones for the deterministic case even without any tuning, see Figure 6
bottle zo1e
frostrite
amidar
jamesbond
6.4 ABLATIONS
ms pecman
treakout
To evaluate the design of our method, we indepen- dently varied a number of the design decisions. Here we present an overview; see Appendix A for detailed results.
seaquest
pihato cye
asseult
gopher
Model architecture and hyperparameters. We evaluated a few choices for the world model and our proposed stochastic discrete model performs best by significant margin. The second most important parameter was the length of world model's training. We verified that longer training would be beneficial, however we had to restrict it in all other ablation stud- all games. As for
hero
demon,
without actiens
bank, heit
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
165
Figure 6: Impact of the environment stochasticiry. The graphs in the same format as Figure 3 each bar illustrates the number of interactions with environment required by Rainbow to achieve the same score as SimPLe (with stochastic discrete world model) using 100k steps environment with and without sticky actions.
ies due to a high cost of training the length of rollouts from simulated env', we use N 50 by default. We experimentally shown that N = 25 performs roughly on par, while N = 100 is slightly worse, likely due to compounding model errors.
The discount factor was set to 7 0.99 unless specified otherwise. We see that = 0.95 is slightly better than other values, and we hypothe- size that is due better tolerance to model imperfections. But overall, all three values of y perform comparably.
Model-based iterations. The iterative process of training the model, training the policy, and collect- ing data is crucial for non-trivial tasks where random data collection is insufficient. In a game-by- by-game analysis, we quantified the number of games where the best results were obtained in later iterations of training. In some games, good policies could be learned very early. While this might have been due to the high variability of training, does suggest the possibility of much faster training (i.e. in fewer step than 100k) with more directed exploration policies. In Tigurepiin in the Appendix we present the cumulative distribution plot for the (first) point during learning when the maximum score for the run was achieved in the main training loop of Algorithm
Random starts. Using short rollouts is crucial to mitigate the compounding errors in the model. To ensure exploration, SimPLe starts rollouts from randomly selected states taken from the real data buffer D. Segure@comparc.. the bascline with an experiment without random starts and rollouts of length 1000 on Seaquest which shows much worse results without random starts.

---

Published as a conference paper at ICLR 2020
7 CONCLUSIONS AND FUTURE WORK
We presented SimPLe, a model- based reinforcement learning approach that operates directly on raw pixel observations and learns effective policies to play games in the Atari Learning Environment. Our experiments demonstrate that SimPLe learns to play many the games with just 100K interactions with the environment, corresponding to hours of play time. In many cases, the number of samples required for prior methods to learn to reach the same reward value several times larger.
Our predictive model has stochastic latent variables it be applied in highly stochastic environ- ments. Studying such environments is an exciting direction for future work, as is the study of other ways in which the predictive neural network model could be used. Our approach learned simulator and directly applies model-free policy learning to acquire the policy. However, we could use the model for planning. Also, since our model is differentiable, the additional information contained in its gradients could be incorporated into the reinforcement learning process. Finally, the representation learned by the predictive model likely be more meaningful by itself than the raw pixel observations from the environment. Incorporating this representation into the policy could further accelerate and improve the reinforcement learning process.
the model
While SimPLe is able to learn more quickly than model-free methods, it does have limitations. First, the final scores are on whole lower than the best state-of-the-art model-free methods. This can be improved with better dynamics models and, while generally common with model-based RL algorithms, suggests an important direction for future work. Another, less obvious limitation is that the performance of our method generally varied substantially between different The complex interactions between the model, policy, and data collection were likely responsible for this. In future work, models that capture uncertainty via Bayesian parameter posteriors or ensembles (Kurutach et al. 2018 Chua et al. 2018) may improve robustness, Finally, the computational and time requirement training inside world model are substantial (see Appendix C). which makes developing lighter models an important research direction.
the same game.
In this paper our focus was demonstrate the capability and generality of SimPLe only across suite of Atari games, however, we believe similar methods can be applied to other environments and tasks which is one of our main directions for future work. As long-term challenge, we believe that bodel- reinforcement learning based on stochastic predictive models represents a promising and highly efficient alternative to model-free RL. Applications of such approaches to both high-fidelity simulated environments and real-world data represent an exciting direction for future work that can enable highly efficient learning of behaviors from raw sensory inputs in domains such as robotics and autonomous driving.
ACKNOWLEDGMENTS
We thank Mare Bellemare and Pablo Castro for their help with Rainbow and Dopamine. The work of Konrad Czechowski, Piotr Kozakowski and Piotr Miloš was supported by the Polish National Science Center grants UMO-2011/26//AST666 / T6/00622. The work of Henryk Michalewski was supported by the Polish National Science Center grant UMO-2018/22 000-20001/////ST6/0299999 This research was sup- ported by the PL-Grid Infrastructure. In particular, Konrad Czechowski. Piotr Kozakowski, Henryk Michalewski, Piotr Mitoš and Blažej Osiński extensively used the Prometheus supercomputer, located in the Academic Computer Center Cyfronet in the AGH University of Science and Technology in Kraków, Poland. Some of the experiments were managed using https: /nnepuuss ai We would like to thank the Neptune team for providing access the team version and technical support.
REFERENCES
Stephan Alaniz. Deep reinforcement learning with model learning and monte carlo tree search in minecraft. arXiv preprint arXiv:1803.088 08456, 2018.
Kamyar Azizzadenesheli. Brandon Yang. Weitang Liu, Emma Brunskill, Zachary C. Lipton, and Animashree Anandkumar. Sample-efficient deep RL with generative adversarial tree search. CoRR, abs/1 00//1806.0788 2018
10

---

Published as a conference paper at ICLR 2020
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. JCLR, 2017a.
Mohammad Babaeizadeh, Turi Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on GPU. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference net/ forum?
Track Proceedings. OpenReview.net, 2017b. URL
1d=r1VGvBcxI
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents (extended abstract). Proceedings of the Toenty-Fourth International Joint Conference Artificial Intelligence, IJCAL, pp. 4148-44444 -1522 2015.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1171-1179, 2015.
Lars Buesing. Theophane Weber, Yori Zwols, Nicolas Heess, Sébastien Racanière, Arthur Guez, and Jean-Baptiste Lespiau. Woulda, coulda, shoulda: Counterfactually -auuiuggg policy search. In 7th International Conference on Leurning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL [ddBJGGvvCCYQ
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A research framework for deep reinforcement learning. CoRR, abs/1812.061 10, 2018. Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
//openrevier net wto.moorddddai// sxvgl
Kurtland Chua, Roberto Calandra, Rowan McAllister and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural
Information Processing Systems, pp. 4759-4770, 2018.
Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2), 2013.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In 1st Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, volume 78 of Proceedings of Machine Learning Research, pp. 344-356. PMLR, 2017.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 18/2. 2.00568, 2018
Mustafa Ersen and Sanem Sariel. Learning behaviors of and interactions among objects through spatio-temporal reasoning. IEEE Transactions on Computational Intelligence and Al in Games, 7
(1):75-87, 2014.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning, ICML, pp. 1406-1415, 2018
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. CoRR, htss/8803.0001.. 2018
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29 June 3, 2017, pp. 3786-27933 IEEF, 2017. doi: 10.1109/CCR..200 2017.7989324.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation, ICRA, pp. 512-519, 2016.
Matthew Guzdial, Boyang Li, and Mark Riedl. Game engine learning from video. In Proceedings of the Twenty-Soe Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 3707-3713, 2017. doi: 10.24963/ijcaii 20177///8....
11

---

Published as a conference paper at ICLR 2020
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 2455-2467, 2018.
Danijar Hafner, Timothy P. Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamies for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565. PMLR, 2019.
Nicolas Heess, Gregory Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2944-2952, 2015.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3215-3222 AAAI Press, 2018. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
G. Zacharias Holland, Erik Talvitie, and Michael Bowling, The effect of planning shape on dyna-style planning in nigh-dimensional state spaces. CoRR, abs/1806.01 0820, 2018.
I Lukasz Kaiser and Samy Bengio. Discrete autoencoders for sequence models. CoRR, 0tbs/1801.099) 2018.
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep rein- forcement learning. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 195-206. PMLR, 13-15 Nov 2017.
Kacper Piotr Kielak. Do recent advancements in based deep reinforcement learning really improve data efficiency?, 2020. URL
net forum? d=
Bke 1FF
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann
LeCun (eds.). 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. uRttttps:: http//ppneeeeeeeeee Felix Leibfried, Nate Kushman, and Katja Hofmann. A deep learning approach for joint video frame and reward prediction in Atari games. CoRR, abs/1611. 1.7777 2016.
Felix Leibfried, Rasul Tutunov, Peter Vrancx, and Haitham Bou-Ammar. Model- based regularization for deep reinforcement learning with transcoder networks. arXiv preprint ww..vv//09.0/// 2018. Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Intell. Res., 61:523-562, 2018. doi: 10. 16111111 1/.16//aiir.669, Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
12

---

Published as a conference paper at ICLR 2020
Volodymyr Mnih, Adrià Puigdomènech Badia, Mchdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference Machine Learning, ICML, pp.
1928-1937, 2016.
Junhyuk Oh, Xiaoxiao Guo. Honglak Lee, Richard L. Lewis, and Satinder P. Singh. Action- conditional video prediction using deep networks in atari games. In NIPS, pp. 2863-2871,2000
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In Guyon, U. V. Luxburg,
S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 61 68--1228, Curran Associates, Inc., 2017.
Chris Paxton. Yotam Barny,, Kapil D. Katyal, Raman Arora, and Gregory D. Hager. Visual robot task planning. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 20/9, pp. 8832-8838. IEEE, 2019. doi: 10. 1 109/ICRA. 2019.77771. A. J. Piergiovanni, Alan Wu, and Michael S. Ryoo. Learning real-world robot policies by dreaming. CoRR, abs/1805.00 805.078 13, 2018.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Vecerík, Matteo Hessel, Rémi Munos, and Olivier Pietquin. Observe and look further: Achieving consistent performance atari. CoRR,
abs/ 0obs/1805.111 1593, 2018.
Oleh Rybkin, Karl Pertsch, Andrew Jaegle, Konstantinos G. Derpanis, and Kostas Daniilidis. Unsuper- vised learning of sensorimotor affordances by stochastic future prediction. CoRR, 0bs//800.09655, 2018.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans. Autonomous Mental Development, 2(3):230-244 14000224.4440 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344 2017.
Shagun Sodhani, Anirudh Goyal, Tristan Deleu, Yoshua Bengio, Sergey Levine, and Jian Tang. Learning powerful policies by using consistent dynamics model. arXiv preprint arXiv: 19066043655
2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ily Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-13))))))) 1958, 2014.
Richard S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2(4):160-18 July 1991.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction, 2nd edition (work in progress). Adaptive computation and machine learning. MIT Press, 2017.
Pedro Tsividis, Thomas Pouncy, Jaqueline L. Xu, Joshua B. Tenenbaum, and Samuel J. Gershman. Human learning in atari. 2017 AAAI Spring Symposia, Stanford University, Palo Alto, California,
USA, March 27-29, 2017, 2017.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio. Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 6306-6315 6315,222 2017.
Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforce- ment learning? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Alché-Buu Emily B. Fox, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 14332-14333, 2019.
Arun Venkatraman, Roberto Capobianco, Lerrel Pinto, Martial Hebert, Daniele Nardi, and J. Andrew Bagnell. Improved learning of dynamics models for control. In International Symposium on Experimental Robotics, ISER 2016, Tokyo, Japan, October 3-6, 2016., pp. 703-713, 2016.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, abs/1907. 00077,0277 2019.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, pp. 2746-2754, 2015.
13

---

Published as a conference paper at ICLR 2020
Yuhuai Wu, Elman Mansimov, Roger B. Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using coneecer-factoree approximation. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5279-5288, 2017.
14

---

Published as a conference paper at ICLR 2020
Table 1: Summary of Le ablations. For each game, configuration assigned score being the over experiments. The best and median scores were calculated per game. The table reports the number of games a given configuration achieved the best score or at least the median score, respectively.
model
best
at least median
deterministic
0
7
det. recurrent
3
13
SD
8
16
SD = 0.9
14
default
10
21
SD 100 steps
0
14
SD 25 steps
4
19
All our code is available as part of the Tensor2Tensor library and it includes instructions on how our experiments: http:///////t/tao/o/oo/oooooo/oa
mmstar/tensor/tensor/
A ABLATIONS
To evaluate the design of our method, we independently varied number of the design decisions: the choice of the model, the y parameter and the length of PPO rollouts. The results for 7 experimental configurations summarized in the Table
Models. To assess the model choice, we evaluated the following models: deterministic, determin- istic recurrent, and stochastic discrete (see Section 4). Based on Table [ can be seen that our proposed stochastic discrete model performs best. Figures 7a and 7b show the role of stochasticity and recurrence.
Steps. See Figure 7 As described in Section 5 every N steps we reinitialize the simulated environment with ground-truth data. By default we use N 50, in some experiments we set N 25 or 100. It is clear from the table above and Figure 7d that 100 is bit worse than either 25 or 50, likely due to compounding model errors, but this effect is much smaller than the effeet of model architecture.
Gamma. See Figure 8b
We used the discount factor 7 = 0.99 unless specified otherwise. see that y = 0.95 is slightly better than other values, and we hypothesize that is due to better tolerance to model imperfections. But overall, all three values of seem to perform comparably at the same number of steps.
Model-based iterations. The iterative process of training the model, training the policy, and collecting data is crucial for non-trivial tasks where simple random data collection is insufficient. In the game-by-game analysis,
obtained quantified the number of games where the best results in later iterations of training. In some games, good policies could be learned very carly. While this might have been due simply to the high variability of training, does suggest the possibility that much faster training - in many fewer than 100k steps could be obtained in future work with more directed exploration policies. We leave this question to future work.
In Figure g maximum score for the run was achieved in the main training loop of Algorithm
we present the cumulative distribution plot for the (first) point during learning when the
On Figure 7c
we show results for experiments in which the number samples was fixed to be 100K but the number of training loop varied. We conclude that 15 is beneficial for training.
Long model training Our best results obtained with much 5 times longer training of the world models, see Figure Ba] comparison with shorter training. Due to our resources constraints other ablations made with the short model training setting.
15

---

Published as a conference paper at ICLR 2020
Random starts. Using short rollouts is crucial to mitigate the compounding To ensure exploration SimPLe starts rollouts from randomly selected states taken from the real data buffer D. In Figure 9 we present a comparison with experiment without random starts and rollouts of length 1000 on Seaquest. These data strongly indicate that ablating random starts substantially deteriorate results.
under the model.
B QUALITATIVE ANALYSIS
This section provides a qualitative analysis and case studies of individual games. We emphasize that we did not adjust the method nor hyperparameters individually for each game, but we provide specific qualitative analysis to better understand the predictions from the model
Solved games. The primary goal of our paper was use model-based methods to achieve good performance within a modest budget of 100k interactions. For two games, Pong and F reeway, our method, SimPLe, was able to achieve the maximum score.
Exploration. Freeway is a particularly interesting game. Though simple, it presents sub- stantial exploration challenge. The chicken, controlled by the agents, is quite slow to ascend when exploring randomly as it constantly gets bumped down by the cars (see the left video https: goo g1/
This makes it very unlikely to fully cross the road and obtain a non-zero reward. Nevertheless, SimPLe is able to capture such rare events, internalize them into the predictive model and then successfully learn successful policy.
However, this good performance did not happen on every run. We conjecture the following scenario in failing cases. If at early stages the entropy of the policy decayed too rapidly the collected experience stayed limited leading to a poor world model, which was not powerful enough to support exploration (e.g. the chicken disappears when moving high). In one of experiments, we observed that the final policy was that the chicken moved up only to the second lane and stayed waiting to be hit by the car and so on so forth.
Pixel-perfect games. In some cases (for Pong, Freeway, Breakout) our models were able to predict the future perfectly, down to every pixel. This property holds for rather short time intervals, we observed episodes lasting up to time-steps. Extending to long sequences would be a very exciting research direction. See videos htttps://go..g/uy//
Benign errors. Despite the aforementioned positive examples, accurate models difficult to acquire for some games, especially at early stages of learning. However, model-based RL should be tolerant modest model errors. Interestingly, some cases our models differed from the original games in way that was harmless or only mildly harmful for policy training.
For example, in Bowling and Pong, the ball sometimes splits into two. While nonphysical, seemingly these errors did not distort much the objective of the game, see Figure 10 and also
https://goo (iooo.g//P11rr
In Kung Fu Master our model's predictions deviate from the real game by spawning a different number opponents, see Figure TT In Crazy Cli immbrrr we observed the bird appearing earlier in the game. These cases are probably to be attributed to the stochasticity in the model. Though not aligned with the true environment, the predicted behaviors are plausible, and the resulting policy can still play the original game.
Failures hard games. On of the games, models simply failed produce useful predictions. We believe that listing such errors may be helpful in designing better training protocols and building better models. The most common failure was due the presence of very small but highly relevant objects. For example, in At Atlantis and Battle Zone bullets are so small that they tend to disappear. Interestingly, Battle Zone has pseudo-3D graphics, which may have added to the difficulty. See videos httph:ttt/sssoo http::::///oooooo http://ooo...occcccc/ooo
"We strongly encourage the reader watch accompanying videos
16

---

Published as a conference paper at ICLR 2020
frowsy
Deeway
20g suster
lang du maate
Sanguro
cloppir
dagper oxezznnd
octrod
Cag clasber
emay diuder
laning
bowing
Treakest
<ber
lettl
bottle
Iano
3000
dezzn
Seavc attade
bank bot
(a) Effect of stochasticity.
(b) Effect of recurrent architecture.
Froway
Orewwyy
pelora
Sang madde
langaro
king
poog
pood stsoe
tamobood
Guay diokes
paztion
Janesbaad
Teakost
dorcn
Iaro
chopper commundd
Hemitone (Cabua))
Semac uttedk
sterstioea
atage (kdestt)
bodig
baak boist
(c) Effect of adjusting of number of epochs.
(d) Effect of adjusting of number of steps.
Figure 7: Ablations part 1. The graphs are in the same format as Figure 3 each bar illustrates the number of interactions with environment required by Rainbow to achieve the same score as particular variant of SimPLe. The red line indicates the 100K interactions threshold which is used by SimPLe.
17

---

Published as a conference paper at ICLR 2020
fronny
king
cag
sateid
pong
dizber
med PERCAT
Ineelete
Teakors
Toc
Bcptax
dazeo attacs
prlcce
deleah
boinn
bonk bosa
(a) Effect of extended model training.
(b) Effect of adjusting y in PPO training
Figure 8: Ablations part 2. The graphs in the format as Figure each bar illustrates the number of interactions with environment required by Rainbow to achieve the same score as a particular variant of SimPLe. The red line indicates the 100K interactions threshold which is used by SimPLe.
seaquest
random starts
08
10
02
0.
10 14
step
Figure 9: (left) CDF of the number of iterations to acquire maximum score. The vertical axis represents fraction of all games. (right) Comparison of random starts vs no random starts on Seaquest (for better readability we clip game rewards to 1, 1.). The vertical axis shows a mean reward and the horizontal axis the number iterations of Algorithm
18

---

Published as a conference paper at ICLR 2020
20
20
8
20
8
20
Figure 10: Frames from the Pong environment.
16-27
1622 11000
15-44
1554
500-
5-0-
11500
Figure 11: Frames from the Kung Fu Master environment (left) and its model (right).
Another interesting example comes from Private Eye in which the agent traverses different scenes, teleporting from one to the other. We found that our model generally struggled to capture such large global changes.
19

---

Published as a conference paper at ICLR 2020
C ARCHITECTURE DETAILS
The world model is a crucial ingredient of our algorithm. Therefore the neural-network archi- tecture of the model plays crucial role. The high-level overview of the architecture is given in Section [ and Figure 2
We stress that the model general, not Atari specific, and we believe could handle other visual prediction tasks. The whole model has around 74M parameters and the inference/haccproppggeee
time is approx. 0.5s/0.7s respectively, where inference is on batch size 16 and backpropagation on batch size 2, running on NVIDIA Tesla P100. This gives us around 32ms per frame from our simulator, in comparison one step of the ALE simulator takes approximately 0.4ms.
Below give more details of the architecture. First, the frame prediction network:
Layer
Number of outputs
Other details
Input frame dense
96
Downscale convolution 1
192
kernel 4x4, stride 2x2
Downscale convolution
384
kernel 4x4, stride 2x2
Downscale convolution 3
768
kernel 4x4, stride 2x2
Downscale convolution 4
768
kernel 4x4, stride 2x2
Downscale convolution 5
768
kernel 4x4, stride 2x2
Downscale convolution 6
768
kernel 4x4, stride 2x2
Action embedding
768
Latent predictor embedding Latent predictor LSTM
128
128
Latent predictor output dense
256
Reward predictor hidden
128
Reward predictor output dense
Middle convolution
768
kernel 3x3, stride 1x1
Middle convolution 2
768
kernel 3x3, stride 1x1
Upscale transposed convolution 1
768
kernel 4x4, stride 2x2
Upscale transposed convolution 2
768
kernel 4x4, stride 2x2
Upscale transposed convolution 3
768
kernel 4x4, stride 2x2
Upscale transposed convolution 4
384
kernel 4x4, stride 2x2
Upscale transposed convolution 5
192
kernel 4x4, stride 2x2
Upscale transposed convolution 6 Output frame dense
96
kernel 4x4, stride 2x2
768
The latent inference network, used just during training:
Layer
Number of outputs
Other details
Downscale convolution
128
kernel 8x8, stride 4x4
Downscale convolution 2
512
kernel 8x8, stride 4x4
All activation functions are ReLU,, except for the layers marked as "output", which have softmax activations, and LSTM internal layers. In the frame prediction network, the downscale layers are connected to the corresponding upscale layers with residual connections. All convolution and transposed convolution layers are preceded by dropout 0.15 and followed by layer normalization. The latent predictor outputs 128 bits sequentially, in chunks
20

---

Published as a conference paper at ICLR 2020
D NUMERICAL RESULTS
Below we present numerical results our experiments. We tested SimPLe on configurations (see description in Section Aj. For each configuration experiments we used the policy given by softmax (lo where и; is the final learnt policy in the experiment and T' is the temperature parameter. We found empirically that T 0.5 worked best in most cases. A tentative explanation is that polices with temperatures smaller than 1 are less stochastic and thus more stable. However, going down to T proved to be detrimental in many cases as, possibly, it makes policies more prone to imperfections of models.
experiments. For the evaluation of the i-th
In Table 2 we present the mean and standard deviation of the experiments. We observed that the median behaves rather similarly, which is reported it in Table In this table we also show maximal scores over runs. Interestingly, many cases they turned out to be much higher. This, we hope, indicates that our methods has a further potential of reaching these higher scores.
Human scores are Avg. Human" from Table 3 in Pohlen et al. (0188).
21

---

Published as a conference paper at ICLR 2020
Table 2: Models comparison, Mean scores and standard deviations over five training runs. Right most columns presents score for random agent and human.
Game
Ours, deterministic
Ours, det. recurrent
Ours, SD long
Ours, SD
Ours, SD
0.90
Ours, SD ry 0.95
Ours, SD 100 steps
Ours, SD 25 steps
random
human
Alien
378.3
(85.5)
321.7
(50.7)
616.9
(252.2)
405.2
(1.30.8
413.0
(89.7)
590.2
(57.8)
435.6
(78.9)
534.8
(166.2)
184.8
7128.0
Amidar
62.4
(15.2)
86.7
(18.8)
74.3
(28.3)
88.0
(23.8)
50.3
(11.7)
78.3
(18.8)
37.7
(15.1)
82.2
(43.0)
11.8
1720.0
Assault
361.4
(166.6)
490.5
(143.6)
527.2
(112.3)
369.3
(107.8)
406.7
(118.7)
549.0
(127.9)
311.7
(88.2)
664.5
(298.2)
233.7
Asterix
668.0
(294.1)
1853.0
(391.8)
1128.3
(211.8)
1089.5
(335.3)
855.0
(176.4)
921.6
(114.2)
777.0
(200.4)
1340.6
(627.5)
248.8
8501.0
Asteroids
743.7
(92.2)
821.7
(115.6)
793.6
(182.2)
731.0
(165.3)
882.0
(24.7)
886.8
(45.2)
821.9
(93.8)
644.5
(110.6)
649.0
47389.0
Atlantis
14623.4
(2122.5)
12584,4
(5823.6)
20992.5
(11062.0)
144816
(2436.9)
18444.1
(4616.0)
14055.6
(6226.1)
14139,7
(2500.9)
11641.2
(3385.0)
16492.0
29028.0
Banklleist
13.8 (2.5)
15.1
(2.2)
34,2
(29.2)
8.2
(4.4)
11.9
(2.5)
12.0
(1.4)
13.1
(3.2)
12.7
(4.7)
15.0
753.0
Battle7one
3306.2 1)
4665.6
(2799.4)
4031.2
(1156.1)
5184,4
(1347.5)
2781.2
(661.7)
4000.0
(788.9)
4068.8
(2912.1)
3746.9
(1426.8)
2895.0
871
BeamRider
463
(29.2)
358.9
(87.4)
621.6
(79.8)
422.7
(103.6)
456.2 (160.8)
415.4
(103.4)
456.0
(60.9)
386.6
(264-4)
372.1
16926.0
Bowling
25.3 (10:4)
22.3
(17.0)
30.0
34,4
(16.3)
27.7 (5.2)
(3.3)
29.3
33.2
(15.5)
24.2
Boxing
-9.3 (10.9)
-3.1
(14.1)
7.8 (10.1)
9.1 (8.8)
11.6 (12.6)
5.1 (10.0)
-2.1
(5.0)
1.6
(14.7)
0.3
12.0
Breakout
6.1 (2.8)
10.2
(5.1)
16.4 (6.2)
12.7
(3.8)
7.3 (2.4)
8.8 (5.1)
11.4
(3.7)
(4.1)
0.9
30.0
ChopperCommant
906.9
(210.2)
709.1
(174.1)
979.4
(172.7)
1246.9
(392.00
725.6
(204.2)
946.6
(49,9)
729.1
(185.1)
1047.2
(2216)
671.0
7388.0
CruzyClimber
19380.0
(6138.8)
54700.3
(14480.5)
62583.6
(16856.8)
39827.8
(22582.6)
49840.9
(11920.9)
34353.1
(31547.2)
48651.2
(14903.5)
25612.2
(14037.5)
7339.5
35829.0
DemonArtack
191.9 (86.3)
120.3
(38.3)
208.1
(56.8)
169.5
(41.8)
187.5
(68.6)
194.9
170.1
(42.4)
202.2
(134.0)
140.0
1971.0
DerbyngDerby
-94.5
(3.0)
96.9
(1.7)
-90.7 (5.3)
-9.55
(2.8)
91.0
(4.1)
-92.6
(3:2)
-90.0
(2.7)
-94.5
(2.5)
-93.6
-39.0
Freeway
(13.1)
23.7
(13.5)
16.7 (15.7)
20.3
(18.5)
(17.2)
(13.3)
(16.7)
27.3 (5.8)
Prostbite
196.4
(4,4)
219.6
(21.4)
236.9
(31.5)
254.7
(4.9)
234.6
(26.8)
239.2
(19.1)
226.8
(16.9)
252.1 (54.4)
74.0
Gopher
510.2 (158.4)
225.2
(105.7)
596.8
(183.5)
771.0
(160.2)
845.6
(230.3)
612.6
(273.9)
698,4
(213.9)
509.7
(273.4)
245.9
2412.0
Gravitar
237.0 (73.1)
213.8
(57.4)
173.4
(54.7)
198.3
(39.9)
219.4
(7.8)
213.0
(37.3)
188.9
(27.6)
116.4
(84.0)
227.2
3351.0
Hero
621.5
(1281.3)
558.3
(1143.3)
2656.6
(483. I)
1295.
(1600.1) I)
2853.9
(539.5)
3503.5
(892.9)
3052.7
(169.3)
1484,8
(1671.7)
224.6
30826.0
IceHockey
-12.6 (2.1)
-14.0
(1.8)
-11.6
(2.5)
-10.5
(2.2)
-12.2
(2.9)
-11.9
(1.2)
-13.5
-13.9
(3:9)
-9.7
1.0
Jamesbond
68.8
(37.2)
100.5
(69.8)
100.5
(36.8)
125.3
(112.5)
28.9
(12.7)
50.5
(21.3)
68.9
(42.7)
163.4
(81.8)
29.2
303.0
Kangaroo
(313.2)
191,9
(301.0)
51.2
(17.8)
323,1
(359.8)
(121.5)
(8.0)
301.2
(593.4)
(470.4)
42.0
Krull
834.9
(166.3)
1778.5
(906.9)
2204.8
(776.5)
4539,9
(2470.4)
2396.5
(962.0)
2620.9
(856.2)
3559,0
(1896.7)
3320.6
(2410.1)
1543,3
2666.0
KungFuMaster
10340.9
(8835.7)
4086.6
(3384.5)
14862.5
(4U31.6)
17257.2
(5502.6)
12587.8
(6810.0)
16926.6
(6598.3)
17121.2
(7211.6)
15541.2
(5086.1)
616.5
22736.0
MsPacian
560.6
(172.2)
1098.1
(450.9)
1480.0
(288.2)
762.8
(331.5)
1197.1
(544.6)
1273.3
(59.5)
921.0
(306.0)
805.8
(261.1)
235.2
6952.0
NaneThisGame
1512.1
(408.3)
2077.9
(367.0)
2420.7
(289.4)
1990.4
(284.7)
2058.1
(103.7)
21.4.8
(387.4)
2067.2
(304.8)
1805.3
(453.4)
2136.8
8049.0
Pong
-17.4
(5.2)
-11.6
12.8
(17.2)
(7.3)
-2.5
(15.4)
-13,9
(14.9)
-20.4
15.0
PrivateEye
16.4
(46.7)
50.8
(43.2)
35.0
(60.2)
58.3
(45.4)
$4.4
(49.0)
67.8
(26.4)
88.3
(19.0)
1334.3
(1794.5)
26.6
69571.0
Qbert
480.4
(158.8)
(150.3)
1288.8
(1677.9)
559.8
(183.8)
899.3
(474.3)
(697.1)
534.4
(162.5)
(138.2)
13455,0
Riverraid
1285.6
(604.6)
1740.7
(458.1)
1957.8
(758.1)
1587.0
(818.0)
1977.4
(332.7)
2115.1
(106.2)
1318.7
(540.4)
1426.0
(374.0)
1451.0
17118.0
RoadRumner
5724.4
(3093.1)
1228.8
(1025.9)
5640.6
(3936.6)
$169.4
(3939.0)
1586.2
(1574.1)
8414.1
(4542.8)
722.2
(627.2)
4366.2
(3867.8)
0.0
7845.0
Senquest
419.5
(236.2)
289.6
(110.4)
683.3
(171.2)
370.9
(128.2)
3646
(138.6)
337.8
(79.0)
247.8
(72.4)
350.0
(136.8)
61.1
42055.0
UpNDown
1329.3
(495.3)
926.7
(335.7)
3350,3
(3540.0)
2152.6
(1192.4)
1291.2
(324.6)
1250.6
(493.0)
1828.4
(688.3)
2136.5
(2095.0)
488.4
11693.0
YarsRovenge
3014.9
(397.4)
3291.4
(1097.3)
5664.3
(1870.5)
2980.2
(778.6)
2934.2
(459.2)
3366.6
(493.0)
2673.7
4666.1
(1889.4)
3121.2
54577.0

---

Published as a conference paper at ICLR 2020
Table 3: Comparison of our method (SimPLe) with model-free benchmarks PPO and Rainbow, trained with 100 thousands/500 thousands/I million steps. (1 step equals 4 frames)
Game
SimPI.c
PPO_1000
PPO_500K
PPO 1m
Rainhow
100k
Rainhow_500
Rainhow Im
random
human
616.9
(252.2)
291.0
(40.3)
(203.4)
(102.0)
290.6
(14.8)
(54.2)
945.0
184.8
7128.0
Amidar
74.1
(28.3)
56.5
(20.8)
93.2
(36.7)
123.8
(19.7)
20.8
(2.3)
194.0
(34.9)
275.8
(66.7)
11.8
1720.0
Assault
527.2
(112.3)
424.2
(55.8) 8)
552.3
(110.4)
1134,4
(798.8)
300.3
(14.6)
1041.5
(92.1)
1581.8
(207.8)
233.7
742.0
Asterix
1128.3
(211.8)
385,0
(104,4)
1085.0
(354.8)
2185.0
(931.))
285.7
(9.3)
1702.7
(162.8)
2151.6
(202.6)
248.8
8503.0
Asteroids
793.6
(182.2)
1134.0
(326.9)
1053,0
(433.3)
1251.0
(377.9)
912.3
(62.7)
895,9
(82.0)
1071.5
(91.7)
649.0
47389.0
Atlantis
20992.5
(1 (11062.0)
34316.7
(5703.8)
4836416.7
(6218247.3)
17881.8
(617.6)
79541.0
(25393.4)
848800.0
(37533.1)
16492.0
29028.0
BankHeist
34.2
(29.2)
16.0
(12.4)
641.0
(352.8)
856.0
(376.7)
34,5
(2.0)
727.3
(198,3)
1053.3
(22.9)
15.0
753.0
BattleZone
4031.2
(1156.1)
(3655.1)
14400.0
(6476.1) 1)
19000.0
(4571.7)
3363.5
(523.8) 8)
19507.1
(3193.3)
22391.4
(7708.9)
37188.0
BeamRider
621.6
(79.8)
563.6
(189.4)
497.6
(103-5)
684.0
(168.8)
365.6
(29.8)
5890.0
(525.6)
6945.3
(1390.8)
372.1
16926.0
Bowling
30.0 (5.8)
17.7
(11.2)
28.5
(3.4)
35.8
(6.2)
24.7
(0.8)
31,0
(1.9)
30.6
(6.2)
24.2
161.0
Boxing
7.8 (10.1)
+3,9
(6.4)
3.5
(3.5)
19.6
(20.9)
0.9
(1.7)
58.2
(16.5)
80.3
(5.6)
0.3
12.0
Breakout
16.4 (6.2)
5.9
(3.3)
66.1
(114.3)
128.0
(153.3)
3.3
(0.1)
26.7
(2.4)
38.7
(3.4)
0.9
30.0
ChopperCommand
979.4
(172.7)
730.0
(199.0)
860.0
(285.3)
970.0
(201.5)
776.6
(59.0)
1765.2
(280.7)
2474.0
(504.5)
671.0
7388.0
CrazyClimber
62583.6
(16856.8)
18400.0
(5275.1)
33420.0
(3628.3)
58000.0
(16994.6)
12558.3
(674.6)
75655.1
(9439.6)
97088.1 1
(9975.4)
7339.5
35829.0
DemonAntak
(56.8)
192.5
(83.1)
(96.2)
(135.0)
431.6
(79.5)
3642.1
(478.2)
(297.9)
140.0
DerhingDerby
-90.7
(5.3)
95,6
(4.3)
-87.2
(5,3)
-8888
(4.0)
-911
(2.1)
-66,7
(6.0)
-23.2
(22.3)
-93.6
-39.0
23
Froeway
16.7
(15.7)
8.0
(9.8)
14.0
(11.5)
20.8
(11.1)
0.1
(0.1)
12.6
(15.4)
13.0
(15.9)
0.0
30.0
Frosibile
236.9
(31.5)
174.0
(40.7)
214.0
(10.2)
229.0
(20.6)
140.1
(2.7)
1386.1
(321.7)
2972.3
(284.9)
74.0
Gopher
596.8
(183.5)
246.0
(103.3)
560.0
(118.8)
696.0
(279.3)
748.3
(105 4)
1640.5
(105.6)
1905.0
(211.1)
245.9
2412.0
Gravitar
173.4
(54.7)
235.0
(197.2)
235.0
(134.7)
325,0
(85.1)
231.4
214.9
(27.6)
260.0
(22.7)
227.2
3351,0
Hero
2656.6
(483.1)
569.0
(1100.9)
1824.0
(1461.2)
3719.0
(1306.0)
2676.3
(93.7)
10664.3
(1060.5)
13295.5
(261.2
224.6
30826.0
IceHockey
-11.6
(2.5)
10.0
(2.1)
(1.6)
(1.7)
-9.5
(0.8)
.9.7
(0.8)
-6.5
(0.5)
-9.7
Jameshond
100.5
(36.8)
65.0
(46.4)
255.0
(101.7)
310.0
(129.0)
61.7
(8.8)
429.7
(27.9)
692.6
(316.2)
29.2
303.0
Kangaroo
51.2
(17.8)
140.0
(102.0)
340,0
(407.9)
840.0
(806.5)
38.7
(9.3)
970.9
(501.9)
4084.6
(1954.1)
42.0
3035,0
Krull
2204.8
(776.5)
3750.4
(3071.9)
3056.1
(1155,5)
$061.8
(1333.4)
2978.8
(148.4)
4139.4
(336.2)
4971.1
(360.3)
1543.3
2666.0
KungeuMaster
14862.5
(4031.6)
4820.0
(983.2)
17370.0
(10707.6) 6)
13780.0
(3971.6)
1019.4
(149.6)
19346.1
(3274.4)
21258.6
(3210.2)
616.5
22736.0
MaPaeman
1480.0
(288.2)
496.0
(379.8)
306.0
(70.2)
(247.9)
364.3
(20.4)
1558.0
(248.9)
18814
(112.0)
235.2
6952.0
Name ThisGamme
2420.7
(289.4)
2225.0
(423.7)
2106.0
(898.8)
2311.0
(547.6)
2368.2
(318.3)
4886.5
(583.1)
4454.2
(338.3)
2136.8
8049.0
Pong
12.8
(17.2)
20.55
(0.6)
(14.9)
14.7
(5.1)
(0.2)
19.9
(0.4)
(0.2)
-0044
PrivateBye
35.0
(60.2)
10.0
(20.0)
20.0
(40.0)
20.0
(40.0)
42.1
(53.8)
-6.2
(89.8)
2336.7
(4732.6)
26.6
69571.0
Obert
1288.8
(1677.9)
362.5
(117.8)
757.5
(78.9)
2675.0
(1701.1)
235.6
(12.9)
4241.7
(193.1)
8585.2
(169119)
166.1
13455.0
Riveraid
1957.8
(758.1)
1398.0
(513.8)
2865.0
(327.1)
2887.0
(807.0)
1904.2
(44.2)
5068.6
(292.6)
7018.9
(334.2)
1451.0
17118.0
RoadRunner
5640.6
(3936.6)
1430.0
(760.0)
5750.0
(5259.9)
8930.0
(4304.0)
$24.1
(147.5)
18415
(5280.0)
31379.7
(3225.8)
0.0
7845.0
Scaquest
683.3
(171.2)
370.0
(103.3)
(48.3)
882.0
(122.7)
206.3
1558.7
(221.2)
3279.9
(683.9)
42055.0
UpNDowa
3350,3
(3540.0)
2874.0
(1105.8)
12126.0
(1389.5)
13777.0
(6766.3)
1346.3
(95.1)
6120.7
(356.8)
8010.9
(907.0)
488.4
11693.0
YarsRevenge
5664.3
(1870.5)
(1209.3)
8064.8
(2859.8)
9495.0
(2638.3)
3649.0
(168.6)
7005.7
(394.2)
(957.9)
3121.2
$4577.0

---

Published as conference paper at ICLR 2020
Table 4: Models comparison. Scores of median (left) and best (right) models out of five training runs. Right most columns presents score for random agent and human.
Game
Ours, deterministic
Ours, det. recurrent
Ours, SD long
Ours, SD
Ours, SD
0.90
Ours, SD
0.95
SD 100 steps
Ours, SD 25 steps
random
human
Alien
354.4
516.5
299.2
381.1
515.9
1030.5
409.2
586.9
411.9
530.5
$67.3
682.7
399.5
$22.3
525.5
792.8
184.8
7128.0
Amidar
58.0
84.8
82.7
118.4
80.2
102.7
85.1
114.0
55.1
58.9
84.3
101 4
45.2
47.5
93.1
137.7
11.8
1720.0
Assault
334.4
560.1
566.6
627.2
509.1
671.1
527.9
614.4
508.4
722.5
322.9
391.1
701.4
1060.3
233.7
742.0
Asterix
529.7
1087.5
1798.4
2282.0
1065.6
1485.2
1158.6
1393.8
805.5
1159.4
923.4
1034.4
813.3
1000.0
1128.1
2313.3
248.8
8503.0
Asteroids
727.3
854.7
827.7
919.8
899.7
955.6
671.2
962.0
885.5
909.1
886.1
949.5
813.8
962.2
657.5
752.7
649.0
47389.0
Atlantis
15587.5
16545.3
15939.1
17778.1
13695.3
3489006
13645.3
18396.9
19367.2
23046.9
12981.2
23579.7
15020.3
16790.6
12196,9
15728.1
16492.0
29028.0
Bunk
14.4
16.2
14.7
18.8
31.9
77.5
8.9
13.9
12.3
14.5
12.3
13.1
12.8
17.2
14.1
17.0
15.0
753.0
BattleZone
3312.5
4140.6
4515.6
9312.5
3484,4
5359.4
5390.6
7093.8
2937.5
3343.8
4421.9
4703.
3500.0
8906.2
3859.4
5734,4
2895.0
37188.0
BeamRider
453.1
515.5
351.4
470.2
580.2
728.8
433.9
512.6
393.5
682.8
446.6
519.2
447.1
544.6
385.7
741.9
372.1
16926.0
Bowling
27.0
36.2
28.4
28.0
22.6
24.2
Boxing
-7.1
0.2
3.5
5.0
9.4
21.0
8.3
21.5
6,4
31.5
2.5
15.0
-0.7
2.2
-0.9
20.8
0.3
12.0
Bacakout
12.5
13.9
16.0
22.8
11.0
19.5
7,4
10.2
14.1
10.5
16.7
6.9
13.0
0.9
30.0
ChopperCommand
942.2
1167.2
748.4
957.8
909.4
1279.7
1139.1
1909.4
682.8
1045.3
954.7
1010.9
751.6
989.1
1031.2
1329.7
671.0
7388.0
CramyClimber
20754.7
23831.2
49854.7
80156.2
55795.3
87593.8
41396.9
67250.D
56875.0
58979.7
19448.4
84070.3
53406.2
64196.9
19345.3
43179.77
7339.5
35829.0
DemonAttak
219.2
263.0
135.8
148.4
191.2
288.9
182.4
223.9
160.3
293.8
204.1
312.8
164.4
222.6
187.5
424.8
140.0
1971.0
DerbyngDerby
-94.3
-90.2
-97.3
-94.2
-91.8
-84.3
-91.6
-88.6
-90.0
-85.7
-92.0
-88.8
-90.6
-85.4
-95.0
-90.7
-93.6
-39.0
Freeway
29.3
33.5
Frostbite
194.5
203.9
213.4
256.2
248.8
266.9
253.1
262.8
246.7
261.7
250.0
255.9
215.8
247.7
249.4
337.5
74.0
Gopher
514.7
740.6
270.3
320.9
525.3
845.6
856,9
934-4
874.1
1167.2
604.1
1001.6
726.9
891.6
526.2
845.0
245.9
2412.0
Gravitar
252.8
310.2
219.5
300,0
156.2
233.6
202.3
252.3
223.4
225.8
228.1
243.8
193.8
218.0
93.0
240.6
227.2
3351,0
Hero
71.5
2913.0
75.0
2601.5
2935.0
3061.6
237.5
3133.8
3135.0
3147.5
3066.2
5092.0
3067.3
3256.9
1487.2
2964.8
224.6
30826.0
IceHockey
-12.4
-9.9
14.8
-12.3
-7.2
-10.0
-7.7
-11.8
-8.5
-11.6
-10.7
-12.9
-10.0
-12.2
-11.0
.9.7
1.0
Jamesbond
64.8
128.9
64.8
219.5
110.9
141.4
87.5
323.4
25.0
46.9
58.6
69.5
61.7
139.1
139,8
261.7
29.2
303.0
Kungaroo
68.8
34.4
1362.5
1128.1
Krull
852.2
1014.3
1783.6
2943.6
1933.7
3317.5
4264.3
7163.2
1874,8
3554.5
2254.0
3827.1
3142.8
6315.2
3198.2
6833.4
1543.3
2666.0
KungFuMaster
7575.0
20450.0
4848.4
8065.6
14318.8
21054.7
17448.4
21943.8
12964.1
21956.2
20195.3
23690.6
19718.8
25375.0
18025.0
20365.6
616.5
22736.0
MsPacman
557.3
818.0
1178.8
1685.9
1525.0
1903.4
751.2
1146.1
1410.5
1538.9
1277.3
1354.5
866.2
1401.9
777.2
1227.8
235.2
6952.0
NameThisGame
1468.
1992.7
1826.7
2614.5
2460.0
2782.8
1919.8
2377.7
2087.3
2155.2
1994.8
2570.3
2153.4
2471.9
1964.2
2314.8
2136.8
8049.0
-19.6
-8.5
-17.3
20.7
6.6
-17.9
-2.0
21.0
-20.4
15.0
Privateliye
0.0
98.9
75.0
82.8
0.0
100.0
76.6
100.0
75.0
96.9
60.9
100.0
96.9
99.3
100.0
4038.7
26.6
6957100
Obent
702.7
475.0
Riverrid
1416.1
1929.4
1784.4
2274.5
2360.0
2639.8
1799.4
2158.4
2053.8
2307.5
2143.6
2221.2
1387.8
1759.8
1345.5
1923.4
1451.0
17118.0
RoudRanner
5901.6
8484.4
781.2
2857.8
$906.2
11176.6
2804.7
10676.6
1620.3
4104.7
7032.8
14978.1
857.8
1342.2
2717.2
8560.9
0.0
7845.0
Seaquest
414.4
768.1
236.9
470.6
711.6
854.1
386.9
497.2
330.9
551.2
332.8
460.9
274.1
317.2
366.9
527.2
61.1
42055.0
UpNDown
1195.9
2071.
1007.5
1315.2
1616.
8614.5
2389.5
3798.3
1433.3
1622.0
1248.6
1999.4
1670.3
2728.0
1825.2
5193.1
488.4
11693.0
YarsRevenge
3047.0
3380.5
3416.3
4230.8
6580.2
7547.4
2435.5
2955.9
3434.8
4276.3
6673.1
$4577.0

---

Published as a conference paper at ICLR 2020
E BASELINES OPTIMIZATION
To assess the performance SimPle we compare it with model-free algorithms. To make this com- parison more reliable tuned Rainbow in the low data regime. To this end hyperparame- ter search over the following parameters from httptttts:///ttuuiiii com cow.ooogloococoogocooooooooooo google/ /dppamiiee blob/ hltp:///t///ppatta////e/////////
in [1, 3), best parameter 3
man_reepaay_history in (500, 5000, 20000), best parameter 20000
indate_periood ( 4), best parameter= 4
target_update__period (50, 100, 1000, 4000), best parameter 8000
replay_scheme in [uniforme prwppriiiiieeeee best parameter = =prioritized
Each set of hyperparameters was used to train 5 Rainbow agents on the game of Pong until 1 million of interactions with the environment. Their average performance used to pick the best hyperparameter set.
For PPO used the standard set of hyperparameters from http:s/s//tttub..cmmpp.....a baselines
F RESULTS AT DIFFERENT NUMBERS OF INTERACTIONS
25

---

Published as a conference paper at ICLR 2020
value numbers
value numbers
kung_fu_master master
kung_ Lung_N_me __mtter
road runner
crazy climber
boxing
qbert
obert
asterix
crazy_climber
roed_runmen runner
freeway
freeway
boxing
ms_pacmman
amider
seaquest
breakout
kangeroo
assault
breakout
nattle_zone
alien
allen
up_n_down
assault
frostbite
chopper command
battie_zone
frostbite
jamesbond
n_dw
@ropper_comandd
omidar
kangaroo
jamesbond
hero
hero
bens_helst
gopher
gopher
bank
demon attack
demon attack
private_eye _aye
private eye
(a) Fraction 100K clipped to 10.
(b) Fraction at 200K
numbers
value numbers
freeway
freeway
uis_pacman
pong
ms pacman
crazy_climber
kung_fo fu_master
sang_tu_master
crazy climber
allen
allen
asterix
asterix
breakout
breakout
down
seaquest
amidar
gopher
amidar
assault
road_rumte Junner
phopper_commane
hero
obert
road runner
krull
seaquest
gopher
chopper command
hero
abert
jamesbond
jamesbond
boxing
boxing
frestbite
frostbite
nontee_onneeeeee
zont
bank heist
bank heist
demon_atten attack
demon_attann attack
private eye
private eye
kangaroo
kangaroo
0.0
0.2
0.4
0.8
0.8
1.0
1.2
0.0
0.2
0.4
0.8
0.8
1.0
1.2
(c) Fraction 500K.
(d) Fraction 1M.
Figure 12: Fructions of the rainbow scores at given number of samples. These were calculate with the formula PSemPLee_s eccoesecreeee random_ random_score) cacoemnsscoeeeeeeee ( _scorsseceeeee random_score denominator is smaller than O, both nominator and denominator are increased
26

---

Published as a conference paper at ICLR 2020
value numbers
value numbers
gopher
pong
pong
esterix
ms. pacman
boxing
bank_hee heist
abert
ms_ papamaa
runner
crazy climber
crazy climber
allen
asterix
road runner
alien
hero
breakout
seaquest
muns_ru_master master
gopher
jamesbond
kung fu master
freeway
freeway
frostbite
seaquest
amicar
frostbite
jamesbond
up n, dowwwn
bank heist
moopprr_command
chapper_comeee
amidar
up_n_down
demon_attack
eetonnattack
krull
krull
zone
karigaroo
kangaroo
private_eyy _eye
privat_eee
(a) Fraction 100K clipped to 10.
(b) Fraction at 200K
value_numbers
value numbers
pong
allen
ms. papmama
ms mns_pacman
alien
crazy_climber
breakout
hero
crazy_ climber
freeway
baxing
seaquest
fung_fu_nng fu_ master
frostbite
frastbite
treeway
road_ jund_r
seaquest
chopper_comee _ommandd
amidar
asterix
abert
asterix
gopher
gopher
kung_ master
amidar
assault
chooper_command
jamesbond
up_n_down
@emon_pttack
Jamesbond
knull
up nn nown
demon_attact attack
battle
battle, Looe
kangareo
bank_heis heist
bank heist
kangaroo
private_eye
private_eyy eeye
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.00
0.25
0.30
0.75
1.00
1.25
1.50
1.75
2.00
(c) Fraction 500K.
(d) Fraction 1M.
Figure 13: Fractions of the ppo scores at given number of samples. These were calculate with the formula SiemPLeescome random__satt ( hspooppp__o___ score random__scoreee if denominator is smaller than both nominator and denominator are increased by
27

---

Published as a conference paper at ICLR 2020
crany_climber
crany crany_climber
road runner
runder
kung_fu, mnnster master
kung fu, master
boxing
bexing
freeway
freeway
breakout
breakout
assault
ms mo__acman
ms_ pacman
assault
jamesbond
obert
asterix
alien
obort
battie_zons zone
alien
seaquest
amider
chapper command
frestbite
frostbite
battle_
battie_zonee zone
kangaroo
goopper_commang
seaquest
private_cd _yee
kangaroo
up_n_down
bank_ heist
hero
private_eyy _eye
jamesbond
hero
bank_heist
gopher
gopher
demon attack
krull
krull
demon_attace attack
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Human-nommalized difference
Human-normaliize difference
SimPI.e compared Rainbow 100K.
(h) SimPI.c compared Rainbow 200K
pong
pong
crezy_climber
crazycllimber
boxing
boxing
roaderunner
ran_rrnner
kung_fu_nee lec_fuummase mustter
breakout
breakkuut
ms_pacman
freeway
kung fu master
assault
freeway
pacman
assault
gopher
asterix
jamesbond
asterix
qbert
up nown
gopher
obert
allen
hero
frostbite
alien
bank
beikt heist
emidar
frestbite
bank heist
chopper_command
jamesbond
seaquest
chopper_commans _ommand
amidar
private_ee _yee
private_es eye
kangaroo
kangaroo
demon attack
demon_attack
demon attack
battle zone
battle zone
down
krull
knlll
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Human-normalized difference
Human-normalized difference
(c) SimPLe compared to PPO 100K.
(d) SimPLe compared to PPO 200K.
Figure 14: Comparison of scores from Simple against Rainbow PPO at different numbers of interactions. The following formula used: (SimPl normalized by average human score in order to be presentable in graph.
boseline bo.selinesccree score) / haman_scome Points are
28


Reviews:
[{"review_id": "SJg3esx3tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper covers the author’s approach to learning a model of a game, which can then be used to train a reinforcement learning agent. The major benefit of the approach is that instead of requiring millions of training steps in the game, the model can be constructed with only 1-2 hours of footage, and then train in the simulated game for millions of training steps. \n\nThis is a well-written paper, and the results are very impressive. The approach builds upon prior work with the same general thrust, but broadly makes clear that it stands above these existing approaches. However, I would have appreciated some clarity in the comparison made to the work of Ha and Schmidhuber (2018). It is unclear if the difference given is just because of the environments employed by Ha and Schmidhuber or if the authors see the approach presented in this paper as fundamentally different or improved in some way. \n\nMy one major technical concern comes down to how this work is framed and what that implies about appropriate baselines. The authors state clearly that the benefit of this work is that it can learn a sufficient model of a game in only 1-2 hours of gameplay footage. As I said above that is very impressive. However, the agents then requires 15.2 million interactions in this environment to learn to play the game. I would have appreciated some clarity then in the computational resource cost in this approach as opposed to just training say Rainbow in the actual game environments with 15.2 million interactions. It’s also not clear if optimizing Rainbow’s performance on 1M steps is a fair comparison. Ideally I would have liked to have seen some variance in the amount of time Rainbow was trained for compared to the associated computational costs. Clarity on this especially in sections like 6.1 would help readers better grasp the tradeoffs of the approach.\n\nThe authors could have also included reference to non-DNN work on learning forward/engine/world models that were then used to play or reason about the game. For example Guzdial and Riedl’s 2017 “Game Engine Learning from Gameplay Video” on Super Mario Bros. or Ersen and Sariel’s 2015 “Learning behaviors of and interactions among objects through spatio–temporal reasoning” on a novel game.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1lwnmopFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses sample-efficient learning (~2 hours of gameplay equivalent) for Atari (ALE) games. Building on the idea of training in a learned world model and the use of a u-net next-frame predictor, the approach is claimed to yield almost comparable performance to other models with only a fraction of the true-environment experience.\n\nSample efficiency is a major concern for DRL, particularly with an eye towards robotics and other physical domains. Although the approach is rather specific to the shapes and qualities of data in the ALE setting, the work is motivated at a high level, and the specific techniques for predicting the next frame explained in the past are explained.\n\nThis reviewer moves for a weak accept on account that the paper is well written (with quite thorough experiments explaining improvements in sample efficiency and possible limits in final task performance) but specifically targets ALE where execution is so cheap. The total number of PPO updates made in the new approach is not much reduced from before even if the number of trajectories evaluated in the true environment is very much reduced. On the problem of how much RL itself is sample efficient, not much progress is made.\n\nQuestion:\n- What is the impact on total wall-clock training time when using this approach? Given that the technique is centered on ALE, the characteristics of ALE compared to the learned world model are relevant (ALE executes very quickly and easily parallelizes whereas the learned world model presumably only runs where you have a GPU).\n- Can this approach be stacked to benefit from training in a lighter-weight approximate model (env'') of the world model (env')?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byg8y2bUKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a model-based reinforcement learning algorithm suitable for high-dimensional visual environments like Atari. The algorithmic loop is conceptually simple and comprises 1) collecting real data with the current policy 2) updating an environment model with old and newly acquired data and 3) updating the policy \"virtually\" inside of the environment model using PPO. The approach is evaluated on 26 games from the Atari benchmark and compared against the model-free baselines Rainbow DQN and PPO. The newly proposed model-based method clearly outperforms both model-free baselines in low training regimes (100,000 steps). Further ablation studies are provided, e.g. similar results are obtained in a more stochastic setting of ALE with sticky actions.\n\nQuality\n\nThis paper has a strong applied focus and needs to be judged based on its experiments. The quality of those are high. The method is evaluated on a suite of 26 games, compared to strong model-free baselines and results are averaged over 5 seeds. One concern I have is that the method is only evaluated in low training regimes. While I do understand that increasing the training horizon is computationally demanding, results in the appendix (Figure 11a) indicate that the proposed model-based method has worse asymptotic performance compared to the model-free baselines. After 500,000 training steps the effect of sample efficiency vanishes and the final performance results are far away from the final performance results of the model-free baselines after 50,000,000 training steps. Also, a plot similar to Figure 11a) from the appendix for Rainbow DQN would be good (but I do understand this might be difficult to obtain in the course of the review period should this require more experiments).\n\nClarity\n\nThe paper is clearly written and easy to follow. However, the authors could state in the main paper more clearly that their method excels in low training regimes and that the sample efficiency effect seems to vanish when increasing training iterations from 100,000 to 500,000 steps. In fact, Figure 11a) from the appendix should go into the main paper, and it should be also mentioned that there is a huge discrepancy between the maximum performance achieved by the proposed model-based method and the maximum performance achieved by the model-free baselines when training for 50,000,000 steps. Based on the experiments, it is not clear at all if the new method will eventually catch up with best model-free results from the literature when training time is increased, or stall in low-performance regimes indefinitely.\n\nOriginality\n\nThe originality of this paper is not very high since the proposed algorithm and its components are not novel (there might be some minor novelty in the environment model architecture). However, this paper should not be judged based on its originality but based on its significance. \n\nSignificance\n\nA working model-based RL algorithm for Atari is clearly a huge gap in the current literature and this paper takes an important step towards this direction. Demonstrating improved sample efficiency compared to strong model-free baselines in low training regimes is a significant result. The significance is however decreased by the fact that the paper does not answer the question how to obtain good asymptotic performance that matches (or comes close to) model-free state-of-the-art results. I therefore vote for weak accept at this stage.\n\nMinor details\n\nOn a side note, there are two citations missing related to model-based RL in visual domains:\n- S. Alaniz. Deep Reinforcement Learning with Model Learning and Monte Carlo Tree Search in Minecraft. In the 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2017.\n- F. Leibfried and P. Vrancx. Model-Based Regularization for Deep Reinforcement Learning with Transcoder Networks. In NIPS Deep Reinforcement Learning Workshop, 2018.\n\nUpdate\n\nAfter the author response, my review remains the same. I think this paper is worthwhile publishing at ICLR.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model-based RL approach to Atari games based on video prediction. The architecture performs remarkably well with a limited amount of interactions.  This is a very significant result on a question that engages many in the research community.

Reviewers all agree that the paper is good and should be published. There is some disagreement about the novelty of it. However, as one reviewer states, the significance of the results is more important than the novelty. Many conference attendees would like to hear about it.

Based on this, I think the paper can be accepted for oral presentation.
Decision: Accept (Spotlight)