SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference
Authors: Lasse Espeholt
Raphaël Marinier
Piotr Stanczyk
Ke Wang
Marcin Michalski‎
Venue:  (2020)
Keywords: machine learning
reinforcement learning
scalability
distributed
DeepMind Lab
ALE
Atari-57
Google Research Football

Abstract: We present a modern scalable reinforcement learning agent called SEED (Scalable, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions of frames per second but also to lower the cost. of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state-of-the-art distributed algorithms, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and is evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 twice as fast in wall-time. For the scenarios we consider, a 40% to 80% cost reduction for running experiments is achieved. The implementation along with experiments is open-sourced so results can be reproduced and novel ideas tried out.

Published as a conference paper at ICLR 2020
SEED RL: SCALABLE AND EFFICIENT DEEP-RL WITH ACCELERATED CENTRAL INFERENCE
Lasse Espeholt; Raphaël Marinier, Piotr Stanczyk; Ke Wang & Marcin Michalski
Brain Team
Google Research
(lespehelt, raphaelm, stanczyk, kewa, michalski @egoggll.coooooooooooooo
ABSTRACT
We present modern scalable reinforcement learning agent called SEED (Scal- able, Efficient Deep-RL). By effectively utilizing modern accelerators, we show that it is not only possible to train on millions frames per second but also to lower the cost of experiments compared to current methods. We achieve this with a simple architecture that features centralized inference and an optimized communication layer. SEED adopts two state of the art distributed algorithms,, IMPALA/V-trace (policy gradients) and R2D2 (Q-learning), and evaluated on Atari-57, DeepMind Lab and Google Research Football. We improve the state of the art on Football and are able to reach state of the art on Atari-57 three times faster in wall-time. For the scenarios for running experiments is achieved. The implementation along with experiments is open-sourced results can be reproduced and novel ideas tried out.
consider, 40% 80% cost reduction
Github: http://ggaaab..cogoogggeeeeeeeee
INTRODUCTION
The field of reinforcement learning (RL) has recently seen impressive results across variety of tasks. This has in part been fueled by the introduction of deep learning in RL and the introduction of accelerators such as GPUs. In the very recent history, focus massive scale has been key to solve a number of complicated games such as AlphaGo (Silver et al. 2016), Dota (OpenAl 2018) and StarCraft 2 (Vinyals et al. (017))
The sheer amount environment data needed to solve tasks trivial to humans, makes distributed machine learning unavoidable for fast experiment turnaround time. RL is inherently comprised of heterogeneous tasks: running environments, model inference, model training, replay buffer, etc. and current state-of-the-ar distributed algorithms do efficiently use compute resources for the tasks. The amount of data and inefficient use of resources makes experiments unreasonably expensive. The two main challenges addressed in this paper are scaling of reinforcement learning and optimizing the use of modern accelerators, CPUs and other resources.
We introduce SEED (Scalable, Efficient, Deep-RL), a modern RL agent that scales well, is flexible and efficiently utilizes available resources. It is a distributed agent where model inference is done centrally combined with fast streaming RPCs to reduce the overhead of inference calls. We show that with simple methods, one can achieve state-of-the-art results faster on number of tasks. For optimal performance, we use TPUs www.ld.gooll.ccool.. to simplify the implementation. The cost of running SEED is analyzed against IMPALA (Espeholt ctt al. 2018) which is a commonly used state-of-the-ar distributed RL algorithm (Vecriah et al. (2019 Li et al. (2019); Deverett al.) (019);; Omidshatiei et al. (2019); Vezhnevets et al. (2019); Hansen et al. 2019); Schaarschmidt et al.; Tirumala et al. (2019). ...). We show cost reductions of up 80% while being significantly faster. When scaling SEED to many accelerators, can train on millions of frames per second. Finally, the implementation is open-sourced together with examples
and TensorFlow (Abadi et al.) 2015)
of rumning it at scale on Google Cloud (see Appendix A.4 for details) making it easy to reproduce results and try novel ideas.
"qqual contribution

---

Published as a conference paper at ICLR 2020
2
RELATED WORK
For value-based methods, an early attempt for scaling DQN was Nair et al. (2015) that used asyn- chronous SGD (Dean et al., 2012) together with a distributed setup consisting of actors, replay buffers, parameter servers and learners. Since then, it has been shown that asynchronous SGD leads to poor sample complexity while not being significantly faster (Chen el al., 2016; [spehhot et al. 2018). Along with advances for Q-learning such as prioritized replay (Schaul et al. | 2011), dueling networks (Wang et al., 2016). and double-Q learning (van Hasselt 2010] Van Hasselt et al. 2016) the state-of-the-an distributed Q-learning was improved with Ape-X (Horgan et al., 2018). Recently, R2D2 (Kapturowski et al., 2018) achieved impressive results across all the Arcade Learning Envi- ronment (ALE) (Bellemare et al., 2013) games by incorporating salue-function rescaling (Pohlen et al. 2018 and LSTMs (Hochreiter & Schmidhuber 1997) top of the ad advancements of Ape-X.
There have also been many approaches for scaling policy gradients methods. A3C (Mnih et al.) 2016) introduced asynchronous single-machine training using asynchronous SGD and relied exclu- sively on CPUs. GPUs were later introduced in GA3C (Mahmood (017)) with improved speed but poor convergence results due to an inherently on on-policy method being used in an off ooolcc setting. This was corrected by V-trace (Espeholt et al. 2018) in the IMPALA agent both for single- machine training and also scaled using simple actor-learner architecture more than thou- sand machines. PPO (Schulman et al., 2017) serves similar purpose to V-trace and was used in OpenAl Rapid (Petrov et al. 2018) with the actor-learner architecture extended with Redis (redis.io), an in-memory data store, and was scaled to I 128,000 CPUs. For inexpensive environments like ALE, single machine with multiple accelerators can achieve results quickly (Stooke & Abbeel 2018). This approach was taken a step further by converting ALE to run GPU (Dalton et al. 2019).
third class of algorithms evolutionary algorithms. With simplicity and massive scale, they have achieved impressive results on a number of tasks (Salimans et al. 2017 Such et al. 2017).
Besides algorithms, there exist a number of useful libraries and frameworks for reinforcement learn- ing. ELF TTan et al.) 2017) is framework for efficiently interacting with environments, avoiding Python global-interprete--lokk contention. Dopamine (Castro et al. 2018) is a flexible research focused RL framework with a strong emphasis on reproducibility. It has state of the art agent imple- mentations such as Rainbow (Hessel et al. (017) but is single-threaded TF-Agents (Guadarramal et al. 2018) and rlpyt (Stooke & Abbeel 2019) both have broader focus with implementations for several classes of algorithms but as of writing, they do not have distributed capability for large- scale RL. RLLib (Liang et al., (017) provides number of composable distributed components and a communication abstraction with a number of algorithm implementations such as IMPALA and Ape-X. Concurrent with this work, TorchBeast (Küttler al., 2019) was released which is an implementation of single-machine IMPALA with remote environments.
SEED closest related to IMPALA, but has number of key differences that combine the benefits of single-machine training with a scalable architecture. Inference is moved to the learner but envi- ronments run remotely. This is combined with a fast communication layer to mitigate latency issues from the increased number of remote calls. The result is significantly faster training at reduced costs by as much as 80% for the scenarios we consider. Along with a policy gradients (V-trace) implemen- tation we also provide an implementation of state of the art Q-learning (R2D2). In the work use TPUs but in principle, any modern accelerator could be used in their place. TPUs are particularly well-suited given they high throughput for machine learning applications and the scalability. Up to 2048 cores are connected with a fast interconnect providing 100+ petaflops of compute.
ARCHITECTURE
Before introducing the architecture of SEED, we first analyze the generic actor-learner architecture used by IMPALA, which is also used in various forms in Ape-X, OpenAl Rapid and others. An overview of the architecture is shown in Figure Tal
A large number of actors repeatedly read model parameters from the learner (or parameter servers). Each actor then proceeds the local model to sample actions and generate a full trajectory of observa- tions, actions, policy logits/Q-values. Finally, this trajectory along with recurrent state is transferred
2

---

Published as a conference paper at ICLR 2020
Artor
Learner
Optlimias
Enviroement
Pacunadies
Model
Obscrvation
Buteh trajectorios
Model
Queese
4. Collec
Weaeetooc
Store
b.Trjjectoyy (State obsevvations)
(a) IMPALA architecture (distributed version)
Leurner
Optirison
Actor
1. Oberrentio
Model
Environment
Taathing Ioper
trajectories
Colieet
Quesue
Store
Trajestury
(b) SEED architecture, see detailed replay architecture in Figure
Figure I: Overview of architectures
to a shared queue or replay buffer. Asynchronously, the learner reads batches trajectories from the queue/replay buffer and optimizes the model.
There are a number of reasons for why this architecture falls short:
1. Using CPUs for neural network inference: (occasionally GPU-based for expensive environments). CPUs are known to be computa- tionally inefficient for neural networks (Raina al. 2009). When the computational needs of model increase, the time spent on inference starts to outweigh the environment step computation. The solution is to increase the number of actors which increases the cost and affects convergence (Espeholt al., 2018).
The actor machines usually CPU-based
2. Inefficient resource utilization:
Actors alternate between two tasks: environment steps and inference steps. The compute requirements for the two tasks are often not similar which leads to poor utilization or slow actors. E.g. some environments are inherently single-threading while neural networks are easily parallelizable.
3. Bandwidth requirements: Model parameters, recurrent state and observations are trans- ferred between actors and learners. Relatively to model parameters, the size of the ob- servation trajectory often only accounts for a few percents Furthermore, memory-based models send large states, increase bandwidth requirements.
While single-machine approaches such as GA3C (Mahmood 2017) and single-machine IMPALA avoid using CPU for inference (T) and do not have network bandwidth requirements (3). they are restricted by resource usage and the scale required for many types of environments.
The architecture used in SEED (Figure Tb) solves the problems mentioned above. Inference and trajectory accumulation is moved to the learner which makes it conceptually a single-machine setup with remote environments (besides handling failures). Moving the logic effectively makes the actors a small loop around the environments. For every single environment step, the observations are sent the learner, which runs the inference and sends actions back the actors. This introduces new problem: 4. Latency.
To minimize latency, we created simple framework that uses gRPC (grpc.io) a high performance RPC library. Specifically, we employ streaming RPCs where the connection from actor to learner is kept open and metadata sent only once. Furthermore, the framework includes a batching module that efficiently batches multiple actor inference calls together. In cases where actors can fit the same machine as learners, gRPC uses unix domain sockets and thus reduces latency, CPU and syscall overhead. Overall, the end-to-end latency, including network and inference, is faster for number of the models we consider (see Appendix A.7
With 100,000 observations send per second (96 72 3 bytes each), trajectory length of 20 and 30MB model, the total bandwidth requirement is 148 GB/s. Transferring observations uses only 2 GB/s.
3

---

Published as a conference paper at ICLR 2020
Upettatttiv step
Lactre
Pras
Aeter
Pestmerriect step
Isterincce
(a) Off-pollicy in IMPALA. For the entire trajectory the policy stays the same. By the time the trajectory is sent the queue for optimization, the policy has changed twice.
(h) Off-policy in SEED. Optimizing model has im- mediate effect the policy. Thus, the trajectory sists of actions sampled from many different policies (Ta, TT.
Figure 2: Variants of "near on on-policy" when evaluating policy x while asynchronously optimizing model parameters 8.
Learner
Reporting and checkpointing
Inferennce TPUs
Training TPUs
Tralning lirrad
Apply gradients
Model
Devion butfer
Data prefeechingg threads
Recurrent states
Incompkete trsjectories
Complete sajeectries queae
Prioritbedd replay bafler
Batching
Figure Detailed Learner architecture in SEED (with optional replay buffer).
The IMPALA and SEED architectures differ in that for SEED, at any point in time, only one copy of the model exists whereas for distributed IMPALA each actor has its own copy. This changes the way the trajectories are off off-col In IMPALA (Figure [agg an actor uses the same policy Ta, for an entire trajectory. For SEED (Figure 2 the policy during unroll of trajectory may change multiple times with later steps using more recent policies closer to the one used at optimization time.
A detailed view of the learner in the SEED architecture is shown on Figure Three types of threads are running: 1. Inference 2. Data prefetching and 3. Training. Inference threads receive a batch of observations, rewards and episode termination flags. They load the recurrent states and send the data to the inference TPU core. The sampled actions and new recurrent states are received, and the actions sent back the actors while the latest recurrent states are stored. When trajectory is fully unrolled it is added to FIFO queue or replay buffer and later sampled by data prefetching threads. Finally, the trajectories are pushed to a device buffer for each of the TPU cores taking part in training. The training thread (the main Python thread) takes the prefetched trajectories, computes gradients using the training TPU cores and applies the gradients on the models of all TPU cores (inference and training) synchronously. The ratio of inference and training cores can be adjusted for maximum throughput and utilization. The architecture scales to TPU pod (2048 cores) by round- robin assigning actors to TPU host machines, and having separate inference threads for each TPU host. When actors wait for a response from the learner, they are idle so in order to fully utilize the single actor.
machines, we run multiple environments
To summarize, we solve the issues listed previously by:
1. Moving inference to the learner and thus eliminating any neural network related computa- tions from the actors. Increasing the model size this architecture will not increase the need for more actors (in fact the opposite is true).
2. Batching inference on the learner and having multiple environments on the actor. This fully utilize both the accelerators the learner and CPUs on the actors. The number of
4

---

Published as a conference paper at ICLR 2020
TPU cores for inference and training is finely tuned to match the inference and training workloads. All factors help reducing the cost of experiments.
3. Everything involving the model stays on the learner and only observations and actions are sent between the actors and the learner. This reduces bandwidth requirements by as much as 99%.
4. Using streaming gRPC that has minimal latency and minimal overhead and integrating batching into the server module.
We provide the following two algorithms implemented in the SEED framework: V-trace and Q- learning.
3.1 V-TRACE
One of the algorithms we adapt into the framework is V-trace (Espeholt et al. 2018). We do not include any the additions that have been proposed on top of IMPALA such as van den Oord al. 2018); Gregor et al. (2019). The additions can also be applied to SEED and since they are more computational expensive, they would benefit from the SEED architecture.
3.2 Q-LEARNING
We show the versatility of SEED's architecture by fully implementing R2D2 Kapturowski al. 2018), a state of the art distributed value-based agent. R2D2 itself builds on a long list of improve- ments over DQN (Mnih al., 2015): double Q-learning (van Hasselt 2010, Van Hasselt et al.) 2016). multi-step bootstrap targets (Sutton 1988 Suttonn & Barto] 1998; Mnih et al. (016), du- eling network architecture (Wang al.) 2016), prioritized distributed replay buffer (Schaul et al.,
2015 Horgan et al. 20118) value-function rescaling (Pohlen et al. 2018). LSTM's (Hochreiter & Schmidhuber 1997 and burn-in (Kapturowski al. 2018).
Instead of a distributed replay buffer, we show that it is possible to keep the replay buffer on the learner with a straightforward flexible implementation. This reduces complexity by removing one type of job in the setup. It has the drawback of being limited by the memory of the learner but it was a problem in our experiments by large margin: a replay buffer of 10 trajectories of length 120 of 84 x 84 uncompressed grayscale observations (following R2D2's hyperparameters) takes 85GBs of RAM, while Google Cloud machines can offer hundreds of GBs. However, nothing prevents the use of distributed replay buffer together with SEED's central inference, in cases where much larger replay buffer is needed.
EXPERIMENTS
We evaluate SEED on number of environments: DeepMind Lab (Beattie et al. 2016), Google Research Football (Kurach et al. (019) and Arcade Learning Environment (Bellemare et al. 2013
4.1 DEEPMIND LAB AND V-TRACE
DeepMind Lab is a 3D environment based on the Quake engine. It features mazes, laser tag and memory tasks. We evaluate on four commonly used tasks. The action set used is from |Espeholt ct al. (2018) although for some tasks, higher return can be achieved with bigger action sets such as the introduced in Hessel et al. (018)). For all experiments, the number of frames plots is listed as environment frames (equivalent to times the number of steps). The same set of 24 hyperparameter sets and the same model (ResNet from IMPALA) was used for both agents. More details can be found in Appendix A.1.2
used action repeat of 4 and
4.1.1 STABILITY
The first experiment evaluates the effect of the change in off-policy behavior described in Figure 2 Exactly the same hyperparameters environments used. As is shown in Figure the stability across hyperparameters of SEED is slightly better than IMPALA, while achieving slightly higher final returns.
used for both IMPALA and SEED, including the number of
5

---

Published as a conference paper at ICLR 2020
IMPALA 175
TPU 175
keys, doors, pozzle
Insortag twree_eoreepeeeeeeeeeee www.ooppooeessssee small
explore_goo Bacee_ogo___aaaaaae locations_ssn small
Final Return
80
200
20
20
100
17 Hyperparameter Combination
13
17
21 Hyperparameter Comhination
Hyperpammeter Combination
Hyperparameter Combination
Figure 4: Comparison of IMPALA and SEED under the exact same conditions (175 actors, same hyperparameters, etc.) The plots show hyperparameter combinations sorted by the final performance across different hyperparameter combinations.
IMPALA- 175
TTU 1560
SEED TPU v3 cures, 312
SEED TPU 61 cores, 12,180
explore_godl Jowpooooootttttssoo snsall
rooms Watermaze
rooms Booms__oys__oo docss_ _
lasertag three_ opponeets ssmall
40
Retum
200
20-
20
300
10
0.00
0.25
0.50
0.75
1.00
0.25
0.50 Frame
0.75
1.00
0.25
0.50
0.75
1.00
Frame
lc9
Frame
1c9
1e9
Prame
169
explore _aalore_gaa______ locations, small
keys, doors, p
lnsertay three opponento small
30
Return
200
20
100
20-
10
5.0
7.5
0.0
2.5
5:0 Hours
7.5
2.5
5.0
7.5
0.0
2.5
5.0
Hours
Hours
Figure 5: Training on DeepMind Lab tasks. Each curve is the best of the 24 runs based on final return following the evaluation procedure in Emspehhot et al. al. (2018). Sample complexity is maintained up to TPU v3 cores, which leads to 11x faster training than the IMPALA baseline. Top Row: X- axis is per frame (number of frames 4x number of steps). Bottom Row: X-axis is hours.
4.1.2 SPEED
For evaluating performance, compare IMPALA using Nvidia P100 with SEED with multiple accelerator setups. They are evaluated on the same set of hyperparameters. 2.5x faster than IMPALA using? 2 TPU v3 cores (see Table D while using only 77% more envi- ronments and 41% less CPU (see section (.4.I). Scaling from 2 to 8 cores results in an additional 4.4x speedup with sample complexity maintained (Figure 5). The speed-up is greater than 4x due to using 6 cores for training and 2 for inference instead core for each, resulting in better utiliza- tion. 5.3x speed-up instead of 4.4x can be obtained by increasing the batch size linearly with the number of training cores, but contrary to related research (You et al. 2017b) Goyal et al. (017) we found that increased batch size hurts sample complexity even with methods like warm-up and actor de-correlation (Stooke & Abbeel 2018). We hypothesize that this is due to the limited actor and environment diversity in DeepMind Lab tasks. McCanddish et al. (2018) they found that Pong scales poorly with batch size but that Dota can be trained effectively with batch size five orders of magnitude larger. Note, for most models, the effective batch size is batch size trajectory length. In Figure [ we include
find that SEED
from a limited sweep "explore goal locations "mall" using 64 cores with an almost linear speed-up. Wall-time performance is improved but sample complexity is heavily penalized.
When using an Nvidia P100, SEED is 1.58x slower than IMPALA. slowdown is expected because SEED performs inference on the accelerator. SEED does however use significantly fewer CPUs and lower cost (see Appendix(A.6) The TPU version of SEED has been optimized but it is likely that improvements can be found for SEED with P100.

---

Published as a conference paper at ICLR 2020
Architecture
Accelerators
Environments
Actor CPUs
Batch Size
FPS
Ratio
DeepMind Lab
IMPALA
Nvidia P100
176
176
32
30K
SEED
Nvidia P100
176
44
19K
0.63x
SEED
TPU v3, cores
312
104
74K
2.5x
SEED
TPU v3, 8 cores
1560
520
48
330K
11.0x
SEED
TPU v3, cores
12,480
4.160
384'
2.4M
80.0x
Google Research Football
IMPALA, Default
2 Nvidia P100
400
400
128
11K
SEED, Default
TPU cores
624
416
128
18K
1.6x
SEED, Default
TPU v3, 8 cores
2,496
1,664
160*
71K
6.5x
SEED, Medium
TPU v3, 8 cores
1,550
1,032
160
44K
SEED, Large
TPU v3, 8 cores
1,260
840
160
29K
SEED, Large
TPU v3, cores
5,040
3,360
640
114K
3.9x
Arcade Learning Environment
R2D2
Nvidia V100
256
N/A
85K²
SEED
Nvidia V100
256
55
64
67K
0.79x
SEED
TPU v3, 8 cores
610
213
64
260K
3.1x
SEED
TPU v3, cores
1200
419
256
440K*
5.2x
cores used for training. Each the 256 R2D2 actors run at 335 (information from R2D2 authors). 8 5/8 cores used for training.
No frame stacking.
Table Performance of SEED, IMPALA and R2D2.
4.2 GOOGLE RESEARCH FOOTBALL AND V-TRACE
Google Research Football is environment similar to FIFA video games (ea.com). We evaluate SEED on the "Hard" task introduced in Kurach et al.] (2019) where the baseline IMPALA agent achieved a positive average score after 500M frames using the engineered "checkpoint" reward function but negative average score using only the score as a reward signal. In all experiments we the model from Kurach et al. (2019) and sweep 40 hyperparameter with 3 seeds each. See all hyperparameters in Appendix A.2.1
4.2.1 SPEED
Compared to the baseline IMPALA using 2 Nvidia P100's (and CPUs for inference) in Kurach ctt al. (2019) we find that using 2 TPU v3 cores SEED improves the speed by 1.6x (see Table []. Additionally, using cores adds another 4 speed-up. A speed-up of 4.5x is achievable if the batch size increased linearly with the number of training cores (5). However, we found that increasing the batch size, like with DeepMind Lab, hurts sample complexity.
4.2.2 INCREASED MAP SIZE
increase the size of the Super Mini Map (SMM) input state. By default its size is 96 72 (x 4) and it represents players, opponents, ball and the active player as 2d bit maps. We evaluated three sizes: (1) Default 96 72, (2) Medium 120 x 90 and (3) Large 144 108. As shown in Table ① switching from Default to Large SMM results in 60% speed reduction. However, increasing the map size improves the final score (Table It may suggest that the bit map representation is not granular enough for the task. For both reward functions, we improve upon the results of Kurach et al. (2019). Finally, training on 4B frames improves the results by a significant margin (an average score of 0.46 vs. 4.76 in case of the scoring reward function).
More compute power allows
4.3 ARCADE LEARNING ENVIRONMENT AND Q-LEARNING

---

Published as a conference paper at ICLR 2020
2000%
R2D2(120h)
R202(120h)
Human- numan-noormalzzs Score
(Median S7 games)
R2D2(72h)
1500%
SEED cores
SEED cores
R2D2(48h)
R2D2(48h)
1000%-
R2D2(24h
(2D2(24
500%
R2D2d
Ape- 20))
Ape X(70h)
(pe-X(120h)
R2D2(12h)
Ape-X(120b)
((0h)
Are x(70)
(120h
(120)
0%
0
20
40
60
80
100
120
2
Training time (Hours)
Environment frames
1e10
Figure 6: Median human-normalized score Atari-57 for SEED and related agents. SEED was run with 1 seed for each game. All agents use up to 30 random no-ops for evaluation. Left: X-axis hours Right: X-axis environment frames (a frame is 1/4th of environment step due action repeat). SEED reaches state of the art performance 3.1x faster (wall-time) than R2D2.
We evaluate implementation of R21>2 in SEED architecture 57 Atari 2600 games from the ALE benchmark. This benchmark has been the testbed for most recent deep reinforcement learning agents because of the diversity of visuals and game mechanics.
We follow the same evaluation pro- cedure as R2D2. particular, we use the full action set, no loss-
Architecture
Accelerators
SMM
Median
Max
ei-iie----epissde---edd heuristic and
Scoring reward
start episodes with up to 30 ran- dom no-ops. We use TPU v3 cores and 610 actors to maximize TPU utilization. This achieves 260K environment FPS and per- forms 9.5 network updates per sec- ond. Other hyperparameters are taken from R2D2, and are fully re- produced appendix [
IMPALA
2 x Nvidia P100
Default
-0.74
0.06
SEED
TPU v3, 2 cores
Default
-0.72
-0.12
SEED
TPU v3, 8 cores
Default
-0.83
-0.02
SEED
TPU v3, 8 cores
Medium
-0.74
0.12
SEED
TPU v3, 8 cores
Large
-0.69
0.46
SEED
TPU v3, 32 cores
Large
n/a
4.76'
Checkpoint reward
IMPALA
2 Nvidia P100
Default
-0.27
1.63
Figure shows the median human- normalized scores for SEED, R2D2, Ape-X and Rainbow. As expected, SEED has similar sample efficiency as R2D2, but it is 3.1x faster (see Table []. This allows us to reach a median human- normalized score of 1880% in just 1.8 days of training instead of 5, establishing a new wall-time state of the art Atari-57.
SEED
TPU v3, 2 cores
Default
-0.44
1.64
SEED
TPU v3, 8 cores
Default
-0.68
1.55
SEED
TPU v3, 8 cores
Medium
-0.52
1.76
SEED
TPU v3, 8 cores
Large
-0.38
1.86
SEED
TPU v3, 32 cores
Large
n/a
7.66
32 core experiments trained 4B frames with limited sweep.
Table 2: Google Research Football "Hard" using two kinds of reward functions. For each reward function, 40 hyperparame- ter sets ran with 3 seeds each which were averaged after 500M frames of training. The table shows the median and maximum of the 40 averaged values. This is similar setup to Kurach et al. (2019) although we ran 40 hyperparameter sets vs. 100 but did not rerun our best models using 5 seeds.
With the number of actors in- creased to 1200, batch size in- creased to 256 and without frame- stacking, we can achieve 440K en- vironment FPS and learn using 16 batches per second. However, this significantly degrades sample efficiency and limits the final me- dian numan-normalle score to approximately 1000%.
4.4 COST COMPARISONS
With growing complexity of environments as well as size of neural networks used in reinforcement learning, the need of running big experiments increases, making cost reductions important. In this
8

---

Published as a conference paper at ICLR 2020
section analyze how increasing complexity of the network impacts training cost for SEED and IMPALA. In our experiments we use the pricing model of Google AI Platform, ML Engine
Our main focus is obtaining lowest possible cost per step, while maintaining training speed. Distributed exper- iments from Espeholt al. (2018) (IMPALA) used be- tween 150 and 500 CPUs, which translates into $7.125 $23.75 of actors' cost per hour. The cost of single-GPU learner is $1.46 per hour. Due to the relatively high ex- pense of the actors, our main focus is reduce number of actors and to obtain high CPU utilization.
Resource
Cost per hour
CPU core
$0.0475
Nvidia Tesla P100
$1 |46
TPU v3 core
$1.00
Table 3: Cost of cloud resources as of Sep. 2019.
4.4.1 DEEPMIND LAB
Our DeepMind Lab experiment is based on the ResNet model from IMPALA. We evaluate increas- ing the number of filters the convolutional layers: (1) Default 1x (2) Medium 2x and (3) Large 4x. Experiments are performed on the rexplore_goal goal_locations_sigg samalll task. IMPALA uses a single Nvidia Tesla P100 GPU for training while inference is done on CPU by the actors. SEED uses one TPU v3 core for training and one for inference.
For IMPALA, actor CPU utilization is close to 100% but in case of SEED, only the environment runs an actor making CPU idle while waiting for inference step. To improve utilization, a single SEED actor runs multiple environments (between 12 and 16) on 4PUUU machine.
Model
Actors
CPUs
Envs.
Speed
Cost/1B
Cost ratio
IMPALA
Default
176
176
176
30k
$90
Medium
130
130
130
16.5k
$128
Large
100
100
100
7.3k'
$236
SEED
Default
104
312
74k
$25
3.60
Medium
12
48
156
34k
$35
3.66
Large 6
24
84
16k
$54
4.37
The batch size was lowered from 32 to 16 due to limited memory Nvidia P100.
Table 4: Training cost on DeepMind Lab for billion frames.
As Table 4 shows, SEED turns out to be not only faster, but also cheaper to The cost ratio between SEED and IMPALA is around 4. Due to high cost of inference on a CPU, IMPALA's cost increases with increasing complexity of the network. In the case of SEED, increased network size has smaller impact on overall costs, as inference accounts for about 30% of the costs (see Table A.5)
4.4.2 GOOGLE RESEARCH FOOTBALL
We evaluate cost of running experiments with Google Research Football with different sizes of the Super Mini Map representation (the size has virtually no impact on environment's speed.) For IMPALA, two Nvidia P100 GPUs were used for training and SEED used one TPU v3 core for training and one for inference.
For IMPALA, for better CPU utilization (8 cores, 12 environments).
one core per actor while SEED's actors run multiple environments per actor
For the default size of the SMM, per experiment training cost differs by only 68%. As the Google Research Football environment is more expensive than DeepMind Lab, training and inference costs
2TPU cores are sold in multiples of 8, by running many experiments at once we use as many cores per experiment needed. See www.gggggg...oommmmonnneeeeeeeee

---

Published as a conference paper at ICLR 2020
Model
Actors
CPUs
Envs.
Speed
Cost/IB
Cost ratio
IMPALA
Default
400
400
400
11k
$553
Medium
300
300
300
7k
$681
Large
300
300
5.3k
$899
SEED
Default
52
416
624
17.5k
$345
1.68
Medium
248
310
10.5k
$365
1.87
Large
168
252
7.5k
$369
2.70
Table 5: Training cost on Google Research Football for billion frames.
have relatively smaller impact on the overall experiment cost. The difference increases when the size of the SMM increases and SEED needing relatively fewer actors.
4.4.3 ARCADE LEARNING ENVIRONMENT
Due lack of baseline implementation for R2D2, do not include comparisons for this environment. However, comparison of relative costs between ALE, DeepMind Lab and Football suggests that savings should be even more significant. ALE is the fastest among the three environ- ments making inference proportionally the most expensive. Appendix A.5 presents training cost split between actors and the learner for different setups.
5 CONCLUSION
We introduced and analyzed a new reinforcement learning agent architecture that is faster and less costly per environment frame than previous distributed architectures by better utilizing modern ac- celerators. It achieved a 1 Ix wall-time speedup on DeepMind Lab compared to strong IMPALA baseline while keeping the same sample efficiency, improved on state the art scores on Google Re- search Football, and achieved state of the art scores on Atari-57 3.1x faster (wall-time) than previous research.
The agent is open-sourced and packaged to easily run Google Cloud. We hope that this will accelerate reinforcement learning research by allowing the community to replicate state-of-the-art results and build on top of them.
As demonstration of the potential of this new agent architecture, we were able to scale it to millions of frames per second in some realistic scenarios (80x speedup compared to previous research). However, this requires increasing the number of environments and using larger batch sizes which hurts sample efficiency in the environments tested. Preserving sample efficiency with larger batch-
sizes has been studied to some extent in RL (Stooke & Abbeel 2018 McCandlish et al. 2018)) and in the context of supervised learning You al.. 201 a 2019 Goyal et al. 2017). We believe it is still an open and increasingly important area of research in order to scale up reinforcement learning.
ACKNOWLEDGMENTS
We would like to thank Steven Kapturowski, Georg Ostrovski, Tim Salimans, Aidan Clark, Manuel Kroiss, Matthieu Geist, Leonard Hussenot, Alexandre Passos, Marvin Ritter, Neil Zeghidour, Marc G. Bellemare and Sylvain Gelly for comments and insightful discussions and Marcin Andrychow- icz, Dan Abolafia, Damien Vincent, Dehao Chen, Eugene Brevdo and Ruoxin Sang for their code contributions.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew
10

---

Software Published as a conference paper at ICLR 2020
Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: /wwp:://wwwww.//. available from tensorflow.org.
/ww. www.ttesorflow.or orgg/ org/
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab, CoRR, abs/1612.03801,, 2016. URL http::/arxii: orgg abs / 1612.11. 03801
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents. J. Artif. Intell. Res. (JAIR), 47:253-279, 2013.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
//arxiiv.crg/aab//11220000
Jianmin Chen, Rajat Monga, Samy Bengio, and Rafal Józefowicz. Revisiting distributed syn- chronous SGD. CoRR, abs/1600.00981, 2016. URL http://arx.v..oog/ab//BBo 00981
Steven Dalton, Iuri Frosio, and Michael Garland. Gpu-accelerated atari emulation reinforcmmentt learning. CoRR, abs/1997.08467, 2019. URL
abaaabass/111// 08467)
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le. and Andrew Y. Ng. Large scale distributed deep networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp- 1223 1231. Curran Associates, Inc., 2012. URL [attp://papers ccppser//ppprr 4687 1687-larger large-
deep networks.pdf
Ben Deverett, Ryan Faulkner, Meire Fortunato, Greg Wayne, and Joel Z Leibo. Interval timing in deep reinforcement learning agents. arXiv preprint arXiv:/905. 13469, 2019.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukeuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamnr architectures. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1406-1415, 2018. URL http:
//proceedings, www.pigsssi....r.aa.e.e..
www.ppressssi... wtrraa//v0/espehottite.. html
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: training im- agenet in I hour. CoRR, abs/1706.02677, 2017. URL httt:://axxiv.orr/abbs//s D2677
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping belief states with generative environment models for RL. CoRR, abs/1906.09237, 2019. URL/attp://axxi> wwtprr//aaaaaaooeeeeeeee /996. C9237
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. https: stttps:/giite htt.tttt.uu...gm com/ Lensorflow agentss 2018. URL https: //tt:///////.../////// http://tttt...omeeeeeonorooo//// agw.:/onrr.oooo//att.oooooee [Online; accessed 25-Jun.-2019).
11

---

Published as a conference paper at ICLR 2020
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farles and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 100K:::906000000. 2019.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.022988, 2017. URL http://
arxiv. wttpv/orrgabbbb/1113331 02298
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van
Hasselt. Mask deep reinforcement learning with popart. CoRR, abs/1809 809.04474, 2018. URL
http://arxi: org. http//b////10// 04474
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In ICLR, 2018.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent expe- rience replay in distributed reinforcement learning. 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference Learning Representations (ICLR), 2014.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michat Zajac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. arXiv preprint arXiv: 2019.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Dis- tributed RL. arXiv preprint 11X:5:1910.03552, 2019. URL httts:://ithhtt com/
Bbeebokkrseercchttrrriti
Ang Li, Huiyi Hu, Piotr Mirowski, and Mehrdad Farajtabar. Cross-view policy learning for street navigation. arXiv preprint arXiv: /906.05930, 2019.
Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael I Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. arXiv preprint 1rXiv:1712.09381, 2017.
Ashique Mahmood.
Incremental Off- off-policy Reinforcement Learning Algorithms. PhD thesis, Uni- versity of Alberta, 2017.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. CoRR, abs/1812.06162, 2018. URL
066 62
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-19371 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods for deep reinforcement learning. arXiv preprint arxXi:1107.00420 2015.
Shayegan Omidshafici. Daniel Hennes, Dustin Morrill, Rémi Munos, Julien Pérolat, Marc Lanctot, Audrunas Gruslys, Jean-Ba can--Bptiste Lespiau, and Karl Tuyls. Neural replicator dynamics. CoRR, 190660019000.... 2019. http: /tttt:://arxtie wwwp:iv.oooooo/gooooooo abs 6006 001 90
12

---

Published as a conference paper at ICLR 2020
OpenAl. Openai five. httpss//bboo...pppaa..ooo
2018.
Michael Petrov, Szymon Sidor, Susan Zhang, Jakub Pachocki, Przemystaw Dębiak, Filip Wolski, Christy Dennison, Henrique Pondé, Greg Brockman, Jie Tang, David Farhi, Brooke Chan, and Jonathan Raiman. Openai rapid.
cemeeai..com/i scom/blogg hhtp:/ppeaaii-ieeee five/ 2018.
Accessed: 2019-09-14.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: 11593, 1593, 2018.
Rajat Raina, Anand Madhavan, and Andrew Ng. Large-scale deep unsupervised learning using graphies processors. In Proceedings of the 26th annual international conference on machine learning, pp. 873-880. ACM, 2009.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.
Michael Schaarschmidt, Sven Mika, Kai Fricke, and Eiko Yoneki. Rlgraph: Modular computation
graphs for deep reinforcement learning.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In Proc. of ICLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, (BV/17070063453 2017. URL http: http://axxv.../ org abs/
17C7.0634
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Adam Stooke and Picter Abbeel. Accelerated methods for deep reinforcement learning. CoRR, abs/1803.02811 2018. URL http:t////aa www.aawwwwwww..aaaaa..a..aoooeoe absa wttp:////aaaa...ooo//// 02811
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv: 5909,01500, 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. CoRR, abs/1712..065e 2017. URL http:
Richard S Sutton. Learning to predict by the methods of temporal diffcrences. Machine learning, (1):9 44, 1988.
Richard Sutton and Andrew G Barto. Reinforcement learning: an introduction mit press. Cam- bridge, MA, 1998.
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitnick. Elf: extensive, lightweight and flexible research platform for real-time strategy games. Advances in Neural Information Processing Systems (NIPS), 2017.
Dhruva Tirumala, Hyeonwoo Noh. Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized RI. CoRR, abs/1900.007438, 2019, URL
07438
Aäron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, 0bss/887..a 507.03748, 2018. http:://t//axx org/ abs/1807 00188077787888
Hado van Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems 23, pp. 2613-2621, 2010.
13

---

Published as a conference paper at ICLR 2020
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. arXiv preprint arXiv: 1909.04999000000 2019.
Alexander Sasha Vezhnevets, Yuhuai Wu, Remi Leblond, and Joel Leibo. Options responses: Grounding behavioural hierarchies multi-agent rl. arXiv preprint 1rXiv:1906.01470, 2019.
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets,
Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: new challenge for reinforcement learning. arXiv preprint 178.:2 2017.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanetot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd In- ternational Conference on Machine Learning, pp. 1995-2003, 2016.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 708. 038888, 2017a.
Yang You, Igor Gitman, and Boris Ginsburg. Scaling sgd batch size to 32k for imagenet training. arXiv preprint 10X..v/708.03000 6, 2017b.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in minutes. arXiv preprint arXiv; Iee...00400990 2019.
A APPENDIX
A.1
DEEPMIND LAB
A.1.1
LEVEL CACHE
We enable Deep.ind Lab's option for using level cache for both SEED and IMPALA which greatly reduces CPU usage and results in stable actor CPU usage at close to 100% for a single core.
A.1.2
HYPERPARAMETERS
Parameter
Range
Action Repetitions
Discount Factor (7)
(.99.-993 .997,997, .999)
Entropy Coefficient
Log-uniform (1e-5, 5, 1e le-3)
Learning Rate
Log-uniform (le-4. 4, le le-3)
Optimizer
Adam
Adam Epsilon
(1c-1, 1e 5, 73
Unroll Length/n-step
32
Value Function Coefficient
V-trace X
199.995.99 .9,, 1.}
Table 6: Hyperparameter ranges used in the stability experiments.
14

---

Published as a conference paper at ICLR 2020
A.2
GOOGLE RESEARCH FOOTBALL
A.2.1 HYPERPARAMETERS
Parameter
Range
Action Repetitions Discount Factor (?)
(.99,.999 .00,,.99...00. .977, }
Entropy Coefficient
Log-uniform (1c-7, 1e-3)
Learning Rate
Log-uniform (le-5, le-3)
Optimizer
Adam
Unroll Length/n-step
32
Value Function Coefficient
V-trace À
{.9, .95, .99, 1.}
Table Hyperparameter ranges used for experiments with scoring and checkpoint rewards.
A.3 ALE
A.3.1 HYPERPARAMETERS
We use the same hyperparameters as R2D2 Kapturowski et al., 2018), except that we more actors in order to best utilize 8 TPU cores. For completeness, agent hyperparameters are in table and environment processing parameters in table / We use the same neural network architecture as R2D2, namely convolutional layers with filter sizes [32, 64, 64]. kernel sizes [8 x 8, x 4, 3 x 3) and strides [4, 2, 1], ReLU activations and "valid" padding. They feed into linear layer with 512 units, feeding into an LSTM layer with 512 hidden units (that also uses the one-hot encoded previous action and the previous environment reward as input), feeding into dueling heads with 512 hidden units. We use Glorot uniform (Glorot & Bengio 2010) initialization.
Parameter
Value
Number of actors
610
Replay ratio
0.75
Sequence length
120 incl. prefix of 40 burn-in transitions
Replay buffer size
10 part-overlapping sequences
Minimum replay buffer size
5000 part-overlapping sequences
Priority exponent
0.9
Importance sampling exponent
0.6
Discount 7
0.997
Training batch size
64
Inference batch size
64
Optimizer
Adam (Ir 10-4, 4, 10-3) (Kingma & Ba 2014
Target network update interval
2500 updates
Value function rescaling
x + sign(<) +1- + cx, = 10
Gradient clipping
80
n-steps
Epsilon-greedy
Training: i-th actor [0, N) uses e; 1-41+x2 7i Evaluation: = 10
Sequence priority
= 17 max, + (1 n75ō where 17 0.9,
8; are per-step absolute TD enrrr..
Table 8: SEED agent hyperparameters for Atari-57.
15

---

Published as a conference paper at ICLR 2020
Parameter
Value
Observation size
84 84
Resizing method
Bilinear
Random no-ops
uniform in [1,30]. Applied before action repetition.
Frame stacking
4
Action repetition
4
Frame pooling
2
Color mode
grayscale
Terminal on loss of life
False
Max frames per episode
108K (30 minutes)
Reward clipping
No
Action set
Full (18 actions)
Sticky actions
No
Table 9: Atari-57 environment processing parameters.
16

---

Published as a conference paper at ICLR 2020
A.3.2
FULL RESULTS ON ATARI-57
Game
Human
R2D2
SEED TPU v3 cores
Alien
7127.7
229496.9
262197.4
Amidar
1719.5
29321.4
28976.4
Assault
742.0
108197.0
102954.7
Asterix
8503.3
999153.3
983821.0
Asteroids
47388.7
357867.7
296783.0
Atlantis
29028.1
1620764.0
1612438.0
BankHeist
7
24235.9
47080.6
BattleZone
37187.5
751880.0
777200.0
BeamRider
16926.5
188257.4
173505.3
Berzerk
2630.4
53318.7
57530.4
Bowling
160.7
219.5
204.2
Boxing
12.1
98.5
100.0
Breakout
30.5
837.7
854.1
Centipede
12017.0
599140.3
574373.1
ChopperCommand
7387.8
986652.0
994991.0
CrazyClimber
35829.4
366690.7
337756.0
Defender
18688.9
665792.0
555427.2 .2
Demon Attack
1971.0
140002.3
143748.6
DoubleDunk
-16.4
23.7
24.0
Enduro
860.5
2372.7
2369.3
FishingDerby
-38.7
85.8
75.0
Freeway
29.6
32.5
33.0
Frostbite
4334.7
315456.4
101726.8
Gopher
2412.5
124776.3
117650.4
Gravitar
3351.4
15680.7
7813.8
Hero
30826.4
39537.1
37223.1
IceHockey
0.9
79.3
79.2
Jamesbond
302.8
25354.0
25987.0
Kangaroo
3035.0
14130.7
13862.0
Krull
2665.5
218448.1
113224.8
KungFuMaster
22736.3
233413.3
239713.0
MontezumaRevenge
4753.3
2061.3
900,0
MsPacman
6951.6
42281.7
43115,4
Name
8049.0
58182.7
68836.2
Phoenix
7242.6
864020.0
915929.6
Pitfall
6463.7
0.0
-0.1
Pong
14.6
21.0
21.0
PrivatelEye
69571.3
5322.7
198.0
Qbert
13455.0
408850.0
546857.5
Riverraid
17118.0
45632.1
36906.4
RoadRunner
7845.0
599246.7
601220.0
Robotank
11.9
100.4
104.8
Seaquest
42054.7
999996.7
999990.2
Skinn
-4336.9
-30021.7
29973.66
Solaris
12326.7
3787.2
861.6
SpaceInvaders
1668.7
43223.4
62957.8
StarGunner
10250.0
717344.0
448480.0
Surround
6.5
9.9
9.8
Tennis
-8.3
-0.1
23.9
TimePilot
5229.2
445377.3
444527.0 0
Tutankham
167.6
395.3
376.5
UpNDown
11693.2
589226.9
549355.4
Venture
1187.5
1970.7
2005.5
VideoPinball
17667.9
999383.2
979432.
WizardOrWar
4756.5
144362.7
136352.5
YarsRevenge
54576.9
995048.4
973319.0
Zaxxon
9173.3
224910.7
168816.5
Table 10: Final performance of SEED TPU cores, 610 actors seed) compared to R2D2 (averaged over seeds) and Human, using up to 30 random no-op steps at the beginning of each episode. SEED was evaluated by averaging returns over 200 episodes for each game after training on 40c9 environment frames.
17

---

Published as a conference paper at ICLR 2020
Allec
Anada:
800000
Asiosiia
15000
Alazb
1000
20000
30000
1200
5000-
Bowlizg
Buaiig
Hozaltkrr
40000
50000
1900
103
60000
Cord prde
100000
CsaycTiober
Dedesde
15000
Deevetrak
19008
20000
20000
5000
10000
FiblagDeity
Ieooway
Pouthie
Crghet
5000 20-
2000
Gloviorr
Beo
Inlllcceyy
Anaokood
Reagave
Kwlll
2000
2000
5000
200-
Minizannkeenng
Maltuczua
nell
35000
KegTatara
2
15000
10000
40000
-02
20000
-4.5
P.lvadEye
Ruboteak
Prag
Uhei
HeatKunc
10000
20000
63-
300
2000
43-
Sclais
sk.ing
400000
.0000
30000
300
2000
12000
-5
Trmeikke
Thrankaam
Verkue
1550
20006
332
12
16
30
23
Yanitrrgg
SEED - TPU v3 8 cores, 610 actors.
1900
2000
Heacs
Figure 7: Learning curves on 57 Atari 2600 games for SEED (8 TPUv3 cores, 610 actors, evaluated with 1 seed). Each point of each curve averages returns over 200 episodes. No curve smoothing was performed. Curves end at approximately 43 hours of training, corresponding to 40e9 environment frames.
18

---

Published as a conference paper at ICLR 2020
A.4 SEED LOCALLY AND ON CLOUD
SEED is open-sourced together with example of running it both local machine and with scale using Al Platform, part of Google Cloud. We provide a public Docker image with low-level components implemented in C++ already pre-compiled to minimize the time needed to start SEED experiments.
The main pre-requisite to running on Cloud is setting up a Cloud Project. The provided startup script uploads the image and runs training for you. For more details please see github. com/
gcogle-research/Se seed rl
A.5
EXPERIMENTS COST SPLIT
Model
Algorithm
Actors cost
Learner cost
Total cost
Arcade Learning Environment
Default
SEED
$10.8
$8.5
$19.3
DeepMind Lab
Default
IMPALA
$77.0
$13.4
$90
Medium
IMPALA
$103.6
$24.4
$128
Large
IMPALA
$180.5
$55.5
$236
Default
SEED
$20.1
$8.2
$28
Medium
SEED
$18.6
$16.4
S35
Large
SEED
$19.6
$35
$54
Google Research Football
Default
IMPALA
$479
$74
$553
Medium
IMPALA
$565.2
$115.8
$681
Large
IMPALA
$746.1
$153
$899
Default
SEED
$313
$32
$345
Medium
SEED
$312
853
$365
Large
SEED
$295
$74
$369
Table 11: Cost of performing 1 billion frames for both IMPALA and SEED split by component.
A.6 COST COMPARISON ON DEEPMIND LAB USING NVIDIA P100 GPUs
compared the of running SEED using TPU v3 cores and IMPALA on single Nvidia P100 GPU. In table 12] we also compare the cost when both agents run on a single Nvidia P100 GPU on DeepMind Lab. Even though this is a sub-optimal setting for SEED because it performs inference on the accelerator and therefore benefits disproportionately from more efficient accelerators such as TPUs, SEED compares favorably, being 1.76x cheaper than IMPALA frame.
In section 4.4.
Architecture
Actors
CPUs
Envs.
Speed
Cost/1B
Cost ratio
IMPALA
176
176
176
30k
$90
SEED
15
44
176
19k
S51
1.76
Table 12: Cost of performing billion frames for both IMPALA and SEED when running on single Nvidia P100 GPU on DeepMind Lab.
19

---

Published as a conference paper at ICLR 2020
A.7
INFERENCE LATENCY
Model
IMPALA
SEED
DeepMind Lab
Default 17.97ms
10.98ms
Medium 25.86ms
12.70ms
Large
48.79ms
14.99ms
Google Research Football
Default
12.59ms
6,50ms
Medium
19.24ms
5.90ms
Large
34.20ms
11.19ms
Arcade Learning Environment
Default
N/A
7.2ms
Table 13: End-to-end inference latency of IMPALA and SEED for different environments and mod- els.
20


Reviews:
[{"review_id": "S1lmYz5Ycr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new reinforcement learning agent architecture which is significantly faster and way less costly than previously distributed architectures. To this end, the paper proposes a new architecture that utilizes modern accelerators more efficiently.  The paper reads very well and the experimental results indeed demonstrate improvement. Nevertheless, even though working in deep learning for years and have also some experience with Reinforcement learning I am not in the position to provide an expert judgment on the novelty of the work. I do not know if ICLR is the right place of the paper (I would probably suggest a system architectures conference for better assessment of the work).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1g1dnlAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents SEED RL, which is a scalable reinforcement learning agent. The approach restructure the interface / division of functionality between the actors (environments) and the learner as compared to the distributed approach in IMPALA (a state-of-the-art distributed RL framework). Most importantly, the model is only in the learner in SEED while it is distributed in IMPALA. \n\nThe architectural change from to IMPALA to SEED feels reasonable, and the results support the choices in a positive way.\n\nSEED is evaluated using a large number of benchmarks using three environments, and the performance is compared to IMPALA. The results are very good, shows good scalability, and significantly reduced training times.  \n\nThe paper is well written, easy to read, and I enjoyed it. \n\nThe code for SEED is released open source, which enables future research to build upon SEED. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byg1qYe5qS", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a scalable reinforcement learning training architecture which combines a number of modern engineering advances to address the inefficiencies of prior methods. The proposed architecture shows good performance on a wide variety of benchmarks from ALE to DeepMind Lab and Google Research Football. Important to the community, authors also open source their code and provide an estimate which shows that the proposed framework is cheaper to run on cloud platforms.\n\nPros:\n1. This work is solid from the engineering perspective. It effectively addresses the problems with prior architectures and the accompanying source code is clear and well structured. It is also extensively tested on several RL benchmarks.\n\n2. The proposed framework is especially suited for training large models as the model parameters are not transferred between actors and learners.\n\n3. The paper is well written and organized.\n\nCons:\n\n1. The gain of the main algorithmic improvement (SEED architecture) over the baseline (IMPALA architecture) is obscured by the usage of different hardware. TPUv3 has different characteristics than Nvidia P100/V100 GPU chips which also might contribute to the speed up.\n\nQuestions:\n\n1. Is it possible to provide more “apple-to-apple” comparison by running SEED and IMPALA on the same hardware (TPUv3 or Nvidia P100/V100 GPU)? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a framework for scalable Deep-RL on really large-scale architecture, which addresses several problems on multi-machine training of such systems with many actors and learners running.  Large-scale experiments and impovements over IMPALA are presented, leading to new SOTA results. The reviewers are very positive over this work, and I think this is an important contribution to the overall learning / RL community.
Decision: Accept (Talk)