Accelerating Reinforcement Learning Through GPU Atari Emulation
Authors: Steven Dalton
Michael Garland
Iuri Frosio
Venue:  (2020)
Keywords: GPU
reinforcement learning

Abstract: We introduce CuLE (CUDA Learning Environment), a CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep reinforcement algorithms.  CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU parallelization to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communication bandwidth. CuLE generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlighting the differences between CPU and GPU emulators in the context of reinforcement learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+V-trace. CuLE is available at [hidden URL].

Under review as a conference paper at ICLR 2020
ACCELERATING REINFORCEMENT THROUGH GPU ATARI EMULATION
LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce CuLE (CUDA Learning Environment), CUDA port of the Atari Learning Environment (ALE) which is used for the development of deep rein- forcement algorithms. CuLE overcomes many limitations of existing CPU-based emulators and scales naturally to multiple GPUs. It leverages GPU paralleliza- tion to run thousands of games simultaneously and it renders frames directly on the GPU, to avoid the bottleneck arising from the limited CPU-GPU communica- tion bandwidth. CuL.E generates up to 155M frames per hour on a single GPU, a finding previously achieved only through a cluster of CPUs. Beyond highlight- ing the differences between CPU and GPU emulators in the context of reinforce- ment learning, we show how to leverage the high throughput of CuLE by effective batching of the training data, and show accelerated convergence for A2C+ V-trace. CuLE is available at [URL revealed upon acceptance].
INTRODUCTION
Initially triggered by the success of DQN Mnih iet al. (2015). research in Deep Reinforcement Learning (DRL) has grown popularity in the last years Lillicrap et al.] (2015); Mnih et al.
actions (a)
states (s)
sur
states (6)
DRL
rewards [r]
(2016) 2015), leading intelligent agents that solve non-trivial tasks in complex environments. But DRI. also soon proved be a challenging computational problem, especially if one wants to achieve peak performance on modern archi- tectures.
states (s)
PCI-e
traim butter
biri
CPU
GPU
inference path
training path (off-policy only)
training path
Fig. iis schematic representation modern DRL algorithm implementation: CPU environ- ments exccute a set actions {a--1} at time t - 1, and produce observable states {st) and rewards {r.) These data are migrated to a Deep Neural Network (DNN) on the GPU eventu- ally select the next action set, {ar}. which is copied back to the CPU. This sequence of operations defines the inference path, whose main aim is generate training data. A training buffer on the GPU stores the states generated on the inference path; this is periodically used to update the DNN's s weights according the training rule of the DRL algorithm (training path). computationally efficient DRL. system should balance the data generation and training processes, while minimizing the communication overhead along the inference path and consuming, along the training path as many data per second as possible Babaeizadeh et al. (2016) 2017). The solution to this problem is however non non-trivial and many DRL implementations do not leverage the full computational potential of modern systems Stooke & Abbeel (0018b).
Figure 1: typical DRL system, environments run on CPUs and DNNs on GPUs. The limited CPU-GPU communication bandwidth and small set of environments prevent full GPU usage.
We focus our attention on the inference path and move from the traditional CPU implementation of the Atari Learning Environment (ALE), set of Atari 2600 games that emerged as excellent DRL benchmark Bellemare /Bllemrrrreee et al. (2013); |Mchado et al. (2017). We show that significant performance bottlenecks primarily stem from CPU environment emulation: the CPU cannot large set of

---

Under review as a conference paper at ICLR 2020
environments simultaneously, the CPU-GPU communication bandwidth is limited, and the GPU is consequently underutilized. To both investigate and mitigate these limitations we introduce CuLE (CUDA Learning Environment), DRL library containing CUDA enabled Atari 2600 emulator, that renders frames directly in the GPU memory, avoids off-chip communication and achieves high GPU utilization by processing thousands of environments in somallel-something only through large and costly distributed systems. Compared to the traditional CPU-based approach, GPU emulation improves the utilization of the computational resources: CuLE on a single GPU gen- erates more Frames Per Second" (FPS) on the inference path (between 39K and 125K, depending on the game, see Table I compared its CPU counterpart (between 12.5K and 19.8 19.8K). Beyond offer- ing CuLE (IURL revealed upon acceptance]) as tool for research in the DRL field, our contribution can be summarized as follow:
so far achievable
(1) We identify common computational bottlenecks in several DRL. implementations that prevent effective utilization of high throughput compute units and effective scaling to distributed systems.
(2) We introduce an effective batching strategy for large environment sets, that allows leveraging the high throughput generated by CuLE to quickly reach convergence with V-trace [speholt et al. 20188, and show effective scaling on multiple GPUs. This leads to the consumption of 26-68K FPS along the training path on a single GPU, and up to 187K FPS using four GPUs, comparable (Table I to those achieved by large clusters Stooke & Abbeel 2018a); Especolt et al. (2018).
(3) We analyze advantages and limitations of GPU emulation with CuLE in DRL, including the effect of thread divergence and of the lower (compared to CPU) number of instructions per second per thread, and hope that our insights may be of value for the development of efficient DRL systems.
Table I: Average training times, raw frames to reach convergence, FPS, and computational resources of existing accelerated DRL schemes, compared to CuLF. Data from |Horgan et al. (2018); FPS are taken from the corresponding papers, if available, and measured on the entire Atari suite for CuLE.
Algorithm
Time
Frames
FPS
Resources
Notes
Apc-X Horgan et al. 2018
5 days
22,800M
50K
376 cores, I GPU GPU 1 GPU 16 cores 16 cores, GPU
Rainhow (eeter
10 days
200M
Distributioal I mellemeee et
2017
10 days
200M
A3C Mnih al. 2
4 days
2K SK
GA3C (aabaezzadeh etal. 2016:
2017
1 day
Priorinzed Ducling Wang 2015
9.5 days
200M
1 GPU 1 GPU
Mnihh
Mnih ot al. 2015
9.5 days
200M
Gorila DON Nair
2015
4 days
100 cores 16 cores
Unreal Inderherg et al,
2016
250M
Stooke (A2C/
Stooke Abbeel 2018b)
hours
200M
40 CPUs, GPUs (DGX-1) 100-200 cores, 1 GPU
IMPALA (A2C + V- V-Taace [Epperot et ai 2018
minshours
200M
250K
CuLE (emulation only)
NIA
N/A
41K-155K
System (1 GPU)
ALES
CuLE (inference only. A2C, single batch)
NIA
N/A
39K-125K
System (11 GPU)
4096 ALEs
CuLE (rraining. A2C+ + V-trace, multiple butches)
hour
200M
26K-68K
System (1 GPU)
ALES
CuLE (training, A2C + V-trace, multiple batches)*
mins
200M
142-187K
System III (4 GPUs)
x ALES
IPPS measured on Asterix, Assault, MsPacman, and Pong
RELATED WORK
The wall clock convergence time of DRL al- gorithm is determined by two main factors: its sample efficiency, and the computational effi- ciency of its implementation. Here we analyze the sample and computational efficiency of dif- ferent DRL algorithms, in connection with their implementation.
Table 2: Systems used for experiments.
System
Intel CPU
NVIDIA
1
12 core Core 17. 59308 6 3.550Gtt
Time V
11
f-ooe Cone 17-9086K @ SGGH
Toda Vi00
II
core Cure ES 2698 v4 22220Cl 2
Tesla V100 B, NVLink
Raw frames are reported here and in the rest of the paper, unless otherwise specified. These are the frames that actually emulated, but only 25% of them are rendered and used for training. Training frames are obtained dividing the raw frames by 4-sce also Espeholt et al. (2018).
2

---

Under review as a conference paper at ICLR 2020
We first divide DRL algorithms into policy gradient and Q-value methods, as in Stooke & Abbeel (2018b). Q-learning optimizes the error on the estimated action values as a proxy for policy opti- mization, whereas policy gradient methods directly learn the relation between state, 8t, and the optimal action, az; since at each update they follow, by definition, the gradient with respect to the policy itself, they improve the policy more efficiently. Policy methods are also considered more gen- eral, e.g. they can handle continuous actions easily. Also the on- or off-policy nature of an algorithm profoundly affects both its sample and computational efficiency. Off-policy methods allow re-using experiences multiple times, which directly improves the sample efficiency; additionally, old data stored in the GPU memory (replay buffer in Fig. I can be used to continuously update the DNN on the GPU, leading to high GPU utilization without saturating the inference puth. The replay buffer has positive effect on the stability of learning as well Mnih et al. (2015). On-policy algorithms saturate the inference path more easily, as frames have to be generated on-the-fly using the current policy and moved from the CPU emulators to the GPU for processing with the DNN, On-policy up- dates are generally effective but they are also more prone to fall into local minima because of noise, especially the number of environment is small - this is the reason why on on-policy algorithms largely benefit (in term of stability) from significant increase of the number of environments.
Among the policy gradient methods, A3C Mnih et al. (2016) is an asynchronous, on on-policy, actor- critic algorithm where 16 CPU agents use copies of the same DNN to interact with CPU environ- ments; N-step bootstrapping is used to reduce the variance of the critic, V (sx; 0): the agents send updates to the DNN after every N actions, and the new set of global @ weights are broadcast to
all agents after each update. A3C can solve an Atari game in approximately 4 days: its hybrid CPU- GPU implementation (GA3C Babaeizadeh et al. (2016 2017)) collects CPU data in system of queues to send them to the GPU in large batches, and achieves convergence in 1 day. Recent (and faster) policy gradient implementations, like PAAC Clementee eet al. (2017). PPO |Schulman et al. (2017). or A2C OpenAl (2017), store training data directly in GPU training buffer, as depicted in Fig. which is well representative of the flow of computation of modern DRL algorithms.
stores single DNN on the GPU,
Policy gradient algorithms are often on-policy: their efficient update strategy is counterbalanced by the bottlenecks in the inference path and competition for the use of the GPU along the inference and training path at the same time. Acceleration by scaling to a distributed system is possible but inefficient in this case: in IMPALA |Espeholt et al. (2018) a cluster with hundreds of CPU cores is needed to accelerate A2C, while training is desynchronized to hide latency. As a consequence, the algorithm becomes off-policy, and V-trace was introduced to deal with off-policy data (see details in the Appendix). Acceleration on DGX-1 has also been demonstrated for A2C and PPO, using large batch sizes to increase the GPU occupancy, and asynchronous distributed models that hide latency, but require periodic updates to remain synchronized Stooke & Abbeel 2018b) and overall achieves sublinear scaling with the number of GPUs. practice, the efficiency generally scales sub-linearly on distributed systems, and though Atari games can be solved in few hours or even minutes, the cost of such systems makes them prohibitive for many researchers.
Q-value methods, like DON Mnih et al. (2015) and its more recent, improved versions (Dou- bleDON van Hasselt et al. (2155), Dueling Networks Wang et al. (2015), Prioritized Replay Schaul et al. (2015). n-step learning Peng & Williams (199)). NoisyNets Fortunato et al.] (2017), Rain- bow Hessel et al. (017)) are generally ooflpo and can be accelerated more casily, as the infer- ence and training path can be decoupled, with the GPU continuously fed with training data from the replay buffer in Fig Mapping the two paths on different devices is natural and effective, and indeed DQN methods have been significantly accelerated
distributed systems (see Nair et al. (2015) and Jaderberg et al. (2016) in T.. The replay buffer increases the sample efficiency, but the overall convergence time has pay the cost of using a Q-value method. Furthermore, imple- mentation on large scale systems still incurs communication overhead and thus generally achieves sub-linear scaling on large distributed systems.
The best compromise between sample efficiency, computational efficiency and effectiveness of the DNN update seems to be achieved by using an on-policy while correcting for stale data, gradient policy methods like A2C+V-trace, which is therefore the algorithm analyzed in detail here, but still suffers from computational bottlenecks in its CPU distributed implementation Espeholt et al. (2018).
3

---

Under review as a conference paper at ICLR 2020
Other Data Generation Engines Frameworks generate training experiences already exist, like the well-known OpenAl Gym, built on top of the C++ Stella emulator Brockman et al. (2016) for Atari games. Other frameworks that target specific problems. such as navigation Mirowski et al. (2016), aim at creating large benchmark for DRI. Tassa CPU with optimized C++ multi-threader implementations, while also providing large, GPU-friendly batches for inference and training Tian et al. (2017). To the best of our knowledge, however, none of these attempts directly address the problem of optimizing the inference path, i.e. trying to increase the number of environments while avoiding the CPU and bandwidth bottlenecks, by providing direct implementation of the environments on the GPU.
| et al. (2018), or accelerate simulation on the
CUDA LEARNING ENVIRONMENT (CULE)
Despite the number and variety of games developed for the Atari 2600, it is rela- tively simple to emulate the functioning of its hardware much faster than real-time. The Atari console has 1.19Mhz CPU (to execute the game instructions) and a 128 bytes RAM (containing the game state); the code of each game is stored in a car- tridge ROM (typically 2 2-4kB). The in- struction set of each game contains both game and rendering instructions; the first ones are executed by the CPU, while the second ones are sent to the Television In- terface Adaptor (TIA), a secondary pro- cessor embedded in the Atari hardware that executes the rendering instructions update a limited set of registers and render the 160 210 game screen using a 128- colour palette.
BPU cxsslant
GPU global memery
mamory
Rundem states
Ateii RAMs (128 bytes much)
Atari RDN
REST stee
I2E stite
VEST sate
CRU Instructions
Atri RAMs (28 bybes each)
state
instructions
came stree
game strte &TEE
CPU instructions
DAU ssae
D4U ratol
LRU stuta AlF)
TA instrustions
TIA instruction buffer
Inst:
frunnebuffer
Atori ERV Aorool
cslaade,
Af Awreal
2) load RDM instruction
3] lond Atari RXM game state / Atari DPU state
instruction bultar
Aari CPU iestruations
2) eascate TIA instructions
stare game in RAM,
state
3) store IBD:210 frome rutput framebuffer
B) store IM instructio in II intrastion balker
Figure 2: Our CUDA-based Atari emulator uses an Atari CPU kernel to emulate the functioning of the Atari CPU and advance the game state, and second TIA kernel to emulate the TIA and render frames di- rectly in GPU memory. For episode resetting we gen- erate and store a cache of random initial states. Mas- sive parallelization on GPU threads allows the parallel emulation of thousands of Atari games.
In CuLE, we emulate the functioning of many Atari consoles in parallel using the CUDA programming model, where a se- quential host program executes parallel programs, known as kernels, on GPU. trivial mapping of the Atari emulator to CUDA, a single thread emulates both the Atari CPU and TIA to execute the ROM code, update the Atari CPU and TIA registers as well as the game state in the 128 bytes RAM, and eventually render the pixels in the output frame. However, the contrasting nature of the game code execution and renderings tasks, the first dominated by reading from the RAM/ROM and writ- ing tens of bytes to RAM, while the second writes hundreds of pixels the framebuffer, poses a serious issue in terms of performance, such as thread divergence and an imbalanced number of registers required by the first and second tasks. To mitigate these issues, CuLE uses two CUDA kernels: the first one first loads data from the GPU global memory, where we store the state of each emulated Atari processor, and the 128 bytes RAM data containing the current state of the game; it also reads ROM instructions from the constant GPU memory, executes them to update the Atari CPU and game states, and stores the updated Atari CPU and game states back into GPU global memory. It is important to notice that this first kernel does not execute the TIA instructions read from the ROM, but copies them into the TIA instruction buffer in GPU global memory, which we implemented to decouple the execution of the Atari CPU and TIA instructions in CuLE. The second CuLE kernel emulates the functioning of the TIA processor: it first reads the instructions stored in the TIA instruction buffer, execute them to update the TIA registers, and renders the 160 210 output framebuffer in global GPU memory. Despite this implementation requires going through the TIA instruction twice, it has several advantages over the single-kernel trivial implementation. First
4

---

Under review as a conference paper at ICLR 2020
of all, the requirements in terms of registers per thread and the chance of having divergent code different for the Atari CPU and TIA kemelss, and the use of different kernels achieves a better GPU usage. A second advantage that we exploit is that not all frames are rendered in ALE: the input of the RI. algorithm the pixelwise maximum between the last two frames in a sequence of four, so avoid calling the TIA kernel when rendering of the screen is not needed. A last advan- tage, not exploited in our implementation yet, is that the TIA kernel may be scheduled one the GPU with more than one thread per game, as rendering of diverse rows on the screen is indeed a parallel operation we leave this optimization for future developments of CuLE.
To better fit our execution model, our game reset strategy also different from the one in the existing CPU emulators, where 64 startup frames are executed at the end of each episode. Furthermore, wrapper interfaces for RI., such as ALE, randomly execute an additional number of frames (up to 30) to introduce randomness into the initial state. This results into up to 94 frames to reset a game, which may cause massive divergence between thousands of emulators executing SIMD fashion on GPU. To address this issue, we generate and store cache of random initial states (30 by default) when a set of environments is initialized in CuLE. At the end of an episode, each emulator randomly selects one of the cached states seed and copies it into the terminal emulator state.
Some of the choices made for the implementation of CuLE are informed by ease of debugging, like associating one state update kernel to one environment, need for flexibility, like emulating the Atari console instead directly writing CUDA code for each Atari game. A 1-to-1 mapping between threads and emulators is not the most computationally efficient way a GPU, but it makes the implementation relatively straightforward and has the additional advantage that the same emulator code can be executed on the CPU for debugging and benchmarking (in the following, we will refer to this implementation as CuLEcru). Despite of this, the computational advantage provided by CuLE over traditional CPU emulation remains significant, shows in the next Section.
Atari games on
EXPERIMENTS
Atari emulation We measure the FPS under different conditions: we get an upper bound on the maximum achievable FPS in the emulation only case, when we emulate the environments and use random policy to select actions. In the inference only case, we measure the FPS along the inference path: a policy DNN selects the actions, CPU-GPU data transfer occur for CPU emulators, while both emulation and DNN inference achievable by off off-policy algorithms, when data generation and consumption are decoupled and run on different devices. In the training case, the entire DRL system is at work: emulation, inference, and training may all run on the same GPU. This is representative of the case of on-policy algorithms, but the FPS are also affected by the computational cost of the specific DRL update algorithm; in experiments we use a vanilla A2C OpenAl (2017), with N-step bootstrapping, and N = 5 as the baseline (for details of A2C and off-policy correction with V-trace, see the Appendix).
the GPU when CuLE is used. This is the maximum throughput
FPS
FPS per environment
2001
200
FPS envirorment
2n*
FPS enviorment
20f
CuL-Rett CuLRR OpenAE
KFPS
KFPS
100
10=
10
10P
10a
10
10F
10a
10
10
10
Environments
Environments
Environments
Environments
(a)
emulation only (b) inference only
(C)
emulation only
(d)
inference only
Figure 3: FPS and FPS/ / environment on System I in Table 2 for OpenAl Gym OpenAl 2017).
Cul.Ecru, and CuLF, as function of the number of environments, under different load conditions: emulation only, and inference only. The boxplots indicate the minimum, 25th 50th 75th percentiles and maximum FPS, for the entire set of 57 Atari games.
5

---

Under review as a conference paper at ICLR 2020
Table 3: Training FPS, DNN's Update Per Second (UPS), time reach given score, and corre-
sponding number of training frames for four Atari games, V-C+e-t and different configurations of the emulation engines, measured on System I Table 2 (System III for the multi-GPU case). The best metric in each row is in bold.
Engine
OpenAl Gym
CuLE,, GPU
CaLEE, GPUs
Enws
120
120
120
1200
1200
1200
1200
1200x
Batches
20
20
5
20x4
N-steps
20
20
5
20
20
Training KFPS
4.2
3.4
3.0
4.9
10.6
11.5
11.0
42.7
Assault
UPS
7.0
28.3
24.7
4.1
1.8
9.6
9.1
8.9
Time [mins]
20.2
42.6
44.2
18.8
9.4
9.9
7.9
Training Mframes (for average 800)
7.5
12.0
6.5
18.0
Training KFPS
12.5
UPS
7.1
27.9
24.8
4.1
2.0
10.4
10.0
9.7
Asteti
Time [mins]
8.1
35.2
14.4
27.1
14.0
3.4
2.5
Training Mframes (for average 1,000)
2.5
8.0
10.5
2.5
7.0
Training
9.6
MoPacman
UPS
6.7
27.1
23.7
4.0
1,5
8.0
7.7
7.4
Time [mins]
16.6
20.5
14.7
12.4
6.9
2.4
Training Mirames (for average 1,500)
4.0
4.0
2.5
3.5
4.0
6.5
3.0
Training KFPS
3.0
4.8
10.5
11.2
41.7K
UPS
7.2
28.1
24.9
4.0
1.8
9.3
8.9
8.7
Pong
Time [mins]
5.9
Training Mirames (for average 18)
5.5
2,5
1.5
2.5
4.0
2.0
6.0
show the FPS generated by OpenAl Gym, CuLEcpu, and CuLE, on the entire set of Atari games, as a function of the number of environments. In the emulation only case, CPU emula- tion is more efficient for number of environments up to 128, when the GPU computational power not leveraged because of the low occupancy. For larger number of environments, CuLE sig- nificantly overcomes OpenAl Gym, for which FPS are mostly stable for 64 environments or more, indicating that the CPU is saturated: the ratio between the median FPS generated by CuLE with 4096 environment (64K) and the peak FPS for OpenAl Gym (18K) is 3.56 In the inference only case there are two additional overheads: CPU-GPU communication (to transfer observations). and DNN inference the GPU. Consequently, CPU emulators achieve lower FPS in inference only when compared to emulation only; the effects of the overheads is more evident for a small number of environments, while the FPS slightly increase with the number environments without reaching the emulation only FPS. CuLE's FPS are also lower for inference only, because of the latency intro- duced by DNN inference, but the FPS grow with the number of environments, suggesting that the computational capability of the GPU is still far from being saturated.
Figs. 3(a) 3(b)
17
M0
Rasetiig amas
Rassdt mg sma
Rasettin
Reseting
0
190
20:1
300
100
1100
200
100
102
2003
2241
410
(a) Asterix, GPU
(b) Pong, GPU
(c) Ms Pacman, GPU
(d) Assault, GPU
Figure 4: FPS as function of the environment step, measured on System I in Table 2 for emulation only four Atari games, 512 environments, for CuLE; each panel also shows the number of reset- ting environments. FPS is higher at the beginning, when all environments are in similar states and thread divergence within warps minimized; after some steps, correlation is lost, FPS decreases and stabilizes. Minor oscillations in FPS are possibly associated to more or less computational demanding phases in the emulation of the environments (e-g., when a goal is scored in Pong).
Factors affecting the FPS Figs. 3(a) 3(b) shows that the throughput varies dramatically across
games: 4096 Cul_Ecro environments at 27K FPS Riverraid, but only 14K FPS for Boxing:

---

Under review as a conference paper at ICLR 2020
1.93x difference, explained by the different complexity of the ROM code of each game. The ratio between the maximum and minimum FPS is amplified in the case of GPU emulation: Riverraid runs in emalation only at 155K FPS when emulated by CuLE and 4096 environments, while UpNDown runs at 41K FPS -a 3.78x ratio.
To better highlight the impact of thread divergence on throughput, measure the FPS for CuLE, emulation only, 512 environments, and four games (Fig. 4 All the environments share the same initial state, but random action selection leads them diverge after some steps. Each environment resets at the end of an episode. The FPS is maximum at the very beginning, when all the envi- ronments are in similar states and the chance to execute the same instruction in all the threads is high. When they move towards different states, code divergence negatively impacts the FPS, until it reaches an asymptotic value. This effect is present all games and particularly evident for MsPac- man in Fig. 4 it is not present in CPU emulation (see Appendix). Although divergence can reduce FPS by 30% in the worst case, this has to be compared with case of complete divergence within each thread and for each instruction, which would yield 1/32 = 3% of the peak performances. Minor oscillations the FPS are also visible especially for games with a repetitive pattern (e.g. Pong), where different environments can be more or less correlated with typical oscillation frequency.
Performances during training Fig. 5 compares the FPS generated by different emulation engines on a specific game (Assault for different load conditions, including the training case, and num- ber of environments. As expected, when the entire training path at work, the FPS decreases even further. However, for CPU emulators, the differ- ence between FPS in the inference only and training cases decreases when the number of environments increases, as the system is bounded by the CPU com- putational capability and CPU-GPU communication bandwidth. In the case of the CPU scaling to mul- tiple GPUs would be ineffective for on-policy algo- rithms, such GA3C Babaeizade et al. (2016 2017).
00
OalE takeaae
42
CalE imatsg
OalE adoaze
HFRS
Opmal
sfnrem
AlpesAi mendes
500
1000
1300
Envirommntts
Figure 5: FPS generated by different emula- tion engines on System I in Table 2 for As- sault, as function of the number of environ- ments, and different load conditions for A2C with N-step bootstrapping.
or sub-optimal. in the case of distributed systems Es- peholt et al. (2018): Stooke & Abbeel (2018b). On
the other hand, the difference between inference only and training FPS increases with the number of envi- ronments for CuLE, because of the additional training overhead on the GPU, The potential speed-up provided by CuLE for vanilla A2C and Assault in Fig. 5 2.53x for 1,024 environments, but the system is bounded by the GPU computational power; as a consequence, better batching strategies that reduce the training computational overhead as well as scaling to multiple GPUs effective to further increase the speed-up ratio, as demonstrated later in this Section.
5).
When data generation and training be decoupled, like for off-policy algorithms, training can be easily moved to a different GPU and the inference path can be used at maximum speed. The potential speed-up provided by CuLE for off-policy algorithms is then given by the ratio between the inference only median FPS for CuLE (56K) and CuLEcPu (18K), which is 3.11 x for 4,096 environments. Furthermore, since the FPS remains flat for CPU emulation, the advantage of CuLE amplifies (for both on- and off- off-policy methods) with the number of environments.
Frames per second per environment Fig. 3 3(d) show the FPS environment for different emulation engines System I, as function of the number of environments. For 128 environments or fewer, CPU emulators generate frames at higher rate (compared to CuLE), because CPUs are optimized for low latency, and execute high number of instructions per second per thread. How- ever, the FPS / environment decrease with the number of environments, that have to share the same CPU cores. Instead, the GPU architecture maximizes the throughput and has lower number of instructions per second per thread. As consequence, the FPS / environment is smaller (compared CPU emulation) for a small number of environments, but they are almost constant up 512 en- vironments, and starts decreasing only after this point. In practice, CuLE environments provide an
Otter games for which we observe similar behavior are reported in the Appendix, for sake of space.

---

Under review as a conference paper at ICLR 2020
efficient means of training with diverse set of data and collect large statistics about the rewards experienced by numerous agents, and consequently lowering the variance of the value estimate. On the other hand, samples are collected less efficiently in the temporal domain, which may worsen the bias on the estimate of the value function by preventing the use of large N in N-step bootstrapping. The last paragraph of this Section shows how to leverage the high throughput generated by Cul.E, considering these peculiarities.
Memory limitations Emulating a massively large number of environments can be problematic considering the relatively small amount of GPU DRAM. Our PyToorc Paszke et al. (2017) imple- mentation of A2C requires each environment to store 4 84x84 frames, plus some additional variables for the emulator state. For 16K environments this translates into IGB of memory, but the primary issue is the combined memory pressure to store the DNN with 4M parameters and the meta-data dur- ing training, including the past states: training with 16K environments easily exhausts the DRAM on a single GPU (while training on multiple GPUs increases the amount of available RAM). Since we did not implement any data compression scheme as Horgan et al.] (2018), we constrain our training configuration to fewer than 5K environments, but peak performance in terms of FPS would be achieved for a higher number of environments this is left as a possible future improvement.
A2C We analyze in detail the case of A2C with CuLE on a single GPU. As baseline, we consider vanilla A2C, using 20 OpenAl Gym CPU environments that send training data to the GPU to update the DNN (Fig. 6(a) every N 5 steps. This configuration takes, on average, 21.2 minutes (and 5.5M training frames) to reach score of 18 for Pong and 16.6 minutes (4.0M training frames) for a score of 1,500 on Ms- Ms-Pacman (Fig. 7 red line; first column of Table 3). CuLE with 1 1,200 environments generates approximately 2.5x more FPS compared to OpenAl Gym, but this alone is not sufficient to improve the convergence speed (blue line, Fig. 7p. CuLE generates larger batches but, because FPS / environment is lower when compared to CPU emulation, fewer Updates Per Second (UPS) are performed for training the DNN (Table 3 which is detrimental for learning.
A2C+V-trace and batching strategy To bet- ter leverage CuLE, and similar in spirit to the approach in IMPALA Espeholt et al. (2018), we employ a different batching strategy, illus-
Plap Forraaat ETC)
bec-1
Intreeniee
Iesh
trated in Fig. 6(b) environment steps occur in parallel on the GPU, but training data are read in batches to update the DNN every Steps Per Update (SPU) steps. This batching strategy sig- nificantly increases the DNN's UPS at the cost of slight decrease in FPS (second columns of OpenAl Gym and CuLE in Table due to the fact that the GPU has to dedicate more time to training. Furthermore, as only the most recent data in batch are generated with the current policy, we use V-trace Espeholt et al. (2018) for off-policy correction. The net result is increase of the overall training time when 120
(a) Single batch strategy
(b) Multi batch strategy
Figure 6: Different batching strategies are defined by the number of batches, N-Steps and Steps Per Update (SPU) parameters. The on-policy single batch case (panel a) a special case of the more general, off-policy multi batch approach (panel b). The batching strategy affects both the computa- tional and convergence aspects of the DRL algo- rithm, as shown in Fig. 7 and in Table
OpenAl Gym CPU environments are used, as this configuration pays for the increased training and communication overhead, while the smaller batch size increases the variance in the estimate of the value function and leads to noisy DNN updates (second column in Table 3 orange lines in Fig. 7 Since CuLE does not suffer from the same computational bottlenecks, and at the same time beneñts from the variance reduction associated with the large number (1,200) of environments, using the same batching strategy with CuLE reduces the time to reach a score of 18 for Pong and 1,500 for Pacman respectively to 5.9 and 6.9 minutes. The number of frames required to reach the same score is sometimes higher for CuLE (Table 3), which can lead to less sample efficient implementation when compared to the baseline, but the higher FPS largely compensates for this. Extending the batch size in the temporal dimension (N-steps bootstrapping, N = 20) increases the GPU compu- tational load and reduces both the FPS and UPS, but it also reduces the bias in the estimate of the value function, making each DNN update more effective, and leads to an overall decrease of the
8

---

Under review as a conference paper at ICLR 2020
wall clock training time, the fastest convergence being achieved by CuLE with 1,200 environments. Using OpenAl Gym with the same configuration results in a longer training time, because of the lower FPS generated by CPU emulation.
Scaling to multiple GPUs The black line in Fig. 7 rep- resents the case of + Trace with CuLE and GPUs, each emulating 1 1,200 environments. The FPS grow almost linearly with the num- ber of GPUs under this train- ing load: CuLE rumning on GPUs allows generating and consuming a comparable number of FPS with respect to IMPALA on large CPU cluster (Table [.). This leads to dramatic reduction of the convergence time, as docu- mented in Table 3 with Pong almost solved in less than minutes.
1160
5339
00.0
1330
1840
3330
410
3300
20
40
83
50
100
20
600
80
100
Time [mins]
Time (mire))
(a) Assault, 20M training frames
(b) Asterix, 20M training frames
axc
300
26 Mestago
Cie Vioraee 3 597
new borege bookinn
C.B. SUSFU
Time
Time Jmnns]
(c) Ms-Pacman, 20M training frames
(d) Pong, 8M training frames
Figure 7: Average testing score and standard deviation on four Atari games as a function of the training time, for A2C+ V-trace, System III in Table 2 and different batching strategies (see also Table 3). Training frames are double for the multi-GPU case (black line). Training performed on CuLE or OpenAl Gym; testing performed on OpenAl Gym environments (see the last paragraph of Section 4
Beyond A2C+V-trace, we re- port in Table the time to generate 200M frames for PPO and Rainbow DQN by running CuLE on 1 to 8 GPUs on single system. We use the PyTorch multiprocessing facilities to launch 1 process for each GPU and update gradients in a distributed manner using the NVIDIA NCCL multi-GPU communication backend. This Table shows efficient scaling of the training FPS with the number of GPUs. As in the case of A2C+V- trace, an efficient batching strategy is then required to fully leverage the high throughput generated by CuLE and can significantly improve these figures.
Generalization for different systems Table reports the FPS for the implemen- tations of vanilla DQN, A2C, and PPO, on System and II in Table - The speed-up in terms of FPS provided by Cul.E is con- sistent across different systems, different algorithms, and larger in percentage for a large number of environments. Different DRL algorithms achieve different FPS de- pending on the complexity and frequency of the training step on the GPU.
Table 4: Hours to complete 50M training frames
(200M raw frames) on System III in Table 2 by GPU count, for two different DRL algorithms, 2,048 envi- ronments per GPU.
Algo
I GPU
2 GPUs
4 GPUs
8 GPUs
PPO
9,38
4.45
2.12
1.1
Rainbow DQN
10.5
5.1
3.2
2.1
The same Table also reports the minimum and maximum GPU utilization measured while running each DRL algorithm. In its vanilla implementation DQN is characterized by long GPU idle times occurring during CPU emulation, leading to low GPU utilization. The GPU utilization increases when emulation is moved to the GPU, while GPU peak utilization is reached during the DNN train- ing step. GPU underutilization can be observed also for CPU emulation; peak utilization is higher for PPO, because of its larger computational complexity. Nearly full GPU utilization is achieved only by CuLE for a large number of environments.
algorithms (A2C, PPO) and
Correctness of the implementation To guarantee that CuLE is compliant with OpenAI Gym, we seed OpenAl Gym and CuLE environments with the same game state and successfully verify that the sequence of game states and frames returned by each environment are equivalent. A more
9

---

Under review as a conference paper at ICLR 2020
Table 5: Average FPS and min/max GPU utilization during training for Pong with different algo- rithms and using different emulation engines on different systems (see Table CuLE consistently leads to higher FPS and GPU utilization.
Algoritha
Emuulaiio engine
FPS [GPU utilization [S]
System [256 envs]
System [1024 envs]
System II (256 enss]
System [1024 envs]
OpenAl
[15-42%%
[0-69%]
10.8K [26-32%
21.2K (28-75%)
DQN
CaL.Eeeu
7.2K [16.43%]
8.6K [0-72%%
6.8K [17-25%)
20.8K [8-21%)
CuLE
14.4K [16-99%]
25.6K [17-99%)
11:2K [48-62%)
33.2K [57-77%)
OpenAl
12.8K [2-15%%
15.2K [0-43%]
24.4K [5-23%]
30.4K [3-45%]
A2C
CaLFçeo
10.4K [2-159)]
14.2K [0-43%]
12.8K [1-18%]
25.6K [3-47%]
CuLE
19.6K [97-98%)
51K [98-100%)]
23.2K (97-98%)
48.0K (98-99%)
OpenAT
12K [3-99%]
10.6K [0-96%]
16.0K [4-33%]
19.2K [4-62%]
PPO
CaLEgro
10K [2-99%]
10.2K [0-965)
9.2K [2-28%]
18.4K [3-61%)
CuL.F
14K [95-99%)
36K 100%]
14.4K [43-98%)
28.0K [45-99%)]
subtle question to answer is the CuLE reset procedure, based on a set of random initial states, can reduce the variability in the environments and consequently be detrimental for the learning and generalization capability of the trained agents. Our code ((URL revealed upon acceptance]) allows the user to test the training agent on CuLE, CuLEcpU. and OpenAI Gym environments: in our tests the results are comparable for the three backends. Results reported in Fig. 7 and the Appendix further suggest that CuLE trained agents do not suffer any performance loss when tested on OpenAl Gym environments.
5 DISCUSSION AND CONCLUSION
The common allocation of the tasks in a DRL system dictates that the environment should run on CPUs, whereas GPUs should be dedicated to DNN operations. With most of the existing frame- works Brockman et al. Mirowski et al. (016);; Tassa et al. (2018): Tian et al. (2017) follow- ing this paradigm, the limited CPU-GPU communication bandwidth and CPU capability to emulate a large number of environments represent two limiting factors to effectively accelerate DRL algo- rithms, even when mapped to expensive distributed systems. By rendering frames directly on the GPU, Cul overcomes these limitations and generate as many FPS as those generated by large, expensive CPU systems. CuLE promises to be an effective tool to develop and test DRL. algorithms by significantly reducing the experiment turnaround time.
As already shown by others in the case of DRL on distributed system, our experiments show that proper batching coupled with a slight off-policy gradient policy algorithm can significantly acceler- ate the wall clock convergence time; CuLE has the additional advantage of allowing effective scaling of this implementation to system with multiple GPUs CuLE effectively allows increasing the num- ber of parallel enviromments but, because of the low number of instructions per second per thread on the GPU, training data can be narrow in the time direction. This can be problematic for problems with sparse temporal rewards, but rather than considering this as a pure limitation of CuLEE, we be- lieve that this peculiarity opens the door new interesting research questions, like active sampling of important states Hessel et al. (2017); Wang et al. (2015) that can then be effectively analyzed on a large number of parallel environments with CuLE. CuLE also hits a new obstacle, which is the limited amount of DRAM available on the GPU; studying new compression schemes, like the one proposed in Hessel et al. (2017). as well as training methods with smaller memory footprints may help extend the utility of CuLEE to even larger environment counts, and design better GPU-based simulator for RL. in the future. Since these are only two of the possible research directions for which CuLE is effective investigation instrument, CuLE comes with python interface that allows easy experimentation and is freely available to any researcher at [URL revealed upon acceptance).
A last note has to be done CuLE's implementation, that is informed by ease of debugging, need for flexibility, and compatibility with standard DRL benchmarks. These choices put a limit on the achievable speed up factor (for instance by using emulation of the Atari 2600 console instead of direct CUDA implementations of Atari games), but the analysis and insights provided in our paper furnish indications for the design of efficient simulators for DRI.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mohammad Babacizadeh, luri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. GA3C; gpu- based A3C for deep reinforcement learning. CoRR, abs/1611 1ens/161100666,, .06256, 2016. URL http://
arxiv. www. abs 611. 16111.00 06256
Mohammad Babaeizadeh, luri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz, Reinforcement learning through asynchronous advantage actor-critic gpu. In ICLR, 2017.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributtional perspective on reinforcement learning. CoRR, abs/1707.06887 2017. URL
htspgg/abs/1000 Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Alfredo V. Clemente, Humberto Nicolás Castejón Martínez, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. CoRR, abs/1705.04862. 2017. URL hRthtt/://aaxii
org/abs/170 1705 04862
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley. Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CoRR, 0bss/882.01535 2018. httttttpp///rrttt wwtp:/wwwwrr///////xeeeeeee
01561
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. CoRR, abs/1706. 10295, 2017. URL http::tt///c httt:://arxiv. org/
abs/1706. 10295
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Daniel Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.02298,, 2017. URL http://
arxiv.org/anl
710. 02298
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. CoRR, abs/1 0ts/1803300933. 2018. URL.
http://arxiv.. wttp:/awwwv....oaaaa.o
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint drXiv:1611.0539) 2016.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, Abs//1009.0291, 2015. URL httttt/pp//aiiii org/ Saegg/abs/150 abss1509 02971
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. CoRR, abs/1 709.06009. 2017.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning navigate in complex environments. CoRR, abs/1611.03677 2016. URL http::/ http:///xxvvvvvo org/ hwtpg//ab///11.....// 03673
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx.. doi wtt.:ddo..ooo...o. 1038 naturel 4236

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforeement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp- 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http:
//proceedings mlr. www.spressssee.. v48 Setess//v8//mihallee mem/manihall html
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg. Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods for deep reinforcement learning. CoRR, abs/1507.04296, 2015. URL htlLtttpp//axxv.cooo /bs/
1507. 04296
OpenAl. Openai baselines: Acktr a2c, 2017. URL bttpa://blog..opne. com/ Daselines-acke a2c/
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-P- W, 2017.
Jing Peng and Ronald Williams, Incremental multi-step q-learning Machine Learning, 22(1): 283-290, Mar 1996. ISSN 1573-0565. doi: 10. 1007/BF00114431. hRL/httss://cc. org/ 10. 10.1007/FF0011473
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, abs/151 1105555, 2015. Utl.http://arxis http://awwvvv..r////////1111111. .0952
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.0634. 06347, 2017. URL http://arxxv.orq//aa
77777 0.0.70000 06347
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. CoRR, abs/1803.02811 2018a. URL /
02811
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. CoRR, abs/1803.00 0883.02811, 2018b. Utl.itttp.//aarav..or///aaa//a/a 02811
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Mar- tin A. Riedmiller. Deepmind control suite. CoRR, wttp://////.....ooooo///
2018. URL http:
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and Larry Zitnick. ELF: extensive, lightweight and flexible research platform for real-time strategy games. CoRR, abs/1 abs/1700.00167 2017. U /ttpa/bbb///1//00 C1067
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double 4- learning. CoRR, abs/1559.06461, 2015. URL sttp:://axxi..or//aaa.. 06461
Ziyu Wang, Nando de Freitas, and Mare Lanctot. Dueling network architectures for deep rein- forcement learning. CoRR, abs/1511.06581,, 2015. URL http://aaxxlv.orrr/bb///
06581
Ronald J Williams.
Simple statistical learning. Machine learning, ((3-4):22-2254 1992.
algorithms for connectionist reinforcement
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 REINFORCEMENT LEARNING, A2C AND V-TRACE
Reinforcement learning In RI., agent observes state 8; at time and follows a policy (8;) to select an action a;; the agent also receives a scalar reward ra from the environment. The goal of RL is optimize such that the sum of the expected rewards is maximized.
In model-free policy gradient methods (ar sy;; 0) is the output of policy DNN with weights 0, and represents the probability of selecting action a₁ the state 8t. Updates the DNN are generally aligned in the direction of the gradient of E Re). where RS = Ees y'rrss is the discounted reward from time t, with discount factor y E (0, 1] (see also REINFORCE Williams (1992)) The vanilla implementation updates 0 along Vo log x (a. 8y; 0) Rg which is an unbiased estimator of F(R)). The training procedure can be improved by reducing the variance the estimator by subtracting learned baseline (x() and using the gradient V, log π (a, sy; [R, - (s())]- One common baseline is the value function (s(s)) = E R4 st], which is the expected return for the policy π starting from st The policy and the baseline b, can be viewed as actor and critic in an actor-critic architecture Sutton & Barto 1998).
A2C A2C OpenAl (017) is the synchronous version of A3C Mnih al. (2016). successful actor-critic algorithm, where a single DNN outputs a softmax layer for the policy क (at |8; 0), and linear layer for V (st; 0). In A2C, multiple agents perform simultaneous steps on of parallel environments, while the DNN is updated every 'ma actions using the experiences collected by all the agents in the last maa steps. This means that the variance of the critic V (st; 0) is reduced (at the price of an increase in the bias) by N N-step bootstrapping, with N = fmar The cost function for
the policy is then:
log (a. RRo V (s(; + 38 [iT (s);
(1)
where 0g are the DNN weights 0 at time t, D 7 + V ($t+4: 0₂) is the bootstrapped
discounted reward from & + k and k is upper-bounded by bymax, and H [Ir (st; 0)] is an entropy term that favors exploration, weighted by the value function is:
(3. The cost function for the estimated
Ri v (sri (sri
(2)
which uses, again, the bootstrapped estimate R. Gradients 40 collected from both of the cost functions; standard optimizers, such as Adam or RMSProp, can be used for optimization.
V-trace In the case where there is a large number of environments, such as in CuLE or IM- PALA Espeholt et al. (2018), the synchronous nature of A2C become detrimental for the learning speed, as one should wait for all the environments to complete fmaz steps before computing single DNN update. Faster convergence is achieved (both in our paper and in EEpehhlt et al.] (2018) by desynchronizing data generation and DNN updates, which in practice means sampling a subset of experiences generated by the agents, and updating the policy using an approximate gradient, which makes the algorithm slightly off policy.
To correct for the off-policy nature of the data, that may lead to inefficiency or, even worse, in- stabilities, in the training process, V-trace introduced in Espeholt et al. (2018). In summary, the aim of off off-policy correction is to give less weight to experiences that have been generated with pol- icy μ. called the behaviour policy, when it differs from the targer policy, क; for a more principled explanation remand the curios reader to Espeholt et al. (208))
For a set of experiences collected from time t = to to time = to + N following some policy μ., the N-steps V-trace target for V (St,; 0) is defined as:
13

---

Under review as a conference paper at ICLR 2020
Resetiin enve
envs
KFPS
KFPS
100
200
300
500
1000
Asterix, CPU
(b) Pong, CPU
Resetting envs
Resetting envs
KFPS
20
8
200
100
000
100
200
300
100
(c) Ms Pacman, CPU
(d) Assault, CPU
Figure 8: FPS as function of the environment step, measured on System I in Table 2 for emulation only on four Atari games, 512 environments, for Cul.Ecru; each panel also shows the number of resetting environments. A peak in the FPS at the beginning of the emulation period, as in the case of GPU emulation in Fig. is not visible this case.
Uto V(sta 0) + > 1
-t= 70-00 [I
/ os ).)o,,
(3)
V Pc(re + VV((+11 V(s(:0)
(4)
Pt min (p. m(ue(s))
(5)
p(ar(se)
min (c, T(a (s)
(6)
Pe and e truncated importance sampling (IS) weights, and I C 1 for t, and p > Notice that, when we adopt the proposed multi-batching strategy, there are multiple behaviour policies y that have been followed to generate the training data - e.g., N different policies used when SPU=1 in Fig. 6(b) Eqs. 5 do not need to be changed in this case, but we have to store all the psa,,ss;) in the training buffer compute the, V-trace corrected, DNN update. In implementation, we compute the V-trace update recursively as:
t2 (ss 0) S.V + yca (1 ((s++100)
At training time f, update / with respect the value output, vs given by:
(2t - V(x; V (o.())))
(8)
whereas the policy gradient is given by:
Va log (a. 74 V(s 0)).
(9)
An entropy regularization term that favors exploration and prevents premature convergence (as in Eq. T is also added.
A.2 THREADD DIVERGENCE IS NOT PRESENT IN THE CASE OF CPU EMULATION
We show here that thread divergence, that affects GPU-based emulation (see Fig. 4) does not affect CPU-based emulation. Fig. shows the FPS four Atari games where all the environments share the same initial state. In constrast with GPU emulation, the CPU FPS do not peak at the beginning of the emulation period, where many environments are correlated.
14

---

Under review as a conference paper at ICLR 2020
A.3 PERFORMANCE DURING TRAINING OTHER GAMES
For sake of space, we only report (Fig. 9 the FPS measured on system in Table for three addi- tional games, function of different load conditions and number of environments.
(a) Pong
(b) MsPacman
(c) Asterix
Figure 9: FPS generated by different emulation engines on System I in Table 2 for different Atari games, as function of the number of environments, and different load conditions (the main A2C OpenAl 2017) loop run here, with N-step bootstrapping, N = 5.
A.4 CORRECTNESS OF THE IMPLEMENTATION
To demonstrate the correctness of implementation, and thus that policies learned with CuLE generalize to the same game emulated by OpenAI Gym. we report in Fig. I0 the average scores achieved in testing, while training an agent with with A2C+V-trace and CuLE. The testing scores measured on Cul and OpenAl Gym environments do not show any relevant statistical differ- ence, even for the case of Ms-Pacman, where the variability of the scores higher because of the nature of the game.
2000
3000
2000
2000
3000
22
Store
Store
Scoe
u00
D00
M
-32
Open
-000
-000
44
X32
X10
=100
xUF
(a) Assault
(b) Asterix
(c) Ms-Pacman
(d) Pong
Figure 10: Average testing scores measured on 10 CuI.Ecpu and OpenAl Gym environments, while training with V-Caeetrace and CuLE, as a function of the training frames; 250 environments used for Ms-Pacman, given its higher variability. The shaded area represents 2 standard deviations.
15


Reviews:
[{"review_id": "ryl7uG2atH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work contributes a library emulating Atari games in GPU in parallel and allowing to speed-up the execution of reinforcement learning algorithms.\n\nI see that the paper qualifies to the conference; in particular there is listed the topic:\n\n- “implementation issues, parallelization, software platforms, hardware”\n\nHowever, this is not a research paper, and I do not really see how I should asses it. What I can say about it is that it is considerable amount of work, not only implementing the simulator but also looking at what RL methods need, and how to optimize the allocation and exchange of the data so that everything would work on GPU more efficiently.\n\nFrom the practical perspective, I am somewhat confused. The speed-up factors in the experiments are rather modest: about 4x for simulating and rendering frames, 2.5x for full RL, on a single GPU. Better with scaling to multi-GPU systems. In Table 1 the total training time per resources used differs dramatically. However if I look at the lines with A2C it is about the same time with 100-200 CPU cores + 1 GPU versus 12 cores + 1 GPU. So this is about factor 10 in the resources, versus CPU parallelization probably suffering overheads.\n\nIt appears that the maximum steed-ups are achieved for a particular type of the reinforcement learning algorithms, and using it in a general case would give a modest improvement.\n\nThe paper itself consists of introduction, related work, 1 page overview of what it means to simulate the Atari games, and experiments. So it is mostly about measuring the speedups, with several implementations / platforms.\n\nI tend to think that this work will not very much boost the research for new RL methods. It is limited to Atari games, mostly helps to sample-inefficient RL methods and if it helps, the speed-up factors are not of the order that would make experiments by the researchers otherwise impossible. \n\nI would also give priority to theoretical contributions at ICLR. In the end, we all are using CUDA and cnDNN, but presentations about how they implement things are rather given at GPU computing conferences. \n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJgL80qkqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a port of the Atari Learning Environment to CUDA, reports on a set of performance comparison, and provides a bottleneck analysis based communication bandwidth and various throughputs required to saturate them for training and inference.\n\nMy first reaction to this paper was, \"So what?\"; but as I read more, I like the paper more and more.  It was the bottleneck analysis that changed my mind.  It was done very thoroughly and it provides deep insight in the challenges that RL faces for both learning and inference in a variety of settings.  I especially liked the analysis of the advantages and limitations of GPU emulation.  I also thought the Discussion section was well written.\n\nThe paper would be better if:\n1) The figure fonts were larger throughout the paper.\n2) The gaps in Table 1 were explained.\n\nMinor issue:  Change \"feed\" to \"fed\" on page 3.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxxYQ73FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a CUDA port of the Atari Learning Environment. The paper goes into detail examining the benefits that come from a GPU-only implementation, including much better per-GPU utilization as well as no need to run a distributed system of CPUs. They show this hardware scaling can be taken advantage of across a variety of state of the art reinforcement learning algorithms and indeed create new batching strategies to utilize their framework and the GPU better. \n\nThe paper is well written and goes into some detail describing the implementation of CuLE as well as various design decisions taken, as with splitting the emulation process across several kernels. Finally, the paper is very explicit about a number of optimizations that are not being exploited by the new framework and serve as markers for future work.\n\nA question that arises and which is not addressed in the experiments is how the authors verified their port is faithful to the original version; there is no mention of correctness in the paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented a detailed discussion on the implementation of a library emulating Atari games on GPU for efficient reinforcement learning. The analysis is very thoroughly done. The major concern is whether this paper is a good fit to this conference. The developed library would be useful to researchers and the discussion is interesting with respect to system design and implementation, but the technical depth seems not sufficient.
Decision: Reject