Accelerating Reinforcement Learning Through GPU Atari Emulation

1 INTRODUCTION Initially triggered by the success of DQN Mnih et al . ( 2015 ) , research in Deep Reinforcement Learning ( DRL ) has grown in popularity in the last years Lillicrap et al . ( 2015 ) ; Mnih et al . ( 2016 ; 2015 ) , leading to intelligent agents that solve non-trivial tasks in complex environments . But DRL also soon proved to be a challenging computational problem , especially if one wants to achieve peak performance on modern architectures . Fig . 1 is a schematic representation of a modern DRL algorithm implementation : CPU environments execute a set of actions { at−1 } at time t − 1 , and produce observable states { st } and rewards { rt } . These data are migrated to a Deep Neural Network ( DNN ) on the GPU to eventu- ally select the next action set , { at } , which is copied back to the CPU . This sequence of operations defines the inference path , whose main aim is to generate training data . A training buffer on the GPU stores the states generated on the inference path ; this is periodically used to update the DNN ’ s weights θ , according to the training rule of the DRL algorithm ( training path ) . A computationally efficient DRL system should balance the data generation and training processes , while minimizing the communication overhead along the inference path and consuming , along the training path , as many data per second as possible Babaeizadeh et al . ( 2016 ; 2017 ) . The solution to this problem is however non-trivial and many DRL implementations do not leverage the full computational potential of modern systems Stooke & Abbeel ( 2018b ) . We focus our attention on the inference path and move from the traditional CPU implementation of the Atari Learning Environment ( ALE ) , a set of Atari 2600 games that emerged as an excellent DRL benchmark Bellemare et al . ( 2013 ) ; Machado et al . ( 2017 ) . We show that significant performance bottlenecks primarily stem from CPU environment emulation : the CPU can not run a large set of environments simultaneously , the CPU-GPU communication bandwidth is limited , and the GPU is consequently underutilized . To both investigate and mitigate these limitations we introduce CuLE ( CUDA Learning Environment ) , a DRL library containing a CUDA enabled Atari 2600 emulator , that renders frames directly in the GPU memory , avoids off-chip communication and achieves high GPU utilization by processing thousands of environments in parallel—something so far achievable only through large and costly distributed systems . Compared to the traditional CPU-based approach , GPU emulation improves the utilization of the computational resources : CuLE on a single GPU generates more Frames Per Second1 ( FPS ) on the inference path ( between 39K and 125K , depending on the game , see Table 1 ) compared to its CPU counterpart ( between 12.5K and 19.8K ) . Beyond offering CuLE ( [ URL revealed upon acceptance ] ) as a tool for research in the DRL field , our contribution can be summarized as follow : ( 1 ) We identify common computational bottlenecks in several DRL implementations that prevent effective utilization of high throughput compute units and effective scaling to distributed systems . ( 2 ) We introduce an effective batching strategy for large environment sets , that allows leveraging the high throughput generated by CuLE to quickly reach convergence with A2C+V-trace Espeholt et al . ( 2018 ) , and show effective scaling on multiple GPUs . This leads to the consumption of 26-68K FPS along the training path on a single GPU , and up to 187K FPS using four GPUs , comparable ( Table 1 ) to those achieved by large clusters Stooke & Abbeel ( 2018a ) ; Espeholt et al . ( 2018 ) . ( 3 ) We analyze advantages and limitations of GPU emulation with CuLE in DRL , including the effect of thread divergence and of the lower ( compared to CPU ) number of instructions per second per thread , and hope that our insights may be of value for the development of efficient DRL systems . 2 RELATED WORK The wall clock convergence time of a DRL algorithm is determined by two main factors : its sample efficiency , and the computational efficiency of its implementation . Here we analyze the sample and computational efficiency of different DRL algorithms , in connection with their implementation . 1Raw frames are reported here and in the rest of the paper , unless otherwise specified . These are the frames that are actually emulated , but only 25 % of them are rendered and used for training . Training frames are obtained dividing the raw frames by 4—see also Espeholt et al . ( 2018 ) . We first divide DRL algorithms into policy gradient and Q-value methods , as in Stooke & Abbeel ( 2018b ) . Q-learning optimizes the error on the estimated action values as a proxy for policy optimization , whereas policy gradient methods directly learn the relation between a state , st , and the optimal action , at ; since at each update they follow , by definition , the gradient with respect to the policy itself , they improve the policy more efficiently . Policy methods are also considered more general , e.g . they can handle continuous actions easily . Also the on- or off-policy nature of an algorithm profoundly affects both its sample and computational efficiency . Off-policy methods allow re-using experiences multiple times , which directly improves the sample efficiency ; additionally , old data stored in the GPU memory ( replay buffer in Fig . 1 ) can be used to continuously update the DNN on the GPU , leading to high GPU utilization without saturating the inference path . The replay buffer has a positive effect on the stability of learning as well Mnih et al . ( 2015 ) . On-policy algorithms saturate the inference path more easily , as frames have to be generated on-the-fly using the current policy and moved from the CPU emulators to the GPU for processing with the DNN . On-policy updates are generally effective but they are also more prone to fall into local minima because of noise , especially if the number of environment is small — this is the reason why on-policy algorithms largely benefit ( in term of stability ) from a significant increase of the number of environments . Among the policy gradient methods , A3C Mnih et al . ( 2016 ) is an asynchronous , on-policy , actorcritic algorithm where 16 CPU agents use copies of the same DNN to interact with CPU environments ; N -step bootstrapping is used to reduce the variance of the critic , V ( st ; θ ) : the agents send updates to the DNN after every N = 5 actions , and the new set of global θ weights are broadcast to all agents after each update . A3C can solve an Atari game in approximately 4 days ; its hybrid CPUGPU implementation ( GA3C Babaeizadeh et al . ( 2016 ; 2017 ) ) stores a single DNN on the GPU , collects CPU data in a system of queues to send them to the GPU in large batches , and achieves convergence in 1 day . Recent ( and faster ) policy gradient implementations , like PAAC Clemente et al . ( 2017 ) , PPO Schulman et al . ( 2017 ) , or A2C OpenAI ( 2017 ) , store training data directly in a GPU training buffer , as depicted in Fig . 1 , which is well representative of the flow of computation of modern DRL algorithms . Policy gradient algorithms are often on-policy : their efficient update strategy is counterbalanced by the bottlenecks in the inference path and competition for the use of the GPU along the inference and training path at the same time . Acceleration by scaling to a distributed system is possible but inefficient in this case : in IMPALA Espeholt et al . ( 2018 ) a cluster with hundreds of CPU cores is needed to accelerate A2C , while training is desynchronized to hide latency . As a consequence , the algorithm becomes off-policy , and V-trace was introduced to deal with off-policy data ( see details in the Appendix ) . Acceleration on a DGX-1 has also been demonstrated for A2C and PPO , using large batch sizes to increase the GPU occupancy , and asynchronous distributed models that hide latency , but require periodic updates to remain synchronized Stooke & Abbeel ( 2018b ) and overall achieves sublinear scaling with the number of GPUs . In practice , the efficiency generally scales sub-linearly on distributed systems , and though Atari games can be solved in a few hours or even minutes , the cost of such systems makes them prohibitive for many researchers . Q-value methods , like DQN Mnih et al . ( 2015 ) and its more recent , improved versions ( DoubleDQN van Hasselt et al . ( 2015 ) , Dueling Networks Wang et al . ( 2015 ) , Prioritized Replay Schaul et al . ( 2015 ) , n-step learning Peng & Williams ( 1996 ) , NoisyNets Fortunato et al . ( 2017 ) , Rainbow Hessel et al . ( 2017 ) ) are generally off-policy and can be accelerated more easily , as the inference and training path can be decoupled , with the GPU continuously fed with training data from the replay buffer in Fig.1 . Mapping the two paths on different devices is natural and effective , and indeed DQN methods have been significantly accelerated on distributed systems ( see Nair et al . ( 2015 ) and Jaderberg et al . ( 2016 ) in Table 1 ) . The replay buffer increases the sample efficiency , but the overall convergence time has to pay the cost of using a Q-value method . Furthermore , implementation on large scale systems still incurs a communication overhead and thus generally achieves sub-linear scaling on large distributed systems . The best compromise between sample efficiency , computational efficiency and effectiveness of the DNN update seems to be achieved by using an on-policy while correcting for stale data , gradient policy methods like A2C+V-trace , which is therefore the algorithm analyzed in detail here , but still suffers from computational bottlenecks in its CPU distributed implementation Espeholt et al . ( 2018 ) . Other Data Generation Engines Frameworks to generate training experiences already exist , like the well-known OpenAI Gym , built on top of the C++ Stella emulator Brockman et al . ( 2016 ) for Atari games . Other frameworks that target specific problems , such as navigation Mirowski et al . ( 2016 ) , aim at creating large benchmark for DRL Tassa et al . ( 2018 ) , or accelerate simulation on the CPU with optimized C++ multi-threaded implementations , while also providing large , GPU-friendly batches for inference and training Tian et al . ( 2017 ) . To the best of our knowledge , however , none of these attempts directly address the problem of optimizing the inference path , i.e. , trying to increase the number of environments while avoiding the CPU and bandwidth bottlenecks , by providing a direct implementation of the environments on the GPU . 3 CUDA LEARNING ENVIRONMENT ( CULE ) . Despite the number and variety of games developed for the Atari 2600 , it is relatively simple to emulate the functioning of its hardware much faster than real-time . The Atari console has a 1.19Mhz CPU ( to execute the game instructions ) and a 128 bytes RAM ( containing the game state ) ; the code of each game is stored in a cartridge ROM ( typically 2–4kB ) . The instruction set of each game contains both game and rendering instructions ; the first ones are executed by the CPU , while the second ones are sent to the Television Interface Adaptor ( TIA ) , a secondary processor embedded in the Atari hardware that executes the rendering instructions to update a limited set of registers and render the 160 × 210 game screen using a 128- colour palette . In CuLE , we emulate the functioning of many Atari consoles in parallel using the CUDA programming model , where a sequential host program executes parallel programs , known as kernels , on a GPU . In a trivial mapping of the Atari emulator to CUDA , a single thread emulates both the Atari CPU and TIA to execute the ROM code , update the Atari CPU and TIA registers as well as the game state in the 128 bytes RAM , and eventually render the pixels in the output frame . However , the contrasting nature of the game code execution and renderings tasks , the first dominated by reading from the RAM/ROM and writing tens of bytes to RAM , while the second writes hundreds of pixels to the framebuffer , poses a serious issue in terms of performance , such as thread divergence and an imbalanced number of registers required by the first and second tasks . To mitigate these issues , CuLE uses two CUDA kernels : the first one first loads data from the GPU global memory , where we store the state of each emulated Atari processor , and the 128 bytes RAM data containing the current state of the game ; it also reads ROM instructions from the constant GPU memory , executes them to update the Atari CPU and game states , and stores the updated Atari CPU and game states back into GPU global memory . It is important to notice that this first kernel does not execute the TIA instructions read from the ROM , but copies them into the TIA instruction buffer in GPU global memory , which we implemented to decouple the execution of the Atari CPU and TIA instructions in CuLE . The second CuLE kernel emulates the functioning of the TIA processor : it first reads the instructions stored in the TIA instruction buffer , execute them to update the TIA registers , and renders the 160 × 210 output framebuffer in global GPU memory . Despite this implementation requires going through the TIA instruction twice , it has several advantages over the single-kernel trivial implementation . First of all , the requirements in terms of registers per thread and the chance of having divergent code are different for the Atari CPU and TIA kernels , and the use of different kernels achieves a better GPU usage . A second advantage that we exploit is that not all frames are rendered in ALE : the input of the RL algorithm is the pixelwise maximum between the last two frames in a sequence of four , so we can avoid calling the TIA kernel when rendering of the screen is not needed . A last advantage , not exploited in our implementation yet , is that the TIA kernel may be scheduled one the GPU with more than one thread per game , as rendering of diverse rows on the screen is indeed a parallel operation - we leave this optimization for future developments of CuLE . To better fit our execution model , our game reset strategy is also different from the one in the existing CPU emulators , where 64 startup frames are executed at the end of each episode . Furthermore , wrapper interfaces for RL , such as ALE , randomly execute an additional number of frames ( up to 30 ) to introduce randomness into the initial state . This results into up to 94 frames to reset a game , which may cause massive divergence between thousands of emulators executing in SIMD fashion on a GPU . To address this issue , we generate and store a cache of random initial states ( 30 by default ) when a set of environments is initialized in CuLE . At the end of an episode , each emulator randomly selects one of the cached states as a seed and copies it into the terminal emulator state . Some of the choices made for the implementation of CuLE are informed by ease of debugging , like associating one state update kernel to one environment , or need for flexibility , like emulating the Atari console instead of directly writing CUDA code for each Atari game . A 1-to-1 mapping between threads and emulators is not the most computationally efficient way to run Atari games on a GPU , but it makes the implementation relatively straightforward and has the additional advantage that the same emulator code can be executed on the CPU for debugging and benchmarking ( in the following , we will refer to this implementation as CuLECPU ) . Despite of this , the computational advantage provided by CuLE over traditional CPU emulation remains significant , as shows in the next Section .

Summary: This paper introduces a CUDA port of the Atari Learning Environment. The paper goes into detail examining the benefits that come from a GPU-only implementation, including much better per-GPU utilization as well as no need to run a distributed system of CPUs. They show this hardware scaling can be taken advantage of across a variety of state of the art reinforcement learning algorithms and indeed create new batching strategies to utilize their framework and the GPU better.