Human-Level Control without Server-Grade Hardware

Deep Q-Network ( DQN ) marked a major milestone for reinforcement learning , demonstrating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization . Even years after its introduction , DQN remains highly relevant to the research community since many of its innovations have been adopted by successor methods . Nevertheless , despite significant hardware advances in the interim , DQN ’ s original Atari 2600 experiments remain extremely costly to replicate in full . This poses an immense barrier to researchers who can not afford state-of-the-art hardware or lack access to large-scale cloud computing resources . To facilitate improved access to deep reinforcement learning research , we introduce a DQN implementation that leverages a novel concurrent and synchronized execution framework designed to maximally utilize a heterogeneous CPU-GPU desktop system . With just one NVIDIA GeForce GTX 1080 GPU , our implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours . The ideas introduced in our paper should be generalizable to a large number of off-policy deep reinforcement learning methods . 1 INTRODUCTION . Reinforcement learning ( Sutton & Barto , 2018 ) has long grappled with the ramifications of the Curse of Dimensionality ( Bellman , 1966 ) , a phenomenon in which exact solution methods become hopelessly intractable in the face of high-dimensional state spaces . As such , Deep Q-Network ( DQN ) ( Mnih et al. , 2013 ; 2015 ) was heralded as a landmark achievement for the field , establishing “ deep ” methods as a promising avenue for controlling environments that emit rich sensory observations . Through an effective combination of Q-Learning ( Watkins , 1989 ) and a deep convolutional neural architecture ( Krizhevsky et al. , 2012 ) , DQN became the first algorithm to achieve human-level performance on a majority of the Atari 2600 games when learning directly from raw pixel inputs . In contrast to previous efforts to integrate neural networks into reinforcement learning ( e.g . Tesauro , 1992 ; Riedmiller , 2005 ) , DQN proved to be robust , efficient , and scalable . Deep reinforcement learning has consequently become an active area of research in recent years . Although DQN ’ s performance on the Atari benchmark ( Bellemare et al. , 2013 ) has been surpassed by later methods ( e.g . Hessel et al. , 2018 ; Badia et al. , 2020 ; Schrittwieser et al. , 2020 ) , the algorithm remains pertinent to ongoing deep reinforcement learning research . Its core elements—minibatched experience replay and a time-delayed target network—have been adopted by most off-policy deep reinforcement learning methods ( e.g . Lillicrap et al. , 2015 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) . As such , the DQN algorithm is a good testbed for new ideas , as improvements to it are often directly transferable to state-of-the-art methods . Furthermore , its relatively straightforward implementation compared to modern successors , as well as its widely replicated results , have made it a reliable baseline for benchmarking and validating new methods . These factors have made DQN crucial to the research community even years after its introduction . In spite of substantial improvements to computing hardware over nearly a decade , DQN is still expensive to train . The large computational cost stems from the need to conduct gradient-based optimization on a multimillion-parameter convolutional neural network . The original Atari 2600 experiments from Mnih et al . ( 2015 ) , in particular , remain prohibitive for many to replicate . Agent training was conducted for 200 million frames on each of the 49 games considered , for a total of 9.8 billion frames ( equivalent to about 5.2 years of real experience ) . Conducting the entirety of these experiments is utterly infeasible without access to costly Graphics Processing Units ( GPUs ) , and can still take many weeks without access to a great number of them . This poses a significant barrier to deep reinforcement learning researchers , particularly those who lack substantial cloud computing resources . Given DQN ’ s importance as a testbed and a baseline , this barrier puts a majority of researchers at an unfair disadvantage when it comes to publishing their ideas . To foster improved accessibility to deep reinforcement learning research , we analyze the algorithmic structure of DQN and look for opportunities to reduce its runtime when executed on a standard CPU-GPU desktop system . In contrast to a separate line of inquiry into distributed DQN methods that focus on scaling to a large number of nodes ( e.g . Nair et al. , 2015 ; Ong et al. , 2015 ; Horgan et al. , 2018 ) , we specifically consider the challenges of optimizing performance for a resource-constrained local system . We develop a modified DQN implementation based on a novel framework of Concurrent Training and Synchronized Execution ; the framework is general and fits nicely into other target network-based methods as well . When trained on a single NVIDIA GeForce GTX 1080 GPU , our implementation reduces the runtime of a full Atari experiment ( 200 million frames ) from 25 hours to just 9 hours compared to a highly optimized baseline DQN implementation . At this rate , all 49 experiments from Mnih et al . ( 2015 ) can be replicated in a relatively short timeframe using highly affordable , off-the-shelf hardware . Our implementation achieves human- and DQN-level performance on a large majority of the games . We plan to publicly release the code after the submission period to aid other researchers in reproducing these results quickly with limited hardware requirements . 2 BACKGROUND . DQN can be understood as the combination of Q-Learning and deep convolutional neural networks , along with supporting infrastructure to make this combination stable . The neural networks ’ generalization ability helps DQN learn effective control policies in the face of high-dimensional sensory inputs ( e.g . images ) where classic reinforcement learning and dynamic programming would be intractable . We provide a brief summary of the DQN algorithm in our notation ( Section 2.1 ) and then establish assumptions about the CPU-GPU hardware model for which we will optimize its runtime ( Section 2.2 ) . 2.1 DQN AND NOTATION . In essence , DQN inherits the same fundamental objective of Q-Learning : to estimate a function Q : S ×A 7→ R such that acting according to a greedy policy π ( s ) = argmaxaQ ( s , a ) maximizes the agent ’ s expected discounted return Eπ [ ∑∞ t=1 γ t−1rt ] for some γ ∈ [ 0 , 1 ] ( Sutton & Barto , 2018 ) . Here , the environment is defined as a Markov Decision Process ( MDP ) of the form ( S , A , T , R ) . The sets S and A contain the environment states and agent actions , respectively , that are permissible in the decision process . The agent executes an action at given the state st at the current timestep t and receives a scalar reward rt : = R ( st , at ) , triggering a stochastic transition to a new state st+1 ∈ S with probability T ( st , at , st+1 ) . Whereas Q-Learning implements the Q-function as a lookup table , DQN implements it as a deep neural network parameterized by a vector θ . Learning then amounts to a first-order minimization with respect to θ of a squared-error loss : L ( θ ) : = 1 2 ( rt + γmax a′∈A Q ( st+1 , a ′ ; θ− ) −Q ( st , at ; θ ) ) 2 ( 1 ) Rather than conducting updates immediately upon collecting a new experience ( as Q-Learning does ) , DQN buffers each experience ( st , at , rt , st+1 ) in a replay memory D. Every F timesteps , the agent conducts a gradient update on a minibatch of replayed experiences ( Lin , 1992 ) sampled randomly from D. This helps to circumvent the strong temporal correlations between successive experiences while also efficiently reusing samples ( Mnih et al. , 2015 ) . For additional stability , the maximization in ( 1 ) is conducted using a stationary “ target ” network parameterized by a separate vector θ− . DQN updates the target network every C timesteps by copying the parameters from the main network : i.e . θ− ← θ . Following Mnih et al . ( 2015 ) , we assume that each action at is selected according to an -greedy policy ( Sutton & Barto , 2018 ) . That is , the agent selects the greedy action argmaxaQ ( st , a ) with probability t ∈ [ 0 , 1 ] and an action randomly from A otherwise . The -greedy strategy linearly interpolates between the uniform-random and greedy policies , helping to balance exploration with exploitation ( Sutton & Barto , 2018 ) . In practice , it is common to start with t = 1 early in training and gradually reduce its value over time ; we discuss the exact -schedules used in our experiments in Section 5 . 2.2 HARDWARE MODEL . Optimizing the performance of any algorithm necessarily requires some assumptions about the type of computer system on which it is being executed . In the interest of generality , we defer a discussion of our particular hardware specifications until Section 5 . For now , it will be more useful to outline the general capabilities of the systems under our consideration here . We define our abstract machine as a heterogeneous system that consists of two components : a Central Processing Unit ( “ CPU ” ) and a coprocessor optimized for massively parallel computation ( “ GPU ” ) .1 We assume that the GPU is suitable only for neural network operations : i.e . Q-value prediction ( inference ) and training ( backpropagation ) . All other faculties are to be handled by the CPU , including but not limited to sampling from the environment , managing the replay memory , and preprocessing input data for the neural network . We also assume that the CPU is capable of executing W program threads simultaneously . As a result , the system can process up toW CPU tasks and one GPU task in parallel . In practice , the original DQN algorithm only executes one task at a time , which is inefficient . The goal in the following sections is to modify the algorithm such that the machine ’ s capabilities are fully utilized . 3 CONCURRENT TRAINING . The DQN algorithm repeatedly alternates between executing F actions in its environment and then conducting a single training update on a minibatch of replayed experiences . While this is effective for finely interleaving data generation with learning , it is not efficient for the heterogeneous CPU-GPU systems we consider here . This is because either the CPU or the GPU is left idle at any given point in the process . Ideally , we would fill these idle intervals by ensuring that a CPU- and GPU-intensive tasks are executed in parallel at all times . Unfortunately , the original DQN algorithm can not be refactored to permit this possibility . The dilemma is that execution and training are sequentially dependent on each other . To see this , suppose that the agent has just completed a gradient update at some timestep t that produces the current network parameters θ . The DQN agent would now interact with its environment for F steps , computing the Q-values Q ( si , ai ; θ ) for i ∈ { t , . . . , t+ F − 1 } in the process . The next update , which produces a new set of parameters θ′ , is scheduled to occur at the beginning of timestep t+F . Note that we could not have conducted this update any earlier , since the Q-values for action selection depended on θ and not θ′ . On the other hand , we can not proceed with the next sequence of Q-values Q ( si , ai ; θ′ ) for i ∈ { t + F , . . . , t + 2F − 1 } 1For ease of presentation , we refer to any such coprocessor as a Graphics Processing Unit ( GPU ) . GPUs are presently the most common form of hardware acceleration for machine learning research ; they were used by Mnih et al . ( 2015 ) and are used for our experiments too ( Section 5 ) . Of course , other matrix-optimized Application-Specific Integrated Circuits ( ASICs ) would be equally suitable ; these may become more prevalent in the future . The optimizations we propose here are sufficiently general to apply to any of these possible implementations . since these will depend on θ′ , which has not yet been prepared by the GPU . Sampling and training are interlocked processes as a consequence , making simultaneous execution impossible . Rather than attempting to preserve DQN ’ s original control flow , let us instead consider a slight modification to it . The sequential dependency in DQN arises from the fact that the training procedure needs to overwrite θ with θ′ , but the sampling procedure requires θ for action selection . If these tasks relied on different sets of parameters , then we would be able to parallelize these tasks . Quite fortunately , DQN already has a second parameter set available : its target network parameters θ− . We therefore propose to substitute θ− as a surrogate for the main parameters θ during execution ; that is , greedy actions are computed by argmaxaQ ( s , a ; θ − ) during -greedy exploration . The mild assumption here is that a policy derived from θ− should perform comparably to a policy derived from θ . Given that θ− is a time-delayed copy of θ that differs by at most C timesteps of experience , this assumption should not be problematic in practice . The result of this subtle change is that execution and training can now be decoupled ( Figure 1 ) , at least until the target parameters θ− must be updated . Instead of sampling F experiences and then training on one minibatch , it is possible to sample C experiences while concurrently training on C / F minibatches using a separate program thread . Overall , the total computation is the same , but both the CPU and GPU are kept busy . After C timesteps have elapsed , the two threads must synchronize to copy θ− from θ as usual . To avoid a race condition between the threads , we also temporarily buffer the experiences collected by the sampler thread , transferring them to the replay memory D only when the threads are synchronized . This ensures that D does not change during training , which could produce non-deterministic results . Related Work We found no previous algorithms that utilize a similar technique to this type of concurrency , in which the target network parameters θ− are used to break the dependency between sampling and training . Daley & Amato ( 2019 ) first demonstrated that DQN with grouped minibatch training—where C / F minibatches of training are conducted every C timesteps—could learn effectively , where it was employed for the purposes of efficient λ-return calculation . However , unlike in our work , actions were still sampled using θ and training was not conducted concurrently . Distributed DQN methods that rely on a centralized training server ( e.g . Nair et al. , 2015 ; Ong et al. , 2015 ; Horgan et al. , 2018 ) could be viewed as an alternative form of concurrent training , since training and sampling occur simultaneously ( although possibly on distinct physical systems ) . This type of concurrency is quite different from ours , since gradient computation is performed at the nodes , and the updates are applied to the parameters asynchronously and in a non-deterministic order . For these reasons , we do not expect that distributed methods would be efficient on a monolithic CPU-GPU system that we consider in our work ( we elaborate on this point in Section 4 ) .

Summary: DQN is known to consume a lot of resources. This paper introduces an optimized version of DQN which speeds up training 25->9 hours. The authors use two main techniques to improve throughput. Firstly, the authors propose to select actions by computing the argmax over the target network. This decouples acting and training and allows these two operations to run asynchronously. Secondly, the authors compute actions for multiple environments in parallel on the GPU.