Year: 2021

Abstract:
Deep Q-Network (DQN) marked a major milestone for reinforcement learning, demonstrating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization. Even years after its introduction, DQN remains highly relevant to the research community since many of its innovations have been adopted by successor methods. Nevertheless, despite significant hardware advances in the interim, DQN’s original Atari 2600 experiments remain extremely costly to replicate in full. This poses an immense barrier to researchers who cannot afford state-of-the-art hardware or lack access to large-scale cloud computing resources. To facilitate improved access to deep reinforcement learning research, we introduce a DQN implementation that leverages a novel concurrent and synchronized execution framework designed to maximally utilize a heterogeneous CPU-GPU desktop system. With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. The ideas introduced in our paper should be generalizable to a large number of off-policy deep reinforcement learning methods.

text: Deep Q-Network (DQN) marked a major milestone for reinforcement learning, demonstrating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization. Even years after its introduction, DQN remains highly relevant to the research community since many of its innovations have been adopted by successor methods. Nevertheless, despite significant hardware advances in the interim, DQN’s original Atari 2600 experiments remain extremely costly to replicate in full. This poses an immense barrier to researchers who cannot afford state-of-the-art hardware or lack access to large-scale cloud computing resources. To facilitate improved access to deep reinforcement learning research, we introduce a DQN implementation that leverages a novel concurrent and synchronized execution framework designed to maximally utilize a heterogeneous CPU-GPU desktop system. With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. The ideas introduced in our paper should be generalizable to a large number of off-policy deep reinforcement learning methods.

heading: 1 INTRODUCTION
text: Reinforcement learning (Sutton & Barto, 2018) has long grappled with the ramifications of the Curse of Dimensionality (Bellman, 1966), a phenomenon in which exact solution methods become hopelessly intractable in the face of high-dimensional state spaces. As such, Deep Q-Network (DQN) (Mnih et al., 2013; 2015) was heralded as a landmark achievement for the field, establishing “deep” methods as a promising avenue for controlling environments that emit rich sensory observations. Through an effective combination of Q-Learning (Watkins, 1989) and a deep convolutional neural architecture (Krizhevsky et al., 2012), DQN became the first algorithm to achieve human-level performance on a majority of the Atari 2600 games when learning directly from raw pixel inputs. In contrast to previous efforts to integrate neural networks into reinforcement learning (e.g. Tesauro, 1992; Riedmiller, 2005), DQN proved to be robust, efficient, and scalable. Deep reinforcement learning has consequently become an active area of research in recent years.
Although DQN’s performance on the Atari benchmark (Bellemare et al., 2013) has been surpassed by later methods (e.g. Hessel et al., 2018; Badia et al., 2020; Schrittwieser et al., 2020), the algorithm remains pertinent to ongoing deep reinforcement learning research. Its core elements—minibatched experience replay and a time-delayed target network—have been adopted by most off-policy deep reinforcement learning methods (e.g. Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018). As such, the DQN algorithm is a good testbed for new ideas, as improvements to it are often directly transferable to state-of-the-art methods. Furthermore, its relatively straightforward implementation compared to modern successors, as well as its widely replicated results, have made it a reliable baseline for benchmarking and validating new methods. These factors have made DQN crucial to the research community even years after its introduction.
In spite of substantial improvements to computing hardware over nearly a decade, DQN is still expensive to train. The large computational cost stems from the need to conduct gradient-based optimization on a multimillion-parameter convolutional neural network. The original Atari 2600 experiments from Mnih et al. (2015), in particular, remain prohibitive for many to replicate. Agent training was conducted for 200 million frames on each of the 49 games considered, for a total of 9.8 billion frames (equivalent to about 5.2 years of real experience). Conducting the entirety of these experiments is utterly infeasible without access to costly Graphics Processing Units (GPUs), and can still take many weeks without access to a great number of them. This poses a significant barrier to deep reinforcement learning researchers, particularly those who lack substantial cloud computing resources. Given DQN’s importance as a testbed and a baseline, this barrier puts a majority of researchers at an unfair disadvantage when it comes to publishing their ideas.
To foster improved accessibility to deep reinforcement learning research, we analyze the algorithmic structure of DQN and look for opportunities to reduce its runtime when executed on a standard CPU-GPU desktop system. In contrast to a separate line of inquiry into distributed DQN methods that focus on scaling to a large number of nodes (e.g. Nair et al., 2015; Ong et al., 2015; Horgan et al., 2018), we specifically consider the challenges of optimizing performance for a resource-constrained local system. We develop a modified DQN implementation based on a novel framework of Concurrent Training and Synchronized Execution; the framework is general and fits nicely into other target network-based methods as well. When trained on a single NVIDIA GeForce GTX 1080 GPU, our implementation reduces the runtime of a full Atari experiment (200 million frames) from 25 hours to just 9 hours compared to a highly optimized baseline DQN implementation. At this rate, all 49 experiments from Mnih et al. (2015) can be replicated in a relatively short timeframe using highly affordable, off-the-shelf hardware. Our implementation achieves human- and DQN-level performance on a large majority of the games. We plan to publicly release the code after the submission period to aid other researchers in reproducing these results quickly with limited hardware requirements.

heading: 2 BACKGROUND
text: DQN can be understood as the combination of Q-Learning and deep convolutional neural networks, along with supporting infrastructure to make this combination stable. The neural networks’ generalization ability helps DQN learn effective control policies in the face of high-dimensional sensory inputs (e.g. images) where classic reinforcement learning and dynamic programming would be intractable. We provide a brief summary of the DQN algorithm in our notation (Section 2.1) and then establish assumptions about the CPU-GPU hardware model for which we will optimize its runtime (Section 2.2).

heading: 2.1 DQN AND NOTATION
text: In essence, DQN inherits the same fundamental objective of Q-Learning: to estimate a function Q : S ×A 7→ R such that acting according to a greedy policy π(s) = argmaxaQ(s, a) maximizes the agent’s expected discounted return Eπ[ ∑∞ t=1 γ
t−1rt] for some γ ∈ [0, 1] (Sutton & Barto, 2018). Here, the environment is defined as a Markov Decision Process (MDP) of the form (S,A, T,R). The sets S and A contain the environment states and agent actions, respectively, that are permissible in the decision process. The agent executes an action at given the state st at the current timestep t and receives a scalar reward rt := R(st, at), triggering a stochastic transition to a new state st+1 ∈ S with probability T (st, at, st+1). Whereas Q-Learning implements the Q-function as a lookup table, DQN implements it as a deep neural network parameterized by a vector θ. Learning then amounts to a first-order minimization with respect to θ of a squared-error loss:
L(θ) := 1
2
( rt + γmax
a′∈A Q(st+1, a
′; θ−)−Q(st, at; θ) )2
(1)
Rather than conducting updates immediately upon collecting a new experience (as Q-Learning does), DQN buffers each experience (st, at, rt, st+1) in a replay memory D. Every F timesteps, the agent conducts a
gradient update on a minibatch of replayed experiences (Lin, 1992) sampled randomly from D. This helps to circumvent the strong temporal correlations between successive experiences while also efficiently reusing samples (Mnih et al., 2015). For additional stability, the maximization in (1) is conducted using a stationary “target” network parameterized by a separate vector θ−. DQN updates the target network every C timesteps by copying the parameters from the main network: i.e. θ− ← θ. Following Mnih et al. (2015), we assume that each action at is selected according to an -greedy policy (Sutton & Barto, 2018). That is, the agent selects the greedy action argmaxaQ(st, a) with probability t ∈ [0, 1] and an action randomly from A otherwise. The -greedy strategy linearly interpolates between the uniform-random and greedy policies, helping to balance exploration with exploitation (Sutton & Barto, 2018). In practice, it is common to start with t = 1 early in training and gradually reduce its value over time; we discuss the exact -schedules used in our experiments in Section 5.

heading: 2.2 HARDWARE MODEL
text: Optimizing the performance of any algorithm necessarily requires some assumptions about the type of computer system on which it is being executed. In the interest of generality, we defer a discussion of our particular hardware specifications until Section 5. For now, it will be more useful to outline the general capabilities of the systems under our consideration here.
We define our abstract machine as a heterogeneous system that consists of two components: a Central Processing Unit (“CPU”) and a coprocessor optimized for massively parallel computation (“GPU”).1 We assume that the GPU is suitable only for neural network operations: i.e. Q-value prediction (inference) and training (backpropagation). All other faculties are to be handled by the CPU, including but not limited to sampling from the environment, managing the replay memory, and preprocessing input data for the neural network. We also assume that the CPU is capable of executing W program threads simultaneously.
As a result, the system can process up toW CPU tasks and one GPU task in parallel. In practice, the original DQN algorithm only executes one task at a time, which is inefficient. The goal in the following sections is to modify the algorithm such that the machine’s capabilities are fully utilized.

heading: 3 CONCURRENT TRAINING
text: The DQN algorithm repeatedly alternates between executing F actions in its environment and then conducting a single training update on a minibatch of replayed experiences. While this is effective for finely interleaving data generation with learning, it is not efficient for the heterogeneous CPU-GPU systems we consider here. This is because either the CPU or the GPU is left idle at any given point in the process.
Ideally, we would fill these idle intervals by ensuring that a CPU- and GPU-intensive tasks are executed in parallel at all times. Unfortunately, the original DQN algorithm cannot be refactored to permit this possibility. The dilemma is that execution and training are sequentially dependent on each other. To see this, suppose that the agent has just completed a gradient update at some timestep t that produces the current network parameters θ. The DQN agent would now interact with its environment for F steps, computing the Q-values Q(si, ai; θ) for i ∈ {t, . . . , t+ F − 1} in the process. The next update, which produces a new set of parameters θ′, is scheduled to occur at the beginning of timestep t+F . Note that we could not have conducted this update any earlier, since the Q-values for action selection depended on θ and not θ′. On the other hand, we cannot proceed with the next sequence of Q-values Q(si, ai; θ′) for i ∈ {t + F, . . . , t + 2F − 1}
1For ease of presentation, we refer to any such coprocessor as a Graphics Processing Unit (GPU). GPUs are presently the most common form of hardware acceleration for machine learning research; they were used by Mnih et al. (2015) and are used for our experiments too (Section 5). Of course, other matrix-optimized Application-Specific Integrated Circuits (ASICs) would be equally suitable; these may become more prevalent in the future. The optimizations we propose here are sufficiently general to apply to any of these possible implementations.
since these will depend on θ′, which has not yet been prepared by the GPU. Sampling and training are interlocked processes as a consequence, making simultaneous execution impossible.
Rather than attempting to preserve DQN’s original control flow, let us instead consider a slight modification to it. The sequential dependency in DQN arises from the fact that the training procedure needs to overwrite θ with θ′, but the sampling procedure requires θ for action selection. If these tasks relied on different sets of parameters, then we would be able to parallelize these tasks. Quite fortunately, DQN already has a second parameter set available: its target network parameters θ−. We therefore propose to substitute θ− as a surrogate for the main parameters θ during execution; that is, greedy actions are computed by argmaxaQ(s, a; θ
−) during -greedy exploration. The mild assumption here is that a policy derived from θ− should perform comparably to a policy derived from θ. Given that θ− is a time-delayed copy of θ that differs by at most C timesteps of experience, this assumption should not be problematic in practice.
The result of this subtle change is that execution and training can now be decoupled (Figure 1), at least until the target parameters θ− must be updated. Instead of sampling F experiences and then training on one minibatch, it is possible to sample C experiences while concurrently training on C / F minibatches using a separate program thread. Overall, the total computation is the same, but both the CPU and GPU are kept busy. After C timesteps have elapsed, the two threads must synchronize to copy θ− from θ as usual. To avoid a race condition between the threads, we also temporarily buffer the experiences collected by the sampler thread, transferring them to the replay memory D only when the threads are synchronized. This ensures that D does not change during training, which could produce non-deterministic results.
Related Work We found no previous algorithms that utilize a similar technique to this type of concurrency, in which the target network parameters θ− are used to break the dependency between sampling and training.
Daley & Amato (2019) first demonstrated that DQN with grouped minibatch training—where C / F minibatches of training are conducted every C timesteps—could learn effectively, where it was employed for the purposes of efficient λ-return calculation. However, unlike in our work, actions were still sampled using θ and training was not conducted concurrently. Distributed DQN methods that rely on a centralized training server (e.g. Nair et al., 2015; Ong et al., 2015; Horgan et al., 2018) could be viewed as an alternative form of concurrent training, since training and sampling occur simultaneously (although possibly on distinct physical systems). This type of concurrency is quite different from ours, since gradient computation is performed at the nodes, and the updates are applied to the parameters asynchronously and in a non-deterministic order. For these reasons, we do not expect that distributed methods would be efficient on a monolithic CPU-GPU system that we consider in our work (we elaborate on this point in Section 4).

heading: 4 SYNCHRONIZED EXECUTION
text: Concurrent training helps utilize the GPU more effectively by conducting training when the device would otherwise be idle. Assuming that sampling from the environment for C timesteps takes longer than training on C / F minibatches (our preliminary experiments indicated that this is the case), then concurrency effectively masks the computational cost of training. We illustrate this effect in Figure 2. Because training is not the performance bottleneck anymore, the algorithm’s critical path has become the agent-environment interaction. To further reduce the overall runtime, we must turn our attention to techniques that allow the agent to gather samples more rapidly.
Since it is assumed that the environment’s interaction speed cannot be optimized itself, a common approach for accelerating sample collection is to instantiate multiple (simulated) environments that can be updated in parallel on the CPU (e.g. Mnih et al., 2016). Based on our hardware assumptions in Section 2.2, up to W environments can be updated at once. Simultaneously sampling from these instances reduces the average time spent collecting each individual sample (Figure 2). These so-called asynchronous methods do not rely on locks or other synchronization primitives when updating the parameters θ, which can theoretically increase overall throughput. However, this creates other issues: updates are applied in a non-deterministic
order, may be outdated with respect to the current parameters (due to latency), and may be clobbered by subsequent updates (thereby wasting samples).
When specifically targeting a monolithic CPU-GPU system, asynchronous learning encounters another problem regarding efficiency: the number of GPU transactions scales linearly with the number of samplers. Since the GPU has limited bandwidth, having too many threads attempting to communicate with it simultaneously can saturate the communication bus. Specifically, the samplers must compete for one shared GPU when computing their greedy actions for environment interaction, leading to diminishing improvements to runtime even when more parallel environments are added.
We propose to resolve this with a Synchronized Execution model (Figure 3). After the W sampler threads each take a step in their respective environments, they are blocked by the main thread, which aggregates the environments’ states into a single minibatch for efficient Q-value prediction on the GPU. The resulting Q-values are then distributed back to the sampler threads; the threads are subsequently unblocked. This process then repeats, with the sampler threads using the precomputed Q-values to select their actions in each iteration. In practice, the states and Q-values can be stored in shared memory arrays, meaning that no explicit message passing actually occurs between the threads. Synchronized Execution yields two major benefits. First, it exploits the massively parallel capabilities of the GPU by computing Q-values in a larger minibatch. Second, the total number of transactions between the CPU and GPU is reduced by a factor of W ; in other words, this number is now independent of the number of samplers. While synchronizing the sampler threads every W timesteps may incur a slight overhead cost, it is far outweighed by these computational benefits.
Synchronized Execution is orthogonal to Concurrent Training (Section 3), meaning that they can be combined into a single, fast DQN algorithm. Altogether, the main program thread becomes responsible for orchestrating W sampler threads and one trainer thread, for a total of W + 1 threads executing concurrently (the main thread does not perform computation). We outline pseudocode for this in Algorithm 1. For ease of
Algorithm 1 DQN with Concurrent Training and Synchronized Execution (main thread) Prepopulate replay memory D with N experiences collected randomly Initialize parameter vector θ randomly Start W sampler threads and one trainer thread Create environment instance with state s(j) for each sampler thread, j ∈ {1, . . . ,W} for t = 1, 2, . . . until convergence do
if t mod C = 1 then Wait for trainer thread and all sampler threads to finish their tasks Flush all sampler threads’ temporary buffers to D Update target network parameters: θ− ← θ Asynchronously dispatch trainer thread for C / F minibatches sampled from D end if i← t modW if i = 1 then
Wait for all sampler threads to finish their tasks In one minibatch, compute Q-values Q(s(j), ·) for each sampler thread, j ∈ {1, . . . ,W} end if Asynchronously dispatch i-th sampler thread for one step in its environment using Q(s(i), ·)
end for
presentation, we omit the specific details of the sampler and trainer threads (which are identical to standard DQN but executed in parallel), focusing instead on the high-level dispatching logic. Related Work The Synchronized Execution model in Section 4 is closely related to the strategy employed by PAAC (Clemente et al., 2017). PAAC is an online actor-critic method that similarly relies on shared memory arrays for efficient inter-thread communication during shared-minibatch synchronous execution. Stooke & Abbeel (2018) later applied this same form of synchronization to various deep reinforcement learning methods including DQN. In contrast to our work, training was conducted asynchronously via a distributed strategy across multiple GPUs, and the replay memory was statically partitioned across workers. We believe we are the first to integrate synchronized execution into a non-distributed, off-policy agent in which training minibatches are conducted in a fully deterministic order and retrieved from a globally shared replay memory. As an alternative to shared memory arrays, other parallel actor-critic methods like GA3C (Babaeizadeh et al., 2016) and Impala (Espeholt et al., 2018) have used queues to dynamically batch experiences, granting them enhanced robustness to slow or faltering workers, but also adding substantial complexity to the algorithms.

heading: 5 EXPERIMENTS
text: We design our experiments to answer two important questions regarding Algorithm 1. First, to what extent do Concurrent Training and Synchronized Execution reduce the runtime given fixed hardware specifications? Our hyperparameters of interest are whether these features are enabled, and how many sampling threads are created. Second, how does the fastest hyperparameter configuration’s performance compare to the scores attained by DQN on the 49 Atari 2600 games tested by Mnih et al. (2015)?
To implement the Atari games, we use OpenAI Gym (Brockman et al., 2016) as an interface to the Arcade Learning Environment (ALE) (Bellemare et al., 2013). Our experiment setup is identical to that of Mnih et al. (2015), except where stated otherwise in Section 5.1.

heading: 5.1 SPEED TEST
text: We begin by conducting an ablation study on our fast DQN implementation, measuring the individual contribution of each component to the overall speed of the algorithm. We consider 14 algorithmic variants based
on whether Concurrent Training or Synchronized Execution is enabled and how many sampler threads are executed (chosen from {1, 2, 4, 8}). For now, we focus solely on wall-clock runtime for these experiments, deferring any performance evaluations until Section 5.2.
Our test hardware consists of a four-core (eight-thread) Intel Core i7-7700K CPU and an NVIDIA GeForce GTX 1080 GPU. Notably, this particular GPU model is not considered state of the art even for desktop hardware, having been released in 2016. As such, comparable or better hardware should be affordable to a large number of researchers and practitioners.
We choose Pong from the 49 Atari games as our test environment, although the choice of game is arbitrary and unlikely to impact the timing results. We make two slight changes to the experiment setup compared to that of Mnih et al. (2015). First, we train the agents for one million timesteps (instead of 50 million) and then multiply the results by a factor of 50. While this may increase the variance of the reported data, it makes the experiments much more feasible to implement. Second, we fix t = 0.1 for the entirety of this shorter training duration. This represents the greediest portion of the -schedule used by Mnih et al. (2015) and, in turn, a worst-case scenario for GPU usage. Furthermore, this value of t corresponds to the vast majority (98%) of the schedule, meaning that the results here closely reflect the full experiments.
We emphasize that our baseline DQN implementation has been heavily optimized in the interest of a fair comparison. All of our tested variations share the same core code—including time-critical portions like the deep neural network and replay memory. This means that any observable speedup is due only to Concurrent Training or Synchronized Execution. Our baseline appears to be significantly faster than popular existing DQN implementations (e.g. Roderick et al., 2017; Dhariwal et al., 2017; Hill et al., 2018) based on their reported results, although we did not compare them on our hardware. In any case, our proposed optimizations are equally applicable to all of these DQN agents.
For each hyperparameter combination, we report the mean runtime and standard deviation computed over five independent trials (Table 1). (Recall that these values were multiplied by 50.) We also alternatively express each mean as a percentage (Table 2) and as a speedup factor (Table 3) with respect to DQN’s mean runtime, in order to help reveal trends in the data. We make several interesting observations. First, for any
number of sampler threads, enabling concurrency or synchronization is always helpful. Their benefits are synergistic too; enabling both is always faster than enabling just one. Interestingly, the reverse is not true; with both concurrency and synchronization disabled, increasing the number of sampler threads beyond four fails to further reduce the runtime. We hypothesize that this is due to intense GPU competition between the threads that we discussed in Section 4. Finally, and perhaps most importantly, the fastest combination is achieved by combining both features and maximizing the number of sampler threads, which almost triples the overall speed. These findings indicate that Concurrent Training and Synchronized Execution work together to enable full utilization of the system resources.

heading: 5.2 ATARI 2600 SUITE
text: We now evaluate the fastest hyperparameter configuration from Section 5.1 on all 49 Atari games, following the experimental procedures from Mnih et al. (2015) as closely as possible. The agents are trained for 50 million timesteps (200 million frames) in each game, and are periodically evaluated during training by executing an -greedy policy ( = 0.05) for 30 episodes in a separate environment instance. We conduct the evaluation every 250,000 timesteps (Roderick et al., 2017) and report the best mean performance attained in each game (see Appendix A). Note that here we used four computers (each with multiple GPUs) to accelerate results collection, although each agent was still trained on exactly one GPU.
Out of the 49 games, our agent achieved human-level performance—i.e. at least 75% human-normalized score (Mnih et al., 2015)—in 33 games. Not only does this constitute a clear majority, but it also exceeds the 29 human-level scores achieved by DQN. Furthermore, our implementation outperformed DQN on 28 of the games—once again, a majority. We attribute the improved performance to the parallel environment instances, which help to diversify experiences via enhanced exploration while reducing correlations between experiences.
Overall, our implementation performed comparably to or better than DQN in 37 games. In the remaining 12 games where performance was not quite as good, our agent converged to relatively stable (but lower) scores roughly halfway through training. We speculate that this is due to the inadequate exploratory capabilities of the -greedy policy. Indeed, several of these games like Private Eye and Venture are considered to be hard-exploration problems (Ostrovski et al., 2017); given that the scores for both DQN and our method consisted of only a single trial on each game, it is not surprising that the results would exhibit high variance due to random chance. Nevertheless, our strong performance across a large majority of the games gives us confidence that our implementation is effective for quickly learning human-level control policies on the Atari benchmark with modest hardware requirements.

heading: 6 CONCLUSION
text: We presented a fast implementation of DQN that is designed to efficiently utilize the resources of a CPUGPU system. When executed on affordable desktop hardware, the implementation reduces the runtime of a full Atari experiment from 25 hours to just 9 hours—nearly triple the speed of our highly optimized DQN baseline. On top of this, the trained agent exceeded both human- and DQN-level performance in a majority of the 49 games tested.
We hope that the impact of our work will be twofold. First, once the code is released publicly, our implementation can function as a baseline or test environment for quickly validating new ideas in deep reinforcement learning. Second, our optimizations described here can inspire analogous adaptations to other important deep reinforcement learning baselines like DDPG (Lillicrap et al., 2015) and Rainbow (Hessel et al., 2018) for execution on desktop hardware. Continued efforts in this direction would further reduce computational barriers to research and be of great benefit to the community.

heading: A ATARI 2600 RESULTS
text: Table 4 (next page) contains the best scores attained across all of the evaluation periods during training, on each of the 49 Atari games. The learners considered are Random, Human, and DQN (reproduced from Mnih et al. (2015)) as well as our fast DQN implementation. We also report the humannormalized performance for DQN and our method. The human-normalized (“norm.”) scores are computed as 100× (Score−Random) / (Human−Random). The higher human-normalized score between DQN and our implementation is emphasized (bold). Parenthetical values represent standard deviation calculated over the 30 evaluation episodes.

heading: B HYPERPARAMETERS
text: Table 5 below summarizes the DQN-specific hyperparameters and their values used in all of our experiments. The networks were trained using “centered” RMSProp (Hinton et al., 2012) with a learning rate of 2.5 × 10−4, first- and second-moment decay values of 0.95, and a small constant of 0.01 added to the denominator. These values are identical to those from Mnih et al. (2015), but are restated here for reference.

Review summaries:
The authors propose an implementation of DQN that focuses on maximising the data / training throughput in a common CPU-GPU machine setting. They do so by (asynchronously) running inference on multiple environments at once (and synchronously blocking on the env batch) and using the DQN target network parameters for inference rather than the default -- latest -- parameters of the network. They show considerable improvements in speed while producing comparable results with previous DQN results on the Atari suite. 

A new method for benchmarking RL algorithms like DQN, focusing on Atari suite and experiments from seminal DQN paper. The focus is to achieve speedup, allowing researchers with a modest setup (pc with gpu) to run the tests in reasonable time. The method combines concurrent data collection in multiple threads (and multiple environment instances) and training and achieves a speedup of x2.7 compared to vanilla DQN. At the same time the method slightly changes the nature of the algorithm by different action sampling (multiple env instances which is reminiscent of Asynchronous Methods paper from 2016) and slightly different action selection (using target network). The results are comparable to ones in the original paper overall, but vary on per game basis probably due to mentioned changes, slight change in hyper parameters and high variance of a single run. 

DQN is known to consume a lot of resources. This paper introduces an optimized version of DQN which speeds up training 25->9 hours. The authors use two main techniques to improve throughput. Firstly, the authors propose to select actions by computing the argmax over the target network. This decouples acting and training and allows these two operations to run asynchronously. Secondly, the authors compute actions for multiple environments in parallel on the GPU.