Human-Level Control without Server-Grade Hardware
Authors: Brett Daley
Christopher Amato
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Reinforcement Learning

Abstract: Deep Q-Network (DQN) marked a major milestone for reinforcement learning, demonstrating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization. Even years after its introduction, DQN remains highly relevant to the research community since many of its innovations have been adopted by successor methods. Nevertheless, despite significant hardware advances in the interim, DQN's original Atari 2600 experiments remain extremely costly to replicate in full. This poses an immense barrier to researchers who cannot afford state-of-the-art hardware or lack access to large-scale cloud computing resources. To facilitate improved access to deep reinforcement learning research, we introduce a DQN implementation that leverages a novel concurrent and synchronized execution framework designed to maximally utilize a heterogeneous CPU-GPU desktop system. With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. The ideas introduced in our paper should be generalizable to a large number of off-policy deep reinforcement learning methods.

Under review conference paper at ICLR 2022
HUMAN-LEVEL CONTROL WITHOUT SERVER-GRADE HARDWARE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Q-Network (DQN) marked a major milestone for reinforcement learning, demon- strating for the first time that human-level control policies could be learned directly from raw visual inputs via reward maximization. Even years after its introduction, DQN re- mains highly relevant to the research community since many of its innovations have been adopted by successor methods. Nevertheless, despite significant hardware advances in the interim, DQN's original Atari 2600 experiments remain extremely costly to replicate in full. This poses an immense barrier to researchers who cannot afford state-of-the- hardware or lack access to large-scale cloud computing resources. To facilitate improved access to deep reinforcement learning research, we introduce a DQN implementation that leverages novel concurrent and synchronized execution framework designed to maxi- mally utilize heterogeneous CPU-GPU desktop system. With just one NVIDIA GeForce GTX 1080 GPU, our implementation reduces the training time of 200-million-fraa Atari experiment from 25 hours to just 9 hours. The ideas introduced in our paper should be generalizable to large number of off-policy deep reinforcement learning methods.
1 INTRODUCTION
Reinforcement learning (Sutton& & Barto 2018) has long grappled with the ramifications of the Curse of Dimensionality (Bellman 1966). phenomenon in which exact solution methods become hopelessly in- tractable in the face of high-dimensional state spaces. As such, Deep Q-Network (DQN) (Mnih et al. 2013] 2015) was heralded avenue for controlling environments that emit rich sensory observations. Through an effective combination of Q-Leamnng (Watkins| 1989) and deep convolutional neural architecture (Krizhevsky et al. 2012). DQN became the first algorithm to achieve human-level performance on majority of the Atari 2600 games when learning directly from raw pixel inputs. In contrast to previous efforts integrate neural networks into rein- forcement learning (c.g. Tesauro 1992 Riedmiller, 2005). DQN proved to be robust, efficient, and scalable. Deep reinforcement learning has consequently become an active area of research in recent years.
landmark achievement for the field, establishing "deep" methods
promising
Although DQN's performance on the Atari benchmark (Bellemare et al. 2013) has been surpassed by later methods (e.g. Hessel et al.] 2018 Badia al., [020); Schrittwieser et al. 2020), the algorithm remains experience re- play and time-delayed target network- have been adopted by most off-policy deep reinforcement learning methods (c.g. Lillicrap et al.) 2015) Fujimoto et al.) 2018 Haarnoja al.) 2018). As such, the DQN algo- rithm is good testbed for new ideas, as improvements to it are often directly transferable to state-of-the-an methods. Furthermore, its relatively straightforward implementation compared to modern successors, well as its widely replicated results, have made a reliable baseline for benchmarking and validating new methods. These factors have made DQN crucial to the research community years after its introduction.
pertinent to ongoing deep reinforcement learning research. Its core

---

Under review conference paper at ICLR 2022
In spite of substantial improvements to computing hardware over nearly decade, DQN is still expensive to train. The large computational cost stems from the need to conduct maudient-basede optimization on a multimilioon-praaetee convolutional neural network. The original Atari 2600 experiments from Mnih et al. (2015), in particular, remain prohibitive for many to replicate. Agent training was conducted for 200 million frames on each of the 49 games considered, for a total of 9.8 billion frames (equivalent to about 5.2 years of real experience). Conducting the entirety of these experiments is utterly infeasible without access to costly Graphics Processing Units (GPUs), and can still take many weeks without access to a great number of them. This poses significant barrier to deep reinforcement learning researchers, particularly those who lack substantial cloud computing resources. Given DQN's importance as a testbed and a baseline, this barrier unfair disadvantage when it comes to publishing their ideas.
puts a majority of researchers
To foster improved accessibility to deep reinforcement learning research, we analyze the algorithmic struc- ture of DQN and look for opportunities reduce its runtime when executed on standard CPU-GPU desktop system. In contrast to a separate line of inquiry into distributed DQN methods that focus on scaling to a large number of nodes (e.g. Nair et al. 2015 Ong et al., 2015 Horgan et al. 2018). we specifically consider the challenges of optimizing performance for a resource-constraing constrained local system. We develop a modified DQN implementation based on novel framework of Concurrent Training and Synchronized Execution; the frame- work is general and fits nicely into other target nework-basee methods as well. When trained on single NVIDIA GeForce GTX 1080 GPU, our implementation reduces the runtime of full Atari experiment (200 million frames) from 25 hours to just 9 hours compared to highly optimized baseline DQN implementa- tion. At this rate, all 49 experiments from Mnih et al. 2015) can replicated a relatively short timeframe using highly affordable, off-the-shelf hardware. Our implementation achieves human- and DQN-level per- formance on a large majority of the games. We plan to publicly release the code after the submission period to aid other researchers in reproducing these results quickly with limited hardware requirements.
2 BACKGROUND
DQN can be understood as the combination of Q-Learning and deep convolutional neural networks, along with supporting infrastructure to make this combination stable. The neural networks' generalization ability helps DQN learn effective control policies in the face of high-dimensional sensory inputs (e.g. images) where classic reinforcement learning and dynamic programming would be intractable. We provide a brief summary of the DQN algorithm in our notation (Section 2.J) and then establish assumptions about the CPU-GPU hardware model for which we will optimize its runtime (Section 2.2
2.1 DQN AND NOTATION
In essence, DQN inherits the same fundamental objective of Q-Learning: to estimate function Q: S A + Ik such that acting according to a greedy policy т (s)) = argmaxx, Q(s, a) maximizes the agent's expected discounted return E, <>> "r="re] for some 7 [0, 1] Suttonn & Barto 2018). Here, the environment is defined as a Markov Decision Process (MDP) of the form (S,A,T.)). The sets S and A contain the environment states and agent actions, respectively, that permissible in the decision process. The agent executes an action 0f given the state 8, at the current timestep and receives a scalar reward T₁ R(s,, at), triggering stochastic transition new state 5t+1 E S with probability T( az Sc+1).
Whereas Q-Learning implements the Q-function as a lookup table, DQN implements it as a deep neural network parameterized by vector 0. Learning then amounts to a first-order minimization with respect to 0 of squared-error loss:
((0) (rs + m ((5++: ((:08))))(((ss(())))) ((s),);;)))
2
(1)
a'EAA
Rather than conducting updates immediately upon collecting a new experience (as Q-Learning does), DQN buffers each experience (St, Gz Te, 8241) in a replay memory D. Every F timesteps, the agent conducts
2

---

Under review conference paper at ICLR 2022
gradient update on a minibatch of replayed experiences (Lin 1992) sampled randomly from D. This helps to circumvent the strong temporal correlations between successive experiences while also efficiently reusing samples (Mnih et al. 2015). For additional stability, the maximization in [[] is conducted using stationary "target" network parameterized by a separate vector f- DQN updates the target network every C' timesteps by copying the parameters from the main network: i.e. 0 +- 0.
Following Mnih et al. (2015), we assume that each action is selected according to an e-greedy policy (Sutton & Barto 2018). That is, the agent selects the greedy action argmax, Q+x; a) with probability E [0, 1] and an action randomly from A otherwise. The e-greedy strategy linearly interpolates between the uniform-random random and greedy policies, helping to balance exploration with exploitation (Sutton & Barto, 2018). In practice, it is common to start with c 1 early in training and gradually reduce its value over time; we discuss the exact e-schedules used in our experiments in Section 5
2.2 HARDWARE MODEL
Optimizing the performance of any algorithm necessarily requires some assumptions about the type of com- puter system on which it is being executed. In the interest of generality, we defer discussion of our particular hardware specifications until Section [ For now, will be more useful to outline the general capabilities of the systems under our consideration here.
We define our abstract machine as a heterogeneous system that consists of two components: a Central Processing Unit ("CPU") and a coprocessor optimized for massively parallel computation ("GPU") We assume that the GPU is suitable only for neural network operations: i.c. Q-value prediction (inference) and training (backpropagation). All other faculties are to be handled by the CPU, including but not limited to sampling from the environment, managing the replay memory, and preprocessing input data for the neural network. We also assume that the CPU is capable of executing W program threads simultaneooslyy
As a result, the system can process up to W CPU tasks and one GPU task in parallel. In practice, the original DQN algorithm only executes one task at time, which is inefficient. The goal in the following sections is to modify the algorithm such that the machine's 's capabilities are fully utilized.
3 CONCURRENT TRAINING
The DQN algorithm repeatedly alternates between executing F actions in its environment and then con- ducting single training update on minibatch of replayed experiences. While this is effective for finely interleaving data generation with learning, it is not efficient for the heterogeneous CPU-GPU systems we consider here. This is because either the CPU or the GPU is left idle at any given point in the process.
Ideally, we would fill these idle intervals by ensuring that a CPU- and GPU-intensive tasks are executed in parallel at all times. Unfortunately, the original DQN algorithm cannot be refactored to permit this possi- bility. The dilemma is that execution and training are sequentially dependent on each other. To see this, suppose that the agent has just completed a gradient update some timestep t that produces the current network parameters 0. The DQN agent would now interact with its environment for F steps, computing the Q-values Q(si, a;; 0) for i E (1, .l, + F 1} in the process. The next update, which produces new set of parameters 0', is scheduled to occur at the beginning of timestep + F. Note that we could not have con- ducted this update any earlier, since the Q-values for action selection depended on and not 0'. On the other hand, we cannot proceed with the next sequence of Q-values (s,,, ag; 0') for E {t + F...., t 2F - 1}
'For ease of presentation, we refer any such coprocessor as a Graphics Processing Unit (GPU). GPUs are presently the most common form of hardware acceleration for machine learning research; they were used by Mmih et al. (2015) and are used for our experiments too (Section 5). Of course, other matrix-optimized (ASICs) would be equally suitable; these may become more prevalent in the future. The optimizations propose here are sufficiently general apply any of these possible implementations.
Ieplication-Speciii Integrated Circuits
3

---

Under review conference paper at ICLR 2022
Sterl
Updele targel network
Updale largel network
Take C environment
stops environment
using forgel
minibatches repley memory
Store transttons in replay memory
Store transitons temporary buffer
Train minbarch from raploy memory
Synchronize
Flush temporary buffor to repleyy memory
limeslape elapsed?
(a) Control flow for DQN.
(b) Control flow for DQN with Concurrent Training.
Figure 1: Algorithm flowcharts for DQN (left) and DQN with our proposed Concurrent Training technique (right). By computing the agent's policy from the target network parameters 0- instead of the main network parameters 0, our method breaks the sequential dependency between training and environment sampling that is inherent to the standard DQN algorithm. This allows training and sampling occur in parallel via multithreading, greatly reducing the overall runtime.
since these will depend on 8', which has not yet been prepared by the GPU. Sampling and training are interlocked processes as a consequence, making simultaneous execution impossible.
Rather than attempting to preserve DQN's original control flow, us instead consider a slight modification to it. The sequential dependency in DQN arises from the fact that the training procedure needs to overwrite 0 with 8', but the sampling procedure requires / for action selection. If these tasks relied on different sets of parameters, then we would be able to parallelize these tasks. Quite fortunately, DQN already has a second parameter set available: its target network parameters 0- We therefore propose to substitute 0- surrogate for the main parameters 0 during execution; that is, greedy actions are computed by argmax, Q(s, a; ))) during c-greedy exploration. The mild assumption here is that a policy derived from 0 should perform comparably to policy derived from é. Given that A is a time-delayed copy of 0 that differs by at most C timesteps of experience, this assumption should not be problematic in practice.
The result of this subtle change that execution and training can be decoupled (Figure [. least until the target parameters 8 must be updated. Instead of sampling F experiences and then training on one minibatch, is possible to sample C' experiences while concurrently training on C. / minibatches using a separate program thread. Overall, the total computation is the same, but both the CPU and GPU are kept busy. After C' timesteps have elapsed, the two threads must synchronize To avoid a race condition between the threads, we also temporarily buffer the experiences collected by the sampler thread, transferring them to the replay memory D only when the threads are synchronized. This ensures that D does not change during training, which could produce non-deterministic results.
copy 0- from usual.
Related Work We found no previous algorithms that utilize a similar technique to this type of concurrency, in which the target network parameters 0 used to break the dependency between sampling and training.
4

---

Under review conference paper at ICLR 2022
aampaing
Standard
training
sampling
Concurrent
training
samp1 1nng
Concurrent Synchronized
training
Figure 2: Abstract timing diagrams illustrating the theoretical speedup of our optimizations in DQN. By overlapping sampling and training, the training cost can be effectively masked (concurrency). To mitigate the remaining sampling bottleneck, multiple environments can be simulated in parallel (synchronization).
Daley & Amato 2019) first demonstrated that DQN with grouped minibatch training-whert C F mini- batches of training are conducted every timesteps-coute learn effectively, where was employed for the purposes of efficient A-retur calculation. However, unlike in our work, actions were still sampled using o and training was not conducted concurrently. Distributed DQN methods that rely on a centralized training server (e.g. Nair et al. 20155 Ong et al.) [015 [Horaan et al. 2018) could be viewed as alternative form of concurrent training, since training and sampling occur simultaneously (although possibly on distinct physi- cal systems). This type of concurrency is quite different from ours, since gradient computation is performed at the nodes, and the updates are applied to the parameters asynchronously and in a non-deterministic order. For these reasons, do expect that distributed methods would be efficient on monolithic CPU-GPU system that we consider in our work (we elaborate on this point in Section 4).
4 SYNCHRONIZED EXECUTION
Concurrent training helps utilize the GPU more effectively by conducting training when the device would otherwise be idle. Assuming that sampling from the environment for C' timesteps takes longer than train- ing on C / minibatches (our preliminary experiments indicated that this the case), then concurrency effectively masks the computational cost of training. We illustrate this effect in Figure 2 Because training is not the performance bottleneck anymore, the algorithm's 's critical path has become the agent-environment interaction. To further reduce the overall runtime, we must turn our attention to techniques that allow the agent to gather samples more rapidly.
Since it is assumed that the environment's interaction speed cannot be optimized itself, common approach for accelerating sample collection to instantiate multiple (simulated) environments that can be updated parallel the CPU (c.g. Mnih et al.) 2016). Based on hardware assumptions in Section 2.2 up to W environments can be updated at once. Simultancously sampling from these instances reduces the average time spent collecting each individual sample (Figure 2). These so-called asynchronous methods do not rely on locks or other synchronization primitives when updating the parameters 6, which can theoretically increase overall throughput. However, this creates other issues: updates applied in coneeteeriiiistic
5

---

Under review conference paper at ICLR 2022
Environment
Environment
Enviroment
Environment
Envronment
Environment
Minibatchh
State
state
state
ptata
state
state
GPU
GPU
50t10n
actico
action
S0t1on
action
action
(a) Asynchronous execution. Threads compete for limited device bandwidth, reducing throughput.
(b) Synchronized execution. Threads synchronize and share device transactions in single minibatch.
Figure 3: Attempting to run asynchronous environments for deep reinforcement learning leads to significant resource competition between threads (left), introducing performance bottleneck. This can be alleviated by first synchronizing the threads and then sharing computation in a single minibatch (right).
order, may be outdated with respect to the current parameters (due to latency), and may be clobbered by subsequent updates (thereby wasting samples).
When specifically targeting monolithic CPU-GPU system, asynchronous learning encounters another prob- lem regarding efficiency: the number of GPU transactions scales linearly with the number of samplers. Since the GPU has limited bandwidth, having too many threads attempting to communicate with simultancously saturate the communication bus. Specifically, the samplers must compete for shared GPU when computing their greedy actions for environment interaction, leading to diminishing improvements to run- time even when more parallel environments are added.
We propose to resolve this with a Synchronized Execution model (Figure 3). After the W sampler threads each take step in their respective environments, they are blocked by the main thread, which aggregates the environments' states into a single minibatch for efficient Q-value prediction on the GPU. The resulting Q-values are then distributed back to the sampler threads; the threads are subsequently unblocked. This process then repeats, with the sampler threads using the precomputed Q-values to select their actions in each iteration. In practice, the states and Q-values can be stored in shared memory arrays, meaning that no explicit message passing actually occurs between the threads. Synchronized Execution yields two major benefits. First, exploits the massively parallel capabilities the GPU by computing Q-values a larger minibatch. Second, the total number of transactions between the CPU and GPU is reduced by factor of W; in other words, this number now independent the number of samplers. While synchronizing the sampler threads every W timesteps may incur slight overhead cost, is far outweighed by these computational benefits.
Synchronized Exccution is orthogonal to Concurrent Training (Section meaning that they can com- bined into single, fast DQN algorithm. Altogether, the main program thread becomes responsible for orchestrating W sampler threads and one trainer thread, for a total of W + threads executing concurrently (the main thread does not perform computation). We outline pseudocode for this in Algorithm T For ease of
6

---

Under review conference paper at ICLR 2022
Algorithm 1 DQN with Concurrent Training and Synchronized Execution (main thread)
Prepopulate replay memory D with N experiences collected randomly
Initialize parameter vector 0 randomly
Start W sampler threads and one trainer thread
Create environment instance with state أاب) for each sampler thread, (1....,W. W] for 1 = 1,2.... until convergence do
if mod C then
Wait for trainer thread and all sampler threads to finish their tasks Flush all sampler threads' temporary buffers
Update target network parameters: 0 +- 0
Asynchronously dispatch trainer thread for C/ minibatches sampled from D
end if
i 4 1 mod W
if = then
Wait for all sampler threads to finish their tasks
In one minibatch, compute Q-values ((s(1),) for each sampler thread, j E (1...., W)
end if
Asynchronously dispatch i-th sampler thread for one step in its environment using Q(s(1,,)
end for
presentation, omit the specific details of the sampler and trainer threads (which are identical to standard DQN but executed in parallel), focusing instead on the high-level dispatching logic.
Related Work The Synchronized Execution model in Section is closely related the strategy employed
by PAAC (Clemente et 2017). PAAC is an online actor-critic method that similarly relies on shared mem- ory arrays for efficient inter-thread communication during seared-minibatch synchronous execution. [Stooke] & Abbeel (2018) later applied this same form of synchronization to various deep reinforcement learning methods including DQN. In contrast to our work, training was conducted asynchronously via a distributed strategy across multiple GPUs, and the replay memory we are the first to integrate synchronized execution into a non- non-diittibuted, off off-policy agent in which training minibatches are conducted in fully deterministic order and retrieved from globally shared replay memory. As an alternative to shared memory arrays, other parallel actor-critic methods like GA3C (Babaeizadeh et al.) 2016) and Impala (Espeholt et al. (208) have used queues dynamically batch experiences, granting them enhanced robustness to slow or faltering workers, but also adding substantial complexity to the algorithms.
statically partitioned across workers. We believe
5 EXPERIMEN IMPERS
We design experiments answer two important questions regarding Algorithm I First, what extent do Concurrent Training and Synchronized Execution reduce the runtime given fixed hardware specifications? Our hyperparameters of interest are whether these features are enabled, and how many sampling threads are created. Second, how does the fastest hyperparameter configuration' 's performance compare to the scores attained by DQN on the 49 Atari 2600 games tested by Mnih al. (2015)
To implement the Atari games, we use OpenAl Gym [Brockman et al. 2016) as an interface to the Arcade Learning Environment (ALE) (Bellemare al. 2013). Our experiment setup is identical to that of Mnih
et al. 2015), except where stated otherwise in Section 5.1
5.1
SPEED TEST
We begin by conducting an ablation study on our fast DQN implementation, measuring the individual con- tribution of each component to the overall speed of the algorithm. We consider 14 algorithmic variants based
7

---

Under review conference paper at ICLR 2022
Table 1:
Measured runtimes for DQN with/without concurrent training and/or synchronized exccution as the number of sampling threads is varied. We report the mean and standard deviation over five trials (units: hours). All variants were evaluated on the same single-GPU system (see Section 5.1
Threads
Standard
Concurrent
Synchronized
Both
1
25.08 + 0.52
20.64 + 0.29
2
19.10 = 0.07
14.00 + 0.00
19.32 = 0.28
14.72 0.13
4
16.84 + 0.09
12.14 + 0.05
15.74 = 0.21
11.08 + 0.08
8
16.92 = 0.23
11.68 + 0.04
14.60 0.12
9.02 0.16
Table 2: Mean runtimes from Table 1 expressed as a percentage of DQN's mean runtime.
Table 3: Mean runtimes from Table expressed as the speedup relative DQN's mean runtime.
Threads
Std.
Conc.
Syne.
Both
Threads
Std.
Cone.
Sync.
Both
100.0%
82.3%
1.00x
1.22x
76.2%
55.8%
77.0%
58.7%
2
1.31x
1.79x
1.30x
1.70x
67.1%
48.4%
62.8%
44.2%
4
1.49x
2.07x
1.59x
2.26
8
67.5%
46.6%
58.2%
36.0%
8
1.48x
2.15x
1.72x
2.78x
on whether Concurrent Training Synchronized Execution is enabled and how many sampler threads are executed (chosen from (1, 2, 83)). For now, we focus solely on wall-clock runtime for these experiments, deferring any performance evaluations until Section 5.2
Our test hardware consists a four-core (eight-thread) Intel Core i7-7700K CPU and an NVIDIA GeForce GTX 1080 GPU. Notably, this particular GPU model is not considered state of the art even for desktop hardware, having been released in 2016. As such, comparable or better hardware should be affordable to a large number of researchers and practitioners.
We choose Pong from the 49 Atari games as our test environment, although the choice of game arbitrary and unlikely impact the timing results. We make that Mnih et al. (2015). First, we train the agents for one million timesteps (instead of 50 million) and then multiply the results by factor of 50. While this may increase the variance of the reported data, makes the experiments much more feasible to implement. Second, we fix é, = 0.1 for the entirety of this shorter training duration. This represents the greediest portion of the e-schedule used by Mnih al. (2015) and, in turn, a worst-case scenario for GPU usage. Furthermore, this value of corresponds to the vast majority (98%) of the schedule, meaning that the results here closely reflect the full experiments.
slight changes to the experiment setup compared
We emphasize that our baseline DQN implementation has been heavily optimized in the interest of fair comparison. All our tested variations share the same core code cedde-iccluding time-critical portions like the deep neural network and replay memory. This means that any observable speedup is due only to Concurrent Training or Synchronized Execution. Our baseline appears to significantly faster than popular existing DQN implementations (e.g. [Roderick al. 2017 Dhariwal et al. 2017 Hill et al. 2018 8) based on their reported results, although we did not compare them on our hardware. In any case, our proposed optimizations are equally applicable all of these DQN agents.
For each hyperparameter combination, we report the mean runtime and standard deviation computed over five independent trials (Table [). (Recall that these values were multiplied by 50.) We also alternatively express each mean as percentage (Table and as a speedup factor (Table with respect to DQN's mean runtime, in order to help reveal trends in the data. We make several interesting observations. First, for any
8

---

Under review conference paper at ICLR 2022
number of sampler threads, enabling concurrency or synchronization is always helpful. Their benefits are synergistic too; enabling both is always faster than enabling, just one. Interestingly, the reverse is not true; with both concurrency and synchronization disabled, increasing the number of sampler threads beyond four fails to further reduce the runtime. We hypothesize that this is due to intense GPU competition between the threads that we discussed in Section 4 Finally, and perhaps most importantly the fastest combination achieved by combining both features and maximizing the number of sampler threads, which almost triples the overall speed. These findings indicate that Concurrent Training and Synchronized Execution work to- gether to enable full utilization of the system resources.
5.2 ATARI 2600 SUITE
We now evaluate the fastest hyperparameter configuration from Section 5.1 on all 49 Atari games, following the experimental procedures from Mnih al. (2015) closely possible. The agents trained for 50 million timesteps (200 million frames) in each game, and are periodically evaluated during training by executing an e-greedy policy (e 0.05) for 30 episodes in separate environment instance. We conduct the evaluation every 250,000 timesteps (Roderick et al. (017) and report the best mean performance attained in each game (see Appendix(A) Note that here we used four computers (each with multiple GPUs) to accelerate results collection, although each agent was still trained on exactly one GPU.
Out of the 49 games, our agent achieved human-level performance-iss -.e. at least 75% numan-normalized score (Mnih et al.) 2015)-in in 33 games. Not only does this constitute a clear majority, but it also exceeds the 29 human-level scores achieved by DQN, Furthermore, our implementation outperformed DQN on 28 of the games again, majority. We attribute the improved performance to the parallel environment instances, which help to diversify experiences via enhanced exploration while reducing correlations between experiences.
Overall, our implementation performed comparably to or better than DQN in 37 games. In the remaining 12 games where performance not quite good, agent converged to relatively stable (but lower) scores roughly halfway through training. We speculate that this is due to the inadequate exploratory capabilities of the e-greedy policy. Indeed, several of these games like Private Eye and Venture are considered to be hard-exploration problems (Ostrovski et al., 2017);; given that the scores for both DQN and our method consisted of only a single trial on each game, it not surprising that the results would exhibit high variance due to random chance. Nevertheless, our strong performance across large majority of the games gives us confidence that our implementation is effective for quickly learning human-leve control policies on the Atari benchmark with modest hardware requirements.
6 CONCLUSION
We presented fast implementation of DQN that is designed to efficiently utilize the resources of CPU- GPU system. When executed on affordable desktop hardware, the implementation reduces the runtime of a full Atari experiment from 25 hours to just 9 hours- nearly triple the speed of our highly optimized DQN baseline. On top of this, the trained agent exceeded both human- and DQN-level performance in a majority of the 49 games tested.
We hope that the impact of our work will be twofold. First, once the code is released publicly, our implemen- tation can function learning. Second, our optimizations described here can inspire analogous adaptations to other important deep reinforcement learning baselines like DDPG (Lillicrap al. (015)) and Rainbow (Hessel et al. 2018) for execution on desktop hardware. Continued efforts in this direction would further reduce computational barriers to research and be of great benefit the community.
baseline test environment for quickly validating ideas in deep reinforcement
9

---

Under review conference paper at ICLR 2022
REFERENCES
Mohammad Babaeizadeh, luri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Ga3c: Gpu-based a3c for deep reinforcement learning. CoRR abs/1611.06256, 2016.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In Interna- tional Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Marc G Bellemaae, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents, Journal of Artificial Intelligence Research, 47:253-279, 2013,
Richard Bellman. Dynamic programming. Science, 153(3731) 3:34333) 1966.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba. OpenAl gym. arXiv preprint arXiv:1606.01540 2016.
Alfredo Clemente, Humberto Castejón, and Arjun Chandra. Efficient parallel methods for deep rein- forcement learning. arXiv preprint arXiv:1705.0482,, 2017.
Brett Daley and Christopher Amato. Reconciling X-returns with experience replay. Advances in Neural Information Processing Systems, 12:1123-1 2019.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines, 2017.
Lasse Espeholt, Hubert Soyer. Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Hor- gan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep rein- forcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Pra- fulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert. Alec Radford. John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. fattps://gitann com hill-a/ Stable-baselines 2018.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.0m 00933, 2018
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
10

---

Under review conference paper at ICLR 2022
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.000 0000000.99..... 2015.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Ma- chine learning, 8(3-4):293-321, 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1928-1937, 2016.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Ve- davyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint 1rXiv:1507.042200 2015.
Hao Yi Ong, Kevin Chavez, and Augustus Hong. 2015.
Distributed deep q-learning.
arXiv preprint
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Martin Riedmiller. Neural fitted q iteration-first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Melrose Roderick, James MacGlashan, and Stefanie Tellex. Implementing the deep q qenetwork. arXivv preprint 17X0v:/711.07475 2017.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604- 609, 2020.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. arXiv preprint arXiv:1803.02811 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2nd edition, 2018.
Gerald Tesauro. Practical issues temporal difference learning. Machine learning. 8(3):257-277, 1992.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, King's 's College, Cambridge, 1989.
11

---

Under review conference paper at ICLR 2022
A ATARI 2600 RESULTS
Table (next page) contains the best scores attained across all of the evaluation periods during train- ing, on each of the 49 Atari games. The learners considered are Random, Human, and DQN (repro- duced from Mnih et al. (2015)) well
fast DQN implementation. We also report the human- normalized performance for DQN and our method. The noraa-normalized ("norm." ") scores are computed 100 (Score- Random) / (Human Random). The higher human- momannnzrmalzzd score between DQN and our implementation emphasized (bold). Parenthetical values represent standard deviation calculated over the 30 evaluation episodes.
HYPERPARAMETERS
Table summarizes the DQN-specific hyperparameters and their values used in all of our experiments. The networks were trained using "centered" RMSProp (Hinton et al. 2012) with a learning rate of 2.5 x 10-4 first- and second-moment decay values of 0.95, and small constant of 0.01 added to the denominator.
These values are identical to those from Mnih et al. (2015). but are restated here for reference.
Table 5: DQN hyperparameters for all experiments.
Hyperparameter
Symbol
Value
minibatch size
replay memory capacity
1000000
target update period
C
10000
training period
F
4
discount factor
0.99 ?
replay memory prepopulation
50000
12

---

Under review conference paper at ICLR 2022
Table 4: Comparison of our fast DQN implementation with DQN on 49 Atari 2600 games.
DQN (norm.)
Ours
Game
Random
Human
DQN
Ours
(norm.)
Alien
227.8 8
6875
3069 (1093)
1319.0 (707.
42.7%
16.4%
Amidar
5.8
1676
739.5 (3024)
1276.7 (210.4)
43.9%
76.1 %
Assault
222.4
1496
3359 (775)
3818.1 (866.8
246.3%
282.3 %
Asterix
210
8503
6012 (1744)
11128.3 (6000.7)
70.0%
131.7% %
Asteroids
719.1
13157
1629 (542)
1475.3 (396.0)
7.3%
6,
Atlantis
12850
29028
85641 (17600)
113926.7 (4267.8)
449.9%
624.8 %
Bank Heist
14.2
734.4
429.7 (650)
668.7 (82.0)
57.7%
90.9%
Battle Zone
2360
37800
26300 (7725)
26300.0 (6309.0)
67.6%
67.6%
Beam Rider
363.9
5775
6846 (1619)
6538.9 (874.4)
119.8%
114.1%
Bowling
23.1
154.8
42.4 (88)
$7.1 (9.5)
14.7%
25.8%
Boxing
0.1
4.3
71.8 (8.4)
96.1 (3.2)
1707.1%
2285.7 %
Breakout
1.7
31.8
401.2 (26.9)
373.4 (95.1)
1327.2%
1234.9%
Centipede
2091
11963
(5237)
5138.2 (2793.3)
63.0%
30.9%
Chopper Command
811
9882
6687 (2916)
1973.3 (955.2)
64.8%
12.8%
Crazy Climber
10781
35411
114103 (22797)
120300.0 (17662.2)
419.5%
444.7%
Demon Attack
152.1
3401
9711 (2406)
10245.0 (1018.5)
294.2%
310.7% %
Double Dunk
-1866
-15.5
-18.1 (2.6)
-15.6 (3.6)
16.1%
96.8% %
Enduro
309.6
301.8 (24.6)
343.6 (30.0)
97.5%
111.0%
Fishing Derby
-9.77
5.5
-0.8 (19.0)
26.9 (10.2)
93.5%
122.0% %
Freeway
0
29.6
30.3 (0.7)
29.5 (1.1)
102.4%
99.7%
Frostbite
65.2
4335
328.3 (250.5)
2606.7 (782.3)
6.2%
59.5%
Gopher
257.6
2321
8520 (3279)
11566.7 7 (3261.3)
400.4%
548.1 %
Gravitar
173
2672
306.7 (223.9)
418.3 (312.0)
5.4%
9.8%
H.F.R.O.
1027
25763
19950 (158)
22900.0 (2671.0)
76.5%
88.4%
Ice Hockey
11.2
0.9
-1.6 (2.5)
-0.3 (1.8)
79.3%
90.1 %
James Bond
29
406.7
576.7 (175.5)
540.0 (124.8)
145.0%
135.3%
Kangaroo
52
3035
6740 (2959)
9720.0 (1897.8)
224.2%
324.1 %
Krull
1598
2395
3805 (1033)
8327.5 (818.0)
276.9%
844,4 %
Kung-Fu Master
258.5
22736
23270 (5955)
33840.0 (5024.7)
102.4%
149.4% %
Montezuma's i Revenge
0
4367
(0)
(0.0)
0.0%
0.0%
Ms. Pacman
307.3
15693
2311 (525)
3105.0 (1100.9)
13.0%
18.2% %
Name This Game
2292
4076
7257 (547)
7428.3 (716.1)
278.3%
287.9 %
Pong
-20.7
9.3
18.9 (1.3)
18.7 (1.4)
132.0% %
131.3%
Private Eye
24.9
69571
1788 (5473)
103.3 (18.3)
2.5%
0.1%
QeBert
163.9
13455
10596 (3294)
12236.7 (1979.1)
78.5%
90.8% %
River Raid
1339
13513
8136 (1049)
13405.0 (2655.5)
57.3%
99.1 %
Road Runner
11.5
7845
18257 (4268)
46873.3 (7020.8)
232.9%
598.2%
Robotank
2.2
11.9
51.6 (4.7)
26.7 (3.7)
509.3%
252.6%
Seaquest
68.4
20182
5286 (1310)
3397.3 (853.1)
25.9%
16.6%
Space Invaders
148
1652
1976 (893)
1124.3 (247.1)
121.5%
64.9%
Star Gunner
664
10250
57997 (3152)
12703.3 (4880.6)
598.1
125.6%
Tennis
-23.8
-8.9
-2.5
(0.3)
143.0%
153.0%
Time Pilot
3568
5925
5947 (1600)
6430.0 (1864.4)
100.9%
121.4% %
Tutankham
11.4
167.6
186.7 (41.9)
151.6 (60.8)
112.2%
89.8%
Up and Down
533.4
9082
(3162)
15653.7 (7689.6)
92.7%
176.9%
Venture
0
1188
380 (238.6)
186.7 (135.8)
32.0%
15.7%
Video Pinball
16257
17298
42684 (16287)
129182.8 (56706.7)
2538.6%
10847,8%
Wizard of Wor
563.5
4757
3393 (2019)
1590.0 (522.8)
67.5%
24.5%
Zaxxon
32.5
9173
4977 (1235)
3406.7 (652.3)
54.1% %
36.9%
13


Reviews:
[{"review_id": "twuLA9u0u2B", "reviewer": "Reviewer_KSo9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "irst and foremost, let me start by saying that I really appreciate the direction that the paper is taking and the problem that it is trying to solve. RL is often not optimized for academia-like hardware requirements, and we need more research and engineering work to figure out how to utilize these kinds of resources better. Kudos to the authors for working on this.\n\nNow, onto the some of the issues that prevent me from recommending the paper for acceptance:\n\n## Novelty\n\nI am a little confused on how we should place this work in the literature. The novelty of the approach seems the clever combination of two frequently used ideas: (1) \"vectorizing\" environment executions so that the env interface can take $A = {a_0, a_1, ..., a_n}$ actions for $E = {e_0, e_1, ..., e_n}$ environments; (2) employing the target network to run inference on CPU whilst training a new set of parameters on default parameters on GPU from experience replay.\n\nPoint (1) however is an implementation details that is pretty much used across the large majority of modern DRL codebases out in the wild (e.g. ACME, stable-baselines, TorchBeast, ...), as in the majority of the cases environments are comparably much cheaper than the rest of the loop to run in parallel. This doesn't seem to be very well reflected in the manuscript (upon first read I was under the impression that this was something else, because I assumed that it was going to be something *beyond* VecEnvs).\n\nPoint (2) technically could be consider novel (AFAIK) when employed to produce this accelerator utilization scheme, but it is also a concept that exists in many existing libraries as an implementation detail. As the current manuscript is lacking in a strong discussion regarding (for instance) how using the target network affects DQN training (say, wrt. different kinds of MDPS -- lava world vs a pendulum optimizer -- or how the target network update frequency might affect the effectiveness of this trick), it feels odd to claim this as a strong contribution of the manuscript.\n\n\n## Clarity of methods\n\nI think the paper could use with a solid paragraph of definitions regarding what the authors mean when they say \"distributed\". One can technically run a distributed agent with extremely good results in local CPU-GPU hardware (see e.g. TorchBeast); it is true that the underlying complexity is greater, but many off-the-shelf frameworks abstract a lot of this away (and frankly speaking completely nail down the RPC/orchestration aspect of this problem -- see e.g. ACME recently). Since the manuscript is lacking a discussion on the tradeoffs that one makes when going that direction -- rather than sticking to a simpler threading / data communication model -- the comparisons with algorithms such as Impala (or R2D2) feel generally handwave-y.\n\nI also found the explanation of the synchronization model to be a little unclear and underwhelming. Considering that these modifications are supposedly simple, I think a little bit of pseudo-python could go a long way towards making the whole methodology section clearer (particularly if other systems are also mentioned in the paper).\n\n\n## Experiments\n\nThe experiments were also a little underwhelming. Firstly, for the speed test to be truly representative of reasoanble in-the-wild workloads, it probably should include comparisons against environments that have different step frequencies, models of wildly different sizes, the addition of recurrence (which is very common now in most RL-trained models), etc.\n\nSecondly, I could not understand the point of the extrapolation done in the Pong experiments. Pong is very cheap to run, especially compared to running the whole Atari suite, so it feels bizarre to stop training early and then *linearly* (!) extrapolate the results. I would really like to understand why this procedure was chosen.\n\nThirdly, I think these sorts of empirical validations (for this particular paper) cannot rely on baselines numbers provided by papers, but rather be based on existing popular (and already well optimized) implementations of DQN. DQN is a relatively old method, and a lot of different public codebases have optimized it greatly with various amounts of tricks. It still remains unclear whether the presented tricks would actually make a significant increase in training speed when used on many of these implementations (I think at least one comparison should be attempted).\n\n-- \n\nPlease do note that I'd be very willing to increase my score provided that we (including the rest of the reviewers) have a productive discussions on how the manuscript can be quickly improved.", "summary_of_the_paper": "The authors propose an implementation of DQN that focuses on maximising the data / training throughput in a common CPU-GPU machine setting. They do so by (asynchronously) running inference on multiple environments at once (and synchronously blocking on the env batch) and using the DQN target network parameters for inference rather than the default -- latest -- parameters of the network. They show considerable improvements in speed while producing comparable results with previous DQN results on the Atari suite.\n", "summary_of_the_review": "The direction of the manuscript is commendable, and the ideas seem simple and cleverly executed. However there remain doubts wrt. novelty and on the validity of the experimental settings. Furthermore, the manuscript could also be significantly improved in terms of clarity.\n\n---\n\nEDIT: bumped score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j6ypjBkLkG", "reviewer": "Reviewer_287J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strength: Nearly three times acceleration of the proposed framework is very attracting that facilitates the complex Atrai games playing with DQN on desktop-level computing resources. \n\nweakness: It is more important to see how the proposed framework can shed light on the implementations of other DRL or even DL models. This important discussion has been largely ignored  in the manuscript. The generalization of the proposed method is highly expected.", "summary_of_the_paper": "This paper introduces a DQN implementation to leverage a novel concurrent and synchronized execution framework for better utilizing a heterogeneous CPU-GPU desktop system. With one NVIDIA GeForce GTX 1080 GPU, the implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. ", "summary_of_the_review": "The paper proposes an acceleration framework for DRL based on synchronized execution of GPU. The performance of the framework is attracting. However, instead of the sole showcase, it is more important to see how the proposed framework can generalize to more DRL tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ebtKF7B2elf", "reviewer": "Reviewer_4Ciy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is written clearly, and presents the main contributions in a straightforward manner. The inclusion of the code for reproducibility is appreciated.\n\nHowever, there are a number of concerns with this paper, which are listed below.\n- The motivation for doing this study is not very convincing; why would researchers need to retrain using DQN (or the variant of DQN introduced here) in Atari? Wouldn't just making the trained models available serve the same task?\n- More generally, it is not convincing that a more efficient re-implementation of DQN (with a change to improve parallelism) is a notable contribution. If this concept could somehow be generalized to other domains, then this would potentially be useful. As it is, there are other algorithms that can achieve much better results than DQN, so it is less meaningful to implement this change. Also, how general are these results, and do they hold on different hardware?\n- What does \"real experience\" refer to in Section 1, paragraph 2? Does this refer to a human playing?\n- Section 1, paragraph 2 refers to GPUs as \"costly\", but this method still uses a GPU.\n- The claimed speedup in terms of wall-clock time is 2.78x. It is not clear how significant this is, especially compared to other DQN implementations.\n- Section 2: since this paper is about a more resource-efficient implementation of DQN, it is reasonable to assume that readers are familiar with DQN, so this discussion could be much more compact.\n- Section 5.1 refers to \"should be affordable\" - it would be useful to have some kind of relative pricing - for instance, what would be the relative cost required to get an analogous speedup just by improving hardware?\n- Section 5.2 refers to \"all 49 Atari games\"; there are actually many more than this, although 57 are commonly studied. There were indeed 49 games in the original DQN paper.\n", "summary_of_the_paper": "This paper proposes a reimplementation of DQN that improves training speed with lower hardware requirements, showing a decrease of running a single Atari experiment from 25 hours to 9 hours on a specific system.", "summary_of_the_review": "The main contribution of this paper is not sufficiently well-motivated or general; the re-implementation of DQN with more parallelism does not pose enough independent interest to be practically useful for most researchers. Suggestions for improvement would be to (1) show that this method could actually extend to settings beyond DQN and improve other RL algorithms; (2) perform a more in-depth hardware study comparing the cost to speedup via more powerful hardware vs. software improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_OCsP1wfId_", "reviewer": "Reviewer_Dq75", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Main Review\n\nThe paper is well-written and well-motivated. The implementation appears to be much faster and uses reasonable methods to achieve this speedup. Having a performant DQN implementation that runs quickly and is publicly accessible would benefit the RL community. I think there are two main issues with the paper: the novelty and the empirical evaluation.\n\n## Novelty\n\nThe first idea that the authors introduce, selecting actions from the target network to enable parallel acting and training, has not been used to speed up training to my knowledge. However, it is similar to the idea of Double Q-learning[1]. Additionally, multiple other researchers have introduced systems that allow for concurrent training and acting [2]. Thus, I would say that the novelty of this idea is limited.\n\nThe second idea, taking multiple actions in parallel, is a standard method as far as I know. I believe it e.g. is used in the stable baselines implementation. See e.g. (https://stable-baselines.readthedocs.io/en/master/guide/examples.html?highlight=vector#recurrent-policies) and (https://github.com/hill-a/stable-baselines/blob/master/stable_baselines/common/vec_env/subproc_vec_env.py#L51). To the best of my knowledge, the proposed method is not novel -- but please clarify how your proposals differ if there's a misunderstanding on my part.\n\n[1] Deep Reinforcement Learning with Double Q-learning, 2016\n\n[2] Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning, 2020\n\n## Empirical Evaluation\n\nThe authors seem to measure speedup of 25->9 hours on their own baseline DQN implementation. It does not seem reasonable to use a baseline you implement yourself when there are plenty of performant DQN implementations available online. Minor implementation details can often have outsized effect to outcome in RL, and the authors might not be incentivized to optimize the baseline as it's easier to improve upon a poor baseline. I would encourage the authors to start from a performant publicly available DQN implementation, e.g. the dopamine repo.\n\nThe authors write that “Our baseline appears to be significantly faster than popular existing DQN implementations (e.g. Roderick et al., 2017; Dhariwal et al., 2017; Hill et al., 2018) based on their reported results, although we did not compare them on our hardware.”. The difference might be entirely due to hardware differences, so this statement does not say much. Additionally, the cited papers are relatively old, and more modern implementations and improved hardware might be available today. \n\nTable 4 contains no standard deviations. Are results just computed over 1 seed per environment? If so, it might not be statistically meaningful.\n\nThe code does not seem to have all dependencies listed (e.g. cv2 is missing), and there's also no example on how to use it. For a paper whose main contribution is on sharing a good implementation, this is not ideal.\n\n## Minor comments\n\nI would recommend that the authors show the learning curves (of e.g. averaged human-normalized scores) in the main paper.\n\nI would encourage the authors to broadener their related work section to other methods that have been proposed for accelerating RL:\n\n[3] Accelerating reinforcement learning through GPU atari emulation, 2019\n\n[4] Large Batch Simulation for Deep Reinforcement Learning, 2021\n\n", "summary_of_the_paper": "DQN is known to consume a lot of resources. This paper introduces an optimized version of DQN which speeds up training 25->9 hours. The authors use two main techniques to improve throughput. Firstly, the authors propose to select actions by computing the argmax over the target network. This decouples acting and training and allows these two operations to run asynchronously. Secondly, the authors compute actions for multiple environments in parallel on the GPU. ", "summary_of_the_review": "The paper aims for a reasonable goal and is both well written and well-motivated. However, the introduced methods are either well-known or incremental variations of known methods. Additionally, the empirical evaluation could be improved by using stronger baselines.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JibvM5-glwh", "reviewer": "Reviewer_pigN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ solves an actual technical/engineering problem that I encountered\n+ reasonably clearly written, easy to read\n+ well executed\n\nCons:\n- changes the algorithms to be benchmarked slightly so it is not really fully equivalent to normal testing (using target network and multithreaded sampler)\n- the speed up is not that impressive, if 25 hours is prohibitive I imagine that 9 might also be\n- the ideas implemented aren't really that innovative or creative, I don't want to be arrogant but these are first things to try and I think I saw similar approaches but in context of synchronous A3C on gpu\n- (not necessarily) the usefulness to the community hinges on the implementation of the framework and no details are mentioned\n\n\nComments/questions:\n- The part where you run first 1mil instead 50mil steps is quite ironic, wasn't it easier to let it run for a bit longer and run it as is?\n-\"The dillema is that exectuion and training are sequentially dependent on each other\" - are they really? Samples are put into replay memory and are randomly sampled every step. Considering that F is usually considerably bigger than the memory size it should not matter if we experience/train in sequence or in parallel. There is much bigger time delay between target and and original network and you explicitly ignore it.\n- Figure 1: \"Synchronize\" step could be more explicit I guess.", "summary_of_the_paper": "A new method for benchmarking RL algorithms like DQN, focusing on Atari suite and experiments from seminal DQN paper. The focus is to achieve speedup, allowing researchers with a modest setup (pc with gpu) to run the tests in reasonable time. The method combines concurrent data collection in multiple threads (and multiple environment instances) and training and achieves a speedup of x2.7 compared to vanilla DQN. At the same time the method slightly changes the nature of the algorithm by different action sampling (multiple env instances which is reminiscent of Asynchronous Methods paper from 2016) and slightly different action selection (using target network). The results are comparable to ones in the original paper overall, but vary on per game basis probably due to mentioned changes, slight change in hyper parameters and high variance of a single run. ", "summary_of_the_review": "The method proposed is not super innovative or creative but it's something that is needed and is well executed. It seems to do what it strived to do (speed-up) and there are no issues with the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a variant of DQN optimized for desktop environments to make large scale experiments more feasible for anyone.

This paper was close. The reviewers appreciated the effort and motivation, but in the end the reviewers all seemed to think that the paper was not ready. The main contribution is framed as making DQN training more feasible, but the reviewers expected the paper to show examples of what the workflow for another architecture would look like and ideally present results for domains beyond Atari. In addition, several reviewers thought the paper could be more precise about (1) ruling out speed differences due to hardware and low-level software, and (2) contextualizing the speedups reported---does 3x matter, what should we expect, etc.

This is certainly an interesting direction. The AC personally thinks that if the authors take some steps to address the points above this will be a great and potentially high impact paper.
Decision: Reject