Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a reimplementation of DQN that improves training speed with lower hardware requirements, showing a decrease of running a single Atari experiment from 25 hours to 9 hours on a specific system.
main_review: This paper is written clearly, and presents the main contributions in a straightforward manner. The inclusion of the code for reproducibility is appreciated.

However, there are a number of concerns with this paper, which are listed below.
- The motivation for doing this study is not very convincing; why would researchers need to retrain using DQN (or the variant of DQN introduced here) in Atari? Wouldn't just making the trained models available serve the same task?
- More generally, it is not convincing that a more efficient re-implementation of DQN (with a change to improve parallelism) is a notable contribution. If this concept could somehow be generalized to other domains, then this would potentially be useful. As it is, there are other algorithms that can achieve much better results than DQN, so it is less meaningful to implement this change. Also, how general are these results, and do they hold on different hardware?
- What does "real experience" refer to in Section 1, paragraph 2? Does this refer to a human playing?
- Section 1, paragraph 2 refers to GPUs as "costly", but this method still uses a GPU.
- The claimed speedup in terms of wall-clock time is 2.78x. It is not clear how significant this is, especially compared to other DQN implementations.
- Section 2: since this paper is about a more resource-efficient implementation of DQN, it is reasonable to assume that readers are familiar with DQN, so this discussion could be much more compact.
- Section 5.1 refers to "should be affordable" - it would be useful to have some kind of relative pricing - for instance, what would be the relative cost required to get an analogous speedup just by improving hardware?
- Section 5.2 refers to "all 49 Atari games"; there are actually many more than this, although 57 are commonly studied. There were indeed 49 games in the original DQN paper.

summary_of_the_review: The main contribution of this paper is not sufficiently well-motivated or general; the re-implementation of DQN with more parallelism does not pose enough independent interest to be practically useful for most researchers. Suggestions for improvement would be to (1) show that this method could actually extend to settings beyond DQN and improve other RL algorithms; (2) perform a more in-depth hardware study comparing the cost to speedup via more powerful hardware vs. software improvements.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an implementation of DQN that focuses on maximising the data / training throughput in a common CPU-GPU machine setting. They do so by (asynchronously) running inference on multiple environments at once (and synchronously blocking on the env batch) and using the DQN target network parameters for inference rather than the default -- latest -- parameters of the network. They show considerable improvements in speed while producing comparable results with previous DQN results on the Atari suite.

main_review: irst and foremost, let me start by saying that I really appreciate the direction that the paper is taking and the problem that it is trying to solve. RL is often not optimized for academia-like hardware requirements, and we need more research and engineering work to figure out how to utilize these kinds of resources better. Kudos to the authors for working on this.

Now, onto the some of the issues that prevent me from recommending the paper for acceptance:

## Novelty

I am a little confused on how we should place this work in the literature. The novelty of the approach seems the clever combination of two frequently used ideas: (1) "vectorizing" environment executions so that the env interface can take $A = {a_0, a_1, ..., a_n}$ actions for $E = {e_0, e_1, ..., e_n}$ environments; (2) employing the target network to run inference on CPU whilst training a new set of parameters on default parameters on GPU from experience replay.

Point (1) however is an implementation details that is pretty much used across the large majority of modern DRL codebases out in the wild (e.g. ACME, stable-baselines, TorchBeast, ...), as in the majority of the cases environments are comparably much cheaper than the rest of the loop to run in parallel. This doesn't seem to be very well reflected in the manuscript (upon first read I was under the impression that this was something else, because I assumed that it was going to be something *beyond* VecEnvs).

Point (2) technically could be consider novel (AFAIK) when employed to produce this accelerator utilization scheme, but it is also a concept that exists in many existing libraries as an implementation detail. As the current manuscript is lacking in a strong discussion regarding (for instance) how using the target network affects DQN training (say, wrt. different kinds of MDPS -- lava world vs a pendulum optimizer -- or how the target network update frequency might affect the effectiveness of this trick), it feels odd to claim this as a strong contribution of the manuscript.


## Clarity of methods

I think the paper could use with a solid paragraph of definitions regarding what the authors mean when they say "distributed". One can technically run a distributed agent with extremely good results in local CPU-GPU hardware (see e.g. TorchBeast); it is true that the underlying complexity is greater, but many off-the-shelf frameworks abstract a lot of this away (and frankly speaking completely nail down the RPC/orchestration aspect of this problem -- see e.g. ACME recently). Since the manuscript is lacking a discussion on the tradeoffs that one makes when going that direction -- rather than sticking to a simpler threading / data communication model -- the comparisons with algorithms such as Impala (or R2D2) feel generally handwave-y.

I also found the explanation of the synchronization model to be a little unclear and underwhelming. Considering that these modifications are supposedly simple, I think a little bit of pseudo-python could go a long way towards making the whole methodology section clearer (particularly if other systems are also mentioned in the paper).


## Experiments

The experiments were also a little underwhelming. Firstly, for the speed test to be truly representative of reasoanble in-the-wild workloads, it probably should include comparisons against environments that have different step frequencies, models of wildly different sizes, the addition of recurrence (which is very common now in most RL-trained models), etc.

Secondly, I could not understand the point of the extrapolation done in the Pong experiments. Pong is very cheap to run, especially compared to running the whole Atari suite, so it feels bizarre to stop training early and then *linearly* (!) extrapolate the results. I would really like to understand why this procedure was chosen.

Thirdly, I think these sorts of empirical validations (for this particular paper) cannot rely on baselines numbers provided by papers, but rather be based on existing popular (and already well optimized) implementations of DQN. DQN is a relatively old method, and a lot of different public codebases have optimized it greatly with various amounts of tricks. It still remains unclear whether the presented tricks would actually make a significant increase in training speed when used on many of these implementations (I think at least one comparison should be attempted).

-- 

Please do note that I'd be very willing to increase my score provided that we (including the rest of the reviewers) have a productive discussions on how the manuscript can be quickly improved.
summary_of_the_review: The direction of the manuscript is commendable, and the ideas seem simple and cleverly executed. However there remain doubts wrt. novelty and on the validity of the experimental settings. Furthermore, the manuscript could also be significantly improved in terms of clarity.

---

EDIT: bumped score to 6.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A new method for benchmarking RL algorithms like DQN, focusing on Atari suite and experiments from seminal DQN paper. The focus is to achieve speedup, allowing researchers with a modest setup (pc with gpu) to run the tests in reasonable time. The method combines concurrent data collection in multiple threads (and multiple environment instances) and training and achieves a speedup of x2.7 compared to vanilla DQN. At the same time the method slightly changes the nature of the algorithm by different action sampling (multiple env instances which is reminiscent of Asynchronous Methods paper from 2016) and slightly different action selection (using target network). The results are comparable to ones in the original paper overall, but vary on per game basis probably due to mentioned changes, slight change in hyper parameters and high variance of a single run. 
main_review: Pros:
+ solves an actual technical/engineering problem that I encountered
+ reasonably clearly written, easy to read
+ well executed

Cons:
- changes the algorithms to be benchmarked slightly so it is not really fully equivalent to normal testing (using target network and multithreaded sampler)
- the speed up is not that impressive, if 25 hours is prohibitive I imagine that 9 might also be
- the ideas implemented aren't really that innovative or creative, I don't want to be arrogant but these are first things to try and I think I saw similar approaches but in context of synchronous A3C on gpu
- (not necessarily) the usefulness to the community hinges on the implementation of the framework and no details are mentioned


Comments/questions:
- The part where you run first 1mil instead 50mil steps is quite ironic, wasn't it easier to let it run for a bit longer and run it as is?
-"The dillema is that exectuion and training are sequentially dependent on each other" - are they really? Samples are put into replay memory and are randomly sampled every step. Considering that F is usually considerably bigger than the memory size it should not matter if we experience/train in sequence or in parallel. There is much bigger time delay between target and and original network and you explicitly ignore it.
- Figure 1: "Synchronize" step could be more explicit I guess.
summary_of_the_review: The method proposed is not super innovative or creative but it's something that is needed and is well executed. It seems to do what it strived to do (speed-up) and there are no issues with the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a DQN implementation to leverage a novel concurrent and synchronized execution framework for better utilizing a heterogeneous CPU-GPU desktop system. With one NVIDIA GeForce GTX 1080 GPU, the implementation reduces the training time of a 200-million-frame Atari experiment from 25 hours to just 9 hours. 
main_review: strength: Nearly three times acceleration of the proposed framework is very attracting that facilitates the complex Atrai games playing with DQN on desktop-level computing resources. 

weakness: It is more important to see how the proposed framework can shed light on the implementations of other DRL or even DL models. This important discussion has been largely ignored  in the manuscript. The generalization of the proposed method is highly expected.
summary_of_the_review: The paper proposes an acceleration framework for DRL based on synchronized execution of GPU. The performance of the framework is attracting. However, instead of the sole showcase, it is more important to see how the proposed framework can generalize to more DRL tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: DQN is known to consume a lot of resources. This paper introduces an optimized version of DQN which speeds up training 25->9 hours. The authors use two main techniques to improve throughput. Firstly, the authors propose to select actions by computing the argmax over the target network. This decouples acting and training and allows these two operations to run asynchronously. Secondly, the authors compute actions for multiple environments in parallel on the GPU. 
main_review: # Main Review

The paper is well-written and well-motivated. The implementation appears to be much faster and uses reasonable methods to achieve this speedup. Having a performant DQN implementation that runs quickly and is publicly accessible would benefit the RL community. I think there are two main issues with the paper: the novelty and the empirical evaluation.

## Novelty

The first idea that the authors introduce, selecting actions from the target network to enable parallel acting and training, has not been used to speed up training to my knowledge. However, it is similar to the idea of Double Q-learning[1]. Additionally, multiple other researchers have introduced systems that allow for concurrent training and acting [2]. Thus, I would say that the novelty of this idea is limited.

The second idea, taking multiple actions in parallel, is a standard method as far as I know. I believe it e.g. is used in the stable baselines implementation. See e.g. (https://stable-baselines.readthedocs.io/en/master/guide/examples.html?highlight=vector#recurrent-policies) and (https://github.com/hill-a/stable-baselines/blob/master/stable_baselines/common/vec_env/subproc_vec_env.py#L51). To the best of my knowledge, the proposed method is not novel -- but please clarify how your proposals differ if there's a misunderstanding on my part.

[1] Deep Reinforcement Learning with Double Q-learning, 2016

[2] Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning, 2020

## Empirical Evaluation

The authors seem to measure speedup of 25->9 hours on their own baseline DQN implementation. It does not seem reasonable to use a baseline you implement yourself when there are plenty of performant DQN implementations available online. Minor implementation details can often have outsized effect to outcome in RL, and the authors might not be incentivized to optimize the baseline as it's easier to improve upon a poor baseline. I would encourage the authors to start from a performant publicly available DQN implementation, e.g. the dopamine repo.

The authors write that “Our baseline appears to be significantly faster than popular existing DQN implementations (e.g. Roderick et al., 2017; Dhariwal et al., 2017; Hill et al., 2018) based on their reported results, although we did not compare them on our hardware.”. The difference might be entirely due to hardware differences, so this statement does not say much. Additionally, the cited papers are relatively old, and more modern implementations and improved hardware might be available today. 

Table 4 contains no standard deviations. Are results just computed over 1 seed per environment? If so, it might not be statistically meaningful.

The code does not seem to have all dependencies listed (e.g. cv2 is missing), and there's also no example on how to use it. For a paper whose main contribution is on sharing a good implementation, this is not ideal.

## Minor comments

I would recommend that the authors show the learning curves (of e.g. averaged human-normalized scores) in the main paper.

I would encourage the authors to broadener their related work section to other methods that have been proposed for accelerating RL:

[3] Accelerating reinforcement learning through GPU atari emulation, 2019

[4] Large Batch Simulation for Deep Reinforcement Learning, 2021


summary_of_the_review: The paper aims for a reasonable goal and is both well written and well-motivated. However, the introduced methods are either well-known or incremental variations of known methods. Additionally, the empirical evaluation could be improved by using stronger baselines.
review_title: null

Meta-review: This paper introduces a variant of DQN optimized for desktop environments to make large scale experiments more feasible for anyone.

This paper was close. The reviewers appreciated the effort and motivation, but in the end the reviewers all seemed to think that the paper was not ready. The main contribution is framed as making DQN training more feasible, but the reviewers expected the paper to show examples of what the workflow for another architecture would look like and ideally present results for domains beyond Atari. In addition, several reviewers thought the paper could be more precise about (1) ruling out speed differences due to hardware and low-level software, and (2) contextualizing the speedups reported---does 3x matter, what should we expect, etc.

This is certainly an interesting direction. The AC personally thinks that if the authors take some steps to address the points above this will be a great and potentially high impact paper.
Decision: Reject