Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper describes a port of the Atari Learning Environment to CUDA, reports on a set of performance comparison, and provides a bottleneck analysis based communication bandwidth and various throughputs required to saturate them for training and inference.

My first reaction to this paper was, "So what?"; but as I read more, I like the paper more and more.  It was the bottleneck analysis that changed my mind.  It was done very thoroughly and it provides deep insight in the challenges that RL faces for both learning and inference in a variety of settings.  I especially liked the analysis of the advantages and limitations of GPU emulation.  I also thought the Discussion section was well written.

The paper would be better if:
1) The figure fonts were larger throughout the paper.
2) The gaps in Table 1 were explained.

Minor issue:  Change "feed" to "fed" on page 3.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The work contributes a library emulating Atari games in GPU in parallel and allowing to speed-up the execution of reinforcement learning algorithms.

I see that the paper qualifies to the conference; in particular there is listed the topic:

- “implementation issues, parallelization, software platforms, hardware”

However, this is not a research paper, and I do not really see how I should asses it. What I can say about it is that it is considerable amount of work, not only implementing the simulator but also looking at what RL methods need, and how to optimize the allocation and exchange of the data so that everything would work on GPU more efficiently.

From the practical perspective, I am somewhat confused. The speed-up factors in the experiments are rather modest: about 4x for simulating and rendering frames, 2.5x for full RL, on a single GPU. Better with scaling to multi-GPU systems. In Table 1 the total training time per resources used differs dramatically. However if I look at the lines with A2C it is about the same time with 100-200 CPU cores + 1 GPU versus 12 cores + 1 GPU. So this is about factor 10 in the resources, versus CPU parallelization probably suffering overheads.

It appears that the maximum steed-ups are achieved for a particular type of the reinforcement learning algorithms, and using it in a general case would give a modest improvement.

The paper itself consists of introduction, related work, 1 page overview of what it means to simulate the Atari games, and experiments. So it is mostly about measuring the speedups, with several implementations / platforms.

I tend to think that this work will not very much boost the research for new RL methods. It is limited to Atari games, mostly helps to sample-inefficient RL methods and if it helps, the speed-up factors are not of the order that would make experiments by the researchers otherwise impossible. 

I would also give priority to theoretical contributions at ICLR. In the end, we all are using CUDA and cnDNN, but presentations about how they implement things are rather given at GPU computing conferences. 



review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper introduces a CUDA port of the Atari Learning Environment. The paper goes into detail examining the benefits that come from a GPU-only implementation, including much better per-GPU utilization as well as no need to run a distributed system of CPUs. They show this hardware scaling can be taken advantage of across a variety of state of the art reinforcement learning algorithms and indeed create new batching strategies to utilize their framework and the GPU better. 

The paper is well written and goes into some detail describing the implementation of CuLE as well as various design decisions taken, as with splitting the emulation process across several kernels. Finally, the paper is very explicit about a number of optimizations that are not being exploited by the new framework and serve as markers for future work.

A question that arises and which is not addressed in the experiments is how the authors verified their port is faithful to the original version; there is no mention of correctness in the paper.
review_title: Official Blind Review #3

Meta-review: The paper presented a detailed discussion on the implementation of a library emulating Atari games on GPU for efficient reinforcement learning. The analysis is very thoroughly done. The major concern is whether this paper is a good fit to this conference. The developed library would be useful to researchers and the discussion is interesting with respect to system design and implementation, but the technical depth seems not sufficient.
Decision: Reject