Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Is the approach well-balanced in terms of utilization / occupancy of the host device?
review_title: Any comment on the CPU utilization?

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a 'GPU-friendly' variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive. 

One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn't provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn't have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made.

I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial.

A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.
review_title: Interesting work, well motivated.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces GA3C, a GPU-based implementation of the A3C algorithm, which was originally designed for multi-core CPUs. The main innovation is the introduction of a system of queues. The queues are used for batching data for prediction and training in order to achieve high GPU occupancy. The system is compared to the authors' own implementation of A3C as well as to published reference scores.

The paper introduces a very natural architecture for implementing A3C on GPUs. Batching requests for predictions and learning steps for multiple actors to maximize GPU occupancy seems like the right thing to do assuming that latency is not an issue. The automatic performance tuning strategy is also really nice to see. 

I appreciate the response showing that the throughput of GA3C is 20% higher than what is reported in the original A3C paper. What is still missing is a demonstration that the learning speed/data efficiency is in the right ballpark. Figure 3 of your paper is comparing scores under very different evaluation protocols. These numbers are just not comparable. The most convincing way to show that the learning speed is comparable would be time vs score plots or data vs score plots that show similar or improved speed to A3C. For example, this open source implementation seems to match the performance on Breakout: https://github.com/muupan/async-rl
One or two plots like that would complete this paper very nicely.

-----------------------------------

I appreciate the additional experiments included in the revised version of the paper. The learning speed comparison makes the paper more complete and I’m slightly revising my score to reflect that. Having said that, there is still no clear demonstration that the higher throughput of GA3C leads to consistently faster learning. With the exception of Pong, the training curves in Figure 6 seem to significantly underperform the original A3C results or even the open source implementation of A3C on Breakout and Space Invaders (https://github.com/muupan/async-rl).
review_title: Final Review

final_score: rating: null
review_content: 1-If I understood correctly, in GA3C there is only one model, is that correct? If that's the case then it's a significant and important divergence from A3C, it would be great if you could go into more detail and explain in it in section 4.1

2-Do you have some learning curves showing and comparing the instability introduced by the change in the objective function. A comparison between A3C, GA3C with and without epsilon. I would be concerned in some environments like Mujoco which is inherently more unstable than Atari, this might become a problem.

review_title: 1-Model explanation 2-Instability plots

final_score: rating: null
review_content: My main question is about throughput and learning speed. Have you compared the throughput (PPS) of GA3C with DeepMind's original CPU A3C implementation? Their paper does not include explicit FPS numbers but rough estimates can be inferred from Figures 3 and 4 of "Asynchronous Methods for Deep Reinforcement Learning". For example, A3C with 16 workers reaches 400 points on breakout after roughly 6 hours and 100 million frames, which implies roughly 4600 frames per second or a PPS of roughly 1150. This throughput is about 3 times higher than what you report for your CPU implementation of A3C and is similar to what is reported for GA3C in Table 2. Similarly, Figure 3 shows GA3C reaching 10-15 points on Pong after 24 hours while Figure 1 of the A3C paper shows it reaching ~18 points in 4 hours.

My other question is about the scores in Table 3. You report numbers with the random starts evaluation while the A3C paper seems to only report numbers with the human starts condition. Are the numbers in Table 3 comparable?
review_title: Pre-review questions

final_score: rating: 7: Good paper, accept
review_content: This paper introduce a variant of A3C model where while agents run on multiple cores on CPU the model computations which is the computationally intensive part is passed to the GPU. And they perform various analysis to show the gained speedup.

Thanks the authors for the replying to the questions and adjusting the paper to make it more clear.
It's an interesting modification the the original algorithm. And section 5 does a through analysis of gpu utilization on different configurations.
The main weakness of the paper is lack of more extensive experiments in more atari domains and non atari domains, also multiple plots for multiple runs for observing the instabilities. Stability is a very important issue in RL, and also the most successful algorithms should be able to achieve good results in various domains. I do understand the computational resource limitation, especially in academia if in fact this work was done outside Nvidia.
review_title: Final Review

Meta-review: This paper suggests modifications to A3C - mainly by adding a system of queues to batch data - to obtain higher learning throughput on a GPU. The paper gives a solid systems analysis of the algorithm and performance. Although the method does not necessarily lead to higher Atari performance, it is a valuable and relevant contribution to the field, since it provides a stable, fast, open-source implementation of the currently best-performing deep RL algorithm.
Decision: Accept (Poster)