Title: Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPU
Authors: Mohammad Babaeizadeh
Iuri Frosio
Stephen Tyree
Jason Clemons
Jan Kautz
Venue: ICLR 2017
Keywords: Reinforcement Learning

Abstract: We introduce a hybrid CPU/GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm, currently the state-of-the-art method in reinforcement learning for various gaming tasks. We analyze its computational traits and concentrate on aspects critical to leveraging the GPU's computational power. We introduce a system of queues and a dynamic scheduling strategy, potentially helpful for other asynchronous algorithms as well. Our hybrid CPU/GPU version of A3C, based on TensorFlow, achieves a significant speed up compared to a CPU implementation; we make it publicly available to other researchers at https://github.com/NVlabs/GA3C.

TL;DR: Implementation and analysis of the computational aspect of a GPU version of the Asynchronous Advantage Actor-Critic (A3C) algorithm

Review discussion:
Is the approach well-balanced in terms of utilization / occupancy of the host device?
This is indeed an interesting question. Since, the environment simulations are running on CPU, CPU usage depends primarily on the simulation. For OpenAI Gym on ATARI, we measured the average CPU occupancy through mpstat for 200s on the three systems considered in the paper for the original A3C DNN and a set of larger networks. The occupancy for the CPU is 32% and 16% on System I, 62% and 49% on System II, and 31% and 17% on System III, for the smallest and largest DNN, respectively. This suggests that the load on the CPU clearly diminishes when using the large network because more time is spent waiting for the GPU between prediction and training updates.
The idle time of the CPU may be useful for either more complex simulations or in some part of prediction/training of the model. The dynamic scheduling of prediction/training tasks between the GPU and CPU goes beyond the scope of this paper, but is a potential future work we would like to investigate. We have updated Section 5. 1 / Table 2 with this utilization information and further comments.
1-If I understood correctly, in GA3C there is only one model, is that correct? If that's the case then it's a significant and important divergence from A3C, it would be great if you could go into more detail and explain in it in section 4.1
2-Do you have some learning curves showing and comparing the instability introduced by the change in the objective function. A comparison between A3C, GA3C with and without epsilon. I would be concerned in some environments like Mujoco which is inherently more unstable than Atari, this might become a problem.
2. We first note that the instability is introduced by computing the derivative of the cost function in Eq. (1) with a delay (as in Eq. (4)), which may generate NaNs in the DNN weights and therefore break the training process, as described in Section 4.4, right before Eq. (5). In this sense, the introduction of \epsilon is needed for the correctness of the algorithm - GA3C cannot be implemented with \epsilon = 0.
Thus we cannot compare versions of GA3C without \epsilon.
In a more general interpretation, \epsilon acts as a regularizer, as it sets a lower bound for log(\pi(a_t) + \epsilon) and therefore a sort of upper bound for the gradient of the cost function in Eq. (5) (recall that d log(x) / dx = 1 / x). Large gradients can generate instabilities and catastrophic forgetting in the learning process, and gradients are therefore explicitly clipped in A3C to avoid problems like these. Introducing \epsilon achieves a similar effect to gradient clipping, and we omit clipping in GA3C.
Comparing the learning curves of GA3C with different values of \epsilon versus A3C (with gradient clipping) is indeed an interesting research question, but one that requires a large number of training runs on different games to make statistically-meaningful conclusions. Unfortunately, given the long training time required by our CPU implementation of A3C (4 days per game), we will be unable to provide the results in time for the review process.
1. This is an excellent observation, and the understanding of the reviewer is mostly correct, with a few caveats.
Unlike A3C, where each agent on a CPU core has its own copy of the DNN model, GA3C keeps a single version of the model on the GPU, akin to A3C’s master mode. In A3C, each agent plays asynchronously, accumulating a gradient update using its local model; after submitting the update to the master model, the agent copies the updated master model to stay in sync with the other agents. Similar to A3C, GA3C has multiple agents playing asynchronously, each using the DNN-defined policy on the GPU. In both cases, agents are operating according to subtly different networks.
The main difference is in the computation and application of gradients. In A3C, each agent has a copy of the model, accumulates a short series of experiences (state/action/reward), calculates a gradient with respect to its local copy using those experiences, and then applies the gradient to the master copy. In GA3C, the agent accumulates a series experiences using several subtly different version of the network, then sends those experiences (not the gradient) to the GPU, where the trainer thread calculates and applies the gradients.
The difference is typically subtle, but occasionally important. In A3C, the experiences and gradients are computed with the same local network parameters. In GA3C, experiences are computed on one or more sets of (slightly old) parameters (those parameters are potentially modified by other agents in the meantime), then the experiences are used to compute gradients using current parameters. In the case that intervening updates have made the probability of an action in the set of experiences nearly zero, there may be numerical issues. We resolve this by adding a small \epsilon>0 to ensure all actions have non-zero probability, when the gradient is computed.
We have added a few sentences in the paper to clarify this point. Thank you for the question.
My main question is about throughput and learning speed. Have you compared the throughput (PPS) of GA3C with DeepMind's original CPU A3C implementation? Their paper does not include explicit FPS numbers but rough estimates can be inferred from Figures 3 and 4 of "Asynchronous Methods for Deep Reinforcement Learning".
For example, A3C with 16 workers reaches 400 points on breakout after roughly 6 hours and 100 million frames, which implies roughly 4600 frames per second or a PPS of roughly 1150. This throughput is about 3 times higher than what you report for your CPU implementation of A3C and is similar to what is reported for GA3C in Table 2.
Similarly, Figure 3 shows GA3C reaching 10-15 points on Pong after 24 hours while Figure 1 of the A3C paper shows it reaching ~18 points in 4 hours.
My other question is about the scores in Table 3. You report numbers with the random starts evaluation while the A3C paper seems to only report numbers with the human starts condition. Are the numbers in Table 3 comparable?
As noted by the reviewer, A3C uses a “human start condition” (described in https://arxiv.org/pdf/1507.04296.pdf), which is not directly reproducible: the starting points have not been shared and some parameters are not specified, including how many games and for how long human player played, from which the 100 starting points are sampled. As an approximation, we use a random initialization (400 frames).
Therefore, it is non-trivial to directly and fairly compare scores to the results reported in DeepMind’s paper.
We would like to emphasize that the results in our scoring table are not intended to demonstrate that we achieve better scores than the original A3C paper. The purpose is instead to demonstrate that, despite subtle implementation changes including the policy lag described in section 4.4, GA3C obtains scores comparable to A3C.
Thank you for the question. Answering this question should help clear up our claims.
We understand the importance of comparing to the A3C implementation described in DeepMind’s paper; unfortunately, this comparison is not simple to achieve. Performance depends heavily on the particular implementation and unfortunately DeepMind’s implementation is not open-source. And we have observed that particular implementation details can significantly alter the overall performance. We used Python and TensorFlow to implement our algorithm, but no details about the implementation language/library are shared in DeepMind’s paper (we believe that a more efficient implementation can be obtained through a low-level programming language like C, but at the cost of flexibility. Since this is still a highly evolving field and for maximum experimental expressiveness, we opt for the flexible approach.)
So, for the sake of an having an apples-to-apples comparison, we compared GA3C to an implementation of A3C in TensorFlow. As the reviewer noted, it is likely our implementation of A3C is not as fast as the one described in DeepMind’s paper, but since it shares the same framework as GA3C, it can be used for an informative evaluation.
However, even in an apple-to-orange comparison, GA3C is still faster although not significantly. We agree with the estimate of the reviewer of roughly 1150 PPS from DeepMind’s paper.
Still the latest implementation of GA3C, reported in Table 2, achieves 1361 PPS that is approximately 1.2x faster than A3C.
But again, we would like to emphasize that this is an “apple-to-orange” comparison because of differences in underneath hardware and software.
Finally, regarding the Pong score in DeepMind’s paper relative to our result, we would caution that Figures 1 in DeepMind’s paper report the average of the best 5 learning rates from a set of 50 training experiments (Figures 3 and 4 report the 3 best learning rates), while our curve was produced from a single training run.
This paper introduces a 'GPU-friendly' variant of A3C which relaxes some synchronicity constraints in the original A3C algorithm to make it more friendly to a high-throughput GPU device. The analysis of the effects of this added latency is thorough. The systems analysis of the algorithm is extensive.
One caveat is that the performance figures in Table 3 are hard to compare since the protocols vary so much. I understand that DeepMind didn't provide reproducible code for A3C, but I gather from the comment that the authors have re-implemented vanilla A3C as well, in which case it would be good to show what this reimplementation of A3C achieves in the same setting used by DM, and in the setting of the experiment conducted using GA3C (1 day). It would be good to clarify in the text that the experimental protocol differed (top 5 out of 50 vs single run), and clarify why the discrepancy, even if the answer is that the authors didn't have time / resources to reproduce the same protocol. A bit more care would go a long way to establishing that indeed, there is no price to pay for the approximations that were made.
I applaud the authors for open-sourcing the code, especially since there is a relative shortage of properly tested open-source implementations in that general area, and getting these algorithms right is non-trivial.
A disclaimer: having never implemented A3C myself, I have a low confidence in my ability to appropriately assess of the algorithmic aspects of the work.
This paper introduce a variant of A3C model where while agents run on multiple cores on CPU the model computations which is the computationally intensive part is passed to the GPU. And they perform various analysis to show the gained speedup.
Thanks the authors for the replying to the questions and adjusting the paper to make it more clear.
It's an interesting modification the the original algorithm. And section 5 does a through analysis of gpu utilization on different configurations.
The main weakness of the paper is lack of more extensive experiments in more atari domains and non atari domains, also multiple plots for multiple runs for observing the instabilities. Stability is a very important issue in RL, and also the most successful algorithms should be able to achieve good results in various domains. I do understand the computational resource limitation, especially in academia if in fact this work was done outside Nvidia.
The paper introduces GA3C, a GPU-based implementation of the A3C algorithm, which was originally designed for multi-core CPUs. The main innovation is the introduction of a system of queues. The queues are used for batching data for prediction and training in order to achieve high GPU occupancy. The system is compared to the authors' own implementation of A3C as well as to published reference scores.
The paper introduces a very natural architecture for implementing A3C on GPUs. Batching requests for predictions and learning steps for multiple actors to maximize GPU occupancy seems like the right thing to do assuming that latency is not an issue. The automatic performance tuning strategy is also really nice to see.
I appreciate the response showing that the throughput of GA3C is 20% higher than what is reported in the original A3C paper. What is still missing is a demonstration that the learning speed/data efficiency is in the right ballpark. Figure 3 of your paper is comparing scores under very different evaluation protocols. These numbers are just not comparable. The most convincing way to show that the learning speed is comparable would be time vs score plots or data vs score plots that show similar or improved speed to A3C.
For example, this open source implementation seems to match the performance on Breakout: https://github.com/muupan/async-rl
One or two plots like that would complete this paper very nicely.
I appreciate the additional experiments included in the revised version of the paper. The learning speed comparison makes the paper more complete and I’m slightly revising my score to reflect that.
Having said that, there is still no clear demonstration that the higher throughput of GA3C leads to consistently faster learning. With the exception of Pong, the training curves in Figure 6 seem to significantly underperform the original A3C results or even the open source implementation of A3C on Breakout and Space Invaders (https://github.com/muupan/async-rl).
We first thank all the reviewers for the interesting comments they provided. All three reviewers agree on the potential importance of the proposed algorithm, but they also suggest that more experimental data are needed to demonstrate that GA3C achieves results in the same ballpark as the original A3C. This turns out to be particularly important in the light of the fact that the policy lag in GA3C can potentially introduce instabilities, as noted by Reviewer #1.
We have included a better description of this in Section 4. 4 of the revised paper, as well as analyzed it in a newly added Section 5. 3, while providing additional experiments to verify the game performance of GA3C and its stability. As pointed out by the reviewers and already highlighted in the paper, establishing a validation protocol that allows comparing the final scores achieved by GA3C with the scores achieved by other training algorithms is complex or even infeasible, as many implementation details are not shared, as well as validation protocols are poorly reproducible. Beyond that, the final score (still reported in Table 3) tells only part of the story, and therefore we espouse the idea shared by Reviewer #1 and #3 of reporting learning curves as a function of the time. The newly added Fig. 6 and 7 show the learning curves for GA3C and several games as a function of the time.
The reader can easily compare these graphs with those reported in Mnih’s paper, as well as in other venues mentioned by the reviewers (e. g. github). The score vs. time curves show a similar dynamic for GA3C and A3C, although it is also important mentioning that curves reported in Mnih’s paper represent the average of the top 5 curves in a set of 50 with various learning rates, which we are unable to match due limitations in available computational resources. Since systems, system loads, environments and hyper-parameters are so widely varying, we also believe that releasing the GA3C code (which we plan to do in the next days) will allow other researchers to systematically evaluate the performances of GA3C and facilitate a fair comparison with new techniques.
Reviewer #1’s main concern is about instabilities introduced in GA3C. The training curves reported in the paper indeed show a few instabilities occurring in the training process, but these represent a common phenomenon in RL. Beyond analyzing in Section 4. 4 how the policy lag in GA3C can generate these instabilities, we now show experimentally that such instabilities do not affect some games (e. g. Pong), while they can invalidate the learning process in other ones (e. g. Breakout), especially in the case of a large learning rate (Fig. 7 and 8).
We found that properly tuning the algorithm hyper-parameters (and in particular the learning rate) dramatically limits the effect of instabilities on the learning process. GA3C would likely benefit from a thorough learning rate search as presented in Mnih’s paper over a set of 50 different learning rates for A3C. Beyond this, we also show how batching on training data (see the newly introduced Section 5. 3, and Fig. 7 and 8 in particular) can be used to limit the effect of these instabilities, at the price of a slower convergence rate.
Specific answers to the individual reviewers follow.
Reviewer 1
We agree with the reviewer that instabilities may affect the learning process. This is indeed a commonly observed phenomenon in RL. Given the fact that GA3C and A3C are indeed (slightly) different algorithms (as now clearly explained in Section 4.4 of the paper), we followed the reviewer’s suggestion and performed a thorough evaluation of the instabilities of the learning process on several games. Experimental results are reported in Section 5.3 and in Fig. 7 and 8 in particular. These show that, for some games (e.g. Pong), the delayed computation of the gradient terms in GA3C (see Section 4.4) does not introduce convergence issues, that are however present in other games (e.g. Breakout).
However, a proper selection of the algorithm hyper-parameters, and of the learning rate in particular, is sufficient to achieve results that are comparable to those achieved by A3C (see Table 3). Quite interestingly, the optimal learning rate for GA3C results to be smaller than the one reported in Mnih’s paper for A3C (as it can be seen in Fig. 6) - this somehow confirms that A3C and GA3C are indeed slightly different algorithms, and GA3C requires careful parameter tuning to obtain optimal results. We also demonstrated that batching of the training data can be used to limit the instabilities, at the price of a slower convergence rate.
Reviewer 2
Following the suggestion of the other two reviewers, and sharing the conviction that replicating the Deep Mind evaluation protocol is non-trivial, we now report in the paper more training curves for several games (Fig. 6) as a function of the training time. This allows the reader to perform a direct comparison between our training curves and the ones reported in Mnih’s paper, independently from the hardware system used for training.
Although, the reader should be careful about this comparison since Mnih’s graphs show top 5 out of 50 runs which is hard to compare with the small number of runs considered in Fig. 6. These curves demonstrate that training time is comparable while achieving a similar training score. The analysis of the theoretical discrepancies between A3C and GA3C (Section 4.4) and its experimental evaluation in terms of instabilities (Section 5.3) should also serve to clarify that, once the hyper-parameters are properly set, GA3C is comparable to A3C in terms of performance.
As for the question about the protocol used for evaluation the score of our trained agents, we now better specify in the caption of table 3 and in the text that we are reporting the average score over 30 different runs for the best agent we trained with GA3C. A direct comparison using the protocol followed in Mnih’s paper is not feasible, as the authors did not share the “human start condition” for each game.
Reviewer 3
We agree with the reviewer that showing score vs. time plot would be more informative than reporting only the final score as in Table 3.
Therefore, we added Fig. 6 to the paper, showing that, with the proper selection of the hyper-parameters, the dynamic of the learning curves of GA3C is similar to that shown in the original A3C paper by Mnih et al. We believe that other, more precise comparisons in terms of learning speed require an optimized implementation as well as a broad hyper-parameter search (in the style of the original A3C plot, showing top 5 results among a set of 50 learners trained with different learning rates), which is out of the scope for this paper. We also decided to keep Table 3 in the paper - although, as explicitly mentioned in the paper, this information is approximate, it gives an idea that GA3C and A3C achieves final score in the same ballpark.
We also suggest that the contribution of our paper is not limited to the implementation of GA3C, but it also includes the analysis of the computational aspects of GA3C and other RL algorithms that can be implemented through a similar architecture. This contribution stands independently from the potential stability issues for GA3C, while the proposed architecture (and the freely available code) can be modified or extended to address these issues, improve the convergence speed, or attack real world problems.
Please note we just updated the latest version of the paper with the latest version of Fig. 5.
This is now coherent with results reported in Figs. 6/7/8, where we achieve higher convergence rate by using a different learning rate.
Please also notice the code has been officially released on Github: https://github.com/NVlabs/GA3C

Meta-review:
role: PC
data: comment: This paper suggests modifications to A3C - mainly by adding a system of queues to batch data - to obtain higher learning throughput on a GPU. The paper gives a solid systems analysis of the algorithm and performance. Although the method does not necessarily lead to higher Atari performance, it is a valuable and relevant contribution to the field, since it provides a stable, fast, open-source implementation of the currently best-performing deep RL algorithm.

Decision: Accept (Poster)