Title: DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY

Abstract: We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.
In this paper we extend prioritized experience replay to the distributed setting and show that this is a highly scalable approach to deep reinforcement learning. We introduce a few key modifications that enable this scalability, and we refer to our approach as Ape-X.

1. INTRODUCTION:

A broad trend in deep learning is that combining more computation (Dean et al., 2012) with more powerful models (Kaiser et al., 2017) and larger datasets (Deng et al., 2009) yields more impressive results. It is reasonable to hope that a similar principle holds for deep reinforcement learning. There are a growing number of examples to justify this optimism: effective use of greater computational resources has been a critical factor in the success of such algorithms as Gorila (Nair et al., 2015) , A3C (Mnih et al., 2016) , GPU Advantage Actor Critic (Babaeizadeh et al., 2017) , Distributed PPO (Heess et al., 2017) and AlphaGo (Silver et al., 2016) .

Deep learning frameworks such as TensorFlow (Abadi et al., 2016) support distributed training, making large scale machine learning systems easier to implement and deploy. Despite this, much current research in deep reinforcement learning concerns itself with improving performance within the computational budget of a single machine, and the question of how to best harness more resources is comparatively underexplored.

In this paper we describe an approach to scaling up deep reinforcement learning by generating more data and selecting from it in a prioritized fashion (Schaul et al., 2016) . Standard approaches to distributed training of neural networks focus on parallelizing the computation of gradients, to more rapidly optimize the parameters (Dean et al., 2012) . In contrast, we distribute the generation and selection of experience data, and find that this alone suffices to improve results. This is complementary to distributing gradient computation, and the two approaches can be combined, but in this work we focus purely on data-generation.

We use this distributed architecture to scale up variants of Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG), and we evaluate these on the Arcade Learning Environment benchmark (Bellemare et al., 2013) , and on a range of continuous control tasks. Our architecture achieves a new state of the art performance on Atari games, using a fraction of the wall-clock time compared to the previous state of the art, and without per-game hyperparameter tuning.

We empirically investigate the scalability of our framework, analysing how prioritization affects performance as we increase the number of data-generating workers. Our experiments include an analysis of factors such as the replay capacity, the recency of the experience, and the use of different data-generating policies for different workers. Finally, we discuss implications for deep reinforcement learning agents that may apply beyond our distributed framework.

2. BACKGROUND:

Distributed Stochastic Gradient Descent Distributed stochastic gradient descent is widely used in supervised learning to speed up training of deep neural networks, by parallelizing the computation of the gradients used to update their parameters. The resulting parameter updates may be applied synchronously (Krizhevsky, 2014) or asynchronously (Dean et al., 2012) . Both approaches have proven effective and are an increasingly standard part of the deep learning toolbox. Inspired by this, Nair et al. (2015) applied distributed asynchronous parameter updates and distributed data generation to deep reinforcement learning. Asynchronous parameter updates and parallel data generation have also been successfully used within a single-machine, in a multi-threaded rather than a distributed context (Mnih et al., 2016) . GPU Asynchronous Actor-Critic (GA3C; Babaeizadeh et al., 2017) and Parallel Advantage Actor-Critic (PAAC; Clemente et al., 2017) adapt this approach to make efficient use of GPUs.

Distributed Importance Sampling A complementary family of techniques for speeding up training is based on variance reduction by means of importance sampling (cf. Hastings, 1970) . This has been shown to be useful in the context of neural networks (Hinton, 2007) . Sampling non-uniformly from a dataset and weighting updates according to the sampling probability in order to counteract the bias thereby introduced can increase the speed of convergence by reducing the variance of the gradients. One way of doing this is to select samples with probability proportional to the L 2 norm of the corresponding gradients. In supervised learning, this approach has been successfully extended to the distributed setting (Alain et al., 2015) . An alternative is to rank samples according to their latest known loss value and make the sampling probability a function of the rank rather than of the loss itself (Loshchilov & Hutter, 2015) .

Prioritized Experience Replay Experience replay (Lin, 1992) has long been used in reinforcement learning to improve data efficiency. It is particularly useful when training neural network function approximators with stochastic gradient descent algorithms, as in Neural Fitted Q-Iteration (Riedmiller, 2005) and Deep Q-Learning (Mnih et al., 2015) . Experience replay may also help to prevent overfitting by allowing the agent to learn from data generated by previous versions of the policy. Prioritized experience replay (Schaul et al., 2016) extends classic prioritized sweeping ideas (Moore & Atkeson, 1993) to work with deep neural network function approximators. The approach is strongly related to the importance sampling techniques discussed in the previous section, but using a more general class of biased sampling procedures that focus learning on the most 'surprising' experiences. Biased sampling can be particularly helpful in reinforcement learning, since the reward signal may be sparse and the data distribution depends on the agent's policy. As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016) , UNREAL (Jaderberg et al., 2017) , DQfD (Hester et al., 2017) , and Rainbow (Hessel et al., 2017) . In an ablation study conducted to investigate the relative importance of several algorithmic ingredients (Hessel et al., 2017) , prioritization was found to be the most important ingredient contributing to the agent's performance. for t = 1 to T do 5:

Learner:



Network:



Replay:



Experiences:

at-1 ← π θ t-1 (st-1) Select an action using the current policy. 6:

(rt, γt, st) ← ENVIRONMENT.STEP(at-1) Apply the action in the environment. 7:

LOCALBUFFER.ADD((st-1, at-1, rt, γt)) As in Gorila (Nair et al., 2015) , we decompose the standard deep reinforcement learning algorithm into two parts, which run concurrently with no high-level synchronization. The first part consists of stepping through an environment, evaluating a policy implemented as a deep neural network, and storing the observed data in a replay memory. We refer to this as acting. The second part consists of sampling batches of data from the memory to update the policy parameters. We term this learning.

In principle, both acting and learning may be distributed across multiple workers. In our experiments, hundreds of actors run on CPUs to generate data, and a single learner running on a GPU samples the most useful experiences (Figure 1 ). Pseudocode for the actors and learners is shown in Algorithms 1 and 2. Updated network parameters are periodically communicated to the actors from the learner.

In contrast to Nair et al. (2015) , we use a shared, centralized replay memory, and instead of sampling uniformly, we prioritize, to sample the most useful data more often. Since priorities are shared, high priority data discovered by any actor can benefit the whole system. Priorities can be defined in various ways, depending on the learning algorithm; two instances are described in the next sections.

In Prioritized DQN (Schaul et al., 2016) priorities for new transitions were initialized to the maximum priority seen so far, and only updated once they were sampled. This does not scale well: due to the large number of actors in our architecture, waiting for the learner to update priorities would result in a myopic focus on the most recent data, which has maximum priority by construction. Instead, we take advantage of the computation the actors in Ape-X are already doing to evaluate their local copies of the policy, by making them also compute suitable priorities for new transitions online. This ensures that data entering the replay has more accurate priorities, at no extra cost.

Sharing experiences has certain advantages compared to sharing gradients. Low latency communication is not as important as in distributed SGD, because experience data becomes outdated less rapidly than gradients, provided the learning algorithm is robust to off-policy data. Across the system, we take advantage of this by batching all communications with the centralized replay, increasing the efficiency and throughput at the cost of some latency. With this approach it is even possible for actors and learners to run in different data-centers without limiting performance.

Finally, by learning off-policy (cf. Sutton & Barto, 1998; 2017) , we can further take advantage of Ape-X's ability to combine data from many distributed actors, by giving the different actors different exploration policies, broadening the diversity of the experience they jointly encounter. As we will see in the results, this can be sufficient to make progress on difficult exploration problems.

3.1. APE-X DQN:

The general framework we have described may be combined with different learning algorithms. First, we combined it with a variant of DQN (Mnih et al., 2015) with some of the components of Rainbow (Hessel et al., 2017) . More specifically, we used double Q-learning (van Hasselt, 2010; van Hasselt et al., 2016) with multi-step bootstrap targets (cf. Sutton, 1988; Sutton & Barto, 1998; 2017; Mnih et al., 2016) as the learning algorithm, and a dueling network architecture (Wang et al., 2016) as the function approximator q(•, •, θ).

This results in computing for all elements in the batch the loss l t (θ) = 1 2 (G tq(S t , A t , θ)) 2 with

G t = R t+1 + γR t+2 + . . . + γ n-1 R t+n + γ n double-Q bootstrap value q(S t+n , argmax a q(S t+n , a, θ), θ -) multi-step return ,

where t is a time index for an experience sampled from the replay starting with state S t and action A t , and θ -denotes parameters of the target network (Mnih et al., 2015) , a slow moving copy of the online parameters. Multi-step returns are truncated if the episode ends in fewer than n steps.

In principle, Q-learning variants are off-policy methods, so we are free to choose the policies we use to generate data. However, in practice, the choice of behaviour policy does affect both exploration and the quality of function approximation. Furthermore, we are using a multi-step return with no off-policy correction, which in theory could adversely affect the value estimation. Nonetheless, in Ape-X DQN, each actor executes a different policy, and this allows experience to be generated from a variety of strategies, relying on the prioritization mechanism to pick out the most effective experiences. In our experiments, the actors use -greedy policies with different values of . Low policies allow exploring deeper in the environment, while high policies prevent over-specialization.

3.2. APE-X DPG:

To test the generality of the framework we also combined it with a continuous-action policy gradient system based on DDPG (Lillicrap et al., 2016) , an implementation of deterministic policy gradients Silver et al. (2014) also similar to older methods (Werbos, 1990; Prokhorov & Wunsch, 1997) , and tested it on continuous control tasks from the DeepMind Control Suite (Tassa et al., 2018) . The Ape-X DPG setup is similar to Ape-X DQN, but the actor's policy is now represented explicitly by a separate policy network, in addition to the Q-network. The two networks are optimized separately, by minimizing different losses on the sampled experience. We denote the policy and Q-network parameters by φ and ψ respectively, and adopt the same convention as above to denote target networks. The Q-network outputs an action-value estimate q(s, a, ψ) for a given state s, and multi-dimensional action a ∈ R m . It is updated using temporal-difference learning with a multi-step bootstrap target. The Q-network loss can be written as l t (ψ) = 1 2 (G tq(S t , A t , ψ)) 2 , where

G t = R t+1 + γR t+2 + . . . + γ n-1 R t+n + γ n q(S t+n , π(S t+n , φ -), ψ -) multi-step return .

The policy network outputs an action A t = π(S t , φ) ∈ R m . The policy parameters are updated using policy gradient ascent on the estimated Q-value, using gradient ∇ φ q(S t , π(S t , φ), ψ) -note that this depends on the policy parameters φ only through the action A t = π(S t , φ) that is input to the critic network. Further details of the Ape-X DPG algorithm are available in the appendix.

4. EXPERIMENTS:



4.1. ATARI:

In our first set of experiments we evaluate Ape-X DQN on Atari, and show state of the art results on this standard reinforcement learning benchmark. We use 360 actor machines (each using one CPU core) to feed data into the replay memory as fast as they can generate it; approximately 139 frames per second (FPS) each, for a total of ∼50K FPS, which corresponds to ∼12.5K transitions (because of a fixed action repeat of 4). The actors batch experience data locally before sending it to the replay: up to 100 transitions may be buffered at a time, which are then sent asynchronously in batches of B = 50. The learner asynchronously prefetches up to 16 batches of 512 transitions, and computes updates for 19 such batches each second, meaning that gradients are computed for ∼9.7K transitions per second on average. To reduce memory and bandwidth requirements, observation data is compressed using a PNG codec when sent and when stored in the replay. The learner decompresses data as it prefetches it, in parallel with computing and applying gradients. The learner also asynchronously handles any requests for parameters from actors. Actors copy the network parameters from the learner every 400 frames (∼2.8 seconds). Each actor i ∈ {0, ..., N -1} executes an i -greedy policy where i = 1+ i N -1 α with = 0.4, α = 7. Each i is held constant throughout training. The episode length is limited to 50000 frames during training.

The capacity of the shared experience replay memory is soft-limited to 2 million transitions: adding new data is always permitted, to not slow down the actors, but every 100 learning steps any excess data above this capacity threshold is removed en masse, in FIFO order. The median actual size of the memory is 2035050. Data is sampled according to proportional prioritization, with a priority exponent of 0.6 and an importance sampling exponent set to 0.4.

In Figure 2 , on the left, we compare the median human normalized score across all 57 games to several baselines: DQN, Prioritized DQN, Distributional DQN (Bellemare et al., 2017) , Rainbow, and Gorila. In all cases the performance is measured at the end of training under the no-op starts testing regime (Mnih et al., 2015) . On the right, we show initial learning curves (taken from the greediest actor) for a selection of 6 games (full learning curves for all games are in the appendix). Given that Ape-X can harness substantially more computation than most baselines, one might expect it to train faster. Figure 2 shows that this was indeed the case. Perhaps more surprisingly, our agent achieved a substantially higher final performance.

In Table 1 we compare the median human-normalized performance of Ape-X DQN on the Atari benchmark to corresponding metrics as reported for other baseline agents in their respective publications. Whenever available we report results both for no-op starts and for human starts. The human-starts regime (Nair et al., 2015) corresponds to a more challenging generalization test, as the agent is initialized from random starts drawn from games played by human experts. Ape-X's performance is higher than the performance of any of the baselines according to both metrics.

4.2. CONTINUOUS CONTROL:

In a second set of experiments we evaluated Ape-X DPG on four continuous control tasks. In the manipulator domain the agent must learn to bring a ball to a specified location. In the humanoid domain the agent must learn to control a humanoid body to solve three distinct tasks of increasing complexity: Standing, Walking and Running. Since here we learn from features, rather than from pixels, the observation space is much smaller than it is in the Atari domain. We therefore use small, fully-connected networks (details in the appendix). With 64 actors on this domain, we obtain ∼14K total FPS (the same number of transitions per second; here we do not use action repeats). We process 86 batches of 256 transitions per second, or ∼22K transitions processed per second.

Figure 3 shows that Ape-X DPG achieved very good performance on all four tasks. The figure shows the performance of Ape-X DPG for different numbers of actors: as the number of actors increases our agent becomes increasingly effective at solving these problems rapidly and reliably, outperforming a standard DDPG baseline trained for over 10 times longer. A parallel paper (Barth-Maron et al., 2018) builds on this work by combining Ape-X DPG with distributional value functions, and the resulting algorithm is successfully applied to further continuous control tasks. 

5. ANALYSIS:

In this section we describe additional Ape-X DQN experiments on Atari that helped improve our understanding of the framework, and we investigate the contribution of different components.

First, we investigated how the performance scales with the number of actors. We trained our agent with different numbers of actors (8, 16, 32, 64, 128 and 256) for 35 hours on a subset of 6 Atari games. In all experiments we kept the size of the shared experience replay memory fixed at 1 million transitions. Figure 4 shows that the performance consistently improved as the number of actors increased. The appendix contains learning curves for additional games, and a comparison of the scalability of the algorithm with and without prioritized replay. It is perhaps surprising that performance improved so substantially purely by increasing the number of actors, without changing the rate at which the network parameters are updated, the structure of the network, or the update rule.

We hypothesize that the proposed architecture helps with a common deep reinforcement learning failure mode, in which the policy discovered is a local optimum in the parameter space, but not a global one, e.g., due to insufficient exploration. Using a large number of actors with varying amounts of exploration helps to discover promising new courses of action, and prioritized replay ensures that when this happens, the learning algorithm focuses its efforts on this important information.

Next, we investigated varying the capacity of the replay memory (see Figure 5 ). We used a setup with 256 actors, for a median of ∼37K total environment frames per second (approximately ∼9K transitions). With such a large number of actors, the contents of the memory is replaced much faster than in most DQN-like agents. We observed a small benefit to using a larger replay capacity. We hypothesize this is due to the value of keeping some high priority experiences around for longer and replaying them. As above, a single learner machine trained the network with median 19 batches per second, each of 512 transitions, for a median of ∼9.7K transitions processed per second.

Finally, we ran additional experiments to disentangle potential effects of two confounding factors in our scalability analysis: recency of the experience data in the replay memory, and diversity of the data-generating policies. The full description of these experiments is confined to the appendix; to summarize, neither factor alone is sufficient to explain the performance we see. We therefore conclude that the results are due substantially to the positive effects of gathering more experience data; namely better exploration of the environment and better avoidance of overfitting.

6. CONCLUSION:

We have designed, implemented, and analyzed a distributed framework for prioritized replay in deep reinforcement learning. This architecture achieved state of the art results in a wide range of discrete and continuous tasks, both in terms of wall-clock learning speed and final performance.

In this paper we focused on applying the Ape-X framework to DQN and DPG, but it could also be combined with any other off-policy reinforcement learning update. For methods that use temporally extended sequences (e.g., Mnih et al., 2016; Wang et al., 2017) , the Ape-X framework may be adapted to prioritize sequences of past experiences instead of individual transitions.

Ape-X is designed for regimes in which it is possible to generate large quantities of data in parallel. This includes simulated environments but also a variety of real-world applications, such as robotic arm farms, self-driving cars, online recommender systems, or other multi-user systems in which data is generated by many instances of the same environment (c.f. Silver et al., 2013) . In applications where data is costly to obtain, our approach will not be directly applicable. With powerful function approximators, overfitting is an issue: generating more training data is the simplest way of addressing it, but may also provide guidance towards data-efficient solutions.

Many deep reinforcement learning algorithms are fundamentally limited by their ability to explore effectively in large domains. Ape-X uses a naive yet effective mechanism to address this issue: generating a diverse set of experiences and then identifying and learning from the most useful events. The success of this approach suggests that simple and direct approaches to exploration may be feasible, even for synchronous agents.

Our architecture illustrates that distributed systems are now practical both for research and, potentially, large-scale applications of deep reinforcement learning. We hope that the algorithms, architecture, and analysis we have presented will help to accelerate future efforts in this direction. fixed set of 6 values for . Blue: full range of values for . In both cases, the curve plotted is from a separate actor that does not add data to the replay memory, and which follows an -greedy policy with = 0.00164.

A RECENCY OF EXPERIENCE:

In our main experiments we do not change the size of the replay memory in proportion to the number of actors, so by changing the number of actors we also increased the rate at which the contents of the replay memory is replaced. This means that in the experiments with more actors, transitions in the replay memory are more recent: they are generated by following policies whose parameters are closer to version of the parameters being optimized by the learner, and in this sense they are more onpolicy. Could this alone be sufficient to explain the improved performance? If so, we might be able to recover the results without needing a large number of actor machines. To test this, we constructed an experiment wherein we replicate the rate at which the contents of the replay memory is replaced in the 256-actor experiments, but instead of actually using 256 actors, we use 32 actors but add each transition they generate to the replay memory 8 times over. In this setup, the contents of the replay memory is similarly generated by policies with a recent version of the network parameters: the only difference is that the data is not as diverse as in the 256-actor case. We observe (see Figure 6 ) that this does not recover the same performance, and therefore conclude that the recency of the experience alone is not sufficient to explain the performance of our method. Indeed, we see that adding the same data multiple times can sometimes harm performance, since although it increases recency this comes at the expense of diversity.

Note: in principle, duplicating the added data in this fashion has a similar effect to reducing the capacity of the replay memory, and indeed, our results with a smaller replay memory in Figure 5 do corroborate the finding. However, we test also by duplicating the data primarily in order to exclude any effects arising from the implementation. In particular, in contrast to simply reducing the replay capacity, duplicating each data point means that the computational demands on the replay server in these runs are the same as when we use the corresponding number of real actors.

B VARYING THE DATA-GENERATING POLICIES:

Another factor that could conceivably contribute to the scalability of our algorithm is the fact that each actor has a different . To determine the extent to which this impacts upon the performance, we ran an experiment (see Figure 7 ) with some simple variations on the mechanism we use to choose the policies that generate the data we train on. The first alternative we tested is to choose a small fixed set of 6 values for , instead of the full range that we typically use. In this test, we use prioritized replay as normal, and we find that the results with the full range of are overall slightly better. However, it is not essential for achieving good results within our distributed framework.

C ATARI: ADDITIONAL DETAILS:

The frames received from the environment are preprocessed on the actor side with the standard transformations introduced by DQN. This includes greyscaling, frame stacking, repeating actions 4 times, and clipping rewards to [-1, 1].

The learner waits for at least 50000 transitions to be accumulated in the replay before starting learning. We use a Centered RMSProp optimizer with a learning rate of 0.00025 / 4, decay of 0.95, epsilon of 1.5e-7, and no momentum to minimize the multi-step loss (with n = 3). Gradient norms are clipped to 40. The target network used in the loss calculation is copied from the online network every 2500 training batches. We use the same network as in the Dueling DDQN agent.

D CONTINUOUS CONTROL: ADDITIONAL DETAILS:

The critic network has a layer with 400 units, followed by a tanh activation, followed by another layer of 300 units. The actor network has a layer with 300 units, followed by a tanh activation, followed by another layer of 200 units. The gradient used to update the actor network is clipped to [-1, 1], element-wise. Training uses the Adam optimizer (Kingma & Ba (2014) ) with learning rate of 0.0001. The target network used in the loss calculation is copied from the online network every 100 training batches.

Replay sampling priorities are set according to the absolute TD error as given by the critic, and are sampled by the learner using proportional prioritized sampling (see appendix F) with priority exponent α sample = 0.6. To maintain a fixed replay capacity of 10 6 , transitions are periodically evicted using proportional prioritized sampling, with priority exponent α evict = -0.4. This is a different strategy for removing data than in the Atari experiments, which simply removed the oldest data first -it remains to be seen which is superior. 

E TUNING:

On Atari, we performed some limited tuning of the learning rate and batch size: we found that larger batch sizes contribute significantly to performance, when using many actors. We tried batch sizes from {32, 128, 256, 512, 1024}, seeing clear benefits up to 512. We attempted increasing the learning rate to 0.00025 with the larger batch sizes but this destabilized training on some games. We also tried a lower learning rate of 0.00025 / 8, but this did not reliably improve results.

Likewise for continuous control, we experimented with batch sizes {32, 128, 256, 512, 1024} and learning rates from 10 -3 to 10 -5 . We also experimented with the prioritization exponents α from 0.0 to 1.0, with results proving essentially consistent within the range [0.3, 0.7] (beyond 0.7, training would sometimes become unstable and diverge).

For the experiments with many actors, we set the period for updating network parameters on the actors to be high enough that the learner was not overloaded with requests, and we set the number of transitions that are locally accumulated on each actor to be high enough that the replay server would not be overloaded with network traffic, but we did not otherwise tune those parameters and have not observed them to have significant impact on the learning dynamics.

F IMPLEMENTATION:

The following section makes explicit some of the more practical details that may be of interest to anyone wishing to implement a similar system.

Data Storage:

The algorithm is implemented using TensorFlow (Abadi et al., 2016) . Replay data is kept in a distributed in-memory key-value store implemented using custom TensorFlow ops, similar to the lookup ops available in core TensorFlow. The ops allow adding, reading, and removing batches of Tensor data efficiently.

Sampling Data We also implemented ops for efficiently maintaining and sampling from a prioritized distribution over the keys, using the algorithm for proportional prioritization described in Schaul et al. (2016) . The probability of sampling a transition is p α k / k p α k where p k is the priority of the transition with key k. The exponent α controls the amount of prioritization, and when α = 0 uniform sampling is recovered. The proportional variant sets priority p k = |δ k | where δ k is the TD error for transition k. Whenever a batch of data is added to or removed from the store, or is processed by the learner, this distribution is correspondingly updated, recording any change to the set of valid keys and the priorities associated with them.

A background thread on the learner fetches batches of sampled data from the remote replay and decompresses it using the learner's CPU, in parallel with the gradients being computed on the GPU. The fetched data is buffered in a TensorFlow queue, so that the GPU always has data available to train on.

Adding Data In order to efficiently construct n-step transition data, each actor maintains a circular buffer of capacity n containing tuples (S t , A t , R t:t+B , γ t:t+B , q(S t , * )), where B is the current size of the buffer. With each step, the new data is appended and the accumulated per-step discounts γ t:t+B and partial returns R t:t+B for all entries in the buffer are updated. If the buffer has reached its capacity, n, then its first element may be combined with the latest state S t+n and value estimates q(S t+n ) to produce a valid n-step transition (with accompanying Q-values).

However, instead of being directly added to the remote replay memory on each step, the constructed transitions (S t , A t , R t:t+B , γ t:t+B , S t+n , q(S t , * ), q(S t+n , * )) are first stored in a local TensorFlow queue, in order to reduce the number of requests to the replay server. The queue is periodically flushed, at which stage the absolute n-step TD-errors (and thus the initial priorities) for the queued transitions are computed in batch, using the buffered Q-values to avoid recomputation. The Q-value estimates from which the initial priorities are derived are therefore based on the actor's copy of the network parameters at the time the corresponding state was obtained from the environment, rather than the latest version on the learner. These Q-values need not be stored after this, since the learner does not require them, although they can be helpful for debugging.

A unique key is assigned to each transition, which records which actor and environment step it came from, and the dequeued transition tuples are stored in the remote replay memory. As mentioned in the previous section, the remote sampling distribution is immediately updated with the newly added keys and the corresponding initial priorities computed by the actor. Note that, since we store both the start and the end state with each transition, we are storing some data twice: this costs more RAM, but simplifies the code.

Contention It is important that the replay server be able to handle all requests in a timely fashion, in order to avoid slowing down the whole system. Possible bottlenecks include CPU, network bandwidth, and any locks protecting the shared data. In our experiments we found CPU to be the main bottleneck, but this was resolved by ensuring all requests and responses use sufficiently large batches. Nonetheless, it is advisable to consider all of these potential performance concerns when designing such systems.

Asynchronicity In our framework, since acting and learning proceed with no synchronization, and performance depends on both, it can be misleading to consider performance with reference to only one of these. For example, the results after a given total number of environment frames have been experienced are highly dependent on the number of updates the learner has performed in that time. For this reason it is important to monitor and report the speeds of all parts of the system and to consider them when analyzing results.

Failure Tolerance In distributed systems with many workers, it is inevitable that interruptions or failures will occur, either due to occasional hardware issues or because shared resources are needed by higher priority jobs. All stateful parts of the system therefore must periodically save their work and be able to resume where they left off when restarted. In our system, actors may be interrupted at any time and this will not prevent continued learning, albeit with a temporarily reduced rate of new data entering the replay memory. If the replay server is interrupted, the data it contains is discarded, and upon resuming, the memory is refilled quickly by the actors. In this event, to avoid overfitting, the learner will pause training briefly, until the minimum amount of data has once again been accumulated. If the learner is interrupted, progress will stall until it resumes. Purple: Rainbow baseline. Orange: A3C baseline. Blue: Ape-X DQN with 360 actors, 1 replay server and 1 Tesla P100 GPU learner. The anomaly in Riverraid is due to an infrastructure error. 

:



Figures and Tables:

Figure1: The Ape-X architecture in a nutshell: multiple actors, each with its own instance of the environment, generate experience, add it to a shared experience replay memory, and compute initial priorities for the data. The (single) learner samples from this memory and updates the network and the priorities of the experience in the memory. The actors' networks are periodically updated with the latest network parameters from the learner.

Figure 2: Left: Atari results aggregated across 57 games, evaluated from random no-op starts. Right: Atari training curves for selected games, against baselines. Blue: Ape-X DQN with 360 actors; Orange: A3C; Purple: Rainbow; Green: DQN. See appendix for longer runs over all games.

Figure 3: Performance of Ape-X DPG on four continuous control tasks, as a function of wall clock time. Performance improves as we increase the numbers of actors. The black dashed line indicates the maximum performance reached by a standard DDPG baseline over 5 days of training.

Figure 5: Varying the capacity of the replay. Agents with larger replay memories perform better on most games. Each curve corresponds to a single run, smoothed over 20 points. The curve for Wizard Of Wor with replay size 250K is incomplete because training diverged; we did not observe this with the other replay sizes.

Figure6: Testing whether improved performance is caused by recency alone: n denotes the number of actors, k the number of each transition is replicated in the replay. The data in the run with n = 32, k = 8 is therefore as recent as the data in the run with n = 256, k = 1, but performance is not as good.

Unlike the original DPG algorithm which applies autocorrelated noise sampled from a Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein (1930)), we apply exploration noise to each action sampled from a normal distribution with σ = 0.3. Evaluation is performed using the noiseless deterministic policy. Hyperparameters are otherwise as per DQN. Benchmarking was performed in two continuous control domains ((a) Humanoid and (b) Manipulator, see Figure 8) implemented in the MuJoCo physics simulator (Todorov et al. (2012)). Humanoid is a humanoid walker with action, state and observation dimensionalities |A| = 21, |S| = 55 and |O| = 67 respectively. Three Humanoid tasks were considered: walk (reward for exceeding a minimum velocity), run (reward proportional to movement speed) and stand (reward proportional to standing height). Manipulator is a 2-dimensional planar arm with |A| = 2, |S| = 22 and |O| = 37, which receives reward for catching a randomly-initialized moving ball. (a) Humanoid domain. (b) Manipulator domain.

Figure 8: Continuous control domains considered for benchmarking Ape-X DPG: (a) Humanoid, and (b) Manipulator. All tasks simulated in the MuJoCo physics simulator (Todorov et al. (2012)).

Figure 9: Training curves for 57 Atari games (performance against wall clock time). Green: DQN baseline.

Figure 10: Training curves for 57 Atari games (performance against environment frames). Only the first billion frames are shown, corresponding to 5-6 hours of training for Ape-X. Green: DQN baseline. Purple: Rainbow baseline. Blue: ApeX-DQN with 360 actors, 1 replay server and 1 Tesla P100 GPU learner.

Median normalized scores across 57 Atari games. a Tesla P100. b >100 CPUs, with a mixed number of cores per CPU machine. c Only evaluated on 49 games. d Hyper-parameters were tuned per game.

Table content: <table><tr><td>Algorithm</td><td colspan="2">Training Environment</td><td>Resources</td><td>Median</td><td>Median</td></tr><tr><td/><td>Time</td><td>Frames</td><td colspan="3">(per game) (no-op starts) (human starts)</td></tr><tr><td>Ape-X DQN</td><td>5 days</td><td colspan="2">22800M 376 cores, 1 GPU a</td><td>434%</td><td>358%</td></tr><tr><td>Rainbow</td><td>10 days</td><td>200M</td><td>1 GPU</td><td>223%</td><td>153%</td></tr><tr><td>Distributional (C51)</td><td>10 days</td><td>200M</td><td>1 GPU</td><td>178%</td><td>125%</td></tr><tr><td>A3C</td><td>4 days</td><td>-</td><td>16 cores</td><td>-</td><td>117%</td></tr><tr><td>Prioritized Dueling</td><td>9.5 days</td><td>200M</td><td>1 GPU</td><td>172%</td><td>115%</td></tr><tr><td>DQN</td><td>9.5 days</td><td>200M</td><td>1 GPU</td><td>79%</td><td>68%</td></tr><tr><td>Gorila DQN c</td><td>∼4 days</td><td>-</td><td>unknown b</td><td>96%</td><td>78%</td></tr><tr><td>UNREAL d</td><td>-</td><td>250M</td><td>16 cores</td><td>331% d</td><td>250% d</td></tr></table>

Peer review (iclr2018):
 A parallel aproach to DQN training is proposed, based on the idea of having multiple actors collecting data in parallel, while a single learner trains the model from experiences sampled from a central replay memory. Experiments on Atari game playing and two MuJoCo continuous control tasks show significant improvements in terms of training time and final performance compared to previous baselines. The core idea is pretty straightforward but the paper does a very good job at demonstrating that it works very well, when implemented efficiently over a large cluster (which is not trivial). I also appreciate the various experiments to analyze the impact of several settings (instead of just reporting a new SOTA). Overall I believe this is definitely a solid contribution that will benefit both practitioners and researchers... as long as they got the computational resources to do so! There are essentially two more things I would have really liked to see in this paper (maybe for future work?): - Using all Rainbow components - Using multiple learners (with actors cycling between them for instance) Sharing your custom Tensorflow implementation of prioritized experience replay would also be a great bonus! Minor points: - Figure 1 does not seem to be referenced in the text - « In principle, Q-learning variants are off-policy methods » => not with multi-step unless you do some kind of correction! I think it is important to mention it even if it works well in practice (just saying « furthermore we are using a multi-step return » is too vague) - When comparing the Gt targets for DQN vs DPG it strikes me that DPG uses the delayed weights phi- to select the action, while DQN uses current weights theta. I am curious to know if there is a good motivation for this and what impact this can have on the training dynamics. - In caption of Fig. 5 25K should be 250K - In appendix A why duplicate memory data instead of just using a smaller memory size? - In appendix D it looks like experiences removed from memory are chosen by sampling instead of just removing the older ones as in DQN. Why use a different scheme? - Why store rewards and gamma’s at each time step in memory instead of just the total discounted reward? - It would have been better to re-use the same colors as in Fig. 2 for plots in the appendix - Would Fig. 10 be more interesting with the full plot and a log scale on the x axis?