Title: REMEMBER AND FORGET FOR EXPERIENCE REPLAY

Abstract: Experience replay (ER) is crucial for attaining high data-efficiency in off-policy deep reinforcement learning (RL). ER entails the recall of experiences obtained in past iterations to compute gradient estimates for the current policy. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors, possibly undermining the effectiveness of ER. Previous off-policy RL algorithms mitigated this issue by tuning their hyper-parameters in order to abate policy changes. We propose ReF-ER, a method for active management of experiences in the Replay Memory (RM). ReF-ER forgets experiences that would be too unlikely with the current policy and constrains policy changes within a trust region of the behaviors in the RM. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods to show that ReF-ER reliably improves the performance of continuous-action off-policy RL. We complement ReF-ER with a novel off-policy actor-critic algorithm (RACER) for continuous-action control. RACER employs a computationally efficient closed-form approximation of the action values and is shown to be highly competitive with state-of-the-art algorithms on benchmark problems, while being robust to large hyper-parameter variations.

1. INTRODUCTION:

Deep Reinforcement Learning (RL) has an ever increasing number of success stories ranging from realistic simulated environments (Schulman et al., 2015; Mnih et al., 2016) , robotics (Levine et al., 2016) and games (Mnih et al., 2015; Silver et al., 2016) . Experience Replay (ER) (Lin, 1992) enhances deep RL algorithms by using information collected in past policy (µ) iterations to compute updates for the current policy (π). ER has become one of the mainstay techniques to improve the sample-efficiency of off-policy deep RL. Sampling from a Replay Memory (RM) stabilizes stochastic gradient descent (SGD) by disrupting temporal correlations and extracts information from useful experiences over multiple updates (Schaul et al., 2015b) . However, when π is parameterized by a Neural Network (NN), SGD updates may result in significant changes to the policy, thereby shifting the distribution of states observed from the environment. In this case sampling the RM for further updates may lead to incorrect gradient estimates, therefore deep RL methods must account for and limit the dissimilarity between π and behaviors in the RM. Previous works employed trust region methods to bound policy updates (Schulman et al., 2015; Wang et al., 2017) . Despite several successes, deep RL algorithms are known to suffer from instabilities and exhibit high-variance of outcomes (Islam et al., 2017; Henderson et al., 2017) , especially continuous-action methods employing the stochastic (Sutton et al., 2000) or deterministic (Silver et al., 2014) Policy Gradients (PG or DPG) .

In this work we redesign ER in order to control the distance between the behaviors µ used to compute the update and the current policy π. More specifically, we classify experiences either as "near-policy" or "far-policy", depending on the ratio ρ of probabilities of selecting the associated action with π and that with µ. The weight ρ appears in many estimators that are used with ER such as the off-policy policy gradients (off-PG) (Degris et al., 2012) and the off-policy return-based evaluation algorithm Retrace (Munos et al., 2016) . From this classification, we introduce measures to limit the fraction of "far-policy" samples in the RM, as well as computing gradient estimates only from "near-policy" experiences. Furthermore, these hyper-parameters can be gradually annealed during training to obtain increasingly accurate updates from nearly on-policy experiences. Remember and Forget Experience Replay (ReF-ER) is a simple algorithm that can be applied to virtually any off-policy RL algorithm with parameterized policies. We show that ReF-ER allows better stability and performance than conventional ER in all three main classes of continuous-actions off-policy deep RL algorithms: methods based on the DPG (ie. DDPG (Lillicrap et al., 2016) ), methods based on Q-learning (ie. NAF (Gu et al., 2016) ), and with off-PG (Degris et al., 2012; Wang et al., 2017) .

In recent years, there has been a growing interest in coupling RL with high-fidelity physics simulations (Reddy et al., 2016; Novati et al., 2017; Colabrese et al., 2017; Verma et al., 2018) . The computational cost of these simulations calls for data-efficient RL methods that are reliable and do not require problem-specific tweaks to the hyper-parameters. Moreover, while on-policy training of simple policy architectures has been shown to be sufficient in some benchmark environments (Rajeswaran et al., 2017) , agents aiming to solve complex problems with highly non-linear dynamics might require deep or recurrent models that can be trained more efficiently with off-policy methods. We address these issues by introducing a simple and computationally efficient off-policy actor-critic architecture (RACER) for continuous-action control problems. We systematically analyze a wide range of hyper-parameters on the OpenAI Gym (Brockman et al., 2016) robotic tasks, and show that RACER combined with ReF-ER reliably obtains results that are competitive with the state-of-the-art.

2. PRELIMINARIES:

Consider the sequential decision process of an agent aiming to optimize its interaction with the environment. At each step t, the agent observes its state s t ∈ R d S , performs an action by sampling a policy a t ∼ µ(a|s t ) ∈ R d A , and transitions to a new state s t+1 ∼ D(s|a t , s t ) with reward r t+1 ∈ R. These interactions {s t , a t , r t , µ t } are stored in a RM, which constitutes the data used by off-policy RL to train the parametric policy π w (a|s). The importance weight ρ t = π w (a t |s t )/µ t is the ratio between the probability of selecting a t with the current π w and with the behavior µ t , which gradually becomes dissimilar from π w as the latter is trained. The on-policy state-action value Q π (s, a) measures the expected returns from (s, a) following the policy π w :

Q π (s, a) = E st∼D, at∼π w ∞ t=0 γ t r t+1 s 0 = s, a 0 = a (1)

Here γ is a discount factor. The value of state s is the on-policy expectation: s, a) ] and the action advantage is A π (s, a) = Q π (s, a)-V π (s), such that E a∼π [A π (s, a)] :=0. We consider algorithms that train parametric approximators Q w from off-policy data. The Q-learning target is qt = r t+1 + γ Ea ∼π Q w (s t+1 , a ). The Retrace algorithm (Munos et al., 2016) includes all the rewards obtained by the behavior µ t in the value estimation:

V π (s) = E a∼π [Q π (

EQUATION

The off-PG (Degris et al., 2012) can be used to update π w by ER: ĝoff-PG

t (w) = ρ t Âπ t ∇ w log π w (a t |s t ).

Here, Âπ:

t is an estimator for the on-policy advantage, such as Âret t := Qret t -V w (s t ).

3. REMEMBER AND FORGET EXPERIENCE REPLAY:

In off-policy RL it is common to maximize on-policy returns averaged over the distribution of states in a RM (Degris et al., 2012) . However, as π w gradually shifts away from previous behaviors µ t , the distribution of states in the RM is increasingly dissimilar from the on-policy distribution, and trying to increase an off-policy performance metric may not improve on-policy outcomes. This issue may be compounded with algorithm-specific concerns. For example, in ACER (Wang et al., 2017) the dissimilarity between µ t and π w may cause vanishing or diverging importance weights ρ t , thereby increasing the variance of the off-PG and deteriorating the convergence speed of Retrace by inducing "trace-cutting" (Munos et al., 2016) . As a remedy, ACER tunes the learning rate and uses a target network (Mnih et al., 2015) , updated as a delayed copy of the policy network, to constrain policy updates. Target networks are also employed in DDPG (Lillicrap et al., 2016) in order to slow down the feedback loop between value-network and policy-network optimizations. This feedback loop causes overestimated action values that can only be corrected by acquiring new on-policy samples. Recent works (Henderson et al., 2017) have shown the opaque variability of outcomes of continuous-action deep RL algorithms depending on hyper-parameters. Target-networks may be one of the sources of this unpredictability. In fact, when using deep approximators, there is no guarantee that the small weight changes imposed by target-networks correspond to small changes in the network's output.

Here we propose a set of three simple rules, collectively referred to as Remember and Forget ER (ReF-ER), to directly control the degree of "off-policyness" of the samples in the RM:

1. Updates are computed from mini-batches drawn uniformly from a RM. We compute the importance weight ρ t of each sample and classify it as "near-policy" if 1/c max <ρ t <c max with c max >1. The samples with vanishing (ρ t <1/c max ) or exploding (ρ t >c max ) importance weights are classified as "far-policy". When computing off-policy estimators with finite batch-sizes, such as Qret or the off-PG, "far-policy" samples may either be irrelevant or increase the variance. ReF-ER allows computing gradient estimates exclusively from "near-policy" samples. In order to efficiently approximate the number of far-policy samples in the RM, we store for each step its most recent ρ t .

2. When acquiring samples from the environment, older episodes with the highest fraction of far-policy samples are removed until the number n obs of samples in the RM is at most N .

3. Policy updates are penalized in order to attract the current policy π w towards µ t :

ĝReF-ER t (w) = β ĝt (w) -(1-β)∇D KL [µ t π w (•|s t )] if 1/c max <ρ t <c max -(1-β)∇D KL [µ t π w (•|s t )] otherwise (3)

Here D KL is the Kullback-Leibler divergence and the coefficient β ∈ [0, 1] is updated after each gradient step such that a fixed fraction D ∈ (0, 1) of the RM are far-policy samples:

EQUATION

where η is the learning rate and n far is the number of far-policy samples. Note that iteratively updating β with Eq. 4 has fixed points in 0 for n far /n obs >D and in 1 otherwise. 

(a):

Neural Network (w) Under review as a conference paper at ICLR 2019 updates with a trust-region scheme, and modifies Retrace to use dA √ ρ instead of ρ. Lacking these techniques, we expect RACER to require ReF-ER to deal with unbounded importance weights.

EQUATION

RACER's network architecture consists of a single Multilayer Perceptron (MLP) with weights w which receives the state s and outputs as shown in Fig. 1b . From the final layer, the first 2d A outputs are taken as the mean m and diagonal covariance Σ of the Gaussian policy π w (a|s). One output estimates the state value V w , and the remaining outputs are the parameters of a closed-form approximator for the action advantage A w . RACER can be extended for problems where the state is received as a visual feed (by inserting convolutional layers), or where the state is only partially observable (by substituting the MLP with a recurrent model). However, these extensions are outside the scope of this paper.

The parameters w are updated from mini-batches of off-policy time steps with ReF-ER. A separate gradient is defined for each component of the actor-critic framework, as sketched in figure 1b . The policy statistics m(s) and Σ(s) are updated with the off-PG (Degris et al., 2012) :

EQUATION

We estimate the on-policy advantage with Retrace. From TBC and Retrace, we obtain an estimator for the on-policy state value

EQUATION

Rather than having a separate MLP with inputs (s, a) to parameterize Q w (as in ACER or DDPG), whose expected value under the policy would be computationally demanding to compute, we employ a closed-form equation for A w inspired by NAF (Gu et al., 2016) . The network outputs the coefficients of a concave function f w (s, a) which is chosen such that its maximum coincides with the mean of the policy m(s), and such that it is possible to derive analytical expectations for a ∼ π w . In App. B we explore multiple choices of f w (s, a). For all results in the main text of the manuscript we used:

EQUATION

Here a -= min [a-m(s), 0] and a + = max [a-m(s), 0] (both element-wise operations). This parameterization requires one MLP output for K(s) and d A outputs for each diagonal matrix L + and L -. In total, given a state s, the MLP computes m, Σ, V , K, L + and L -. From Eq.7, for any action a the advantage is uniquely defined:

A w (s, a):= f w (s, a)-E a ∼π [f w (s, a )].

Like the exact on-policy advantage A π , A w has by design expectation zero under the policy. The parameterization coefficients K and L are updated to minimize the L2 error from Âret t :

EQUATION

Here, ρ t reduces the weight of estimation errors for unlikely actions, where A w is expected to be less accurate. To ensure that Σ, K, L + and L -are positive definite, the respective outputs are mapped onto R + by a Softplus rectifier. The analytical derivation of E a ∼π [f w (s, a )] can be found in App. A. Collectively, ĝπ t (w), ĝV t (w), and ĝA t (w) form a vector ĝAC t (w) with the same size as the MLP output. ĝAC t (w) is weighted with the ReF-ER penalization (Eq. 3) and then back-propagated to the MLP. In order to estimate Qret t for a sampled time step t, Retrace (Eq. 2) requires V w and Q w for all future steps in sample t's episode. These are naturally computed when training from batches of episodes (as in ACER) rather than time steps. However, the information contained in consecutive steps is highly correlated, worsening the quality of the gradient estimate, and episodes can be composed of thousands of time steps, increasing the computational cost. In order to efficiently train from uncorrelated time steps, RACER stores for each sample in the RM the most recently computed estimations of V w (s t ), A w (s t , a t ), ρ t and Q ret t . When a batch of time steps is sampled, the stored Q ret t are used to compute the gradient update. At the same time, the stored V w (s t ), A w (s t , a t ) and ρ t are updated with the current NN outputs and used to correct Q ret for all prior time-steps in the respective episodes with Eq. 2. A description of all remaining implementation details is provided in App. D.

5. RELATED WORK:

The rules that determine which samples are kept in the RM and how they are used for training can be designed to address several objectives. For example, properly designed ER may prove necessary to prevent lifelong learning agents from forgetting previously mastered tasks (Isele & Cosgun, 2018) . Prioritized Experience Replay (Schaul et al., 2015b) (PER) improves the performance of DQN (Mnih et al., 2015) by biasing sampling in favor of experiences that cause large temporal-difference (TD) errors. TD errors may signal rare events that would convey useful information to the learner. ER can be used to train transition models in planning-based RL (Pan et al., 2018) , or to train off-policy learners on auxiliary tasks (Schaul et al., 2015a; Jaderberg et al., 2017) helping to shape the network features. When rewards are very sparse, RL agents can be trained to repeat previous outcomes (Andrychowicz et al., 2017) or to reproduce successful states or episodes (Oh et al., 2018; Goyal et al., 2018) .

de Bruin et al. (2015) proposes a modification to ER that increases the diversity of behaviors contained in the RM, which is the opposite of what ReF-ER achieves. The ideas proposed by de Bruin et al. (2015) cannot readily be applied to complex tasks (they consider a low-dimensional problem which can be learned with the RM holding a handful of episodes) and the authors concede that their method is not suitable when the policy is advanced for many iterations. For these reasons, we compare ReF-ER only to PER and conventional ER. We assume that if increasing the diversity of experiences in the RM were beneficial to off-policy RL then either PER or ER would outperform ReF-ER.

ReF-ER is inspired by the techniques developed for on-policy RL to bound policy updates of PPO (Schulman et al., 2017) . Rule 1 of ReF-ER is similar to the clipped objective function of PPO (gradients are zero if ρ lies outside of some range). However, Rule 1 is not affected by the sign of the advantage estimate and clips both policy and value gradients. Like Rule 3, one variant of PPO penalizes D KL (µ t ||π w ) (also Schulman et al. (2015) and Wang et al. (2017) employ trust-region schemes in the on-and off-policy setting respectively). While PPO picks one of the two techniques, in ReF-ER Rules 1 and 3 complement each other and can be applied to most off-policy RL methods with parametric policies. In the ER setting, samples remain in the RM over many iterations. If only Rule 1 is included, any PG could push the sample's ρ outside of the clipping range without means to recover, leading to zero-valued gradients. Including only Rule 3 would not reduce the number of hyper-parameters (a target D KL would be needed), and would not prevent unbounded ρ.

The first method to combine ER and PG was the Off-Policy Actor Critic (Degris et al., 2012) . Further contributions were introduced with ACER ( Wang et al., 2017) , such as extension to NN, estimation of on-policy returns with Retrace, and the variance truncation and bias correction trick (TBC). In the continuous-action domain, the proposed RACER offers several advantages over ACER: (a) reduced complexity by employing a single NN and closed-form A w (continuous-ACER uses 9 NN evaluations per gradient). (b) relies on ReF-ER rather than constraining policy updates around a target network. (c) samples time steps rather than episodes (which may consist of thousands of steps). (d) uses the original Retrace estimator, which has better convergence guarantees than the value-estimators used by continuous-ACER. Because of (a) and (c), RACER is two orders of magnitude faster than ACER.

6. RESULTS:

In this section we couple ReF-ER, conventional ER and PER with one method from each of the three main classes of deep continuous-action RL algorithms: DDPG, NAF, and RACER. The performance of each combination of algorithms is measured on the MuJoCo (Todorov et al., 2012) tasks of OpenAI Gym (Brockman et al., 2016) by plotting the mean cumulative reward R = t r t . Each plot tracks the average R among all episodes entering the RM within intervals of 2•10 5 time steps and averaged again among five differently seeded training trials. In the appendix we include contours of the 20 th to 80 th percentiles of R obtained by DDPG and RACER on the OpenAI Gym and on the DeepMind Control Suite (Tassa et al., 2018) . The code to reproduce all present results is available on GitHub. 1 6.1 RESULTS FOR DDPG DDPG (Lillicrap et al., 2016) trains two networks by ER. The value network outputs Q w (s, a) and is trained to minimize the L2 distance from the Q-learning target (Sec. 2). The policy network outputs a deterministic policy m w (s) and is trained with the DPG (Silver et al., 2014) :

ĝDPG t (w) = ∇ w m w (s) ∇ a Q w (s, a) a=m w (s) (9)

1 The repository is hidden to maintain anonymity during the review process. Noise is added to the deterministic policy π w =m w +N (0, σ 2 I) for exploration. We consider two variants of DDPG: one with m w bounded to the unit box [-1, 1] d A (as in the original) and one without bounds (referred to as u-DDPG, implementation details can be found in App. D). Bounding the policy may lead to lower returns in the OpenAI MuJoCo benchmarks, which are defined for unbounded actions. We note that, without measures to constrain policy updates or without careful tuning of the hyper-parameters (we found the critic's weight decay and temporally-correlated action noise to be necessary), u-DDPG is unstable. The returns for u-DDPG can fall to large negative values, especially in tasks that include a control-cost penalty in the reward such as Ant. This behavior is explained by the value network not having learned local maxima with respect to the action (Silver et al., 2014) .

By replacing ER with ReF-ER we can stabilize u-DDPG and greatly improve its performance, especially for tasks with complex dynamics such as Humanoid or Ant. The hyper-parameter c max determines how much the policy is allowed to change from the behaviors in the RM. By annealing c max we can allow faster improvements at the beginning of training, when an inaccurate policy gradient might be sufficient to estimate a good direction for the update. Conversely, during the later stages of training, precise updates can be computed from almost on-policy samples. We anneal c max and the learning rate according to:

EQUATION

)

Here k is the gradient step index, η is the initial learning rate (10 -4 and 10 -5 for the value and policy networks respectively, as when using regular ER). We found that annealing η worsened the instability of u-DDPG with regular ER. Lower values of C reduce the speed of policy improvements, but after 10 7 time steps C = 2 achieves the best performance in most tasks. In Fig. 3 we report for a subset of problems the average D KL (µ t ||π w ). D KL decreases for lower values of C and further decreases during training due to the annealing process. With regular ER, even after lowering η by one order of magnitude from that of the original paper, the distance between π and µ may span the entire action space. In fact, in most tasks shown in Fig. 3 the D KL (µ t ||π w ) of DDPG is of similar order of magnitude as its maximum d A 2 (2/σ) 2 (for example, since σ = 0.2, the maximum D KL is 850 for Humanoid and 300 for Walker2d and it oscillates during training around 100 and 50 respectively).

ReF-ER maintains a RM of mostly near-policy samples, providing the value-network with multiple examples of trajectories that are possible with the current policy. This focuses the predictive capabilities of the value-network, enabling it to extrapolate the effect of a marginal change of action on the expected returns, and therefore increasing the accuracy of the DPG. Any misstep of the DPG is weighted with a penalization term that attracts the policy towards past behaviors. This allows time for the learner to gather experiences with the new policy, improve the value-network, and correct the misstep. This reasoning is almost diametrically opposed to that behind PER. In PER observations that are associated with larger TD errors are sampled more frequently. In the continuous-action setting, however, TD errors may be result from actions that are farther from the current policy. Therefore, precisely estimating their value might not help the value network in yielding an accurate estimate of the DPG. We obtained better results with the rank-based variant of PER, which has similar performance to that of DDPG with ER. The main benefits over ReF-ER arise in tasks that require more exploration, such as Swimmer and HumanoidStandup.

6.2. RESULTS FOR NAF:

Normalized Advantage Functions (NAF) is the state-of-the-art of Q-learning based algorithms for continuous-action problems. It employs a quadratic-form approximation of the advantage A w , analogous to the one employed by RACER:

EQUATION

NAF trains A w with the Q-learning target (Sec. 2) and uses the location m w of its maximum as the mean of the policy, with added Gaussian noise for exploration. When the actual A π is not well approximated by a quadratic (e.g. when the return landscape is multi-modal), NAF may fail to choose good actions. In contrast, RACER updates π w with the off-PG, which is independent of the error in approximating A π (s t , a t ) (but depends on the error at t+1). Figure 4 shows how NAF is affected by the choice of ER algorithm. While Q-learning based methods are thought to be less sensitive than PG-based methods to the dissimilarity between policy and stored behaviors owing to the bootstrapped Q-learning target, NAF benefits from REF-ER. This is because A π is likely to be approximated well by a quadratic in a small neighborhood near its local maxima. ReF-ER constrains learning from actions within this neighborhood and prevents large TD errors from disrupting the locally-accurate approximation of A w . This intuition is supported by observing that rank-based PER (the better performing variant of PER also in this case), often worsens the performance of NAF. PER aims at biasing sampling in favor of larger TD errors, which are more likely to be farther from m(s) (note that f w NAF is unbounded), and their accurate prediction might not help the learner in fine-tuning the policy by improving a local approximation of the advantage.

6.3. RESULTS FOR RACER:

Here we compare RACER to ACER and PPO, an algorithm that owing to its simplicity and good performance on MuJoCo tasks is often used as baseline. We omit results from coupling RACER with ER or PER as it yields large negative R values. In fact, without ReF-ER, the unbounded importance weights cause off-PG estimates to diverge, disrupting prior learning progress. Similarly to ReF-ER, ACER's techniques (Sec. 4) guard against the numerical instability of the off-PG. However, bounding policy updates to a target-network does not ensure similarity between π w and RM behaviors (as shown in Fig. 7 ). In fact, when using deep approximators, simply enforcing slow parameter updates does not guarantee small changes in the output. As D KL (µ t π w ) grows, off-PG estimates may become inaccurate, causing ACER to be outperformed by RACER and PPO. We note that, due to ACER's cost, we could only optimize some of its hyper-parameters relative to the original paper (see App. D). We do not exclude that ACER could have performed better after more extensive tuning.

The two hyper-parameters that most strongly affect the performance of RACER are the RM size N and c max , annealed with Eq. 10 (for a discussion of the other parameters see App. B). RACER uses the Retrace estimator Q ret which is expected to converge to the on-policy Q π (Munos et al., 2016) . Because π w is gradually changing during training, it is crucial to maximize the convergence speed of Retrace to obtain accurate estimates of the off-PG. Large values of c max increase the variance of the policy gradient and increase the amount of "trace-cutting" as the importance weights are allowed to diverge from 1. The cumulative rewards reported in Fig. 5 show that "stable" tasks, where the agent's success is less predicated on avoiding mistakes that would cause it to trip, are more tolerant to high values of c max (e.g. HalfCheetah). In most other tasks, especially those that require precise controls (e.g. Walker), the best results are obtained for values of c max that strike a balance between strict penalty terms, which slow down policy improvements, and high-variance gradient estimates. The RM size N has a parallel effect. A small RM may not contain enough diversity of samples for the learner to accurately estimate the gradients. Conversely, a large RM is composed of episodes obtained with increasingly older versions of π w . In this case, the penalty terms required to preserve a sufficient fraction of near-policy samples may prevent the policy from improving. A discussion of all the secondary hyper-parameters and the choice of function parameterizing A w can be found in App. B. For most combinations of hyper-parameters and tasks presented in this section, RACER outperforms the best result from DDPG (Sec. 6.1), PPO, and is competitive with the best results found in the published literature, which to our knowledge were achieved by the on-policy algorithms TRPO (Schulman et al., 2015) and Policy Search with Natural Gradient (Rajeswaran et al., 2017) .

7. CONCLUSION:

Many RL algorithms update a policy π w from past experiences collected with off-policy behaviors µ. We present evidence that off-policy continuous-action deep RL methods benefit from actively maintaining similarity between policy and replay behaviors. We propose a novel ER algorithm (ReF-ER) that extends these benefits to off-policy PG, deterministic PG and Q-learning methods. ReF-ER characterizes past behaviors either as "near-policy" or "far-policy" by the deviation from one of the importance weight ρ=π w (a|s)/µ(a|s). ReF-ER consists of: 1) Computing gradients only from near-policy experiences. 2) Forgetting far-policy samples when new observations are obtained from the environment. 3) Regulating the pace at which π w is allowed to deviate from µ through penalty terms that reduce D KL (µ||π w ). This allows time for the learner to gather experiences with the new policy, improve the value estimators, and increase the accuracy of the next steps. The extension of ReF-ER to discrete-action methods is subject of ongoing work.

ReF-ER is combined with a novel method based on the off-policy PG (RACER). RACER uses a closed-form parameterization of the advantage and it estimates on-policy returns with a bootstrapped formulation of Retrace. These innovations greatly increase the computational efficiency of RACER compared to similar methods. The application of ReF-ER and RACER to OpenAI Gym benchmarks produces state-of-the-art results while being robust to large variations in their hyper-parameters.

A EXPECTATIONS OF THE PARAMETERIZED ADVANTAGE FUNCTIONS:

We considered three parameterizations for the action advantage:

EQUATION

We recall a -= min [a-m(s), 0] and a + = max [a-m(s), 0]. The first parameterization f w Q is identical to the one employed by NAF (Gu et al., 2016) , and requires training (d 2 A + d A )/2 coefficients of the lower triangular matrix L Q . Its quadratic complexity makes the choice of f w Q unfavorable for high-dimensional action spaces (e.g. it requires 153 parameters for the 17-dimensional Humanoid tasks of OpenAI Gym, against the 35 of f w DG ). In order to preserve bijection between L Q and L Q L Q , the diagonal terms are mapped to R + with a Softplus function. The expectation under a Gaussian policy can be computed as (Petersen et al., 2008) :

EQUATION

Here Tr denotes the trace of a matrix.

The second parameterization requires training the 1 + d A coefficients for K(s) and the diagonal matrix L(s). The expectation can be easily derived from the properties of products of Gaussian densities:

EQUATION

Here | • | denotes a determinant.

To derive the expectation of the third parameterization, which was used for most results in the paper, we recall that the expectation of a product of independent variables is the product of the expectations.

For one component i of the action vector:

EQUATION

Here we exploited the symmetry of the Gaussian policy around the mean. Since Σ, L + , and L -are all diagonal, we can compute the expectation:

EQUATION

Finally, we note that all these parameterizations are differentiable.

B SENSITIVITY TO HYPER-PARAMETERS:

Figure 6 shows the robustness of RACER to various hyper-parameters and to the choice of the advantage parameterizations introduced in App. A. Moreover, we can bypass the advantage approximator (i.e. A w :=0) and rely exclusively on the state value approximator. We recall the on-policy value estimator obtained with "variance truncation and bias correction" (TBC) (Wang et al., 2017) (Sec. 4):

EQUATION

From Eq. 2 and 19 we obtain Qret (s t , a t ) = r t+1 + γ V tbc (s t+1 ). By neglecting A w , from this relation and Eq. 19, we obtain a recursive Retrace-based estimator for the on-policy state value that depends on V w alone:

EQUATION

This target is equivalent to the recently proposed V-trace estimator (Espeholt et al., 2018) when all importance weights are clipped at 1, which was empirically found by the authors to be the best-performing solution. As expected, lacking a model for A w (s, a), this architecture (denoted The y-axes are magnified around the final returns to clarify often minor differences in performance.

f DG (s,a) f SG (s,a) f Q (s,a) V-

as V-RACER in the first column of results of Fig. 6 ) yielded worse performance than the original RACER. However the difference is often minor and the increased simplicity of this architecture might justify its adoption for some problems.

The tolerance D for far-policy samples in the RM has a similar effect as c max : low values tend to delay learning while high values reduce the fraction of the RM that is used to compute updates and may decrease the accuracy of gradient estimates. Increasing the number F of time steps per gradient step could either cause a rightward shift of the expected returns because the learner computes fewer updates for the same budget of observations, or it could aid the learner by providing more on-policy samples. The actual outcomes depend on the problem: tasks with simpler dynamics (e.g. HalfCheetah) can be learned more quickly by performing more gradient steps, while problems with more complex dynamics (e.g. Ant, Humanoid) benefit from more on-policy samples. The batch-size B and learning rate η have a minor effect on performance.

C STATE, ACTION AND REWARD PREPROCESSING:

Several authors have employed state (Henderson et al., 2017) and reward (Duan et al., 2016) (Gu et al., 2017) rescaling to improve the learning results. This technique has not systematically been studied but reflects a deeper challenge of RL. The success of any RL algorithm is intrinsically linked to the design of the control problem. For example, the stability of DDPG is affected by the L2 weight decay of the value-network. Depending on the numerical values of the distribution of rewards provided by the environment and the choice of weight decay coefficient, the L2 penalization can be either negligible or dominate the Bellman error. Similarly, the distribution of values describing the state variables can increase the challenge of learning by gradient descent.

We propose partially addressing these issues by rescaling both rewards and state vectors depending on the the experiences contained in the RM. At the beginning of training we prepare the RM by collecting N start observations and then we compute:

µ s = 1 nobs nobs t=0 s t (21) σ s = 1 nobs nobs t=0 (s t -µ s ) 2 (22)

Throughout training, µ s and σ s are used to standardize all state vectors ŝt = (s t -µ s )/(σ s + ) before feeding them to the network approximators. Moreover, every 1000 steps, chosen as the smallest power of ten that did not affect the run time, we loop over the n obs observations stored in the RM to compute:

EQUATION

)

This value is used to scale the rewards rt = r t /(σ r + ) used by the Q-learning target of DDPG and the Retrace algorithm for RACER. We use = 10 -7 to ensure numerical stability.

The actions sampled by the learner may need to be rescaled or bounded to some interval depending on the environment. For the OpenAI Gym tasks this amounts to a linear scaling a = a (upper_value -lower_value)/2, where the values specified by the Gym library are ±0.4 for the Humanoid tasks, ±8 for the Pendulum tasks, and ±1 for all others. The tasks of DeepMind Control Suite are defined with all actions bounded do the interval [-1, 1] d A . In this case we can learn Gaussian policies, as if the action space were unbounded, and then map the actions sent to the environment with an hyperbolic tangent: a t = tanh a t . This approach, however, might prevent efficiently learning policies that behave like bang-bang controllers. Close to the bounds of the control space, actions that should be similar may be mapped to distant positions in the unbounded action space. Therefore, learning bang-bang controls is more likely to be hindered by the D KL penalties of ReF-ER. This issue may explain some poor results on the DeepMind Control Suite shown in Fig. 10 and 11. These results show that successfully tackling control problems with bounded action spaces may require policies parameterized as Beta distribution (Chou et al., 2017) .

D IMPLEMENTATION AND NETWORK ARCHITECTURE DETAILS:

We implemented all presented learning algorithms within smarties,2 our open source C++ RL framework, and optimized for high CPU-level efficiency through fine-grained multi-threading, strict control of cache-locality, and computation-communication overlap. On every step, we asynchronously obtain on-policy data by sampling the environment with π, which advances the index t of observed time steps, and we compute updates by sampling from the Replay Memory (RM), which advances the index k of gradient steps (Fig. 1a ). During training, the ratio of time and update steps is usually equal to a constant: t/k = F . This parameter affects the data efficiency of the algorithm; by lowering F each sample is used more times to improve the policy before being replaced by newer samples. Upon completion of all tasks, we apply the gradient update and proceed to the next step. The pseudo-codes in App. E neglect parallelization details as they do not affect execution.

In order to evaluate all algorithms on equal footing, we use the same baseline network architecture for RACER, DDPG and NAF, consisting of an MLP with two hidden layers of 128 units each. For the sake of computational efficiency, we employed Softsign activation functions. The weights of the hidden layers are initialized according to U -6/ √

f i + f o , 6/ √ f i + f o ,

where f i and f o are respectively the layer's fan-in and fan-out (Glorot & Bengio, 2010) . The weights of the linear output layer are initialized from the distribution U -0.1/ √ f i , 0.1/ √ f i , such that the MLP has near-zero outputs at the beginning of training. When sampling the components of the action vectors, the policies are treated as truncated normal distributions with symmetric bounds at three standard deviations from the mean. Finally, we optimize the network weights with the Adam algorithm (Kingma & Ba, 2015) . RACER We note that the values of the diagonal covariance matrix are shared among all states and initialized to Σ = 0.2I. The remaining hyper-parameters of RACER are listed in table 1. DDPG In its original formulation, DDPG transforms the policy-network's output onto the bounded interval [-1, 1] d A with an hyperbolic tangent function. We found that bounding the action space may limit the performance of DDPG with the OpenAI MuJoCo tasks. In order to stabilize the unboundedaction version of DDPG (u-DDPG) we set the learning rate for the policy-network to 1 • 10 -5 and that of the value-network to 1 • 10 -4 with L2 weight decay coefficient of 1 • 10 -4 . These changes lead to better performance also with the bounded-action DDPG and therefore were used for all numerical experiments. The RM is set to contain N =2 19 observations and we follow Henderson et al. (2017) for the remaining hyper-parameters: mini-batches of B = 128 samples, γ=0.995, soft target network -4 for the policy and 3 • 10 -4 for the critic) with the same annealing as used in the other experiments.

ACER We kept most hyper-parameters as described in the original paper (Wang et al., 2017) : the TBC clipping parameter is c = 5, the trust-region update parameter is δ = 1, and five samples of the advantage network are used to compute A w estimates under π. We use a RM of 1e5 samples, each gradient is computed from 24 uniformly sampled episodes, and we perform one gradient step per environment step. Because here learning is not from pixels, each network (value, advantage, and policy) is an MLP with 2 layers and 64 units per layer (we tried training runs with 96 and 128 units per layer and did not observe improvements). Accordingly, we reduced the soft target-network update coefficient (α = 0.001) and the learning rates for the value-(η = 3 • 10 -4 ) and for the policy-network (η = 1 • 10 -4 ). Despite these reduced learning rates, Fig. 7 shows that ACER's techniques do not prevent the policy from becoming disconnected from prior behaviors. We note that hyper-parameters are known to affect performance (Henderson et al., 2017) and we do not exclude that ACER could have performed better with more extensive tuning.

E PSEUDO-CODES:

Remarks on algorithm 1: 1) It describes the general structure of the ER-based off-policy RL algorithms implemented for this work (i.e. RACER, DDPG, and NAF). 2) This algorithm can be adapted to conventional ER, PER (by modifying the sampling algorithm to compute the gradient estimates), or ReF-ER (by following Sec. 3)).

3) The algorithm requires 3 hyper-parameters: the ratio of time step to gradient steps F (usually set to 1 as in DDPG), the maximal size of the RM N , and the minimal size of the RM before we begin gradient updates N start .

Remarks on algorithm 2: 1) The reward for an episode's initial state, before having performed any action, is zero by definition.

2) The value V w (s t ) for the last state of an episode is computed if the episode has been truncated due the task's time limits or is set to zero if s t is a terminal state. 3) Each time step we use the learner's updated policy network and we store µ t = {m(s t ), Σ(s t )}.

Remarks on algorithm 3: 1) In order to compute the gradients ĝAC ti (w), we rely on advantage estimates Q ret ti that were computed when subsequent time steps in t i 's episode were previously drawn by ER. Not having to compute the quantities A w , V w , and ρ for all following steps comes with clear computational 

:

The repository is hidden to maintain anonymity during the review process.

Figures and Tables:

Figure 1: (a) Diagram of asynchronous ER-based RL algorithms. (b) Neural network architecture employed by RACER. Blue arrows connect each output with the elements of the actor-critic framework for which it is used. Red arrows represent the flow of the error signals.

Figure 2: Cumulative rewards on OpenAI MuJoCo tasks for DDPG (green line), DDPG with rankbased PER (gray line), u-DDPG with regular ER (blue), and ReF-ER with C = 8 (purple), C = 4 (red), C = 2 (orange). Implementation details in App. D.

Figure 4: Cumulative rewards on the OpenAI MuJoCo tasks for NAF (blue line), NAF with rank-based PER (gray line), NAF with ReF-ER and C = 8 (purple), C = 4 (red), C = 2 (orange).

Figure 5: Average cumulative rewards on MuJoCo OpenAI Gym tasks obtained with PPO (dashed black lines), ACER (dash-dot purple lines) and with RACER by independently varying the two main hyper-parameters of ReF-ER: the RM size N and C (colored lines).

Figure 6: Mean cumulative rewards with RACER on OpenAI Gym tasks exploring the effect of the choice of advantage parameterization, ReF-ER tolerance factor D, mini-batch size B, number of time steps per gradient step F , and learning rate η. Dashed-black lines refer to the baseline parameters.The y-axes are magnified around the final returns to clarify often minor differences in performance.

Figure 8: 20 th and 80 th percentiles of cumulative rewards for episodes of OpenAI Gym tasks ended within intervals of 2 • 10 5 time steps across 5 independent training runs. The figures includes results for PPO (black contours), RACER baseline (green contours, C = 4, N = 10 19 ), RACER with conservative ReF-ER constraints and large RM size (blue contours, C = 2, N = 10 20 ), and RACER with relaxed ReF-ER constraints and smaller RM size (red contours, C = 8, N = 10 18 ).

Figure 9: 20 th and 80 th percentiles of cumulative rewards for episodes of OpenAI Gym tasks ended within intervals of 2•10 5 time steps across 5 independent training runs. The figures includes results for DDPG (green contours), DDPG with PER (red contours), and u-DDPG with ReF-ER (blue contours, C = 4, and all other hyper-parameters set according to App. D).

Figure10: 20 th and 80 th percentiles of cumulative rewards for trajectories of DeepMind Control Suite tasks ended within intervals of 2 • 10 5 time steps across 5 independent training runs using the baseline RACER hyper-parameters (Table1).

For c max →1 and D→0, ReF-ER becomes asymptotically equivalent to computing updates from on-policy samples. ReF-ER aims to reduce the sensitivity on the network architecture and hyperparameters by controlling the rate at which the policy can deviate from the behaviors in the RM.

Table content: <table><tr><td colspan="4">4 RACER: REGULARIZED ACTOR-CRITIC WITH EXPERIENCE REPLAY</td></tr><tr><td colspan="4">This work analyzes ReF-ER with three types of continuous-actions off-policy deep RL algorithms:</td></tr><tr><td colspan="4">methods based on the DPG (i.e. DDPG), based on Q-learning (i.e. NAF), and based on the off-PG</td></tr><tr><td colspan="4">(e.g. ACER (Wang et al., 2017) and IMPALA (Espeholt et al., 2018)). Here we introduce RACER, an</td></tr><tr><td colspan="4">off-PG based architecture with some improvements over ACER and IMPALA which aid our analysis</td></tr><tr><td colspan="4">of ReF-ER. First, RACER does not require expensive network architectures (easing reproducibility</td></tr><tr><td colspan="4">and exploration of the hyper-parameters). Second, it samples time steps rather than episodes (like</td></tr><tr><td colspan="4">DDPG and NAF and unlike ACER and IMPALA), further reducing its cost. Third, it does not introduce</td></tr><tr><td colspan="4">any technique that would interfere with ReF-ER and affect its analysis. Specifically, ACER uses</td></tr><tr><td colspan="4">the variance truncation and bias correction trick (TBC), employs a target network to bound policy</td></tr><tr><td>s t+1 r t+1 CPU i</td><td colspan="2">Spawn independent tasks Learner process</td><td>CPU j</td></tr><tr><td>Environment</td><td>Data acquisition</td><td>Gradient estimation</td></tr><tr><td>a t ~ π w</td><td colspan="2">Weight update Gradients summation</td><td>Reduction</td></tr></table>

RACER architecture's baseline hyper-parameters. We note that while DDPG is the only algorithm employing two networks, choosing half the batch-size as RACER and NAF makes the compute cost roughly equal among the three methods. Finally, when using ReF-ER we add exploratory Gaussian noise to the deterministic policy: π w =m w +N (0, σ 2 I) with σ=0.2. When performing regular ER or PER we sample the exploratory noise from an Ornstein-Uhlenbeck process with σ=0.2 and θ=0.15.NAF We use the same baseline MLP architecture and learning rate η = 10 -4 , batch-size B = 256, discount γ = 0.995, RM size N = 2 19 , and soft target network update coefficient 0.01. Gaussian noise is added to the deterministic policy π w =m w +N (0, σ 2 I) with σ=0.2.PPO We tuned the hyper-parameters asHenderson et al. (2017): γ=0.995, GAE λ=0.97, policy clipping at ∆ρ t =0.2, and we alternate performing 2048 environment steps and 10 optimizer epochs with batch-size 64 on the obtained data. Both the policy-and the value-network are 2-layer MLPs with 64 units per layer. We further improved results by having separate learning rates (10

Table content: <table><tr><td colspan="2">Short-hand Description</td><td>Baseline value</td></tr><tr><td>C</td><td>Annealing parameter of c max .</td><td>4</td></tr><tr><td>N</td><td>Size of the Replay Memory.</td><td>2 19</td></tr><tr><td>N start</td><td>Number of samples gathered before training starts.</td><td>2 18</td></tr><tr><td>D</td><td>Fraction of far-policy samples allowed in the RM.</td><td>0.1</td></tr><tr><td>B</td><td>Mini-batch size.</td><td>256</td></tr><tr><td>F</td><td colspan="2">Ratio between observed time steps and gradient steps. 1</td></tr><tr><td>η</td><td>Learning rate.</td><td>10 -4</td></tr><tr><td>γ</td><td>Discount factor.</td><td>0.995</td></tr><tr><td colspan="2">update coefficient 0.01.</td><td/></tr></table>

Peer review (iclr2019):
 This paper first proposed a variant of experience replay to achieve better data efficiency in off-policy RL. The RACER algorithm was then developed, by modifying the approximated advantage function in the NAF algorithm. The proposed methods were finally tested on the MuJoCo environment to show the competitive performance. This paper is in general well written. The ideas look interesting, even though they are mostly small modification of the previous works. The experiments also show the promise of the proposed methods. One of my concerns is regarding the generality of ReF-ER. I am wondering if it can be also applied to the Atari domain to boost the performance there, similar to the prioritized experience replay paper. I understand that the requirement of GPUs is beyond the hardware configuration in this work, but that would be an important contribution to the community. My other questions and comments are as follows. - Regarding the parametric form of f^w in Eq. (7), what are the definitions for L_+ and L_-? What are the benefits of introducing min and max there, compared with the form in Eq. (11), as used in NAF? Does it cause any problems during optimization? - The y axis in Figure 3 is for KL (.pi || .mu), while the text below used KL(.mu || .pi) and the description regarding the change of C also seems to be inaccurate. - In Figure 4, do you have any explanation why using PER leads to worse performance for NAF? - For the implementation, did you use any parallelization to speed up the algorithm?