Remember and Forget for Experience Replay

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has an ever increasing number of success stories ranging from realistic simulated environments ( Schulman et al. , 2015 ; Mnih et al. , 2016 ) , robotics ( Levine et al. , 2016 ) and games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . Experience Replay ( ER ) ( Lin , 1992 ) enhances deep RL algorithms by using information collected in past policy ( µ ) iterations to compute updates for the current policy ( π ) . ER has become one of the mainstay techniques to improve the sample-efficiency of off-policy deep RL . Sampling from a Replay Memory ( RM ) stabilizes stochastic gradient descent ( SGD ) by disrupting temporal correlations and extracts information from useful experiences over multiple updates ( Schaul et al. , 2015b ) . However , when π is parameterized by a Neural Network ( NN ) , SGD updates may result in significant changes to the policy , thereby shifting the distribution of states observed from the environment . In this case sampling the RM for further updates may lead to incorrect gradient estimates , therefore deep RL methods must account for and limit the dissimilarity between π and behaviors in the RM . Previous works employed trust region methods to bound policy updates ( Schulman et al. , 2015 ; Wang et al. , 2017 ) . Despite several successes , deep RL algorithms are known to suffer from instabilities and exhibit high-variance of outcomes ( Islam et al. , 2017 ; Henderson et al. , 2017 ) , especially continuous-action methods employing the stochastic ( Sutton et al. , 2000 ) or deterministic ( Silver et al. , 2014 ) Policy Gradients ( PG or DPG ) . In this work we redesign ER in order to control the distance between the behaviors µ used to compute the update and the current policy π . More specifically , we classify experiences either as “ near-policy '' or “ far-policy '' , depending on the ratio ρ of probabilities of selecting the associated action with π and that with µ . The weight ρ appears in many estimators that are used with ER such as the off-policy policy gradients ( off-PG ) ( Degris et al. , 2012 ) and the off-policy return-based evaluation algorithm Retrace ( Munos et al. , 2016 ) . From this classification , we introduce measures to limit the fraction of “ far-policy '' samples in the RM , as well as computing gradient estimates only from “ near-policy '' experiences . Furthermore , these hyper-parameters can be gradually annealed during training to obtain increasingly accurate updates from nearly on-policy experiences . Remember and Forget Experience Replay ( ReF-ER ) is a simple algorithm that can be applied to virtually any off-policy RL algorithm with parameterized policies . We show that ReF-ER allows better stability and performance than conventional ER in all three main classes of continuous-actions off-policy deep RL algorithms : methods based on the DPG ( ie . DDPG ( Lillicrap et al. , 2016 ) ) , methods based on Q-learning ( ie . NAF ( Gu et al. , 2016 ) ) , and with off-PG ( Degris et al. , 2012 ; Wang et al. , 2017 ) . In recent years , there has been a growing interest in coupling RL with high-fidelity physics simulations ( Reddy et al. , 2016 ; Novati et al. , 2017 ; Colabrese et al. , 2017 ; Verma et al. , 2018 ) . The computational cost of these simulations calls for data-efficient RL methods that are reliable and do not require problem-specific tweaks to the hyper-parameters . Moreover , while on-policy training of simple policy architectures has been shown to be sufficient in some benchmark environments ( Rajeswaran et al. , 2017 ) , agents aiming to solve complex problems with highly non-linear dynamics might require deep or recurrent models that can be trained more efficiently with off-policy methods . We address these issues by introducing a simple and computationally efficient off-policy actor-critic architecture ( RACER ) for continuous-action control problems . We systematically analyze a wide range of hyper-parameters on the OpenAI Gym ( Brockman et al. , 2016 ) robotic tasks , and show that RACER combined with ReF-ER reliably obtains results that are competitive with the state-of-the-art . 2 PRELIMINARIES . Consider the sequential decision process of an agent aiming to optimize its interaction with the environment . At each step t , the agent observes its state st ∈ RdS , performs an action by sampling a policy at ∼ µ ( a|st ) ∈ RdA , and transitions to a new state st+1 ∼ D ( s|at , st ) with reward rt+1 ∈ R. These interactions { st , at , rt , µt } are stored in a RM , which constitutes the data used by off-policy RL to train the parametric policy πw ( a|s ) . The importance weight ρt = πw ( at|st ) /µt is the ratio between the probability of selecting at with the current πw and with the behavior µt , which gradually becomes dissimilar from πw as the latter is trained . The on-policy state-action value Qπ ( s , a ) measures the expected returns from ( s , a ) following the policy πw : Qπ ( s , a ) = E st∼D , at∼πw [ ∞∑ t=0 γtrt+1 ∣∣∣∣∣ s0 = s , a0 = a ] ( 1 ) Here γ is a discount factor . The value of state s is the on-policy expectation : V π ( s ) = Ea∼π [ Qπ ( s , a ) ] and the action advantage is Aπ ( s , a ) = Qπ ( s , a ) −V π ( s ) , such that Ea∼π [ Aπ ( s , a ) ] : =0 . We consider algorithms that train parametric approximators Qw from off-policy data . The Q-learning target is q̂t = rt+1 + γ Ea′∼π Qw ( st+1 , a′ ) . The Retrace algorithm ( Munos et al. , 2016 ) includes all the rewards obtained by the behavior µt in the value estimation : Q̂rett = rt+1 + γV w ( st+1 ) + γmin { 1 , ρt+1 } [ Q̂rett+1 −Qw ( st+1 , at+1 ) ] ( 2 ) The off-PG ( Degris et al. , 2012 ) can be used to update πw by ER : ĝoff-PGt ( w ) = ρtÂ π t∇w log πw ( at|st ) . Here , Âπt is an estimator for the on-policy advantage , such as Â ret t : = Q̂ ret t − V w ( st ) . 3 REMEMBER AND FORGET EXPERIENCE REPLAY . In off-policy RL it is common to maximize on-policy returns averaged over the distribution of states in a RM ( Degris et al. , 2012 ) . However , as πw gradually shifts away from previous behaviors µt , the distribution of states in the RM is increasingly dissimilar from the on-policy distribution , and trying to increase an off-policy performance metric may not improve on-policy outcomes . This issue may be compounded with algorithm-specific concerns . For example , in ACER ( Wang et al. , 2017 ) the dissimilarity between µt and πw may cause vanishing or diverging importance weights ρt , thereby increasing the variance of the off-PG and deteriorating the convergence speed of Retrace by inducing “ trace-cutting ” ( Munos et al. , 2016 ) . As a remedy , ACER tunes the learning rate and uses a target network ( Mnih et al. , 2015 ) , updated as a delayed copy of the policy network , to constrain policy updates . Target networks are also employed in DDPG ( Lillicrap et al. , 2016 ) in order to slow down the feedback loop between value-network and policy-network optimizations . This feedback loop causes overestimated action values that can only be corrected by acquiring new on-policy samples . Recent works ( Henderson et al. , 2017 ) have shown the opaque variability of outcomes of continuous-action deep RL algorithms depending on hyper-parameters . Target-networks may be one of the sources of this unpredictability . In fact , when using deep approximators , there is no guarantee that the small weight changes imposed by target-networks correspond to small changes in the network ’ s output . Here we propose a set of three simple rules , collectively referred to as Remember and Forget ER ( ReF-ER ) , to directly control the degree of “ off-policyness ” of the samples in the RM : 1 . Updates are computed from mini-batches drawn uniformly from a RM . We compute the importance weight ρt of each sample and classify it as “ near-policy '' if 1/cmax < ρt < cmax with cmax > 1 . The samples with vanishing ( ρt < 1/cmax ) or exploding ( ρt > cmax ) importance weights are classified as “ far-policy '' . When computing off-policy estimators with finite batch-sizes , such as Q̂ret or the off-PG , “ far-policy '' samples may either be irrelevant or increase the variance . ReF-ER allows computing gradient estimates exclusively from “ near-policy '' samples . In order to efficiently approximate the number of far-policy samples in the RM , we store for each step its most recent ρt . 2 . When acquiring samples from the environment , older episodes with the highest fraction of far-policy samples are removed until the number nobs of samples in the RM is at most N . 3 . Policy updates are penalized in order to attract the current policy πw towards µt : ĝReF-ERt ( w ) = { β ĝt ( w ) − ( 1−β ) ∇DKL [ µt‖πw ( ·|st ) ] if 1/cmax < ρt < cmax − ( 1−β ) ∇DKL [ µt‖πw ( ·|st ) ] otherwise ( 3 ) Here DKL is the Kullback–Leibler divergence and the coefficient β ∈ [ 0 , 1 ] is updated after each gradient step such that a fixed fraction D ∈ ( 0 , 1 ) of the RM are far-policy samples : β ← { ( 1− η ) β if nfar/nobs > D ( 1− η ) β + η , otherwise ( 4 ) where η is the learning rate and nfar is the number of far-policy samples . Note that iteratively updating β with Eq . 4 has fixed points in 0 for nfar/nobs > D and in 1 otherwise . For cmax→1 and D→0 , ReF-ER becomes asymptotically equivalent to computing updates from on-policy samples . ReF-ER aims to reduce the sensitivity on the network architecture and hyperparameters by controlling the rate at which the policy can deviate from the behaviors in the RM . 4 RACER : REGULARIZED ACTOR-CRITIC WITH EXPERIENCE REPLAY This work analyzes ReF-ER with three types of continuous-actions off-policy deep RL algorithms : methods based on the DPG ( i.e . DDPG ) , based on Q-learning ( i.e . NAF ) , and based on the off-PG ( e.g . ACER ( Wang et al. , 2017 ) and IMPALA ( Espeholt et al. , 2018 ) ) . Here we introduce RACER , an off-PG based architecture with some improvements over ACER and IMPALA which aid our analysis of ReF-ER . First , RACER does not require expensive network architectures ( easing reproducibility and exploration of the hyper-parameters ) . Second , it samples time steps rather than episodes ( like DDPG and NAF and unlike ACER and IMPALA ) , further reducing its cost . Third , it does not introduce any technique that would interfere with ReF-ER and affect its analysis . Specifically , ACER uses the variance truncation and bias correction trick ( TBC ) , employs a target network to bound policy updates with a trust-region scheme , and modifies Retrace to use dA √ ρ instead of ρ . Lacking these techniques , we expect RACER to require ReF-ER to deal with unbounded importance weights . RACER ’ s network architecture consists of a single Multilayer Perceptron ( MLP ) with weights w which receives the state s and outputs as shown in Fig . 1b . From the final layer , the first 2dA outputs are taken as the mean m and diagonal covariance Σ of the Gaussian policy πw ( a|s ) . One output estimates the state value V w , and the remaining outputs are the parameters of a closed-form approximator for the action advantage Aw . RACER can be extended for problems where the state is received as a visual feed ( by inserting convolutional layers ) , or where the state is only partially observable ( by substituting the MLP with a recurrent model ) . However , these extensions are outside the scope of this paper . The parameters w are updated from mini-batches of off-policy time steps with ReF-ER . A separate gradient is defined for each component of the actor-critic framework , as sketched in figure 1b . The policy statistics m ( s ) and Σ ( s ) are updated with the off-PG ( Degris et al. , 2012 ) : ĝπt ( w ) = ρtÂ ret t ∇ { m , Σ } log πw ( at | st ) ( 5 ) We estimate the on-policy advantage with Retrace . From TBC and Retrace , we obtain an estimator for the on-policy state value V̂ tbct = V w ( st ) + min { 1 , ρt } ( Q̂rett −Qw ( st , at ) ) , used as a target for V w : ĝVt ( w ) = V̂ tbc ( s ) −V w ( st ) = min { 1 , ρt } [ Q̂rett −Qw ( st , at ) ] ( 6 ) Rather than having a separate MLP with inputs ( s , a ) to parameterize Qw ( as in ACER or DDPG ) , whose expected value under the policy would be computationally demanding to compute , we employ a closed-form equation for Aw inspired by NAF ( Gu et al. , 2016 ) . The network outputs the coefficients of a concave function fw ( s , a ) which is chosen such that its maximum coincides with the mean of the policy m ( s ) , and such that it is possible to derive analytical expectations for a ∼ πw . In App . B we explore multiple choices of fw ( s , a ) . For all results in the main text of the manuscript we used : fw ( s , a ) = K ( s ) exp [ −1 2 aᵀ+ L −1 + ( s ) a+ − 1 2 aᵀ− L −1 − ( s ) a− ] ( 7 ) Here a−= min [ a−m ( s ) ,0 ] and a+= max [ a−m ( s ) ,0 ] ( both element-wise operations ) . This parameterization requires one MLP output for K ( s ) and dA outputs for each diagonal matrix L+ and L− . In total , given a state s , the MLP computes m , Σ , V , K , L+ and L− . From Eq.7 , for any action a the advantage is uniquely defined : Aw ( s , a ) : = fw ( s , a ) −Ea′∼π [ fw ( s , a′ ) ] . Like the exact on-policy advantage Aπ , Aw has by design expectation zero under the policy . The parameterization coefficients K and L are updated to minimize the L2 error from Ârett : ĝAt ( w ) = ρt [ Ârett −Aw ( st , at ) ] ∇ { K , L } Aw ( st , at ) ( 8 ) Here , ρt reduces the weight of estimation errors for unlikely actions , where Aw is expected to be less accurate . To ensure that Σ , K , L+ and L− are positive definite , the respective outputs are mapped onto R+ by a Softplus rectifier . The analytical derivation of Ea′∼π [ fw ( s , a′ ) ] can be found in App . A . Collectively , ĝπt ( w ) , ĝ V t ( w ) , and ĝ A t ( w ) form a vector ĝ AC t ( w ) with the same size as the MLP output . ĝACt ( w ) is weighted with the ReF-ER penalization ( Eq . 3 ) and then back-propagated to the MLP . In order to estimate Q̂rett for a sampled time step t , Retrace ( Eq . 2 ) requires V w and Qw for all future steps in sample t ’ s episode . These are naturally computed when training from batches of episodes ( as in ACER ) rather than time steps . However , the information contained in consecutive steps is highly correlated , worsening the quality of the gradient estimate , and episodes can be composed of thousands of time steps , increasing the computational cost . In order to efficiently train from uncorrelated time steps , RACER stores for each sample in the RM the most recently computed estimations of V w ( st ) , Aw ( st , at ) , ρt and Qrett . When a batch of time steps is sampled , the stored Q ret t are used to compute the gradient update . At the same time , the stored V w ( st ) , Aw ( st , at ) and ρt are updated with the current NN outputs and used to correct Qret for all prior time-steps in the respective episodes with Eq . 2 . A description of all remaining implementation details is provided in App . D .

Summary: The authors introduce two new algorithms: remember and forget experience replay (ReF-ER), and an actor-critic architecture for continuous-action problems which is significantly more computationally efficient than previous approaches (RACER). ReF-ER manages the experience in the replay memory more directly and removes trajectories (episodes) that follow policies less related to the current parameterized policy (based on the importance weights). RACER's main contribution is provides a closed form approximation of the action values, enabling significant gains computationally. They provide several empirical studies in benchmark domains showing the competitiveness of their approach, and the provided more stability to various continuous control algorithms (NAF, PG, u-DDPG).