Remember and Forget for Experience Replay

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has an ever increasing number of success stories ranging from realistic simulated environments ( Schulman et al. , 2015 ; Mnih et al. , 2016 ) , robotics ( Levine et al. , 2016 ) and games ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . Experience Replay ( ER ) ( Lin , 1992 ) enhances deep RL algorithms by using information collected in past policy ( µ ) iterations to compute updates for the current policy ( π ) . ER has become one of the mainstay techniques to improve the sample-efficiency of off-policy deep RL . Sampling from a Replay Memory ( RM ) stabilizes stochastic gradient descent ( SGD ) by disrupting temporal correlations and extracts information from useful experiences over multiple updates ( Schaul et al. , 2015b ) . However , when π is parameterized by a Neural Network ( NN ) , SGD updates may result in significant changes to the policy , thereby shifting the distribution of states observed from the environment . In this case sampling the RM for further updates may lead to incorrect gradient estimates , therefore deep RL methods must account for and limit the dissimilarity between π and behaviors in the RM . Previous works employed trust region methods to bound policy updates ( Schulman et al. , 2015 ; Wang et al. , 2017 ) . Despite several successes , deep RL algorithms are known to suffer from instabilities and exhibit high-variance of outcomes ( Islam et al. , 2017 ; Henderson et al. , 2017 ) , especially continuous-action methods employing the stochastic ( Sutton et al. , 2000 ) or deterministic ( Silver et al. , 2014 ) Policy Gradients ( PG or DPG ) . In this work we redesign ER in order to control the distance between the behaviors µ used to compute the update and the current policy π . More specifically , we classify experiences either as “ near-policy '' or “ far-policy '' , depending on the ratio ρ of probabilities of selecting the associated action with π and that with µ . The weight ρ appears in many estimators that are used with ER such as the off-policy policy gradients ( off-PG ) ( Degris et al. , 2012 ) and the off-policy return-based evaluation algorithm Retrace ( Munos et al. , 2016 ) . From this classification , we introduce measures to limit the fraction of “ far-policy '' samples in the RM , as well as computing gradient estimates only from “ near-policy '' experiences . Furthermore , these hyper-parameters can be gradually annealed during training to obtain increasingly accurate updates from nearly on-policy experiences . Remember and Forget Experience Replay ( ReF-ER ) is a simple algorithm that can be applied to virtually any off-policy RL algorithm with parameterized policies . We show that ReF-ER allows better stability and performance than conventional ER in all three main classes of continuous-actions off-policy deep RL algorithms : methods based on the DPG ( ie . DDPG ( Lillicrap et al. , 2016 ) ) , methods based on Q-learning ( ie . NAF ( Gu et al. , 2016 ) ) , and with off-PG ( Degris et al. , 2012 ; Wang et al. , 2017 ) . In recent years , there has been a growing interest in coupling RL with high-fidelity physics simulations ( Reddy et al. , 2016 ; Novati et al. , 2017 ; Colabrese et al. , 2017 ; Verma et al. , 2018 ) . The computational cost of these simulations calls for data-efficient RL methods that are reliable and do not require problem-specific tweaks to the hyper-parameters . Moreover , while on-policy training of simple policy architectures has been shown to be sufficient in some benchmark environments ( Rajeswaran et al. , 2017 ) , agents aiming to solve complex problems with highly non-linear dynamics might require deep or recurrent models that can be trained more efficiently with off-policy methods . We address these issues by introducing a simple and computationally efficient off-policy actor-critic architecture ( RACER ) for continuous-action control problems . We systematically analyze a wide range of hyper-parameters on the OpenAI Gym ( Brockman et al. , 2016 ) robotic tasks , and show that RACER combined with ReF-ER reliably obtains results that are competitive with the state-of-the-art . 2 PRELIMINARIES . Consider the sequential decision process of an agent aiming to optimize its interaction with the environment . At each step t , the agent observes its state st ∈ RdS , performs an action by sampling a policy at ∼ µ ( a|st ) ∈ RdA , and transitions to a new state st+1 ∼ D ( s|at , st ) with reward rt+1 ∈ R. These interactions { st , at , rt , µt } are stored in a RM , which constitutes the data used by off-policy RL to train the parametric policy πw ( a|s ) . The importance weight ρt = πw ( at|st ) /µt is the ratio between the probability of selecting at with the current πw and with the behavior µt , which gradually becomes dissimilar from πw as the latter is trained . The on-policy state-action value Qπ ( s , a ) measures the expected returns from ( s , a ) following the policy πw : Qπ ( s , a ) = E st∼D , at∼πw [ ∞∑ t=0 γtrt+1 ∣∣∣∣∣ s0 = s , a0 = a ] ( 1 ) Here γ is a discount factor . The value of state s is the on-policy expectation : V π ( s ) = Ea∼π [ Qπ ( s , a ) ] and the action advantage is Aπ ( s , a ) = Qπ ( s , a ) −V π ( s ) , such that Ea∼π [ Aπ ( s , a ) ] : =0 . We consider algorithms that train parametric approximators Qw from off-policy data . The Q-learning target is q̂t = rt+1 + γ Ea′∼π Qw ( st+1 , a′ ) . The Retrace algorithm ( Munos et al. , 2016 ) includes all the rewards obtained by the behavior µt in the value estimation : Q̂rett = rt+1 + γV w ( st+1 ) + γmin { 1 , ρt+1 } [ Q̂rett+1 −Qw ( st+1 , at+1 ) ] ( 2 ) The off-PG ( Degris et al. , 2012 ) can be used to update πw by ER : ĝoff-PGt ( w ) = ρtÂ π t∇w log πw ( at|st ) . Here , Âπt is an estimator for the on-policy advantage , such as Â ret t : = Q̂ ret t − V w ( st ) . 3 REMEMBER AND FORGET EXPERIENCE REPLAY . In off-policy RL it is common to maximize on-policy returns averaged over the distribution of states in a RM ( Degris et al. , 2012 ) . However , as πw gradually shifts away from previous behaviors µt , the distribution of states in the RM is increasingly dissimilar from the on-policy distribution , and trying to increase an off-policy performance metric may not improve on-policy outcomes . This issue may be compounded with algorithm-specific concerns . For example , in ACER ( Wang et al. , 2017 ) the dissimilarity between µt and πw may cause vanishing or diverging importance weights ρt , thereby increasing the variance of the off-PG and deteriorating the convergence speed of Retrace by inducing “ trace-cutting ” ( Munos et al. , 2016 ) . As a remedy , ACER tunes the learning rate and uses a target network ( Mnih et al. , 2015 ) , updated as a delayed copy of the policy network , to constrain policy updates . Target networks are also employed in DDPG ( Lillicrap et al. , 2016 ) in order to slow down the feedback loop between value-network and policy-network optimizations . This feedback loop causes overestimated action values that can only be corrected by acquiring new on-policy samples . Recent works ( Henderson et al. , 2017 ) have shown the opaque variability of outcomes of continuous-action deep RL algorithms depending on hyper-parameters . Target-networks may be one of the sources of this unpredictability . In fact , when using deep approximators , there is no guarantee that the small weight changes imposed by target-networks correspond to small changes in the network ’ s output . Here we propose a set of three simple rules , collectively referred to as Remember and Forget ER ( ReF-ER ) , to directly control the degree of “ off-policyness ” of the samples in the RM : 1 . Updates are computed from mini-batches drawn uniformly from a RM . We compute the importance weight ρt of each sample and classify it as “ near-policy '' if 1/cmax < ρt < cmax with cmax > 1 . The samples with vanishing ( ρt < 1/cmax ) or exploding ( ρt > cmax ) importance weights are classified as “ far-policy '' . When computing off-policy estimators with finite batch-sizes , such as Q̂ret or the off-PG , “ far-policy '' samples may either be irrelevant or increase the variance . ReF-ER allows computing gradient estimates exclusively from “ near-policy '' samples . In order to efficiently approximate the number of far-policy samples in the RM , we store for each step its most recent ρt . 2 . When acquiring samples from the environment , older episodes with the highest fraction of far-policy samples are removed until the number nobs of samples in the RM is at most N . 3 . Policy updates are penalized in order to attract the current policy πw towards µt : ĝReF-ERt ( w ) = { β ĝt ( w ) − ( 1−β ) ∇DKL [ µt‖πw ( ·|st ) ] if 1/cmax < ρt < cmax − ( 1−β ) ∇DKL [ µt‖πw ( ·|st ) ] otherwise ( 3 ) Here DKL is the Kullback–Leibler divergence and the coefficient β ∈ [ 0 , 1 ] is updated after each gradient step such that a fixed fraction D ∈ ( 0 , 1 ) of the RM are far-policy samples : β ← { ( 1− η ) β if nfar/nobs > D ( 1− η ) β + η , otherwise ( 4 ) where η is the learning rate and nfar is the number of far-policy samples . Note that iteratively updating β with Eq . 4 has fixed points in 0 for nfar/nobs > D and in 1 otherwise . For cmax→1 and D→0 , ReF-ER becomes asymptotically equivalent to computing updates from on-policy samples . ReF-ER aims to reduce the sensitivity on the network architecture and hyperparameters by controlling the rate at which the policy can deviate from the behaviors in the RM . 4 RACER : REGULARIZED ACTOR-CRITIC WITH EXPERIENCE REPLAY This work analyzes ReF-ER with three types of continuous-actions off-policy deep RL algorithms : methods based on the DPG ( i.e . DDPG ) , based on Q-learning ( i.e . NAF ) , and based on the off-PG ( e.g . ACER ( Wang et al. , 2017 ) and IMPALA ( Espeholt et al. , 2018 ) ) . Here we introduce RACER , an off-PG based architecture with some improvements over ACER and IMPALA which aid our analysis of ReF-ER . First , RACER does not require expensive network architectures ( easing reproducibility and exploration of the hyper-parameters ) . Second , it samples time steps rather than episodes ( like DDPG and NAF and unlike ACER and IMPALA ) , further reducing its cost . Third , it does not introduce any technique that would interfere with ReF-ER and affect its analysis . Specifically , ACER uses the variance truncation and bias correction trick ( TBC ) , employs a target network to bound policy updates with a trust-region scheme , and modifies Retrace to use dA √ ρ instead of ρ . Lacking these techniques , we expect RACER to require ReF-ER to deal with unbounded importance weights . RACER ’ s network architecture consists of a single Multilayer Perceptron ( MLP ) with weights w which receives the state s and outputs as shown in Fig . 1b . From the final layer , the first 2dA outputs are taken as the mean m and diagonal covariance Σ of the Gaussian policy πw ( a|s ) . One output estimates the state value V w , and the remaining outputs are the parameters of a closed-form approximator for the action advantage Aw . RACER can be extended for problems where the state is received as a visual feed ( by inserting convolutional layers ) , or where the state is only partially observable ( by substituting the MLP with a recurrent model ) . However , these extensions are outside the scope of this paper . The parameters w are updated from mini-batches of off-policy time steps with ReF-ER . A separate gradient is defined for each component of the actor-critic framework , as sketched in figure 1b . The policy statistics m ( s ) and Σ ( s ) are updated with the off-PG ( Degris et al. , 2012 ) : ĝπt ( w ) = ρtÂ ret t ∇ { m , Σ } log πw ( at | st ) ( 5 ) We estimate the on-policy advantage with Retrace . From TBC and Retrace , we obtain an estimator for the on-policy state value V̂ tbct = V w ( st ) + min { 1 , ρt } ( Q̂rett −Qw ( st , at ) ) , used as a target for V w : ĝVt ( w ) = V̂ tbc ( s ) −V w ( st ) = min { 1 , ρt } [ Q̂rett −Qw ( st , at ) ] ( 6 ) Rather than having a separate MLP with inputs ( s , a ) to parameterize Qw ( as in ACER or DDPG ) , whose expected value under the policy would be computationally demanding to compute , we employ a closed-form equation for Aw inspired by NAF ( Gu et al. , 2016 ) . The network outputs the coefficients of a concave function fw ( s , a ) which is chosen such that its maximum coincides with the mean of the policy m ( s ) , and such that it is possible to derive analytical expectations for a ∼ πw . In App . B we explore multiple choices of fw ( s , a ) . For all results in the main text of the manuscript we used : fw ( s , a ) = K ( s ) exp [ −1 2 aᵀ+ L −1 + ( s ) a+ − 1 2 aᵀ− L −1 − ( s ) a− ] ( 7 ) Here a−= min [ a−m ( s ) ,0 ] and a+= max [ a−m ( s ) ,0 ] ( both element-wise operations ) . This parameterization requires one MLP output for K ( s ) and dA outputs for each diagonal matrix L+ and L− . In total , given a state s , the MLP computes m , Σ , V , K , L+ and L− . From Eq.7 , for any action a the advantage is uniquely defined : Aw ( s , a ) : = fw ( s , a ) −Ea′∼π [ fw ( s , a′ ) ] . Like the exact on-policy advantage Aπ , Aw has by design expectation zero under the policy . The parameterization coefficients K and L are updated to minimize the L2 error from Ârett : ĝAt ( w ) = ρt [ Ârett −Aw ( st , at ) ] ∇ { K , L } Aw ( st , at ) ( 8 ) Here , ρt reduces the weight of estimation errors for unlikely actions , where Aw is expected to be less accurate . To ensure that Σ , K , L+ and L− are positive definite , the respective outputs are mapped onto R+ by a Softplus rectifier . The analytical derivation of Ea′∼π [ fw ( s , a′ ) ] can be found in App . A . Collectively , ĝπt ( w ) , ĝ V t ( w ) , and ĝ A t ( w ) form a vector ĝ AC t ( w ) with the same size as the MLP output . ĝACt ( w ) is weighted with the ReF-ER penalization ( Eq . 3 ) and then back-propagated to the MLP . In order to estimate Q̂rett for a sampled time step t , Retrace ( Eq . 2 ) requires V w and Qw for all future steps in sample t ’ s episode . These are naturally computed when training from batches of episodes ( as in ACER ) rather than time steps . However , the information contained in consecutive steps is highly correlated , worsening the quality of the gradient estimate , and episodes can be composed of thousands of time steps , increasing the computational cost . In order to efficiently train from uncorrelated time steps , RACER stores for each sample in the RM the most recently computed estimations of V w ( st ) , Aw ( st , at ) , ρt and Qrett . When a batch of time steps is sampled , the stored Q ret t are used to compute the gradient update . At the same time , the stored V w ( st ) , Aw ( st , at ) and ρt are updated with the current NN outputs and used to correct Qret for all prior time-steps in the respective episodes with Eq . 2 . A description of all remaining implementation details is provided in App . D .

Summary: This paper presents a method for forgetting and re-weighting experiences from a buffer during updates. It is well quantified experimentally and has some interesting tricks to improve performance in DDPG and other methods in continuous control which make use of a replay buffer. The authors also present another method “RACER” which makes use of this.