Off-Policy Actor-Critic with Shared Experience Replay

1 INTRODUCTION . Value-based and actor-critic policy gradient methods are the two leading techniques of constructing general and scalable reinforcement learning agents ( Sutton et al. , 2018 ) . Both have been combined with non-linear function approximation ( Tesauro , 1995 ; Williams , 1992 ) , and have achieved remarkable successes on multiple challenging domains ; yet , these algorithms still require large amounts of data to determine good policies for any new environment . To improve data efficiency , experience replay agents store experience in a memory buffer ( replay ) ( Lin , 1992 ) , and reuse it multiple times to perform reinforcement learning updates ( Riedmiller , 2005 ) . Experience replay allows to generalize prioritized sweeping ( Moore & Atkeson , 1993 ) to the non-tabular setting ( Schaul et al. , 2015 ) , and can also be used to simplify exploration by including expert ( e.g. , human ) trajectories ( Hester et al. , 2017 ) . Overall , experience replay can be very effective at reducing the number of interactions with the environment otherwise required by deep reinforcement learning algorithms ( Schaul et al. , 2015 ) . Replay is often combined with the value-based Q-learning ( Mnih et al. , 2015 ) , as it is an off-policy algorithm by construction , and can perform well even if the sampling distribution from replay is not aligned with the latest agent ’ s policy . Combining experience replay with actor-critic algorithms can be harder due to their on-policy nature . Hence , most established actor-critic algorithms with replay such as ( Wang et al. , 2017 ; Gruslys et al. , 2018 ; Haarnoja et al. , 2018 ) employ and maintain Q-functions to learn from the replayed off-policy experience . In this paper , we demonstrate that off-policy actor-critic learning with experience replay can be achieved without surrogate Q-function approximators using V-trace by employing the following approaches : a ) off-policy replay experience needs to be mixed with a proportion of on-policy experience . We show experimentally ( Figure 2 ) and theoretically that the V-trace policy gradient is otherwise not guaranteed to converge to a locally optimal solution . b ) a trust region scheme ( Conn et al. , 2000 ; Schulman et al. , 2015 ; 2017 ) can mitigate bias and enable efficient learning in a strongly off-policy regime , where distinct agents share experience through a commonly shared replay module . Sharing experience permits the agents to benefit from parallel exploration ( Kretchmar , 2002 ) ( Figures 1 and 3 ) . Our paper is structured as follows : In Section 2 we revisit pure importance sampling for actor-critic agents ( Degris et al. , 2012 ) and V-trace , which is notable for allowing to trade off bias and variance in its estimates . We recall that variance reduction is necessary ( Figure 4 left ) but is biased in V-trace . We derive proposition 2 stating that off-policy V-trace is not guaranteed to converge to a locally optimal solution – not even in an idealized scenario when provided with the optimal value function . Through theoretical analysis ( Section 3 ) and experimental validation ( Figure 2 ) we determine that mixing on-policy experience into experience replay alleviates the problem . Furthermore we propose a trust region scheme ( Conn et al. , 2000 ; Schulman et al. , 2015 ; 2017 ) in Section 4 that enables efficient learning even in a strongly off-policy regime , where distinct agents share the experience replay module and learn from each others experience . We define the trust region in policy space and prove that the resulting estimator is correct ( i.e . estimates an improved return ) . As a result , we present state-of-the-art data efficiency in Section 5 in terms of median human normalized performance across 57 Atari games ( Bellemare et al. , 2013 ) , as well as improved learning efficiency on DMLab30 ( Beattie et al. , 2016 ) ( Table 1 ) . 2 THE ISSUE WITH IMPORTANCE SAMPLING : BIAS AND VARIANCE IN V-TRACE . V-trace importance sampling is a popular off-policy correction for actor-critic agents ( Espeholt et al. , 2018 ) . In this section we revisit how V-trace controls the ( potentially infinite ) variance that arises from naive importance sampling . We note that this comes at the cost of a biased estimate ( see Proposition 1 ) and creates a failure mode ( see Proposition 2 ) which makes the policy gradient biased . We discuss our solutions for said issues in Section 4 . 2.1 REINFORCEMENT LEARNING . We follow the notation of Sutton et al . ( 2018 ) where an agent interacts with its environment , to collect rewards . On each discrete time-step t , the agent selects an action at ; it receives in return a reward rt and an observation ot+1 , encoding a partial view of the environment ’ s state st+1 . In the fully observable case , the RL problem is formalized as a Markov Decision Process ( Bellman , 1957 ) : a tuple ( S , A , p , γ ) , where S , A denotes finite sets of states and actions , p models rewards and state transitions ( so that rt , st+1 ∼ p ( st , at ) ) , and γ is a fixed discount factor . A policy is a mapping π ( a|s ) from states to action probabilities . The agent seeks an optimal policy π∗ that maximizes the value , defined as the expectation of the cumulative discounted returns Gt = ∑∞ k=0 γ krt+k . Off-policy learning is the problem of finding , or evaluating , a policy π from data generated by a different policy µ . This arises in several settings . Experience replay ( Lin , 1992 ) mixes data from multiple iterations of policy improvement . In large-scale RL , decoupling acting from learning ( Nair et al. , 2015 ; Horgan et al. , 2018 ; Espeholt et al. , 2018 ) causes the experience to lag behind the latest agent policy . Finally , it is often useful to learn multiple general value functions ( Sutton et al. , 2011 ; Mankowitz et al. , 2018 ; Lample & Chaplot , 2016 ; Mirowski et al. , 2017 ; Jaderberg et al. , 2017b ) or options ( Sutton et al. , 1999 ; Bacon et al. , 2017 ) from a single stream of experience . 2.2 NAIVE IMPORTANCE SAMPLING . On-policy n-step bootstraps give more accurate value estimates in expectation with larger n ( Sutton et al. , 2018 ) . They are used in many reinforcement learning agents ( Mnih et al. , 2016 ; Schulman et al. , 2017 ; Hessel et al. , 2017 ) . Unfortunately n must be chosen suitably as the estimates variance increases with n too . It is desirable to obtain benefits akin to n-step returns in the off-policy case . To this end multi-step importance sampling ( Kahn , 1955 ) can be used . This however adds another source of ( potentially infinite ( Sutton et al. , 2018 ) ) variance to the estimate . Importance sampling can estimate the expected return V π from trajectories sampled from µ 6= π , as long as µ is non-zero whereever π is . We employ a previously estimated value function V as a bootstrap to estimate expected returns . Following Degris et al . ( 2012 ) , a multi-step formulation of the expected return is V π ( st ) = Eµ [ V ( st ) + K−1∑ k=0 γk ( k∏ i=0 πt+i µt+i ) δt+kV ] ( 1 ) where Eµ denotes the expectation under policy µ up to an episode termination , δtV = rt + γV ( st+1 ) − V ( st ) is the temporal difference error in consecutive states st+1 , st , and πt = πt ( at|st ) . Importance sampling estimates can have high variance . Tree Backup ( Precup et al. , 2000 ) , and Q ( λ ) ( Sutton et al. , 2014 ) address this , but reduce the number of steps before bootstrapping even when this is undesirable ( as in the on-policy case ) . RETRACE ( Munos et al. , 2016 ) makes use of full returns in the on-policy case , but it introduces a zero-mean random variable at each step , adding variance to empirical estimates in both on- and off-policy cases . 2.3 BIAS-VARIANCE ANALYSIS & FAILURE MODE OF V-TRACE IMPORTANCE SAMPLING . V-trace ( Espeholt et al. , 2018 ) reduces the variance of importance sampling by trading off variance for a biased estimate of the return – resulting in a failure mode ( see Proposition 2 ) . It uses clipped importance sampling ratios to approximate V π by V π̃ ( st ) = V ( st ) + ∑K−1 k=0 γ k ( ∏k−1 i=0 ci ) ρtδt+kV where V is a learned state value estimate used to bootstrap , and ρt = min [ πt/µt , ρ̄ ] , ct = min [ πt/µt , c̄ ] are the clipped importance ratios . Note that , differently from RETRACE , V-trace fully recovers the Monte Carlo return when on policy . It similarly reweights the policy gradient as : ∇V π̃ ( st ) def = Eµ [ ρt∇ ( log πt ) ( rt + γV π̃ ( st+1 ) ) ] ( 2 ) Note that ∇V π̃ ( st ) recovers the naively importance sampled policy gradient for ρ̄ → ∞ . In the literature , it is common to subtract a baseline from the action-value estimate rt + γV π̃ ( st+1 ) to reduce variance ( Williams , 1992 ) , omitted here for simplicity . The constants ρ̄ ≥ c̄ ≥ 1 ( typically chosen ρ̄ = c̄ = 1 ) define the level of clipping , and improve stability by ensuring a bounded variance . For any given ρ̄ , the bias introduced by V-trace in the value and policy gradient estimates increases with the difference between π and µ . We analyze this in the following propositions . Proposition 1 . The V-trace value estimate V π̃ is biased : It does not match the expected return of π but the return of a related implied policy π̃ defined by equation 3 that depends on the behaviour policy µ : π̃µ ( a|x ) = min [ ρ̄µ ( a|x ) , π ( a|x ) ] ∑ b∈A min [ ρ̄µ ( b|x ) , π ( b|x ) ] ( 3 ) Proof . See Espeholt et al . ( 2018 ) . Note that the biased policy π̃µ can be very different from π . Hence the V-trace value estimate V π̃ may be very different from V π as well . As an illustrative example , consider two policies over a set of two actions , e.g . “ left ” and “ right ” represented as a tuple of probabilities . Let us investigate µ = ( φ , 1− φ ) and π = ( 1− φ , φ ) defined for any suitably small φ ≤ 1 . Observe that π and µ share no trajectories ( state-action sequences ) in the limit as φ→ 0 and they get more focused on one action . A practical example of this could be two policies , one almost always taking a left turn and one always taking the right . Given sufficient data of either policy it is possible to estimate the value of the other e.g . with naive importance sampling . However observe that V-trace with ρ̄ = 1 will always estimate a biased value - even given infinite data . Observe that min [ µ ( a|x ) , π ( a|x ) ] = min [ φ , 1− φ ] for both actions . Thus π̃µ is uniform rather than resembling π the policy . The V-trace estimate V π̃ would thus compute the average value of `` left '' and `` right '' – poorly representing the true V π . Proposition 2 . The V-trace policy gradient is biased : given the the optimal value function V ∗ the V-trace policy gradient does not converge to a locally optimal π∗ for all off-policy behaviour distributions µ . Proof . See Appendix C .

Summary: This paper investigates off-policy actor critic (AC) learning with experience replay using V-trace. It shows that V-trace policy gradient is not guaranteed to converge to a local optimal solution. To mitigate the bias and variance problem of V-trace and importance sampling, a trust region approach is proposed to adaptively selects only suitable behavior distributions when estimating the state-value of a policy. To this end, a behavior relevance function (KL divergence) is introduced to classify behavior as relevant. The proposed learning method LASER demonstrates the state-of-the-art data efficiency in Atari among agents trained up until 200M frames. In all, this paper is well motivated and technically sound. The draft can be improved by making it more self-contained by providing a sketch of the proof rather than refer everything to the appendix. Also it might be helpful to provide a pseudocode of LASER to help readers better understand the technical details.