Off-Policy Actor-Critic with Shared Experience Replay

1 INTRODUCTION . Value-based and actor-critic policy gradient methods are the two leading techniques of constructing general and scalable reinforcement learning agents ( Sutton et al. , 2018 ) . Both have been combined with non-linear function approximation ( Tesauro , 1995 ; Williams , 1992 ) , and have achieved remarkable successes on multiple challenging domains ; yet , these algorithms still require large amounts of data to determine good policies for any new environment . To improve data efficiency , experience replay agents store experience in a memory buffer ( replay ) ( Lin , 1992 ) , and reuse it multiple times to perform reinforcement learning updates ( Riedmiller , 2005 ) . Experience replay allows to generalize prioritized sweeping ( Moore & Atkeson , 1993 ) to the non-tabular setting ( Schaul et al. , 2015 ) , and can also be used to simplify exploration by including expert ( e.g. , human ) trajectories ( Hester et al. , 2017 ) . Overall , experience replay can be very effective at reducing the number of interactions with the environment otherwise required by deep reinforcement learning algorithms ( Schaul et al. , 2015 ) . Replay is often combined with the value-based Q-learning ( Mnih et al. , 2015 ) , as it is an off-policy algorithm by construction , and can perform well even if the sampling distribution from replay is not aligned with the latest agent ’ s policy . Combining experience replay with actor-critic algorithms can be harder due to their on-policy nature . Hence , most established actor-critic algorithms with replay such as ( Wang et al. , 2017 ; Gruslys et al. , 2018 ; Haarnoja et al. , 2018 ) employ and maintain Q-functions to learn from the replayed off-policy experience . In this paper , we demonstrate that off-policy actor-critic learning with experience replay can be achieved without surrogate Q-function approximators using V-trace by employing the following approaches : a ) off-policy replay experience needs to be mixed with a proportion of on-policy experience . We show experimentally ( Figure 2 ) and theoretically that the V-trace policy gradient is otherwise not guaranteed to converge to a locally optimal solution . b ) a trust region scheme ( Conn et al. , 2000 ; Schulman et al. , 2015 ; 2017 ) can mitigate bias and enable efficient learning in a strongly off-policy regime , where distinct agents share experience through a commonly shared replay module . Sharing experience permits the agents to benefit from parallel exploration ( Kretchmar , 2002 ) ( Figures 1 and 3 ) . Our paper is structured as follows : In Section 2 we revisit pure importance sampling for actor-critic agents ( Degris et al. , 2012 ) and V-trace , which is notable for allowing to trade off bias and variance in its estimates . We recall that variance reduction is necessary ( Figure 4 left ) but is biased in V-trace . We derive proposition 2 stating that off-policy V-trace is not guaranteed to converge to a locally optimal solution – not even in an idealized scenario when provided with the optimal value function . Through theoretical analysis ( Section 3 ) and experimental validation ( Figure 2 ) we determine that mixing on-policy experience into experience replay alleviates the problem . Furthermore we propose a trust region scheme ( Conn et al. , 2000 ; Schulman et al. , 2015 ; 2017 ) in Section 4 that enables efficient learning even in a strongly off-policy regime , where distinct agents share the experience replay module and learn from each others experience . We define the trust region in policy space and prove that the resulting estimator is correct ( i.e . estimates an improved return ) . As a result , we present state-of-the-art data efficiency in Section 5 in terms of median human normalized performance across 57 Atari games ( Bellemare et al. , 2013 ) , as well as improved learning efficiency on DMLab30 ( Beattie et al. , 2016 ) ( Table 1 ) . 2 THE ISSUE WITH IMPORTANCE SAMPLING : BIAS AND VARIANCE IN V-TRACE . V-trace importance sampling is a popular off-policy correction for actor-critic agents ( Espeholt et al. , 2018 ) . In this section we revisit how V-trace controls the ( potentially infinite ) variance that arises from naive importance sampling . We note that this comes at the cost of a biased estimate ( see Proposition 1 ) and creates a failure mode ( see Proposition 2 ) which makes the policy gradient biased . We discuss our solutions for said issues in Section 4 . 2.1 REINFORCEMENT LEARNING . We follow the notation of Sutton et al . ( 2018 ) where an agent interacts with its environment , to collect rewards . On each discrete time-step t , the agent selects an action at ; it receives in return a reward rt and an observation ot+1 , encoding a partial view of the environment ’ s state st+1 . In the fully observable case , the RL problem is formalized as a Markov Decision Process ( Bellman , 1957 ) : a tuple ( S , A , p , γ ) , where S , A denotes finite sets of states and actions , p models rewards and state transitions ( so that rt , st+1 ∼ p ( st , at ) ) , and γ is a fixed discount factor . A policy is a mapping π ( a|s ) from states to action probabilities . The agent seeks an optimal policy π∗ that maximizes the value , defined as the expectation of the cumulative discounted returns Gt = ∑∞ k=0 γ krt+k . Off-policy learning is the problem of finding , or evaluating , a policy π from data generated by a different policy µ . This arises in several settings . Experience replay ( Lin , 1992 ) mixes data from multiple iterations of policy improvement . In large-scale RL , decoupling acting from learning ( Nair et al. , 2015 ; Horgan et al. , 2018 ; Espeholt et al. , 2018 ) causes the experience to lag behind the latest agent policy . Finally , it is often useful to learn multiple general value functions ( Sutton et al. , 2011 ; Mankowitz et al. , 2018 ; Lample & Chaplot , 2016 ; Mirowski et al. , 2017 ; Jaderberg et al. , 2017b ) or options ( Sutton et al. , 1999 ; Bacon et al. , 2017 ) from a single stream of experience . 2.2 NAIVE IMPORTANCE SAMPLING . On-policy n-step bootstraps give more accurate value estimates in expectation with larger n ( Sutton et al. , 2018 ) . They are used in many reinforcement learning agents ( Mnih et al. , 2016 ; Schulman et al. , 2017 ; Hessel et al. , 2017 ) . Unfortunately n must be chosen suitably as the estimates variance increases with n too . It is desirable to obtain benefits akin to n-step returns in the off-policy case . To this end multi-step importance sampling ( Kahn , 1955 ) can be used . This however adds another source of ( potentially infinite ( Sutton et al. , 2018 ) ) variance to the estimate . Importance sampling can estimate the expected return V π from trajectories sampled from µ 6= π , as long as µ is non-zero whereever π is . We employ a previously estimated value function V as a bootstrap to estimate expected returns . Following Degris et al . ( 2012 ) , a multi-step formulation of the expected return is V π ( st ) = Eµ [ V ( st ) + K−1∑ k=0 γk ( k∏ i=0 πt+i µt+i ) δt+kV ] ( 1 ) where Eµ denotes the expectation under policy µ up to an episode termination , δtV = rt + γV ( st+1 ) − V ( st ) is the temporal difference error in consecutive states st+1 , st , and πt = πt ( at|st ) . Importance sampling estimates can have high variance . Tree Backup ( Precup et al. , 2000 ) , and Q ( λ ) ( Sutton et al. , 2014 ) address this , but reduce the number of steps before bootstrapping even when this is undesirable ( as in the on-policy case ) . RETRACE ( Munos et al. , 2016 ) makes use of full returns in the on-policy case , but it introduces a zero-mean random variable at each step , adding variance to empirical estimates in both on- and off-policy cases . 2.3 BIAS-VARIANCE ANALYSIS & FAILURE MODE OF V-TRACE IMPORTANCE SAMPLING . V-trace ( Espeholt et al. , 2018 ) reduces the variance of importance sampling by trading off variance for a biased estimate of the return – resulting in a failure mode ( see Proposition 2 ) . It uses clipped importance sampling ratios to approximate V π by V π̃ ( st ) = V ( st ) + ∑K−1 k=0 γ k ( ∏k−1 i=0 ci ) ρtδt+kV where V is a learned state value estimate used to bootstrap , and ρt = min [ πt/µt , ρ̄ ] , ct = min [ πt/µt , c̄ ] are the clipped importance ratios . Note that , differently from RETRACE , V-trace fully recovers the Monte Carlo return when on policy . It similarly reweights the policy gradient as : ∇V π̃ ( st ) def = Eµ [ ρt∇ ( log πt ) ( rt + γV π̃ ( st+1 ) ) ] ( 2 ) Note that ∇V π̃ ( st ) recovers the naively importance sampled policy gradient for ρ̄ → ∞ . In the literature , it is common to subtract a baseline from the action-value estimate rt + γV π̃ ( st+1 ) to reduce variance ( Williams , 1992 ) , omitted here for simplicity . The constants ρ̄ ≥ c̄ ≥ 1 ( typically chosen ρ̄ = c̄ = 1 ) define the level of clipping , and improve stability by ensuring a bounded variance . For any given ρ̄ , the bias introduced by V-trace in the value and policy gradient estimates increases with the difference between π and µ . We analyze this in the following propositions . Proposition 1 . The V-trace value estimate V π̃ is biased : It does not match the expected return of π but the return of a related implied policy π̃ defined by equation 3 that depends on the behaviour policy µ : π̃µ ( a|x ) = min [ ρ̄µ ( a|x ) , π ( a|x ) ] ∑ b∈A min [ ρ̄µ ( b|x ) , π ( b|x ) ] ( 3 ) Proof . See Espeholt et al . ( 2018 ) . Note that the biased policy π̃µ can be very different from π . Hence the V-trace value estimate V π̃ may be very different from V π as well . As an illustrative example , consider two policies over a set of two actions , e.g . “ left ” and “ right ” represented as a tuple of probabilities . Let us investigate µ = ( φ , 1− φ ) and π = ( 1− φ , φ ) defined for any suitably small φ ≤ 1 . Observe that π and µ share no trajectories ( state-action sequences ) in the limit as φ→ 0 and they get more focused on one action . A practical example of this could be two policies , one almost always taking a left turn and one always taking the right . Given sufficient data of either policy it is possible to estimate the value of the other e.g . with naive importance sampling . However observe that V-trace with ρ̄ = 1 will always estimate a biased value - even given infinite data . Observe that min [ µ ( a|x ) , π ( a|x ) ] = min [ φ , 1− φ ] for both actions . Thus π̃µ is uniform rather than resembling π the policy . The V-trace estimate V π̃ would thus compute the average value of `` left '' and `` right '' – poorly representing the true V π . Proposition 2 . The V-trace policy gradient is biased : given the the optimal value function V ∗ the V-trace policy gradient does not converge to a locally optimal π∗ for all off-policy behaviour distributions µ . Proof . See Appendix C .

Summary: This paper aims to improve the efficiency of the actor-critic method. The authors first analyzed the cause of instability in the prior work, from the perspective of bias and variance. Two remedies were then presented: (i) mixing the experience replay with online learning; (ii) proposing a trust region scheme to select the behavior policies. The authors finally tested the proposed method on Atari games, and showed the better results, compared with the state-of-the-art methods.