Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective

1 INTRODUCTION . A practical reinforcement learning ( RL ) algorithm is often in an actor-critic setting ( Lin , 1992 ; Precup et al. , 2000 ) where the policy ( actor ) generates actions and the Q/value function ( critic ) evaluates the policy ’ s performance . Under this setting , off-policy RL uses transitions sampled from a replay buffer to perform Q function updates , yielding a new policy π . Then , a finite-length trajectory under π is added to the buffer , and the process repeats . Notice that sampling from a replay buffer is an offline operation and that the growth of replay buffer is an online operation . This implies off-policy actor-critic RL lies between offline RL ( Yu et al. , 2020 ; Levine et al. , 2020 ) and onpolicy RL ( Schulman et al. , 2015 ; 2017 ) . From a bias-variance perspective , offline RL experiences large policy mismatch bias but low sampling variance , while on-policy RL has a low bias but high variance . Hence , with a careful choice of the sampling from its replay buffer , off-policy actor-critic RL may achieve a better bias-variance trade-off . This is the direction we explore this work . To reduce policy mismatch bias , off-policy RL employs importance sampling with the weight given by the probability ratio of the current to behavior policy ( the policy that samples the trajectories ) ( Precup et al. , 2000 ; Xie et al. , 2019 ; Schmitt et al. , 2020 ) . However , because the behavior policy is usually not given in practice , one can either estimate the probability ratio from the data ( Lazic et al. , 2020 ; Yang et al. , 2020 ; Sinha et al. , 2020 ) or use other reasonable quantities , such as the Bellman error ( Schaul et al. , 2016 ) , as the sampling weight . Even using a naive uniform sampling from the replay buffer , some off-policy actor-critic algorithms can achieve a nontrivial performance ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) . These observations suggest we need to better understand the success of off-policy actor-critic algorithms , especially in practical situations where a fixed behavior policy is unavailable . Our contributions are as follows . To understand the actor-critic setting without a fixed behavior policy , we construct a non-stationary policy that generates the averaged occupancy measure . We use this policy as a reference and show the policy evaluation error in an off-policy actor-critic setting decomposes into the Bellman error , the policy mismatch bias , and the variance from sampling . Since supervised learning during the Q function update only controls the Bellman error , we need careful sampling strategies to mitigate bias and variance . We show that the 1/age weighting or its variants like the Emphasizing Recent Experience ( ERE ) strategy ( Wang & Ross , 2019 ) are preferable because both their biases and variances are smaller than that of uniform weighting . To ensure the applicability of our explanation to practical off-policy actor-critic algorithms , we adopt weak but verifiable assumptions such as Lipschitz Q functions , bounded rewards , and a bounded action space . We avoid strong assumptions such as a fixed well-explored behavior policy , concentration coefficients , bounded probability ratios ( e.g. , ratios of current to behavior policy ) , and tabular or linear function approximation . Hence our analysis is more applicable to practical settings . In addition , our analysis suggests that 1/age-based samplings ( e.g. , ERE , ERE apx , 1/age ) have advantages in bias and variance . Our experiments verify that SAC ( Haarnoja et al. , 2018 ) with 1/age-based samplings outperform the prior work . Thus , our results not only provide theoretical foundations for practical off-policy actor-critic RL algorithms but also achieve better performances . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . Consider an infinite-horizon Markov Decision Process ( MDP ) 〈S , A , T , r , γ〉 , where S , A are finite-dimensional continuous state and action spaces , r ( s , a ) is a deterministic reward function , γ ∈ ( 0 , 1 ) is the discount factor , and T ( s′|s , a ) is the state transition density ; i.e. , the density of the next state s′ given the current state and action ( s , a ) . Given an initial state distribution ρ0 , the objective of RL is to find a policy π that maximizes the γ-discounted cumulative reward when the actions along the trajectory follow π : max π J ( π ) = max π E [ ∞∑ i=0 γir ( si , ai ) ∣∣∣s0 ∼ ρ0 , ai ∼ π ( ·|si ) , si+1 ∼ T ( ·|si , ai ) ] . ( 1 ) Let ρi ( s|ρ0 , π , T ) be the state distribution under π at trajectory step i . Define the normalized state occupancy measure by ρπρ0 ( s ) , ( 1− γ ) ∞∑ i=0 γiρi ( s|ρ0 , π , T ) . ( 2 ) Ideally , the maximization in ( 1 ) is achieved using a parameterized policy πθ and policy gradient updates with ( Sutton et al. , 1999 ; Silver et al. , 2014 ) : ∇θJ ( πθ ) = ( 1− γ ) −1E ( s , a ) ∼ρπθρ0 [ ( ∇θ log πθ ( a|s ) ) Q πθ ( s , a ) ] . ( 3 ) Here ρπθρ0 ( s , a ) = ρ πθ ρ0 ( s ) πθ ( a|s ) is given in ( 2 ) , and Q πθ is the Q function under policy πθ : Qπ ( s , a ) = E [ ∞∑ i=0 γir ( si , ai ) ∣∣∣s0 = s , a0 = a , ai ∼ π ( ·|si ) , si+1 ∼ T ( ·|si , ai ) ] . ( 4 ) Off-policy RL estimates Qπ by approximating the solution of the Bellman fixed-point equation : ( BπQπ ) ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) , a′∼π ( a|s ) Qπ ( s′ , a′ ) = Qπ ( s , a ) . It is well-known that Qπ is the unique fixed point of the Bellman operator Bπ . Hence if ( BπQ̂ ) ( s , a ) ≈ Q̂ ( s , a ) , then Q̂ may be a “ good ” estimate of Qπ . In the next subsection , we will see that an off-policy actor-critic algorithm encourages this to hold for the replay buffer . 2.2 OFF-POLICY ACTOR-CRITIC ALGORITHM . We study an off-policy actor-critic algorithm of the form shown in Alg . 1 . In line 2 , for episode index e , πe samples one trajectory of length L. The transitions in this trajectory are then added to the replay buffer . Since the policies for different episodes are distinct , the collection of transitions in the replay buffer are generally inconsistent with the current policy . Notice the trajectory is simply collected by πe , not a perturbed version of πe ( e.g. , gaussian corrupted version of pie ) . Line 3 is a supervised learning that minimizes the Bellman error of Q̂ using gradient descents , making ( BπeQ̂ ) ( s , a ) ≈ Q̂ ( s , a ) for ( s , a ) in the replay buffer . When this holds , we will prove that the “ distance ” between Q̂ and Qπ e become smaller over the replay buffer . This is a crucial step for line 4 to be truly useful . In line 4 , Qπ e is replaced by Q̂ , and the policy is updated accordingly . In practice , line 3 is replaced by mini-batch updates ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) where the summation of J ( Q̂φ ) is approximated by a sum over the mini-batches . In section 5.2 , we show that a uniform-weighted mini-batch sampling biases learning towards older samples ; this motivates the need for a countermeasure . Algorithm 1 Off-policy Actor-critic Algorithm Require : Parameterized policy πe = πθe and Q function estimate Q̂φ . Learning rate α . 1 : for episode e = 1 , 2 , ... do 2 : Sample a length-L trajectory { ( sei , aei ) } L−1 i=0 with initial distribution ρ0 ( s ) and policy π e ( a|s ) . 3 : Evaluate the Bellman error as J ( Q̂φ ) = ∑e j=1 ∑L−1 i=0 ( Q̂φ ( s j i , a j i ) − ( Bπ e Q̂φ ) ( s j i , a j i ) ) 2 and update Q̂φ as φ← φ− α∇φJ ( Q̂φ ) . 4 : Approximate Eq . ( 3 ) as ∇θĴ ( πθ ) = ( 1 − γ ) −1E ( s , a ) ∼ρπθρ0 [ ( ∇θ log πθ ( a|s ) ) Q̂ ( s , a ) ] and update as θe+1 = θe + α∇θĴ ( πθ ) . 5 : end for Note that Alg . 1 seems to be inconsistent in the horizon because the Q function , Eq . ( 4 ) , is defined in the infinite horizon but the trajectories are finite-length . In Corollary 1 , we will address this inconsistency by approximating the Q function using finite-length trajectories . 2.3 THE CONSTRUCTION OF OUR BEHAVIOR POLICY . Although Alg . 1 doesn ’ t have a fixed behavior policy , in Lemma 1 , we construct a non-stationary policy that generates the averaged occupancy measure at every step . This describes the averagedover-episode behavior at every trajectory step i of the historical trajectory { ( sei , aei ) } L−1 , N i , e=0,1 . We hence define it to be our behavior policy and use it as a reference when analyzing Alg . 1 . The construction is as follows . Denote the state distribution at trajectory step i in episode e as ρei ( s ) = ρi ( s|ρ0 , πe , T ) . By Eq . ( 2 ) , ρπ e ρ0 is the state occupancy measure generated by ( ρ0 , π e , T ) . Intuitively , ρπ e ρ0 describes the discounted state distribution starting at trajectory step 0 in episode e. More generally , define ρπ e ρei as the state occupancy measure generated by ( ρei , π e , T ) . Then , ρπ e ρei describes the discounted state distribution starting at trajectory step i in episode e. ρπ e ρei ( s ) , ( 1− γ ) ∞∑ j=i γj−iρej ( s ) ( 5 ) Behavior policy through averaging . Since Alg . 1 considers trajectories from all episodes , the average-over-episodes distribution % ( i ) = N−1 ∑ e ρ πe ρei will be of interest . Namely , % ( i ) is the average of all occupancy measures starting at step i . To describe the average-over-episodes behavior at step i , we want a policy that generates % ( i ) , and Eq . ( 5 ) helps construct such a policy . Concretely , let ρi , π D i ( a|s ) be the averaged state distribution and the averaged policy at step i , respectively . % ( i ) ( s ) , N−1 N∑ e=1 ρπ e ρei ( s ) , ρi ( s ) , N −1 N∑ e=1 ρei ( s ) , π D i ( a|s ) , ∑N e=1 π e ( a|s ) ρπeρei ( s ) ∑N e=1 ρ πe ρei ( s ) . ( 6 ) Lemma 1 shows πDi in Eq . ( 6 ) is a notion of behavior policy in the sense that ( ρi , π D i , T ) generates % ( i ) ; i.e. , πDi generates the averaged occupancy measures when the initial state follows ρi . Since π D i describes the averaged discounted behavior starting at step i , we define it to be our behavior policy . This is a key to analyzing the policy evaluation error in Alg . 1 . Lemma 1 . Let ρπ D i ρi ( s ) be the normalized state occupancy measure generated by ( ρi , π D i , T ) . Then % ( i ) ( s ) = ρ πDi ρi ( s ) a.e . and hence from Eq . ( 6 ) , N−1 ∑N e=1 ρ πe ρei ( s ) πe ( a|s ) = ρπ D i ρi ( s ) πDi ( a|s ) a.e .

Summary: The paper seeks to improve our understanding of actor-critic methods in deep RL by providing an analysis of the error in estimating the value function. The main theoretical result is an error bound which depends on the Bellman error, variance from sampling and a bias term from the policy mismatch. This error bound suggests that a replay buffer sampling strategy that focuses on recent samples will have an advantage. Experiments validate this hypothesis and using weighted sampling with 1/age weights improves performance. Morover, the theory helps explain the success of the ERE algorithm which also emphasizes recent transitions.