Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective

1 INTRODUCTION . A practical reinforcement learning ( RL ) algorithm is often in an actor-critic setting ( Lin , 1992 ; Precup et al. , 2000 ) where the policy ( actor ) generates actions and the Q/value function ( critic ) evaluates the policy ’ s performance . Under this setting , off-policy RL uses transitions sampled from a replay buffer to perform Q function updates , yielding a new policy π . Then , a finite-length trajectory under π is added to the buffer , and the process repeats . Notice that sampling from a replay buffer is an offline operation and that the growth of replay buffer is an online operation . This implies off-policy actor-critic RL lies between offline RL ( Yu et al. , 2020 ; Levine et al. , 2020 ) and onpolicy RL ( Schulman et al. , 2015 ; 2017 ) . From a bias-variance perspective , offline RL experiences large policy mismatch bias but low sampling variance , while on-policy RL has a low bias but high variance . Hence , with a careful choice of the sampling from its replay buffer , off-policy actor-critic RL may achieve a better bias-variance trade-off . This is the direction we explore this work . To reduce policy mismatch bias , off-policy RL employs importance sampling with the weight given by the probability ratio of the current to behavior policy ( the policy that samples the trajectories ) ( Precup et al. , 2000 ; Xie et al. , 2019 ; Schmitt et al. , 2020 ) . However , because the behavior policy is usually not given in practice , one can either estimate the probability ratio from the data ( Lazic et al. , 2020 ; Yang et al. , 2020 ; Sinha et al. , 2020 ) or use other reasonable quantities , such as the Bellman error ( Schaul et al. , 2016 ) , as the sampling weight . Even using a naive uniform sampling from the replay buffer , some off-policy actor-critic algorithms can achieve a nontrivial performance ( Haarnoja et al. , 2018 ; Fujimoto et al. , 2018 ) . These observations suggest we need to better understand the success of off-policy actor-critic algorithms , especially in practical situations where a fixed behavior policy is unavailable . Our contributions are as follows . To understand the actor-critic setting without a fixed behavior policy , we construct a non-stationary policy that generates the averaged occupancy measure . We use this policy as a reference and show the policy evaluation error in an off-policy actor-critic setting decomposes into the Bellman error , the policy mismatch bias , and the variance from sampling . Since supervised learning during the Q function update only controls the Bellman error , we need careful sampling strategies to mitigate bias and variance . We show that the 1/age weighting or its variants like the Emphasizing Recent Experience ( ERE ) strategy ( Wang & Ross , 2019 ) are preferable because both their biases and variances are smaller than that of uniform weighting . To ensure the applicability of our explanation to practical off-policy actor-critic algorithms , we adopt weak but verifiable assumptions such as Lipschitz Q functions , bounded rewards , and a bounded action space . We avoid strong assumptions such as a fixed well-explored behavior policy , concentration coefficients , bounded probability ratios ( e.g. , ratios of current to behavior policy ) , and tabular or linear function approximation . Hence our analysis is more applicable to practical settings . In addition , our analysis suggests that 1/age-based samplings ( e.g. , ERE , ERE apx , 1/age ) have advantages in bias and variance . Our experiments verify that SAC ( Haarnoja et al. , 2018 ) with 1/age-based samplings outperform the prior work . Thus , our results not only provide theoretical foundations for practical off-policy actor-critic RL algorithms but also achieve better performances . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . Consider an infinite-horizon Markov Decision Process ( MDP ) 〈S , A , T , r , γ〉 , where S , A are finite-dimensional continuous state and action spaces , r ( s , a ) is a deterministic reward function , γ ∈ ( 0 , 1 ) is the discount factor , and T ( s′|s , a ) is the state transition density ; i.e. , the density of the next state s′ given the current state and action ( s , a ) . Given an initial state distribution ρ0 , the objective of RL is to find a policy π that maximizes the γ-discounted cumulative reward when the actions along the trajectory follow π : max π J ( π ) = max π E [ ∞∑ i=0 γir ( si , ai ) ∣∣∣s0 ∼ ρ0 , ai ∼ π ( ·|si ) , si+1 ∼ T ( ·|si , ai ) ] . ( 1 ) Let ρi ( s|ρ0 , π , T ) be the state distribution under π at trajectory step i . Define the normalized state occupancy measure by ρπρ0 ( s ) , ( 1− γ ) ∞∑ i=0 γiρi ( s|ρ0 , π , T ) . ( 2 ) Ideally , the maximization in ( 1 ) is achieved using a parameterized policy πθ and policy gradient updates with ( Sutton et al. , 1999 ; Silver et al. , 2014 ) : ∇θJ ( πθ ) = ( 1− γ ) −1E ( s , a ) ∼ρπθρ0 [ ( ∇θ log πθ ( a|s ) ) Q πθ ( s , a ) ] . ( 3 ) Here ρπθρ0 ( s , a ) = ρ πθ ρ0 ( s ) πθ ( a|s ) is given in ( 2 ) , and Q πθ is the Q function under policy πθ : Qπ ( s , a ) = E [ ∞∑ i=0 γir ( si , ai ) ∣∣∣s0 = s , a0 = a , ai ∼ π ( ·|si ) , si+1 ∼ T ( ·|si , ai ) ] . ( 4 ) Off-policy RL estimates Qπ by approximating the solution of the Bellman fixed-point equation : ( BπQπ ) ( s , a ) = r ( s , a ) + γEs′∼T ( ·|s , a ) , a′∼π ( a|s ) Qπ ( s′ , a′ ) = Qπ ( s , a ) . It is well-known that Qπ is the unique fixed point of the Bellman operator Bπ . Hence if ( BπQ̂ ) ( s , a ) ≈ Q̂ ( s , a ) , then Q̂ may be a “ good ” estimate of Qπ . In the next subsection , we will see that an off-policy actor-critic algorithm encourages this to hold for the replay buffer . 2.2 OFF-POLICY ACTOR-CRITIC ALGORITHM . We study an off-policy actor-critic algorithm of the form shown in Alg . 1 . In line 2 , for episode index e , πe samples one trajectory of length L. The transitions in this trajectory are then added to the replay buffer . Since the policies for different episodes are distinct , the collection of transitions in the replay buffer are generally inconsistent with the current policy . Notice the trajectory is simply collected by πe , not a perturbed version of πe ( e.g. , gaussian corrupted version of pie ) . Line 3 is a supervised learning that minimizes the Bellman error of Q̂ using gradient descents , making ( BπeQ̂ ) ( s , a ) ≈ Q̂ ( s , a ) for ( s , a ) in the replay buffer . When this holds , we will prove that the “ distance ” between Q̂ and Qπ e become smaller over the replay buffer . This is a crucial step for line 4 to be truly useful . In line 4 , Qπ e is replaced by Q̂ , and the policy is updated accordingly . In practice , line 3 is replaced by mini-batch updates ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018 ) where the summation of J ( Q̂φ ) is approximated by a sum over the mini-batches . In section 5.2 , we show that a uniform-weighted mini-batch sampling biases learning towards older samples ; this motivates the need for a countermeasure . Algorithm 1 Off-policy Actor-critic Algorithm Require : Parameterized policy πe = πθe and Q function estimate Q̂φ . Learning rate α . 1 : for episode e = 1 , 2 , ... do 2 : Sample a length-L trajectory { ( sei , aei ) } L−1 i=0 with initial distribution ρ0 ( s ) and policy π e ( a|s ) . 3 : Evaluate the Bellman error as J ( Q̂φ ) = ∑e j=1 ∑L−1 i=0 ( Q̂φ ( s j i , a j i ) − ( Bπ e Q̂φ ) ( s j i , a j i ) ) 2 and update Q̂φ as φ← φ− α∇φJ ( Q̂φ ) . 4 : Approximate Eq . ( 3 ) as ∇θĴ ( πθ ) = ( 1 − γ ) −1E ( s , a ) ∼ρπθρ0 [ ( ∇θ log πθ ( a|s ) ) Q̂ ( s , a ) ] and update as θe+1 = θe + α∇θĴ ( πθ ) . 5 : end for Note that Alg . 1 seems to be inconsistent in the horizon because the Q function , Eq . ( 4 ) , is defined in the infinite horizon but the trajectories are finite-length . In Corollary 1 , we will address this inconsistency by approximating the Q function using finite-length trajectories . 2.3 THE CONSTRUCTION OF OUR BEHAVIOR POLICY . Although Alg . 1 doesn ’ t have a fixed behavior policy , in Lemma 1 , we construct a non-stationary policy that generates the averaged occupancy measure at every step . This describes the averagedover-episode behavior at every trajectory step i of the historical trajectory { ( sei , aei ) } L−1 , N i , e=0,1 . We hence define it to be our behavior policy and use it as a reference when analyzing Alg . 1 . The construction is as follows . Denote the state distribution at trajectory step i in episode e as ρei ( s ) = ρi ( s|ρ0 , πe , T ) . By Eq . ( 2 ) , ρπ e ρ0 is the state occupancy measure generated by ( ρ0 , π e , T ) . Intuitively , ρπ e ρ0 describes the discounted state distribution starting at trajectory step 0 in episode e. More generally , define ρπ e ρei as the state occupancy measure generated by ( ρei , π e , T ) . Then , ρπ e ρei describes the discounted state distribution starting at trajectory step i in episode e. ρπ e ρei ( s ) , ( 1− γ ) ∞∑ j=i γj−iρej ( s ) ( 5 ) Behavior policy through averaging . Since Alg . 1 considers trajectories from all episodes , the average-over-episodes distribution % ( i ) = N−1 ∑ e ρ πe ρei will be of interest . Namely , % ( i ) is the average of all occupancy measures starting at step i . To describe the average-over-episodes behavior at step i , we want a policy that generates % ( i ) , and Eq . ( 5 ) helps construct such a policy . Concretely , let ρi , π D i ( a|s ) be the averaged state distribution and the averaged policy at step i , respectively . % ( i ) ( s ) , N−1 N∑ e=1 ρπ e ρei ( s ) , ρi ( s ) , N −1 N∑ e=1 ρei ( s ) , π D i ( a|s ) , ∑N e=1 π e ( a|s ) ρπeρei ( s ) ∑N e=1 ρ πe ρei ( s ) . ( 6 ) Lemma 1 shows πDi in Eq . ( 6 ) is a notion of behavior policy in the sense that ( ρi , π D i , T ) generates % ( i ) ; i.e. , πDi generates the averaged occupancy measures when the initial state follows ρi . Since π D i describes the averaged discounted behavior starting at step i , we define it to be our behavior policy . This is a key to analyzing the policy evaluation error in Alg . 1 . Lemma 1 . Let ρπ D i ρi ( s ) be the normalized state occupancy measure generated by ( ρi , π D i , T ) . Then % ( i ) ( s ) = ρ πDi ρi ( s ) a.e . and hence from Eq . ( 6 ) , N−1 ∑N e=1 ρ πe ρei ( s ) πe ( a|s ) = ρπ D i ρi ( s ) πDi ( a|s ) a.e .

Summary: The paper studies the policy evaluation error in off-policy actor critic algorithms. The core idea is to decompose the error into three terms: bellman error, policy mismatch bias and sampling variance. Based on the decomposition, the paper justifies ERE, a recently proposed replay sampling heuristic and empirically studies the properties of a number of other sampling methods.