Adaptive N-step Bootstrapping with Off-policy Data

1 INTRODUCTION . The goal of reinforcement learning ( RL ) is to find an optimal policy by interacting with the environment . In order to do that , a RL algorithm needs to define a target , e.g. , Q function or value function , and update it iteratively to bootstrap from scratch . The challenge of designing an efficient update target manifests both on the sample complexity and computation complexity . Ideally , the target should only be updated by the data generated by the corresponding policy to obtain an unbiased estimate ( Sutton & Barto , 2018 ) , while the amount of the data needs to reach a certain scale to control the variance ( Schulman et al. , 2015 ) . These two requirements limit the update frequency and lead to a high sample complexity finally . On the computational part , the consideration is to make a trade-off between the computation cost of each step and the number of total steps . Monte-Carlo returns has advantages on generalization ( behave well with function approximation ) and exploration ( a quick propagation of new findings ) at the cost of computing the whole trajectory on each step ( Sutton & Barto , 2018 ) . Bootstrapping methods apply readily on off-policy data and control the trace length ( Sutton & Barto , 2018 ) . However , they require more update steps to converge comparing with Monte-Carlo returns . Those design concerns are nested together , which makes it hard to achieve a good balance . N-step returns ( Sutton & Barto , 2018 ) serves as the basis of various update targets , due to its flexibility and simple implementation . Together with off-policy learning ( Sutton & Barto , 2018 ) and the replay buffer ( Mnih et al. , 2015 ) , n-step returns is able to update the target frequently while ensures that the variance is in a controllable range . However , a systematical study ( Fedus et al. , 2020 ) reveals that the performance of n-step returns highly relies on the exact value of n. Since the underlying working mechanism is unclear , previous research can only give some vague suggestion based on empirical results , that simply increases the value from one to a larger number , e.g . 3 or 4 . In this paper , we illustrate that the estimation error of n-step returns can be decomposed into offpolicy bias ( under-estimation part ) and approximation error ( over-estimation part ) , and the selection of n controls the balance between them . Data stored in the replay buffer are generated by previous policies . Thus , adopting them for update introduces the off-policy bias . Since the current policy is better than previous policies , the off-policy bias is an underestimation . Replay buffer is not the only source of the off-policy bias , epsilon-greedy exploration also introduces the off-policy issue . On the other hand , n-step returns adopts a max operator explicitly ( on Q-learning based algorithms ) or implicitly ( on actor-critic algorithms ) on an existing function to approximate the real target . The max operator brings the approximation error , which is an overestimation . Sec 4 gives the formal definition of the decomposition and verifies the conclusion by experiments . According to our analysis , the quantity of the off-policy bias and approximation error varies a lot on different data points . Thus , a fixed value of n is just a rough average , and there is plenty of room for improvement . We introduce a new metric , policy age , to quantify the off-policyness of each data point . As the policy age grows , the off-policy bias increases linearly , while the approximation error decreases exponentially . Based on this observation , we propose a novel algorithm , named adaptive n-step bootstrapping . Given the policy age of each data point , adaptive n-step calculates the optimal n by an exponential function . Hyperparameter of the function is determined by the tree-structured parzen estimator ( Bergstra et al. , 2011 ) . We conduct extensive experiments on both MuJoCo and Atari games . Adaptive n-step bootstrapping outperforms all fixed-value n settings with a large margin , in terms of both data efficiency and final reward . For the other update target definitions , we select Retrace ( Munos et al. , 2016 ) as a representative . Compared with Retrace , our method maintains the performance advantage under the premise of low computational complexity and simple implementation . 2 RELATED WORKS . 2.1 RESEARCH ON N-STEP RETURNS . The recent works on n-step returns focus on finding the optimal value of n. In Ape-X ( Horgan et al. , 2018 ) and R2D2 ( Kapturowski et al. , 2019 ) , the value of n is fixed , which is set by manual tuning or hyper-parameter search . Rainbow ( Hessel et al. , 2018 ) figures out that the final performance is sensitive to the value of n , and n = 3 achieves the best score in most cases on Atari games . Fedus et al . ( 2020 ) verifies that setting n to 3 is a good choice , and further reveals that the replay buffer must also be large enough to gain performance benefits . Those researches give some heuristic rules of setting the value of n , but the underlying mechanism of why n = 3 performs better than one-step temporal difference is still unclear . 2.2 OTHER UPDATE TARGETS . To improve the performance of the vanilla n-step returns , there are many other update target definitions in the literature ( Hernandez-Garcia & Sutton , 2019 ) . Importance sampling ( IS ) ( Precup et al. , 2000 ) provides a simple way to correct the off-policy bias . It can be seen as a weighted average of multiple one-step TD ( 0 ) target . However , IS brings large ( and possibly infinite ) variance , which makes it impractical on large-scale problems . Retrace ( Munos et al. , 2016 ) clips the IS ratio to a maximum value of 1 to reduce the large variance of IS targets . It has many applications in recent reinforcement learning agents , like distributed offpolicy learning agent Reactor ( Gruslys et al. , 2017 ) . The most serious disadvantage of Retrace is its high computation cost . Retrace needs to calculate O ( n ) times of Q and O ( n ) times of π in each time , compared with only O ( 1 ) from n-step returns ( n is trace length ) . In large-scale problems , evaluating Q and π requires a forward pass in the neural network , which is slow and expensive . Reactor ( Gruslys et al. , 2017 ) calculates the Retrace target as a linear combination of many n-step targets and dispatches those calculation workloads into different nodes for acceleration . Since the computation complexity is still high , reactor can not work under limited resources . Furthermore , even without considering the calculation cost , the application scope of Retrace is not as good as n-step returns , as reported in Hernandez-Garcia & Sutton ( 2019 ) . 3 PRELIMINARIES . Reinforcement learning ’ s goal is to find an optimal policy π∗ with maximal discounted returns Rπ = Eπ [ ∑ t γ t−1rt ] given the Markov Decision Process ( MDP ) . To achieve this , agents often estimate the state-action value function qπ ( s , a ) = Eπ [ ∑ t γ t−1rt|s0 = s , a0 = a ] . Let Qπ ( s , a ) denote the estimation of qπ ( s , a ) . In tabular settings , Qπ can be represented by a table of all stateaction pairs 〈s , a〉 , while in large-scale settings , Qπ is often approximated by a deep neural network ( DNN ) with parameter θ , written as Qπ ; θ . During the training process , Qπ is continuously updated by the update target Ĝπ , which is calculated from data points ( s , a , r , s′ ) . In tabular settings ( Watkins & Dayan , 1992 ) , the update equation can be written as : Qπ ( s , a ) ← Qπ ( s , a ) + α [ Ĝπ ( s , a ) − Qπ ( s , a ) ] , where α is the learning rate . In large-scale settings ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ) , Qπ ; θ is updated by mini-batch gradient descent on the neural network parameter θ as : θ ← θ − α 1N ∑N i=1∇θL ( Qπ ; θ ( si , ai ) , Ĝπ ( si , ai ) ) , where L is the loss function . Off-policy learning adopts two policies , the behavior policy µ for generating data points and the target policy π for learning from the data points . Replay buffer ( Fedus et al. , 2020 ) is often used together with the off-policy to handle the increasing sample complexity brought by large-scale problems . Agents draw data points from the replay buffer to update the estimator Qπ . This enables agents to learn from past experiences , thus yields higher sample efficiency . In the sequel , the term off-policy learning refers to adopt off-policy learning together with the replay buffer , as used by most recent off-policy algorithms . N-step returns ( Sutton & Barto , 2018 ) together with off-policy learning have strong empirical performance ( Hessel et al. , 2018 ) , as well as being easy to calculate . It calculates the Ĝnπ as : Ĝnπ ( s , a ) = r0 + γr1 + · · ·+ γn−1rn−1 + γnEan∼π ( sn ) [ Qπ ( sn , an ) ] . ( 1 ) It can be seen as a mix of Monte-Carlo ( MC ) estimation of qπ ≈ ∑ t γ t−1rt ( n → ∞ case ) and one-step TD ( 0 ) estimation r0 + γEa∼π ( s1 ) [ Qπ ( s1 , a ) ] ( n = 1 case ) . In off-policy learning , to calculate n-step returns Ĝnπ ( s0 , a0 ) for data point ( s0 , a0 , r0 , s ′ 0 ) , we draw consecutive transitions ( st , at , rt , s ′ t ) t=0,1,2 , ... in the same trajectory τ as current data point ( s0 , a0 , r0 , s ′ 0 ) from replay buffer . As n-step returns estimates qπ using trajectory τ generated by behavior policy µ , the discrepancy between π and µ makes it biased . We define this off-policy induced bias as off-policy bias , and difference between π and µ as off-policyness . 4 UNDERLYING WORKING MECHANISM OF N-STEP BOOTSTRAPPING . N-step returns lays in the center of designing the update target . It not only unifies the Monte-Carlo returns and one-step temporal difference but also lays the foundation of the eligibility traces ( Singh & Sutton , 1996 ) . Together with off-policy learning , n-step bootstrapping works well because it achieves a good balance between the bias ( TD ) and variance ( MC ) . Considering its importance and wide application , the underlying mechanism of n-step returns has not been studied in detail . In this section , we give a careful analysis of why n-step bootstrapping works and what property the optimal selection of n should satisfy . 4.1 DECOMPOSITION OF THE ESTIMATION ERROR . To understand how n-step bootstrapping works , we conduct a systematical analysis of the estimation error . We formalize the estimation error of an update target Ĝπ as its difference with ground truth qπ for every ( s , a ) pair that the agent experienced : E ( Ĝπ ) = Eµ [ Ĝπ ( s , a ) − qπ ( s , a ) ] . ( 2 ) As shown in Eq . 1 , the estimation error consists of two parts — the off-policy bias and approximation error . The off-policy bias is introduced by the difference of π and µ , as we mentioned before , while the approximation error comes as agent ’ s estimation of Qπ is not perfect , e.g . Qπ 6= qπ . Note that , the behavior policy µ is an older version of the target policy π . We split these two types of error by defining two intermediate targets , Ĝnπ , qπ ; τ and Ĝ n π , Qπ ; τ̃ , to eliminate the other type of error . Ĝnπ , qπ ; τ adopts the ground truth qπ instead of the estimated Qπ to eliminate approximation error : Ĝnπ , qπ ; τ ( s , a ) = r0 + γr1 + · · ·+ γ n−1rn−1 + γ nEan∼π ( sn ) [ qπ ( sn , an ) ] . Ĝnπ , Qπ ; τ̃ uses the trajectory τ̃ = ( s̃t , ãt , r̃t , s̃ ′ t ) t=0,1,2 , ... which is generated by the current policy π instead of the old trajectory τ to remove the off-policy bias : Ĝnπ , Qπ ; τ̃ ( s , a ) = r̃0 + γr̃1 + · · ·+ γ n−1r̃n−1 + γ nEãn∼π ( s̃n ) [ Qπ ( s̃n , ãn ) ] . Then we can quantify the off-policy bias Eoffpolicy and the approximation error Eapprox independently as : Eoffpolicy ( Ĝnπ ) = E ( Ĝnπ , qπ ; τ ) , Eapprox ( Ĝ n π ) = E ( Ĝnπ , Qπ ; τ̃ ) . Now , we can decompose the total error E ( Ĝnπ ) into the sum of off-policy bias and approximation error , plus a negligible small residual term Eresidual ( Ĝnπ ) : Eresidual ( Ĝnπ ) = E ( Ĝnπ ) − Eoffpolicy ( Ĝnπ ) − Eapprox ( Ĝnπ ) = γn ( Ea∼π ( sn ) [ Qπ ( sn , a ) − qπ ( sn , a ) ] − Eã∼π ( s̃n ) [ Qπ ( s̃n , ã ) − qπ ( s̃n , ã ) ] ) . The residual term Eresidual ( Ĝnπ ) is a difference of the discounted error γnEa∼π ( s ) [ Qπ ( s , a ) − qπ ( s , a ) ] on terminal states sn and s̃n . If n is small , the difference between Qπ and qπ on sn will be close to the difference between Qπ and qπ on s̃n . Otherwise , the discount factor γn will shrink exponentially , making the residual term very small . The experimental results in Sec 4.2 show that the residual term is an order of magnitude smaller than the two main sources in practice . Thus , it can be ignored and we get an approximate decomposition : E ( Ĝnπ ) ≈ Eoffpolicy ( Ĝnπ ) + Eapprox ( Ĝnπ ) . ( 3 )

Summary: The paper deals with an intriguing point in RL -- how to correctly choose an adaptive $n$ for $n$-step bootstrap. From such a paper one might expect theoretical results on the tradeoff between bias and variance in the presence of off-policyness. However, as opposed to the picture portrayed in the first section, no such analysis follows.