PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method

1 INTRODUCTION . As a major value function evaluation method , temporal difference ( TD ) learning ( Sutton , 1988 ; Dayan , 1992 ) has been widely used in various planning problems in reinforcement learning . Although TD learning performs successfully in the on-policy settings , where an agent can interact with environments under the target policy , it can perform poorly or even diverge under the off-policy settings when the agent only has access to data sampled by a behavior policy ( Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ; Mahmood et al. , 2015 ) . To address such an issue , the gradient temporal-difference ( GTD ) ( Sutton et al. , 2008 ) and least-squares temporal difference ( LSTD ) ( Yu , 2010 ) algorithms have been proposed , which have been shown to converge in the off-policy settings . However , since GTD and LSTD consider an objective function based on the behavior policy , which adjusts only the distribution mismatch of the action and does not adjust the distribution mismatch of the state , their converging points can be largely biased from the true value function due to the distribution mismatch between the target and behavior policies , even when the express power of the function approximation class is arbitrarily large ( Kolter , 2011 ) . In order to provide a more accurate evaluation , Sutton et al . ( 2016 ) proposed the emphatic temporal difference ( ETD ) algorithm , which introduces the follow-on trace to address the distribution mismatch issue and thus adjusts both state and action distribution mismatch . The stability of ETD was then shown in Sutton et al . ( 2016 ) ; Mahmood et al . ( 2015 ) , and the asymptotic convergence guarantee for ETD was established in Yu ( 2015 ) , it has also achieved great success in many tasks ( Ghiassian et al. , 2016 ; Ni , 2021 ) . However , although ETD can address the distribution mismatch issue to yield a more accurate evaluation , it often suffers from very large variance error due to the follow-on trace estimation over a long or infinite time horizon ( Hallak et al. , 2016 ) . Consequently , the convergence of ETD can be unstable . It can be shown that the variance of ETD can grow exponentially fast as the number of iterations grow so that ETD requires exponentially large number of samples to converge . Hallak et al . ( 2016 ) proposed an ETD method to keep the follow-on trace bounded but at the cost of a possibly large bias error . This thus poses the following intriguing question : Can we design a new ETD method , which overcomes its large variance without introducing a large bias error , and improves its exponential sample complexity to be polynomial at the same time ? In this work , we provide an affirmative answer . 1.1 MAIN CONTRIBUTIONS . We propose a novel ETD approach , called PER-ETD ( i.e. , PEriodically Restarted-ETD ) , in which for each update of the value function parameter we restart the follow-on trace iteration and update it only for b times ( where we call b as the period length ) . Such a periodic restart effectively reduces the variance of the follow-on trace . More importantly , with the design of the period length b to increase logarithmically with the number of iterations , PER-ETD attains the polynomial rather than exponential sample complexity required by ETD . We provide the theoretical guarantee of the sample efficiency of PER-ETD via the finite-time analysis . We show that PER-ETD ( both PER-ETD ( 0 ) and PER-ETD ( λ ) ) converges to the same fixed points of ETD ( 0 ) and ETD ( λ ) , respectively , but with only polynomial sample complexity ( whereas ETD takes exponential sample complexity ) . Our analysis features the following key insights . ( a ) The period length b plays the role of trading off between the variance ( of the follow-on trace ) and bias error ( with respect to the fixed point of ETD ) , and its optimal choice of logarithmical increase with the number of iterations achieves the best tradeoff and keeps both errors vanishing sublinearly . ( b ) Our analysis captures how the mismatch between the behavior and target policies affects the convergence rate of PER-ETD . Interestingly , the mismatch level determines a phase-transition phenomenon of PER-ETD : as long as the mismatch is below a certain threshold , then PER-ETD achieves the same convergence rate as the on-policy TD algorithm ; and if the mismatch is above the threshold , the converge rate of PER-ETD gradually decays as the level of mismatch increases . Experimentally , we demonstrate that PER-ETD converges in the case that neither TD nor ETD converges . Further , our experiments provide the following two interesting observations . ( a ) There does exist a choice of the period length for PER-ETD , which attains the best tradeoff between the variance and bias errors . Below such a choice , the bias error is large so that evaluation is not accurate , and above it the variance error is large so that the convergence is unstable . ( b ) Under a small period length b , it is not always the case that PER-ETD ( λ ) with λ = 1 attains the smallest error with respect to the ground truth value function . The best λ depends on the geometry of the locations of fixed points of PER-ETD ( λ ) for 0 ≤ λ ≤ 1 , which is determined by chosen features . 1.2 RELATED WORKS . TD learning and GTD : The asymptotic convergence of TD learning was established by Sutton ( 1988 ) ; Jaakkola et al . ( 1994 ) ; Dayan & Sejnowski ( 1994 ) ; Tsitsiklis & Van Roy ( 1997 ) , and its nonasymptotic convergence rate was further characterized recently in Dalal et al . ( 2018a ) ; Bhandari et al . ( 2018 ) ; Kotsalis et al . ( 2020 ) ; Chen et al . ( 2019 ) ; Kaledin et al . ( 2020 ) ; Hu & Syed ( 2019 ) ; Srikant & Ying ( 2019 ) . The gradient temporal-difference ( GTD ) was proposed in Sutton et al . ( 2008 ) for off-policy evaluation and was shown to converge asymptotically . Then , Dalal et al . ( 2018b ) ; Gupta et al . ( 2019 ) ; Wang et al . ( 2018 ) ; Xu et al . ( 2019 ) ; Xu & Liang ( 2021 ) provided the finite-time analysis of GTD and its variants . Emphatic Temporal Difference ( ETD ) Learning : The ETD approach was originally proposed in the seminal work Sutton et al . ( 2016 ) , which introduced the follow-on trace to overcome the distribution mismatch between the behavior and target policies . Yu ( 2015 ) provided the asymptotic convergence guarantee for ETD . Hallak et al . ( 2016 ) showed that the variance of the follow-on trace may be unbounded . They further proposed an ETD method with a variable decay rate to keep the follow-on trace bounded but at the cost of a possibly large bias error . Our approach is different and keeps both the variance and bias vanishing sublinearly with the number of iterations . Imani et al . ( 2018 ) developed a new policy gradient theorem , where the emphatic weight is used to correct the distribution shift . Zhang et al . ( 2020b ) provided a new variant of ETD , where the emphatic weights are estimated through function approximation . Van Hasselt et al . ( 2018 ) ; Jiang et al . ( 2021 ) studied ETD with deep neural function class . Comparison to concurrent work : During our preparation of this paper , a concurrent work ( Zhang & Whiteson , 2021 ) was posted on arXiv , and proposed a truncated ETD ( which we refer to as T-ETD for short here ) , which truncates the update of the follow-on trace to reduce the variance of ETD . While T-ETD and our PER-ETD share a similar design idea , there are several critical differences between our work from Zhang & Whiteson ( 2021 ) . ( a ) Our PER-ETD features a design of the logarithmical increase of the restart period with the number of iterations , which guarantees the convergence to the original fixed point of ETD , with both the variance and bias errors vanishing sublinearly . However , T-ETD is guaranteed to converge only to a truncation-length-dependent fixed point , where the convergence is obtained by treating the truncation length as a constant . A careful review of the convergence proof indicates that the variance term scales exponentially fast with the truncation length , and hence the polynomial efficiency is not guaranteed as the truncation length becomes large . ( b ) Our convergence rate for PER-ETD does not depend on the cardinality of the state space and has only polynomial dependence on the mismatch parameter of the behavior and target policies . However , the convergence rate in Zhang & Whiteson ( 2021 ) scales with the cardinality of the state space , and increases exponentially fast with the mismatch parameter of behavior and target policies . ( c ) This paper further studies PER-ETD ( λ ) and the impact of λ on the converge rate and variance and bias errors , whereas Zhang & Whiteson ( 2021 ) considers further the application of T-ETD to the control problem . 2 BACKGROUND AND PRELIMINARIES . 2.1 MARKOV DECISION PROCESS . We consider the infinite-horizon Markov decision process ( MDP ) defined by the five tuple ( S , A , r , P , γ ) . Here , S and A denote the state and action spaces respectively , which are both assumed to be finite sets , r : S ×A → R denotes the reward function , P : S ×A → ∆ ( S ) denotes the transition kernel , where ∆ ( S ) denotes the probability simplex over the state space S , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π : S → ∆ ( A ) of an agent maps from the state space to the probability simplex over the action space A , i.e. , π ( a|s ) represents the probability of taking the action a under the state s. At any time t , given that the system is at the state st , the agent takes an action at with the probability π ( at|st ) , and receives a reward r ( st , at ) . The system then takes a transition to the next state st+1 at time t+ 1 with the probability P ( st+1|st , at ) . For a given policy π , we define the value function corresponding to an initial state s0 = s ∈ S as Vπ ( s ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , π ] . Then the value function over the state space can be expressed as a vector Vπ = ( Vπ ( 1 ) , Vπ ( 2 ) , . . . , Vπ ( |S| ) ) > ∈ R|S| . Here , Vπ is a deterministic function of the policy π . We use capitalized characters to be consistent with the literature . When the state space is large , we approximate the value function Vπ via a linear function class as Vθ ( s ) = φ > ( s ) θ , where φ ( s ) ∈ Rd denotes the feature vector , and θ ∈ Rd denotes the parameter vector to be learned . We further let Φ = [ φ ( 1 ) , φ ( 2 ) , . . . , φ ( |S| ) ] > denote the feature matrix , and then Vθ = Φθ . We assume that the feature matrix Φ has linearly independent columns and each feature vector has bounded ` 2-norm , i.e. , ‖φ ( s ) ‖2 ≤ Bφ for all s ∈ S .

Summary: The authors propose an improved variant of emphatic temporal difference (ETD) aimed at addressing issues of high variance when faced with a large mismatch between behavior and target policies. The main improvement of PER-ETD, the proposed algorithm, comes from periodically clearing the follow-on trace at logarthmically increasing periods. The authors present a finite-time analysis for both the PER-ETD(0) and PER-ETD($\lambda$) case showing how bias and variance depends on the time between clearing the trace. This is used to derive a schedule that effectively minimizes the variance and bias to achieve polynomial sample complexity. The authors conclude by illustrating these improved properties in Baird's counter-example MDP. The results confirm the effect on bias and variance of the period parameter and highlight's that $\lambda=1$ no longer results in the closest fixed-point to the optimal solution.