PER-ETD: A Polynomially Efficient Emphatic Temporal Difference Learning Method

1 INTRODUCTION . As a major value function evaluation method , temporal difference ( TD ) learning ( Sutton , 1988 ; Dayan , 1992 ) has been widely used in various planning problems in reinforcement learning . Although TD learning performs successfully in the on-policy settings , where an agent can interact with environments under the target policy , it can perform poorly or even diverge under the off-policy settings when the agent only has access to data sampled by a behavior policy ( Baird , 1995 ; Tsitsiklis & Van Roy , 1997 ; Mahmood et al. , 2015 ) . To address such an issue , the gradient temporal-difference ( GTD ) ( Sutton et al. , 2008 ) and least-squares temporal difference ( LSTD ) ( Yu , 2010 ) algorithms have been proposed , which have been shown to converge in the off-policy settings . However , since GTD and LSTD consider an objective function based on the behavior policy , which adjusts only the distribution mismatch of the action and does not adjust the distribution mismatch of the state , their converging points can be largely biased from the true value function due to the distribution mismatch between the target and behavior policies , even when the express power of the function approximation class is arbitrarily large ( Kolter , 2011 ) . In order to provide a more accurate evaluation , Sutton et al . ( 2016 ) proposed the emphatic temporal difference ( ETD ) algorithm , which introduces the follow-on trace to address the distribution mismatch issue and thus adjusts both state and action distribution mismatch . The stability of ETD was then shown in Sutton et al . ( 2016 ) ; Mahmood et al . ( 2015 ) , and the asymptotic convergence guarantee for ETD was established in Yu ( 2015 ) , it has also achieved great success in many tasks ( Ghiassian et al. , 2016 ; Ni , 2021 ) . However , although ETD can address the distribution mismatch issue to yield a more accurate evaluation , it often suffers from very large variance error due to the follow-on trace estimation over a long or infinite time horizon ( Hallak et al. , 2016 ) . Consequently , the convergence of ETD can be unstable . It can be shown that the variance of ETD can grow exponentially fast as the number of iterations grow so that ETD requires exponentially large number of samples to converge . Hallak et al . ( 2016 ) proposed an ETD method to keep the follow-on trace bounded but at the cost of a possibly large bias error . This thus poses the following intriguing question : Can we design a new ETD method , which overcomes its large variance without introducing a large bias error , and improves its exponential sample complexity to be polynomial at the same time ? In this work , we provide an affirmative answer . 1.1 MAIN CONTRIBUTIONS . We propose a novel ETD approach , called PER-ETD ( i.e. , PEriodically Restarted-ETD ) , in which for each update of the value function parameter we restart the follow-on trace iteration and update it only for b times ( where we call b as the period length ) . Such a periodic restart effectively reduces the variance of the follow-on trace . More importantly , with the design of the period length b to increase logarithmically with the number of iterations , PER-ETD attains the polynomial rather than exponential sample complexity required by ETD . We provide the theoretical guarantee of the sample efficiency of PER-ETD via the finite-time analysis . We show that PER-ETD ( both PER-ETD ( 0 ) and PER-ETD ( λ ) ) converges to the same fixed points of ETD ( 0 ) and ETD ( λ ) , respectively , but with only polynomial sample complexity ( whereas ETD takes exponential sample complexity ) . Our analysis features the following key insights . ( a ) The period length b plays the role of trading off between the variance ( of the follow-on trace ) and bias error ( with respect to the fixed point of ETD ) , and its optimal choice of logarithmical increase with the number of iterations achieves the best tradeoff and keeps both errors vanishing sublinearly . ( b ) Our analysis captures how the mismatch between the behavior and target policies affects the convergence rate of PER-ETD . Interestingly , the mismatch level determines a phase-transition phenomenon of PER-ETD : as long as the mismatch is below a certain threshold , then PER-ETD achieves the same convergence rate as the on-policy TD algorithm ; and if the mismatch is above the threshold , the converge rate of PER-ETD gradually decays as the level of mismatch increases . Experimentally , we demonstrate that PER-ETD converges in the case that neither TD nor ETD converges . Further , our experiments provide the following two interesting observations . ( a ) There does exist a choice of the period length for PER-ETD , which attains the best tradeoff between the variance and bias errors . Below such a choice , the bias error is large so that evaluation is not accurate , and above it the variance error is large so that the convergence is unstable . ( b ) Under a small period length b , it is not always the case that PER-ETD ( λ ) with λ = 1 attains the smallest error with respect to the ground truth value function . The best λ depends on the geometry of the locations of fixed points of PER-ETD ( λ ) for 0 ≤ λ ≤ 1 , which is determined by chosen features . 1.2 RELATED WORKS . TD learning and GTD : The asymptotic convergence of TD learning was established by Sutton ( 1988 ) ; Jaakkola et al . ( 1994 ) ; Dayan & Sejnowski ( 1994 ) ; Tsitsiklis & Van Roy ( 1997 ) , and its nonasymptotic convergence rate was further characterized recently in Dalal et al . ( 2018a ) ; Bhandari et al . ( 2018 ) ; Kotsalis et al . ( 2020 ) ; Chen et al . ( 2019 ) ; Kaledin et al . ( 2020 ) ; Hu & Syed ( 2019 ) ; Srikant & Ying ( 2019 ) . The gradient temporal-difference ( GTD ) was proposed in Sutton et al . ( 2008 ) for off-policy evaluation and was shown to converge asymptotically . Then , Dalal et al . ( 2018b ) ; Gupta et al . ( 2019 ) ; Wang et al . ( 2018 ) ; Xu et al . ( 2019 ) ; Xu & Liang ( 2021 ) provided the finite-time analysis of GTD and its variants . Emphatic Temporal Difference ( ETD ) Learning : The ETD approach was originally proposed in the seminal work Sutton et al . ( 2016 ) , which introduced the follow-on trace to overcome the distribution mismatch between the behavior and target policies . Yu ( 2015 ) provided the asymptotic convergence guarantee for ETD . Hallak et al . ( 2016 ) showed that the variance of the follow-on trace may be unbounded . They further proposed an ETD method with a variable decay rate to keep the follow-on trace bounded but at the cost of a possibly large bias error . Our approach is different and keeps both the variance and bias vanishing sublinearly with the number of iterations . Imani et al . ( 2018 ) developed a new policy gradient theorem , where the emphatic weight is used to correct the distribution shift . Zhang et al . ( 2020b ) provided a new variant of ETD , where the emphatic weights are estimated through function approximation . Van Hasselt et al . ( 2018 ) ; Jiang et al . ( 2021 ) studied ETD with deep neural function class . Comparison to concurrent work : During our preparation of this paper , a concurrent work ( Zhang & Whiteson , 2021 ) was posted on arXiv , and proposed a truncated ETD ( which we refer to as T-ETD for short here ) , which truncates the update of the follow-on trace to reduce the variance of ETD . While T-ETD and our PER-ETD share a similar design idea , there are several critical differences between our work from Zhang & Whiteson ( 2021 ) . ( a ) Our PER-ETD features a design of the logarithmical increase of the restart period with the number of iterations , which guarantees the convergence to the original fixed point of ETD , with both the variance and bias errors vanishing sublinearly . However , T-ETD is guaranteed to converge only to a truncation-length-dependent fixed point , where the convergence is obtained by treating the truncation length as a constant . A careful review of the convergence proof indicates that the variance term scales exponentially fast with the truncation length , and hence the polynomial efficiency is not guaranteed as the truncation length becomes large . ( b ) Our convergence rate for PER-ETD does not depend on the cardinality of the state space and has only polynomial dependence on the mismatch parameter of the behavior and target policies . However , the convergence rate in Zhang & Whiteson ( 2021 ) scales with the cardinality of the state space , and increases exponentially fast with the mismatch parameter of behavior and target policies . ( c ) This paper further studies PER-ETD ( λ ) and the impact of λ on the converge rate and variance and bias errors , whereas Zhang & Whiteson ( 2021 ) considers further the application of T-ETD to the control problem . 2 BACKGROUND AND PRELIMINARIES . 2.1 MARKOV DECISION PROCESS . We consider the infinite-horizon Markov decision process ( MDP ) defined by the five tuple ( S , A , r , P , γ ) . Here , S and A denote the state and action spaces respectively , which are both assumed to be finite sets , r : S ×A → R denotes the reward function , P : S ×A → ∆ ( S ) denotes the transition kernel , where ∆ ( S ) denotes the probability simplex over the state space S , and γ ∈ ( 0 , 1 ) is the discount factor . A policy π : S → ∆ ( A ) of an agent maps from the state space to the probability simplex over the action space A , i.e. , π ( a|s ) represents the probability of taking the action a under the state s. At any time t , given that the system is at the state st , the agent takes an action at with the probability π ( at|st ) , and receives a reward r ( st , at ) . The system then takes a transition to the next state st+1 at time t+ 1 with the probability P ( st+1|st , at ) . For a given policy π , we define the value function corresponding to an initial state s0 = s ∈ S as Vπ ( s ) = E [ ∑∞ t=0 γ tr ( st , at ) |s0 = s , π ] . Then the value function over the state space can be expressed as a vector Vπ = ( Vπ ( 1 ) , Vπ ( 2 ) , . . . , Vπ ( |S| ) ) > ∈ R|S| . Here , Vπ is a deterministic function of the policy π . We use capitalized characters to be consistent with the literature . When the state space is large , we approximate the value function Vπ via a linear function class as Vθ ( s ) = φ > ( s ) θ , where φ ( s ) ∈ Rd denotes the feature vector , and θ ∈ Rd denotes the parameter vector to be learned . We further let Φ = [ φ ( 1 ) , φ ( 2 ) , . . . , φ ( |S| ) ] > denote the feature matrix , and then Vθ = Φθ . We assume that the feature matrix Φ has linearly independent columns and each feature vector has bounded ` 2-norm , i.e. , ‖φ ( s ) ‖2 ≤ Bφ for all s ∈ S .

Summary: This paper presents a technique of reducing the variance of emphatic algorithms by resetting the trace periodically, and that increasing the period logarithmicaly can result in an optimal way of trading off bias and variance of the resulting learned value function. The authors prove such a method obtains the optimal tradeoff and results in convergence in settings where predecessor variants fails (TD and ETD). The paper proves these claims theoretically, and show an illustration on the Baird domain for empirical support.