Reanalysis of Variance Reduced Temporal Difference Learning

1 INTRODUCTION . In reinforcement learning ( RL ) , policy evaluation aims to obtain the expected long-term reward of a given policy and plays an important role in identifying the optimal policy that achieves the maximal cumulative reward over time Bertsekas and Tsitsiklis ( 1995 ) ; Dayan and Watkins ( 1992 ) ; Rummery and Niranjan ( 1994 ) . The temporal difference ( TD ) learning algorithm , originally proposed by Sutton ( 1988 ) , is one of the most widely used policy evaluation methods , which uses the Bellman equation to iteratively bootstrap the estimation process and continually update the value function in an incremental way . In practice , if the state space is large or infinite , function approximation is often used to find an approximate value function efficiently . Theoretically , TD with linear function approximation has been shown to converge to the fixed point solution with i.i.d . samples and Markovian samples in Sutton ( 1988 ) ; Tsitsiklis and Van Roy ( 1997 ) . The finite sample analysis of TD has also been studied in Bhandari et al . ( 2018 ) ; Dalal et al . ( 2018a ) ; Cai et al . ( 2019 ) ; Srikant and Ying ( 2019 ) . Since each iteration of TD uses one or a mini-batch of samples to estimate the mean of the pseudogradient 1 , TD learning usually suffers from the inherent variance , which substantially degrades the convergence accuracy . Although a diminishing stepsize or very small constant stepsize can reduce the variance Bhandari et al . ( 2018 ) ; Srikant and Ying ( 2019 ) , they also slow down the convergence significantly . Two approaches have been proposed to reduce the variance . The first approach is the so-called batch TD , which takes a fixed sample set and transforms the empirical mean square projected Bellman error ( MSPBE ) into an equivalent convex-concave saddle-point problem Du et al . ( 2017 ) . Due to the finite-sample nature of such a problem , stochastic variance reduction techniques for conventional optimization can be directly applied here to reduce the variance . In particular , Du et al . ( 2017 ) showed that SVRG Johnson and Zhang ( 2013 ) and SAGA Defazio et al . ( 2014 ) can be applied to improve 1We call the increment in each iteration of TD as `` pseudo-gradient '' because such a quantity is not a gradient of any objective function , but the role that it serves in the TD algorithm is analogous to the role of the gradient in the gradient descent algorithm . the performance of batch TD algorithms , and Peng et al . ( 2019 ) proposed two variants of SVRG to further save the computation cost . However , the analysis of batch TD does not take into account the statistical nature of the training samples , which are generated by a MDP . Hence , there is no guarantee of such obtained solutions to be close to the fixed point of TD learning . The second approach is the so-called TD with centering ( CTD ) algorithm proposed in Korda and La ( 2015 ) , which introduces the variance reduction idea to the original TD learning algorithm . For the sake of better reflecting its major feature , we refer to CTD as Variance Reduced TD ( VRTD ) throughout this paper . Similarly to the SVRG in Johnson and Zhang ( 2013 ) , VRTD has outer and inner loops . The beginning of each inner-loop ( i.e . each epoch ) computes a batch of sample pseudogradients so that each subsequent inner loop iteration modifies only one sample pseudo-gradient in the batch pseudo-gradients to reduce the variance . The main difference between VRTD and batch TD is that VRTD applies the variance reduction directly to TD learning rather than to a transformed optimization problem in batch TD . Though Korda and La ( 2015 ) empirically verified that VRTD has better convergence accuracy than vanilla TD learning , some technical errors in the analysis in Korda and La ( 2015 ) have been pointed out in follow up studies Dalal et al . ( 2018a ) ; Narayanan and Szepesvári ( 2017 ) . Furthermore , as we discuss in Section 3 , the technical proof in Korda and La ( 2015 ) regarding the convergence of VRTD also has technical errors so that their results do not correctly characterize the impact of variance reduction on TD learning . Given the recent surge of interest in the finite time analysis of the vanilla TD Dalal et al . ( 2018a ) ; Bhandari et al . ( 2018 ) ; Dalal et al . ( 2018b ) ; Srikant and Ying ( 2019 ) , it becomes imperative to reanalyze the VRTD and accurately understand whether and how variance reduction can help to improve the convergence accuracy over vanilla TD . Towards this end , this paper specifically addresses the following central questions . • For i.i.d . sampling , it has been shown in Dalal et al . ( 2018a ) ; Bhandari et al . ( 2018 ) that vanilla TD converges only to a neighborhood of the fixed point for a constant stepsize and suffers from a constant error term caused by the variance of the stochastic pseudo-gradient at each iteration . For VRTD , does the variance reduction help to reduce such an error and improve the accuracy of convergence ? How does the error depend on the variance reduction parameter , i.e. , the batch size for variance reduction ? • For Markovian sampling , it has been shown in Bhandari et al . ( 2018 ) that the convergence of vanilla TD further suffers from a bias error due to the correlation among samples in addition to the variance error as in i.i.d . sampling . Does VRTD , which was designed to have reduced variance , also enjoy reduced bias error ? If so , how does the bias error depend on the batch size for variance reduction ? • For both i.i.d . and Markovian sampling , to attain an -accurate solution , what is the overall computational complexity ( the total number of computations of pseudo-gradients ) of VRTD , and does VRTD have a reduced overall computational complexity compared to TD ? 1.1 OUR CONTRIBUTIONS . Our main contributions are summarized in Table 1 and are described as follows . For i.i.d . sampling , we show that a slightly modified version of VRTD ( for avoiding bias error ) converges linearly to a neighborhood of the fixed point solution for a constant stepsize α , with the variance error at the order ofO ( α/M ) , where M is the batch size for variance reduction . To attain an -accurate solution , the overall computational complexity ( i.e. , the total number of pseudo-gradient computations ) of VRTD outperforms the vanilla TD algorithm Bhandari et al . ( 2018 ) if the condition number is small . For Markovian sampling , we show that VRTD has the same linear convergence and the same variance error reduction over the vanilla TD Dalal et al . ( 2018a ) ; Bhandari et al . ( 2018 ) as i.i.d . sampling . More importantly , the variance reduction in VRTD also attains a substantially reduced bias error at the order of O ( 1/M ) over the vanilla TD Bhandari et al . ( 2018 ) , where the bias error is at the order of O ( α ) . As a result , VRTD outperforms vanilla TD in terms of the total computational complexity by a factor of log 1 . At the technical level , our analysis of bias error for Markovian sampling takes a different path from how the existing analysis of TD handles Markovian samples in Bhandari et al . ( 2018 ) ; Wang et al . ( 2017 ) ; Srikant and Ying ( 2019 ) . Due to the batch average of stochastic pseudo-gradients adopted by VRTD to reduce the variance , the correlation among samples in different epochs is eliminated . Such an analysis explicitly explains why the variance reduction step helps to further reduce the bias error . 1.2 RELATED WORK . On-policy TD learning and variance reduction . On-policy TD learning aims to minimize the Mean Squared Bellman Error ( MSBE ) Sutton ( 1988 ) when samples are drawn independently from the stationary distribution of the corresponding MDP . The non-asymptotic convergence under i.i.d . sampling has been established in Dalal et al . ( 2018a ) for TD with linear function approximation and for TD with overparameterized neural network approximation Cai et al . ( 2019 ) . The convergence of averaged linear SA with constant stepsize has been studied in Lakshminarayanan and Szepesvari ( 2018 ) . In the Markovian setting , the non-asymptotic convergence has been studied for on-policy TD in Bhandari et al . ( 2018 ) ; Karmakar and Bhatnagar ( 2016 ) ; Wang et al . ( 2019 ) ; Srikant and Ying ( 2019 ) . Korda and La ( 2015 ) proposed a variance reduced CTD algorithm ( called VRTD in this paper ) , which directly applies variance reduction technique to the TD algorithm . The analysis of VRTD provided in Korda and La ( 2015 ) has technical errors . The aim of this paper is to provide a technically solid analysis for VRTD to characterize the advantage of variance reduction . Variance reduced batch TD learning . Batch TD Lange et al . ( 2012 ) algorithms are generally designed for policy evaluation by solving an optimization problem on a fixed dataset . In Du et al . ( 2017 ) , the empirical MSPBE is first transformed into a quadratic convex-concave saddle-point optimization problem and variance reduction methods of SVRG Johnson and Zhang ( 2013 ) and SAGA Defazio et al . ( 2014 ) were then incorporated into a primal-dual batch gradient method . Furthermore , Peng et al . ( 2019 ) applied two variants of variance reduction methods to solve the same saddle point problems , and showed that those two methods can save pseudo-gradient computation cost . We note that due to the extensive research in TD learning , we include here only studies that are highly related to our work , and can not cover many other interesting topics on TD learning such as asymptotic convergence of TD learning Tadić ( 2001 ) ; Hu and Syed ( 2019 ) , off-policy TD learning Sutton et al . ( 2008 ; 2009 ) ; Liu et al . ( 2015 ) ; Wang et al . ( 2017 ) ; Karmakar and Bhatnagar ( 2017 ) , two time-scale TD algorithms Xu et al . ( 2019 ) ; Dalal et al . ( 2018b ) ; Yu ( 2017 ) , fitted TD algorithms Lee and He ( 2019 ) , SARSA Zou et al . ( 2019 ) etc . The idea of the variance reduction algorithm proposed in Korda and La ( 2015 ) as well as the analysis techniques that we develop in this paper can potentially be useful for these algorithms . 2 PROBLEM FORMULATION AND PRELIMINARIES . 2.1 ON-POLICY VALUE FUNCTION EVALUATION . We describe the problem of value function evaluation over a Markov decision process ( MDP ) ( S , A , P , r , γ ) , where each component is explained in the sequel . Suppose S ⊂ Rd is a compact state space , and A is a finite action set . Consider a stationary policy π , which maps a state s ∈ S to the actions in A via a probability distribution π ( ·|s ) . At time-step t , suppose the process is in some state st ∈ S , and an action at ∈ A is taken based on the policy π ( ·|st ) . Then the transition kernel P = P ( st+1|st , at ) determines the probability of being at state st+1 ∈ S in the next time-step , and the reward rt = r ( st , at , st+1 ) is received , which is assumed to be bounded by rmax . We denote the associated Markov chain by p ( s′|s ) = ∑ a∈A p ( s ′|s , a ) π ( a|s ) , and assume that it is ergodic . Let µπ be the induced stationary distribution , i.e. , ∑ s p ( s ′|s ) µπ ( s ) = µπ ( s′ ) . We define the value function for a policy π as vπ ( s ) = E [ ∑∞ t=0 γ tr ( st , at , st+1 ) |s0 = s , π ] , where γ ∈ ( 0 , 1 ) is the discount factor . Define the Bellman operator Tπ for any function ξ ( s ) as Tπξ ( s ) : = rπ ( s ) + γEs′|sξ ( s′ ) , where rπ ( s ) = Ea , s′|sr ( s , a , s′ ) is the expected reward of the Markov chain induced by the policy π . It is known that vπ ( s ) is the unique fixed point of the Bellman operator Tπ , i.e. , vπ ( s ) = Tπvπ ( s ) . In practice , since the MDP is unknown , the value function vπ ( s ) can not be directly obtained . The goal of policy evaluation is to find the value function vπ ( s ) via sampling the MDP .

Summary: In this paper, the authors study the variance reduced TD (VRTD) algorithm, by Korda and Prashanth (2015)  (KP15), for policy evaluation in RL. They first highlight technical errors in the analysis of KP15, and then provide new convergence analysis for this algorithm. The new analysis is based on a new technique to bound the bias of the VRTD gradient estimator, and shows the advantage of VRTD over vanilla TD (the analyses by Bhandari et al. 2018 and Srikant and Ying 2019), both in terms of variance and bias, that are reduced by increasing the batch size. The authors show that while the variance and bias of vanilla TD are both of O(\alpha) (where \alpha is the step-size), they are of O(\alpha/M) and O(1/\sqrt{M}) (where M is the batch size) in VRTD. This shows that a good convergence is possible for VRTD without reducing the step-size \alpha, that causes slow convergence, by increasing the batch size M. In the middle of their analysis, the authors propose a slight modification of VRTD for the case that the samples are obtained iid from the stationary distribution of the evaluating policy, and provide its analysis. Finally, the authors provide simple experiments to support their theoretical findings.