Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximation

√ T ) ( up to a factor of O ( log T ) ) . Our approach can be applied to a wide range of value-based reinforcement learning algorithms with general smooth function approximation . 1 Introduction . In reinforcement learning ( RL ) , an agent interacts with a stochastic environment in order to maximize the total reward [ Sutton and Barto , 2018 ] . Towards this goal , it is often needed to evaluate how good a policy performs , and more specifically , to learn its value function . Temporal difference ( TD ) learning algorithm is one of the most popular policy evaluation approaches . However , when applied with function approximation approach and/or under the off-policy setting , the TD learning algorithm may diverge [ Baird , 1995 , Tsitsiklis and Van Roy , 1997 ] . To address this issue , a family of gradient-based TD ( GTD ) algorithms , e.g. , GTD , GTD2 , temporal-difference learning with gradient correction ( TDC ) and Greedy-GQ , were developed for the case with linear function approximation [ Maei , 2011 , Sutton et al. , 2009b , Maei et al. , 2010 , Sutton et al. , 2009a , b ] . These algorithms were later extended to the case with general smooth function approximation in [ Bhatnagar et al. , 2009 ] , where asymptotic convergence guarantee was established for the on-policy setting with i.i.d . samples . Despite the success of the GTD methods in practice , previous theoretical studies only showed that these algorithms converge asymptotically , and did not suggest how fast these algorithms converge and how the accuracy of the solution depends on various parameters of the algorithms . Not until 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recently have the non-asymptotic error bounds for these algorithms been investigated , e.g. , [ Dalal et al. , 2020 , Karmakar and Bhatnagar , 2018 , Wang and Zou , 2020 , Xu et al. , 2019 , Kaledin et al. , 2020 , Dalal et al. , 2018 , Wang et al. , 2017 ] , which mainly focus on the case with linear function approximation . These results thus can not be directly applied to more practical applications with general smooth function approximation , e.g. , neural networks , which have greater representation power , do not need to construct feature mapping , and are widely used in practice . In this paper , we develop a non-asymptotic analysis for the TDC algorithm with general smooth function approximation ( which we refer to as non-linear TDC ) for both i.i.d . and Markovian samples . Technically , the analysis in this paper is not a straightforward extension of previous studies on those GTD algorithms with linear function approximation . First of all , different from existing studies with linear function approximation whose objective functions are convex and the updates are linear , the objective function of the non-linear TDC algorithm is non-convex , and the two time-scale updates are non-linear functions of the parameters . Second , the objective function of the non-linear TDC algorithm , the mean-square projected Bellman error ( MSPBE ) , involves a projection onto a timevarying tangent plane which depends on the sample trajectory , whereas for GTD algorithms with linear function approximation , this projection is time-invariant . Third , due to the two time-scale structure of the algorithm and the Markovian noise , novel techniques to deal with the stochastic bias and the tracking error need to be developed . 1.1 Challenges and Contributions . In this section , we summarize the technical challenges and our contributions . Analysis for two time-scale non-linear updates and non-convex objective . Unlike many existing results on two time-scale stochastic approximation , e.g. , [ Konda et al. , 2004 , Gupta et al. , 2019 , Kaledin et al. , 2020 ] and the studies of linear GTD algorithms in [ Xu et al. , 2019 , Ma et al. , 2020 , Wang et al. , 2017 , Dalal et al. , 2020 ] , the objective function of the non-linear TDC is non-convex , and its two time-scale updates are non-linear . Therefore , existing studies on linear two time-scale algorithms can not be directly applied . Moreover , the convergence to global optimum can not be guaranteed for the non-linear TDC algorithm , and therefore , we study the convergence to stationary points . In this paper , we develop a novel non-asymptotic analysis of the non-linear TDC algorithm , which solves RL problems from a non-convex optimization perspective . We note that our analysis is not a straightforward extension of analyses of non-convex optimization , as the update rule here is two time-scale and the noise is Markovian . The framework we develop in this paper can be applied to analyze a wide range of value-based RL algorithms with general smooth function approximation . Time-varying projection . For the MSPBE , a projection of the Bellman error onto the parameterized function class is involved . However , unlike linear function approximation , the projection onto a general smooth class of functions usually does not have a closed-form solution . Thus , a projection onto the tangent plane at the current parameter is used instead , which incurs a time-varying projection that depends on the current parameter and thus the sample trajectory . This brings in additional challenges in the bias and variance analysis due to such dependency . We develop a novel approach to decouple such a dependency and characterize the bias by exploiting the uniform ergodicity of the underlying MDP and the smoothness of the parameterized function . The new challenges posed by the time-varying projection and the dependence between the projection and the sample trajectory are not special to the non-linear TDC investigated in this paper , and they exist in a wide range of value-based algorithms with general smooth function approximation , where our techniques can be applied . A tight tracking error analysis . Due to the two time-scale structure of the update rule , the tracking error , which measures how fast the fast time-scale tracks its own limit , needs to be explicitly bounded . Unlike the studies on two time-scale linear stochastic approximation [ Dalal et al. , 2020 , Kaledin et al. , 2020 , Konda et al. , 2004 ] , where a linear transformation can asymptotically decouple the dependence between the fast and slow time-scale updates , it is non-trivial to construct such a transformation for non-linear updates . To develop a tight bound on the tracking error , we develop a novel technique that bounds the tracking error as a function of the gradient of the MSPBE . This leads to a tighter bound on the tracking error compared to many existing works on two time-scale analysis , e.g. , [ Wu et al. , 2020 , Hong et al. , 2020 ] . Although we do not decouple the fast and slow time-scale updates , we still obtain a desired convergence rate of O ( 1/ √ T ) ( up to a factor of log T ) , which matches with the complexity of stochastic gradient descent for non-convex problems [ Ghadimi and Lan , 2013 ] . 1.2 Related Work . TD , Q-learning and SARSA . The asymptotic convergence of TD with linear function approximation was shown in [ Tsitsiklis and Van Roy , 1997 ] , and the non-asymptotic analysis of TD was developed in [ Srikant and Ying , 2019 , Lakshminarayanan and Szepesvari , 2018 , Bhandari et al. , 2018 , Dalal et al. , 2018 , Sun et al. , 2020 ] . Moreover , [ Cai et al. , 2019 ] further studied the non-asymptotic error bound of TD learning with neural function approximation . Q-learning and SARSA are usually used for solving the optimal control problem and were shown to converge asymptotically under some conditions in [ Melo et al. , 2008 , Perkins and Precup , 2003 ] . Their non-asymptotic error bounds were also studied in [ Zou et al. , 2019 ] . The non-asymptotic analysis of Q-learning under the neural function approximation was developed in [ Cai et al. , 2019 , Xu and Gu , 2020 ] . Note that all these algorithms are one time-scale , while the TDC algorithm we study is a two time-scale algorithm . GTD methods with linear function approximation . A class of GTD algorithms were proposed to address the divergence issue for off-policy training [ Baird , 1995 ] and arbitrary smooth function approximation [ Tsitsiklis and Van Roy , 1997 ] , e.g. , GTD , GTD2 and TDC [ Maei , 2011 , Sutton et al. , 2009b , Maei et al. , 2010 , Sutton et al. , 2009a , b ] . Recent studies established their non-asymptotic convergence rate , e.g. , [ Dalal et al. , 2018 , Wang et al. , 2017 , Liu et al. , 2015 , Gupta et al. , 2019 , Xu et al. , 2019 , Dalal et al. , 2020 , Kaledin et al. , 2020 , Ma et al. , 2020 , Wang and Zou , 2020 , Ma et al. , 2021 ] under i.i.d . and Markovian settings . These studies focus on the case with linear function approximation , and thus the objective functions are convex , and the updates are linear . In this paper , we focus on the non-linear TDC algorithm with general smooth function approximation , where the two time-scale update rule is non-linear , the objective is non-convex , and the projection is time-varying , and thus new techniques are required to develop the non-asymptotic analysis . Non-linear two time-scale stochastic approximation . There are also studies on asymptotic convergence rate and non-asymptotic analysis for non-linear two time-scale stochastic approximation , e.g. , [ Mokkadem et al. , 2006 , Doan , 2021 ] . Although the non-linear update rule is investigated , it is assumed that the algorithm converges to the global optimum . In this paper , we do not make such an assumption on the global convergence , which may not necessarily hold for the non-linear TDC algorithm , and instead , we study the convergence to stationary points , which is a widely used convergence criterion for non-convex optimization problems . We also note that there is a resent work studying the batch-based non-linear TDC in [ Xu and Liang , 2021 ] , where at each update , a batch of samples is used . To achieve a sample complexity of O ( −2 ) , a batch size of O ( −1 ) is required in [ Xu and Liang , 2021 ] to control the bias and variance . We note that by setting the batch size being one in [ Xu and Liang , 2021 ] , the desired sample complexity can not be obtained , and their error bound will be a constant . In this paper , we focus on the non-linear TDC algorithm without using the batch method , where the parameters update in an online and incremental fashion and at each update only one sample is used . Our error analysis is novel and more refined as it does not require a large batch size of O ( −1 ) while still achieving the same sample complexity . 2 Preliminaries . 2.1 Markov Decision Process . A Markov decision process ( MDP ) is a tuple ( S , A , P , r , γ ) , where S and A are the state and action spaces , P = P ( s′|s , a ) is the transition kernel , r : S×A× S→ R+ is the reward function bounded by rmax , and γ ∈ [ 0 , 1 ] is the discount factor . A stationary policy π maps a state s ∈ S to a probability distribution π ( ·|s ) over the action space A . At each time-step t , suppose the process is at some state st ∈ S , and an action at ∈ A is taken . Then the system transits to the next state st+1 following the transition kernel P ( ·|st , at ) , and the agent receives a reward r ( st , at , st+1 ) . For a given policy π and any initial state s ∈ S , we define its value function as V π ( s ) = E [ ∑∞ t=0 γ tr ( St , At , St+1 ) |S0 = s , π ] . The goal of policy evaluation is to use the samples generated from the MDP to estimate the value function . The value function satisfies the Bellman equation : V π ( s ) = TπV π ( s ) for any s ∈ S , where the Bellman operator Tπ is defined as TπV ( s ) = ∑ s′∈S , a∈A P ( s′|s , a ) π ( a|s ) r ( s , a , s′ ) + γ ∑ s′∈S , a∈A P ( s′|s , a ) π ( a|s ) V ( s′ ) . ( 1 ) Hence the value function V π is the fixed point of the Bellman operator Tπ [ Bertsekas , 2011 ] .

Summary: This paper analyses the convergence of two time-scale TDC with general smooth function approximation. Previous analysis mainly focuses the linear approximation, but the nonlinear approximation is more powerful and its convergence analysis is more challenging; thus, this work is theoretically solid. The analysis covers both i.i.d. settings and Markov settings.