Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators

1 Introduction . Reinforcement learning ( RL ) demonstrated its success in learning effective policies for a variety of decision making problems such as autonomous driving [ 25 , 26 ] , recommender systems [ 1 , 41 ] , and game-related problems [ 23 , 27 , 39 ] . In RL , there is an important sub-problem – called the policy evaluation problem – of estimating the expected long term reward of a given policy . Solving the policy evaluation problem is usually an itermediate step in many existing RL algorithms to ultimately find an optimal policy , such as approximate policy iteration and actor-critic framework . The policy evaluation problem is usually solved with the TD-learning method [ 30 ] . A key ingredient in TD-learning is the policy used to collect samples ( called the behavior policy ) . Ideally , we want to generate samples from the target policy whose value function we want to estimate , and this is called on-policy sampling . However , in many cases such on-policy sampling is not possible due to practical reasons [ 16 , 40 ] , and hence we need to work with historical data that is generated by a possibly different policy ( i.e. , off-policy sampling ) . Although off-policy sampling is more practical than on-policy sampling , it is more challenging to analyze and is known to have high variance [ 15 ] , which is a fundamental difficulty in off-policy learning . To overcome this difficulty , many variants of off-policy TD-learning algorithms have been proposed in the literature , such as Qπ ( λ ) [ 17 ] , Tree-Backup ( λ ) ( henceforth denoted by TB ( λ ) ) [ 24 ] , Retrace ( λ ) [ 22 ] , and Q-trace [ 19 ] , etc . 1.1 Main Contributions . In this work , we establish finite-sample bounds of a general n-step off-policy TD-learning algorithm that also subsumes several algorithms presented in the literature . The key step is to show that such 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) algorithm can be modeled as a Markovian stochastic approximation ( SA ) algorithm for solving a generalized Bellman equation . We present sufficient conditions under which the generalized Bellman operator is contractive with respect to a weighted ` p-norm for every p ∈ [ 1 , ∞ ) , with a uniform contraction factor for all p. Our result shows that the sample complexity scales as Õ ( −2 ) , where is the required accuracy . It also involves a factor that depends on the problem parameters , in particular , the generalized importance sampling ratios , and explicitly demonstrates the bias-variance trade-off . Our result immediately gives finite-sample guarantees for variants of multi-step off-policy TDlearning algorithms including Qπ ( λ ) , TB ( λ ) , Retrace ( λ ) , and Q-trace . For Qπ ( λ ) , TB ( λ ) , and Retrace ( λ ) , we establish the first-known results in the literature , while for Q-trace , we improve the best known results in [ 19 ] in terms of the dependency on the size of the state-action space . The weighted ` p-norm contraction property with a uniform contraction factor for all p ∈ [ 1 , ∞ ) is crucial for us to establish the improved sample complexity . Based on the finite-sample bounds , we show that all four algorithms overcome the high variance issue in Vanilla off-policy TD-learning , but their convergence rates are all affected to varying degrees . 1.2 Generalized Bellman Operator and Stochastic Approximation . In this section , we illustrate the interpretation of off-policy multi-step TD-learning as an SA algorithm for solving a generalized Bellman equation . Consider the policy evaluation problem where the goal is to estimate the state-action value function Qπ of a given policy π . In the simplest setting where TD ( 0 ) with on-policy sampling is employed , it is well known that the algorithm is an SA algorithm for solving the Bellman equationQ = Hπ ( Q ) , whereHπ ( · ) is the Bellman operator . The generalized Bellman operator B ( · ) we consider in this paper is defined by : B ( Q ) = T ( H ( Q ) −Q ) +Q , ( 1 ) where T ( · ) and H ( · ) are two auxiliary operators . In the special case where T ( · ) = I ( · ) and H ( · ) = Hπ ( · ) , the generalized Bellman operator B ( · ) reduces to the regular Bellman operatorHπ ( · ) . Note that any fixed point ofH ( · ) is also a fixed point of B ( · ) , as long as T ( · ) is such that T ( 0 ) = 0 . Thus , the operatorH ( · ) controls the fixed-point of the generalized Bellman operator B ( · ) , and as we will see later , the operator T ( · ) can be used to control its contraction properties . To further understand the operatorB ( · ) , we demonstrate in the following that both on-policy n-step TD and TD ( λ ) can be viewed as SA algorithms for solving the generalized Bellman equation B ( Q ) = Q , with different auxiliary operators T ( · ) and H ( · ) . On-policy n-step TD is designed to solve the n-step Bellman equation Q = Hnπ ( Q ) , which can be explicitly written as Q = ∑n−1 i=0 ( γPπ ) iR + ( γPπ ) nQ . Here R is the reward vector , γ is the discount factor , and Pπ is the transition probability matrix under policy π . By reverse telescoping , the n-step Bellman equation is equivalent to Q =∑n−1 i=0 ( γPπ ) i ( R + γPπQ − Q ) + Q = T ( Hπ ( Q ) − Q ) + Q , where T ( Q ) = ∑n−1 i=0 ( γPπ ) iQ . Similarly , one can formulate the TD ( λ ) Bellman equation in the form of B ( Q ) = Q , where T ( Q ) = ( 1− λ ) ∑∞ i=0 λ i ∑i−1 j=0 ( γPπ ) iQ andH ( · ) = Hπ ( · ) . In these examples , the operator T ( · ) determines the contraction factor of B ( · ) by controlling the degree of bootstrapping . In this work , we show that in addition to on-policy TD-learning , variants of off-policy TD-learning with multi-step bootstrapping and generalized importance sampling ratios can also be interpreted as SA algorithms for solving the generalized Bellman equation . Moreover , under some mild conditions , we show that the generalized Bellman operator B ( · ) is a contraction mapping with respect to some weighted ` p-norm for any p ∈ [ 1 , ∞ ) , with a common contraction factor . This enables us to establish finite-sample bounds of general multi-step off-policy TD-like algorithms . 1.3 Related Literature . The TD-learning method was first proposed in [ 30 ] for solving the policy evaluation problem . Since then , there is an increasing interest in theoretically understanding TD-learning and its variants . On-Policy TD-Learning . The most basic TD-learning method is the TD ( 0 ) algorithm [ 30 ] . Later it was extended to using multi-step bootstrapping ( i.e. , the n-step TD-learning algorithm [ 11 , 37 , 38 ] ) , and using eligibility trace ( i.e. , the TD ( λ ) algorithm [ 28 , 30 ] ) . The asymptotic convergence of TDlearning was established in [ 13 , 18 , 35 ] . As for finite-sample analysis , a unified Lyapunov approach is presented in [ 10 ] . To overcome the curse of dimensionality in RL , TD-learning is usually incorporated with function approximation in practice . In the basic setting where a linear parametric architecture is used , the asymptotic convergence of TD-learning was established in [ 36 ] , and finite-sample bounds in [ 5 , 12 , 29 , 34 ] . Very recently , the convergence and finite-sample guarantee of TD-learning with neural network approximation were studied in [ 7 , 8 ] . Off-Policy TD-Learning . In the off-policy setting , since the samples are not necessarily generated by the target policy , usually importance sampling ratios ( or “ generalized ” importance sampling ratios ) are introduced in the TD-learning algorithm . The resulting algorithms are Qπ ( λ ) [ 24 ] , TB ( λ ) [ 17 ] , Retrace ( λ ) [ 22 ] , and Q-trace [ 19 ] ( which is an extension of V -trace [ 14 ] ) , etc . The asymptotic convergence of these algorithms has been established in the papers in which they were proposed . To the best of our knowledge , finite-sample guarantees are established only for Q-trace and V -trace [ 9 , 10 , 19 ] . In the function approximation setting , TD-learning with off-policy sampling and function approximation is a typical example of the deadly triad [ 31 ] , and can be unstable [ 2 , 31 ] . To achieve convergence , one needs to significantly modify the original TD-learning algorithm , resulting in two time-scale algorithms such as GTD [ 21 ] , TDC [ 32 ] , and emphatic TD [ 33 ] , etc . 1.4 Preliminaries . In this section , we cover the background of RL and the TD-learning method for solving the policy evaluation problem . The RL problem is usually modeled as a Markov decision process ( MDP ) . In this work , we consider an MDP with a finite set of states S , a finite set of actionsA , a set of unknown action dependent transition probability matrices P = { Pa ∈ R|S|×|S| | a ∈ A } , an unknown reward functionR : S ×A 7→ [ 0 , 1 ] , and a discount factor γ ∈ ( 0 , 1 ) . In order for an MDP to progress , we must specify the policy of selecting actions based on the state of the environment . Specifically , a policy π is a mapping from the state-space to probability distributions supported on the action space , i.e. , π : S 7→ ∆|A| . The state-action value function Qπ associated with a policy π is defined by Qπ ( s , a ) = Eπ [ ∑∞ k=0 γ kR ( Sk , Ak ) | S0 = s , A0 = a ] for all ( s , a ) . The goal in policy evaluation is to estimate the state-action value function Qπ for a given policy π . Since the transition probabilities as well as the reward function are unknown , such state-action value function can not be directly computed . The TD-learning algorithm is designed to estimate Qπ using the SA method . Specifically , in TD-learning , we first collect a sequence of samples { ( Sk , Ak ) } from the model using some behavior policy πb . Then the value function Qπ is iteratively estimated using the samples { ( Sk , Ak ) } . When πb = π , the algorithm is called on-policy TD-learning , otherwise the algorithm is referred to as off-policy TD-learning . 2 Finite-Sample Analysis of General Off-Policy TD-Learning . In this section , we present our unified framework for finite-sample analysis of off-policy TD-learning algorithms using generalized importance sampling ratios and multi-step bootstrapping . The proofs of all technical results presented in this paper are provided in the Appendix .

Summary: The authors propose a unified framework for understanding several existing tabular off-policy TD algorithms for policy evaluation. Central to this new framework is a generalized Bellman equation and its contraction. The unified framework provides new insights about the bias-variance trade-off of those existing algorithms, as well as finite sample analysis.