Finite Sample Analysis of Average-Reward TD Learning and $Q$-Learning

1 Introduction . The average-reward setting is a classical setting for formulating the goal in an infinite-horizon Markov decision process ( MDP ) [ 1 ] . The need to maximize the average reward has been demonstrated in many applications , including scheduling automatic guided vehicles [ 2 ] , inventory management in supply chains [ 3 ] , communication system control and routing [ 4 ] , cooperative multi-robot learning [ 5 ] and queuing network control [ 6 ] . In these problems , the discounted-reward criterion usually leads to poor long-time performance since the system operates over an extended period of time and the goal is to optimize long-term behavior , whereas the discounted objective biases the optimal policy to choose actions that lead to high near-term performance . Even though there is a well developed theory of average-reward MDPs [ 7 , 8 , 9 ] , the theoretical understanding of average-reward RL methods is still quite limited . Most existing results are focused only on asymptotic convergence [ 10 , 11 , 12 , 13 ] . The focus of this paper is to understand the sample efficiency . How much data is required to guarantee a given level of accuracy ? Recent literature obtains finite sample guarantees for discounted-reward TD learning and Q-learning algorithms by developing novel analytical techniques [ 14 , 15 , 16 , 17 , 18 ] . Such a study of averagereward RL algorithms is not undertaken . Analysis of average-reward RL algorithms is known to be more challenging to study than their discounted-reward counterparts . The key property that is exploited in the study of discounted-reward problems is the contraction property of the underlying Bellman operator . In the average-reward setting , such a contraction property does not hold under any norm , and the Bellman equation is known to have multiple fixed points . ∗Equal contribution . Correspondence to Sheng Zhang < shengzhang @ gatech.edu | shengzhangblog.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we take the first step toward understanding finite sample guarantees of ( i ) averagereward TD ( λ ) with linear function approximation for policy evaluation , and ( ii ) average-reward tabular Q-learning in the synchronous setting for the control problem . 1.1 Contributions and Summary of Our Techniques . We establish the first finite sample convergence guarantees of average-reward TD ( λ ) with linear function approximation and average-reward tabular Q-learning in the literature . TD ( λ ) Results . We study the average-reward TD ( λ ) with linear function approximation under a general asynchronous update . We present finite sample bounds under both constant and diminishing step sizes . With a constant step-size , the iterates converge at an exponential rate to a small cylinder around the set of TD fixed points . With properly chosen diminishing step sizes , the mean-square distance of the iterates to the set of TD fixed points converges with an Õ ( 1 T ) rate , and this leads to a sample complexity of Õ ( 1 2 ) . Our sample complexity bound also suggests that an intermediate value of λ yields the best performance . The dependence on the effective horizon plays a key role in the study of discounted-reward RL algorithms [ 19 , 20 ] . There is no such effective horizon in average-reward problems , and the spectral gap of an appropriately defined matrix plays a key role instead . TD ( λ ) Analysis . A major challenge in the analysis is that the projected Bellman operator is not a contraction under any norm . Moreover , even though the projected Bellman equation can be written as a linear set of equations , they are underdetermined . So existing techniques [ 14 , 15 ] are not directly applicable . Since the value function is unique up to an additive constant , we have a unique solution of the projected Bellman equation when restricted to an appropriately defined subspace . We exploit this property and work in this subspace , and use a quadratic Lyapunov function to obtain finite sample guarantees . Q-learning Results . We consider a J-step synchronous Q-learning algorithm . We present finite sample error bounds for both constant and diminishing step sizes . The span of a vector is defined to be the difference between the maximum and minimum element . Since the optimal action-value function , Q∗ , is agnostic up to an additive constant , we show that , with at most O ( 1/ 2 ) samples , the expected span of the error QT −Q∗ converges to for both decreasing and constant step sizes . Q-learning Analysis . While the corresponding Bellman operator is not a contraction under any norm , it is known to be a contraction under the span seminorm . The span seminorm can be interpreted as the ` ∞-distance to the subspace spanned by the all-ones vector . Finite sample bounds for stochastic approximation of ` ∞-norm contractive operators were obtained in [ 18 ] by using generalized Moreau envelop as a smooth Lyapunov function . Here , we generalize this approach and introduce a new Lyapunov function to study span seminorm contractive operators . Our Lyapunov function is obtained by applying an infimal convolution with respect to an indicator function to the generalized Moreau envelop used in [ 18 ] . 1.2 Related Literature . Average-Reward MDP . There is an extensive body of literature on average-reward MDPs . Several authors have made early contributions to average-reward problems [ 21 , 7 , 8 , 22 , 23 ] . There are well known dynamic programming algorithms for finding optimal policies such as policy iteration [ 7 ] and value iteration [ 24 ] . However , these algorithms require complete knowledge of the MDP , and are also computationally intractable in large state spaces [ 25 ] . Average-Reward Policy Evaluation . Tsitsiklis and Van Roy [ 10 ] proved the asymptotic convergence of the average-reward TD ( λ ) with linear function approximation , and provided approximation error bounds . Yu and Bertsekas [ 26 ] proved the asymptotic convergence of the average-reward LSPE ( λ ) , and provided the rate of convergence for constant step size . Both TD ( λ ) with linear function approximation and LSPE ( λ ) aim to solve the same projected Bellman equation . However , TD ( λ ) is based on stochastic approximation while LSPE ( λ ) is based on least squares . In addition , the papers above assumed that the set of basis functions are independent of the all-ones vector , which apparently does not hold in the tabular setting . We do not require such a restrictive assumption in this paper . Recent work has also established the asymptotic convergence of the off-policy average-reward TD learning algorithm in the tabular setting [ 12 ] , and finite sample guarantees of average-reward gradient TD algorithms with linear function approximation [ 13 ] . Note that , [ 13 ] needed either the restrictive assumption aforementioned or the ridge regularization to obtain the finite sample results . Average-Reward Control . The earliest control algorithms were those introduced by Schwartz [ 27 ] and Singh [ 28 ] without convergence proofs . The first provably convergent algorithms are RVI Qlearning and SSP Q-learning , introduced by Abounadi , Bertsekas , and Borkar [ 11 ] . SSP Q-learning and the algorithm introduced later by Gosavi [ 29 ] are limited to MDPs with a special state that is recurrent under all stationary policies , whereas RVI Q-learning is convergent for more general MDPs . Recently , Wan et al . [ 12 ] introduced an algorithm without a reference function , which is needed in RVI Q-learning , and proved its asymptotic convergence with the techniques which are a slight generalization of those in [ 11 ] . To the best of our knowledge , our paper is the first work in the literature that studies the finite sample guarantees of a general average-reward Q-learning algorithm . Stochastic Approximation . Many RL algorithms can be viewed through the lens of stochastic approximation ( SA ) . There is a well developed asymptotic theory of SA [ 30 , 31 , 32 ] . The ODE method is a dominant approach used in most asymptotic convergence proofs in RL [ 33 ] . However , this is a coarse tool , since it is not able to generate insight into an algorithm ’ s sensitivity to noise in the system and step-size choices . Driven by the interest in finite sample guarantees of RL algorithms , recent years have witnessed a focus shifted from asymptotic analysis to non-asymptotic analysis of SA schemes . For example , a finite-time bound for linear SA was given in [ 15 ] , which leads to finite-time bounds for asynchronous TD learning . [ 17 ] provided a finite-time analysis of asynchronous nonlinear SA , which yields finite-time bounds for asynchronous Q-learning . Others . There are other related papers which are beyond the scope of the present paper . For instance , there is a line of work [ 34 , 35 , 36 ] on regret guarantees , which is a different focus compared to our work , for learning in average-reward MDPs . In addition , there are RL methods based on linear programming [ 37 , 38 ] , or learning automata [ 39 , 40 ] . 2 The Average-Reward Problem Setting . We consider an infinite-horizon average-reward MDP described by ( S , A , R , p ) , where S = { 1 , 2 , · · · , |S| } is a finite state space , A = { 1 , 2 , · · · , |A| } is a finite action space , R : S×A → [ 0 , 1 ] is the reward function , and p : S × S × A → [ 0 , 1 ] is the transition dynamics of the environment . An agent interacts with the environment according to the following protocol : at each time step t = 0 , 1 , 2 , · · · , the agent is in a state St ∈ S and selects an action At ∈ A , then receives from the environment an immediate rewardR ( St , At ) and the next state St+1 which is a sample drawn from p ( ·|St , At ) . The average reward of a deterministic stationary policy µ : S → A starting from state s ∈ S is defined as rµ ( s ) : = lim inf T→∞ 1 T E [ T−1∑ t=0 R ( St , µ ( St ) ) |S0 = s ] . ( 2.1 ) Let r∗ ( s ) : = supµ∈M r µ ( s ) , where M is the set of deterministic stationary policies . A policy µ∗ ∈M is said to be optimal if it satisfies rµ∗ ( s ) = r∗ ( s ) for all s ∈ S . 3 Policy Evaluation Algorithm : TD Learning . 3.1 Problem Formulation . We consider the problem of evaluating a given policy µ ∈M when the data is generated by applying the policy µ in the MDP . Since the system is an induced Markov reward process ( MRP ) , for simplicity , we employ the notation R ( i ) : = R ( i , µ ( i ) ) for rewards , and P ( i , j ) : = p ( j|i , µ ( i ) ) for transition probabilities . We make the following standard assumption to ensure the existence and uniqueness of a stationary distribution π : = [ π1 , · · · , π|S| ] > . Assumption 1 . The Markov chain associated with P is irreducible and aperiodic . It is a standard assumption in studying convergence of TD learning algorithms with linear function approximation [ 41 , 10 , 14 ] and guarantees that all states are visited an infinite number of times during an infinitely long trajectory . Notice that π satisfies π > P = π > , with πi > 0 for all i ∈ S. Let Eπ [ · ] denote expectation with respect to π and define D = diag ( π1 , · · · , π|S| ) ∈ R|S|×|S| . It is easy to see that 〈x , y〉D : = x > Dy is a D-weighted inner product and we denote its induced norm by ‖x‖D : = √ x > Dx . Under Assumption 1 , the average reward in ( 2.1 ) satisfies rµ ( s ) = r ( µ ) : = π > R for all s ∈ S. A differential value function v : S → R for policy µ satisfies the Bellman equation Tv = v , where the Bellman operator T : R|S| → R|S| is defined by Tv : = R− r ( µ ) e+ Pv . Here , e ∈ R|S| is the all-ones vector . Under Assumption 1 , it is known that the set of differential value functions takes the form { vµ + ce|c ∈ R } , where vµ : S → R , known as the basic differential value function , is given by vµ : = ∑∞ t=0 P t ( R− r ( µ ) e ) . Most modern applications have large state spaces , so due to the curse of dimensionality , exact value function learning may be intractable . To mitigate this , we consider a linear function approximation Vθ ( i ) = φ ( i ) > θ to differential value functions , where φ ( i ) : = [ φ1 ( i ) , · · · , φd ( i ) ] > ∈ Rd is the feature vector for state i ∈ S and θ ∈ Rd is a tunable parameter vector . Here , { φk : S → R|k = 1 , 2 , · · · , d } is a set of d basis functions to be viewed as vectors of dimension |S| . With this notation , Vθ can be expressed compactly in the form Vθ = Φθ , where Φ is an |S|×dmatrix whose k-th column is φk . We assume that Φ has full column rank ; that is , the basis functions { φk|k = 1 , 2 , · · · , d } are linearly independent . This results in no loss of generality because if some basis function φk is a linear combination of the others , it can be eliminated without changing the power of the approximation architecture . Additionally , we assume that ‖φ ( i ) ‖2 ≤ 1 for all i ∈ S , which can be ensured through feature normalization .

Summary: This paper focuses on the theoretical analysis (especially on the non-asymptotic sample efficiency) of two algorithms under the average-reward setting: (1) TD($\lambda$) with linear function approximation under Markovian observation noise (2) tabular Q-learning in the synchronous setting.  Previous works have obtained similar finite-sample guarantees under the discounted setting. However, this work focuses on the average-reward setting where the Bellman equation is known to have multiple fixed points. For the TD($\lambda$) analysis, the authors work in a subspace to get the unique solution of the projected Bellman equation. For the Q-Learning case, the Bellman operator is a contraction under the span seminorm.