Finite Sample Analysis of Average-Reward TD Learning and $Q$-Learning

1 Introduction . The average-reward setting is a classical setting for formulating the goal in an infinite-horizon Markov decision process ( MDP ) [ 1 ] . The need to maximize the average reward has been demonstrated in many applications , including scheduling automatic guided vehicles [ 2 ] , inventory management in supply chains [ 3 ] , communication system control and routing [ 4 ] , cooperative multi-robot learning [ 5 ] and queuing network control [ 6 ] . In these problems , the discounted-reward criterion usually leads to poor long-time performance since the system operates over an extended period of time and the goal is to optimize long-term behavior , whereas the discounted objective biases the optimal policy to choose actions that lead to high near-term performance . Even though there is a well developed theory of average-reward MDPs [ 7 , 8 , 9 ] , the theoretical understanding of average-reward RL methods is still quite limited . Most existing results are focused only on asymptotic convergence [ 10 , 11 , 12 , 13 ] . The focus of this paper is to understand the sample efficiency . How much data is required to guarantee a given level of accuracy ? Recent literature obtains finite sample guarantees for discounted-reward TD learning and Q-learning algorithms by developing novel analytical techniques [ 14 , 15 , 16 , 17 , 18 ] . Such a study of averagereward RL algorithms is not undertaken . Analysis of average-reward RL algorithms is known to be more challenging to study than their discounted-reward counterparts . The key property that is exploited in the study of discounted-reward problems is the contraction property of the underlying Bellman operator . In the average-reward setting , such a contraction property does not hold under any norm , and the Bellman equation is known to have multiple fixed points . ∗Equal contribution . Correspondence to Sheng Zhang < shengzhang @ gatech.edu | shengzhangblog.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we take the first step toward understanding finite sample guarantees of ( i ) averagereward TD ( λ ) with linear function approximation for policy evaluation , and ( ii ) average-reward tabular Q-learning in the synchronous setting for the control problem . 1.1 Contributions and Summary of Our Techniques . We establish the first finite sample convergence guarantees of average-reward TD ( λ ) with linear function approximation and average-reward tabular Q-learning in the literature . TD ( λ ) Results . We study the average-reward TD ( λ ) with linear function approximation under a general asynchronous update . We present finite sample bounds under both constant and diminishing step sizes . With a constant step-size , the iterates converge at an exponential rate to a small cylinder around the set of TD fixed points . With properly chosen diminishing step sizes , the mean-square distance of the iterates to the set of TD fixed points converges with an Õ ( 1 T ) rate , and this leads to a sample complexity of Õ ( 1 2 ) . Our sample complexity bound also suggests that an intermediate value of λ yields the best performance . The dependence on the effective horizon plays a key role in the study of discounted-reward RL algorithms [ 19 , 20 ] . There is no such effective horizon in average-reward problems , and the spectral gap of an appropriately defined matrix plays a key role instead . TD ( λ ) Analysis . A major challenge in the analysis is that the projected Bellman operator is not a contraction under any norm . Moreover , even though the projected Bellman equation can be written as a linear set of equations , they are underdetermined . So existing techniques [ 14 , 15 ] are not directly applicable . Since the value function is unique up to an additive constant , we have a unique solution of the projected Bellman equation when restricted to an appropriately defined subspace . We exploit this property and work in this subspace , and use a quadratic Lyapunov function to obtain finite sample guarantees . Q-learning Results . We consider a J-step synchronous Q-learning algorithm . We present finite sample error bounds for both constant and diminishing step sizes . The span of a vector is defined to be the difference between the maximum and minimum element . Since the optimal action-value function , Q∗ , is agnostic up to an additive constant , we show that , with at most O ( 1/ 2 ) samples , the expected span of the error QT −Q∗ converges to for both decreasing and constant step sizes . Q-learning Analysis . While the corresponding Bellman operator is not a contraction under any norm , it is known to be a contraction under the span seminorm . The span seminorm can be interpreted as the ` ∞-distance to the subspace spanned by the all-ones vector . Finite sample bounds for stochastic approximation of ` ∞-norm contractive operators were obtained in [ 18 ] by using generalized Moreau envelop as a smooth Lyapunov function . Here , we generalize this approach and introduce a new Lyapunov function to study span seminorm contractive operators . Our Lyapunov function is obtained by applying an infimal convolution with respect to an indicator function to the generalized Moreau envelop used in [ 18 ] . 1.2 Related Literature . Average-Reward MDP . There is an extensive body of literature on average-reward MDPs . Several authors have made early contributions to average-reward problems [ 21 , 7 , 8 , 22 , 23 ] . There are well known dynamic programming algorithms for finding optimal policies such as policy iteration [ 7 ] and value iteration [ 24 ] . However , these algorithms require complete knowledge of the MDP , and are also computationally intractable in large state spaces [ 25 ] . Average-Reward Policy Evaluation . Tsitsiklis and Van Roy [ 10 ] proved the asymptotic convergence of the average-reward TD ( λ ) with linear function approximation , and provided approximation error bounds . Yu and Bertsekas [ 26 ] proved the asymptotic convergence of the average-reward LSPE ( λ ) , and provided the rate of convergence for constant step size . Both TD ( λ ) with linear function approximation and LSPE ( λ ) aim to solve the same projected Bellman equation . However , TD ( λ ) is based on stochastic approximation while LSPE ( λ ) is based on least squares . In addition , the papers above assumed that the set of basis functions are independent of the all-ones vector , which apparently does not hold in the tabular setting . We do not require such a restrictive assumption in this paper . Recent work has also established the asymptotic convergence of the off-policy average-reward TD learning algorithm in the tabular setting [ 12 ] , and finite sample guarantees of average-reward gradient TD algorithms with linear function approximation [ 13 ] . Note that , [ 13 ] needed either the restrictive assumption aforementioned or the ridge regularization to obtain the finite sample results . Average-Reward Control . The earliest control algorithms were those introduced by Schwartz [ 27 ] and Singh [ 28 ] without convergence proofs . The first provably convergent algorithms are RVI Qlearning and SSP Q-learning , introduced by Abounadi , Bertsekas , and Borkar [ 11 ] . SSP Q-learning and the algorithm introduced later by Gosavi [ 29 ] are limited to MDPs with a special state that is recurrent under all stationary policies , whereas RVI Q-learning is convergent for more general MDPs . Recently , Wan et al . [ 12 ] introduced an algorithm without a reference function , which is needed in RVI Q-learning , and proved its asymptotic convergence with the techniques which are a slight generalization of those in [ 11 ] . To the best of our knowledge , our paper is the first work in the literature that studies the finite sample guarantees of a general average-reward Q-learning algorithm . Stochastic Approximation . Many RL algorithms can be viewed through the lens of stochastic approximation ( SA ) . There is a well developed asymptotic theory of SA [ 30 , 31 , 32 ] . The ODE method is a dominant approach used in most asymptotic convergence proofs in RL [ 33 ] . However , this is a coarse tool , since it is not able to generate insight into an algorithm ’ s sensitivity to noise in the system and step-size choices . Driven by the interest in finite sample guarantees of RL algorithms , recent years have witnessed a focus shifted from asymptotic analysis to non-asymptotic analysis of SA schemes . For example , a finite-time bound for linear SA was given in [ 15 ] , which leads to finite-time bounds for asynchronous TD learning . [ 17 ] provided a finite-time analysis of asynchronous nonlinear SA , which yields finite-time bounds for asynchronous Q-learning . Others . There are other related papers which are beyond the scope of the present paper . For instance , there is a line of work [ 34 , 35 , 36 ] on regret guarantees , which is a different focus compared to our work , for learning in average-reward MDPs . In addition , there are RL methods based on linear programming [ 37 , 38 ] , or learning automata [ 39 , 40 ] . 2 The Average-Reward Problem Setting . We consider an infinite-horizon average-reward MDP described by ( S , A , R , p ) , where S = { 1 , 2 , · · · , |S| } is a finite state space , A = { 1 , 2 , · · · , |A| } is a finite action space , R : S×A → [ 0 , 1 ] is the reward function , and p : S × S × A → [ 0 , 1 ] is the transition dynamics of the environment . An agent interacts with the environment according to the following protocol : at each time step t = 0 , 1 , 2 , · · · , the agent is in a state St ∈ S and selects an action At ∈ A , then receives from the environment an immediate rewardR ( St , At ) and the next state St+1 which is a sample drawn from p ( ·|St , At ) . The average reward of a deterministic stationary policy µ : S → A starting from state s ∈ S is defined as rµ ( s ) : = lim inf T→∞ 1 T E [ T−1∑ t=0 R ( St , µ ( St ) ) |S0 = s ] . ( 2.1 ) Let r∗ ( s ) : = supµ∈M r µ ( s ) , where M is the set of deterministic stationary policies . A policy µ∗ ∈M is said to be optimal if it satisfies rµ∗ ( s ) = r∗ ( s ) for all s ∈ S . 3 Policy Evaluation Algorithm : TD Learning . 3.1 Problem Formulation . We consider the problem of evaluating a given policy µ ∈M when the data is generated by applying the policy µ in the MDP . Since the system is an induced Markov reward process ( MRP ) , for simplicity , we employ the notation R ( i ) : = R ( i , µ ( i ) ) for rewards , and P ( i , j ) : = p ( j|i , µ ( i ) ) for transition probabilities . We make the following standard assumption to ensure the existence and uniqueness of a stationary distribution π : = [ π1 , · · · , π|S| ] > . Assumption 1 . The Markov chain associated with P is irreducible and aperiodic . It is a standard assumption in studying convergence of TD learning algorithms with linear function approximation [ 41 , 10 , 14 ] and guarantees that all states are visited an infinite number of times during an infinitely long trajectory . Notice that π satisfies π > P = π > , with πi > 0 for all i ∈ S. Let Eπ [ · ] denote expectation with respect to π and define D = diag ( π1 , · · · , π|S| ) ∈ R|S|×|S| . It is easy to see that 〈x , y〉D : = x > Dy is a D-weighted inner product and we denote its induced norm by ‖x‖D : = √ x > Dx . Under Assumption 1 , the average reward in ( 2.1 ) satisfies rµ ( s ) = r ( µ ) : = π > R for all s ∈ S. A differential value function v : S → R for policy µ satisfies the Bellman equation Tv = v , where the Bellman operator T : R|S| → R|S| is defined by Tv : = R− r ( µ ) e+ Pv . Here , e ∈ R|S| is the all-ones vector . Under Assumption 1 , it is known that the set of differential value functions takes the form { vµ + ce|c ∈ R } , where vµ : S → R , known as the basic differential value function , is given by vµ : = ∑∞ t=0 P t ( R− r ( µ ) e ) . Most modern applications have large state spaces , so due to the curse of dimensionality , exact value function learning may be intractable . To mitigate this , we consider a linear function approximation Vθ ( i ) = φ ( i ) > θ to differential value functions , where φ ( i ) : = [ φ1 ( i ) , · · · , φd ( i ) ] > ∈ Rd is the feature vector for state i ∈ S and θ ∈ Rd is a tunable parameter vector . Here , { φk : S → R|k = 1 , 2 , · · · , d } is a set of d basis functions to be viewed as vectors of dimension |S| . With this notation , Vθ can be expressed compactly in the form Vθ = Φθ , where Φ is an |S|×dmatrix whose k-th column is φk . We assume that Φ has full column rank ; that is , the basis functions { φk|k = 1 , 2 , · · · , d } are linearly independent . This results in no loss of generality because if some basis function φk is a linear combination of the others , it can be eliminated without changing the power of the approximation architecture . Additionally , we assume that ‖φ ( i ) ‖2 ≤ 1 for all i ∈ S , which can be ensured through feature normalization .

Summary: The authors provide a finite sample analysis for average reward TD with linear function approximation. Importantly, the authors succeeded in eliminating a commonly used assumption. The authors also provide a J-step synchronous average reward Q-learning and a corresponding finite sample analysis. One central idea to the success of the paper is to work on equivalent classes directly to deal with the challenge that differential Bellman equation has multiple solutions.