Title: Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: Average Reward MDPs

Reinforcement Learning Theory

Approximate Policy Iteration

Policy Based Methods

Performance Bounds
Abstract: Many policy-based reinforcement learning (RL) algorithms can be viewed as instantiations of approximate policy iteration (PI), i.e., where policy improvement and policy evaluation are both performed approximately. In applications where the average reward objective is the meaningful performance metric, often discounted reward formulations are used with the discount factor being close to $1,$ which is equivalent to making the expected horizon very large. However, the corresponding theoretical bounds for error performance scale with the square of the horizon. Thus, even after dividing the total reward by the length of the horizon, the corresponding performance bounds for average reward problems go to infinity. Therefore, an open problem has been to obtain meaningful performance bounds for approximate PI and RL algorithms for the average-reward setting.  In this paper, we solve this open problem by obtaining the first non-trivial finite time error bounds for average-reward MDPs which go to zero in the limit as policy evaluation and policy improvement errors go to zero.

Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms

Abstract

Many policy-based reinforcement learning (RL) algorithms can be viewed as instantiations of approximate policy iteration (PI), i.e., where policy improvement and policy evaluation are both performed approximately. In applications where the average reward objective is the meaningful performance metric, discounted reward formulations are often used with the discount factor being close to 1 , which is equivalent to making the expected horizon very large. However, the corresponding theoretical bounds for error performance scale with the square of the horizon. Thus, even after dividing the total reward by the length of the horizon, the corresponding performance bounds for average reward problems go to infinity. Therefore, an open problem has been to obtain meaningful performance bounds for approximate PI and RL algorithms for the average-reward setting. In this paper, we solve this open problem by obtaining the first finite-time error bounds for average-reward MDPs, and show that the asymptotic error goes to zero in the limit as policy evaluation and policy improvement errors go to zero.

1 Introduction

Reinforcement Learning algorithms can be broadly classified into value-based methods and policybased methods. In the case of discounted-reward Markov Decision Processes (MDPs), value-based methods such as Q-learning [WD92, Tsi94, JJS93, SB18, BT96], fitted value iteration [MS08] and target network Q-learning [CCM22] can be viewed as approximately solving the fixed point of the Bellman optimality equation to find the value function and an approximately optimal control policy. In other words value iteration is approximately implemented for discounted reward MDPs [Ber11, Ber12, Put14]. In policy-gradient methods [AKLM21], a gradient step is used to improve the policy, somewhat similar to the policy improvement step in policy iteration for discounted-reward MDPs. In a recent paper [CM22], it has been shown that many policy-based methods for discountedreward MDPs can be viewed as special cases of approximate policy iteration. The classical results on approximate policy iteration [BT96] assume that the error in the policy evaluation and improvement steps are constant, independent of the iteration. The key idea in [CM22] is to show that a simple modification of the proof in [BT96] can be used to allow for iteration-dependent policy evaluation and improvement errors, which is then used to make the connection to policy-based methods. Our

goal in this paper is to derive similar results for average-reward problems. Average-reward problems [ABB01, WNS21, ZWSW21, Mah96, YB09, Gos04, Sin94] are, in some sense, harder to study than their discounted-reward counterparts. We now discuss why this is so by recalling the error bounds for discounted-reward problems and examining them in an appropriate limit to study their applicability to average-reward problems.

The fundamental result on approximate policy iteration for discounted reward MDPs which allows the connection to policy-based methods is the following [BT96]:

where α is the discount factor, H α = 1 / (1 -α ) , J α ∗ is the optimal value function, J µ k is the value function associated with the policy obtained after k iterations of approximate policy iteration, ϵ is the policy improvement error, and δ is the policy evaluation error. Thus, as ϵ, δ → 0 , we recover the result that standard policy iteration converges. On the other hand, to understand whether the above bound is useful to study average-reward problems, we write Equation (1) as

Under mild conditions [Ber11, Ros14], it is well known that each element of the value function vector approaches the average-reward for each fixed policy and for the optimal policy, in the limit α → 1 . Hence, the left-hand side of the above equation is an approximation to the error in approximate policy iteration for average-reward MDPs; the right-hand side which gives an upper bound on this error blows up to infinity, i.e., when α → 1 . Thus, unlike in the discounted-reward case, the above bound fails to even recover the well-known convergence of standard policy iteration in average-reward case in the limit ϵ, δ → 0 (since we have to let α → 1 before letting ϵ, δ → 0 ). However, as mentioned in [BT96], it is also well known that approximate policy iteration performs much better than the bound suggests. The main goal of our paper is to resolve this discrepancy between theory and practice.

1.1 Contributions

It is well known that the error bound for discounted-reward, approximate policy iteration is tight for unichain MDPs [BT96]. Thus, it is impossible to improve upon the bound in general. However, as will be explained in a later section, it is easy to convert most reasonable unichain MDPs to MDPs where every stationary policy results in an irreducible Markov chain with an arbitrarily small loss of reward. So the natural question to ask is whether the bound can be dramatically improved for MDPs where every stationary policy results in an irreducible Markov chain. To the best of our knowledge, no such bound was available in the prior literature.

Our main contributions are as follows:

Under the assumption that every stationary policy induces an irreducible Markov chain, we first perform a Schweitzer transformation of the MDP [Sch71] and obtain finite-time error bounds for average-reward approximate policy iteration.

Using the above finite-time error bounds, we prove an error bound of the form

for average-reward approximate policy iteration for a function f such that f ( ϵ, δ ) → 0 in the limit as ϵ, δ → 0 . Note that this is in sharp contrast to the result that one can obtain from the discounted-reward analysis where the error bound blows up to infinity when applied to discount factors approaching 1 , which is the appropriate limit to convert discounted-reward problems to average-reward problems.

The main difficulty in obtaining the above bound when compared to the discounted-reward case is the lack of infinity-norm contraction property for the Bellman operator in the averagereward case. In most analysis of average-reward problems, this difficulty is circumvented by the use of a span-contraction property [Put14]. However, the span contraction property is insufficient for our purpose and therefore, we use a technique due to [VdW80] for the study of a different algorithm called modified policy iteration in [Put14].

Next, we extend the above analysis to obtain finite-iteration error bounds for the case where the policy evaluation and improvement errors can potentially vary from iteration to iteration. Further, we allow these errors to be random (as would be the case, for example, when one uses TD learning for policy evaluation and soft policy improvement) and obtain expected error bounds on approximate policy iteration after a finite number of iterations.

Our error bounds are in a form such that one can then apply them to many policy-based RL algorithms, i.e., we can simply plug in error bounds for specific policy evaluation or policy improvement algorithms. We illustrate the applicability of the results to several different policy-based RL methods (softmax update, greedy update and mirror descent update) studied in the literature [MXSS20, BT96, AYBB + 19, AKLM21].

While the main focus of our paper is on offline learning, we provide connections to regret bounds for online learning in average-reward MDPS as in [AYBB + 19, LYAYS21].

1.2 Related Work

Our work in this paper is most closely related to the work in [CM22]. The main contribution there was to recognize that an extension of the approximate policy iteration results in [BT96] can be used to derive finite-time error bounds for many RL-based algorithms. The idea of considering iteration-dependent policy evaluation and improvement bounds was also used to study policy iteration with function approximation in discounted-reward MDPs in [WLLS21]. Our contribution here is to derive the first error bound for approximate policy iteration for average-reward MDPs and then write it in a form which can be applied to policy-based RL methods for average-reward TD learning. To the best of our knowledge, no known error bounds existed in prior literature for average-reward approximate policy iteration due to the fact that the corresponding bounds for discounted MDPs increase proportional to the square of length of the horizon. Thus, in the limit as the horizon increases to infinity, we have essentially reduced the dependence of the error from the square of the length of the horizon to linear in the length of the horizon.

RL algorithms have not been studied as extensively for the average-reward case as they are for discounted-reward MDPs. There is recent work [ZZM21] on average-reward TD-learning which we leverage to illustrate the applicability of our results. There are also asymptotic convergence results for policy gradient and actor-critic methods for average-reward MDPs [TVR99, KT99, BSGL09, MMT00]. However, these results only prove convergence to a stationary point and there are no global convergence results. The original paper on natural policy gradient (NPG) is written for average-reward MDPs, but the extensive performance analysis of NPG in subsequent work such as [AKLM21] seems to only consider the discounted-reward case. Recent work [AYBB + 19, LYAYS21] considers mirror descent in average reward RL but do not provide performance results for other RL algorithms. In a later section, we compare our results to [AYBB + 19, LYAYS21] in the special case of mirror descent-type policy updates

2 Model and Preliminaries

2.1 Average Reward Formulation

We consider the class of infinite horizon MDPs with finite state space S , finite action space A , and transition kernel P , where |S| = n and |A| = m . Let ∆ A denote the probability simplex over actions. We consider the class of randomized policies Π = { µ : S → ∆ A} , so that a policy µ assigns a probability vector over actions to each state. Given a policy µ , the transition kernel for the underlying Markov process is denoted by P µ : S → S , where P µ ( s ′ | s ) := ∑ a ∈A µ ( a | s ) P ( s ′ | s, a ) is the probability of moving to state s ′ ∈ S from s ∈ S upon taking action µ ( s ) ∈ A . Associated with each state-action pair ( s, a ) , is a one-step reward which is denoted by r µ ( s ) := E a ∼ µ ( s ) r ( s, a ) .

Let J µ ∈ R be the average reward associated with the policy µ ∈ Π , i.e. J µ is defined as:

Here the expectation is taken with respect to the measure P µ associated with the policy µ . Let h µ ∈ R n be the relative value function associated with the policy µ . Defining 1 ∈ R n to be the

vector of all ones, the pair ( J µ , h µ ) satisfies the following average reward Bellman equation:

Let π µ ∈ R n denote the stationary distribution associated with the kernel P µ . Let P ∗ µ = 1 π ⊤ µ ∈ R n × n denote the matrix whose rows are π ⊤ µ . Using P ∗ µ = P ∗ µ P µ , we get the following characterization of the average reward by multiplying both sides of Equation (2) with P ∗ µ : J µ 1 = P ∗ µ r µ .

Let J ∗ := max µ ∈ Π J µ be the optimal average reward. From standard MDP theory, there exists h ∗ ∈ R n , for which the pair ( J ∗ , h ∗ ) satisfies the following Bellman optimality equation:

Let µ ∗ be the optimizing policy in Equation (3). Then, similar reasoning shows that J ∗ 1 = P ∗ µ ∗ r µ ∗ .

The goal of dynamic programming and reinforcement learning is to determine this optimal average reward J ∗ and its corresponding optimal policy µ ∗ . Policy iteration is one of the most widely used dynamic programming algorithms for this purpose. It consists of two steps: (i) evaluation of the value function associated with a policy, and (ii) determining a greedy policy with respect to this evaluation. There are primarily two challenges that arise when applying such an algorithm: (i) high memory and time complexity when the state space is too large, and (ii) the unknown transition probability kernel that governs the dynamics of the MDP.

When the state space is too large, it may be computationally infeasible to perform policy iteration exactly. Existing literature suggests that approximate policy iteration in the context of discounted reward MDPs yields good performance. However, such an algorithm has not been studied in the context of average reward MDPs. In fact, the results obtained for the discounted reward MDPs yield vacuous performance bounds for average reward MDPs given the standard relationship between discounted reward and average reward MDPs. We explore this further in Section 2.2.

2.2 Relationship to the Discounted Reward MDP

To provide some background, we first discuss the approximate policy iteration algorithm in the context of discounted reward MDPs. Let α ∈ [0 , 1) denote the discount factor. The discounted reward associated with a deterministic policy µ ∈ Π starting from some state s ∈ S is denoted J α µ ( s ) and is defined as:

The value function J α µ ∈ R n is the unique fixed point of the Bellman operator T α µ : R n → R n associated with the policy µ , which is defined as T α µ J = r µ + α P µ J . For each state s ∈ S , let J α ∗ ( s ) denote the optimal discounted reward, i.e., J α ∗ ( s ) = max µ ∈ Π J α µ ( s ) . Similarly, J α ∗ ∈ R n is the unique fixed point of the optimality Bellman operator T α : R n → R n , which is defined as T α J = max µ ∈ Π ( r µ + α P µ J ) . Algorithm 1 is Approximate Policy Iteration for the discounted reward MDP:

Algorithm 1 Approximate Policy Iteration: Discounted Reward

Require J 0 ∈ R n for k = 0 , 1 , 2 , . . . do

Compute µ k +1 ∈ Π such that ∥ T α J k -T α µ k +1 J k ∥ ∞ ≤ ϵ ▷ Approximate Policy Improvement

Choose J k +1 such that ∥ J k +1 -J µ k +1 ∥ ∞ ≤ δ ▷ Approximate Policy Evaluation where J µ k +1 = T α µ k +1 J µ k +1

end for

The proof of this theorem can be found in [Ber11]

Proof.

From literature [Ros14, Ber11], we know that the average reward J µ associated with any policy µ is related to its discounted reward J α µ ( s ) counterpart as follows:

Note that the above relation is independent of the state s ∈ S , as the average reward does not depend on the initial state. Multiplying Equation (4) with (1 -α ) , and letting α → 1 , still yields an approximation performance bound with (1 -α ) in the denominator. This term blows up to infinity as α → 1 . Note that this bound is known to be tight under unichain Markov structure of the probability transition kernel [Ber11]. However, in practice it is observed that approximate policy iteration works well in the average reward MDP case although these theoretical bounds are not representative of this performance. To the best of our knowledge, we are unaware of an average reward approximate policy iteration performance bound, when the policies induce an irreducible Markov chain. We bridge this gap between theory and practice by providing a theoretical analysis of approximate policy iteration in the average reward MDP setting, with non trivial performance bounds.

3 Approximate Policy Iteration for Average Reward

A crucial component of the proof of convergence of approximate policy iteration in the context of discounted reward MDPs is the contraction due to the discount factor α which is absent in the average reward setting (since α = 1 ). To get some source of contraction, we have to make some assumptions on the MDP.

Assumption 3.1. We make the following assumptions:

(a) Every deterministic policy µ ∈ Π induces an irreducible Markov Chain P µ .

(b) For all policies, the diagonal elements of the probability transition matrix are positive, i.e., the Markov chain stays in the same state with non-zero probability.

Assumption (a) need not be satisfied by all MDPs. However, in order to satisfy this assumption, we consider a modified MDP where at every time step with probability ε, an action is chosen from the set of all possible actions with equal probability. Simultaneously, with probability 1 -ε , we choose an action dictated by some policy. The problem then is to choose this policy optimally. For most MDPs of interest, this small modification will satisfy our assumption with a small O ( ε ) loss in performance. It is straightforward to show the O ( ε ) loss, but we include the proof in the appendix for completeness. Assumption (b) is without loss of generality in the following sense: there exists a simple transformation which essentially leaves the MDP unchanged but ensures that this assumption is satisfied. This transformation known as the aperiodicity transformation or Schweitzer transformation was introduced in [Sch71]. For more details, the reader is referred to the appendix. In the remainder of this paper, we assume that all MDPs are transformed accordingly to ensure the assumptions are satisfied. One consequence of Assumption (b) is the following lemma which will be useful to us later.

Lemma 3.2. There exists a γ > 0 such that, for any policy µ ∈ Π ,

where π µ is stationary distribution over states associated with the policy µ .

Proof. This lemma has been proved in a more general sense in the context of deterministic policies in [VdW80]. Since the aperiodicity transformation ensures a non-zero probability of staying in the same state, the expected return times are bounded and the lemma holds true for randomized policies as well.

Prior to presenting the algorithm, consider the following definitions. The average-reward Bellman operator T µ : R n → R n corresponding to a policy µ is defined as T µ h = r µ + P µ h. The averagereward optimal Bellman operator T : R n → R n is defined as T h = max µ ∈ Π r µ + P µ h. The value function h µ associated with a policy µ satisfies the Bellman Equation (2) with single step reward r µ and the transition kernel P µ . Note that h µ is unique up to an additive constant.

Solving for ( J µ , h µ ) using Equation (2) involves setting the value function for some fixed state x ∗ as 0 (since this reduces the system from being underdetermined to one with a unique solution). Further, the value function h µ can be alternatively expressed as the lim m →∞ T m µ h 0 for any h 0 ∈ R n as is the case with discounted reward MDPs. However, unlike the discounted reward Bellman Equation, there is no discount factor α < 1 preventing the value function from exploding to infinity. Hence, we consider value function computed using the relative Bellman operator ˜ T µ defined as

We now state the algorithm for Approximate Policy Iteration for Average Reward MDPs.

3.1 Approximate Policy Iteration Algorithm for Average Reward MDPs

Algorithm 2 Approximate Policy Iteration: Average Reward

1: Require h 0 ∈ R n

2: for k = 0 , 1 , 2 , . . . do

3: 1. Compute µ k +1 ∈ Π such that ∥ T h k -T µ k +1 h k ∥ ∞ ≤ ϵ ▷ Approx. Policy Improvement

4:

2. Compute

h

k

+1

such that

∥

h

k

+1

-

h

5: where h µ k +1 = lim m →∞ ˜ T m µ k +1 h k

6: end for

3.2 Performance bounds for Average Reward Approximate Policy Iteration

We are now ready to present the main result of this work.

Theorem 3.3. Let J ∗ be the optimal average reward of an MDP that satisfies Assumption 3.1. The sequence of policies µ k generated by Algorithm 2 and their associated average rewards J µ k satisfy the following bound:

Interpretation of the Bound: Since 0 < γ < 1 , as k → ∞ , the error due to initial condition h 0 drops to zero. The approximation error consists of two components, ϵ and δ . Here, ϵ represents the approximation error due to a suboptimal policy and δ represents the approximation error associated with evaluating the value function corresponding to a policy. The limiting behavior of the average reward associated with the policies obtained as a consequence of the algorithm is captured in the following corollary.

Corollary 3.4. The asymptotic performance of the policies obtained from Algorithm 2 satisfies:

Note that the asymptotic bound in Equation (5) is in a very similar form as the asymptotic performance bound for the discounted reward MDP in Equation (4) where γ plays the role of (1 -α ) 2 . However, when ϵ and δ got to zero, this bound also goes to zero which is not the case if we try to approximate average-reward problems by using the results for discounted-reward problems in the limit where the horizon goes to infinity.

4 Application to Reinforcement Learning

Approximate policy iteration is closely related to RL. In general, approximate policy improvement and approximate policy evaluation in Algorithm 2 are executed through approximations to policy improvement (such as greedy update, mirror descent, softmax) and TD learning for value function approximation. First, we present a generic framework to analyze policy-based RL algorithms by

µ

k

+1

∥

∞

δ

▷

Approx. Policy Evaluation

building upon the theory presented in the last section. For this purpose, we define the state-action relative value function Q (instead of the relative state value function h as before) to evaluate a policy. Let T Q µ Q denote the Bellman operator with respect to Q and policy µ where ( T Q µ Q )( s, a ) = r ( s, a ) + ( Q µ Q ) ( s, a ) , where Q µ ( s ′ , a ′ | s, a ) = µ ( a ′ | s ′ ) P ( s ′ | s, a ) . The relative state action value function corresponding to policy µ is represented by Q µ and is the solution to the following Bellman equation: J µ + Q µ ( s, a ) = r ( s, a ) + ∑ s ′ ∈S ,a ′ ∈A Q µ ( s ′ , a ′ | s, a ) Q µ ( s ′ , a ′ ) , for all state action pairs ( s, a ) ∈ ( S , A ) . We present a generic policy-based algorithm for average-reward problem below.

Algorithm 3 Generic Policy Based Algorithm: Q -function Average Reward

Require Q 0 ∈ R |S||A|

Determine µ k +1 ∈ Π as a function of Q k using a possibly random policy improvement step

2. Compute

Q

k

+1

as an approximation to trajectory generated by policy

end for

Note that the error in Steps 1 (policy improvement) and 2 (policy evaluation) of the above algorithm could be random. Thus, the analysis for Theorem 3.3 has to be adapted to Algorithm 3. The resulting expected deviation from optimality is characterized in the lemma below.

Lemma 4.1. Let µ k and Q k be the sequence of policies and relative state-action value function iterates generated by Algorithm 3. For all k ∈ 0 , . . . , T , we have:

where

µ

k

+1

Q

µ

k

+1

using (state, action, reward) samples from a

and Q ∗ µ is a matrix with identical rows corresponding to the invariant distribution of Q µ .

Iteratively applying Lemma 4.1 yields the following proposition for a generic policy-based algorithm. Proposition 4.2. For any T > 0 , the iterates of Algorithm 3 satisfy

We now illustrate the applicability of the above result to specific RL algorithms. More specifically, we characterize finite time performance bounds when state-action relative value functions are learnt through TD-Learning [ZZM21], while several different update rules are employed for policy improvement.

4.1 Performance bounds for RL algorithms

For every iteration of Algorithm 3, it is necessary to observe trajectories of length τ in order to learn the state action value function associated with any policy. Using linear function approximations for value functions, the error in policy evaluation at iteration k in Algorithm 3 can be expressed in the following form:

where Φ ∈ R |S||A|× d corresponds to the feature vector matrix and θ ∈ R d is the parameter vector. Further, θ k is the parameter vector obtained as a result of the TD learning algorithm. The projection of Q µ k onto the span of Φ is given by Φ θ ∗ µ k . The last term in Equation (6) corresponds to the function approximation error which depends on the span of the feature matrix Φ . We denote this error term by δ 0 ,k . The error due to TD learning (first term in Equation (6)) is denoted by δ TD ,k . Assuming max ( s,a ) ∈S×A ∥ ϕ ( s, a ) ∥ 2 ≤ 1 , and ∥ θ ∗ µ k ∥ 2 is bounded uniformly in k , [ZZM21] shows that there exists C > 0 such that the length of the sample trajectory τ and δ TD ,k are related as follows:

Let c 0 = ( J ∗ -min i ∈S ( T Q Q 0 -Q 0 ) ( i ) ) be the error due to initial condition and δ 0 = max t δ 0 ,t be the maximum function approximation error across all iterations. Then we obtain the following performance bounds when TD learning is used for policy evaluation.

Corollary 4.3. (Greedy policy update) Let ˜ a ( s ) = argmax a ∈A Q k ( s, a ′ ) . Let β > 0 be such that the sequence of policies in Algorithm 3 corresponding to all state action pairs ( s, a ) are obtained using the following algorithm, which we call the greedy policy update.

̸

Let η = max s ∈S ,a ∈A k ∈ 1 ...T | Q k ( s, a ) | . Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:

Corollary 4.4. (Softmax policy update) Let β > 0 be such that the sequences of policies in Algorithm 3 corresponding to all state action pairs ( s, a ) are obtained using the following algorithm, which we call the softmax policy update.

Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:

Corollary 4.5. (Mirror descent update) Let β > 0 be such that the sequences of policies in Algorithm 3 corresponding to all state action pairs ( s, a ) are obtained using the following algorithm, which we call the mirror descent policy update.

Let ω = min k ∈ 1 ...T s ∈S µ k ( a ∗ ( s ) | s ) where a ∗ ( s ) is the optimal action at state s. Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:

Remark 4.6 . The following remarks are in order:

In each of the above performance bounds, there is an inevitable constant error ( δ 0 ) due to function approximation. If the value function lies in the class of functions chosen, then this error will be zero.

The error due to policy update (characterized by β ) can be made arbitrarily small by choosing β large. However, choosing a large β may increase C due to the fact that C is a function of the mixing time of each policy, which is affected by the choice of β. However, C cannot be arbitrarily large due to Assumption (a).

The error due to the initial condition decays exponentially fast but the rate of decay can be slow because γ can be very small. In this regard, it is interesting to compare our work with the results in [AYBB + 19, LYAYS21]. We do so in the next subsection.

4.2 Comparison with [AYBB + 19, LYAYS21]

Our model for approximate policy iteration is intended to be a general framework to study policybased RL algorithms. A specific class of such algorithms has been studied in [AYBB + 19, LYAYS21], where regret bounds are presented for a mirror descent like algorithm referred to as POLITEX. Although our model is intended for offline algorithms, it is possible to adapt the techniques in [AYBB + 19, LYAYS21] to obtain regret bounds for our model as well.

Let r ( s t , a t ) be the reward obtained at time t. Let K be the total number of time units for which Algorithm 3 is executed. Hence K = Tτ , where τ is the length of the sample trajectory for every iteration of policy evaluation. The regret, defined as R ( K ) = ∑ K t =1 ( J ∗ -r ( s t , a t )) can be further decomposed as:

The first term i.e., ∑ K t =1 ( J ∗ -J µ t ) is referred to as the pseudo regret R PS ( K ) . The second term is essentially bounding the average value from its estimate and can be bounded using a martingale analysis identical to [AYBB + 19, LYAYS21]. Here, we focus on obtaining an upper bound on the pseudo regret with mirror descent update and TD learning. The pseudo regret can be further decomposed as:

For the mirror descent policy update in Corollary 4.5, we get the following regret bound

Let β = √ τ , then the above regret can be expressed as:

Optimizing for the regret yields τ = O ( K 2 3 ) and the following pseudo regret upper bound,

We now compare our regret bounds to the ones in [AYBB + 19, LYAYS21]. Since all of the bounds will have an inevitable error ( δ 0 ) due to function approximation, the comparison is with respect to the other terms in the regret bound.

We get a better order-wise bound on the regret compare to [AYBB + 19] which obtains a bound of O ( K 3 / 4 ) . However, they have better constants which are in terms of mixing-time coefficients rather than in terms of γ.

[LYAYS21] gets a better regret bound ( O ( √ K )) than us.

The results in [AYBB + 19, LYAYS21] use Monte Carlo policy evaluation, which allows for a closedform expression for the value function estimate. The structure of the closed-form expressions allows them to exploit the fact that the policy varies slowly from one iteration to the next (which is specific to the mirror descent policy update) and hence γ does not appear in their analysis. However, the Monte Carlo policy evaluation algorithms use matrix inversion which is often too expensive to implement in practice unlike the TD learning algorithm in our analysis which is widely used. Nevertheless, the ideas in [AYBB + 19, LYAYS21] provide a motivation to investigate whether one can extend their analysis to more general policy evaluation and policy improvement algorithms. This is an interesting topic for future research.

5 Conclusions

We present a general framework for analyzing policy-based reinforcement learning algorithms for average-reward MDPs. Our main contribution is to obtain performance bounds without using the natural source of contraction available in discounted reward formulations. We apply our general framework to several well-known RL algorithms to obtain finite-time error bounds on the average reward. We conclude with a comparative regret analysis and an outline of possible future work.

References

[ABB01] Jinane Abounadi, Dimitri Bertsekas, and Vivek S Borkar. Learning algorithms for markov decision processes with average cost. SIAM Journal on Control and Optimization , 40(3):681-698, 2001.

[AKLM21] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research , 22(1):4431-4506, 2021.

[AYBB + 19] Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning , pages 3692-3702. PMLR, 2019.

[Ber11] Dimitri P Bertsekas. Dynamic programming and optimal control 3rd edition, volume ii. Belmont, MA: Athena Scientific , 2011.

[Ber12] Dimitri Bertsekas. Dynamic programming and optimal control: Volume I , volume 1. Athena scientific, 2012.

[BSGL09] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica , 45(11):2471-2482, 2009.

[BT96] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

[CCM22] Zaiwei Chen, John Paul Clarke, and Siva Theja Maguluri. Target network and truncation overcome the deadly triad in q -learning. arXiv preprint arXiv:2203.02628 , 2022.

[CM22] Zaiwei Chen and Siva Theja Maguluri. Sample complexity of policy-based methods under off-policy sampling and linear function approximation. In International Conference on Artificial Intelligence and Statistics , pages 11195-11214. PMLR, 2022.

[Gos04] Abhijit Gosavi. Reinforcement learning for long-run average cost. European journal of operational research , 155(3):654-674, 2004.

[JJS93] Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems , 6, 1993.

[KT99] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

[LYAYS21] Nevena Lazic, Dong Yin, Yasin Abbasi-Yadkori, and Csaba Szepesvari. Improved regret bound and experience replay in regularized policy iteration. In International Conference on Machine Learning , pages 6032-6042. PMLR, 2021.

[Mah96] Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. Recent advances in reinforcement Learning , pages 159-195, 1996.

[MMT00] Peter Marbach, Oliver Mihatsch, and John N Tsitsiklis. Call admission control and routing in integrated services networks using neuro-dynamic programming. IEEE Journal on selected areas in communications , 18(2):197-208, 2000.

[MS08] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

[MXSS20] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning , pages 6820-6829. PMLR, 2020.

[Put14] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

[Ros14] Sheldon M Ross. Introduction to stochastic dynamic programming . Academic press, 2014.

[SB18] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

[Sch71] Paul J Schweitzer. Iterative solution of the functional equations of undiscounted markov renewal programming. Journal of Mathematical Analysis and Applications , 34(3):495-501, 1971.

[Sin94] Satinder P Singh. Reinforcement learning algorithms for average-payoff markovian decision processes. In AAAI , volume 94, pages 700-705, 1994.

[Tsi94] John N Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine learning , 16:185-202, 1994.

[TVR99] John N Tsitsiklis and Benjamin Van Roy. Average cost temporal-difference learning. Automatica , 35(11):1799-1808, 1999.

[VdW80] J Van der Wal. Successive approximations for average reward markov games. International Journal of Game Theory , 9(1):13-24, 1980.

[WD92] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8:279-292, 1992.

[WLLS21] Anna Winnicki, Joseph Lubars, Michael Livesay, and R Srikant. The role of lookahead and approximate policy evaluation in reinforcement learning with linear value function approximation. arXiv preprint arXiv:2109.13419 , 2021.

[WNS21] Yi Wan, Abhishek Naik, and Richard S Sutton. Learning and planning in average-reward markov decision processes. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10653-10662. PMLR, 18-24 Jul 2021.

[YB09] Huizhen Yu and Dimitri P Bertsekas. Convergence results for some temporal difference methods based on least squares. IEEE Transactions on Automatic Control , 54(7):15151531, 2009.

[ZWSW21] Shangtong Zhang, Yi Wan, Richard S Sutton, and Shimon Whiteson. Average-reward off-policy policy evaluation with function approximation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 12578-12588. PMLR, 18-24 Jul 2021.

[ZZM21] Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of averagereward td learning and q -learning. Advances in Neural Information Processing Systems , 34:1230-1242, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives some performance bounds on some approximate policy iteration type algorithms and later considers the application in the context of reinforcement learning. The bounds derived appear correct and meaningful but I have several comments which I will list below.\n\nSTRENGTHS:\nThis paper derives some performance bounds on some approximate policy iteration type algorithms and later considers the application in the context of reinforcement learning. The bounds derived appear correct and meaningful but I have several comments which I will list below.\n\nWEAKNESSES:\nOne of the weaknesses is a lack of discussion on previous bounds that are available in the context of average reward problems.\nA central class of algorithms in the RL literature based on the policy iteration method have been the actor-critic algorithms.\n I would have preferred a thorough comparison and discussion on the significance of the bounds that the authors derive and how different are these from the bounds obtained for the actor-critic schemes.\n\nA second weakness is of the lack of any experimentation. That would have certainly added value to the paper even though the primary focus of the authors has been to derive some bounds for the algorithms.\n\nA third weakness is the lack of a comprehensive analysis of asymptotic convergence for these algorithms.\n\nQUESTIONS:\n1. In the discussion following (1), the authors try to argue out a bound for average reward problems after stating a bound for the discounted reward problems. Do we really have to do it this way to argue out the problems that might result. Why do we even need to use the bounds for discounted MDPs in the average reward setting?\n\n2. For the equation above (2), the authors would need to assume certain conditions on the underlying Markov chain in order for the limit to exist.\n\n3. Para above Sec 2.2 has incorrect statements. It suggests that algorithms based on approximate DP for average reward MDPs do not exist. The authors should look up references such as [BSGL09] and [KT99] cited their paper. There are several other papers that deal with average reward MDPs and use function approximation in particular.\n\n4. In Algorithm 1, the authors should specify what \\epsilon and \\delta are. Also, in step 2, how do you decide on J_{k+1}? Is there a systematic way of it's selection.\n5. In Theorem 3.3, is \\gamma the same as in Lemma 3.2? Mention also whether \\gamma >0.\n\n6. Second line after Corollary 3.4: Substituting \\gamma = (1-\\alpha)^2 makes the denominator same in (4)-(5) but numerators are quite different. So, not clear what the authors mean by saying \\gamma plays the role of (1-\\alpha)^2.\n\n7. In Lemma 4.1, what is T? Is this the time of episode termination. Also, is T deterministic or random? How is T chosen and what is the effect of T on the performance since it is an important parameter?\n\n8. Is the \\gamma in Lemma 4.1 the same as \\gamma in Lemma 3.2? I guess that is not true because this \\gamma is based on the state-action Markov chain unlike the state-valued chain previously. Why should the invariant distribution of the state-action Markov chain with transition probability matrix Q_\\mu exist?\n\n9. How does the inequality (6) hold? What is Q_k on the LHS and how did it vanish on the RHS?\n\n10. In Corollaries 4.3-4.5, what is the parameter \\tau and is it related to T? One would have expected the inequalities in these results to go to 0 as T goes to infinity, but that is not happening. Why does this problem arise and so how meaningful are the presented bounds in these corollaries?\n\n11. In (15), the RHS is multiplied by \\tau which was not the case earlier (see line below (14)). This makes the role of \\tau confusing?\n\n*********\nThe following comments are post the rebuttal from the authors.\n\n1. I am again looking at the paper. For one, things are not well presented at all. For instance, Algorithm 3 does not involve a theta_k parameter but in (6), the terms \\theta_k, \\theta^*_{\\mu_k} appear on the RHS saying that Algorithm 3 gives this. The authors begin by talking of approximate policy iteration and then say in the case of RL, the terms \\theta_k etc appear and there is no algorithm given for the update of the theta_k.\n\n2. The authors say that they are able to prove convergence to global optima even with function approximation but it's not clear how? In Remark 4.6, they mention that there is an error due to function approximation which is also the case with other algorithms in the literature that use function approximation. \n\n3. Also, the authors say they show asymptotic convergence. Is the convergence shown in the almost sure sense? If so, where is that result? Also, somewhere they said that [ZZM21] showed the bound in (7) assuming that \\theta^*_{\\mu_k} are uniformly bounded. It is not stated if the authors also assume this bound. Also, what about a bound on the \\theta_k sequence itself? Do you not require to prove stability of your stochastic iterate before claiming asymptotic convergence? \n\n4. Assumption 3.1a clearly is too strong because it is only for the case of deterministic policies. Following 3.1a, the authors say that they consider instead a modified policy where with some positive probability, they select an action uniformly at random and with the remaining probability, they select actions as per a deterministic policy. Clearly, this is not a deterministic policy. Then the assumption makes no sense. I find this whole thing a rather strange argument.\n\n5. Assumption 3.1b also requires a special structure on the Markov chain which may not always be the case - that the diagonal elements in the probability matrix have non-zero entries under every policy. This is also the reason why experiments are important. The authors can easily demonstrate in a future version of their work whether such unrealistic assumptions are needed even for empirical studies.\n*****\n\nThe authors have been able to convince me that theirs is a good contribution. I am therefore revising my score.\n\nLIMITATIONS:\nI have already listed my comments above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper proposed an Approximate Policy Iteration Algorithm for average-reward MDPs, and further extended to RL setting.

STRENGTHS:
The paper is well-written and clear. The approximated PI approach is new and novel; The theoretical results are also comprehensive.

WEAKNESSES:
The algorithm seems unclear to me. How do the updates implement (e.g., Lines 3,4 in Alg 2)? 

How does the approach handle the "contraction" in the average-reward setting?

Overall, this paper seems a little bit unsurprising to me, the author should highlight the novelty and contribution in the work.

QUESTIONS:
See above.

LIMITATIONS:
It is unclear if the algorithm can be implemented in practice.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on finite-sample convergence analysis for average-reward reinforcement learning. The authors firstly propose finite-time  error bounds for approximate policy iteration. The major technique ingredient is performing Schweitzer transformation to the MDP. By this transformation, the stationary distribution of any policy could cover the state-space uniformly with a condition number $\\gamma>0$. Therefore some contraction arguments in the analysis for discounted MDP could be applied.\n\nSTRENGTHS:\n1. The authors firstly achieve a finite-time error bounds for approximation policy iteration in average-reward RL with using Schweitzer transformation;\n\n2. The authors also present a more refined analysis with respect to time-varying policy evaluation and improvement errors. Some extensions on practical algorithms (e.g, greedy and softmax) are established.\n\nWEAKNESSES:\nMy major concern is about the technical novelty. The authors main contribution is to generalize the finite-time error analysis for $\\alpha$-discounted MDP to the average-reward MDP, and the major technique is using the Schweitzer transformation, which I think is roughly equivalent to  learning $\\lambda_1 \\mathbf{I}+\\lambda_2 \\bar{P}+(1-\\lambda_1-\\lambda_2)P$ instead of learning $P$ for some proper $\\lambda_1,\\lambda_2\\in (0,1)$, where $\\bar{P}$ is the averaged transition model over all actions, and $P$ is the true transition model. In this way, the contraction arguments could work directly.  \n\nI have learned similar arguments before in literature on online regret minimization for average-reward MDP (e.g. [Wei, 2020], [Wang, 2022]), where $\\lambda_1 \\mathbf{I}+(1-\\lambda_1)P$ is considered instead of the original transition model $P$. Moreover, [Wang, 2022] described the relationship between near-optimal policies for $\\alpha$-discounted MDP and that for average-reward MDP. More precisely, it was proven that an $\\epsilon$-optimal policy for $\\alpha$-discounted MDP is at least $\\epsilon+(1-\\alpha)\\mathrm{sp}(h^*)$-optimal for the average-reward MDP as long as the MDP is weak-communicating. Here $\\mathrm{sp}(h^*) = 2\\min_{x\\in \\mathbb{R}}\\|h^* - r\\cdot \\textbf{1}\\|_{\\infty}$. \n\nOne can also show that $\\|V^*_{\\alpha}-\\min_{s}V^*_{\\alpha}(s) \\textbf{1} - h^*+\\min_{s}h^*(s)\\cdot \\textbf{1}\\|\\leq O((1-\\alpha)D\\mathrm{sp}(h^*))$ , where $V^*_{\\alpha}$ is the optimal value function for the $\\alpha$-discounted MDP and $D$ is the diameter of the MDP.  \nIn this work, the authors introduce another term $\\bar{P}$ into the transition model to make the MDP irreducible for any policy. But I do not think it is a good choice since the mixing time could be exponentially large, which might make $1/\\lambda$ is exponentially large. \n\n\nReferences:\n\n[Wei, 2020] Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes\n\n[Wang, 2022] Near Sample-Optimal Reduction-based Policy Learning for Average Reward MDP\n\nQUESTIONS:\n1. In line 18 in the appendix, why $\\hat{V}_{\\mu}$ is bounded? And what is the exact bound? In general, $\\hat{V}$ could be large if the mixing time of the MDP is large. Do you need $\\mathrm{sp}(h^*)$ to bound $\\hat{V}_{\\mu}$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper obtains performance bounds for policy iteration in the average reward setting, where instead of the usual discontinued factor formulation, the limiting ratio between total reward divided by time is used as a performance metric, which is a commonly used metric in the regret-based literature.The paper explains why obtaining such bounds has been difficult, and utilizes novel techniques and shows how we can obtain the bound.

STRENGTHS:
- The technical claims seem sound, and the need for this bound in the average setting is well motivated. 
- The authors do a good job of explaining why it has been difficult to use commonly used techniques for analyzing performance bounds for PI.

WEAKNESSES:
- It would be nice if the authors could mention meaningful problems that minimize for the average setting instead of the discounted setting. In other words, the author's mathematical contribution is well conveyed, but it would be good to motivate why we should care about the average setting in the first place despite the difficulty of analyzing them.

QUESTIONS:
I am not familiar with RL theory and do not have the necessary background to ask questions on the topic.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence of approximate policy iteration for the average reward setting. First, finite suboptimality bounds are provided for average reward approximate policy iteration under suitable ergodicity properties of the set of policy-induced Markov chains. Next, these results are extended to Q-iteration-type algorithms, building a bridge to reinforcement learning methods. Finally, using this bridge, corresponding bounds are provided for a variety of existing policy iteration-based reinforcement learning methods, including greedy policy updates, softmax policy updates, and mirror descent updates.\n\nSTRENGTHS:\nThis paper is well-written, it builds the necessary technical background thoroughly but concisely, its contributions and results are clearly stated, and it addresses a longstanding open issue in classical policy iteration methods for average-reward MDPs. The overall plan of the paper is to rigorously establish a connection between average-reward approximate policy iteration methods and policy iteration-based average-reward reinforcement learning methods, inspired by the work [Z. Chen & S. T. Maguluri, _Sample complexity of policy-based methods under off-policy sampling and linear function approximation_. AISTATS, 2022], which achieved this for the discounted setting. In the present work, this is achieved largely via an analysis based on [J. Van der Wal,  _Successive approximations for average reward Markov games_. International Journal of Game Theory, 1980], but suitably modified to handle the presence of approximation errors in the policy recovery and policy evaluation steps. I did not check the appendix in detail, but the general thrust of the arguments appears to makes sense. The work has a somewhat \"classical\" flavor, as access to the transition dynamics, rewards, and approximate policy improvement and evaluation oracles appears to be assumed in the algorithms and analysis, but it should nonetheless be of interest to the theoretical RL community, especially those interested in RL policy iteration and policy evaluation underpinnings.\n\nWEAKNESSES:\nMy main concerns are the following:\n* The apparent access to transition dynamics, rewards, and approximate policy improvement and evaluation oracles will likely appear unreasonable to those unfamiliar with, e.g., [D. Bertsekas & J. N. Tsitsiklis, _Neuro-dynamic programming._ 1996]. For this reason, a discussion of how the approximate policy improvement and approximate policy evaluation steps can be carried out in practice would help allay this concerns and strengthen the paper.\n* The key innovation and insight in the analysis remains somewhat unclear. Lines 119-125 in the appendix mention that the analysis of [Van der Wal, 1980] must be modified by carefully bounding the errors arising from approximate policy evaluation and improvement, but it is still unclear what the magnitude of the effort was and whether any specific, innovative techniques were used.\n\nQUESTIONS:\n* How can the approximate policy evaluation and improvement oracles be instantiated in practice?\n* Did the careful bounding mentioned in lines 119-125 of the appendix consist solely of manipulations of inequalities and algebraic expressions? Or was there some key, interesting insight or innovative application of some unlooked-for technique required?\n\nLIMITATIONS:\nAside from the weaknesses and questions above, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms\" tries to solve the open problem of obtaining meaningful performance bounds for approximate policy iteration algorithms for the average reward setting. If one tries to use the bounds for PI from discounted reward setting and set the discount factor equal to one to obtain the average reward setting, due to the horizon dependence, the bounds tend to infinity. While this is so, it is well known that API performs better in practice than this bound suggests. This paper tries to bridge this gap between theory and practice. It first obtains finite time error bounds for average reward approximate policy iteration. Then, it extends to obtain finite-iteration error bounds for the case where the errors can vary over iterations. Later, it illustrates the use of these results for various policy-based RL methods.\n\nSTRENGTHS:\n1. Directly using the bounds for PI from discounted reward setting by setting the discount factor equal to 1 and obtaining the average reward setting, the bounds tend to infinity. This is due to the horizon dependence.  From the literature, it is well known that API performs well in practice, in contrast to what this bound suggests. This paper tries to bridge this gap between theory and practice by presenting meaningful performance bounds for approximate policy iteration algorithms for the average reward setting.\n\n2. Their bounds show that the error goes to zero, unlike the transformation of bounds from discounted to average reward.\n\n3. The motivation is clear and the paper is very well written.\n\n4. Obtained performance bounds for various RL based methods by plugging the above bounds.\n\nWEAKNESSES:\n1. In the performance bounds for RL based methods, the errors don't seem to be small. They all depend on $\\gamma$, which can make them worse. (This is discussed in remark 4.6, but I don't understand the use of these results if this is the case, please clarify this).\n\n2. In section 4.2, the regret of an online algorithm is discussed. In equation $16$, the regret is of the linear order, while the earlier work has $\\sqrt{K}$ dependence. Moreover, this bound depends upon $\\gamma$. Then again, what is the use of these performance bounds?\n\nA minor issue:\n\n$\\gamma$ is not clear or defined before Theorem 3.3.\n\nQUESTIONS:\nPlease provide clarifications to the questions above.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}