SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the asymptotic bound in Equation (5) is in a very similar form as the asymptotic performance bound for the discounted reward MDP in Equation (4) where  $\gamma$  plays the role of  $(1 - \alpha)^{2}$ . However, when  $\epsilon$  and  $\delta$  got to zero, this bound also goes to zero which is not the case if we try to approximate average-reward problems by using the results for discounted-reward problems in the limit where the horizon goes to infinity.
4 Application to Reinforcement Learning
Approximate policy iteration is closely related to RL. In general, approximate policy improvement and approximate policy evaluation in Algorithm 2 are executed through approximations to policy improvement (such as greedy update, mirror descent, softmax) and TD learning for value function approximation. First, we present a generic framework to analyze policy-based RL algorithms by
building upon the theory presented in the last section. For this purpose, we define the state-action relative value function  $Q$  (instead of the relative state value function  $h$  as before) to evaluate a policy. Let  $\mathsf{T}_{\mu}^{\mathsf{Q}}Q$  denote the Bellman operator with respect to  $Q$  and policy  $\mu$  where  $(\mathsf{T}_{\mu}^{\mathsf{Q}}Q)(s,a) = r(s,a) + (\mathbb{Q}_{\mu}Q)(s,a)$ , where  $\mathbb{Q}_{\mu}(s',a'|s,a) = \mu (a'|s')\mathbb{P}(s'|s,a)$ . The relative state action value function corresponding to policy  $\mu$  is represented by  $Q_{\mu}$  and is the solution to the following Bellman equation:  $J_{\mu} + Q_{\mu}(s,a) = r(s,a) + \sum_{s'\in S,a'\in A}\mathbb{Q}_{\mu}(s',a'|s,a)Q_{\mu}(s',a')$ , for all state action pairs  $(s,a)\in (\mathcal{S},\mathcal{A})$ . We present a generic policy-based algorithm for average-reward problem below.
Algorithm 3 Generic Policy Based Algorithm:  $Q$ -function Average Reward
Require  $Q_0\in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$
for  $k = 0,1,2,\ldots ,T$  do
1. Determine  $\mu_{k + 1}\in \Pi$  as a function of  $Q_{k}$  using a possibly random policy improvement step
2. Compute  $Q_{k + 1}$  as an approximation to  $Q_{\mu_{k + 1}}$  using (state, action, reward) samples from a trajectory generated by policy  $\mu_{k + 1}$
Note that the error in Steps 1 (policy improvement) and 2 (policy evaluation) of the above algorithm could be random. Thus, the analysis for Theorem 3.3 has to be adapted to Algorithm 3. The resulting expected deviation from optimality is characterized in the lemma below.
Lemma 4.1. Let  $\mu_{k}$  and  $Q_{k}$  be the sequence of policies and relative state-action value function iterates generated by Algorithm 3. For all  $k\in \mathbf{0},\ldots ,T$  we have:
$$
\begin{array}{l} \left(J ^ {*} - \left(\min  _ {(s, a)} \left(\mathsf {T} ^ {\mathsf {Q}} Q _ {k} - Q _ {k}\right) (s, a)\right)\right) \leq (1 - \gamma) \left(J ^ {*} - \left(\min  _ {(s, a)} \left(\mathsf {T} ^ {\mathsf {Q}} Q _ {k - 1} - Q _ {k - 1}\right) (s, a)\right)\right) \\ + \left\| \mathsf {T} ^ {\mathsf {Q}} Q _ {k - 1} - \mathsf {T} _ {\mu_ {k}} ^ {\mathsf {Q}} Q _ {k - 1} \right\| _ {\infty} + 2 \| Q _ {k} - Q _ {\mu_ {k}} \| _ {\infty}, \\ \end{array}
$$
$$
\gamma = \min_{\substack{\mu \in \mu_{1},\mu_{2},\ldots ,\mu_{T}\\ (s,a)\in \mathcal{S}\times \mathcal{A}}}\mathbb{Q}_{\mu}^{*}(s,a) > 0,
$$
and  $\mathbb{Q}_{\mu}^{*}$  is a matrix with identical rows corresponding to the invariant distribution of  $\mathbb{Q}_{\mu}$ .
Iteratively applying Lemma 4.1 yields the following proposition for a generic policy-based algorithm.
Proposition 4.2. For any  $T > 0$ , the iterates of Algorithm 3 satisfy
$$
\mathbb{E}\bigl[J^{*} - J_{\mu_{T + 1}}\bigr ]\leq \underbrace{(1 - \gamma)^{T}\mathbb{E}\Bigl{\lbrack}J^{*} - \min_{i}\left(\mathsf{T}^{\mathsf{Q}}Q_{0} - Q_{0}\right)(i)\Bigr]}_{\text{Initial condition error}} + \underbrace{2\sum_{\ell = 0}^{T - 1}(1 - \gamma)^{\ell}\mathbb{E}\big[\big\| Q_{T - \ell} - Q_{\mu_{T - \ell}}\big\|_{\infty}\big]}_{\text{Error}}
$$
Policy evaluation error
$$
+ \underbrace {\sum_ {\ell = 1} ^ {T - 1} (1 - \gamma) ^ {\ell - 1} \mathbb {E} \left[ \left\| \mathsf {T} _ {\mu_ {T + 1 - \ell}} ^ {\mathsf {Q}} Q _ {T - \ell} - \mathsf {T} ^ {\mathsf {Q}} Q _ {T - \ell} \right\| _ {\infty} \right]} _ {\leqslant T _ {\mu_ {T + 1}} ^ {\mathsf {Q}} Q _ {T} - \mathsf {T} ^ {\mathsf {Q}} Q _ {T} \left\| _ {\infty} \right.}.
$$
Policy improvement error
We now illustrate the applicability of the above result to specific RL algorithms. More specifically, we characterize finite time performance bounds when state-action relative value functions are learnt through TD-Learning [ZZM21], while several different update rules are employed for policy improvement.
4.1 Performance bounds for RL algorithms
For every iteration of Algorithm 3, it is necessary to observe trajectories of length  $\tau$  in order to learn the state action value function associated with any policy. Using linear function approximations for value functions, the error in policy evaluation at iteration  $k$  in Algorithm 3 can be expressed in the following form:
$$
\left\| Q _ {k} - Q _ {\mu_ {k}} \right\| _ {\infty} \leq \left\| \Phi \left(\theta_ {k} - \theta_ {\mu_ {k}} ^ {*}\right) \right\| _ {\infty} + \left\| Q _ {\mu_ {k}} - \Phi \theta_ {\mu_ {k}} ^ {*} \right\| _ {\infty}, \tag {6}
$$
where  $\Phi \in \mathbb{R}^{|S||\mathcal{A}| \times d}$  corresponds to the feature vector matrix and  $\theta \in \mathbb{R}^d$  is the parameter vector. Further,  $\theta_k$  is the parameter vector obtained as a result of the TD learning algorithm. The projection of  $Q_{\mu_k}$  onto the span of  $\Phi$  is given by  $\Phi \theta_{\mu_k}^*$ . The last term in Equation (6) corresponds to the function approximation error which depends on the span of the feature matrix  $\Phi$ . We denote this error term by  $\delta_{0,k}$ . The error due to TD learning (first term in Equation (6)) is denoted by  $\delta_{\mathrm{TD},k}$ . Assuming  $\max_{(s,a) \in S \times \mathcal{A}} \| \phi(s,a) \|_2 \leq 1$ , and  $\| \theta_{\mu_k}^* \|_2$  is bounded uniformly in  $k$ , [ZZM21] shows that there exists  $C > 0$  such that the length of the sample trajectory  $\tau$  and  $\delta_{\mathrm{TD},k}$  are related as follows:
$$
\mathbb {E} \left[ \delta_ {\mathrm {T D}, k} \right] = \frac {C}{\sqrt {\tau}}. \tag {7}
$$
Let  $c_{0} = \left(J^{*} - \min_{i\in S}\left(\mathsf{T}^{\mathsf{Q}}Q_{0} - Q_{0}\right)(i)\right)$  be the error due to initial condition and  $\bar{\delta}_0 = \max_t\delta_{0,t}$  be the maximum function approximation error across all iterations. Then we obtain the following performance bounds when TD learning is used for policy evaluation.
Corollary 4.3. (Greedy policy update) Let  $\tilde{a}(s) = \operatorname{argmax}_{a \in \mathcal{A}} Q_k(s, a')$ . Let  $\beta > 0$  be such that the sequence of policies in Algorithm 3 corresponding to all state action pairs  $(s, a)$  are obtained using the following algorithm, which we call the greedy policy update.
$$
\mu_ {k + 1} (a | s) = \left\{ \begin{array}{l l} \frac {1}{\beta | \mathcal {A} |}, & \text {i f} a \neq \tilde {a} (s) \\ \frac {1}{\beta | \mathcal {A} |} + 1 - \frac {1}{\beta}, & \text {i f} a = \tilde {a} (s), \end{array} \right. \tag {8}
$$
Let  $\eta = \max_{\substack{s\in S, a\in \mathcal{A}\\ k\in 1\dots T}}|Q_k(s,a)|$ . Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:
$$
\mathbb {E} \left[ J ^ {*} - J _ {\mu_ {T + 1}} \right] \leq (1 - \gamma) ^ {T} c _ {0} + \left(\frac {1 + \gamma}{\gamma}\right) \frac {2 \eta}{\beta} + \frac {2}{\gamma} \left(\bar {\delta} _ {0} + \frac {C}{\sqrt {\tau}}\right). \tag {9}
$$
Corollary 4.4. (Softmax policy update) Let  $\beta > 0$  be such that the sequences of policies in Algorithm 3 corresponding to all state action pairs  $(s, a)$  are obtained using the following algorithm, which we call the softmax policy update.
$$
\mu_ {k + 1} (a | s) = \frac {\exp (\beta Q _ {k} (s , a))}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp (\beta Q _ {k} (s , a ^ {\prime}))}. \tag {10}
$$
Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:
$$
\mathbb {E} \left[ J ^ {*} - J _ {\mu_ {T + 1}} \right] \leq (1 - \gamma) ^ {T} c _ {0} + \left(\frac {1 + \gamma}{\gamma}\right) \frac {\log (| \mathcal {A} |)}{\beta} + \frac {2}{\gamma} \left(\bar {\delta} _ {0} + \frac {C}{\sqrt {\tau}}\right). \tag {11}
$$
Corollary 4.5. (Mirror descent update) Let  $\beta > 0$  be such that the sequences of policies in Algorithm 3 corresponding to all state action pairs  $(s, a)$  are obtained using the following algorithm, which we call the mirror descent policy update.
$$
\mu_ {k + 1} (a | s) \propto \mu_ {k} (a | s) \exp \left\{\beta Q _ {k} (s, a) \right\}. \tag {12}
$$
Let  $\omega = \min_{\substack{k\in 1\dots T\\ s\in S}}\mu_k(a^* (s)|s)$  where  $a^* (s)$  is the optimal action at state  $s$ . Using TD Learning with linear value function approximation for policy evaluation, we get the following finite time performance bound:
$$
\mathbb {E} \left[ J ^ {*} - J _ {\mu_ {T + 1}} \right] \leq (1 - \gamma) ^ {T} c _ {0} + \left(\frac {1 + \gamma}{\gamma \beta}\right) \log \left(\frac {1}{\omega}\right) + \frac {2}{\gamma} \left(\bar {\delta} _ {0} + \frac {C}{\sqrt {\tau}}\right). \tag {13}
$$
Remark 4.6. The following remarks are in order:
- In each of the above performance bounds, there is an inevitable constant error  $(\overline{\delta}_0)$  due to function approximation. If the value function lies in the class of functions chosen, then this error will be zero.
- The error due to policy update (characterized by  $\beta$ ) can be made arbitrarily small by choosing  $\beta$  large. However, choosing a large  $\beta$  may increase  $C$  due to the fact that  $C$  is a function of the mixing time of each policy, which is affected by the choice of  $\beta$ . However,  $C$  cannot be arbitrarily large due to Assumption (a).
- The error due to the initial condition decays exponentially fast but the rate of decay can be slow because  $\gamma$  can be very small. In this regard, it is interesting to compare our work with the results in [AYBB+19, LYAYS21]. We do so in the next subsection.
4.2 Comparison with  $\left[\mathrm{AYBB}^{+}19, \mathrm{LYAYS21}\right]$
Our model for approximate policy iteration is intended to be a general framework to study policy-based RL algorithms. A specific class of such algorithms has been studied in [AYBB+19, LYAYS21], where regret bounds are presented for a mirror descent like algorithm referred to as POLITEX. Although our model is intended for offline algorithms, it is possible to adapt the techniques in [AYBB+19, LYAYS21] to obtain regret bounds for our model as well.
Let  $r(s_{t},a_{t})$  be the reward obtained at time  $t$ . Let  $K$  be the total number of time units for which Algorithm 3 is executed. Hence  $K = T\tau$ , where  $\tau$  is the length of the sample trajectory for every iteration of policy evaluation. The regret, defined as  $R(K) = \sum_{t=1}^{K} (J^{*} - r(s_{t},a_{t}))$  can be further decomposed as:
$$
R (K) = \sum_ {t = 1} ^ {K} \left(J ^ {*} - J _ {\mu_ {t}}\right) + \left(J _ {\mu_ {t}} - r \left(s _ {t}, a _ {t}\right)\right). \tag {14}
$$
The first term i.e.,  $\sum_{t=1}^{K}(J^{*} - J_{\mu_t})$  is referred to as the pseudo regret  $R_{\mathrm{PS}}(K)$ . The second term is essentially bounding the average value from its estimate and can be bounded using a martingale analysis identical to [AYBB+19, LYAYS21]. Here, we focus on obtaining an upper bound on the pseudo regret with mirror descent update and TD learning. The pseudo regret can be further decomposed as:
$$
R _ {\mathrm {P S}} (K) = \tau \sum_ {t = 1} ^ {T} \left(J ^ {*} - J _ {\mu_ {t}}\right).
$$
For the mirror descent policy update in Corollary 4.5, we get the following regret bound
$$
R _ {\mathrm {P S}} (K) \leq \tau \sum_ {t = 1} ^ {T} (1 - \gamma) ^ {T} c _ {0} + \frac {1 + \gamma}{\gamma \beta} \log \left(\frac {1}{\omega}\right) + \frac {2}{\gamma} \left(\bar {\delta} _ {0} + \frac {c _ {3}}{\sqrt {\tau}}\right). \tag {15}
$$
Let  $\beta = \sqrt{\tau}$ , then the above regret can be expressed as:
$$
R _ {\mathrm {P S}} (K) \leq \frac {1}{\gamma} \left(\tau c _ {0} + 2 K \bar {\delta} _ {0} + \frac {K c _ {5}}{\sqrt {\tau}}\right).
$$
Optimizing for the regret yields  $\tau = O(K^{\frac{2}{3}})$  and the following pseudo regret upper bound,
$$
R _ {\mathrm {P S}} (K) \leq \frac {1}{\gamma} \left(2 K \bar {\delta} _ {0} + c _ {6} K ^ {\frac {2}{3}}\right). \tag {16}
$$
We now compare our regret bounds to the ones in  $\left[\mathrm{AYBB}^{+}19, \mathrm{LYAYS21}\right]$ . Since all of the bounds will have an inevitable error  $(\bar{\delta}_0)$  due to function approximation, the comparison is with respect to the other terms in the regret bound.
- We get a better order-wise bound on the regret compare to  $\left[\mathrm{AYBB}^{+}19\right]$  which obtains a bound of  $O(K^{3/4})$ . However, they have better constants which are in terms of mixing-time coefficients rather than in terms of  $\gamma$ .
- [LYAYS21] gets a better regret bound  $(O(\sqrt{K}))$  than us.
The results in  $\left[\mathrm{AYBB}^{+}19, \mathrm{LYAYS21}\right]$  use Monte Carlo policy evaluation, which allows for a closed-form expression for the value function estimate. The structure of the closed-form expressions allows them to exploit the fact that the policy varies slowly from one iteration to the next (which is specific to the mirror descent policy update) and hence  $\gamma$  does not appear in their analysis. However, the Monte Carlo policy evaluation algorithms use matrix inversion which is often too expensive to implement in practice unlike the TD learning algorithm in our analysis which is widely used. Nevertheless, the ideas in  $\left[\mathrm{AYBB}^{+}19, \mathrm{LYAYS21}\right]$  provide a motivation to investigate whether one can extend their analysis to more general policy evaluation and policy improvement algorithms. This is an interesting topic for future research.
We present a general framework for analyzing policy-based reinforcement learning algorithms for average-reward MDPs. Our main contribution is to obtain performance bounds without using the natural source of contraction available in discounted reward formulations. We apply our general framework to several well-known RL algorithms to obtain finite-time error bounds on the average reward. We conclude with a comparative regret analysis and an outline of possible future work.
[ABB01] Jinane Abounadi, Dimitri Bertsekas, and Vivek S Borkar. Learning algorithms for markov decision processes with average cost. SIAM Journal on Control and Optimization, 40(3):681-698, 2001.
[AKLM21] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431-4506, 2021.
[AYBB+19] Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szeptesvari, and Gellert Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning, pages 3692-3702. PMLR, 2019.
[Ber11] Dimitri P Bertsekas. Dynamic programming and optimal control 3rd edition, volume ii. Belmont, MA: Athena Scientific, 2011.
[Ber12] Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 1. Athena scientific, 2012.
[BSGL09] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.
[BT96] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
[CCM22] Zaiwei Chen, John Paul Clarke, and Siva Theja Maguluri. Target network and truncation overcome the deadly triad in  $q$ -learning. arXiv preprint arXiv:2203.02628, 2022.
[CM22] Zaiwei Chen and Siva Theja Maguluri. Sample complexity of policy-based methods under off-policy sampling and linear function approximation. In International Conference on Artificial Intelligence and Statistics, pages 11195-11214. PMLR, 2022.
[Gos04] Abhijit Gosavi. Reinforcement learning for long-run average cost. European journal of operational research, 155(3):654-674, 2004.
[JJS93] Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems, 6, 1993.
[KT99] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[LYAYS21] Nevena Lazic, Dong Yin, Yasin Abbasi-Yadkori, and Csaba Szepesvari. Improved regret bound and experience replay in regularized policy iteration. In International Conference on Machine Learning, pages 6032-6042. PMLR, 2021.
[Mah96] Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical results. *Recent advances in reinforcement Learning*, pages 159–195, 1996.
[MMT00] Peter Marbach, Oliver Mihatsch, and John N Tsitsiklis. Call admission control and routing in integrated services networks using neuro-dynamic programming. IEEE Journal on selected areas in communications, 18(2):197-208, 2000.
[MS08] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.
[MXSS20] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning, pages 6820-6829. PMLR, 2020.
[Put14] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[Ros14] Sheldon M Ross. Introduction to stochastic dynamic programming. Academic press, 2014.
[SB18] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[Sch71] Paul J Schweitzer. Iterative solution of the functional equations of undiscounted markov renewal programming. Journal of Mathematical Analysis and Applications, 34(3):495-501, 1971.
[Sim94] Satinder P Singh. Reinforcement learning algorithms for average-payoff markovian decision processes. In AAAI, volume 94, pages 700-705, 1994.
[Tsi94] John N Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine learning, 16:185-202, 1994.
[TVR99] John N Tsitsiklis and Benjamin Van Roy. Average cost temporal-difference learning. Automatica, 35(11):1799-1808, 1999.
[VdW80] J Van der Wal. Successive approximations for average reward markov games. International Journal of Game Theory, 9(1):13-24, 1980.
[WD92] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
[WLLS21] Anna Winnicki, Joseph Lubars, Michael Livesay, and R Srikant. The role of lookahead and approximate policy evaluation in reinforcement learning with linear value function approximation. arXiv preprint arXiv:2109.13419, 2021.
[WNS21] Yi Wan, Abhishek Naik, and Richard S Sutton. Learning and planning in average-reward markov decision processes. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10653–10662. PMLR, 18–24 Jul 2021.
[YB09] Huizhen Yu and Dimitri P Bertsekas. Convergence results for some temporal difference methods based on least squares. IEEE Transactions on Automatic Control, 54(7):1515-1531, 2009.
[ZWSW21] Shangtong Zhang, Yi Wan, Richard S Sutton, and Shimon Whiteson. Average-reward off-policy policy evaluation with function approximation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12578–12588. PMLR, 18–24 Jul 2021.
[ZZM21] Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite sample analysis of average-reward td learning and  $q$ -learning. Advances in Neural Information Processing Systems, 34:1230-1242, 2021.