SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the next proposition, we show that the subintervals of  $[0,1)$  where a policy is discount-optimal may be much more complex than usually alluded to in the literature. In particular, there exists a simple MDP instance with only two policies, but where a Blackwell-optimal policy may be discount-optimal in an arbitrary number of arbitrary disjoint subintervals of  $[0,1)$ .
Theorem 3.7. For any odd integer  $N \in \mathbb{N}$  and any sequence  $0 = \gamma_0 < \gamma_1 < \dots < \gamma_{N-1} < \gamma_N = 1$ , there exists an MDP instance  $(S, A, r, P)$  with  $|S| = N + 1$  and  $|\mathcal{A}| = 2$ , and two policies  $\pi_1, \pi_2$  such that  $\pi_1$  is the unique optimal policy on any of the intervals  $(\gamma_{2i}, \gamma_{2i+1})$  for  $i = 0, \dots, (N-1)/2$  and  $\pi_2$  is the unique optimal policy on  $(\gamma_{2i-1}, \gamma_{2i})$ , for  $i = 1, \dots, (N-1)/2$ .
Theorem 3.7 shows that the algorithm that explore the entire interval of  $(0,1)$  to compute discount-optimal policies (Hordijk et al., 1985) may visit a number of subintervals that is impractical. We present a detailed proof in Appendix B. The proof relies on interpreting value functions as polynomials and using Lagrange interpolation polynomials to tune the instantaneous rewards to ensure that the value functions intersect at the given discount factors. Overall, our results in this section highlight the pitfalls of the existing approach to Blackwell optimality and the potential pathological behaviors of the value functions, even in simple MDP instances. We ameliorate this issue in the next section.
4 Introducing the Blackwell discount factor
This section introduces the notion of the Blackwell discount factor, which we use to reduce Blackwell optimality and average optimality to discounted optimality. This reduction leads to algorithms to compute Blackwell-optimal and average policies that are significantly simpler than the state-of-the-art. Intuitively, we need the following condition to reduce Blackwell optimality to discounted optimality: there must exist a discount factor  $\gamma_{\mathrm{bw}} \in [0,1)$  such that any  $\gamma$ -discount-optimal policy for  $\gamma > \gamma_{\mathrm{bw}}$  is also  $\gamma'$ -discount-optimal for any other  $\gamma' > \gamma_{\mathrm{bw}}$ . The following definition formalizes this intuition.
Definition 4.1. The Blackwell discount factor  $\gamma_{\mathrm{bw}} \in [0,1)$  is equal to  $\gamma_{\mathrm{bw}} = \inf \{\gamma \in [0,1) \mid \Pi_{\gamma'}^{\star} = \Pi_{\mathrm{bw}}^{\star}, \forall \gamma' \in (\gamma,1)\}$ , where  $\Pi_{\mathrm{bw}}^{\star}$  is the set of Blackwell-optimal policies.
We establish the existence of a Blackwell discount factor in the next theorem.
Theorem 4.2. The Blackwell discount factor  $\gamma_{\mathrm{bw}}$  in Definition 4.1 exists in any finite MDP.
Proof. We show that there exists a discount factor  $\gamma \in [0,1)$  such that  $\Pi_{\gamma'}^{\star} = \Pi_{\mathrm{bw}}^{\star}, \forall \gamma' \in (\gamma, 1)$ . Let  $\bar{\gamma}$  defined as in Equation (3.1). We show  $\forall \gamma \in [\bar{\gamma}, 1)$ ,  $\Pi_{\gamma}^{\star} = \Pi_{\mathrm{bw}}^{\star}$ . Let  $\gamma' \in (\bar{\gamma}, 1)$  and let  $\pi$  be a policy that is  $\gamma'$ -discount-optimal. By definition, we have  $v_{\gamma', s}^{\pi} \geq v_{\gamma', s}^{\pi'}$ ,  $\forall \pi' \in \Pi, \forall s \in S$ . Since  $\gamma' > \bar{\gamma}$ , the map  $\phi_s^{\pi, \pi'}$  does not change sign on  $[\bar{\gamma}, 1)$ . This shows that  $\pi$  is  $\gamma$ -discount-optimal for all  $\gamma \in (\bar{\gamma}, 1)$ . Therefore,  $\pi$  is Blackwell optimal, and any  $\gamma$ -discount-optimal policy is Blackwell optimal, for any  $\gamma \in (\bar{\gamma}, 1)$ , i.e., this shows  $\Pi_{\bar{\gamma}}^{\star} \subset \Pi_{\mathrm{bw}}^{\star}$ . The inclusion  $\Pi_{\mathrm{bw}}^{\star} \subset \Pi_{\bar{\gamma}}^{\star}$  follows from the definition of  $\bar{\gamma}$ : if  $\pi$  is Blackwell-optimal but not discount-optimal for  $\bar{\gamma}$ , then it must become discount-optimal for a larger  $\gamma' > \bar{\gamma}$ , which is impossible since  $\bar{\gamma}$  is the largest discount factors where the value functions of any two stationary policies can intersect.
Difference from the existing definition. It is important to elaborate on the difference between Definition 3.1 (classical definition of Blackwell optimality) and Definition 4.1 (Blackwell discount factor). While the proof for the existence of  $\gamma_{\mathrm{bw}}$  is relatively concise, the distinction between  $\gamma_{\mathrm{bw}}$  and  $\gamma (\pi)$  has been utterly overlooked in the literature, where it is common to find statements that suggest that  $\gamma >\gamma (\pi)$  implies Blackwell optimality of all discount-optimal policies, e.g. in Dewanto and Gallagher (2021), Wang et al. (2023). To the best of our knowledge, we are the first to properly introduce the Blackwell discount factor  $\gamma_{\mathrm{bw}}$ , to show its sufficiency to compute Blackwell-optimal policies, to emphasize the shortcomings of the classical approach to Blackwell optimality, and to clarify the distinction between  $\gamma_{\mathrm{bw}}$  and  $\gamma (\pi)$ . In particular, in Definition 3.1, a Blackwell-optimal policy  $\pi$  is optimal for any  $\gamma \in [\gamma (\pi),1)$ . However, for some  $\gamma \in [\gamma (\pi),1)$ , there may be other optimal policies that are not Blackwell-optimal, as shown in Proposition 3.5. We show an MDP instance like this in Example 3.6, where  $\gamma_{\mathrm{bw}} = 3 / 4$  but where  $\gamma (a_1) = 1 / 2$ , and  $a_1$  is the only Blackwell-optimal policy. Hence in all generality, we may have  $\gamma (\pi) < \gamma_{\mathrm{bw}}$ , and  $\gamma (\pi)\neq \gamma_{\mathrm{bw}}$ . Note that the authors in (Dewanto and Gallagher, 2021, Dewanto et al., 2020) also introduce the notation “ $\gamma_{\mathrm{bw}}$ ” but they use it to denote  $\gamma (\pi)$ .
Reduction to discounted optimality. If  $\gamma_{\mathrm{bw}}$  is known for a given MDP instance, it is straightforward to compute a Blackwell-optimal policy, by solving a discounted MDP with  $\gamma > \gamma_{\mathrm{bw}}$ . Therefore, the notion of Blackwell discount factor provides a method to reduce the criteria of Blackwell optimality and average optimality to the well-studied criterion of discounted optimality. As we have discussed before, efficient methods for solving discounted MDPs such as value iteration or linear programming have been extensively studied. These algorithms are much simpler than the two existing algorithms for computing Blackwell-optimal policies. Note that it is enough to compute an upper bound on  $\gamma_{\mathrm{bw}}$ . In particular, if we are able to show that  $\gamma_{\mathrm{bw}} < \gamma'$  for some  $\gamma' \in [0,1)$ , then following the definition of  $\gamma_{\mathrm{bw}}$ , we can compute a Blackwell-optimal policy by solving a discounted MDP with a discount factor  $\gamma = \gamma'$ . Therefore, in the rest of Section 4, we focus on obtaining an upper bound on  $\gamma_{\mathrm{bw}}$ .
4.1 Upper bound on  $\gamma_{\mathrm{bw}}$
We now obtain an instance-dependent upper bound on  $\gamma_{\mathrm{bw}}$ , i.e., we construct a scalar  $\eta(\mathcal{M}) \in (0,1)$  for each MDP instance  $\mathcal{M} = (\mathcal{S}, \mathcal{A}, r, P)$ , such that  $\gamma_{\mathrm{bw}} < 1 - \eta(\mathcal{M})$ . Our main contribution in this section is Theorem 4.4, which gives a closed-form expression for  $\eta(\mathcal{M})$  as a function of the maximum bit-size of the data of the MDP instance  $\mathcal{M}$ . We start by showing that it is impossible to obtain a bound on  $\gamma_{\mathrm{bw}}$  that is independent of  $r$  or  $P$ .
Proposition 4.3. For any  $\eta >0$ , there exists an MDP instance  $\mathcal{M} = (\mathcal{S},\mathcal{A},\boldsymbol {r},\boldsymbol {P})$  with  $|\mathcal{S}| = 2,|\mathcal{A}| = 2$  and deterministic transitions, such that  $\gamma_{\mathrm{bw}} > 1 - \eta$ .
Proof. Let  $\mathcal{S} = \{s_1, s_2\}, \mathcal{A} = \{a_1, a_2\}$ . In state  $s_1$ , action  $a_1$  transitions to  $s_1$  (with reward 0) and action  $a_2$  transitions to  $s_2$  (with reward -1). There is no action to choose in state  $s_2$  which is absorbing with a reward  $\epsilon > 0$ . It is straightforward to check that  $a_2$  is Blackwell optimal, with  $\gamma_{\mathrm{bw}} = (1 + \epsilon)^{-1}$ , so that  $\gamma_{\mathrm{bw}}$  can be chosen arbitrarily close to 1 by choosing small values for  $\epsilon$ .
We show that Proposition 4.3 still holds even under the assumption that the MDP instances are weakly-communicating in Appendix C. Proposition 4.3 shows that an instance-dependent bound on  $\gamma_{\mathrm{bw}}$  must depend on the "coarseness" of  $r$  and  $P$ . This suggests parametrizing our upper bound by the bit-sizes of the MDP instance. MDPs with finite bit-sizes parameters are the MDP instances that can be exactly encoded in a computer and practically solved by existing algorithms. We first recall the definitions pertaining to bit-size, necessary to describe the complexity of classical weakly-polynomial time algorithms like interior-point methods (section 4.6 in (Ben-Tal and Nemirovski, 2001)) and the ellipsoid method (Bland et al., 1981). The bit-size of  $r \in \mathbb{N}$  is  $\lfloor \log_2(r) \rfloor$ , the number of bits necessary to represent  $r$  with standard binary encoding. The bit-size of a rational number is the sum of the bit-size of its numerator and its denominator. The maximum bit-size of an MDP instance is the maximum bit-size of any  $r_{sa}$  and  $P_{sa'}$  for  $(s, a, s') \in S \times \mathcal{A} \times S$ . Its total bit-size is the sum of the bit-sizes of the components of  $r$  and  $P$ . For instance, in the riverswim instance, the maximum bit-size of the reward is 14, since the largest rewards are bounded by  $10^4$  in the terminal states. Our main theorem in this section provides a strict upper bound on  $\gamma_{\mathrm{bw}}$  as follows.
Theorem 4.4. Let  $\mathcal{M} = (\mathcal{S},\mathcal{A},\boldsymbol {r},\boldsymbol {P})$  be an MDP instance with finite bit-size and let  $m\in \mathbb{N}$  be the maximum bit-size of the instance  $\mathcal{M}$ . Then we have  $\gamma_{\mathrm{bw}} < 1 - \eta (\mathcal{M})$ , with  $\eta (\mathcal{M})\in (0,1)$  defined as
$$
\eta (\mathcal {M}) = \frac {1}{2 N ^ {N / 2 + 2} (L + 1) ^ {N}}, N = 2 | \mathcal {S} | - 1, L = 2 \cdot | \mathcal {S} | \cdot r _ {\infty} \cdot m ^ {2 | \mathcal {S} |} \cdot 4 ^ {| \mathcal {S} |}.
$$
Our proof uses ideas that are new in the MDP literature, such as the separation of algebraic numbers. We provide an outline of the proof below and defer the full statement to Appendix D.
In the first step of the proof, by carefully inspecting the proofs of Theorem 3.2 and of Theorem 4.2, we note that an upper bound for  $\gamma_{\mathrm{bw}}$  is  $\bar{\gamma}$ , as defined in (3.1):  $\bar{\gamma} = \max_{\pi, \pi' \in \Pi, s \in S} \gamma(\pi, \pi', s)$ , where for  $\pi, \pi' \in \Pi$  and  $s \in S$ ,  $\gamma(\pi, \pi', s)$  is the largest discount factor  $\gamma$  in [0, 1) for which  $\phi_s^{\pi, \pi'}(\gamma) = 0$  when  $\phi_s^{\pi, \pi'}: \gamma \mapsto v_{\gamma, s}^{\pi} - v_{\gamma, s}^{\pi'}$  is not identically equal to 0, and 0 otherwise. Therefore, we focus on obtaining an upper bound on  $\gamma(\pi, \pi', s)$  for any two policies  $\pi, \pi' \in \Pi$  and any state  $s \in S$ .
In the second step, following Lemma 3.3, the value functions  $\gamma \mapsto v_s^\pi$ ,  $\gamma \mapsto v_s^{\pi'}$  are rational functions, i.e., they are ratios of two polynomials. Therefore, we interpret  $\phi_s^{\pi,\pi'}(\gamma) = 0$  as a polynomial equation in  $\gamma$ , i.e., as  $p(\gamma) = 0$  for a certain polynomial  $p$ . With this notation,  $\gamma(\pi, \pi', s) \in [0,1)$  is a root of  $p$ . We show that  $\gamma = 1$  is always a root of  $p$ , even though value functions are a priori not defined for  $\gamma = 1$ . We then precisely characterize the degree  $N$  and the sum  $L$  of the absolute values of the coefficients of the polynomial  $p$ , depending on the MDP instance  $\mathcal{M}$ .
Theorem 4.5. The polynomial  $p$  has degree  $N = 2|\mathcal{S}| - 1$ . Moreover,  $m^{2|\mathcal{S}|}p$  has integral coefficients. The sum of the absolute values of the coefficients of  $m^{2|\mathcal{S}|}p$  is bounded by  $L = 2\cdot |\mathcal{S}|\cdot r_{\infty}\cdot m^{2|\mathcal{S}|}\cdot 4^{|S|}$ .
In the third step, we lower-bound the distance between any two distinct roots of  $p$ . To do this, we rely on the following separation bounds of algebraic numbers.
Theorem 4.6 ((Rump, 1979)). Let  $p$  be a polynomial of degree  $N$  with integer coefficients. Let  $L$  be the sum of the absolute values of its coefficients. The distance between any two distinct roots of  $p$  is strictly larger than  $\eta > 0$ , with  $\eta = 2N^{-N/2 + 2} (L + 1)^{-N}$ .
Recall that  $\gamma (\pi ,\pi^{\prime},s)$  and 1 are two always roots of  $p$  with  $\gamma (\pi ,\pi^{\prime},s) < 1$ . Combining Theorem 4.5 with Theorem 4.6, we conclude that  $\gamma (\pi ,\pi^{\prime},s) < 1 - \eta (\mathcal{M})$  for  $\eta (\mathcal{M}) > 0$  defined as in Theorem 4.4. Therefore,  $\bar{\gamma} < 1 - \eta (\mathcal{M})$ , and  $\gamma_{\mathrm{bw}} < 1 - \eta (\mathcal{M})$ . This concludes our proof of Theorem 4.4.
Discussion. Using Theorem 4.4, we obtain the first reduction from Blackwell optimality to discounted optimality: solving a discounted MDP with  $\gamma \geq 1 - \eta(\mathcal{M})$  returns a Blackwell-optimal policy. Blackwell optimality implies average optimality, so we also obtain the first reduction from average optimality to discounted optimality without any assumptions on the structure of the underlying Markov chains of the MDP. We also discuss the complexity results for computing a Blackwell-optimal policy using our reduction. Policy iteration returns a discounted optimal policy in  $O\left(\frac{|S|^2|A|}{1 - \gamma}\log \left(\frac{1}{1 - \gamma}\right)\right)$  iterations (Scherrer, 2013), but it may be slow to converge when  $\gamma = 1 - \eta(\mathcal{M})$  as in Theorem 4.4, since  $\eta(\mathcal{M})$  may be close to 0. Various algorithms exist to obtain convergence faster than  $O(1/(1 - \gamma))$ , such as accelerated value iteration (Goyal and Grand-Clement, 2023a) and Anderson acceleration (Zhang et al., 2020). However, note that  $\left\lfloor \log_2(\eta(\mathcal{M}))\right\rfloor$ , the bit-size of the scalar  $\eta(\mathcal{M})$ ,
is polynomial in the bit-size of the MDP instance  $\mathcal{M}$ . Since discounted MDPs can be formulated as linear programs, which can be solved in polynomial-time in the input size of the MDP (Ye, 2011), we obtain a weakly-polynomial time algorithm for computing Blackwell-optimal policies. We present the proof of the following theorem in Appendix E.
Theorem 4.7. Let  $\mathcal{M} = (\mathcal{S},\mathcal{A},\boldsymbol {r},\boldsymbol {P})$  be an MDP instance with total bit-size  $Q(\boldsymbol {r},\boldsymbol {P})\in \mathbb{N}$ . Then we can compute a Blackwell-optimal policy in  $O\left(|S|^5 |\mathcal{A}|^2 Q(\boldsymbol {r},\boldsymbol {P})\right)$  arithmetic operations.
Note that with Theorem 4.4 and Theorem 4.7, we have reduced the complex problem of computing a Blackwell optimal policy to a much simpler and well-studied problem: solving a linear program, which can be done in weakly-polynomial time. Potential improvements for our upper bound on  $\gamma_{\mathrm{bw}}$  are an important future direction: more precise separation bounds than Theorem 4.6 could be obtained for the specific polynomial  $p$  appearing in the proof of Theorem 4.4, or for a specific MDP instances, e.g. ergodic or unichain MDPs. Going beyond the case of finite sets of states and actions is interesting but this may be difficult, as in both cases there may not exist a Blackwell optimal policy anymore (Chitashvili, 1976, Maitra, 1965).
4.2 The case of robust MDPs
In practice, the value function  $\pmb{v}_{\gamma}^{\pi}$  may be very sensitive to the values of the transition probabilities  $\pmb{P}$ . To emphasize this dependence, in this section we note  $\pmb{v}_{\gamma}^{\pi, P}$  for the value function associated with a policy  $\pi$  and a transition probability  $\pmb{P}$ , defined similarly as in Section 2. Robust MDPs (RMDPs) ameliorate this issue by considering an uncertainty set  $\mathcal{U}$ , which can be seen as a plausible region for the transition probabilities  $\pmb{P} \in \mathcal{U}$ . We focus on the case of sa-rectangular MDPs (Iyengar, 2005), where  $\mathcal{U} = \times_{(s,a) \in S \times \mathcal{A}} \mathcal{U}_{sa}$  for  $\mathcal{U}_{sa} \subseteq \Delta(S)$ . The worst-case value function  $v_{\gamma}^{\pi, \mathcal{U}} \in \mathbb{R}^{S}$  of a policy  $\pi$  is defined as  $v_{\gamma,s}^{\pi, \mathcal{U}} = \min_{\pmb{P} \in \mathcal{U}} v_{\gamma,s}^{\pi, P}, \forall s \in S$ . In discounted RMDPs, the goal is to compute a robust discounted optimal policy, defined as follows.
Definition 4.8. Given  $\gamma \in [0,1)$ , a policy  $\pi \in \Pi$  is robust  $\gamma$ -discount-optimal if  $v_{\gamma,s}^{\pi,\mathcal{U}} \geq v_{\gamma,s}^{\pi',\mathcal{U}}, \forall \pi' \in \Pi, \forall s \in S$ . We write  $\Pi_{\gamma,\mathrm{rob}}^{\star}$  the set of robust  $\gamma$ -discount-optimal policies.
Robust Blackwell optimality is studied in (Goyal and Grand-Clément, 2023b, Tewari and Bartlett, 2007), to address the sensitivity of the robust value functions as regards the discount factors. Its connection to average reward RMDPs is discussed in (Tewari and Bartlett, 2007, Wang et al., 2023).
Definition 4.9. A policy  $\pi \in \Pi$  is robust Blackwell-optimal if there exists  $\gamma \in [0,1)$ , such that  $\pi \in \Pi_{\gamma',\mathbf{r}}^{\star}, \forall \gamma' \in [\gamma,1)$ . We call  $\Pi_{\mathrm{bw,r}}^{\star}$  the set of robust Blackwell-optimal policies.
(Goyal and Grand-Clément, 2023b) shows the existence of a Blackwell-optimal policy for RMDPs, under the condition that  $\mathcal{U}$  is sa-rectangular and has finitely many extreme points. This is the case for popular polyhedral uncertainty sets, e.g., when  $\mathcal{U}_{sa}$  is based on the  $\ell_p$  distance, for  $p\in \{1,\infty \}$  (Givan et al., 1997, Ho et al., 2018, Iyengar, 2005), for some estimated kernel  $P^0$  and some radius  $\alpha_{sa} > 0$ :
$$
\mathcal {U} _ {s a} = \left\{\boldsymbol {p} \in \Delta (\mathcal {S}) \mid \| \boldsymbol {p} - \boldsymbol {P} _ {s a} ^ {0} \| _ {p} \leq \alpha_ {s a} \right\}. \tag {4.1}
$$
Definition 4.10. We define the robust Blackwell discount factor  $\gamma_{\mathrm{bw,r}} \in [0,1)$  as  $\gamma_{\mathrm{bw,r}} = \inf \{\gamma \in [0,1) \mid \Pi_{\gamma',\mathrm{r}}^{\star} = \Pi_{\mathrm{bw,r}}^{\star}, \forall \gamma' \in (\gamma,1)\}$ .
We provide a detailed proof of the existence of the robust Blackwell discount factor in Appendix F. The proof strategy is the same as for the existence of the Blackwell discount factor for MDPs. We can obtain the same upper bound on  $\gamma_{\mathrm{bw,r}}$ , by studying the values of  $\gamma$  for which  $\gamma \mapsto v_{\gamma,s}^{\pi,\pmb{P}} - v_{\gamma,s}^{\pi',\pmb{P'}}$  cancels, for any two policies  $\pi, \pi' \in \Pi$  and any two extreme points  $\pmb{P}, \pmb{P}'$  of  $\mathcal{U}$ . Writing  $\gamma(\pi, \pi', s, \pmb{P}, \pmb{P}')$  for the largest zero in  $[0,1)$  of the function  $\gamma \mapsto v_{\gamma,s}^{\pi,\pmb{P}} - v_{\gamma,s}^{\pi',\pmb{P'}}$  if it is not identically equal to zero, or  $\gamma(\pi, \pi', s, \pmb{P}, \pmb{P}') = 0$  otherwise, an upper bound on  $\gamma_{\mathrm{bw,r}}$  for RMDPs can be computed as  $\bar{\gamma}_{\mathrm{r}}$ , defined as  $\bar{\gamma}_{\mathrm{r}} = \max_{\pi, \pi' \in \Pi, s \in S} \max_{\pmb{P}, \pmb{P}' \in \mathcal{U}_{\mathrm{ext}}} \gamma(\pi, \pi', s, \pmb{P}, \pmb{P}')$  with  $\mathcal{U}_{\mathrm{ext}}$  the set of extreme points of  $\mathcal{U}$ . This leads to the following theorem.
Theorem 4.11. Let  $\mathcal{M} = (\mathcal{S},\mathcal{A},\boldsymbol {r},\boldsymbol{P}^0)$  be an MDP instance with maximum bit-size  $m\in \mathbb{N}$ . Assume that  $\mathcal{U}$  is sa-rectangular, where for each  $(s,a)\in S\times \mathcal{A}$ ,  $\mathcal{U}_{sa}$  is constructed as in (4.1) based on  $\ell_{1}$  or  $\ell_{\infty}$  distance, and with the scalars  $(\alpha_{sa})_{s,a}$  of maximum bit-size  $m$ . Then  $\gamma_{\mathrm{bw,r}}\leq 1 - \eta (\mathcal{M})$ , with  $\eta (\mathcal{M})$  defined as in Theorem 4.4 with  $m^{\prime} = 2m$  instead of  $m$ .
Based on Theorem 4.11, we obtain the first reduction from robust Blackwell optimality to robust discounted optimality. Since discounted RMDPs can be solved with value iteration or policy iteration, we provide the first algorithms to compute a robust Blackwell-optimal policy for RMDPs with sa-rectangular uncertainty, when the uncertainty set is based on the  $\ell_1$  or the  $\ell_{\infty}$  distance. Note that there is no known convex (or linear) formulation for RMDPs (Grand-Clément and Petrik, 2022), so we are not able to provide a complexity statement akin to Theorem 4.7.
We highlight the shortcomings of the existing approach to Blackwell optimality and we introduce the Blackwell discount factor to ameliorate this issue. We provide an upper bound for MDPs and RMDPs in all generality, parametrized by the bit-sizes of the instances. Any progress in solving discounted MDPs, one of the most active research directions in RL, can be combined with our results to obtain new algorithms for computing average- and Blackwell-optimal policies. Our work also opens new research avenues for MDPs and RMDPs: the proof techniques for our bound on  $\gamma_{\mathrm{bw}}$  and  $\gamma_{\mathrm{bw},r}$ , based on the separation of algebraic numbers, are novel and they could be tightened for specific instances or different optimality criteria, such as bias optimality or  $n$ -discount optimality. The notion of approximate Blackwell optimality as well as the existence of the robust Blackwell discount factor for other uncertainty sets, e.g., s-rectangular or non-polyhedral sa-rectangular uncertainty sets, or for distributionally robust MDPs, are also interesting directions of research.
Funding. J. Grand-Clément is supported by the Agence Nationale de la Recherche [Grant 11-LABX-0047] and by Hi! Paris. M. Petrik's work was supported, in part, by NSF grants 2144601 and 1815275.
M. Akian and S. Gaubert. Policy iteration for perfect information stochastic mean payoff games with bounded first return times is strongly polynomial. arXiv preprint arXiv:1310.4953, 2013.
M. Akian, S. Gaubert, J. Grand-Clément, and J. Guillaud. The operator approach to entropy games. Theory of Computing Systems, 63(5):1089-1130, 2019.
J. Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319-350, 2001.
B. Behzadian, M. Petrik, and C. P. Ho. Fast algorithms for  $l_{\infty}$ -constrained s-rectangular robust MDPs. Advances in Neural Information Processing Systems, 34, 2021.
A. Ben-Tal and A. Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications. SIAM, 2001.
S. Bhatnagar, M. Ghavamzadeh, M. Lee, and R. S. Sutton. Incremental natural actor-critic algorithms. Advances in neural information processing systems, 20, 2007.
D. Blackwell. Discrete dynamic programming. The Annals of Mathematical Statistics, pages 719-726, 1962.
R. G. Bland, D. Goldfarb, and M. J. Todd. The ellipsoid method: A survey. Operations research, 29 (6):1039-1091, 1981.
V. Boone and B. Gaujal. Identification of Blackwell policies for deterministic MDPs. In 23ème congress annuel de la Société Française de Recherche Opérationnelle et d'Aide à la Décision, 2022.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016.