SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Julien Grand-Clement
ISOM Department HEC Paris
grand-clement@hec.fr
We introduce the Blackwell discount factor for Markov Decision Processes (MDPs). Classical objectives for MDPs include discounted, average, and Blackwell optimality. Many existing approaches to computing average-optimal policies solve for discount-optimal policies with a discount factor close to 1, but they only work under strong or hard-to-verify assumptions such as unichain or ergodicity. We highlight the shortcomings of the classical definition of Blackwell optimality, which does not lead to simple algorithms for computing Blackwell-optimal policies and overlooks the pathological behaviors of optimal value functions with respect to the discount factors. To resolve this issue, we show that when the discount factor is larger than the Blackwell discount factor  $\gamma_{\mathrm{bw}}$ , all discount-optimal policies become Blackwell- and average-optimal, and we derive a general upper bound on  $\gamma_{\mathrm{bw}}$ . Our upper bound on  $\gamma_{\mathrm{bw}}$ , parametrized by the bit-size of the rewards and transition probabilities of the MDP instance, provides the first reduction from average and Blackwell optimality to discounted optimality, without any assumptions, along with new polynomial-time algorithms. Our work brings new ideas from polynomials and algebraic numbers to the analysis of MDPs. Our results also apply to robust MDPs, enabling the first algorithms to compute robust Blackwell-optimal policies.
Markov Decision Processes (MDPs) provide a widely-used framework for modeling sequential decision-making problems (Puterman, 2014). In a (finite) MDP, the decision maker repeatedly interacts with an environment characterized by a finite set of states and a finite set of available actions. The decision maker follows a policy that prescribes an action at a state at every period. An instantaneous reward is obtained at every period, depending on the current state-action pair, and the system transitions to the next state at the next period. MDPs provide the underlying model for the applications of reinforcement learning (RL), ranging from healthcare (Gottesman et al., 2019) to game solving (Mnih et al., 2013) and finance (Deng et al., 2016).
There are several optimality criteria that measure a decision maker's performance in an MDP. In discounted optimality, the decision maker optimizes the discounted return, defined as the sum of the instantaneous rewards over the infinite horizon, where future rewards are discounted with a discount factor  $\gamma \in [0,1)$ . In average optimality, the decision maker optimizes the average return, defined as the average of the instantaneous rewards obtained over the infinite horizon. The average return ignores any return gathered in finite time, i.e., it does not reflect the transient performance of a policy and it only focuses on the steady-state behavior. The most selective optimality criterion in MDPs is Blackwell optimality (Puterman, 2014). A policy is Blackwell-optimal if it optimizes the discounted return simultaneously for all discount factors sufficiently close to 1. Since a discount factor close
to 1 can be interpreted as a preference for rewards obtained in later periods, Blackwell-optimal policies are also average-optimal. However, average-optimal policies need not be Blackwell-optimal. Blackwell optimality can be a useful criterion in environments with no natural, or known, discount factor. Also, any algorithm that computes a Blackwell-optimal policy also immediately computes an average-optimal policy. This is one of the reasons why better understanding the Blackwell optimality criterion is mentioned as "one of the pressing questions in RL" in the list of open research problems from a recent survey on RL for average reward optimality (Dewanto et al., 2020).
Average-optimal policies can be computed via linear programming (section 9.3, (Puterman, 2014)). However, virtually all of the recent algorithms for computing average-optimal policies require strong assumptions on the underlying Markov chains associated with the policies in the MDP instance, such as ergodicity (Wang, 2017), the unichain and aperiodicity properties (Schneckenreither, 2020), weakly communicating MDPs (Wang et al., 2022), or assumptions on the mixing time associated with any deterministic policies (Jin and Sidford, 2020, 2021). These assumptions are motivated by technical considerations (e.g., ensuring that the average reward is uniform across all states) and can be restrictive in practice (Puterman, 2014) and NP-hard to verify, such as unichain (Tsitsiklis, 2007). Existing methods for computing Blackwell-optimal policies rely on linear programming over the field of power series including negative coefficients (Hordijk et al., 1985), or on an algorithm based on a nested sequence of optimality equations (O'Sullivan and Veinott Jr, 2017) which requires to solve multiple linear programs sequentially. These algorithms are complex, difficult to implement, and have no complexity guarantees or known implementations.
In summary, existing algorithms for average optimality require restrictive assumptions, and algorithms for Blackwell-optimality are very complex. This is in stark contrast with the vast literature on solving discounted MDPs, where general and well-understood methods exist, including value iteration, policy iteration, and linear programming (chapter 6, (Puterman, 2014)). This is the starting point of this paper, which aims to develop new algorithms for computing average-optimal and Blackwell-optimal policies through a reduction to discounted MDPs. We make the following three main contributions.
third main contribution, we show that the robust Blackwell discount factor  $\gamma_{\mathrm{bw},\mathrm{r}}$  exists for popular models of uncertainty, such as sa-rectangular robust MDPs with polyhedral uncertainty (Goyal and Grand-Clement, 2023b, Iyengar, 2005). For this setting, we generalize our upper bound on  $\gamma_{\mathrm{bw}}$  for MDPs to an upper bound on  $\gamma_{\mathrm{bw,r}}$  for robust MDPs. Since robust MDPs with discounted optimality can be solved via value iteration and policy iteration, we provide the very first algorithms to compute Blackwell-optimal policies for robust MDPs.
We conclude this section with a discussion on related works. Several papers study the reduction of average optimality policy to discounted optimality under strong assumptions. Early attempts include (Ross, 1968), assuming that all transition probabilities are lower bounded by  $\epsilon > 0$ . Recent extensions assume bounded times of first returns (Akian and Gaubert, 2013, Huang, 2016), or weakly-communicating MDPs (Wang et al., 2022). Note that checking that an MDP instance is weakly-communicating can be done in polynomial-time (Kallenberg, 2002), in contrast to the unichain assumption (Tsitsiklis, 2007). The case of deterministic MDPs is treated in (Friedmann, 2011, Perotto and Vercouter, 2018, Zwick and Paterson, 1996). Other reductions require assumptions on the mixing times of the Markov chains induced by deterministic policies (Jin and Sidford, 2021). (Boone and Gaujal, 2022) propose a sampling algorithm to learn a Blackwell-optimal policy, in a special case in which it reduces to bias optimality. Under the condition that the robust MDP is unichain and that there is a unique average optimal policy, (Wang et al., 2023) show the existence of Blackwell-optimal policies for sa-rectangular robust MDPs, which is connected to the existence results in (Tewari and Bartlett, 2007) and (Goyal and Grand-Clément, 2023b) for polyhedral uncertainty. In contrast to the existing literature, one of the core strengths of our results is that we do not need any structural assumption on the Markov chains of the underlying MDP to obtain our reduction from Blackwell optimality and average optimality to discounted optimality.
2 Preliminaries on MDPs
An MDP instance is characterized by a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},\boldsymbol {r},\boldsymbol {P})$ , where  $\mathcal{S}$  is a finite set of states and  $\mathcal{A}$  is a finite set of actions. The instantaneous rewards are denoted by  $\boldsymbol {r}\in \mathbb{R}^{\mathcal{S}\times \mathcal{A}}$  and the transition probabilities are denoted by  $P\in (\Delta (S))^{S\times A}$ , where  $\Delta (S)$  is the simplex over  $\mathcal{S}$ . At any time period  $t$ , the decision maker is in a state  $s_t\in S$ , chooses an action  $a_{t}\in \mathcal{A}$ , obtains an instantaneous reward  $r_{s_t a_t}\in \mathbb{R}$ , and transitions to state  $s_{t + 1}$  with probability  $P_{s_t a_st_{t + 1}}\in [0,1]$ . A deterministic stationary policy  $\pi \colon S\to \mathcal{A}$  assigns an action to each state. Importantly, there exists an optimal deterministic stationary policy for all the criteria considered in this paper (discounted, Blackwell, and average optimality) (Puterman, 2014), so we simply refer to them as policies and denote them as  $\Pi = \mathcal{A}^S$ . A policy  $\pi \in \Pi$  induces a vector of expected instantaneous reward  $\boldsymbol {r}_{\pi}\in \mathbb{R}^{S}$ , defined as  $r_{\pi ,s} = r_{s\pi (s)},\forall s\in S$ , as well as a Markov chain over  $\mathcal{S}$ , evolving via a transition matrix  $P_{\pi}\in \mathbb{R}^{S\times S}$ , defined as  $P_{\pi ,ss'} = P_{s\pi (s)s'},\forall s,s'\in S$ . We also write  $r_\infty = \max \{|r_{sa}||(s,a)\in S\times \mathcal{A}\}$ .
Given a discount factor  $\gamma \in [0,1)$  and a policy  $\pi \in \Pi$ , the value function  $v_{\gamma}^{\pi} \in \mathbb{R}^{S}$  represents the discounted value obtained starting from each state:  $v_{\gamma ,s}^{\pi} = \mathbb{E}^{\pi ,P}\left[\sum_{t = 0}^{+\infty}\gamma^{t}r_{st,a_{t}}\Big|s_{0} = s\right], \forall s \in S$ . We start with discounted optimality, the most popular optimality criterion in RL.
Definition 2.1. Given  $\gamma \in [0,1)$ , a policy  $\pi \in \Pi$  is  $\gamma$ -discount-optimal if  $v_{\gamma,s}^{\pi} \geq v_{\gamma,s}^{\pi'}, \forall \pi' \in \Pi, \forall s \in S$ . We call  $\Pi_{\gamma}^{\star} \subset \Pi$  the set of  $\gamma$ -discount-optimal policies.
The discount factor  $\gamma \in [0,1)$  represents the preference for current rewards compared to future rewards. The difficulty of choosing the discount factor  $\gamma$  is well recognized in RL (Tang et al., 2021). In some applications, it is reasonable to choose values of  $\gamma$  close to 1, e.g., in finance (Deng et al., 2016), in healthcare (Garcia et al., 2021, Neumann et al., 2016) or in game solving (Brockman et al., 2016). In other applications,  $\gamma$  is merely treated as a parameter introduced for algorithmic purposes, e.g., controlling the variance of the policy gradient estimates (Baxter and Bartlett, 2001), or ensuring convergence of algorithms. A discount-optimal policy can be computed efficiently with value iteration, policy iteration, and linear programming (Puterman, 2014). Notably, these algorithms do not require any assumptions on the MDP instance  $\mathcal{M}$ .
Another fundamental optimality criterion is average optimality, where the average reward  $\pmb{g}^{\pi} \in \mathbb{R}^{S}$  of a policy  $\pi \in \Pi$  is  $g_{s}^{\pi} = \lim_{T \to +\infty} \frac{1}{T+1} \mathbb{E}^{\pi, \pmb{P}} \left[ \sum_{t=0}^{T} r_{s_{t},a_{t}} \Big| s_{0} = s \right]$ ,  $\forall s \in S$ . This limit always
exists for stationary policies (Puterman, 2014). A policy  $\pi$  is average-optimal if  $g^{\pi} \geq g^{\pi'}$ ,  $\forall \pi' \in \Pi$ . Average optimality has been extensively studied in the RL literature, as it alleviates the introduction of a potentially artificial discount factor. Classical algorithms include relative value iteration (Dong et al., 2019, Yang et al., 2016), and gradient-based methods (Bhatnagar et al., 2007, Iwaki and Asada, 2019). We refer to (Dewanto et al., 2020) for a survey on average optimality in RL.
Several technical complications arise from considering average optimality instead of discounted optimality. First, the average reward  $g^{\pi}$  of a policy is not a continuous function of the policy  $\pi$  (e.g., chapter 4, (Feinberg and Shwartz, 2012)). This can make gradient-based methods inefficient, since a small change in the policy may result in drastic changes in the average reward. Additionally, the Bellman operator associated with the average optimality criterion is not a contraction and may have multiple fixed points. These complications can be circumvented by assuming structural properties on the MDP instance, such as bounded times of first returns and weakly-communicating MDPs (Akian and Gaubert, 2013, Wang et al., 2022). Some of these assumptions may be hard to verify in a simulation environment where only samples are available, or NP-hard to verify even when the MDP instance is fully known, as is the case for the unichain assumption (Tsitsiklis, 2007). One of our goals in this paper is to provide a method to compute average-optimal policies via solving discounted MDPs, without any restrictive structural assumptions on the MDP instance. We will do so via the notion of Blackwell optimality.
3 Classical theory of Blackwell optimality
In this section, we describe the classical definition of Blackwell optimality in MDPs and summarize its main limitations. We first give this definition of a Blackwell-optimal policy and outline the proof of its existence. This proof will serve as a building block of our main result in Section 4. We then highlight the main limitations of the existing definition of Blackwell optimality.
Existing definition and algorithms. We start with the following classical definition.
Definition 3.1. A policy  $\pi$  is Blackwell-optimal if there exists  $\gamma \in [0,1)$ , such that  $\pi \in \Pi_{\gamma'}^\star$ ,  $\forall \gamma' \in [\gamma,1)$ . We call  $\Pi_{\mathrm{bw}}^\star$  the set of Blackwell-optimal policies.
In short, a Blackwell-optimal policy is  $\gamma$ -discount-optimal for all discount factors  $\gamma$  sufficiently close to 1 (Blackwell, 1962). This notion has become popular in the field of reinforcement learning, mainly due to its connection to average optimality (Dewanto and Gallagher, 2021). Blackwell optimality bridges the gap between the different optimality criteria: it is defined in terms of discounted optimality, yet, crucially, Blackwell-optimal policies are average-optimal (theorem 10.1.5, (Puterman, 2014)). Therefore, any advances in computing Blackwell-optimal policies transfer to advances in computing average-optimal policies. A Blackwell-optimal policy is guaranteed to exist for finite MDPs.
Theorem 3.2 ((Blackwell, 1962)). When  $|\mathcal{S}| < +\infty$ ,  $|\mathcal{A}| < +\infty$ , there exists at least one Blackwell-optimal policy:  $\Pi_{\mathrm{bw}}^{\star} \neq \emptyset$ .
We highlight the proof of Theorem 3.2 based on section 10.1.1 in (Puterman, 2014). Summarizing this proof is important because it is not well-known and serves as a building block for our results.
Step 1. Let  $\pi, \pi' \in \Pi, s \in S$ . Through this paper use the notation  $\phi_s^{\pi, \pi'}$  for  $\phi_s^{\pi, \pi'}: \gamma \mapsto v_{\gamma, s}^{\pi} - v_{\gamma, s}'$ . We first show that  $\phi_s^{\pi, \pi'}$  has finitely many zeros in  $[0, 1)$ . This is a consequence of the next lemma.
Lemma 3.3. For  $\pi \in \Pi$  and  $s\in S$ ,  $\gamma \mapsto v_{\gamma ,s}^{\pi}$  is a rational function on  $[0,1)$ , i.e., it is the ratio of two polynomials.
Lemma 3.3 follows from the Bellman equation for the value function  $\pmb{v}^{\pi} \colon \pmb{v}^{\pi} = \pmb{r}_{\pi} + \gamma \pmb{P}_{\pi} \pmb{v}^{\pi}$ . Therefore,  $\pmb{v}^{\pi}$  is the unique solution to the equation  $\pmb{A}\pmb{x} = \pmb{b}$ , for  $\pmb{b} = \pmb{r}_{\pi}$  and  $\pmb{A} = \pmb{I} - \gamma \pmb{P}_{\pi}$ . Lemma 3.3 then follows directly from Cramer's rule for the solution of a system of linear equations: since  $\pmb{A}$  is invertible, then  $A\pmb{x} = \pmb{b}$  has a unique solution  $\pmb{x}$ , which satisfies  $x_{s} = \operatorname*{det}(A_{s}) / \operatorname*{det}(A)$ ,  $\forall s \in S$ , with  $\operatorname*{det}(\cdot)$  the determinant of a matrix and  $A_{s}$  the matrix formed by replacing the  $s$ -th column of  $A$  by the vector  $\pmb{b}$ . A consequence of Lemma 3.3 is that the function  $\phi_s^{\pi ,\pi '}$  is a rational function, and therefore its zeros are the zeros of a polynomial. This shows that  $\phi_s^{\pi ,\pi '}$  is either identically equal to 0, or it has only finitely many roots in [0, 1).
Step 2. We now conclude the proof of Theorem 3.2. Let  $\pi, \pi' \in \Pi$ ,  $s \in S$ . If  $\phi_s^{\pi, \pi'}$  is not identically equal to  $0$ , let  $\gamma(\pi, \pi', s) \in [0,1)$  be its the largest zero of  $\phi_s^{\pi, \pi'}$  in  $[0,1)$ :  $\gamma(\pi, \pi', s) = \max\{\gamma \in [0,1) | v_{\gamma, s}^{\pi} - v_{\gamma, s}^{\pi'} = 0\}$ . We let  $\gamma(\pi, \pi', s) = 0$  if  $\phi_s^{\pi, \pi'}$  is identically equal to  $0$  in  $[0,1)$ . We now let
$$
\bar {\gamma} = \max  _ {\pi , \pi^ {\prime} \in \Pi , s \in \mathcal {S}} \gamma \left(\pi , \pi^ {\prime}, s\right). \tag {3.1}
$$
We have  $\bar{\gamma} < 1$  since there is a finite number of (stationary, deterministic) policies and  $|\mathcal{S}| < +\infty$ . Let  $\pi$  be  $\gamma$ -discount-optimal for a certain  $\gamma > \bar{\gamma}$ . We have, for any  $s \in S$ ,  $v_{\gamma,s}^{\pi} \geq v_{\gamma,s}^{\pi'}, \forall \pi' \in \Pi$ . By definition of  $\bar{\gamma}$ , the map  $\phi_s^{\pi,\pi'}$  cannot change a sign on  $[\bar{\gamma}, 1)$  (because it cannot be equal to 0), for any policy  $\pi' \in \Pi$  and any state  $s \in S$ , i.e., we have  $v_{\gamma',s}^{\pi} \geq v_{\gamma',s}^{\pi'}, \forall \pi' \in \Pi, \forall \gamma' \in (\gamma, 1)$ . This shows that  $\pi$  remains  $\gamma'$ -discount-optimal for all  $\gamma' > \gamma$ , and, therefore,  $\pi$  is Blackwell-optimal.
Remark 3.4. At this point, the reader may wonder if some Blackwell optimal policies are "better" than others, e.g., for instance, if we can find a Blackwell optimal policy that is  $\gamma$ -discount optimal for  $\gamma$  as small as possible. Interestingly, all Blackwell optimal policies are  $\gamma$ -discount optimal (or not) for the same discount factors. This follows from the key property that the value functions of Blackwell optimal policies coincide for all  $\gamma \in (0,1)$  at all states  $s \in S$ . Indeed, these value functions must coincide on an entire interval close enough to 1, and they are rational functions. Hence, if they are equal for an infinite number of discount factors, they are equal on the entire interval  $(0,1)$ .
To the best of our knowledge, there are only two existing algorithms to compute a Blackwell-optimal policy. The first algorithm (Hordijk et al., 1985) formulates MDPs with varying discount factors as linear programs (LPs) over the field of power series with potentially negative coefficients, known as Laurent series. The simplex method for solving LPs over power series explores  $[0,1)$  and computes the subintervals of  $[0,1)$  where an optimal policy can be chosen constant (as a function of  $\gamma$ ). It returns a Blackwell-optimal policy in a finite number of operations, but there are no complexity guarantees for this algorithm. The second algorithm is based on  $n$ -discount optimality, described with a set of  $(|S| + 1)$ -nested equations indexed by  $n = -1,\dots,|S| - 1$  that need to be solved sequentially by solving three LPs at each stage  $n$  (O'Sullivan and Veinott Jr, 2017). This gives a polynomial-time algorithm for computing Blackwell-optimal policies, requiring solving  $3(|S| + 1)$  linear programs of dimension  $O(|S|)$ . A simpler description is in section 10.3.4 in (Puterman, 2014), but only finite convergence is proved. We are not aware of any available implementations of these algorithms.
Limitations of existing approaches. We now highlight the shortcomings of the existing definition of Blackwell optimality. In particular, we demonstrate that the current approach is insufficient to reduce Blackwell optimality to discount optimality, we show that it does not lead to simple algorithms, and we show that it completely overlooks the potential pathological behaviors of the value functions.
First, Definition 3.1 leads to methods that are significantly more involved than solving discounted MDPs. The two existing algorithms for computing Blackwell-optimal policies handle complex objects, e.g., the simplex algorithm over the field of power series and nested optimality equations with multiple subproblems that need to be solved sequentially. The intricacy of both algorithms makes them difficult to implement, and these algorithms are not widely used in practice.
Second, Definition 3.1 implicitly introduces, for each Blackwell-optimal policy  $\pi \in \Pi_{\mathrm{bw}}^{\star}$ , a discount factor  $\gamma (\pi)\in [0,1)$ , defined as the smallest discount factor after which  $\pi$  remains discount-optimal:
$$
\gamma (\pi) = \min  \left\{\gamma \in [ 0, 1) \mid \pi \in \Pi_ {\gamma^ {\prime}} ^ {\star}, \forall \gamma^ {\prime} \in [ \gamma , 1) \right\}. \tag {3.2}
$$
We now show that  $\gamma (\pi)$  provides insufficient information to compute a Blackwell-optimal policy.
Proposition 3.5. There exists an MDP instance  $\mathcal{M}$ , a Blackwell-optimal policy  $\pi \in \Pi_{\mathrm{bw}}^{\star}$ , and discount factors  $\gamma_1, \gamma_2 \in [0,1)$  with  $\gamma_1 < \gamma(\pi) < \gamma_2$  such that:
1. the policy  $\pi$  is  $\gamma_{1}$ -discount-optimal, and
2. there exists  $\pi' \neq \pi$  that is  $\gamma_2$ -discount-optimal and not Blackwell-optimal.
Proposition 3.5 shows the naive approach of solving a  $\gamma$ -discounted MDP for discount factor  $\gamma > \gamma(\pi)$  does not compute a Blackwell-optimal policy. That is, the policy  $\pi'$  in Proposition 3.5 is optimal for  $\gamma_2 > \gamma(\pi)$  but is not Blackwell-optimal. It also shows that  $\gamma(\pi)$  is not even the smallest discount factor for which  $\pi$  is discount-optimal. Note that we are the first to highlight this shortcoming of the classical definition of Blackwell optimality. We also note that Proposition 3.5 remains true even under the assumption that MDP instance is unichain, as we prove in Appendix A.Overall, we have
shown that the discount factor  $\gamma (\pi)$ , appearing in the classical definition of Blackwell optimality, cannot be exploited to compute a Blackwell-optimal policy.
The limitation outlined above calls for the definition of another discount factor that can adequately describe when does the set of discount-optimal policies equals to the set of Blackwell optimal policies. We introduce this Blackwell discount factor in the next section. The proof of Proposition 3.5 is based on the following very simple example, with  $|\mathcal{S}| = 8$ ,  $|\mathcal{A}| = 3$ , and deterministic transitions.
Example 3.6. We consider the MDP instance from Figure 1. The decision maker starts in state 0 and chooses one of three actions  $\{a_1, a_2, a_3\}$ ; there is no choice in other states, all transitions are deterministic, and the rewards are indicated above the transition arcs. The reward for  $a_1$  is 1 and the process transitions to the absorbing state 7, which gives a reward of 0. The reward for  $a_2$  is 0, and the process transitions to states 1, 2, 3 before reaching the absorbing state 7. The value functions equal to  $v_{\gamma}^{a_2} = r_1\gamma + r_2\gamma^2$ ,  $v_{\gamma}^{a_3} = r_4\gamma + r_5\gamma^2$ ,  $v_{\gamma}^{a_1} = 1$ . Choosing  $(r_1, r_2) = (6, -8)$  and  $(r_4, r_5) = (8/3, -16/9)$  gives the value functions shown in Figure 1 (left figure). In particular,  $v_{\gamma}^{a_2}$  is the parabola that is equal to 0 at  $\gamma = 0$ , and equal to 1 at  $\gamma \in \{1/4, 1/2\}$ , and  $v_{\gamma}^{a_3}$  is the parabola that is equal to 0 at  $\gamma = 0$  and equal to its maximum 1 at  $\gamma = 3/4$ . This shows that  $a_1$  is Blackwell-optimal with  $\gamma(a_1) = 1/2$ . Additionally, for  $\gamma_1 \in [0, 1/4]$ ,  $a_1$  is  $\gamma_1$ -discount-optimal. Finally,  $a_3$  is  $\gamma_2$ -discount-optimal for  $\gamma_2 = 3/4$ , but it is not Blackwell-optimal.
Figure 1: MDP instance (left) and value functions (right) for Example 3.6.