Title: Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factor
Venue: NeurIPS 2023
Accepted: yes
Keywords: Markov Decision Process

Blackwell optimality

average optimality

robust optimization
Abstract: We introduce the Blackwell discount factor for Markov Decision Processes (MDPs). Classical objectives for MDPs include discounted, average, and Blackwell optimality. Many existing approaches to computing average-optimal policies solve for discount-optimal policies with a discount factor close to $1$, but they only work under strong or hard-to-verify assumptions on the MDP structure  such as unichain or ergodicity. We are the first to highlight the shortcomings of the classical definition of Blackwell optimality, which does not lead to simple algorithms for computing Blackwell-optimal policies and overlooks the pathological behaviors of optimal policies as regards the discount factors. To resolve this issue, in this paper, we show that when the discount factor is larger than  the Blackwell discount factor $\gamma_{\sf bw}$, all discount-optimal policies become Blackwell- and average-optimal, and we derive a general upper bound on $\gamma_{\sf bw}$. Our upper bound on $\gamma_{\sf bw}$, parametrized by the bit-size of the rewards and transition probabilities of the MDP instance, provides the first reduction from average and Blackwell optimality to discounted optimality, without any assumptions, along with new polynomial-time algorithms. Our work brings new ideas from polynomials and algebraic numbers to the analysis of MDPs. Our results also apply to robust MDPs, enabling the first algorithms to compute robust Blackwell-optimal policies.

Reducing Blackwell and Average Optimality to Discounted MDPs via the Blackwell Discount Factor

Abstract

We introduce the Blackwell discount factor for Markov Decision Processes (MDPs). Classical objectives for MDPs include discounted, average, and Blackwell optimality. Many existing approaches to computing average-optimal policies solve for discount-optimal policies with a discount factor close to 1 , but they only work under strong or hard-to-verify assumptions such as unichain or ergodicity. We highlight the shortcomings of the classical definition of Blackwell optimality, which does not lead to simple algorithms for computing Blackwell-optimal policies and overlooks the pathological behaviors of optimal value functions with respect to the discount factors. To resolve this issue, we show that when the discount factor is larger than the Blackwell discount factor γ bw , all discount-optimal policies become Blackwell- and average-optimal, and we derive a general upper bound on γ bw . Our upper bound on γ bw , parametrized by the bit-size of the rewards and transition probabilities of the MDP instance, provides the first reduction from average and Blackwell optimality to discounted optimality, without any assumptions , along with new polynomial-time algorithms. Our work brings new ideas from polynomials and algebraic numbers to the analysis of MDPs. Our results also apply to robust MDPs, enabling the first algorithms to compute robust Blackwell-optimal policies.

1 Introduction

Markov Decision Processes (MDPs) provide a widely-used framework for modeling sequential decision-making problems (Puterman, 2014). In a (finite) MDP, the decision maker repeatedly interacts with an environment characterized by a finite set of states and a finite set of available actions. The decision maker follows a policy that prescribes an action at a state at every period. An instantaneous reward is obtained at every period, depending on the current state-action pair, and the system transitions to the next state at the next period. MDPs provide the underlying model for the applications of reinforcement learning (RL), ranging from healthcare (Gottesman et al., 2019) to game solving (Mnih et al., 2013) and finance (Deng et al., 2016).

There are several optimality criteria that measure a decision maker's performance in an MDP. In discounted optimality , the decision maker optimizes the discounted return, defined as the sum of the instantaneous rewards over the infinite horizon, where future rewards are discounted with a discount factor γ ∈ [0 , 1) . In average optimality , the decision maker optimizes the average return, defined as the average of the instantaneous rewards obtained over the infinite horizon. The average return ignores any return gathered in finite time, i.e., it does not reflect the transient performance of a policy and it only focuses on the steady-state behavior. The most selective optimality criterion in MDPs is Blackwell optimality (Puterman, 2014). A policy is Blackwell-optimal if it optimizes the discounted return simultaneously for all discount factors sufficiently close to 1 . Since a discount factor close

to 1 can be interpreted as a preference for rewards obtained in later periods, Blackwell-optimal policies are also average-optimal. However, average-optimal policies need not be Blackwell-optimal. Blackwell optimality can be a useful criterion in environments with no natural, or known, discount factor. Also, any algorithm that computes a Blackwell-optimal policy also immediately computes an average-optimal policy. This is one of the reasons why better understanding the Blackwell optimality criterion is mentioned as 'one of the pressing questions in RL' in the list of open research problems from a recent survey on RL for average reward optimality (Dewanto et al., 2020).

Average-optimal policies can be computed via linear programming (section 9.3, (Puterman, 2014)). However, virtually all of the recent algorithms for computing average-optimal policies require strong assumptions on the underlying Markov chains associated with the policies in the MDP instance, such as ergodicity (Wang, 2017), the unichain and aperiodicity properties (Schneckenreither, 2020), weakly communicating MDPs (Wang et al., 2022), or assumptions on the mixing time associated with any deterministic policies (Jin and Sidford, 2020, 2021). These assumptions are motivated by technical considerations (e.g., ensuring that the average reward is uniform across all states) and can be restrictive in practice (Puterman, 2014) and NP-hard to verify, such as unichain (Tsitsiklis, 2007). Existing methods for computing Blackwell-optimal policies rely on linear programming over the field of power series including negative coefficients (Hordijk et al., 1985), or on an algorithm based on a nested sequence of optimality equations (O'Sullivan and Veinott Jr, 2017) which requires to solve multiple linear programs sequentially. These algorithms are complex, difficult to implement, and have no complexity guarantees or known implementations.

In summary, existing algorithms for average optimality require restrictive assumptions, and algorithms for Blackwell-optimality are very complex. This is in stark contrast with the vast literature on solving discounted MDPs, where general and well-understood methods exist, including value iteration, policy iteration, and linear programming (chapter 6, (Puterman, 2014)). This is the starting point of this paper, which aims to develop new algorithms for computing average-optimal and Blackwell-optimal policies through a reduction to discounted MDPs. We make the following three main contributions.

1) A new definition of Blackwell optimality via the Blackwell discount factor γ bw ∈ [0 , 1) . Our first main contribution is to highlight that the standard definition of Blackwell optimality cannot be used to compute Blackwell-optimal policies with simple algorithms. Standard definitions have focused on necessary condition for Blackwell optimal policies to be discount optimal. However, we show that this condition needs to be revised when one seeks to compute a Blackwell-optimal policy. We do so by highlighting the potential pathological behaviors of the value functions: a Blackwell-optimal policy may be optimal on an arbitrary number of arbitrary disjoint intervals, and other non-Blackwell optimal policies may also be discount-optimal for some discount factors very close to 1 . Demonstrating this issue is important because previous literature has repeatedly overlooked it. To address this issue, we introduce and show the existence of a discount factor γ bw such that discount optimality for γ > γ bw is sufficient for Blackwell optimality. Knowing the discount factor γ bw is vital because it enables one to compute Blackwell- and average-optimal policies simply by solving a discounted MDP with γ ∈ ( γ bw , 1) , for which there exist well-studied, simple, and efficient algorithms.

2) Upper-bound the Blackwell discount factor. As our second main contribution, we provide a strict upper bound on γ bw given an MDP instance. We show that an upper bound must depend on r and P , and we compute a bound that is parametrized by the number of states and the number of bits required to represent the MDP instance. Solving a discounted MDP with a discount factor larger or equal than our strict upper bound returns a Blackwell-optimal policy. Crucially, our strict upper bound requires no assumptions on the underlying structure of the MDP , which is a significant improvement on existing literature. Interestingly, the construction of our upper bound relies on novel techniques for analyzing MDPs. We interpret γ bw ∈ [0 , 1) as the root of a polynomial equation p ( γ ) = 0 in γ , show p (1) = 0 , and use a lower bound sep ( p ) on the distance between any two roots of a polynomial p , known as the separation of algebraic numbers . This shows that γ bw < 1 -sep ( p ) , where sep ( p ) depends on the MDP instance. Since Blackwell optimality implies average optimality, we also obtain the first reduction from average optimality to discounted optimality, without any assumption on the MDP structure. Our upper bound on γ bw is itself of polynomial size in the bit-size of the MDP data. Combining this bound with interior-point methods for solving discounted MDPs, we obtain new weakly-polynomial time algorithms for computing Blackwell-optimal and average-optimal policies.

3) Blackwell discount factor for robust MDPs. We consider the case of robust reinforcement learning where the transition probabilities are unknown and, instead, belong to an uncertainty set. As our

third main contribution, we show that the robust Blackwell discount factor γ bw , r exists for popular models of uncertainty, such as sa-rectangular robust MDPs with polyhedral uncertainty (Goyal and Grand-Clément, 2023b, Iyengar, 2005). For this setting, we generalize our upper bound on γ bw for MDPs to an upper bound on γ bw , r for robust MDPs. Since robust MDPs with discounted optimality can be solved via value iteration and policy iteration, we provide the very first algorithms to compute Blackwell-optimal policies for robust MDPs.

We conclude this section with a discussion on related works . Several papers study the reduction of average optimality policy to discounted optimality under strong assumptions. Early attempts include (Ross, 1968), assuming that all transition probabilities are lower bounded by ϵ > 0 . Recent extensions assume bounded times of first returns (Akian and Gaubert, 2013, Huang, 2016), or weakly-communicating MDPs (Wang et al., 2022). Note that checking that an MDP instance is weakly-communicating can be done in polynomial-time (Kallenberg, 2002), in contrast to the unichain assumption (Tsitsiklis, 2007). The case of deterministic MDPs is treated in (Friedmann, 2011, Perotto and Vercouter, 2018, Zwick and Paterson, 1996). Other reductions require assumptions on the mixing times of the Markov chains induced by deterministic policies (Jin and Sidford, 2021). (Boone and Gaujal, 2022) propose a sampling algorithm to learn a Blackwell-optimal policy, in a special case in which it reduces to bias optimality. Under the condition that the robust MDP is unichain and that there is a unique average optimal policy, (Wang et al., 2023) show the existence of Blackwell-optimal policies for sa-rectangular robust MDPs, which is connected to the existence results in (Tewari and Bartlett, 2007) and (Goyal and Grand-Clément, 2023b) for polyhedral uncertainty. In contrast to the existing literature, one of the core strengths of our results is that we do not need any structural assumption on the Markov chains of the underlying MDP to obtain our reduction from Blackwell optimality and average optimality to discounted optimality.

2 Preliminaries on MDPs

An MDP instance is characterized by a tuple M = ( S , A , r , P ) , where S is a finite set of states and A is a finite set of actions. The instantaneous rewards are denoted by r ∈ R S×A and the transition probabilities are denoted by P ∈ (∆( S )) S×A , where ∆( S ) is the simplex over S . At any time period t , the decision maker is in a state s t ∈ S , chooses an action a t ∈ A , obtains an instantaneous reward r s t a t ∈ R , and transitions to state s t +1 with probability P s t a t s t +1 ∈ [0 , 1] . A deterministic stationary policy π : S → A assigns an action to each state. Importantly, there exists an optimal deterministic stationary policy for all the criteria considered in this paper (discounted, Blackwell, and average optimality) (Puterman, 2014), so we simply refer to them as policies and denote them as Π = A S . A policy π ∈ Π induces a vector of expected instantaneous reward r π ∈ R S , defined as r π,s = r sπ ( s ) , ∀ s ∈ S , as well as a Markov chain over S , evolving via a transition matrix P π ∈ R S×S , defined as P π,ss ′ = P sπ ( s ) s ′ , ∀ s, s ′ ∈ S . We also write r ∞ = max {| r sa | | ( s, a ) ∈ S × A} .

Given a discount factor γ ∈ [0 , 1) and a policy π ∈ Π , the value function v π γ ∈ R S represents the discounted value obtained starting from each state: v π γ,s = E π, P [ ∑ + ∞ t =0 γ t r s t ,a t ∣ ∣ ∣ s 0 = s ] , ∀ s ∈ S . We start with discounted optimality, the most popular optimality criterion in RL.

Definition 2.1. Given γ ∈ [0 , 1) , a policy π ∈ Π is γ -discount-optimal if v π γ,s ≥ v π ′ γ,s , ∀ π ′ ∈ Π , ∀ s ∈ S . We call Π ⋆ γ ⊂ Π the set of γ -discount-optimal policies.

The discount factor γ ∈ [0 , 1) represents the preference for current rewards compared to future rewards. The difficulty of choosing the discount factor γ is well recognized in RL (Tang et al., 2021). In some applications, it is reasonable to choose values of γ close to 1 , e.g., in finance (Deng et al., 2016), in healthcare (Garcia et al., 2021, Neumann et al., 2016) or in game solving (Brockman et al., 2016). In other applications, γ is merely treated as a parameter introduced for algorithmic purposes, e.g., controlling the variance of the policy gradient estimates (Baxter and Bartlett, 2001), or ensuring convergence of algorithms. A discount-optimal policy can be computed efficiently with value iteration, policy iteration, and linear programming (Puterman, 2014). Notably, these algorithms do not require any assumptions on the MDP instance M .

Another fundamental optimality criterion is average optimality , where the average reward g π ∈ R S of a policy π ∈ Π is g π s = lim T → + ∞ 1 T +1 E π, P [ ∑ T t =0 r s t ,a t ∣ ∣ ∣ s 0 = s ] , ∀ s ∈ S . This limit always

exists for stationary policies (Puterman, 2014). A policy π is average-optimal if g π ≥ g π ′ , ∀ π ′ ∈ Π . Average optimality has been extensively studied in the RL literature, as it alleviates the introduction of a potentially artificial discount factor. Classical algorithms include relative value iteration (Dong et al., 2019, Yang et al., 2016), and gradient-based methods (Bhatnagar et al., 2007, Iwaki and Asada, 2019). We refer to (Dewanto et al., 2020) for a survey on average optimality in RL.

Several technical complications arise from considering average optimality instead of discounted optimality. First, the average reward g π of a policy is not a continuous function of the policy π (e.g., chapter 4, (Feinberg and Shwartz, 2012)). This can make gradient-based methods inefficient, since a small change in the policy may result in drastic changes in the average reward. Additionally, the Bellman operator associated with the average optimality criterion is not a contraction and may have multiple fixed points. These complications can be circumvented by assuming structural properties on the MDP instance, such as bounded times of first returns and weakly-communicating MDPs (Akian and Gaubert, 2013, Wang et al., 2022). Some of these assumptions may be hard to verify in a simulation environment where only samples are available, or NP-hard to verify even when the MDP instance is fully known, as is the case for the unichain assumption (Tsitsiklis, 2007). One of our goals in this paper is to provide a method to compute average-optimal policies via solving discounted MDPs, without any restrictive structural assumptions on the MDP instance . We will do so via the notion of Blackwell optimality .

3 Classical theory of Blackwell optimality

In this section, we describe the classical definition of Blackwell optimality in MDPs and summarize its main limitations. We first give this definition of a Blackwell-optimal policy and outline the proof of its existence. This proof will serve as a building block of our main result in Section 4. We then highlight the main limitations of the existing definition of Blackwell optimality.

Existing definition and algorithms. We start with the following classical definition.

Definition 3.1. A policy π is Blackwell-optimal if there exists γ ∈ [0 , 1) , such that π ∈ Π ⋆ γ ′ , ∀ γ ′ ∈ [ γ, 1) . We call Π ⋆ bw the set of Blackwell-optimal policies.

In short, a Blackwell-optimal policy is γ -discount-optimal for all discount factors γ sufficiently close to 1 (Blackwell, 1962). This notion has become popular in the field of reinforcement learning, mainly due to its connection to average optimality (Dewanto and Gallagher, 2021). Blackwell optimality bridges the gap between the different optimality criteria: it is defined in terms of discounted optimality, yet, crucially, Blackwell-optimal policies are average-optimal (theorem 10.1.5, (Puterman, 2014)). Therefore, any advances in computing Blackwell-optimal policies transfer to advances in computing average-optimal policies. A Blackwell-optimal policy is guaranteed to exist for finite MDPs.

̸

Theorem 3.2 ((Blackwell, 1962)) . When |S| < + ∞ , |A| < + ∞ , there exists at least one Blackwelloptimal policy: Π ⋆ bw = ∅ .

We highlight the proof of Theorem 3.2 based on section 10.1.1 in (Puterman, 2014). Summarizing this proof is important because it is not well-known and serves as a building block for our results.

Step 1. Let π, π ′ ∈ Π , s ∈ S . Through this paper use the notation ϕ π,π ′ s for ϕ π,π ′ s : γ ↦→ v π γ,s -v π ′ γ,s . We first show that ϕ π,π ′ s has finitely many zeros in [0 , 1) . This is a consequence of the next lemma.

Lemma 3.3. For π ∈ Π and s ∈ S , γ ↦→ v π γ,s is a rational function on [0 , 1) , i.e., it is the ratio of two polynomials.

Lemma 3.3 follows from the Bellman equation for the value function v π : v π = r π + γ P π v π . Therefore, v π is the unique solution to the equation Ax = b , for b = r π and A = I -γ P π . Lemma 3.3 then follows directly from Cramer's rule for the solution of a system of linear equations: since A is invertible, then Ax = b has a unique solution x , which satisfies x s = det( A s ) / det( A ) , ∀ s ∈ S , with det( · ) the determinant of a matrix and A s the matrix formed by replacing the s -th column of A by the vector b . A consequence of Lemma 3.3 is that the function ϕ π,π ′ s is a rational function, and therefore its zeros are the zeros of a polynomial. This shows that ϕ π,π ′ s is either identically equal to 0 , or it has only has finitely many roots in [0 , 1) .

Step 2. We now conclude the proof of Theorem 3.2. Let π, π ′ ∈ Π , s ∈ S . If ϕ π,π ′ s is not identically equal to 0 , let γ ( π, π ′ , s ) ∈ [0 , 1) be its the largest zero of ϕ π,π ′ s in [0 , 1) : γ ( π, π ′ , s ) = max { γ ∈ [0 , 1) | v π γ,s -v π ′ γ,s = 0 } . We let γ ( π, π ′ , s ) = 0 if ϕ π,π ′ s is identically equal to 0 in [0 , 1) . We now let

We have ¯ γ < 1 since there is a finite number of (stationary, deterministic) policies and |S| < + ∞ . Let π be γ -discount-optimal for a certain γ > ¯ γ . We have, for any s ∈ S , v π γ,s ≥ v π ′ γ,s , ∀ π ′ ∈ Π . By definition of ¯ γ , the map ϕ π,π ′ s cannot change a sign on [¯ γ, 1) (because it cannot be equal to 0 ), for any policy π ′ ∈ Π and any state s ∈ S , i.e., we have v π γ ′ ,s ≥ v π ′ γ ′ ,s , ∀ π ′ ∈ Π , ∀ γ ′ ∈ ( γ, 1) . This shows that π remains γ ′ -discount-optimal for all γ ′ > γ , and, therefore, π is Blackwell-optimal.

Remark 3.4. At this point, the reader may wonder if some Blackwell optimal policies are 'better" than others, e.g., for instance, if we can find a Blackwell optimal policy that is γ -discount optimal for γ as small as possible. Interestingly, all Blackwell optimal policies are γ -discount optimal (or not) for the same discount factors. This follows from the key property that the value functions of Blackwell optimal policies coincide for all γ ∈ (0 , 1) at all states s ∈ S . Indeed, these value functions must coincide on an entire interval close enough to 1 , and they are rational functions. Hence, if they are equal for an infinite number of discount factors, they are equal on the entire interval (0 , 1) .

To the best of our knowledge, there are only two existing algorithms to compute a Blackwell-optimal policy. The first algorithm (Hordijk et al., 1985) formulates MDPs with varying discount factors as linear programs (LPs) over the field of power series with potentially negative coefficients, known as Laurent series. The simplex method for solving LPs over power series explores [0 , 1) and computes the subintervals of [0 , 1) where an optimal policy can be chosen constant (as a function of γ ). It returns a Blackwell-optimal policy in a finite number of operations, but there are no complexity guarantees for this algorithm. The second algorithm is based on n -discount optimality, described with a set of ( |S| +1) -nested equations indexed by n = -1 , ..., |S| 1 that need to be solved sequentially by solving three LPs at each stage n (O'Sullivan and Veinott Jr, 2017). This gives a polynomial-time algorithm for computing Blackwell-optimal policies, requiring solving 3( |S| +1) linear programs of dimension O ( |S| ) . A simpler description is in section 10.3.4 in (Puterman, 2014), but only finite convergence is proved. We are not aware of any available implementations of these algorithms.

Limitations of existing approaches. We now highlight the shortcomings of the existing definition of Blackwell optimality. In particular, we demonstrate that the current approach is insufficient to reduce Blackwell optimality to discount optimality, we show that it does not lead to simple algorithms, and we show that it completely overlooks the potential pathological behaviors of the value functions.

First, Definition 3.1 leads to methods that are significantly more involved than solving discounted MDPs. The two existing algorithms for computing Blackwell-optimal policies handle complex objects, e.g., the simplex algorithm over the field of power series and nested optimality equations with multiple subproblems that need to be solved sequentially. The intricacy of both algorithms makes them difficult to implement, and these algorithms are not widely used in practice.

Second, Definition 3.1 implicitly introduces, for each Blackwell-optimal policy π ∈ Π ⋆ bw , a discount factor γ ( π ) ∈ [0 , 1) , defined as the smallest discount factor after which π remains discount-optimal:

We now show that γ ( π ) provides insufficient information to compute a Blackwell-optimal policy.

Proposition 3.5. There exists an MDP instance M , a Blackwell-optimal policy π ∈ Π ⋆ bw , and discount factors γ 1 , γ 2 ∈ [0 , 1) with γ 1 < γ ( π ) < γ 2 such that:

the policy π is γ 1 -discount-optimal, and

̸

there exists π ′ = π that is γ 2 -discount-optimal and not Blackwell-optimal.

Proposition 3.5 shows the naive approach of solving a γ -discounted MDP for discount factor γ > γ ( π ) does not compute a Blackwell-optimal policy. That is, the policy π ′ in Proposition 3.5 is optimal for γ 2 > γ ( π ) but is not Blackwell-optimal. It also shows that γ ( π ) is not even the smallest discount factor for which π is discount-optimal. Note that we are the first to highlight this shortcoming of the classical definition of Blackwell optimality. We also note that Proposition 3.5 remains true even under the assumption that MDP instance is unichain, as we prove in Appendix A.Overall, we have

shown that the discount factor γ ( π ) , appearing in the classical definition of Blackwell optimality, cannot be exploited to compute a Blackwell-optimal policy.

The limitation outlined above calls for the definition of another discount factor that can adequately describe when does the set of discount-optimal policies equals to the set of Blackwell optimal policies. We introduce this Blackwell discount factor in the next section. The proof of Proposition 3.5 is based on the following very simple example, with |S| = 8 , |A| = 3 , and deterministic transitions.

Example 3.6. We consider the MDP instance from Figure 1. The decision maker starts in state 0 and chooses one of three actions { a 1 , a 2 , a 3 } ; there is no choice in other states, all transitions are deterministic, and the rewards are indicated above the transition arcs. The reward for a 1 is 1 and the process transitions to the absorbing state 7 , which gives a reward of 0 . The reward for a 2 is 0 , and the process transitions to states 1 , 2 , 3 before reaching the absorbing state 7 . The value functions equal to v a 2 γ = r 1 γ + r 2 γ 2 , v a 3 γ = r 4 γ + r 5 γ 2 , v a 1 γ = 1 . Choosing ( r 1 , r 2 ) = (6 , -8) and ( r 4 , r 5 ) = (8 / 3 , -16 / 9) gives the value functions shown in Figure 1 (left figure). In particular, v a 2 γ is the parabola that is equal to 0 at γ = 0 , and equal to 1 at γ ∈ { 1 / 4 , 1 / 2 } , and v a 3 γ is the parabola that is equal to 0 at γ = 0 and equal to its maximum 1 at γ = 3 / 4 . This shows that a 1 is Blackwell-optimal with γ ( a 1 ) = 1 / 2 . Additionally, for γ 1 ∈ [0 , 1 / 4] , a 1 is γ 1 -discount-optimal. Finally, a 3 is γ 2 -discount-optimal for γ 2 = 3 / 4 , but it is not Blackwell-optimal.

0

7

1

a

1

4

5

6

0

1

2

3

0

r

1

0

a

2

a

3

0

r

2

r

4

r

5

0.0

0.2

0.4

0.6

0.8

1.0

Discount factor

γ

-2

-1

0

1

Value functions

v a

1

γ

v a

2

γ

v a

3

γ

In the next proposition, we show that the subintervals of [0 , 1) where a policy is discount-optimal may be much more complex than usually alluded to in the literature. In particular, there exists a simple MDP instance with only two policies, but where a Blackwell-optimal policy may be discount-optimal in an arbitrary number of arbitrary disjoint subintervals of [0 , 1) .

Theorem 3.7. For any odd integer N ∈ N and any sequence 0 = γ 0 < γ 1 < ... < γ N -1 < γ N = 1 , there exists an MDP instance ( S , A , r , P ) with |S| = N +1 and |A| = 2 , and two policies π 1 , π 2 such that π 1 is the unique optimal policy on any of the intervals ( γ 2 i , γ 2 i +1 ) for i = 0 , ..., ( N -1) / 2 and π 2 is the unique optimal policy on ( γ 2 i -1 , γ 2 i ) , for i = 1 , ..., ( N -1) / 2 .

Theorem 3.7 shows that the algorithm that explore the entire interval of (0 , 1) to compute discountoptimal policies (Hordijk et al., 1985) may visit a number of subintervals that is impractical. We present a detailed proof in Appendix B. The proof relies on interpreting value functions as polynomials and using Lagrange interpolation polynomials to tune the instantaneous rewards to ensure that the value functions intersect at the given discount factors. Overall, our results in this section highlight the pitfalls of the existing approach to Blackwell optimality and the potential pathological behaviors of the value functions, even in simple MDP instances. We ameliorate this issue in the next section.

4 Introducing the Blackwell discount factor

This section introduces the notion of the Blackwell discount factor , which we use to reduce Blackwell optimality and average optimality to discounted optimality. This reduction leads to algorithms to compute Blackwell-optimal and average policies that are significantly simpler than the state-of-the-art. Intuitively, we need the following condition to reduce Blackwell optimality to discounted optimality: there must exist a discount factor γ bw ∈ [0 , 1) such that any γ -discount-optimal policy for γ > γ bw is also γ ′ -discount-optimal for any other γ ′ > γ bw . The following definition formalizes this intuition.

Definition 4.1. The Blackwell discount factor γ bw ∈ [0 , 1) is equal to γ bw = inf { γ ∈ [0 , 1) | Π ⋆ γ ′ = Π ⋆ bw , ∀ γ ′ ∈ ( γ, 1) } , where Π ⋆ bw is the set of Blackwell-optimal policies.

We establish the existence of a Blackwell discount factor in the next theorem.

Theorem 4.2. The Blackwell discount factor γ bw in Definition 4.1 exists in any finite MDP .

Proof. We show that there exists a discount factor γ ∈ [0 , 1) such that Π ⋆ γ ′ = Π ⋆ bw , ∀ γ ′ ∈ ( γ, 1) . Let ¯ γ defined as in Equation (3.1). We show ∀ γ ∈ [¯ γ, 1) , Π ⋆ γ = Π ⋆ bw . Let γ ′ ∈ (¯ γ, 1) and let π be a policy that is γ ′ -discount-optimal. By definition, we have v π γ ′ ,s ≥ v π ′ γ ′ ,s , ∀ π ′ ∈ Π , ∀ s ∈ S . Since γ ′ > ¯ γ , the map ϕ π,π ′ s does not change sign on [¯ γ, 1) . This shows that π is γ -discount-optimal for all γ ∈ (¯ γ, 1) . Therefore, π is Blackwell optimal, and any γ -discount-optimal policy is Blackwell optimal, for any γ ∈ (¯ γ, 1) , i.e., this shows Π ⋆ ¯ γ ⊂ Π ⋆ bw . The inclusion Π ⋆ bw ⊂ Π ⋆ ¯ γ follows from the definition of ¯ γ : if π is Blackwell-optimal but not discount-optimal for ¯ γ , then it must become discount-optimal for a larger γ ′ > ¯ γ , which is impossible since ¯ γ is the largest discount factors where the value functions of any two stationary policies can intersect.

̸

Difference from the existing definition. It is important to elaborate on the difference between Definition 3.1 (classical definition of Blackwell optimality) and Definition 4.1 (Blackwell discount factor). While the proof for the existence of γ bw is relatively concise, the distinction between γ bw and γ ( π ) has been utterly overlooked in the literature, where it is common to find statements that suggest that γ > γ ( π ) implies Blackwell optimality of all discount-optimal policies, e.g. in Dewanto and Gallagher (2021), Wang et al. (2023). To the best of our knowledge, we are the first to properly introduce the Blackwell discount factor γ bw , to show its sufficiency to compute Blackwell-optimal policies, to emphasize the shortcomings of the classical approach to Blackwell optimality, and to clarify the distinction between γ bw and γ ( π ) . In particular, in Definition 3.1, a Blackwell-optimal policy π is optimal for any γ ∈ [ γ ( π ) , 1) . However, for some γ ∈ [ γ ( π ) , 1) , there may be other optimal policies that are not Blackwell-optimal, as shown in Proposition 3.5. We show an MDP instance like this in Example 3.6, where γ bw = 3 / 4 but where γ ( a 1 ) = 1 / 2 , and a 1 is the only Blackwell-optimal policy. Hence in all generality, we may have γ ( π ) < γ bw , and γ ( π ) = γ bw . Note that the authors in (Dewanto and Gallagher, 2021, Dewanto et al., 2020) also introduce the notation ' γ bw ' but they use it to denote γ ( π ) .

Reduction to discounted optimality. If γ bw is known for a given MDP instance, it is straightforward to compute a Blackwell-optimal policy, by solving a discounted MDP with γ > γ bw . Therefore, the notion of Blackwell discount factor provides a method to reduce the criteria of Blackwell optimality and average optimality to the well-studied criterion of discounted optimality. As we have discussed before, efficient methods for solving discounted MDPs such as value iteration or linear programming have been extensively studied. These algorithms are much simpler than the two existing algorithms for computing Blackwell-optimal policies. Note that it is enough to compute an upper bound on γ bw . In particular, if we are able to show that γ bw < γ ′ for some γ ′ ∈ [0 , 1) , then following the definition of γ bw , we can compute a Blackwell-optimal policy by solving a discounted MDP with a discount factor γ = γ ′ . Therefore, in the rest of Section 4, we focus on obtaining an upper bound on γ bw .

4.1 Upper bound on γ bw

We now obtain an instance-dependent upper bound on γ bw , i.e., we construct a scalar η ( M ) ∈ (0 , 1) for each MDP instance M = ( S , A , r , P ) , such that γ bw < 1 -η ( M ) . Our main contribution in this section is Theorem 4.4, which gives a closed-form expression for η ( M ) as a function of the maximum bit-size of the data of the MDP instance M . We start by showing that it is impossible to obtain a bound on γ bw that is independent of r or P .

Proposition 4.3. For any η > 0 , there exists an MDP instance M = ( S , A , r , P ) with |S| = 2 , |A| = 2 and deterministic transitions, such that γ bw > 1 -η.

Proof. Let S = { s 1 , s 2 } , A = { a 1 , a 2 } . In state s 1 , action a 1 transitions to s 1 (with reward 0 ) and action a 2 transitions to s 2 (with reward -1 ). There is no action to choose in state s 2 which is absorbing with a reward ϵ > 0 . It is straightforward to check that a 2 is Blackwell optimal, with γ bw = (1 + ϵ ) -1 , so that γ bw can be chosen arbitrarily close to 1 by choosing small values for ϵ .

We show that Proposition 4.3 still holds even under the assumption that the MDP instances are weakly-communicating in Appendix C. Proposition 4.3 shows that an instance-dependent bound on γ bw must depend on the 'coarseness' of r and P . This suggests parametrizing our upper bound by the bit-sizes of the MDP instance. MDPs with finite bit-sizes parameters are the MDP instances that can be exactly encoded in a computer and practically solved by existing algorithms. We first recall the definitions pertaining to bit-size, necessary to describe the complexity of classical weakly-polynomial time algorithms like interior-point methods (section 4.6 in (Ben-Tal and Nemirovski, 2001)) and the ellipsoid method (Bland et al., 1981). The bit-size of r ∈ N is ⌊ log 2 ( r ) ⌋ , the number of bits necessary to represent r with standard binary encoding. The bit-size of a rational number is the sum of the bit-size of its numerator and its denominator. The maximum bit-size of an MDP instance is the maximum bit-size of any r sa and P sas ′ for ( s, a, s ′ ) ∈ S × A × S . Its total bit-size is the sum of the bit-sizes of the components of r and P . For instance, in the riverswim instance, the maximum bit-size of the reward is 14 , since the largest rewards are bounded by 10 4 in the terminal states. Our main theorem in this section provides a strict upper bound on γ bw as follows.

Theorem 4.4. Let M = ( S , A , r , P ) be an MDP instance with finite bit-size and let m ∈ N be the maximum bit-size of the instance M . Then we have γ bw < 1 -η ( M ) , with η ( M ) ∈ (0 , 1) defined as

Our proof uses ideas that are new in the MDP literature, such as the separation of algebraic numbers. We provide an outline of the proof below and defer the full statement to Appendix D.

In the first step of the proof, by carefully inspecting the proofs of Theorem 3.2 and of Theorem 4.2, we note that an upper bound for γ bw is ¯ γ , as defined in (3.1): ¯ γ = max π,π ′ ∈ Π ,s ∈S γ ( π, π ′ , s ) , where for π, π ′ ∈ Π and s ∈ S , γ ( π, π ′ , s ) is the largest discount factor γ in [0 , 1) for which ϕ π,π ′ s ( γ ) = 0 when ϕ π,π ′ s : γ ↦→ v π γ,s -v π ′ γ,s is not identically equal to 0 , and 0 otherwise. Therefore, we focus on obtaining an upper bound on γ ( π, π ′ , s ) for any two policies π, π ′ ∈ Π and any state s ∈ S .

In the second step, following Lemma 3.3, the value functions γ ↦→ v π s , γ ↦→ v π ′ s are rational functions, i.e., they are ratios of two polynomials. Therefore, we interpret ϕ π,π ′ s ( γ ) = 0 as a polynomial equation in γ , i.e., as p ( γ ) = 0 for a certain polynomial p . With this notation, γ ( π, π ′ , s ) ∈ [0 , 1) is a root of p . We show that γ = 1 is always a root of p , even though value functions are a priori not defined for γ = 1 . We then precisely characterize the degree N and the sum L of the absolute values of the coefficients of the polynomial p , depending on the MDP instance M .

Theorem 4.5. The polynomial p has degree N = 2 |S|1 . Moreover, m 2 |S| p has integral coefficients. The sum of the absolute values of the coefficients of m 2 |S| p is bounded by L = 2 ·|S|· r ∞ · m 2 |S| · 4 |S| .

In the third step, we lower-bound the distance between any two distinct roots of p . To do this, we rely on the following separation bounds of algebraic numbers .

Theorem 4.6 ((Rump, 1979)) . Let p be a polynomial of degree N with integer coefficients. Let L be the sum of the absolute values of its coefficients. The distance between any two distinct roots of p is strictly larger than η > 0 , with η = 2 N -N/ 2+2 ( L +1) -N .

Recall that γ ( π, π ′ , s ) and 1 are two always roots of p , with γ ( π, π ′ , s ) < 1 . Combining Theorem 4.5 with Theorem 4.6, we conclude that γ ( π, π ′ , s ) < 1 -η ( M ) for η ( M ) > 0 defined as in Theorem 4.4. Therefore, ¯ γ < 1 -η ( M ) , and γ bw < 1 -η ( M ) . This concludes our proof of Theorem 4.4.

Discussion. Using Theorem 4.4, we obtain the first reduction from Blackwell optimality to discounted optimality: solving a discounted MDP with γ ≥ 1 -η ( M ) returns a Blackwell-optimal policy. Blackwell optimality implies average optimality, so we also obtain the first reduction from average optimality to discounted optimality without any assumptions on the structure of the underlying Markov chains of the MDP. We also discuss the complexity results for computing a Blackwell-optimal policy using our reduction. Policy iteration returns a discounted optimal policy in O ( |S| 2 |A| 1 -γ log ( 1 1 -γ )) iterations (Scherrer, 2013), but it may be slow to converge when γ = 1 -η ( M ) as in Theorem 4.4, since η ( M ) may be close to 0 . Various algorithms exist to obtain convergence faster than O (1 / (1 -γ )) , such as accelerated value iteration (Goyal and Grand-Clément, 2023a) and Anderson acceleration (Zhang et al., 2020). However, note that ⌊ log 2 ( η ( M )) ⌋ , the bit-size of the scalar η ( M ) ,

is polynomial in the bit-size of the MDP instance M . Since discounted MDPs can be formulated as linear programs, which can be solved in polynomial-time in the input size of the MDP (Ye, 2011), we obtain a weakly-polynomial time algorithm for computing Blackwell-optimal policies. We present the proof of the following theorem in Appendix E.

Theorem 4.7. Let M = ( S , A , r , P ) be an MDP instance with total bit-size Q ( r , P ) ∈ N . Then we can compute a Blackwell-optimal policy in O ( |S| 5 |A| 2 Q ( r , P ) ) arithmetic operations.

Note that with Theorem 4.4 and Theorem 4.7, we have reduced the complex problem of computing a Blackwell optimal policy to a much simpler and well-studied problem: solving a linear program, which can be done in weakly-polynomial time. Potential improvements for our upper bound on γ bw are an important future direction: more precise separation bounds than Theorem 4.6 could be obtained for the specific polynomial p appearing in the proof of Theorem 4.4, or for a specific MDP instances, e.g. ergodic or unichain MDPs. Going beyond the case of finite sets of states and actions is interesting but this may be difficult, as in both cases there may not exist a Blackwell optimal policy anymore (Chitashvili, 1976, Maitra, 1965).

4.2 The case of robust MDPs

In practice, the value function v π γ may be very sensitive to the values of the transition probabilities P . To emphasize this dependence, in this section we note v π, P γ for the value function associated with a policy π and a transition probability P , defined similarly as in Section 2. Robust MDPs (RMDPs) ameliorate this issue by considering an uncertainty set U , which can be seen as a plausible region for the transition probabilities P ∈ U . We focus on the case of sa-rectangular MDPs (Iyengar, 2005), where U = × ( s,a ) ∈S×A U sa for U sa ⊆ ∆( S ) . The worst-case value function v π, U γ ∈ R S of a policy π is defined as v π, U γ,s = min P ∈U v π, P γ,s , ∀ s ∈ S . In discounted RMDPs, the goal is to compute a robust discounted optimal policy, defined as follows.

Definition 4.8. Given γ ∈ [0 , 1) , a policy π ∈ Π is robust γ -discount-optimal if v π, U γ,s ≥ v π ′ , U γ,s , ∀ π ′ ∈ Π , ∀ s ∈ S . We write Π ⋆ γ, rob the set of robust γ -discount-optimal policies.

Robust Blackwell optimality is studied in (Goyal and Grand-Clément, 2023b, Tewari and Bartlett, 2007), to address the sensitivity of the robust value functions as regards the discount factors. Its connection to average reward RMDPs is discussed in (Tewari and Bartlett, 2007, Wang et al., 2023).

Definition 4.9. A policy π ∈ Π is robust Blackwell-optimal if there exists γ ∈ [0 , 1) , such that π ∈ Π ⋆ γ ′ , r , ∀ γ ′ ∈ [ γ, 1) . We call Π ⋆ bw , r the set of robust Blackwell-optimal policies.

(Goyal and Grand-Clément, 2023b) shows the existence of a Blackwell-optimal policy for RMDPs, under the condition that U is sa-rectangular and has finitely many extreme points. This is the case for popular polyhedral uncertainty sets, e.g., when U sa is based on the ℓ p distance, for p ∈ { 1 , ∞} (Givan et al., 1997, Ho et al., 2018, Iyengar, 2005), for some estimated kernel P 0 and some radius α sa > 0 :

Definition 4.10. We define the robust Blackwell discount factor γ bw , r ∈ [0 , 1) as γ bw , r = inf { γ ∈ [0 , 1) | Π ⋆ γ ′ , r = Π ⋆ bw , r , ∀ γ ′ ∈ ( γ, 1) } .

We provide a detailed proof of the existence of the robust Blackwell discount factor in Appendix F. The proof strategy is the same as for the existence of the Blackwell discount factor for MDPs. We can obtain the same upper bound on γ bw , r , by studying the values of γ for which γ ↦→ v π, P γ,s -v π ′ , P ′ γ,s cancels, for any two policies π, π ′ ∈ Π and any two extreme points P , P ′ of U . Writing γ ( π, π ′ , s, P , P ′ ) for the largest zero in [0 , 1) of the function γ ↦→ v π, P γ,s -v π ′ , P ′ γ,s if it is not identically equal to zero, or γ ( π, π ′ , s, P , P ′ ) = 0 otherwise, an upper bound on γ bw , r for RMDPs can be computed as ¯ γ r , defined as ¯ γ r = max π,π ′ ∈ Π ,s ∈S max P , P ′ ∈U ext γ ( π, π ′ , s, P , P ′ ) with U ext the set of extreme points of U . This leads to the following theorem.

Theorem 4.11. Let M = ( S , A , r , P 0 ) be an MDP instance with maximum bit-size m ∈ N . Assume that U is sa-rectangular, where for each ( s, a ) ∈ S × A , U sa is constructed as in (4.1) based on ℓ 1 or ℓ ∞ distance, and with the scalars ( α sa ) s,a of maximum bit-size m . Then γ bw , r ≤ 1 -η ( M ) , with η ( M ) defined as in Theorem 4.4 with m ′ = 2 m instead of m .

Based on Theorem 4.11, we obtain the first reduction from robust Blackwell optimality to robust discounted optimality. Since discounted RMDPs can be solved with value iteration or policy iteration, we provide the first algorithms to compute a robust Blackwell-optimal policy for RMDPs with sa-rectangular uncertainty, when the uncertainty set is based on the ℓ 1 or the ℓ ∞ distance. Note that there is no known convex (or linear) formulation for RMDPs (Grand-Clément and Petrik, 2022), so we are not able to provide a complexity statement akin to Theorem 4.7.

5 Conclusion

We highlight the shortcomings of the existing approach to Blackwell optimality and we introduce the Blackwell discount factor to ameliorate this issue. We provide an upper bound for MDPs and RMDPs in all generality, parametrized by the bit-sizes of the instances. Any progress in solving discounted MDPs, one of the most active research directions in RL, can be combined with our results to obtain new algorithms for computing average- and Blackwell-optimal policies. Our work also opens new research avenues for MDPs and RMDPs: the proof techniques for our bound on γ bw and γ bw , r , based on the separation of algebraic numbers, are novel and they could be tightened for specific instances or different optimality criteria, such as bias optimality or n -discount optimality. The notion of approximate Blackwell optimality as well as the existence of the robust Blackwell discount factor for other uncertainty sets, e.g., s-rectangular or non-polyhedral sa-rectangular uncertainty sets, or for distributionally robust MDPs, are also interesting directions of research.

Funding. J. Grand-Clément is supported by the Agence Nationale de la Recherche [Grant 11LABX-0047] and by Hi! Paris. M. Petrik's work was supported, in part, by NSF grants 2144601 and 1815275.

References

Akian and S. Gaubert. Policy iteration for perfect information stochastic mean payoff games with bounded first return times is strongly polynomial. arXiv preprint arXiv:1310.4953 , 2013.

Akian, S. Gaubert, J. Grand-Clément, and J. Guillaud. The operator approach to entropy games. Theory of Computing Systems , 63(5):1089-1130, 2019.

Baxter and P. L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research , 15:319-350, 2001.

Behzadian, M. Petrik, and C. P. Ho. Fast algorithms for l ∞ -constrained s-rectangular robust MDPs. Advances in Neural Information Processing Systems , 34, 2021.

Ben-Tal and A. Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications . SIAM, 2001.

Bhatnagar, M. Ghavamzadeh, M. Lee, and R. S. Sutton. Incremental natural actor-critic algorithms. Advances in neural information processing systems , 20, 2007.

Blackwell. Discrete dynamic programming. The Annals of Mathematical Statistics , pages 719-726, 1962.

G. Bland, D. Goldfarb, and M. J. Todd. The ellipsoid method: A survey. Operations research , 29 (6):1039-1091, 1981.

Boone and B. Gaujal. Identification of Blackwell policies for deterministic MDPs. In 23ème congrès annuel de la Société Française de Recherche Opérationnelle et d'Aide à la Décision , 2022.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540 , 2016.

Y. Chitashvili. A controlled finite markov chain with an arbitrary set of decisions. Theory of Probability & Its Applications , 20(4):839-847, 1976.

Delage and S. Mannor. Percentile optimization for Markov decision processes with parameter uncertainty. Operations research , 58(1):203-213, 2010.

Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai. Deep direct reinforcement learning for financial signal representation and trading. IEEE transactions on neural networks and learning systems , 28(3): 653-664, 2016.

Dewanto and M. Gallagher. Examining average and discounted reward optimality criteria in reinforcement learning. arXiv preprint arXiv:2107.01348 , 2021.

Dewanto, G. Dunn, A. Eshragh, M. Gallagher, and F. Roosta. Average-reward model-free reinforcement learning: a systematic review and literature mapping. arXiv preprint arXiv:2010.08920 , 2020.

Dong, Y. Wang, X. Chen, and L. Wang. Q-learning with UCB exploration is sample efficient for infinite-horizon MDP. arXiv preprint arXiv:1901.09311 , 2019.

A. Feinberg and A. Shwartz. Handbook of Markov decision processes: methods and applications , volume 40. Springer Science & Business Media, 2012.

Friedmann. An exponential lower bound for the latest deterministic strategy iteration algorithms. Logical Methods in Computer Science , 7, 2011.

Garcia, L. Steimle, W. Marrero, and J. B. Sussman. Interpretable policies and the price of interpretability in hypertension treatment planning. Optimization Online , pages 1-46, 2021.

Givan, S. Leach, and T. Dean. Bounded parameter Markov decision processes. In European Conference on Planning , pages 234-246. Springer, 1997.

Goh, M. Bayati, S. A. Zenios, S. Singh, and D. Moore. Data uncertainty in Markov chains: Application to cost-effectiveness analyses of medical innovations. Operations Research , 66(3): 697-715, 2018.

Gottesman, F. Johansson, M. Komorowski, A. Faisal, D. Sontag, F. Doshi-Velez, and L. A. Celi. Guidelines for reinforcement learning in healthcare. Nature medicine , 25(1):16-18, 2019.

Goyal and J. Grand-Clément. A first-order approach to accelerated value iteration. Operations Research , 71(2):517-535, 2023a.

Goyal and J. Grand-Clément. Robust Markov decision processes: Beyond rectangularity. Mathematics of Operations Research , 48(1):203-226, 2023b.

Grand-Clément and M. Petrik. On the convex formulations of robust Markov decision processes. arXiv preprint arXiv:2209.10187 , 2022.

Grand-Clément, C. W. Chan, V. Goyal, and G. Escobar. Robustness of proactive intensive care unit transfer policies. Operations Research , 71(5):1653-1688, 2022.

P. Ho, M. Petrik, and W. Wiesemann. Fast Bellman updates for robust MDPs. In International Conference on Machine Learning , pages 1979-1988. PMLR, 2018.

P. Ho, M. Petrik, and W. Wiesemann. Partial policy iteration for l 1 -robust Markov decision processes. Journal of Machine Learning Research , 22(275):1-46, 2021.

Hordijk, R. Dekker, and L. C. M. Kallenberg. Sensitivity-analysis in discounted Markovian decision problems. Operations-Research-Spektrum , 7(3):143-151, 1985.

A. Horn and C. R. Johnson. Matrix analysis . Cambridge university press, 2012.

Hu, C. W. Chan, J. R. Zubizarreta, and G. J. Escobar. An examination of early transfers to the icu based on a physiologic risk score. Manufacturing & Service Operations Management , 20(3): 531-549, 2018.

Huang. Complexity Estimates and Reductions to Discounting for Total and Average-Reward Markov Decision Processes and Stochastic Games . PhD thesis, State University of New York at Stony Brook, 2016.

Iwaki and M. Asada. Implicit incremental natural actor critic algorithm. Neural Networks , 109: 103-112, 2019.

N. Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Jin and A. Sidford. Efficiently solving MDPs with stochastic mirror descent. In International Conference on Machine Learning , pages 4890-4900. PMLR, 2020.

Jin and A. Sidford. Towards tight bounds on the sample complexity of average-reward MDPs. In International Conference on Machine Learning , pages 5055-5064. PMLR, 2021.

Kallenberg. Classification problems in mdps. Markov processes and controlled Markov chains , pages 151-165, 2002.

Maitra. Dynamic programming for countable state systems. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 241-248, 1965.

Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

J. Neumann, G. D. Sanders, L. B. Russell, J. E. Siegel, and T. G. Ganiats. Cost-effectiveness in health and medicine . Oxford University Press, 2016.

O'Sullivan and A. F. Veinott Jr. Polynomial-time computation of strong and n-present-value optimal policies in Markov decision chains. Mathematics of Operations Research , 42(3):577-598, 2017.

S. Perotto and L. Vercouter. Tuning the discount factor in order to reach average optimality on deterministic MDPs. In International Conference on Innovative Techniques and Applications of Artificial Intelligence , pages 92-105. Springer, 2018.

L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley and Sons, 2014.

M. Ross. Non-discounted denumerable Markovian decision models. The Annals of Mathematical Statistics , 39(2):412-423, 1968.

M. Rump. Polynomial minimum root separation. Mathematics of Computation , 33(145):327-336, 1979.

Scherrer. Improved and generalized upper bounds on the complexity of policy iteration. Advances in Neural Information Processing Systems , 26, 2013.

Schneckenreither. Average reward adjusted discounted reinforcement learning: Near-Blackwelloptimal policies for real-world applications. arXiv preprint arXiv:2004.00857 , 2020.

N. Steimle, D. L. Kaufman, and B. T. Denton. Multi-model markov decision processes. IISE Transactions , 53(10):1124-1139, 2021.

Tang, M. Rowland, R. Munos, and M. Valko. Taylor expansion of discount factors. In International Conference on Machine Learning , pages 10130-10140. PMLR, 2021.

Tewari and P. L. Bartlett. Bounded parameter Markov decision processes with average reward criterion. In International Conference on Computational Learning Theory , pages 263-277. Springer, 2007.

N. Tsitsiklis. Np-Hardness of checking the unichain condition in average cost MDPs. Operations research letters , 35(3):319-323, 2007.

Wang, M. Wang, and L. F. Yang. Near sample-optimal reduction-based policy learning for average reward MDP. arXiv preprint arXiv:2212.00603 , 2022.

Wang. Primal-dual \ π learning: Sample complexity and sublinear run time for ergodic Markov decision problems. arXiv preprint arXiv:1710.06100 , 2017.

Wang, A. Velasquez, G. Atia, A. Prater-Bennette, and S. Zou. Robust average-reward Markov decision processes. arXiv preprint arXiv:2301.00858 , 2023.

Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Yang, Y. Gao, B. An, H. Wang, and X. Chen. Efficient average reward reinforcement learning using constant shifting values. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Ye. The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. Mathematics of Operations Research , 36(4):593-603, 2011.

Zhang, B. O'Donoghue, and S. Boyd. Globally convergent type-I Anderson acceleration for nonsmooth fixed-point iterations. SIAM Journal on Optimization , 30(4):3170-3197, 2020.

Zwick and M. Paterson. The complexity of mean payoff games on graphs. Theoretical Computer Science , 158(1-2):343-359, 1996.

A Unichain instance for Proposition 3.5

We can extend Example 3.5 to a unichain MDP as follows: we add a transition from state 7 to state 0 , with a reward of 0 . We also add three intermediate states from 0 to 7 for action a 1 , so that it takes as many periods to reach state 7 from state 0 for the three actions a 1 , a 2 , a 3 . Note that this new MDP is unichain. We represent it in Figure 3a. Additionally, for this new MDP instance, we have v a 1 γ = 1 / (1 -γ 5 ) , v a 2 γ = ( r 1 γ + r 2 γ 2 ) / (1 -γ 5 ) , v a 3 γ = ( r 4 γ + r 5 γ 2 ) / (1 -γ 5 ) , which are the same expressions as in Example 3.5, up to the common denominator (1 -γ 5 ) -1 . Therefore, we have proved that the same conclusion as Proposition 3.4 holds for unichain MDPs.

B Proof of Theorem 3.7

Proof. Consider the following MDP instance, represented in Figure 2a. The initial state is state 0 , where there are two actions to be chosen, a 1 or a 2 . Action a 1 yields an instantaneous reward of 1 and then the decision maker transitions to the absorbing state N , where there is a reward of 0 . Otherwise, choosing action a 2 yields an instantaneous reward r 0 and takes the decision maker through a deterministic sequence of states 1 , ..., N -1 with rewards r 1 , ..., r N -1 , before transitioning to state N . For a given γ ∈ [0 , 1) , the closed-form expressions for the value functions v a 1 γ , v a 2 γ are v a 1 γ = 1 and v a 2 γ = ∑ N -1 t =0 r t γ t .

Note that γ ↦→ v a 2 γ is a polynomial of degree N -1 . Using Lagrange interpolation polynomials (section 0.9.11, (Horn and Johnson, 2012)), we can find coefficients r 0 , ..., r N -1 such that γ ↦→ v a 1 γ is equal to 1 for all N -1 discount factors γ 1 , ..., γ N -1 and equal to 0 . 9 at γ 0 = 0 . The value function v a 2 γ resulting from this construction is highlighted in Figure 2b for N = 5 and ( γ 0 , γ 1 , γ 2 , γ 3 , γ 4 , γ 5 ) = (0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 . 0) . Let us note q : γ ↦→ v a 1 γ -v a 2 γ . Our choice of the rewards ensures that q is a polynomial of degree N -1 , with q (0) > 0 , and q ( γ ) = 0 for γ ∈ { γ 1 , ..., γ N -1 } . Because γ ↦→ q ( γ ) -1 is a polynomial of degree N -1 with N -1 different real roots, it changes signs at every root. This shows that γ ↦→ v a 1 γ -v a 2 γ is positive on ( γ 0 , γ 1 ) , negative on ( γ 1 , γ 2 ) , then positive on ( γ 2 , γ 3 ) , etc.. Action a 1 is optimal on ( γ N -1 , γ N ) = ( γ N -1 , 1) because N is odd. This concludes the proof of Theorem 3.7.

1

a

2

/uni22EF

1

r

1

r

N

-1

r

0

0

N

-1

N

a

1

(a)

0.0

0.2

0.4

0.6

0.8

1.0

γ

0.90

0.92

0.94

0.96

0.98

1.00

Value functions

v a

1

γ

v a

2

γ

(b)

C Weakly-communicating instances for Proposition 4.3

Consider the MDP instance from the proof of Proposition 4.3. We now add a deterministic transition from state s 2 to state s 1 , with a reward of 0 for action a 1 and a reward of ϵ for action a 2 . The new MDP instance is represented in Figure 3b.

0

7

1

a

1

4

5

6

0

1

2

3

0

r

1

0

a

2

a

3

0

r

2

r

4

r

5

8

9

10

0

0

0

0

-1

s

1

s

2

a

2

a

1

0

/uni03F5

a

2

a

1

0

First, this MDP instance is weakly-communicating since { s 1 , s 2 } is strongly connected under policy a 2 . In this new MDP instance, we still have v a 1 γ = 0 but v a 2 γ = ( -1 + ϵγ ) / (1 -γ ) . Hence a 2 is Blackwell optimal when γ ≥ 1 /ϵ . By choosing ϵ larger than 1 and ϵ → 1 , we obtain γ bw → 1 . This shows that we can extend Proposition 4.3 to weakly-communicating MDPs.

D Proof of Theorem 4.4

In this appendix, we provide the proof for Theorem 4.4. As noted in Section 4.1, to bound γ bw , it is enough to obtain an upper bound on γ ( π, π ′ , s ) for any π, π ′ ∈ Π and s ∈ S such that γ ↦→ v π γ,s -v π ′ γ,s is not identically equal to 0 , since γ bw ≤ max π,π ′ ∈ Π ,s ∈S γ ( π, π ′ , s ) . Since m is the maximum bit-size of the input data, we can write, for any ( s, a, s ′ ) ∈ S × A × S , P sas ′ = n sas ′ /m , for n sas ′ ∈ N , n sas ′ ≤ m , and r sa = q sa /m, | q sa | ≤ r ∞ . Examples of MDPs with finite bit-sizes include any real instances used for applications where the transition probabilities are estimated as empirical frequencies from some data, e.g. examining patients' transfers in hospitals as in (Hu et al., 2018) and (Grand-Clément et al., 2022), MDPs for hypertension treatment (Garcia et al., 2021), diabetes management (Steimle et al., 2021) and cancer detection (Goh et al., 2018), as well as the machine maintenance studied in (Wiesemann et al., 2013) and (Delage and Mannor, 2010). We now proceed to proving Theorem 4.4.

Step 1. We start by studying in more detail the properties of the value functions. The following lemma follows directly from Cramer's rule, as explained in Section 3.

Lemma D.1. We have

with M ( γ, s, π ) the matrix formed by replacing the s -th column of I -γ P π by the vector r π .

From Lemma D.1, we have

for n ( γ, s, π ) = det ( M ( γ, s, π )) and d ( γ, π ) = det ( I -γ P π ) . We choose the letter n for nominator and the letter d for denominator .

Note that γ ↦→ n ( γ, s, π ) is a polynomial of degree at most |S|1 , while γ ↦→ d ( γ, π ) is a polynomial of degree at most |S| .

We have, by definition,

Therefore, v π γ,s -v π ′ γ,s = 0 for γ ∈ [0 , 1) implies that γ is a root of the following polynomial equation in γ :

for p the polynomial defined as

Step 2. We now study the properties of the polynomial p . Note that it is straightforward that p is a polynomial of degree N = 2 |S| 1 . We first study the properties of the polynomial γ ↦→ d ( π, γ ) . We have the following lemma.

Lemma D.2. We have and d (1 , π ) = 0 , ∀ π ∈ Π .

Proof of Lemma D.2. This lemma follows from the relation between the determinant of a matrix and its eigenvalues, through the characteristic polynomial:

with α λ the algebraic multiplicity of the (potentially complex) eigenvalue λ in the spectrum Sp ( P π ) of P π . Since P π is the transition matrix of a Markov chain, we know that the modulus of any eigenvalue λ of P π is smaller or equal to 1 . This shows that d ( γ, π ) > 0 , ∀ γ ∈ [0 , 1) , ∀ π ∈ Π . To show d (1 , π ) = 0 , we simply note that 1 ∈ Sp ( P π ) since P π is the transition matrix of a Markov chain.

From Lemma D.2 and the definition of p as in (D.3), it is straightforward that p (1) = 0

Lemma D.3. γ = 1 is a root of p

. .

We now bound the sum of the absolute values of the coefficients of p . We have the following theorem.

Theorem D.4. The polynomial m 2 |S| · p has integral coefficients, potentially negative. The sum of the absolute values of the coefficients of m 2 |S| p is bounded by

Theorem D.4 is based on the following three propositions. We note C k ℓ the binomial coefficient defined as C k ℓ = ℓ ! /k !( ℓ -k )! .

Proposition D.5. For any π ∈ Π , the function γ ↦→ d ( π, γ ) is a polynomial of degree |S| . Moreover, γ ↦→ m |S| · d ( π, γ ) is a polynomial with integral coefficients (potentially negative), and the absolute value of its coefficient of degree k is bounded by m |S| C k |S| .

Therefore, the sum of the absolute values of the coefficients of γ ↦→ m |S| · d ( π, γ ) is upper bounded by

Proposition D.6. For any policy π ∈ Π and any state s ∈ S , the function γ ↦→ n ( γ, s, π ) is a polynomial of degree |S| 1 . Moreover, γ ↦→ m |S| · n ( γ, s, π ) is a polynomial with integral coefficients (potentially negative), and the absolute value of its coefficient of degree k is bounded by m |S| · |S| · r ∞ · C k |S|1 · 2 .

Therefore, the sum of the absolute values of the coefficients of γ ↦→ m |S| · n ( γ, s, π ) is upper bounded by

Proposition D.7. Let P = ∑ n i =0 a i X i , Q = ∑ m j =0 b j X j . Then PQ = ∑ n + m k =0 c k X k , c k = ∑ i,j ; i + j = k a i b j . Additionally, suppose that ∑ n i =0 | a i | ≤ L P , ∑ m j =0 | b j | ≤ L Q . Then

Combining Proposition D.5, Proposition D.6 and Proposition D.7 with the definition of the polynomial p in (D.3) yields Theorem D.4.

To conclude Step 2 of our proof, let us prove Proposition D.5 and Proposition D.6. Proposition D.7 simply follows from the multiplication rule for polynomials.

Proof of Proposition D.5. By definition,

where M ↦→ a k ( M ) is the ( |S| k ) -th coefficient of the characteristic polynomial of a matrix M . By definition, a k ( M ) is the sum of all the principal minors of size k of M (section 0.7.1, (Horn and Johnson, 2012)). This first shows that a k ( γ P π ) = γ k a k ( P π ) , and therefore, that

We will show that

Let g be a principal minor of P π of size k . By definition, g is the determinant of a submatrix M of size k of P π , obtained by deleting rows and columns with the same indices: g = det( M ) . For any matrix square M , we always have det( M ) = det( M ⊤ ) . Now Hadamard's inequality shows that det( M ⊤ ) ≤ ∏ k i =1 ∥ Col i ( M ⊤ ) ∥ 2 , with Col i ( M ⊤ ) the i -th column of M ⊤ , and therefore we have det( M ⊤ ) ≤ ∏ k i =1 ∥ Col i ( M ⊤ ) ∥ 1 . Note that the columns of M ⊤ have ℓ 1 -norm smaller than 1 , since P π is a stochastic matrix, and M is a submatrix of P π . Therefore, g ≤ 1 . Because there are C k n possible principal minors of size k of P π , we have a k ( P π ) ≤ C k n , ∀ k = 1 , ..., n.

Of course, we may have a k ( P π ) / ∈ Z . However, for any principal minor g = det( M ) of P π , we have, by definition the determinant,

where ε ( σ ) is the signature of the permutation σ and S k is the symmetric group, i.e., the group of all permutations of { 1 , ..., k } . This shows, by definition m as the maximum bit-size of the input data, that m |S| det( M ) ∈ Z , and therefore that m |S| a k ( P π ) ∈ Z and that m |S| a k ( P π ) ≤ m |S| C k |S| .

Proof of Proposition D.6. Using Laplace cofactor expansions (section 0.3.1, (Horn and Johnson, 2012)), we have that n ( γ, s, π ) is equal to

where ( I -γ P π ) S\{ s ′ }×S\{ s } is the matrix obtained from I -γ P π by removing the s -th column and the s ′ -th row.

Note that γ ↦→ det ( ( I -γ P π ) S\{ s ′ }×S\{ s } ) is a polynomial of degree |S| 1 in γ . Similarly as for the proof of Proposition D.5, γ ↦→ m |S| n ( γ, s, π ) is a polynomial of degree |S| 1 with integral coefficients.

Let us consider I \{ s ′ ,s } the matrix of dimension ( |S| 1) × ( |S| 1) , obtained by removing the s -th column and the s ′ -th row from the identity matrix of dimension |S| , and let us call E s ′ the matrix of dimension ( |S| 1) × ( |S| 1) , where all rows are 0 ⊤ , except the s -th row, equal to e ⊤ s ′ .

Then det ( ( I -γ P π ) S\{ s ′ }×S\{ s } ) is equal to

and therefore is equal to

We notice that I \{ s ′ ,s } + E s ′ is a matrix whose rows are exactly the rows of the identity matrix of R |S|1 , up to a certain permutation σ ∈ S |S|1 . Let P σ ∈ R ( |S|1) × ( |S|1) the permutation matrix defined as P ij = 1 if σ ( j ) = i and 0 otherwise. Then for any matrix M , we have det( P σ M ) = det( P σ ) det( M ) = ε ( σ ) det( M ) , with ε ( σ ) the signature of the permutation σ . Since we always

where similarly as for the proof of Proposition D.5, a k ( M ) is the k -th coefficient of the characteristic polynomial of a matrix M , i.e., a k ( M ) is equal to the sum of all the principal minors of M of dimension k × k . Let

Note that ( P π ) S\{ s ′ }×S\{ s } is a substochastic matrix, i.e., it has non-negative entries and the sum of the entries of each row is smaller or equal to 1 . Note that M differs from ( γ P π ) S\{ s ′ }×S\{ s } only at the coefficient of index ( s, s ′ ) . Using Hadamard's inequality, we find that that

We conclude by combining Equation (D.5) with Equation (D.4).

Step 3. We now lower bound the distance between any two roots of p by a scalar η > 0 . Since we know that for γ ( π, π ′ , s ) ∈ [0 , 1) and 1 are two roots of P , this will show that γ ( π, π ′ , s ) < 1 -η .

Our proof is based on the following theorem.

Theorem D.8 ((Rump, 1979)) . Let p be a polynomial of degree N with integer coefficients, possibly with multiple roots. Let L be the sum of the absolute values of its coefficients. Then the distance between any two distinct roots of p is strictly larger

Recall that both γ ( π, π ′ , s ) ∈ [0 , 1) and 1 are roots of the polynomial p . Therefore, we can combine Theorem D.8 with Theorem D.4 to obtain γ ( π, π ′ , s ) < 1 -η ( M ) , with

This concludes the proof of Theorem 4.4.

Remark D.9. Note that (Akian et al., 2019) use Theorem D.8 to obtain a lower bound on the average rewards of any two different policies, in the setting of two-player stochastic games.

Remark D.10. Theorem 1 in (Rump, 1979) provides a separation bound in the case where the polynomial p has complex coefficients. Unfortunately, the separation bound from Theorem 1 in (Rump, 1979) is not directly usable here, because it depends on the discriminant D ( p ) of the polynomial p , a quantity that is hard to lower-bound (in all generality). We decide to use the bound from Theorem 3 in (Rump, 1979) because it does not depend on D ( p ) but directly on the ℓ 1 -norm of p and of the degree of p , which can be computed in closed-form and can be bounded as in Proposition D.6 and Proposition D.5.

with

E Proof of Theorem 4.7

Proof of Theorem 4.7. Following table 4 in (Ye, 2011), we know that interior-point methods for the linear programming formulation of MDPs return an optimal policy in O ( |S| 3 |A| 2 ( Q ( r , P , γ )) ) arithmetic operations, with Q ( r , P , γ ) equal to the total bit-size of the MDP instance, i.e., the sum of the bit-sizes of all instantaneous rewards, transition probabilities, and the discount factor. By choosing γ = 1 -η ( M ) and noticing that log( η ( M )) = O ( |S| log( r ∞ ) + |S| 2 log( m ) ) = O ( |S| 2 log( m ) ) , we see that interior-point methods for the linear programming formulation of MDPs return an optimal policy in O ( |S| 5 |A| 2 ( Q ( r , P )) ) , where Q ( r , P ) is the total bit-size of MDP instance.

F Proof of Section 4.2

Proof of the existence of γ bw , r . Let

where γ ( π, π ′ , s, P , P ′ ) is the largest zero of the function γ ↦→ v π, P γ,s -v π ′ , P ′ γ,s if it is not identically equal to zero, or γ ( π, π ′ , s, P , P ′ ) = 0 otherwise. Recall that U ext is the (finite) set of extreme points of U . We will show that Π ⋆ γ, r = Π ⋆ bw , r , ∀ γ > ¯ γ r . Let π be a robust discount-optimal policy for some γ > ¯ γ r . We will prove that π is a Blackwell-optimal policy. Since π is robust γ -discount-optimal, we have

By definition v π, U γ,s = min P ∈U v π, P γ,s , ∀ s ∈ S . From (Iyengar, 2005), we know that the arg min in min P ∈U v π, P γ,s is attained at an extreme point of U . Therefore, by definition of ¯ γ r , the function γ ↦→ v π, U γ,s -v π ′ , U γ,s cannot be equal to 0 on (¯ γ r , 1) , and therefore it does not change sign, since it is a continuous function. This shows that for all γ > ¯ γ r , we have

This shows the existence of the robust Blackwell discount factor γ bw , r and that γ bw , r < ¯ γ r .

Proof of Theorem 4.11. We start by showing the following lemma.

Lemma F.1. Let M = ( S , A , r , P 0 ) be an MDP instance with maximum bit-size m ∈ N . Assume that U is sa-rectangular, where for each ( s, a ) ∈ S × A , U sa is constructed as in (4.1) , with the scalars ( α sa ) s,a of maximum bit-size m .

Then the maximum bit-size complexity to describe the transition probabilities associated with the extreme points of U sa is m ′ for p = ∞ and 2 m ′ for p = 1 .

Proof of Lemma F.1. In the proof of this lemma, we use the fact that the worst-case kernel P ⋆ of a policy π can be chosen as the arg min of the optimization problem min p ∈U sπ ( s ) p ⊤ v π, U γ , where v π, U γ is the worst-case value function of π . In particular, let v ∈ R S .

The case p = ∞ . In this case, there exists a sorting solution to min p ∈U sa p ⊤ v for any ( s, a ) ∈ S ×A and any v ∈ R S , by sorting v , see for instance proposition 3 in (Goh et al., 2018), equation (9) in (Givan et al., 1997), or appendix C in (Behzadian et al., 2021). In particular, let ( s, a ) ∈ S × A and define σ the permutation of S such that v σ (1) ≤ ... ≤ v σ ( |S| ) , and define i as the smaller integer in { 1 , ..., |S|} such that

Then a solution to min p ∈U sa p ⊤ v is p σ ( s ′ ) = P 0 saσ ( s ′ ) + α sa if s ′ < i , p σ ( s ′ ) = P 0 saσ ( s ′ ) -α sa if s ′ > i , and

This closed-form shows that for any vector v ∈ R S , a solution of min p ∈U sa p ⊤ v can be found as a vector with rational entries with a denominator of at most m .

̸

The case p = 1 . In this case, one can show that the optimization problem min p ∈U sa p ⊤ v can be formulated as a linear program. Therefore, there exists an optimal basic feasible solution p which has the following form by lemma 5.4 and lemma 5.5 in (Ho et al., 2021). There exist j 1 , j 2 ∈ S such that j 1 = j 2 and for each i ∈ I = S \ { j 1 , j 2 } :

Then, in order for p ∈ U sa we need the following equalities to hold

Combining the equalities above yields that

Because the right-hand side of the equation above is a sum of rational numbers with a denominator of at most m , p j 1 is also rational with a denominator at most 2 m . Using an analogous argument for p j 2 , we get that there exists an optimal solution that is rational with a denominator of at most 2 m .

Theorem 4.11 then follows by applying Theorem 4.4 with on the MDP instance ( S , A , r , P ′ ) with P ′ an extreme point of U . Lemma F.1 exactly describes the maximum bit-size of any transition P ′ sas ′ for ( s, a, s ′ ) ∈ S × A × S in the case of sa-rectangular uncertainty set based on ℓ 1 -distance or ℓ ∞ -distance as in (4.1). This concludes the proof of Theorem 4.11.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Blackwell discount factor MDPs.\nThe authors show that when the discount factor is larger than the Blackwell discount factor $\\gamma_{bw}$, all discount-optimal policies become Blackwell- and average-optimal, and derive a general upper bound on $\\gamma_{bw}$. The authors claim to provide the first reduction from average and Blackwell optimality to discounted optimality via polynomial-time algorithms. \n\nI am not much familiar with the literature regarding Blackwell optimality criterion and thus might not fully understand the significant of the results. I would be happy that the authors would highlight for significant of results in compression to the known results in for discounted value criterion and averaged value criterion.\n\nSTRENGTHS:\n1)\tThe assumptions and contributions are clearly stated.\n2)\tIt seems that the presented results improve the known results in the study of Blackwell optimality criterion, remove commonly used assumptions and provide efficient algorithms.  \n3)\tThe application of algebraic techniques to RL might be useful in other settings.\n\nWEAKNESSES:\n1)\tNo concrete application or motivation for the work is given.\n2)\tIt is unclear to me whether the new definition of Blackwell optimality via the Blackwell discount factor proposed in the paper is better/ outperform/ more useable than the discounted value criterion in standard RL applications.\n3)\tThe authors claim throughout the paper that previous literature repeatedly mentioned the discussed problem as important, but not clearly mentioned why.\n4)\tThe focus of the paper is mainly about previous works (and their limitations),  instead of the new approach (they get to it only in page 6 out of 9!). More elaborated proof sketches would benefit the reader.\n\nQUESTIONS:\n1)\tMay you please provide concrete motivation for the use in Blackwell optimality criterion and/or a concrete application? \n2)\tErgodicity and aperiodicity properties are standard assumption in RL even in simple planning task. It is unclear to me how can you remove them and why it is important. In most application the assumptions are reasonable.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the a Blackwell optimal policy can be obtained from an average-optimal policy by introducing the Blackwell discount factor $\\gamma_{bw}$. The author further showed that the discount factor can be upper bounded by the bit-size of an MDP instance.\n\nSTRENGTHS:\n1. The paper is well written and organized.  The insights behind the main results are well presented.\n2. The introducing of the Blackwell discount factor is interesting, it may lead to efficient algorithms in future studies.\n3. An upper bound for $\\gamma_{bw}$ is provided.\n\nWEAKNESSES:\n1. Providing the upper bound of $\\gamma_{bw}$ is not enough, still we don’t have a efficient algorithm for finding the Blackwell policy. Also I am afraid for calculating it needs have sufficient information of the MDP structure.\n2. Proposition 4.3 is only for the deterministic transitions with $S=2, A=2.$\n3. The $ \\eta$ in Theorem is quite close to zero, which means the upper bound for $\\gamma_{bw}$ is almost 1. \n4. No simulation results are provided, it is better the show the advantage of the proposed approach with even some simple simulations.\n\nQUESTIONS:\n1. I don’t agree with the authors that they criticize on the existing definition of the Blackwell optimality. The definition looks good to me, we just don’t have an efficient algorithm yet. What if the MDP is unichain or ergodic (different as your example 3.5), does proposition 3.4 still hold?  As far as I know, the weakly communicating assumption or the ergodic assumption, is in fact known to be necessary for learning infinite-horizon MDPs with low regret.\n2. what is m in your theorem 4.4? \n3. Proposition 4.3: \\eta >0 → 1>\\eta>0\n\nLIMITATIONS:\nMore discussions on the limitations of your approach should be included. Also it is better to make more comparisons between the two discounted factors when there are more assumptions on the MDP.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to reduce the computation of Blackwell-optimal policies to that of discount-optimal policies. The authors introduce the notion of \"Blackwell discount factor\", a value $\\gamma_{bw} \\in [0,1)$ s.t. any discount-optimal policy for $\\gamma > \\gamma_{bw}$ is also Blackwell optimal. They show that $\\gamma_{bw}$ always exists for finite MDPs and upper bound it as a function of the size of the MDP. Finally, they extend these results to robust MDPs.\n\nSTRENGTHS:\n1. The paper studies a relevant problem, that of computing Blackwell optimal policies, which hasn't received much research attention so far (eg, as compared to other popular optimality criteria)\n2. The contribution is significant and, to my knowledge, novel. Being able to reduce the computation of Blackwell optimal policies to discounted MDPs is absolutely relevant given the theoretical understanding and efficient algorithms that we currently have for the latter setting\n3. The paper is extremely well written. Despite the complexity of the studied topic (which is also not very well known in the RL community), the authors did a very nice job in providing the necessary basics and intuitively discussing all formal results, while providing comprehensive proof sketches. Related works are also discussed with sufficient detail.\n\nWEAKNESSES:\n1. While it is nice to have an upper bound on $\\gamma_{bw}$ that depends only on quantities known by the learner, the quantity $\\eta(M)$ is exponentially small in the size of the MDP (eg in the number of states). This essentially means that, while we can solve a discounted MDP with $\\gamma = 1-\\eta(M)$ to compute a Blackwell optimal policy, we need to use a discount factor which is extremely close to 1 even for MDPs of moderate size. In practice, such a large value of $\\gamma$ may simply make the learning process extremely inefficient. It is thus natural to wonder how conservative the proposed bound is. For instance, do the authors believe that there exist MDPs with a \"simple structure\" in which $\\gamma_{bw}$ is way smaller than the stated bound?\n2. There is no numerical simulation. It would have been nice to see some experiments comparing (eg in terms of run-time) existing algorithms for computing Blackwell optimal policies to a method for solving discount MDPs with the (upper bound on the) Blackwell discount factor\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nLimitations have been discussed. No potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a new concept called Blackwell discount factor $\\gamma_{bw}$, which enjoys good properties: any policy that is $\\gamma_{bw}$-discount-optimal will be Blackwell optimal as well. If $\\gamma_{bw}$ is known, then one can reduce the problem of find average-optimal policy to finding discount-optimal policies, which is practically easier. The authors provide theoretical justifications for Blackwell discount factor and a proper instance-dependent upper bound. In the end, they extend the results to Robust MDP setting, where the definitions and results continue to hold.\n\nSTRENGTHS:\nThe paper tackles a significant shortcoming in the previous literature on Blackwell optimality: the current definition does not allow us to find Blackwell optimal policy in a naive way that is to choose a large enough $\\gamma$ and simply find its discount optimal policy. The paper has a clear and strong motivation to study the problem with a minor question I will mention in the Weakness section.\n\nThe theoretical results are clearly delivered and significant. Especially some impossible results in Proposition 3.4 and Theorem 3.6 are intriguing and it reveals something fundamental about the Blackwell optimality.\n\nWEAKNESSES:\nCertain writings and motivation may be taken care:\n1. Line 5: computing average-optimal policies requires only the weakly-communication assumption (instead of unichain or ergodicity), which is a reasonable assumption as weakly-communication is necessary for a unique optimal average rewards.\n2. Line 43-61: There exists many value iteration algorithm for average-reward setting. I don't fully understand the motivation to find the optimal average-optimal policy through finding discounted-optimal policies. The well-known [UCRL2](https://proceedings.neurips.cc/paper_files/paper/2008/file/e4a6222cdb5b34375400904f03d8e6a5-Paper.pdf) calls value iteration as a sub-routine.\n3. Saying the classical definition of Blackwell optimality has shortcomings can be confusing, because the paper does not have a new definition for Blackwell optimality. Instead, it proposes a new concept named Blackwell discount factor. \n4. I am not fully convinced that Blackwell discount factor is the only discount factor of interest. In some circumstances, we may want to find the Blackwell optimal policy that is $\\gamma$-discount-optimal for as small as possible $\\gamma$. \n5. I don't see a strong connection between Blackwell optimality and Robust MDP. It is nice to see that similar results continue to hold for Robust MDP. However, the two topics seem orthogonal to me. Before reading Section 4.2, I was expecting to see results on how Blackwell optimality improves robustness, since Blackwell optimality in general sense is one type of robustness w.r.t. the change of discount factors.\n\nMinor comments:\n1. Line 247: the authors mentioned “in the next proposition”, while it is a Theorem that follows.\n2. Theorem 4.5, Theorem 4.6 should be stated as a Lemma.\n\nQUESTIONS:\n1. Is $\\gamma_{bw} = \\max_{\\pi} \\gamma(\\pi)$? Intuitively, this makes sense as if $\\gamma_{bw} < \\gamma(\\pi')$, then $\\pi'$ is not $\\gamma_{bw}$-discount-optimal, which violates the definition. If this is true, then can I understand the paper as finding the smallest $\\gamma$ such that all Blackwell optimal policies are simultaneously optimal on this $\\gamma$?\n2. Is it NP-Hard to verify weakly-communicating assumption?\n\nLIMITATIONS:\nThe authors mentioned approximate Blackwell optimality and robust Blackwell discount factor for other types of uncertainty sets. No societal impacts that need to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

comments: SUMMARY:
This paper defines the Blackwell discount factor: when the discount factor is larger than this factor, discount-optimal policies become Blackwell and average-optimal. An analytical solution for such factor is also derived. The results are also extended to the setting of sa-rectangular RMDPs.

STRENGTHS:
Motivated by the original definition of Blackwell optimality, the authors propose the Blackwell discount factor, via which we can solve for the optimal policy of an average-reward MDP by solving a corresponding discounted-reward one, where the restrictive assumptions in average-reward setting is bypassed. This should be an exciting result. The writing is clear and the contents are well organized.

WEAKNESSES:
I only have some minor comments. Please see my questions below.

QUESTIONS:
Line 177: It should be "we use the notation...".

Line 319: It is not clear to me what "coarseness" is and how Proposition 4.3 shows this dependence.

LIMITATIONS:
Please see the questions above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a new class of objective for MDPs. Instead of aiming to find the policy which maximizes the long-term average reward, the authors propose a new approach to find a Blackwell optimal policy. A Blackwell optimal policy also maximizes the average reward, but Blackwell optimality has not been studied much in the literature due to the intractability. In this paper, the authors show that there exists a ‘Blackwell discount factor’ such that any policy which is discount optimal for this specific choice of discount factor is also Blackwell optimal and average reward optimal. However, finding this Blackwell discount factor is quite challenging, although the authors provide a sufficient condition for it, providing an upper bound on the Blackwell discount factor, and showing that any policy which is optimal wrt this upper bound on the discount factor is Blackwell optimal. However, this upper bound depends on properties of the MDP which are may not be known in RL. The authors also provide an extension to robust MDPs.\n\nSTRENGTHS:\nThe notion of Blackwell optimality is interesting and indeed not studied by the RL community, so it is nice to see it being brought to attention. It is also nice that this provides a method for finding an average-optimal policy in a known MDP without assumptions on the MDP, and it is pleasing that there is an extension to robust MDPs. The paper is reasonably well written and organised.\n\nWEAKNESSES:\nIt seems that Blackwell optimality has been considered in MDPs before, so the main contribution seems to be defining this Blackwell discount factor and showing solving an MDP with that discount factor leads to an average-optimal policy. From the discussion of the literature, it sounds like the existence of the Blackwell discount factor was folklore so it is nice to see it proven. However, the Blackwell discount factor itself seems difficult to find and the proposed upper bound also depends on quantities such as the maximum bit size of the MDP which I imagine is typically not known. However, there was only limited discussion of the proposed bound so I found it somewhat difficult to interpret. The bound on the discount factor also seems to scale with 1-1/(4^S)^S, so as S gets bigger this will approach 1 very quickly (indeed when I plotted it on my laptop, for S=4, it was rounded to 1). Therefore, I am concerned that for most reasonably sized MDPs, we will need to find an optimal policy with a discount factor arbitrarily close to 1 which will still be challenging and may require more assumptions, so I am not completely sure of the benefit of this approach. \nLastly, it is not clear how to translate this approach to an RL setting where we do not know the MDP. There was a bit of discussion about this, but not sufficient to understand how to actually use it in RL. There is an attempt to generalise to robust MDPs where we only know a set of transition functions, but I did not think sufficient details were provided to fully understand the usefulness of this result. I also wonder whether this work could be better suited to another venue.\n\nQUESTIONS:\nCan more details of calculating the bit-size of an MDP be given? In particular, it would be nice to see an example of the bit-size of an MDP (e.g. riverswim or another classical example) be given? Can this also be translated into the bound on the discount factor? \nIt would also be good to see a plot of the bound on the discount factor with size of the state space to see when it leads to bounds that are essentially 1. \nIt may also be good to have a comparison in terms of computation time/number of iterations of this method and other methods, even if there are different assumptions. More generally, it would be good to have the benefits of the proposed method clearly demonstrated. \nCan the results for robust MDPs be used in the setting where we define confidence sets using data?\n\nLIMITATIONS:\nLimitations (e.g. what to do when we don’t know the MDP) are discussed briefly but could be discussed in more detail.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}