Title: Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: reinforcement learning theory

average reward

sample complexity
Abstract: We study the sample complexity of learning an $\varepsilon$-optimal policy in an average-reward Markov decision process (MDP) under a generative model. For weakly communicating MDPs, we establish the complexity bound $\widetilde{O}\left(SA\frac{\mathsf{H}}{\varepsilon^2} \right)$, where $\mathsf{H}$ is the span of the bias function of the optimal policy and $SA$ is the cardinality of the state-action space. Our result is the first that is minimax optimal (up to log factors) in all parameters $S,A,\mathsf{H}$, and $\varepsilon$, improving on existing work that either assumes uniformly bounded mixing times for all policies or has suboptimal dependence on the parameters. We also initiate the study of sample complexity in general (multichain) average-reward MDPs. We argue a new transient time parameter $\mathsf{B}$ is necessary, establish an $\widetilde{O}\left(SA\frac{\mathsf{B} + \mathsf{H}}{\varepsilon^2} \right)$ complexity bound, and prove a matching (up to log factors) minimax lower bound. Both results are based on reducing the average-reward MDP to a discounted MDP, which requires new ideas in the general setting. To optimally analyze this reduction, we develop improved bounds for $\gamma$-discounted MDPs, showing that $\widetilde{O}\left(SA\frac{\mathsf{H}}{(1-\gamma)^2\varepsilon^2} \right)$ and $\widetilde{O}\left(SA\frac{\mathsf{B} + \mathsf{H}}{(1-\gamma)^2\varepsilon^2} \right)$ samples suffice to learn $\varepsilon$-optimal policies in weakly communicating and in general MDPs, respectively. Both these results circumvent the well-known minimax lower bound of $\widetilde{\Omega}\left(SA\frac{1}{(1-\gamma)^3\varepsilon^2} \right)$ for $\gamma$-discounted MDPs, and establish a quadratic rather than cubic horizon dependence for a fixed MDP instance.

Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPs

Abstract

We study the sample complexity of learning an ε -optimal policy in an averagereward Markov decision process (MDP) under a generative model. For weakly communicating MDPs, we establish the complexity bound ˜ O ( SA H ε 2 ) , where H is the span of the bias function of the optimal policy and SA is the cardinality of the state-action space. Our result is the first that is minimax optimal (up to log factors) in all parameters S, A, H , and ε , improving on existing work that either assumes uniformly bounded mixing times for all policies or has suboptimal dependence on the parameters. We also initiate the study of sample complexity in general (multichain) average-reward MDPs. We argue a new transient time parameter B is necessary, establish an ˜ O ( SA B + H ε 2 ) complexity bound, and prove a matching (up to log factors) minimax lower bound. Both results are based on reducing the average-reward MDP to a discounted MDP, which requires new ideas in the general setting. To optimally analyze this reduction, we develop improved bounds for γ -discounted MDPs, showing that ˜ O ( SA H (1 -γ ) 2 ε 2 ) and ˜ O ( SA B + H (1 -γ ) 2 ε 2 ) samples suffice to learn ε -optimal policies in weakly communicating and in general MDPs, respectively. Both these results circumvent the well-known minimax lower bound of ˜ Ω ( SA 1 (1 -γ ) 3 ε 2 ) for γ -discounted MDPs, and establish a quadratic rather than cubic horizon dependence for a fixed MDP instance.

1 Introduction

The paradigm of Reinforcement learning (RL) has demonstrated remarkable successes in various sequential learning and decision-making problems. Empirical successes have motivated extensive theoretical study of RL algorithms and their fundamental limits. The RL environment is commonly modeled as a Markov decision process (MDP), where the objective is to find a policy π that maximizes the expected cumulative rewards. Different reward criteria are considered, such as the finite horizon total reward E π [ ∑ T t =0 R t ] and the infinite horizon total discounted reward E π [ ∑ ∞ t =0 γ t R t ] with a discount factor γ < 1 . The finite horizon criterion only measures performance for T steps, and the discounted criterion is dominated by rewards from the first 1 1 -γ time steps. In many situations where the long-term performance of the policy π is of interest, we may prefer to evaluate policies by their long-run average reward lim T →∞ (1 /T ) E π [∑ T -1 t =0 R t ] .

A foundational theoretical problem in RL is the sample complexity for learning a near-optimal policy using a generative model of the MDP [10], meaning the ability to obtain independent samples of the next state given any initial state and action. For the finite horizon and discounted reward criteria, the sample complexity of this task has been thoroughly studied (e.g., [2, 3, 15, 19, 1, 12]). However, despite significant effort (reviewed in Section 1.1), the sample complexity of the average reward setting is unresolved in existing literature.

Our contributions In this paper, we resolve the sample complexity of weakly communicating Average-Reward MDPs (AMDP) in terms of H := ∥ h ⋆ ∥ span , the span of the bias (a.k.a. relative value function) of the optimal policy. We show that ˜ O ( SA H /ε 2 ) samples suffice to find an ε -optimal policy of a weakly communicating MDP with S states and A actions. This bound, presented in Theorem 2, is the first that matches the minimax lower bound ˜ Ω ( SA H /ε 2 ) up to log factors.

Furthermore, we initiate the study of sample complexity for average-reward general MDPs , which refers to the class of all finite-space MDPs without any restrictions [14]. General MDPs are not necessarily weakly communicating and all their optimal policies may be multichain . In this general setting, we demonstrate the span H alone cannot characterize the sample complexity, as the lower bound in Theorem 4 exhibits instances which require ≫ H SA/ε 2 samples. This observation motivates our introduction of a new transient time bound parameter B , which in conjunction with H captures the sample complexity of general average-reward MDPs. Specifically, our Theorem 8 shows that ˜ O ( SA B + H ε 2 ) samples suffice to learn an ε -optimal policy, and Theorem 4 provides a matching minimax lower bound of Ω ( SA B + H ε 2 ) . We remark that it is trivially impossible to achieve low regret in standard online settings of general MDPs, since the agent may become trapped in a closed class of low reward states [4]. The simulator setting is natural for studying general MDPs since it avoids this fatal issue, although the existence of multiple closed classes with different long-run rewards still plays a fundamental role in the minimax sample complexity, as reflected in the dependence on B .

To establish the above upper bounds, we adopt the reduction-to-discounted-MDP approach [9, 20], and improve on prior work by developing enhanced sample complexity bounds for γ -discounted MDPs (DMDPs). We improve the analysis of variance parameters related to DMDPs using a new multistep variance Bellman equation, which is applied in a recursive manner to bound the variance of near-optimal policies. For general (multichain) MDPs, we further utilize law-of-totalvariance ideas to bound the total variance contribution from transient states, which present new challenges significantly different to their behavior in the weakly communicating setting. Our averageto-discounted reduction also requires new techniques, because many structural properties used in earlier reduction arguments no longer hold for general MDPs. Our analysis leads to DMDP sample complexities of ˜ O ( SA H (1 -γ ) 2 ε 2 ) and ˜ O ( SA B + H (1 -γ ) 2 ε 2 ) to learn ε -optimal policies in weakly communicating and general MDPs, respectively. Notably, the latter bound, valid for all MDPs, circumvents the existing lower bound ˜ Ω ( SA (1 -γ ) 3 ε 2 ) [3, 15]. Whereas this minimax lower bound allows the adversary to choose the transition matrix P based on γ with B ≈ 1 1 -γ [3, Theorem 3], our result reflects the complexity of a fixed MDP P through its parameters H , B and a quadratic dependence on the effective horizon 1 1 -γ . This fixedP complexity is essential for our particular algorithmic approach, where the reduction discount γ is chosen depending on P . It is also a more relevant framework in general for many RL problems where the discount factor is tuned for best performance on a particular instance.

1.1 Comparison with related work on average-reward MDPs

We summarize in Table 1 existing sample complexity results for average reward MDPs.

Various parameters have been used to characterize the sample complexity of average reward MDPs, including the diameter D of the MDP, the uniform mixing time bound τ unif for all policies, and the span H of the optimal bias; formal definitions are provided in Section 2. All sample complexity upper bounds involving τ unif require the strong assumption that all stationary deterministic policies have finite mixing times. Otherwise, τ unif = ∞ by definition, which for example occurs if some policy induces a periodic Markov chain. It is also possible to have D = ∞ , while H and our newly introduced B are always finite for finite state-action spaces. As shown in [20], there is generally no relationship between D and τ unif ; they can each be arbitrarily larger than the other. On the other hand, it has been shown that H ≤ D [4] and that H ≤ 8 τ unif [20]. Therefore, either of the first two minimax lower bounds in Table 1 (which both use hard instances that are weakly communicating) imply a lower bound of ˜ Ω ( SA H ε 2 ) and thus the minimax optimality of our Theorem 2.

To the best of our knowledge, no prior work has considered the average-reward sample complexity of general (potentially multichain) MDPs. Existing results make assumptions at least as strong as weakly communicating or uniformly bounded mixing times.

The work [9] was the first to develop an algorithm based on reduction to a discounted MDP with a discount factor of γ = 1 -ε τ unif . Their argument was improved in [20], which improved the uniform mixing assumption to only assuming a weakly communicating MDP, and used a smaller discount factor γ = 1 -ε H . These arguments both make essential use of the fact that the optimal gain is independent of the starting state, which does not hold for general MDPs. After analyzing the reductions, both [9] and [20] then solved the discounted MDPs by appealing to the algorithm from [12]. To the best of our knowledge, the algorithm of [12] is the only known algorithm for discounted MDPs which could work with either reduction, as the reductions each require a ε 1 -γ -optimal policy from the discounted MDP, and other known algorithms for discounted MDPs do not permit such large suboptimality levels. (We discuss algorithms for discounted MDPs in more detail below.) Other algorithms for average-reward MDPs are considered in [9, 13, 26]. The above results fall short of matching the minimax lower bounds.

While preparing this manuscript, we became aware of [22], which considers the uniform mixing setting and obtains a minimax optimal sample complexity ˜ O ( SA τ unif ε 2 ) in terms of τ unif . Although developed independently, their work and ours have several similarities. We both utilize discounted reductions and observe that it is possible to improve the sample complexity of the resulting DMDP task by improving the analysis of variance parameters. They accomplish the improvement by leveraging the uniform mixing assumption, whereas we make use of the low span of the optimal policy. Note that H ≤ 8 τ unif holds in general and there exist MDPs with H ≪ τ unif = ∞ , so our Theorem 2 is strictly stronger than the result of [22].

1.2 Comparison with related work on discounted MDPs

We discuss a subset of results for discounted MDPs in the generative setting. Several works [15, 19, 1, 12] obtain the minimax optimal sample complexity of ˜ O ( SA 1 (1 -γ ) 3 ε 2 ) for finding an ε -optimal policy w.r.t. the discounted reward. However, only [12] is able to show this bound for the full range of ε ∈ (0 , 1 1 -γ ] . As mentioned, the reduction from average reward MDPs requires a large ε in the resulting discounted MDP, making it unsurprising that all of [9, 20, 22] as well as our Algorithm 1 essentially use their algorithm. The matching lower bound is established in [15, 3].

As mentioned earlier, both we and the authors of [22, 21] independently observed that the ˜ Ω ( SA 1 (1 -γ ) 3 ε 2 ) sample complexity lower bound can be circumvented in the settings that arise

under the average-to-discounted reductions. The authors of [22, 21] assume uniform mixing and obtain a discounted MDP sample complexity of ˜ O ( SA τ unif (1 -γ ) 2 ε 2 ) , first in [21] by modifying the algorithm of [19], and then in [22] under a wider range of ε by instead modifying the analysis of [12]. The work [21] also proves a matching lower bound. Our Theorem 1 for discounted MDPs attains a sample complexity of ˜ O ( SA H (1 -γ ) 2 ε 2 ) assuming only that the MDP is weakly communicating. Again, in light of the relationship that H ≤ 8 τ unif , our results are strictly better (ignoring constants), and their lower bound also establishes the optimality of our Theorem 1.

2 Problem setup and preliminaries

A Markov decision process (MDP) is given by a tuple ( S , A , P, r ) , where S is the finite set of states, A is the finite set of actions, P : S × A → ∆( S ) is the transition kernel with ∆( S ) denoting the probability simplex over S , and r : S×A → [0 , 1] is the reward function. Let S := |S| and A := |A| denote the cardinality of the state and action spaces, respectively. Unless otherwise noted, all policies considered are stationary Markovian policies of the form π : S → ∆( A ) . For any initial state s 0 ∈ S and policy π , we let E π s 0 denote the expectation with respect to the probability distribution over trajectories ( S 0 , A 0 , S 1 , A 1 , . . . ) where S 0 = s 0 , A t ∼ π ( S t ) , and S t +1 ∼ P ( · | S t , A t ) . Equivalently, this is the expectation with respect to the Markov chain induced by π starting in state s 0 , with the transition probability matrix P π given by ( P π ) s,s ′ := ∑ a ∈A π ( a | s ) P ( s ′ | s, a ) . We also define ( r π ) s := ∑ a ∈A π ( a | s ) r ( s, a ) . We occasionally treat P as an ( S × A ) -byS matrix where P sa,s ′ = P ( s, a, s ′ ) . We also let P sa denote the row vector such that P sa ( s ′ ) = P ( s, a, s ′ ) . For any s ∈ S and any bounded function X of the trajectory, we define the variance V π s [ X ] := E π s ( X -E π s [ X ]) 2 , with its vector version V π [ X ] ∈ R S given by ( V π [ X ]) s = V π s [ X ] . For s ∈ S , let e s ∈ R S be the vector that is all 0 except for a 1 in entry s . Let 1 ∈ R S be the all-one vector. For each v ∈ R S , define the span semi-norm ∥ v ∥ span := max s ∈S v ( s ) -min s ∈S v ( s ) .

)

Discounted reward criterion A discounted MDP is a tuple ( S , A , P, r, γ ) , where γ ∈ (0 , 1) is the discount factor. For a stationary policy π , the (discounted) value function V π γ : S → [0 , ∞ is defined, for each s ∈ S , as V π γ ( s ) := E π s [ ∑ ∞ t =0 γ t R t ] , where R t = r ( S t , A t ) is the reward received at time t . It is well-known that there exists an optimal policy π ⋆ γ that is deterministic and satisfies V π ⋆ γ γ ( s ) = V ⋆ γ ( s ) := sup π V π γ ( s ) for all s ∈ S [14]. In discounted MDPs the goal is to compute an ε -optimal policy, which we define as a policy π satisfying ∥ ∥ V π γ -V ⋆ γ ∥ ∥ ∞ ≤ ε . We define one more variance parameter V P π [ V π γ ] ∈ R S , specific to a given policy π , by ( V P π [ V π γ ]) s := ∑ s ′ ∈S ( P π ) s,s ′ [ V π γ ( s ′ ) -∑ s ′′ ( P π ) s,s ′′ V π γ ( s ′′ ) ] 2 .

Average-reward criterion In an MDP ( S , A , P, r ) , the average reward per stage or the gain of a policy π starting from state s is defined as ρ π ( s ) := lim T →∞ 1 T E π s [ ∑ T -1 t =0 R t ] . The bias function of any stationary policy π is h π ( s ) := C-lim T →∞ E π s [ ∑ T -1 t =0 ( R t -ρ π ( S t )) ] , where C-lim denotes the Cesaro limit. When the Markov chain induced by P π is aperiodic, C-lim can be replaced with the usual limit. For any policy π , its ρ π and h π satisfy ρ π = P π ρ π and ρ π + h π = r π + P π h π .

Apolicy π ⋆ is Blackwell-optimal if there exists some discount factor ¯ γ ∈ (0 , 1) such that for all γ ≥ ¯ γ we have V π ⋆ γ ≥ V π γ for all policies π . Henceforth we let π ⋆ denote some fixed Blackwell-optimal policy, which is guaranteed to exist when S and A are finite [14]. We define the optimal gain ρ ⋆ ∈ R S by ρ ⋆ ( s ) = sup π ρ π ( s ) and note that we have ρ ⋆ = ρ π ⋆ . For all s ∈ S , ρ ⋆ ( s ) ≥ max a ∈A P sa ρ ⋆ , or equivalently ρ ⋆ ≥ P π ρ ⋆ for all policies π (and this maximum is achieved by π ⋆ ). We also define h ⋆ = h π ⋆ (and we note that this definition does not depend on which Blackwell-optimal π ⋆ is used, if there are multiple). For all s ∈ S , ρ ⋆ and h ⋆ satisfy ρ ⋆ ( s ) + h ⋆ ( s ) = max a ∈A : P sa ρ ⋆ = ρ ⋆ ( s ) r sa + P sa h ⋆ , known as the (unmodified) Bellman equation.

A weakly communicating MDP is such that the states can be partitioned into two disjoint subsets S = S 1 ∪ S 2 such that all states in S 1 are transient under any stationary policy and within S 2 , any state is reachable from any other state under some stationary policy. In weakly communicating MDPs ρ ⋆ is a constant vector (all entries are equal), and thus ( ρ ⋆ , h ⋆ ) are also a solution to the modified Bellman equation ρ ⋆ ( s ) + h ⋆ ( s ) = max a ∈A r sa + P sa h ⋆ . When discussing weakly communicating MDPs we occasionally abuse notation and treat ρ ⋆ as a scalar. A stationary policy is multichain if it

induces multiple closed irreducible recurrent classes, and an MDP is called multichain if it contains such a policy. Weakly-communicating MDPs always contain some gain-optimal policy which is unichain (not multichain), but in general MDPs, all gain-optimal policies may be multichain and ρ ⋆ may not be a constant vector. All uniformly mixing MDPs are weakly communicating. In the average reward setting, our goal is find an ε -optimal policy, defined as a policy π such that ∥ ρ ⋆ -ρ π ∥ ∞ ≤ ε .

Complexity parameters Our most important complexity parameter is the span of the optimal bias function H := ∥ h ⋆ ∥ span . In addition, for general MDPs we introduce a new transient time parameter B , defined as follows. Let Π be the set of deterministic stationary policies. For each π ∈ Π , let R π be the set of states which are recurrent in the Markov chain P π , and let T π = S \ R π be the set of transient states. Let T R π = inf { t : S t ∈ R π } be the first hitting time of a state which is recurrent under π . We say an MDP satisfies the bounded transient time property with parameter B if for all policies π and states s ∈ S we have E π s [ T R π ] ≤ B , or in words, the expected time spent in transient states (with respect to the Markov chain induced by π ) is bounded by B .

̸

We recall several other parameters used in the literature to characterize sample complexity. The diameter is defined as D := max s 1 = s 2 inf π ∈ Π E π s 1 [ η s 2 ] , where η s denotes the hitting time of a state s ∈ S . For each policy π , if the Markov chain induced by P π has a unique stationary distribution ν π , we define the mixing time of π as τ π := inf { t ≥ 1 : max s ∈S ∥ ∥ ∥ e ⊤ s ( P π ) t -ν ⊤ π ∥ ∥ ∥ 1 ≤ 1 2 } . If all policies π ∈ Π satisfy this assumption, we define the uniform mixing time τ unif := sup π ∈ Π τ π . Note that D and τ unif are generally incomparable [20], while we always have H ≤ D [4] and H ≤ 8 τ unif [20]. It is possible for τ unif = ∞ , for instance if there are any policies which induce periodic Markov chains. Also, D = ∞ if there are any states which are transient under all policies. However, H and B are finite in any MDP with S, A < ∞ . Also if τ unif is finite, Lemma 27 shows B ≤ 4 τ unif .

We assume access to a generative model [10], also known as a simulator. This means we can obtain independent samples from P ( · | s, a ) for any given s ∈ S , a ∈ A , but P itself is unknown. We assume the reward function r is deterministic and known, which is standard in generative settings (e.g., [1, 12]) since otherwise estimating the mean rewards is relatively easy. Specifically, to learn an ε -optimal policy for the discounted MDP, we would need to estimate each entry of r to accuracy O ((1 -γ ) ε ) , which requires a lower order number of samples ˜ O ( SA (1 -γ ) 2 ε 2 ) . For this reason we assume (as in [20]) that H ≥ 1 . Using samples from the generative model, our Algorithm 1 constructs an empirical transition kernel ̂ P . For a policy π , we use ̂ V π γ ( s ) to denote the value function computed with respect to the Markov chain with transition matrix ̂ P π (as opposed to P π ). Our Algorithm 1 also utilizes a perturbed reward function ˜ r , and we use the notation V π γ, p ( s ) to denote a value function computed using this reward (and P π ); more concretely, we replace R t with ˜ R t = ˜ r ( S t , A t ) in the definition above of V π γ . We use the notation ̂ V π γ, p when using ̂ P and ˜ r simultaneously.

3 Main results for weakly communicating MDPs

Our approach is based on reducing the average-reward problem to a discounted problem. We first present our algorithm and guarantees for the discounted MDP setting. As discussed in Subsection 1.1, our algorithm of choice, Algorithm 1, is essentially the same as the one presented in [12], with a slightly different perturbation level ξ . Algorithm 1 constructs an empirical transition kernel ̂ P using n samples per state-action pair from the generative model, and then solves the resulting empirical (perturbed) MDP ( ̂ P, ˜ r, γ ) . As noted in [12], the perturbation ensures ̂ π ⋆ γ, p can be computed exactly in poly ( 1 1 -γ , S, A, log(1 /δε )) time by multiple standard MDP solvers. We remark in passing that the SA -byS transition matrix ̂ P has at most nSA nonzero entries.

Our Theorem 1 provides an improved sample complexity bound for Algorithm 1 under the setting that the MDP is weakly communicating.

Theorem 1 (Sample Complexity of Weakly Communicating DMDP) . Suppose the discounted MDP ( P, r, γ ) is weakly communicating, H ≤ 1 1 -γ , and ε ≤ H . There exists a constant C 2 > 0 such that, for any δ ∈ (0 , 1) , if n ≥ C 2 H (1 -γ ) 2 ε 2 log ( SA (1 -γ ) δε ) , then with probability at least 1 -δ , the policy ̂ π ⋆ γ, p output by Algorithm 1 satisfies ∥ ∥ ∥ V ⋆ γ -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ε.

Algorithm 1 Perturbed Empirical Model-Based Planning

input: Sample size per state-action pair n , target accuracy ε , discount factor γ

1: for each state-action pair ( s, a ) ∈ S × A do

2: Collect n samples S 1 s,a , . . . , S n s,a from P ( · | s, a )

3: Form the empirical transition kernel ̂ P ( s ′ | s, a ) = 1 n ∑ n i =1 I { S i s,a = s ′ } , for all s ′ ∈ S

4: end for

5: Set perturbation level ξ = (1 -γ ) ε/ 6

6: Form perturbed reward r = r + Z where Z ( s, a ) i.i.d. ∼ Unif (0 , ξ )

8: return ̂ π ⋆ γ, p

˜ 7: Compute a policy ̂ π ⋆ γ, p which is optimal for the perturbed empirical discounted MDP ( ̂ P, ˜ r, γ )

Since we observe n samples for each state-action pair, Theorem 1 shows that a total number of ˜ O ( H SA (1 -γ ) 2 ε 2 ) samples suffices to learn an ε -optimal policy. This bound improves on the ˜ O ( SA (1 -γ ) 3 ε 2 ) complexity bound from [12] when the span H is no larger than the effective horizon 1 1 -γ . This assumption holds in many situations, as can be seen by using the relationships H ≤ D or H ≤ 8 τ unif . On the other hand, in the regime with H > 1 1 -γ , the existing bound ˜ O ( SA (1 -γ ) 3 ε 2 ) , also achieved by Algorithm 1, is superior. In this regime, the discounting effectively truncates the MDP at a short horizon 1 1 -γ before the long-run behavior of the optimal policy (as captured by H ) kicks in.

Proof highlights for Theorem 1. The key to obtaining this improved complexity is a careful analysis of certain instance-specific variance parameters. It suffices to bound ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ and ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ by O ( ε ) . The prior DMDP complexity of SA (1 -γ ) 3 ε 2 is obtained using the wellknown law-of-total-variance argument [3, 1, 12], which ultimately yields a sample complexity like ˜ O (√ SA (1 -γ ) ε 2 ∥ ∥ V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] ∥ ∥ ∞ ) to bound ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ O ( ε ) . From here, the variance of the cumulative discounted reward ∥ ∥ ∥ V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] ∥ ∥ ∥ ∞ is bounded by 1 (1 -γ ) 2 , since the total reward in a trajectory is within [0 , 1 1 -γ ] . We instead seek to bound ∥ ∥ ∥ V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] ∥ ∥ ∥ ∞ ≤ O ( H 1 -γ ) . Assume H is an integer. The first step is to decompose V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] recursively like

(see our Lemma 13). This is a multi-step version of the standard variance Bellman equation (e.g., [16, Theorem 1]). Ordinarily an H -step expansion would not be useful, since the term V π ⋆ γ γ ( S H ) by itself appears to have fluctuations on the order of 1 1 -γ in the worst case depending on S H (note S H is the random state encountered at time H ). However, in our setting, we should have V π ⋆ γ γ ( S H ) ≈ 1 1 -γ ρ ⋆ + h ⋆ ( S H ) , reducing the magnitude of the random fluctuations to order H = ∥ h ⋆ ∥ span . (See Lemma 11 for a formalization of this approximation which first appeared in [23].) Therefore expansion to H steps achieves the optimal tradeoff between maintaining V π ⋆ γ [ ∑ H -1 t =0 γ t R t + γ H V π ⋆ γ γ ( S H ) ] ≤ O ( H 2 ) and minimizing γ 2 H . As desired this yields ∥ ∥ ∥ V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] ∥ ∥ ∥ ∞ ≤ O ( H 2 1 -γ 2 H ) = O ( H 1 -γ ) , where 1 1 -γ 2 H ≤ O ( 1 H (1 -γ ) ) requires 1 1 -γ ≥ H . See Lemma 15 for the complete argument.

We would like to use a similar argument as above to bound the second term ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ , which is the 'evaluation error' of the empirically optimal policy ̂ π ⋆ γ, p . However, applying the same argument would give a bound in terms of ∥ ∥ ∥ V ̂ π ⋆ γ, p γ ∥ ∥ ∥ span , which, unlike for the analogous term involving the true optimal policy π ⋆ γ , is not a priori bounded in terms of H . (If we instead assumed uniform mixing, we could immediately bound this by O ( τ unif ) .) Thus, to control the variance associated with evaluating ̂ π ⋆ γ, p , we are able to recursively bound ∥ ∥ ∥ V ̂ π ⋆ γ, p γ ∥ ∥ ∥ span ≤ O ( H + ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ) , which can be shown to yield the desired sample complexity.

Now we present our main result for the average-reward problem in the weakly communicating setting. Applied in this setting with a DMDP target accuracy of ε = H , our Algorithm 2 reduces the problem to γ -discounted MDP with γ = 1 -ε 12 H and then calls Algorithm 1 with target accuracy H .

Algorithm 2 Average-to-Discount Reduction

input: Sample size per state-action pair n , target accuracy ε ∈ (0 , 1] , DMDP target accuracy ε

1: Set γ = 1 -ε 12 ε

3: return ̂ π ⋆

2: Obtain ̂ π ⋆ from Algorithm 1 with sample size per state-action pair n , accuracy ε , discount γ

We have the following sample complexity bound for Algorithm 2.

Theorem 2 (Sample Complexity of Weakly Communicating AMDP) . Suppose the MDP ( P, r ) is weakly communicating. There exists a constant C 1 > 0 such that for any δ, ε ∈ (0 , 1) , if n ≥ C 1 H ε 2 log ( SA H δε ) and we call Algorithm 2 with ε = H , then with probability at least 1 -δ , the output policy ̂ π ⋆ satisfies the elementwise inequality ρ ⋆ -ρ ̂ π ⋆ ≤ ε 1 .

Again, since we observe n samples for each state-action pair, this result shows that ˜ O ( H SA ε 2 ) total samples suffice to learn an ε -optimal policy for the average reward MDP. This bound matches the minimax lower bound in [20] and is superior to existing results for weakly communicating MDPs (see Table 1). We note that the proof of Theorem 1 works so long as H is any upper bound of ∥ h ⋆ ∥ span , hence Algorithm 2 also only needs an upper bound for ∥ h ⋆ ∥ span .

We show in the following theorem that it is in general impossible to obtain a useful upper bound on ∥ h ⋆ ∥ span with a sample complexity that is a function of only ∥ h ⋆ ∥ span . This suggests that it is not easy to remove the need for knowledge of ∥ h ⋆ ∥ span .

Theorem 3. For any given n, T ≥ 1 , there exist two MDPs M 0 and M 1 with S = 4 , A = 1 such that M 0 has optimal bias span 1 , M 1 has optimal bias span T , and it is impossible to distinguish between M 0 and M 1 with probability ≥ 3 4 with n samples from each state-action pair.

Thus even for an MDP with a small span, there exists another MDP that has an arbitrarily large span and is arbitrarily statistically close (that is, cannot be distinguished even with a large sample size n ). We emphasize that all previous algorithms in Table 1 also require knowledge of their respective complexity parameters, and such assumptions are pervasive throughout the literature on averagereward RL. The only exception of which we are aware is the contemporaneous work [7], which achieves a suboptimal ˜ O ( SA τ 8 unif ε 8 ) sample complexity without knowledge of τ unif in the uniformly mixing setting. It is unclear if H -based sample complexities are possible without knowing H . Besides the evidence offered by Theorem 3, in the online setting, it has been conjectured that knowledge of H is necessary to obtain an H -dependent regret bound [6, 5, 25]. Moreover, even with knowledge of H , the only known online algorithm with optimal regret is computationally inefficient [25], making it somewhat surprising that our Theorem 2 uses a simple and efficient algorithm.

Nevertheless, when H is unknown, one can replace H with the diameter D (since H ≤ D ). The diameter is known to be estimable [25, 17] and is often a more refined complexity parameter than τ unif . Our Theorem 2 is the first to imply the optimal diameter-based complexity ˜ O ( SAD ε 2 ) , given knowledge of D or using a constant-factor upper bound obtained from some estimation procedure.

4 Main results for general MDPs

Our starting point for general MDPs is that unlike the weakly communicating setting, their complexity cannot be captured solely by ∥ h ⋆ ∥ span . We first argue this point informally using the simple example in Figure 1, which is parameterized by a value T > 1 . Only state 1 contains multiple actions, and action 2 is optimal since it leads to state 2 which collects reward 0 . 5 forever, while taking action 1 will always eventually lead to state 3 where the reward is 0 forever. We thus have ρ ⋆ = [0 . 5 , 0 . 5 , 0] ⊤ and ∥ h ⋆ ∥ span = 0 . However, clearly Ω( T ) samples are required to even observe a transition 1 → 3 , so the sample complexity must depend on T ≫ H (without observing a transition 1 → 3 , we cannot determine that action 1 is not optimal). Taking action 1 leads to a large reward of 1 in the short

term (for T steps in expectation), so even if we had perfect knowledge of the environment, the optimal γ -discounted policy would not choose the optimal action a = 2 until the effective horizon 1 1 -γ ≥ Ω( T ) . Thus 1 1 -γ ≈ H is insufficient for the reduction to discounted MDP. Note that this instance has its bounded transient time parameter B = T . This example reflects that transient states play a categorically different role in general MDPs: in the weakly communicating setting, states which are transient under all policies can be completely ignored, whereas in this example our action at state 1 fully determines our reward even though state 1 is transient under all policies.

1

2

3

a

= 1

, R

= 1

P

(1

|

1

,

1) = 1

-

1

P

(3

|

1

,

1) =

1

T

a

= 2

, R

= 0

.

5

R

= 0

.

5

R

= 0

T

The statistical hardness is formally captured by the following theorem, which uses improved instances to obtain the correct dependence on ε .

Theorem 4 (Lower Bound for General AMDPs) . For any ε ∈ (0 , 1 / 4) , B ≥ 1 , A ≥ 4 and S ∈ 8 N , for any algorithm Alg which is guaranteed to return an ε/ 3 -optimal policy for any input average-reward MDP with probability at least 3 4 , there exists an MDP M = ( P, r ) such that:

M has S states and A actions.

Letting h ⋆ be the bias of the Blackwell-optimal policy for M , we have ∥ h ⋆ ∥ span = 0 .

M satisfies the bounded transient time assumption with parameter B .

Alg requires Ω ( B log( SA ) ε 2 ) samples per state-action pair on M .

A similar minimax lower bound holds for the discounted setting.

Theorem 5 (Lower Bound for General DMDP) . For any ε ∈ (0 , 1 / 4) , B ≥ 1 , A ≥ 4 and S ∈ 8 N for any algorithm Alg which is guaranteed to return an ε/ 3 -optimal policy for any input discounted MDP with probability at least 3 4 , there exists a discounted MDP M = ( P, r, γ ) such that:

M has S states and A actions.

M satisfies the bounded transient time assumption with parameter B .

Alg requires Ω ( B log( SA ) (1 -γ ) 2 ε 2 ) samples per state-action pair on M .

The lower bounds of ˜ O ( H ε 2 ) from the weakly communicating setting still apply in the general setting. Together with Theorem 4 they imply a ˜ O ( H + B ε 2 ) lower bound for general average-reward MDPs.

Figure 1 demonstrates that, unlike the weakly communicating setting, discounted reduction with 1 1 -γ set in terms of only H cannot succeed for general MDPs. (Contrast with Lemma 9 for the analogous theorem from [20] for weakly communicating MDPs.) We remedy this issue and lay the foundation for our matching upper bound by proving a new reduction theorem in terms of H and B ; in particular, B measures how much farther ahead we must look in order to determine which closed communicating class will be reached. By Lemma 27 B ≤ 4 τ unif , although B is always finite unlike τ unif .

Theorem 6 (Average-to-Discount Reduction for General MDP) . Suppose ( P, r ) is a general MDP, has an optimal bias function h ⋆ satisfying ∥ h ⋆ ∥ span ≤ H , and satisfies the bounded transient time assumption with parameter B . Fix ε ∈ (0 , 1] and set γ = 1 -ε B + H . For any ε γ ∈ [0 , 1 1 -γ ] , if π is any ε γ -optimal policy for the discounted MDP ( P, r, γ ) , then ρ ⋆ -ρ π ≤ ( 3 + 2 ε γ B + H ) ε 1 .

Proof highlights. Letting π ⋆ γ be the optimal policy for the γ -discounted MDP, our first key observation is that ρ ⋆ is constant within any irreducible closed recurrent block of the Markov chain P π ⋆ γ , essentially

because all states in this block must be reachable from each other with probability one (see Lemma 17). Leveraging the optimality of π ⋆ γ , this enables us to bound both ∣ ∣ V π ⋆ γ γ ( s ) -1 1 -γ ρ ⋆ ( s ) ∣ ∣ and ∣ ∣ V π ⋆ γ γ ( s ) -1 1 -γ ρ π ⋆ γ ( s ) ∣ ∣ by O ( ∥ h ⋆ ∥ span ) for any s which is recurrent under π ⋆ γ , which when combined demonstrate that the gain ρ π ⋆ γ ( s ) of π ⋆ γ is near-optimal for its recurrent states. See Lemma 21. We then leverage the bounded transient time assumption to guarantee that for transient s , V π ⋆ γ γ ( s ) is dominated by the expected returns from recurrent states, since at most O ( B ) time is spent in transient states. We complete the proof of Theorem 6 by combining these facts, as well as extending them to accommodate approximately optimal policies.

Next we establish an improved sample complexity for the discounted problem in the setting relevant to this reduction. This bound matches the lower bound in Theorem 5 up to log factors.

Theorem 7 (Sample Complexity of General DMDP) . Suppose B + H ≤ 1 1 -γ and ε ≤ B + H . There exists a constant C 3 > 0 such that, for any δ ∈ (0 , 1) , if n ≥ C 3 B + H (1 -γ ) 2 ε 2 log ( SA (1 -γ ) δε ) , then with probability 1 -δ , the policy ̂ π ⋆ γ, p output by Algorithm 1 satisfies ∥ ∥ V ⋆ γ -V ̂ π ⋆ γ, p γ ∥ ∥ ∞ ≤ ε.

Finally, we present our result for the sample complexity of general average-reward MDPs, matching the lower bound in Theorem 4 up to log factors. We again use the reduction Algorithm 2, this time with the larger DMDP target accuracy ε = B + H , leading to a discount factor of γ = 1 -ε 12( B + H ) .

Theorem 8 (Sample Complexity of General AMDP) . There exists a constant C 4 > 0 such that for any δ, ε ∈ (0 , 1) , if n ≥ C 4 B + H ε 2 log ( SA ( B + H ) δε ) and we call Algorithm 2 with ε = B + H , then with probability at least 1 -δ , the output policy ̂ π ⋆ satisfies the elementwise inequality ρ ⋆ -ρ ̂ π ⋆ ≤ ε 1 .

Proof highlights. Similarly to Theorem 2, we seek to bound certain variance parameters, and this time it would suffice to bound the variance of the cumulative discounted reward starting from any state s like ∣ ∣ V π ⋆ γ s [ ∑ ∞ t =0 γ t R t ] ∣ ∣ ≤ O ( H + B 1 -γ ) . Such a bound indeed holds for states s that are recurrent under π ⋆ γ , because ρ ⋆ ( S t ) will remain constant to ρ ⋆ ( s ) for all t , since, as mentioned above, ρ ⋆ is constant on closed irreducible recurrent blocks, and all ( S t ) t ≥ 0 will stay in the same block as s . Therefore, we can almost reuse our argument from the weakly communicating case. However, if s is transient, it is easy to see that ∣ ∣ ∣ V π ⋆ γ s [ ∑ ∞ t =0 γ t R t ] ∣ ∣ ∣ = Ω ( ( 1 1 -γ ) 2 ) in general (even under the bounded transient time assumption), as we can consider an example where from s we transition to either an absorbing reward 1 state or an absorbing reward 0 state. Thus, when s is transient, instead of bounding ∣ ∣ V π ⋆ γ s [ ∑ ∞ t =0 γ t R t ] ∣ ∣ , we directly work with the sharper variance parameter ∣ ∣ ∣ e ⊤ s ( I -γP π ⋆ γ ) -1 √ V P π ⋆ γ [ V π ⋆ γ γ ] ∣ ∣ ∣ , which is also common to the analysis of DMDPs [3, 1, 12] (and in these previous works is bounded in terms of ∥ ∥ ∥ V π ⋆ γ [ ∑ ∞ t =0 γ t R t ] ∥ ∥ ∥ ∞ ; see Lemma 12 for this relationship). We instead develop a novel law-of-total-variance-style argument which limits the total contribution of transient states to this sharper variance parameter. See Lemma 26 for details.

5 Conclusion

In this paper we obtained optimal sample complexities for weakly communicating and general average reward MDPs by improving the analysis of discounted MDPs, revealing a quadratic rather than cubic dependence on the effective horizon for a fixed instance. A limitation of our results (as well as of all previous results) is that the average-to-discounted reduction requires prior knowledge of parameters for optimal complexity, and an interesting open question is whether it is possible to remove this assumption. In conclusion, we believe our results shed greater light on the relationship between the discounted and average reward settings as well as the fundamental complexity of the discounted setting, and we hope that our technical developments can be useful in future work, such as leading to efficient optimal algorithms in the online setting.

Acknowledgments and Disclosure of Funding

Y. Chen and M. Zurek were supported in part by National Science Foundation CCF-2233152 and DMS-2023239.

References

Alekh Agarwal, Sham Kakade, and Lin F. Yang. Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal, April 2020. arXiv:1906.03804 [cs, math, stat] version: 3.

Mohammad Gheshlaghi Azar, Remi Munos, and Bert Kappen. On the Sample Complexity of Reinforcement Learning with a Generative Model, June 2012. arXiv:1206.6461 [cs, stat].

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J. Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning , 91(3):325-349, June 2013.

Peter L. Bartlett and Ambuj Tewari. REGAL: A Regularization based Algorithm for Reinforcement Learning in Weakly Communicating MDPs, May 2012. arXiv:1205.2661.

Ronan Fruit, Matteo Pirotta, and Alessandro Lazaric. Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes, March 2019. arXiv:1807.02373 [cs, stat].

Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Ronald Ortner. Efficient Bias-SpanConstrained Exploration-Exploitation in Reinforcement Learning, July 2018. arXiv:1802.04020 [cs, stat].

Ying Jin, Ramki Gummadi, Zhengyuan Zhou, and Jose Blanchet. Feasible $Q$-Learning for Average Reward Reinforcement Learning. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages 1630-1638. PMLR, April 2024. ISSN: 2640-3498.

Yujia Jin and Aaron Sidford. Efficiently Solving MDPs with Stochastic Mirror Descent, August 2020. arXiv:2008.12776.

Yujia Jin and Aaron Sidford. Towards Tight Bounds on the Sample Complexity of Averagereward MDPs, June 2021. arXiv:2106.07046 [cs, math].

Michael Kearns and Satinder Singh. Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms. In Advances in Neural Information Processing Systems , volume 11. MIT Press, 1998.

David A. Levin and Yuval Peres. Markov Chains and Mixing Times . American Mathematical Soc., October 2017.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model. In Advances in Neural Information Processing Systems , volume 33, pages 12861-12872. Curran Associates, Inc., 2020.

Tianjiao Li, Feiyang Wu, and Guanghui Lan. Stochastic first-order methods for average-reward Markov decision processes, September 2024. arXiv:2205.05800.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, August 2014.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin Yang, and Yinyu Ye. Near-Optimal Time and Sample Complexities for Solving Markov Decision Processes with a Generative Model. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Matthew J. Sobel. The variance of discounted Markov decision processes. Journal of Applied Probability , 19(4):794-802, December 1982. Publisher: Cambridge University Press.

Jean Tarbouriech, Matteo Pirotta, Michal Valko, and Alessandro Lazaric. A Provably Efficient Sample Collection Strategy for Reinforcement Learning, November 2021. arXiv:2007.06437 [cs, stat].

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 1 edition, February 2019.

Martin J. Wainwright. Variance-reduced $Q$-learning is minimax optimal, August 2019. arXiv:1906.04697 [cs, math, stat].

Jinghan Wang, Mengdi Wang, and Lin F. Yang. Near Sample-Optimal Reduction-based Policy Learning for Average Reward MDP, December 2022. arXiv:2212.00603 [cs].

Shengbo Wang, Jose Blanchet, and Peter Glynn. Optimal Sample Complexity of Reinforcement Learning for Mixing Discounted Markov Decision Processes, September 2023. arXiv:2302.07477.

Shengbo Wang, Jose Blanchet, and Peter Glynn. Optimal Sample Complexity for Average Reward Markov Decision Processes, February 2024. arXiv:2310.08833.

Chen-Yu Wei, Mehdi Jafarnia-Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Modelfree Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes, February 2020. arXiv:1910.07072 [cs, stat].

Bin Yu. Assouad, Fano, and Le Cam. In Festschrift for Lucien Le Cam , pages 423-435. Springer, 1997.

Zihan Zhang and Xiangyang Ji. Regret Minimization for Reinforcement Learning by Evaluating the Optimal Bias Function, December 2019. arXiv:1906.05110 [cs, stat] version: 3.

Zihan Zhang and Qiaomin Xie. Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes, June 2023. arXiv:2306.16394 [cs].

A Proofs for weakly communicating MDPs

In this section, we provide the proofs for our main results in Section 3 for weakly communicating MDPs. Before beginning, we note that given that H ≥ 1 , we may assume that H is an integer by setting H ←⌈ H ⌉ , which only affects the sample complexity by a constant multiple < 2 relative to the original parameter H . Let ∥ M ∥ ∞→∞ := sup v : ∥ v ∥ ∞ ≤ 1 ∥ Mv ∥ ∞ denote the ℓ ∞ operator norm of a matrix M . Werecord the standard and useful fact that ∥ ∥ ( I -γP ′ ) -1 ∥ ∥ ∞→∞ ≤ 1 1 -γ for any transition probability matrix P ′ , which follows from the Neumann series ( I -γP ′ ) -1 = ∑ t ≥ 0 ( γP ′ ) t and the elementary fact that ∥ P ′ ∥ ∞→∞ ≤ 1 .

A.1 Technical lemmas

First we formally state the main theorem from [20], which gives a reduction from weakly communicating average-reward problems to discounted problems.

Lemma 9. Suppose ( P, r ) is an MDP which is weakly communicating and has an optimal bias function h ⋆ satisfying ∥ h ⋆ ∥ span ≤ H . Fix ε ∈ (0 , 1] and set γ = 1 -ε H . For any ε γ ∈ [0 , 1 1 -γ ] , if π is any ε γ -optimal policy for the discounted MDP ( P, r, γ ) , then

From here, we will first establish lemmas which are useful for proving Theorem 1 on discounted MDPs, and then we will apply the reduction approach of Lemma 9 to prove Theorem 2 on averagereward MDPs. As mentioned in the introduction, a key technical component of our approach is to establish superior bounds on a certain instance-dependent variance quantity which replace a factor of 1 1 -γ with a factor of H . Before reaching this step however, to make use of such a bound, we require an algorithm for discounted MDPs which enjoys a variance-dependent guarantee.

The work [12] obtains bounds with variance dependence that suffice for our purposes. However, they do not directly present said variance-dependent bounds, so we must slightly repackage their arguments in the form we require.

Lemma 10. There exist absolute constants c 1 , c 2 such that for any δ ∈ (0 , 1) , if n ≥ c 2 1 -γ log ( SA (1 -γ ) δε ) , then with probability at least 1 -δ , after running Algorithm 1, we have

and

Proof. First we establish equation (1). The proof of [12, Lemma 1] shows that when n ≥ 16 e 2 1 -γ 2 log ( 4 S log e 1 -γ δ ) , with probability at least 1 -δ we have

Now since

we can obtain equation (1) by triangle inequality (although we will choose the constant c 1 below). Next we establish equation (2). Using [12, Lemma 6], with probability at least 1 -δ we have that

̸

uniformly over all s and all a = ̂ π ⋆ γ, p ( s ) . From this separation condition (4), the assumptions of [12, Lemma 5] hold (with ω = εδ (1 -γ ) 2 18 SA 2 in their notation) for the MDP with the perturbed reward ˜ r . The proof of [12, Lemma 5] shows that under the event (4) holds, the conditions for [12, Lemma 2] are satisfied (with, in their notation, β 1 = 2log ( 32 (1 -γ ) 2 ωδ SA log e 1 -γ ) = 2log ( 576 S 2 A 3 (1 -γ ) 4 δ 2 ε log e 1 -γ ) ) with additional failure probability ≤ δ . The proof of [12, Lemma 2] then shows that, assuming n > 16 e 2 1 -γ 2 log ( 576 S 2 A 3 (1 -γ ) 4 δ 2 ε log e 1 -γ ) , we have

where we abbreviated β 1 = 2log ( 576 S 2 A 3 (1 -γ ) 4 δ 2 ε log e 1 -γ )

for notational convenience. We can again calculate that

Finally, to choose the constants c 1 and c 2 , we first note that 2 log ( 4 S log e 1 -γ δ ) ≤ β 1 < c ′ 1 log ( SA (1 -γ ) δε ) for some absolute constant c ′ 1 , and therefore also all our requirements on n are fulfilled when n ≥ 16 e 2 1 -γ c ′ 1 log ( SA (1 -γ ) δε ) = c ′ 2 1 -γ log ( SA (1 -γ ) δε ) for another absolute constant c ′ 2 . Lastly we note that by the union bound the total failure probability is at most 3 δ , so to obtain a failure probability of δ ′ we may set δ = δ ′ / 3 and absorb the additional constant when defining c 1 , c 2 in terms of c ′ 1 , c ′ 2 , and we also then increase c 1 by a factor of 4 to absorb the factor of 4 appearing in the first terms within (3) and (5).

Now we can analyze the variance parameters

which appear in the error bounds in Lemma 10. We begin by reproducing the following inequality from [23, Lemma 2].

Lemma 11. In a weakly communicating MDP, for all γ ∈ [0 , 1) , it holds that

The following relates the variance parameter of interest to another parameter, the variance of the total discounted rewards. This result essentially appears in [1, Lemma 4] (which was in turn inspired by [3, Lemma 8]), but since their result pertains to objects slightly different than P π and V P π [ V π γ ] , we provide the full argument for completeness.

Lemma 12. For any deterministic stationary policy π , we have

Proof. First we note the well-known variance Bellman equation (see for instance [16, Theorem 1]):

Nowwecanbasically identically follow the argument of [1, Lemma 4]. The matrix (1 -γ )( I -γP π ) -1 has rows which are each probability distributions (are non-negative and sum to 1 ). Therefore, by Jensen's inequality and the concavity of the function x ↦→ √ x , for each row s ∈ S we have

Using this fact we can calculate that, abbreviating v = V P π [ V π γ ] ,

In order to relate ∥ ∥ ( I -γP π ) -1 v ∥ ∥ ∞ to ∥ ∥ ( I -γ 2 P π ) -1 v ∥ ∥ ∞ in order to apply the variance Bellman equation (6), we calculate

Combining these calculations with the variance Bellman equation (6), we conclude that

as desired.

The following is a multi-step version of the variance Bellman equation, which we will later apply with T = H but holds for arbitrary T .

Lemma 13. For any integer T ≥ 1 , for any deterministic stationary policy π , we have

and consequently

Proof. Fix a state s 0 ∈ S . Letting F T be the σ -algebra generated by ( S 1 , . . . , S T ) , we calculate that

Using the above shorthands and opening the square, we obtain

where we used the tower property, the Markov property, and the fact that E π s 0 [ B |F T ] = 0 (which is immediate from the definition of V π γ ). Since e ⊤ s 0 P T π is a probability distribution, it follows from Holder's inequality that ∣ ∣ e ⊤ s 0 P T π V π [ ∑ ∞ t =0 γ t R t ] ∣ ∣ ≤ ∥ V π [ ∑ ∞ t =0 γ t R t ] ∥ ∞ . Therefore, it holds that

and we can obtain the desired conclusion after rearranging terms.

We also need the following elemetary inequality.

Lemma 14. If γ ≥ 1 -1 T for some integer T ≥ 1 , then

Proof. Fixing T ≥ 1 , we have

which is increasing in γ , so inf γ ≥ 1 -1 T 1 -γ 2 T 1 -γ is attained at γ = 1 -1 T . Now allowing T ≥ 1 to be arbitrary, note 1 -( 1 -1 T ) 2 T 1 -( 1 -1 T ) = T ( 1 -( 1 -1 T ) 2 T ) so it suffices to show that 1 -( 1 -1 T ) 2 T ≥ 1 -e 2 for all T ≥ 1 . By computing the derivative, one finds that 1 -( 1 -1 T ) 2 T is monotonically decreasing, so

We can now provide a bound on the variance of the total discounted rewards under π ⋆ γ .

Lemma15. Letting π ⋆ γ be the optimal policy for the weakly communicating discounted MDP ( P, r, γ ) , if γ ≥ 1 -1 H , we have

Proof. By using the multi-step variance Bellman equation in Lemma 13, it suffices to bound the quantity ∥ ∥ ∥ V π ⋆ γ [ ∑ H -1 t =0 γ t R t + γ H V π ⋆ γ γ ( S H ) ]∥ ∥ ∥ ∞ .

Fixing a state s 0 ∈ S ,

where in the final inequality we used Lemma 11. Taking the maximum over all states s and combining with Lemma 13 we obtain

Combining this bound with the elementary inequality in Lemma 14, which can be rearranged to show that 1 1 -γ 2 H ≤ 5 4 1 (1 -γ ) H , we complete the proof.

We also need to control the variance under ̂ π ⋆ γ, p , which requires additional steps. This is done in the following lemma.

Proof. In light of the multi-step variance Bellman equation in Lemma 13, it suffices to give a bound on ∥ ∥ ∥ V ̂ π ⋆ γ, p [ ∑ H -1 t =0 γ t ˜ R t + γ H V ̂ π ⋆ γ, p γ, p ( S H ) ]∥ ∥ ∥ ∞ . We have for any state s 0 that

where we have used triangle inequality and the inequalities ( a + b ) 2 ≤ 2 a 2 +2 b 2 and ( a + b + c ) 2 ≤ 3 a 2 +3 b 2 +3 c 2 . Now we bound each term of (7). First, we have

where we had (1 -γ ) ε 6 ≤ ε 6 H ≤ 1 6 because 1 1 -γ ≥ H and ε ≤ H . Clearly it holds that

By an argument identical to those used in the proof of the error bounds in Lemma 10, we get

so 6 γ 2 H ∥ ∥ ∥ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ 2 ∞ ≤ ε 2 6 ≤ H 2 6 since ε ≤ H . Finally, using Lemma 11, we obtain

Using all these bounds in (7), we have

Finally, we use the elementwise inequality

from which it follows that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ + ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ . Combining this with (8), we conclude

Now combining with Lemma 13 and then using Lemma 14, we have

as desired.

A.2 Proofs of Theorems 1 and 2

With the above lemmas we can complete the proof of Theorem 1 on discounted MDPs.

Proof of Theorem 1. Our approach will be to utilize our variance bounds within the error bounds from Lemma 10. We will find a value for n which guarantees that ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ and ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ are both ≤ ε/ 2 , which guarantees that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ ε .

First we note that the conclusions of Lemma 10 require n ≥ c 2 1 -γ log ( SA (1 -γ ) δε ) so we assume n is large enough that this holds.

Now we bound ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ . Starting with inequality (1) from Lemma 10 and then applying our variance bounds through Lemma 12 and then Lemma 15, we have

where in the last inequality we used the facts that ∥ ∥ ∥ V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ 1 1 -γ and γ ≤ 1 . Now if we assume n ≥ 360 c 1 H (1 -γ ) 2 ε 2 log ( SA (1 -γ ) δε ) , we have

due to the fact that ε ≤ H .

Next, to bound ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ , starting from inequality (2) in Lemma 10 and then analogously applying Lemma 12 and then Lemma 16, we obtain

Combining with the fact from above that ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ H 2 , as well as the facts that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ, p ∥ ∥ ∥ ∞ ≤ 1 1 -γ , γ ≤ 1 , and √ a + b ≤ √ a + √ b , we have

Rearranging terms gives

Assuming n ≥ 120 c 1 H (1 -γ ) 2 ε 2 log ( SA (1 -γ ) δε ) , we have

since ε ≤ H . Also assuming n ≥ (75 / 2) · 24 2 c 1 H (1 -γ ) 2 ε 2 log ( SA (1 -γ ) δε ) we have similarly to before that

Combining these two calculations, we have 1 2 ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ε 4 , so ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ε 2 as desired.

Since we have established that ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ , ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ε 2 , since also ̂ V ̂ π ⋆ γ, p γ, p ≥ ̂ V π ⋆ γ γ, p , we can conclude that

that is that ̂ π ⋆ γ, p is ε -optimal for the discounted MDP ( P, r, γ ) .

We finally note that all our requirements on the size of n can be satisfied by requiring

where we used that H (1 -γ ) 2 ε 2 ≥ H 2 (1 -γ ) ε 2 ≥ 1 1 -γ (since 1 1 -γ ≥ H and H ≥ ε ).

We next use Theorem 1 to prove Theorem 2 on average-reward MDPs.

Proof of Theorem 2. Using Theorem 1 with target accuracy H and discount factor γ = 1 -ε 12 H , we obtain a H -optimal policy for the discounted MDP ( P, r, γ ) with probability at least 1 -δ as long as

which is satisfied when n ≥ C 1 H ε 2 log ( SA H δε ) for sufficiently large C 1 .

Applying Lemma 9 (with error parameter ε 12 since we have chosen γ = 1 -ε/ 12 H ), we have that

as desired.

A.3 Proof of Theorem 3

Proof of Theorem 3. Fix T, n ≥ 1 . First we define the instances M 0 and M 1 , which have parameters B and ε which we will choose later, using Figure 2. Note that in both MDPs, all states have only one action. The only difference is in the state transition distribution at state 1 : For M 0 this is a Cat ( 1 2 , 1 2 ) distribution and for M 1 this is a Cat ( 1 2 + ε, 1 2 -ε ) distribution, where Cat ( p 1 , p 2 ) denotes the categorical distribution with event probabilities p 1 and p 2 = 1 -p 1 .

1

2

3

4

R

=

1

2

1

2

1

2

R

= 1

R

= 0

R

=

1

2

1

-

1

B

1

B

Instance

M

0

1

2

3

4

R

=

1

2

1

2

+

ε

1

2

-

ε

R

= 1

R

= 0

R

=

1

2

1

-

1

B

1

B

Instance

M

1

Now we calculate the bias of instance M 1 . It is easy to check the stationary distribution is µ = [ 1 2 , 1 4 + ε 2 , 1 4 -ε 2 , 0] . Therefore it has optimal gain ρ ⋆ = 1 2 1 2 + 1 4 + ε 2 = 1 2 + ε 2 . Now we claim that the optimal bias is

We can check this by showing that µh ⋆ = 0 and that ρ ⋆ 1 + h ⋆ = r + Ph ⋆ , where P is the transition matrix of the above MDP (again, note that each state has only one action, so there is only one policy, and we use this policy to induce the markov chain with transition matrix P ). First,

It is also easy to check the first three rows of the equality ρ ⋆ 1 + h ⋆ = r + Ph ⋆ . For the fourth row, we have

Thus ∥ h ⋆ ∥ span = 1 2 -ε/ 2 -( -( B +1) ε 2 ) = 1 2 ( Bε +1) . If we set B = 2 T ε -1 2 , we have ∥ h ⋆ ∥ span = T . Also note that the calculation for h ⋆ holds for any ε , so the optimal bias span of M 0 is [0 , 1 2 , -1 2 , 0] ⊤ , and thus M 0 has optimal bias span 1 .

Finally, to distinguish between the two MDPs M 0 and M 1 , we must be able to determine the next-state distribution of state 1 , that is, to distinguish between the two hypotheses Q 1 = Cat ( 1 2 , 1 2 ) and Q 2 = Cat ( 1 2 + ε, 1 2 -ε ) . Given n i.i.d. observations from the transition distribution of state 1 , this is a binary hypothesis testing problem between the product distributions Q n 1 and Q n 2 . By Le

Cam's bound [24], the testing failure probability is lower bounded by

where ∥ Q n 1 -Q n 2 ∥ TV and DKL ( Q n 1 | Q n 2 ) denote the total variation distance and Kullback-Leibler (KL) divergence between Q n 1 and Q n 2 , respectively, and the last two (in)equalities follow from Pinsker's inequality and tensorization of KL divergence. By direct calculation, we have

Combining the last two equations, we see that the testing failure probability is at least 1 2 ( 1 -√ 4 nε 2 ) . Thus, if we set ε = 1 4 √ n , the failure probability is at least 1 4 .

B Proofs for general MDPs

In this section, we provide the proofs for our main results in Section 4 for general MDPs. Again, we can assume that H + B is an integer, which only affects the sample complexity by a constant multiple < 2 .

First we develop more notation which will be useful in the setting of general MDPs. Recall we defined, for any policy π , that R π is the set of states which are recurrent in the Markov chain P π , and T π = S \ R π is the set of transient states. We now present a standard decomposition of Markov chains [14, Appendix A]. For any policy π , possibly after reordering states so that the recurrent states appear first (and are grouped into disjoint irreducible closed sets), we can decompose

such that X π are probabilities of transitions between states which are recurrent under π , Y π are probabilities of transitions from T π into R π , and Z π are probabilities of transitions between states within T π . Furthermore, supposing there are k irreducible closed blocks within R π , X π is blockdiagonal of the form

The limiting matrix of the Markov chain induced by policy π is defined as the matrix

P ∞ π is a stochastic matrix (all rows positive and sum to 1 ) since S is finite. We also have P π P ∞ π = P ∞ π = P ∞ π P π . Additionally, ρ π = P ∞ π r π . In terms of our decomposition, we have

where

each X ∞ π,i = 1 x ⊤ π,i for some stochastic row vector x ⊤ π,i , and Y ∞ π = ( I -Z π ) -1 Y π X ∞ π . Also we have ( I -Z π ) -1 = ∑ ∞ t =0 Z t π , and ∑ ∞ t =0 Z t π Y π = ( I -Z π ) -1 Y π has stochastic rows (each row is a probability distribution, that is all entries are positive and sum to 1 ).

With the same arrangement of states as within the above decomposition of P π (10), let

decompose V π γ into recurrent and transient states, and generally we use this same notation for any vector x ∈ R S : we let x list the values of x s for recurrent x ∈ R π , x contain x s for s ∈ T π , and we assume the entire x has been rearranged so that x = [ x x ] ⊤ . Note that the rearrangement of states depends on the policy π so this notation has potential for confusion if applied to objects relating to multiple policies at once, but the policy determining the rearrangement will always be clear from context in our arguments.

The main reason we decompose P π into recurrent and transient states is the following key observation. Lemma 17. For any policy π , if s, s ′ are in the same recurrent block of the Markov chain with transition matrix P π , then ρ ⋆ ( s ) = ρ ⋆ ( s ′ ) .

Proof. Define the history-dependent policy ˜ π which follows π until its history first contains s ′ , after which point it follows π ⋆ . Since ρ ⋆ ( s ) is the optimal gain achievable starting at s by following any history-dependent policy [14], we have ρ ⋆ ( s ) ≥ ρ ˜ π ( s ) := lim T →∞ 1 T E ˜ π s ∑ T -1 t =0 R t (where E ˜ π s is defined in the natural way from the distribution over trajectories ( S 0 , A 0 , . . . ) where A t ∼ ˜ π ( S 0 , A 0 , . . . , S t ) and S t +1 ∼ P ( · | S t , A t ) ). Let T s ′ = inf { t ≥ 1 : S t = s ′ } be the hitting time of state s ′ and let F T s ′ be the stopped σ -algebra (with respect to the filtration where for all nonnegative integers t , F t is the σ -algebra generated by S 0 , A 0 , . . . , S t , A t ). Then

where g ( T, k ) := E π ⋆ s ′ [ ∑ T -k -1 t =0 R t ] , and we used the tower property, F T s ′ -measurability of ∑ T s ′ -1 t =0 R t , the strong Markov property, and the definition of ˜ π . Now note that T s ′ < ∞ almost surely since s and s ′ are in the same recurrent block, and on the event { T s ′ = k } for any natural number k , we have that

because we can bound

and both sides converge to ρ ⋆ ( s ′ ) . Therefore g ( T,T s ′ ) T converges almost surely to the constant ρ ⋆ ( s ′ ) , and also this random variable is bounded by 1 , so by the dominated convergence theorem we have

Thus we have shown that ρ ⋆ ( s ) ≥ ρ ⋆ ( s ′ ) . Since s and s ′ were arbitrary states in the same recurrent block we also have ρ ⋆ ( s ′ ) ≥ ρ ⋆ ( s ) , and thus ρ ⋆ ( s ) = ρ ⋆ ( s ′ ) as desired.

Lemma 18. For any state s which is transient under a policy π , if the MDP satisfies the bounded transient time assumption with parameter B , we have

Proof. Let T = inf { t : S t ∈ R π } . Notice that ∥ ∥ e ⊤ s Z t π ∥ ∥ 1 = P π s ( T > t ) . Therefore, we have

where we used a well-known formula for the expectation of nonnegative-integer-valued random variables, and the bounded transient time assumption.

Lemma 19. Let s be a transient state under P π . Then

Proof. Using the decomposition of P π , we can calculate for any integer t ≥ 1 that

Therefore, we have

Note that we are able to rearrange the order of the summation in the third equality because all summands are (elementwise) positive.

B.1 Proof of Theorem 6

Theorem 6, our result which helps reduce general average reward MDPs to discounted MDPs, is proven as a straightforward consequence of the following sequence of lemmas, some of which will also be needed for the proof of our discounted MDP sample complexity bound Theorem 7.

Lemma 20. We have

Proof. We begin by observing that π ⋆ satisfies

Therefore, it holds that

Since P π ⋆ ρ ⋆ = ρ ⋆ , we can calculate that

It also holds that

and ∑ t ≥ 0 γ t +1 -γ t = ( γ -1) ∑ t ≥ 0 γ t = -1 . Therefore (12) is the difference of two stochastic matrices, and so it follows that

Lemma 21. If π ⋆ γ is optimal for the discounted MDP ( P, r, γ ) and s is recurrent under π ⋆ γ , then

and

These facts can be written as ∥ ∥ ∥ ∥ V π ⋆ γ γ -1 1 -γ ρ ⋆ ∥ ∥ ∥ ∥ ∞ ≤ ∥ h ⋆ ∥ span and ∥ ∥ ∥ ∥ V π ⋆ γ γ -1 1 -γ ρ π ⋆ γ ∥ ∥ ∥ ∥ ∞ ≤ 2 ∥ h ⋆ ∥ span respectively.

Proof. First note that if s is recurrent for the Markov chain P π ⋆ γ , then all states in the support of e ⊤ s P π ⋆ γ are in the same recurrent block as state s , and ρ ⋆ is constant (and equal to ρ ⋆ ( s ) ) within this recurrent block by Lemma 17. The (unmodified) Bellman equation states that

Since we established that e ⊤ s P π ⋆ γ ρ ⋆ = ρ ⋆ ( s ) , all actions a in the support of π ⋆ γ ( a | s ) satisfy P sa ρ ⋆ = ρ ⋆ ( s ) , and therefore

Since this holds for all s ∈ R π ⋆ γ , we can rearrange to obtain that

Now we can follow an argument which is similar to that of [23, Lemma 2]. We have

using monotonicity of ( I -X π ⋆ γ ) -1 in the final inequality. Due to the observation above that for all s ∈ R π ⋆ γ , all actions a in the support of π ⋆ γ ( a | s ) satisfy P sa ρ ⋆ = ρ ⋆ ( s ) , we have X π ⋆ γ ρ ⋆ = ρ ⋆ . Therefore we have

For the second term, by using an argument which is completely analogous to that used in Lemma 20 we have ∥ ∥ ∥ ( I -X π ⋆ γ ) -1 ( h ⋆ -X π ⋆ γ h ⋆ )∥ ∥ ∥ ∞ ≤ ∥ h ⋆ ∥ span . Combining these steps we obtain that

To obtain a lower bound, we can combine the optimality of π ⋆ γ for the γ -discounted problem with Lemma 20 to obtain the bound

Therefore we can conclude that ∥ ∥ ∥ ∥ V π ⋆ γ γ -1 1 -γ ρ ⋆ ∥ ∥ ∥ ∥ ∞ ≤ ∥ h ⋆ ∥ span .

For the second bound in the lemma statement, we first note that, as observed in [20],

Also, as discussed previously, if s ∈ R π ⋆ γ then e ⊤ s P π ⋆ γ ρ ⋆ = ρ ⋆ ( s ) , so then we also have e ⊤ s P ∞ π ⋆ γ ρ ⋆ = ρ ⋆ ( s ) (which can be seen directly from the definition of the limiting matrix P ∞ π ⋆ γ ). Equivalently, e ⊤ s ( I -P ∞ π ⋆ γ ) ρ ⋆ = 0 . Using both of these two observations, we have

Therefore, we obtain

using the first bound from the lemma statement in the final inequality.

Lemma 22. We have and

and

Proof. Note that by combining with Lemma 21, it suffices to prove for any transient state s ∈ T π ⋆ γ that

Let s be transient under π ⋆ γ . Then starting by using Lemma 19, we can calculate

Now we can obtain the two bounds in the lemma statement by bounding the second term of (13) in two different ways. For the first bound in the lemma statement, we can use the first bound in Lemma

21 to calculate that

where we used the fact that X ∞ π ⋆ γ ρ ⋆ = ρ ⋆ and then that e ⊤ s P ∞ π ⋆ γ ρ ⋆ ≤ ρ ⋆ ( s ) . This gives an upper bound of

Combining with the lower bound

which is the first bound in the lemma statement.

we obtain that

To obtain the second bound in the lemma statement, using the second bound from Lemma 21, we can calculate for the second term in (13) that

where in the second equality we used the fact that ( ∑ ∞ t =0 e ⊤ s Z t π ⋆ γ Y π ⋆ γ ) is a probability distribution, and in the final steps we used the decomposition of P ∞ π ⋆ γ and the fact that ρ π ⋆ γ = P ∞ π ⋆ γ r π ⋆ γ .

Therefore by combining these steps we obtain that

Combining with the lower bound

we obtain the desired bound

Lemma 23. If π satisfies V π γ ≥ V π ⋆ γ γ -δ 1 , then

Proof. Similar to the proof of Lemmas 21 and 22, we will first establish a bound for the states which are recurrent under π . Specifically, we will first show that if s is recurrent under π we have

Letting s ∈ R π , following steps which are similar to the proof of the second part of Lemma 21, we have

using the fact discussed in Lemma 21 that e ⊤ s ( I -P ∞ π ) ρ ⋆ = 0 since s is recurrent under π . Then by triangle inequality, we obtain

where we used the facts that ∥·∥ span ≤ 2 ∥·∥ ∞ and that V π ⋆ γ γ ≥ V π γ ≥ V π ⋆ γ γ -δ 1 .

Having established (14), we now extend to transient states using arguments similar to those for the second bound of Lemma 22. Let s be transient under π . Then starting by using Lemma 19, we can calculate

using the bounded transient time assumption via Lemma 18 in the final step. Then we can calculate

where in the first equality we used the fact that ( ∑ ∞ t =0 e ⊤ s Z t π Y π ) is a probability distribution, in the second inequality we used the bound (14), and in the final steps we used the decomposition of P ∞ π and the fact that ρ π = P ∞ π r π .

Therefore by combining this last bound with the bound (15), we have

Combining with the lower bound

we conclude that as desired.

Proof of Theorem 6. Suppose π is ε γ -optimal for the discounted MDP ( P, r, γ ) . We can calculate that

where in the first inequality we used Lemma 23, in the second inequality we used the fact that π is ε γ -optimal, in the third inequality we used the optimality of π ⋆ γ for the discounted MDP, and in the final inequality we used Lemma 20. Therefore by mulitplying both sides by 1 -γ , we have that

B.2 Proof of Theorem 7 (Discounted MDP Bounds)

In this section, we provide our main result on the sample complexity of general discounted MDPs.

Our proof relies on three lemmas that provide bounds on relevant variance parameters. The first lemma controls the variance for π ⋆ on recurrent states.

γ

Lemma 24. Letting π ⋆ γ be the optimal policy for the discounted MDP ( P, r, γ ) , if γ ≥ 1 -1 B + H , we have

Proof. First, using the decomposition (10), we can calculate for any s ∈ R π ⋆ γ that

Also due to the decomposition, notice that set R π ⋆ γ is a closed set for the Markov chain with transition matrix P π ⋆ γ , and furthermore when restricting to the entries corresponding to this closed set we obtain the transition matrix X π ⋆ γ . Therefore we can apply Lemma 12 to this subchain to obtain that

Abbreviating L = B + H , we can also then apply Lemma 13 to bound

We can repeat a similar argument as within Lemma 15 to bound this term. Fixing an initial state s 0 ∈ R π ⋆ γ , the key observation is that ρ ⋆ is constant on the recurrent block of X π ⋆ γ containing s 0 , and therefore any state trajectory S 0 = s 0 , S 1 , S 2 , . . . under the transition matrix P π ⋆ γ will have ρ ⋆ ( S L ) = ρ ⋆ ( s 0 ) . Therefore for this fixed s 0 we have

where we used Lemma 21 in the penultimate inequality. Applying this argument to all s 0 ∈ R π ⋆ γ we obtain

Therefore by combining with our initial bounds we have that

where in the penultimate inequality we used Lemma 14 to bound 1 1 -γ 2 L ≤ 5 4 1 (1 -γ ) L .

The next lemma controls the variance for ̂ π ⋆ γ, p on recurrent states.

Lemma 25. Letting ̂ π ⋆ γ, p be the optimal policy for the discounted MDP ( ̂ P, ˜ r, γ ) , if γ ≥ 1 -1 B + H , we have

Proof. Let L = B + H . By the same arguments as in the beginning of the proof of Lemma 24, we have

so it again suffices to bound V ̂ π ⋆ γ, p [ ∑ L -1 t =0 γ t ˜ R t + γ L V ̂ π ⋆ γ, p γ, p ( S L ) ] . Fix s 0 ∈ R ̂ π ⋆ γ, p . Again, as observed in Lemma 24, ρ ⋆ is constant on the recurrent block of X ̂ π ⋆ γ, p containing s 0 , so we will have

ρ ⋆ ( S L ) = ρ ⋆ ( s 0 ) with probability one. Therefore (mostly following the steps of Lemma 16)

using the inequalities ( a + b + c ) 2 ≤ 3 a 2 +3 b 2 +3 c 2 and ( a + b ) 2 ≤ 2 a 2 +2 b 2 . Now we bound each term of (16) analogously to the steps of Lemma 16. For the first term of (16),

where we had (1 -γ ) ε 6 ≤ ε 6 L ≤ 1 6 because 1 1 -γ ≥ L and ε ≤ L . For the second term of (16),

where we used ( a + b ) 2 ≤ 2 a 2 +2 b 2 and the fact that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ + ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ which was shown in Lemma 16. For the third term of (16),

where the fact that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ξ 1 -γ is identical to the arguments used in the proof of Lemma 10, and the final inequality is due to the assumption that ε ≤ L . For the fourth term of (16),

using Lemma 22 for the second inequality. Using all these bounds in (16), we obtain

and so (since this holds for arbitrary s 0 ∈ R ̂ π ⋆ γ, p ), we have

Therefore, combining with our initial arguments,

where we used Lemma 14 to bound 1 -γ 2 L ≤ 4 (1 -γ ) L .

The next lemma controls the variance on all states.

Lemma 26. Under the settings of Lemmas 24 and 25, we have

and

Proof. First we establish the first bound in the lemma statement. As we have already bounded the entries corresponding to the recurrent states of π ⋆ γ by Lemma 24, it remains to bound the transient states. Let s ∈ T π ⋆ γ be an arbitrary transient state. Using Lemma 19, we have

Now we bound each of the terms in (17). For the first term, we can calculate

where we used the fact that e s ⊤ ∑ ∞ k =1 Z k -1 π ⋆ γ Y π ⋆ γ is a probability distribution and Lemma 24.

For the second term of (17), we have

where we used Jensen's inequality since x ↦→ √ x is concave and ∑ ∞ t =0 γ t e s ⊤ Z t π ⋆ γ ∥ ∥ ∥ ∥ e s ⊤ ∑ ∞ t =0 γ t Z t π ⋆ γ ∥ ∥ ∥ ∥ 1 is a probability distribution (all entries of this row vector are positive and they sum to 1 due to our normalization). Now we bound each factor in (18). Using Lemma 18, we have

For the second factor in (18), we have

where the equality step is due to Lemma 19. Now we can apply two steps which are used within Lemma 12 to obtain the desired bound on this term. Abbreviating v = V P π ⋆ γ [ V π ⋆ γ γ ] , it is shown within Lemma 12 that

(where the final inequality is because the total discounted return is within [0 , 1 1 -γ ] ). Therefore we can bound the second factor in (18) as

Combining all of these bounds back into (17), we have

Thus we have established the first inequality from the lemma statement.

For the second inequality, the argument is entirely analogous, except that we use Lemma 25 instead of Lemma 24, and also in the MDP with the perturbed reward ˜ r we have the bound

where we used the fact that (1 -γ ) ε 6 ≤ ε 6( B + H ) ≤ 1 6 because 1 1 -γ ≥ B + H and ε ≤ B + H . Thus we can obtain the bound

This completes the proof of the lemma.

We are now ready to prove Theorem 7 on the sample complexity of general discounted MDPs.

Proof of Theorem 7. To prove Theorem 7 we will combine our bounds of the variance parameters in Lemma 26 with Lemma 10. First, starting with (1) from Lemma 10 and combining with the first bound from Lemma 26, we have that there exist absolute constants c 1 , c 2 such that for any δ ∈ (0 , 1) ,

if n ≥ c 2 1 -γ log ( SA (1 -γ ) δε ) , then with probability at least 1 -δ

where the penultimate inequality is under the assumption that n ≥ 16 · 6 2 c 1 B + H ε 2 (1 -γ ) 2 log ( SA (1 -γ ) δε ) , and the final inequality makes use of the fact that ε ≤ B + H . Next, still using Lemma 10, under the same event, we also have

using the second inequality from Lemma 26 for the second inequality, and then we use the fact that ∥ ∥ ∥ V ̂ π ⋆ γ, p γ, p ∥ ∥ ∥ ∞ ≤ 7 6 1 1 -γ which was argued in Lemma 26, as well as the fact from above that

If n ≥ 6 2 · 10 2 c 1 B + H ε 2 (1 -γ ) 2 log ( SA (1 -γ ) δε ) , then the RHS of (19) is bounded by

using the assumption that ε ≤ B + H . Under the same condition on n , we also have that

where again we used the assumption that ε ≤ B + H . Combining these two bounds with the inequality (19), we obtain that

Since we have established that ∥ ∥ ∥ ̂ V π ⋆ γ γ, p -V π ⋆ γ γ ∥ ∥ ∥ ∞ ≤ ε 2 and that ∥ ∥ ∥ ̂ V ̂ π ⋆ γ, p γ, p -V ̂ π ⋆ γ, p γ ∥ ∥ ∥ ∞ ≤ ε 2 , since also ̂ V ̂ π ⋆ γ, p γ, p ≥ ̂ V π ⋆ γ γ, p , we can conclude that

that is that ̂ π ⋆ γ, p is ε -optimal.

Finally, we check that all of our conditions on n can be satisfied if

and since 1 1 -γ ≥ B + H and B + H ≥ ε , we have B + H ε 2 (1 -γ ) 2 ≥ ( B + H ) 2 ε 2 (1 -γ ) ≥ 1 1 -γ , so the above is guaranteed if we set C 3 = max { 6 2 · 10 2 c 1 , c 2 } and require n ≥ C 3 B + H ε 2 (1 -γ ) 2 log ( SA (1 -γ ) δε ) .

which implies that

B.3 Proof of Theorem 8 (General Average-Reward MDP Bounds)

In this section, we prove our main result on the sample complexity of general average-reward MDPs.

Proof of Theorem 8. Wecan combine our bound for discounted MDPs, Theorem 7, with our reduction from average-reward MDPs to discounted MDPs, Theorem 6.

Using Theorem 7 with target accuracy B + H and discount factor γ = 1 -ε 12( B + H ) , we obtain a ( B + H ) -optimal policy for the discounted MDP ( P, r, γ ) with probability at least 1 -δ as long as

which is satisfied when n ≥ C 4 B + H ε 2 log ( SA ( B + H ) δε ) for sufficiently large C 4 .

Applying Theorem 6 (with error parameter ε 12 ), we obtain

as desired.

B.4 Proof of Theorems 4 and 5 (Lower Bounds)

In this section, we prove our minimax lower bounds on the sample complexity of general averagereward MDPs (Theorem 4) and discounted MDPs (Theorem 5).

Proof of Theorem 4. First consider the MDP instances M a ⋆ indexed by a ⋆ ∈ { 1 , . . . , A } shown in Figure 3. In all instances, states 2 , 3 and 4 are absorbing states, and state 1 is a transient state. State 1 has A actions and is the only state with multiple actions. At state 1 , taking action a = 1 will take the agent to state 4 deterministically; taking action 2 will take the agent back to state 1 with probability P (1 | 1 , 2) = 1 -1 T , to state 2 with probability P (2 | 1 , 2) , and to state 3 with probability P (3 | 1 , 2) = 1 -P (1 | 1 , 2) -P (2 | 1 , 2) . The instances differ only in the values of P (2 | 1 , a ) and P (3 | 1 , a ) , which are shown in Figure 3 along with the reward R for each state-action pair.

̸

For the MDP instance M 1 , the optimal policy is taking action a = 1 at state 1 , leading to an average reward of 1 / 2 ; taking any other action leads to a sub-optimal average reward of 1 -2 ε 2 . Similarly, for the instance M a ⋆ with a ⋆ ∈ { 2 , . . . , A } , the optimal action is a = a ⋆ with average reward 1+2 ε 2 , the action a = 1 has average reward 1 2 , and all other actions have average reward 1 -2 ε 2 . By direct calculation, we find that the span of the optimal policy is ∥ h ⋆ ∥ span = 0 in all instances. Moreover, by taking any action a = 1 , the agent will stay in state 1 for B steps in expectation before transitioning to state 2 or 3 , so the bounded transient time is satisfied with parameter B.

Wenext define ( A -1) S/ 4 master MDPs M s ⋆ ,a ⋆ indexed by s ⋆ ∈ { 1 , . . . , S/ 4 } and a ⋆ ∈ { 2 , . . . , A } as follows. Each master MDP M s ⋆ ,a ⋆ has S/ 4 copies of sub-MDPs such that the s ⋆ th sub-MDP is equal to M a ⋆ and all other sub-MDPs are equal to M 1 . We rename the states so that the states of the s th sub-MDP has states 4 s +1 , 4 s +2 , 4 s +3 , 4 s +4 corresponding to states 1 , 2 , 3 , 4 of the instances shown in Figure 3. Note each of these master MDPs has S states and A actions, satisfies the bounded transient time property with parameter B , and has the span of the bias of its Blackwell optimal policy equal to 0 . Note that for a given policy π to be ε/ 3 -average optimal in master MDP M s ⋆ ,a ⋆ , it must take action a ⋆ in state 4 s ⋆ +1 with probability at least 2 / 3 , and it must take action 1 in states 4 s +1 for s ∈ { 1 , . . . , S/ 4 } \ { s ⋆ } with probability at least 2 / 3 .

̸

Thus, for an algorithm Alg to output an ε/ 3 -average optimal policy π , it must identify the master MDP instance M s ⋆ ,a ⋆ (equivalently, the values of s ⋆ and a ⋆ ), in the sense that there must be exactly one state 4 s +1 where an action a = 1 is taken with probability ≥ 2 / 3 . Therefore it suffices to lower bound the failure probability of any algorithm Alg for this ( A -1) S/ 4 -way testing problem. By

1

2

3

4

a

∈ {

2

, . . . , A

}

, R

= (1 + 2

ε

)

/

2

P

(1

|

1

, a

) = 1

-

1

B

P

(2

|

1

, a

) =

1

-

2

ε

2

B

P

(3

|

1

, a

) =

1+2

ε

2

B

a

= 1

, R

= 1

/

2

R

= 1

R

= 0

R

= 1

/

2

Instance

M

1

1

2

3

4

a

∈ {

2

, . . . , A

} \ {

a

⋆

}

, R

= (1 + 2

ε

)

/

2

P

(1

|

1

, a

) = 1

-

1

B

P

(2

|

1

, a

) =

1

-

2

ε

2

B

P

(3

|

1

, a

) =

1+2

ε

2

B

a

= 1

, R

= 1

/

2

R

= 1

R

= 0

R

= 1

/

2

a

=

a

⋆

, R

= (1 + 2

ε

)

/

2

P

(1

|

1

, a

⋆

) = 1

-

1

B

P

(2

|

1

, a

⋆

) =

1+2

ε

2

B

P

(3

|

1

, a

⋆

) =

1

-

2

ε

2

B

Instance

M

a

⋆

, for

a

⋆

∈ {

2

, . . . , A

}

construction, for any two distinct index pairs ( s ⋆ 1 , a ⋆ 1 ) and ( s ⋆ 2 , a ⋆ 2 ) , the master MDPs M s ⋆ 1 ,a ⋆ 1 and M s ⋆ 2 ,a ⋆ 2 differ only in the state-action pairs (4 s ⋆ 1 , a ⋆ 1 ) and (4 s ⋆ 2 , a ⋆ 2 ) , and we have

where Cat ( p 1 , p 2 , p 3 ) denotes the categorical distribution with event probabilities p i 's (and vice versa for the distributions of the state action pair (4 s ⋆ 2 , a ⋆ 2 ) ).

Now we use Fano's method [18] to lower bound this failure probability. Choose an index J uniformly at random from the set J := { 1 , . . . , S/ 4 } × { 2 , . . . , A } and suppose that we draw n iid samples X = ( X 1 , . . . , X n ) from the master MDP M J ; note that under the generative model, each random variable X i represents an ( S × A ) -byS transition matrix with exactly one nonzero entry in each row. Letting I ( J ; X ) denote the mutual information between J and X , Fano's inequality yields that the failure probability is lower bounded by

We can calculate using the fact that the P i 's are i.i.d., the chain rule of mutual information, and the form of the construction that

̸

By direct calculation, we have

Also note that DKL ( Q 2 | Q 1 ) = DKL ( Q 1 | Q 2 ) in this case. Therefore the failure probability is at least

where in the second inequality we assumed A and S are at least a sufficiently large constant. For the above RHS to be smaller than 1 / 4 , we therefore require n ≥ Ω( B log( SA ) ε 2 ) .

Proof of Theorem 5. The desired DMDP lower bound follows from combining our AMDP lower bound Theorem 4 with the average-to-discount reduction in Theorem 6.

B.5 Relationship between transient time and mixing time

Lemma 27. In any uniformly mixing MDP, we have B ≤ 4 τ unif .

Proof. Fix a deterministic stationary policy π . Notice that since all states in the support of the stationary distribution ν π are recurrent, for any s ∈ S we have

where the final inequality uses standard properties of mixing [11, Chapter 4]. Now define T = inf { t : S t ∈ R π } . Then, using a standard formula for the expectation of nonnegative-integer-values random

variables, we have for any s ∈ S that

Since this bound holds for all s ∈ S and all deterministic stationary policies π , we conclude that B ≤ 4 τ unif .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction match the theoretical results provided in the main results Sections 3 and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The conclusion (Section 5) mentions the main limitation, of the necessity of knowledge of H / B for the optimal average-reward complexity results to hold, and this point is elaborated upon in Section 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are provided with their respective theorems and within the problem setup Section 2, and formal proofs of all results are provided in Appendices A and B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not involve any human subjects or datasets, and as a foundational theoretical paper it does not have any direct potentially harmful societal consequences.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is foundational research on the sample complexity of average-reward and discounted MDPs, and thus is not directly tied to any negative applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not provide any data nor models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use any code, model, nor data assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a model-based reinforcement learning algorithm for tabular MDPs with the average reward criteria. The authors assume a generative model (each state-action pairs can be simulated) and studies learning algorithm that sample each state-action pairs n times.  In the weakly communicating setting, the authors provides the first algorithm that achieves the minimax lower-bound to find an \\varepsilon-policy. The authors study also the more general case of multi-chain MDPs for which the authors introduce a parameter B (time to visite a recurrent-state) and propose both an analysis of a minimax lower bound and an algorithm that achieves this bound.\n\nSTRENGTHS:\nThe paper is really well written. The related work section show that the paper is well documented, and the authors make a good use of the related work via the various references throughout the text. The choice of the examples and their use is nicely done. All in all, the paper is nice to ready.\n\nThe paper improves on related work and obtain an algorithm that matches the lower bound.\n\nThe paper is very precise in its definitions (contrary to many papers on the same subjects).\n\nThe paper studies learning algorithm for multi-chain MDPs, which is rarely done.\n\nWEAKNESSES:\nThe algorithmic novelty of the paper seems limited. Only the analysis seems new.\n\nThe authors focus on a generative model (although this is probably unavoidable in the multichain case).\n\nTo me, the discussion on the lower bound is not complete.\n\nQUESTIONS:\nAre there any novelty in the algorithmic part or is it just the analysis that is new?\n\nThe use of a generative model drastically simplifies the analysis. Would any of these results translate to a navigating model?\n\nThe lower bound seem to imply that *all* state-action pairs have to be visited n times. I am surprized that the sampling of state-action pairs is not adaptive. Would it change anything?\n\nLIMITATIONS:\nNA.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work obtains the first minimax optimal sample complexity bound of weakly communicating and general average reward MDPs, without uniform mixing assumption, by introducing new transient time parameter and obtaining tighter minimax optimal sample complexity bound for discounted MDP.\n\nSTRENGTHS:\nThis theoretical work provides comprehensive lit review. I did not check the proof, but the theoretical results look reasonable and strong based on my knowledge about discounted MDP theory. The presentation is clear.\n\nWEAKNESSES:\nA few points are to be clarified as shown in the questions below.\n\nQUESTIONS:\n(1) You may change the following citation which has been accepted by ICLR 2024. \n\n[21] Shengbo Wang, Jose Blanchet, and Peter Glynn. Optimal Sample Complexity for Average Reward Markov Decision Processes, October 2023.\n\nSome other citations like [13] lack location (ArXiv, conference, journal, etc.). \n\n(2) In lines 152-153 about the definition of Blackwell-optimal 154 policy, do you mean for all $\\gamma\\in[\\overline{\\gamma},1)$ or $\\gamma\\in[\\overline{\\gamma},1]$? Does $V_{\\gamma}^{\\pi^*}\\ge V_{\\gamma}^{\\pi}$ mean $V_{\\gamma}^{\\pi^*}(s)\\ge V_{\\gamma}^{\\pi}(s), \\forall s$?\n\n(3) In lines 155-156, what does $P_{sa}\\rho^*$ mean? Does $\\rho^*(s)\\ge P_{\\pi}\\rho^*$ mean $\\rho^*(s)\\ge (P_{\\pi}\\rho^*)(s):=\\sum_{s\\in\\mathcal{S}}P_{\\pi}(s,s')\\rho^*(s')$? The meaning of $\\rho^*(s)\\ge {\\rm a~vector}$ is not clear to me.\n\n(4) In Theorem 2, the accuracy $\\overline{\\epsilon}=H$ for DMDP is not arbitrarily small. Why can the accuracy for AMDP be arbitrarily small $\\epsilon$?\n\nLIMITATIONS:\nI agree with the following two statements made in the checklist: \n\n(1) Limitations: \"The conclusion (Section 5) mentions the main limitation, of the necessity of knowledge of H/B for the optimal average-reward complexity results to hold, and this point is elaborated upon in Section 3.\" \n\n(2) Negative societal impact: \"Our work is foundational research on the sample complexity of average-reward and discounted MDPs, and thus is not directly tied to any negative applications.\"", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm with optimal sample complexity in general average reward MDPs.\n\nSTRENGTHS:\nThe algorithm proposed is sample optimal for the general class of MDPs (possibly  multichain) that are much harder to learn than uni-chain or ergodic MDPs. \nIt also introduces a new parameter, the transient time that helps one assess  the sample complexity for multi-chain MDPs.\n\nWEAKNESSES:\nAlthough the following papers only appear recently on Arxiv ( i am not an author of either of them)\nI think they should be mentioned because they answer some questions raised in the paper.\n\nFirst XXBoone shows that regret bounds  using H are possible without prior knowledge of H, disproving the supposed conejcture\n in papers [5,4,25] cited in this submission. Actually, the same paper answers the point about the computational eficiency of the optimal algorithm.\n\nSecond, the results of the current submission should be compared to XXKauffman which seems to solve the same problem.\n\nIn the ergodic case, i agree that navigating and generative models are almost similar but in the general case, the generative model looks very strong.\n\nQUESTIONS:\n1. The authors discuss the fact that H cannot estimated while D can be. However they do not mention anything about B ?\nMy first guess would be that B cannot be estimated either because it looks discontinuous in the paramaters on the MDP.\n\n2. Maybe I am mistaken but I did not see a proper definition of the transient time B, used in the statement of Theorems 4 and 5.\n\nLIMITATIONS:\nAgain, the generative model looks like a strong assumptions.\n\nThe lack of numerical experiments is classical in this domain. It seems that MDPs of size 10 are already impossible to learn, which limits strongly the practical aspect of this type of algorithms. Can the authors comment on this?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

comments: SUMMARY:
The paper resolves the open problem of designing an algorithm for the generative tabular average reward setting for weakly communicating MDPs that achieves optimal span-dependent sample complexity with known span. This is done by an original observation that is concerned with discounted MDPs: Existing sample complexity bounds for the discounted setting are refined and the result is obtained from this refinement by reducing the average reward setting to the discounted setting, just like it was done in previous works. A second result is to give the first sample complexity results for general MDPs; with matching lower and upper bounds.

STRENGTHS:
Solving a major open problem based on an interesting insight: This is a breakthrough paper.

WEAKNESSES:
None

QUESTIONS:
n.a.

LIMITATIONS:
n.a.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15