Title: Achieving Tractable Minimax Optimal Regret in Average Reward MDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Markov decision processes

Regret

Average reward

Minimax

Optimism

Model-based
Abstract: In recent years, significant attention has been directed towards learning average-reward Markov Decision Processes (MDPs).
However, existing algorithms either suffer from sub-optimal regret guarantees or computational inefficiencies.
In this paper, we present the first *tractable* algorithm  with minimax optimal regret of $\mathrm{O}\left(\sqrt{\mathrm{sp}(h^*) S A T \log(SAT)}\right)$ where $\mathrm{sp}(h^*)$ is the span of the optimal bias function $h^*$, $S\times A$  is the size of the state-action space and $T$ the number of learning steps. 
Remarkably, our algorithm does not require prior information on $\mathrm{sp}(h^*)$. 

Our algorithm relies on a novel subroutine, **P**rojected **M**itigated **E**xtended **V**alue **I**teration (`PMEVI`), to compute bias-constrained optimal policies efficiently. 
This subroutine can be applied to various previous algorithms to obtain improved regret bounds.

Achieving Tractable Minimax Optimal Regret in Average Reward MDPs

Abstract

In recent years, significant attention has been directed towards learning averagereward Markov Decision Processes (MDPs). However, existing algorithms either su ff er from sub-optimal regret guarantees or computational ine ffi ciencies. In this paper, we present the first tractable algorithm with minimax optimal regret of e O GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> , 1 where sp ( h ∗ ) is the span of the optimal bias function h ∗ , S × A is the size of the state-action space and T the number of learning steps. Remarkably, our algorithm does not require prior information on sp ( h ∗ ).

Our algorithm relies on a novel subroutine, P rojected M itigated E xtended V alue I teration ( PMEVI ), to compute bias-constrained optimal policies e ffi ciently. This subroutine can be applied to various previous algorithms to improve regret bounds.

1 Introduction

Reinforcement learning (RL) Burnetas and Katehakis [1997], Sutton and Barto [2018] has become a popular approach for solving complex sequential decision-making tasks and has recently achieved notable advancements in diverse fields of application. RL problems are generally formulated with Markov Decision Processes (MDPs) Puterman [1994], where a learning agent seeks to maximize the rewards that are gathered by interacting with an unknown environment.

This paper focuses on average reward MDPs where the learning agent must maximize the sum of rewards in the long run without any reset mechanism. In this setting, the proper balancing between exploration (i.e., playing sub-optimally to learn the unknown environment) and exploitation (i.e., planning optimally according to the current knowledge), usually known as the exploration-exploitation trade-o ff , is key to learn e ffi ciently. The measure of learning performance that we adopt throughout is the regret , that compares the aggregate rewards collected by the learning agent during the learning process to the expected performance of an omniscient agent that knows everything in advance. The seminal work of Auer et al. [2009] provides a minimax regret lower bound of Ω GLYPH<16> √ DSAT GLYPH<17> , where D is the diameter (the maximal distance between two di ff erent states), S the number of states, A the number of actions and T the learning horizon. They also provide an algorithm achieving regret e O GLYPH<16> √ D 2 S 2 AT GLYPH<17> , where e O( -). Ever since Auer et al. [2009], many works have been devoted to close the gap between the regret lower and upper bounds in the average reward setting Auer et al. [2009], Bartlett and Tewari [2009], Filippi et al. [2010], Talebi and Maillard [2018], Fruit et al. [2018, 2020], Bourel et al. [2020], Zhang and Ji [2019], Ouyang et al. [2017], Agrawal and Jia [2023],

Abbasi-Yadkori et al. [2019], Wei et al. [2020] and more. Subsequent works Fruit et al. [2018], Zhang and Ji [2019] refined the minimax regret lower bound to Ω GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> where sp ( h ∗ ) is the span of the bias function, which is the maximal gap of the long-term accumulative rewards starting from two di ff erent states. The di ff erence is significant, since sp ( h ∗ ) ≤ D and the gap between the two can be arbitrarily large. However, no existing work achieves the following three requirements simultaneously:

The method achieves minimax optimal regret guarantees e O GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> ;

The proposed method is tractable;

No prior knowledge on the model is required.

Most algorithms simply fail to achieve minimax optimal regret, and the only method achieving it Zhang and Ji [2019] is intractable because it relies an oracle to solve di ffi cult optimization problems along the learning process. Naturally, we raise the question of whether these three requirements can be met all at once:

Is there a tractable algorithm with e O GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> minimax regret without prior knowledge?

Contributions. In this paper, we answer the above question a ffi rmatively, by proposing a polynomial time algorithm with regret guarantees e O GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> for average-reward MDPs. Our method can further incorporate almost arbitrary prior bias information H∗ ⊆ R S to further improve its regret.

Theorem 1 (Informal) . Provided that the confidence region used by PMEVI-DT satisfy mild regularity conditions (see Assumption 1-3), then for every weakly communicating model M with sp( h ∗ ) ≤ T 1 / 5 and sp ( h ∗ ) ∈ H∗ , PMEVI-DT ( H∗ , δ, T ) achieves regret:

with probability 1 -26 δ . Moreover, if PMEVI-DT runs with the same confidence regions that UCRL2 Auer et al. [2009] on a communicating environment, it has a time complexity O( DS 3 AT ) .

Taking δ = √ 1 / T , we also obtain a e O GLYPH<16>p sp ( h ∗ ) S AT GLYPH<17> regret bound in expectation. The geometry of the prior bias region H∗ that PMEVI-DT can support is discussed later (see Assumption 4). It can be taken trivial with H∗ = R S to obtain a completely prior-less algorithm.

To the best of our knowledge, PMEVI-DT is the first tractable algorithm with minimax optimal regret bounds (up to logarithmic factors). The algorithm does not necessitate any prior knowledge of sp ( h ∗ ), thus circumventing the potentially high cost associated with learning sp ( h ∗ ). On the technical side, a key novelty of our method is the subroutine named PMEVI (see Algorithm 2) that improves and can replace EVI Auer et al. [2009] in any algorithm that relies on it Auer et al. [2009], Fruit et al. [2018], Filippi et al. [2010], Fruit et al. [2020], Bourel et al. [2020] to boost its performance and achieve minimax optimal regret.

Related works on average reward MDPs. For communicating MDPs, the notable work of Auer et al. [2009] proposes the famous UCRL2 algorithm, a mature version of their prior UCRL Auer and Ortner [2006], achieving a regret bound of e O( DS √ AT ). This paper pioneered the use optimistic methods to learn MDPs e ffi ciently. A line of papers Filippi et al. [2010], Fruit et al. [2020], Bourel et al. [2020] developed this direction by tightening the confidence region that UCRL2 relies on, and sharpened the analysis through the use of local properties of MDPs, such as local diameters and local bias variances. However, none of these works went beyond regret guarantees of order S √ DAT and su ff er from an extra √ S . A parallel direction was initiated by Bartlett and Tewari [2009] with REGAL , obtaining regret bounds scaling with sp( h ∗ ) instead of D , and extending the regret bounds to weakly-communicating MDPs in the mean time. The computational intractability of REGAL is addressed by Fruit et al. [2018] with SCAL , and regret guarantees are further improved by Zhang and Ji [2019] with EBF , eventually reaching optimal minimax regret but loosing tractability.

Another successful design approach is Bayesian-flavored sampling, derived from Thompson Sampling Thompson [1933], that usually replaces optimism. The regret guarantees of these algorithms usually stick to the Bayesian setting however Ouyang et al. [2017], Theocharous et al. [2017], although

Agrawal and Jia [2023] also enjoys e O( S √ DAT ) high probability regret by coupling posterior sampling with optimism. Another line of research focuses on the study of ergodic MDPs, where the environment is such that all states are visited infinitely often under every policy. To name a few, the model-free algorithm Politex Abbasi-Yadkori et al. [2019] attains a regret of e O(( t mix) 3 t hit √ S AT 3 4 ) where t mix and t hit are respectively the mixing and the hitting times of the ergodic environment. By leveraging an optimistic mirror descent algorithm, Wei et al. [2020] achieve an enhanced regret of e O( p ( t mix) 2 t hit AT ).

We refer the readers to Table 1 for a (non-exhaustive) list of existing algorithms.

2 Preliminaries

We fix a finite state-action space structure X : = S s ∈S { s } × A ( s ), and denote M the collection of all MDPs with state-action space X and rewards supported in [0 , 1].

Infinite-horizon MDP. An element M ∈ M is a tuple ( S , A , p , r ) where p is the transition kernel and r the reward function. The random state-action pair played by the agent at time t is denoted Xt ≡ ( S t , At ), and the achieved reward is Rt . A policy is a deterministic rule π : S → A and we write Π the space of policies. When coupled with a MDP M ∈ M , a policy properly defines the distribution of ( Xt , Rt ) whose associated probability probability and expectation operators are denoted P π s , E π s , where s ∈ S is the initial state. Under M , a fixed policy has a reward function r π ( s ) : = r ( s , π ( s )), a transition matrix P π , a gain g π ( s ) : = lim 1 T E π s [ R 0 + . . . + RT -1] and a bias h π ( s ) : = Cesàro- lim E π s [ P T -1 t = 0 ( Rt -g ( S t ))], that all together satisfy the Poisson equation h π + g π = r π + P π h π , see Puterman [1994]. The Bellman operator of the MDP is:

The optimal gain is g ∗ ( s ) : = max π g π ( s ) and the optimal bias is h ∗ ( s ) : = max { h π ( s ) : π s.t. g π = g ∗ } .

Weakly-communicating MDPs. M is weakly-communicating Puterman [1994], Bartlett and Tewari [2009] if the state space can be divided into two sets: (1) the transient set, consisting in states that are transient under all policies; (2) the non-transient set, where every state is reachable starting from any other non-transient state. In this case, h ∗ is a span-fixpoint of L (see Puterman [1994]), i.e., Lh ∗ -h ∗ ∈ R e where e is the vector full of ones. We write h ∗ ∈ Fix( L ). Then g ∗ = Lh ∗ -h ∗ and every policy π satisfies r π + P π h ∗ ≤ g ∗ + h ∗ . We accordingly define the Bellman gaps :

Another important concept is the diameter , that describes the maximal distance from one state to another state. It is given by D : = sup s , s ′ inf π E π s [inf { t ≥ 1 : S t = s ′ } ] . An MDP is said communicating

if its diameter D is finite, in which case sp( h ∗ ) ≤ sp( r ) D , see Bartlett and Tewari [2009], Fruit [2019], where sp( -) is the span function given by sp ( u ) : = max( u ) -min( u ).

Reinforcement learning. The learner is only aware that M ∈ M but doesn't have a clue about what M further looks like. From the past observations and the current state S t , the agent picks an available action A ( S t ), receives a reward Rt and observe the new state S t + 1. The regret of the agent is:

Its expected value satisfies E [Reg( T )] = E [ P T -1 t = 0 ∆ ∗ ( Xt )] + E [ h ∗ ( S 0 ) -h ∗ ( S T )] and the quantity P T -1 t = 0 ∆ ∗ ( Xt ) will be referred to as the pseudo-regret . This paper focuses on minimax regret guarantees . Specifically, for c ≥ 1, denote M c : = GLYPH<8> M ∈ M : ∃ h ∗ ∈ Fix( L ( M )) , sp ( h ∗ ) ≤ c GLYPH<9> the set of weaklycommunicating MDPs that admit a bias function with span at most c . Following Auer et al. [2009], every algorithm A , for all c > 0, we have

The goal of this work is to reach this lower bound with a tractable algorithm.

3 Algorithm PMEVI-DT

The algorithm PMEVI-DT that we present in this work is actually a general method can be applied to improve various existing algorithms Auer et al. [2009], Filippi et al. [2010], Fruit et al. [2018], Bourel et al. [2020], Tewari and Bartlett [2007]. All these algorithms work episodically, by maintaining a policy π k that drives play during a time-window { t k , . . . , t k + 1 -1 } called an episode. An episode rule determines when π k should be considered obsolete and defines the time t k + 1 at which the policy is renewed. To compute π k , these algorithms follow the optimism-in-face-of-certainty (OFU) design principle, by choosing π k that achieves the largest possible gain that is plausible under their current information. This is done by building a confidence region M t ⊆ M for the hidden model M , then searching for a policy π solving the optimization problem:

The design of the confidence region M t varies from a work to another. Given a confidence region ( M t ) t ≥ 0, OFU-algorithms work as follows: At the start of episode k , the optimization problem (5) is solved, and its solution π k is played until the end of episode. The duration of episodes can be managed in various ways, although the most popular is arguably the doubling trick (DT), that essentially waits until a state-action pair is about to double the visit count it had at the beginning of the current episode (see Algorithm 1).

Notations. In the rest of this section, we use ˆ pt ( s , a ) (and ˆ rt ( s , a )) to denote the empirical transition (and reward) of the latest doubling update before the t -th step, and further denote ˆ Mt : = (ˆ rt , ˆ pt ).

Extended Bellman operators and EVI . To solve (5) e ffi ciently, the celebrated work Auer et al. [2009] introduce the extended value iteration algorithm ( EVI ), that can be run whenever M t is a ( s , a )-rectangular confidence region, meaning that M t ≡ Q s , a ( R t ( s , a ) × P t ( s , a )) where R t ( s , a ) and P t ( s , a ) are respectively the confidence region for r ( s , a ) and p ( s , a ) after t learning steps. EVI is the algorithm computing the sequence defined by:

until sp ( vi + 1 -vi ) < ϵ where ϵ > 0 is the numerical precision. When the process stops, it is known that any policy π such that π ( s ) achieves L t vi in (6) satisfies g π ( M t ) ≥ g ∗ ( M ) -ϵ , hence is nearly optimistically optimal. This process gets its name from the observation that L t is the Bellman operator of M t seen as a MDP, hence EVI is just the Value Iteration algorithm Puterman [1994] ran in M t . A choice of action from s ∈ S in M t consists in (1) a choice of action a ∈ A ( s ), (2) a choice of reward ˜ r ( s , a ) ∈ R t ( s , a ) and (3) a choice of transition ˜ p ( s , a ) ∈ P t ( s , a ); It is an extended version of A ( s ).

Towards Projected Mitigated EVI . Obviously, the regret of an OFU-algorithm is directly related to the quality of the confidence region M t . That is why most previous works tried to approach the regret lower bound √ DSAT of Auer et al. [2009] by refining M t . The older works of Auer et al. [2009], Bartlett and Tewari [2009], Filippi et al. [2010] have been improved with a variance aware analysis Talebi and Maillard [2018], Fruit et al. [2018, 2020], Bourel et al. [2020] that essentially make use of tightened kernel confidence regions P t . While all these algorithms successively reduce the gap between the regret upper and lower bounds, they fail to achieve optimal regret √ DSAT . Meanwhile, the EBF algorithm of Zhang and Ji [2019] is minimax optimal but (1) the algorithm is intractable because it relies on an oracle to retrieve optimistically optimal policies and (2) needs prior information on the bias function. Nonetheless, the method of Zhang and Ji [2019] strongly suggests that inferring bias information from the available data is key to achieve minimax optimal regret.

Rather surprisingly and in opposition to this previous line of work, our work suggests that the choice of the confidence region M t has little importance. Instead, our algorithm takes an arbitrary (wellbehaved) confidence region in, infer bias information similarly to Zhang and Ji [2019] and makes use of it to refine the extended Bellman operator (6) associated to the input confidence region. Our algorithm can further take arbitrary prior information (possibly none) on the bias vector to tighten its bias confidence region. The pseudo-code given in Algorithm 1 is the high level structure our algorithm PMEVI-DT . In the next Section 3.1, we explain how (6) is refined using bias information.

Algorithm 1: PMEVI-DT ( H∗ , T , t 7→M t )

Parameters: Bias prior H∗ , horizon T , a system of confidence regions t 7→M t

1: for k = 1 , 2 , . . . do

2: Set t k ← t , update confidence region M t ;

4: H t ←H∗ ∩{ u : sp ( u ) ≤ T 1 / 5 } ∩ H ′ t ;

3: H ′ t ← BiasEstimation ( F t , M t , δ ):

5: Γ t ← BiasProjection ( H t , -);

6: β t ← VarianceApprox ( H ′ t , F t );

7: h k ← PMEVI ( M t , β t , Γ t , p log( t ) / t ) ;

8: g k ← L t h k -h k ;

9: Update policy π k ← Greedy ( M t , h k , β t );

10: repeat

11: Play At ← π k ( S t ), observe Rt , S t + 1 ;

12: Increment t ← t + 1;

13: until (DT) Nt ( S t , π k ( S t )) ≥ 1 ∨ 2 Ntk ( Xt ).

14: end for

3.1 Projected mitigated extended value iteration ( PMEVI )

Assume that an external mechanism provides a confidence region H t for the bias function h ∗ . Provided that M t is correct ( M ∈ M t ) and that H t is correct ( h ∗ ∈ H t ), we want to find a policy-model pair ( π, e M ) that maximizes the gain among pairs with h π ( e M ) ∈ H t . This is done with an improved version of (6) combining two ideas, that are both necessary to achieve minimax optimal regret in the analysis.

Projection ( Section 3.2 ). Whenever it is correct, the bias confidence region H t informs the learner that the search of an optimistic model can be constrained to those with bias within H t . This is done by projecting L β t (see mitigation ) using an operator Γ t : R S →H t , that has to satisfy a few non-trivial regularity conditions that are specified in Proposition 2.

Mitigation ( Section 3.3 ). When one is aware that h ∗ ∈ H t , the dynamical bias update ˜ p ( s , a ) vi in (6) can be controlled better, by trying to restrict (6) to the ˜ p ( s , a ) such that ˜ p ( s , a ) vi ≤ ˆ pt ( s , a ) vi + ( p ( s , a ) -ˆ pt ( s , a )) vi with the knowledge that vi ∈ H t . However, controlling the error ( p ( s , a ) -ˆ pt ( s , a )) vi by doing a union-bound on all possible values of vi is equivalent to building a confidence region for p ( s , a ), which produces an extra S 1 / 2 in the error term that cannot be a ff orded by a minimax optimal algorithm.

We take a di ff erent approach instead. For a fixed u ∈ R S , the empirical Bernstein inequality (Lemma 38) provides a variance bound of the form ( ˆ pt ( s , a ) -p ( s , a )) u ≤ β t ( s , a , u ). By

Algorithm 2: PMEVI ( M , β, Γ , ϵ )

Parameters: region M , mitigation β , projection Γ , precision ϵ , initial vector v 0 (optional)

1: if v 0 not initialized then set v 0 ← 0;

2: n ← 0

3: L ← extended operator associated to M ;

4: repeat

5: v n + 1 ←L β vn ;

7: n ← n + 1;

2 6: vn + 1 ← Γ v n + 1 2 ;

8: until sp ( vn -vn -1 ) < ϵ

9: return vn .

estimating β t ( s , a ) : = max u ∈H t β t ( s , a , u ), the search makes sure that ( ˆ pt ( s , a ) -p ( s , a )) u ≤ β t ( s , a ) holds with high probability for u = h ∗ , even though h ∗ is unknown. For β ∈ R X + , we introduce the β -mitigated extended Bellman operator:

The mitigation β ( s , a ) is independent of u , which is crucial for L β t to be well-behaved.

The proposition below shows how well-behaved the composition L t : = Γ t ◦L β t is. Its proof requires to build a complete analysis of projected mitigated Bellman operators. This is deferred to the appendix.

Proposition 2. Fix β ∈ R X + and assume that there exists a projection operator Γ t : R X →H t which is ( O1 ) monotone: u ≤ v ⇒ Γ u ≤ Γ v; ( O2 ) non span-expansive: sp ( Γ u -Γ v ) ≤ sp ( u -v ) ; ( O3 ) linear: Γ ( u + λ e ) = Γ u + λ e and ( O4 ) Γ u ≤ u. Then, the projected mitigated extended Bellman operator L t : = Γ t ◦ L β t has the following properties:

There exists a unique g t ∈ R e such that ∃ h t ∈ H t , L t h t = h t + g t ;

If M ∈ M t , h ∗ ∈ H t and ( ˆ pt ( s , a ) -p ( s , a )) h ∗ ≤ β ( s , a ) , then g t ≥ g ∗ ( M ) ;

If M t is convex, then for all u ∈ R S , the policy π = : Greedy ( M t , u , β ) picking the actions achieving L β t u satisfies L tu = ˜ r π + ˜ P π u for ˜ r π ( s ) ≤ sup R t ( s , π ( s )) and ˜ P π ( s ) ∈ P t ( s , π ( s )) ;

The property (1) guarantees that L t has a fix-point while (2) states that this fix-point corresponds to an optimistic gain g t if the model and the bias confidence region are correct and the mitigation isn't too aggressive. Combined with (3), the Poisson equation of a policy corresponds to this fix-point, i.e., ˜ r π + ˜ P π h t = h t + g t , so that g t is the gain and h t ∈ H t is a legal bias for π under the model (˜ r π , ˜ P π ). Lastly, the property (4) guarantees that the iterates L n t u converge to a fix-point of L at least as quickly as L n t u goes to a fix-point of L t ; The convergence of ( L t ) n u is already guaranteed by existing studies and is discussed in the appendix.

Provided that the bias confidence region is constructed, Proposition 2 foreshadows how powerful the construction is: The algorithm PMEVI , obtained by iterating L t instead of L t in EVI , can replace the well-known EVI within any algorithm of the literature that relies on it ( UCRL2 Auer et al. [2009], UCRL2B Fruit et al. [2020] or KL-UCRL Filippi et al. [2010]) for an immediate improvement of its theoretical guarantees.

3.2 Building the bias confidence region and its projection operator

The bias confidence region used by PMEVI-DT is obtained as a collection of constraints of the form:

Such constraints include (1) prior bias constraints (if any) of the form of h ( s ) -h ( s ′ ) ≤ c ∗ ( s , s ′ ); (2) span constraints of the form h ( s ) -h ( s ′ ) ≤ c 0 : = T 1 / 5 spawning the span semi-ball { u : sp ( u ) ≤ T 1 / 5 } ; and (3) pair-wise constraints obtained by estimating bias di ff erences in the style of Zhang and Ji [2019], Zhang and Xie [2023] that we further improve. We start by defining a bias di ff erence estimator.

Definition 1 (Bias di ff erence estimator) . Given a pair of states s , s ′ , their sequence of commute times ( τ s ↔ s ′ i ) i ≥ 0 is defined by τ s ↔ s ′ 2 i : = inf { t > τ s ↔ s ′ 2 i -1 : S t = s } and τ s ↔ s ′ 2 i + 1 : = inf { t > τ s ↔ s ′ 2 i : S t = s ′ } with the convention that τ s ↔ s ′ -1 = -∞ . The number of commutations up to time t is Nt ( s ↔ s ′ ) : = inf { i : τ s ↔ s ′ i ≤ t } , and ˆ g ( t ) : = 1 t P t -1 i = 0 Ri is the empirical gain. The bias di ff erence estimator at time T is any quantity cT ( s , s ′ ) ∈ R such that:

Lemma 3. With probability 1 -2 δ , for all T ′ ≤ T, we have

Lemma 3 says that the quality of the estimator cT ( s , s ′ ) is directly linked to the number of observed commutes between s and s ′ as well as the regret. The idea is that if the algorithm makes many commutes between s and s ′ and if its regret is small, then the algorithm mostly takes optimal paths from s to s ′ . The bound provided by Lemma 3 is not accessible to the learner however, because sp ( h ∗ ) is unknown in general. To overcome this issue, sp ( h ∗ ) is upper-bounded by c 0 : = T 1 / 5 . Overall, this leads to the design of the algorithm estimating the bias confidence region as specified in Algorithm 3.

6:

return

(

ct

,

error(

ct

,

-

,

-

)), (8) defines

H ′

t

.

Coupled with prior information and span constraints, the bias confidence region H t is a polyhedron of the same kind as the one encountered in Zhang and Xie [2023]. When generated by constraints of the form (8), following [Zhang and Xie, 2023, Proposition 3], one can project onto H t in polynomial time with Algorithm 4. Moreover, the resulting projection operator satisfies the prerequisites ( O1-4 ) of Proposition 2, making PMEVI (Algorithm 2) well-behaved. See Appendix B.2 for proofs.

Lemma 4. Assume that H is a set of h ∈ R S satisfying a system of equations of the form of (8) . If H is non empty, then the operator Γ u : = BiasProjection ( H , u ) (see Algorithm 4) is a projection on H and satisfies the properties ( O1-4 ) defined in Proposition 2.

3.3 Mitigation using finer bias dynamical error

The fact that h ∗ ∈ H t with high probability is used in PMEVI-DT to restrict the search of EVI by reducing the dynamical bias error. This reduction is based on a empirical Bernstein inequality (see Lemma 38) applied to ( ˆ p ( s , a ) -p ( s , a )) u . Here, it gives that with probability 1 -δ , we have:

where V ( ˆ pt ( s , a ) , u ) is the variance of u under the probability vector ˆ pt ( s , a ). More specifically, if q is a probability on S and q ∈ R S , we set V ( q , u ) : = P s q ( s )( u ( s ) -q · u ) 2 . In (11), u ∈ R S , ( s , a ) ∈ X and T ≥ 1 are fixed. Once is tempted to use (11) directly to mitigate the extended Bellman operator, but the resulting operator is ill-behaved because it loses monotony. This issue is avoided by changing β t ( s , a , u ) to max u ∈H t β t ( s , a , u ) in (11). The resulting inequality is not guaranteed to hold simultaneously for all u ∈ H t and with high probability; However, it is guaranteed to hold with high probability for u = h ∗ , which will be enough.

The variance maximization problem max u ∈H t β t ( s , a , u ) is a convex maximization problem with linear constraints. Even in very simple settings, such optimization problems are NP-hard Pardalos and Schnitger [1988] hence computing max u ∈H t β t ( s , a , u ) is not reasonable in general. Thankfully, this value can be upper-bounded by a tractable quantity that is enough in the regret analysis. The mitigation β t used by PMEVI-DT is provided by Algorithm 5. See Lemma 12 and Appendix A.2.2 for details.

4 Regret guarantees

Theorem 5 thereafter shows that PMEVI-DT has minimax optimal regret under regularity assumptions on the used confidence region M t . Assumption 1 asserts that the confidence region holds uniformly with high probability. Assumption 2 asserts that the reward confidence region is sub-Weissman (see

Algorithm 5: VarianceApproximation ( H ′ t , F t ) Parameters: Bias region H ′ t , history F t 1: Extract constraints ( c , error( c , -, -)) ←H ′ t ; 2: Set c 0 ← T 1 5 ; 3: Pick a reference point h 0 ← BiasProjection ( H t , c ( -, s 0 )); 4: for ( s , a ) ∈ X do 5: ρ ← log GLYPH<16> S AT δ GLYPH<17> / max { 1 , Nt ( s , a ) } ; 6: var( s , a ) ← V ( ˆ pt ( s , a ) , h 0 ) + 8 c 0 P s ′ ∈S ˆ pt ( s ′ | s , a ) c ( s ′ , s ); 7: β t ( s , a ) ← p 2var( s , a ) ρ + 3 c 0 ρ or + ∞ if Nt ( s , a ) = 0; 8: end for 9: return β t .

Lemma 35) and Assumption 3 assumes that the model confidence region makes sure that EVI (6) converges in the first place. Assumption 4 asserts that the prior bias region is correct.

Assumption 1. With probability 1 -δ , we have M ∈ T K ( T ) k = 1 M t k .

Assumption 2. There exists a constant C > 0 such that for all ( s , a ) ∈ S , for all t ≤ T , we have:

Assumption 3. For t ≥ 0, M t is a ( s , a )-rectangular convex region and L n t u converges a fix-point.

Assumption 4. The prior bias region H∗ contains h ∗ ( M ) and is generated by constraints of the form:

with c ∗ ( s , s ′ ) ∈ [ -∞ , ∞ ] (possibly infinite).

Refer to Appendix A.2 for the feasibility of Assumption 1, Appendix A.2.3 for Assumption 2, and Appendix A.3 for Assumption 3.

Theorem 5 (Main result) . Let c > 0 . Assume that PMEVI-DT runs with a confidence region system t 7→M t that guarantees Assumptions 1-3. If T ≥ c 5 , then for every weakly communicating model with sp ( h ∗ ) ≤ c and such that Assumption 4 is satisfied (h ∗ ∈ H∗ ), PMEVI-DT achieves regret:

with probability 1 -26 δ , and in expectation if δ < √ 1 / T. Moreover, if PMEVI-DT runs with the same confidence regions that UCRL2 Auer et al. [2009], then it enjoys a time complexity O( DS 3 AT ) .

To have a completely prior-less algorithm, pick H∗ = R S . The proof of Theorem 5 is tedious and its details are deferred to appendix. We will focus here on the main ideas.

Notations. At episode k , the played policy is denoted π k . As a greedy response to h k , by Proposition 2 (3), there exists ˜ rk ( s ) ≤ sup R t k ( s , π k ( s )) and ˜ Pk ( s ) ∈ P t k ( s , π ( x )) such that h k + g k = ˜ rk + ˜ Pk h k . The reward-kernel pair ˜ Mk = (˜ rk , ˜ Pk ) is referred to as the optimistic model of π k . We write Pk : = P π k ( M ) the true kernel and ˆ Pk : = P π k ( ˆ Mtk ) the empirical kernel. Likewise, we define the reward functions rk and ˆ rk . The optimistic gain and bias satisfy g k = g ( π k , e Mk ) and h k = h ( π k , e Mk ). We further denote c 0 = T 1 5 .

The regret is first decomposed episodically with Reg( T ) = P k P t k + 1 -1 t = t k ( g ∗ -Rt ). The first step goes back to the analysis of UCRL2 Auer et al. [2009], and consists in upper-bounding the regret of the episode k with optimistic quantities that are exclusive to that episode.

Lemma 6 (Reward optimism) . With probabililty 1 -6 δ , we have:

Weintroduce the two optimistic regrets B ( T ) : = P k P t k + 1 -1 t = t k ( g k -Rt ) and ˜ B ( T ) : = P k P t k + 1 -1 t = t k ( g k -˜ rk ( Xt )). Rewriting the summand g k -˜ rk ( Xt ) using the Poisson equation h k + g k = ˜ rk + ˜ Pk h k , we get:

Model

region

M

t

region

H

t

Mitigation

β

t

Projected Mitigated EVI (Alg.1)

T

)

P

k

P

t

k

+

1

-

1

t

=

t

k

(

g

∗ -

Rt

)

Optimistic re-

gret (Lemma 6)

P

k

P

t

k

+

1

-

1

t

=

t

k

(

g

k

-

Rt

)

O

GLYPH<16>p

S AT

log(

T

)

GLYPH<17>

Navigation error (Lemma 7)

P

k

,

t

(

p

(

)

-

eSt

+

1

)

h

k

Emp. bias error (Lemma 8)

P

k

,

t

( ˆ

ptk

(

)

-

p

(

))

h

∗

Optimism over-

shoot (Lemma 9)

P

k

,

t

( ˜

pt k

(

S t

)

-

ˆ

pt k

(

S t

))

h

k

2nd order error (Lemma 10)

P

k

,

t

( ˆ

pt k

(

S t

)

-

p

(

))(

h

k

-

h

∗

)

(in expectation)

O(

c

0

log(

S AT

))

p

sp (

h

∗

)

S AT

log(

S AT

)

S

2

Ac

0 log

2

(

T

)

p

T

)

O

GLYPH<16>

c

0

S

2

A

2

T

1

/

4

log

2

(

T

)

GLYPH<17>

The analysis proceeds by decomposing the above expression of ˜ B ( T ) in the style of Zhang and Ji [2019]. We write P t k + 1 -1 t = t k ( ˜ pk ( S t ) -eSt ) h k as:

Each error term is bounded separately. Below, we denote V ( q , u ) : = P s q ( s )( u ( s ) -q · u ) 2 .

Lemma 7 (Navigation error) . With probability 1 -7 δ , the navigation error is bounded by:

Lemma 8 (Empirical bias error) . With probability 1 -δ , the empirical bias error is bounded by:

Lemma 9 (Optimistic overshoot) . With probability 1 -6 δ , the optimistic overshoot is bounded by:

Lemma 10 (Second order error) . With probability 1 -6 δ , the second order error is bounded by:

We see that the empirical bias error (Lemma 8) and the optimistic overshoot (Lemma 9) both involve the sum of variances P T -1 t = 0 V ( p ( Xt ) , h ∗ ), which is shown in Lemma 29 to be of order sp ( h ∗ )sp ( r ) T + P T -1 t = 0 ∆ ∗ ( Xt ). The pseudo-regret term P T -1 t = 0 ∆ ∗ ( Xt ) is bounded with the regret using Corollary 31, then by B ( T ). With high probability, we obtain an equation of the form:

where C is a constant. Setting α : = CS 2 A (1 + c 0) log 2 ( T ) and β : = C p (1 + sp ( h ∗ )) S AT log( T /δ ) + ˜ O( T 1 / 4 ), the above equation is of the form B ( T ) ≤ β + α √ B ( T ). Solving in B ( T ), we find B ( T ) ≤ β + 2 √ αβ + α 2 . The dominant term is β , hence we readily obtain:

Since c 0 = o( T 1 4 ), we conclude that B ( T ) = O GLYPH<16>p sp ( h ∗ ) S AT log( T /δ ) GLYPH<17> , ending the proof.

5 Experimental illustrations

To get a grasp of how PMEVI-DT behaves in practice, we provide in Fig. 2 a first round of illustrative experiments. In both, the environment is a river-swim which is a model known to be hard to learn despite its size, with high diameter and bias span, see Appendix D for the model's description.

UCRL2

PMEVI(c=2)

PMEVI(c=0.5)

PMEVI(c=1)

UCRL2B & PMEVI-UCRL2B

UCRL2 & PMEVI-UCRL2

KLUCRL

&

PMEVI-KLUCRL

On the first experiment, we observe that PMEVI can exploit prior bias knowledge e ff ectively and drastically improve the regret performance, depending on the quality of the prior region.

On the second experiment however, we observe that without prior knowledge, PMEVI has nearly the same regret performance that its EVI counterparts, meaning that the bias confidence region is too large to e ff ectively improve the regret performance. This observation is first to be taken with caution. Indeed, the regret that is being estimated above is model specific, hence is not an estimate of the minimax regret - This being said, it undoubtedly shows that the bias confidence region is ine ff ective and this can be explained as follows. On experiments, we see that most of the regret is due to the early phase of the learning process, where proper bias information is nearly impossible to get. Indeed, the regret is still growing linearly, so no bias information can be inferred. But in addition, this 'bad' early data pollutes the bias estimator for a long duration. In other words, while the theoretical regret guarantees of PMEVI-DT are better than its EVI analogues, there is room to improve the bias estimation mechanism and the practical performance.

6 Conclusion

In this work, we have shown that regret guarantees of order p sp( h ∗ ) S AT log( T ) can be achieved for weakly communicating MDPs without prior knowledge, nor exponential computational cost. In particular, regret guarantees can scale with the bias span rather than the diameter without prior knowledge. This is in opposition to the recent results that the sample complexity cannot be bounded in term of bias span without prior knowledge for average reward MDPs Tuynman et al. [2024], Wang et al. [2024], Zurek and Chen [2024a,b]. This di ff erence lies in the fact ( ϵ, δ )-PAC algorithm must produce a policy πτ after τ learning steps where τ is a stopping time, with P ( g πτ ≤ g ∗ -ϵ ) ≤ δ . Implicitly, these algorithms must hereby certify that the output policy is approximately optimal. In opposition, regret robust algorithms have no need to assess that deployed policies are indeed optimal.

In the end, the regret advantages of PMEVI-DT over pure EVI -based methods remain theoretical, and the experimental shortcomings displayed in Section 5 leave a few opportunities for future work. Can bias information be inferred more e ffi ciently? Or, do the experiments indicate that the regret analysis of EVI -based methods may be drastically improved?

References

Yasin Abbasi-Yadkori, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. Exploration-enhanced POLITEX. 2019. arXiv:1908.10479.

Shipra Agrawal and Randy Jia. Optimistic Posterior Sampling for Reinforcement Learning: WorstCase Regret Bounds. Mathematics of Operations Research , 48(1):363-392, 2023. Publisher: INFORMS.

Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeo ff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009. Publisher: Elsevier.

Peter Auer and Ronald Ortner. Logarithmic Online Regret Bounds for Undiscounted Reinforcement Learning. Proceedings of the 19th International Conference on Neural Information Processing Systems , December 2006.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-Time Analysis of the Multiarmed Bandit Problem. Mach. Learn. , 47(2-3):235-256, May 2002.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal Regret Bounds for Reinforcement Learning. In Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2009.

Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal , 19(3):357 - 367, 1967. Publisher: Tohoku University, Mathematical Institute.

Peter L. Bartlett and Ambuj Tewari. REGAL: a regularization based algorithm for reinforcement learning in weakly communicating MDPs. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , UAI '09, pages 35-42, Arlington, Virginia, USA, June 2009. AUAI Press.

Hippolyte Bourel, Odalric Maillard, and Mohammad Sadegh Talebi. Tightening Exploration in Upper Confidence Reinforcement Learning. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1056-1066. PMLR, July 2020.

Apostolos Burnetas and Michael Katehakis. Optimal Adaptive Policies for Markov Decision Processes. Mathematics of Operations Research - MOR , 22:222-255, February 1997.

Michael B. Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time, 2020.

Sarah Filippi, Olivier Cappé, and Aurélien Garivier. Optimism in Reinforcement Learning and Kullback-Leibler Divergence. 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 115-122, September 2010. arXiv: 1004.5229.

Ronan Fruit. Exploration-exploitation dilemma in Reinforcement Learning under various form of prior knowledge . PhD Thesis, Université de Lille 1, Sciences et Technologies; CRIStAL UMR 9189, 2019.

Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Ronald Ortner. E ffi cient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning. Proceedings of the 35 th International Conference on Machine Learning , 2018.

Ronan Fruit, Matteo Pirotta, and Alessandro Lazaric. Improved Analysis of UCRL2 with Empirical Bernstein Inequality. 2020. arXiv:2007.05456.

Anders Jonsson, Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Edouard Leurent, and Michal Valko. Planning in markov decision processes with gap-dependent sample complexity. Advances in Neural Information Processing Systems , 33:1253-1263, 2020.

Yi Ouyang, Mukul Gagrani, Ashutosh Nayyar, and Rahul Jain. Learning Unknown Markov Decision Processes: A Thompson Sampling Approach. September 2017. arXiv: 1709.04570.

Panos M. Pardalos and Georg Schnitger. Checking local optimality in constrained quadratic programming is NP-hard. Operations Research Letters , 7:33-35, 1988.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . Wiley Series in Probability and Statistics. Wiley, 1 edition, April 1994. ISBN 978-0-471-61977-2 978-0-470-31688-7.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPs. Journal of Machine Learning Research , pages 1-36, April 2018. Publisher: Microtome Publishing.

Ambuj Tewari and P. Bartlett. Optimistic Linear Programming gives Logarithmic Regret for Irreducible MDPs. In NIPS , 2007.

Georgios Theocharous, Zheng Wen, Yasin Abbasi-Yadkori, and Nikos Vlassis. Posterior sampling for large scale reinforcement learning. 2017. arXiv:1711.07979.

William R Thompson. On the Likelihood that One Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika , 25(3-4):285-294, December 1933. ISSN 0006-3444.

Adrienne Tuynman, Rémy Degenne, and Emilie Kaufmann. Finding good policies in average-reward Markov Decision Processes without prior knowledge, May 2024.

Shengbo Wang, Jose Blanchet, and Peter Glynn. Optimal Sample Complexity for Average Reward Markov Decision Processes, February 2024.

Chen-Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Modelfree Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 10170-10180. PMLR, July 2020.

Zihan Zhang and Xiangyang Ji. Regret Minimization for Reinforcement Learning by Evaluating the Optimal Bias Function. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Zihan Zhang and Qiaomin Xie. Sharper Model-free Reinforcement Learning for Average-reward Markov Decision Processes. In The Thirty Sixth Annual Conference on Learning Theory , pages 5476-5477. PMLR, 2023.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decomposition. June 2020. arXiv: 2004.10019 [cs, stat].

Matthew Zurek and Yudong Chen. The Plug-in Approach for Average-Reward and Discounted MDPs: Optimal Sample Complexity Analysis, 2024a. arXiv:2410.07616 [cs.LG].

Matthew Zurek and Yudong Chen. Span-Based Optimal Sample Complexity for Weakly Communicating and General Average Reward MDPs, 2024b. arXiv:2403.11477 [cs.LG].

A Construction of PMEVI-DT

This section provides the technical details required to understand the design of PMEVI-DT in Section 3. We further discuss the assumptions 1-4 appearing in Theorem 5 and provide su ffi cient conditions so that they are met.

A.1 Proof of Lemma 3, estimation of the bias error

Fix s , s ′ ∈ S . We denote α T : = NT ( s ↔ s ′ )( h ∗ ( s ) -h ∗ ( s ′ ) -cT ( s , s ′ )). We will start by considering the better estimator c ′ T ( s , s ′ ) that satisfies the same equation (9) than cT ( s , s ′ ) but with ˆ g ( T ) changed to g ∗ , readily:

To avoid a typographical clutter, we write τ i instead of τ s ↔ s ′ i in the remaining of the proof and we write α ′ T : = NT ( s ↔ s ′ )( h ∗ ( s ) -h ∗ ( s ′ ) -c ′ T ( s , s ′ ).

( STEP 1 ) We start by relating the two estimators. Intuitively, ˆ g ( T ) is a good estimator for g ∗ when the regret is small. Recall that ˆ g ( T ) : = 1 T P T -1 t = 0 Rt , hence:

Therefore,

We are left with upper-bounding GLYPH<12> GLYPH<12> GLYPH<12> α ′ T GLYPH<12> GLYPH<12> GLYPH<12> .

( STEP 2 ) If i is even, then S τ i and S τ i + 1 = s ′ ; otherwise S τ i = s ′ and S τ i + 1 = s . In both cases, we have h ∗ ( S τ i + 1 ) -h ∗ ( S τ i ) = ( -1) i ( h ∗ ( s ′ ) -h ∗ ( s )). Therefore, using Bellman's equation, the quantity A : = P τ i + 1 -1 t = τ i ( g ∗ -Rt ) satisfies:

Multiplying by ( -1) i and rearranging, h ∗ ( s ′ ) -h ∗ ( s ) + ( -1) i + 1 P τ i + 1 -1 t = τ i ( g ∗ -Rt ) appears to be equal to:

Proceed by summing over i . By triangular inequality, we obtain:

Because all Bellman gaps ∆ ∗ are non-negative, the second term is upper-bounded by the pseudo-regret P T -1 t = 0 ∆ ∗ ( Xt ). The first term is a martingale, and the martingale di ff erence sequence ( -1) i + 1 (( p ( Xt ) -eSt + 1 ) h ∗ + r ( Xt ) -Rt has span at most sp ( h ∗ ) + 1 since rewards are supported in [0 , 1]. Although the number of involved terms is random, it is upper-bounded by T , hence by the maximal version of Azuma-Hoe ff ding's inequality (Lemma 32), we have that with probability at least 1 -δ and uniformly for T ′ ≤ T ,

( STEP 3 ) We conclude that with probability 1 -δ , for all T ′ ≤ T ,

Weare left with relating both P T ′ -1 t = 0 ∆ ∗ ( Xt ) and GLYPH<12> GLYPH<12> GLYPH<12> Reg( T ′ ) GLYPH<12> GLYPH<12> GLYPH<12> to P T ′ -1 t = 0 (˜ g -Rt ). Using the Bellman equation again, we find that:

where the last inequality holds with probability 1 -δ uniformly over T ′ ≤ T by Azuma-Hoe ff ding's inequality again (Lemma 32). Remark that if y -z ≤ x ≤ y + z , then | x | ≤ | y | + | z | , hence we conclude that with probability 1 -δ , for all T ′ ≤ T :

where the last inequality invokes ˜ g ≥ g ∗ . We conclude that, with probability 1 -2 δ , for all T ′ ≤ T , we have:

This concludes the proof.

A.2 The confidence region of PMEVI-DT

The algorithm PMEVI-DT can be instantiated with a large panel of possibilities, depending on the type of confidence region one is willing to use for rewards and kernels. In this work, we allow for four types of confidence regions, described below. For conciseness, q ∈ { r , p } is a symbolic letter that can be a reward or a kernel and we denote Q t ( s , a ) the confidence region for q ( s , a ) at time t . If q = r , then dim( q ) = 2 (Bernoulli rewards) with Q ( s , a ) = [0 , 1]; and if q = p , then dim( q ) = S with Q ( s , a ) = P ( S ).

( C1 ) Azuma-Hoe ff ding or Weissman type confidence regions, with Q t ( s , a ) taken as:

( C2 ) Empirical Bernstein type confidence regions, with Q t ( s , a ) taken as:

with the convention that x / 0 = + ∞ for x > 0.

( C3 ) Empirical likelihood type confidence regions, with Q t ( s , a ) taken as:

( C4 ) Trivial confidence region with Q t ( s , a ) = Q ( s , a ).

A few remarks are in order. When rewards are not Bernoulli, only the confidence regions ( C1 ) and ( C4 ) are elligible among the above. Then, Weissman's inequality must be changed to Azuma's inequality for σ -sub-Gaussian random variables, see Lemma 34. Since rewards are supported in [0 , 1], Hoe ff ding's Lemma guarantees that reward distributions are σ -sub-Gaussian with σ = 1 2 .

□

A.2.1 Correctness of the model confidence region M t and Assumption 1

The confidence regions Q t ( s , a ) described with ( C1-4 ) are tuned so that the following result holds:

Lemma 11. Assume that, for all q ∈ { r , p } and ( s , a ) ∈ X , we choose Q t ( s , a ) among ( C1-4 ). Then Assumption 1 holds. More specifically, the region of models M t : = Q s , a ( R t ( s , a ) × P t ( s , a )) satisfies P ( ∃ t ≤ T : M < M t ) ≤ δ .

Proof. We show that, for all q ∈ { r , q } and ( s , a ) ∈ X , if Q t ( s , a ) is chosen amoung ( C1-4 ), then

If Q t ( s , a ) is chosen with ( C1 ), this is a direct application of Lemma 35; with ( C2 ), this is Lemma 36; with ( C3 ), this is Lemma 37; and with ( C4 ) this is by definition. □

A.2.2 Simultaneous correctness of bias confidence region H t , mitigation β t and optimism

In this section, we show that if Assumption 1 holds, then the bias confidence region constructed by PMEVI-DT is correct with high probability, and that the mitigation is not too strong. Recall that ( g k , h k ) are the optimistic gain and bias of the policy deployed in episode k (see Algorithm 1). In particular, we have g k = L t k h k -h k with h k ∈ H t k . We start by a result on the deviation of the variance, which is what the variance approximation Algorithm 5 is based on. Recall that the bias confidence region H t is obtained as the collection of constraints:

prior constraints (if any) h ( s ) -h ( s ′ ) ≤ c ∗ ( s , s ′ );

span constraints h ( s ) -h ( s ′ ) ≤ c 0 : = T 1 / 5 ;

dynamically inferred constraints | h ( s ) -h ( s ′ ) -ct ( s ′ , s ) | ≤ error( ct , s ′ , s ) (see Algorithm 3).

We start with the technical result that is behind the variance approximation Algorithm 5.

Lemma 12. Let u , v ∈ H t and fix p a probability distribution on S . Then for all s ∈ S ,

Proof. We start by establishing the following result: If p is a probability distribution on S and u , v ∈ R S , we have:

where · is the dot product, u 2 the Hadamard product uu and | u | the vector whose entry s is | u ( s ) | . (14) is obtained with a straight forward computation:

Observe that v can be changed to v + λ e , where e is the vector full of ones, without changing the result. The same goes for u . We now move to the proof of the main statement. First, translate u and v such that u ( s ) = v ( s ) = 0. Then, we have:

Conclude using that max( u + v ) ≤ max( u ) + max( v ) + 2 c 0 for u , v ∈ H such that u ( s ) = v ( s ) = 0. □

Lemma 13. Assume that Assumption 1 holds and that c 0 ≥ sp ( h ∗ ) . Then, with probability 1 -4 δ , for all k ≤ K ( T ) , (1) g k ≥ g ∗ and (2) h ∗ ∈ H t k and (3) for all ( s , a ) , ( ˆ ptk ( s , a ) -p ( s , a )) h ∗ ≤ β t k ( s , a ) .

Proof. Let E 1 the event ( ∀ k ≤ K ( T ) , M ∈ M t k ). Let E 2 the event stating that, for all T ′ ≤ T ,

and let E 3 the event stating that, for all T ′ ≤ T and for all ( s , a ) ∈ X , we have:

By Assumption 1, we have P ( E 1 ) ≥ 1 -δ . By Lemma 3, we have P ( E 2 ) ≥ 1 -2 δ and by Lemma 36, we have P ( E 3 ) ≥ 1 -δ , so P ( E 1 ∩ E 2 ∩ E 3 ) ≥ 1 -4 δ . We prove by induction on k ≤ K ( T ) that, on E 1 ∩ E 2 ∩ E 3 , (1) g k ≥ g ∗ , (2) h ∗ ∈ H t k (3) and for all ( s , a ), ( ˆ ptk ( s , a ) -p ( s , a )) h ∗ ≤ β t k ( s , a ), where g k is the optimistic gain of the policy deployed at episode k .

It is obvious for k = 0. Indeed, N 0 ( s ↔ s ′ ) = 0 for all s , s ′ hence c 0 ( s , s ′ ) = c 0 ≥ sp ( h ∗ ). Therefore,

so contains h ∗ , proving (2). Moreover, since N 0 ( s , a ) = 0, we have β 0 ( s , a ) = + ∞ , proving (3). Finally, since M ∈ M 0 on E 1, by the statement (2) of Proposition 2, we have g k ≥ g ∗ , hence proving (1).

Now assume that k ≥ 1. By induction g ℓ ≥ g ∗ for all ℓ < k , so on E 2 we have:

By design of H t k (see Algorithm 3), we deduce that (2) h ∗ ∈ H t k . Denote h 0 ∈ H t k the reference point used by Algorithm 5. We have, for all ( s , a ) ∈ X , on E 1 ∩ E 2 ∩ E 3, we have:

by construction of Algorithm 5. Accordingly, (3) is satisfied. Finally, M ∈ M t k on E 1 so by Proposition 2, we have (1) g k ≥ g ∗ . □

Corollary 14. Assume that, for all q ∈ { r , p } and ( s , a ) ∈ X , we choose Q t ( s , a ) among ( C1-4 ). Then, with probability 1 -4 δ , for all k ∈ K ( T ) , we have g k ≥ g ∗ and (2) h ∗ ∈ H t k and (3) for all ( s , a ) , ( ˆ ptk ( s , a ) -p ( s , a )) h ∗ ≤ β t k ( s , a ) .

Proof. By Lemma 11, Assumption 1 is satisfied. Apply Lemma 13.

A.2.3 Sub-Weissman reward confidence region and Assumption 2

Although the kernel confidence region can even chosen to be trivial with ( C4 ), in order to work, PMEVI-DT needs the reward confidence region to be sub-Weissman in the following sense:

Assumption 2. There exists a constant C > 0 such that for all ( s , a ) ∈ S , for all t ≤ T , we have:

This is indeed the case if R t ( s , a ) is chosen among ( C1-3 ).

A.3 Convergence of EVI and Assumption 3

We start with a preliminary lemma on the speed of convergence of EVI . The Lemma 15 is thought to be applied to extended MDPs. Below, when we claim that the action space is compact, we further claim that a ∈ A ( s ) 7→ p ( s , a ) is a continuous map, so that the Bellman operator is continuous and that g ∗ and h ∗ are well-defined, see Puterman [1994].

□

Lemma 15. Let M a weakly-communicating MDP with finite state space R S and compact action space, and let L its Bellman operator. Assume that there exists γ > 0 such that, ∀ u ∈ R S ,

with q u s ∈ P ( S ) . Then, for all u ∈ R S and all ϵ > 0 , if sp GLYPH<16> L n + 1 u -L n u GLYPH<17> ≥ ϵ , then:

Proof. Since M is weakly communicating, has finitely many states and compact action space, it has well-defined gain g ∗ and bias h ∗ functions. Denote un + 1 : = L n u .

Observe that the policy achieving the maximum is the one achieving un = r π + P π un -1. Remark that r ′ π ( s ) = -∆ ∗ ( s , π ( s )) ≤ 0 is the Bellman gap of the pair ( s , π ( s )), that we more simply write ∆ π . For all n , there exists π n ∈ Π such that wn + 1 = -∆ π n + P π n wn . Moreover, by assumption, we have P π n = γ · e ⊤ sn e + (1 -γ ) Qn where Qn is a stochastic matrix. Moreover,

Hence, sp ( wn + 1 ) ≤ (1 -γ )sp ( wn ) + sp GLYPH<0> ∆ π n GLYPH<1> . In addition, wn = L n u -L n h ∗ , so by non-expansiveness of L in span semi-norm, sp ( wn + 1 ) ≤ sp ( wn ). Overall,

Fix ϵ > 0, and let n ϵ : = inf GLYPH<8> n : sp ( wn ) < ϵ GLYPH<9> .

Let π ∗ an optimal policy. We have wn + 1 ≥ P π ∗ wn so by induction, wn + 1 ≥ P n + 1 π ∗ w 0 ≥ min( w 0 ) e . Meanwhile, we see that ∥ wn ∥ 1 ≥ P n -1 k = 0 GLYPH<13> GLYPH<13> GLYPH<13> ∆ π k GLYPH<13> GLYPH<13> GLYPH<13> 1 + S min( w 0 ), so P n -1 k = 0 GLYPH<13> GLYPH<13> GLYPH<13> ∆ π k GLYPH<13> GLYPH<13> GLYPH<13> 1 ≤ sp ( w 0). Since ∆ π k ≤ 0 for all k , we have sp GLYPH<0> ∆ π k GLYPH<1> ≤ GLYPH<13> GLYPH<13> GLYPH<13> ∆ π k GLYPH<13> GLYPH<13> GLYPH<13> 1 so P n -1 k = 0 sp GLYPH<0> ∆ π k GLYPH<1> ≤ sp ( w 0 ).

By (15), either sp ( wn + 1 ) ≤ (1 -1 2 γ ) max( ϵ, sp ( wn )) or sp GLYPH<0> ∆ π n GLYPH<1> ≥ 1 2 γϵ , but because P + ∞ k = 0 sp GLYPH<0> ∆ π k GLYPH<1> ≤ sp ( w 0), the second case can happen at most 2sp( w 0 ) γϵ times. We deduce that, for all n ≤ n ϵ ,

In particular, for n = n ϵ -1, we get:

We obtain:

To conclude, check that sp GLYPH<16> L n + 1 u -L n u GLYPH<17> = sp ( wn + 1 -wn ) ≤ 2sp ( wn ). □

Before moving to the application of interest, remark that this result can be greatly improved if the supremum sup { ∆ ∗ ( s , a ) : ∆ ∗ ( s , a ) < 0 } is not zero, to change the dominant term 4sp( w 0 ) γϵ for a constant independent of ϵ .

Corollary 16. Assume that the M t has non-empty interior, and that its Bellman operator satisfies the requirement of Lemma 15, i.e., there exists γ > 0 such that, ∀ u ∈ R S , ∀ s ∈ S , ∃ a ∈ A ( s ) , ∃ ˜ rt ( s , a ) ∈ R t ( s , a ) , ∃ ˜ pt ( s , a ) ∈ P t ( s , a ) :

for some q u s ∈ P ( S ) . Then Assumption 3 is satisfied, and span fix-points ˜ ht of L t are such that g ∗ ( M t ) = L t ˜ ht -˜ ht .

Proof. If M t is has non-empty interior, it means that for all ( s , a ), P t ( s , a ) has non-empty interior. Therefore, for all state-action pair, there exists ˜ pt ( s , a ) ∈ P t ( s , a ) that is fully supported. It follows that M t is communicating, and it follows from standard results Puterman [1994] that its span fix-points ˜ h do exist and that ˜ gt : = L ˜ ht -˜ ht ∈ R e does not depend on the initial state.

Moreover, if e M ∈ M t and π ∈ Π with ˜ g π ≡ g ( π, M t ) ∈ R e , letting ˜ r π : = r π ( ˜ M ) and ˜ P π : = P π ( ˜ M ), we have:

So by induction and since L t is obviously monotone and linear, we show that:

Dividing by n and letting it go to infinity, we obtain g ( π, M t ) ≤ ˜ gt . Observe that we have equility by taking the policy achieving (˜ gt , ˜ ht ).

To see that EVI converges indeed, simply observe that Lemma 15 provides a finite bound on how much time is required until the sp GLYPH<16> L n + 1 t u - L n t u GLYPH<17> ≤ ϵ . Hence sp GLYPH<16> L n + 1 t u - L n t u GLYPH<17> vanishes to 0. □

About Assumption 3. The assumptions made by Corollary 16 are met if the kernel confidence regions are:

Built out of Weissman's inequality ( C1 ) (see the next section, also Auer et al. [2009]);

Built out of Bernstein's inequality ( C2 ) (because the maximization algorithm to compute ˜ pt ( s , a ) ui in EVI has the same greedy properties than with Weissman's inequality);

Trivial ( C4 ).

For confidence regions build with empirical likelihood estimates ( C3 ), there is no guarantee of convergence (although we conjecture that one could be established), although the gain is still welldefined because M t remains communicating. However, just like the original work of Filippi et al. [2010], the convergence is always met numerically.

A.4 Proof of Theorem 5: Complexity of PMEVI with Weissman confidence regions

In this section, we show that when one is using Weissman confidence regions for kernels ( C1 ), then the iterates of L t converge to an ϵ span-fix-point quickly.

Proposition 17. Assume that PMEVI-DT uses kernel confidence regions of Weissman type ( C1 ) satisfying Assumption 1. Then with probability 1 -δ , the number of iterations of PMEVI (see Algorithm 2) is O GLYPH<16> D √ S AT GLYPH<17> , hence the algorithm has polynomial per-step amortized complexity.

Proof. With Weissman type confidence regions for kernels, for all t ≤ T and ( s , a ) ∈ X , we have

It follows that, for all t ≤ T , the extended Bellman operator L t satisfies the prerequisite ( ∗ ) of Lemma 15 with

Under Assumption 1, we have M ∈ M t with probability 1 -δ . Under this event, M t is weakly communicating and sp ( h ∗ ( M t )) ≤ D ( M ), we can apply Lemma 15 and conclude that every calls to PMEVI (Algorithm 2) takes

where we use that ϵ = q log( S AT /δ ) T , that sp ( w 0 ) = O GLYPH<0> sp ( h ∗ ( M t )) GLYPH<1> = O( D ( M )) and that δ ≥ 1 T . Since the number of episodes under the doubling trick (DT) is O( S A log( T )), we conclude accordingly. □

Every call to the projection operator solves a linear program. Although in theory, this time is polynomial (relying on recent work on the complexity of LP such as Cohen et al. [2020], it is the current matrix multiplication time O( S 2 . 38 )), in practice, reducing the number of calls to the projection operator is key to run PMEVI-DT in reasonable time.

B Analysis of the projected mitigated Bellman operator

In this section, we fix the model region M , the bias region H and the mitigation vector β , dropping the sub-script t for conciseness. We denote ˆ r , ˆ p the respective empirical reward and kernel. Further assume that H = H 0 + R e with H 0 a compact convex set. The associated projection operation (see Appendix B.2) is denoted Γ . The (vanilla) extended Bellman operator L associated to M is given by L u ( s ) : = max a ∈A ( s ) GLYPH<8> sup R ( s , a ) + sup P ( s , a ) u GLYPH<9> . The β -mitigated extended Bellman operator associated to M is:

The function Greedy( M , u , β ) returns a stationary deterministic policy that picks its actions among the one reaching the maximum above. The projection of L β to H is

The goal of this section is to establish Proposition 2 and

Proposition 2 statement (1) is a consequence of Lemma 22;

Proposition 2 statement (2) follows from Theorem 25;

Proposition 2 statement (3) follows from Corollary 27;

Proposition 2 statement (4) follows from Corollary 21;

Proposition 2 prerequisites on the projection operator and Lemma 4 follows from Lemma 19.

B.1 Finding an optimistic policy under bias constraints

The main goal is to find and optimistic policy under bias constraints (projection) and bias error constraints (mitigation). The bias constraints imply that we search for a policy π together with a model e M such that h π ( e M ) ∈ H . The bias error means that, for ˜ h ≡ h π ( e M ), we want in addition ˜ p ( s , π ( s )) ˜ h ≤ ˆ p ( s , π ( s )) ˜ h + β ( s , π ( s )) where ˜ p is the transition kernel of e M . In the end, our goal is to track the solution of the following optimization problem:

where the supremum is taken with respect to the product order R S . In particular, if U ⊆ R S , check that u ∗ = sup U is obtained as u ∗ ( s ) : = sup { v ( s ) : v ∈ U} . The constraint sp( g π ( e M )) = 0 is suggested by the work of Fruit et al. [2018], Fruit [2019] and is key for the problem to be solvable.

The bias constraint and the constraint involving β make the problem impossible to handle with a 'pure' extended MDP solution, which is why the extended Bellman operators are mitigated (with β ) then projected (with Γ ). The mitigation operation guarantees that the constraint involving β is satisfied, while the projection on H makes sure that the bias constraint is satisfied. It is important for both operations to be compatible, i.e., that the constraint involving β that L β forces is not lost when applying Γ . As a matter of fact, projecting then mitigating would not work.

We now explain why L can be used to solve (18).

B.2 Projection operation and definition of L

We start by discussing why L is well-defined at all. The well-definition of L β is obvious. The point is to explain why the projection onto H is possible while preserving mandatory structural properties such as monotony, non-expansivity, linearity and more. For general H , such properties are impossible to meet. But the bias confidence region constructed with Algorithm 3 has a specific shape that makes the projection possible. The central property is the one below:

( A1 ) The downward closure { v ≤ u : v ∈ H} of every u ∈ R S has a maximum in H .

The only order that we will be considering is the product order on R S . Recall that a set U ⊆ R S has a maximum if there exists u ∈ U such that v ≤ u for all u ∈ U . A supremum of U is a

minimal upper-bound of U , i.e., u such that (1) v ≤ u for all v ∈ U and (2) no w satisfying (1) can be smaller than u . For the product order, the supremum of a subset U is unique and of the form u ( s ) = sup { v ( s ) : v ∈ U} .

Define the projection Γ : R S →H as such:

In general, Assumption ( A1 ) is satisfied when H admits a join, i.e., is stable by finite supremum: u , v ∈ H ⇒ sup( u , v ) ∈ H .

Lemma 18. If H is generated by constraints of the form h ( s ) -h ( s ′ ) -c ( s , s ′ ) ≤ d ( s , s ′ ) , then it has a join and ( A1 ) is satisfied. Moreover, Γ is then correctly computed with Algorithm 4.

Proof. The first half of the result is well-known, see Zhang and Xie [2023], but we recall a proof for self-containedness. Let v 1 , v 2 ∈ H and define v 3 : = sup( v 1 , v 2 ). Observe that v 3 ( s ) -v 3 ( s ′ ) ≤ max( v 1 ( s ) -v 1 ( s ′ ) , v 2 ( s ) -v 2 ( s ′ )) ≤ c ( s , s ′ ) + d ( s , s ′ ). So v 3 ∈ H .

We continue by showing that if H has a join, then (19) is well-defined. For s ∈ S , take a sequence v s n such that v s n ( s ) → α ( s ) : = sup { v ( s ) : v ≤ u , v ∈ H} . Because the span of every element of H is upper-bounded by c : = sup GLYPH<8> sp ( v ) : v ∈ H GLYPH<9> , it follows that v s n evolves in the compact region { v ≤ u : v ∈ H} ∩ { v : ∥ v -α se ∥∞ = 1 + c } . We can therefore extract a convergent sequence of v s n , converging v s ∗ that belongs to H since the latter is closed. By construction, v s ∗ ( s ) = α ( s ). Because H has a join, v ∗ : = sup GLYPH<8> v s ∗ : s ∈ S GLYPH<9> ∈ H . □

Lemma 19. Under assumption ( A1 ), the operator Γ u : = max { v ≤ u : v ∈ H} is well-defined, and is:

monotone: u ≤ v ⇒ Γ u ≤ Γ v;

non span-expansive: sp ( Γ u -Γ v ) ≤ sp ( u -v ) ;

linear: Γ ( u + λ e ) = Γ u + λ e;

Γ u ≤ u.

Proof. The well-definition of Γ is obvious from ( A1 ). For (2), if u ≤ v then w ≤ u ⇒ w ≤ v . Hence Γ u : = max { w ≤ u : w ∈ H} ≤ max { w ≤ v : w ∈ H} = : Γ v . For (3), check that it follows from H = H + R e . For (4), we obviously have Γ u : = max { v ≤ u : v ∈ H} ≤ u .

The more di ffi cult point is (2) span non-expansivity. Pick u , v ∈ R S . By linearity, it su ffi ces to show the result for P s u ( s ) = P s v ( s ). In that case, we have sp ( v -u ) = max( v -u ) + max( u -v ). Observe that for all w ≤ u , we have w + min( v -u ) e ≤ v . Since H = H + R e , it follows that:

Similarly, we have max { w ≤ u : w ∈ H} ≥ max { w ≤ v : w ∈ H} + min( v -u ) e . Using them both at once, we find sp ( Γ u -Γ v ) ≤ sp ( v -u ). □

The properties (1), (3) and (4) are essential for L to properly address the optimization problem (18). The property (2) is just as important, because it plays a central part in the convergence of value iteration. The next result shows similar properties for the β -mitigated extended Bellman operator L β . From now on, we will assume (A1), because it is almost-surely satisfied by the bias confidence region generated by Algorithm 3.

Lemma 20. The β -mitigated extended Bellman operator L β is (1) monotone, (2) non-span-expansive and (3) linear.

Proof. The properties (1) and (3) directly follow from the definition. We focus on (2). Fix u , u ′ ∈ R S . By Lemma 26, we can write L β u = ˜ r π + ˜ P π u and L β u ′ = ˜ r π ′ + ˜ P π ′ u ′ . In the following, we write βπ ( s ) : = β ( s , π ( s )). Check that:

If the minimum is reached with ˜ P u ′ , then:

π

If the minimum is reached with ˆ P π u ′ + βπ , then upper-bound ˜ P π u by ˆ P π u + βπ to obtain:

Overall, we find that there exists Q π ∈ P π such that L β u - L β u ′ ≤ Q π ( u -u ′ ). Similarly, we find Q π ′ ∈ P π ′ such that L β u - L β u ′ ≥ Q π ′ ( u -u ′ ). We conclude that:

This concludes the proof.

By composition, we obtain the following result.

Corollary 21. L is (1) monotone, (2) non-span-expansive and (3) linear. Moreover, sp ( L u -L v ) ≤ sp ( L u - L v ) for all u , v ∈ R S .

B.3 Fix-points of L and (weak) optimism

Lemma 22. L has a fix-point in span semi-norm, i.e., ∃ u ∈ H , sp ( L u -u ) = 0 .

Proof. The idea is to apply Brouwer's fix-point theorem in R S quotiented by the equivalence relation u ∼ v ⇔ sp ( u -v ) = 0, where sp ( -) becomes a norm. By linearity (Corollary 21), L is well-defined in this quotient space, and if L is shown continuous on R S , so will it be on the quotient.

We show that L is sequentially continuous on H . Consider a sequence un ∈ H N converging to u ∈ H and fix ϵ > 0. Provided that n > N ϵ for N ϵ large enough, we have ∥ un -u ∥∞ < ϵ , i.e., un -ϵ e ≤ un ≤ u + ϵ e . Therefore, in the one hand, for all v ≤ un , we have v -ϵ e ≤ u so max { v ≤ un : v ∈ H} ≤ max { v ≤ u : v ∈ H} + ϵ e ; And on the other hand, for all v ≤ u , v + ϵ e ≤ un so max { v ≤ u : v ∈ H} ≤ max { v ≤ un : v ∈ H} + ϵ e . Hence:

It shows that Γ is continuous. The operator L β is obviously continuous as well, so L = Γ ◦ L β is continuous by composition. Since H = H 0 + R e with H 0 compact and ocnvex, the quotient H / ∼ is compact and convex, and is preserved by L / ∼ . By Brouwer's fix-point theorem, L / ∼ has a fix-point in H / ∼ . So L has a span fix-point in H . □

We write Fix( L ) the span fix-points of L .

Lemma 23. L has well-defined growth. Specifically, if L u = u + g e, then:

There exists c > 0 , s.t., for all v ∈ H 0 , ( n g -c ) e + u ≤ L n v ≤ ( n g + c ) e + u;

If u ′ ∈ Fix( L ) , then L u ′ -u ′ = g e.

Proof. Setting c : = max v ∈H 0 ∥ v -u ∥∞ < ∞ , one can check that u -ce ≤ v ≤ u + ce for all v ∈ H 0 . this proves (1) for n = 0 and we then proceed by induction on n ≥ 0. By induction, L n v ≤ u + ( n g + c ) e and by Corollary 21, L is monotone, so we have:

where the last inequality use the linearity of L together with L u = u + g e . The lower bound of L n v is shown similarly, establishing (1).

For (2), pick u ′ ∈ Fix( L ) with L u ′ = u ′ + g ′ e . Up to translating u ′ , we can assume that u ′ ∈ H 0 and apply (1). We get:

Divided by n and let it go to infinity. We conclude that g = g ′ .

We finally have everything in hand to claim that L solves (18).

Corollary 24. The growth of L given by g = L u -u for u ∈ Fix( L ) is well-defined, and:

□

Moreover, g ≥ g ∗ ( H , β, M ) .

Proof. The growth property is a direct consequence of Lemma 23. We show g ≥ g ∗ ( H , β, M ) which is defined in (18). Pick π ∈ Π , e M ∈ M its model with ˜ h ≡ h ( π, e M ) and ˜ P π ˜ h ≤ ˆ P π ˜ h + βπ where βπ ( s ) : = β ( s , π ( s )). Up to translation, we can assume that ˜ h ∈ H 0 .

We have g ( π, e M ) = ˜ ge for ˜ g ∈ R , so

by definition. By monotony of L , see Corollary 21, n ˜ ge + ˜ h ≤ L n ˜ h follows by induction on n ≥ 0. By Lemma 23, we further have L n ˜ h ≤ n ( g + c ) e + u where u ∈ Fix( L ). In tandem,

Letting n →∞ , we deduce that ˜ g ≤ g . Conclude by taking the best π and e M .

The next theorem follows directly with the same proof technique, and guarantees optimism.

Theorem 25. Assume that g ∗ + h ∗ ≤ L h ∗ . Then g ≥ g ∗ .

The condition ' g ∗ + h ∗ ≤ L h ∗ ' can be referred to as a weak form of optimism. We qualify this version of optimism as weak because it is much weaker than optimism property suggested by Fruit [2019] L ≥ L where L is the Bellman operator of the true MDP. Here, we only ask for L h ∗ ≥ Lh ∗ , i.e., optimism at a span fix-point of L . This condition is met as soon as M ∈ M , h ∗ ∈ H and β large enough.

B.4 Modelization of the projected mitigated Bellman operator L

The aim of this paragraph is to establish Corollary 27, stating that L u can be viewed as a policy produced by Greedy ( M , u , β ).

Lemma 26 (Modelization) . For π ∈ Π , denote βπ ( s ) : = β ( s , π ( s )) , R π : = Q s R ( s , π ( s )) and P π : = Q s P ( s , π ( s )) . Fix u ∈ R S and let π : = Greedy ( M , u , β ) .

If P is convex, then there exists (˜ r π, ˜ P π ) ∈ R π × P π such that L β u = ˜ r π + ˜ P π u.

Assume that L β u = ˜ r π + ˜ P π u. There exists r ′ π ≤ ˜ r π such that L u = r ′ π + ˜ P π u.

The convexity requirement of (1) is always true if the kernel confidence region is chosen via ( C1-4 ).

Proof. For (1), fix a state s ∈ S , let a : = π ( s ) and ρ : = min(sup P ( s , a ) u , ˆ p ( s , a ) u + β ( s , a )). If ρ = sup P ( s , a ) u , then there is nothing to say because P is compact, hence the sup is a max and ρ is of the form ˜ p ( s , a ) u . Otherwise, let ˜ p ( s , a ) u > ˆ p ( s , a ) u + β ( s , a ) with ˜ p ( s , a ) ∈ P ( s , a ). Introduce, for λ ∈ [0 , 1],

By continuity, there exists λ ∈ (0 , 1) such that ˜ p λ ( s , a ) u = ˆ p ( s , a ) u + β ( s , a ) and by convexity of P ( s , a ), ˜ p λ ( s , a ) ∈ P ( s , a ). This proves (1).

For (2), recall that L u = Γ L β u = Γ (˜ r π + ˜ P π u ). Since Γ v ≤ v , for v ∈ R S , we have:

Set r ′ π : = Γ (˜ r π + ˜ P π u ) -˜ P π u . Check that r ′ π satisfies r ′ π ≤ ˜ r π and L u = r ′ π + ˜ P π u . □

The last corollary bellow is crucial to claim that greedy policies are good choices in PMEVI-DT .

Corollary 27 (Greedy modelization) . Let u ∈ R S and fix π : = Greedy ( M , u , β ) . If P is convex, then with the notations of Lemma 26, there exists ˜ r π ≤ sup R π and ˜ P π ∈ P π such that L u = ˜ r π + ˜ P π u.

□

C Proof of Theorem 5: Regret analysis of PMEVI-DT

Notations. At episode k , the played policy is denoted π k . As a greedy response to h k , by Proposition 2 (3), there exists ˜ rk ( s ) ≤ sup R t k ( s , π k ( s )) and ˜ Pk ( s ) ∈ P t k ( s , π ( x )) such that h k + g k = ˜ rk + ˜ Pk h k . The reward-kernel pair ˜ Mk = (˜ rk , ˜ Pk ) is referred to as the optimistic model of π k . We write Pk : = P π k ( M ) the true kernel and ˆ Pk : = P π k ( ˆ Mtk ) the empirical kernel. Likewise, we define the reward functions rk and ˆ rk . The optimistic gain and bias satisfy g k = g ( π k , e Mk ) and h k = h ( π k , e Mk ). We further denote c 0 = T 1 5 .

Important remark. To slightely simplify the analysis, we assume that PMEVI is run with perfect precision ϵ = 0, i.e., that h k = PMEVI ( M t k , β t k , Γ t k , 0) hence is a span fix-point of L t k . This assumption is mild and can be dropped by adding an extra error term that has to be carried out in the calculations.

C.1 Number of episodes under doubling trick (DT)

Lemma 28 (Number of episodes, Auer et al. [2009]) . The number of episodes up to time T ≥ S A is upper-bounded by:

C.2 Sum of bias variances

The Lemma 29 below shows that P T -1 t = 0 V ( p ( Xt ) , h ∗ ) scales as T sp ( h ∗ )sp ( r ) + sp ( h ∗ ) Reg( T ) in probability.

Lemma 29. With probability at least 1 -δ , we have:

Proof. Using the Bellman equation h ∗ ( s ) + g ∗ ( s ) = r ( s , a ) + p ( s , a ) h ∗ + ∆ ∗ ( s , a ), we have:

Since sp GLYPH<16> h ∗ 2 GLYPH<17> ≤ sp ( h ∗ ) 2 , we get:

where the last inequality holds with probability 1 -δ . This concludes the proof. □

C.3 Regret and pseudo-regret: A tight relation

In this paragraph, we bound the regret with respect to the pseudo-regret (and conversely) up to a factor of order (sp ( h ∗ )sp ( r ) log( T δ )) 1 / 2 . Hence, in proofs, the pseudo-regret can be changed to the regret with ease.

Lemma 30. With probability 1 -4 δ , the regret and the pseudo-regret and linked as follows:

Proof. We rely again on the Poisson equation g ∗ ( S t ) -r ( Xt ) -∆ ∗ ( Xt ) = ( p ( Xt ) -eSt ) h ∗ , so:

Up to the constant sp ( h ∗ ), the two error terms are respectively a navigation and a reward error. The second is bounded using Azuma's inequality (Lemma 32), showing that with probability 1 -2 δ , we have:

We continue by using Freedman's inequality, instantiated in the form of Lemma 33. With probability 1 -δ , we have:

The quantity P T -1 t = 0 V ( p ( Xt ) , h ∗ ) is a classical one that appears at several places throughout the analysis. Using Lemma 29, we bount it explicitely. Further simplifying the bound with √ a + b ≤ √ a + √ b , we get that with probability 1 -4 δ , we have:

Bound log( 1 δ ) by log( T δ ) and use √ a + √ b ≤ 2 √ a + b to merge the terms in q T log( T δ ) under a single square-root. □

Overall, Lemma 30 states that the regret P T -1 t = 0 ( g ∗ -Rt ) and the pseudo-regret P T -1 t = 0 ∆ ∗ ( Xt ) di ff er by about (sp ( h ∗ ) T log( T δ )) 1 / 2 in probability (up to asymptotically negligible additional terms). In general, the precise form of Lemma 30 is not convenient to use because it is of form form x ≤ y + α √ y + β that is not linear in y . Corollary 31 factorizes the result into one which will be more convenient in proofs.

Corollary 31. Denote x : = P T -1 t = 0 ( g ∗ -Rt ) and y : = P T -1 t = 0 ∆ ∗ ( Xt ) . Further introduce:

Then, with probability 1 -4 δ , we have √ x ≤ √ y + 1 2 α + √ β and √ y ≤ √ x + α + √ β .

Proof. This is straight forward algebra from the result of Lemma 30.

C.4 Proof of Lemma 6, reward optimism

We start by getting rid of the reward noise. We have:

with probability 1 -δ by Azuma's inequality (Lemma 32). We are left with P T -1 t = 0 ( g ∗ -r ( Xt )). We continue by splitting the regret episodically and invoking optimism. By Lemma 13, with probability 1 -4 δ , we have P T -1 t = 0 ( g ∗ -r ( Xt )) ≤ P k P t k + 1 -1 t = t k ( g k -r ( Xt )). Introduce

We focus on bounding B 0 ( T ). By Assumption 2, ˜ rk ( s , a ) is of the form ˆ rk ( s , a ) + p C log(2 S AT /δ ) / Ntk ( s , a ) -η k ( s , a ) with η k ( s , a ) ∈ R . By the statement (3) of Proposition 2, η k ( s , a ) ≥ 0. Therefore,

□

where ( ∗ ) holds with probability 1 -δ following Lemma 35. By the doubling trick rule (DT), we have Nt ( Xt ) ≤ 2 Ntk ( Xt ) for t < t k + 1, so, with probability 1 -δ ,

We conclude that with probability 1 -6 δ , we have:

This concludes the proof.

C.5 Proof of Lemma 7, navigation error

We have:

The last term is O( c 0 S A log( T )) by Lemma 28, hence is O( T 1 / 5 log( T )).

( STEP 1 ) We start by bounding A1. By Lemma 13, with probability 1 -4 δ , we have h ∗ ∈ H t k for all k ≤ K ( T ). So sp ( h k -h ∗ ) ≤ sp ( h k ) + sp ( h ∗ ) ≤ 2 c 0. By Freedman's inequality invoked in the form of Lemma 33, we have with probability 1 -5 δ ,

It su ffi ces to bound the first term. Recall that e is the vector full of ones. We have:

□

Here the inequality ( ∗ ) holds with probability 1 -δ following Lemma 40. We will bound the summand with the bias estimation error error( ck , s , s ′ ) that spawns the inner regret estimation B 0 ( t k ) = P k -1 ℓ = 1 P t ℓ + 1 -1 t = t ℓ ( g ℓ -Rt ). This inner estimation is linked to B ( T ) : = P k , t ( g k -Rt ) the overall optimistic regret by:

In the above, ( ∗ ) holds with probability 1 -4 δ uniformly on k following Lemma 13 and ( † ) holds, also uniformly on k , with probability 1 -δ by applying Azuma-Hoe ff ding's inequality (Lemma 32). Continuing, still on the event specified by Lemma 13, we have with probability 1 -6 δ :

( STEP 2 ) For A2, by Freedman's inequality invoked in the form of Lemma 33 again, we have with probability 1 -δ ,

We recognize the sum of variance P T -1 t = 0 V ( p ( Xt ) , h ∗ ) that we leave as is.

( STEP 3 ) As a result, with probability 1 -7 δ , we have:

when c 0 = T .

C.6 Proof of Lemma 8, empirical bias error

Because h ∗ is a fixed vector, Bennett's inequality (see Lemma 39) guarantees that ( ˆ pk ( S t ) -pk ( S t ) h ∗ is small as follows. By doing a union bound over Lemma 39 with confidence δ S AT over all pairs ( s , a ) and visits counts N ( s , a ) ≤ T , we see that with probability 1 -δ , for all k , we have:

Summing this over k and factorizing over state-action pairs, we get that with probability 1 -δ ,

We recognize the sum of variances P T -1 t = 0 V ( p ( Xt ) , h ∗ ), that is left to be upper-bounded later on. □

C.7 Proof of Lemma 9, optimistic overshoot

Because of the β -mitigation generated by Algorithm 5, the quantity ( ˜ pk ( S t ) -ˆ pk ( S t )) h k is shown to be directly related to V ( p ( Xt ) , h ∗ ) up to a provably negligible error. Denote h ′ k the reference point BiasProjection ( H t k , ctk ( -, s 0)) used in Algorithm 5 (denoted h 0 in the algorithm). By Lemma 13, with probability 1 -4 δ , we have h ∗ ∈ H t k for all k . To lighten up notations, we write dtk ( s ′ , s ) instead of error( ctk , s ′ , s ).

( STEP 1 ) Denote A : = ( ˜ pk ( S t ) -ˆ pk ( S t )) h k . By construction of ˜ pk , we have A ≤ β t k ( Xt ), so:

The rightmost term of A is of order O(log 2 ( T )) hence is negligible. We focus on the other two. The analysis of A1 will spawn a term similar to A2, hence we start by the second. Recall that dtk is the bias error provided by Algorithm 3 and that the inner regret estimation is B 0 ( t k ) = P k -1 ℓ = 1 P t ℓ + 1 -1 t = t ℓ ( g ℓ -Rt ). Now, remark that:

In the above, ( ∗ ) holds with probability 1 -4 δ uniformly on k following Lemma 13 and ( † ) holds, also uniformly on k , with probability 1 -δ by applying Azuma-Hoe ff ding's inequality (Lemma 32). Therefore, with probability 1 -5 δ , for all k and t ∈ { t k , . . . , t k + 1 -1 } , we have:

This bound will be enough. We move on to A1. We have:

where ( ∗ ) is obtained by applying Lemma 12 and ( † ) holds with probability 1 -δ by applying Weissman's inequality, see Lemma 35. All together, with probability 1 -6 δ , A is upper-bounded by:

( STEP 2 ) The number of visits Nk ( Xt ) is lower-bounded by 1 2 Nt ( Xt ) when Nk ( Xt ) ≥ 1 by doubling trick (DT). By summing over t and k , we find that with probability 1 -6 δ ,

where the last inequality is obtained with computations that are similar to those detailed in the proof of Lemma 8. We recognize the variance that we will leave as is. We finish the proof by bounding the lower order terms A2 and A3.

( STEP 3 ) We start with A2. We have:

( STEP 4 ) We are left with A3. We have:

This concludes the proof.

C.8 Proof of Lemma 10, second order error

Recall that by Lemma 13, with probability 1 -4 δ , h ∗ ∈ H t k for all k , hence sp ( h k -h ∗ ) ≤ 2 c 0 for all k on the same event. Therefore, with probability 1 -4 δ ,

where ( ∗ ) uses that h ∗ ∈ H t k , and ( † ) is obtained by applying the empirical Bernstein's inequality, see Lemma 36, to ˆ pk ( s ′ | S t ) -pk ( s ′ | S t ), and holds with probability 1 -δ . The rightmost term's sum is

□

upper-bounded by:

For the other term, follow the line of the proof of Lemma 9 (term A2). We have with probability 1 -5 δ (4 δ of which is by invoking Lemma 13):

Therefore,

Summing over k , t , s ′ , with probability 1 -6 δ , we have:

This concludes the proof.

D More details on experiments

D.1 River swim as a hard communicating environment

Experiments of Fig. 2 are run on n -states river-swim. Such MDPs are, despite their size, known to be hard to learn. They consists in n states aligned in a straight line with two playable actions right and left whose dynamics are given in the figure below. Rewards are Bernoulli and null everywhere excepted for r ( sn , right ) = 0 . 95 and r ( s 0 , left ) = 0 . 05.

s

1

s

2

· · ·

· · ·

sn

0

.

6

0

.

4

0

.

6

0

.

35

0

.

05

0

.

05

0

.

35

0

.

95

0

.

05

1

1

1

1

3 -state river-swim. The gain is g ∗ ≈ 0 . 82 and h ∗ ≈ ( -4 . 28 , -2 . 24 , 0 . 4).

5 -state river-swim. The gain is g ∗ ≈ 0 . 82 and h ∗ ≈ ( -9 . 62 , -7 . 58 , -4 . 96 , -2 . 27 , 0 . 45).

D.2 Experiments in weakly-communicating environments

Beyond communicating models, PMEVI-DT is superior to EVI -based methods. On Fig. 4, we see that PMEVI-DT can learn in environments of infinite diameter while UCRL2 cannot. The gain is due to the truncation operation, that makes sure that the optimistic bias vector has span less than T 1 / 5 .

s

0

s

1

1

0

.

5

0

.

1

UCRL2

PMEVI

Without the truncation operation, EVI -based methods can never reject the plausibility that the reward at s 0 is maximal (equal to one) and that there is a positive probability ϵ to switch from s 1 to s 0 when taking the sub-optimal action from s 1 . More precisely, denote the actions a , b where r ( s 0 , a ) = 1, r ( s 1 , a ) = 0 . 5 and r ( s 1 , b ) = 0 . 1, so that a is the optimal action and b is sub-optimal. There are two policies π a and π b choosing action a and b from s 1 respectively. Because s 0 is only visited once, the best reward that can be achieved from s 0 is ˜ rt ( s 0 , a ) = 1 at all times. If one runs UCRL2 , the confidence region for transition kernels is roughly of the form P t ( x ) = { ˜ p ( x ) : Nt ( x ) ∥ ˜ p ( x ) -ˆ pt ( x ) ∥ 2 1 ≤ Cp log( t ) } and the plausible transition kernel ˜ p ( x ) ∈ P t ( x ) that goes the quickest from s 1 to s 0 is of the form:

After running EVI for i steps, the current vector vi (see (6)) is V ∗ i ( M t ), the maximal amount of reward that one can collect in i steps on M t seen as an extended Markov decision process, see Auer et al. [2009]. If the data is well concentrated, the optimal reward from s 1 are respectively ˜ rt ( s 1 , a ) ≈ 0 . 5 + p Cr log( t ) / Nt ( s 1 , a ) and ˜ rt ( s 1 , b ) = 0 . 1 + p Cr log( t ) / Nt ( s 1 , b ). From all this, one can argue that when i is large enough, the optimistic scores of π a and π b over i steps are roughly equal to:

So V π a i ( s 1 ; M t ) ≤ V π b i ( s 1 ; M t ) if Nt ( s , b ) ≪ 25 81 Nt ( s , a ).

This means that EVI will output π b if Nt ( s , b ) ≪ 25 81 Nt ( s , a ), leading to Nt ( s , a ) ≍ Nt ( s , b ) so both growing linearly with t . This informal argument can be generalized to EVI -based algorithms with other types of confidence regions: EVI -based methods such as UCRL Auer et al. [2002], UCRL2B Fruit et al. [2020] and KLUCRL Filippi et al. [2010] will su ff er from Nt ( s , b ) = Θ ( t ) and their regret will grow linearly.

In opposition, PMEVI -based methods use truncation, making sure that | vi ( s 1 ) -vi ( s 0 ) | ≤ T 1 / 5 at all times. Intuitively, it makes PMEVI -based method 'think' that ˜ pt ( s 0 | s 1 , b ) cannot be as small as p Cp log( t ) / Nt ( s 1 , b ), because the optimistic bias of π b would be too large otherwise; Or, equivalently, that ˜ pt ( s 0 | s 1 , b ) = p Cp log( t ) / Nt ( s 1 , b ) but with ˜ rt ( s 0 , a ) ≪ 1, hence killing the optimistic reward at ( s 0 , a ) to meet the bias constraints.

Overall, these features of PMEVI -based methods are shared with algorithms such as REGAL Bartlett and Tewari [2009] and SCAL Fruit et al. [2018]. The di ff erence is that these methods require precise prior information on sp( h ∗ ) that PMEVI-DT does not need.

E Standard concentration inequalities

Lemma 32 (Azuma's inequality, Azuma [1967]) . Let ( Ut ) t ≥ 0 a martingale di ff erence sequence such that sp ( Ut ) ≤ c a.s., i.e., there exists a t ∈ R such that at ≤ Ut ≤ at + c a.s. Then, for all δ > 0 ,

Lemma 33 (Freedman's inequality, Zhang et al. [2020]) . Let ( Ut ) t ≥ 0 a martingale di ff erence sequence such that | Ut | ≤ c a.s., and denote its conditional variance Vt : = E [ U 2 t |F t -1 ] . Then, for all δ > 0 ,

Lemma 34 (Time-uniform Azuma, Bourel et al. [2020]) . Let ( Ut ) a martingale di ff erence sequence such that, for all λ ∈ R , E [exp( λ Ut ) | U 1 , . . . , Ut -1 ] ≤ exp( λ 2 σ 2 2 ) . Then:

Lemma 35 (Time-uniform Weissman) . Let q a distribution over { 1 , . . . , d } . Let ( Ut ) a sequence of i.i.d. random variables of distribution q. Then:

Proof. Remark that GLYPH<13> GLYPH<13> GLYPH<13> P n k = 1 ( eUk -q ) GLYPH<13> GLYPH<13> GLYPH<13> 1 = max v ∈{-1 , 1 } d P n k = 1 GLYPH<10> eUk -q , v GLYPH<11> . Let W v k : = GLYPH<10> eUk -q , v GLYPH<11> . Remark that for each v ∈ {-1 , 1 } d , ( W v k ) is a family of i.i.d. random variables with -⟨ q , v ⟩ ≤ W v k ≤ 1 - ⟨ q , v ⟩ , so E [exp( λ W v k )] ≤ exp( λ 2 8 ) by Hoe ff ding's Lemma. By Lemma 34, we have:

This concludes the proof.

□

Lemma 36 (Time-uniform Empirical Bernstein) . Let ( Uk ) k ≥ 1 a martingale di ff erence sequence such that sp ( Un ) ≤ c a.s., let ˆ Un : = 1 n P n k = 1 Uk the empirical mean and ˆ Vn : = 1 n P n k = 1 ( Uk -ˆ Un ) 2 the population variance. Then,

Proof. This is obtained with a union bound on the values of n ≤ T , then applying Lemma 38. □

Lemma 37 (Time-uniform Empirical Likelihoods, Jonsson et al. [2020]) . Let q a distribution on { 1 , . . . , d } . Let ( Ut ) a sequence of i.i.d. random variables of distribution q. Then:

Lemma 38 (Empirical Bernstein inequality, Audibert et al. [2009]) . Let ( Uk ) k ≥ 1 a martingale di ff erence sequence such that sp ( Un ) ≤ c a.s., let ˆ Un : = 1 n P n k = 1 Uk the empirical mean and ˆ Vn : = 1 n P n k = 1 ( Uk -ˆ Un ) 2 the population variance. Then,

Lemma39 (Bennett's inequality, Audibert et al. [2009]) . Let ( Ut ) t ≥ 0 a martingale di ff erence sequence such that | Ut | ≤ c a.s., and denote its conditional variance Vt : = E [ U 2 t |F t -1 ] . Then,

Lemma 40 (Lemma 3 of Zhang and Xie [2023]) . Let ( Ut ) be a sequence of random variables such that 0 ≤ Ut ≤ c a.s., and let F t : = σ ( U 0 , U 1 , . . . , Ut -1 ) . Then:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that the algorithm reaches minimax optimal regret, and it does. We claim that our algorithm can e ff ectively use bias information, and the experimental illustration supports the claim.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and / or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: The paper is a theoretical one that covers all weakly communicating MDPs, and although it may help the design of algorithms intended to be applicable in real-life scenarios, our method is not meant to be directly applied.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational e ffi ciency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: To every result is attached a proof or a reference.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it a ff ects the main claims and / or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The algorithmic content in the main body is detailed enough to implement the algorithm, the exception being the confidence intervals and the implementation of EVI (because those come from already existing works). The confidence intervals are discussed in a dedicated section in the Appendix, while the works of Auer et al. [2009], Filippi et al. [2010], Fruit et al. [2020] etc. provide detailed ready-to-implement pseudo-code for EVI . Our code is mostly written in Python. Experiments took a few dozen minutes on a low end laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and / or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might su ffi ce, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with su ffi cient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the supplementary material, together with the scripts to reproduce the exact figures of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting / Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: People that are unfamiliar with Markov decision processes may not know the river-swim setting in which experiments are driven, but it is not required to understand the figures and the discussion. Moreover, a description of the environment (river-swim) is provided in the appendix in addition to be referenced. See also Experimental Result Reproducibility for more.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did a hundred runs for every algorithm, which is enough to get statistical significance for the small sized problems that we experiment on. Also, experiments are mostly illustrative and do not account for an extensive numerical validation of the theoretical results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train / test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide su ffi cient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: This is mostly a theoretical paper. Most of the computational cost of the work is processing L A T E X and rendering a PDF. Like said earlier, experiments all together took less than a hour on a low end laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: -

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: It's mainly a theoretical work on small sized Markov decision processes.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the e ffi ciency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: -

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing e ff ective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith e ff ort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: -

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: -

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset / code / model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: -

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval / review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: -

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the first tractable algorithm that achieves minimax optimal regret for average reward tabular MDPs. The algorithm does not require prior information on the span of the optimal bias function.\n\nSTRENGTHS:\nThe paper proposes the first tractable algorithm that achieves minimax optimal regret for average reward tabular MDPs.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. The minimax lower bound in [4] is sqrt{DSAT}. Because H=span(h^*) <= D, I wonder why sqrt{HSAT} is even achievable by the algorithm developed in the paper. \n\n2. The authors cited [14, 25] for the lower bound sqrt{HSAT}, but I can not find this result/theorem in these two papers with a rigorous proof.  In fact, [14] mentioned it is an open problem whether the “actual” lower bound depends on diameter D or the bias span H. Please provide a proper reference if the actual lower bound depends on H. \n\n3. As shown in Figure 2, PMEVI behaves almost the same with its EVI counterparts when no prior information on bias is given. This does not illustrate the advantage of PMEVI compared with UCRL2 or KLUCRL, even though the regret of the algorithm in this paper is claimed to be better. \n\n4. In Figure 2 (to the right), PMEVI is run with bias information c, but it is compared with UCLR2 which does not use c. This is not fair comparison. If PMEVI is given bias information, then it should be compared with other algorithms such as SCAL or UCB-AVG which require bias information for implementation. \n\n5. Computing max_u beta_t(s,a,u) may not be tractable in general. Algorithm 5 provides a way to bound/approximate this quantity. I wonder what is the intuition? I also can not find the proof that it is indeed a bound that does not impact the regret efficiency.  \n\n6. The regret proof in Section 4 does not highlight how the key components projection and mitigation in the algorithm impact the regret. In particular, it is not clear from the proof sketch where these components are used, and how the beta-mitigated extended bellman operator is applied to achieve the minimax optimal regret.\n\nLIMITATIONS:\nI do not find the discussions of limitations in the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel algorithm for weakly communicating Markov Decision Processes (MDPs) with an average-reward objective. This algorithm, at the same time, is a) tractable since it does not rely on the exact solution to a high-dimensional non-convex optimization problem as prior work, and b) achieves minimax optimal regret up to logarithmic factors.\n\nSTRENGTHS:\n- First tractable algorithm that achieves minimax optimal regret bound;\n- The algorithm does not require any prior knowledge of the span of optimal bias function;\n- The algorithm does not require any reduction to a discounted setting and works directly in the average-reward setting.\n\nWEAKNESSES:\n- The algorithm requires solving the linear programs for each doubling epoch, which is tractable but non-generalizable beyond the tabular setting;\n- The large second-order term is dominated by the main asymptotic optimal term if $T > S^{20}$. This effect of a large second-order term is also observable during the experimental section.\n\nQUESTIONS:\n- Discussion in Section 3.3 is a little bit misleading since it is formulated in terms of arbitrary vector u and may think that, afterward, the union bound over possible u should be applied to write down a bound of type $\\max_{u} \\beta_{t}(s,a,u)$ (and thus inducing additional $\\sqrt{S}$ factor due to equivalent with purely $\\ell_1$-confidence regions),  whereas you need to do it only for $u = h^\\star$. I would suggest rewriting this discussion a little bit to avoid the confusion I experienced during the first reading of the text.\n- Appendix C6. It's not clear what (2k) means in the equation after line 796. I have the same question about (3k) and (4k) after line 816 and 825.\n- Additionally, I would appreciate the additional discussion on the difference between regret minimization and best policy identification settings for average-reward MDPs, especially in a glance at existing concurrent work Tuynman et al. 2024.\nTuynman, A., Degenne, R., & Kaufmann, E. (2024). Finding good policies in average-reward Markov Decision Processes without prior knowledge. arXiv preprint arXiv:2405.17108.\n\nLIMITATIONS:\nThe paper is of a theoretical nature and thus does not have any direct impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning in average reward MDPs, and presents the first minimax optimal algorithm (in terms of $sp(h^*)$) which is computationally tractable, and also simultaneously the first which does not require prior knowledge of $sp(h^*)$.\n\nSTRENGTHS:\nThe main theorem resolves a longstanding open problem which has been the subject of extensive research effort. Removal of knowledge of $sp(h^*)$ is a challenging problem in many settings beyond the one considered in this paper, and thus I hope that this work might lead to progress on this issue in other areas.\n\nWEAKNESSES:\nThe algorithm is rather complicated and has many components. The presentation also does not make it easy to determine the key ingredients of this method. While it is interesting that this method may incorporate different confidence regions, I wonder if it might be more clear to present things in less generality.\n\nIt is nice that experiments are included, but if anything they seem to contradict the claim that the method achieves minimax regret without prior knowledge of $sp(h^*)$ (due to the left plot in Figure 2). Since the bias estimation subroutine seems to be a key ingredient, it would be much better if an experiment was included where this subroutine was making a nontrivial contribution (by my understanding, in the left plot the bias estimation routine is not doing anything since it hasn't been given enough time, and in the right plot, it is not doing anything since much better prior information has been given to the algorithm).\n\nQUESTIONS:\nWhy is the main theorem stated with the parameter $c$? It seems like the best choice is always $c=sp(h^*)$? (And if $c \\gg sp(h^*)$, then the theorem is not giving the minimax optimal regret.) Does the theorem hold for all $c$ simultaneously or is this parameter used somewhere that I didn't notice?\n\nThe main theorem mentions a confidence region system for communicating MDPs with a diameter-dependent computational complexity, but since the paper is for weakly communicating MDPs which generally have infinite diameter, I think a different system would be a better choice? Can the author(s) comment on the complexity when the diameter is infinite?\n \nMany times notation like $h^*(\\mathcal{M})$ is used, but is this well-defined? There can be multiple solutions to the Bellman optimality equation.\n\nThe paper https://arxiv.org/pdf/1905.12425 claims to achieve the optimal diameter-based $\\sqrt{DSAT}$ regret, maybe it should be added to the related work.\n\nIn the display below line 785, the number of commutes between $S_t$ and $S_{t+1}$ is lower bounded by the number of transitions between these states. A similar bound is used under line 830. Does this suggest that the bias difference estimator could actually be formulated using only direct transitions between pairs of states?\n\n\nLine 397: $h^*$ should be $g^*$?\n\nLine 404: Some character I believe meant to be $S$ was used\n\nLIMITATIONS:\nNo major limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that by replacing the extended value iteration in optimistic methods, like UCRL2, it is possible to obtain regret that scales optimally with the number of states, actions, the time horizon and the span of the optimal value function, instead of the diameter, despite not knowing the diameter, assuming weakly communicating MDPs. In addition, a novel analysis is presented that also gives a polynomial bound on the compute cost of the algorithm. The new extended value iteration method is designed based on new ideas to refine the set of plausible MDPs considered in a given step: For this, an inequality is provided that relates the deviation of the values assigned to two states to observable quantities; creating a new set of constraints on the MDPs considered.\n\nSTRENGTHS:\nA major breakthrough if the proof holds up. Interesting insight.\n\nWEAKNESSES:\nThe presentation is not great; the paper feels rushed, the paper is full of (minor) grammatical and typographical errors (starting with the abstract: \"encounter suffer\"). While I did not go through the whole proof, there are many minor problems that are apparent: The paper is not very careful in tracking error events. For example, \\tilde{beta} is sometimes appearing in a condition, but it is also constructed by the algorithm, perhaps things work out, but details are definitely missing.\nThe experimental results are unconvincing: Experiments that show scaling the optimal span would have been more convincing. The presented experiments do not help at all.\n\nQUESTIONS:\nAre you sure you tracked all the error events correctly? (E.g. in Lemma 13, Lemma 3 is used, which needs $\\tilde{g}\\ge g^*$, but the proof of Lemma 13 does not mention why this would hold and exactly what value of $\\tilde{g}$ is used here.)\nOne of the confusing aspects of the presentation of the insight was that on page 5, where the mitigation is explained, beta_t(s,a,u) is used and then a maximum over u is taken; while it is not explained whether beta_t(s,a,u) bounds the deviation that is a function of u for all u (needs a covering?). I guess this covering is not needed, but then why can we take the max as suggested in this paragraph (lines 155-161).\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}