Title: Achieving Constant Regret in Linear Markov Decision Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement learning

Constant regret

Misspecified linear MDP
Abstract: We study the constant regret guarantees in reinforcement learning (RL). Our objective is to design an algorithm that incurs only finite regret over infinite episodes with high probability. We introduce an algorithm, Cert-LSVI-UCB, for misspecified linear Markov decision processes (MDPs) where both the transition kernel and the reward function can be approximated by some linear function up to misspecification level $\zeta$. At the core of Cert-LSVI-UCB is an innovative certified estimator, which facilitates a fine-grained concentration analysis for multi-phase value-targeted regression, enabling us to establish an instance-dependent regret bound that is constant w.r.t. the number of episodes. Specifically, we demonstrate that for a linear MDP characterized by a minimal suboptimality gap $\Delta$, Cert-LSVI-UCB has a cumulative regret of $\tilde{\mathcal{O}}(d^3H^5/\Delta)$ with high probability, provided that the misspecification level $\zeta$ is below $\tilde{\mathcal{O}}(\Delta / (\sqrt{d}H^2))$. Here $d$ is the dimension of the feature space and $H$ is the horizon. Remarkably, this regret bound is independent of the number of episodes $K$. To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

Achieving Constant Regret in Linear Markov Decision Processes

Abstract

We study the constant regret guarantees in reinforcement learning (RL). Our objective is to design an algorithm that incurs only finite regret over infinite episodes with high probability. We introduce an algorithm, Cert-LSVI-UCB , for misspecified linear Markov decision processes (MDPs) where both the transition kernel and the reward function can be approximated by some linear function up to misspecification level ζ . At the core of Cert-LSVI-UCB is an innovative certified estimator, which facilitates a fine-grained concentration analysis for multi-phase value-targeted regression, enabling us to establish an instance-dependent regret bound that is constant w.r.t. the number of episodes. Specifically, we demonstrate that for a linear MDP characterized by a minimal suboptimality gap ∆ , Cert-LSVI-UCB has a cumulative regret of ˜ O ( d 3 H 5 / ∆) with high probability, provided that the misspecification level ζ is below ˜ O (∆ / ( √ dH 2 )) . Here d is the dimension of the feature space and H is the horizon. Remarkably, this regret bound is independent of the number of episodes K . To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a constant, instance-dependent, high-probability regret bound in RL with linear function approximation without relying on prior distribution assumptions.

1 Introduction

Reinforcement learning (RL) has been a popular approach for teaching agents to make decisions based on feedback from the environment. RL has shown great success in a variety of applications, including robotics (Kober et al., 2013), gaming (Mnih et al., 2013), and autonomous driving. In most of these applications, there is a common expectation that RL agents will master tasks after making only a bounded number of mistakes, even over indefinite runs. However, theoretical support for this expectation is limited in RL literature: in the worst case, existing works such as Jin et al. (2020); Ayoub et al. (2020); Wang et al. (2019) only provided ˜ O ( √ K ) regret upper bounds with K being the number of episodes; in the instance-dependent case, Simchowitz and Jamieson (2019); Yang et al. (2021); He et al. (2021a) achieved logarithmic high-probability regret upper bounds (e.g.,

˜ O (∆ -1 log K ) ) for both tabular MDPs and MDPs with linear function approximations, provided a minimal suboptimality gap ∆ . However, these findings suggest that an agent's regret increases with the number of episodes K , contradicting to the expectation of finite mistakes in practice. To close this gap between theory and practice, there is a recent line of work proving constant regrets bound for RL and bandits, suggesting that an RL agent's regret may remain bounded even when it encounters an indefinite number of episodes. Papini et al. (2021a); Zhang et al. (2021) have provided instance-dependent constant regret bound under certain coverage assumptions on the data distribution. However, verifying these data distribution assumptions can be difficult or even infeasible. On the other hand, it is known that high-probability constant regret bound can be achieved unconditionally in multi-armed bandits (Abbasi-Yadkori et al., 2011) and contextual linear bandits if and only if the misspecification is sufficiently small with respect to the minimal sub-optimality gap (Zhang et al., 2023b). This raises a critical question:

Is it possible to design a reinforcement learning algorithm that incurs only constant regret under minimal assumptions?

To answer this question, we introduce a novel algorithm, which we refer to as Cert-LSVI-UCB , for reinforcement learning with linear function approximation. To encompass a broader range of realworld scenarios characterized by large state-action spaces and the need for function approximation, we consider the misspecified linear MDP (Jin et al., 2020) setting, where both the transition kernel and reward function can be approximated by a linear function with approximation error ζ . We show that, with our innovative design of certified estimator and novel analysis, Cert-LSVI-UCB achieves constant regret without relying on any prior assumption on data distributions. Our key contributions are summarized as follows:

We introduce a parameter-free algorithm, referred to as Cert-LSVI-UCB , featuring a novel certified estimator for testing when the confidence set fails. This certified estimator enables Cert-LSVI-UCB to achieve a constant, instance-dependent, high probability regret bound of ˜ O ( d 3 H 5 / ∆) for tasks with a suboptimality gap ∆ , under the condition that the misspecification level ζ is bounded by ζ < ˜ O ( ∆ / ( √ dH 2 ) ) . This bound is termed a high probability constant regret bound , indicating that it does not depend on the number of episodes K . We note that this constant regret bound matches the logarithmic expected regret lower bound of Ω(∆ -1 log K ) , suggesting that our result is valid and optimal in terms of the dependence on the suboptimality gap ∆ .

When restricted to a well-specified linear MDP (i.e., ζ = 0 ), the constant high probability regret bound improves the previous logarithmic result ˜ O ( d 3 H 5 ∆ -1 log K ) in He et al. (2021a) by a log K factor. Our results suggest that the total suboptimality incurred by Cert-LSVI-UCB remains constantly bounded, regardless of the number of episodes K . In contrast to the previous constant regret bound achieved by Papini et al. (2021a), our regret bound does not require any prior assumption on the feature mapping, such as the UniSOFT assumption made in Papini et al. (2021a). To the best of our knowledge, Cert-LSVI-UCB is the first algorithm to achieve a high probability constant regret bound for MDPs without prior assumptions on data distributions. We further show that this constant regret high-probability bound does not violate the logarithmic expected regret bound by letting δ = 1 /K 2 .

Notation. Vectors are denoted by lower case boldface letters such as x , and matrices by upper case boldface letters such as A . We denote by [ k ] the set { 1 , 2 , · · · , k } for positive integers k . We use log x to denote the logarithm of x to base 2 . For two non-negative sequence { a n } , { b n } , a n ≤ O ( b n ) means that there exists a positive constant C such that a n ≤ Cb n ; a n ≤ ˜ O ( b n ) means there exists a positive constant k such that a n ≤ O ( b n log k b n ) ; a n ≥ Ω( b n ) means that there exists a positive constant C such that a n ≥ Cb n ; a n ≥ ˜ Ω( b n ) means there exists a positive constant k such that a n ≥ Ω( b n log -k b n ) ; a n ≥ ω ( b n ) means that lim n →∞ b n /a n = 0 . For a vector x ∈ R d and a positive semi-definite matrix A ∈ R d × d , we define ∥ x ∥ 2 A = x ⊤ Ax . For any set C , we use |C| to denote its cardinality. We denote the identity matrix by I and the empty set by ∅ . The total variation distance of two distribution measures P ( · ) and Q ( · ) is denoted by ∥ P ( · ) -Q ( · ) ∥ TV .

2 Related Work

Instance-dependent regret bound in RL. Although most of the theoretical RL works focus on worst-case regret bounds, instance-dependent (a.k.a., problem-dependent, gap-dependent) regret bound is another important bound to understanding how the hardness of different instance can affect the sample complexity of the algorithm. For tabular MDPs, Jaksch et al. (2010) proved a ˜ O ( D 2 S 2 A ∆ -1 log K ) instance-dependent regret bound for average-reward MDP where D is the diameter of the MDP and ∆ is the policy suboptimal gap. Simchowitz and Jamieson (2019) provided a lower bound for episodic MDP which suggests that the any algorithm will suffer from Ω(∆ -1 ) regret bound. Yang et al. (2021) analyzed the optimistic Q -learning and proved a O ( SAH 6 ∆ -1 log K ) logarithmic instance-dependent regret bound. In the domain of linear function approximation, He et al. (2021a) provided instance-dependent regret bounds for both linear MDPs (i.e., ˜ O ( d 3 H 5 ∆ -1 log K ) ) and linear mixture MDPs (i.e., ˜ O ( d 2 H 5 ∆ -1 log K ) ). Furthermore, Dann et al. (2021) provided an improved analysis for this instance-dependent result with a redefined suboptimal gap. Zhang et al. (2023a) proved a similar logarithmic instance-dependent bound with He et al. (2021a) in misspecified linear MDPs, showing the relationship between misspecification level and suboptimality bound. Despite all these bounds are logarithmic depended on the number of episode K , many recent works are trying to remove this logarithmic dependence. Papini et al. (2021a) showed that under the linear MDP assumption, when the distribution of contexts ϕ ( s, a ) satisfies the 'diversity assumption' (Hao et al., 2020) called 'UniSOFT', then LSVI-UCB algorithm may achieve an expected constant regret w.r.t. K . Zhang et al. (2021) showed a similar result on bilinear MDP (Yang and Wang, 2020), and extended this result to offline setting, indicating that the algorithm only need a finite offline dataset to learn the optimal policy. Table 1 summarizes the most relevant results mentioned above for the ease of comparison with our results.

RL with model misspecification. All of the aforementioned works consider the well-specified setting and ignore the approximation error in the MDP model. To better understand this misspecification issue, Du et al. (2019) showed that having a good representation is insufficient for efficient RL unless the approximation error (i.e., misspecification level) by the representation is small enough. In particular, Du et al. (2019) showed that an ˜ Ω( √ H/d ) misspecification will lead to Ω(2 H ) sample complexity for RL to identify the optimal policy, even with a generative model. On the other hand, a series of work (Jin et al., 2020; Zanette et al., 2020b,a) provided ˜ O ( √ K + ζK ) -type regret bound for RL in various settings, where ζ is the misspecification level 3 and we ignore the dependence on the dimension of the feature mapping d and the planing horizon H for simplicity. These algorithms, however, require the knowledge of misspecification level ζ , thus are not parameterfree . Another concern for these algorithms is that some of the algorithms (Jin et al., 2020) would possibly suffer from a trivial asymptotic regret , i.e., Regret ( k ) > ω ( kζ · poly ( d, H, log(1 /δ ))) , as suggested by Vial et al. (2022). This means the performance of the RL algorithm will possibly degenerate as the number of episodes k grows. To tackle these two issues, Vial et al. (2022) propose the Sup-LSVI-UCB algorithm which requires a parameter ε tol. When ε tol = d/ √ K , the proposed

algorithm is parameter-free but will have a trivial asymptotic regret bound . When ε tol = ζ , the algorithm will have a non-trivial asymptotic regret bound but is not parameter-free since it requires knowledge of the misspecification level. Another series of works (He et al., 2022b; Lykouris et al., 2021; Wei et al., 2022) are working on the corruption robust setting. In particular, Lykouris et al. (2021); Wei et al. (2022) are using the model-selection technique to ensure the robustness of RL algorithms under adversarial MDPs.

3 Preliminaries

We consider episodic Markov Decision Processes, which are denoted by M ( S , A , H, { r h } , { P h } ) . Here, S is the state space, A is the finite action space, H is the length of each episode, r h : S×A ↦→ [0 , 1] is the reward function at stage h and P h ( ·| s, a ) is the transition probability function at stage h . The policy π = { π h } H h =1 denotes a set of policy functions π h : S ↦→ A for each stage h . For given policy π , we define the state-action value function Q π h ( s, a ) and the state value function V π h ( s ) as

where s h ′ +1 ∼ P h ( ·| s h ′ , a h ′ ) . The optimal state-action value function Q ∗ h and the optimal state value function V ∗ h are defined by Q ∗ h ( s, a ) = max π Q π h ( s, a ) , V ∗ h ( s ) = max π V π h ( s ) .

By definition, both the state-action value function Q π h ( s, a ) and the state value function V π h ( s ) are bounded by [0 , H ] for any state s , action a and stage h . For any function V : S ↦→ R , we denote by [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ′ ) the expected value of V after transitioning from state s given action a at stage h and [ B h V ]( s, a ) = r h ( s, a ) + [ P h V ]( s, a ) where B is referred to as the Bellman operator . For each stage h ∈ [ H ] and policy π , the Bellman equation, as well as the Bellman optimality equation, are presented as follows

We use regret to measure the performance of RL algorithms. It is defined as Regret ( K ) = ∑ K k =1 ( V ∗ 1 ( s k 1 ) -V π k 1 ( s k 1 ) ) , where π k represents the agent's policy at episode k . This definition quantifies the cumulative difference between the expected rewards that could have been obtained by following the optimal policy and those achieved under the agent's policy across the first K episodes, measuring the total loss in performance due to suboptimal decisions.

We consider linear function approximation in this work, where we adopt the misspecified linear MDP assumption, which is firstly proposed in Jin et al. (2020).

Assumption 3.1 ( ζ -Approximate Linear MDP, Jin et al. 2020) . For any ζ ≤ 1 , we say a MDP M ( S , A , H, { r h } , { P h } ) is a ζ -approximate linear MDP with a feature map ϕ : S × A ↦→ R d , if for any h ∈ [ H ] , there exist d unknown (signed) measures µ h = ( µ (1) h , · · · , µ ( d ) h ) over S and an unknown vector θ h ∈ R d such that for any ( s, a ) ∈ S × A , we have

w.l.o.g. we assume ∀ ( s, a ) ∈ S × A : ∥ ϕ ( s, a ) ∥ ≤ 1 and ∀ h ∈ [ H ] : ∥ µ h ( S ) ∥ ≤ √ d, ∥ θ h ∥ ≤ √ d .

The ζ -approximate linear MDP suggests that for any policy π , the state-action value function Q π h can be approximated by a linear function of the given feature mapping ϕ up to some misspecification level, which is summarized in the following proposition.

Proposition 3.2 (Lemma C.1, Jin et al. 2020) . For a ζ -approximate linear MDP , for any policy π , there exist corresponding weights { w π h } h ∈ [ H ] where w π h = θ h + ∫ V π h +1 ( s ′ )d µ h ( s ′ ) such that for any ( s, a, h ) ∈ S × A × [ H ] , ∣ ∣ Q π h ( s, a ) -⟨ ϕ ( s, a ) , w π h ⟩ ∣ ∣ ≤ 2 Hζ . We have ∥ w π h ∥ 2 ≤ 2 H √ d .

Next, we introduce the definition of the suboptimal gap as follows.

̸

Definition 3.3 (Minimal suboptimality gap) . For each s ∈ S , a ∈ A and step h ∈ [ H ] , the suboptimality gap gap h ( s, a ) is defined by ∆ h ( s, a ) = V ∗ h ( s ) -Q ∗ h ( s, a ) and the minimal suboptimality gap ∆ is defined by ∆ = min h,s,a { ∆ h ( s, a ) : ∆ h ( s, a ) = 0 } .

Algorithm 1 Cert-LSVI-UCB

̸

Notably, a task with a larger ∆ means it is easier to distinguish the optimal action π ∗ h ( s ) from other actions a ∈ A , while a task with lower gap ∆ means it is more difficult to distinguish the optimal action.

4 Proposed Algorithms

4.1 Main algorithm: Cert-LSVI-UCB

We begin by introducing our main algorithm Cert-LSVI-UCB , which is a modification of the Sup-LSVI-UCB (Vial et al., 2022). As presented in Algorithm 1, for each episode k , our algorithm maintains a series of index sets C l k,h for each stage h ∈ [ H ] and phase l . The algorithm design ensures that for any episode k , the maximum number of phases l is bounded by L k ≤ max {⌈ log 4 ( k/d ) ⌉ , 0 } . During the exploitation step, for each phase l associated with the index set C l k -1 ,h , the algorithm constructs the estimator vector w k h,l by solving the following ridge regression problem in Line 6 and Line 7:

After calculating the estimator vector w k h,l in Line 8, the algorithm quantilizes w k h,l and ( U k h,l ) -1 to the precision of κ l . Similar to Sup-LSVI-UCB (Vial et al., 2022), we note ˜ U k, -1 h,l is the quantized version of inverse covariance matrix ( U k h,l ) -1 rather than the inverse of quantized covariance matrix ( ˜ U k h,l ) -1 . The main difference between our implementation and that in Vial et al. (2022) is that we use a layer-dependent quantification precision κ l instead of the global quantification precision κ = 2 -4 L /d , which enables our algorithm get rid of the dependence on O (log K ) in the maximum number of phases L k .

After obtaining ˜ w k h,l and ˜ U k, -1 h,l , a subroutine, Cert-LinUCB , is called to calculate an optimistic value function ̂ V k h ( s τ h ) for all historical states s τ h in Line 10. Then the algorithm transits to stage h -1 and iteratively computes ˜ w k h,l and ˜ U k, -1 h,l for all phase l and stage h ∈ [ H ] .

In the exploration step, the algorithm starts to do planning from the initial state s k 1 . For each observed state s k h , the same subroutine, Cert-LinUCB , will be called in Line 14 for the policy π k h ( s k h ) , the

corresponding phase l k h ( s k h ) , and a flag f k h ( s k h ) . If the flag f k h ( s k h ) = 1 , the algorithm adds the index k to the index set C k h,l k h ( s k h ) in Line 15. Otherwise, the algorithm skips the current index k and all index sets remain unchanged. Finally, the algorithm executes policy π k h ( s k h ) , receives reward r k h and observes the next state s k h +1 in Line 17.

4.2 Subroutine: Cert-LinUCB

Next we introduce subroutine Cert-LinUCB , improved from Sup-Lin-UCB-Var (Vial et al., 2022) that computes the optimistic value function ̂ V k h . The algorithm is described as follows. Starting from phase l = 1 , the algorithm first calculates the estimated state-action function Q k h,l ( s, a ) as a linear function over the quantified parameter ˜ w k h,l and feature mapping ϕ ( s, a ) , following Proposition 3.2. After calculating the estimated state-action value function Q k h,l ( s ) , the algorithm computes the greedy policy π k h,l ( s ) and its corresponding value function V k h,l ( s ) .

Similar to Sup-Lin-UCB-Var (Vial et al., 2022), our algorithm has several conditions starting from Line 7 to determine whether to stop at the current phase or to eliminate the actions and proceed to the next phase l +1 , which are listed in the following conditions.

Condition 1 : In Line 7, if the current phase l is greater than the maximum phase L , we directly stop at that phase and take the greedy policy on previous phase π k h ( s ) = π k h,l -1 ( s ) .

Condition 2 : In Line 9, if there exists an action whose uncertainty ∥ ϕ ( s, a ) ∥ ˜ U k. -1 h,l is greater than the threshold 2 -l γ -1 l , our algorithm will perform exploration by selecting that action.

Condition 3 : In Line 11, we compare the value of the pessimistic value function q V k h,l ( s ) and the optimistic value function ̂ V k h,l ( s ) which will be assigned in Line 14 and Line 15, if the pessimistic estimation will be greater than the optimistic estimation, we will stop at that phase and take the greedy policy on previous phase π k h ( s ) = π k h,l -1 ( s ) . Only in this case, the Algorithm 2 outputs flag f k h ( s ) = 0 , which means this observation will not be used in Line 15 in Algorithm 1.

Condition 4 : In the default case in Line 16, the algorithm proceeds to the next phase after eliminating actions.

Notably, in Condition 4 , since the expected estimation precision in the l -th phase is about ˜ O (2 -l ) , our algorithm can eliminate the actions whose state-action value is significantly less than others, i.e., less than ˜ O (2 -l ) , while retaining the remaining actions for the next phase.

Specially, our algorithm differs from that in Vial et al. (2022) in terms of Condition 3 to certify the performance of the estimation. In particular, a well-behaved estimation should always guarantee that the optimistic estimation is greater than the pessimistic estimation. According to Line 14 and Line 15, this is equivalent to the confidence region for l -th phase has intersection of the previous confidence region [ q V k h,l -1 ( s ) , ̂ V k h,l -1 ( s )] . Otherwise, we hypothesis the estimation on l -th phase is corrupted by either misspecification or bad concentration event, thus will stop the algorithm. We will revisit the detail of this design later.

It's important to highlight that our algorithms provide unique approaches when compared with previous works. In particular, He et al. (2021b) does not eliminate actions and combines estimations from all layers by considering the minimum estimated optimistic value function. This characteristic prevents their algorithm from achieving a uniform PAC guarantee in the presence of misspecification. For a more detailed comparison with He et al. (2021b), please refer to Appendix B.1. Additionally, Lykouris et al. (2021); Wei et al. (2022) focus on a model-selection regime where a set of base learners are employed in the algorithms, whereas we adopt a multi-phase approach similar with SupLinUCB rather than conducting model selection over base learners.

5 Constant Regret Guarantee

Theorem 5.1. Under Assumption 3.1, let γ l = 5( l + 20 + ⌈ log( ld ) ⌉ ) dH √ log(16 ldH/δ ) for some fixed 0 < δ < 1 / 4 . With probability at least 1 -4 δ , if misspecification level ζ is below ˜ O ( ∆ / ( √ dH 2 ) ) where ∆ is the minimal suboptimality gap, then for all K ∈ N + , the regret of Algorithm 1 is upper bounded by

This regret bound is constant w.r.t. the episode K .

Theorem 5.1 demonstrates a constant regret bound with respect to number of episodes K . Compared with Papini et al. (2021a), our regret bound does not require any prior assumption on the feature mapping ϕ , such as the UniSOFT assumption made in Papini et al. (2021a). In addition, compared with the previous logarithmic regret bound He et al. (2021a) in the well-specified setting, our constant regret bound removes the log K factor, indicating the cumulative regret no longer grows w.r.t. the number of episode K , with high probability.

Remark 5.2. As discussed in Zhang et al. (2023b) in the misspecified linear bandits, Our high probability constant regret bound does not violate the lower bound proved in Papini et al. (2021a), which says that certain diversity condition on the contexts is necessary to achieve an expected constant regret bound. When extending this high probability constant regret bound to the expected regret bound, we have

which depends on the number of episodes k . To obtain a sub-linear expected regret, we can choose δ = 1 /K , which yields a logarithmic expected regret ˜ O ( d 3 H 5 ∆ -1 log K ) and does not violate the lower bound in Papini et al. (2021a).

Remark 5.3. Du et al. (2019) provide a lower bound showing the interplay between the misspecification level ζ and suboptimality gap ∆ in a weaker setting, which we discuss in detail in Appendix B.2. Along with the result from Du et al. (2019), our results suggests that ignoring the dependence on H , ζ = ˜ O (∆ / √ d ) plays an important seperation for if a misspeficied model can be efficiently learned. This result is also aligned with the positive result and negative result for linear bandits (Lattimore et al., 2020; Zhang et al., 2023b).

6 Technical Challenges and Highlight of Proof Techniques

In this section, we highlight several major challenges in obtaining the constant regret under misspecified linear MDP assumption and how our method, especially the certified estimator, tackles these challenges.

6.1 Challenge 1. Achieving layer-wise local estimation error.

In the analysis of the value function under misspecified linear MDPs, we follow the multi-phase estimation strategy (Vial et al., 2022) to eliminate suboptimal actions and improve the robustness of the next phase estimation. Similar approaches have been observed in Zhang et al. (2023b); Chu et al. (2011) within the framework of (misspecified) linear bandits. However, unlike linear bandits, when constructing the empirical value function ̂ V h for stage h in linear MDPs, Jin et al. (2020) requires a covering statement on value functions to ensure the convergence of the regression, which is written by: (see Lemma D.4 in Jin et al. (2020) for details)

where we employ notation ˜ O H to obscure the dependence on H to simplify the presentation. We use the notation V k h +1 to denote as an κ -covering, (or quantification in Takemura et al. (2021); Vial et al. (2022)) for the value functions ̂ V k h +1 . However, in the multi-phase algorithm, the empirical value function ̂ V k h +1 from the subsequent stage h + 1 , which is formulated using all pairs of parameters { w k h,ℓ , U k h,ℓ } ℓ in L phases. Consequently, the covering number log |V k h +1 | is directly proportional to the number of phases L = O (log K ) .

Therefore, when analyzing any single phase l , prior analysis cannot eliminate the log K term from (6.1) to achieve a local estimation error independent that is independent of the logarithmic number of global episodes log K . Furthermore, due to the algorithm design of previous methods (Vial et al., 2022), additional log K terms may be introduced by global quantification (i.e., ε tol = d/ √ K ) .

Our approach: Cert-LinUCB . To tackle this challenge, we introduce the certified estimator into Algorithm 2 and use a 'local quantification' to ensure the quantification error of each phase l depend on the local phase ˜ O ( l ) instead of the global parameter log K . The certified estimator works as follows: Considering the concentration term we need to control for each phase l :

as discussed in Challenge 1 , the function class ̂ V k h +1 ∋ ̂ V k h +1 involves L = O (log K ) parameters, leading to a log K dependence in the results when using traditional routines. The idea of certified estimator is to get rid of this by not directly controlling log |V k h +1 | . Instead, certified estimator establishes a covering statement for the value function class V k h +1 ,l + ∋ ̂ V k h +1 ,l + , where ̂ V k h +1 ,l + is the value function that only incorporates the first l + phases of parameters { w k h,ℓ , U k h,ℓ } ℓ . Under this framework, the covering statement becomes:

Lemma 6.1 (Lemma C.4, informal) . Let ̂ V k h +1 ,l + be the output of Algorithm 2 terminated at phase l + ∈ N + , then with probability is at least 1 -2 δ ,

Lemma 6.1 suggests a concentration inequality at any phase l + , and the following lemma suggests that this procedure will only introduce an ˜ O (2 -l + ) error, under some faithful extension of the ̂ V k h,l + ( s ) :

Therefore, if a large enough l + can be reached in Algorithm 2, combining Lemma 6.1 and Lemma 6.2 allow us to bound (6.2) without introducing log K factors. The next lemma shows that the Line 11 will only never be triggered in shallow layer l .

Lemma 6.3 (Lemma C.8, informal) . With probability at least 1 -2 δ , for any ( k, h ) ∈ [ K ] × [ H ] , Line 11 in Algorithm 2 can only be triggered on phase l ≥ ˜ Ω ( log(1 /ζ ) ) .

Lemma 6.3 delivers a clear message: In the well-specified setting, Line 11 will never be triggered ( l ≥ ∞ ). When the misspecification level is large, then Line 11 will be more likely triggered, indicating it's harder for the algorithm to proceed to deeper layer. The contribution of the certified estimator yields the following important lemma regarding the 'local estimation error': √

Lemma 6.4 (Lemma C.12, Informal) . With high probability, for any ε > ˜ Ω( dH 2 ζ ) and h ∈ [ H ] , Cert-LSVI-UCB ensures ∑ ∞ k =1 1 [ V ∗ h ( s k h ) -V π k h ( s k h ) ≥ ε ] ≤ ˜ O ( d 3 H 4 ε -2 ) .

Remark 6.5. He et al. (2021b) achieved a similar O ( d 3 H 5 ε -2 ) uniform-PAC bound for (wellspecified) linear MDP. Comparing with Lemma 6.4 with ζ = 0 , one can find that our result is better than He et al. (2021b). In addition, Lemma 6.4 ensures this uniform-PAC result under all stage h ∈ [ H ] while He et al. (2021b) only ensure the h = 1 . This improvement is achieved by a more efficient data selection strategy which we will discuss in detail in Appendix B.1.

6.2 Challenge 2. Achieving constant regret from local estimation error

In misspecified linear bandits, Zhang et al. (2023b) concludes their proof by controlling ∑ ∞ k =1 1 [ V ∗ 1 ( s k 1 ) -V π 1 ( s k 1 ) ≥ ∆] 4 . Although it is trivial showing that rounds with instantaneous regret V ∗ 1 ( s k 1 ) -V π 1 ( s k 1 ) < ∆ is optimal in bandits (i.e., V ∗ 1 ( s k 1 ) = V π 1 ( s k 1 ) ), previous works fail to reach a similar result for RL settings. This difficulty arises from the randomness inherent in MDPs: Consider a policy π that is optimal at the initial stage h = 1 . After the initial state and action, the MDP may transition to a state s ′ 2 with a small probability p where the policy π is no longer optimal, or to another state s 2 where π remains optimal until the end. In this context, the gap between V ∗ 1 ( s 1 ) and V π 1 ( s 1 ) can be arbitrarily small, given a sufficiently small p > 0 :

Therefore, one cannot easily draw a constant regret conclusion simply by controlling ∑ ∞ k =1 1 [ V ∗ 1 ( s k 1 ) -V π 1 ( s k 1 ) ≥ ∆] since the gap between V ∗ 1 ( s k 1 ) -V π 1 ( s k 1 ) needs to be further fine-grained controlled. In short, the existence of ∆ describing the minimal gap between V ∗ ( s ) -Q ∗ ( s, a ) cannot be easily applied to controlling regret V ∗ ( s ) -V π ( s ) .

Our approach: A fine-grained concentration analysis We address this challenge by providing a fine-grained concentration analysis in connecting the gap with the regret. Notice that the regret V ∗ h ( s h ) -V π k h ( s h ) in episode k is the expectation of cumulative suboptimality gap E [ ∑ H h =1 ∆ k h ] taking over trajectory { s k h } H h =1 . In addition, the variance of the random variable can be self-bounded according to

Denote η k be the difference between V ∗ h ( s h ) -V π k h ( s h ) and the actual ∑ H h =1 ∆ k h . Freedman inequality (Lemma H.5) implies that ∑ T t =1 η t ≥ aC and ∑ T t =1 Var[ η t ] ≤ vC happens at the same with a small probability for certain constant a and v . Using a fine-grained union bound statement over C , we can reach the following statement indicates the cumulative regret can be upper bounded using the cumulative suboptimality gap:

Lemma 6.6 (Lemma C.14, Informal) . The following statement holds with high probability:

Comparing with Lemma 6.1 in He et al. (2021a), Lemma 6.6 eliminates the log K dependence, which is achieved by the aforementioned fine-grained union bound. As a result, together with Lemma 6.4, we reach the desired statement that Cert-LSVI-UCB achieves constant regret bound when the misspecification is sufficiently small against the minimal suboptimality gap.

7 Conclusions and Limitations

In this work, we proposed a new algorithm, called certified estimator, for reinforcement learning with a misspecified linear function approximation. Our algorithm is parameter-free and does not

require prior knowledge of misspecification level ζ or the suboptimality ∆ . Our algorithm is based on a novel certified estimator and provides the first constant regret guarantee for misspecified linear MDPs and (well-specified) linear MDPs.

Limitations. Despite these advancements, several aspects of our algorithm and analysis warrant further investigation. One significant open question is whether the dependency on the planning horizon and dimension d, H can achieve optimal instance-dependent regret bounds. For the gapindependent regret bounds, the regret lower bound is Ω( d √ H 3 K ) as shown by Zhou et al. (2021a), and this benchmark has recently been met by works such as He et al. (2022a); Agarwal et al. (2022). Additionally, our analysis assumes uniform misspecification across all actions. Investigating other types of misspecifications could lead to more sophisticated results, enhancing the algorithm's robustness and applicability to diverse real-world scenarios. This exploration remains an important direction for future research.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their helpful comments. This work was done while WZ was a PhD student at UCLA. WZ is partially supported by UCLA dissertation year fellowship and the research fund from UCLA-Amazon Science Hub. JH and QG are partially supported by the research fund from UCLA-Amazon Science Hub. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

References

ABBASI-YADKORI, Y., PÁL, D. and SZEPESVÁRI, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24 2312-2320.

AGARWAL, A., JIN, Y. and ZHANG, T. (2022). Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. arXiv preprint arXiv:2212.06069 .

AYOUB, A., JIA, Z., SZEPESVARI, C., WANG, M. and YANG, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning . PMLR.

CESA-BIANCHI, N. and LUGOSI, G. (2006). Prediction, learning, and games . Cambridge university press.

CHU, W., LI, L., REYZIN, L. and SCHAPIRE, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics . JMLR Workshop and Conference Proceedings.

DANN, C., MARINOV, T. V., MOHRI, M. and ZIMMERT, J. (2021). Beyond value-function gaps: Improved instance-dependent regret bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems 34 .

DU, S. S., KAKADE, S. M., WANG, R. and YANG, L. F. (2019). Is a good representation sufficient for sample efficient reinforcement learning? In International Conference on Learning Representations .

HAO, B., LATTIMORE, T. and SZEPESVARI, C. (2020). Adaptive exploration in linear contextual bandit. In International Conference on Artificial Intelligence and Statistics . PMLR.

HE, J., ZHAO, H., ZHOU, D. and GU, Q. (2022a). Nearly minimax optimal reinforcement learning for linear markov decision processes. arXiv preprint arXiv:2212.06132 .

HE, J., ZHOU, D. and GU, Q. (2021a). Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

HE, J., ZHOU, D. and GU, Q. (2021b). Uniform-PAC bounds for reinforcement learning with linear function approximation. In Advances in Neural Information Processing Systems .

HE, J., ZHOU, D., ZHANG, T. and GU, Q. (2022b). Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. In Advances in Neural Information Processing Systems .

HOEFFDING, W. (1963). Probability inequalities for sum of bounded random variables.

ISHFAQ, H., CUI, Q., NGUYEN, V., AYOUB, A., YANG, Z., WANG, Z., PRECUP, D. and YANG, L. (2021). Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning . PMLR.

JAKSCH, T., ORTNER, R. and AUER, P. (2010). Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research 11 .

JIA, Z., YANG, L., SZEPESVARI, C. and WANG, M. (2020). Model-based reinforcement learning with value-targeted regression. In Learning for Dynamics and Control . PMLR.

JIN, C., YANG, Z., WANG, Z. and JORDAN, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory . PMLR.

KOBER, J., BAGNELL, J. A. and PETERS, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32 1238-1274.

LATTIMORE, T., SZEPESVARI, C. and WEISZ, G. (2020). Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning . PMLR.

LYKOURIS, T., SIMCHOWITZ, M., SLIVKINS, A. and SUN, W. (2021). Corruption-robust exploration in episodic reinforcement learning. In Conference on Learning Theory . PMLR.

MNIH, V., KAVUKCUOGLU, K., SILVER, D., GRAVES, A., ANTONOGLOU, I., WIERSTRA, D. and RIEDMILLER, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 .

MODI, A., JIANG, N., TEWARI, A. and SINGH, S. (2020). Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics .

PAPINI, M., TIRINZONI, A., PACCHIANO, A., RESTELLI, M., LAZARIC, A. and PIROTTA, M. (2021a). Reinforcement learning in linear mdps: Constant regret and representation selection. Advances in Neural Information Processing Systems 34 16371-16383.

PAPINI, M., TIRINZONI, A., RESTELLI, M., LAZARIC, A. and PIROTTA, M. (2021b). Leveraging good representations in linear contextual bandits. In International Conference on Machine Learning . PMLR.

SIMCHOWITZ, M. and JAMIESON, K. G. (2019). Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems 32 1153-1162.

TAKEMURA, K., ITO, S., HATANO, D., SUMITA, H., FUKUNAGA, T., KAKIMURA, N. and KAWARABAYASHI, K.-I. (2021). A parameter-free algorithm for misspecified linear contextual bandits. In International Conference on Artificial Intelligence and Statistics . PMLR.

VIAL, D., PARULEKAR, A., SHAKKOTTAI, S. and SRIKANT, R. (2022). Improved algorithms for misspecified linear markov decision processes. In International Conference on Artificial Intelligence and Statistics . PMLR.

WANG, Y., WANG, R., DU, S. S. and KRISHNAMURTHY, A. (2019). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations .

WEI, C.-Y., DANN, C. and ZIMMERT, J. (2022). A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory . PMLR.

YANG, K., YANG, L. and DU, S. (2021). Q-learning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics . PMLR.

YANG, L. and WANG, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning . PMLR.

ZANETTE, A., BRANDFONBRENER, D., BRUNSKILL, E., PIROTTA, M. and LAZARIC, A. (2020a). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics .

ZANETTE, A., LAZARIC, A., KOCHENDERFER, M. and BRUNSKILL, E. (2020b). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning . PMLR.

ZHANG, W., HE, J., FAN, Z. and GU, Q. (2023a). On the interplay between misspecification and sub-optimality gap: From linear contextual bandits to linear MDPs.

ZHANG, W., HE, J., FAN, Z. and GU, Q. (2023b). On the interplay between misspecification and sub-optimality gap in linear contextual bandits. arXiv preprint arXiv:2303.09390 .

ZHANG, W., HE, J., ZHOU, D., ZHANG, A. and GU, Q. (2021). Provably efficient representation learning in low-rank markov decision processes. arXiv preprint arXiv:2106.11935 .

ZHOU, D., GU, Q. and SZEPESVARI, C. (2021a). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

ZHOU, D., HE, J. and GU, Q. (2021b). Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning . PMLR.

A Additional Related Work

RL with linear function approximation. Recent years have witnessed a line of work focusing on RL with linear function approximation to tackle RL tasks in large state space. A widely studied MDPmodel is linear MDP (Jin et al., 2020), where both the transition kernel and the reward function are linear functions of a given feature mapping of the state-action pairs ϕ ( s, a ) . Several works have developed RL algorithms with polynomial sample complexity or sublinear regret bound in this setting. For example, LSVI-UCB (Jin et al., 2020) has an ˜ O ( √ d 3 H 4 K ) regret bound, randomized LSVI (Zanette et al., 2020a) has an ˜ O ( √ d 4 H 5 K ) regret bound and Ishfaq et al. (2021) achieved an ˜ O ( √ d 3 H 4 K ) . He et al. (2022a) then improves this regret bound to a nearly minimax-optimal result ˜ O ( d √ H 3 K ) while Agarwal et al. (2022) provides a general function approximation extension given the above result. Linear mixture/kernel MDPs (Modi et al., 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021b) have also emerged as another model that enables model-based RL with linear function approximation. In this setting, the transition kernel is a linear function of a feature mapping on the triplet of state, action, and next state ϕ ( s, a, s ′ ) . Nearly minimax optimal regrets can be achieved for both finite-horizon episodic MDPs (Ayoub et al., 2020; Zhou et al., 2021a) and infinite-horizon discounted MDPs (Zhou et al., 2021b) under this assumption.

B Additional Discussions on Algorithm Design and Result

B.1 Comparison with He et al. (2021b)

It is worth comparing our algorithm with He et al. (2021b), which also provides a uniform PAC bound for linear MDPs. Both our algorithm and theirs utilize a multi-phase structure that maintains multiple regression-based value function estimators at different phases. Despite this similarity, there are several major differences between our algorithm and that in He et al. (2021b), which are highlighted as follows:

In Line 7 of Algorithm 1, when calculating the regression-based estimator, for different phase l , we use the same regression target ̂ V k h +1 , while their algorithm uses different V k h +1 ,l for different phase l .

When aggregating the regression estimators over all different L k phases, we follow the arm elimination method as in Chu et al. (2011), while He et al. (2021b) simply take the point-wise minimum of all estimated state-action functions, i.e., Q ( s, a ) = min l ∈ [ L k ] Q l k,h ( s, a ) .

When calculating the phase l k h ( s k h ) for a trajectory s k 1 , s k 2 , · · · , s k H , He et al. (2021b) require that the phase l k h ( s k h ) to be monotonically decreasing with respect to the stage h , i.e., l k h ( s k h ) ≤ l k h -1 ( s k h -1 ) (see line 19 in Algorithm 2 in He et al. (2021b)). Such a requirement will lead to a poor estimation for later stages and thus increase the sample complexity. In contrast, we do not have this requirement or any other requirements related to l k h ( s k h ) and l k h -1 ( s k h -1 ) .

As a result, by (3), He et al. (2021b) have to sacrifice some sample complexity to make their algorithm work for different target value functions V k h +1 ,l . As a comparison, since we use the same regression target for different phase l , we do not have to make such a sacrifice in (3). Moreover, by (2), He et al. (2021b) cannot deal with linear MDPs with misspecification, while our algorithm can handle misspecification as in Vial et al. (2022).

B.2 Discussion on Lower Bounds of Sample Complexity

We present a lower bound from Du et al. (2019) to better illustrate the interplay between the misspecification level ζ and the suboptimality gap ∆ .

Assumption B.1 (Assumption 4.3, Du et al. 2019, ζ -Approximate Linear MDP) . There exists ζ > 0 , θ h ∈ R d and µ h : S ↦→ R d for each stage h ∈ [ H ] such that for any ( s, a, s ′ ) ∈ S × A × S , we have ∣ ∣ P h ( s ′ | s, a ) -⟨ ϕ ( s, a ) , µ h ( s ′ ) ⟩ ∣ ∣ ≤ ζ and ∣ ∣ r ( s, a ) -⟨ ϕ ( s, a ) , θ h ⟩ ∣ ∣ ≤ ζ .

Theorem B.2 (Theorem 4.2, Du et al. 2019) . There exists a family of hard-to-learn linear MDPs with action space |A| = 2 and a feature mapping ϕ ( s, a ) satisfying Assumption B.1, such that for any algorithm that returns a 1 / 2 -optimal policy with probability 0 . 9 needs to sample at least Ω(min {|S| , 2 H , exp( dζ 2 / 16) } ) episodes.

Remark B.3. As claimed in Du et al. (2019), Theorem B.2 suggests that when misspecification in the ℓ ∞ norm satisfies ζ = Ω(∆ √ H/d ) , the agent needs an exponential number of episodes to find a near-optimal policy, where ∆ = 1 / 2 in their setting. It is worth noting that Assumption B.1 is a ℓ ∞ approximation for the transition matrix. Such a ℓ ∞ guarantee ( ∥ · ∥ ∞ ≤ ζ ) is weaker than the ℓ 1 guarantee ( ∥ · ∥ 1 ≤ ζ ) provided in Assumption 3.1. So it's natural to observe a positive result when making a stronger assumption and a negative result when making a weaker assumption. In addition, despite of this difference, one could find that ζ ∼ ∆ / √ d plays a vital role in determining if the task can be efficiently learned. Similar positive and negative results are also provided in Lattimore et al. (2020); Zhang et al. (2023b) in the linear contextual bandit setting (a special case of linear MDP with H = 1 ).

C Constant Regret Guarantees for Cert-LSVI-UCB

In this section, we present the proof of Theorem 5.1. To begin with, we recap the notations used in the algorithm and introduce several shorthand notations that would be employed for the simplicity of latter proof. The notation table is presented in Table 2.Any proofs not included in this section are deferred to Appendix D.

C.1 Quantized State Value Function set V k h,l .

To begin our proof, we first extend the definition of ̂ V k h,l to arbitrary l and give a formal definition of the state value function class V k h,l as we skip the detail of this definition in Section 6.

Definition C.1. We extend the definition of state value function ̂ V k h,l to any tuple ( k, h, l ) ∈ [ K ] × [ H ] × N + by

We also define the state value function family V k h,l be the set of all possible ̂ V k h,l .

where { ˜ w · · ,ℓ } l ℓ =1 and { ˜ U · , -1 · ,ℓ } l ℓ =1 are referring to any possible parameters generated by Line 8 in Algorithm 1.

It is worth noting that one can check the definition of ̂ V k h,l here is consistent with those computed in Algorithm 2 with l < l k h ( s ) . Therefore, we will not distinguish between the notations in the remainder of the proof.

The following lemma controls the distance between ̂ V k h ( s ) and ̂ V k h,l ( s ) for any phase l .

Lemma C.2. For any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [ l k ( s ) -1] , it holds that

Moreover, for any tuple ( k, h, s, l + ) ∈ [ K ] × [ H ] × S × N + , the difference | ̂ V k h ( s ) -̂ V k h,l + ( s ) | is bounded by 6 · 2 -l + , following the extension of the definition scope of ̂ V k h,l + as outlined in Definition C.1.

Lemma C.2 suggests that given any phase l + , ̂ V k h,l is close to ̂ V k h . This enables us to construct covering on ̂ V k h using the covering on ̂ V k h,l .

C.2 Concentration of State Value Function ̂ V k h ( s )

In this subsection, we provide a new analysis for bounding the self-normalized concentration of ∥ ∥ ∥ ∑ τ ϕ τ h ( [ P h ̂ V k h ]( s τ h , a τ h ) -̂ V k h ( s τ h +1 ) ) ∥ ∥ ∥ U -1 to get rid of the log k factor in Vial et al. (2022).

To facilitate our proof, we define the filtration list F k h = { { s j i , a j i } H,k -1 i =1 ,j =1 , { s k i , a k i } h i =1 } . It is easy to verify that s k h , a k h are both F k h -measurable. Also, for any function V built on F k h , [ P h V ]( s k h , a k h ) -V ( s k h +1 ) is F k h +1 -measurable and it is also a zero-mean random variable conditioned on F k h .

The first lemma we provide is similar with Vial et al. (2022), which shows the self-normalized concentration property for each phase l and any function V ∈ V k h,l .

Definition C.3. For some fixed mapping l ↦→ l + = l + ( l ) that l + ≥ l , we define the bad event as

The good event is defined by G 1 = ⋂ K k =1 ⋂ H h =1 ⋂ l ≥ 1 ⋂ V ∈V k h,l + B ∁ 1 ( k, h, l, V ) where we define γ l,l + = 5 l + dH √ log(16 ldH/δ ) = ˜ O ( ldH log( δ -1 )) .

Lemma C.4. The good event G 1 defined in Definition C.3 happens with probability at least 1 -2 δ .

Lemma C.4 establishes the concentration bounds for any given phase l . However, the total number of phases for the state value function V k h ( s ) can be bounded only trivially by l = O (log K ) , resulting in log K dependence. To address this issue, the following lemma proposes a method to eliminate this logarithmic factor:

Lemma C.5. Under event G 1 , for any ( k, h, l ) ∈ [ K ] × [ H ] × N + ,

where we set γ l = γ l,l + with l + = l +20 + ⌈ log( ld ) ⌉ .

Then Lemma C.5 immediately yields the following lemma regarding the estimation error of the state-action value function Q k h,l :

Lemma C.6. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [ l k h ( s ) -f k h ( s )] , a l ∈ A k h,l ( s ) ,

where we define χ = 12 √ dH .

Lemma C.6 build an estimation error for any l ∈ [ l k h ( s ) -1] . As we mentioned in the algorithm design, a larger l here will lead to more precise estimation (a smaller 2 -l term in (C.2)) but will suffer from a larger covering number (a larger γ l term in (C.2)). Following a similar proof sketch from Vial et al. (2022), the next lemma shows that any action that is not eliminated has a low regret,

Lemma C.7. Fix some arbitrary L 0 ≥ 1 and let χ = 12 √ dH . Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] , a l +1 ∈ A k h,l +1 ( s ) ,

C.3 The Impact of Misspecification Level ζ

Next, we are ready to show the criteria where Line 11 in Algorithm 2 will be triggered, which shows the impact of misspecification on this multi-phased estimation.

Lemma C.8. Under event G 1 , for any ( k, h ) ∈ [ K ] × [ H ] such that f k h ( s k h ) = 0 , we have l k h ( s k h ) > L ζ where L ζ is the maximal integer satisfying 2 -L ζ ≥ χL 1 . 5 ζ ζ for χ = 12 √ dH , i.e., L ζ = Ω(log(1 /ζ )) .

Equipped with Lemma C.8, the following lemma suggests that how much estimation precision ε can be achieved by accumulating the error 2 -l k h ( s k h ) that occurred in Lemma C.6.

Lemma C.9. Under event G 1 and for all ε > 0 , define L ε to be the minimal integer satisfying 2 -L ε ≤ 0 . 01 ε/H , i.e., L ε = ⌈-log(0 . 01 ε/H ) ⌉ . When L ε ≤ L ζ , then for any K ⊆ [ K ] , h ∈ [ H ] ,

The relationship between L ε ≤ L ζ can be translated to the relationship between ε and ζ . We characterize this condition as follows:

Definition C.10. Condition G ε is defined for a given ε , and is satisfied if L ζ ≥ L ε where L ε is the minimal integer satisfying 2 -L ε ≤ 0 . 01 ε/H and L ζ is the maximal integer satisfying 2 -L ζ ≥ χL 1 . 5 ζ ζ .

Lemma C.11. If ε ≥ Ω ( √ dH 2 ζ log 2 (1 /ζ ) ) , then G ε is satisfied.

Proof. If ε ≥ Ω ( √ dH 2 ζ log 2 (1 /ζ ) ) , we have

where the first inequality is given by the definition of L ε , the last inequality is given by the definition of L ζ , and the second inequality holds since HχL 1 . 5 ζ ≤ O ( √ dH 2 log 2 (1 /ζ ) ) , and the last inequality is given by the definition of L ε and L ζ , respectively. Since 2 -l decreases as l increases, we can conclude that L ε ≤ L ζ .

The above analysis of the interplay between misspecification level ζ and precision ε yields the following important lemma in our proof, showing a local decision error across all h ∈ [ H ] :

Lemma C.12. Under Assumption 3.1, let γ l = 5( l +20+ ⌈ log( ld ) ⌉ ) dH √ log(16 ldH/δ ) , for some fixed 0 < δ < 1 / 3 . With probability at least 1 -3 δ , for any ε > Ω ( √ dH 2 ζ log 2 (1 /ζ ) ) and h ∈ [ H ] , we have

where ι refers to some polynomial of log log( dHε -1 δ -1 ) . This can also be written as

with ε 0 = ˜ Ω( √ dH 2 ζ ) and f ( ε, δ ) = ˜ O ( d 3 H 4 ε -2 log( δ -1 )) .

C.4 From Local Step-wise Decision Error to Constant Regret

The next lemma shows that the total incurred suboptimality gap is constant if the minimal suboptimality gap ∆ satisfies ∆ > ε 0 .

Lemma C.13. Suppose an RL algorithm Alg. satisfies

such that f ( ε, δ ) = ˜ O ( C 1 /ε + C 2 /ε 2 ) where C 1 , C 2 > 0 are constant in ε , but may depend on other quantities such as d, H, log( δ -1 ) . If the minimal suboptimality gap ∆ satisfies ∆ > ε 0 , then

where ∆ k h = ∆ h ( s k h , π k h ( s k h ) ) = V ∗ h ( s k h ) -Q ∗ h ( s k h , π k h ( s k h ) ) is the suboptimality gap suffered in stage h of episode k .

The following Lemma is a refined version of Lemma 6.1 in He et al. (2021a) that removes the dependence between regret and number of episodes K .

Lemma C.14. For each MDP M ( S , A , H, { r h } , { P h } ) and any δ > 0 , with probability at least 1 -δ , we have

We are now ready to prove Theorem 5.1:

Proof of Theorem 5.1. By plugging in Lemma C.12 and Lemma C.13 into Lemma C.14, we can reach the desired statement.

D Proof of Lemmas in Appendix C

In this section, we prove lemmas outlined in Appendix C. Any proofs not included in this section are deferred to Appendix E.

D.1 Proof of Lemma C.2

Proof of Lemma C.2. According to the criteria for Line 11, we have q V k h,l ( s ) ≤ ̂ V k h,l ( s ) for any l ∈ [ l k h ( s ) -1] . From the definition of q V k h,l ( s ) and ̂ V k h,l ( s ) , they are monotonic in l that ̂ V k h,l -1 ( s ) ≤ ̂ V k h,l ( s ) and ̂ V k h,l ( s ) ≤ ̂ V k h,l -1 ( s ) hold. Combining with ̂ V k h +1 ( s ) = ̂ V k h,l k h ( s ) -1 , we have

From the definition of ̂ V k h,l ( s ) and q V k h,l ( s ) , we have

Plugging (D.1) into (D.2), we conclude that for any phase l ∈ [ l k h ( s ) -1] , it holds that | ̂ V k h ( s ) -̂ V k h,l ( s ) | ≤ 6 · 2 -l .

Now consider the extended state value function ̂ V k h,l + with an arbitrary l + ∈ N + . For every s where l + ≤ l k h ( s ) -1 , we have | ̂ V k h ( s ) -V k h,l + ( s ) | ≤ 6 · 2 -l + as reasoned above. For the other s ∈ S where l + ≥ l k h ( s ) , we have ̂ V k h,l ( s ) = ̂ V k h ( s ) following the procedure of Algorithm 2. This suggest that | ̂ V k h ( s ) -̂ V k h,l + ( s ) | ≤ 6 · 2 -l + always holds.

D.2 Proof of Lemma C.4

The following Lemma shows the rounding only cast bounded effects on the recovered parameters. Lemma D.1. For any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [ l k h ( s ) -f k h ( s )] , a ∈ A k h,l ( s ) , it holds that

The following lemma shows the number of episodes that are taken into regression |C k h,l | is bounded independently from the number of episodes k .

The following lemma shows the number of possible state value functions |V k h,l | is bounded independently from the number of episodes k .

Lemma D.3. For any tuple ( k, h, l ) ∈ [ K ] × [ H ] × N + , we have |V k h,l | ≤ (2 22 d 6 H 4 ) l 2 d 2 .

Now we are ready to prove Lemma C.4.

Proof of Lemma C.4. Recall in Definition C.3, the good event defined by the union of each single bad event:

where each single bad event is given by

in which [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ) .

Consider some fixed ( h, l ) ∈ [ H ] × N + , V ∈ V K h,l + . Arrange elements of C k h,l in ascending order as { τ i } i . Since the environment sample s τ i h +1 according to P h ( ·| s τ i h , a τ i h ) , we have [ P h V ]( s τ i h , a τ i h ) -V ( s τ i h +1 ) is F τ i h -measurable with E [ [ P h V ]( s τ i h , a τ i h ) -V ( s τ i h +1 ) ∣ ∣ F τ i h ] = 0 . Since 0 ≤ V ( s τ i h +1 ) ≤ H , we have | [ P h V ]( s τ i h , a τ i h ) -V ( s τ i h +1 ) | ≤ H . This further leads to

where the first inequality holds following from the good event of probability 1 -δ/ ( l 2 H |V K h,l + | ) defined in Lemma H.2 over filtration {F τ i h } i , the second inequality is derived from combining Lemma D.2 and Lemma D.3, and the last inequality is given by Lemma G.3. According to Lemma H.2, we have the bad event ⋃ K k =1 B 1 ( k, h, l, V ) happens with probability at most δ/ ( l 2 H |V K h,l + | ) . Taking union bound over all ( h, l ) ∈ [ H ] × N + , V ∈ V K h,l + , we have the bad event happens with probability at most

where the last inequality holds due to ∑ n ≥ 1 n -2 = π 2 / 6 . This completes our proof.

D.3 Proof of Lemma C.5

Proof of Lemma C.5. Denote the martingale difference between ̂ V k h,l + -̂ V k h as:

By triangle inequality:

According to the definition of event G 1 , we can upper bound the first term by

According to Lemma C.2, we have | ̂ V k h,l + ( s ) -̂ V k h +1 ( s ) | ≤ 6 · 2 -l + for any s ∈ S . Thus, the difference can be bounded by | µ τ h,l + | ≤ 6 · 2 -l + . Consequently, we can bound the second term by

where the first inequality is provided by Lemma H.3, utilizing the condition | µ τ h,l + | ≤ 6 · 2 -l + , the second inequality is from Lemma D.2. By plugging in the definition of l + , we can further bound the final term of (D.5) by

Plugging (D.4) and (D.6) into (D.3) yields the desired statement such that

which concludes our proof.

D.4 Proof of Lemma C.6

The following lemma shows the state-action value function Q k h,l ( s, a ) is always well estimated.

Lemma D.4. Under event G 1 , for any ( k, h, l, s, a ) ∈ [ K ] × [ H ] × N + ×S × A ,

Equipped with Lemma D.1 and Lemma D.4, we are ready to prove Lemma C.6.

Proof of Lemma C.6. In case that l ≤ l k h ( s ) -f k h ( s ) , for any a ∈ A k h,l ( s ) , we have that

where the first inequality holds due to triangle inequality, and in the second inequality, the first term is satisfied since state s passes the criterion in Line 9 in phase l and the second term follows from Lemma D.1, and the last inequality is given by Lemma G.2 which implies 2 l > γ l . Plugging (D.7) into Lemma D.4 gives

which proves the desired statement.

D.5 Proof of Lemma C.7

Equipped with Lemma C.6, we are able to show several properties of the state value function V k h,l throught the arm-elimination process. The first lemma suggests that for any action a l ∈ A k h,l ( s ) , there is at least one action a l +1 ∈ A k h,l +1 ( s ) close to a l in terms of the Bellman operator [ B h ̂ V k h +1 ]( s, a ) after the elimination.

Lemma D.5. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] , a l ∈ A k h,l ( s ) , there exists a l +1 ∈ A k h,l +1 ( s ) that

where χ = 12 √ dH for arbitrary L 0 ≥ 1 .

Then the following lemma shows that by induction on stage h ∈ [ H ] , we can show the elimination process keep at least one near-optimal action a l +1 ∈ A k h,l +1 ( s ) .

Lemma D.6. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] × S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] , there exists a l +1 ∈ A k h,l +1 ( s ) that,

where χ = 12 √ dH for arbitrary L 0 ≥ 1 .

The following two lemmas indicate that the state value function V k h,l ( s ) on stage h is a good estimation for the state value function given by Bellman operator V ( s ) = max a ∈A [ B h ̂ V k h +1 ]( s, a ) .

Lemma D.7. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] ,

where χ = 12 √ dH for arbitrary L 0 ≥ 1 .

Lemma D.8. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] ,

where χ = 12 √ dH for arbitrary L 0 ≥ 1 .

Now we are ready to show any actions remaining in the elimination process are near-optimal.

Proof of Lemma C.7. First, according to Lemma D.7, we can write

Any action a l +1 ∈ A k h,l +1 ( s ) passes the elimination process will satisfy:

According to Lemma C.6 with the condition that l ≤ L 0 , we have that the empirical state-action value function Q k h,l ( s, · ) is a good estimation for [ B h ̂ V k h +1 ]( s, · ) among every a l +1 ∈ A k l ( s ) under event G 1 :

Combining (D.8), (D.9), and (D.10) gives

which proves the desired statement.

D.6 Proof of Lemma C.8

The following two lemmas demonstrate that, at stage h , both the optimistic state value function, ̂ V k h,l ( s ) , and the pessimistic state value function, q V k h,l ( s ) , exhibit a gap relative to the state value function determined by the Bellman operator, given as V ( s ) = max a ∈A [ B h ̂ V k h +1 ]( s, a ) .

Lemma D.9. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] ,

where χ = 12 √ dH for arbitrary L 0 ≥ 1 . In case that l ≤ l k h ( s ) -1 , the inequality is equivalent to

Lemma D.10. Under event G 1 , for any ( k, h, s ) ∈ [ K ] × [ H ] ×S , l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] ,

where χ = 12 √ dH for arbitrary L 0 ≥ 1 . In case that l ≤ l k h ( s ) -1 , the inequality is equivalent to

Proof of Lemma C.8. Set L 0 = L ζ be the maximal integer satisfying 2 -L ζ -χL 1 . 5 ζ ζ ≥ 0 . Combining Lemma D.10 and Lemma D.9, for any l ∈ [min { L 0 , l k h ( s ) -f k h ( s ) } ] , we have that

where the second inequality holds since 2 -l decreases as l increases and the last inequality holds according to the selection of L 0 .

When f k h ( s ) = 0 , consider l = l k h ( s ) . The above reasoning indicates the criterion in Line 11 can never satisfied. Thus f k h ( s ) = 0 can only happen if l k h ( s ) > L 0 = L ζ .

D.7 Proof of Lemma C.9

By partitioning [ K ] based on whether Algorithm 2 stops before phase L ε , we can prove Lemma C.9. Specifically, Lemma D.2 bounds the number of episodes in which Algorithm 2 stops before phase L ε . This allows us to establish an upper bound for the desired summation over these episodes. Furthermore, for episodes that stop after phase L ε , the contribution of 2 -l k h ( s k h ) γ l k h ( s k h ) is small according to the definition of L ε .

Proof of Lemma C.9. Denote C K h, + = [ K ] -⋃ L ε -1 l =1 C K h,l . In this sense, we have

From the construction of C K h,l , we have l k h ( s k h ) = l for any k ∈ C K h,l . Fix some k ∈ C K h, + . If f k h ( s k h ) = 0 , we have l k h ( s k h ) ≥ L ζ ≥ L ε where the first inequality is given by Lemma C.8 and the second inequality is given by the assignment of L ε . Otherwise, we have l k h ( s k h ) ≥ L ε according to

the definition of C K h,l . This indicates l k h ( s k h ) ≥ L ε holds for any k ∈ C K h, + . This allow is to bound the first term by

where the first inequality holds since l k h ( s k h ) > L ε and the second inequality holds from both 2 -L ε ≤ 0 . 01 ε/H and |K ∩ C K h, + | ≤ |K| .

Furthermore, we can bound the second term by

where the second inequality is given by Lemma D.2, and the last inequality holds due to 0 . 005 ε/H ≤ 2 -L ε which is because L ε is a minimal integer such that 2 -L ε ≤ 0 . 01 ε/H .

Finally, plugging (D.12) and (D.13) into (D.11) gives

D.8 Proof of Lemma C.12

The following lemma provides an upper bound for the underestimation error of the empirical state value function ̂ V k h with respect to the optimal state value function V ∗ h .

LemmaD.11. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h, s ) ∈ [ K ] × [ H ] ×S ,

As ̂ V k h represents an empirical state value function with potentially optimal policy π k h ( s ) , the following lemma provides an upper bound for the overestimation error of ̂ V k h with respect to deploying the policy π k h ( s ) on the ground-truth transition kernel.

LemmaD.12. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h, s ) ∈ [ K ] × [ H ] ×S ,

To start with, we define a good event according to:

Definition D.13. For some ε > 0 , let K ε h = { k ∈ [ K ] : V ∗ h ( s k h ) -V π k h ( s k h ) ≥ ε } . We define the bad event as

where η k h = [ P h ( ̂ V k h +1 -V π k h +1 )]( s k h , π k h ( s k h )) -( ̂ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) ) . The good event is defined as G 2 = ⋂ H h =1 ⋂ l ≥ 1 B ∁ 2 ( h, 2 -l ) .

The following lemma provides the concentration property such that the cumulative sample error is small with high probability.

Lemma D.14. Event G 2 happens with probability at least 1 -δ .

Using the above results, we can bound the instantaneous regret of any subsets once the misspecification level is appropriately controlled,

Lemma D.15. Under event G 1 , G 2 and for all ε > 0 that G ε is satisfied, for any K ⊆ [ K ] and h ∈ [ H ] , it satisfies that

With all lemmas stated above, we can show Cert-LSVI-UCB achieves constant step-wise decision error. The following lemma gives a sufficient condition that G ε defined in Definition C.10 is satisfied.

Now, we are ready to prove Lemma C.12.

Proof of Lemma C.12. Wefocus on the case where the good event G 1 ∩G 2 ∩G ε occurs. By the union bound statement over Lemma C.4 and Lemma D.14, and Lemma C.11, this good event happens with a probability of at least 1 -3 δ and requires ε ≥ Ω( ζ √ dH 2 log 2 ( dHζ -1 )) . W.l.o.g, consider K ε h for some h ∈ [ H ] and ε = 2 -l where l > 0 is an integer. On the one hand, we have

On the other hand, Lemma D.15 gives

Combining (D.14) and (D.15) gives

Plugging the value of γ L ε , we have

According to Lemma G.4, (D.16) implies

where ι refers to a polynomial of log log( dHε -1 δ -1 ) . With the definition of L ε , we conclude that

D.9 Proof of Lemma C.13

Proof of Lemma C.13. From the definition of suboptimality gap, we have

According to the assumption,

holds for every ε > ε 0 with probability at least 1 -δ , replacing the V ∗ h ( s k 1 ) -V π k h ( s k h ) with its lower bound ∆ k h yields for every ε > ε 0 ,

In addition, according to the definition of minimal suboptimality gap ∆ in Definition 3.3, we have ∆ k h is either 0 or no less than ∆ . Since for any x ∈ { 0 } ∪ [∆ , H ] , it holds that x ≤ ∆ · 1 [ x ≥ ∆] + ∫ H ∆ 1 [ x ≥ ε ] d ε , we decompose the total suboptimality incurred in stage h by

In case that ε 0 ≤ ∆ , the first term in (D.18) can be bounded by

We can further bound the second term by

Plugging (D.19) and (D.20) into (D.18) with summation over h ∈ [ H ] , we conclude that the total suboptimality incurred in stage h is bounded by

D.10 Proof of Lemma C.14

Proof of Lemma C.14. For a given policy π and any state s h ∈ S , we have

where the second equality is given by the definition of suboptimality gap ∆ h ( · , · ) in Definition 3.3. Taking expectation on both sides with respect to the randomness of state-transition and taking telescoping sum over all h ∈ [ H ] gives

where s h +1 ∼ P h ( ·| s h , π h ( s h )) . Let the filtration list be F k = { { s j i , a j i } H,k -1 i =1 ,j =1 } , we have

Denote random variable η k = ( V ∗ 1 ( s k 1 ) -V π k h ( s k 1 ) ) -∑ H h =1 ∆ k h . Wecan see η k is F k +1 -measurable with | E [ η k |F k ] | = 0 . Furthermore, for the variance of η k , we have

where the first inequality holds due to Var[ X ] ≤ E [( X -t ) 2 ] for any fixed t , the second inequality follows 0 ≤ ∆ k h ≤ H . As a result, the total variance of the random variables { η k } can be bounded by

Let F ( x ) = √ 2 xH 2 log( x/δ ) + H 2 log( x/δ ) , using peeling technique, we can write

where the last inequality follows Lemma H.5. Plugging F ( x ) = √ 2 xH 2 log( x/δ ) + H 2 log( x/δ ) back into (D.21) yields

Therefore, whenever Regret ( K ) > 1 , with probability at least 1 -δ , we have

Combining with the fact that Regret ( K ) = ∑ K k =1 η k + ∑ K k =1 ∑ H h =1 ∆ k h , we have

whenever Regret ( K ) > 1 . Taking x = Regret ( K ) , a = ∑ K k =1 ∑ H h =1 ∆ k h + H 2 log( Regret ( K ) /δ ) , and b = 2 H 2 log( Regret ( K ) /δ ) , inequality (1) yields

According to the fact that x ≤ a log x + b ⇒ x ≤ 4 a log(2 a ) + 2 b , letting x = Regret ( K ) , a = 2 ∑ K k =1 ∑ H h =1 ∆ k h +6 H 2 log(1 /δ ) and b = 6 H 2 , (D.22) becomes

Hiding the logarithmic factors within the ˜ O notation yields

E Proof of Lemmas in Appendix D

In this section, we prove lemmas outlined in Appendix D. Any proofs not included in this section are deferred to Appendix F.

E.1 Proof of Lemma D.1

We first introduce the claim from Vial et al. (2022) controlling the rounding error:

Lemma E.1 (Claim 1, Vial et al. 2022, restate) . For any ( k, h, l, s, a ) ∈ [ K ] × [ H ] × N + ×S ×A , we have

where κ l is used to quantify the vector w k h,l and inverse matrix ( U l h,l ) -1 .

Proof of Lemma D.1. From Lemma E.1 we have

where the first inequality is due to Lemma E.1, and the second inequality is valid due to κ l = 0 . 01 · 2 -4 l . Similarly, we have

E.2 Proof of Lemma D.2

Proof of Lemma D.2. First, both l τ h ( s τ h ) = l and f τ h ( s τ h ) = 1 held for any τ ∈ C k h,l . This implies that the criteria for either Line 7 or Line 9 holds as l = l τ h ( s τ h ) . For τ that satisfies the first criterion, we have l τ h ( s τ h ) > L τ . Note that L τ = max {⌈ log 4 ( τ/d ) ⌉ , 0 } , so this only happens for τ < 4 l d . For other τ that satisfies the second criterion, we have that

where the first inequality holds due to the triangle inequality. In the second inequality, the first term ∥ ϕ τ h ∥ ˜ U τ, -1 h,l is bounded by criterion in Line 9 while the second term ∣ ∣ ∥ ϕ τ h ∥ ˜ U τ, -1 h,l - ∥ ϕ τ h ∥ ( U τ h,l ) -1 ∣ ∣ follows from Lemma D.1.

Arrange elements of C k h,l in ascending order as { τ i } i . According to the above reasoning, the number of elements τ ∈ C k h,l that ∥ ϕ τ h ∥ ( U τ h,l ) -1 ≥ 0 . 9 · 2 -l γ -1 l is at least |C k h,l | -4 l d . This gives

On the other hand, Lemma H.1 upper bounds the LHS of (E.1) by

Combining (E.1) and (E.2) gives

From algebra analysis in Lemma G.1, a necessary condition for (E.3) is |C k h,l | ≤ 16 l · 4 l γ 2 l d .

E.3 Proof of Lemma D.3

We first present a claim from Vial et al. (2022) controlling the infinite norm of coefficient w . Lemma E.2 (Claim 10, Vial et al. 2022) . For any ( k, h, l ) ∈ [ K ] × [ H ] × N + , we have ∥ w k h,l ∥ ∞ ≤ ∥ w k h,l ∥ 2 ≤ (2 l dH ) 4 .

Proof of Lemma D.3. Denote X ℓ as the set of all ˜ w k h,ℓ and Y ℓ as the set of all ˜ U k, -1 h,ℓ . From the definition of V k h,l , we have that |V k h,l | ≤ ∏ l ℓ =1 ( |X ℓ | · |Y ℓ | ) . From Lemma E.2, we have ∥ w k h,ℓ ∥ ∞ ≤ (2 ℓ dH ) 4 . Note that w k h,ℓ ∈ R d , we have the number of different ˜ w k h,ℓ controlled by

In addition, we have ∥ ( U k h,l ) -1 ∥ ∞ ≤ 1 /λ = 1 / 16 . So we can bound the number of ˜ U k, -1 h,ℓ by

As a result, we can conclude that

E.4 Proof of Lemma D.4

Proof of Lemma D.4. According to Proposition 3.2, there exists a parameter w h such that for any ( s, a ) ∈ S ×A , it holds that ∣ ∣ ⟨ ϕ ( s, a ) , w h ⟩ -[ B h ̂ V k h +1 ]( s, a ) ∣ ∣ ≤ 2 Hζ . Denoting η τ h = ⟨ ϕ τ h , w h ⟩ -[ B h ̂ V k h +1 ]( s τ h , a τ h ) and ε τ h = ( ̂ V k h +1 ( s τ h +1 ) -[ P h ̂ V k h +1 ]( s τ h , a τ h ) ) , we have

where the first equality holds due to the definition of U k h,l , w k h,l , the second equality holds by rearranging the terms, the third equality holds according the definition of η τ h , and the last equality holds from the relationship that [ B h ̂ V k h +1 ]( s τ h , a τ h ) = r τ h + [ P h ̂ V k h +1 ]( s τ h , a τ h ) . Therefore, for any vector ϕ ∈ R d , it holds that

where the second equality follows (E.4) and the inequality holds from Cauchy-Schwarz inequality (i.e., | x ⊤ Uy | ≤ ∥ x ∥ U ∥ y ∥ U ). From the triangle inequality, we have

There are three terms which we will bound respectively. For the first term, we have

√

where the first inequality holds due to the fact that ∥ w h ∥ 2 ≤ 2 H d as of Proposition 3.2 and the fact that U k h,l ⪰ λ I . Under the good event G 1 and Lemma C.5, the second term can be bounded by the following:

And the last term can be bounded by:

where the first inequality is due to Lemma H.3, and the second inequality follows from Lemma D.2. Plugging (E.6), (E.7), (E.8), and (E.9) into (E.5) gives

So for any ( s, a ) ∈ S × A , we have

where the first inequality holds from the triangle inequality, and there are three terms in the second inequality which we will bound them respectively: the first term is given by Lemma D.1, the second term follows (E.10), and the third term holds from the definition of w h .

E.5 Proof of Lemma D.5

Proof of Lemma D.5. We prove by doing case analysis. In case that action a l ∈ A k h,l +1 ( s ) , we can assign a l +1 = a l ∈ A k h,l +1 ( s ) so that

On the other hand, in the case that a l / ∈ A k h,l +1 ( s ) , the action a l is eliminated with Q k h,l ( s, a l ) < V k h,l ( s ) -4 · 2 -l . Note in this case, there exists a l +1 = π k h,l ( s ) ∈ A k h,l +1 ( s ) such that

According to Lemma C.6 and the condition that l ≤ L 0 , we have that empirical state-value function Q k h,l ( s, · ) is a good estimation for [ B h ̂ V k h +1 ]( s, · ) on actions a l , a l +1 ∈ A k l ( s ) under event G 1 :

Moreover,

where the first inequality is derived from combining (E.13), (E.14), and (E.15). So from (E.12) and (E.16), we have that [ B h ̂ V k h +1 ]( s, a l ) -[ B h ̂ V k h +1 ]( s, a l +1 ) ≤ 2 χ √ L 0 ζ holds in both cases.

E.6 Proof of Lemma D.6

Proof of Lemma D.6. We prove by induction on l . The induction basis holds at l = 0 by selecting a 1 = argmax a ∈A [ B h ̂ V k h +1 ]( s, a ) ∈ A which ensures max a ∈A [ B h ̂ V k h +1 ]( s, a ) -[ B h ̂ V k h +1 ]( s, a 1 ) = 0 . Additionally, if the induction hypothesis holds for l -1 , we have that

where the first inequality term is derived from combining induction hypothesis with Lemma D.5. We can then reach desired statement holds for all l in the range by induction.

E.7 Proof of Lemma D.7

Proof of Lemma D.7. According to Lemma D.6, there exists some action a l ∈ A k h,l ( s ) that

Moreover, we have

where the first inequality comes from the definition V k h,l ( s ) = max a ∈A k h,l Q k h,l ( s, a ) and the second inequality holds according to Lemma C.6 with l ≤ L 0 . Adding up (E.17) and (E.18) leads to

This completes the proof.

E.8 Proof of Lemma D.8

Proof of Lemma D.8. The statement holds by simply checking:

where the first inequality holds from max a ∈A [ B h ̂ V k h +1 ]( s, a ) ≥ [ B h ̂ V k h +1 ]( s, π k h,l ( s )) , the equality is from the definition V k h,l ( s ) = Q k h,l ( s, π k h,l ( s )) , and the last inequality holds according to Lemma C.6 with the condition l ≤ L 0 .

E.9 Proof of Lemma D.9

Proof of Lemma D.9. The statement holds by checking

where the first equality holds due to ̂ V k h,l ( s ) = min l ℓ =1 { V k h,ℓ ( s ) + 3 · 2 -ℓ } , the inequality holds according to Lemma D.7, and the last equality holds since 2 -l decreases as l increases.

E.10 Proof of Lemma D.10

Proof of Lemma D.10. The statement holds by checking

where the first equality holds due to the design of Algorithm 2 such that q V k h,l ( s ) = max l ℓ =1 { V k h,ℓ ( s ) -3 · 2 -ℓ } , the inequality holds according to Lemma D.8, and the last equality holds since 2 -l decreases as l increases.

E.11 Proof of Lemma D.11

We prove Lemma D.11 in this subsection. The first lemma which we introduce establishes an upper bound on the underestimation of the state value function ̂ V k h for every action and every state through a categorised discussion based on whether Algorithm 2 reaches phase L ε for state s . Specifically, if the process does not reach phase L ε , we can substantiate the statement by applying Lemma D.9 to phase l k h ( s ) -1 . Conversely, if the process reaches phase L ε , the statement can be proven by applying Lemma D.7 to phase L ε .

Lemma E.3. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h, s ) ∈ [ K ] × [ H ] ×S ,

Now we are ready to prove Lemma D.11 by induction.

Proof of Lemma D.11. We prove by induction on stage h ∈ [ H ] . It is sufficient to show for any h ∈ [ H ] , s ∈ S ,

We use induction on h from H +1 to 1 to prove the statement. The induction basis holds from the definition that V ∗ H +1 ( s ) = ̂ V k H +1 ( s ) = 0 . Assume the induction hypothesis (E.19) holds for h +1 , we have

So for level h , it holds that

where the first inequality holds by combining (E.20) with Lemma E.3. This proves the induction statement (E.19) for h , which leads to the desired statement.

E.12 Proof of Lemma D.12

We prove Lemma D.12 in this subsection, the first lemma we use establishes an upper bound on the overestimation of the state value function ̂ V k h for the executed policy π k h ( s ) across all states.

Lemma E.4. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h, s ) ∈ [ K ] × [ H ] ×S ,

Then the following lemma establishes an upper bound on the decision error induced by the armelimination process with respect to the state-action value function given by the ground-truth transform.

Lemma E.5. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h, s ) ∈ [ K ] × [ H ] ×S ,

Proof of Lemma D.12. We can directly reach the desired result by taking summation on Lemma E.4 and Lemma E.5:

E.13 Proof of Lemma D.14

We can prove the statement by applying a union bound to the concentration event, as given by the Azuma-Hoeffding inequality.

Proof of Lemma D.14. Consider some fixed h ∈ [ H ] and ε = 2 -l > 0 . List the episodes index k such that V ∗ h ( s k h ) -V π k h ( s k h ) > ε holds in ascending order as { τ i } i . Recall that

Since the environment sample s τ i h ′ +1 according to P h ′ ( ·| s τ i h ′ , a τ i h ′ ) , we have η τ i h ′ is F τ i h ′ +1 -measurable with E [ η τ i h ′ ∣ ∣ F τ i h ′ ] = 0 . Since both 0 ≤ ̂ V τ i h ′ +1 ( s τ i h ′ +1 ) ≤ H and 0 ≤ V π τ i h ′ +1 ( s τ i h ′ +1 ) ≤ H hold, we have | η τ i h ′ | ≤ 2 H . According to Lemma H.4 over filtration

for some fixed S = |K ε h | , the good event that

happens with probability at least 1 -δ/ (4 HS 2 l 2 ) . By the union bound statement over all ( h, S, l ) ∈ [ H ] × [ K ] × N + , we have the bad event happens with probability at most

where the last inequality holds from ∑ n ≥ 1 n -2 = π 2 / 6 , which reach the desired statement.

E.14 Proof of Lemma D.15

We first provide the following instantaneous regret upper bound by combining Lemma D.11 and Lemma D.12.

Lemma E.6. Under event G 1 and for all ε > 0 that G ε is satisfied, for any ( k, h ) ∈ [ K ] × [ H ] ,

where η k h = [ P h ( ̂ V k h +1 -V π k h +1 )]( s k h , π k h ( s k h )) -( ̂ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) ) is a F k h +1 -measurable random variable that E [ η k h |F k h ] = 0 and | η k h | ≤ H .

Together with Lemma C.9 and the definition of G 2 , we can provide an upper bound for arbitrary subsets.

Proof of Lemma D.15. Taking summation on result given by Lemma E.6 to all k ∈ K gives

We can bound the second term according to Lemma C.9,

Under event G 2 , the third term satisfies that

Plugging (E.22) and (E.23) into (E.21) gives

F Proof of Lemmas in Appendix E

F.1 Proof of Lemma E.3

Proof of Lemma E.3. We start the proof by discussing different cases. First, if l k h ( s ) ≤ L ε , we have l k h ( s ) -1 ≤ min { L ε , l k h ( s ) -1 } , according to the definition of ̂ V k h,l ( s ) ,

where the first inequality holds from Lemma D.9, and the last inequality holds due to χL 1 . 5 ε ζ ≤ 2 -L ε ≤ 0 . 01 ε/H given by G ε .

On the other hand, when l k h ( s ) > L ε , we have L ε ≤ min { L ε , l k h ( s ) -1 } and thus

where the first inequality is due to Lemma C.2 and the second inequality holds due to the definition of q V k h,L ε ( s ) . Therefore, L ε ≤ min { L ε , l k h ( s ) -1 } yields

where the first inequality is given by (F.2), the second inequality is given by Lemma D.7, and the last inequality holds from χL 1 . 5 ε ζ ≤ 2 -L ε ≤ 0 . 01 ε/H given by G ε . So considering both (F.1) and (F.3), we have the first statement

always holds under event G 1 .

F.2 Proof of Lemma E.4

We prove Lemma E.4 by applying Lemma C.7 on phase min { L ε , l k h ( s ) -1 } , in this subsection.

Proof of Lemma E.4. Note we have π k h,l k h ( s ) -1 ( s ) ∈ A k h,l k h ( s ) ( s ) according to the definition of A k h,l +1 ( s ) . This implies π k h ( s ) ∈ A k h,l k h ( s ) ( s ) during the elimination process.

If l k h ( s ) ≤ L ε , we have l k h ( s ) -1 ≤ min { L ε , l k h ( s ) -1 } . Thus,

where the first inequality follows from Lemma C.7 with π k h ( s ) ∈ A k h,l k h ( s ) ( s ) and the last inequality holds due to χL 1 . 5 ε ζ ≤ 0 . 01 ε/H given by G ε .

Otherwise, we have L ε ≤ min { L ε , l k h ( s ) -1 } . In this case, we have

where the first inequality follows from Lemma C.7 with π k h ( s ) ∈ A k h,l k h ( s ) ( s ) ⊆ A k h,L ε ( s ) according to the elimination routine and the final inequality holds due to χL 1 . 5 ε ζ ≤ 2 -L ε ≤ 0 . 01 ε/H given by G ε . So by combining (F.4) and (F.5), we have the desired statement that

F.3 Proof of Lemma E.5

We prove Lemma E.5 in this section by applying Lemma D.8 on phase min { L ε , l k h ( s ) -1 } .

Proof of Lemma E.5. If l k ( s ) ≤ L ε , we have l k ( s ) -1 ≤ min { L ε , l k ( s ) -1 } . Firstly, we have

where the first inequality is given by Lemma C.2 and the second inequality follows from the definition of ̂ V k h,l k h ( s ) -1 ( s ) . This leads to

where in the second inequality, the first term is given by (F.6) and the second term holds according to Lemma D.8, and the third inequality holds from χ √ L ε ζ ≤ 0 . 01 ε/H given by G ε .

Otherwise, we have L ε ≤ min { L ε , l k h ( s ) -1 } , this leads to

where in the second inequality, the first term is given by the definition of ̂ V k h ( s ) and the second term holds according to Lemma D.8, and the third inequality holds from χL 1 . 5 ε ζ ≤ 2 -L ε ≤ 0 . 01 ε/H given by G ε . Combining (F.7) and (F.8) gives the desired statement

F.4 Proof of Lemma E.6

Proof of Lemma E.6. According to the definition in which V π k h ( s k h ) = [ B h V π k h +1 ]( s k h , π k h ( s k h )) and η k h +[ P h ( ̂ V k h +1 -V π k h +1 )]( s k h , π k h ( s k h )) -( ̂ V k h +1 ( s k h +1 ) -V π k h +1 ( s k h +1 ) ) . We can write

By a telescoping statement from h to H with the final terminal value ̂ V k H +1 ( · ) = V π k H +1 ( · ) = 0 , we reach

As a result, we can bound the desired term by

where the first inequality is given by Lemma D.11, the first equality is given by (F.9), and the final inequality is given by Lemma D.12.

G Technical Numerical Lemmas

Lemma G.1. If |C k h,l | ≤ 4 l d +2 . 5 · 4 l γ 2 l d ln ( 1 + |C k h,l | / (16 d ) ) , then |C k h,l | ≤ 16 l · 4 l γ 2 l d .

Proof. Denote c = |C k h,l | / ( l · 4 l γ 2 l d ) . We have that

Dividing both sides by 4 l γ 2 l d , we have that

Since l ≥ 1 and γ l ≥ 1 , we can further conclude that

The necessary condition for the above inequality is c ≤ 16 , which proves the desired statement.

Lemma G.2. For any l ≥ 1 , γ l +1 /γ l ≤ 1 . 4 .

Proof. Firstly, we have that

where the first inequality holds due to log( x +1) ≤ 0 . 2 x +2 . In addition, we have

where the first inequality holds due to log( x +1) ≤ log( x )+1 . As a result, we can reach the desired statement according to

where the third inequality holds from plugging both (G.1) and (G.2).

Lemma G.3.

Proof. By calculation, we have that

Lemma G.4. If some constant c 1 , c 2 > 0 that

Then, there exists c 3 > 0 such that

where ι is a polynomial of log log( L ε dHδ -1 ) .

Proof. Let x = |K ε h | / log( |K ε h | ) . We have that

Since x < a + √ bx implies x < 2 a +2 b , so the above inequality implies

Moreover, since y/ log( y ) < a implies y < 2 a log a , we can conclude that there exists c 3 > 0 that

where ι is a polynomial of log log( L ε dHε -1 δ -1 ) .

H Auxiliary Lemmas

This section provides some auxiliary concentration lemmas frequently used in the proof.

Lemma H.1 (Lemma 11, Abbasi-Yadkori et al. (2011)) . Let { ϕ k } ∞ k =1 be any bounded sequence such that ϕ k ∈ R d and ∥ ϕ k ∥ 2 ≤ B for some constant B > 0 . For k ≥ 1 , let U k = λ I + ∑ k -1 τ =1 ϕ τ ( ϕ τ ) ⊤ . Let λ > 0 , then for all k ∈ [ K ] , we have that

Lemma H.2 (Self-Normalized Martingale, Abbasi-Yadkori et al. (2011)) . Let {F k } ∞ k =1 be a filtration, and { ϕ k , η k } ∞ k =1 be a stochastic process where ϕ k ∈ R d is G k -measurable and η k is F k +1 -measurable such that

for some constant B,R > 0 . Let λ > 0 . For k ≥ 1 , let U k = λ I + ∑ k -1 τ =1 ϕ τ ( ϕ τ ) ⊤ . Then for any δ ∈ (0 , 1) , with probability at least 1 -δ , for all k ≥ 1 , we have that

Lemma H.3 (Lemma 8, Zanette et al. (2020b)) . Let { ϕ k , η k } ∞ k =1 be any bounded sequence satisfying ϕ k ∈ R d and | η k | ≤ ζ for some constant ζ > 0 . For k ≥ 1 , let U k = λ I + ∑ k -1 τ =1 ϕ τ ( ϕ τ ) ⊤ . Then, for all k ≥ 1 , we have that

LemmaH.4 (Azuma-Hoeffding inequality, Hoeffding (1963)) . Let { η k } K k =1 be a martingale difference sequence with respect to a filtration {F k } K k =1 satisfying | η k | ≤ M for some constant M > 0 and η k is F k +1 -measurable with | E [ η k |F k ] | = 0 . Then for some fixed k ∈ [ K ] and any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Lemma H.5 (Freedman inequality, Cesa-Bianchi and Lugosi (2006)) . Let { η k } K k =1 be a martingale difference sequence with respect to a filtration {F k } K k =1 satisfying | η k | ≤ M for some constant M > 0 and η k is F k +1 -measurable with | E [ η k |F k ] | = 0 . Then for some fixed k ∈ [ K ] , a > 0 and v > 0 , we have

I Numerical Simulation

We added experiments on synthetic datasets to verify the performance of the algorithm and the contribution of each component. Specifically, we consider a linear MDP with S = 4 , A = 5 , H = 2 , and d = 8 . Each element in the feature vector ϕ ( s, a ) and µ ( s ′ ) is generated by a uniform distribution U (0 , 1) . Subsequently, ϕ is normalized to ensure that P ( s ′ | s, a ) is a probability measure, i.e., ϕ ( s, a ) = ϕ ( s, a ) / ∑ s ′ ϕ ⊤ ( s, a ) µ ( s ′ ) . The reward is defined by r ( s, a ) = ϕ ⊤ ( s, a ) θ , where θ ∼ N (0 , I d ) . The model misspecification is also added to the transition P and reward function r . For a given misspecification ζ , the ground truth reward function is defined by r ( s, a ) = ϕ ⊤ ( s, a ) θ + Z ( s, a ) , where Z ( s, a ) ∼ U ( -ζ, ζ ) . When adding the model misspecification to the transition kernel, we first random sample a subset S + ⊂ S such that |S + | = | S | / 2 . Then the misspecified transition kernel is then generated by

we can verify that ∥ P ( ·| s, a ) -P ′ ( ·| s, a ) ∥ TV = ζ . We investigated the misspecification level from ζ = 0 , 0 . 01 , · · · , 0 . 3 in 16 randomly generated environments over 2000 episodes. We report the cumulative regret and runtime with respect to different misspecification levels. Additionally, we performed an ablation study by 1) removing the certified estimation (Algorithm 2, Line 11) and 2) removing the quantization (Algorithm 1, Line 8).

The results of these configurations are presented in the following table. The detailed regret for all misspecification level is presented in Table 3 and Figure 1, we plot the cumulative regret for 2000 episodes with respect to the misspecification level ζ . The cumulative regret curve is plotted in Figure 2.

The experimental results suggest several key findings that support our theoretical analysis:

When the misspecification level is low, it is possible to achieve constant regret, where the instantaneous regret in the final rounds is approximately zero.

The certified estimator and the quantization do not significantly affect the algorithm's runtime. In contrast, the certified estimator provides an 'early-stopping' condition in Algorithm 2, which slightly reduces the algorithm's runtime. In particular, our algorithm yields a computational complexity of O ( d 2 AHK 2 log K ) , which is the same as Vial et al. (2022) and only log K greater than the vanilla LSVI-UCB (Jin et al., 2020) due to the multi-phased algorithm.

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Misspecification level

200

300

400

500

600

700

Average Cumulative Regret

Average Cumulative Regret vs. Misspecification level

Cert.Est.(×), Quant. (

)

Cert.Est.(

)

(a) With quantization

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Misspecification level

200

300

400

500

600

700

Average Cumulative Regret

Average Cumulative Regret vs. Misspecification level

Cert.Est.(×), Quant. (×)

Cert.Est.(

(b) Without quantization

The certified estimator helps the algorithm by providing robust estimation in the presence of misspecification. As shown in the table, using the certified estimator does not make a significant difference when the misspecification level ζ is low, but it becomes significant as the misspecification level increases.

The quantization does not contribute significantly to the results, as the numerical results are intrinsically discrete and quantized. In Figure 2, the regret curve with quantization and the one without quantization are highly overlapped.

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

25

50

75

100

125

150

175

200

Cumulative Regret

Cumulative Regret over Rounds for  =0

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(a)

ζ

= 0

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

50

100

150

200

Cumulative Regret

Cumulative Regret over Rounds for

=0.05

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(b)

ζ

= 0

.

05

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

50

100

150

200

250

300

Cumulative Regret

Cumulative Regret over Rounds for

=0.1

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(c)

ζ

= 0

.

10

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

100

200

300

400

Cumulative Regret

Cumulative Regret over Rounds for

=0.15

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(d)

ζ

= 0

.

15

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

100

200

300

400

500

Cumulative Regret

Cumulative Regret over Rounds for

=0.2

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(e)

ζ

= 0

.

20

0

250

500

750

1000

1250

1500

1750

2000

Number of Rounds

0

100

200

300

400

500

Cumulative Regret

Cumulative Regret over Rounds for

=0.25

Cert.Est.(×), Quant. (×)

Cert.Est.(×), Quant. (

)

Cert.Est.(

Cert.Est.(

)

(f)

ζ

= 0

.

25

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We study the constant regret analysis in reinforcement learning from a theoretical perceptive. The contribution, assumptions and scope are clearly claimed in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We discussed the limitations and future potential directions in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: We provide a detailed proof for all theorems in the appendix, starting from Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our submission paper does not include experiments

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with

the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work provides the theoretical understanding of reinforcement learning. Although there might be some potential social impacts on reinforcement learning applications, according to the guidelines, we believe our result does not have a direct connection with these issues.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We place this paper on the theoretical understand of reinforcement learning thus the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use existing assets in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies constant regret guarantees for linear MDPs. The main result is the first algorithm with this guarantee. The algorithm is a modification of LSVI-UCB with a careful quantization technique and some form of action elimination. This improves upon the regret of previous works by a factor of $\\log K$ where $K$ is the number of episodes.\nUnlike previous works, the algorithm is robust to some degree of model mispecification and does not need to know it in advance.\n\nSTRENGTHS:\n1. The method is robust to some model mispecification and does not need to know it in advance.\n2. The regret guarantee is tighter compared to previous works.\n3. The algorithm is computationally efficient in the sense that it runs in polynomial time in the problem parameters.\n4. There are significant technical challenges in the work. In particular the careful quantization required to avoid dependence on $K$ and the refined analysis of the regret bound by the suboptimality gaps.\n\nWEAKNESSES:\n1. While it's challenging to achieve regret that is independent of $K$, I do not see this as a significant improvement over the logarithmic dependence. This is especially true since it is not clear whether the correct polynomial dependence on the remaining parameters has been established. Moreover, this improvement is only relevant in the regime of a fixed success probability $\\delta$, which is often dependent on $K$, e.g., if bounding the expected regret.\n2. The improvement to the regret comes at a significant increase to the algorithm's complexity. It's not clear whether this is necessary. For example, are you certain that the quantization cannot be replaced by a more careful analysis of the value cover? Why do you need both bonuses and action elimination and sample rejection? The runtime complexity also seems higher (though this is a minor concern). If you think these modifications are indeed necessary, perhaps you could demonstrate this with experiments.\n3. While I am not familiar with better results, the dependence on the worst case gap weakens the result. Do you think it is possible to achieve a better dependence? e.g. on the average inverse gap with respect to the optimal policy.\n4. The writing is not very consistent with some places having phrasing and grammatical issues (e.g., quantification should be quantization and many more).\n5. The algorithm needs to be better explained. While it seems the authors made significant efforts in this regard, many parts of the algorithm remain rather unclear. For example, the bonuses are not immediately apparent. As I understand it, you use a quantization of the standard bonus. It would be much easier for the reader if this is made explicit in the algorithm and explained. Other examples are the action elimination and sample rejection whose purpose is unclear.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a constant regret learning algorithm for linear MDPs with approximation error (misspecification). Specifically, it proved an instance dependent regret that is independent of the number of episodes. In addition, the algorithm does not require prior knowledge of misspecification level or suboptimality gap.\n\nSTRENGTHS:\n1.\tThe background of the problem is stated clearly with well summarized review of the literature.\n2.\tThere is a comprehensive comparison of the new constant regret results with existing works.\n\nWEAKNESSES:\n1.\tThe proposed algorithm and analysis follow the algorithm in Vial et al. (2022) closely. The main differences seem to be the layer-dependent quantification in Algorithm 1 and the exit condition 3 in Algorithm 2. It would be better to discuss more clearly how these two modifications contribute to getting rid of the log(K) dependence. \n2.\tSome of notations are used without explanation. For example, $\\lambda$ in algorithm 1, $\\gamma_l$ in Algorithm 2 and Theorem 5.1. Also, how the certified estimator is defined in Algorithm 2 is not clear.\n3.\tSome analysis and results require more clarification: (a) The $\\delta$ condition used in Theorem 5.1 does not match the one used in Lemma C12. (b) It is not very clear how the key results in getting rid of log(k) (Lemma C14 line 765) is derived using $x\\leq a+\\sqrt{bx}$.\n\nQUESTIONS:\n1.\tThe new algorithm does not require UniSOFT assumption made by Papini et al. (2021a). However, in Thm 5 of that paper, they showed that UniSOFT is a necessary condition for constant regret. Can you explain how this does not apply to the current setup? \n2.\tIt is not very intuitive to understand the constant regret with no dependency on the number of episodes. Are there any simple examples that can run the algorithm empirically to verify this behavior?\n\nLIMITATIONS:\nThere is no potential negative societal impact.\n\nThe authors addressed the limitations in Section 7 such as the tightness of the bound and uniform misspecification assumption.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives a constant regret (total regret independent of the number of episodes $K$) algorithm for online reinforcement learning of linear MDPs. The environment is considered to be a $\\zeta$-approxinate linear MDP as in (Jin et al 2019), with an assumption that the misspecification level ($\\zeta$) is not too large. The algorithm is a more sophisticated variant of the original LSVI-UCB, which is a further improvement of earlier work that improves the regret of LSVI-UCB (but does not achieve constant regret).\n\nThe regret is parametrized by the instance-dependent minimal suboptimality gap $\\Delta := \\min_{h,s,a} \\\\{V_h^\\ast(s) - Q_h^*(s,a) : V_h^\\ast(s) - Q_h^*(s,a) \\neq 0\\\\}$, which also appears in earlier instance dependent regret bounds for linear MDPs (and also in general episodic MDPs).\n\nTheir algorithm, Cert-LSVI-UCB, has a regret bound of $\\tilde{O}(d^3 H^5 \\Delta^{-1})$. The dependence on $\\Delta$ is optimal, whereas the dependence on $d$ and $H$ may be suboptimal.\n\nThis paper is also an improvement on the previous work that provides \"constant regret\" bounds for linear MDPs (Papini et al 2021) in that it does not require the \"UniSoft\" assumption on the feature mapping (only requires the standard assumptions of misspecified linear MDPs as in Jin et al 2019, plus that the misspecification level $\\zeta \\leq \\tilde{O}(\\Delta/\\sqrt{d}H^2)$).\n\nSTRENGTHS:\n* The algorithm and its analysis gives a substantial improvement over earlier algorithms with instance-dependent bounds for linear MDPs, in that it is the first algorithm with a constant regret bound that requires minimal assumptions, even for misspecified linear MDPs.\n* The algorithm is cleanly based on earlier linear MDP algorithms starting from LSVI-UCB, applied in more sophisticated ways (multiple regression phases per episode etc), and the improvements in the algorithm and challenges in the analysis have been highlighted in the main paper itself (especially, in avoiding the dependency of the regret on the number of phases).\n\nWEAKNESSES:\n* The contribution of the paper is entirely theoretical, but most of the proof is in the appendix (which is unavoidable).\n* Some experimental results (even on synthetic data) would definitely enhance the paper.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\n* The limitations have been adequately addressed and there is no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
The paper presents improved regret bounds for Linear MDPs when the suboptimality gap \Delta is known. Most crucially (a bit surprisingly), the regret bounds are constant in a number of episodes K

STRENGTHS:
The contributions are broadly theoretical, and the removal of dependence on the number of episodes K seems to be a good contribution to the academic understanding of the gap-dependent (instance-dependent)learning bounds for reinforcement learning.

The paper provides a good coverage of related works in this area and describes the setting well.

WEAKNESSES:
1) b_h^\pi seems to be undefined in proposition 3.2
2) \kapp_\ell in Algorithm 1, line 8, as claimed by the author, helps to get rid of log(K). Can you explain more about how it helps to remove log(K)? is this the reason why the regret bounds do not have log(K) dependence?
3)I would urge the authors to provide a full expression of regret without O notation, including the log factors either in the main text or in the appendix (I could not find a full expression for the result in theorem 5.1). Specifically, it is surprising, perhaps I am missing something, that the regret is O(1) when Delta = 1/(d^3H^5) when K=1. I suspect some lower-order terms are hidden, or maybe the result is only valid for K \geq something. The asymptotic dependence can be right still.

QUESTIONS:
Please see above, I am willing to engage during the rebuttal phase.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17