Title: A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement learning

function approximation
Abstract: The exploration-exploitation dilemma has been a central challenge in reinforcement learning (RL) with complex model classes. In this paper, we propose a new algorithm, Monotonic  Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation. Our key algorithmic design includes (1) a general deterministic policy-switching strategy that achieves low switching cost, (2) a monotonic value function structure with carefully controlled function class complexity, and (3) a variance-weighted regression scheme that exploits historical trajectories with high data efficiency. MQL-UCB achieves minimax optimal regret of $\tilde{O}(d\sqrt{HK})$ when $K$ is sufficiently large and near-optimal policy switching cost of $\tilde{O}(dH)$, with $d$ being the eluder dimension of the function class, $H$ being the planning horizon, and $K$ being the number of episodes. 
   Our work sheds light on designing provably sample-efficient and deployment-efficient Q-learning with nonlinear function approximation.

A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation

Abstract

The exploration-exploitation dilemma has been a central challenge in reinforcement learning (RL) with complex model classes. In this paper, we propose a new algorithm, Monotonic Q-Learning with Upper Confidence Bound (MQL-UCB) for RL with general function approximation. Our key algorithmic design includes (1) a general deterministic policy-switching strategy that achieves low switching cost, (2) a monotonic value function structure with carefully controlled function class complexity, and (3) a variance-weighted regression scheme that exploits historical trajectories with high data efficiency. MQL-UCB achieves minimax optimal regret of ˜ O ( d √ HK ) when K is sufficiently large and near-optimal policy switching cost of ˜ O ( dH ) , with d being the eluder dimension of the function class, H being the planning horizon, and K being the number of episodes. Our work sheds light on designing provably sample-efficient and deployment-efficient Q-learning with nonlinear function approximation.

1 Introduction

In reinforcement learning (RL), a learner interacts with an unknown environment and aims to maximize the cumulative reward. As one of the most mainstream paradigms for sequential decisionmaking, RL has extensive applications in many real-world problems (Kober et al., 2013; Mnih et al., 2015; Lillicrap et al., 2015; Zoph and Le, 2016; Zheng et al., 2018). Theoretically, the RL problem is often formulated as a Markov Decision Process (MDP) (Puterman, 2014). Achieving the optimal regret bound for various MDP settings has been a long-standing fundamental problem in RL research. In tabular MDPs where the state space S and the action space A are finite, the optimal regret bound has been well-established, ranging from the episodic setting (Azar et al., 2017; Zanette and Brunskill, 2019; Simchowitz and Jamieson, 2019; Zhang et al., 2020), average-reward setting (Zhang and Ji, 2019) to the discounted setting (He et al., 2021b). Nevertheless, these regret guarantees are intolerably large in many real-world applications, where the state space S and the action space A are often large or even infinite.

As is commonly applied in applications, function approximation has been widely studied by theorists to demonstrate the generalization across large state-action spaces, proving the performance guarantees of various RL algorithms for specific function classes. There are recent works on RL with linear function approximation under different assumptions such as linear MDPs (Yang and Wang, 2019; Jin et al., 2020), linear mixture MDPs (Modi et al., 2020; Ayoub et al., 2020; Zhou et al., 2021a). Among them, Zhou et al. (2021a) achieved nearly optimal regret bounds for linear mixture MDPs through a model-based approach adopting variance-weighted linear regression. Later, Hu et al. (2022) proposed LSVI-UCB+ algorithm, making an attempt to improve the regret for linear MDP through an over-optimistic value function approach. However, their analysis was later discovered to suffer from a technical issue (Agarwal et al., 2022; He et al., 2022). To fix this issue, Agarwal et al. (2022) introduced similar over-optimistic value functions to construct a monotonic variance estimator and a non-Markov policy, achieving the first statistically optimal regret for linear MDPs. They also proposed a novel algorithm dubbed VO Q L for RL with general function approximation. Concurrently, He et al. (2022) proposed LSVI-UCB++, which takes a different approach and employs a rare-switching technique to obtain the optimal regret. In a parallel line of research, there has been a growing body of literature proposing more general frameworks to unify sample efficient RL algorithms, e.g., Bellman rank (Jiang et al., 2017), Witness rank (Sun et al., 2019), eluder dimension (Russo and Van Roy, 2013), Bellman eluder dimension (Jin et al., 2021), Bilinear Classes (Du et al., 2021), Decision-Estimation Coefficient (Foster et al., 2021), Admissible Bellman Characterization (Chen et al., 2022) and Decoupling Coefficient (Agarwal and Zhang, 2022a,b). However, when applying these frameworks to linear MDPs, none of them can achieve the minimax optimal regret. To our knowledge, Agarwal et al. (2022) is the only algorithmic framework achieving optimal regret for RL beyond linear function approximation. Since VO Q L requires a non-Markov planning procedure, where the resulting policy does not act greedily with respect to a single optimistic value function, it is natural to ask

Can we develop a RL algorithm with Markov policy 1 to solve MDPs with general function approximation and achieve the optimal regret?

While the sample efficiency of RL algorithms for MDPs with nonlinear function classes has been comprehensively researched, deployment efficiency (Matsushima et al., 2020) is also a major concern in many real-world application scenarios. For example, in recommendation systems (Afsar et al., 2022), it may take several weeks to deploy a new recommendation policy. On the other hand, the system is capable of collecting an enormous amount of data every minute implementing a fixed policy. As a result, it is computationally inefficient to change the executed policy after each data point is collected as is demanded by most of the online RL algorithms in theoretical studies. To resolve this issue, Bai et al. (2019) first introduced the concept of switching cost, defined as the number of policy updates. Following this concept, a series of RL algorithms have been proposed on the theoretical side with low switching cost guarantees (e.g., Zhang et al., 2020; Wang et al., 2021; Qiao et al., 2022; Kong et al., 2021; Velegkas et al., 2022; Li et al., 2023). Xiong et al. (2023) considered low-switching RL with general function approximation, which achieves ˜ O ( dH ) switching cost. However, their algorithm has an intractable planning phase and is not statistically optimal. Kong et al. (2021), Velegkas et al. (2022), and Li et al. (2023) considered RL with general function approximation, all of which achieved the switching cost of O ( d 2 H polylog ( K )) , where d is the eluder dimension of the underlying function class, H is the planning horizon, and K is the number of episodes. In contrast, Gao et al. (2021) proved an Ω( dH/ log d ) lower bound of switching cost for any deterministic algorithms in learning linear MDPs. Therefore, the following question remains open:

Can we design an efficient algorithm with ˜ O ( dH ) switching cost for MDPs with bounded eluder dimension?

In this paper, we answer the above two questions simultaneously by proposing a novel algorithm Monotonic Q-Learning with UCB (MQL-UCB) with all the aforementioned appealing properties. At the core of our algorithmic design are the following innovative techniques:

We propose a novel policy-switching strategy based on the cumulative sensitivity of historical data. To the best of our knowledge, this is the first computationally-tractable deterministic rare-switching algorithm for RL with general function approximation which achieves ˜ O ( dH ) switching cost. We also prove a nearly matching lower bound for any algorithm with arbitrary policies including both deterministic and stochastic policies (See Theorem B.1). Previous approaches for low switching cost in RL with general function approximation are sampling-based (Kong et al., 2021; Velegkas et al., 2022; Li et al., 2023) and highly coupled with a sub-sampling technique used for regression, making it less efficient. When restricted to the linear case, sampling-based rare-switching has a worse switching cost of ˜ O ( d 2 H ) (Section 3.3 in Kong et al. 2021; Theorem 3.3 in Velegkas et al. 2022; Section C in Li et al. 2023) than that in Wang et al. (2021).

With the novel policy-switching scheme, we illustrate how to reduce the complexity of value function classes while maintaining a series of monotonic value functions, strictly generalizing the LSVI-UCB++ algorithm (He et al., 2022) to general function class with bounded eluder dimension. Based on the structure of the value functions, we further demonstrate how MQL-UCB achieves a variance-dependent regret bound that can gracefully reduce to a nearly constant regret for deterministic MDPs. While in the worst case, our algorithm still attains a nearly minimax optimal regret guarantee. Our work is the first work for RL with general function approximation that achieves the nearly minimax optimal regret when specialized to linear MDPs, while still enjoys simple Markov planning phase.

It is worth noting that recently, Qiao et al. (2023) also considered RL with low-switching cost beyond linear function approximation, i.e., MDPs with low inherent Bellman error (Zanette et al., 2020b) and generalized linear MDPs (Wang et al., 2020b). Their approach can be seen as a slight extension of RL with low-switching cost for linear MDPs, since in both settings, the covariance matrix still exists and they can still use the determinant of the covariance as a criterion for policy switching.

Notation. We use lower case letters to denote scalars and use lower and upper case bold face letters to denote vectors and matrices respectively. We denote by [ n ] the set { 1 , . . . , n } . For two positive sequences { a n } and { b n } with n = 1 , 2 , . . . , we write a n = O ( b n ) if there exists an absolute constant C > 0 such that a n ≤ Cb n holds for all n ≥ 1 and write a n = Ω( b n ) if there exists an absolute constant C > 0 such that a n ≥ Cb n holds for all n ≥ 1 . We use ˜ O ( · ) to further hide the polylogarithmic factors except log-covering numbers. We use 1 {·} to denote the indicator function.

2 Preliminaries

2.1 Time-Inhomogeneous Episodic MDP

We consider a time-inhomogeneous episodic Markov Decision Process (MDP), denoted by a tuple M = M ( S , A , H, { P h } H h =1 , { r h } H h =1 ) . Here, S and A are the spaces of state and action, respectively, H is the length of each episode, P h : S × A × S → [0 , 1] is the transition probability function at stage h which denotes the probability for state s to transfer to the next state s ′ with current action a , and r h : S × A → [0 , 1] is the deterministic reward function at stage h . A policy π := { π h } H h =1 is a collection of mappings π h from an observed state s ∈ S to the simplex of action space A . For any policy π = { π h } H h =1 and stage h ∈ [ H ] , we define the value function V π h ( s ) and the action-value function Q π h ( s, a ) as follows:

where s h ′ +1 ∼ P h ′ ( ·| s h ′ , a h ′ ) . Then, we further define the optimal value function V ∗ h and the optimal action-value function Q ∗ h as V ∗ h ( s ) = max π V π h ( s ) and Q ∗ h ( s, a ) = max π Q π h ( s, a ) . For simplicity, we assume the total reward for each possible trajectory ( s 1 , a 1 , ..., s H , a H ) satisfies ∑ H h =1 r h ( s h , a h ) ≤ 1 . Under this assumption, the value function V π h ( · ) and Q π h ( · , · ) are bounded in [0 , 1] . For any function V : S → R and stage h ∈ [ H ] , we define the following first-order Bellman operator T h and second-order Bellman operator T 2 h on function V :

where s h +1 ∼ P h ( ·| s h , a h ) and r h = r h ( s h , a h ) . For simplicity, we further define [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ′ ) and [ V h V ]( s, a ) = T 2 h V ( s h , a h ) -( T h V ( s h , a h ) ) 2 . Using this notation, for each stage h ∈ [ H ] , the Bellman equation and Bellman optimality equation take the following forms:

where V π H +1 ( · ) = V ∗ H +1 ( · ) = 0 . At the beginning of each episode k ∈ [ K ] , the agent selects a policy π k to be executed throughout the episode, and an initial state s k 1 is arbitrarily selected by the environment. For each stage h ∈ [ H ] , the agent first observes the current state s k h , chooses an action following the policy π k h , then transits to the next state with s k h +1 ∼ P h ( ·| s k h , a k h ) and reward r h ( s h , a h ) . Based on the protocol, we defined the suboptimality gap in episode k as the difference between the value function for selected policy π k and the optimal value function V ∗ 1 ( s k 1 ) -V π k 1 ( s k 1 ) . Based on these definitions, we can define the regret in the first K episodes as follows:

Definition 2.1. For any RL algorithm Alg , the regret in the first K episodes is denoted by the sum of the suboptimality for episode k = 1 , . . . , K ,

where π k is the agent's policy at episode k .

2.2 Function Classes and Covering Numbers

Assumption 2.2 (Completeness) . Given F := {F h } H h =1 which is composed of bounded functions f h : S × A → [0 , L ] . We assume that for any function V : S → [0 , 1] there exists f 1 , f 2 ∈ F h such that for any ( s, a ) ∈ S × A ,

We assume that L = O (1) throughout the paper.

Remark 2.3. Completeness is a fundamental assumption in RL with general function approximation, as recognized in Wang et al. (2020b); Jin et al. (2021); Agarwal et al. (2022). Our assumption is the same as that in Agarwal et al. (2022) and is slightly stronger than that in Wang et al. (2020a) and

Jin et al. (2021). More specifically, in Wang et al. (2020a), completeness is only required for the first-order Bellman operator. In contrast, we necessitate completeness with respect to the second-order Bellman operator, which becomes imperative during the construction of variance-based weights. Jin et al. (2021) only requires the completeness for the function class F h +1 ( T h F h +1 ⊆ F h ). However, the GOLF algorithm (Jin et al., 2021) requires solving an intricate optimization problem across the entire episode. In contrast, we employ pointwise exploration bonuses as an alternative strategy, which requires the completeness for function class V = { V : S → [0 , 1] } , i.e., T h V ⊆ F h . The completeness assumption on the second moment is first introduced by Agarwal et al. (2022), and is crucial for obtaining a tighter regret bound. More specifically, making use of the variance of the value function at the next state is known to be crucial to achieve minimax-optimal regret bound in RL ranging from tabular MDPs (Azar et al., 2017) to linear mixture MDPs (Zhou et al., 2021a) and linear MDPs (He et al., 2022). In RL with general function approximation, the second-moment compleness assumption makes the variance of value functions computationally tractable.

Definition 2.4 (Generalized Eluder dimension, Agarwal et al. 2022) . Let λ ≥ 0 , a sequence of state-action pairs Z = { z i } i ∈ [ T ] and a sequence of positive numbers σ = { σ i } i ∈ [ T ] . The generalized Eluder dimension of a function class F : S × A → [0 , L ] with respect to λ is defined by dim α,T ( F ) := sup Z , σ : | Z | = T, σ ≥ α dim( F , Z , σ ) ,

We write dim α,T ( F ) := H -1 · ∑ h ∈ [ H ] dim α,T ( F h ) for short when F is a collection of function classes F = {F h } H h =1 in the context.

Remark 2.5. The D 2 F quantity has been introduced in Agarwal et al. (2022) and Ye et al. (2023) to quantify the uncertainty of a state-action pair given a collected dataset with corresponding weights. It was inspired by Gentile et al. (2022) where an unweighted version of uncertainty has been defined for active learning. Prior to that, Wang et al. (2020a) introduced a similar 'sensitivity' measure to determine the sampling probability in their sub-sampling framework. As discussed in Agarwal et al. (2022), when specialized to linear function classes, D 2 F ( z t ; z [ t -1] , σ [ t -1] ) can be written as the elliptical norm ∥ z t ∥ 2 Σ -1 t -1 , where Σ t -1 is the weighted covariance matrix of the feature vectors z [ t -1] .

Definition 2.6 (Bonus oracle ¯ D 2 F ) . In this paper, the bonus oracle is denoted by ¯ D 2 F , which computes the estimated uncertainty of a state-action pair z = ( s, a ) ∈ S × A with respect to historical data z [ t -1] and corresponding weights σ [ t -1] . In detail, we assume that a computable function ¯ D 2 F ( z ; z [ t -1] , σ [ t -1] ) satisfies ¯ D F ( z ; z [ t -1] ,σ [ t -1] ) ∈ [1 , C ] , where C is a fixed constant.

Remark 2.7. Agarwal et al. (2022) also assumed access to such a bonus oracle defined in Definition 2.6, where they assume that the bonus oracle finds a proper bonus from a finite bonus class (Definition 3, Agarwal et al. 2022). Our definition is slightly different in the sense that the bonus class is not assumed to be finite but with a finite covering number. Previous works by Kong et al. (2021) and Wang et al. (2020a) proposed a sub-sampling idea to compute such a bonus function efficiently in general cases, which is also applicable in our framework. In a similar nonlinear RL setting, Ye et al. (2023) assumed that the uncertainly D 2 F can be directly computed, which is a slightly stronger assumption. But essentially, these differences in bonus assumption only lightly affect the algorithm structure.

Definition 2.8 (Covering numbers of function classes) . For any ϵ > 0 , we define the following covering numbers of the involved function classes:

For each h ∈ [ H ] , there exists an ϵ -cover C ( F h , ϵ ) ⊆ F h with size |C ( F h , ϵ ) | ≤ N ( F h , ϵ ) , such that for any f ∈ F , there exists f ′ ∈ C ( F h , ϵ ) , such that ∥ f -f ′ ∥ ∞ ≤ ϵ . For any ϵ > 0 , we define the uniform covering number of F with respect to ϵ as N F ( ϵ ) := max h ∈ [ H ] N ( F h , ϵ ) .

There exists a bonus class B : S × A → R such that for any t ≥ 0 , z [ t ] ∈ ( S × A ) t , σ [ t ] ∈ R t , the oracle defined in Definition 2.6 ¯ D F ( · ; z [ t ] , σ [ t ] ) is in B .

For bonus class B , there exists an ϵ -cover C ( B , ϵ ) ⊆ B with size |C ( B , ϵ ) | ≤ N ( B , ϵ ) , such that for any b ∈ B , there exists b ′ ∈ C ( B , ϵ ) , such that ∥ b -b ′ ∥ ∞ ≤ ϵ .

Remark 2.9. In general function approximation, it is common to introduce the additional assumption on the covering number of bonus function classes. For example, in Ye et al. (2023), Agarwal and Zhang (2022a), and Di et al. (2023), the covering number of the bonus function class is bounded.

3 Algorithm and Key Techniques

In this section, we will introduce our new algorithm, MQL-UCB. The detailed algorithm is provided in Algorithm 1. Our algorithm's foundational framework follows the Upper Confidence Bound (UCB) approach. In detail, for each episode k ∈ [ K ] , we construct an optimistic value function Q k,h ( s, a ) during the planning phase. Subsequently, in the exploration phase, the agent interacts with the environment, employing a greedy policy based on the current optimistic value function Q k,h ( s, a ) . Once the agent obtains the reward r k h and transitions to the next state s k h +1 , these outcomes are incorporated into the dataset, contributing to the subsequent planning phase. We will proceed to elucidate the essential components of our method.

3.1 Rare Policy Switching

For MQL-UCB algorithm, the value functions Q k,h , q Q k,h , along with their corresponding policy π k , undergo updates when the agent collects a sufficient number of trajectories within the dataset that could significantly diminish the uncertainty associated with the Bellman operator T h V ( · , · ) through the weighted regression. In the context of linear bandits (Abbasi-Yadkori et al., 2011) or linear MDPs (He et al., 2022), the uncertainty pertaining to the least-square regression is quantified by the covariance matrix Σ k . In this scenario, the agent adjusts its policy once the determinant of the covariance matrix doubles, employing a determinant-based criterion. Nevertheless, in the general function approximation setting, such a method is not applicable in the absence of the covariance matrix which serves as a feature extractor in the linear setting. Circumventing this issue, Kong et al. (2021) proposed a sub-sampling-based method to achieve low-switching properties in nonlinear RL. Their subsampling technique is inspired by Wang et al. (2021), which showed that one only needs to maintain a small subset of historical data to obtain a sufficiently accurate least-square estimator. Such a subset can be generated sequentially according to the sensitivity of a new coming data point. However, their approach leads to a switching cost of ˜ O ( d 2 H ) , which does not match the lower bound in linear MDPs proposed by Gao et al. (2021).

To resolve this issue, we proposed a more general deterministic policy-updating framework for nonlinear RL. In detail, we use ¯ D 2 F h ( z i,h ; z [ k last -1] ,h , ¯ σ [ k last -1] ,h ) to evaluate the information collected at the episode i , given the last updating k last . Once the collected information goes beyond a threshold χ from last updating, i.e.,

the agent will perform updates on both the optimistic estimated value function and the pessimistic value function. Utilizing the D 2 F h -based criterion, we will show that the number of policy updates can be bounded by O ( H · dim α,K ( F )) . This further reduces the complexity of the optimistic value function class and removes additional factors from a uniform convergence argument over the function class. Specifically, we showcase under our rare-switching framework, the ϵ -covering number of the optimistic and the pessimistic value function class at episode k is bounded by

where p β K is the maximum confidence radius as shown in Algorithm 1, which will be specified in Lemmas G.3 and G.4, l k is the number of policy switches before the end of the k -th episode according to Algorithm 1.

3.2 Weighted Regression

The estimation of Q function in MQL-UCB extends LSVI-UCB proposed by Jin et al. (2020) to general function classes. While the estimators in LSVI-UCB are computed from the classic least

squares regression, we construct the estimated value function p f k,h for general function classes by solving the following weighted regression:

In the weighted regression, we set the weight ¯ σ k,h as

where σ k,h is the estimated variance for the stochastic transition process, ¯ D F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) denotes the uncertainty of the estimated function p f k,h conditioned on the historical observations and

is used to properly balance the uncertainty across various state-action pairs. It is worth noting that Ye et al. (2023) also introduced the uncertainty-aware variance in the general function approximation with a distinct intention to deal with the adversarial corruption from the attacker. In addition, according to the weighted regression, with high probability, the Bellman operator T h V k,h satisfies:

where p β k is the exploration radius for the Q functions, which will be specified in Theorem 4.1. According to the definition of Generalized Eluder dimension, the estimation error between the estimated function p f k,h and the Bellman operator is upper bounded by:

Therefore, we introduce the exploration bonus b k,h and construct the optimistic value function Q k,h ( s, a ) ,i.e., Q k,h ( s, a ) ≈ p f k,h ( s, a ) + b k,h ( s, a ) , where b k,h ( s, a ) = p β k · ¯ D F ( ( s, a ); z [ k -1] ,h , σ [ k -1] ,h ) . Inspired by Hu et al. (2022); He et al. (2022); Agarwal et al. (2022), in order to estimate the gap between the optimistic value function V k,h ( s ) and the optimal value function V ∗ h ( s ) , we further construct the pessimistic estimator q f k,h by the following weighted regression

and introduce a negative exploration bonus when generating the pessimistic estimator. q Q k,h ( s, a ) ≈ q f k,h ( s, a ) -q b k,h ( s, a ) , where q b k,h ( s, a ) = q β k · ¯ D F (( s, a ); z [ k -1] ,h , σ [ k -1] ,h . Different from Agarwal et al. (2022), the pessimistic value function q f k,h is computed from a similar weighted-regression scheme as in the case of the optimistic estimator, leading to a tighter confidence set.

3.3 Variance Estimator

In this subsection, we provide more details about the variance estimator σ k,h , which measures the variance of the value function V k,h +1 ( s k h +1 ) caused by the stochastic transition from state-action pair ( s k h , a k h ) . According to the definition of p f k,h , the difference between the estimator p f k,h and T h V k,h +1 satisfies

where the noise p η k h ( V ) = r k h + V ( s k h +1 ) -E s ′ ∼ P h ( s k h ,a k h ) [ r h ( s k h , a k h , s ′ ) + V ( s ′ )] denotes the stochastic transition noise for the value function V . However, the generation of the target function V k,h +1 relies on previously collected data z [ k last ] , thus violating the conditional independence property. Consequently, the noise term p η k h ( V k,h +1 ) may not be unbiased. To address this challenge, it becomes imperative to establish a uniform convergence property over the potential function class, which is first introduced in linear MDPs by Jin et al. (2020).

Following the previous approach introduced by Azar et al. (2017); Hu et al. (2022); Agarwal et al. (2022); He et al. (2022), we decompose the noise of optimistic value function p η k h ( V k,h +1 ) into the

Algorithm 1 Monotonic Q-Learning with UCB (MQL-UCB)

Require: Regularization parameter λ , confidence radius { ˜ β k } k ∈ [ K ] , { p β k } k ∈ [ K ] and { q β k } k ∈ [ K ] .

1: Initialize k last = 0 . For each stage h ∈ [ H ] and state-action ( s, a ) ∈ S × A , set Q 0 ,h ( s, a ) ← H, q Q 0 ,h ( s, a ) ← 0 .

2: for episodes k = 1 , . . . , K do

3: Received the initial state s k 1 .

4: for stage h = H,.. . , 1 do

5: if there exists a stage h ′ ∈ [ H ] such that (3.1) holds then

11: Set the last updating episode k last ← k and number of policies as l k ← l k -1 +1 .

12: else

14: end if

15: Set the policy π k as π k h ( · ) ← argmax a ∈A Q k,h ( · , a ) . V k,h ( s ) ← max a Q k,h ( s, a ) , q V k,h ( s ) ← max a q Q k,h ( s, a ) .

16: end for

17: for stage h = 1 , . . . , H do

18: Take action a k h ← π k h ( s k h ) and receive next state s k h +1 .

19: Set the estimated variance σ k,h as in (3.4) and set ¯ σ k,h ← max { σ k,h , α, γ · D 1 / 2 F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) } .

20: end for

21: end for

noise of optimal value function p η k h ( V ∗ h +1 ) and the noise p η k h ( V k,h +1 -V ∗ h +1 ) to reduce the extra log ( N ϵ ( K ) ) dependency in the confidence radius. With the noise decomposition, we evaluate the variances [ V h V ∗ h +1 ]( s, a ) and [ V h ( V k,h +1 -V ∗ h +1 ) ] ( s, a ) separately.

For the variance of the optimal value function [ V h V ∗ h +1 ]( s, a ) , since the optimal value function V ∗ h +1 is independent with the collected data z [ k last ] , it prevents a uniform convergence-based argument over the function class. However, the optimal value function V ∗ h +1 is unobservable, and it requires several steps to estimate the variance. In summary, we utilize the optimistic function V k,h +1 to approximate the optimal value function V ∗ h +1 and calculate the estimated variance [ ¯ V h V k,h ] as the difference between the second-order moment and the square of the first-order moment of V k,h

Here, the approximate second-order moment ˜ f k,h and the approximate first-order moment p f k,h is generated by the least-square regression (Lines 6 and 8). In addition, we introduce the exploration bonus E k,h to control the estimation error between the estimated variance and the true variance of V k,h +1 and F k,h to control the sub-optimality gap between V k,h +1 and V ∗ h +1 :

For the variance of the sub-optimality gap, [ V h ( V k,h +1 -V ∗ h +1 ) ] ( s, a ) , based on the structure of optimistic and pessimistic value function, it can be approximate and upper bounded by

where the approximate first-order moments p f k,h , q f k,h are generated by the least-square regression (Lines 6 and 7) and can be dominated by the exploration bonus F k,h .

In summary, we construct the estimated variance σ k,h as:

3.4 Monotonic Value Function

As we discussed in the previous subsection, we decompose the value function V k,h and evaluate the variance [ V h V ∗ h +1 ]( s, a ) , [ V h ( V k,h +1 -V ∗ h +1 ) ] ( s, a ) separately. However, for each state-action pair ( s k h , a k h ) and any subsequent episode i > k , the value function V i,h and corresponding variance [ V h ( V i,h +1 -V ∗ h +1 ) ] ( s k h , a k h ) may differ from the previous value function V k,h and variance [ V h ( V k,h +1 -V ∗ h +1 ) ] ( s k h , a k h ) . Extending the idea proposed by He et al. (2022) for linear MDPs, we ensure that the pessimistic value function q Q k,h maintains a monotonically increasing property during updates, while the optimistic value function Q k,h maintains a monotonically decreasing property. Leveraging these monotonic properties, we can establish an upper bound on the variance as follows:

In this scenario, the previously employed variance estimator σ k,h offers a consistent and uniform upper bound for the variance across all subsequent episodes.

4 Main Results

In this section, we present the main theoretical results. In detail, we provide the regret upper bound of Algorithm MQL-UCB in Theorem 4.1. As a complement, in Section B.1, we provide a lower bound on the communication complexity for cooperative linear MDPs. Finally, in Section B.2, we discussed the connection between the generalized eluder dimension and the standard eluder dimension.

The following theorem provides the regret upper bound of Algorithm MQL-UCB.

Theorem 4.1. Suppose Assumption 2.2 holds for function classes F := {F h } H h =1 and Definition 2.4 holds with λ = 1 . If we set α = 1 / √ KH , ϵ = ( KLH ) -1 , and set p β 2 k = q β 2 k := O ( log 2 k 2 ( 2 log( L 2 k/α 4 )+2 ) · ( log(4 L/α 2 )+2 ) δ/H ) · [log( N F ( ϵ )) + 1] + O ( λ ) + O ( ϵkL/α 2 ) , then with probability 1 -O ( δ ) , the regret of MQL-UCB is upper bounded as follows:

where Var K := ∑ K k =1 ∑ H h =1 [ V h V π k h +1 ]( s k h , a k h ) = ˜ O ( K ) , we denote the covering number of bonus function class by N b , the covering number of function class F by N , and the dimension dim α,K ( F ) by dim( F ) . Meanwhile, the switching cost of Algorithm 1 is O (dim α,K ( F ) · H ) .

In the worst case, when the number of episodes K is sufficiently large, the leading term in our regret bound is ˜ O (√ dim( F ) log N · HK ) . Our result matches the optimal regret achieved by Agarwal et al. (2022). While their proposed algorithm involves the execution of a complicated and nonMarkovian policy with an action selection phase based on two series of optimistic value functions and the prefix trajectory, MQL-UCB only requires the knowledge of the current state and a single optimistic state-action value function Q to make a decision over the action space. In addition, our theorem also provides a variance-dependent regret bound, which is adaptive to the randomness of the underlying MDP encountered by the agent. Our definition of Var K is inspired by Zhao et al. (2023) and Zhou et al. (2023), which studied variance-adaptive RL under tabular MDP setting and linear mixture MDP setting, respectively.

As a direct application, we also present the regret guarantee of MQL-UCB for linear MDPs.

Corollary 4.2. Under the same conditions of Theorem 4.1, assume that the underlying MDP is a linear MDP such that F := {F h } h ∈ [ H ] is composed of linear function classes with a known feature mapping over the state-action space ϕ : S × A → R d . If we set λ = 1 , α = 1 / √ K , then with probability 1 -O ( δ ) , the following cumulative regret guarantee holds for MQL-UCB:

Remark 4.3. The leading term in our regret bound, as demonstrated in Corollary 4.2, matches the lower bound proved in Zhou et al. (2021a) for linear MDPs. Similar optimal regrets have also been accomplished by He et al. (2022) and Agarwal et al. (2022) for linear MDPs. Since we also apply weighted regression to enhance the precision of our pessimistic value functions, the lower order term (i.e., H 2 . 5 d 5 √ H + d 2 ) in our regret has a better dependency on H than VO Q L (Agarwal et al., 2022) and LSVI-UCB++ (He et al., 2022), which may be of independent interest when considering long-horizon MDPs. In addition, the switching cost of Algorithm 1 is bounded by ˜ O ( dH ) , which matches the lower bound in Gao et al. (2021) for deterministic algorithms and our new lower bound in Theorem B.1 for arbitrary algorithms up to logarithmic factors. For more details about our lower bound, please refer to Appendix E.

5 Conclusion and Future Work

In this paper, we delve into the realm of RL with general function approximation. We proposed the MQL-UCBalgorithm with an innovative uncertainty-based rare-switching strategy in general function approximation. Notably, our algorithm only requires ˜ O ( dH ) updating times, which matches with the lower bound established by Gao et al. (2021) up to logarithmic factors, and obtains a ˜ O ( d √ HK ) regret guarantee, which is near-optimal when restricted to the linear cases.

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their helpful comments. HZ is partially supported by the research award from Cisco. JH and QG are partially supported by the research fund from UCLAAmazon Science Hub. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

References

ABBASI-YADKORI, Y., PÁL, D. and SZEPESVÁRI, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24 2312-2320.

AFSAR, M. M., CRUMP, T. and FAR, B. (2022). Reinforcement learning based recommender systems: A survey. ACM Computing Surveys 55 1-38.

AGARWAL, A., JIN, Y. and ZHANG, T. (2022). Voql: Towards optimal regret in model-free rl with nonlinear function approximation. arXiv preprint arXiv:2212.06069 .

AGARWAL, A. and ZHANG, T. (2022a). Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. Advances in Neural Information Processing Systems 35 3528435297.

AGARWAL, A. and ZHANG, T. (2022b). Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. In Conference on Learning Theory . PMLR.

AYOUB, A., JIA, Z., SZEPESVARI, C., WANG, M. and YANG, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning . PMLR.

AZAR, M. G., OSBAND, I. and MUNOS, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning . PMLR.

BAI, Y., XIE, T., JIANG, N. and WANG, Y.-X. (2019). Provably efficient q-learning with low switching cost. Advances in Neural Information Processing Systems 32 .

CHEN, Z., LI, C. J., YUAN, H., GU, Q. and JORDAN, M. (2022). A general framework for sample-efficient function approximation in reinforcement learning. In The Eleventh International Conference on Learning Representations .

DANN, C., JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2018). On oracle-efficient pac rl with rich observations. Advances in neural information processing systems 31 .

DI, Q., ZHAO, H., HE, J. and GU, Q. (2023). Pessimistic nonlinear least-squares value iteration for offline reinforcement learning. arXiv preprint arXiv:2310.01380 .

DU, S., KAKADE, S., LEE, J., LOVETT, S., MAHAJAN, G., SUN, W. and WANG, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning . PMLR.

DU, S. S., KAKADE, S. M., WANG, R. and YANG, L. F. (2019). Is a good representation sufficient for sample efficient reinforcement learning? In International Conference on Learning Representations .

FOSTER, D. J., KAKADE, S. M., QIAN, J. and RAKHLIN, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

GAO, M., XIE, T., DU, S. S. and YANG, L. F. (2021). A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 .

GENTILE, C., WANG, Z. and ZHANG, T. (2022). Achieving minimax rates in pool-based batch active learning. In International Conference on Machine Learning . PMLR.

HAN, Y., ZHOU, Z., ZHOU, Z., BLANCHET, J., GLYNN, P. W. and YE, Y. (2020). Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321 .

HE, J., ZHAO, H., ZHOU, D. and GU, Q. (2022). Nearly minimax optimal reinforcement learning for linear markov decision processes. arXiv preprint arXiv:2212.06132 .

HE, J., ZHOU, D. and GU, Q. (2021a). Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

HE, J., ZHOU, D. and GU, Q. (2021b). Nearly minimax optimal reinforcement learning for discounted mdps. Advances in Neural Information Processing Systems 34 22288-22300.

HU, P., CHEN, Y. and HUANG, L. (2022). Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

HUANG, J., CHEN, J., ZHAO, L., QIN, T., JIANG, N. and LIU, T.-Y. (2022). Towards deploymentefficient reinforcement learning: Lower bound and optimality. arXiv preprint arXiv:2202.06450

.

JIA, Z., YANG, L., SZEPESVARI, C. and WANG, M. (2020). Model-based reinforcement learning with value-targeted regression. In Learning for Dynamics and Control . PMLR.

JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning . PMLR.

JIN, C., ALLEN-ZHU, Z., BUBECK, S. and JORDAN, M. I. (2018). Is q-learning provably efficient? Advances in neural information processing systems 31 .

JIN, C., LIU, Q. and MIRYOOSEFI, S. (2021). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems 34 13406-13418.

JIN, C., YANG, Z., WANG, Z. and JORDAN, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory . PMLR.

KOBER, J., BAGNELL, J. A. and PETERS, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32 1238-1274.

KONG, D., SALAKHUTDINOV, R., WANG, R. and YANG, L. F. (2021). Online sub-sampling for reinforcement learning with general function approximation. arXiv preprint arXiv:2106.07203 .

LI, Y., WANG, Y., CHENG, Y. and YANG, L. (2023). Low-switching policy gradient with exploration via online sensitivity sampling. In International Conference on Machine Learning . PMLR.

LILLICRAP, T. P., HUNT, J. J., PRITZEL, A., HEESS, N., EREZ, T., TASSA, Y., SILVER, D. and WIERSTRA, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

MATSUSHIMA, T., FURUTA, H., MATSUO, Y., NACHUM, O. and GU, S. (2020). Deploymentefficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations .

MNIH, V., KAVUKCUOGLU, K., SILVER, D., RUSU, A. A., VENESS, J., BELLEMARE, M. G., GRAVES, A., RIEDMILLER, M., FIDJELAND, A. K., OSTROVSKI, G. ET AL. (2015). Human-level control through deep reinforcement learning. nature 518 529-533.

MODI, A., JIANG, N., TEWARI, A. and SINGH, S. (2020). Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics .

PUTERMAN, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons.

QIAO, D., YIN, M., MIN, M. and WANG, Y.-X. (2022). Sample-efficient reinforcement learning with loglog (t) switching cost. In International Conference on Machine Learning . PMLR.

QIAO, D., YIN, M. and WANG, Y.-X. (2023). Logarithmic switching cost in reinforcement learning beyond linear mdps. arXiv preprint arXiv:2302.12456 .

RUAN, Y., YANG, J. and ZHOU, Y. (2021). Linear bandits with limited adaptivity and learning distributional optimal design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing .

RUSSO, D. and VAN ROY, B. (2013). Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems 26 .

RUSSO, D. and VAN ROY, B. (2014). Learning to optimize via posterior sampling. Mathematics of Operations Research 39 1221-1243.

SIMCHOWITZ, M. and JAMIESON, K. G. (2019). Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems 32 .

SUN, W., JIANG, N., KRISHNAMURTHY, A., AGARWAL, A. and LANGFORD, J. (2019). Modelbased rl in contextual decision processes: Pac bounds and exponential improvements over modelfree approaches. In Conference on learning theory . PMLR.

VELEGKAS, G., YANG, Z. and KARBASI, A. (2022). Reinforcement learning with logarithmic regret and policy switches. Advances in Neural Information Processing Systems 35 36040-36053.

WANG, R., SALAKHUTDINOV, R. R. and YANG, L. (2020a). Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems 33 6123-6135.

WANG, T., ZHOU, D. and GU, Q. (2021). Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems 34 13524-13536.

WANG, Y., WANG, R., DU, S. S. and KRISHNAMURTHY, A. (2020b). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations .

XIONG, N., YANG, Z. and WANG, Z. (2023). A general framework for sequential decision-making under adaptivity constraints. arXiv preprint arXiv:2306.14468 .

YANG, L. and WANG, M. (2019). Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning . PMLR.

YANG, L. and WANG, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning . PMLR.

YE, C., XIONG, W., GU, Q. and ZHANG, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning . PMLR.

ZANETTE, A., BRANDFONBRENER, D., BRUNSKILL, E., PIROTTA, M. and LAZARIC, A. (2020a). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics .

ZANETTE, A. and BRUNSKILL, E. (2019). Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning . PMLR.

ZANETTE, A., LAZARIC, A., KOCHENDERFER, M. and BRUNSKILL, E. (2020b). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning . PMLR.

ZHANG, Z. and JI, X. (2019). Regret minimization for reinforcement learning by evaluating the optimal bias function. Advances in Neural Information Processing Systems 32 .

ZHANG, Z., JI, X. and DU, S. (2021). Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory . PMLR.

ZHANG, Z., JIANG, Y., ZHOU, Y. and JI, X. (2022). Near-optimal regret bounds for multi-batch reinforcement learning. Advances in Neural Information Processing Systems 35 24586-24596.

ZHANG, Z., ZHOU, Y. and JI, X. (2020). Almost optimal model-free reinforcement learningvia reference-advantage decomposition. Advances in Neural Information Processing Systems 33 15198-15207.

ZHAO, H., HE, J., ZHOU, D., ZHANG, T. and GU, Q. (2023). Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. arXiv preprint arXiv:2302.10371 .

ZHENG, G., ZHANG, F., ZHENG, Z., XIANG, Y., YUAN, N. J., XIE, X. and LI, Z. (2018). Drn: A deep reinforcement learning framework for news recommendation. In Proceedings of the 2018 world wide web conference .

ZHOU, D. and GU, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507 .

ZHOU, D., GU, Q. and SZEPESVARI, C. (2021a). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

ZHOU, D., HE, J. and GU, Q. (2021b). Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning . PMLR.

ZHOU, R., ZIHAN, Z. and DU, S. S. (2023). Sharp variance-dependent bounds in reinforcement learning: Best of both worlds in stochastic and deterministic environments. In International Conference on Machine Learning . PMLR.

ZOPH, B. and LE, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 .

A Additional Related Work

A.1 RL with Linear Function Approximation

In recent years, a substantial body of research has emerged to address the challenges of solving Markov Decision Processes (MDP) with linear function approximation, particularly to handle the vast state and action spaces (Jiang et al., 2017; Dann et al., 2018; Yang and Wang, 2019; Du et al., 2019; Sun et al., 2019; Jin et al., 2020; Wang et al., 2020b; Zanette et al., 2020a; Yang and Wang, 2020; Modi et al., 2020; Ayoub et al., 2020; Zhou et al., 2021a; He et al., 2021a; Zhou and Gu, 2022; He et al., 2022; Zhao et al., 2023). These works can be broadly categorized into two groups based on the linear structures applied to the underlying MDP. One commonly employed linear structure is known as the linear MDP (Jin et al., 2020), where the transition probability function P h and reward function r h are represented as linear functions with respect to a given feature mapping ϕ : S × A → R d . Under this assumption, the LSVI-UCB algorithm (Jin et al., 2020) has been shown to achieve a regret guarantee of ˜ O ( √ d 3 H 4 K ) . Subsequently, Zanette et al. (2020a) introduced the RLSVI algorithm, utilizing the Thompson sampling method, to attain a regret bound of ˜ O ( √ d 4 H 5 K ) . More recently, He et al. (2022) improved the regret guarantee to ˜ O ( √ d 2 H 3 K ) with the LSVI-UCB++ algorithm, aligning with the theoretical lower bound in Zhou et al. (2021a) up to logarithmic factors. Another line of research has focused on linear mixture MDPs (Modi et al., 2020; Yang and Wang, 2020; Jia et al., 2020; Ayoub et al., 2020; Zhou et al., 2021a), where the transition probability is expressed as a linear combination of basic models P 1 , P 2 , .., P d . For linear mixture MDPs, Jia et al. (2020) introduced the UCRL-VTR algorithm, achieving a regret guarantee of ˜ O ( √ d 2 H 4 K ) . Subsequently, Zhou et al. (2021a) enhanced this result to ˜ O ( √ d 2 H 3 K ) , reaching a nearly minimax optimal regret bound. Recently, several works focused on time-homogeneous linear mixture MDPs, and removed the dependency on the episode length (horizon-free) (Zhang and Ji, 2019; Zhou et al., 2021b; Zhao et al., 2023).

A.2 RL with General Function Approximation

Recent years have witnessed a flurry of progress on RL with nonlinear function classes. To explore the theoretical limits of RL algorithms, various complexity measures have been developed to characterize the hardness of RL instances such as Bellman rank (Jiang et al., 2017), Witness rank (Sun et al., 2019), eluder dimension (Russo and Van Roy, 2013), Bellman eluder dimension (Jin et al., 2021), Bilinear Classes (Du et al., 2021), Decision-Estimation Coefficient (Foster et al., 2021), Admissible Bellman Characterization (Chen et al., 2022), generalized eluder dimension (Agarwal et al., 2022). Among them, only Agarwal et al. (2022) yields a near-optimal regret guarantee when specialized to linear MDPs. In their paper, they proposed a new framework named generalized eluder dimension to handle the weighted objects in weighted regression, which can be seen as a variant of eluder dimension. In their proposed algorithm VO Q L, they adopted over-optimistic and over-pessimistic value functions in order to bound the variance of regression targets, making it possible to apply a weighted regression scheme in the model-free framework.

A.3 RL with Low Switching Cost

Most of the aforementioned approaches necessitate updating both the value function and the corresponding policy in each episode, a practice that proves to be inefficient when dealing with substantial datasets. To overcome this limitation, a widely adopted technique involves dividing the time sequence into several epochs and updating the policy only between different epochs. In the context of the linear bandit problem, Abbasi-Yadkori et al. (2011) introduced the rarely-switching OFUL algorithm, where the agent updates the policy only when the determinant of the covariance matrix doubles. This method enables the algorithm to achieve near-optimal regret of ˜ O ( √ K ) while maintaining policy updates to ˜ O ( d log K ) times. When the number of arms, denoted as |D| , is finite, Ruan et al. (2021) proposed an algorithm with regret bounded by ˜ O ( √ dK ) and a mere ˜ O ( d log d log K ) policy-updating times. In the realm of episodic reinforcement learning, Bai et al. (2019) and Zhang et al. (2021) delved into tabular Markov Decision Processes, introducing algorithms that achieve a regret of ˜ O ( √ T ) and ˜ O ( SA log K ) updating times. Wang et al. (2021) later extended these results to linear MDPs, unveiling the LSVI-UCB-RareSwitch algorithm. LSVI-UCB-RareSwitch delivers a regret bound

of ˜ O ( d 3 H 4 K ) with a policy switching frequency of ˜ O ( d log T ) , which matches the lower bound of the switching cost (Gao et al., 2021) up to logarithmic terms. Furthermore, if the policy is updated within fixed-length epochs (Han et al., 2020), the method is termed batch learning model, but this falls beyond the scope of our current work. In addition, with the help of stochastic policies, Zhang et al. (2022) porposed an algorithm with ˜ O ( √ K ) regret guarantee and only ˜ O ( H ) swithcing cost for learning tabular MDPs. Later, Huang et al. (2022) employed the stochastic policy in learning linear MDPs, which is able to find an ϵ -optimal policy with only ˜ O ( H ) switching cost.

B Additional Results

B.1 Lower Bound for the Switching Cost

As a complement, we prove a new lower bound on the switching cost for RL with linear MDPs. Note that linear MDPs lie in the class of MDPs studied in our paper with bounded generalized eluder dimension. In particular, the generalized eluder dimension of linear MDPs is ˜ O ( d ) .

Theorem B.1. For any algorithm Alg with expected switching cost less than dH/ (16 log K ) , there exists a hard-to-learn linear MDP, such that the expected regret of Alg is at least Ω( K ) .

Remark B.2. Theorem B.1 suggests that, to achieve a sublinear regret guarantee, an ˜ Ω( dH ) switching cost is inevitable. This lower bound does not violate the minimax upper bound of ˜ O ( H ) proved in Zhang et al. (2022); Huang et al. (2022), which additionally assume that the initial state s k 1 is either fixed or sampled from a fixed distribution. In contrast, our work and Gao et al. (2021) allow the initial state to be adaptively chosen by an adversarial environment, bringing more challenges to the learning of linear MDPs. When comparing our lower bound with the result in Gao et al. (2021), it is worth noting that their focus is solely on deterministic algorithms, and they suggest that an ˜ Ω( dH ) switching cost is necessary. As a comparison, our result holds for any algorithm with arbitrary policies including both deterministic and stochastic policies.

B.2 Connection Between D 2 F -Uncertainty and Eluder Dimension

Previous work by Agarwal et al. (2022) achieved the optimal regret bound ˜ O ( √ dim( F ) log N · HK ) , where dim( F ) is defined as the generalized eluder dimension as stated in Definition 2.4. However, the connection between generalized eluder dimension and eluder dimension proposed by Russo and Van Roy (2013) is still under-discussed 2 . Consequently, their results could not be directly compared with the results based on the classic eluder dimension measure (Wang et al., 2020a) or the more general Bellman eluder dimension (Jin et al., 2021).

In this section, we make a first attempt to establish a connection between generalized eluder dimension and eluder dimension in Theorem B.3.

Theorem B.3. For a function space G , α > 0 , let dim be defined in Definition 2.4. WLOG, we assume that for all g ∈ G and z ∈ Z , | g ( z ) | ≤ 1 . Then the following inequality between dim( F , Z , σ ) and dim E ( G , 1 / √ T ) holds for all Z := { z i } i ∈ [ T ] with z i ∈ Z and σ := { σ i } i ∈ [ T ] s.t. α ≤ σ i ≤ M ∀ i ∈ [ T ] :

According to Theorem B.3, the generalized eluder dimension is upper bounded by eluder dimension up to logarithmic terms. When the number of episodes K is sufficiently large, the leading term in our regret bound in Theorem 4.1 is ˜ O (√ dim E ( F ) log N · HK ) , where dim E ( F ) is the eluder dimension of the function class F .

C Proof of Theorem B.3

To start with, we first recall the concept of eluder dimension as follows.

Definition C.1 (Definition of eluder dimension, Russo and Van Roy 2013) . The eluder dimension of a function class G with domain Z is defined as follows:

A point z ∈ Z is ϵ -dependent on z 1 , z 2 , · · · , z k ∈ Z with respect to G , if for all g 1 , g 2 ∈ G such that ∑ k i =1 [ g 1 ( z i ) -g 2 ( z i )] 2 ≤ ϵ , it holds that | g 1 ( z ) -g 2 ( z ) | ≤ ϵ .

Further z is said to be ϵ -independent of z 1 , z 2 , · · · , z k with respect to G , if z is not dependent on z 1 , z 2 , · · · , z k .

The eluder dimension of G , denoted by dim E ( G , ϵ ) , is the length of the longest sequence of elements in Z such that every element is ϵ ′ -independent of its predecessors for some ϵ ′ ≥ ϵ .

With this definition, we can prove Theorem B.3.

Proof of Theorem B.3. Let I j ( 1 ≤ j ≤ ⌈ log 2 M/α ⌉ ) be the index set such that

Then we focus on the summation over I j for each j . For simplicity, we denote the subsequence { z i } i ∈I j by { x i } i ∈ [ |I j | ] . Then we have

∑ i ∈I j min ( 1 , 1 σ 2 i D 2 G ( z i ; z [ i -1] , σ [ i -1] ) ) ≤ ∑ i ∈ [ |I j | ] min ( 1 , 4 D 2 G ( x i ; x [ i -1] , 1 [ i -1] )) . To bound ∑ i ∈ [ |I j | ] min ( 1 , 4 D 2 G ( x i ; x [ i -1] , 1 [ i -1] )) , ∑ i ∈ [ |I j | ] min ( 1 , 4 D 2 G ( x i ; x [ i -1] , 1 [ i -1] )) ≤ 4 /λ + ∑ i ∈ [ |I j | ] 4 ∫ 1 1 /λT 1 { D 2 G ( x i ; x [ i -1] , 1 [ i -1] ) ≥ ρ } d ρ ≤ 4 /λ +4 ∫ 1 1 /λT ∑ i ∈ [ |I j | ] 1 { D 2 G ( x i ; x [ i -1] , 1 [ i -1] ) ≥ ρ } d ρ. (C.1)

Then we proceed by bounding the summation ∑ i ∈ [ |I j | ] 1 { D 2 G ( x i ; x [ i -1] , 1 [ i -1] ) ≥ ρ } for each ρ ≥ 1 / ( λT ) . For short, let d := dim E ( G , 1 / √ T ) . Essentially, it suffices to provide an upper bound of the cardinality of the subset J j := { i ∈ I j | D 2 G ( x i ; x [ i -1] , 1 [ i -1] ) ≥ ρ } .

For each i ∈ J j , since D 2 G ( x i ; x [ i -1] , 1 [ i -1] ) ≥ ρ , there exists g 1 , g 2 in G such that

Here ( g 1 ( x i ) -g 2 ( x i )) 2 ≥ λρ ≥ 1 /T . Denote such value of ( g 1 ( x i ) -g 2 ( x i )) 2 by L ( x i ) . Then we consider split cJ j into ⌈ log 2 T ⌉ layers such that in each layer J k j ( k ∈ [ ⌈ log 2 T ⌉ ]) , we have 1 /T ≤ ξ ≤ L ( x i ) ≤ 2 ξ for some ξ .

Denote the cardinality of J k j by A and the subsequence { x i } i ∈I k j by { y i } i ∈ [ A ] . For the elements in { y } , we dynamically maintain ⌊ A/d ⌋ queues of elements. We enumerate through { y } in its original order and put the current element y i into the queue Q such that y i is ξ -independent of the current elements in Q . By the Pigeonhole principle and the definition of eluder dimension d , we can find an element y i in { y } such that y i is ξ -dependent of all the ⌊ A/d ⌋ queues before i .

Then by the definition of L ( y i ) and ξ , we can choose g 1 , g 2 such that

By the ξ -dependencies, we have ∑ t ∈ [ i -1] ( g 1 ( y t ) -g 2 ( y t )) 2 ≥ ⌊ A/d ⌋ · ξ . At the same time, ∑ t ∈ [ i -1] ( g 1 ( y t ) -g 2 ( y t )) 2 ≤ 4 ξ/ρ due to (C.2). Thus, we deduce that A = |J k j | ≤ O ( d/ρ ) for all k ∈ [ ⌈ log 2 T ⌉ ] . Substituting it into (C.1), we have

Here j is chosen arbitrarily. Hence, the generalized eluder dimension can be further bounded as follows:

In the following part, we will also discuss the issue in Agarwal et al. (2022) about the relationship between the standard eluder dimension dim E and the generalized eluder dimension dim . In detail, Agarwal et al. (2022) proposed the concept of Generalized Eluder dimension and made the following claim:

Lemma C.2 (Remark 4, Agarwal et al. 2022) . If we set the weight σ = 1 , then the Generalized Eluder dimension dim = sup Z , σ : | Z | = T dim( F , Z , 1 ) satisfied

where dim E denotes the standard Eluder dimension proposed in Russo and Van Roy (2013). In the proof of Remark 4, Agarwal et al. (2022) claims that given the standard Eluder dimension dim E ( F , √ λ ) = n , there are at most n different (sorted) indices { t 1 , t 2 , .. } such that D 2 F ( z t i ; z t i -1] , 1 ) ≥ ϵ 2 /λ . However, according to the definition of D 2 F -uncertainty, we only have

However, z t i is ϵ -dependence with z 1 , .., z t i -1 is only a sufficient condition for the uncertainty D 2 F ( z ; z [ t -1] , 1 ) rather than necessary condition. In cases where both ( f 1 ( z ) -f 2 ( z )) 2 ≥ ϵ 2 and ∑ s ∈ [ t -1] ( f 1 ( z s ) -f 2 ( z s )) 2 ≥ ϵ 2 hold, the uncertainty D 2 F ( z ; z [ t -1] , 1 ) may exceed ϵ 2 /λ , yet it will not be counted within the longest ϵ -independent sequence for the standard Eluder dimension.

D Proof of Theorem 4.1

In this section, we provide the proof of Theorem 4.1.

D.1 High Probability Events

In this subsection, we define the following high-probability events:

where

Then we define the following bellman-type concentration events for p f k,h , q f k,h , which implies a tighter confidence set due to carefully designed variance estimators and an inverse-variance weighted regression scheme for the general function classes.

where

We also define the following events which are later applied to prove the concentration of p f k,h and q f k,h by induction.

where p 1 is the shorthand for the following product of indicator functions i,h p 1 i,h := 1 ( [ V h V ∗ h +1 ]( s i h , a i h ) ≤ ¯ σ 2 i,h ) · 1 ( [ T h V i,h +1 -T V ∗ h +1 ] ≤ (log N F ( ϵ ) + log N ϵ ( K )) -1 ¯ σ 2 i,h ) · 1 ( V i,h +1 ( s ) ≥ V ∗ h +1 ( s ) ∀ s ∈ S ) . (D.6)

where q 1 is the shorthand for the following product of indicator functions i,h q 1 i,h := 1 ( [ V h V ∗ h +1 ]( s i h , a i h ) ≤ ¯ σ 2 i,h ) · 1 ( [ T V ∗ h +1 -T h q V i,h +1 ] ≤ (log N F ( ϵ ) + log N ϵ ( K )) -1 ¯ σ 2 i,h ) · 1 ( q V i,h +1 ( s ) ≤ V ∗ h +1 ( s ) ∀ s ∈ S ) . (D.8)

The following Lemmas suggest that previous events hold with high probability.

Lemma D.1. Let ˜ f k,h be defined as in line 8 of Algorithm 1, we have E ˜ f := ⋂ k ≥ 1 ,h ∈ [ H ] E ˜ f k,h holds with probability at least 1 -δ , where E ˜ f k,h is defined in (D.1).

Lemma D.2. Let p f k,h be defined as in line 6 of Algorithm 1, we have E p f := ⋂ k ≥ 1 ,h ∈ [ H ] E p f k,h holds with probability at least 1 -δ , where E p f k,h is defined in (D.2).

Lemma D.3. Let q f k,h be defined as in line 7 of Algorithm 1, we have E q f := ⋂ k ≥ 1 ,h ∈ [ H ] E q f k,h holds with probability at least 1 -δ , where E q f k,h is defined in (D.3).

Lemma D.4. Let p f k,h be defined as in line 6 of Algorithm 1, we have ¯ E p f := ⋂ k ≥ 1 ,h ∈ [ H ] ¯ E p f k,h holds with probability at least 1 -2 δ , where E p f k,h is defined in (D.5).

Lemma D.5. Let q f k,h be defined as in line · of Algorithm 1, we have ¯ E q f holds with probability at least 1 -2 δ .

D.2 Optimism and Pessimism

Based on the high probability events, we have the following lemmas for the function p f k,h ( s, a ) and q f k,h ( s, a )

Lemma D.6. On the events E p f h and E q f h , for each episode k ∈ [ K ] , we have

where z = ( s, a ) and z [ k -1] ,h = { z 1 ,h , z 2 ,h , .., z k -1 ,h } .

Lemma D.7. On the events E p f h +1 and E q f h +1 , for each stage h ≤ h ′ ≤ H and episode k ∈ [ K ] , we have Q k,h ( s, a ) ≥ Q ∗ h ( s, a ) ≥ q Q k,h ( s, a ) . Furthermore, for the value functions V k,h ( s ) and q V k,h ( s ) , we have V k,h ( s ) ≥ V ∗ h ( s ) ≥ q V k,h ( s ) .

D.3 Monotonic Variance Estimator

In this subsection, we introduce the following lemmas to construct the monotonic variance estimator.

Lemma D.8. On the events E ˜ f , E p f and E q f , for each episode k ∈ [ K ] and stage h ∈ [ H ] , we have

where z = ( s, a ) and z [ k -1] ,h = { z 1 ,h , z 2 ,h , .., z k -1 ,h } .

Lemma D.9. On the events E ˜ f , E p f , E q f , E p f h +1 , E q f h +1 , for each episode k ∈ [ K ] , the empirical variance [ ¯ V h V k,h +1 ]( s k h , a k h ) satisfies the following inequalities:

Lemma D.10. On the events E p f , E q f , E p f h +1 , E q f h +1 , for each episode k ∈ [ K ] and i > k , we have

Based on previous lemmas, we can construct an optimistic estimator σ k,h for the transition variance. Under this situation, the weighted regression have the following guarantee.

Lemma D.11. If the events E ˜ f , E p f , E q f , ¯ E p f and ¯ E q f hold, then events E p f = E p f 1 and E q f = E q f 1 hold.

D.4 Proof of Regret Bound

In the subsection, we first define the following high proSbability events to control the stochastic noise from the transition process:

According to Freedman inequality (Lemma H.5), we directly have the following results.

Lemma D.12. Events E 1 and E 2 hold with probability at least 1 -2 δ .

Proof. Fix an arbitrary h ∈ [ H ] . Applying Lemma H.5, we have the following inequality:

Applying a union bound for (D.9) across all h ∈ [ H ] , k ≥ 0 , we have Event E 1 holds with probability at least 1 -2 δ .

Similarly, we also have the corresponding high-probability bound for E 2 .

Next, we need the following lemma to control the summation of confidence radius.

Lemma D.13. For any parameters β ≥ 1 and stage h ∈ [ H ] , the summation of confidence radius over episode k ∈ [ K ] is upper bounded by

where z = ( s, a ) and z [ k -1] ,h = { z 1 ,h , z 2 ,h , .., z k -1 ,h } .

Then, we can decompose the total regret in the first K episode to the summation of variance ∑ K k =1 ∑ H h =1 σ 2 k,h as the following lemma.

Lemma D.14. On the events E p f = E p f 1 , E q f = E q f 1 and E 1 , for each stage h ∈ [ H ] , the regret in the first K episodes can be decomposed and controlled as:

and for all stage h ∈ [ H ] , we further have

In addition, the gap between the optimistic value function V k,h ( s ) and pessimistic value function q V k,h ( s ) can be upper bounded by the following lemma.

Lemma D.15. On the events E p f = E p f 1 , E q f = E q f 1 and E 2 , for each stage h ∈ [ H ] , the regret in the first K episodes can be decomposed and controlled as:

and for all stage h ∈ [ H ] , we further have

We define the following high probability event E 3 to control the summation of variance.

According to Jin et al. (2018) (Lemma C.5) 3 , with probability at least 1 -δ . Condition on this event, the summation of variance ∑ K k =1 ∑ H h =1 σ 2 k,h can be upper bounded by the following lemma.

Lemma D.16. On the events E 1 , E 2 , E 3 , E p f = E p f 1 and E q f = E q f 1 , the total estimated variance is upper bounded by:

With all previous lemmas, we can prove Theorem 4.1

Proof of Theorem 4.1. The low switching cost result is given by Lemma G.1.

After taking a union bound, the high probability events E 1 , E 2 , E 3 , E ˜ f , E p f , E q f , ¯ E p f and ¯ E q f hold with probability at least 1 -10 δ . Conditioned on these events, the regret is upper bounded by

where the first inequality holds due to Lemma D.7, the second inequality holds due to Lemma D.14, the third inequality follows from Lemma D.7 and Var[ X ] ≤ E [ X ] · M if random variable X is in [0 , M ] , the fourth inequality holds due to (F.41) and 2 ab ≤ a 2 + b 2 , the last inequality holds due to Lemma D.16. Thus, we complete the proof of Theorem 4.1.

We can reorganize (D.11) into the following upper bound for regret,

where we denote the covering number of bonus function class N ( B , 1 2 KHL p β K ) by N b , the covering number of function class F by N and the dimension dim α,K ( F ) by dim( F ) . Since E 3 occurs with high probability according to Jin et al. (2018) (Lemma C.5), we have Var K = ˜ O ( K ) , which matches the worst-case minimax optimal regret bound of MDPs with general function approximation.

D.5 Proof Sketch of Regret Bound

In this subsection, we provide a proof sketch on the regret bound and show how our policy-switching strategy works.

First, we introduce the following lemma which illustrates the stability of the bonus function after using the uncertainty-based policy-switching condition.

Lemma D.17 (Restatement of Lemma G.2) . If the policy is not updated at episode k , the uncertainty of all state-action pair z = ( s, a ) ∈ S ×A and stage h ∈ [ H ] satisfies the following stability property:

With Lemma G.2, we can then convert D 2 F h ( z ; z [ k last -1] ,h , ¯ σ [ k last -1] ,h in the bonus term into D 2 F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) with a cost of an additional 1 + χ constant.

As a direct consequence of Lemma G.2, We have the following lemma, which controls the gap between optimistic value functions and real value functions resulting from the exploration bonuses.

Lemma D.18 (Restatement of Lemma D.14) . On the events E p f = E p f 1 , E q f = E q f 1 and E 1 , for each stage h ∈ [ H ] , the regret in the first K episodes can be decomposed and controlled as:

and for all stage h ∈ [ H ] , we further have

As a need to bound the sum of bonuses corresponding to a weighted regression target, we derive the following results on the sum of inverse weights.

Lemma D.19 (Informal version of Lemma D.16) . With high probability, the total estimated variance is upper bounded by:

Proof sketch of Theorem 4.1. As the result of optimism, we have shown that with high probability, for all k, h ∈ [ K ] × [ H ] , V k,h is an upper bound for V ∗ h . Hence, we further have,

To further proceed, we apply Lemma D.14 and obtain

where the second term can be bounded by applying Lemma D.14 repeatedly and the third term can be controlled by Lemma D.16.

Then after substituting the value of p beta K , γ and α , we conclude that with high probability,

Regret( K ) = ˜ O ( √ dim( F ) log N · H Var K ) + ˜ O ( H 2 . 5 dim 2 ( F ) √ log N log( N · N b ) · √ H log N +dim( F ) log( N · N b ) ) . Please refer to Subsection D.4 for the detailed calculation of this part.

E Proof of Theorem B.1

In this section, we provide the proof of Theorem B.1. To prove the lower bound, we create a series of hard-to-learn MDPs as follows. Each hard-to-learn MDP comprises d/ 4 distinct sub-MDPs denoted as M 1 , .., M d/ 4 . Each sub-MDP M i is characterized by two distinct states, initial state s i, 0 and absorbing state s i, 1 , and shares the same action set A = { a 0 , a 1 } . Since the state and action spaces are finite, these tabular MDPs can always be represented as linear MDPs with dimension |S| × |A| = d .

To generate each sub-MDP M i , for all stage h ∈ [ H ] , a special action a i,h is uniformly randomly selected from the action set { a 0 , a 1 } . Given the current state s i, 0 , the agent transitions to the state s i, 0 if it takes the special action a i,h . Otherwise, the agent transitions to the absorbing state s i, 1 and remains in that state in subsequent stages. The agent will receive the reward 1 if it takes the special action a i,H at the state s i, 0 during the last stage H . Otherwise, the agent always receives reward 0 . In this scenario, for sub-MDP M i , the optimal policy entails following the special action sequence ( a i, 1 , a i, 2 , ..., a i,H ) to achieve a total reward of 1 . In contrast, any other action sequence fails to yield any reward.

Now, we partition the K episodes to d/ 4 different distinct epochs. For each epoch (ranging from episodes 4( i -1) K/d +1 to episode 4 iK/d ), we initialize the state as s i, 0 and exclusively focus on the sub-MDP M i . The regret in each epoch can be lower bounded separately as follows:

Lemma E.1. For each epoch i ∈ [ d/ 4] and any algorithm Alg capable of deploying arbitrary policies, if the expected switching cost in epoch i is less than H/ (2 log K ) , the expected regret of Alg in the i -th epoch is at least Ω( K/d ) .

Proof of Lemma E.1. Given that each sub-MDP is independently generated, policy updates before epoch i only offer information for the sub-MDPs M 1 , ..., M i -1 and do not provide any information for the current epoch i . In this scenario, there is no distinction between epochs and for simplicity, we only focus on the first epoch, encompassing episodes 1 to 4 K/d .

Now, let k 0 = 0 and we denote K = { k 1 , k 2 , ... } as the set of episodes where the algorithm Alg updates the policy. If Alg does not update the policy i times, we set k i = 4 K/d +1 . For simplicity, we set C = 2log K and for each i ≤ H/C , we define the events E as the algorithm Alg has not reached the state s 1 , 0 at the stage iC before the episode k i .

Conditioned on the events E 1 , ..., E i -1 , the algorithm Alg does not gather any information about the special action a 1 ,h for stage h ≥ ( i -1) C +1 . In this scenario, the special actions can still be considered as uniformly randomly selected from the action set a 0 , a 1 . For each episode between k i -1 and k i , the probability that a policy π arrives at state s 1 , 0 at stage iC is upper-bounded by:

where the first inequality holds due to π ( s 1 , 0 ) = a 1 ,h ) ≤ 1 and the second equation holds due to the random generation process of the special actions. Notice that there are at most K episodes between the k i -1 and k i and after applying an union bound for all episodes, we have

Furthermore, we have

Notice that the agent cannot receive any reward unless it has reached the state s i, 0 at the last stage H . Therefore, the expected regret for the algorithm Alg for the sub-MDP M can be bounded by the switching cost δ :

where the first inequality holds due to the fact that the agent receives no reward before k H/C conditioned on the event E H/C , the second inequality holds due to (E.1) with k H/C -1 ≤ K , the third inequality holds due to the definition of k H/C and the last inequality holds due to E [ 1 ( x ≥ a )] ≤ E [ x ] /a for any non-negative random variable x . According to the result in (E.1), if the expected switching E [ δ ] ≤ H/ (2 C ) , the expected regret is at least Ω( K ) , when K is large enough compared to H .

With Lemma B.1, we can prove Theorem B.1.

Proof of Theorem B.1. For these constructed hard-to-learn MDPs and any given algorithm Alg , we denote the expected switching cost for sub-MDP M i as δ i .

According to Lemma E.1, we have

where the first inequality holds due to lemma B.1 and last inequality holds due to E [ 1 ( x ≥ a )] ≤ E [ x ] /a for any non-negative random variable x . Taking a summation of (E.3) for all sub-Mdps, we have

where δ is the total expected switching cost. Therefore, for any algorithm with total expected switching cost less than dH/ (16 log K ) , the expected is lower bounded by

Thus, we finish the proof of Theorem B.1.

F Proof of Lemmas in Appendix D

In this section, we provide the detailed proof of lemmas in Appendix D.

F.1 Proof of High Probability Events

F.1.1 Proof of Lemma D.1

Proof of Lemma D.1. We first prove that for an arbitrarily chosen h ∈ [ H ] , E ˜ f k,h holds with probability at least 1 -δ/H for all k .

For simplicity, in this proof, we denote T 2 h V k,h +1 ( s i h , a i h ) as ˜ f ∗ k ( s i h , a i h ) where ˜ f ∗ k ∈ F h exists due to our assumption. For any function V : S → [0 , 1] , let ˜ η k h ( V ) = ( r k h + V ( s k h +1 ) ) 2 -E s ′ ∼ s k h ,a k h [ ( r ( s k h , a k h , s ′ ) + V ( s ′ ) ) 2 ] .

By simple calculation, for all f ∈ F h , we have

Due to the definition of ˜ f k,h , we have

Then we give a high probability bound for -I ( f, f ∗ , V ) through the following calculation.

Applying Lemma H.4, for fixed f , f and V , with probability at least 1 -δ ,

By the definition of V in line · of Algorithm 1, V k,h +1 lies in the optimistic value function class V k defined in (G.4). Applying a union bound for all the value functions V c in the corresponding ϵ -net V c , we have

holds for all k with probability at least 1 -δ .

For all V such that ∥ V -V c ∥ ∞ ≤ ϵ , we have | η i h ( V ) -η i h ( V c ) | ≤ 4 ϵ . Therefore, with probability 1 -δ , the following bound holds for I ( f, ¯ f, V k,h +1 ) :

To further bound I ( ˜ f k,h , ˜ f ∗ k , V k,h +1 ) in (F.1), we apply an ϵ -covering argument on F h and show that with probability at least 1 -δ ,

for probability at least 1 -δ .

Substituting (F.2) into (F.1) and rearranging the terms,

for all k with probability at least 1 -δ/H .

Finally, we apply a union bound over all h ∈ [ H ] to conclude that E ˜ f holds with probability at least 1 -δ .

F.1.2 Proof of Lemmas D.2 and D.3

Proof of Lemma D.2. Similar to the proof of Lemma D.1, we first prove that for an arbitrarily chosen h ∈ [ H ] , E p f k,h holds with probability at least 1 -δ/H for all k .

In this proof, we denote T h V k,h +1 ( s i h , a i h ) as p f ∗ k ( s i h , a i h ) where p f ∗ k ∈ F h exists due to our assumption. For any function V : S → [0 , 1] , let p η k h ( V ) = ( r k h + V ( s k h +1 ) ) -E s ′ ∼ s k h ,a k h [ r h ( s k h , a k h , s ′ ) + V ( s ′ ) ] . For all f ∈ F h , we have

By the definition of p f k,h , we have

Then we give a high probability bound for -I ( p f k,h , p f ∗ k , V k,h +1 ) through the following calculation. Applying Lemma H.4, for fixed f , ¯ f and V , with probability at least 1 -δ ,

By the definition of V in line · of Algorithm 1, V k,h +1 lies in the optimistic value function class V k defined in (G.4). Applying a union bound for all the value functions V c in the corresponding ϵ -net V c , we have

holds for all k with probability at least 1 -δ .

For all V such that ∥ V -V c ∥ ∞ ≤ ϵ , we have | η i h ( V ) -η i h ( V c ) | ≤ 4 ϵ . Therefore, with probability 1 -δ , the following bound holds for I ( f, ¯ f, V k,h +1 ) :

To further bound I ( p f k,h , p f ∗ k , V k,h +1 ) in (F.3), we apply an ϵ -covering argument on F h and show that with probability at least 1 -δ ,

Substituting (F.4) into (F.3) and rearranging the terms, we have

for all k with probability at least 1 -δ/H . Then we can complete the proof by using a union bound over all h ∈ [ H ] .

Proof of Lemma D.3. The proof is almost identical to the proof of Lemma D.2.

F.1.3 Proof of Lemmas D.4 and D.5

Proof of Lemma D.4. Similar to the proof of Lemma D.1, we first prove that for an arbitrary h ∈ [ H ] , ¯ E p f k,h holds with probability at least 1 -δ/H for all k .

In this proof, we denote T h V k,h +1 ( s i h , a i h ) as p f ∗ k ( s i h , a i h ) where p f ∗ k ∈ F exists due to our assumption. For any function V : S → [0 , 1] , let p η k h ( V ) = ( r k h + V ( s k h +1 ) ) -E s ′ ∼ s k h ,a k h [ r h ( s k h , a k h , s ′ ) + V ( s ′ ) ] .

For all f ∈ F h , we have

Due to the definition of p f k,h , we have

Then it suffices to bound the value of I ( f, ¯ f, V ) for all f, ¯ f ∈ F .

Unlike the proof for Lemma D.1, we decompose I ( f, f, V ) into two parts:

k,h +1 ¯ k,h +1

Then we bound the two terms separately.

For the first term, we first check the following conditions before applying Lemma H.2, which is a variant of Freedman inequality. For fixed f and ¯ f , we have

since s i h +1 is sampled from P h ( ·| s i h , a i h ) .

Next, we need to derive a bound for the maximum absolute value of each 'weighted' transition noise:

where the second inequality follows from the definition of D F h in Definition 2.4, the last inequality holds since ¯ σ i,h ≥ γ · D 1 / 2 F h ( z i,h ; z [ i -1] ,h , ¯ σ [ i -1] ,h ) according to line 19 in Algorithm 1. From the definition of p 1 i,h in (D.6) we directly obtain the following upper bound of the variance:

Applying Lemma H.2 with V = 4 L 2 k/α 2 , M = 2 L/α 2 and v = m = 1 , for fixed f , ¯ f , k , with probability at least 1 -δ/ (2 k 2 · N F ( ϵ ) · H ) ,

Using a union bound across all f, ¯ f ∈ C ( F h , ϵ ) and k ≥ 1 , with probability at least 1 -δ/H ,

holds for all f, ¯ f ∈ C ( F h , ϵ ) and k . Due to the definition of ϵ -net, we deduce that for p f k,h and p f ∗ k ,

For the second term in (F.6), the following inequality holds for all V ∈ V k,h +1 ,

where the calculation is similar to that in (F.7).

We denote the sum of variance by Var( V -V ∗ h +1 ) as follows for simplicity:

For V k,h +1 , we have

where the second inequality holds due to the definition of p 1 i,h in (D.6).

With a similar argument as shown in (F.8) ∼ (F.12), we have with probability at least 1 -δ/ (2 k 2 · N F ( ϵ ) · N ϵ ( k -1) · H ) , for a fixed f , ¯ f , k and V , (applying Lemma H.2, with V = k · L 2 /α 4 and M = 2 L/α 2 , v = (log N F ( ϵ ) + log N ϵ ( K )) -1 / 2 , m = v 2 . )

where

Using a union bound over all ( f, ¯ f, V ) ∈ C ( F h , ϵ ) ×C ( F h , ϵ ) ×V c k,h +1 and all k ≥ 1 , we have the inequality above holds for all such f, ¯ f, V, k with probability at least 1 -δ/H .

There exists a V c k,h +1 in the ϵ -net such that ∥ V k,h +1 -V c k,h +1 ∥ ∞ ≤ ϵ . Then we have

for all k with at least probability 1 -δ/H .

Substituting (F.15) and (F.12) into (F.5), we can conclude that

Proof of Lemma D.5. The proof is almost identical to the proof of Lemma D.4.

F.2 Proof of Optimism and Pessimism

F.2.1 Proof of Lemma D.6

Proof of Lemma D.6. According to the definition of D 2 F function, we have

where the first inequality holds due the definition of D 2 F function with the Assumption 2.2 and the second inequality holds due to the events E p f h . Thus, we have

With a similar argument, for the pessimistic value function q f k,h , we have

where the first inequality holds due to the definition of D 2 F function with the Assumption 2.2 and the second inequality holds due to the events E q f h . In addition, we have,

Thus, we complete the proof of Lemma D.6.

F.2.2 Proof of Lemma D.7

Proof of Lemma D.7. We use induction to prove the optimistic and pessimistic property. First, we study the basic case with the last stage H + 1 . In this situation, Q k,H +1 ( s, a ) = Q ∗ h ( s, a ) = q Q k,h ( s, a ) = 0 and V k,h ( s ) = V ∗ h ( s ) = q V k,h ( s ) = 0 hold for all state s ∈ S and action a ∈ A . Therefore, Lemma D.7 holds for the basic case (stage H +1 ).

Second, if Lemma D.7 holds for stage h +1 , then we focus on the stage h . Notice that the event ˜ E h directly implies the event ˜ E h +1 . Therefore, according to the induction assumption, the following inequality holds for all state s ∈ S and episode k ∈ [ K ] .

Thus, for all episode k ∈ [ K ] and state-action pair ( s, a ) ∈ S × A , we have

where the first inequality holds due to Lemma D.6, the second inequality holds due to the definition of the exploration bonus b k,h and the last inequality holds due to the (F.16). Therefore, the optimal value function Q ∗ h ( s, a ) is upper bounded by

where the first inequality holds due to (F.17) with the fact that Q ∗ h ( s, a ) ≤ 1 and the second inequality holds due to the update rule of value function Q k,h .

With a similar argument, for the pessimistic estimator q f k,h , we have

where the first inequality holds due to Lemma D.6, the second inequality holds due to the definition of the exploration bonus b k,h and the last inequality holds due to the (F.19). Therefore, the optimal value function Q ∗ h ( s, a ) is lower bounded by

where the first inequality holds due to (F.19) with the fact that Q ∗ h ( s, a ) ≥ 0 and the second inequality holds due to the update rule of value function q Q k,h .

Furthermore, for the value functions V k,h and q V k,h , we have

where the first inequality holds due to (F.18) and the second inequality holds due to (F.20). Thus, by induction, we complete the proof of Lemma D.7.

F.3 Proof of Monotonic Variance Estimator

F.3.1 Proof of Lemma D.8

Proof of Lemma D.8. According to the definition of D 2 F function, we have

where the first inequality holds due the definition of D 2 F function with the Assumption 2.2 and the second inequality holds due to the events E p f . Thus, we have

For the pessimistic value function q f k,h , we have

(

q

k,h

f

(

D

β

s, a

2

F

h

)

(

z

2

k

×

;

D

h

-T

z

[

k

2

F

h

q

-

(

k,h

V

1]

z

;

,h

,

(

¯

σ

z

[

k

s, a

[

-

)

k

)

1]

-

1]

,h

,

2

,h

¯

σ

)

[

k

×

-

1]

,h

(

λ

)

,

where the first inequality holds due the definition of D 2 F function with the Assumption 2.2 and the second inequality holds due to the events E q f . In addition, we have

With a similar argument, for the second-order estimator ˜ f k,h , we have

where the first inequality holds due the definition of D 2 F function with the Assumption 2.2 and the second inequality holds due to the events E ˜ f . Therefore, we have

Now, we complete the proof of Lemma D.8.

+

k

1

-

∑

i

=1

(¯

σ

1

i,h

)

2

(

q

f

k,h

(

s

i

h

i

h

, a

)

-T

h

q

V

k,h

+1

(

s

i

h

i

h

, a

)

)

2

)

F.3.2 Proof of Lemma D.9

Proof of Lemma D.9. First, according to Lemma D.8, we have

where the first inequality holds due to | a + b | ≤ | a | + | b | , the second inequality holds due to | a 2 -b 2 | = | a -b | · | a + b | ≤ | a -b | · 2 max( | a | , | b | ) and the last inequalirtnholds due to Lemma D.8.

For the difference between variances [ V h V k,h +1 ]( s k h , a k h ) and [ V h V ∗ h +1 ]( s k h , a k h ) , it can be upper bounded by

where the first inequality holds due to | a + b | ≤ | a | + | b | , the second inequality holds due to 1 ≥ V k,h +1 ( · ) ≥ V ∗ h +1 ( · ) ≥ 0 (Lemma D.7), the third inequality holds due to V ∗ h +1 ( · ) ≥ q V k,h +1 ( · ) (Lemma D.7) and the last inequality holds due to Lemma D.8. Combining the results in (F.21) and (F.22) with the fact that 0 ≤ V k,h +1 ( · ) , V ∗ h +1 ( · ) ≤ 1 , we have

Thus, we complete the proof of Lemma D.9.

F.3.3 Proof of Lemma D.10

Proof of Lemma D.10. On the events E p f and E p f h +1 , we have

where the first equation holds since the reward function is deterministic, the second inequality holds due to Lemma D.7 with the fact that 0 ≤ V i,h +1 ( s ) , V ∗ h +1 ( s ) ≤ 1 and the third inequality holds due to Lemma D.7 and the last inequality holds due to the fact that V k,h +1 ≥ V i,h +1 .

With a similar argument, on the events E q f and E q f h +1 , we have

where the first inequality holds since the reward function is deterministic, the second and third inequality holds due to Lemma D.7 with the fact that 0 ≤ q V i,h +1 ( s ) , V ∗ h +1 ( s ) ≤ H , the last inequality the fact q V k,h +1 ( s ) ≤ q V i,h +1 ( s ) . For both variances [ V h ( V i,h +1 -V ∗ h +1 ) ] ( s k h , a k h ) and [ V h ( V ∗ h +1 -q V i,h +1 ) ] ( s k h , a k h ) , they are upper bounded by

where the first inequality holds due to Lemma D.8 and the second inequality holds due to the definition of F k,h . Substituting the result in (F.25) into (F.23), (F.24) and combining the fact that [ V h ( V i,h +1 -V ∗ h +1 ) ] ( s k h , a k h ) , [ V h ( V ∗ h +1 -q V i,h +1 ) ] ( s k h , a k h ) ≤ 1 , we finish the proof of Lemma D.10.

F.4 Proof of Lemmas in Section D.4

F.4.1 Proof of Lemma D.11

Proof of Lemma D.11. We use induction to shows that the conclusion in Lemma D.7 and events E p f h , E q f H hold for each stage h ∈ [ H ] . First, for the basic situation (stage H +1 ), Q k,H +1 ( s, a ) = Q ∗ h ( s, a ) = q Q k,h ( s, a ) = 0 and V k,h ( s ) = V ∗ h ( s ) = q V k,h ( s ) = 0 hold for all state s ∈ S and action a ∈ A . Therefore, Lemma D.7 holds for the basic case (stage H +1 )

Second, if Lemma D.7 holds for stage h +1 , then we focus on the stage h . According to Lemmas D.10 and Lemma D.9, we have the following inequalities:

where the first inequality holds due to Lemma D.9, the second and third inequality holds due to Lemma D.10. Thus, the indicator function in events ¯ E p f h and ¯ E q f h hold, which implies events E p f h , E q f H hold. Furthermore, when events E p f h , E q f H hold, then Lemma D.7 holds for stage h . Thus, we complete the proof of Lemma D.11 by induction.

F.4.2 Proof of Lemma D.13

Proof of Lemma D.13. For each stage h , we divide the episodes { 1 , 2 , .., K } to the following sets:

The number of episode in set I 1 is upper bounded by

where the equation holds due to D 2 F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) / ¯ σ 2 k,h ≥ 1 and the inequality holds due to the definition of Generalized Eluder dimension. Thus, for set I 1 , the summation of confidence radius is upper bounded by

For set I 2 , the summation of confidence radius is upper bounded by

where the second inequality holds due to Cauchy-Schwartz inequality with σ k,h = ¯ σ k,h and the last inequality holds due to the definition of Generalized Eluder dimension with the fact that D F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) / ¯ σ k,h < 1 .

With a similar argument, the summation of confidence radius over set I 3 is upper bounded by

where the second inequality holds due to Cauchy-Schwartz inequality with ¯ σ k,h = α and the last inequality holds due to the definition of Generalized Eluder dimension with the fact that D F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) / ¯ σ k,h < 1 .

Finally, the summation of confidence radius over set I 4 is upper bounded by With a similar argument, the summation of confidence radius over set I 3 is upper bounded by

where the first equation holds due to ¯ σ k,h = γ × D 1 / 2 F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) and the last inequality holds due to the definition of Generalized Eluder dimension with D F h ( z ; z [ k -1] ,h , ¯ σ [ k -1] ,h ) / ¯ σ k,h < 1 .

Combining the results in (F.26), (F.27), (F.28) and (F.29), we have

Thus, we complete the proof of Lemma D.13.

F.4.3 Proof of Lemma D.14

Proof of Lemma D.14. First, for each stage h ∈ [ H ] and episode k ∈ [ K ] , the gap between V k,h ( s k h ) and V π k h ( s k h ) can be decomposed as:

where the first inequality holds due to the definition of value function Q k,h ( s k h , a k h ) , the second inequality holds due to Lemma D.6, the third inequality holds due to b k last ,h ( s k h , a k h ) ≤ C · D F h ( z ; z [ k last -1] ,h , ¯ σ [ k last -1] ,h ) and the last inequality holds due to Lemma G.2. Taking a summation of (F.30) over all episode k ∈ [ K ] and stage h ′ ≥ h , we have

where the first inequality holds due to (F.30), the second inequality holds due to event E 1 , the third inequality holds due to Lemma D.13, the fourth inequality holds due to Cauchy-Schwartz inequality and the last inequality holds due to ∑ H h ′ = h dim α,K ( F h ′ ) ≤ ∑ H h ′ =1 dim α,K ( F h ′ ) = H dim α,K ( F ) . Furthermore, taking a summation of (F.31), we have

where the first inequality holds due to event E 1 and the second inequality holds due to (F.31). Thus, we complete the proof of Lemma D.14.

F.4.4 Proof of Lemma D.15

Proof of Lemma D.15. Similar to the proof of Lemma D.14, for each stage h ∈ [ H ] and episode k ∈ [ K ] , the gap between V k,h ( s k h ) and q V k,h ( s k h ) can be decomposed as:

where the first and second inequalities hold due to the definition of q V k,h ( s k h ) and V k,h ( s k h ) , the third inequality holds due to Lemma D.6 with p β k last = q β k last , the fourth inequality holds due to b k last ,h ( s k h , a k h ) ≤ C · D F h ( z ; z [ k last -1] ,h , ¯ σ [ k last -1] ,h ) and the last inequality holds due to Lemma G.2. Taking a summation of (F.32) over all episode k ∈ [ K ] and stage h ′ ≥ h , we have

where the first inequality holds due to (F.32), the second inequality holds due to event E 2 , the third inequality holds due to Lemma D.13, the fourth inequality holds due to Cauchy-Schwartz inequality and the last inequality holds due to ∑ H h ′ = h dim α,K ( F h ′ ) ≤ ∑ H h ′ =1 dim α,K ( F h ′ ) = H dim α,K ( F ) .

Furthermore, taking a summation of (F.33), we have

where the first inequality holds due to event E 2 and the last inequality holds due to (F.33). Thus, we complete the proof of Lemma D.15.

F.4.5 Proof of Lemma D.16

Proof of Lemma D.16. According to the definition of estimated variance σ k,h , the summation of variance can be decomposed as following:

For the term I 1 , it can be upper bounded by

where the inequality holds due to Lemma D.9.

For the second term I 2 = ∑ K k =1 ∑ H h =1 E k,h , we have

where the inequality holds due to Lemma D.13 and the second inequality holds due to CauchySchwartz inequality.

For the term I 3 , we have

where the first inequality holds due to Lemma D.8, the second inequality holds due to V k,h +1 ( · ) ≥ V ∗ h +1 ( · ) ≥ q V k,h +1 ( · ) , the third inequality holds due to Lemma D.13 with Cauchy-Schwartz inequality. By Lemma D.15,

where the last inequality follows from Lemma D.7.

Notice that for each variable x , x ≤ a √ x + b implies x ≤ a 2 +2 b , from (F.38), we further have

Substituting (F.39) into (F.37), we obtain the following upper bound for I 3 ,

For the term I 4 , we have

where the first inequality holds due to V k,h +1 ( · ) ≥ V ∗ h +1 ( · ) ≥ V π k h +1 ( · ) , the second inequality holds due to 0 ≤ V ∗ h +1 ( · ) , V π k h +1 ( · ) ≤ 1 , the third inequality holds due to Lemma D.14, and the last inequality follows from Lemma D.7 and the fact that x ≤ a √ x + b implies x ≤ a 2 +2 b .

For the term I 5 , according to the definition of Var K , we have

Substituting the results in (F.35), (F.36), (F.40), (F.41) and (F.42) into (F.34), we have

Notice that for each variable x , x ≤ a √ x + b implies x ≤ a 2 +2 b . With this fact, we have

Thus, we complete the proof of Lemma D.16.

G Covering Number Argument

G.1 Rare Switching

Based on the policy-updating criterion, the following lemma provides a upper bound of the switching cost.

Lemma G.1. The number of episodes when the algorithm updates the value function is at most O (dim α,K ( F ) · H ) .

Proof. According to line 9, the policy is updated at episode k only when there exists a stage h ∈ [ H ] such that

and

Then the following inequality holds,

where the first inequality holds due to the definition of D F h (Definition 2.4), the second inequality follows from (G.1).

(G.3) further gives a lower bound for the summation

Note that χ/C 1+ χ ≤ 1 , we also have

Then we have an upper bound and lower bound for the following summation:

Therefore, the number of policy switching l K is of order O (dim α,K ( F ) · H ) .

Lemma G.2 (Stability of uncertainty under rare switching strategy) . If the policy is not updated at episode k , the uncertainty of all state-action pair z = ( s, a ) ∈ S × A and stage h ∈ [ H ] satisfies the following stability property:

Proof. Due to the definition of k last in Algorithm 1, we have

As is shown in (G.3), here we also have

From the definition of D F h ,

The proof is then completed due to the arbitrariness of h .

G.2 Value Function Class and Its Covering Number

The optimistic value functions at episode k and stage h ∈ [ H ] in our construction belong to the following function class:

where l k is the number of updated policies as defined in Algorithm 1, f i ∈ F h and b ∈ B . Similarly, we also define the following pessimistic value function classes for all k ≥ 1 :

Lemma G.3 ( ϵ -covering number of optimistic value function classes) . For optimistic value function class V k,h defined in (G.4), we define the distance between two value functions V 1 and V 2 as ∥ V 1 -V 2 ∥ ∞ := max s ∈S | V 1 ( s ) -V 2 ( s ) | . Then the ϵ -covering number with respect to the distance function can be upper bounded by

Proof. By the definition of N ( F , ϵ ) , there exists an ϵ/ 2 -net of F , denoted by C ( F , ϵ/ 2) , such that for any f ∈ F , we can find f ′ ∈ C ( F , ϵ/ 2) such that ∥ f -f ′ ∥ ∞ ≤ ϵ/ 2 . Also, there exists an ϵ/ 2 β -net of B , C ( B , ϵ/ 2 β ) .

Then we consider the following subset of V k ,

Consider an arbitrary V ∈ V where V = max a ∈A min 1 ≤ i ≤ l k +1 min(1 , f i ( · , a ) + β · b i ( · , a )) . For each f i , there exists f c i ∈ C ( F , ϵ/ 2) such that ∥ f i -f c i ∥ ∞ ≤ ϵ/ 2 . There also exists b c ∈ C ( B , ϵ/ 2 β ) such that ∥ b -b c ∥ ∞ ≤ ϵ/ 2 β . Let V c = max a ∈A min 1 ≤ i ≤ l k +1 min(1 , f c i ( · , a ) + β · b c ( · , a )) ∈ V c . It is then straightforward to check that ∥ V -V c ∥ ∞ ≤ ϵ/ 2 + β · ϵ/ 2 β = ϵ .

By direct calculation, we have |V c | = [ N ( F h , ϵ/ 2) · N ( B , ϵ/ 2 β )] l k +1 .

LemmaG.4 ( ϵ -covering number of pessimistic value function classes) . For pessimistic value function class q V k,h defined in (G.5), we define the distance between two value functions V 1 and V 2 as ∥ V 1 -V 2 ∥ ∞ := max s ∈S | V 1 ( s ) -V 2 ( s ) | . Then the ϵ -covering number of q V k with respect to the distance function can be upper bounded by N ϵ ( k ) defined in (G.6).

Proof. The proof is nearly the same as that of Lemma G.3.

H Auxiliary Lemmas

Lemma H.1 (Azuma-Hoeffding inequality) . Let { x i } n i =1 be a martingale difference sequence with respect to a filtration {G i } satisfying | x i | ≤ M for some constant M , x i is G i +1 -measurable, E [ x i |G i ] = 0 . Then for any 0 < δ < 1 , with probability at least 1 -δ , we have

Lemma H.2 (Corollary 2, Agarwal et al. 2022) . Let M > 0 , V > v > 0 be constants, and { x i } i ∈ [ t ] be stochastic process adapted to a filtration {H i } i ∈ [ t ] . Suppose E [ x i |H i -1 ] = 0 , | x i | ≤ M and ∑ i ∈ [ t ] E [ x 2 i |H i -1 ] ≤ V 2 almost surely. Then for any δ, ϵ > 0 , let ι = √ log (2 log( V/v )+2) · (log( M/m )+2) δ we have

Lemma H.3 (Lemma 7, Russo and Van Roy 2014) . Consider random variables ( Z n | n ∈ N ) adapted to the filtration ( H n : n = 0 , 1 , ... ) . Assume E [exp { λZ i } ] is finite for all λ . Define the conditional mean µ i = E [ Z i |H i -1 ] . We define the conditional cumulant generating function of the centered random variable [ Z i -µ i ] by ψ i ( λ ) = log E [exp ( λ [ Z i -µ i ]) |H i -1 ] . For all x ≥ 0 and λ ≥ 0 ,

Lemma H.4 (Self-normalized bound for scalar-valued martingales) . Consider random variables ( v n | n ∈ N ) adapted to the filtration ( H n : n = 0 , 1 , ... ) . Let { η i } ∞ i =1 be a sequence of real-valued random variables which is H i +1 -measurable and is conditionally σ -sub-Gaussian. Then for an arbitrarily chosen λ > 0 , for any δ > 0 , with probability at least 1 -δ , it holds that

Lemma H.5 (Lemma 10, Zhang et al. 2020) . Let ( M n ) n ≥ 0 be a martingale such that M 0 = 0 and | M n -M n -1 | ≤ c for some c > 0 and any n ≥ 1 . Let Var n = ∑ n k =1 E [( M k -M k -1 ) 2 |F k -1 ] for n ≥ 0 , where F k = σ ( M 1 , M 2 , . . . , M k ) . Then for any positive integer n and any ε, p > 0 , we have that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer:[Yes]

Justification: In both abstract and introduction, we highlight the contribution in our paper. The proposed algorithm and the corresponding theoretical results are discussed in the followed sections.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explicitly list all the necessary assumptions for our theoretical analysis.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state all the assumptions in Section 2 and give the proof of theorems in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification:The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper is a theoretical work with no societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have described the related works, especially those work which our work is based on with proper citations in corresponding sections.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not include crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not include crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers reinforcement learning with low switching cost. The authors design a new algorithm named MQL-UCB for RL with general function approximation. The key algorithmic design includes a general deterministic policy-switching strategy that achieves low switching cost, a monotonic value function structure with carefully controlled function class complexity, and a variance-weighted regression scheme that exploits historical trajectories with high data efficiency. MQL-UCB achieves minimax optimal regret and a near optimal switching cost.\n\nSTRENGTHS:\n1. The problem of reinforcement learning with low switching cost and general function approximation is interesting.\n2. The paper is solid.  The proof looks correct to me. \n3. The bounds are strong. The switching cost bound is optimal according to the lower bound.\n4. The presentation is clear in general.\n\nWEAKNESSES:\n1. Some of the assumptions look very strong.  The completeness of all functions $V:\\mathcal{S}\\rightarrow [0,1]$ , the second-order completeness and the existence of bonus oracle are not standard to my knowledge. It seems that the weighted regression is possible only if these assumptions hold.\n\n2. The paper [1] also studies low switching reinforcement learning with general function approximation. It seems that the setting in [1] is slightly more general (please correct me if there is misunderstanding) and their algorithm is cleaner.\n\n[1] Xiong et al. A general framework for sequential decision-making under adaptivity constraints.\n\n3. The algorithm itself is very complicated and whether it can be implemented in practice is unclear. Since low switching RL is a very practical problem, some experiments (even under the simplified linear MDP) to show the performance of the algorithm will be helpful. I am curious about how this algorithm performs compared to the standard approach: LSVI-UCB with a doubling trick under linear MDP.\n\nQUESTIONS:\nSee the weakness above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the reinforcement learning under general function approximation. This paper attempts to find an algorithm which achieves optimal regret and maintains low switching cost in the mean time. \n\nIn the algorithm, first they calculated the empirical value function using the weighted ERM, where the weights are chosen according to the variance of the estimator. Next, they utilize optimism and pessimism over the Q-function. The algorithm only collects new samples if the existing dataset does not have enough coverage. \n\nThis paper brings in the technique of constructing monotonic value function (originally from He et al. (2022)) into the cases of general function approximation. \n\nThis paper achieves the optimal rate (matching the lower bound) in both the regret and also the number of switches.\n\nSTRENGTHS:\nThis paper studies MDP with general function approximation, which is more general than the common linear MDP and tabular MDP setting. \n\nAssumptions made in this paper include the completeness and finite Eluder dimension, which are common in related literatures as well.\n\nFrom the regret and switching cost perspective, this paper matches the optimal rate for both of the terms. This paper seems to be the first paper which achives this goal.\n\nCompared to Agarwal el al. (2022), the algorithm and analysis are similar to theirs, but this paper has the following advantages:\n(1) The policies output by this paper is Markovian, while Agarwal el al. (2022) has non-Markov policies.\n(2) The switching cost is better than Agarwal el al. (2022) by a factor of d_elu.\n\nWEAKNESSES:\nThe assumptions made on completeness and eluder dimension are somehow stronger than the usual assumptions in related literatures. Specifically, the completeness assumption requires closeness after executing Bellman operator for any value function, and the Eluder dimension requires additional parameter $\\sigma$. This completeness assumption is difficult to check unless under strong assumption over dynamics, e.g. linear structure or linear mixture MDPs, etc.\n\nCompared to the related work Agarwal el al. (2022), the techniques does not have significant improvement. The algorithm idea is very similar to the algorithm in Agarwal el al. (2022). Specifically, the technique of calculating the variance and monotonicity of value functions are both used in Agarwal el al. (2022).\n\nQUESTIONS:\nI have the following questions:\n\n1. Regarding the completeness assumption, I am curious is there a lower bound showing that the normal completeness assumption $T_h F_{h+1}\\subset F_h$ is not enough?\n\n2. In the algorithm, calculating $D_F$ requires solving an optimization problem over $(f_1, f_2)\\in F$, which is not computational tractable due to the non-convexity of the function. Is there any way to bypass this?\n\nLIMITATIONS:\nYes. The authors addressed all the limitations listed in the guidelines.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies reinforcement learning with general functional approximation and proposes a near-optimal algorithm with low policy switching times.\n\nSTRENGTHS:\nThe paper is very well-written, and the main results of the paper are of high quality. The authors improved prior works on RL with general functional approximations to get an algorithm that achieves both the near-optimal regret and the lowest possible switching cost when the number of episodes is large. Besides, the proposed algorithm is intuitive and clean, and the proofs are well-written and easy to follow.\n\nWEAKNESSES:\nIt would be helpful to provide a bit more details for readers (like me) who are not very familiar with the literature on RL with policy switching cost on how to obtain the counting of switches in Table 1. For example, [1] doesn't seem to optimize the number of policy switches (while only trying to optimize the regret); since the paper is directly improving upon [1], it would be helpful to provide a more detailed discussion on why [1]'s algorithm needs $\\tilde{O}(dim(\\mathcal{F})^2H)$ number of switches and why the authors' algorithm can strictly improve upon the number of switches to $\\tilde{O}(dim(\\mathcal{F})H)$ without worsening the regret.\n\n[1] Agarwal, Alekh, Yujia Jin, and Tong Zhang. \"VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximation.\" The Thirty-Sixth Annual Conference on Learning Theory. PMLR, 2023.\n\nQUESTIONS:\nYou have proposed an algorithm that achieves optimality of regret of switching costs simultaneously under the condition that $K$ is large. Also, the lower bound from Theorem B.1 suggests $\\tilde{O}(dim(\\mathcal{F})H)$ is unavoidable when one wants to achieve a regret sublinear in $K$. Obviously, the current results will not make sense if the number of episodes $K$ is relatively small compared to the length of episode $H$. I am thus wondering: do you have any insights (what can be done and what cannot be done) in this different regime?\n\nLIMITATIONS:\nThere is no obvious limitation in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}