Title: Model-free Posterior Sampling via Learning Rate Randomization
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

exploration

q-learning
Abstract: In this paper, we introduce Randomized Q-learning (RandQL), a novel randomized model-free algorithm for regret minimization in episodic Markov Decision Processes (MDPs). To the best of our knowledge, RandQL is the first tractable model-free posterior sampling-based algorithm. We analyze the performance of RandQL in both tabular and non-tabular metric space settings. In tabular MDPs, RandQL achieves a regret bound of order $\widetilde{\mathcal{O}}(\sqrt{H^{5}SAT})$, where $H$ is the planning horizon, $S$ is the number of states, $A$ is the number of actions, and $T$ is the number of episodes. For a metric state-action space, RandQL enjoys a regret bound of order $\widetilde{\mathcal{O}}(H^{5/2} T^{(d_z+1)/(d_z+2)})$, where $d_z$ denotes the zooming dimension. Notably, RandQL achieves optimistic exploration without using bonuses, relying instead on a novel idea of learning rate randomization. Our empirical study shows that RandQL outperforms existing approaches on baseline exploration environments.

Model-free Posterior Sampling via Learning Rate Randomization

Abstract

In this paper, we introduce Randomized Q-learning ( RandQL ), a novel randomized model-free algorithm for regret minimization in episodic Markov Decision Processes (MDPs). To the best of our knowledge, RandQL is the first tractable model-free posterior sampling-based algorithm. We analyze the performance of RandQL in both tabular and non-tabular metric space settings. In tabular MDPs, RandQL achieves a regret bound of order ˜ O ( √ H 5 SAT ) , where H is the planning horizon, S is the number of states, A is the number of actions, and T is the number of episodes. For a metric state-action space, RandQL enjoys a regret bound of order ˜ O ( H 5 / 2 T ( d z +1) / ( d z +2) ) , where d z denotes the zooming dimension. Notably, RandQL achieves optimistic exploration without using bonuses , relying instead on a novel idea of learning rate randomization . Our empirical study shows that RandQL outperforms existing approaches on baseline exploration environments.

1 Introduction

In reinforcement learning (RL, Sutton and Barto 1998), an agent learns to interact with an unknown environment by acting, observing the next state, and receiving a reward. The agent's goal is to maximize the sum of the collected rewards. To achieve this, the agent can choose to use model-based or model-free algorithms. In model-based algorithms, the agent builds a model of the environment by inferring the reward function and the transition kernel that produces the next state. The agent then plans in this model to find the optimal policy. In contrast, model-free algorithms directly learn the optimal policy, which is the mapping of a state to an optimal action, or equivalently, the optimal Q-values, which are the mapping of a state-action pair to the expected return of an optimal policy starting by taking the given action at the given state.

Although empirical evidence suggests that model-based algorithms are more sample efficient than model-free algorithms [Deisenroth and Rasmussen, 2011, Schulman et al., 2015]; model-free approaches offer several advantages. These include smaller time and space complexity, the absence of a need to learn an explicit model, and often simpler algorithms. As a result, most of the recent breakthroughs in deep RL, such as those reported by Mnih et al. [2013], Schulman et al. [2015, 2017], Haarnoja et al. [2018], have been based on model-free algorithms, with a few notable exceptions, such as Schrittwieser et al. [2020], Hessel et al. [2021]. Many of these model-free algorithms [Mnih et al., 2013, Van Hasselt et al., 2016, Lillicrap et al., 2016] are rooted in the well-known Q-learning algorithm of Watkins and Dayan [1992]. Q-learning is an off-policy learning technique where the agent follows a behavioral policy while simultaneously incrementally learning the optimal Q-values

by combining asynchronous dynamic programming and stochastic approximation. Until recently, little was known about the sample complexity of Q-learning in the setting where the agent has no access to a simulator allowing to sample an arbitrary state-action pair. In this work, we consider such challenging setting where the environment is modelled by an episodic Markov Decision Process (MDP) of horizon H . After T episodes, the performance of an agent is measured through regret which is the difference between the cumulative reward the agent could have obtained by acting optimally and what the agent really obtained during the interaction with the MDP.

This framework poses the famous exploration-exploitation dilemma where the agent must balance the need to try new state-action pairs to learn an optimal policy against exploiting the current observations to collect the rewards. One effective approach to resolving this dilemma is to adopt the principle of optimism in the face of uncertainty. In finite MDPs, this principle has been successfully implemented in the model-based algorithm using bonuses [Jaksch et al., 2010, Azar et al., 2017, Fruit et al., 2018, Dann et al., 2017, Zanette and Brunskill, 2019]. Specifically, the upper confidence bounds (UCBs) on the optimal Q-value are built by adding bonuses and then used for planning. Building on this approach, Jin et al. [2018] proposed the OptQL algorithm, which applies a similar bonus-based technique to Q-learning, achieving efficient exploration. Recently, Zhang et al. [2020] introduced a simple modification to OptQL that achieves optimal sample complexity, making it competitive with model-based algorithms.

Another class of methods for optimistic exploration is Bayesian-based approaches. An iconic example among this class is the posterior sampling for reinforcement learning ( PSRL ,Strens 2000, Osband et al. 2013) algorithm. This model-based algorithm maintains a surrogate Bayesian model of the MDP, for instance, a Dirichlet posterior on the transition probability distribution if the rewards are known. At each episode, a new MDP is sampled (i.e., a transition probability for each state-action pair) according to the posterior distribution of the Bayesian model. Then, the agent plans in this sampled MDP and uses the resulting policy to interact with the environment. Notably, an optimistic variant of PSRL , named optimistic posterior sampling for reinforcement learning ( OPSRL , Agrawal and Jia, 2017, Tiapkin et al., 2022a) also enjoys an optimal sample complexity [Tiapkin et al., 2022a]. The random least square value iteration ( RLSVI , Osband et al. [2013]) is another well-known model-based algorithm that leverages a Bayesian-based technique for exploration. Precisely, RLSVI directly sets a Gaussian prior on the optimal Q-values and then updates the associated posterior trough value iteration in a model [Osband et al., 2013, Russo, 2019]. A close variant of RLSVI proposed by Xiong et al. [2022], using a more sophisticated prior/posterior couple, is also proven to be near-optimal.

It is noteworthy that Bayesian-based exploration techniques have shown superior empirical performance compared to bonus-based exploration, at least in the tabular setting [Osband et al., 2013, Osband and Van Roy, 2017]. Furthermore, these techniques have also been successfully applied to the deep RL setting [Osband et al., 2016, Azizzadenesheli et al., 2018, Fortunato et al., 2018, Li et al., 2022, Sasso et al., 2023]. Finally, Bayesian methods allow for the incorporation of apriori information into exploration (e.g. by giving more weight to important states). However, most of the theoretical studies on Bayesian-based exploration have focused on model-based algorithms, raising the natural question of whether the PSRL approach can be extended to a provably efficient model-free algorithm that matches the good empirical performance of its model-based counterparts. Recently, Dann et al. [2021] proposed a model-free posterior sampling algorithm for structured MDPs, however, it is not computationally tractable. Therefore, a provably tractable model-free posterior sampling algorithm has remained a challenge.

In this paper, we aim to resolve this challenge. We propose the randomized Q-learning ( RandQL ) algorithm that achieves exploration without bonuses, relying instead on a novel idea of learning rate randomization. RandQL is a tractable model-free algorithm that updates an ensemble of Q-values via Q-learning with Beta distributed step-sizes. If tuned appropriately, the noise introduced by the random learning rates is similar to the one obtained by sampling from the posterior of the PSRL algorithm. Thus, one can see the ensemble of Q-values as posterior samples from the same induced posterior on the optimal Q-values as in PSRL . Then, RandQL chooses among these samples in the same optimistic fashion as OPSRL . We prove that for tabular MDPs, a staged version [Zhang et al., 2020] of RandQL , named Staged-RandQL enjoys the same regret bound as the OptQL algorithm, that is, ˜ O ( √ H 5 SAT ) where S is the number of states and A the number of actions. Furthermore, we extend Staged-RandQL beyond the tabular setting into the Net-Staged-RandQL algorithm to deal with metric state-action spaces [Domingues et al., 2021c, Sinclair et al., 2019]. Net-Staged-RandQL operates similarly to Staged-RandQL but over a fixed discretization of the state-action space and

uses a specific prior tuning to handle the effect of discretization. We prove that Net-Staged-RandQL enjoys a regret bound of order ˜ O ( H 5 / 2 T ( d c +1) / ( d c +2) ) , where d c denotes the covering dimension. This rate is of the same order as the one of Adaptive-QL by Sinclair et al. [2019, 2023], an adaptation of OptQL to metric state-action space and has a better dependence on the budget T than one of the model-based kernel algorithms such that Kernel-UCBVI by Domingues et al. [2021c]. We also explain how to adapt Net-Staged-RandQL and its analysis to work with an adaptive discretization as by Sinclair et al. [2019, 2023]. Finally, we provide preliminary experiments to illustrate the good performance of RandQL against several baselines in finite and continuous environments.

We highlight our main contributions:

The RandQL algorithm, a new tractable (provably efficient) model-free Q-learning adaptation of the PSRL algorithm that explores through randomization of the learning rates. √

Aregret bound of order ˜ O ( H 5 SAT ) for a staged version of the RandQL algorithm in finite MDPs where S is the number of states and A the number of actions, H the horizon and T the budget.

A regret bound of order ˜ O ( H 5 / 2 T ( d c +1) / ( d c +2) ) for an adaptation of RandQL to metric spaces where d c denotes the covering dimension.

Adaptive version of metric space extension of RandQL algorithm that achieves a regret bound of order ˜ O ( H 5 / 2 T ( d z +1) / ( d z +2) ) , where d z is a zooming dimension.

Experiments in finite and continuous MDPs that show that RandQL is competitive with model-based and model-free baselines while keeping a low time-complexity.

2 Setting

We consider an episodic MDP ( S , A , H, { p h } h ∈ [ H ] , { r h } h ∈ [ H ] ) , where S is the set of states, A is the set of actions, H is the number of steps in one episode, p h ( s ′ | s, a ) is the probability transition from state s to state s ′ upon taking action a at step h, and r h ( s, a ) ∈ [0 , 1] is the bounded deterministic reward received after taking the action a in state s at step h . Note that we consider the general case of rewards and transition functions that are possibly non-stationary, i.e., that are allowed to depend on the decision step h in the episode.

Policy & value functions A deterministic policy π is a collection of functions π h : S → A for all h ∈ [ H ] , where every π h maps each state to a single action. The value functions of π , denoted by V π h , as well as the optimal value functions, denoted by V ⋆ h are given by the Bellman and the optimal Bellman equations,

where by definition, V ⋆ H +1 ≜ V π H +1 ≜ 0 . Furthermore, p h f > ( s, a ) ≜ E s ′ ∼ p h ( ·| s,a ) [ f ( s ′ )] denotes the expectation operator with respect to the transition probabilities p h and π h g ( s ) ≜ g ( s, π h ( s )) denotes the composition with the policy π at step h .

Learning problem The agent, to which the transitions are unknown (the rewards are assumed to be known 1 for simplicity), interacts with the environment during T episodes of length H , with a fixed initial state s 1 . 2 Before each episode t the agent selects a policy π t based only on the past observed transitions up to episode t -1 . At each step h ∈ [ H ] in episode t , the agent observes a state s t h ∈ S , takes an action π t h ( s t h ) = a t h ∈ A and makes a transition to a new state s t h +1 according to the probability distribution p h ( s t h , a t h ) and receives a deterministic reward r h ( s t h , a t h ) .

Regret The quality of an agent is measured through its regret, that is the difference between what it could obtain (in expectation) by acting optimally and what it really gets,

Additional notation For N ∈ N ++ , we define the set [ N ] ≜ { 1 , . . . , N } . We denote the uniform distribution over this set by Unif[ N ] . We define the beta distribution with parameters α, β as Beta( α, β ) . Appendix A references all the notation used.

3 Randomized Q-learning for Tabular Environments

In this section we assume that the state space S is finite of size S as well as the action space A of size A . We first provide some intuitions for RandQL algorithm.

3.1 Concept

The main idea of RandQL is to perform the usual Q-learning updates but instead of adding bonuses to the targets as OptQL to drive exploration, RandQL injects noise into the updates of the Q-values through noisy learning rates . Precisely, for J ∈ N , we maintain an ensemble of size J of Qvalues 3 ( Q n,j ) j ∈ [ J ] updated with random independent Beta-distributed step-sizes ( w n,j ) j ∈ [ J ] where w n,j ∼ Beta( H,n ) . Then, policy Q-values Q n are obtained by taking the maximum among the Q-values of the ensemble

where s n h +1 stands for the next (in time) state after n -th visitation of ( s, a ) at step h .

Note that the policy Q-values Q n are designed to be upper confidence bound on the optimal Q-values. The policy used to interact with the environment is greedy with respect to the policy Q-values π n h ( s ) ∈ arg max a Q n h ( s, a ) . We provide a formal description of RandQL in Appendix B.

Connection with OptQL We observe that the learning rates of RandQL are in expectation of the same order E [ w n,j ] = H/ ( n + H ) as the ones used by the OptQL algorithm. Thus, we can view our randomized Q-learning as a noisy version of the OptQL algorithm that doesn't use bonuses.

Connection with PSRL If we unfold the recursive formula above we can express the Q-values Q n +1 ,j as a weighted sum

To compare, we can unfold the corresponding formula for PSRL algorithm using the aggregation properties of the Dirichlet distribution (see e.g. Section 4 of Tiapkin et al. [2022b] or Appendix C)

where weights ( ˜ W 0 n , . . . , ˜ W n n ) follows Dirichlet distribution Dir( n 0 , 1 , . . . , 1) and n 0 is a weight for the prior distribution. In particular, one can represent these weights as partial products of other weights w n ∼ Beta(1 , n + n 0 ) . If we use (1) to construct a model-free algorithm, this would require recomputing the targets r h ( s, a ) + V n +1 ( s k h +1 ) in each iteration. To make algorithm more efficient and model-free, we approximate V n +1 by V k , and, as a result, obtain RandQL algorithm with weight distribution w n,j ∼ Beta(1 , n + n 0 ) .

Note that in expectation this algorithm is equivalent to OptQL with the uniform step-sizes which are known to be sub-optimal due to a high bias (see discussion in Section 3 of [Jin et al., 2018]). There are two known ways to overcome this sub-optimality for Q-learning: to introduce more aggressive

learning rates w n,j ∼ Beta( H,n + n 0 ) leading to RandQL algorithm, or to use stage-dependent framework by Bai et al. [2019], Zhang et al. [2020] resulting in Staged-RandQL algorithm.

The aforementioned transition from PSRL to RandQL is similar to the transition from UCBVI [Azar et al., 2017] to Q-learning. To make UCBVI model-free, one has to to keep old targets in Q-values. This, however, introduces a bias that could be eliminated either by more aggressive step-size [Jin et al., 2018] or by splitting on stages [Bai et al., 2019]. Our algorithms ( RandQL and Staged-RandQL ) perform the similar tricks for PSRL and thus could be viewed as model-free versions of it. Additionally, RandQL shares some similarities with the OPSRL algorithm [Agrawal and Jia, 2017, Tiapkin et al., 2022a] in the way of introducing optimism (taking maximum over J independent ensembles of Q-values). Let us also mention a close connection to the theory of Dirichlet processes in the proof of optimism for the case of metric spaces (see Remark 1 in Appendix E.4).

Prior As remarked above, in expectation, RandQL has a learning rate of the same order as OptQL . In particular, it implies that the first (1 -1 /H ) fraction of the the target will be forgotten exponentially fast in the estimation of the Q-values, see Jin et al. [2018], Ménard et al. [2021]. Thus we need to re-inject prior targets, as explained in Appendix B, in order to not forget too quickly the prior and thus replicate the same exploration mechanism as in the PSRL algorithm.

3.2 Algorithm

In this section, following Bai et al. [2019], Zhang et al. [2020], we present the Staged-RandQL algorithm a scheduled version of RandQL that is simpler to analyse. The main idea is that instead of using a carefully tuned learning rate to keep only the last 1 /H fraction of the targets we split the learning of the Q-values in stages of exponentially increasing size with growth rate of order 1 + 1 /H . At a given stage, the estimate of the Q-value relies only on the targets within this stage and resets at the beginning of the next stage. Notice that the two procedures are almost equivalent. A detail description of Staged-RandQL is provided in Algorithm 1.

Counts and stages Let n t h ( s, a ) ≜ ∑ t -1 i =1 1 { ( s i h , a i h ) = ( s, a ) } be the number of visits of state-action pair ( s, a ) at step h before episode t . We say that a triple ( s, a, h ) belongs to the k -th stage at the beginning of episode t if n t h ( s, a ) ∈ [ ∑ k -1 i =0 e i , ∑ k i =0 e i ) . Here e k = ⌊ (1 + 1 /H ) k · H ⌋ is the length of the stage k ≥ 0 and, by convention, e -1 = 0 . Let ˜ n t h ( s, a ) ≜ n t h ( s, a ) -∑ k -1 i =0 e i be the number of visits of state-action pair ( s, a ) at step h during the current stage k .

Temporary Q-values At the beginning of a stage, let say time t , we initialize J temporary Q-values as ˜ Q t,j h ( s, a ) = r h ( s, a ) + r 0 ( H -h -1) for j ∈ [ J ] and r 0 some pseudo-reward. Then as long as ( s t h , a t h , h ) remains within a stage we update recursively the temporary Q-values

where ˜ n = ˜ n t h ( s, a ) is the number of visits, w j, ˜ n is a sequence of i.i.d. random variables w j, ˜ n ∼ Beta(1 /κ, ( ˜ n + n 0 ) /κ ) with κ > 0 being some posterior inflation coefficient and n 0 a number of pseudo-transitions.

Policy Q-values Next we define the policy Q-values that is updated at the end of a stage. Let say for state-action pair ( s, a ) at step h an stage ends at time t . This policy Q-values is then given by the maximum of temporary Q-values Q t +1 h = max j ∈ [ J ] ˜ Q t +1 ,j h ( s, a ) . Then the policy Q-values is constant within a stage. The value used to defined the targets is V t +1 h ( s ) = max a ∈A Q t +1 h ( s, a ) . The policy used to interact with the environment is greedy with respect to the policy Q-values π t +1 h ( s ) ∈ arg max a ∈A Q t +1 h ( s, a ) (we break ties arbitrarily).

3.3 Regret bound

We fix δ ∈ (0 , 1) and the number of posterior samples J ≜ ⌈ c J · log(2 SAHT/δ ) ⌉ , where c J = 1 / log(2 / (1 + Φ(1))) and Φ( · ) is the cumulative distribution function (CDF) of a normal distribution. Note that J has a logarithmic dependence on S, A, H, T, and 1 /δ .

We now state the regret bound of Staged-RandQL with a full proof in Appendix D.

Algorithm 1 Tabular Staged-RandQL

Theorem 1. Consider a parameter δ ∈ (0 , 1) . Let κ ≜ 2(log(8 SAH/δ )+3log(e π (2 T +1))) , n 0 ≜ ⌈ κ ( c 0 +log 17 / 16 ( T )) ⌉ , r 0 ≜ 2 , where c 0 is an absolute constant defined in (5) ; see Appendix D.3. Then for Staged-RandQL , with probability at least 1 -δ ,

Discussion The regret bound of Theorem 1 coincides (up to a logarithmic factor) with the bound of the OptQL algorithm with Hoeffding-type bonuses from Jin et al. [2018]. Up to a H factor, our regret matches the information-theoretic lower bound Ω( √ H 3 SAT ) [Jin et al., 2018, Domingues et al., 2021b]. This bound could be achieved (up to logarithmic terms) in model-free algorithms by using Bernstein-type bonuses and variance reduction [Zhang et al., 2020]. We keep these refinements for future research as the main focus of our paper is on the novel randomization technique and its use to construct computationally tractable model-free algorithms.

Computational complexity Staged-RandQL is a model-free algorithm, and thus gets the ˜ O ( HSA ) space complexity as OptQL , recall that we set J = ˜ O (1) . The per-episode time-complexity is also similar and of order ˜ O ( H ) .

4 Randomized Q-learning for Metric Spaces

In this section we present a way to extend RandQL to general state-action spaces. We start from the simplest approach with predefined ε -net type discretization of the state-action space S × A (see Song and Sun 2019), and then discuss an adaptive version of the algorithm, similar to one presented by Sinclair et al. [2019].

4.1 Assumptions

To pose the first assumption, we start from a general definition of covering numbers.

Definition 1 (Covering number and covering dimension) . Let ( M,ρ ) be a metric space. A set M of open balls of radius ε is called an ε -cover of M if M ⊆ ⋃ B ∈M B . The cardinality of the minimal ε -cover is called covering number N ε of ( M,ρ ) . We denote the corresponding minimal ε -covering by N ε . A metric space ( M,ρ ) has a covering dimension d c if ∀ ε > 0 : N ε ≤ C N ε -d c , where C N is a constant.

The last definition extends the definition of dimension beyond vector spaces. For example, is case of M = [0 , 1] d the covering dimension of M is equal to d . For more details and examples see e.g. Vershynin [2018, Section 4.2].

Next we are ready to introduce the first assumption.

Assumption 1 (Metric Assumption) . Spaces S and A are separable compact metric spaces with the corresponding metrics ρ S and ρ A . The joint space S × A endowed with a product metric ρ that satisfies ρ (( s, a ) , ( s ′ , a ′ )) ≤ ρ S ( s, s ′ ) + ρ A ( a, a ′ ) . Moreover, the diameter of S × A is bounded by d max , and S × A has covering dimension d c with a constant C N .

̸

This assumption is, for example, satisfied for the finite state and action spaces endowed with discrete metrics ρ S ( s, s ′ ) = 1 { s = s ′ } , ρ A ( a, a ′ ) = 1 { a = a ′ } with d c = 0 , C N = SA and S and A being the cardinalities of the state and action spaces respectively. The above assumption also holds in the case S ⊆ [0 , 1] d S and A ⊆ [0 , 1] d A with d c = d S + d A .

̸

The next two assumptions describe the regularity conditions of transition kernel and rewards.

Assumption 2 (Reparametrization Assumption) . The Markov transition kernel could be represented as an iterated random function. In other words, there exists a measurable space (Ξ , F Ξ ) and a measurable function F h : ( S × A ) × Ξ → S , such that s h +1 ∼ p h ( s h , a h ) ⇐⇒ s h +1 = F h ( s h , a h , ξ h ) for a sequence of independent random variables { ξ h } h ∈ [ H ] .

This assumption is naturally satisfied for a large family of probabilistic model, see Kingma and Welling [2014]. Moreover, it has been utilized by the RL community both in theory [Ye and Zhou, 2015] and practice [Heess et al., 2015, Liu et al., 2018]. Essentially, this assumption holds for Markov transition kernels over a separable metric space, see Theorem 1.3.6 by Douc et al. [2018]. However, the function F h could be ill-behaved. To avoid this behaviour, we need the following assumption.

Assumption 3 (Lipschitz Assumption) . The function F h ( · , ξ h ) is L F -Lipschitz in the first argument for almost every value of ξ h . Additionally, the reward function r h : S × A → [0 , 1] is L r -Lipschitz.

This assumption is commonly used in studies of the Markov processes corresponding to iterated random functions, see Diaconis and Freedman [1999], Ghosh and Marecek [2022]. Moreover, this assumption holds for many cases of interest. As main example, it trivially holds in tabular and Lipschitz continuous deterministic MDPs [Ni et al., 2019]. Notably, this observation demonstrates that Assumption 3 does not necessitate Lipschitz continuity of the transition kernels in total variation distance, since deterministic Lipschitz MDPs are not continuous in that sense. Additionally, incorporation of an additive noise to deterministic Lipschitz MDPs will lead to Assumption 3 with L F = 1 .

Furthermore, it is possible to show that Assumption 3 implies other assumptions stated in the literature. For example, it implies that the transition kernel is Lipschitz continuous in 1 -Wasserstein metric, and that Q ⋆ and V ⋆ are both Lipschitz continuous.

Lemma 1. Let Assumption 1,2,3 hold. Then the transition kernels p h ( s, a ) are L F -Lipschitz continuous in 1 -Wasserstein distance

where 1 -Wasserstein distance between two probability measures on the metric space ( M,ρ ) is defined as W 1 ( ν, η ) = sup f is 1 -Lipschitz ∫ M f d ν -∫ M f d η .

Lemma 2. Let Assumption 1,2,3 hold. Then Q ⋆ h and V ⋆ h are Lipschitz continuous with Lipschitz constant L V,h ≤ ∑ H h ′ = h L h ′ -h F L r .

The proof of these lemmas is postponed to Appendix E. For a more detailed exposition on 1Wasserstein distance we refer to the book by Peyré and Cuturi [2019]. The first assumption was studied by Domingues et al. [2021c], Sinclair et al. [2023] in the setting of model-based algorithms in metric spaces. We are not aware of any natural examples of MDPs with a compact state-action space where the transition kernels are Lipschitz in W 1 but fail to satisfy Assumption 3.

4.2 Algorithms

In this section, following Song and Sun [2019], we present Net-Staged-RandQL algorithm that combines a simple non-adaptive discretization and an idea of stages by Bai et al. [2019], Zhang et al. [2020].

We assume that we have an access to all Lipschitz constants L r , L F , L V ≜ L V, 1 . Additionally, we have access to the oracle that computes ε -cover N ε of the space S × A for any predefined ε > 0 4 .

Counts and stages Let n t h ( B ) ≜ ∑ t -1 i =1 1 { ( s i h , a i h ) ∈ B } be the number of visits of the ball B ∈ N ε at step h before episode t . Let e k = ⌊ (1 + 1 /H ) k · H ⌋ be length of the stage k ≥ 0 and, by convention, e -1 = 0 . We say that ( B,h ) belongs to the k -th stage at the beginning of episode t if n t h ( B ) ∈ [ ∑ k -1 i =0 e i , ∑ k i =0 e i ) . Let ˜ n t h ( B ) ≜ n t h ( s, a ) -∑ k -1 i =0 e i be the number of visits of the ball B at step h during the current stage k .

Temporary Q-values At the beginning of a stage, let say time t , we initialize J temporary Q-values as ˜ Q t,j h ( B ) = r 0 H for j ∈ [ J ] and r 0 some pseudo-reward. Then within a stage k we update recursively the temporary Q-values

where ˜ n = ˜ n t h ( B ) is the number of visits, w j, ˜ n is a sequence of i.i.d random variables w j, ˜ n ∼ Beta(1 /κ, ( ˜ n + n 0 ( k )) /κ ) with κ > 0 some posterior inflation coefficient and n 0 ( k ) a number of pseudo-transitions. The important difference between tabular and metric settings is the dependence on the pseudo-count n 0 ( k ) on k in the latter case, since here the prior is used to eliminate the approximation error.

Policy Q-values Next, we define the policy Q-values that are updated at the end of a stage. Let us fix a ball B at step h and suppose that the currents stage ends at time t . Then the policy Q-values are given by the maximum of the temporary Q-values Q t +1 h ( B ) = max j ∈ [ J ] ˜ Q t +1 ,j h ( B ) . The policy Q-values are constant within a stage. The value used to define the targets is computed on-flight using the formula V t h ( s ) = max a ∈A Q t h ( ψ ε ( s, a )) , where ψ ε : S × A → N ε is a quantization map, that assigns each state-action pair ( s, a ) to a ball B ∋ ( s, a ) . The policy used to interact with the environment is greedy with respect to the policy Q-values and also computed on-flight π t h ( s ) ∈ arg max a ∈A Q t h ( ψ ε ( s, a )) (we break ties arbitrarily).

A detail description of Net-Staged-RandQL is provided in Algorithm 4 in Appendix E.2.

4.3 Regret Bound

We fix δ ∈ (0 , 1) , the discretization level ε > 0 and the number of posterior samples

where ˜ c J = 1 / log(4 / (3 + Φ(1))) and Φ( · ) is the cumulative distribution function (CDF) of a normal distribution. Note that J has a logarithmic dependence on H,T, 1 /ε and 1 /δ . For the regret-optimal discretization level ε = T -1 / ( d c +2) , the number J is almost independent of d c . Let us note that the role of prior in metric spaces is much higher than in the tabular setting. Another important difference is dependence of the prior count on the stage index. In particular, we have

where c 0 is an absolute constant defined in (5) ( see Appendix D.3), κ is the posterior inflation coefficient and L = L r +(1+ L F ) L V is a constant. We now state the regret bound of Net-Staged-RandQL with a full proof being postponed to Appendix E.

Theorem 2. Suppose that N ε ≤ C N ε -d c for all ε > 0 and some constant C N > 0 . Consider a parameter δ ∈ (0 , 1) and take an optimal level of discretization ε = T -1 / ( d c +2) . Let κ ≜ 2(log(8 HC N /δ )+ d c log(1 /ε )+3log(e π (2 T +1))) , r 0 ≜ 2 . Then it holds for Net-Staged-RandQL , with probability at least 1 -δ ,

We can restore the regret bound in the tabular setting by letting d c = 0 and C N = SA , where S is the cardinality of the state-space, and A is the cardinality of the action-space.

Discussion From the point of view of instance-independent bounds, our algorithm achieves the same result as Net-QL [Song and Sun, 2019] and Adaptive-QL [Sinclair et al., 2019], that matches the lower bound Ω( HT dc +1 dc +2 ) by Sinclair et al. [2023] in dependence on budget T and covering dimension d c . Notably, as discussed by Sinclair et al. [2023], the model-based algorithm such as Kernel-UCBVI [Domingues et al., 2021c] does not achieves optimal dependence in T due to hardness of the transition estimation problem.

Computational complexity For a fixed level of discretization ε , our algorithm has a space complexity of order ˜ O ( H N ε ) . Assuming that the computation of a quantization map ψ ε has ˜ O (1) time complexity, we achieve a per-episode time complexity of ˜ O ( HA ) for a finite action space and O ( HN ε ) for an infinite action space in the worst case due to computation of arg max a ∈A Q h ( ψ ε ( s, a )) . However, this can be improved to ˜ O ( H ) if we consider adaptive discretization [Sinclair et al., 2019].

Adaptive discretization Additionally, we propose a way to combine RandQL with adaptive discretization by Cao and Krishnamurthy [2020], Sinclair et al. [2023]. This combination results in two algorithms: Adaptive-RandQL and Adaptive-Staged-RandQL . The second one could achieve the instance-dependent regret bound that scales with a zooming dimension, the instance-dependent measure of dimension. We will follow Sinclair et al. [2023] in the exposition of the required notation.

Definition 2. For any ( s, a ) ∈ S × A , the stage-dependent sub-optimality gap is defined as gap h ( s, a ) = V ⋆ h ( s ) -Q ⋆ h ( s, a ) .

This quantity is widely used in the theoretical instance-dependent analysis of reinforcement learning and contextual bandit algorithms.

Definition 3. The near-optimal set of S × A for a given value ε defined as Z ε h = { ( s, a ) ∈ S × A | gap h ( s, a ) ≤ ( H +1) ε } .

The main insight of this definition is that essentially we are interested in a detailed discretization of the near-optimal set Z ε h for small ε , whereas all other state-action pairs could be discretized in a more rough manner. Interestingly enough, Z ε h could be a lower dimensional manifold, leading to the following definition.

Definition 4. The steph zooming dimension d z,h with a constant C N,h and a scaling factor ρ > 0 is given by

Under some additional structural assumptions on Q ⋆ h , it is possible to show that the zooming dimension could be significantly smaller than the covering dimension, see, e.g., Lemma 2.8 in Sinclair et al. [2023]. However, at the same time, it has been shown that d z,h ≥ d S -1 , where d S is a covering dimension of the state space. Thus, the zooming dimension allows adaptation to a rich action space but not a rich state space.

Given this definition, it is possible to define define an adaptive algorithm Adaptive-Staged-RandQL that attains the following regret guarantees

Theorem 3. Consider a parameter δ ∈ (0 , 1) . For a value κ that depends on T, d c ad δ , for Adaptive-Staged-RandQL the following holds with probability at least 1 -δ ,

where d z,h is the steph zooming dimension and we ignore all multiplicative factors in the covering dimension d c , log( C N ) , and Lipschitz constants.

We refer to Appendix F to a formal statement and a proof.

5 Experiments

In this section we present the experiments we conducted for tabular environments using rlberry library [Domingues et al., 2021a]. We also provide experiments in non-tabular environment in Appendix I.

Environment We use a grid-world environment with 100 states ( i, j ) ∈ [10] × [10] and 4 actions (left, right, up and down). The horizon is set to H = 50 . When taking an action, the agent moves in the corresponding direction with probability 1 -ε , and moves to a neighbor state at random with probability ε = 0 . 2 . The agent starts at position (1 , 1) . The reward equals to 1 at the state (10 , 10) and is zero elsewhere.

Variations of randomized Q-learning For the tabular experiment we use the RandQL algorithm, described in Appendix B as it is the version of randomized Q-learning that is the closest to the baseline OptQL . Note that, we compare the different versions of randomized Q-learning in Appendix B.

Baselines We compare RandQL algorithm to the following baselines: (i) OptQL [Jin et al., 2018] (ii) UCBVI [Azar et al., 2017] (iii) Greedy-UCBVI , a version of UCBVI using realtime dynamic programming [Efroni et al., 2019] (iv) PSRL [Osband et al., 2013] and (v) RLSVI [Russo, 2019]. For the hyper-parameters used for these baselines refer to Appendix I.

0

5000

10000

15000

20000

25000

30000

episode

0

100000

200000

300000

400000

500000

600000

name

Greedy UCBVI

OptQL

PSRL

Randomized QL

RLSVI

UCBVI

regret

Results Figure 1 shows the result of the experiments. Overall, we see that RandQL outperforms OptQL algorithm on tabular environment, but still degrades in comparison to model-based approaches, that is usual for model-free algorithms in tabular environments. Indeed, using a model and backward induction allows new information to be more quickly propagated. But as counterpart, RandQL has a better time-complexity and space-complexity than model-based algorithm, see Table 2 in Appendix I.

6 Conclusion

This paper introduced the RandQL algorithm, a new model-free algorithm that achieves exploration without bonuses. It utilizes a novel idea of learning rate randomization, resulting in provable sample efficiency with regret of order ˜ O ( √ H 5 SAT ) in the tabular case. We also extend RandQL to the case of metric state-action space by using proper discretization techniques. The proposed algorithms inherit the good empirical performance of model-based Bayesian algorithm such that PSRL while keeping the small space and time complexity of model-free algorithm. Our result rises following interesting open questions for a further research.

Optimal rate for RandQL We conjecture that RandQL could get optimal regret in the tabular setting if coupled with variance reductions techniques as used by Zhang et al. [2020]. However, obtaining such improvements is not straightforward due to the intricate statistical dependencies involved in the analysis of RandQL .

Beyond one-step learning We observe a large gap in the experiments between Q-learning type algorithm that do one-step planning and e.g. UCBVI algorithm that does full planning or Greedy-UCBVI that does one-step planning with full back-up (expectation under transition of the model) for all actions. Therefore, it would interesting to study also algorithms that range between these two extremes [Efroni et al., 2018, 2019].

Acknowledgments

The work of D. Tiapkin, A. Naumov, and D. Belomestny were supported by the grant for research centers in the field of AI provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE University No. 70-202100139. E. Moulines received support from the grant ANR-19-CHIA-002 SCAI and parts of his work has been done under the auspices of Lagrange Center for maths and computing. P. Ménard acknowledges the Chaire SeqALO (ANR-20-CHIA-0020-01). This research was supported in part through computational resources of HPC facilities at HSE University.

References

Shipra Agrawal and Navin Goyal. Further optimal regret bounds for thompson sampling. In Carlos M. Carvalho and Pradeep Ravikumar, editors, Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics , volume 31 of Proceedings of Machine Learning Research , pages 99-107, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR. URL https: //proceedings.mlr.press/v31/agrawal13a.html .

Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3621f1454cacf995530ea53652ddf8fb-Paper.pdf .

Duncan Alfers and Hermann Dinges. A normal approximation for beta and gamma tail probabilities. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete , 65:399-420, 1984. URL https://link.springer.com/content/pdf/10.1007/BF00533744.pdf .

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , 2017. URL https: //arxiv.org/pdf/1703.05449.pdf .

Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop, ITA 2018, San Diego, CA, USA, February 11-16, 2018 , pages 1-9. IEEE, 2018. doi: 10.1109/ITA.2018. 8503252. URL https://doi.org/10.1109/ITA.2018.8503252 .

Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost, 2019. URL https://arxiv.org/abs/1905.12849 .

Tongyi Cao and Akshay Krishnamurthy. Provably adaptive reinforcement learning in metric spaces. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 9736-9744. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 6ef1173b096aa200158bfbc8af3ae8e3-Paper.pdf .

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In Neural Information Processing Systems , 2017. URL https://arxiv.org/pdf/1703.07710.pdf .

Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12040-12051. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 649d45bf179296e31731adfd4df25588-Paper.pdf .

Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning , ICML'11, page 465-472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.

Persi Diaconis and David Freedman. Iterated random functions. SIAM Review , 41(1):45-76, 1999. doi: 10.1137/S0036144598338446. URL https://doi.org/10.1137/S0036144598338446 .

Omar Darwiche Domingues, Yannis Flet-Berliac, Edouard Leurent, Pierre Ménard, Xuedong Shang, and Michal Valko. rlberry - A Reinforcement Learning Library for Research and Education, 10 2021a. URL https://github.com/rlberry-py/rlberry .

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132 of Proceedings of Machine Learning Research , pages 578-598. PMLR, 16-19 Mar 2021b. URL https://proceedings.mlr.press/v132/domingues21a.html .

Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. Kernel-based reinforcement learning: A finite-time analysis. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2783-2792. PMLR, 18-24 Jul 2021c. URL https://proceedings.mlr.press/v139/domingues21a.html .

Randal Douc, Eric Moulines, Pierre Priouret, and Philippe Soulier. Markov chains . Springer, 2018.

Yonathan Efroni, Gal Dalal, Bruno Scherrer, and Shie Mannor. Multiple-step greedy policies in approximate and online reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/ paper_files/paper/2018/file/3f998e713a6e02287c374fd26835d87e-Paper.pdf .

Yonathan Efroni, Nadav Merlis, Mohammad Ghavamzadeh, and Shie Mannor. Tight regret bounds for model-based reinforcement learning with greedy policies. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper_files/paper/2019/file/25caef3a545a1fff2ff4055484f0e758-Paper.pdf .

Thomas S Ferguson. A bayesian analysis of some nonparametric problems. The annals of statistics , pages 209-230, 1973.

Claude-Nicolas Fiechter. Efficient reinforcement learning. In Conference on Learning Theory , 1994. URL http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid= 7F5F8FCD1AA7ED07356410DDD5B384FE?doi=10.1.1.49.8652&rep=rep1&type=pdf .

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alexander Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In Proceedings of the International Conference on Representation Learning (ICLR 2018) , Vancouver (Canada), 2018.

Ronan Fruit, Matteo Pirotta, Alessandro Lazaric, and Ronald Ortner. Efficient bias-span-constrained exploration-exploitation in reinforcement learning. In International Conference on Machine Learning , pages 1578-1586. PMLR, 2018.

Aurélien Garivier, Hédi Hadiji, Pierre Menard, and Gilles Stoltz. Kl-ucb-switch: optimal regret bounds for stochastic bandits from both a distribution-dependent and a distribution-free viewpoints. arXiv preprint arXiv:1805.05071 , 2018.

Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparametric Bayesian inference , volume 44. Cambridge University Press, 2017.

Ramen Ghosh and Jakub Marecek. Iterated function systems: A comprehensive survey, 2022.

Senlin Guo, Feng Qi, and Hari Srivastava. Necessary and sufficient conditions for two classes of functions to be logarithmically completely monotonic. Integral Transforms and Special Functions , 18:819-826, 11 2007. doi: 10.1080/10652460701528933.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_ files/paper/2015/file/148510031349642de5ca0c544f31b2ef-Paper.pdf .

Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado van Hasselt. Muesli: Combining improvements in policy optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 4214-4226. PMLR, 2021. URL http://proceedings.mlr.press/v139/hessel21a.html .

Junya Honda and Akimichi Takemura. An asymptotically optimal bandit algorithm for bounded support models. In Adam Tauman Kalai and Mehryar Mohri, editors, COLT , pages 67-79. Omnipress, 2010. ISBN 978-0-9822529-2-5. URL http://dblp.uni-trier.de/db/conf/ colt/colt2010.html#HondaT10 .

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research , 99:1563-1600, 2010. URL http://www.jmlr. org/papers/volume11/jaksch10a/jaksch10a.pdf .

Chi Jin, Zeyuan Allen-Zhu, Sébastien Bubeck, and Michael I. Jordan. Is Q-learning provably efficient? In Neural Information Processing Systems , 2018. URL https://arxiv.org/pdf/ 1807.03765.pdf .

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/ abs/1312.6114 .

Ziniu Li, Yingru Li, Yushun Zhang, Tong Zhang, and Zhi-Quan Luo. Hyperdqn: A randomized exploration method for deep reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=X0nrKAXu7g-.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/1509.02971 .

Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. In ICLR 2018 Conference , February 2018. URL https://www.microsoft.com/en-us/research/publication/ action-dependent-control-variates-policy-optimization-via-stein-identity/ .

Pierre Ménard, Omar Darwiche Domingues, Xuedong Shang, and Michal Valko. Ucb momentum qlearning: Correcting the bias without forgetting. In International Conference on Machine Learning , pages 7609-7618. PMLR, 2021.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop . 2013.

Chengzhuo Ni, Lin F Yang, and Mengdi Wang. Learning to control in metric space with optimal regret. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 726-733. IEEE, 2019.

Ian Osband and Benjamin Van Roy. Bootstrapped thompson sampling and deep exploration. CoRR , abs/1507.00300, 2015. URL http://arxiv.org/abs/1507.00300 .

Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 2701-2710. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/ osband17a.html .

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper/2016/file/ 8d8818c8e140c64c743113f563cf750f-Paper.pdf .

Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends® in Machine Learning , 11(5-6):355-607, 2019. ISSN 1935-8237. doi: 10.1561/2200000073. URL http://dx.doi.org/10.1561/2200000073 .

Iosif Pinelis. Optimum Bounds for the Distributions of Martingales in Banach Spaces. The Annals of Probability , 22(4):1679 - 1706, 1994. doi: 10.1214/aop/1176988477. URL https://doi.org/ 10.1214/aop/1176988477 .

Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 451ae86722d26a608c2e174b2b2773f1-Paper.pdf .

Remo Sasso, Michelangelo Conserva, and Paulo E. Rauber. Posterior sampling for deep reinforcement learning. CoRR , abs/2305.00477, 2023. doi: 10.48550/arXiv.2305.00477. URL https://doi. org/10.48550/arXiv.2305.00477 .

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. URL http://dblp.uni-trier.de/ db/journals/corr/corr1707.html#SchulmanWDRK17 .

Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/10a5ab2db37feedfdeaab192ead4ac0e-Paper.pdf .

Sean R. Sinclair, Siddhartha Banerjee, and Christina Lee Yu. Adaptive discretization for episodic reinforcement learning in metric spaces. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 3(3):1-44, dec 2019. doi: 10.1145/3366703. URL https://doi.org/10. 1145%2F3366703 .

Sean R. Sinclair, Siddhartha Banerjee, and Christina Lee Yu. Adaptive discretization in online reinforcement learning. Operations Research , 71(5):1636-1652, 2023. doi: 10.1287/opre.2022. 2396. URL https://doi.org/10.1287/opre.2022.2396 .

Maciej Skorski. Bernstein-type bounds for beta distribution. Modern Stochastics: Theory and Applications , 10(2):211-228, 2023. ISSN 2351-6046. doi: 10.15559/23-VMSTA223.

Zhao Song and Wen Sun. Efficient model-free reinforcement learning in metric spaces, 2019.

Malcolm J. A. Strens. A bayesian framework for reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning , ICML '00, page 943-950, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.

Sutton and A. Barto. Reinforcement Learning: an Introduction . MIT press, 1998.

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Mark Rowland, Michal Valko, and Pierre Ménard. Optimistic posterior sampling for reinforcement learning with few samples and tight guarantees. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 10737-10751. Curran Associates, Inc., 2022a. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 45e15bae91a6f213d45e203b8a29be48-Paper-Conference.pdf .

Daniil Tiapkin, Denis Belomestny, Eric Moulines, Alexey Naumov, Sergey Samsonov, Yunhao Tang, Michal Valko, and Pierre Menard. From Dirichlet to rubin: Optimistic exploration in RL without bonuses. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21380-21431. PMLR, 17-23 Jul 2022b. URL https://proceedings.mlr.press/v162/tiapkin22a.html .

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Chris J. Watkins and Peter Dayan. Q-learning. Machine Learning , 8(3-4):279-292, 1992. URL https://link.springer.com/content/pdf/10.1007/BF00992698.pdf .

Tzu-Tsung Wong. Generalized dirichlet distribution in bayesian analysis. Applied Mathematics and Computation , 97(2):165-181, 1998. ISSN 0096-3003. doi: https://doi.org/10.1016/ S0096-3003(97)10140-0. URL https://www.sciencedirect.com/science/article/pii/ S0096300397101400 .

Zhihan Xiong, Ruoqi Shen, Qiwen Cui, Maryam Fazel, and Simon S Du. Near-optimal randomized exploration for tabular markov decision processes. Advances in Neural Information Processing Systems , 35:6358-6371, 2022.

Fan Ye and Enlu Zhou. Information relaxation and dual formulation of controlled markov diffusions. IEEE Transactions on Automatic Control , 60(10):2676-2691, 2015.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , 2019. URL https://arxiv.org/pdf/1901.00210.pdf .

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. arXiv preprint arXiv:2004.10019 , 2020. ISSN 23318422. URL https://arxiv.org/pdf/2004.10019.pdf .

Appendix

Table of Contents

A Notation

Let ( X , X ) be a measurable space and P ( X ) be the set of all probability measures on this space. For p ∈ P ( X ) we denote by E p the expectation w.r.t. p . For random variable ξ : X → R notation ξ ∼ p means Law( ξ ) = p . We also write E ξ ∼ p instead of E p . For independent (resp. i.i.d.) random variables ξ ℓ ind ∼ p ℓ (resp. ξ ℓ i . i . d ∼ p ), ℓ = 1 , . . . , d , we will write E ξ ℓ ind ∼ p ℓ (resp. E ξ ℓ i . i . d ∼ p ), to denote expectation w.r.t. product measure on ( X d , X ⊗ d ) . For any x ∈ X we denote δ x a Dirac measure supported at point x .

For any p, q ∈ P ( X ) the Kullback-Leibler divergence KL( p, q ) is given by

For any p ∈ P ( X ) and f : X → R , pf = E p [ f ] . In particular, for any p ∈ ∆ d and f : { 0 , . . . , d } → R , pf = ∑ d ℓ =0 f ( ℓ ) p ( ℓ ) . Define Var p ( f ) = E s ′ ∼ p [ ( f ( s ′ ) -pf ) 2 ] = p [ f 2 ] -( pf ) 2 . For any ( s, a ) ∈ S , transition kernel p ( s, a ) ∈ P ( S ) and f : S → R define pf ( s, a ) = E p ( s,a ) [ f ] and Var p [ f ]( s, a ) = Var p ( s,a ) [ f ] .

Let ( X , ρ ) be a metric space, then the 1-Wasserstein distance between p, q ∈ P ( X ) is defined as W 1 ( p, q ) = sup f is 1 -Lipschitz E p [ f ] -E q [ f ] .

We write f ( S, A, H, T ) = O ( g ( S, A, H, T, δ )) if there exist S 0 , A 0 , H 0 , T 0 , δ 0 and constant C f,g such that for any S ≥ S 0 , A ≥ A 0 , H ≥ H 0 , T ≥ T 0 , δ < δ 0 , f ( S, A, H, T, δ ) ≤ C f,g · g ( S, A, H, T, δ ) . We write f ( S, A, H, T, δ ) = ˜ O ( g ( S, A, H, T, δ )) if C f,g in the previous definition is poly-logarithmic in S, A, H, T, 1 /δ .

For α, β > 0 , we define Beta( α, β ) as a beta distribution with parameters α, β . For set X such that | X | < ∞ define Unif( X ) as a uniform distribution over this set. In particular, Unif[ N ] is a uniform distribution over a set [ N ] .

For a measure p ∈ P ([0 , b ]) supported on a segment [0 , b ] (equipped with a Borel σ -algebra) and a number µ ∈ [0 , b ] we define

As the Kullback-Leibler divergence this quantity admits a variational formula by Lemma 18 of Garivier et al. [2018] up to rescaling for any u ∈ (0 , b )

B Description of RandQL

In this appendix we describe RandQL and Sampled-RandQL algorithms.

B.1 RandQL algorithm

We recall that n t h ( s, a ) = ∑ t -1 i =1 1 { ( s i h , a i h ) = ( s, a ) } is the number of visits of state-action pair ( s, a ) at step h before episode t .

Westart by initializing the ensemble of Q-values, the policy Q-values, and values to an optimistic value ˜ Q t,j h ( s, a ) = Q 1 h ( s, a ) = V 1 h ( s, a ) = r h ( s, a ) + r 0 ( H -h ) for all ( j, h, s, a ) ∈ [ J ] × [ H ] ×S ×A and r 0 > 0 some pseudo-rewards.

At episode t we update the ensemble of Q-values as follows, denoting by n = n t h ( s, a ) the count, w j,n ∼ Beta( H,n ) the independent learning rates,

where we defined the target ˚ Q t,j h ( s, a ) as a mixture between the usual target and some prior target with mixture coefficient ˚ w n,j ∼ Beta( n, n 0 ) and n 0 the number of prior samples,

It is important to note that in our approach, we need to re-inject prior targets to avoid forgetting their effects too quickly due to the aggressive learning rate. Indeed, the exponential decay of the prior effect can hurt exploration. We observe that the ensemble Q-value only averages uniformly over the last 1 /H fraction of the targets, as the expected value of the learning rate is E [ w j,n ] = H/ ( n + H ) . Since E [1 -˚ w j,n ] = n 0 ( n + n 0 ) the weight put on the prior sample in expectation, when we unfold

the definition of ˜ Q t +1 ,j h , is of order H/n · n/H · n 0 / ( n + n 0 ) = n 0 / ( n + n 0 ) , which is consistent with the usual prior forgetting in Bayesian learning. In Staged-RandQL , we avoid forgetting the prior too quickly by resetting the temporary Q-value to a prior value at the beginning of each stage.

The policy Q-values are obtained by taking the maximum among the ensemble of Q-values

The policy is then greedy with respect to the policy Q-values π t +1 h ( s ) ∈ arg max a ∈A Q t +1 h ( s, a ) and the value is V t +1 h ( s ) = max a ∈A Q t +1 h ( s, a ) . The complete RandQL procedure is detailed in Algorithm 2.

Algorithm 2 RandQL

1: Input: J ensemble size, number of prior transitions n 0 , prior reward r 0 .

2: Initialize: V h ( s ) = Q h ( s, a ) = ˜ Q j h ( s, a ) = r ( s, a )+ r 0 ( H -h ) , initialize counters n h ( s, a ) = 0 for h, s, a ∈ [ H ] ×S × A .

3: for t ∈ [ T ] do

4: for h ∈ [ H ] do

5: Play a h ∈ arg max a Q h ( s h , a ) .

6: Observe reward and next state s h +1 ∼ p h ( s h , a h ) .

7: Sample ˚ w j ∼ Beta( n, n 0 ) for n = n h ( s h , a h ) .

8: Build targets for all j ∈ [ J ]

9: Sample learning rates w j ∼ Beta( H,n ) .

10: Update ensemble Q -functions for all j ∈ [ J ]

11: Update policy Q -function Q h ( s h , a h ) := max j ∈ [ J ] ˜ Q j h ( s h , a h ) .

12: Update value function V h ( s h ) := max a ∈A Q h ( s h , a ) .

13: end for

14: end for

B.2 Sampled-RandQL algorithm

To create an algorithm that is more similar to PSRL , it is possible to select a Q-value at random from the ensemble of Q-values, rather than using the maximum Q-value

In this case we also need to update each Q-value in the ensemble with its corresponding target, see Osband and Van Roy [2015],

where ˜ V t,j h ( s ) = max a ∈A ˜ Q t,j h ( s, a ) . We name this new procedure Sampled-RandQL and detail it in Algorithm 3.

Algorithm 3 Sampled-RandQL

1: Input: J ensemble size, number of prior transitions n 0 , prior reward r 0 .

2: Initialize: V h ( s ) = Q h ( s, a ) = ˜ Q j h ( s, a ) = r ( s, a )+ r 0 ( H -h ) , initialize counters n h ( s, a ) = 0 for h, s, a ∈ [ H ] ×S × A .

3: for t ∈ [ T ] do

4: Sample ensemble index i ∼ Unif[ J ]

5: for h ∈ [ H ] do

6: Play a h ∈ arg max a Q h ( s h , a ) .

7: Observe reward and next state s h +1 ∼ p h ( s h , a h ) .

8: Sample ˚ w j ∼ Beta( n, n 0 ) for n = n h ( s h , a h ) .

9: Build targets for all j ∈ [ J ]

10: Sample learning rates w j ∼ Beta( H,n ) .

11: Update ensemble Q -functions for all j ∈ [ J ]

12: Update value function V h ( s h ) := max a ∈A ˜ Q j h ( s h , a ) for all j ∈ [ J ] .

13: Update policy Q -function Q h ( s h , a h ) := ˜ Q i h ( s h , a h ) .

14: end for

15: end for

C Weight Distribution in RandQL

In this section we study the joint distribution of weights over all targets in RandQL algorithm, described in details in Appendix B. To do it, we describe a very useful distribution, defined by Wong [1998].

Definition 5. We say that a random vector ( X 1 , . . . , X n , X n +1 ) has a generalized Dirichlet distribution GDir( α 1 , . . . , α n ; β 1 , . . . , β n ) if X n +1 = 1 -( X 1 + . . . + X n ) and ( X 1 , . . . , X n ) it has the following density over the simplex { x 1 , . . . , x n : x 1 + . . . + x n ≤ 1 } ,

for x 1 + . . . + x n ≤ 1 , x j ≥ 0 for j = 1 , . . . , n, and γ j = β j -α j +1 -β j +1 for j = 1 , . . . , n -1 and γ n = β n -1 . If we set x n +1 = 1 -( x 1 + . . . + x n ) then we obtain a homogeneous formula

Alternative characterization of generalized Dirichlet distribution could be given using independent beta-distributed random variables Z 1 , . . . , Z n with Z i ∼ Beta( α i , β i ) as follows

Therefore, for RandQL algorithm without prior re-injection we have the following formula

for n = n t h ( s, a ) and weights are defined as follows

And, moreover, we have that this vector of weights has the generalized Dirichlet distribution

That is, weights generated by the RandQL procedure is an inverted generalized Dirichlet random vector, that induces additional similarities with a usual posterior sampling approaches. Notably, that for H = 1 we recover exactly usual Dirichlet distribution, as in the setting of Staged-RandQL .

In the setting of the analysis, the main feature of this distribution is asymmetry in attitude to the order of components. In particular, the expectation of the prior weight W 0 n,j is ∏ n i =1 ( 1 -H i + H ) ∼ n -H that leads to too rapid forgetting of the prior information.

D Proofs for Tabular algorithm

D.1 Algorithm

In this section we describe in detail the tabular algorithms and the ways we will analyze them. We also provide some notations that will be used in the sequel.

Let n t h ( s, a ) be the number of visits of ( s, a, h ) (i.e., of the state-action pair ( s, a ) at step h ) at the beginning of episode t : n t h ( s, a ) = ∑ t -1 i =1 1 { ( s i h , a i h ) = ( s, a ) } . In particular, n T +1 h ( s, a ) is the number of visits of ( s, a, h ) after all episodes.

Let e k = ⌊ (1 + 1 /H ) k · H ⌋ be the length of each stage for any k ≥ 0 and, by convention, e -1 = 0 . We will say that at the beginning of episode t a triple ( s, a, h ) is in k -th stage if n t h ( s, a ) ∈ [ ∑ k -1 i =0 e i , ∑ k i =0 e i ) .

Let ˜ n t h ( s, a ) be the number of visits of state-action pair during the current stage at the beginning of episode t . Formally, it holds ˜ n t h ( s, a ) = n t h ( s, a ) -∑ k -1 i =0 e i , where k is the index of current stage.

Let κ > 0 be the posterior inflation coefficient, n 0 be the number of prior transitions, and J be the number of temporary Q -functions. Let ˜ Q t,j h be the j -th temporary Q-function and Q t h be the policy Q-function at the beginning of episode t . We initialize them as follows

We can treat this initialization as a setting prior over n 0 pseudo-transitions to artificial state s 0 with r 0 > 1 reward for each interaction.

For each transition we perform the following update of temporary Q-functions

where ˜ n = ˜ n t h ( s, a ) is the number of visits of ( s, a, h ) during the current stage at the beginning of episode t , k is the index of the current stage, and w k j, ˜ n is a sequence of independent beta-distribution random variables w k j, ˜ n ∼ Beta(1 /κ, ( ˜ n + n 0 ) /κ ) . Here we slightly abuse the notation by dropping the dependence of weights w k j, ˜ n on the triple ( h, s, a ) in order to simplify the exposition. In the case that the explicit dependence is required, we will call these weights as w k,h j, ˜ n ( s, a ) .

Next we define the stage update as follows

where k is the current stage. In other words, we update Q t +1 with temporary values of ˜ Q t +1 / 2 ,j , and then, if the change of stage is triggered, reinitialize ˜ Q t +1 ,j h ( s, a ) for all j . For episode t we will call k t h ( s, a ) the index of stage where Q t h ( s, a ) was updated (and k t h ( s, a ) = -1 if there was no update). For all t we define τ t h ( s, a ) ≤ t as an episode when the stage update happens. In other words, for any t the following holds

where τ t h ( s, a ) = 0 and e k = 0 if there was no updates. To simplify the notation we will omit dependence on ( s, a, h ) where it is deducible from the context.

To simplify the notation, we can extend the state space S by an additional state s 0 that will be purely technical and used in the proofs. This state has the prescribed value function V ⋆ h ( s 0 ) = r 0 ( H -h ) and could be treated as a absorbing pseudo-state with reward r 0 .

We notice that in this case we use e k samples to compute ˜ Q τ t h ( s,a )+1 / 2 ,j for k = k t h ( s, a ) . For this k we define ℓ i k,h ( s, a ) as a time of i -th visit of state-action pair ( s, a ) during k -th stage. Then we have the following decomposition

where we drop dependence on k and ( s, a, h ) in ℓ i to simplify notations, and use the convention s ℓ 0 k,h ( s,a ) h +1 = s 0 , and the following aggregated weights

We will omit the dependent on the stage index k when it is not needed for the statement. However, we notice that these vectors, for different stage k , will be independent.

By the properties of generalized Dirichlet distribution it is possible to show the following result

Lemma 3. For any fixed n > 0 , the random vector ( W 0 j,n , W 1 j,n , . . . , W n j,n ) has a Dirichlet distribution Dir( n 0 /κ, 1 /κ, . . . , 1 /κ ) .

Proof. Using the Dirichlet random variate generation from marginal beta distributions, it is sufficient to prove that for all i ∈ { 0 , . . . , n } , W n -i j,n,k = (1 -W n j,n,k - · · · -W n -i +1 j,n,k ) w k j,n -i -1 , with the convention w k j, -1 = 1 . This is trivial for i = 0 , as W n j,n,k = w k j,n -1 . Now, if this is true for some i , then, for i +1 ∈ { 0 , . . . , n } , we have

which finishes the proof.

Notably, the expression (3) shows a significant similarity between our method and OPSRL . It is the reason why we can call this method a model-free posterior sampling, where posterior sampling is performed over the model in a lazy and model-free fashion.

D.2 Concentration

Let β ⋆ : (0 , 1) × N → R + and β B , β conc , β : (0 , 1) → R + be some function defined later on in Lemma 4. We define the following favorable events

We also introduce the intersection of these events, G ( δ ) ≜ E ⋆ ( δ ) ∩ E B ( δ ) ∩ E conc ( δ ) ∩ E ( δ ) . We prove that for the right choice of the functions β ⋆ , β KL , β conc , β, β Var the above events hold with high probability.

Lemma 4. For any δ ∈ (0 , 1) and for the following choices of functions β,

In particular, P [ G ( δ )] ≥ 1 -δ/ 2

Proof. From the fact that s ℓ i h +1 are i.i.d. generated from p h ( s, a ) , Theorem 4, and union bound S × A × [ H ] it holds P [ E ⋆ ( δ )] ≥ 1 -δ/ 8 .

Next we fix all t, h, s, a, j , and denote n = e k t h ( s,a ) . First, we define a filtration of σ -algebras F τ that is sigma-algebra generated by all random variables appeared untill the update (2) in the episode t and step h , before newly generated random weights but after receiving new state s t h +1 . Formally, we can define it as follows

it holds that

where we drop dependence on state-action pairs everywhere where it is deducible from the context.

Consider a sequence ℓ 1 < . . . < ℓ n be an excursion of the state-action pair ( s, a ) at the step h . Each ℓ i is a stopping time w.r.t F t,h , so we can consider a stopped filtration (with a shift by 1 in indices) ˜ F i -1 = F ℓ i ,h . In other words, this filtration at time-stamp i -1 contains all the information that is available just before generation of random weights for i -th update of temporary Q-functions inside the last stage. We notice that under this definition we have

Next, we notice that the joint vector of weights follows the Dirichlet distribution, applying aggregation property and extending the filtration backward by adding fake transitions we can extend sum to n + n 0 summands defining s ℓ -i h +1 = s 0

Finally, we notice that marginals of Dirichlet random vector follow Beta distribution, therefore by Proposition 7 and union bound we conclude P [ E B ( δ )] ≥ 1 -δ/ 8 .

To show that P ( E conc ( δ )) > 1 -δ/ 8 , it is enough to apply Hoeffding inequality for a fixed number of samples e k used in empirical mean, and then use union bound of all possible values of ( s, a, h ) ∈ S × A × [ H ] and e k ∈ [ T ] .

Next, define the following sequence

It is easy to see that these sequences form a martingale-difference w.r.t filtration F t,h = σ { { ( s ℓ h ′ , a ℓ h ′ , π ℓ ) , ℓ < t, h ′ ∈ [ H ] } ∪ { ( s t h ′ , a t h ′ , π t ) , h ′ ≤ h } } . Moreover, | Z t,h | ≤ 2e r 0 H for all t ∈ [ T ] and h ∈ [ H ] . Hence, the Azuma-Hoeffding inequality implies

therefore P [ E ( δ )] ≥ 1 -δ/ 8 .

D.3 Optimism

In this section we prove that our estimate of Q -function Q t h ( s, a ) is optimistic, that is the event

holds with high probability on the event E ⋆ ( δ ) .

Define constants and

where Φ( · ) is a CDF of a normal distribution.

Proposition 1. Assume that J = ⌈ c J · log(2 SAHT/δ ) ⌉ , κ = 2 β ⋆ ( δ, T ) , r 0 = 2 , and n 0 = ⌈ ( c 0 +1+log 17 / 16 ( T )) · κ ⌉ . Then conditionally on E ⋆ ( δ ) the event

holds with probability at least 1 -δ/ 2 .

Proof. Let us fix t ∈ [ T ] , h ∈ [ H ] , ( s, a ) ∈ S × A , and j ∈ [ J ] . By Lemma 3, we have that the vector ( W i j,e k ,k ) i =0 ,...,e k has Dirichlet distribution. Note that V ⋆ h +1 ( s ℓ 0 h +1 ) = r 0 ( H -h -1) is an upper bound on V -function and the weight of the first atom is α 0 ≜ n 0 /κ ≥ c 0 +log 17 / 16 ( T ) for c 0 defined in (5). Define a measure ¯ ν e k = n 0 -1 e k + n 0 -1 δ V ⋆ h +1 ( s 0 ) + ∑ e k i =1 1 e k + n 0 -1 δ V ⋆ h +1 ( s ℓ i h +1 ) . Since p h V ⋆ h +1 ( s, a ) ≤ H -h -1 , we can apply Lemma 10 with a fixed ε = 1 / 2 conditioned on independent samples { s ℓ i h +1 } e k i =1 from p h ( s, a )

where Φ is a CDF of a normal distribution. Combining Lemma 12 and the event E ⋆ ( δ )

̂ h +1 h +1

̂ where ν e k = 1 e k ∑ e k i =1 δ V ⋆ ( s ℓ i ) , and, as a corollary

By taking κ = 2 β ⋆ ( δ, T ) we have a constant probability of being optimistic

Next, using a choice J = ⌈ log(2 SAHT/δ ) / log(1 / (1 -γ )) ⌉ = ⌈ c J · log(2 SAHT/δ ) ⌉

By a union bound we conclude the statement.

Next we provide a connection between E anticonc and E opt .

Proposition 2. It holds that E opt ⊆ E anticonc .

Proof. We proceed by a backward induction over h . Base of induction h = H +1 is trivial. Next by Bellman equations for Q t h and Q ⋆ h

where n = e k t h ( s,a ) and we drop dependence on k, t, h, s, a in ℓ i . By induction hypothesis we have V ℓ i h +1 ( s ′ ) ≥ Q ℓ i h +1 ( s ′ , π ⋆ ( s ′ )) ≥ Q ⋆ h +1 ( s ′ , π ⋆ ( s ′ )) = V ⋆ h +1 ( s ′ ) for any i , thus

By the definition of event E anticonc ( δ ) we conclude the statement.

Proposition 3 (Optimism) . Assume that J = ⌈ c J · log(2 SAHT/δ ) ⌉ , κ = 2 β ⋆ ( δ, T ) , r 0 = 2 , and n 0 = ⌈ ( c 0 +1+log 17 / 16 ( T )) · κ ⌉ , where c 0 is defined in (5) and c J is defined in (6). Then P ( E opt | E ⋆ ( δ )) ≥ 1 -δ/ 2 .

D.4 Regret Bound

Let us define the main event G ′ ( δ ) = G ( δ ) ∩E opt . On this event we have the following corollary that connects RandQL with OptQL with Hoeffding bonuses.

Define the following quantity

Corollary 1. Assume conditions of Proposition 3 hold. Let t ∈ [ T ] , h ∈ [ H ] , ( s, a ) ∈ S × A . Define k = k t h ( s, a ) and let ℓ 1 < . . . < ℓ e k be a excursions of ( s, a, h ) until the previous stage. Then on the event G ′ ( δ ) the following bound holds for k ≥ 0

where

Proof. The lower bound follows from the definition of the event E opt . For the upper bound we first apply the decomposition for Q t h ( s, a ) and the definition of event E B ( δ ) from Lemma 4

Then, by Bellman equations,

By the definition of event E conc ( δ ) we conclude the statement.

Lemma 5. Assume conditions of Proposition 3 hold. Then on event G ′ ( δ ) = G ( δ ) ∩ E opt , where G ( δ ) is defined in Lemma 4, the following upper bound on regret holds

where ξ t h = p h [ V ⋆ h +1 -V π t h +1 ]( s t h , a t h ) -[ V ⋆ h +1 -V π t h +1 ]( s t h +1 ) and B t h = B t h ( s t h , a t h ) · 1 { k t h ( s t h , a t h ) ≥ 0 } for B t h defined in Corollary 1.

Proof. We notice that on the event E opt the following upper bound holds

Next we analyze δ t h . By the choice of a t h = arg max a ∈A Q t h ( s t h , a ) , Corollary 1, and Bellman equations, we have

where k t h = k t h ( s t h , a t h ) , N t h = e k t h , ℓ i t,h is episode of the i -th visitation of the state-action pair ( s t h , a t h ) during the stage k t h , and additionally by the convention 0 / 0 = 0 . Let ξ t h = p h [ V ⋆ h +1 -V π t h +1 ]( s t h , a t h ) -[ V ⋆ h +1 -V π t h +1 ]( s t h +1 ) be a martingale-difference sequence, and B t h = B t h ( s t h , a t h ) 1 { N t h > 0 } then

and, as a result

Next we have to analyze the second term, following the approach by Zhang et al. [2020],

Notice that ∑ N t h i =1 1 { ℓ i t,h = q } ≤ 1 since all visitations are increasing in i , and, moreover, it turns to equality if and only if ( s q h , a q h ) = ( s t h , a t h ) and this visitation happens in stage k t h , where k t h is equal to the stage of episode q with respect to ( s q h , a q h , h ) . Since the sum is over all the next episodes with respect to stage of q , we have that the number of non-zero elements in the sum over t is bounded by (1 + 1 /H ) N t h . Thus

After a simple algebraic manipulations and using the fact that ζ t h ≤ δ t h ,

By rolling out the upper bound on regret (8) and using inequality (1 + 1 /H ) H -h ≤ e we have

Proof of Theorem 1. First, we notice that the event G ′ ( δ ) defined in Lemma 5, holds with probability at least 1 -δ by Lemma 4 and Proposition 3. Thus, we may assume that G ′ ( δ ) holds.

We start from the decomposition given by Lemma 5

The first term is upper bounded by e SAH 3 , since there is no more than H visits of each state-actionstep triple before the update for the first stage. The second term is bounded by ˜ O ( √ H 3 T ) by a definition of the event E ( δ ) in Lemma 4. To upper bound the last term we have to analyze the following sum

where

therefore by Cauchy inequality

where we used the definition of the previous stage k T +1 h ( s, a )

thus by Cauchy inequality and inequality log(1 + 1 /H ) ≥ 1 / (4 H ) for H ≥ 1

Using this upper bound, we have

Combining this upper bound with the previous ones, we conclude the statement.

E Proofs for Metric algorithm

E.1 Assumptions

In this section we proof Lemma 2 and Lemma 1.

Proof of Lemma 1. By the dual formula for 1-Wasserstein distance (see e.g. Section 6 of Peyré and Cuturi [2019]) we have

By Assumption 2 we have

Proof of Lemma 2. Let us proceed by a backward induction over h . For h = H + 1 we have Q ⋆ H +1 ( s, a ) = V ⋆ H +1 ( s ) = 0 , therefore they are 0 -Lipchitz.

Next we assume that have for any h ′ > h the statement of Lemma 2 holds. Then by Bellman equations

By Assumption 2 we can represent the action of the transition kernel as follows

Since by induction hypothesis V ⋆ h +1 is ∑ H h ′ = h +1 L h ′ -h F L r -Lipschitz and F h ( · , ξ h ) is L F -Lipschitz, therefore

To show that V ⋆ h is also Lipchitz, we have that there is some action a ⋆ equal to π ⋆ ( s ) or π ⋆ ( s ′ ) , such that

where in the end we used the sub-additivity assumption on metric over joint space (see Assumption 1).

E.2 Algorithm

Next we describe a simple non-adaptive version of our algorithm that works with metric spaces. We assume that for any ε > 0 we can compute a minimal ε -cover of state-action space N ε . 5

Next we will use the same notation but with state-action pairs replaces with balls from a fixed cover N ε . To unify the notation, we define ψ ε : S × A → N ε that maps any point ( s, a ) to any ball from ε -cover that contains it.

For any t, h we define B t h = ψ ε ( s t h , a t h ) . Next, let n t h ( B ) be a number of visits of ball B before the episode t : n t h ( B ) = ∑ t -1 k =1 1 { B k h = B } .

Let e k = ⌊ (1+1 /H ) k · H ⌋ be length of each stage for any k ≥ 0 and, by convention, e -1 = 0 . Wewill call that in the beginning of episode t a pair ( B,h ) is in k -th stage if n t h ( B ) ∈ [ ∑ k -1 i =0 e i , ∑ k i =0 e i ) .

Algorithm 4 Metric Net-Staged-RandQL

1: Input: inflation coefficient κ , J ensemble size, number of prior transitions n 0 ( k ) , prior reward r 0 , dicretization level ε .

2: Initialize: ε -net N ε , Q h ( B ) = ˜ Q j h ( B ) = r 0 H, initialize counters ˜ n h ( B ) = 0 for j, h, B ∈ [ J ] × [ H ] ×N ε , stage q h ( B ) = 0 , quantization map ψ ε : S × A → N ε .

3: for t ∈ [ T ] do

4: for h ∈ [ H ] do

5: Play a h ∈ arg max a Q h ( ψ ε ( s h , a )) and define B h = ψ ε ( s h , a h ) .

6: Observe reward and next state s h +1 ∼ p h ( s h , a h ) .

7: Sample learning rates w j ∼ Beta(1 /κ, ( ˜ n + n 0 ( q h ( B h )) /κ ) for ˜ n = ˜ n h ( B h ) .

8: Compute value function V h +1 ( s h +1 ) = max a ∈A Q h +1 ( ψ ε ( s h +1 , a )) .

9: Update temporary Q -values for all j ∈ [ J ]

10: Update counter n h ( B h ) := n h ( B h ) + 1

12: Update policy Q -values Q h ( B h ) := max j ∈ [ J ] ˜ Q j h ( B h ) .

˜ ˜ 11: if ˜ n h ( B h ) = ⌊ (1 + 1 /H ) q H ⌋ for q = q h ( B h ) is the current stage then

13: Reset temporary Q -values ˜ Q j h ( B h ) := r 0 H .

15: end if

14: Reset counter ˜ n h ( B h ) := 0 and change stage k h ( B h ) := k h ( B h ) + 1 .

16: end for

17: end for

Let ˜ n t h ( B ) be a number of visits of state-action pair during the current stage in the beginning of episode t . Formally, ˜ n t h ( B ) = n t h ( B ) -∑ k -1 i =0 e i , where k is an index of current stage.

Define κ > 0 be a posterior inflation coefficient, n 0 is a number of pseudo-transitions, and J as a number of temporary Q -functions. Let ˜ Q t,j h be a j -th temporary Q-value and Q t h be a policy Q-value at the beginning of episode t , defined over the ε -cover. We initialize them as follows

Additionally, we define to the value function as follows

Notice that we cannot precomute it as in the tabular setting, however, it is possible to use its values in lazy fashion.

For each transition we preform the following update of temporary Q-values over balls B ∈ N ε

where ˜ n = ˜ n t h ( B ) is the number of visits of ( B,h ) in the beginning of episode t , and w j, ˜ n is a sequence of independent beta-distribution random variables w j, n ∼ Beta(1 /κ, ( n + n 0 ) /κ ) .

Next we define the stage update as follows

where k is the current stage. A detailed description of the algorithm is presented in Algorithm 4.

For episode t we will call k t h ( B ) the index of stage where Q t h ( B ) were updated (and k t h ( B ) = -1 if there was no update). For all t we define τ t h ( B ) ≤ t as the episode when the stage update happens. In other words, for any t the following holds

where τ t h ( B ) = 0 and e k = 0 if there was no updates. To simplify the notation we will omit dependence on ( s, a, h ) where it is deducible from the context.

We notice that in this case we use e k samples to compute ˜ Q τ t h ( B )+1 / 2 ,j for k = k t h ( s, a ) . For this k we define ℓ i k,h ( s, a ) as the time of i -th visit of state-action pair ( s, a ) during k -th stage. Then we have the following decomposition

where we drop dependence on k and ( B,h ) in ℓ i to simplify notations, using the convention r h ( s ℓ 0 h , a ℓ 0 h ) = r 0 , V ℓ 0 h +1 ( s ℓ 0 h +1 ) = r 0 ( H -1) and the following aggregated weights

E.3 Concentration

Let β ⋆ : (0 , 1) × N × (0 , d max ) → R + and β B , β conc , β : (0 , 1) × (0 , d max ) → R + be some function defined later on in Lemma 6. We define the following favorable events

We also introduce the intersection of these events, G ( δ ) ≜ E ⋆ ( δ ) ∩ E B ( δ ) ∩ E conc ( δ ) ∩ E ( δ ) . We prove that for the right choice of the functions β ⋆ , β KL , β conc , β, β Var the above events hold with high probability.

Lemma 6. For any δ ∈ (0 , 1) and ε ∈ (0 , d max ) and for the following choices of functions β,

it holds that

In particular, P [ G ( δ )] ≥ 1 -δ/ 2 .

Proof. Let us describe the changes from the similar statement in Lemma 4.

Regarding event E ⋆ ( δ, ε ) , for any fixed ball B we have exactly the same structure of the problem thanks to Assumption 2 and a sequence of i.i.d. random variables ξ ℓ i h . Thus, Theorem 4 combined with a union bound over B ∈ N ε and H ∈ [ H ] concludes P ( E ⋆ ( δ, ε )) ≥ 1 -δ/ 8 .

The proof for the event E B ( δ, ε ) remains the almost the same, with two differences: the predictable weights slightly changed but the upper bound for them remain the same, and we have take a union bound not over all state-action pairs ( s, a ) ∈ S × A but all over balls B ∈ N ε .

To show that P ( E conc ( δ, ε )) ≥ 1 -δ/ 8 , let us fix B ∈ N ε , h ∈ [ H ] and e k ∈ [ T ] . Then we can define a filtration F t,h = σ { { ( s ℓ h ′ , a ℓ h ′ , π ℓ ) , ℓ < t, h ′ ∈ [ H ] } ∪ { ( s t h ′ , a t h ′ , π t ) , h ′ ≤ h } } and, since ℓ i k,h ( B ) are stopping times for all i = 1 , . . . , e k , we can define the stopped filtration ˜ F i = F ℓ i ,h .

Then we notice that X i = V ⋆ h +1 ( s ℓ i k,h ( B ) h +1 ) -p h V ⋆ h +1 ( s ℓ i k,h ( B ) h , a ℓ i k,h ( B ) h ) forms a martingale-difference sequence with respect to ˜ F i,h . Thus, by Azuma-Hoeffding inequality and a union bound we have P ( E conc ( δ, ε )) ≥ 1 -δ/ 8 .

The proof of P ( E ( δ )) ≥ 1 -δ/ 8 remains exactly the same as in Lemma 4.

E.4 Optimism

In this section we prove that our estimate of Q -function Q t h ( s, a ) is optimistic that is the event

holds with high probability on the event E ⋆ ( δ, ε ) .

Define constants

and slightly another constant

where Φ( · ) is a CDF of a normal distribution.

Proposition 4. Define a constant L = L r + L V (1 + L F ) . Assume that J = ⌈ ˜ c J · (log(2 HT/δ ) + log( N ε ) ⌉ , κ = 2 β ⋆ ( δ, T, ε ) , r 0 = 2 , and a prior count n 0 ( k ) = ⌈ ˜ n 0 + κ + εL H -1 · ( e k + ˜ n 0 + κ ) ⌉ dependent on the stage k , where ˜ n 0 = ( c 0 +1+log 17 / 16 ( T )) · κ .

Then on event E ⋆ ( δ, ε ) the following event

holds with probability at least 1 -δ/ 2 .

Remark 1. We notice that the obtained result is connected to the theory of Dirichlet processes.

First, let us define the Dirichlet process, following Ferguson [1973]. The stochastic process G , indexed by elements B of X , is a Dirichlet Process with parameter ν ( G ∼ DP( ν ) ) if

for any measurable partition ( B 1 , . . . , B d ) of X .

Let ̂ P n = 1 n ∑ n i =1 δ Z i be an empirical measure of an i.i.d. sample Z 1 , . . . , Z n ∼ P . Let ν be a finite (not necessarily probability) measure on X and ˜ P n ∼ DP( ν + n ̂ P n ) . Then we have the following representation for the expectations of a function f : X → R over a sampled measure ˜ P n (see Theorem 14.37 of Ghosal and Van der Vaart [2017] with σ = 0 for a proof)

where V n ∼ Beta( | ν | , n ) , Q ∼ DP( ν ) , and a vector ( W 1 , . . . , W n ) follows uniform Dirichlet distribution Dir(1 , . . . , 1) . If we take ν = n 0 · δ Z 0 for some Z 0 ∈ X such that f ( Z 0 ) = r 0 ( H -1) 6 , then by a stick-breaking process representation of the Dirichlet distribution we have

By taking an appropriate X and f we have that Proposition 4 could be interpret as a deriving a lower bound on the probability of P [ ˜ P n f ≥ Pf + εL | { Z i } n i =1 ] .

Proof. First for all, let us fix t ∈ [ T ] , h ∈ [ H ] and B ∈ N ε and, consequently, k = k t h ( B ) . Also, let fix j ∈ [ J ] . To simplify the notation in the sequel, define X 0 = r 0 ( H -1) and X i = V ⋆ h +1 ( F h ( s, a, ξ ℓ i h )) for i > 0 . Notice that X i for i > 0 is a sequence of i.i.d. random variables supported on [0 , H -h -1] .

By Lemma 3 we have ( W 0 j,e k ,k , . . . , W e k j,e k ,k ) ∼ Dir( n 0 ( k ) /κ, 1 /κ, . . . , 1 /κ ) . Then we use the aggregation property of Dirichlet distribution: there is a vector ( ˜ W -1 j , . . . , ˜ W e k j ) ∼ Dir(( n 0 ( k ) -˜ n 0 ) /κ, ˜ n 0 /κ, 1 /κ, . . . , 1 /κ ) such that

Next we are going to represent the Dirichlet random vector ˜ W by a stick breaking process (or, equivalently, represent via the generalized Dirichlet distribution)

where ξ j and ̂ W j are independent. Therefore, we have the final decomposition

By independence of ξ j and ̂ W j we have

We split our problem to lower bound the two separate probabilities.

Approximation error To deal with approximation error, we first of all notice that p h V ⋆ h +1 ( s, a ) ≤ H -1 , therefore we have

Next we assume that ε < ( H -1) /L , since ξ j ∼ Beta(( n 0 ( k ) -˜ n 0 ) /κ, ( e k + ˜ n 0 ) /κ ) , we may apply Alfers and Dinges [1984, Theorem 1.2']

where p = ( n 0 ( k ) -˜ n 0 -κ ) / ( e k + ˜ n 0 -κ ) and µ = εL/ ( H -1) . Since n 0 ( k ) = ⌈ ˜ n 0 + κ + εL H -1 · ( e k + ˜ n 0 + κ ) ⌉ , we have P [ T approx ≥ 0] ≥ 1 / 2 .

Stochastic error Since X 0 = r 0 ( H -1) is an upper bound on V -function, and we have that the weight of the first atom α 0 ≜ ˜ n 0 /κ -1 = c 0 +log 17 / 16 ( T ) -1 for c 0 defined in (11).

Define a measure ¯ ν e k = ˜ n 0 -κ e k + ˜ n 0 -κ δ X 0 + ∑ e k i =1 1 e k + n 0 -1 δ X i . Since p h V ⋆ h +1 ( s, a ) ≤ H -h -1 , we can apply Lemma 10 with a fixed ε = 1 / 2 conditioned on independent random variables X i

where Φ is a CDF of a normal distribution. By Lemma 12 and the event E ⋆ ( δ, ε )

where ̂ ν e k = 1 e k ∑ e k i =1 δ V ⋆ h +1 ( F ( s,a,ξ ℓ i h +1 )) , and, as a corollary

By taking κ = 2 β ⋆ ( δ, T, ε ) we have a constant probability of being optimistic for stochastic error

Overall, combining two lower bound for approximation and stochastic terms, we have

Next, using a choice J = ⌈ (log(2 HT/δ ) + log( N ε )) / log(1 / (1 -γ )) ⌉ = ⌈ ˜ c J · (log(2 HT/δ ) + log( N ε )) ⌉

By a union bound we conclude the statement.

Next we provide a connection between E anticonc and E opt .

Proposition 5. It holds E opt ⊆ E anticonc .

Proof. We proceed by a backward induction over h . Base of induction h = H +1 is trivial. Fix state-action pair ( s, a ) and let us call ( s ′ , a ′ ) a center of the ball ψ ε ( s, a ) that is the ball where ( s, a ) contains.

Next by the update formula for Q t h , and Bellman equations

where n = e k t h ( B ) and we drop dependence on k, t, h, s, a in ℓ i . By induction hypothesis we have V ℓ i h +1 ( s ′ ) ≥ Q ℓ i h +1 ( ψ ε ( s ′ , π ⋆ ( s ′ ))) ≥ Q ⋆ h +1 ( s ′ , π ⋆ ( s ′ )) = V ⋆ h +1 ( s ′ ) for any i , thus combining it with Lipchitz continuity of reward function and Q ⋆ , and the value of r h ( s ℓ 0 , a ℓ 0 ) = r 0 > r h ( s, a ) ,

Next we apply Lipschitz continuity of F h and V ⋆ h +1 and obtain

By the definition of event E anticonc we conclude the statement.

Proposition 6 (Optimism) . Define a constant L = L r + L V (1 + L F ) . Assume that J = ⌈ ˜ c J · (log(2 HT/δ ) + log( N ε ) ⌉ , κ = 2 β ⋆ ( δ, T, ε ) , r 0 = 2 , and a prior count n 0 ( k ) = ⌈ ˜ n 0 + κ + εL H -1 · ( e k + ˜ n 0 + κ ) ⌉ dependent on the stage k , where ˜ n 0 = ( c 0 +1+log 17 / 16 (2 e k )) · κ , c 0 is defined in (11), ˜ c J is defined in (12). Then P ( E opt | E ⋆ ( δ, ε )) ≥ 1 -δ/ 2 .

E.5 Regret Bounds

As in the tabular setting, we first connect our algorithm to the algorithm by Song and Sun [2019], using the following corollary. Define an event G ′ ( δ, ε ) = G ( δ, ε ) ∩ E opt .

Let us define the logarithmic term as follows

that has dependence of order O (log( TH/δ ) + log N ε ) .

Corollary 2. Fix ε ∈ (0 , L V /H ) and assume conditions of Proposition 6. Let t ∈ [ T ] , h ∈ [ H ] , B ∈ N ε . Define k = k t h ( B ) and let ℓ 1 < . . . < ℓ e k be a excursions of ( B,h ) till the end of the previous stage. Then on the event G ′ ( δ ) the following bound holds for k ≥ 0 and any ( s, a ) ∈ B

where

Proof. The lower bound follows from the definition of the event E opt . For the upper bound we first apply the decomposition for Q t h ( s, a ) and the definition of event E B ( δ, ε ) from Lemma 6

Additionally, by Bellman equations

Finally, the applications of event E conc ( δ, ε ) concludes the statement.

Let us define δ t h = V t h ( s t h ) -V π t h ( s t h ) and ζ t h = V t h ( s t h ) -V ⋆ h ( s t h ) . Lemma 7. Assume conditions of Proposition 6. Then on event G ′ ( δ, ε ) = G ( δ, ε ) ∩ E opt , where G ( δ, ε ) is defined in Lemma 6, the following upper bound on regret holds

Proof. As in the tabular setting, we notice that on the event E opt we can upper bound the regret in terms of δ t 1 .

Next we analyze δ t h . Since a t h = arg max a ∈A Q t h ( ψ ε ( s t h , a )) , we can use Corollary 2 and Bellman equations in the following way

where k t h = k t h ( B t h ) , N t h = e k t h , ℓ i t,h is an i -th visitation of the ball B t h during an stage k t h , and additionally by a convention 0 / 0 = 0 .

Define ξ t h = p h [ V ⋆ h +1 -V π t h +1 ]( s t h , a t h ) -[ V ⋆ h +1 -V π t h +1 ]( s t h +1 ) a martingale-difference sequence, and B t h = B t h ( k t h ) 1 { N t h > 0 } then

and, as a result

For the second term we may repeat arguments as in the proof of Lemma 5 and obtain

After a simple algebraic manipulations and using the fact that ζ t h ≤ δ t h

By rolling out the upper bound on regret (13) we have

Proof of Theorem 2. First, we notice that the event G ′ ( δ, ε ) defined in Lemma 7, holds with probability at least 1 -δ by Lemma 6 and Proposition 6. Thus, we may assume that G ′ ( δ, ε ) holds for ε > 0 that we will specify later.

The first term is upper bounded by 2e H 3 · N ε , since there is no more than H visits of each ball in ε -net before the update for the first stage. The second term is bounded by O ( √ H 3 Tβ max ( δ, ε )) by a definition of the event E ( δ ) in Lemma 6.

To analyze the last term, consider the following sum

where therefore

Notice that

thus from the Cauchy-Schwarz inequality

By the similar arguments we have

Using this upper bound, we have for L = L r +(1 + L F ) L V

Overall, for any fixed ε > 0 we have

Next we finally use that S × A have covering dimension d c that means N ε ≤ C N · ε -d c , thus our regret bound transforms as follows

By taking ε = T -1 / ( d c +2) we conclude the statement

F Adaptive RandQL

In this section we describe how to improve the dependence in our algorithm from covering dimension to zooming dimension, and describe all required notation.

F.1 Additional Notation

In this section we introduce an additional notation that is needed for introducing an adaptive version of RandQL algorithm for metric spaces.

Hierarchical partition Next, we define all required notation to describe an adaptive partition, as Sinclair et al. [2019, 2023]. Finally, we define the following general framework of hierarchical partition. Instead of balls, we will use a more general notion of regions that will induce a better structure from the computational point of view. We recall for any compact set A ⊆ S × A we call diam( A ) = max x,y ∈ A ρ ( x, y ) .

Definition 6. A hierarchical partition of S × A of a depth d > 0 is a collection of regions P d and their centers such that

Each region B ∈ P d is of the form S ( B ) ×A ( B ) , where S ( B ) ⊆ S , A ( B ) ⊆ A ;

P d is a cover of S × A : ⋃ B ∈P d B = S × A ;

For every B ∈ P d , we have diam( B ) ≤ d max · 2 -d ;

̸

Let B 1 , B 2 ∈ P d . If B 1 = B 2 then ρ (center( B 1 ) , center( B 2 )) ≥ d max · 2 -d ;

For any B ∈ P d , there exists a unique A ∈ P d -1 (called the parent of B ) such that B ⊆ A .

and, for d = 0 we define it as P 0 = {S × A} .

We call the tree generated by the structure of T = {P d } d ≥ 0 a tree of this hierarchical partition. The main example of this partition is the dyadic partition of S×A in the case of S = [0 , 1] d S , A = [0 , 1] d A and the metric induced by the infinity norm ρ (( s, a ) , ( s ′ , a ′ )) = max {∥ s -s ′ ∥ ∞ , ∥ a -a ′ ∥ ∞ } . For examples we refer to [Sinclair et al., 2023].

F.2 Algorithm

In this section we describe two algorithms: Adaptive-RandQL which is an adaptive metric counterpart of RandQL , and Adaptive-Staged-RandQL which is an adaptive metric counterpart of Staged-RandQL . First, we start from the notation and algorithmic parts that will be common for both algorithms.

Algorithms maintain an adaptive partition P t h of S × A , that is a sub-tree of an (infinite) tree of the hierarchical partition T = {P d } d ≥ 0 . We initialize P 1 h = {P 0 } , and then we refine the tree P t h be adding new nodes that corresponding to nodes of T . The leaf nodes of P t h represent the active balls, and for B ∈ P t h the set of its inactive parent balls is defined as { B ′ ∈ P t h | B ⊂ B ′ } . For any B ∈ P t h we define d ( B ) as a depth of B in the tree under a convention d ( S × A ) = 0 .

.

Additionally, we need to define so-called selection rule and splitting rule . For any state s ∈ S we define the set of all relevant balls as R t h ( s ) = { active b ∈ P t h | ( s, a ) ∈ B for some a ∈ A} Then for the current state s t h we define the current ball as B t h = arg max B ∈R t h ( s t h ) Q t h ( B ) and the corresponding action as a t h . To define the splitting rule we maintain the counters n t h ( B ) for all B ∈ P t h as a number of visits of a node B and all its parent nodes. Then we will perform splitting of the current ball B t h if √ d 2 max /n t h ( B t h ) ≤ diam( B t h ) . During splitting, we extend P t +1 h by its child nodes in the hierarchical partition tree T . For more details we refer to [Sinclair et al., 2023], up to small changes in notation. In particular, their constant ˜ C is equal to d max in our setting to make the construction exactly the same for both Adaptive-RandQL and Adaptive-Staged-RandQL algorithms.

Algorithm 5 Adaptive-RandQL

1: Input: ensemble size J , number of prior transitions n 0 , prior reward r 0 .

2: Initialize: Q h ( B ) = ˜ Q j h ( B ) = r 0 H, initialize counters n h ( s, a ) = 0 for h, s, a ∈ [ H ] ×S×A .

3: for t ∈ [ T ] do

4: for h ∈ [ H ] do

5: Compute B h = arg max B ∈R t h ( s h ) Q h ( B ) and play a h for ( s h , a h ) = center( B h ) ;

6: Observe reward and next state s h +1 ∼ p h ( s h , a h ) .

7: Sample ˚ w j ∼ Beta( n, n 0 ) for n = n h ( B h ) .

8: Compute value V h +1 ( s h +1 ) = max B ∈R t h ( s h +1 ) Q h +1 ( B ) .

9: Build targets for all j ∈ [ J ]

10: Sample learning rates w j ∼ Beta( H,n ) .

11: Update ensemble Q -functions for all j ∈ [ J ]

12: Update policy Q -function Q h ( s h , a h ) := max j ∈ [ J ] ˜ Q j h ( s h , a h ) .

13: Update counters n h ( B h ) := n h ( B h ) + 1 ;

14: If √ d 2 max /n h ( B h ) ≤ diam( B h ) , then refine partition B h (see Sinclair et al. [2023]).

15: end for

16: end for

Adaptive-RandQL This algorithm is an adaptive metric version of RandQL algorithm. We recall that for B ∈ P t h we define n t h ( B ) = ∑ t -1 i =1 1 { ( B i h ) is a parent of B } is the number of visits of the ball B and its parent balls at step h before episode t . We start by initializing the ensemble of Q-values, the policy Q-values, and values to an optimistic value ˜ Q t,j h ( B ) = Q 1 h ( B ) = V 1 h ( B ) = r 0 H for all ( j, h ) ∈ [ J ] × [ H ] and the unique ball in the partition B = S × A and r 0 > 0 some pseudo-rewards.

At episode t we update the ensemble of Q-values as follows, denoting by n = n t h ( B ) the count, w j,n ∼ Beta( H,n ) the independent learning rates,

where we defined the target ˚ Q t,j h ( s t h , a t h ) as a mixture between the usual target and some prior target with mixture coefficient ˚ w n,j ∼ Beta( n, n 0 ) and n 0 the number of prior samples,

For a discussion on prior re-injection we refer to Appendix B. The value function is computed on-flight by the rule V t h ( s ) = max B ∈R t h Q t h ( B ) .

The policy Q-values are obtained by taking the maximum among the ensemble of Q-values

The policy is then greedy with respect to the policy Q-values and selection rule ( s, π t +1 h ( s )) = center( B ) , where B = arg max B ∈R t +1 h Q t +1 h ( B ) . After the update of Q-values, algorithm verifies the splitting rule. If the splitting rule is triggered, then all new balls are defined by counter and Q-values of its parent. We notice that all Q-values could be efficiently computed on the nodes of the adaptive partition. The complete and detailed description is presented in Algorithm 6.

Adaptive-Staged-RandQL The notation for this algorithm is very close to Net-Staged-RandQL and we describe only differences between them. The main difference is a way to compute value V t h ( s ) = max B ∈R t h ( s ) Q t h ( B ) and policy ( s, π t h ( s )) = center( B ) for B =

Algorithm 6 Adaptive-Staged-RandQL

1: Input: inflation coefficient κ , J ensemble size, number of prior transitions n 0 ( k ) , prior reward r 0 .

2: Initialize: Q h ( B ) = ˜ Q j h ( B ) = r 0 H, initialize counters ˜ n h ( B ) = 0 for j, h, B ∈ [ J ] × [ H ] ×N ε , stage q h ( B ) = 0 .

3: for t ∈ [ T ] do

4: for h ∈ [ H ] do

5: Compute B h = arg max B ∈R t h ( s h ) Q h ( B ) and play a h for ( s h , a h ) = center( B h ) ;

6: Observe reward and next state s h +1 ∼ p h ( s h , a h ) .

7: Sample learning rates w j ∼ Beta(1 /κ, ( ˜ n + n 0 ( q h ( B h )) /κ ) for ˜ n = ˜ n h ( B h ) .

8: Compute value V h +1 ( s h +1 ) = max B ∈R t h ( s h +1 ) Q h +1 ( B ) .

9: Update temporary Q -values for all j ∈ [ J ]

10: Update counters ˜ n h ( B h ) := ˜ n h ( B h ) + 1 and n h ( B h ) := n h ( B h ) + 1 .

12: Update policy Q -values Q h ( B h ) := max j ∈ [ J ] ˜ Q j h ( B h ) .

11: if ˜ n h ( B h ) = ⌊ (1 + 1 /H ) q H ⌋ for q = q h ( B h ) is the current stage then

13: Reset temporary Q -values ˜ Q j h ( B h ) := r 0 H .

15: end if

14: Reset counter ˜ n h ( B h ) := 0 and change stage k h ( B h ) := k h ( B h ) + 1 .

16: If √ d 2 max /n h ( B h ) ≤ diam( B h ) , then refine partition B h (see Sinclair et al. [2023]).

17: end for

18: end for

arg max B ∈R t h ( s ) Q t h ( B ) . Additionally, all counters including temporary will move to the child nodes after splitting, as it performed in Adaptive-RandQL . The detailed description is presented in Algorithm 6.

F.3 Regret Bound

In this section we state the regret bounds for Adaptive-Staged-RandQL and derive a proof. The given proof shares a lot of similarities with the proof of Net-Staged-RandQL in the first half and to the proof of Adaptive-QL by Sinclair et al. [2023] in the second half.

We fix δ ∈ (0 , 1) , and the number of posterior samples

where ˜ c J = 1 / log(4 / (3 + Φ(1))) and Φ( · ) is the cumulative distribution function (CDF) of a normal distribution.

Additionally we select

where c 0 is an absolute constant defined in (5) (see Appendix D.3), κ is the posterior inflation coefficient and L = L r +(1 + L F ) L V is a constant. Next we restate the regret bound result for Adaptive-Staged-RandQL algorithm.

Theorem (Restatement of Theorem 3) . Consider a parameter δ ∈ (0 , 1) . Let κ ≜ 2(log(8 HC N /δ )+ d c log 2 (8 T/d max ) + 3 log(e π (2 T + 1))) , r 0 ≜ 2 . Then it holds for Adaptive-Staged-RandQL , with probability at least 1 -δ ,

where d z,h is the steph zooming dimension and we ignore all multiplicative factors in the covering dimension d c .

Proof. We divide the proof to four main parts, a little bit different proof of Staged-RandQL and Net-Staged-RandQL since we also need to apply clipping techniques.

Concentration events We can define (almost) the same set of events as in Appendix E.3, where union bound over balls is taken over all the hierarchical partition tree up to depth D that we define as T D .

To apply the union bound argument, we have to bound the size of T D . First, we notice that relation between centers of balls in each layer P d implies that there at least |P d | non-intersecting balls of radius d max · 2 -d -2 . Thus, the size of this sub-tree could be bounded as

using the relation between covering and packing numbers, see e.g. Lemma 4.2.8 by Vershynin [2018]. The only undefined quantity here is D , that can be upper-bounded given budget T . To do it, we apply Lemma B.2 by Sinclair et al. [2023] for any B ∈ P t h

Our goal is to find a value D such that P T +1 h ⊆ T D for any MDPs and correct interactions. To do it, we notice that it is equivalent to show that diam( B ) ≥ d max 2 -D , that could be guaranteed since

which implies that D = 1 + log 2 ( T ) is enough. Finally, since for the value of interest

we can define the β -functions as follows follows

and following line-by-line the proof of Lemma 6, for an event G ( δ ) = E ⋆ ( δ ) ∩E B ( δ, T ) ∩E conc ( δ, T ) ∩ E ( δ ) we have P ( G ( δ )) ≥ 1 -δ/ 2 .

Optimism Next, we state the required analog of Proposition 4. We can show that with probability at least 1 -δ/ 2 on the event E ⋆ ( δ ) the following event

under the choice J = ⌈ ˜ c J · (log(2 HT/δ ) + log( |T D | )) ⌉ , κ = 2 β ⋆ ( δ, T ) , r 0 = 2 , and a prior count

dependent on the stage k , where ˜ n 0 = ( c 0 + 1 + log 17 / 16 ( T )) · κ , L = L r + L V (1 + L F ) . In particular, the proof exactly the same as the proof of Proposition 4 for ε dependent on k . At the same time, it is possible to show that E anticonc implies

Indeed, in the proof of Proposition 5 we actively uses the bound ρ (( s ℓ i h , a ℓ i h ) , ( s, a )) ≤ ε . In the adaptive setting, we have to, at first, use an upper bound ρ (( s ℓ i h , a ℓ i h ) , ( s, a )) ≤ diam( B ℓ i h ) by a construction B ⊆ B ℓ i h , and then apply Lemma B.2 by Sinclair et al. [2023] by defining an upper bound

for k = k t h ( B ) for a particular ball B ∈ P t h in the case He k -k -H 2 ≥ 0 .

By combining event E opt and the event E B ( δ ) we can prove the same statement as Corollary 2.

Let t ∈ [ T ] , h ∈ [ H ] , B ∈ P t h . Define k = k t h ( B ) and let ℓ 1 < . . . < ℓ e k be a excursions of ( B,h ) till the end of the previous stage. Then on the event G ′ ( δ ) = G ( δ ) ∩ E opt the following bound holds for k ≥ 0 and for any ( s, a ) ∈ B

where

where k = k t h ( B t h ) and β max ( δ, T ) = max { β ⋆ ( δ, T ) , β B ( δ ) , β conc ( δ ) , β ( δ ) } . Also we can express this bound in terms of a diameter of B t h as follows

thus and we have

where we define ρ ( H,δ, L ) ≜ 7566e 2 H 3 / 2 ( β max ( δ, T )) 4 /d max +15 L .

As a additional corollary, we have for all t ∈ [ T ] , h ∈ [ H ]

where B ⋆ is a ball that contains a pair ( s, π ⋆ ( s )) .

This upper and lower bound have the similar structure as Lemma D.2 by Sinclair et al. [2023] and the rest of the proof directly follows [Sinclair et al., 2023].

Clipping techniques Next we introduce the required clipping techniques developed by Simchowitz and Jamieson [2019], Cao and Krishnamurthy [2020]. Definition 2 introduces the quantity gap h ( s, a ) = V ⋆ h ( s ) -Q ⋆ h ( s, a ) , and for any compact set B ⊆ S × A we define gap h ( B ) = min ( s,a ) ∈ B gap h ( s, a ) . Finally, we define clipping operator for any µ, ν ∈ R

In particular, this operator satisfies the following important property

Lemma 8 (Lemma E.2. of Sinclair et al. [2023]) . Suppose that gap h ( B ) ≤ ψ ≤ µ 1 + µ 2 for any ψ, µ 1 , µ 2 . Then

Now we apply this lemma to our update rules, producing a result similar to Lemma E.3 of Sinclair et al. [2023]. We notice that

Thus, denoting ψ = Q t h ( B t h ) -Q ⋆ h ( s t h , a t h ) and, by (18),

we apply Lemma 8 and obtain

for k t h = k t h ( B t h ) and B t h defined in (19).

Regret decomposition The rest of the analysis we preform conditionally on event G ′ ( δ ) = G ( δ ) ∩ E opt that holds with probability at least 1 -δ .

By defining δ t h = V t h ( s t h ) -V π t ( s t h ) and ζ t h = V t h ( s t h ) -V ⋆ h ( s t h ) we have

and, at the same time, by Bellman equations

where ξ t h = p h [ V ⋆ h +1 -V π t h +1 ]( s t h , a t h ) -[ V ⋆ h +1 -V π t h +1 ]( s t h +1 ) is a martingale-difference sequence. By (23) we have

where k t h = k t h ( B t h ) . Repeating argument of Lemma 5 and Zhang et al. [2020]

Using an upper bound ζ t h ≤ δ t h we have for any h ≥ 1

and, rolling out starting with h = 1 we have the following regret decomposition

Term ( A ) For this term we notice that for any fixed h the following event

that is guaranteed if

Thus, indicator can be equal to 1 no more than H log(2 T +3) times for any t ∈ [ T ] . As a result,

Term ( B ) Let us rewrite this term using a definition of clipping operator and use the definition of near-optimal set (see Definition 3)

Next we consider the summation for a fixed h . Here we follow Theorem F.3 by Sinclair et al. [2023] and obtain

where we applied an additional rescaling by a function ρ defined in (20).

Next we fix a constant r 0 > 0 and break a summation into two parts: r ≥ r 0 and r ≤ r 0 .

Case r ≤ r 0 . In this situation we have can apply (20)

Case r ≥ r 0 . In this situation we also apply (20) under the indicator function

To upper bound the last sum we repeat the argument of (14) and apply (16), using the fact that diam( B ) = r · ρ ( H,δ, L )

As a result, we have by (19)

and

Finally, by an arbitrary choice of r 0 and a definition of zooming dimension with a scaling ρ = ρ ( H,δ, L ) (Definition 4)

Term ( C ) For this term we just apply definition of the main event G ( δ ) ⊇ E ( δ ) and obtain

Final regret bound First, we notice that β max( δ,T ) = ˜ O ( d c ) , therefore we have

Taking r 0 = K -d z,h +1 / 2 for each h and summing the geometric series we conclude the statement.

G Deviation and Anti-Concentration Inequalities

G.1 Deviation inequality for K inf

For a measure ν ∈ P ([0 , b ]) supported on a segment [0 , b ] (equipped with a Borel σ -algebra) and a number µ ∈ [0 , b ] we recall the definition of the minimum Kullback-Leibler divergence

As the Kullback-Leibler divergence this quantity admits a variational formula.

Lemma 9 (Lemma 18 by Garivier et al., 2018) . For all ν ∈ P ([0 , b ]) , u ∈ [0 , b ) ,

moreover if we denote by λ ⋆ the value at which the above maximum is reached, then

Remark 2. Contrary to Garivier et al. [2018] we allow that u = 0 but in this case Lemma 9 is trivially true, indeed

Let ( X t ) t ∈ N ⋆ be i.i.d. samples from a measure ν supported on [0 , b ] . We denote by ̂ ν n ∈ P ([0 , b ]) the empirical measure ̂ ν n = ∑ n i =1 δ X i , where δ X i is a Dirac measure on X i ∈ [0 , b ] .

We are now ready to state the deviation inequality for the K inf by Tiapkin et al. [2022b] which is a self-normalized version of Proposition 13 by Garivier et al. [2018]. Notice that this inequality is stated in terms of slightly less general definition of K inf, however, the proof remains completely the same.

Theorem 4. For all ν ∈ P ([0 , b ]) and for all δ ∈ [0 , 1] ,

G.2 Anti-concentration Inequality for Dirichlet Weighted Sums

In this section we state anti-concentration inequality by Tiapkin et al. [2022a] in terms of slightly different definition of K inf .

Theorem 5 (Lower bound) . For any α = ( α 0 + 1 , α 1 , . . . , α m ) ∈ R m +1 ++ define p ∈ ∆ m such that p ( ℓ ) = α ℓ /α, ℓ = 0 , . . . , m , where α = ∑ m j =0 α j . Let ε ∈ (0 , 1) . Assume that α 0 ≥ c 0 ( ε ) + log 17 / 16 ( α ) for c 0 ( ε ) defined in (24) , and α ≥ 2 α 0 . Then for any f : { 0 , . . . , m } → [0 , b 0 ] such that f (0) = b 0 , f ( j ) ≤ b < b 0 / 2 , j ∈ { 1 , . . . , m } and µ ∈ ( pf, b 0 )

Next we formulate a simple corollary of Theorem 5, that slightly relaxes assumptions of this theorem under assumption µ < b ≤ b 0 / 2 .

Lemma 10. For any α = ( α 0 +1 , α 1 , . . . , α m ) ∈ R m +1 ++ define p ∈ ∆ m such that p ( ℓ ) = α ℓ /α, ℓ = 0 , . . . , m , where α = ∑ m j =0 α j . Also define a measure ¯ ν = ∑ m i =0 p ( i ) · δ f ( i ) .

Let ε ∈ (0 , 1) . Assume that α 0 ≥ c 0 ( ε ) + log 17 / 16 (2( α -α 0 )) for c 0 ( ε ) defined in (24) . The for any f : { 0 , . . . , m } → [0 , b 0 ] such that f (0) = b 0 , f ( j ) ≤ b ≤ b 0 / 2 , j ∈ [ m ] , and any µ ∈ (0 , b )

Proof. Assume that assumption α ≥ 2 α 0 holds.

Then we show that the Theorem 5 also holds for µ ≤ pf . First, we notice that for any γ > 0

By continuity of K inf in its second argument (see Theorem 7 by Honda and Takemura [2010]) we can tend γ to zero, and then use an equality K inf (¯ ν, pf ) = K inf (¯ ν, µ ) = 0 .

Next, assume α ≤ 2 α 0 . In this case we have pf ≥ b , thus for any 0 ≤ µ ≤ b

where we first apply a lower bound f ( j ) ≥ 0 for all j > 0 and f (0) = b 0 , and second apply a bound µ ≤ b 0 / 2 . Here we may apply the result of Alfers and Dinges [1984, Theorem 1.2'] and obtain the following lower bound

where we used α 0 /α > 1 / 2 .

G.3 Rosenthal-type inequality

In this section we state Rosenthal-type inequality for martingale differences by [Pinelis, 1994, Theorem 4.1]. The exact constants could be derived from the proof.

Theorem 6. Let X 1 , . . . , X n be a martingale-difference sequence adapted to a filtration {F i } i =1 ,...,n : E [ X i |F i ] = 0 . Define V i = E [ X 2 i |F i -1 ] . Then for any p ≥ 2 the following holds

where C 1 = 60e , C 2 = 60 .

Additionally, we need some additional lemma to use this inequality in our setting.

Definition 7. A random variable X is called sub-exponential with parameters ( σ 2 , b ) if the following tail condition holds for any t > 0

By Theorem 1 of Skorski [2023] we have for any ξ ∈ B ( α, β ) with β ≥ α and any t > 0

where

so ξ is ( α/ ( α + β ) 2 , 2 / (3( α + β ))) sub-exponential.

Lemma 11. Let X 1 , . . . , X n be a sequence of centred ( σ 2 , b ) sub-exponential random variables, not necessarily independent. Then for any p ≥ 2

Proof. By Fubini theorem we have for any η ≥ 0 : E [ η p ] = p ∫ ∞ 0 u p -1 P [ η ≥ u ]d u , thus for any a > 0 the following holds

By selecting a = max { √ 8 σ 2 log n, 8 b log n } we have

for any u ≥ a , thus

By the bounds on Gamma-function we have

Proposition 7. Let W 1 , . . . , W n be a sequence of Beta-distributed random variables W i ∼ Beta(1 /κ, ( n -1) /κ ) for κ > 0 . Let {F i } i ∈ [ n ] be a filtration such that W i is independent from F i -1 : E [ W i |F i -1 ] = E [ W i ] , and X 1 , . . . , X n be a sequence of bounded predictable random variables: E [ X i |F i -1 ] = X i , | X i | ≤ B .

Then with probability at least 1 -δ the following holds

Proof. First we notice that Z i = ( W i -E [ W i ]) · X i forms a martingale-difference sequence: E [ Z i |F i -1 ] = 0 . Therefore, we can apply Theorem 6

where V i = E [ Z 2 i |F i -1 ] = X 2 i Var( W i ) . We can easily upper bound the variance of Beta-distributed random variable and obtain

For the second term we apply Lemma 11 since W i are ( κ/n 2 , 2 κ/ (3 n )) -sub-exponential

Therefore we have

Next we turn from moments to tails. By Markov inequality with p = log(1 /δ )

H Technical Lemmas

Lemma 12. Let ν ∈ P ([0 , b ]) be a probability measure over the segment [0 , b ] and let ¯ ν = (1 -α ) δ b 0 + α · ν be a mixture between ν and a Dirac measure on b 0 > b . Then for any µ ∈ (0 , b )

Proof. By a variational formula for K inf (see Lemma 9)

Since ¯ ν is a mixture, we have for any λ ∈ [0 , 1 / ( b 0 -µ )]

Notice that max λ> 0 log(1 -λ ( b 0 -µ )) = 0 . Thus, maximizing each term separately over λ , we have

I Experimental details

In this section we detail the experiments we conducted for tabular and non-tabular environments. For all experiments we used 2 CPUs (Intel Xeon CPU 2 . 20 GHz), and no GPU was used. Each experiment took approximately one hour.

I.1 Tabular experiments

In our initial experiment, we investigated a simple grid-world environment.

0

5000

10000

15000

20000

25000

30000

episode

0

100000

200000

300000

400000

500000

600000

700000

800000

regret

name

Randomized QL

Randomized QL with stage

Sampled randomized QL

Environments For tabular experiments we use two environments.

The first one is a grid-world environment with 100 states ( i, j ) ∈ [10] × [10] and 4 actions (left, right, up and down). The horizon is set to H = 50 . When taking an action, the agent moves in the corresponding direction with probability 1 -ε , and moves to a neighbor state at random with probability ε = 0 . 2 . The agent starts at position (1 , 1) . The reward equals to 1 at the state (10 , 10) and is zero elsewhere.

The second one is a chain environment described by Osband et al. [2016] with L = 15 states and 2 actions (left or right). The horizon is equal to 30 , the probability of moving into wrong direction is equal to 0 . 1 . The agent starts in the leftmost state with reward 0 . 05 , also the largest reward is equal to 1 is the rightmost state.

Variations of randomized Q-learning First we compare the different variations of randomized Q-learning on grid-world environment. Precisely we consider:

RandQL a randomized version of OptQL , detailed in Appendix B.

Staged-RandQL a staged version of RandQL , described in Section 3.2.

Sampled-RandQL a version of RandQL which samples one Q-value function in the ensemble to act, described in Appendix B.

For these algorithms we used the same parameters: posterior inflation κ = 1 . 0 , n 0 = 1 /S prior sample (same as PSRL , see below), ensemble size J = 10 . We use a similar ensemble size as the one used for the experiments with OPSRL by Tiapkin et al. [2022a]. For Staged-RandQL we use stage of sizes ( (1 + 1 /H ) k ) k ≥ 1 without the H factor, in order to have several epochs per state-action pair even for few episodes.

The comparison is presented in Figure 2. We observe that RandQL and Sampled-RandQL behave similarly with slightly better performance for Sampled-RandQL . This is coherent with the experiment on the comparison between OPSRL and PSRL Tiapkin et al. [2022a] where the optimistic version performs worst than the fully randomized algorithm. We also note that even with the aggressive stage schedule, Staged-RandQL needs more episode to converge. We conclude that despite that stage simplifies the analysis, it artificially slows down the learning in practice.

To ease the comparison with the baselines, for the rest of the experiments we only use RandQL because of its similarity with OptQL .

Baselines We compare RandQL algorithm to the following baselines:

OptQL [Jin et al., 2018] a model-free optimistic Q-learning.

0

5000

10000

15000

20000

25000

30000

episode

0

100000

200000

300000

400000

500000

600000

regret

name

Greedy UCBVI

OptQL

PSRL

Randomized QL

RLSVI

UCBVI

0

2000

4000

6000

8000

10000

episode

0

2500

5000

7500

10000

12500

15000

17500

regret

name

Greedy UCBVI

OptQL

PSRL

Randomized QL

RLSVI

UCBVI

UCBVI [Azar et al., 2017] a model-based optimistic dynamic programming.

Greedy-UCBVI [Efroni et al., 2019] optimistic real-time dynamic programming.

PSRL [Osband et al., 2013] model-based posterior sampling.

RLSVI [Russo, 2019] model-based randomized dynamic programming.

The selection of parameters can have a significant impact on the empirical regrets of an algorithm. For example, adjusting the multiplicative constants in the bonus of UCBVI or the scale of the noise in RLSVI can result in vastly different regrets. To ensure a fair comparison between algorithms, we have made the following parameter choices:

For bonus-based algorithm, UCBVI , OptQL we use simplified bonuses from an idealized Hoeffding inequality of the form

As explained by Ménard et al. [2021], this bonus does not necessarily result in a true upperconfidence bound on the optimal Q-value. However, it is a valid upper-confidence bound for n t h ( s, a ) = 0 which is important in order to discover new state-action pairs.

For RLSVI we use the variance of Gaussian noise equal to simplified Hoeffding bonuses described above in (25).

For PSRL , we use a Dirichlet prior on the transition probability distribution with parameter (1 /S, . . . , 1 /S ) and for the rewards a Beta prior with parameter (1 , 1) . Note that since the reward r is not necessarily in { 0 , 1 } we just sample a new randomized reward r ′ ∼ Ber( r ) accordingly to a Bernoulli distribution of parameter r , to update the posterior, see Agrawal and Goyal [2013].

Results Figure 3 shows the result of the experiments. Overall, we see that RandQL outperforms OptQL algorithm on tabular environment, but still degrades in comparison to model-based approaches, that is usual for model-free algorithms in tabular environments. Indeed, as explained by Ménard et al. [2021], using a model and backward induction allows new information to be more quickly propagated. For example UCBVI needs only one episode to propagate information about the last step h = H to the first step h = 1 whereas OptQL or RandQL need at least H episodes. But as counterpart, RandQL has a better time-complexity and space- complexity than model based algorithm, see Table 2.

I.2 Non-tabular experiments

The second experiment was performed on a set of two dimensional continuous environments [Domingues et al., 2021a] with levels of increasing exploration difficulty.

Environment We use a ball environment with the 2-dimensional unit Euclidean ball as state-space S = { s ∈ R 2 , ∥ s ∥ 2 ≤ 1 } and of horizon H = 30 . The action space is a list of 2-dimensional vectors A = { [0 . 0 , 0 . 0] , [ -0 . 05 , 0 . 0] , [0 . 05 , 0 . 0] , [0 . 0 , 0 . 05] , [0 . 0 , -0 . 05] } that can be associated with the action of staying at the same place, moving left, right, up or down. Given a state s h and an action a h the next state is

where z h ∼ N ([0 , 0] , I 2 ) is some independent Gaussian noise with zero mean and identity covariance matrix and proj B is the euclidean projection on the unit ball S . The initial position s 1 = σ 1 z 1 with z 1 ∼ N ([0 , 0] , I 2 ) and σ 1 = 0 . 001 , is sampled at random from a Gaussian distribution. The reward function independent of the action and the step

where s ′ = [0 . 5 , 0 . 5] ∈ S is the reward center and c > 0 is some smoothness parameter. We distinguish 3 levels by increasing exploration difficulty:

Level 1 , dense reward and small noise. The smoothness parameter is c = 0 . 5 · √ 2 ≈ 0 . 71 and the transition standard deviation is σ = 0 . 01 .

Level 2 , sparse reward and small noise. The smoothness parameter is c = 0 . 2 and the transition standard deviation is σ = 0 . 01 .

Level 3 , sparse reward and large noise. The smoothness parameter is c = 0 . 2 and the transition standard deviation is σ = 0 . 025 .

RandQL algorithm Among the different versions of RandQL for continuous state-action space, see Section 4, we pick the Adaptive-RandQL algorithm, described in Appendix F, as it is the closest version to the Adaptive-QL algorithm. It combines the RandQL algorithm and adaptive discretization. For Adaptive-RandQL we used an ensemble of size J = 10 ≈ log( T ) , κ = 10 ≈ log( T ) and a prior number of samples of n 0 = 0 . 33 . Note that we increased the number of prior samples in comparison to the tabular case as explained in Section 4.

Baselines We compare Adaptive-RandQL algorithm to the following baselines:

Adaptive-QL [Sinclair et al., 2019, 2023], an adaptation of OptQL algorithm to continuous state-space thanks to adaptive discretization;

Kernel-UCBVI [Domingues et al., 2021c], a kernel-based version of the UCBVI algorithm;

DQN [Mnih et al., 2013], a deep RL algorithm;

BootDQN [Osband and Van Roy, 2015], a deep RL algorithm with an additional exploration given by bootstraping several Q-networks;

For Adaptive-QL and Kernel-UCBVI baselines we employ the same simplified bonuses (25) used for the tabular experiments. For Kernel-UCBVI we used Gaussian kernel of bandwidth 0 . 025 and the representative states technique, with 300 representative states, described by Domingues et al. [2021c].

For DQN and BootDQN we use as netwrok a 2-layer multilayer perceptron (MLP) with hidden layer size equals to 64 . For exploration, DQN utilizes ε -greedy exploration with coefficient annealing from 1 . 0 to 0 . 1 during the first 10 , 000 steps. For BootDQN we use ensemble of 10 heads and do not use ε -greedy exploration.

0

2500

5000

7500

10000

12500

15000

17500

20000

episode

0

50000

100000

150000

200000

250000

300000

cumultative reward

name

Adaptive Randomized QL

Adaptive Q-Learning

Bootstrap DQN

DQN

Kernel UCBVI

0

2500

5000

7500

10000

12500

15000

17500

20000

episode

0

20000

40000

60000

80000

100000

120000

cumultative reward

name

Adaptive Randomized QL

Adaptive Q-Learning

Bootstrap DQN

DQN

Kernel UCBVI

0

2500

5000

7500

10000

12500

15000

17500

20000

episode

0

20000

40000

60000

80000

100000

120000

cumultative reward

name

Adaptive Randomized QL

Adaptive Q-Learning

Bootstrap DQN

DQN

Kernel UCBVI

Results Figure 4 shows the results of non-tabular experiments. Overall, we see that Adaptive-RandQL outperforms Adaptive-QL in all environments, especially in the sparse reward setting. However, we see that model-based algorithm is much more sample efficient than model-free algorithm, as it was shown by Domingues et al. [2021c]. This is connected to low dimension of the presented environment, where the difference in theoretical regret bounds is not so large. However, this performance come at the price of 3 -times larger time complexity, see Table 3.

Regarding the comparison to neural-network based algorithms, we see that approaches based on adaptive discretization always outperforms DQN and BootDQN on an environment with non-sparse rewards. We connect this phenomenon to the fact that neural network algorithms are solving two problems at the same time: exploration and optimization, whereas discretization-based approaches solve only exploration problem.

In the setup of sparse rewards it turns out that neural network-based approaches are competitive with Adaptive-QL and Adaptive-RandQL . Notably, DQN shows itself as the worst one, whereas Adaptive-RandQL and BootDQN show similar performance, additionally justifying exploration effect of ensemble learning and randomized exploration.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Randomized Q-learning (RandQL), a novel model-free algorithm for regret minimization in episodic MDPs. Leveraging a new idea of learning rate randomization, two respect algorithms are proposed for tabular and non-tabular metric space settings, achieving near-optimal regret upper bound. Proof-of-concept experiments are conducted.\n\nSTRENGTHS:\n- The concept of learning rate randomization is very interesting, which enables the algorithm to explore without bonuses. \n\n- Near-optimal regret upper bounds are obtained, showing the efficiency of proposed algorithms.\n\n- Empirical results show that the algorithm outperforms the existing model-free approach (OptQL) and support their theoretical results.\n\nWEAKNESSES:\n- The experiments are too simple. Although the theoretical contribution of this paper has outweighed the limited empirical verification and made this paper relatively solid, conducting additional experiments on more complex environments would be advantageous and could further enhance the overall research.\n\n- I was unable to find an explicit explanation of the intuition of the value of $J$. It would be helpful to understand the implications if $J$ is too small or too large. I noticed that you set $J$ to be of logarithmic order, but a high-level explanation of this choice would be beneficial.\n\n- The policy $Q$ value $\\overline{Q}_h(s_h,a_h)$ is computed by taking the maximum of $J$ temporary $Q$-values. Will it incur a large estimation error? Recall that in double Q-learning, one common trick is to take a minimum of two networks in order to reduce the estimation error. Although I am not sure how this issue affects your theoretical analysis, this seems a real problem in practice.\n\n- The Net-Staged-RandQL replies on the exact computation of covering, which may be considered a drawback. However, it is worth noting that many existing works also depend on this computation, suggesting that it could be an acceptable requirement.\n\nQUESTIONS:\nWhile most questions are shown in the \"weaknesses\" section, there are a few additional smaller questions:\n\n- In Line 141, shouldn't $\\overline{V}_h^{n+1}$ be computed from $\\overline{Q}^{n+1}_h$? Is it a typo?\n\n- What is the intuition of setting $r_0=2$ (e.g., in line 210)?\n\nLIMITATIONS:\nNo negative societal impact was identified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to develop computationally efficient, posterior sampling-style, model-free RL algorithms. They propose a family of algorithms based on the idea of learning rate randomization, which they show are able to provably learn and mimic to some degree the updates of posterior sampling. In the tabular setting, they show that the regret of this approach scales with $O(\\sqrt{H^5 SAT})$. They also extend the approach to RL in metric spaces, and show that their algorithm is provably efficient in that setting as well.\n\nSTRENGTHS:\n1. The algorithmic technique (exploration via learning rate randomization) presented here is interesting and to my knowledge novel. A common technique in RL is to add bonuses to the value function estimates to induce exploration via optimism. A related approach inspired from posterior sampling is to randomize the rewards in training, which has been shown to induce exploration and is provably efficient. Rather than adding bonuses or randomizing rewards, this paper instead induces exploration by updating the $Q$-function at each step with a random learning rate with carefully chosen distribution. They show that the resulting $Q$ function has a form very similar to that which would be generated by posterior sampling, and that that the algorithm is provably efficient. \n2. To my knowledge there are no existing works which propose provably efficient posterior sampling-inspired algorithms for RL in metric spaces, so the results in this setting are the first of their kind.\n3. The technique of learning rate randomization proposed here seems like it could be implemented in a straightforward way in deep RL algorithms, and may therefore have practical implications (though this is not explored in this work---some experiments are given, but these are restricted to gridworld environments).\n4. Experimental results demonstrate that on gridworld MDPs the approach proposed here performs better than OptQL (but worse than model-based algorithms).\n\nWEAKNESSES:\n1. This work is motivated by noting that “a provably tractable model-free posterior sampling algorithm has remained a challenge”. However, I’m not entirely sure this is correct. The RLSVI algorithm of (Russo, 2019) is inspired by posterior sampling, is computationally efficient, and is model free (in that it never requires computing or storing an estimate of the transition dynamics). While I do think the algorithmic techniques are interesting and the extension to the metric RL setting is novel, it’s not clear to me what the contributions of this paper are beyond that.\n2. The guarantee obtained in the metric RL setting does not scale with the zooming dimension, but instead scales with the ambient dimension. While this is optimal in the worst-case, in general one would hope to see a scaling in the zooming dimension for learning in metric spaces (as is shown, for example, in (Sinclair et al., 2022)). \n3. Furthermore, in the metric RL setting the proposed approach is not in general computationally efficient (as the computational complexity scales with $N_\\epsilon$, which in general could be exponentially large in the dimension). In contrast, the metric RL algorithm of (Sinclair et al., 2022) is computationally efficient. Both this and the previous issue arise because the algorithm applies a fixed rather than adaptive discretization technique.\n4. The assumptions required in the metric RL setting (in particular Assumptions 2 and 3) are somewhat stronger than what has been required in previous work on metric RL (Sinclair et al., 2022). I do not believe the extra assumptions are that restrictive, but some more discussion on them would be helpful.\n5. It would greatly strengthen this paper if experiments could be provided using learning rate randomization on deep RL benchmarks, and illustrating that this approach is able to effectively induce exploration in such settings.\n6. It would also be helpful to include the algorithm of (Russo, 2019) in the experimental evaluation of Section 5.\n\nQUESTIONS:\nPlease clarify the contributions of this paper and why the algorithm of (Russo, 2019) does not satisfy the desiderate (in the tabular setting).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
The paper introduces a novel algorithm for regret minimization in episodic Markov Decision Processes. The proposed algorithm, RandQL, achieves optimistic exploration without using bonuses, relying on learning rate randomization. The authors provide theoretical analysis of the algorithm's regret bound and demonstrate its effectiveness in experiments.

STRENGTHS:
1. The paper presents a novel algorithm that achieves optimistic exploration with learning rate randomization.
2. Both theoretical analysis of the algorithm and experimental results are provided with discussions to compare with baseline methods.

WEAKNESSES:
1. Lack of baseline comparison: using ensembles of Q values is not new, for example, bootstrapped DQN [1]. It would be an essential baseline method to show the potential difference in learning rate randomization compared with other ensemble methods.
2. The experimental section would benefit from a more complex setting other than the grid world, for example, the chain environment in [1] which requires some “deep” exploration.

[1] Deep Exploration via Bootstrapped DQN. Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy

QUESTIONS:
1. If the learning rate randomization is to approximate the posterior distribution, how is it different from other ensemble methods like bootstrapped DQN[1], which also uses multiple Q values to estimate the posterior distribution?
2. The proposed method does not necessarily need to be restricted in the tabular case, is it possible to extend it to more practical scenarios since its advantage also comes from computation?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled Model-free Posterior Sampling via Learning Rate Randomization proposes a novel model-free algorithm called Randomized Q-learning (RandQL) for regret minimization in episodic Markov Decision Processes (MDPs). The algorithm introduces the concept of learning rate randomization to achieve optimistic exploration without using bonuses. The performance of RandQL is analyzed in both tabular and non-tabular metric space settings, and it outperforms existing approaches in baseline exploration environments.\n\nThe authors address the challenge of sample complexity in model-free algorithms where the agent has no access to a simulator. They propose RandQL, a tractable model-free algorithm that achieves exploration without using bonuses. RandQL updates an ensemble of Q-values using Q-learning with Beta distributed step-sizes, introducing noise similar to the posterior sampling approach. The ensemble of Q-values can be seen as posterior samples, and RandQL chooses among these samples in an optimistic fashion. The authors analyze the regret bounds of RandQL in both tabular and metric state-action spaces.\n\nThe paper highlights that Bayesian-based exploration techniques have shown superior empirical performance compared to bonus-based exploration, but most theoretical studies have focused on model-based algorithms. The authors aim to extend the posterior sampling approach to a provably efficient model-free algorithm that matches the empirical performance of its model-based counterparts. Previous attempts at model-free posterior sampling algorithms have not been computationally tractable, making this a challenge to overcome.\n\nSTRENGTHS:\n1. The idea of relating learning rate randomization to PSRL Dirichlet sampling is interesting.\n\n2. The first computationally tractable model-free (optimistic) posterior sampling algorithm with regret guarantees in tabular setting and metric RL setting.\n\n3. Experiments in finite and continuous MDPs that show that RandQL is competitive with model-based and model-free baselines while keeping a low time-complexity.\n\nWEAKNESSES:\nAlthough the author claims the algorithm do not require bonus, the learning rate randomization still needs (s, a) counting. Counting is not easy to generalise to practical RL with function approximation.\n\nQUESTIONS:\n1. RLSVI related algorithm also do not require bonus but can be can be easily compatible with complex function approximation. Also it utilise all history data in previous theoretical analysis (which can be thought as a definition of model-based), it still only track the optimal value function. Could you give more detailed comparison to this line of work in both computation and sample complexity consideration.\n\n2. See weakness.\n\nI could consider increasing score if the questions are properly answered.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new model-free Q-learning algorithm that utilizes a randomized learning rate. The effectiveness of this algorithm is theoretically validated by demonstrating a promising regret bound of $\\widetilde{O}(\\sqrt{H^{5} SA T})$ under the tabular setting. Furthermore, a theoretical bound is established for the metric state-action space in this paper.\n\nSTRENGTHS:\n- Introduces a new model-free algorithm with random exploration.\n- Provides provable guarantees under both tabular and non-tabular settings.\n\nWEAKNESSES:\nThe research significance is not well addressed. What are the benefits of eliminating the exploration bonus? What are the promising parts (for future work to develop new methods and apply to practical tasks) compared with alternative methods (e.g., RLSVI, UCB)?\n\nQUESTIONS:\n1. The idea of using multiple ensembles and maximizing to incentivize exploration has been mentioned in previous works. See e.g., [R1, R2] (the reference of [R1] is missing).  The reviewer wants to know the difference and whether this is a general technique that theorists and practitioners should follow in the future. \n\n2. Why is RLSVI viewed as a model-based algorithm in the introduction? What are the computational benefits compared with RLSVI?\n\n\n[R1] Ishfaq, Haque, et al. \"Randomized exploration in reinforcement learning with general value function approximation.\" *International Conference on Machine Learning*. PMLR, 2021.\n\n[R2] Xiong, Zhihan, et al. \"Near-Optimal Randomized Exploration for Tabular Markov Decision Processes.\" *Advances in Neural Information Processing Systems* 35 (2022): 6358-6371.\n\nLIMITATIONS:\nLimitations:\n\n- The claim in the introduction that \"Although empirical evidence suggests that model-based algorithms are more sample efficient than model-free algorithms...\" is somewhat misleading. In the field of deep RL, there is no definitive conclusion yet. Please consider revising this argument.\n\n- The vanilla Q-learning algorithm is rarely used in practice. Practitioners typically utilize the target Q-learning algorithm. Please refer to [R3] for further discussion. It is encouraged to investigate this algorithm with a randomized learning rate.\n\n[R3] Zanette, Andrea, and Martin Wainwright. \"Stabilizing Q-learning with Linear Architectures for Provable Efficient Learning.\" *International Conference on Machine Learning*. PMLR, 2022.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm, Randomized Q-learning (RandQL), for regret minimization in episodic Markov Decision Processes (MDPs). RandQL is the first manageable model-free posterior sampling-based algorithm. The authors study RandQL's performance in both tabular and non-tabular metric space settings. In tabular MDPs, RandQL gives a regret bound of $O(\\sqrt{H^5SAT})$, where $H$ is the planning horizon, $S$ the number of states, $A$ the number of actions, and $T$ the number of episodes. RandQL has a regret bound of order $O(\\sqrt{H^5T^{(d_c+1)/(d_c+2)}})$, where $d_c$ is the covering dimension, in a metric state-action space. Interestingly, RandQL achieves optimistic exploration without bonuses and uses a new idea of learning rate randomization. The authors also show that RandQL does better than current methods on baseline exploration environments in their empirical study.\n\nSTRENGTHS:\n1. The introduction of learning rate randomization is novel and interesting, that lets RandQL achieve optimistic exploration without bonuses. This indeed reduce the memory cost.\n\n2. RandQL does well in both tabular and non-tabular metric space settings. In both cases, the authors show that it gives good regret bounds. That's a pretty solid result.\n\nWEAKNESSES:\nIn the experiments, the performance gap between model-free algorithms and model-based algorithms seems too large. It would be better to provide stronger evidence in experiments of the benefits of RandQL such as lower computation cost, lower memory cost, or possibly has a closer performance in a more complex environment compared to model-based methods.\n\nQUESTIONS:\nCan authors discuss why the randomized learning rate can work in a high-level idea? For example, is it possible that this randomization brings some optimistic?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}