Title: Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Thompson Sampling

Reinforcement Learning

Bayesian Regret
Abstract: In this paper, we prove state-of-the-art Bayesian regret bounds for Thompson Sampling in reinforcement learning in a multitude of settings. We present a refined analysis of the information ratio, and show an upper bound of order $\widetilde{O}(H\sqrt{d_{l_1}T})$ in the time inhomogeneous reinforcement learning problem where $H$ is the episode length and $d_{l_1}$ is the Kolmogorov $l_1-$dimension of the space of environments. We then find concrete bounds of $d_{l_1}$ in a variety of settings, such as tabular, linear and finite mixtures, and discuss how our results improve the state-of-the-art.

Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learning

Abstract

In this paper, we prove the first Bayesian regret bounds for Thompson Sampling in reinforcement learning in a multitude of settings. We simplify the learning problem using a discrete set of surrogate environments, and present a refined analysis of the information ratio using posterior consistency. This leads to an upper bound of order ˜ O ( H √ d l 1 T ) in the time inhomogeneous reinforcement learning problem where H is the episode length and d l 1 is the Kolmogorov l 1 -dimension of the space of environments. We then find concrete bounds of d l 1 in a variety of settings, such as tabular, linear and finite mixtures, and discuss how how our results are either the first of their kind or improve the state-of-the-art.

1 Introduction

Reinforcement Learning (RL) is a sequential decision-making problem in which an agent interacts with an unknown environment typically modeled as a Markov Decision Process (MDP) [38, 8]. The goal of the agent is to maximize its expected cumulative reward. This problem has a variety of applications, including robotics, game playing, resource management, and medical treatments. The key challenge in RL is to balance the so-called exploration-exploitation trade-off efficiently: exploring unseen state-action pairs to gain more knowledge about the unknown environment or exploiting the current knowledge to maximize the expected cumulative reward. Two efficient approaches have been developed to control this trade-off: optimism in the face of uncertainty (OFU) and Thompson Sampling (TS) (or Posterior Sampling (PS)). OFU constructs a confidence set of statistically plausible MDPs that includes the true MDP with high probability and plays an optimistic policy according to the MDP with maximum gain from this set [5, 40]. TS samples a statistically plausible MDP from a posterior distribution and plays the optimistic policy of the sampled MDP [29, 31]. In this work, we focus on the latter, and by combining an information theoretical approach first introduced by [36] with analysis based on posterior consistency tools, we prove state-of-the-art Bayesian regret bounds in a variety of settings.

In this paper, we start by defining the Bayesian RL problem, where transition and reward functions are Bayesian and time inhomogeneous. The Bayesian RL problem we consider is more comprehensive than in previous works, as we allow for both Bayesian transition and Bayesian rewards, and do not make any assumption on their individual prior. To simplify the learning problem, we utilize the notion of surrogate environments, which is a discretization of the environments space, and its learning task and TS regret is a proxy to that of the main problem. The construction of the surrogate environments was first introduced by [18] with an incorrect proof, which is fixed in our work by

This work was supported in part by the National Science Foundation under grant CCF-2149588 and Cisco, Inc.

∗ §

defining the surrogate environments through an optimization. Of main importance is the size of this new environment space. The Bayesian regret decomposes to the product of two terms, one being the cumulative mutual information of the environment and history traversed by the policy. By the well-known entropy estimation of the mutual information, this significant factor in the regret is connected to the l 1 -dimensions ( d l 1 ) of the transition and reward functions space, which can be more succinctly interpreted as the l 1 -dimension d l 1 of the environment space. The latter is in turn estimated by the size of the space of surrogate environments.

The information ratio, representing a trade-off of exploration/exploitation, is the other significant term in the decomposition of the TS Bayesian regret. In an improvement to [18], our novel analysis of this ratio based on posterior consistency tools, shows that this trade-off is bounded by H 3 / 2 , where H is the episode length. This bound is general and independent of the dimension of transition/reward function space at each step, which is is a key factor behind the advantage of our regret bound, such as the √ SA advantage in the tabular case compared to [18], or the lack of any restriction on the prior (e.g., Dirichlet prior) compared to [31]. Following a further refined approach, we finally estimate the TS Bayesian regret to be ˜ O ( λ √ d l 1 T ) for large enough T in the time inhomogeneous setting. Here, a new term 'value diameter' λ , which is the average difference of the optimal value functions at different states, is used in bounding the information ratio, where instead of H 3 / 2 , we have the smaller term λH 1 / 2 . Bounding the information ratio with λ is a conceptual contribution of our work, which shows that the ratio is bounded by a value-dependent term, which is in nature different from H but always ≤ H +1 . Further, there exists another bound for λ ; in environments where states are reachable from one another in D steps, we have λ ≤ D +1 . In 'well-connected' MDPs, one could have D ≪ H , implying an improvement over the H 3 / 2 information ratio bound.

Our generic bound is abstract in terms of d l 1 , so we estimate it in more explicit terms for useful applications. [18] have bounded d l 1 in the tabular and linear case without formalizing this notion, and while for tabular MDPs, d l 1 was bounded by SAH , for linear MDPs with feature space dimension d f , we investigate their claim of the bound d f H . Detailed in Appendix G, we show a counterexample to their analysis, and we manage to find a correct estimate in this setting. We also introduce finite mixtures MDPs and are the first to prove a TS Bayesian regret of order ˜ O ( λ √ HmT ) , where m is the number of mixtures.

Lastly, we note that our regret bound of order ˜ O ( λ √ d l 1 T ) is the first in the general nonlinear time inhomogeneous Bayesian RL setting for TS, and generalizing [31, Conj. 1], we conjecture it to be optimal if λ can be replaced by ˜ O ( √ H ) .

Related work. Since the introduction of information ratio by [35, 36], a new line of research has emerged to provide tighter regret bounds for TS. The general approach involves factoring the Bayesian regret into two components: an information ratio that captures the trade-off between optimal action selection and information gain, and a cumulative information gain term that depends on the target environment and the history of previous observations. Then, both components are bounded separately using information theoretic tools.

In the bandit setting, this analysis has been used to bound Bayesian regret for TS [14, 9], as well as that of a new algorithm called information-directed sampling (IDS) [35, 26, 23, 19, 20]. This analysis has also been used in partial monitoring [25, 24] and RL with a specific Dirichlet prior and additional assumptions [28, 27] or when the true environment is too complicated to learn [4]. More recently, [18] studied the Bayesian regret of TS in RL without any prior assumptions for tabular MDP. This is the closest work to our paper and we discuss our generalization in detail in Section 5.

The Bayesian tabular MDP case has also been studied with the additional Dirichlet prior assumption in [31], where they achieve a regret bound matching ours. In an independent approach, the first non-linear Bayesian RL model was considered by [16] with a regret bound of dH 3 / 2 T 1 / 2 where d is a notion of dimension of their model, but their results were limited to Gaussian process settings with linear kernels. Finally, [11] considered general non-linear Bayesian RL models and introduced an algorithm that obtains dH 1+ α/ 2 T 1 -α/ 2 where α is a tuning parameter and d is the dimension of S × A × S .

It is worth noting that there is another line of work that incorporates confidence regions into TS to achieve Bayesian regret bounds that can match the best possible frequentist regret bounds by

UCB in both bandit settings [35] and RL [31, 30, 32, 12]. However, this technique often results in a sub-optimal Bayesian regret, as the best bound known for UCB itself is not optimal.

As discussed in Section 4.3 of [16], the Lipschitz term L ∗ , which is used in the grayed papers in the table, may grow exponentially in episode length. Note that [18] claims a regret bound for the linear setting. However, as discussed in Appendix G.1, their proof is incorrect.

While our work's emphasis is on theoretical guarantees for TS, we discuss here the experiments using this algorithm. Previous works on PSRL [35, 26, 23, 20, 31] come with extensive experiments on TS (and/or its variants), and discussions on computational efficiency of PSRL. In particular, experiments in [31] support the assertion that 'PSRL dramatically outperforms existing algorithms based on OFU'. In addition, PSRL with oracle access has been shown to be the most performant, esp. when compared to recent OFU based UCBVI/UCBVI-B, or even variants of PSRL such as Optimistic PSRL [39, Fig. 1.3]. However, an important limitation in experiments is the need for oracle access to an optimal policy, and that can not be always satisfied efficiently. Nevertheless, clever engineering can make TS work even in large scale Deep RL. Indeed, for general RL settings, the recent work [37] shows how to implement TS in Deep RL on the Atari benchmark and concludes that 'Posterior Sampling Deep RL (PSDRL) significantly outperforms previous state-of-the-art randomized value function approaches, its natural model-free counterparts, while being competitive with a state-of-the-art (model-based) reinforcement learning method in both sample efficiency and computational efficiency'. In summary, experiments in the literature provide enough support for the empirical performance of TS.

2 Preliminaries

2.1 Finite-horizon MDP

We follow the literature's conventions in our notation and terminology to avoid confusion when comparing results. The environment is a tuple E = ( S , µ S , A , µ A , H, { P h } H h =1 , { r h } H h =1 ) , where S is the topological measurable state space, A is the topological measurable action space, µ S and µ A are base probability measures on S and A respectively, H is the episode length, P h : S×A → ∆ S ,µ S is the transition probability kernel, and r h : S × A → ∆ [0 , 1] , Lebesgue is the reward function, where we fix the convention r ( s, a ) := E x [ r ( x | s, a )] = ∫ 1 0 xr ( x | s, a ) d x as we mostly deal with its mean value. Notice that ∆ X,µ is the set of probability distributions over X that are absolutely continuous with respect to µ . We will use ∆ X when the base measure is clear from the context. We assume S , A are known and deterministic while the transition probability kernel and reward are unknown and random. Throughout the paper, the implicit dependence of P h and r h on E should be clear from the context.

Let Θ P h be the topological function space of P h and Θ P = Θ P 1 ×···× Θ P H be the full function space. The space Θ P h is assumed to be separable and equipped with prior probability measure ρ P h yielding the product prior probability measure ρ P = ρ P 1 ⊗··· ⊗ ρ P H for Θ P . The exact same definition with similar notations Θ R h , ρ R h , ρ R , Θ R applies for the reward function. Notice the explicit assumption of time inhomogeneity in these definitions, with all 'layers' h being independent. The two sets define the set of all environments parametrized by Θ = Θ 1 ×··· × Θ H where Θ h = Θ P h × Θ R h . Note that the prior is assumed to be known to the learner. This setting implies that an environment E sampled according to the prior ρ = ρ P ⊗ ρ R is essentially determined by its transition and reward functions pair { ( P h , r h ) } H h =1 . We simplify the notation to view Θ as the set of all environments, i.e., saying

E ∈ Θ should be viewed as { ( P h , r h ) } H h =1 ∈ Θ . The space of all possible real-valued functions { ( P h , r h ) } H h =1 has a natural vector space structure. Therefore it is meaningful to discuss the notion of the convex combination of environments. We assume that Θ is a convex subspace of the space of all possible environments. This assumption is not restrictive, since we may replace any environment space with its convex hull. Note that we do not assume that the support of the prior is convex.

Remark 1 . The case of joint prior may be of interest, but to our knowledge all prior works also take ρ P , ρ R to be independent.

Agent, policy and history. An agent starts at an initial state s ℓ 1 , which is fixed for all episodes ℓ . It observes a state s ℓ h at layer h episode ℓ , takes action a ℓ h , and receives reward r ℓ h . The environment changes to the next random state s ℓ h +1 with probability P h ( s ℓ h +1 | s ℓ h , a ℓ h ) . The agent stops acting at s H +1 and the environment is reset to its initial state.

We define H ℓ,h as the history ( s ℓ 1 , a ℓ 1 , r ℓ 1 , . . . , s ℓ h , a ℓ h , r ℓ h ) . Denote by D ℓ = ( H 1 ,H , . . . , H ℓ -1 ,H ) the history up to episode ℓ , where D 1 := ∅ . Finally, let Ω h = ∏ h i =1 ( S × A × [0 , 1]) be the set of all possible histories up to layer h .

A policy π is represented by stochastic maps ( π 1 , . . . , π H ) where each π h : Ω h -1 ×S → ∆ A ,µ A . Let Π S denote the entire stationary policy class, stationary meaning a dependence only on the current state and layer and let Π ⊆ Π S .

Value and state occupancy functions. Define the value function V E h,π as the value of the policy π interacting with E at layer h :

where E E π denotes the expectation over the trajectory under policy, transition, and reward functions π, P h , r h . The value function at step H +1 is set to null, V E H +1 ,π ( · ) := 0 . We assume there is a measurable function π ∗ E : Θ → Π such that V E h,π ∗ E ( s ) = max π ∈ Π V E h,π ( s ) , ∀ s ∈ S , h ∈ [ H ] . The optimal policy π ∗ is a function of E , making it a random variable in the Bayesian setting. Lastly, let the state-action occupancy probability measure be P E π ( s h = s, a h = a ) , also known as the state occupancy measure under policy π and environment E . It follows from the definitions that this measure is absolutely continuous with respect to µ S×A := µ S × µ A . Let d E h,π ( s, a ) denote the Radon-Nikodym derivative so that we have d E h,π ( s, a ) d µ S×A = d P E π ( s h = s, a h = a ) . We will assume throughout the paper that this density d E h,π ( s, a ) is measurable and upper bounded for all π, E , s, a, h . The upper bound is a reasonable assumption, and it happens trivially in the tabular case ( d E h,π ( s, a ) ≤ SA ). This also happens, e.g., when one assumes that the maps ( E , s, a, s ′ , h ) ↦→ P E h ( s ′ | s, a ) and ( π, s, a, h ) ↦→ π h ( a | s ) are continuous and Θ , S , A and the set of all optimal policies (as a subset of Π ) are compact.

2.2 Bayesian regret

We formulate the expected regret over L episodes and T = LH total steps in an environment E as

where the expectation is over the randomness of π = { π ℓ } ℓ . The Bayesian regret is BR L ( π ) = E [ R L ( E , π )] . For Thompson Sampling (TS), the algorithm selects the optimal policy of a given sample E ℓ picked from the posterior E ℓ ∼ P ( E ∈ ·|D ℓ ) :

Importantly, the law of TS aligns with the posterior, i.e., P ( E|D ℓ ) = P ( π ℓ TS = π ∗ E |D ℓ ) .

Remark 2 . Note that P ( π ℓ TS = π ∗ E |D ℓ ) is a probability for a specific measure on the space of optimal policies. To ensure that ∫ Π ∗ P ( π ∗ |D ℓ ) d ρ Π ∗ = 1 , we need an appropriate measure ρ Π ∗ on Π ∗ . Given the law of TS, the natural choice for this measure is the push-forward of the prior measure ρ under the map star : Θ → Π ∗ , where star ( E ) = π ∗ E .

2.3 Notations

For Bayesian RL, conditional expressions involving a given history D ℓ are widely used. We adopt the notation in [18] to refer to such conditionals; let P ℓ ( · ) := P ( ·|D ℓ ) , E ℓ [ · ] := E [ ·|D ℓ ] . We can rewrite the Bayesian regret as

and define the conditional mutual information I ℓ ( X ; Y ) := D KL ( P (( X,Y ) ∈ ·|D ℓ ) || P ( X ∈ ·|D ℓ ) ⊗ P ( Y ∈ ·|D ℓ )) . For a random variable χ and random policy π , the following will be involved in the information ratio:

Note that E [ I ℓ ( X ; Y )] = I ( X ; Y |D ℓ ) . To clarify, P ℓ ( H ℓ,h ∈ ·| π ) is the probability of H ℓ,h being generated under π within some environment. Given that the histories under consideration are generated by the TS algorithm, they are always generated in the true environment E under an optimal policy π ∗ E ′ . For π = π ℓ TS , this can be computed as P ℓ ( H ℓ,h | π ) = ∫ E P ( H ℓ,h | π, E ) d P ℓ ( E ) , where P ( H ℓ,h | π, E ) is an expression in terms of transition and reward functions of E and π .

Finally, we define ¯ E ℓ as the mean MDP where P ¯ E ℓ h ( ·| s, a ) = E ℓ [ P E h ( ·| s, a )] is the mean of posterior measure, and similarly for r ¯ E ℓ h ( ·| s, a ) = E ℓ [ r E h ( ·| s, a )] . We note that under the independence assumption across layers, the same is given for the state-occupancy density d ¯ E ℓ h,π = E ℓ [ d E h,π ] .

3 Bayesian RL problems

Definition 1. A Bayesian RL in this paper refers to the time-inhomogeneous finite-horizon MDP with independent priors on transition and reward functions, as described in Section 2.1.

The Bayesian RL problem is the task of finding an algorithm π with optimal Bayesian regret as defined in Eq. (4). Below we list the variations of this problem. A setting considered by most related works such as [31, 16] is the following:

Definition 2. The time (reward) homogeneous Bayesian RL refers to the Bayesian RL setting where the prior ρ P ( ρ R ) is over the space Θ P ( Θ R ) containing the single transition (reward) function P ( r ) defining E , i.e., all layers have the same transition (reward) functions.

Definition 3. The tabular Bayesian RL is a Bayesian RL where S , A are finite sets.

Definition 4 (Linear MDP [41, 22]) . Let ϕ P : S × A → R d P f , ϕ R : S × A → R d R f be feature maps with bounded norm ∥ ϕ P ( s, a ) ∥ 2 , ∥ ϕ R ( s, a ) ∥ 2 ≤ 1 . The linear Bayesian RL is a Bayesian RL where for any E = { ( P E h , r E h ) } H h =1 ∈ Θ , there exists vector-valued maps ψ P, E h ( s ) , ψ R, E h ( s ) with bounded l 2 -norm such that for any ( s, a ) ∈ S × A ,

A restricted version of the finite mixtures called linear mixture was first considered in [6] in the frequentist setting. Here, we consider the general setting.

Definition 5. The finite mixtures Bayesian RL is a Bayesian RL where for any h ∈ [ H ] there exists fixed conditional distributions { Z P h,i : S ×A → ∆ S } m P h i =1 and { Z R h,i : S ×A → ∆ [0 , 1] } m R h i =1 , such that for any environment E given by { ( P E h , r E h ) } H h =1 , there exists parametrized probability distributions a P, E h : S × A → ∆ m P h , a R, E h : S × A → ∆ m R h such that

4 Surrogate learning

Next, we define the discretized surrogate learning problem, and bound the size of the surrogate environments space, a significant term in the regret. To do so, we need to first define the Kolmogorov dimension of a set of parametrized distributions, esp. working out the case of l 1 -distance. In the definitions below, we implicitly assume any required minimal measurability assumptions on the involved sets.

Definition 6. Given a set F of Oparametrized distributions P : O → ∆( S ) over a set S where both O , S are measurable. Let M ( · , · ) : F × F → R ≥ 0 be a distance , i.e., M ( P, Q ) ≥ 0 = ← → P = Q . Then its right ε -covering number is the size K M ( ε ) of the smallest set C M ( ε ) = { P 1 , . . . , P K M ( ε ) } ⊂ F such that

The potential asymmetry of M (e.g., KL-divergence) requires the notion of left/right covering number. The right covering number will be the default, so covering number will always refer to that.

Definition 7. Let d M ( ε ) = log( K M ( ε )) . Define the Kolmogorov Mdimension d M of F as

For l 1 ( P, Q ) := sup o ∈O || P ( ·| o ) -Q ( ·| o ) || 1 , applying Definition 6 to the sets Θ P h , Θ R h with O = S × A , and denote the respective covering numbers by L P h ( ε ) , L R h ( ε ) corresponding to covering sets C P h ( ε ) , C R h ( ε ) . Similarly applying Eq. (9) and denote the corresponding l 1 -dimensions by d P l 1 ,h ( ε ) , d R l 1 ,h ( ε ) , d P l 1 ,h , d R l 1 ,h and d P l 1 := ∑ h d P l 1 ,h , d R l 1 := ∑ h d R l 1 ,h . The sums d l 1 ,h := d P l 1 ,h + d R l 1 ,h , d l 1 := d P l 1 + d R l 1 can be interpreted as the l 1 -dimension of Θ h and Θ , i.e., the environment space.

Remark 3 . We can also apply this framework to the KL-divergence, by M KL ( P, Q ) := sup o ∈O D KL ( P ( ·| o ) || Q ( ·|| o )) . This was implicitly used by [18] to prove their regret bound in the tabular case. Note that Pinsker's lemma (Lemma 9) implies that the KL-divergence is larger than the squared total variance, and the latter is trivially larger than the l 1 distance. Therefore, l 1 -dimension is smaller than d M KL , allowing for tighter regret bounds.

We now revisit the definition of ε -value partitions and show their existence is guaranteed by finite l 1 -covering numbers. These partitions are the origins of surrogate environments.

Definition 8. Given ε > 0 , an ε -value partition for a Bayesian RL problem is a partition { Θ k } K k =1 over Θ such that for any k ∈ [ K ] and E , E ′ ∈ Θ k ,

A layered ε -value partition is one where the transition functions are independent over layers after conditioning on k . Throughout this paper, we will only consider layered ε -value partition. We define K surr ( ε ) as the minimum K for which there exists a layered ε -value partition.

Inspired by Eq. (9), we define the surrogate dimension as d surr = lim sup ε → 0 K surr ( ε ) log(1 /ε ) .

Lemma 1. Given a Bayesian RL, we have K surr ( ε ) ≤ ∏ h L P h ( ε/ (2 H ) 2 ) × L R h ( ε/ (4 H )) . This implies d surr ≤ d l 1 .

The above is proved in Appendix B. It is hard to find d surr , but one can estimate d l 1 , and according to the above, this acts as a proxy for K surr . This is useful as the regret relates to K surr . But to show this, we need to construct surrogate environments inside each partition, and show that learning those is almost equivalent to the original problem. Let ζ be a discrete random variable taking values in { 1 , · · · , K surr ( ε ) } that indicates the partition E lies in, such that ζ = k if and only if E ∈ Θ k .

Lemma 2. For any ε -value partition and any ℓ ∈ [ L ] , there are random environments ˜ E ∗ ℓ ∈ Θ with their laws only depending on ζ, D ℓ , such that

The expectation in both equations is over E and π ℓ TS ∈ { π ∗ E ′ } E ′ ∈ Θ , with both sampled independently ∼ P ℓ ( · ) , and the K different values of ˜ E ∗ ℓ . The second expectation over ( ˜ E ∗ ℓ , E ) is over pairs that are in the same partition, i.e., ˜ E ∗ ℓ , E are independent only after conditioning on ζ .

We note that the proof in [18, App. B.1] contains the use of a lemma that does not apply to construct the law of the environment ˜ E ∗ ℓ . More details is provided in Appendix C, where we find ˜ E ∗ ℓ by minimizing an expected value of π ℓ TS .

5 Bayesian regret bounds for Thompson Sampling

5.1 General Bayesian regret bound

We start by introducing the notion of value diameter.

Definition 9. Given the environment E , its value diameter is defined as

where r sup h ( s, a ) (and r inf h ( s, a ) ) is the supremum (and infimum) of the set of rewards that are attainable under the distribution r h ( s, a ) with non-zero probability. As a special case, if rewards are deterministic, then we have r sup h ( s, a ) = r inf h ( s, a ) for all s, a . The (average) value diameter over Θ is denoted by λ := E E∼ ρ [ λ 2 E ] 1 / 2 .

As the value function is between 0 and H , we have λ E ≤ H +1 implying λ ≤ H +1 . Note that value diameter is closely related to the notion of diameter commonly defined in finite RL problems. Strictly speaking, for a time-homogeneous RL, it is straightforward to see that the value diameter is bounded from above by one plus the diameter [33].

We now discuss the assumptions surrounding our results. The main technical assumption of this paper is the existence of consistent estimators, which as we will see in Appendix K, is closely related to the notion of posterior consistency:

Assumption 1. There exists a strongly consistent estimator of the true environment given the history.

Roughly speaking, we assume that with unlimited observations under TS, it is possible to find the true environment. For this assumption to fail, we need to have two environments that produce the same distribution over histories under TS and are therefore indistinguishable from the point of view of TS. The precise description of this assumption is detailed in Appendix K.

Another necessary technical assumption is that almost all optimal policies visit almost all state action pairs in their respective environment.

Assumption 2. For almost every environment E ∈ Θ and almost every ( s, a ) ∈ S × A and every h ∈ [ H ] , we have

̸

Recall that, for any environment E ∈ Θ , the policy π ∗ E is the optimal policy of E within the policy class Π . Therefore, one example of how the above assumption holds is when Π is the set of ε -greedy algorithms and transition functions of environments assign non-zero probability to every state. Under these assumptions, we discuss our main result and its corollaries.

Theorem 3. Given a Bayesian RL problem, for all ε > 0 , we have

does not depend on

T

. This can be further upper bounded by

for large enough T . Given a homogeneous l 1 dimension d hom = d l 1 ,h , ∀ h , this simplifies to

Remark 4 . For all regret bounds, we will replace λ ≤ H +1 to compare our result. For the case of homogeneous dimensions, we obtain ˜ O ( H 3 / 2 √ d hom T ) . Crucially, our main result shows a new conceptual understanding of the information ratio by bounding it by two terms of different nature: H and λ , where the latter can be bounded by either the largest diameter of the environments or H .

where

T

0

Remark 5 . Despite not impacting the asymptotics, the impact of T 0 can be large depending on the structure of the RL problem, and could be dominant even for large T s in practice.

Remark 6 . Considering time as a part of the state observation, one could apply this regret analysis to particular time-homogeneous settings. However, this mapping of time-inhomogeneous RLs to homogeneous ones is not surjective, hence the result above does not readily extend to timehomogeneous settings.

While [16] were the first to consider a nonlinear Bayesian RL model, their bound is limited to the Gaussian process (with linear kernel) setting, while ours in the nonlinear time inhomogeneous setting makes no assumptions on the prior and is the first such bound. Our novel analysis allow us to upper bound the information ratio by λ √ H instead of, for example H 3 / 2 √ SA ([18]) in the tabular case, improving the regret bound by a square root relevant to the dimension d of the problem.

The detailed proof is given in Appendix D. Following [18], the regret (4) is rewritten using Lemma 2 to reduce the problem into its surrogate, and we use the well-known information-ratio trick by multiplying and dividing by the mutual information. We follow that with a Cauchy-Schwarz, summarized below

Note the cost ε at each episode (Lemma 2) in the first inequality, yielding the overall error Lε . Then, we can bound the mutual information appearing in the regret term by E [ ∑ L ℓ =1 I π ℓ TS ℓ ( ˜ E ∗ ℓ ; H ℓ,H ) ] =

I π ℓ TS ℓ ( ˜ E ∗ ℓ ; D ℓ ) ≤ I π ℓ TS ℓ ( ζ ; D ℓ ) ≤ H ( ζ ) ≤ log( K surr ( ε )) , where we used the mutual information chain rule, followed by data processing inequality to substitute ˜ E ∗ ℓ → ζ , and finally used the trivial bound by the entropy. But the main novelty of our approach lies in our control of the first term

called the information ratio. In our analysis, we have the following bound on its expectation.

where the average is taken over all histories D ℓ that are generated from running TS on the true environment E 0 , and we have introduced the smaller term λ E instead of H in [18]. While [18] essentially bound the above only in the tabular setting with SAH 3 , we manage to generally bound the above with a more precise bound using Doob's consistency theorem. Assumption 1 allows us to use Doob's consistency theorem to conclude that for almost every environment E 0 , almost every infinite sequence of histories ( D ℓ ) ∞ ℓ =1 sampled from E 0 , and every integrable function f , the posterior mean E ℓ [ f ( E )] = E [ f ( E ) | D ℓ ] converges to f ( E 0 ) . In particular, we conclude that E [Γ ℓ ( π ℓ TS ) | E 0 ] tends to λ 2 E 0 H in the limit, allowing us to claim that for large enough ℓ , the expected information ratio E [Γ ℓ ( π ℓ TS )] is uniformly bounded by 2 E [ λ 2 E ] H = 2 λ 2 H . As there are L many such ratios, the two bounds together yield 2 √ λ 2 HL · √ log( K surr ( ε ))+ Lε . This bound is true for large enough ℓ , giving the additional additive term T 0 in the theorem. Since this term is additive, applying Lemma 1 to bound log( K surr ( ε )) , we have successfully shown the asymptotic behavior of the regret, independent of the prior, is of order ˜ O ( H √ d l 1 T ) .

5.2 Applications

In each application below, the challenge is to bound d l 1 using the specifics of the model, and except for the case of tabular Bayesian RL, such analysis has not been carried out rigorously. We formalize the corollaries and show they are state-of-the-art compared to the literature.

Tabular RL. The result below follows from Theorem 3; the main contribution comes from our new information ratio bound, followed by the estimate ˜ O (( 1 ε ) SAH ) of K surr ( ε ) ([18]).

Corollary 4. Given a tabular Bayesian RL problem, for large enough T ,

where the polylogarithmic terms are explicitly in terms of H,S,A,L .

We observe that our result matches [31] when their result in the time homogeneous setting (Definition 2) is extended to time inhomogeneous. However, in that paper, the authors assume a Dirichlet based prior which we do not.

Linear RL. A previous state-of-the-art ˜ O ( d f H 3 / 2 √ T ) was claimed by [18] to hold for linear Bayesian RLs with deterministic reward. We note:

As in the previous cases, their proof in bounding their information ratio includes a factor of d f , which ours avoids.

We show that the proof bounding K surr ( ε ) in [18, App. B.4] is incorrect, starting with a wrong application of Cauchy-Schwarz and a wrong mutual information in their definition of information ratio. We provide counterexamples for the estimates found therein to substantiate our claim (see Appendix G.1).

To state our own corollary in this case, we need to define a few notions. Let d f l 1 = d P,f l 1 + d R,f l 1 be the sum of the l 1 -dimensions of the feature map space { ψ P, E h } E∈ Θ , { ψ R, E h } E∈ Θ where the l 1 -distance between feature maps is defined as l 1 ( ψ E h , ψ E ′ h ) = ∫ s ∥ ψ E h -ψ E ′ h ∥ 1 µ S . Our corollary also provides a concrete bound in the case of mixture linear Bayesian RL where the feature maps are themselves a sum of finitely many fixed feature maps. This means for all E ∈ Θ , we have

where { Ψ P h,i ( s ) } m P h i =1 , { Ψ R h,i ( s ) } m R h i =1 are finitely many fixed feature maps and ∀E , h : ∑ i | a P, E h,i | 2 , ∑ i | a R, E h,i | 2 ≤ C a for some constant C a > 0 . Let M = M P + M R = ∑ h m P h + ∑ h m R h .

Corollary 5. For a linear Bayesian RL, for large enough T ,

Given a linear Bayesian RL with finitely many states and total feature space dimension d f = d P f + d R f , we have d l 1 ≤ 2 d f HS , yielding for large enough T ,

Given a mixture linear Bayesian RL, for large enough T ,

The proof is given in Appendix G. The fact that d l 1 appears instead of d f in the general bound is not counter-intuitive, as we should expect the complexity of the feature map space { ψ P, E h ( s ) } E∈ Θ ,h ∈ [ H ] , { ψ R, E h ( s ) } E∈ Θ ,h ∈ [ H ] to play a role in the regret, especially as this space can be very complex, and model very different environments that can not be grouped in the same ε -value partition.

Therefore, opposite to the claim made by [18], this complexity can not be captured by simply d f except maybe in degenerate cases, such as when S is finite, which is our second statement. More generally, if each feature map ψ P, E h ( s ) , ψ R, E h ( s ) can be characterized with a vector of uniformly bounded norm a P, E h ∈ R m P h , a R, E h ∈ R m R h , then we can bound the regret in terms of m P h , m R h 's, as is done in Eq. (22) (the finite state case corresponds to m P h = d P f S, m R h = d R f S ).

Finite mixtures RL. To state our finite mixtures model result, we need to set the following notations. Let d l m 1 = d m,P l 1 + d m,R l 1 = ∑ h d m,P l 1 ,h + ∑ h d m,R l 1 ,h correspond to the total l 1 -dimension of the space of mixtures coefficient maps { a P, E h ( s, a ) } E∈ Θ , { a R, E h ( s, a ) } E∈ Θ with l 1 -distance defined as l 1 ( a E h , a E ′ h ) = sup s,a ∥ a E h ( s, a ) -a E ′ h ( s, a ) ∥ 1 . Define also the restricted finite mixtures model where a P, E h , a R, E h are vectors in R m P h , R m R h independent of ( s, a ) and let M = M P + M R = ∑ h m P h + ∑ h m R h .

Corollary 6. Given a finite mixtures Bayesian RL problem, for large enough T ,

Assuming the restricted finite mixtures model, for large enough T ,

which, given a uniform dimension m = m P h = m R h , yields ˜ O ( λ √ HmT ) .

We prove the above in Appendix H, deriving it from our generic bound, after relating the l 1 -dimension d l 1 of the environment space to that of the mixtures coefficients. To the best of our knowledge, this is the first bound for finite mixtures Bayesian RL problems. We note that in a previous work ([6]), a restricted version of finite mixtures, like in Eq. (24), was considered in the frequentist setting.

We finish this section by proposing the following conjecture, in line with [31, Conj. 1].

Conjecture 7. For the Bayesian RL, the following is true and optimal for all T :

where the constant factor is independent of the prior. This means there exists a Bayesian RL problem such that BR L ( π TS ) = ˜ Ω( √ Hd surr T ) . All polylogarithmic terms are in terms of H,d surr , T .

Note that the above coincides with the lower bound for the (model-based) time inhomogeneous frequentist setting; see e.g., [21] for the proven lower bound for the tabular case. This is also √ H higher (this factor being baked in d surr ) than that of the time homogeneous frequentist setting, which is expected, according to [21, App. D]. Note that in this conjecture, the λ in our bound is replaced by √ H , and the conjecture is not for T large enough, but for all T . Supporting this conjecture requires experiments where TS can be exactly implemented assuming access to an oracle which provides the optimal policy for a query environment. Simulations have been performed for the similar [31, Conj. 1] in the time homogeneous case. Our conjecture is similar but with the additional expected factor of √ H due to time inhomogeneity, thus their simulation also supports the above.

6 Conclusions

In this paper, we have addressed the Bayesian Reinforcement Learning (RL) problem in the context of time inhomogeneous transition and reward functions. By considering both Bayesian transition and Bayesian rewards without prior assumptions, we have extended the scope of previous works, making our formulation more comprehensive. To simplify the learning problem, we have introduced surrogate environments, which discretize the environment space. We have established a connection between the size of this new environment space and the l 1 -dimensions of the transition and reward functions space, providing insights into the l 1 -dimension of the environment space denoted by d l 1 . We have employed posterior consistency tools to analyze the information ratio, which captures the trade-off between exploration and exploitation. We conjecture that (at least a weakened version of) our posterior consistency assumption should hold in general, which is left for future work. Our analysis has resulted in a refined approach to estimate the Bayesian regret in Thompson Sampling (TS), yielding a regret bound of ˜ O ( λ √ d l 1 T ) for large enough time steps T . The result is specialized to linear, tabular, and finite mixtures MDPs.

Limitations: While the paper provides asymptotic generic regret bound for TS in a generalized setup which improve the state of the art results, finding lower bounds, esp. one dependent on λ , are left open. In addition, the issue of prior misspecificity is not discussed and left for future studies.

References

Amirhesam Abedsoltan, Mikhail Belkin, and Parthe Pandit. Toward large kernel models. arXiv preprint arXiv:2302.02605 , 2023.

Alekh Agarwal, Yujia Jin, and Tong Zhang. VO Q L: Towards optimal regret in model-free rl with nonlinear function approximation. arXiv preprint arXiv:2212.06069 , 2022.

Shipra Agrawal and Randy Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. Advances in Neural Information Processing Systems , 30, 2017.

Dilip Arumugam and Benjamin Van Roy. Deciding what to model: Value-equivalent sampling for reinforcement learning. 35:9024-9044, 2022.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

Sébastien Bubeck and Mark Sellke. First-order bayesian regret analysis of thompson sampling. In Algorithmic Learning Theory , pages 196-233. PMLR, 2020.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Souradip Chakraborty, Amrit Bedi, Alec Koppel, Pratap Tokekar, Furong Huang, and Dinesh Manocha. Posterior coreset construction with kernelized stein discrepancy for model-based reinforcement learning. In NeurIPS 2022 Workshop on Score-Based Methods , 2022.

Sayak Ray Chowdhury and Aditya Gopalan. Online learning in kernelized markov decision processes. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages 3197-3205. PMLR, 2019.

Christoph Dann, Mehryar Mohri, Tong Zhang, and Julian Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems , 34:12040-12051, 2021.

Shi Dong and Benjamin Van Roy. An information-theoretic analysis for thompson sampling with many actions. Advances in Neural Information Processing Systems , 31, 2018.

Yaqi Duan, Mengdi Wang, and Martin J Wainwright. Optimal policy evaluation using kernelbased temporal difference methods. arXiv preprint arXiv:2109.12002 , 2021.

Ying Fan and Yifei Ming. Model-based reinforcement learning for continuous control with posterior sampling. In International Conference on Machine Learning , pages 3078-3087. PMLR, 2021.

Subhashis Ghosal and Aad Van der Vaart. Fundamentals of nonparametric Bayesian inference , volume 44. Cambridge University Press, 2017.

Botao Hao and Tor Lattimore. Regret bounds for information-directed reinforcement learning. arXiv preprint arXiv:2206.04640 , 2022.

Botao Hao, Tor Lattimore, and Wei Deng. Information directed sampling for sparse linear bandits. Advances in Neural Information Processing Systems , 34:16738-16750, 2021.

Botao Hao, Tor Lattimore, and Chao Qin. Contextual information-directed sampling. In International Conference on Machine Learning , pages 8446-8464. PMLR, 2022.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 21372143. PMLR, 2020.

Johannes Kirschner, Tor Lattimore, Claire Vernade, and Csaba Szepesvári. Asymptotically optimal information-directed sampling. In Conference on Learning Theory , pages 2777-2821. PMLR, 2021.

Tor Lattimore and Andras Gyorgy. Mirror descent and the information ratio. In Conference on Learning Theory , pages 2965-2992. PMLR, 2021.

Tor Lattimore and Csaba Szepesvári. An information-theoretic approach to minimax regret in partial monitoring. In Conference on Learning Theory , pages 2111-2139. PMLR, 2019.

Fang Liu, Swapna Buccapatnam, and Ness Shroff. Information directed sampling for stochastic bandits with graph feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Xiuyuan Lu. Information-directed sampling for reinforcement learning . Stanford University, 2020.

Xiuyuan Lu and Benjamin Van Roy. Information-theoretic confidence bounds for reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learning , pages 2701-2710. PMLR, 2017.

Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res. , 20(124):1-62, 2019.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Daniel Russo. Worst-case regret bounds for exploration via randomized value functions. Advances in Neural Information Processing Systems , 32, 2019.

Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. Advances in Neural Information Processing Systems , 27, 2014.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Remo Sasso, Michelangelo Conserva, and Paulo Rauber. Posterior sampling for deep reinforcement learning. arXiv preprint arXiv:2305.00477 , 2023.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Éric Moulines, Remi Munos, Alexey Naumov, Mark Rowland, Michal Valko, and Pierre Ménard. Optimistic posterior sampling for reinforcement learning with few samples and tight guarantees. Advances in Neural Information Processing Systems , 35:10737-10751, 2022.

Aristide Tossou, Debabrota Basu, and Christos Dimitrakakis. Near-optimal optimistic reinforcement learning using empirical bernstein inequalities. arXiv preprint arXiv:1905.12425 , 2019.

Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics , pages 1954-1964. PMLR, 2020.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learningvia reference-advantage decomposition. Advances in Neural Information Processing Systems , 33:15198-15207, 2020.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a refined analysis of TS in Bayesian regret reinforcement learning. Regret bounds are derived for tabular, linear, and finite mixture MDPs. The paper uses an information theoretical approach: the information ratio, representing the exploration-exploitation trade-off, is analyzed and bounded by the episode length. The paper uses discretization of the environment space defined by fixing a previous proof from the literature. The paper concludes by discussing related work and the optimality of the obtained regret bounds.\n\nSTRENGTHS:\nThe paper provides state-of-the-art Bayesian regret bounds for Thompson Sampling in reinforcement learning through a refined analysis of the surrogate environments and information ratio in many different settings, including tabular, linear, and finite mixtures. The obtained bound is general and independent of the dimension of transition/reward function space. Proof sketches are provided.\n\nWEAKNESSES:\nThe proposed analysis and bounds may have limited applicability to real-world RL problems beyond the considered settings.\nThe paper does not provide experimental results or empirical validation of the derived bounds.\nSome assumptions and prior works are discussed without providing detailed explanations or comparisons.\nOverall, the paper makes significant contributions to the theoretical understanding of TS in RL by providing refined analysis and state-of-the-art Bayesian regret bounds in various settings. However, practical applicability and empirical validation of the derived bounds need further investigation.\n\nQUESTIONS:\nCould you please provide more references that specifically highlight the theoretical analysis of TS in the context of RL and Bandit problems? Usually, bayesian regret is easier than frequentist to analyse. Do you know if your approach could be applicable to frequentist regret ?\n\nWhat is the relationship between the Kolmogorov dimension for $l_1$-distance, the \"value partition for surrogate learning,\" and the time-inhomogeneous Bayesian RL problem? \n\nHow does leveraging this relationship help in bounding the information ratio and achieving regret bounds in TS ? \n\nCan concrete estimates of the dimension be provided for linear, tabular, and finite mixtures applications? What is the significance of isolating the contributions of the information ratio and the cumulative mutual information terms? \n\nCan you clarify the specific experimental results or empirical evidence that support the conjecture regarding the optimality of the regret bound when substituting $H$ in place of the variable? \n\nAre there any specific experiments conducted with TS, assuming access to an oracle, that provide insights into the performance of the regret bound in practical scenarios?\n\nYou mention related work on bounding Bayesian regret for TS, including the use of confidence regions and algorithms such as UCBVI and OPPO. How does the proposed work in this paper compare to these approaches in terms of regret bounds and optimality? Are there any notable advantages or limitations of the proposed approach compared to the existing literature?\n\nLIMITATIONS:\nCan you provide a motivation for using TS-based policies rather than OFU ones ? If this is about empirical performances, then it makes sense to run some experimental comparisons.\n\nI am not sure to clearly understand what evidence or rationale is provided to support the claim that the regret in surrogate environments serves as a proxy for the main problem.\n\nI think you could elaborate more on the methodology used in this analysis. How does it contribute to a better understanding of the trade-off between exploration and exploitation? Are there any limitations or assumptions associated with this analysis?\n\nminor:\nLemma 11 : fix the X vs x notation", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
The paper shows a refined analysis of Thompson Sampling in RL. The analysis leverages the notion of Kolmogorov dimension, and results in an improved rate of the regret. The authors further presented the bounds in terms of several specific settings, which match the state-of-the-art results.

STRENGTHS:
The writing of the paper is clear and easy to follow. The paper studies an important problem in RL. The notion of Kolmogorov dimension as well as the corresponding analysis is novel. The paper also presents a complete discussion for specific settings, and makes sufficient comparisons with previous works, which shows the significance of its results.

WEAKNESSES:
1. While the bound relies on a new notion $\lambda$, the term can still be as big as $H$ in many cases. It hard to to quantify how much improvement is made by this notion, and therefore, in the discussion of specific RL settings, the corresponds bounds only match the state-of-the-art results, but don't improve them. 

2. The paper doesn't provide a lower bound in terms of the proposed notion, which weakens the significance of this notion.

QUESTIONS:
1. The authors mentioned the incorrectness of [12] in terms of surrogate loss. Can the authors make some discussion on it and how did they fix the issue?

LIMITATIONS:
The paper has addressed it limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The paper presents uniform Bayesian regret bounds for Thompson Sampling by utilizing a uniform bound of information ratio and specific bounds of the Kolmogorov dimension in different settings.

STRENGTHS:
1. The paper presents a uniform Bayesian regret bound for Thompson Sampling which yields results in a variety of settings, improving upon previous approaches in some scenarios.
2. The authors incorporate a comprehensive discussion with previous works which helps to understand the contribution of the proposed bounds.

WEAKNESSES:
1. Potential overclaim: in the introduction, the authors state that they first define Bayesian RL with time inhomogeneous settings, which might be an overclaim since there are also previous works discussing this setting like [1].

[1]Variational Bayesian Reinforcement Learning with Regret Bounds. Brendan O'Donoghue

2. The presentation of the paper would benefit from a table that includes all the results discussed in the paper for a comprehensive comparison.

QUESTIONS:
Why is time inhomogeneous specially highlighted in this work? If we consider time also as a part of the state observation, it would be homogeneous (i.e., share the same model across all timesteps). With that being considered, is it possible just to extend the time inhomogeneous bounds to this setting, and is the proposed bound also better than those?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the novel Bayesian regret analysis for posterior sampling for reinforcement learning algorithm. The proposed regret bounds are applicable in a large variety of different RL settings, such as tabular, linear and finite mixture MDPs.\n\nSTRENGTHS:\n- The novel analysis for posterior sampling algorithm in the setting of Bayesian regret;\n- The presented result holds not only in the setting of tabular MDPs but also in linear and finite mixture MDPs.\n\nWEAKNESSES:\n- The weak notion of Bayesian regret is the main weakness of the presented result. Currently, there exists near-optimal results in posterior-sampling based algorithms in the frequentist setting (see section Questions for precise references).\n- The computational side of the presented algorithm was not discusses.\n- The upper bound in linear setup seems to be a contradiction with established lower bound in the setup of linear contextual bandits (see reference below for example). This effect requires additional explanations why is it possible in the presented setting.\n- Lattimore, Tor, and Csaba Szepesvári. *Bandit algorithms*. Cambridge University Press, 2020.\n\nQUESTIONS:\n- Is there any results in literature where this type of dimension were called “Kolmogorov”? Seems that this definition is a just usual covering dimension (at least in the presented setup of $\\ell_1$ distance).\n- Is there examples where $\\lambda$ is much smaller than $H$?\n- What is the limitation to show this analysis for a more general class of MDPs such as MDPs with a finite Eluder dimension?\n- Where is topological structure of $S$ and $A$ were used during the proofs? What is the topological structure of them?\n- The missed references of frequentist regret bounds for TS-based exploration in tabular and linear MDPs:\n    - Zanette, Andrea, et al. \"Frequentist regret bounds for randomized least-squares value iteration.\" *International Conference on Artificial Intelligence and Statistics*. PMLR, 2020.\n    - Agrawal, Shipra, and Randy Jia. \"Optimistic posterior sampling for reinforcement learning: worst-case regret bounds.\" *Advances in Neural Information Processing Systems* 30 (2017).\n    - Tiapkin, Daniil, et al. \"Optimistic posterior sampling for reinforcement learning with few samples and tight guarantees.\" *Advances in Neural Information Processing Systems* 35 (2022): 10737-10751.\n    - Agrawal, Priyank, Jinglin Chen, and Nan Jiang. \"Improved worst-case regret bounds for randomized least-squares value iteration.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 35. No. 8. 2021.\n- Line 83: In [32] they consider not episodic setup. The state-of-the-art result in the episodic setup were presented in [4].\n\nLIMITATIONS:\nThe paper presented the theoretical research on Bayesian regret for posterior-sampling algorithms for reinforcement learning, thus it does not require discussion of ethical limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem setting of Bayesian reinforcement learning, in which both the transition function and the reward function are sampled from a known prior distribution. The authors study Thompson sampling in this setting and prove a Bayesian regret bound of order O(\\lambda \\sqrt{ d T}) where \\lambda is the average value diameter induced by the prior, d is the Kolmogorov dimension (with respect to l1 distance) of the environment, and T is the number of episodes the learner interacts with the environment.\n\nThe authors instantiate their general result to two previously studied settings. \n\n- In the tabular setting, the authors show that their main result implies a regret bound that matches the previous state of the art by Osband and Van Roy (2017), but generalized to hold over any prior distribution rather than specific to the product of Dirichlet distribution prior considered by Osband and Van Roy.\n\n- In the linear MDP setting, the authors give a regret bound of $O(\\lambda \\sqrt{d^f T})$, where $d^f$ is the Kolmogorov l1 dimension of the feature space. The authors also present a counterexample showing that the previously claimed state of the art due to Hao and Lattimore (2022) was in fact incorrect.\n\n\nThe paper also provides corollaries for specialized finite mixtures settings.\n\nSTRENGTHS:\nThe paper's main contribution is a general treatment of Thompson sampling in MDPs. Specifically, the paper claims to provide the most general results to date on the Bayesian regret of Thompson sampling in RL. These results appear to generalize the results of Osband and Van Roy (2017) in the tabular case, and, when their counterexample to Hao and Lattimore (2022) is taken into account, provides the tightest upper bounds in the linear MDP case. The paper clearly signposts these contributions and provides proofs for their claims.\n\nWEAKNESSES:\nSome of the theorem statements/assumptions could be more explicit. For example, the strong consistency assumption (Assumption 1) is not rigorously defined until Appendix J. Also, in my reading of the proof of Theorem 4, it appears that there is another assumption needed in the statement (Assumption 2 from Appendix D). I think it is also worth pointing out that T_0 in Theorem 4 is doing a lot of heavy lifting, as it seems like it could actually be a very large constant, depending on the prior and the structure of the RL problem.\n\nQUESTIONS:\nOn line 138, it is claimed that the law of Thompson sampling aligns with the true posterior distribution. Is this true with no other assumptions? For example, if two environments induce the same optimal policy, then it seems like this would not be true.\n\nLIMITATIONS:\nThe paper does point out that they do not have lower bounds to substantiate the tightness of their upper bounds. Another limitation of the paper, not mentioned by the authors, is that the analysis is limited to the setting where the prior is specified with perfect accuracy.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}