SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ahmadreza Moradipari* † Mohammad Pedramfar* ‡ Modjtaba Shokrian Zini* §
In this paper, we prove the first Bayesian regret bounds for Thompson Sampling in reinforcement learning in a multitude of settings. We simplify the learning problem using a discrete set of surrogate environments, and present a refined analysis of the information ratio using posterior consistency. This leads to an upper bound of order  $\tilde{O}(H\sqrt{d_{l_1}T})$  in the time inhomogeneous reinforcement learning problem where  $H$  is the episode length and  $d_{l_1}$  is the Kolmogorov  $l_1$ -dimension of the space of environments. We then find concrete bounds of  $d_{l_1}$  in a variety of settings, such as tabular, linear and finite mixtures, and discuss how our results are either the first of their kind or improve the state-of-the-art.
Reinforcement Learning (RL) is a sequential decision-making problem in which an agent interacts with an unknown environment typically modeled as a Markov Decision Process (MDP) [38, 8]. The goal of the agent is to maximize its expected cumulative reward. This problem has a variety of applications, including robotics, game playing, resource management, and medical treatments. The key challenge in RL is to balance the so-called exploration-exploitation trade-off efficiently: exploring unseen state-action pairs to gain more knowledge about the unknown environment or exploiting the current knowledge to maximize the expected cumulative reward. Two efficient approaches have been developed to control this trade-off: optimism in the face of uncertainty (OFU) and Thompson Sampling (TS) (or Posterior Sampling (PS)). OFU constructs a confidence set of statistically plausible MDPs that includes the true MDP with high probability and plays an optimistic policy according to the MDP with maximum gain from this set [5, 40]. TS samples a statistically plausible MDP from a posterior distribution and plays the optimistic policy of the sampled MDP [29, 31]. In this work, we focus on the latter, and by combining an information theoretical approach first introduced by [36] with analysis based on posterior consistency tools, we prove state-of-the-art Bayesian regret bounds in a variety of settings.
In this paper, we start by defining the Bayesian RL problem, where transition and reward functions are Bayesian and time inhomogeneous. The Bayesian RL problem we consider is more comprehensive than in previous works, as we allow for both Bayesian transition and Bayesian rewards, and do not make any assumption on their individual prior. To simplify the learning problem, we utilize the notion of surrogate environments, which is a discretization of the environments space, and its learning task and TS regret is a proxy to that of the main problem. The construction of the surrogate environments was first introduced by [18] with an incorrect proof, which is fixed in our work by
*Authors have equal contribution.
$\dagger$  Toyota Motor North America, InfoTech Labs, Mountain View, CA, USA, ahmadreza.moradipari@toyota.com
$\ddagger$  Purdue University, West Lafayette, IN, USA, mpedramf@purdue.edu
$ modjtaba.shokrianzini@gmail.com
Purdue University, West Lafayette, IN, USA, vaneet@purdue.edu
This work was supported in part by the National Science Foundation under grant CCF-2149588 and Cisco, Inc.
defining the surrogate environments through an optimization. Of main importance is the size of this new environment space. The Bayesian regret decomposes to the product of two terms, one being the cumulative mutual information of the environment and history traversed by the policy. By the well-known entropy estimation of the mutual information, this significant factor in the regret is connected to the  $l_{1}$ -dimensions  $(d_{l_1})$  of the transition and reward functions space, which can be more succinctly interpreted as the  $l_{1}$ -dimension  $d_{l_1}$  of the environment space. The latter is in turn estimated by the size of the space of surrogate environments.
The information ratio, representing a trade-off of exploration/exploitation, is the other significant term in the decomposition of the TS Bayesian regret. In an improvement to [18], our novel analysis of this ratio based on posterior consistency tools, shows that this trade-off is bounded by  $H^{3/2}$ , where  $H$  is the episode length. This bound is general and independent of the dimension of transition/reward function space at each step, which is a key factor behind the advantage of our regret bound, such as the  $\sqrt{SA}$  advantage in the tabular case compared to [18], or the lack of any restriction on the prior (e.g., Dirichlet prior) compared to [31]. Following a further refined approach, we finally estimate the TS Bayesian regret to be  $\widetilde{O}(\lambda \sqrt{d_{l_1}T})$  for large enough  $T$  in the time inhomogeneous setting. Here, a new term 'value diameter'  $\lambda$ , which is the average difference of the optimal value functions at different states, is used in bounding the information ratio, where instead of  $H^{3/2}$ , we have the smaller term  $\lambda H^{1/2}$ . Bounding the information ratio with  $\lambda$  is a conceptual contribution of our work, which shows that the ratio is bounded by a value-dependent term, which is in nature different from  $H$  but always  $\leq H + 1$ . Further, there exists another bound for  $\lambda$ ; in environments where states are reachable from one another in  $D$  steps, we have  $\lambda \leq D + 1$ . In 'well-connected' MDPs, one could have  $D \ll H$ , implying an improvement over the  $H^{3/2}$  information ratio bound.
Our generic bound is abstract in terms of  $d_{l_1}$ , so we estimate it in more explicit terms for useful applications. [18] have bounded  $d_{l_1}$  in the tabular and linear case without formalizing this notion, and while for tabular MDPs,  $d_{l_1}$  was bounded by  $SAH$ , for linear MDPs with feature space dimension  $d_f$ , we investigate their claim of the bound  $d_fH$ . Detailed in Appendix G, we show a counterexample to their analysis, and we manage to find a correct estimate in this setting. We also introduce finite mixtures MDPs and are the first to prove a TS Bayesian regret of order  $\widetilde{O}(\lambda \sqrt{HmT})$ , where  $m$  is the number of mixtures.
Lastly, we note that our regret bound of order  $\widetilde{O} (\lambda \sqrt{d_{l_1}T})$  is the first in the general nonlinear time inhomogeneous Bayesian RL setting for TS, and generalizing [31, Conj. 1], we conjecture it to be optimal if  $\lambda$  can be replaced by  $\widetilde{O} (\sqrt{H})$ .
Related work. Since the introduction of information ratio by [35, 36], a new line of research has emerged to provide tighter regret bounds for TS. The general approach involves factoring the Bayesian regret into two components: an information ratio that captures the trade-off between optimal action selection and information gain, and a cumulative information gain term that depends on the target environment and the history of previous observations. Then, both components are bounded separately using information theoretic tools.
In the bandit setting, this analysis has been used to bound Bayesian regret for TS [14, 9], as well as that of a new algorithm called information-directed sampling (IDS) [35, 26, 23, 19, 20]. This analysis has also been used in partial monitoring [25, 24] and RL with a specific Dirichlet prior and additional assumptions [28, 27] or when the true environment is too complicated to learn [4]. More recently, [18] studied the Bayesian regret of TS in RL without any prior assumptions for tabular MDP. This is the closest work to our paper and we discuss our generalization in detail in Section 5.
The Bayesian tabular MDP case has also been studied with the additional Dirichlet prior assumption in [31], where they achieve a regret bound matching ours. In an independent approach, the first non-linear Bayesian RL model was considered by [16] with a regret bound of  $dH^{3/2}T^{1/2}$  where  $d$  is a notion of dimension of their model, but their results were limited to Gaussian process settings with linear kernels. Finally, [11] considered general non-linear Bayesian RL models and introduced an algorithm that obtains  $dH^{1+\alpha/2}T^{1-\alpha/2}$  where  $\alpha$  is a tuning parameter and  $d$  is the dimension of  $\mathcal{S} \times \mathcal{A} \times \mathcal{S}$ .
It is worth noting that there is another line of work that incorporates confidence regions into TS to achieve Bayesian regret bounds that can match the best possible frequentist regret bounds by
UCB in both bandit settings [35] and RL [31, 30, 32, 12]. However, this technique often results in a sub-optimal Bayesian regret, as the best bound known for UCB itself is not optimal.
Table 1: Bayesian regret bounds for TS (i.e. PSRL)
Reference	Tabular	Linear	General	Comments
[29]	\(\sqrt{H^3S^2AL}\)	-	-	-
[30]	-	-	\(L^* \sqrt{d_K d_E HL}\)	Uses Eluder dimension
Lipschitz assumption
[31]	\(\sqrt{H^3 SAL}\)	-	-	Dirichlet prior
[28]	\(\sqrt{H^3 SAL}\)	-	-	Assumptions on prior
[12]	\(L^* \sqrt{H^3 S^2 A^2 L}\)	-	\(L^* \gamma \sqrt{HL}\)	Assumptions on regularity & noise
Lipschitz assumption
[18]	\(\sqrt{H^4 S^2 A^2 L}\)	-	-	-
This paper	\(\lambda \sqrt{H^2 SAL}\)	\(\lambda \sqrt{d_{l_1} HL}\)	\(\lambda \sqrt{d_{l_1} HL}\)	Assumptions 1 & 2
Holds in the limit L → ∞
As discussed in Section 4.3 of [16], the Lipschitz term  $L^{*}$ , which is used in the grayed papers in the table, may grow exponentially in episode length. Note that [18] claims a regret bound for the linear setting. However, as discussed in Appendix G.1, their proof is incorrect.
While our work's emphasis is on theoretical guarantees for TS, we discuss here the experiments using this algorithm. Previous works on PSRL [35, 26, 23, 20, 31] come with extensive experiments on TS (and/or its variants), and discussions on computational efficiency of PSRL. In particular, experiments in [31] support the assertion that "PSRL dramatically outperforms existing algorithms based on OFU". In addition, PSRL with oracle access has been shown to be the most performant, esp. when compared to recent OFU based UCBVI/UCBVI-B, or even variants of PSRL such as Optimistic PSRL [39, Fig. 1.3]. However, an important limitation in experiments is the need for oracle access to an optimal policy, and that can not be always satisfied efficiently. Nevertheless, clever engineering can make TS work even in large scale Deep RL. Indeed, for general RL settings, the recent work [37] shows how to implement TS in Deep RL on the Atari benchmark and concludes that "Posterior Sampling Deep RL (PSDRL) significantly outperforms previous state-of-the-art randomized value function approaches, its natural model-free counterparts, while being competitive with a state-of-the-art (model-based) reinforcement learning method in both sample efficiency and computational efficiency". In summary, experiments in the literature provide enough support for the empirical performance of TS.
2.1 Finite-horizon MDP
We follow the literature's conventions in our notation and terminology to avoid confusion when comparing results. The environment is a tuple  $\mathcal{E} = (\mathcal{S},\mu_{\mathcal{S}},\mathcal{A},\mu_{\mathcal{A}},H,\{P_h\}_{h = 1}^H,\{r_h\}_{h = 1}^H)$ , where  $\mathcal{S}$  is the topological measurable state space,  $\mathcal{A}$  is the topological measurable action space,  $\mu_{\mathcal{S}}$  and  $\mu_{\mathcal{A}}$  are base probability measures on  $\mathcal{S}$  and  $\mathcal{A}$  respectively,  $H$  is the episode length,  $P_{h}:S\times \mathcal{A}\to \Delta_{\mathcal{S},\mu_{\mathcal{S}}}$  is the transition probability kernel, and  $r_h:\mathcal{S}\times \mathcal{A}\rightarrow \Delta_{[0,1],\mathrm{Lebesgue}}$  is the reward function, where we fix the convention  $r(s,a)\coloneqq \mathbb{E}_x[r(x|s,a)] = \int_0^1 x r(x|s,a)\mathrm{d}x$  as we mostly deal with its mean value. Notice that  $\Delta_{X,\mu}$  is the set of probability distributions over  $X$  that are absolutely continuous with respect to  $\mu$ . We will use  $\Delta_X$  when the base measure is clear from the context. We assume  $\mathcal{S}$ ,  $\mathcal{A}$  are known and deterministic while the transition probability kernel and reward are unknown and random. Throughout the paper, the implicit dependence of  $P_{h}$  and  $r_h$  on  $\mathcal{E}$  should be clear from the context.
Let  $\Theta_h^P$  be the topological function space of  $P_{h}$  and  $\Theta^{P} = \Theta_{1}^{P}\times \dots \times \Theta_{H}^{P}$  be the full function space. The space  $\Theta_h^P$  is assumed to be separable and equipped with prior probability measure  $\rho_h^P$  yielding the product prior probability measure  $\rho^P = \rho_1^P\otimes \dots \otimes \rho_H^P$  for  $\Theta^P$ . The exact same definition with similar notations  $\Theta_h^R,\rho_h^R,\rho^R,\Theta^R$  applies for the reward function. Notice the explicit assumption of time inhomogeneity in these definitions, with all 'layers'  $h$  being independent. The two sets define the set of all environments parametrized by  $\Theta = \Theta_{1}\times \dots \times \Theta_{H}$  where  $\Theta_{h} = \Theta_{h}^{P}\times \Theta_{h}^{R}$ . Note that the prior is assumed to be known to the learner. This setting implies that an environment  $\mathcal{E}$  sampled according to the prior  $\rho = \rho^{P}\otimes \rho^{R}$  is essentially determined by its transition and reward functions pair  $\{(P_h,r_h)\}_{h = 1}^H$ . We simplify the notation to view  $\Theta$  as the set of all environments, i.e., saying
$\mathcal{E} \in \Theta$  should be viewed as  $\{(P_h, r_h)\}_{h=1}^H \in \Theta$ . The space of all possible real-valued functions  $\{(P_h, r_h)\}_{h=1}^H$  has a natural vector space structure. Therefore it is meaningful to discuss the notion of the convex combination of environments. We assume that  $\Theta$  is a convex subspace of the space of all possible environments. This assumption is not restrictive, since we may replace any environment space with its convex hull. Note that we do not assume that the support of the prior is convex.
Remark 1. The case of joint prior may be of interest, but to our knowledge all prior works also take  $\rho^P,\rho^R$  to be independent.
Agent, policy and history. An agent starts at an initial state  $s_1^\ell$ , which is fixed for all episodes  $\ell$ . It observes a state  $s_h^\ell$  at layer  $h$  episode  $\ell$ , takes action  $a_h^\ell$ , and receives reward  $r_h^\ell$ . The environment changes to the next random state  $s_{h+1}^\ell$  with probability  $P_h(s_{h+1}^\ell | s_h^\ell, a_h^\ell)$ . The agent stops acting at  $s_{H+1}$  and the environment is reset to its initial state.
We define  $\mathcal{H}_{\ell,h}$  as the history  $(s_1^\ell, a_1^\ell, r_1^\ell, \ldots, s_h^\ell, a_h^\ell, r_h^\ell)$ . Denote by  $\mathcal{D}_\ell = (\mathcal{H}_{1,H}, \ldots, \mathcal{H}_{\ell-1,H})$  the history up to episode  $\ell$ , where  $\mathcal{D}_1 := \emptyset$ . Finally, let  $\Omega_h = \prod_{i=1}^h (S \times \mathcal{A} \times [0,1])$  be the set of all possible histories up to layer  $h$ .
A policy  $\pi$  is represented by stochastic maps  $(\pi_1,\dots ,\pi_H)$  where each  $\pi_h:\Omega_{h - 1}\times S\to \Delta_{A,\mu_A}$ . Let  $\Pi_S$  denote the entire stationary policy class, stationary meaning a dependence only on the current state and layer and let  $\Pi \subseteq \Pi_S$ .
Value and state occupancy functions. Define the value function  $V_{h,\pi}^{\mathcal{E}}$  as the value of the policy  $\pi$  interacting with  $\mathcal{E}$  at layer  $h$ :
$$
V _ {h, \pi} ^ {\mathcal {E}} (s) := \mathbb {E} _ {\pi} ^ {\mathcal {E}} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s \right], \tag {1}
$$
where  $\mathbb{E}_{\pi}^{\mathcal{E}}$  denotes the expectation over the trajectory under policy, transition, and reward functions  $\pi, P_h, r_h$ . The value function at step  $H + 1$  is set to null,  $V_{H + 1,\pi}^{\mathcal{E}}(\cdot) \coloneqq 0$ . We assume there is a measurable function  $\pi_{\mathcal{E}}^{*}: \Theta \to \Pi$  such that  $V_{h,\pi_{\mathcal{E}}^{*}}^{\mathcal{E}}(s) = \max_{\pi \in \Pi} V_{h,\pi}^{\mathcal{E}}(s)$ ,  $\forall s \in S, h \in [H]$ . The optimal policy  $\pi^{*}$  is a function of  $\mathcal{E}$ , making it a random variable in the Bayesian setting. Lastly, let the state-action occupancy probability measure be  $\mathbb{P}_{\pi}^{\mathcal{E}}(s_h = s, a_h = a)$ , also known as the state occupancy measure under policy  $\pi$  and environment  $\mathcal{E}$ . It follows from the definitions that this measure is absolutely continuous with respect to  $\mu_{S \times A} \coloneqq \mu_S \times \mu_A$ . Let  $d_{h,\pi}^{\mathcal{E}}(s, a)$  denote the Radon-Nikodym derivative so that we have  $d_{h,\pi}^{\mathcal{E}}(s, a) \mathrm{d}\mu_{S \times A} = \mathrm{d}\mathbb{P}_{\pi}^{\mathcal{E}}(s_h = s, a_h = a)$ . We will assume throughout the paper that this density  $d_{h,\pi}^{\mathcal{E}}(s, a)$  is measurable and upper bounded for all  $\pi, \mathcal{E}, s, a, h$ . The upper bound is a reasonable assumption, and it happens trivially in the tabular case ( $d_{h,\pi}^{\mathcal{E}}(s, a) \leq SA$ ). This also happens, e.g., when one assumes that the maps  $(\mathcal{E}, s, a, s', h) \mapsto P_h^{\mathcal{E}}(s'|s, a)$  and  $(\pi, s, a, h) \mapsto \pi_h(a|s)$  are continuous and  $\Theta, S, A$  and the set of all optimal policies (as a subset of  $\Pi$ ) are compact.
2.2 Bayesian regret
We formulate the expected regret over  $L$  episodes and  $T = LH$  total steps in an environment  $\mathcal{E}$  as
$$
\Re_ {L} (\mathcal {E}, \pi) = \mathbb {E} \left[ \sum_ {\ell = 1} ^ {L} \left(V _ {1, \pi_ {\mathcal {E}} ^ {*}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) - V _ {1, \pi^ {\ell}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right)\right) \right], \tag {2}
$$
where the expectation is over the randomness of  $\pi = \{\pi^{\ell}\}_{\ell}$ . The Bayesian regret is  $\mathfrak{B}\Re_L(\pi) = \mathbb{E}[\Re_L(\mathcal{E},\pi)]$ . For Thompson Sampling (TS), the algorithm selects the optimal policy of a given sample  $\mathcal{E}_{\ell}$  picked from the posterior  $\mathcal{E}_{\ell} \sim \mathbb{P}(\mathcal{E} \in \cdot|\mathcal{D}_{\ell})$ :
$$
\pi_ {\mathrm {T S}} ^ {\ell} = \operatorname {a r g m a x} _ {\pi \in \Pi} V _ {1, \pi} ^ {\mathcal {E} _ {\ell}} \left(s _ {1} ^ {\ell}\right). \tag {3}
$$
Importantly, the law of TS aligns with the posterior, i.e.,  $\mathbb{P}(\mathcal{E}|\mathcal{D}_{\ell}) = \mathbb{P}(\pi_{\mathrm{TS}}^{\ell} = \pi_{\mathcal{E}}^{*}|\mathcal{D}_{\ell})$
Remark 2. Note that  $\mathbb{P}(\pi_{\mathrm{TS}}^{\ell} = \pi_{\mathcal{E}}^{*}|\mathcal{D}_{\ell})$  is a probability for a specific measure on the space of optimal policies. To ensure that  $\int_{\Pi^{*}}\mathbb{P}(\pi^{*}|\mathcal{D}_{\ell})\mathrm{d}\rho_{\Pi^{*}} = 1$ , we need an appropriate measure  $\rho_{\Pi^{*}}$  on  $\Pi^{*}$ . Given the law of TS, the natural choice for this measure is the push-forward of the prior measure  $\rho$  under the map  $star:\Theta \to \Pi^{*}$ , where  $star(\mathcal{E}) = \pi_{\mathcal{E}}^{*}$ .
2.3 Notations
For Bayesian RL, conditional expressions involving a given history  $\mathcal{D}_{\ell}$  are widely used. We adopt the notation in [18] to refer to such conditionals; let  $\mathbb{P}_{\ell}(\cdot)\coloneqq \mathbb{P}(\cdot |\mathcal{D}_{\ell})$ $\mathbb{E}_{\ell}[\cdot ]\coloneqq \mathbb{E}[\cdot |\mathcal{D}_{\ell}]$ . We can rewrite the Bayesian regret as
$$
\mathfrak {B R} _ {L} (\pi) = \sum_ {\ell = 1} ^ {L} \mathbb {E} \left[ \mathbb {E} _ {\ell} \left[ V _ {1, \pi_ {\varepsilon} ^ {*}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) - V _ {1, \pi} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) \right] \right] \tag {4}
$$
and define the conditional mutual information  $\mathbb{I}_{\ell}(X;Y)\coloneqq D_{\mathrm{KL}}(\mathbb{P}((X,Y)\in \cdot |\mathcal{D}_{\ell})||\mathbb{P}(X\in \cdot |\mathcal{D}_{\ell})\otimes$ $\mathbb{P}(Y\in \cdot |\mathcal{D}_\ell))$  . For a random variable  $\chi$  and random policy  $\pi$  , the following will be involved in the information ratio:
$$
\mathbb {I} _ {\ell} ^ {\pi} (\chi ; \mathcal {H} _ {\ell , h}): = \mathbb {I} _ {\ell} (\chi ; \mathcal {H} _ {\ell , h} | \pi) = \mathbb {E} _ {\pi} \left[ D _ {\mathrm {K L}} \left(\mathbb {P} _ {\ell} \left(\left(\chi , \mathcal {H} _ {\ell , h}\right) \in \cdot | \pi\right) \mid \mid \mathbb {P} _ {\ell} \left(\chi \in \cdot | \pi\right) \otimes \mathbb {P} _ {\ell} \left(\mathcal {H} _ {\ell , h} \in \cdot | \pi\right)\right) \right], \tag {5}
$$
Note that  $\mathbb{E}[\mathbb{I}_{\ell}(X;Y)] = \mathbb{I}(X;Y|\mathcal{D}_{\ell})$ . To clarify,  $\mathbb{P}_{\ell}(\mathcal{H}_{\ell,h}\in \cdot |\pi)$  is the probability of  $\mathcal{H}_{\ell,h}$  being generated under  $\pi$  within some environment. Given that the histories under consideration are generated by the TS algorithm, they are always generated in the true environment  $\mathcal{E}$  under an optimal policy  $\pi_{\mathcal{E}'}^*$ . For  $\pi = \pi_{\mathrm{TS}}^{\ell}$ , this can be computed as  $\mathbb{P}_{\ell}(\mathcal{H}_{\ell,h}|\pi) = \int_{\mathcal{E}}P(\mathcal{H}_{\ell,h}|\pi,\mathcal{E})\mathrm{d}\mathbb{P}_{\ell}(\mathcal{E})$ , where  $P(\mathcal{H}_{\ell,h}|\pi,\mathcal{E})$  is an expression in terms of transition and reward functions of  $\mathcal{E}$  and  $\pi$ .
Finally, we define  $\bar{\mathcal{E}}_{\ell}$  as the mean MDP where  $P_h^{\bar{\mathcal{E}}_\ell}(\cdot |s,a) = \mathbb{E}_\ell [P_h^{\mathcal{E}}(\cdot |s,a)]$  is the mean of posterior measure, and similarly for  $r_h^{\bar{\mathcal{E}}_\ell}(\cdot |s,a) = \mathbb{E}_\ell [r_h^{\mathcal{E}}(\cdot |s,a)]$ . We note that under the independence assumption across layers, the same is given for the state-occupancy density  $d_{h,\pi}^{\bar{\mathcal{E}}_{\ell}} = \mathbb{E}_{\ell}[d_{h,\pi}^{\mathcal{E}}]$ .
3 Bayesian RL problems
Definition 1. A Bayesian RL in this paper refers to the time-inhomogeneous finite-horizon MDP with independent priors on transition and reward functions, as described in Section 2.1.
The Bayesian RL problem is the task of finding an algorithm  $\pi$  with optimal Bayesian regret as defined in Eq. (4). Below we list the variations of this problem. A setting considered by most related works such as [31, 16] is the following:
Definition 2. The time (reward) homogeneous Bayesian RL refers to the Bayesian RL setting where the prior  $\rho^P (\rho^R)$  is over the space  $\Theta^P (\Theta^R)$  containing the single transition (reward) function  $P(r)$  defining  $\mathcal{E}$ , i.e., all layers have the same transition (reward) functions.
Definition 3. The tabular Bayesian RL is a Bayesian RL where  $S, \mathcal{A}$  are finite sets.
Definition 4 (Linear MDP [41, 22]). Let  $\phi^P: \mathcal{S} \times \mathcal{A} \to \mathbb{R}^{d_f^P}$ ,  $\phi^R: \mathcal{S} \times \mathcal{A} \to \mathbb{R}^{d_f^R}$  be feature maps with bounded norm  $\| \phi^P(s,a)\|_2$ ,  $\| \phi^R(s,a)\|_2 \leq 1$ . The linear Bayesian RL is a Bayesian RL where for any  $\mathcal{E} = \{(P_h^\mathcal{E}, r_h^\mathcal{E})\}_{h=1}^H \in \Theta$ , there exists vector-valued maps  $\psi_h^{P,\mathcal{E}}(s), \psi_h^{R,\mathcal{E}}(s)$  with bounded  $l_2$ -norm such that for any  $(s,a) \in S \times \mathcal{A}$ ,
$$
P _ {h} ^ {\mathcal {E}} (\cdot | s, a) = \left\langle \phi^ {P} (s, a), \psi_ {h} ^ {P, \mathcal {E}} (\cdot) \right\rangle , r _ {h} ^ {\mathcal {E}} (\cdot | s, a) = \left\langle \phi^ {R} (s, a), \psi_ {h} ^ {R, \mathcal {E}} (\cdot) \right\rangle \tag {6}
$$
A restricted version of the finite mixtures called linear mixture was first considered in [6] in the frequentist setting. Here, we consider the general setting.