SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 5. The finite mixtures Bayesian RL is a Bayesian RL where for any  $h \in [H]$  there exists fixed conditional distributions  $\{Z_{h,i}^{P}: \mathcal{S} \times \mathcal{A} \to \Delta_{\mathcal{S}}\}_{i=1}^{m_{h}^{P}}$  and  $\{Z_{h,i}^{R}: \mathcal{S} \times \mathcal{A} \to \Delta_{[0,1]}\}_{i=1}^{m_{h}^{R}}$ , such that for any environment  $\mathcal{E}$  given by  $\{(P_{h}^{\mathcal{E}}, r_{h}^{\mathcal{E}})\}_{h=1}^{H}$ , there exists parametrized probability distributions  $\pmb{a}_{h}^{P,\mathcal{E}}: \mathcal{S} \times \mathcal{A} \to \Delta_{m_{h}^{P}}, \pmb{a}_{h}^{R,\mathcal{E}}: \mathcal{S} \times \mathcal{A} \to \Delta_{m_{h}^{R}}$  such that
$$
P _ {h} ^ {\mathcal {E}} (\cdot | s, a) = \sum_ {i = 1} ^ {m _ {h} ^ {P}} a _ {h, i} ^ {P, \mathcal {E}} (s, a) Z _ {h, i} ^ {P} (\cdot | s, a), r _ {h} ^ {\mathcal {E}} (\cdot | s, a) = \sum_ {i = 1} ^ {m _ {h} ^ {R}} a _ {h, i} ^ {R, \mathcal {E}} (s, a) Z _ {h, i} ^ {R} (\cdot | s, a) \tag {7}
$$
4 Surrogate learning
Next, we define the discretized surrogate learning problem, and bound the size of the surrogate environments space, a significant term in the regret. To do so, we need to first define the Kolmogorov dimension of a set of parametrized distributions, esp. working out the case of  $l_{1}$ -distance. In the definitions below, we implicitly assume any required minimal measurability assumptions on the involved sets.
Definition 6. Given a set  $\mathcal{F}$  of  $\mathcal{O}$ -parametrized distributions  $P: \mathcal{O} \to \Delta(\mathcal{S})$  over a set  $\mathcal{S}$  where both  $\mathcal{O}, \mathcal{S}$  are measurable. Let  $\mathcal{M}(\cdot, \cdot): \mathcal{F} \times \mathcal{F} \to \mathbb{R}^{\geq 0}$  be a distance, i.e.,  $\mathcal{M}(P, Q) \geq 0 \stackrel{=}{\longleftrightarrow} P = Q$ . Then its right  $\varepsilon$ -covering number is the size  $K_{\mathcal{M}}(\varepsilon)$  of the smallest set  $\mathcal{C}_{\mathcal{M}}(\varepsilon) = \{P_1, \ldots, P_{K_{\mathcal{M}}(\varepsilon)}\} \subset \mathcal{F}$  such that
$$
\forall P \in \mathcal {F}, \exists P _ {j} \in \mathcal {C} _ {\mathcal {M}} (\varepsilon): \mathcal {M} (P, P _ {j}) \leq \varepsilon . \tag {8}
$$
The potential asymmetry of  $\mathcal{M}$  (e.g., KL-divergence) requires the notion of left/right covering number. The right covering number will be the default, so covering number will always refer to that.
Definition 7. Let  $d_{\mathcal{M}}(\varepsilon) = \log (K_{\mathcal{M}}(\varepsilon))$ . Define the Kolmogorov  $\mathcal{M}$ -dimension  $d_{\mathcal{M}}$  of  $\mathcal{F}$  as
$$
d _ {\mathcal {M}} = \lim  _ {\varepsilon \rightarrow 0} \sup  _ {\varepsilon} \frac {d _ {\mathcal {M}} (\varepsilon)}{\log \left(\frac {1}{\varepsilon}\right)}. \tag {9}
$$
For  $l_{1}(P,Q)\coloneqq \sup_{o\in \mathcal{O}}||P(\cdot |o) - Q(\cdot |o)||_{1}$ , applying Definition 6 to the sets  $\Theta_h^P,\Theta_h^R$  with  $\mathcal{O} = S\times \mathcal{A}$ , and denote the respective covering numbers by  $L_h^P (\varepsilon),L_h^R (\varepsilon)$  corresponding to covering sets  $\mathcal{C}_h^P (\varepsilon),\mathcal{C}_h^R (\varepsilon)$ . Similarly applying Eq. (9) and denote the corresponding  $l_{1}$ -dimensions by  $d_{l_1,h}^P (\varepsilon),d_{l_1,h}^R (\varepsilon),d_{l_1,h}^P,d_{l_1,h}^R$  and  $d_{l_1}^P\coloneqq \sum_h d_{l_1,h}^P,d_{l_1}^R\coloneqq \sum_h d_{l_1,h}^R$ . The sums  $d_{l_1,h}\coloneqq d_{l_1,h}^P + d_{l_1,h}^R,d_{l_1}\coloneqq d_{l_1}^P + d_{l_1}^R$  can be interpreted as the  $l_{1}$ -dimension of  $\Theta_h$  and  $\Theta$ , i.e., the environment space.
Remark 3. We can also apply this framework to the KL-divergence, by  $\mathcal{M}_{\mathrm{KL}}(P,Q) := \sup_{o\in \mathcal{O}}D_{\mathrm{KL}}(P(\cdot |o)||Q(\cdot ||o))$ . This was implicitly used by [18] to prove their regret bound in the tabular case. Note that Pinsker's lemma (Lemma 9) implies that the KL-divergence is larger than the squared total variance, and the latter is trivially larger than the  $l_{1}$  distance. Therefore,  $l_{1}$ -dimension is smaller than  $d_{\mathcal{M}_{\mathrm{KL}}}$ , allowing for tighter regret bounds.
We now revisit the definition of  $\varepsilon$ -value partitions and show their existence is guaranteed by finite  $l_{1}$ -covering numbers. These partitions are the origins of surrogate environments.
Definition 8. Given  $\varepsilon >0$ , an  $\varepsilon$ -value partition for a Bayesian RL problem is a partition  $\{\Theta_k\}_{k = 1}^K$  over  $\Theta$  such that for any  $k\in [K]$  and  $\mathcal{E},\mathcal{E}'\in \Theta_{k}$
$$
V _ {1, \pi_ {\varepsilon} ^ {*}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) - V _ {1, \pi_ {\varepsilon} ^ {*}} ^ {\mathcal {E} ^ {\prime}} \left(s _ {1} ^ {\ell}\right) \leq \varepsilon . \tag {10}
$$
A layered  $\varepsilon$ -value partition is one where the transition functions are independent over layers after conditioning on  $k$ . Throughout this paper, we will only consider layered  $\varepsilon$ -value partition. We define  $K_{\mathrm{surr}}(\varepsilon)$  as the minimum  $K$  for which there exists a layered  $\varepsilon$ -value partition.
Inspired by Eq. (9), we define the surrogate dimension as  $d_{\mathrm{surr}} = \lim \sup_{\varepsilon \to 0} \frac{K_{\mathrm{surr}}(\varepsilon)}{\log(1 / \varepsilon)}$ .
Lemma 1. Given a Bayesian RL, we have  $K_{\mathrm{surr}}(\varepsilon) \leq \prod_h L_h^P (\varepsilon /(2H)^2) \times L_h^R (\varepsilon /(4H))$ . This implies  $d_{\mathrm{surr}} \leq d_{l_1}$ .
The above is proved in Appendix B. It is hard to find  $d_{\mathrm{surr}}$ , but one can estimate  $d_{l_1}$ , and according to the above, this acts as a proxy for  $K_{\mathrm{surr}}$ . This is useful as the regret relates to  $K_{\mathrm{surr}}$ . But to show this, we need to construct surrogate environments inside each partition, and show that learning those is almost equivalent to the original problem. Let  $\zeta$  be a discrete random variable taking values in  $\{1,\dots ,K_{\mathrm{surr}}(\varepsilon)\}$  that indicates the partition  $\mathcal{E}$  lies in, such that  $\zeta = k$  if and only if  $\mathcal{E}\in \Theta_k$ .
Lemma 2. For any  $\varepsilon$ -value partition and any  $\ell \in [L]$ , there are random environments  $\tilde{\mathcal{E}}_{\ell}^{*} \in \Theta$  with their laws only depending on  $\zeta, \mathcal{D}_{\ell}$ , such that
$$
\mathbb {E} _ {\ell} \left[ V _ {1, \pi_ {\varepsilon} ^ {*}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) - V _ {1, \pi_ {T S} ^ {\ell}} ^ {\mathcal {E}} \left(s _ {1} ^ {\ell}\right) \right] - \mathbb {E} _ {\ell} \left[ V _ {1, \pi_ {\varepsilon} ^ {*}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) - V _ {1, \pi_ {T S} ^ {\ell}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) \right] \leq \varepsilon . \tag {11}
$$
The expectation in both equations is over  $\mathcal{E}$  and  $\pi_{TS}^{\ell} \in \{\pi_{\mathcal{E}'}^*\}_{\mathcal{E}' \in \Theta}$ , with both sampled independently  $\sim \mathbb{P}_{\ell}(\cdot)$ , and the  $K$  different values of  $\tilde{\mathcal{E}}_{\ell}^{*}$ . The second expectation over  $(\tilde{\mathcal{E}}_{\ell}^{*}, \mathcal{E})$  is over pairs that are in the same partition, i.e.,  $\tilde{\mathcal{E}}_{\ell}^{*}, \mathcal{E}$  are independent only after conditioning on  $\zeta$ .
We note that the proof in [18, App. B.1] contains the use of a lemma that does not apply to construct the law of the environment  $\tilde{\mathcal{E}}_{\ell}^{*}$ . More details is provided in Appendix C, where we find  $\tilde{\mathcal{E}}_{\ell}^{*}$  by minimizing an expected value of  $\pi_{\mathrm{TS}}^{\ell}$ .
5 Bayesian regret bounds for Thompson Sampling
5.1 General Bayesian regret bound
We start by introducing the notion of value diameter.
Definition 9. Given the environment  $\mathcal{E}$ , its value diameter is defined as
$$
\lambda_ {\mathcal {E}} := \max  _ {1 \leq h \leq H} \left(\sup  _ {s} V _ {h, \pi_ {\mathcal {E}} ^ {*}} ^ {\mathcal {E}} (s) - \inf  _ {s} V _ {h, \pi_ {\mathcal {E}} ^ {*}} ^ {\mathcal {E}} (s)\right) + \max  _ {1 \leq h \leq H, s \in \mathcal {S}, a \in \mathcal {A}} \left(r _ {h} ^ {\sup } (s, a) - r _ {h} ^ {\inf } (s, a)\right),
$$
where  $r_h^{\mathrm{sup}}(s,a)$  (and  $r_h^{\inf}(s,a)$ ) is the supremum (and infimum) of the set of rewards that are attainable under the distribution  $r_h(s,a)$  with non-zero probability. As a special case, if rewards are deterministic, then we have  $r_h^{\mathrm{sup}}(s,a) = r_h^{\inf}(s,a)$  for all  $s,a$ . The (average) value diameter over  $\Theta$  is denoted by  $\lambda \coloneqq \mathbb{E}_{\mathcal{E}\sim \rho}[\lambda_{\mathcal{E}}^2 ]^{1 / 2}$ .
As the value function is between 0 and  $H$ , we have  $\lambda_{\mathcal{E}} \leq H + 1$  implying  $\lambda \leq H + 1$ . Note that value diameter is closely related to the notion of diameter commonly defined in finite RL problems. Strictly speaking, for a time-homogeneous RL, it is straightforward to see that the value diameter is bounded from above by one plus the diameter [33].
We now discuss the assumptions surrounding our results. The main technical assumption of this paper is the existence of consistent estimators, which as we will see in Appendix K, is closely related to the notion of posterior consistency:
Assumption 1. There exists a strongly consistent estimator of the true environment given the history.
Roughly speaking, we assume that with unlimited observations under TS, it is possible to find the true environment. For this assumption to fail, we need to have two environments that produce the same distribution over histories under TS and are therefore indistinguishable from the point of view of TS. The precise description of this assumption is detailed in Appendix K.
Another necessary technical assumption is that almost all optimal policies visit almost all state action pairs in their respective environment.
Assumption 2. For almost every environment  $\mathcal{E} \in \Theta$  and almost every  $(s, a) \in S \times \mathcal{A}$  and every  $h \in [H]$ , we have
$$
d _ {h, \pi_ {\mathcal {E}} ^ {*}} ^ {\mathcal {E}} (s, a) \neq 0.
$$
Recall that, for any environment  $\mathcal{E} \in \Theta$ , the policy  $\pi_{\mathcal{E}}^{*}$  is the optimal policy of  $\mathcal{E}$  within the policy class  $\Pi$ . Therefore, one example of how the above assumption holds is when  $\Pi$  is the set of  $\varepsilon$ -greedy algorithms and transition functions of environments assign non-zero probability to every state. Under these assumptions, we discuss our main result and its corollaries.
Theorem 3. Given a Bayesian RL problem, for all  $\varepsilon >0$ , we have
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq 2 \lambda \sqrt {\log \left(K _ {\text {s u r r}} (\varepsilon)\right) T} + L \varepsilon + T _ {0} \tag {12}
$$
where  $T_0$  does not depend on  $T$ . This can be further upper bounded by
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {d _ {l _ {1}} T}\right). \tag {13}
$$
for large enough  $T$ . Given a homogeneous  $l_{1}$  dimension  $d_{\mathrm{hom}} = d_{l_1,h}, \forall h$ , this simplifies to
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \tilde {O} \left(\lambda \sqrt {H d _ {\text {h o m}} T}\right). \tag {14}
$$
Remark 4. For all regret bounds, we will replace  $\lambda \leq H + 1$  to compare our result. For the case of homogeneous dimensions, we obtain  $\widetilde{O}(H^{3/2}\sqrt{d_{\mathrm{hom}}T})$ . Crucially, our main result shows a new conceptual understanding of the information ratio by bounding it by two terms of different nature:  $H$  and  $\lambda$ , where the latter can be bounded by either the largest diameter of the environments or  $H$ .
Remark 5. Despite not impacting the asymptotics, the impact of  $T_0$  can be large depending on the structure of the RL problem, and could be dominant even for large  $T_s$  in practice.
Remark 6. Considering time as a part of the state observation, one could apply this regret analysis to particular time-homogeneous settings. However, this mapping of time-inhomogeneous RLs to homogeneous ones is not surjective, hence the result above does not readily extend to time-homogeneous settings.
While [16] were the first to consider a nonlinear Bayesian RL model, their bound is limited to the Gaussian process (with linear kernel) setting, while ours in the nonlinear time inhomogeneous setting makes no assumptions on the prior and is the first such bound. Our novel analysis allows us to upper bound the information ratio by  $\lambda \sqrt{H}$  instead of, for example  $H^{3/2} \sqrt{SA}$  ([18]) in the tabular case, improving the regret bound by a square root relevant to the dimension  $d$  of the problem.
The detailed proof is given in Appendix D. Following [18], the regret (4) is rewritten using Lemma 2 to reduce the problem into its surrogate, and we use the well-known information-ratio trick by multiplying and dividing by the mutual information. We follow that with a Cauchy-Schwarz, summarized below
$$
\begin{array}{l} \mathfrak {B R} _ {L} \left(\pi_ {\mathrm {T S}}\right) \leq \mathbb {E} \left[ \sum_ {\ell = 1} ^ {L} \frac {\mathbb {E} _ {\ell} \left[ V _ {1 , \pi_ {\varepsilon} ^ {*}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) - V _ {1 , \pi_ {\mathrm {T S}} ^ {\ell}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) \right]}{\sqrt {\mathbb {I} _ {\ell} ^ {\pi_ {\mathrm {T S}} ^ {\ell}} \left(\tilde {\mathcal {E}} _ {\ell} ^ {*} ; \mathcal {H} _ {\ell , H}\right)}} \sqrt {\mathbb {I} _ {\ell} ^ {\pi_ {\mathrm {T S}} ^ {\ell}} \left(\tilde {\mathcal {E}} _ {\ell} ^ {*} ; \mathcal {H} _ {\ell , H}\right)} \right] + L \varepsilon (15) \\ \leq \sqrt {\mathbb {E} \left[ \sum_ {\ell = 1} ^ {L} \frac {\left(\mathbb {E} _ {\ell} \left[ V _ {1 , \pi_ {\varepsilon} ^ {*}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) - V _ {1 , \pi_ {\ell} ^ {*}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) \right]\right) ^ {2}}{\mathbb {I} _ {\ell} ^ {\pi_ {\mathrm {T S}} ^ {\ell}} \left(\tilde {\mathcal {E}} _ {\ell} ^ {*} ; \mathcal {H} _ {\ell , H}\right)} \right] \mathbb {E} \left[ \sum_ {\ell = 1} ^ {L} \mathbb {I} _ {\ell} ^ {\pi_ {\mathrm {T S}} ^ {\ell}} \left(\tilde {\mathcal {E}} _ {\ell} ^ {*} ; \mathcal {H} _ {\ell , H}\right) \right]} + L \varepsilon (16) \\ \end{array}
$$
Note the cost  $\varepsilon$  at each episode (Lemma 2) in the first inequality, yielding the overall error  $L\varepsilon$ . Then, we can bound the mutual information appearing in the regret term by  $\mathbb{E}\left[\sum_{\ell=1}^{L}\mathbb{I}_{\ell}^{\pi_{\mathrm{TS}}^{\ell}}(\tilde{\mathcal{E}}_{\ell}^{*};\mathcal{H}_{\ell,H})\right] = I_{\ell}^{\pi_{\mathrm{TS}}^{\ell}}(\tilde{\mathcal{E}}_{\ell}^{*};\mathcal{D}_{\ell}) \leq I_{\ell}^{\pi_{\mathrm{TS}}^{\ell}}(\zeta; \mathcal{D}_{\ell}) \leq H(\zeta) \leq \log(K_{\mathrm{surr}}(\varepsilon))$ , where we used the mutual information chain rule, followed by data processing inequality to substitute  $\tilde{\mathcal{E}}_{\ell}^{*} \to \zeta$ , and finally used the trivial bound by the entropy. But the main novelty of our approach lies in our control of the first term
$$
\Gamma_ {\ell} \left(\pi_ {\mathrm {T S}} ^ {\ell}\right) := \frac {\left(\mathbb {E} _ {\ell} \left[ V _ {1 , \pi_ {\varepsilon} ^ {*}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) - V _ {1 , \pi_ {\mathrm {T S}} ^ {\ell}} ^ {\tilde {\mathcal {E}} _ {\ell} ^ {*}} \left(s _ {1} ^ {\ell}\right) \right]\right) ^ {2}}{\mathbb {I} _ {\ell} ^ {\pi_ {\mathrm {T S}} ^ {\ell}} \left(\tilde {\mathcal {E}} _ {\ell} ^ {*}; \mathcal {H} _ {\ell , H}\right)} \tag {17}
$$
called the information ratio. In our analysis, we have the following bound on its expectation.
$$
\mathbb {E} \left[ \Gamma_ {\ell} \left(\pi_ {\mathrm {T S}} ^ {\ell}\right) \mid \mathcal {E} _ {0} \right] \leq \mathbb {E} \left[ \sum_ {h} \int \frac {\mathbb {E} _ {\ell} \left[ \left(\lambda_ {\mathcal {E}} d _ {h , \pi^ {*}} ^ {\bar {\mathcal {E}} _ {\ell}} (s , a)\right) ^ {2} \right]}{\mathbb {E} _ {\ell} \left[ d _ {h , \pi^ {*}} ^ {\bar {\mathcal {E}} _ {\ell}} (s , a) \right]} \mu_ {\mathcal {S} \times \mathcal {A}} \mid \mathcal {E} _ {0} \right],
$$
where the average is taken over all histories  $\mathcal{D}_{\ell}$  that are generated from running TS on the true environment  $\mathcal{E}_0$ , and we have introduced the smaller term  $\lambda_{\mathcal{E}}$  instead of  $H$  in [18]. While [18] essentially bound the above only in the tabular setting with  $SAH^3$ , we manage to generally bound the above with a more precise bound using Doob's consistency theorem. Assumption 1 allows us to use Doob's consistency theorem to conclude that for almost every environment  $\mathcal{E}_0$ , almost every infinite sequence of histories  $(\mathcal{D}_{\ell})_{\ell=1}^{\infty}$  sampled from  $\mathcal{E}_0$ , and every integrable function  $f$ , the posterior mean  $\mathbb{E}_{\ell}[f(\mathcal{E})] = \mathbb{E}[f(\mathcal{E}) \mid \mathcal{D}_{\ell}]$  converges to  $f(\mathcal{E}_0)$ . In particular, we conclude that  $\mathbb{E}[\Gamma_{\ell}(\pi_{\mathrm{TS}}^{\ell}) \mid \mathcal{E}_0]$  tends to  $\lambda_{\mathcal{E}_0}^2 H$  in the limit, allowing us to claim that for large enough  $\ell$ , the expected information ratio  $\mathbb{E}[\Gamma_{\ell}(\pi_{\mathrm{TS}}^{\ell})]$  is uniformly bounded by  $2\mathbb{E}[\lambda_{\mathcal{E}}^2]H = 2\lambda^2H$ . As there are  $L$  many such ratios, the two bounds together yield  $2\sqrt{\lambda^2HL} \cdot \sqrt{\log(K_{\mathrm{surr}}(\varepsilon))} + L\varepsilon$ . This bound is true for large enough  $\ell$ , giving the additional additive term  $T_0$  in the theorem. Since this term is additive, applying Lemma 1 to bound  $\log(K_{\mathrm{surr}}(\varepsilon))$ , we have successfully shown the asymptotic behavior of the regret, independent of the prior, is of order  $\widetilde{O}(H\sqrt{d_{l_1}T})$ .
5.2 Applications
In each application below, the challenge is to bound  $d_{l_1}$  using the specifics of the model, and except for the case of tabular Bayesian RL, such analysis has not been carried out rigorously. We formalize the corollaries and show they are state-of-the-art compared to the literature.
Tabular RL. The result below follows from Theorem 3; the main contribution comes from our new information ratio bound, followed by the estimate  $\widetilde{O}((\frac{1}{\varepsilon})^{SAH})$  of  $K_{\mathrm{surr}}(\varepsilon)$  ([18]).
Corollary 4. Given a tabular Bayesian RL problem, for large enough  $T$ ,
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {H S A T}\right), \tag {18}
$$
where the polylogarithmic terms are explicitly in terms of  $H, S, A, L$ .
We observe that our result matches [31] when their result in the time homogeneous setting (Definition 2) is extended to time inhomogeneous. However, in that paper, the authors assume a Dirichlet based prior which we do not.
Linear RL. A previous state-of-the-art  $\widetilde{O}(d_f H^{3/2} \sqrt{T})$  was claimed by [18] to hold for linear Bayesian RLs with deterministic reward. We note:
- As in the previous cases, their proof in bounding their information ratio includes a factor of  $d_f$ , which ours avoids.
- We show that the proof bounding  $K_{\mathrm{surr}}(\varepsilon)$  in [18, App. B.4] is incorrect, starting with a wrong application of Cauchy-Schwarz and a wrong mutual information in their definition of information ratio. We provide counterexamples for the estimates found therein to substantiate our claim (see Appendix G.1).
To state our own corollary in this case, we need to define a few notions. Let  $d_{l_1}^f = d_{l_1}^{P,f} + d_{l_1}^{R,f}$  be the sum of the  $l_{1}$ -dimensions of the feature map space  $\{\psi_h^{P,\mathcal{E}}\}_{\mathcal{E}\in \Theta},\{\psi_h^{R,\mathcal{E}}\}_{\mathcal{E}\in \Theta}$  where the  $l_{1}$ -distance between feature maps is defined as  $l_{1}(\psi_{h}^{\mathcal{E}},\psi_{h}^{\mathcal{E}^{\prime}}) = \int_{s}\| \psi_{h}^{\mathcal{E}} - \psi_{h}^{\mathcal{E}^{\prime}}\|_{1}\mu_{\mathcal{S}}$  Our corollary also provides a concrete bound in the case of mixture linear Bayesian RL where the feature maps are themselves a sum of finitely many fixed feature maps. This means for all  $\mathcal{E}\in \Theta$ , we have
$$
\psi_ {h} ^ {P, \mathcal {E}} = \sum_ {i = 1} ^ {m _ {h} ^ {P}} a _ {h, i} ^ {P, \mathcal {E}} \Psi_ {h, i} ^ {P} (s), \quad \psi_ {h} ^ {R, \mathcal {E}} = \sum_ {i = 1} ^ {m _ {h} ^ {R}} a _ {h, i} ^ {R, \mathcal {E}} \Psi_ {h, i} ^ {R} (s) \tag {19}
$$
where  $\{\Psi_{h,i}^{P}(s)\}_{i = 1}^{m_h^P},\{\Psi_{h,i}^R (s)\}_{i = 1}^{m_h^R}$  are finitely many fixed feature maps and  $\forall \mathcal{E},h:\sum_{i}|a_{h,i}^{P,\mathcal{E}}|^{2},\sum_{i}|a_{h,i}^{R,\mathcal{E}}|^{2}\leq C_{a}$  for some constant  $C_a > 0$  . Let  $M = M^{P} + M^{R} = \sum_{h}m_{h}^{P} + \sum_{h}m_{h}^{R}$
Corollary 5. For a linear Bayesian RL, for large enough  $T$ ,
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {d _ {l _ {1}} ^ {f} T}\right). \tag {20}
$$
Given a linear Bayesian RL with finitely many states and total feature space dimension  $d_{f} = d_{f}^{P} + d_{f}^{R}$ , we have  $d_{l_1} \leq 2d_fHS$ , yielding for large enough  $T$
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {H d _ {f} S T}\right). \tag {21}
$$
Given a mixture linear Bayesian RL, for large enough  $T$ ,
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} (\lambda \sqrt {M T}), \tag {22}
$$
The proof is given in Appendix G. The fact that  $d_{l_1}$  appears instead of  $d_f$  in the general bound is not counter-intuitive, as we should expect the complexity of the feature map space  $\{\psi_h^{P,\mathcal{E}}(s)\}_{\mathcal{E}\in \Theta ,h\in [H]}$ ,  $\{\psi_h^{R,\mathcal{E}}(s)\}_{\mathcal{E}\in \Theta ,h\in [H]}$  to play a role in the regret, especially as this space can be very complex, and model very different environments that can not be grouped in the same  $\varepsilon$ -value partition.
Therefore, opposite to the claim made by [18], this complexity can not be captured by simply  $d_{f}$  except maybe in degenerate cases, such as when  $\mathcal{S}$  is finite, which is our second statement. More generally, if each feature map  $\psi_h^{P,\mathcal{E}}(s),\psi_h^{R,\mathcal{E}}(s)$  can be characterized with a vector of uniformly bounded norm  $\pmb{a}_{h}^{P,\mathcal{E}}\in \mathbb{R}^{m_{h}^{P}},\pmb{a}_{h}^{R,\mathcal{E}}\in \mathbb{R}^{m_{h}^{R}}$  , then we can bound the regret in terms of  $m_h^P,m_h^R$ 's, as is done in Eq. (22) (the finite state case corresponds to  $m_h^P = d_f^P S,m_h^R = d_f^R S$
Finite mixtures RL. To state our finite mixtures model result, we need to set the following notations. Let  $d_{l_1}^m = d_{l_1}^{m,P} + d_{l_1}^{m,R} = \sum_h d_{l_1,h}^{m,P} + \sum_h d_{l_1,h}^{m,R}$  correspond to the total  $l_1$ -dimension of the space of mixtures coefficient maps  $\{\pmb{a}_h^{P,\mathcal{E}}(s,a)\}_{\mathcal{E}\in \Theta}, \{\pmb{a}_h^{R,\mathcal{E}}(s,a)\}_{\mathcal{E}\in \Theta}$  with  $l_1$ -distance defined as  $l_1(\pmb{a}_h^\mathcal{E},\pmb{a}_h^{\mathcal{E}'}) = \sup_{s,a}\| \pmb{a}_h^\mathcal{E}(s,a) - \pmb{a}_h^{\mathcal{E}'}(s,a)\| _1$ . Define also the restricted finite mixtures model where  $\pmb{a}_h^{P,\mathcal{E}},\pmb{a}_h^{R,\mathcal{E}}$  are vectors in  $\mathbb{R}^{m_h^P},\mathbb{R}^{m_h^R}$  independent of  $(s,a)$  and let  $M = M^P +M^R = \sum_h m_h^P +\sum_h m_h^R$ .
Corollary 6. Given a finite mixtures Bayesian RL problem, for large enough  $T$ ,
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {d _ {l _ {1}} ^ {m} T}\right). \tag {23}
$$
Assuming the restricted finite mixtures model, for large enough  $T$ ,
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq \widetilde {O} \left(\lambda \sqrt {M T}\right). \tag {24}
$$
which, given a uniform dimension  $m = m_h^P = m_h^R$ , yields  $\widetilde{O} (\lambda \sqrt{HmT})$