SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We prove the above in Appendix H, deriving it from our generic bound, after relating the  $l_{1}$ -dimension  $d_{l_{1}}$  of the environment space to that of the mixtures coefficients. To the best of our knowledge, this is the first bound for finite mixtures Bayesian RL problems. We note that in a previous work ([6]), a restricted version of finite mixtures, like in Eq. (24), was considered in the frequentist setting.
We finish this section by proposing the following conjecture, in line with [31, Conj. 1].
Conjecture 7. For the Bayesian RL, the following is true and optimal for all  $T$ :
$$
\mathfrak {B R} _ {L} \left(\pi_ {T S}\right) \leq O \left(\inf  _ {\varepsilon > 0} \left(\sqrt {H \log \left(K _ {\text {s u r r}} (\varepsilon)\right) T} + L \varepsilon\right)\right). \tag {25}
$$
where the constant factor is independent of the prior. This means there exists a Bayesian RL problem such that  $\mathfrak{B}\Re_L(\pi_{TS}) = \tilde{\Omega} (\sqrt{Hd_{\mathrm{surr}}T})$ . All polylogarithmic terms are in terms of  $H, d_{\mathrm{surr}}, T$ .
Note that the above coincides with the lower bound for the (model-based) time inhomogeneous frequentist setting; see e.g., [21] for the proven lower bound for the tabular case. This is also  $\sqrt{H}$  higher (this factor being baked in  $d_{\mathrm{surr}}$ ) than that of the time homogeneous frequentist setting, which is expected, according to [21, App. D]. Note that in this conjecture, the  $\lambda$  in our bound is replaced by  $\sqrt{H}$ , and the conjecture is not for  $T$  large enough, but for all  $T$ . Supporting this conjecture requires experiments where TS can be exactly implemented assuming access to an oracle which provides the optimal policy for a query environment. Simulations have been performed for the similar [31, Conj. 1] in the time homogeneous case. Our conjecture is similar but with the additional expected factor of  $\sqrt{H}$  due to time inhomogeneity, thus their simulation also supports the above.
In this paper, we have addressed the Bayesian Reinforcement Learning (RL) problem in the context of time inhomogeneous transition and reward functions. By considering both Bayesian transition and Bayesian rewards without prior assumptions, we have extended the scope of previous works, making our formulation more comprehensive. To simplify the learning problem, we have introduced surrogate environments, which discretize the environment space. We have established a connection between the size of this new environment space and the  $l_{1}$ -dimensions of the transition and reward functions space, providing insights into the  $l_{1}$ -dimension of the environment space denoted by  $d_{l_1}$ . We have employed posterior consistency tools to analyze the information ratio, which captures the trade-off between exploration and exploitation. We conjecture that (at least a weakened version of) our posterior consistency assumption should hold in general, which is left for future work. Our analysis has resulted in a refined approach to estimate the Bayesian regret in Thompson Sampling (TS), yielding a regret bound of  $\widetilde{O} (\lambda \sqrt{d_{l_1}T})$  for large enough time steps  $T$ . The result is specialized to linear, tabular, and finite mixtures MDPs.
Limitations: While the paper provides asymptotic generic regret bound for TS in a generalized setup which improve the state of the art results, finding lower bounds, esp. one dependent on  $\lambda$ , are left open. In addition, the issue of prior misspecificity is not discussed and left for future studies.
[40] Aristide Tossou, Debabrota Basu, and Christos Dimitrakakis. Near-optimal optimistic reinforcement learning using empirical bernstein inequalities. arXiv preprint arXiv:1905.12425, 2019.
[41] Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995-7004. PMLR, 2019.
[42] Andrea Zanette, David Brandfonbrener, Emma Brunskill, Matteo Pirotta, and Alessandro Lazaric. Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954–1964. PMLR, 2020.
[43] Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. Advances in Neural Information Processing Systems, 33:15198–15207, 2020.
[44] Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.