SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daniil Tiapkin, Denis Belomestny, Eric Moulines, Alexey Naumov, Sergey Samsonov, Yunhao Tang, Michal Valko, and Pierre Menard. From Dirichlet to rubin: Optimistic exploration in RL without bonuses. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21380-21431. PMLR, 17-23 Jul 2022b. URL https://proceedings.mlr.press/v162/tiapkin22a.html.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Chris J. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3-4):279-292, 1992. URL https://link.springer.com/content/pdf/10.1007/BF00992698.pdf.
Tzu-Tsung Wong. Generalized dirichlet distribution in bayesian analysis. Applied Mathematics and Computation, 97(2):165-181, 1998. ISSN 0096-3003. doi: https://doi.org/10.1016/S0096-3003(97)10140-0. URL https://www.sciencedirect.com/science/article/pii/S0096300397101400.
Zhihan Xiong, Ruoqi Shen, Qiwen Cui, Maryam Fazel, and Simon S Du. Near-optimal randomized exploration for tabular markov decision processes. Advances in Neural Information Processing Systems, 35:6358-6371, 2022.
Fan Ye and Enlu Zhou. Information relaxation and dual formulation of controlled markov diffusions. IEEE Transactions on Automatic Control, 60(10):2676-2691, 2015.
Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning, 2019. URL https://arxiv.org/pdf/1901.00210.pdf.
Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition. arXiv preprint arXiv:2004.10019, 2020. ISSN 23318422. URL https://arxiv.org/pdf/2004.10019.pdf.
A Notation 17
B Description of RandQL 18
B.1 RandQL algorithm 18
B.2 Sampled-RandQL algorithm 19
C Weight Distribution in RandQL 20
D Proofs for Tabular algorithm 22
D.1 Algorithm 22
D.2 Concentration 23
D.3 Optimism 25
D.4 Regret Bound 27
E Proofs for Metric algorithm 30
E.1 Assumptions 30
E.2 Algorithm 30
E.3 Concentration 32
E.4 Optimism 33
E.5 Regret Bounds 36
F Adaptive RandQL 40
F.1 Additional Notation 40
F.2 Algorithm 40
F.3 Regret Bound 42
G Deviation and Anti-Concentration Inequalities 49
G.1 Deviation inequality for  $\kappa_{\mathrm{inf}}$  49
G.2 Anti-concentration Inequality for Dirichlet Weighted Sums 49
G.3 Rosenthal-type inequality 50
H Technical Lemmas 52
I Experimental details 52
I.1 Tabular experiments 52
I.2 Non-tabular experiments 54
Table 1: Table of notation use throughout the paper for the tabular setting
Notation	Meaning
S	state space of size S
A	action space of size A
H	length of one episode
T	number of episodes
J	number of posterior samples
rh(s,a)	reward
ph(s'|s,a)	probability transition
Qhπ(s,a)	Q-function of a given policy π at step h
Vhπ(s)	V-function of a given policy π at step h
Qh(s,a)	optimal Q-function at step h
Vh(s)	optimal V-function at step h
RTh	regret
n0and n0(k)	number of pseudo-transitions
s0	optimistic pseudo-state
r0	pseudo-reward
κ	posterior inflation parameter
st	state that was visited at h step during t episode
ah	action that was picked at h step during t episode
Bt	a ball that contains a pair (sh,t,ah)
nh(t,s,a)	number of visits of state-action at the beginning of episode t
nh(t,s,a) = ∑k=1t-11{ (sh,k,ah) = (s,a)}
nh(B)	number of visits of a ball B at the beginning of episode t
ek	length of k-th stage ek = [(1+1/H)kH] for k ≥ 0 and e-1=0
kh(t,s,a)	index of stage previous to time t at step h and state-action pair (s,a):
kh(t,s,a) = max{k : nh(t,s,a) ≥ ∑i=0k ei}
tn(h)(s,a)	number of visits of state-action during the current stage:
tn(h)(s,a) = nh(t,s,a) - ∑i=0k(t,s,a)e i
tn(h)(B)	number of visits of a ball B during the current stage:
Vt(s)	upper approximation of the optimal V-value
Qt(s,a)	upper approximation of the optimal Q-value
Qt(B)	upper approximation of the optimal Q-value for all (s,a) ∈ B
Qt,j(s,a)	temporary estimate of the optimal Q-value
Qt,j(B)	temporary estimate of the optimal Q-value for all (s,a) ∈ B
wn,j	random learning rates
ρS,ρA,ρ	metrics on S, A and S × A correspondingly
Nε	minimal ε-cover if S × A of size Nε
dc	covering dimension of space S × A: ∀ε > 0 : Nε ≤ CNε−dc
dmax	diameter of S × A
Fh(s,a,ξh)	reparametrization function sh+1 ~ ph(s,a) ⇌ sh+1 = Fh(s,a,ξh)
Lr,LF	Lipschitz constants of rewards and reparametrization function
LV	Lipschitz constants of Qh* and Vh*
Let  $(\mathsf{X},\mathcal{X})$  be a measurable space and  $\mathcal{P}(\mathsf{X})$  be the set of all probability measures on this space. For  $p\in \mathcal{P}(\mathsf{X})$  we denote by  $\mathbb{E}_p$  the expectation w.r.t.  $p$ . For random variable  $\xi :\mathsf{X}\to \mathbb{R}$  notation  $\xi \sim p$  means  $\operatorname {Law}(\xi) = p$ . We also write  $\mathbb{E}_{\xi \sim p}$  instead of  $\mathbb{E}_p$ . For independent (resp. i.i.d.) random variables  $\xi_{\ell}\stackrel {\mathrm{ind}}{\sim}p_{\ell}$  (resp.  $\xi_{\ell}\stackrel {\mathrm{i.i.d}}{\sim}p$ ),  $\ell = 1,\ldots ,d$ , we will write  $\mathbb{E}_{\xi_{\ell}\stackrel {\mathrm{ind}}{\sim}p_{\ell}}$  (resp.  $\mathbb{E}_{\xi_{\ell}\stackrel {\mathrm{i.i.d}}{\sim}p}$ ), to denote expectation w.r.t. product measure on  $(\mathsf{X}^d,\mathcal{X}^{\otimes d})$ . For any  $x\in \mathsf{X}$  we denote  $\delta_x$  a Dirac measure supported at point  $x$ .
For any  $p,q\in \mathcal{P}(\mathsf{X})$  the Kullback-Leibler divergence  $\mathrm{KL}(p,q)$  is given by
$$
\operatorname {K L} (p, q) \triangleq \left\{ \begin{array}{l l} \mathbb {E} _ {p} \left[ \log \frac {\mathrm {d} p}{\mathrm {d} q} \right], & p \ll q, \\ + \infty , & \text {o t h e r w i s e}. \end{array} \right.
$$
For any  $p \in \mathcal{P}(\mathsf{X})$  and  $f: \mathsf{X} \to \mathbb{R}$ ,  $pf = \mathbb{E}_p[f]$ . In particular, for any  $p \in \Delta_d$  and  $f: \{0, \dots, d\} \to \mathbb{R}$ ,  $pf = \sum_{\ell=0}^{d} f(\ell) p(\ell)$ . Define  $\operatorname{Var}_p(f) = \mathbb{E}_{s' \sim p}\big[(f(s') - pf)^2\big] = p[f^2] - (pf)^2$ . For any  $(s, a) \in S$ , transition kernel  $p(s, a) \in \mathcal{P}(S)$  and  $f: S \to \mathbb{R}$  define  $pf(s, a) = \mathbb{E}_{p(s, a)}[f]$  and  $\operatorname{Var}_p[f](s, a) = \operatorname{Var}_{p(s, a)}[f]$ .
Let  $(\mathsf{X},\rho)$  be a metric space, then the 1-Wasserstein distance between  $p,q\in \mathcal{P}(\mathsf{X})$  is defined as  $\mathcal{W}_1(p,q) = \sup_{f\text{is 1 - Lipschitz}}\mathbb{E}_p[f] - \mathbb{E}_q[f]$ .
We write  $f(S, A, H, T) = \mathcal{O}(g(S, A, H, T, \delta))$  if there exist  $S_0, A_0, H_0, T_0, \delta_0$  and constant  $C_{f,g}$  such that for any  $S \geq S_0, A \geq A_0, H \geq H_0, T \geq T_0, \delta < \delta_0, f(S, A, H, T, \delta) \leq C_{f,g} \cdot g(S, A, H, T, \delta)$ . We write  $f(S, A, H, T, \delta) = \widetilde{\mathcal{O}}(g(S, A, H, T, \delta))$  if  $C_{f,g}$  in the previous definition is poly-logarithmic in  $S, A, H, T, 1/\delta$ .
For  $\alpha, \beta > 0$ , we define  $\operatorname{Beta}(\alpha, \beta)$  as a beta distribution with parameters  $\alpha, \beta$ . For set  $X$  such that  $|\mathsf{X}| < \infty$  define  $\mathrm{Unif}(X)$  as a uniform distribution over this set. In particular,  $\mathrm{Unif}[N]$  is a uniform distribution over a set  $[N]$ .
For a measure  $p \in \mathcal{P}([0, b])$  supported on a segment  $[0, b]$  (equipped with a Borel  $\sigma$ -algebra) and a number  $\mu \in [0, b]$  we define
$$
\mathcal {K} _ {\inf } (p, \mu) \triangleq \inf  \left\{\mathrm {K L} (p, q): q \in \mathcal {P} ([ 0, b ]), p \ll q, \mathbb {E} _ {X \sim q} [ X ] \geq \mu \right\}.
$$
As the Kullback-Leibler divergence this quantity admits a variational formula by Lemma 18 of Garivier et al. [2018] up to rescaling for any  $u \in (0, b)$
$$
\mathcal {K} _ {\inf } (p, \mu) = \max  _ {\lambda \in [ 0, 1 / (b - \mu) ]} \mathbb {E} _ {X \sim p} [ \log (1 - \lambda (X - \mu)) ].
$$
B Description of RandQL
In this appendix we describe Redis and Sampled-RandQL algorithms.
B.1 RandQL algorithm
We recall that  $n_h^t (s,a) = \sum_{i = 1}^{t - 1}\mathbb{1}\{(s_h^i,a_h^i) = (s,a)\}$  is the number of visits of state-action pair  $(s,a)$  at step  $h$  before episode  $t$ .
We start by initializing the ensemble of Q-values, the policy Q-values, and values to an optimistic value  $\widetilde{Q}_h^{t,j}(s,a) = \overline{Q}_h^1 (s,a) = \overline{V}_h^1 (s,a) = r_h(s,a) + r_0(H - h)$  for all  $(j,h,s,a)\in [J]\times [H]\times \mathcal{S}\times \mathcal{A}$  and  $r_0 > 0$  some pseudo-rewards.
At episode  $t$  we update the ensemble of Q-values as follows, denoting by  $n = n_h^t (s,a)$  the count,  $w_{j,n} \sim \mathrm{Beta}(H,n)$  the independent learning rates,
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (s, a) = \left\{ \begin{array}{l l} (1 - w _ {j, n}) \widetilde {Q} _ {h} ^ {t, j} (s, a) + w _ {j, n} \mathring {Q} _ {h} ^ {t, j} (s, a), & (s, a) = (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ \widetilde {Q} _ {h} ^ {t, j} (s, a) & \text {o t h e r w i s e ,} \end{array} \right.
$$
where we defined the target  $\hat{Q}_h^{t,j}(s,a)$  as a mixture between the usual target and some prior target with mixture coefficient  $\hat{w}_{n,j}\sim \mathrm{Beta}(n,n_0)$  and  $n_0$  the number of prior samples,
$$
\mathring {Q} _ {h} ^ {t, j} (s, a) = \mathring {w} _ {j, n} [ r _ {h} (s, a) + \overline {{V}} _ {h + 1} ^ {t} (s _ {h + 1} ^ {t}) ] + (1 - \mathring {w} _ {j, n}) [ r _ {h} (s, a) + r _ {0} (H - h - 1) ].
$$
It is important to note that in our approach, we need to re-inject prior targets to avoid forgetting their effects too quickly due to the aggressive learning rate. Indeed, the exponential decay of the prior effect can hurt exploration. We observe that the ensemble Q-value only averages uniformly over the last  $1 / H$  fraction of the targets, as the expected value of the learning rate is  $\mathbb{E}[w_{j,n}] = H / (n + H)$ . Since  $\mathbb{E}[1 - \mathring{w}_{j,n}] = n_0(n + n_0)$  the weight put on the prior sample in expectation, when we unfold
the definition of  $\widetilde{Q}_h^{t + 1,j}$ , is of order  $H / n\cdot n / H\cdot n_0 / (n + n_0) = n_0 / (n + n_0)$ , which is consistent with the usual prior forgetting in Bayesian learning. In Staged-RandQL, we avoid forgetting the prior too quickly by resetting the temporary Q-value to a prior value at the beginning of each stage.
The policy Q-values are obtained by taking the maximum among the ensemble of Q-values
$$
\overline {{Q}} _ {h} ^ {t + 1} (s, a) = \max _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {t + 1, j} (s, a).
$$
The policy is then greedy with respect to the policy Q-values  $\pi_h^{t + 1}(s)\in \arg \max_{a\in \mathcal{A}}\overline{Q}_h^{t + 1}(s,a)$  and the value is  $\overline{V}_h^{t + 1}(s) = \max_{a\in \mathcal{A}}\overline{Q}_h^{t + 1}(s,a)$ . The complete RandQL procedure is detailed in Algorithm 2.
Algorithm 2 RandQL
1: Input:  $J$  ensemble size, number of prior transitions  $n_0$ , prior reward  $r_0$ .
2: Initialize:  $\overline{V}_h(s) = \overline{Q}_h(s,a) = \widetilde{Q}_h^j (s,a) = r(s,a) + r_0(H - h)$ , initialize counters  $n_h(s,a) = 0$  for  $h,s,a\in [H]\times \mathcal{S}\times \mathcal{A}$ .
3: for  $t \in [T]$  do
4: for  $h\in [H]$  do
5: Play  $a_{h}\in \arg \max_{a}\overline{Q}_{h}(s_{h},a)$
6: Observe reward and next state  $s_{h + 1} \sim p_h(s_h, a_h)$ .
7: Sample  $\hat{w}_j\sim \mathrm{Beta}(n,n_0)$  for  $n = n_h(s_h,a_h)$
8: Build targets for all  $j \in [J]$
$$
\overset {\circ} {Q} _ {h} ^ {j} = \overset {\circ} {w} _ {j} \left[ r _ {h} \left(s _ {h}, a _ {h}\right) + \overline {{V}} _ {h + 1} \left(s _ {h + 1}\right) \right] + \left(1 - \overset {\circ} {w} _ {j}\right) \left[ r _ {h} \left(s _ {h}, a _ {h}\right) + r _ {0} (H - h) \right].
$$
9: Sample learning rates  $w_{j} \sim \mathrm{Beta}(H, n)$ .
10: Update ensemble  $Q$ -functions for all  $j \in [J]$
$$
\widetilde {Q} _ {h} ^ {j} \left(s _ {h}, a _ {h}\right) := \left(1 - w _ {j}\right) \widetilde {Q} _ {h} ^ {j} \left(s _ {h}, a _ {h}\right) + w _ {j} \overset {\circ} {Q} _ {h} ^ {j}.
$$
11: Update policy  $Q$ -function  $\overline{Q}_h(s_h, a_h) \coloneqq \max_{j \in [J]} \widetilde{Q}_h^j(s_h, a_h)$ .
12: Update value function  $\overline{V}_h(s_h) \coloneqq \max_{a \in \mathcal{A}} \overline{Q}_h(s_h, a)$ .
13: end for
14: end for
B.2 Sampled-RandQL algorithm
To create an algorithm that is more similar to PSRL, it is possible to select a Q-value at random from the ensemble of Q-values, rather than using the maximum Q-value
$$
\overline {{Q}} _ {h} ^ {t} (s, a) = \widetilde {Q} _ {h} ^ {t, j _ {t}} (s, a) \qquad \text {w i t h} j _ {t} \sim \operatorname {U n i f} [ J ].
$$
In this case we also need to update each Q-value in the ensemble with its corresponding target, see Osband and Van Roy [2015],
$$
\mathring {Q} _ {h} ^ {t, j} (s, a) = \mathring {w} _ {j, n} [ r _ {h} (s, a) + \widetilde {V} _ {h + 1} ^ {t, j} (s _ {h + 1} ^ {t}) ] + (1 - \mathring {w} _ {j, n}) [ r _ {h} (s, a) + r _ {0} (H - h - 1) ]
$$
where  $\widetilde{V}_h^{t,j}(s) = \max_{a\in \mathcal{A}}\widetilde{Q}_h^{t,j}(s,a)$ . We name this new procedure Sampled-RandQL and detail it in Algorithm 3.
Algorithm 3 Sampled-RandQL
1: Input:  $J$  ensemble size, number of prior transitions  $n_0$ , prior reward  $r_0$ .  
2: Initialize:  $\overline{V}_h(s) = \overline{Q}_h(s,a) = \widetilde{Q}_h^j (s,a) = r(s,a) + r_0(H - h)$ , initialize counters  $n_h(s,a) = 0$  for  $h,s,a\in [H]\times S\times A$ .  
3: for  $t\in [T]$  do  
4: Sample ensemble index  $i\sim \mathrm{Unif}[J]$   
5: for  $h\in [H]$  do  
6: Play  $a_{h}\in \arg \max_{a}\overline{Q}_{h}(s_{h},a)$ .  
7: Observe reward and next state  $s_{h + 1}\sim p_h(s_h,a_h)$ .  
8: Sample  $\mathring{w}_j\sim \mathrm{Beta}(n,n_0)$  for  $n = n_h(s_h,a_h)$ .  
9: Build targets for all  $j\in [J]$ $\mathring{Q}_h^j = \mathring{w}_j[r_h(s_h,a_h) + \widetilde{V}_{h + 1}^j (s_{h + 1})] + (1 - \mathring{w}_j)[r_h(s_h,a_h) + r_0(H - h)]$ .  
10: Sample learning rates  $w_{j}\sim \mathrm{Beta}(H,n)$ .  
11: Update ensemble  $Q$  functions for all  $j\in [J]$ $\widetilde{Q}_h^j (s_h,a_h)\coloneqq (1 - w_j)\widetilde{Q}_h^j (s_h,a_h) + w_j\mathring{Q}_h^j$ .  
12: Update value function  $\overline{V}_h(s_h)\coloneqq \max_{a\in \mathcal{A}}\widetilde{Q}_h^j (s_h,a)$  for all  $j\in [J]$ .  
13: Update policy  $Q$  function  $\overline{Q}_h(s_h,a_h)\coloneqq \widetilde{Q}_h^i (s_h,a_h)$ .  
14: end for  
15: end for
C Weight Distribution in RandQL
In this section we study the joint distribution of weights over all targets in RandQL algorithm, described in details in Appendix B. To do it, we describe a very useful distribution, defined by Wong [1998].
Definition 5. We say that a random vector  $(X_{1},\ldots ,X_{n},X_{n + 1})$  has a generalized Dirichlet distribution  $\mathrm{GDir}(\alpha_1,\dots ,\alpha_n;\beta_1,\dots ,\beta_n)$  if  $X_{n + 1} = 1 - (X_{1} + \ldots +X_{n})$  and  $(X_{1},\ldots ,X_{n})$  it has the following density over the simplex  $\{x_{1},\ldots ,x_{n}:x_{1} + \ldots +x_{n}\leq 1\}$ ,
$$
p (x) = \prod_ {i = 1} ^ {n} \frac {1}{B (\alpha_ {i} , \beta_ {i})} x _ {i} ^ {\alpha_ {i} - 1} (1 - x _ {1} - \ldots - x _ {i}) ^ {\gamma_ {i}}
$$
for  $x_{1} + \ldots + x_{n} \leq 1$ ,  $x_{j} \geq 0$  for  $j = 1, \ldots, n$ , and  $\gamma_{j} = \beta_{j} - \alpha_{j+1} - \beta_{j+1}$  for  $j = 1, \ldots, n-1$  and  $\gamma_{n} = \beta_{n}-1$ . If we set  $x_{n+1} = 1 - (x_{1} + \ldots + x_{n})$  then we obtain a homogeneous formula
$$
p (x) = \prod_ {i = 1} ^ {n} \frac {1}{B \left(\alpha_ {i} , \beta_ {i}\right)} x _ {i} ^ {\alpha_ {i} - 1} \left(\sum_ {j = i + 1} ^ {n + 1} x _ {j}\right) ^ {\gamma_ {i}}
$$
Alternative characterization of generalized Dirichlet distribution could be given using independent beta-distributed random variables  $Z_{1},\ldots ,Z_{n}$  with  $Z_{i}\sim \mathrm{Beta}(\alpha_{i},\beta_{i})$  as follows
$$
X _ {1} = Z _ {1},
$$
$$
X _ {j} = Z _ {j} \left(1 - X _ {1} - \dots - X _ {j - 1}\right) = Z _ {j} \prod_ {i = 1} ^ {j - 1} \left(1 - Z _ {i}\right) \quad \text {f o r} j = 2, 3, \dots , n
$$
$$
X _ {n + 1} = 1 - X _ {1} - \ldots - X _ {n} = \prod_ {i = 1} ^ {n} (1 - Z _ {i})
$$
Therefore, for RandQL algorithm without prior re-injection we have the following formula
$$
\widetilde {Q} _ {h} ^ {t, j} (s, a) = \sum_ {i = 0} ^ {n _ {h} ^ {t} (s, a)} W _ {j, n} ^ {i} \left(r _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right) + \bar {V} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right)\right),
$$
for  $n = n_h^t (s,a)$  and weights are defined as follows
$$
W _ {j, n} ^ {0} = \prod_ {q = 0} ^ {n - 1} (1 - w _ {j, q}), \quad W _ {j, n} ^ {i} = w _ {j, i - 1} \cdot \prod_ {q = i} ^ {n - 1} (1 - w _ {j, q}), i \geq 1.
$$
And, moreover, we have that this vector of weights has the generalized Dirichlet distribution
$$
(W _ {n, j} ^ {n}, W _ {n, j} ^ {n - 1}, \ldots , W _ {n, j} ^ {1}, W _ {n, j} ^ {0}) \sim \mathrm {G D i r} (H, H, \ldots , H; n + n _ {0}, \ldots , n _ {0} + 1, n _ {0}).
$$
That is, weights generated by the RandQL procedure is an inverted generalized Dirichlet random vector, that induces additional similarities with a usual posterior sampling approaches. Notably, that for  $H = 1$  we recover exactly usual Dirichlet distribution, as in the setting of Staged-RandQL.
In the setting of the analysis, the main feature of this distribution is asymmetry in attitude to the order of components. In particular, the expectation of the prior weight  $W_{n,j}^{0}$  is  $\prod_{i=1}^{n}\left(1 - \frac{H}{i + H}\right) \sim n^{-H}$  that leads to too rapid forgetting of the prior information.
D Proofs for Tabular algorithm
In this section we describe in detail the tabular algorithms and the ways we will analyze them. We also provide some notations that will be used in the sequel.
Let  $n_h^t (s,a)$  be the number of visits of  $(s,a,h)$  (i.e., of the state-action pair  $(s,a)$  at step  $h$ ) at the beginning of episode  $t$ :  $n_h^t (s,a) = \sum_{i = 1}^{t - 1}\mathbb{1}\{(s_h^i,a_h^i) = (s,a)\}$ . In particular,  $n_h^{T + 1}(s,a)$  is the number of visits of  $(s,a,h)$  after all episodes.
Let  $e_k = \left\lfloor (1 + 1 / H)^k \cdot H \right\rfloor$  be the length of each stage for any  $k \geq 0$  and, by convention,  $e_{-1} = 0$ . We will say that at the beginning of episode  $t$  a triple  $(s, a, h)$  is in  $k$ -th stage if  $n_h^t(s, a) \in [\sum_{i=0}^{k-1} e_i, \sum_{i=0}^k e_i)$ .
Let  $\widetilde{n}_h^t (s,a)$  be the number of visits of state-action pair during the current stage at the beginning of episode  $t$ . Formally, it holds  $\widetilde{n}_h^t (s,a) = n_h^t (s,a) - \sum_{i = 0}^{k - 1}e_i$ , where  $k$  is the index of current stage.
Let  $\kappa > 0$  be the posterior inflation coefficient,  $n_0$  be the number of prior transitions, and  $J$  be the number of temporary  $Q$ -functions. Let  $\widetilde{Q}_h^{t,j}$  be the  $j$ -th temporary  $Q$ -function and  $\overline{Q}_h^t$  be the policy  $Q$ -function at the beginning of episode  $t$ . We initialize them as follows
$$
\overline {{Q}} _ {h} ^ {1} (s, a) = r _ {h} (s, a) + r _ {0} (H - h - 1), \quad \widetilde {Q} _ {h} ^ {1, j} (s, a) = r _ {h} (s, a) + r _ {0} (H - h - 1),
$$
We can treat this initialization as a setting prior over  $n_0$  pseudo-transitions to artificial state  $s_0$  with  $r_0 > 1$  reward for each interaction.
For each transition we perform the following update of temporary Q-functions
$$
\widetilde {Q} _ {h} ^ {t + 1 / 2, j} (s, a) = \left\{ \begin{array}{l l} \left(1 - w _ {j, \widetilde {n}} ^ {k}\right) \cdot \widetilde {Q} _ {h} ^ {t, j} (s, a) + w _ {j, \widetilde {n}} ^ {k} \left[ r _ {h} (s, a) + \overline {{V}} _ {h + 1} ^ {t} \left(s _ {h + 1} ^ {t}\right) \right], & (s, a) = \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) \\ \widetilde {Q} _ {h} ^ {t, j} (s, a) & \text {o t h e r w i s e ,} \end{array} \right. \tag {2}
$$
where  $\widetilde{n} = \widetilde{n}_h^t (s,a)$  is the number of visits of  $(s,a,h)$  during the current stage at the beginning of episode  $t$ ,  $k$  is the index of the current stage, and  $w_{j,\widetilde{n}}^k$  is a sequence of independent beta-distribution random variables  $w_{j,\widetilde{n}}^k\sim \mathrm{Beta}(1 / \kappa ,(\widetilde{n} +n_0) / \kappa)$ . Here we slightly abuse the notation by dropping the dependence of weights  $w_{j,\widetilde{n}}^k$  on the triple  $(h,s,a)$  in order to simplify the exposition. In the case that the explicit dependence is required, we will call these weights as  $w_{j,\widetilde{n}}^{k,h}(s,a)$ .
Next we define the stage update as follows
$$
\overline {{Q}} _ {h} ^ {t + 1} (s, a) = \left\{ \begin{array}{l l} \max  _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {t + 1 / 2, j} (s, a) & \widetilde {n} _ {h} ^ {t} (s, a) = \lfloor (1 + 1 / H) ^ {k} H \rfloor \\ \overline {{Q}} _ {h} ^ {t} (s, a) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (s, a) = \left\{ \begin{array}{l l} r _ {h} (s, a) + r _ {0} (H - h + 1) & \widetilde {n} _ {h} ^ {t} (s, a) = \lfloor (1 + 1 / H) ^ {k} H \rfloor \\ \widetilde {Q} _ {h} ^ {t + 1 / 2, j} (s, a) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\overline {{V}} _ {h} ^ {t + 1} (s) = \max _ {a \in \mathcal {A}} \overline {{Q}} _ {h} ^ {t + 1} (s, a)
$$
$$
\pi_{h}^{t + 1}(s)\in \operatorname *{arg  max}_{a\in \mathcal{A}}\overline{Q}_{h}^{t + 1}(s,a),
$$
where  $k$  is the current stage. In other words, we update  $\overline{Q}^{t + 1}$  with temporary values of  $\widetilde{Q}^{t + 1 / 2,j}$ , and then, if the change of stage is triggered, reinitialize  $\widetilde{Q}_h^{t + 1,j}(s,a)$  for all  $j$ . For episode  $t$  we will call  $k_h^t (s,a)$  the index of stage where  $\overline{Q}_h^t (s,a)$  was updated (and  $k_h^t (s,a) = -1$  if there was no update). For all  $t$  we define  $\tau_h^t (s,a)\leq t$  as an episode when the stage update happens. In other words, for any  $t$  the following holds
$$
\overline {{Q}} _ {h} ^ {t + 1} (s, a) = \max _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {\tau_ {h} ^ {t} (s, a) + 1 / 2, j} (s, a),
$$
where  $\tau_h^t (s,a) = 0$  and  $e_k = 0$  if there was no updates. To simplify the notation we will omit dependence on  $(s,a,h)$  where it is deducible from the context.
To simplify the notation, we can extend the state space  $S$  by an additional state  $s_0$  that will be purely technical and used in the proofs. This state has the prescribed value function  $V_h^\star(s_0) = r_0(H - h)$  and could be treated as a absorbing pseudo-state with reward  $r_0$ .
We notice that in this case we use  $e_k$  samples to compute  $\widetilde{Q}^{\tau_h^t (s,a) + 1 / 2,j}$  for  $k = k_h^t (s,a)$ . For this  $k$  we define  $\ell_{k,h}^{i}(s,a)$  as a time of  $i$ -th visit of state-action pair  $(s,a)$  during  $k$ -th stage. Then we have the following decomposition
$$
\widetilde {Q} _ {h} ^ {\tau^ {t} + 1 / 2, j} (s, a) = r _ {h} (s, a) + \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} \bar {V} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right), \tag {3}
$$
where we drop dependence on  $k$  and  $(s,a,h)$  in  $\ell^i$  to simplify notations, and use the convention  $s_{h + 1}^{\ell_{k,h}^0 (s,a)} = s_0$ , and the following aggregated weights
$$
W _ {j, n, k} ^ {0} = \prod_ {q = 0} ^ {n - 1} (1 - w _ {j, q} ^ {k}), \quad W _ {j, n, k} ^ {i} = w _ {j, i - 1} ^ {k} \cdot \prod_ {q = i} ^ {n - 1} (1 - w _ {j, q} ^ {k}), i \geq 1.
$$
We will omit the dependent on the stage index  $k$  when it is not needed for the statement. However, we notice that these vectors, for different stage  $k$ , will be independent.
By the properties of generalized Dirichlet distribution it is possible to show the following result
Lemma 3. For any fixed  $n > 0$ , the random vector  $(W_{j,n}^{0}, W_{j,n}^{1}, \ldots, W_{j,n}^{n})$  has a Dirichlet distribution  $\mathrm{Dir}(n_0 / \kappa, 1 / \kappa, \ldots, 1 / \kappa)$ .