SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Using the Dirichlet random variate generation from marginal beta distributions, it is sufficient to prove that for all  $i \in \{0, \dots, n\}$ ,  $W_{j,n,k}^{n - i} = (1 - W_{j,n,k}^n - \dots - W_{j,n,k}^{n - i + 1}) w_{j,n - i - 1}^k$ , with the convention  $w_{j, -1}^k = 1$ . This is trivial for  $i = 0$ , as  $W_{j,n,k}^n = w_{j,n - 1}^k$ . Now, if this is true for some  $i$  then, for  $i + 1 \in \{0, \dots, n\}$ , we have
$$
\begin{array}{l} W _ {j, n, k} ^ {n - i - 1} = w _ {j, n - i - 2} ^ {k} \prod_ {q = n - i - 1} ^ {n - 1} \left(1 - w _ {j, q} ^ {k}\right) \\ = w _ {j, n - i - 2} ^ {k} \left(1 - w _ {j, n - i - 1} ^ {k}\right) \left(1 - W _ {j, n, k} ^ {n} - \dots - W _ {j, n, k} ^ {n - i + 1}\right) \\ = w_{j,n - i - 2}^{k}(1 - W_{j,n,k}^{n} - \dots -W_{j,n,k}^{n - i + 1} - \underbrace{w_{j,n - i - 1}^{k}(1 - W_{j,n,k}^{n} - \dots - W_{j,n,k}^{n - i + 1})}_{= W_{j,n,k}^{n - i}}), \\ \end{array}
$$
Notably, the expression (3) shows a significant similarity between our method and OPSRL. It is the reason why we can call this method a model-free posterior sampling, where posterior sampling is performed over the model in a lazy and model-free fashion.
D.2 Concentration
Let  $\beta^{\star}\colon (0,1)\times \mathbb{N}\to \mathbb{R}_{+}$  and  $\beta^{B},\beta^{\mathrm{conc}},\beta \colon (0,1)\to \mathbb{R}_{+}$  be some function defined later on in Lemma 4. We define the following favorable events
$$
\mathcal {E} ^ {\star} (\delta) \triangleq \left\{\forall t \in \mathbb {N}, \forall h \in [ H ], \forall (s, a) \in \mathcal {S} \times \mathcal {A}, k = k _ {h} ^ {t} (s, a): \right.
$$
$$
\left. \mathcal {K} _ {\inf } \left(\frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \delta_ {V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell^ {i}}\right)}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq \frac {\beta^ {\star} (\delta , e _ {k})}{e _ {k}} \right\},
$$
$$
\begin{array}{l} \mathcal {E} ^ {B} (\delta) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \forall j \in [ J ], k = k _ {h} ^ {t} (s, a): \right. \\ \left| \sum_ {i = 0} ^ {e _ {k}} \left(W _ {j, e _ {k}, k} ^ {i} - \mathbb {E} [ W _ {j, e _ {k}, k} ^ {i} ]\right) \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) \right| \leq 6 0 \mathrm {e} ^ {2} \sqrt {\frac {r _ {0} ^ {2} H ^ {2} \kappa \beta^ {B} (\delta)}{e _ {k} + n _ {0} \kappa}} \\ \left. + 1 2 0 0 \mathrm {e} \frac {r _ {0} H \kappa \log \left(e _ {k} + n _ {0} \kappa\right) \left(\beta^ {B} (\delta)\right) ^ {2}}{e _ {k} + n _ {0} \kappa} \right\}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} ^ {\text {c o n c}} (\delta) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall (s, a) \in \mathcal {S} \times \mathcal {A}, k = k _ {h} ^ {t} (s, a): \right. \\ \left| \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {k, h} ^ {i} (s, a)}\right) - p _ {h} V _ {h + 1} ^ {\star} (s, a) \right| \leq \sqrt {\frac {2 r _ {0} ^ {2} H ^ {2} \beta^ {\text {c o n c}} (\delta)}{e _ {k}}} \Bigg \} \\ \mathcal {E} (\delta) \triangleq \left\{\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \left| p _ {h} \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) - \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h + 1} ^ {t}\right) \right| \right. \\ \left. \leq 2 \operatorname {e r} _ {0} H \sqrt {2 H T \beta (\delta)}. \right\}. \\ \end{array}
$$
We also introduce the intersection of these events,  $\mathcal{G}(\delta) \triangleq \mathcal{E}^{\star}(\delta) \cap \mathcal{E}^{B}(\delta) \cap \mathcal{E}^{\mathrm{conc}}(\delta) \cap \mathcal{E}(\delta)$ . We prove that for the right choice of the functions  $\beta^{\star}, \beta^{\mathrm{KL}}, \beta^{\mathrm{conc}}, \beta, \beta^{\mathrm{Var}}$  the above events hold with high probability.
Lemma 4. For any  $\delta \in (0,1)$  and for the following choices of functions  $\beta$ ,
$$
\beta^ {\star} (\delta , n) \triangleq \log (8 S A H / \delta) + 3 \log \left(\mathrm {e} \pi (2 n + 1)\right),
$$
$$
\beta^ {B} (\delta) \triangleq \log (8 S A H / \delta) + \log (T J),
$$
$$
\beta^ {\text {c o n c}} (\delta) \triangleq \log (8 S A H / \delta) + \log (2 T),
$$
$$
\beta (\delta) \triangleq \log (1 6 / \delta),
$$
$$
\mathbb {P} \left[ \mathcal {E} ^ {\star} (\delta) \right] \geq 1 - \delta / 8, \quad \mathbb {P} \left[ \mathcal {E} ^ {B} (\delta) \right] \geq 1 - \delta / 8,
$$
$$
\mathbb {P} [ \mathcal {E} ^ {\mathrm {c o n c}} (\delta) ] \geq 1 - \delta / 8, \quad \mathbb {P} [ \mathcal {E} (\delta) ] \geq 1 - \delta / 8.
$$
In particular,  $\mathbb{P}[\mathcal{G}(\delta)]\geq 1 - \delta /2$
Proof. From the fact that  $s_{h+1}^{\ell^i}$  are i.i.d. generated from  $p_h(s, a)$ , Theorem 4, and union bound  $S \times \mathcal{A} \times [H]$  it holds  $\mathbb{P}[\mathcal{E}^\star(\delta)] \geq 1 - \delta/8$ .
Next we fix all  $t, h, s, a, j$ , and denote  $n = e_{k_h^t (s,a)}$ . First, we define a filtration of  $\sigma$ -algebras  $\mathcal{F}_{\tau}$  that is sigma-algebra generated by all random variables appeared until the update (2) in the episode  $t$  and step  $h$ , before newly generated random weights but after receiving new state  $s_{h + 1}^{t}$ . Formally, we can define it as follows
$$
\begin{array}{l} \mathcal {F} _ {t, h} = \sigma \left( \right.\left\{\left(s _ {h ^ {\prime}} ^ {\tau}, a _ {h ^ {\prime}} ^ {\tau}, w _ {j, \overline {{n}} _ {h ^ {\prime}}} ^ {k _ {h ^ {\prime}} + 1, h ^ {\prime}} \left(s _ {h ^ {\prime}} ^ {\tau}, a _ {h ^ {\prime}} ^ {\tau}\right)\right), \forall \tau <   t, \left(h ^ {\prime}, j\right) \in [ H ] \times [ J ] \right\} \\ \cup \left\{\left(s _ {h ^ {\prime}} ^ {t}, a _ {h ^ {\prime}} ^ {t}, s _ {h ^ {\prime} + 1} ^ {t}\right), \forall h ^ {\prime} \leq h \right\} \cup \left\{w _ {j, \widetilde {n} _ {h ^ {\prime}} ^ {t}} ^ {k _ {h ^ {\prime}} ^ {t} + 1, h ^ {\prime}} \left(s _ {h ^ {\prime}} ^ {t}, a _ {h ^ {\prime}} ^ {t}\right), \forall h ^ {\prime} <   h, j \in [ J ] \right\}\left. \right), \\ \end{array}
$$
where we drop dependence on state-action pairs everywhere where it is deducible from the context. Consider a sequence  $\ell^1 < \ldots < \ell^n$  be an excursion of the state-action pair  $(s, a)$  at the step  $h$ . Each  $\ell^i$  is a stopping time w.r.t  $\mathcal{F}_{t,h}$ , so we can consider a stopped filtration (with a shift by 1 in indices)  $\widetilde{\mathcal{F}}_{i-1} = \mathcal{F}_{\ell^i, h}$ . In other words, this filtration at time-stamp  $i-1$  contains all the information that is available just before generation of random weights for  $i$ -th update of temporary Q-functions inside the last stage. We notice that under this definition we have
$$
\begin{array}{l} \mathbb {E} \big [ \overline {{V}} _ {h + 1} ^ {\ell_ {i}} \big (s _ {h + 1} ^ {\ell_ {i}} \big) | \widetilde {\mathcal {F}} _ {i - 1} \big ] = \overline {{V}} _ {h + 1} ^ {\ell_ {i}} \big (s _ {h + 1} ^ {\ell_ {i}} \big), \\ \mathbb {E} [ W _ {j, n, k} ^ {i} | \widetilde {\mathcal {F}} _ {i - 1} ] = \mathbb {E} \bigg [ w _ {j, i - 1} ^ {k} \prod_ {\ell = i} ^ {n - 1} (1 - w _ {j, \ell} ^ {k}) | \mathcal {F} _ {i - 1} \bigg ] = \mathbb {E} [ W _ {j, n, k} ^ {i} ], \\ \end{array}
$$
Next, we notice that the joint vector of weights follows the Dirichlet distribution, applying aggregation property and extending the filtration backward by adding fake transitions we can extend sum to  $n + n_0$  summands defining  $s_{h+1}^{\ell^{-i}} = s_0$
$$
\sum_ {i = 0} ^ {n} \big (W _ {j, n, k} ^ {i} - \mathbb {E} [ W _ {j, n, k} ^ {i} ] \big) \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) = \sum_ {q = - n _ {0} + 1} ^ {n} \Big (\widetilde {W _ {q}} - \mathbb {E} [ \widetilde {W _ {q}} ] \Big) \overline {{V}} _ {h + 1} ^ {\ell^ {q}} (s _ {h + 1} ^ {\ell^ {q}}).
$$
Finally, we notice that marginals of Dirichlet random vector follow Beta distribution, therefore by Proposition 7 and union bound we conclude  $\mathbb{P}[\mathcal{E}^B (\delta)]\geq 1 - \delta /8$
To show that  $\mathbb{P}(\mathcal{E}^{\mathrm{conc}}(\delta)) > 1 - \delta /8$ , it is enough to apply Hoeffding inequality for a fixed number of samples  $e_k$  used in empirical mean, and then use union bound of all possible values of  $(s,a,h)\in S\times \mathcal{A}\times [H]$  and  $e_k\in [T]$ .
Next, define the following sequence
$$
Z _ {t, h} \triangleq (1 + 1 / H) ^ {H - h} \Big ([ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi^ {t}} ] (s _ {h + 1} ^ {t}) - p _ {h} [ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi^ {t}} ] (s _ {h} ^ {t}, a _ {h} ^ {t}) \Big), t \in [ T ], h \in [ H ],
$$
It is easy to see that these sequences form a martingale-difference w.r.t filtration  $\mathcal{F}_{t,h} = \sigma\big\{\{(s_{h'}^\ell, a_{h'}^\ell, \pi^\ell), \ell < t, h' \in [H]\} \cup \{(s_{h'}^t, a_{h'}^t, \pi^t), h' \leq h\}\big\}$ . Moreover,  $|Z_{t,h}| \leq 2\mathrm{er}_0H$  for all  $t \in [T]$  and  $h \in [H]$ . Hence, the Azuma-Hoeffding inequality implies
$$
\mathbb {P} \left(\left| \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} Z _ {t, h} \right| > 2 \mathrm {e r} _ {0} H \sqrt {2 t H \cdot \beta (\delta)}\right) \leq 2 \exp (- \beta (\delta)) = \delta / 8,
$$
therefore  $\mathbb{P}[\mathcal{E}(\delta)]\geq 1 - \delta /8$
D.3 Optimism
In this section we prove that our estimate of  $Q$ -function  $\overline{Q}_h^t (s,a)$  is optimistic, that is the event
$$
\mathcal {E} _ {\text {o p t}} \triangleq \left\{\forall t \in [ T ], h \in [ H ], (s, a) \in \mathcal {S} \times \mathcal {A}: \overline {{Q}} _ {h} ^ {t} (s, a) \geq Q _ {h} ^ {\star} (s, a) \right\}. \tag {4}
$$
holds with high probability on the event  $\mathcal{E}^{\star}(\delta)$
Define constants
$$
c _ {0} \triangleq \frac {8}{\pi} \left(\frac {4}{\sqrt {\log (1 7 / 1 6)}} + 8 + \frac {4 9 \cdot 4 \sqrt {6}}{9}\right) ^ {2} + 1. \tag {5}
$$
$$
c _ {J} \triangleq \frac {1}{\log \left(\frac {2}{1 + \Phi (1)}\right)}, \tag {6}
$$
where  $\Phi (\cdot)$  is a CDF of a normal distribution.
Proposition 1. Assume that  $J = \lceil c_J \cdot \log(2SAHT / \delta) \rceil$ ,  $\kappa = 2\beta^{\star}(\delta, T)$ ,  $r_0 = 2$ , and  $n_0 = \lceil (c_0 + 1 + \log_{17/16}(T)) \cdot \kappa \rceil$ . Then conditionally on  $\mathcal{E}^{\star}(\delta)$  the event
$$
\begin{array}{l} \mathcal {E} _ {\text {a n t i c o n c}} \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall (s, a) \in \mathcal {S} \times \mathcal {A}: \right. \\ \left. \max  _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {t, h} ^ {i} (s, a)}\right) \right\} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a), k = k _ {h} ^ {t} (s, a) \right\} \\ \end{array}
$$
holds with probability at least  $1 - \delta /2$
Proof. Let us fix  $t \in [T]$ ,  $h \in [H]$ ,  $(s, a) \in S \times \mathcal{A}$ , and  $j \in [J]$ . By Lemma 3, we have that the vector  $(W_{j, e_k, k}^i)_{i=0,\dots,e_k}$  has Dirichlet distribution. Note that  $V_{h+1}^\star(s_{h+1}^{\ell^0}) = r_0(H - h - 1)$  is an upper bound on  $V$ -function and the weight of the first atom is  $\alpha_0 \triangleq n_0 / \kappa \geq c_0 + \log_{17/16}(T)$  for  $c_0$  defined in (5). Define a measure  $\bar{\nu}_{e_k} = \frac{n_0 - 1}{e_k + n_0 - 1} \delta_{V_{h+1}^\star(s_0)} + \sum_{i=1}^{e_k} \frac{1}{e_k + n_0 - 1} \delta_{V_{h+1}^\star(s_{h+1}^{\ell^i})}$ . Since  $p_h V_{h+1}^\star(s, a) \leq H - h - 1$ , we can apply Lemma 10 with a fixed  $\varepsilon = 1/2$  conditioned on independent samples  $\{s_{h+1}^{\ell_i}\}_{i=1}^{e_k}$  from  $p_h(s, a)$
$$
\begin{array}{l} \mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {t, h} ^ {i} (s, a)}\right) \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \left\{s _ {h + 1} ^ {\ell_ {i}} \right\} _ {i = 1} ^ {e _ {k}} \right] \\ \geq \frac {1}{2} \left(1 - \Phi \left(\sqrt {\frac {2 \left(e _ {k} + n _ {0} - \kappa\right) \mathcal {K} _ {\inf } \left(\bar {\nu} _ {e _ {k}} , p _ {h} V _ {h + 1} ^ {\star} (s , a)\right)}{\kappa}}\right)\right), \tag {7} \\ \end{array}
$$
where  $\Phi$  is a CDF of a normal distribution. Combining Lemma 12 and the event  $\mathcal{E}^{\star}(\delta)$
$$
\left(e _ {k} + n _ {0} - \kappa\right) \mathcal {K} _ {\inf } \left(\bar {\nu} _ {e _ {k}}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq e _ {k} \mathcal {K} _ {\inf } \left(\widehat {\nu} _ {e _ {k}}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq \beta^ {\star} (\delta , T),
$$
where  $\widehat{\nu}_{e_k} = \frac{1}{e_k}\sum_{i = 1}^{e_k}\delta_{V_{h^+}^1 (s_{h + 1}^\ell i)}$ , and, as a corollary
$$
\mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {t, h} ^ {i} (s, a)}\right) \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \mathcal {E} ^ {\star} (\delta), \{s _ {h + 1} ^ {\ell^ {i}} \} _ {i = 1} ^ {e _ {k}} \right] \geq \frac {1}{2} \left(1 - \Phi \left(\sqrt {\frac {2 \beta^ {\star} (\delta , T)}{\kappa}}\right)\right).
$$
By taking  $\kappa = 2\beta^{\star}(\delta, T)$  we have a constant probability of being optimistic
$$
\mathbb {P} \left(\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {t, h} ^ {i} (s, a)}\right) \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \mathcal {E} ^ {\star} (\delta)\right) \geq \frac {1 - \Phi (1)}{2} \triangleq \gamma .
$$
Next, using a choice  $J = \lceil \log (2SAHT / \delta) / \log (1 / (1 - \gamma)) \rceil = \lceil c_J \cdot \log (2SAHT / \delta) \rceil$
$$
\mathbb {P} \left[ \max  _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} (s _ {h + 1} ^ {\ell_ {t, h} ^ {i} (s, a)}) \right\} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \mathcal {E} ^ {\star} (\delta) \right] \geq 1 - (1 - \gamma) ^ {J} \geq 1 - \frac {\delta}{2 S A H T}.
$$
By a union bound we conclude the statement.
Next we provide a connection between  $\mathcal{E}^{\mathrm{anticonc}}$  and  $\mathcal{E}^{\mathrm{opt}}$ .
Proposition 2. It holds that  $\mathcal{E}^{\mathrm{opt}}\subseteq \mathcal{E}^{\mathrm{anticonc}}$
Proof. We proceed by a backward induction over  $h$ . Base of induction  $h = H + 1$  is trivial. Next by Bellman equations for  $\overline{Q}_h^t$  and  $Q_h^\star$
$$
[ \overline {{Q}} _ {h} ^ {t} - Q _ {h} ^ {\star} ] (s, a) = \max _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {n} W _ {j, n} ^ {i} \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) \right\} - p _ {h} V _ {h + 1} ^ {\star} (s, a),
$$
where  $n = e_{k_h^i (s,a)}$  and we drop dependence on  $k,t,h,s,a$  in  $\ell^i$ . By induction hypothesis we have  $\overline{V}_{h + 1}^{\ell^i}(s')\geq \overline{Q}_{h + 1}^{\ell^i}(s',\pi^\star (s'))\geq Q_{h + 1}^\star (s',\pi^\star (s')) = V_{h + 1}^\star (s')$  for any  $i$ , thus
$$
\left[ \overline {{Q}} _ {h} ^ {t} - Q _ {h} ^ {\star} \right] (s, a) \geq \max  _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {n} W _ {j, n} ^ {i} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell^ {i}}\right) \right\} - p _ {h} V _ {h + 1} ^ {\star} (s, a).
$$
By the definition of event  $\mathcal{E}^{\mathrm{anticonc}}(\delta)$  we conclude the statement.
Proposition 3 (Optimism). Assume that  $J = \lceil c_J \cdot \log(2SAHT / \delta) \rceil$ ,  $\kappa = 2\beta^{\star}(\delta, T)$ ,  $r_0 = 2$ , and  $n_0 = \lceil (c_0 + 1 + \log_{17/16}(T)) \cdot \kappa \rceil$ , where  $c_0$  is defined in (5) and  $c_J$  is defined in (6). Then  $\mathbb{P}(\mathcal{E}^{\mathrm{opt}} \mid \mathcal{E}^{\star}(\delta)) \geq 1 - \delta / 2$ .
D.4 Regret Bound
Let us define the main event  $\mathcal{G}'(\delta) = \mathcal{G}(\delta) \cap \mathcal{E}^{\mathrm{opt}}$ . On this event we have the following corollary that connects RandQL with OptQL with Hoeffding bonuses.
Define the following quantity
$$
\beta^ {\max } (\delta) = \max  \left\{\kappa , n _ {0} / \kappa , \beta^ {B} (\delta), \beta^ {\operatorname {c o n c}} (\delta), \beta (\delta), \log (T + n _ {0}) \right\} = \mathcal {O} \left(\log (S A T H / \delta)\right).
$$
Corollary 1. Assume conditions of Proposition 3 hold. Let  $t \in [T]$ ,  $h \in [H]$ ,  $(s, a) \in S \times \mathcal{A}$ . Define  $k = k_h^t(s, a)$  and let  $\ell^1 < \ldots < \ell^{e_k}$  be a excursion of  $(s, a, h)$  until the previous stage. Then on the event  $\mathcal{G}'(\delta)$  the following bound holds for  $k \geq 0$
$$
0 \leq \overline {{Q}} _ {h} ^ {t} (s, a) - Q _ {h} ^ {\star} (s, a) \leq \frac {1}{n} \sum_ {i = 1} ^ {n} [ \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) - V _ {h + 1} ^ {\star} (s _ {h + 1} ^ {\ell^ {i}}) ] + \mathcal {B} _ {h} ^ {t} (k),
$$
$$
\mathcal {B} _ {h} ^ {t} (k) = 6 1 \mathrm {e} ^ {2} \frac {r _ {0} H (\beta^ {\max} (\delta))}{\sqrt {e _ {k}}} + 1 2 0 1 \mathrm {e} \frac {r _ {0} H (\beta^ {\max} (\delta)) ^ {4}}{e _ {k}}.
$$
Proof. The lower bound follows from the definition of the event  $\mathcal{E}^{\mathrm{opt}}$ . For the upper bound we first apply the decomposition for  $\overline{Q}_h^t (s,a)$  and the definition of event  $\mathcal{E}^B (\delta)$  from Lemma 4
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} (s, a) = r _ {h} (s, a) + \max _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}} ^ {i} \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) \right\} \\ \leq r _ {h} (s, a) + \frac {1}{e _ {k} + n _ {0}} \sum_ {i = 1} ^ {e _ {k}} \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) + \frac {n _ {0} \kappa \cdot r _ {0} H}{e _ {k} + n _ {0}} + 6 0 \mathrm {e} ^ {2} \sqrt {\frac {r _ {0} ^ {2} H ^ {2} \kappa \beta^ {B} (\delta)}{e _ {k} + n _ {0}}} \\ + 1 2 0 0 \mathrm {e} ^ {\frac {r _ {0} H \kappa \log (e _ {k} + n _ {0}) (\beta^ {B} (\delta)) ^ {2}}{e _ {k} + n _ {0}}}. \\ \end{array}
$$
Then, by Bellman equations,
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} (s, a) - Q _ {h} ^ {\star} (s, a) \leq \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \left[ \overline {{V}} _ {h + 1} ^ {\ell^ {i}} - V _ {h + 1} ^ {\star} \right] \left(s _ {h + 1} ^ {\ell^ {i}}\right) + \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \left[ V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell^ {i}}\right) - p _ {h} V _ {h + 1} ^ {\star} (s, a) \right] \\ + (1 2 0 0 \mathrm {e} + 1) \frac {r _ {0} H (\beta^ {\max } (\delta)) ^ {4}}{e _ {k} + n _ {0}} + 6 0 \mathrm {e} ^ {2} \cdot \frac {r _ {0} H \beta^ {\max } (\delta)}{\sqrt {e _ {k} + n _ {0}}} \\ \end{array}
$$
By the definition of event  $\mathcal{E}^{\mathrm{conc}}(\delta)$  we conclude the statement.
Let us define  $\delta_h^t = \overline{V}_h^t (s_h^t) - V_h^{\pi^t}(s_h^t)$  and  $\zeta_h^t = \overline{V}_h^t (s_h^t) - V_h^\star (s_h^t)$ .
Lemma 5. Assume conditions of Proposition 3 hold. Then on event  $\mathcal{G}'(\delta) = \mathcal{G}(\delta) \cap \mathcal{E}^{\mathrm{opt}}$ , where  $\mathcal{G}(\delta)$  is defined in Lemma 4, the following upper bound on regret holds
$$
\mathfrak {R} ^ {T} \leq \mathrm {e} H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {1} \left\{k _ {h} ^ {t} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) = - 1 \right\} + \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \xi_ {h} ^ {t} + \mathrm {e} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t},
$$
where  $\xi_h^t = p_h[V_{h + 1}^\star -V_{h + 1}^{\pi^t}](s_h^t,a_h^t) - [V_{h + 1}^\star -V_{h + 1}^{\pi^t}](s_{h + 1}^t)$  and  $\mathcal{B}_h^t = \mathcal{B}_h^t (s_h^t,a_h^t)\cdot \mathbb{1}\{k_h^t (s_h^t,a_h^t)\geq 0\}$  for  $\mathcal{B}_h^t$  defined in Corollary 1.
Proof. We notice that on the event  $\mathcal{E}^{\mathrm{opt}}$  the following upper bound holds
$$
\Re^ {T} \leq \sum_ {t = 1} ^ {T} \delta_ {1} ^ {t}. \tag {8}
$$
Next we analyze  $\delta_h^t$ . By the choice of  $a_h^t = \arg \max_{a\in \mathcal{A}}\overline{Q}_h^t (s_h^t,a)$ , Corollary 1, and Bellman equations, we have
$$
\begin{array}{l} \delta_ {h} ^ {t} = \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - V _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t}) = \overline {{Q}} _ {h} ^ {t} (s _ {h} ^ {t}, a _ {h} ^ {t}) - Q _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ = \overline {{Q}} _ {h} ^ {t} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) - Q _ {h} ^ {\star} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) + Q _ {h} ^ {\star} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) - Q _ {h} ^ {\pi^ {t}} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) \\ \leq H \mathbb {1} \{N _ {h} ^ {t} = 0 \} + \mathbb {1} \{N _ {h} ^ {t} > 0 \} \left(\frac {1}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} + \mathcal {B} _ {h} ^ {t} (s _ {h} ^ {t}, a _ {h} ^ {t}) + p _ {h} [ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi^ {t}} ] (s _ {h} ^ {t}, a _ {h} ^ {t})\right). \\ \end{array}
$$
where  $k_h^t = k_h^t(s_h^t, a_h^t)$ ,  $N_h^t = e_{k_h^t}$ ,  $\ell_{t,h}^i$  is episode of the  $i$ -th visitation of the state-action pair  $(s_h^t, a_h^t)$  during the stage  $k_h^t$ , and additionally by the convention  $0/0 = 0$ . Let  $\xi_h^t = p_h[V_{h+1}^\star - V_{h+1}^{\pi^t}] (s_h^t, a_h^t) - [V_{h+1}^\star - V_{h+1}^{\pi^t}] (s_{h+1}^t)$  be a martingale-difference sequence, and  $\mathcal{B}_h^t = \mathcal{B}_h^t(s_h^t, a_h^t) \mathbb{1}\{N_h^t > 0\}$  then
$$
\delta_ {h} ^ {t} \leq H \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \frac {\mathbb {1} \left\{N _ {h} ^ {t} > 0 \right\}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} - \zeta_ {h + 1} ^ {t} + \delta_ {h + 1} ^ {t} + \xi_ {h} ^ {t} + \mathcal {B} _ {h} ^ {t}.
$$
and, as a result
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} \leq H \sum_ {t = 1} ^ {T} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \left\{N _ {h} ^ {t} > 0 \right\}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} \\ - \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t}. \\ \end{array}
$$
Next we have to analyze the second term, following the approach by Zhang et al. [2020],
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \left\{N _ {h} ^ {t} > 0 \right\}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} = \sum_ {q = 1} ^ {T} \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \left\{N _ {h} ^ {t} > 0 \right\}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} \mathbb {1} \left\{\ell_ {t, h} ^ {i} = q \right\} \\ = \sum_ {q = 1} ^ {T} \zeta_ {h + 1} ^ {q} \cdot \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \{k _ {h} ^ {t} \geq 0 \}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \mathbb {1} \{\ell_ {t, h} ^ {i} = q \}. \\ \end{array}
$$
Notice that  $\sum_{i=1}^{N_h^t} \mathbb{1}\{\ell_{t,h}^i = q\} \leq 1$  since all visitations are increasing in  $i$ , and, moreover, it turns to equality if and only if  $(s_h^q, a_h^q) = (s_h^t, a_h^t)$  and this visitation happens in stage  $k_h^t$ , where  $k_h^t$  is equal to the stage of episode  $q$  with respect to  $(s_h^q, a_h^q, h)$ . Since the sum is over all the next episodes with respect to stage of  $q$ , we have that the number of non-zero elements in the sum over  $t$  is bounded by  $(1 + 1 / H) N_h^t$ . Thus
$$
\sum_ {q = 1} ^ {T} \zeta_ {h + 1} ^ {q} \cdot \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \left\{k _ {h} ^ {t} \geq 0 \right\}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \mathbb {1} \left\{\ell_ {t, h} ^ {i} = q \right\} \leq \left(1 + \frac {1}{H}\right) \sum_ {q = 1} ^ {T} \zeta_ {h + 1} ^ {q}.
$$
After a simple algebraic manipulations and using the fact that  $\zeta_h^t\leq \delta_h^t$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} \leq H \sum_ {t = 1} ^ {T} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \sum_ {t = 1} ^ {T} (1 + 1 / H) \zeta_ {h + 1} ^ {t} - \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t} \\ \leq H \sum_ {t = 1} ^ {T} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \left(1 + \frac {1}{H}\right) \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t}. \\ \end{array}
$$