SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \widetilde {W} _ {j} ^ {- 1} = \xi_ {j} \qquad \qquad \qquad \xi_ {j} \sim \mathrm {B e t a} ((n _ {0} (k) - \widetilde {n} _ {0}) / \kappa , (e _ {k} + \widetilde {n} _ {0}) / \kappa), \\ (\widetilde {W} _ {j} ^ {0}, \ldots , \widetilde {W} _ {j} ^ {e _ {k}}) = (1 - \xi_ {j}) \cdot (\widehat {W} _ {j} ^ {0}, \ldots , \widehat {W} _ {j} ^ {e _ {k}}), \qquad \qquad \widehat {W} _ {j} \sim \mathrm {D i r} (\widetilde {n} _ {0} / \kappa , 1 / \kappa , \ldots , 1 / \kappa), \\ \end{array}
$$
<sup>6</sup>We can augment the space  $\times$  with this additional point if needed.
where  $\xi_{j}$  and  $\widehat{W_j}$  are independent. Therefore, we have the final decomposition
$$
\begin{array}{l} \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} X _ {i} - p _ {h} V _ {h + 1} ^ {\star} (s, a) - \varepsilon L = \underbrace {\xi_ {j} \left(r _ {0} (H - 1) - p _ {h} V _ {h + 1} ^ {\star} (s , a)\right) - \varepsilon L} _ {T _ {\text {a p p r o x}}} \\ + (1 - \xi_ {j}) \cdot \underbrace {\left(\sum_ {i = 0} ^ {e _ {k}} \widehat {W} _ {j} ^ {i} X _ {i} - p _ {h} V _ {h + 1} ^ {\star} (s , a)\right)} _ {T _ {\text {s t o c h}}}. \\ \end{array}
$$
By independence of  $\xi_{j}$  and  $\widehat{W}_j$  we have
$$
\mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} X _ {i} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) + \varepsilon L | \{X _ {i} \} _ {i = 1} ^ {e _ {k}} \right] \geq \mathbb {P} [ T _ {\text {a p p r o x}} \geq 0 ] \cdot \mathbb {P} [ T _ {\text {s t o c h}} \geq 0 ].
$$
We split our problem to lower bound the two separate probabilities.
Approximation error To deal with approximation error, we first of all notice that  $p_h V_{h+1}^\star(s, a) \leq H - 1$ , therefore we have
$$
\mathbb {P} \left[ T _ {\text {a p p r o x}} \geq 0 \right] = \mathbb {P} \left[ \xi_ {j} \geq \frac {\varepsilon L}{H - 1} \right].
$$
Next we assume that  $\varepsilon < (H - 1) / L$ , since  $\xi_{j} \sim \mathrm{Beta}((n_{0}(k) - \widetilde{n}_{0}) / \kappa, (e_{k} + \widetilde{n}_{0}) / \kappa)$ , we may apply Alfers and Dinges [1984, Theorem 1.2"]
$$
\mathbb {P} \left[ T _ {\text {a p p r o x}} \geq 0 \right] \geq \Phi \left(- \operatorname {s i g n} (p - \mu) \cdot \sqrt {2 \bar {\alpha} \operatorname {k l} (p , \mu)}\right),
$$
where  $p = (n_0(k) - \widetilde{n}_0 - \kappa) / (e_k + \widetilde{n}_0 - \kappa)$  and  $\mu = \varepsilon L / (H - 1)$ . Since  $n_0(k) = \lceil \widetilde{n}_0 + \kappa + \frac{\varepsilon L}{H - 1} \cdot (e_k + \widetilde{n}_0 + \kappa) \rceil$ , we have  $\mathbb{P}[T_{\mathrm{approx}} \geq 0] \geq 1/2$ .
Stochastic error Since  $X_0 = r_0(H - 1)$  is an upper bound on  $V$ -function, and we have that the weight of the first atom  $\alpha_0 \triangleq \widetilde{n}_0 / \kappa - 1 = c_0 + \log_{17/16}(T) - 1$  for  $c_0$  defined in (11).
Define a measure  $\bar{\nu}_{e_k} = \frac{\tilde{n}_0 - \kappa}{e_k + \tilde{n}_0 - \kappa}\delta_{X_0} + \sum_{i = 1}^{e_k}\frac{1}{e_k + n_0 - 1}\delta_{X_i}$ . Since  $p_hV_{h + 1}^\star (s,a)\leq H - h - 1$ , we can apply Lemma 10 with a fixed  $\varepsilon = 1 / 2$  conditioned on independent random variables  $X_{i}$
$$
\begin{array}{l} \mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} \widehat {W} _ {j} ^ {i} X _ {i} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \{X _ {i} \} _ {i = 1} ^ {e _ {k}} \right] \\ \geq \frac {1}{2} \left(1 - \Phi \left(\sqrt {\frac {2 \left(e _ {k} + n _ {0} - \kappa\right) \mathcal {K} _ {\inf } \left(\bar {\nu} _ {e _ {k}} , p _ {h} V _ {h + 1} ^ {\star} (s , a)\right)}{\kappa}}\right)\right), \\ \end{array}
$$
where  $\Phi$  is a CDF of a normal distribution. By Lemma 12 and the event  $\mathcal{E}^{\star}(\delta, \varepsilon)$
$$
\left(e _ {k} + n _ {0} - \kappa\right) \mathcal {K} _ {\inf } \left(\bar {\nu} _ {e _ {k}}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq e _ {k} \mathcal {K} _ {\inf } \left(\widehat {\nu} _ {e _ {k}}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq \beta^ {\star} (\delta , T, \varepsilon),
$$
where  $\widehat{\nu}_{e_k} = \frac{1}{e_k}\sum_{i = 1}^{e_k}\delta_{V_{h + 1}^* (F(s,a,\xi_{h + 1}^{\ell i}))}$ , and, as a corollary
$$
\mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} \widehat {W} _ {j} ^ {i} X _ {i} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) \mid \mathcal {E} ^ {\star} (\delta , \varepsilon), \{X _ {i} \} _ {i = 1} ^ {e _ {k}} \right] \geq \frac {1}{2} \left(1 - \Phi \left(\sqrt {\frac {2 \beta^ {\star} (\delta , T , \varepsilon)}{\kappa}}\right)\right).
$$
By taking  $\kappa = 2\beta^{\star}(\delta, T, \varepsilon)$  we have a constant probability of being optimistic for stochastic error
$$
\mathbb {P} \left[ T _ {\text {s t o c h}} \geq 0 \mid \mathcal {E} ^ {\star} (\delta , \varepsilon) \right] \geq \frac {1 - \Phi (1)}{2}.
$$
Overall, combining two lower bound for approximation and stochastic terms, we have
$$
\mathbb {P} \left[ \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} X _ {i} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) + \varepsilon L | \mathcal {E} ^ {\star} (\delta , \varepsilon) \right] \geq \frac {1 - \Phi (1)}{4} = \gamma .
$$
Next, using a choice  $J = \lceil (\log (2HT / \delta) + \log (N_{\varepsilon})) / \log (1 / (1 - \gamma)) \rceil = \lceil \tilde{c}_J \cdot (\log (2HT / \delta) + \log (N_{\varepsilon})) \rceil$
$$
\mathbb {P} \left[ \max  _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} X _ {i} \right\} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) + \varepsilon L | \mathcal {E} ^ {\star} (\delta , \varepsilon) \right] \geq 1 - (1 - \gamma) ^ {J} \geq 1 - \frac {\delta}{2 N _ {\varepsilon} H T}.
$$
Proposition 5. It holds  $\mathcal{E}^{\mathrm{opt}}\subseteq \mathcal{E}^{\mathrm{anticonc}}$
Proof. We proceed by a backward induction over  $h$ . Base of induction  $h = H + 1$  is trivial. Fix state-action pair  $(s, a)$  and let us call  $(s', a')$  a center of the ball  $\psi_{\varepsilon}(s, a)$  that is the ball where  $(s, a)$  contains.
Next by the update formula for  $\overline{Q}_h^t$ , and Bellman equations
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} (\psi_ {\varepsilon} (s, a)) - Q _ {h} ^ {\star} (s, a) = \max _ {j \in [ J ]} \Biggl \{\sum_ {i = 0} ^ {n} W _ {j, n} ^ {i} [ r _ {h} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) - r _ {h} (s ^ {\prime}, a ^ {\prime}) ] \\ \left. \right.\left. + \sum_ {i = 0} ^ {n} W _ {j, n} ^ {i} \bar {V} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right) - p _ {h} V _ {h + 1} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right)\right\} + \left[ Q _ {h} ^ {\star} (s, a) - Q _ {h} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right)\right], \\ \end{array}
$$
where  $n = e_{k_h^t (B)}$  and we drop dependence on  $k,t,h,s,a$  in  $\ell^i$ . By induction hypothesis we have  $\overline{V}_{h + 1}^{\ell^i}(s')\geq \overline{Q}_{h + 1}^{\ell^i}(\psi_\varepsilon (s',\pi^\star (s')))\geq Q_\varepsilon^\star (s',\pi^\star (s')) = V_\varepsilon^\star (s')$  for any  $i$ , thus combining it with Lipschitz continuity of reward function and  $Q^{\star}$ , and the value of  $r_h(s^{\ell^0},a^{\ell^0}) = r_0 > r_h(s,a)$
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} \left(\psi_ {\varepsilon} (s, a)\right) - Q _ {h} ^ {\star} (s, a) \geq \max  _ {j \in [ J ]} \left\{W _ {j, n} ^ {0} r _ {0} (H - 1) + \sum_ {i = 1} ^ {n} W _ {j, n} ^ {i} V _ {h + 1} ^ {\star} \left(F _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}, \xi_ {h} ^ {\ell^ {i}}\right)\right) \right\} \\ - p _ {h} V _ {h + 1} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) - \left(L _ {r} + L _ {V}\right) \varepsilon . \\ \end{array}
$$
Next we apply Lipschitz continuity of  $F_{h}$  and  $V_{h + 1}^{\star}$  and obtain
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} \left(\psi_ {\varepsilon} (s, a)\right) - Q _ {h} ^ {\star} (s, a) \geq \max  _ {j \in [ J ]} \left\{W _ {j, n} ^ {0} r _ {0} (H - 1) + \sum_ {i = 1} ^ {n} W _ {j, n} ^ {i} V _ {h + 1} ^ {\star} \left(F _ {h} \left(s ^ {\prime}, a ^ {\prime}, \xi_ {h} ^ {\ell^ {i}}\right)\right) \right\} \\ - p _ {h} V _ {h + 1} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) - \left(L _ {r} + L _ {V} \left(1 + L _ {F}\right)\right) \varepsilon . \\ \end{array}
$$
By the definition of event  $\mathcal{E}^{\mathrm{anticonc}}$  we conclude the statement.
Proposition 6 (Optimism). Define a constant  $L = L_{r} + L_{V}(1 + L_{F})$ . Assume that  $J = \lceil \tilde{c}_{J} \cdot (\log(2HT / \delta) + \log(N_{\varepsilon}) \rceil$ ,  $\kappa = 2\beta^{\star}(\delta, T, \varepsilon)$ ,  $r_0 = 2$ , and a prior count  $n_0(k) = \lceil \widetilde{n}_0 + \kappa + \frac{\varepsilon L}{H - 1} \cdot (e_k + \widetilde{n}_0 + \kappa) \rceil$  dependent on the stage  $k$ , where  $\widetilde{n}_0 = (c_0 + 1 + \log_{17/16}(2e_k)) \cdot \kappa$ ,  $c_0$  is defined in (11),  $\tilde{c}_{J}$  is defined in (12). Then  $\mathbb{P}(\mathcal{E}^{\mathrm{opt}} \mid \mathcal{E}^{\star}(\delta, \varepsilon)) \geq 1 - \delta / 2$ .
E.5 Regret Bounds
As in the tabular setting, we first connect our algorithm to the algorithm by Song and Sun [2019], using the following corollary. Define an event  $\mathcal{G}'(\delta, \varepsilon) = \mathcal{G}(\delta, \varepsilon) \cap \mathcal{E}^{\mathrm{opt}}$ .
Let us define the logarithmic term as follows
$$
\beta^ {\max } (\delta , \varepsilon) = \max  \left\{\kappa , \widetilde {n} _ {0} / \kappa , \beta^ {B} (\delta , \varepsilon), \beta (\delta , \varepsilon), \beta^ {\operatorname {c o n c}} (\delta , \varepsilon) \right\}
$$
that has dependence of order  $\mathcal{O}(\log (TH / \delta) + \log N_{\varepsilon})$
Corollary 2. Fix  $\varepsilon \in (0, L_V / H)$  and assume conditions of Proposition 6. Let  $t \in [T]$ ,  $h \in [H]$ ,  $B \in \mathcal{N}_{\varepsilon}$ . Define  $k = k_h^t(B)$  and let  $\ell^1 < \ldots < \ell^{e_k}$  be a excursion of  $(B, h)$  till the end of the previous stage. Then on the event  $\mathcal{G}'(\delta)$  the following bound holds for  $k \geq 0$  and any  $(s, a) \in B$
$$
0 \leq \overline {{Q}} _ {h} ^ {t} (B) - Q _ {h} ^ {\star} (s, a) \leq \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} [ \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) - V _ {h + 1} ^ {\star} (s _ {h + 1} ^ {\ell^ {i}}) ] + \mathcal {B} _ {h} ^ {t} (k),
$$
$$
\mathcal {B} _ {h} ^ {t} (k) = 1 2 1 \mathrm {e} ^ {2} \cdot \sqrt {\frac {H ^ {2} (\beta^ {\max} (\delta , \varepsilon)) ^ {2}}{e _ {k}}} + 2 4 0 \mathrm {e} \cdot \frac {H (\beta^ {\max} (\delta , \varepsilon)) ^ {4}}{e _ {k}} + 3 (L _ {r} + (1 + L _ {F}) L _ {V}) \varepsilon .
$$
Proof. The lower bound follows from the definition of the event  $\mathcal{E}^{\mathrm{opt}}$ . For the upper bound we first apply the decomposition for  $\overline{Q}_h^t (s,a)$  and the definition of event  $\mathcal{E}^B (\delta ,\varepsilon)$  from Lemma 6
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} (B) = \max _ {j \in [ J ]} \left\{\sum_ {i = 0} ^ {e _ {k}} W _ {j, n} ^ {i} \left(r _ {h} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) + \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}})\right) \right\} \\ \leq \frac {1}{e _ {k} + n _ {0} (k)} \sum_ {i = 1} ^ {e _ {k}} \left(r _ {h} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) + \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}})\right) + \frac {n _ {0} (k) \cdot 2 H}{e _ {k} + n _ {0} (k)} \\ + 1 2 0 \mathrm {e} ^ {2} \sqrt {\frac {H ^ {2} \kappa \beta^ {B} (\delta , \varepsilon)}{e _ {k} + n _ {0} (k)}} + 2 4 0 0 \mathrm {e} \frac {H \kappa \log (n + n _ {0} (k)) (\beta^ {B} (\delta , \varepsilon)) ^ {2}}{e _ {k} + n _ {0} (k)}. \\ \end{array}
$$
Additionally, by Bellman equations
$$
\begin{array}{l} Q _ {h} ^ {\star} (s, a) = \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} Q _ {h} ^ {\star} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) + \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \Big (Q _ {h} ^ {\star} (s, a) - Q _ {h} ^ {\star} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) \Big) \\ \geq \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \left(r _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right) + p _ {h} V _ {h + 1} ^ {\star} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right)\right) - 2 \varepsilon L _ {V}. \\ \end{array}
$$
Combining and using the fact that  $n_0(k) \leq \frac{L\varepsilon}{H - 1} \cdot (e_k + n_0(k)) + \widetilde{n}_0 + \kappa$  for  $L = L_r + (1 + L_F)L_V$ ,
$$
\begin{array}{l} \overline {{Q}} _ {h} ^ {t} (s, a) - Q _ {h} ^ {\star} (s, a) \leq \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \bigg [ \overline {{V}} _ {h + 1} ^ {\ell^ {i}} - V _ {h + 1} ^ {\star} \bigg ] (s _ {h + 1} ^ {\ell^ {i}}) + \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \bigg [ V _ {h + 1} ^ {\star} (s _ {h + 1} ^ {\ell^ {i}}) - p _ {h} V _ {h + 1} ^ {\star} (s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}) \bigg ] \\ + 1 2 0 \mathrm {e} ^ {2} \cdot \sqrt {\frac {H ^ {2} (\beta^ {\max} (\delta , \varepsilon)) ^ {2}}{e _ {k}}} + (2 4 0 0 \mathrm {e} + 2) \frac {H (\beta^ {\max} (\delta , \varepsilon)) ^ {4}}{e _ {k}} + 3 L \varepsilon . \\ \end{array}
$$
Finally, the applications of event  $\mathcal{E}^{\mathrm{conc}}(\delta ,\varepsilon)$  conclude the statement.
Lemma 7. Assume conditions of Proposition 6. Then on event  $\mathcal{G}'(\delta, \varepsilon) = \mathcal{G}(\delta, \varepsilon) \cap \mathcal{E}^{\mathrm{opt}}$ , where  $\mathcal{G}(\delta, \varepsilon)$  is defined in Lemma 6, the following upper bound on regret holds
$$
\mathfrak {R} ^ {T} \leq 2 \mathrm {e} H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \sum_ {t = 1} ^ {t} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \xi_ {h} ^ {t} + \mathrm {e} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t}.
$$
where  $\xi_h^t = p_h[V_{h + 1}^\star -V_{h + 1}^\pi ](s_h^t,a_h^t) - [V_{h + 1}^\star -V_{h + 1}^\pi ](s_{h + 1}^t)$  and  $\mathcal{B}_h^t = \mathcal{B}_h^t (k_h^t (s_h^t,a_h^t))\cdot$ $\mathbb{1}\{k_h^t (s_h^t,a_h^t)\geq 0\}$  for  $\mathcal{B}_h^t$  defined in Corollary 2.
Proof. As in the tabular setting, we notice that on the event  $\mathcal{E}^{\mathrm{opt}}$  we can upper bound the regret in terms of  $\delta_1^t$ .
$$
\Re^ {T} \leq \sum_ {t = 1} ^ {T} \delta_ {1} ^ {t}. \tag {13}
$$
Next we analyze  $\delta_h^t$ . Since  $a_h^t = \arg \max_{a\in \mathcal{A}}\overline{Q}_h^t (\psi_\varepsilon (s_h^t,a))$ , we can use Corollary 2 and Bellman equations in the following way
$$
\begin{array}{l} \delta_ {h} ^ {t} = \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - V _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t}) = \overline {{Q}} _ {h} ^ {t} (B _ {h} ^ {t}) - Q _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ = \overline {{Q}} _ {h} ^ {t} (B _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) + Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) - Q _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ \leq r _ {0} H \mathbb {1} \{N _ {h} ^ {t} = 0 \} + \mathbb {1} \{N _ {h} ^ {t} > 0 \} \left(\frac {1}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} + \mathcal {B} _ {h} ^ {t} (k _ {h} ^ {t}) + p _ {h} [ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi^ {t}} ] (s _ {h} ^ {t}, a _ {h} ^ {t})\right). \\ \end{array}
$$
where  $k_h^t = k_h^t (B_h^t)$ ,  $N_{h}^{t} = e_{k_{h}^{t}}$ ,  $\ell_{t,h}^{i}$  is an  $i$ -th visitation of the ball  $B_{h}^{t}$  during an stage  $k_h^t$ , and additionally by a convention  $0 / 0 = 0$ .
Define  $\xi_h^t = p_h[V_{h + 1}^\star -V_{h + 1}^\pi ](s_h^t,a_h^t) - [V_{h + 1}^\star -V_{h + 1}^\pi ](s_{h + 1}^t)$  a martingale-difference sequence, and  $\mathcal{B}_h^t = \mathcal{B}_h^t (k_h^t)\mathbb{1}\{N_h^t >0\}$  then
$$
\delta_ {h} ^ {t} \leq r _ {0} H \mathbb {1} \{N _ {h} ^ {t} = 0 \} + \frac {\mathbb {1} \{N _ {h} ^ {t} > 0 \}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} - \zeta_ {h + 1} ^ {t} + \delta_ {h + 1} ^ {t} + \xi_ {h} ^ {t} + \mathcal {B} _ {h} ^ {t}.
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} \leq r _ {0} H \sum_ {t = 1} ^ {T} \mathbb {1} \{N _ {h} ^ {t} = 0 \} + \sum_ {t = 1} ^ {T} \frac {\mathbb {1} \{N _ {h} ^ {t} > 0 \}}{N _ {h} ^ {t}} \sum_ {i = 1} ^ {N _ {h} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {t, h} ^ {i}} \\ - \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t}. \\ \end{array}
$$
For the second term we may repeat arguments as in the proof of Lemma 5 and obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} \leq H \sum_ {t = 1} ^ {T} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \sum_ {t = 1} ^ {T} (1 + 1 / H) \zeta_ {h + 1} ^ {t} - \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t} \\ \leq H \sum_ {t = 1} ^ {T} \mathbb {1} \{N _ {h} ^ {t} = 0 \} + \left(1 + \frac {1}{H}\right) \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} + \sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t}. \\ \end{array}
$$
By rolling out the upper bound on regret (13) we have
Proof of Theorem 2. First, we notice that the event  $\mathcal{G}'(\delta, \varepsilon)$  defined in Lemma 7, holds with probability at least  $1 - \delta$  by Lemma 6 and Proposition 6. Thus, we may assume that  $\mathcal{G}'(\delta, \varepsilon)$  holds for  $\varepsilon > 0$  that we will specify later.
By Lemma 7
$$
\mathfrak {R} ^ {T} \leq 2 \mathrm {e} H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {1} \left\{k _ {h} ^ {t} = - 1 \right\} + \sum_ {t = 1} ^ {t} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \xi_ {h} ^ {t} + \mathrm {e} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t}.
$$
The first term is upper bounded by  $2\mathrm{e}H^{3}\cdot N_{\varepsilon}$ , since there is no more than  $H$  visits of each ball in  $\varepsilon$ -net before the update for the first stage. The second term is bounded by  $\mathcal{O}(\sqrt{H^3T\beta^{\max}(\delta,\varepsilon)})$  by a definition of the event  $\mathcal{E}(\delta)$  in Lemma 6.
To analyze the last term, consider the following sum
$$
\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \left\{e _ {k _ {h} ^ {t} \left(B _ {h} ^ {t}\right)} > 0 \right\}}{\sqrt {e _ {k _ {h} ^ {t} \left(B _ {h} ^ {t}\right)}}} \leq \sum_ {(B, h) \in \mathcal {N} _ {\varepsilon} \times [ H ]} \sum_ {k = 0} ^ {k _ {h} ^ {T} (B)} \frac {e _ {k + 1}}{\sqrt {e _ {k}}},
$$
$$
e _ {k} = \left\lfloor \left(1 + \frac {1}{H}\right) ^ {k} H \right\rfloor \Rightarrow \frac {e _ {k + 1}}{\sqrt {e _ {k}}} \leq 2 \sqrt {H} \left(1 + \frac {1}{H}\right) ^ {k / 2},
$$
$$
\sum_ {h = 0} ^ {k _ {h} ^ {T} (B)} \frac {e _ {k + 1}}{\sqrt {e _ {k}}} \leq 4 \sqrt {H} \frac {(1 + 1 / H) \left(k _ {h} ^ {T} (B) + 1\right) / 2}{\sqrt {1 + 1 / H} - 1} = 4 H \sqrt {e ^ {k _ {h} ^ {T} (B) + 1}}. \tag {14}
$$
$$
N _ {h} ^ {T + 1} (B) \geq \sum_ {k = 0} ^ {k _ {h} ^ {T} (B)} e ^ {k} = H (e ^ {k _ {h} ^ {T} (B) + 1} - 1) \Rightarrow e ^ {k _ {h} ^ {T} (B) + 1} \leq \frac {N _ {h} ^ {T + 1} (B) + 1}{H}
$$
thus from the Cauchy-Schwarz inequality
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \left\{e _ {k _ {h} ^ {t}} \left(B _ {h} ^ {t}\right) > 0 \right\}}{\sqrt {e k _ {h} ^ {t} \left(B _ {h} ^ {t}\right)}} \leq 4 \sqrt {H} \sum_ {(B, h) \in \mathcal {N} _ {\varepsilon} \times [ H ]} \sqrt {N _ {h} ^ {T + 1} (B) + 1} \\ \leq 4 \sqrt {S A H ^ {2}} \sqrt {\sum_ {(B , h)} \left(N _ {h} ^ {T + 1} (B) + 1\right)} \leq 4 \sqrt {H ^ {3} T \cdot N _ {\varepsilon}} + 4 N _ {\varepsilon} H ^ {2}. \\ \end{array}
$$
By the similar arguments we have
$$
\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \left\{e _ {k _ {h} ^ {t}} \left(B _ {h} ^ {t}\right) > 0 \right\}}{e _ {k _ {h} ^ {t}} \left(B _ {h} ^ {t}\right)} \leq \mathcal {O} (H N _ {\varepsilon} \log (T)).
$$
Using this upper bound, we have for  $L = L_{r} + (1 + L_{F})L_{V}$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t} = \mathcal {O} \left(H \beta^ {\max } (\delta , \varepsilon) \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \{e _ {k _ {h} ^ {t} (s _ {h} ^ {t} , a _ {h} ^ {t})} > 0 \}}{\sqrt {e _ {k _ {h} ^ {t} (s _ {h} ^ {t} , a _ {h} ^ {t})}}} ^ {\prime}}\right) \\ + \mathcal {O} \left(H (\beta^ {\max} (\delta , \varepsilon)) ^ {4} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \{e _ {k _ {h} ^ {t}} (s _ {h} ^ {t} , a _ {h} ^ {t}) > 0 \}}{\sqrt {\mathcal {E} k _ {h} ^ {t} (s _ {h} ^ {t} , a _ {h} ^ {t})}}\right) \\ + \mathcal {O} (L T H \varepsilon) \\ \leq \mathcal {O} \Big (\sqrt {H ^ {5} T N _ {\varepsilon} \cdot (\beta^ {\max} (\delta , \varepsilon)) ^ {2}} + H ^ {3} N _ {\varepsilon} (\beta^ {\max} (\delta , \varepsilon)) ^ {4} + L T H \varepsilon \Big). \\ \end{array}
$$
Overall, for any fixed  $\varepsilon > 0$  we have
$$
\mathfrak {R} ^ {T} \leq \mathcal {O} \Big (\sqrt {H ^ {5} T N _ {\varepsilon} \cdot (\beta^ {\max } (\delta , \varepsilon)) ^ {2}} + H ^ {3} N _ {\varepsilon} (\beta^ {\max } (\delta , \varepsilon)) ^ {4} + L T H \varepsilon + \sqrt {H ^ {3} T} \Big).
$$
Next we finally use that  $\mathcal{S} \times \mathcal{A}$  have covering dimension  $d_{c}$  that means  $N_{\varepsilon} \leq C_N \cdot \varepsilon^{-d_c}$ , thus our regret bound transforms as follows
$$
\begin{array}{l} \mathfrak {R} ^ {T} \leq \mathcal {O} \left(\sqrt {H ^ {5} T C _ {N} \varepsilon^ {- d _ {c}} \cdot (\log (T C _ {N} H / \delta) + d _ {c} \log (1 / \varepsilon)) ^ {2}} \right. \\ \left. + H ^ {3} C _ {N} \varepsilon^ {- d _ {c}} \left(\log \left(T C _ {N} H / \delta\right) + d _ {c} \log (1 / \varepsilon)\right) ^ {4} + L T H \varepsilon\right). \\ \end{array}
$$
By taking  $\varepsilon = T^{-1 / (d_c + 2)}$  we conclude the statement
F Adaptive RandQL
In this section we describe how to improve the dependence in our algorithm from covering dimension to zooming dimension, and describe all required notation.
F.1 Additional Notation
In this section we introduce an additional notation that is needed for introducing an adaptive version of RandQL algorithm for metric spaces.
Hierarchical partition Next, we define all required notation to describe an adaptive partition, as Sinclair et al. [2019, 2023]. Finally, we define the following general framework of hierarchical partition. Instead of balls, we will use a more general notion of regions that will induce a better structure from the computational point of view. We recall for any compact set  $A \subseteq S \times \mathcal{A}$  we call  $\mathrm{diam}(A) = \max_{x,y \in A} \rho(x,y)$ .
Definition 6. A hierarchical partition of  $S \times \mathcal{A}$  of a depth  $d > 0$  is a collection of regions  $\mathcal{P}_d$  and their centers such that
Each region  $B\in \mathcal{P}_d$  is of the form  $\mathcal{S}(B)\times \mathcal{A}(B)$  , where  $\mathcal{S}(B)\subseteq \mathcal{S},\mathcal{A}(B)\subseteq \mathcal{A}$
-  $\mathcal{P}_d$  is a cover of  $S \times \mathcal{A}$ :  $\bigcup_{B \in \mathcal{P}_d} B = S \times \mathcal{A}$ ;
- For every  $B \in \mathcal{P}_d$ , we have  $\mathrm{diam}(B) \leq d_{\max} \cdot 2^{-d}$ ;
- Let  $B_{1}, B_{2} \in \mathcal{P}_{d}$ . If  $B_{1} \neq B_{2}$  then  $\rho(\operatorname{center}(B_{1}), \operatorname{center}(B_{2})) \geq d_{\max} \cdot 2^{-d}$ ;
- For any  $B \in \mathcal{P}_d$ , there exists a unique  $A \in \mathcal{P}_{d - 1}$  (called the parent of  $B$ ) such that  $B \subseteq A$ .
and, for  $d = 0$  we define it as  $\mathcal{P}_0 = \{\mathcal{S}\times \mathcal{A}\}$
We call the tree generated by the structure of  $\mathcal{T} = \{\mathcal{P}_d\}_{d\geq 0}$  a tree of this hierarchical partition. The main example of this partition is the dyadic partition of  $S\times \mathcal{A}$  in the case of  $S = [0,1]^{d_S},\mathcal{A} = [0,1]^{d_{\mathcal{A}}}$  and the metric induced by the infinity norm  $\rho ((s,a),(s^{\prime},a^{\prime})) = \max \{\| s - s^{\prime}\|_{\infty},\| a - a^{\prime}\|_{\infty}\}$ . For examples we refer to [Sinclair et al., 2023].
F.2 Algorithm
In this section we describe two algorithms: Adaptive-RandQL which is an adaptive metric counterpart of RandQL, and Adaptive-Staged-RandQL which is an adaptive metric counterpart of Staged-RandQL. First, we start from the notation and algorithmic parts that will be common for both algorithms.