SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithms maintain an adaptive partition  $\mathcal{P}_h^t$  of  $\mathcal{S} \times \mathcal{A}$ , that is a sub-tree of an (infinite) tree of the hierarchical partition  $\mathcal{T} = \{\mathcal{P}_d\}_{d \geq 0}$ . We initialize  $\mathcal{P}_h^1 = \{\mathcal{P}_0\}$ , and then we refine the tree  $\mathcal{P}_h^t$  by adding new nodes that corresponding to nodes of  $\mathcal{T}$ . The leaf nodes of  $\mathcal{P}_h^t$  represent the active balls, and for  $B \in \mathcal{P}_h^t$  the set of its inactive parent balls is defined as  $\{B' \in \mathcal{P}_h^t \mid B \subset B'\}$ . For any  $B \in \mathcal{P}_h^t$  we define  $d(B)$  as a depth of  $B$  in the tree under a convention  $d(\mathcal{S} \times \mathcal{A}) = 0$ .
Additionally, we need to define so-called selection rule and splitting rule. For any state  $s \in S$  we define the set of all relevant balls as  $\mathcal{R}_h^t(s) = \{ \text{active } b \in \mathcal{P}_h^t \mid (s, a) \in B \text{ for some } a \in \mathcal{A} \}$ . Then for the current state  $s_h^t$  we define the current ball as  $B_h^t = \arg \max_{B \in \mathcal{R}_h^t(s_h^t)} \overline{Q}_h^t(B)$  and the corresponding action as  $a_h^t$ . To define the splitting rule we maintain the counters  $n_h^t(B)$  for all  $B \in \mathcal{P}_h^t$  as a number of visits of a node  $B$  and all its parent nodes. Then we will perform splitting of the current ball  $B_h^t$  if  $\sqrt{d_{\max}^2 / n_h^t(B_h^t)} \leq \mathrm{diam}(B_h^t)$ . During splitting, we extend  $\mathcal{P}_h^{t+1}$  by its child nodes in the hierarchical partition tree  $\mathcal{T}$ . For more details we refer to [Sinclair et al., 2023], up to small changes in notation. In particular, their constant  $\tilde{C}$  is equal to  $d_{\max}$  in our setting to make the construction exactly the same for both Adaptive-RandQL and Adaptive-Staged-RandQL algorithms.
Algorithm 5 Adaptive-RandQL
1: Input: ensemble size  $J$ , number of prior transitions  $n_0$ , prior reward  $r_0$ .  
2: Initialize:  $\overline{Q}_h(B) = \widetilde{Q}_h^j(B) = r_0H$ , initialize counters  $n_h(s,a) = 0$  for  $h,s,a \in [H] \times S \times \mathcal{A}$ .  
3: for  $t \in [T]$  do  
4: for  $h \in [H]$  do  
5: Compute  $B_h = \arg \max_{B \in \mathcal{R}_h^t(s_h)} \overline{Q}_h(B)$  and play  $a_h$  for  $(s_h,a_h) = \text{center}(B_h)$ ;  
6: Observe reward and next state  $s_{h+1} \sim p_h(s_h,a_h)$ .  
7: Sample  $\mathring{w}_j \sim \text{Beta}(n,n_0)$  for  $n = n_h(B_h)$ .  
8: Compute value  $\overline{V}_{h+1}(s_{h+1}) = \max_{B \in \mathcal{R}_h^t(s_{h+1})} \overline{Q}_{h+1}(B)$ .  
9: Build targets for all  $j \in [J]$ $\mathring{Q}_h^j = \mathring{w}_j[r_h(s_h,a_h) + \overline{V}_{h+1}(s_{h+1})] + (1 - \mathring{w}_j)r_0H$ .  
10: Sample learning rates  $w_j \sim \text{Beta}(H,n)$ .  
11: Update ensemble  $Q$ -functions for all  $j \in [J]$ $\widetilde{Q}_h^j(B_h) := (1 - w_j)\widetilde{Q}_h^j(s_h,a_h) + w_j\mathring{Q}_h^j$ .  
12: Update policy  $Q$ -function  $\overline{Q}_h(s_h,a_h) := \max_{j \in [J]} \widetilde{Q}_h^j(s_h,a_h)$ .  
13: Update counters  $n_h(B_h) := n_h(B_h) + 1$ ;  
14: If  $\sqrt{d_{\text{max}}^2 / n_h(B_h)} \leq \text{diam}(B_h)$ , then refine partition  $B_h$  (see Sinclair et al. [2023]).  
15: end for  
16: end for
Adaptive-RandQL This algorithm is an adaptive metric version of RandQL algorithm. We recall that for  $B \in \mathcal{P}_h^t$  we define  $n_h^t(B) = \sum_{i=1}^{t-1} \mathbb{1}\{(B_h^i) \text{ is a parent of } B\}$  is the number of visits of the ball  $B$  and its parent balls at step  $h$  before episode  $t$ . We start by initializing the ensemble of Q-values, the policy Q-values, and values to an optimistic value  $\widetilde{Q}_h^{t,j}(B) = \overline{Q}_h^1(B) = \overline{V}_h^1(B) = r_0H$  for all  $(j,h) \in [J] \times [H]$  and the unique ball in the partition  $B = \mathcal{S} \times \mathcal{A}$  and  $r_0 > 0$  some pseudo-rewards. At episode  $t$  we update the ensemble of Q-values as follows, denoting by  $n = n_h^t(B)$  the count,  $w_{j,n} \sim \mathrm{Beta}(H,n)$  the independent learning rates,
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (B) = \left\{ \begin{array}{l l} (1 - w _ {j, n}) \widetilde {Q} _ {h} ^ {t, j} (B) + w _ {j, n} \mathring {Q} _ {h} ^ {t, j} (s _ {h} ^ {t}, a _ {h} ^ {t}), & B = B _ {h} ^ {t} \\ \widetilde {Q} _ {h} ^ {t, j} (B) & \text {o t h e r w i s e}, \end{array} \right.
$$
where we defined the target  $\hat{Q}_h^{t,j}(s_h^t,a_h^t)$  as a mixture between the usual target and some prior target with mixture coefficient  $\tilde{w}_{n,j}\sim \mathrm{Beta}(n,n_0)$  and  $n_0$  the number of prior samples,
$$
\mathring {Q} _ {h} ^ {t, j} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) = \mathring {w} _ {j, n} \left[ r _ {h} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) + \overline {{V}} _ {h + 1} ^ {t} \left(s _ {h + 1} ^ {t}\right) \right] + \left(1 - \mathring {w} _ {j, n}\right) r _ {0} H.
$$
For a discussion on prior re-injection we refer to Appendix B. The value function is computed on-flight by the rule  $\overline{V}_h^t (s) = \max_{B\in \mathcal{R}_h^t}\overline{Q}_h^t (B)$ .
$$
\overline {{Q}} _ {h} ^ {t + 1} (B) = \max  _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {t + 1, j} (B).
$$
The policy is then greedy with respect to the policy Q-values and selection rule  $(s, \pi_h^{t+1}(s)) = \text{center}(B)$ , where  $B = \arg \max_{B \in \mathcal{R}_h^{t+1}} \overline{Q}_h^{t+1}(B)$ . After the update of Q-values, algorithm verifies the splitting rule. If the splitting rule is triggered, then all new balls are defined by counter and Q-values of its parent. We notice that all Q-values could be efficiently computed on the nodes of the adaptive partition. The complete and detailed description is presented in Algorithm 6.
Adaptive-Staged-RandQL The notation for this algorithm is very close to Net-Staged-RandQL and we describe only differences between them. The main difference is a way to compute value  $\overline{V}_h^t(s) = \max_{B \in \mathcal{R}_h^t(s)} \overline{Q}_h^t(B)$  and policy  $(s, \pi_h^t(s)) = \text{center}(B)$  for  $B =$
Algorithm 6 Adaptive-Staged-RandQL
1: Input: inflation coefficient  $\kappa$ ,  $J$  ensemble size, number of prior transitions  $n_0(k)$ , prior reward  $r_0$ .  
2: Initialize:  $\overline{Q}_h(B) = \widetilde{Q}_h^j(B) = r_0H$ , initialize counters  $\widetilde{n}_h(B) = 0$  for  $j,h,B\in [J]\times [H]\times \mathcal{N}_{\varepsilon}$ , stage  $q_h(B) = 0$ .  
3: for  $t\in [T]$  do  
4: for  $h\in [H]$  do  
5: Compute  $B_h = \arg \max_{B\in \mathcal{R}_h^t (s_h)}\overline{Q}_h(B)$  and play  $a_{h}$  for  $(s_h,a_h) = \mathrm{center}(B_h)$ ;  
6: Observe reward and next state  $s_{h + 1}\sim p_h(s_h,a_h)$ .  
7: Sample learning rates  $w_{j}\sim \mathrm{Beta}(1 / \kappa ,(\tilde{n} +n_{0}(q_{h}(B_{h})) / \kappa)$  for  $\tilde{n} = \tilde{n}_h(B_h)$ .  
8: Compute value  $\overline{V}_{h + 1}(s_{h + 1}) = \max_{B\in \mathcal{R}_h^t (s_{h + 1})}\overline{Q}_{h + 1}(B)$ .  
9: Update temporary  $Q$ -values for all  $j\in [J]$ $\widetilde{Q}_h^j (B)\coloneqq (1 - w_j)\widetilde{Q}_h^j (B) + w_j(r_h(s_h,a_h) + \overline{V}_{h + 1}(s_{h + 1}))$ .  
10: Update counters  $\widetilde{n}_h(B_h)\coloneqq \widetilde{n}_h(B_h) + 1$  and  $n_h(B_h)\coloneqq n_h(B_h) + 1$ .  
11: if  $\widetilde{n}_h(B_h) = \lfloor (1 + 1 / H)^q H\rfloor$  for  $q = q_{h}(B_{h})$  is the current stage then  
12: Update policy  $Q$ -values  $\overline{Q}_h(B_h)\coloneqq \max_{j\in [J]}\widetilde{Q}_h^j (B_h)$ .  
13: Reset temporary  $Q$ -values  $\widetilde{Q}_h^j (B_h)\coloneqq r_0H$ .  
14: Reset counter  $\widetilde{n}_h(B_h)\coloneqq 0$  and change stage  $k_{h}(B_{h})\coloneqq k_{h}(B_{h}) + 1$ .  
15: end if  
16: If  $\sqrt{d_{\mathrm{max}}^2 / n_h(B_h)}\leq \mathrm{diam}(B_h)$ , then refine partition  $B_{h}$  (see Sinclair et al. [2023]).  
17: end for  
18: end for
$\arg \max_{B\in \mathcal{R}_h^t (s)}\overline{Q}_h^t (B)$  . Additionally, all counters including temporary will move to the child nodes after splitting, as it performed in Adaptive-RandQL. The detailed description is presented in Algorithm 6.
F.3 Regret Bound
In this section we state the regret bounds for Adaptive-Staged-RandQL and derive a proof. The given proof shares a lot of similarities with the proof of Net-Staged-RandQL in the first half and to the proof of Adaptive-QL by Sinclair et al. [2023] in the second half.
We fix  $\delta \in (0,1)$ , and the number of posterior samples
$$
J \triangleq \left[ \tilde {c} _ {J} \cdot \left(\log \left(2 C _ {N} H T / \delta\right) + d _ {c} \log_ {2} \left(8 T / d _ {\max }\right)\right) \right], \tag {15}
$$
where  $\tilde{c}_J = 1 / \log (4 / (3 + \Phi (1)))$  and  $\Phi (\cdot)$  is the cumulative distribution function (CDF) of a normal distribution.
Additionally we select
$$
n _ {0} (k) = \left\lceil \widetilde {n} _ {0} + \kappa + \frac {L \cdot d _ {\mathrm {m a x}}}{H - 1} \cdot \frac {e _ {k} + \widetilde {n} _ {0} + \kappa}{\sqrt {H e _ {k} - k - H ^ {2}}} \right\rceil , \quad \widetilde {n} _ {0} = (c _ {0} + 1 + \log_ {1 7 / 1 6} (T)) \cdot \kappa
$$
where  $c_{0}$  is an absolute constant defined in (5) (see Appendix D.3),  $\kappa$  is the posterior inflation coefficient and  $L = L_{r} + (1 + L_{F})L_{V}$  is a constant. Next we restate the regret bound result for Adaptive-Staged-RandQL algorithm.
Theorem (Restatement of Theorem 3). Consider a parameter  $\delta \in (0,1)$ . Let  $\kappa \triangleq 2(\log(8HC_N/\delta) + d_c\log_2(8T/d_{\max}) + 3\log(\mathrm{e}\pi(2T + 1))$ ,  $r_0 \triangleq 2$ . Then it holds for Adaptive-Staged-RandQL, with probability at least  $1 - \delta$ ,
$$
\mathfrak {R} ^ {T} = \widetilde {\mathcal {O}} \left(L H ^ {3 / 2} \sum_ {h = 1} ^ {H} T ^ {\frac {d _ {z , h} + 1}{d _ {z , h} + 2}}\right),
$$
where  $d_{z,h}$  is the step- $h$  zooming dimension and we ignore all multiplicative factors in the covering dimension  $d_c$ .
Proof. We divide the proof to four main parts, a little bit different from that of Staged-RandQL and Net-Staged-RandQL since we also need to apply clipping techniques.
Concentration events We can define (almost) the same set of events as in Appendix E.3, where union bound over balls is taken over all the hierarchical partition tree up to depth  $D$  that we define as  $\mathcal{T}_D$ .
$$
\mathcal {E} ^ {\star} (\delta) \triangleq \left\{\forall t \in \mathbb {N}, \forall h \in [ H ], \forall B \in \mathcal {T} _ {D}, k = k _ {h} ^ {t} (B), (s, a) = \operatorname {c e n t e r} (B): \right.
$$
$$
\begin{array}{l} \mathcal {E} ^ {B} (\delta , T) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall B \in \mathcal {T} _ {D}, \forall j \in [ J ], k = k _ {h} ^ {t} (B): \right. \\ \left| \sum_ {i = 0} ^ {e _ {k}} \left(W _ {j, e _ {k}, k} ^ {i} - \mathbb {E} \left[ W _ {j, e _ {k}, k} ^ {i} \right]\right) \left(r _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right) + \overline {{V}} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right)\right) \right| \\ \leq 6 0 \mathrm {e} ^ {2} \sqrt {\frac {r _ {0} ^ {2} H ^ {2} \kappa \beta^ {B} (\delta , \varepsilon)}{e _ {k} + n _ {0} (k)}} + 1 2 0 0 \mathrm {e} \frac {r _ {0} H \kappa \log (e _ {k} + n _ {0} (k)) (\beta^ {B} (\delta , \varepsilon)) ^ {2}}{e _ {k} + n _ {0} (k)} \Bigg \}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} ^ {\text {c o n c}} (\delta , T) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall B \in \mathcal {T} _ {D}, k = k _ {h} ^ {t} (B): \right. \\ \left| \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {k, h} ^ {i} (B)}\right) - p _ {h} V _ {h + 1} ^ {\star} \left(s _ {h} ^ {\ell_ {k, h} ^ {i} (B)}, a _ {h} ^ {\ell_ {k, h} ^ {i} (B)}\right) \right| \leq \sqrt {\frac {2 r _ {0} ^ {2} H ^ {2} \beta^ {\text {c o n c}} (\delta , \varepsilon)}{e _ {k}}} \Bigg \}, \\ \mathcal {E} (\delta) \triangleq \left\{\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 3 / H) ^ {H - h} \left| p _ {h} \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) - \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h + 1} ^ {t}\right) \right| \right. \\ \leq 2 \mathrm {e} ^ {3} r _ {0} H \sqrt {2 H T \beta (\delta)}. \\ \end{array}
$$
To apply the union bound argument, we have to bound the size of  $\mathcal{T}_D$ . First, we notice that relation between centers of balls in each layer  $\mathcal{P}_d$  implies that there at least  $|\mathcal{P}_d|$  non-intersecting balls of radius  $d_{\mathrm{max}} \cdot 2^{-d - 2}$ . Thus, the size of this sub-tree could be bounded as
$$
\left| \mathcal {T} _ {D} \right| \leq \sum_ {d = 0} ^ {D} N _ {d _ {\max } 2 ^ {- d - 2}} \leq C _ {N} \sum_ {d = 0} ^ {D} \left(2 ^ {d + 2} / d _ {\max }\right) ^ {d _ {c}} \leq \left(8 / d _ {\max }\right) ^ {d _ {c}} C _ {N} \cdot 2 ^ {d _ {c} \cdot D}.
$$
using the relation between covering and packing numbers, see e.g. Lemma 4.2.8 by Vershynin [2018]. The only undefined quantity here is  $D$ , that can be upper-bounded given budget  $T$ . To do it, we apply Lemma B.2 by Sinclair et al. [2023] for any  $B \in \mathcal{P}_h^t$
$$
\left(\frac {d _ {\operatorname* {m a x}}}{2 \cdot \operatorname {d i a m} (B)}\right) ^ {2} \leq n _ {h} ^ {t} (B) \leq \left(\frac {d _ {\operatorname* {m a x}}}{\operatorname {d i a m} (B)}\right) ^ {2}. \tag {16}
$$
Our goal is to find a value  $D$  such that  $\mathcal{P}_h^{T + 1}\subseteq \mathcal{T}_D$  for any MDPs and correct interactions. To do it, we notice that it is equivalent to show that  $\mathrm{diam}(B)\geq d_{\max}2^{-D}$ , that could be guaranteed since
$$
\operatorname {d i a m} (B) \geq \frac {d _ {\max}}{2 \sqrt {n _ {h} ^ {T + 1} (B)}} \geq \frac {d _ {\max}}{2 T},
$$
which implies that  $D = 1 + \log_2(T)$  is enough. Finally, since for the value of interest
$$
\log | \mathcal {T} _ {D} | \leq d _ {c} \log_ {2} (T) + \log C _ {N} + d _ {c} \log \left(8 / d _ {\max }\right),
$$
we can define the  $\beta$ -functions as follows
$$
\beta^ {\star} (\delta) \triangleq \log \left(8 C _ {N} H / \delta\right) + d _ {c} \log_ {2} (8 T / d _ {\max }) + 3 \log \left(\mathrm {e} \pi (2 n + 1)\right),
$$
$$
\beta^ {B} (\delta , T) \triangleq \log \left(8 C _ {N} H / \delta\right) + d _ {c} \log_ {2} \left(8 T / d _ {\max }\right) + \log (T J),
$$
$$
\beta^ {\text {c o n c}} (\delta , T) \triangleq \log (8 C _ {N} H / \delta) + d _ {c} \log_ {2} (8 T / d _ {\max }) + \log (2 T),
$$
$$
\beta (\delta) \triangleq \log (1 6 C _ {N} H / \delta) + d _ {c} \log_ {2} (8 T / d _ {\max}),
$$
and following line-by-line the proof of Lemma 6, for an event  $\mathcal{G}(\delta) = \mathcal{E}^{\star}(\delta)\cap \mathcal{E}^{B}(\delta ,T)\cap \mathcal{E}^{\mathrm{conc}}(\delta ,T)\cap$ $\mathcal{E}(\delta)$  we have  $\mathbb{P}(\mathcal{G}(\delta))\geq 1 - \delta /2$
Optimism Next, we state the required analog of Proposition 4. We can show that with probability at least  $1 - \delta /2$  on the event  $\mathcal{E}^{\star}(\delta)$  the following event
$$
\mathcal {E} _ {\text {a n t i c o n c}} \triangleq \left\{\forall t \in [ T ] \forall h \in [ H ] \forall B \in \mathcal {T} _ {D}: \text {f o r} k = k _ {h} ^ {t} (B), (s, a) = \operatorname {c e n t e r} (B): \right.
$$
$$
\max  _ {j \in [ J ]} \left\{W _ {j, e _ {k}, k} ^ {0} r _ {0} (H - 1) + \sum_ {i = 1} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} \left(F _ {h} (s, a, \xi_ {h} ^ {\ell^ {i}})\right) \right\} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) + L \cdot \operatorname {d i a m} \left(B _ {h} ^ {t}\right) \Bigg \}
$$
under the choice  $J = \lceil \tilde{c}_J\cdot (\log (2HT / \delta) + \log (|\mathcal{T}_D|))\rceil ,\kappa = 2\beta^{\star}(\delta ,T),r_0 = 2$  , and a prior count
$$
n _ {0} (k) = \lceil \widetilde {n} _ {0} + \kappa + \frac {L \cdot d _ {\operatorname* {m a x}}}{H - 1} \cdot \frac {e _ {k} + \widetilde {n} _ {0} + \kappa}{\sqrt {H e _ {k} - k - H ^ {2}}} \rceil
$$
dependent on the stage  $k$ , where  $\widetilde{n}_0 = (c_0 + 1 + \log_{17/16}(T)) \cdot \kappa$ ,  $L = L_r + L_V(1 + L_F)$ . In particular, the proof exactly the same as the proof of Proposition 4 for  $\varepsilon$  dependent on  $k$ .
At the same time, it is possible to show that  $\mathcal{E}_{\mathrm{anticonc}}$  implies
$$
\mathcal {E} _ {\text {o p t}} \triangleq \left\{\forall t \in [ T ], h \in [ H ], \forall B \in \mathcal {P} _ {h} ^ {t}, \forall (s, a) \in B: \overline {{Q}} _ {h} ^ {t} (B) \geq Q _ {h} ^ {\star} (s, a) \right\}. \tag {17}
$$
Indeed, in the proof of Proposition 5 we actively uses the bound  $\rho((s_h^{\ell^i}, a_h^{\ell^i}), (s, a)) \leq \varepsilon$ . In the adaptive setting, we have to, at first, use an upper bound  $\rho((s_h^{\ell^i}, a_h^{\ell^i}), (s, a)) \leq \mathrm{diam}(B_h^{\ell^i})$  by a construction  $B \subseteq B_h^{\ell^i}$ , and then apply Lemma B.2 by Sinclair et al. [2023] by defining an upper bound
$$
\operatorname {d i a m} \left(B _ {h} ^ {\ell^ {i}}\right) \leq \frac {d _ {\max}}{\sqrt {n _ {h} ^ {\ell^ {i}} \left(B _ {h} ^ {\ell^ {i}}\right)}} \leq \frac {d _ {\max}}{\sqrt {\sum_ {i = 0} ^ {k - 1} e _ {i}}} \leq \frac {d _ {\max}}{\sqrt {H \sum_ {i = 0} ^ {k - 1} (1 + 1 / H) ^ {i} - k}} \leq \frac {d _ {\max}}{\sqrt {H e _ {k} - k - H ^ {2}}}
$$
for  $k = k_h^t(B)$  for a particular ball  $B \in \mathcal{P}_h^t$  in the case  $H e_k - k - H^2 \geq 0$ .
By combining event  $\mathcal{E}_{\mathrm{opt}}$  and the event  $\mathcal{E}^B (\delta)$  we can prove the same statement as Corollary 2.
Let  $t \in [T], h \in [H], B \in \mathcal{P}_h^t$ . Define  $k = k_h^t(B)$  and let  $\ell^1 < \ldots < \ell^{e_k}$  be a excursion of  $(B, h)$  till the end of the previous stage. Then on the event  $\mathcal{G}'(\delta) = \mathcal{G}(\delta) \cap \mathcal{E}_{\mathrm{opt}}$  the following bound holds for  $k \geq 0$  and for any  $(s, a) \in B$
$$
0 \leq \overline {{Q}} _ {h} ^ {t} (B) - Q _ {h} ^ {\star} (s, a) \leq H \mathbb {1} \{H e _ {k} / 2 \leq k + H ^ {2} \} + \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \left[ V _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right) - V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell^ {i}}\right) \right] + \mathcal {B} _ {h} ^ {t}, \tag {18}
$$
$$
\mathcal {B} _ {h} ^ {t} = 1 2 1 \mathrm {e} ^ {2} \cdot \sqrt {\frac {H ^ {2} \left(\beta^ {\max } (\delta , T)\right) ^ {2}}{e _ {k}}} + 2 4 0 1 \mathrm {e} \cdot \frac {H \left(\beta^ {\max } (\delta , T)\right) ^ {4}}{e _ {k}} + \frac {5 L \cdot d _ {\max }}{\sqrt {H e _ {k}}} \tag {19}
$$
where  $k = k_h^t (B_h^t)$  and  $\beta^{\mathrm{max}}(\delta ,T) = \max \{\beta^{\star}(\delta ,T),\beta^B (\delta),\beta^{\mathrm{conc}}(\delta),\beta (\delta)\}$ . Also we can express this bound in terms of a diameter of  $B_{h}^{t}$  as follows
$$
\begin{array}{l} \mathrm {d i a m} (B _ {h} ^ {t}) \geq \frac {d _ {\max}}{2 \sqrt {n _ {h} ^ {t} (B _ {h} ^ {t})}} \geq \frac {d _ {\max}}{2 \sqrt {\sum_ {i = 0} ^ {k} e _ {i}}} \geq \frac {d _ {\max}}{2 \sqrt {H \sum_ {i = 0} ^ {k} (1 + 1 / H) ^ {i}}} \geq \frac {d _ {\max}}{2 \sqrt {H}} \\ \geq \frac {d _ {\mathrm {m a x}}}{2 \sqrt {H ^ {2} (1 + 1 / H) ^ {k + 1}}} \geq \frac {d _ {\mathrm {m a x}}}{2 \sqrt {2 H e _ {k}}}, \\ \end{array}
$$
$$
\frac {1}{\sqrt {H e _ {k}}} \leq \frac {3 \mathrm {d i a m} (B _ {h} ^ {t})}{d _ {\max}},
$$
$$
\begin{array}{l} \mathcal {B} _ {h} ^ {t} \leq 7 5 6 6 \mathrm {e} ^ {2} H ^ {3 / 2} \left(\beta^ {\max } (\delta , T)\right) ^ {4} \operatorname {d i a m} \left(B _ {h} ^ {t}\right) / d _ {\max } + 1 5 L \operatorname {d i a m} \left(B _ {h} ^ {t}\right) \tag {20} \\ \leq \rho (H, \delta , L) \cdot \operatorname {d i a m} \left(B _ {h} ^ {t}\right), \\ \end{array}
$$
where we define  $\rho (H,\delta ,L)\triangleq 7566\mathrm{e}^2 H^{3 / 2}(\beta^{\max}(\delta ,T))^4 /d_{\max} + 15L.$
As a additional corollary, we have for all  $t \in [T], h \in [H]$
$$
\bar {V} _ {h} ^ {t} (s) = \max  _ {B \in \mathcal {R} _ {h} ^ {t} (s)} \bar {Q} _ {h} ^ {t} (B) = \bar {Q} _ {h} ^ {t} \left(B ^ {\star}\right) \geq Q _ {h} ^ {\star} \left(s, \pi^ {\star} (s)\right) = V _ {h} ^ {\star} (s), \tag {21}
$$
where  $B^{\star}$  is a ball that contains a pair  $(s,\pi^{\star}(s))$
This upper and lower bound have the similar structure as Lemma D.2 by Sinclair et al. [2023] and the rest of the proof directly follows [Sinclair et al., 2023].
**Clipping techniques** Next we introduce the required clipping techniques developed by Simchowitz and Jamieson [2019], Cao and Krishnamurthy [2020]. Definition 2 introduces the quantity  $\mathrm{gap}_h(s,a) = V_h^\star (s) - Q_h^\star (s,a)$ , and for any compact set  $B\subseteq S\times \mathcal{A}$  we define  $\mathrm{gap}_h(B) = \min_{(s,a)\in B}\mathrm{gap}_h(s,a)$ . Finally, we define clipping operator for any  $\mu ,\nu \in \mathbb{R}$
$$
\operatorname {c l i p} (\mu | \nu) = \mu \mathbb {1} \left\{\mu \leq \nu \right\}. \tag {22}
$$
In particular, this operator satisfies the following important property
Lemma 8 (Lemma E.2. of Sinclair et al. [2023]). Suppose that  $\mathrm{gap}_h(B) \leq \psi \leq \mu_1 + \mu_2$  for any  $\psi, \mu_1, \mu_2$ . Then
$$
\psi \leq \operatorname {c l i p} \left[ \mu_ {1} \left| \frac {\operatorname {g a p} _ {h} (B)}{H + 1} \right] + \left(1 + \frac {1}{H}\right) \mu_ {2} \right.
$$
Now we apply this lemma to our update rules, producing a result similar to Lemma E.3 of Sinclair et al. [2023]. We notice that
$$
\begin{array}{l} \operatorname {g a p} _ {h} (B _ {h} ^ {t}) \leq \operatorname {g a p} _ {h} (s _ {h} ^ {t}, a _ {h} ^ {t}) = V _ {h} ^ {\star} (s _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ \leq \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) = \overline {{Q}} _ {h} ^ {t} (B _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}). \\ \end{array}
$$
Thus, denoting  $\psi = \overline{Q}_h^t (B_h^t) - Q_h^\star (s_h^t,a_h^t)$  and, by (18),
$$
\mu_ {1} = H \mathbb {1} \{H e _ {k _ {h} ^ {t}} / 2 > k _ {h} ^ {t} + H ^ {2} \} + \mathcal {B} _ {h} ^ {t}, \quad \mu_ {2} = \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} [ \overline {{V}} _ {h + 1} ^ {\ell^ {i}} (s _ {h + 1} ^ {\ell^ {i}}) - V _ {h + 1} ^ {\star} (s _ {h + 1} ^ {\ell^ {i}}) ]
$$
we apply Lemma 8 and obtain
$$
\begin{array}{l} \overline {{V}} _ {h} ^ {t} \left(s _ {h} ^ {t}\right) - Q _ {h} ^ {\star} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) \leq \operatorname {c l i p} \left[ H 1 \left\{\operatorname {H e} _ {k _ {h} ^ {t}} / 2 \leq k _ {h} ^ {t} + H ^ {2} \right\} + \mathcal {B} _ {h} ^ {t} \mid \frac {\operatorname {g a p} _ {h} \left(B _ {h} ^ {t}\right)}{H + 1} \right] \\ + \left(1 + \frac {1}{H}\right) \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \left[ \bar {V} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right) - V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell^ {i}}\right) \right] \tag {23} \\ \end{array}
$$
for  $k_h^t = k_h^t (B_h^t)$  and  $\mathcal{B}_h^t$  defined in (19).
Regret decomposition The rest of the analysis we preform conditionally on event  $\mathcal{G}'(\delta) = \mathcal{G}(\delta) \cap \mathcal{E}_{\mathrm{opt}}$  that holds with probability at least  $1 - \delta$ .
By defining  $\delta_h^t = \overline{V}_h^t (s_h^t) - V^{\pi^t}(s_h^t)$  and  $\zeta_h^t = \overline{V}_h^t (s_h^t) - V_h^\star (s_h^t)$  we have
$$
\mathfrak {R} ^ {T} = \sum_ {t = 1} ^ {T} V _ {1} ^ {\star} \left(s _ {1} ^ {t}\right) - V _ {1} ^ {\pi^ {t}} \left(s _ {1} ^ {t}\right) \leq \sum_ {t = 1} ^ {T} \delta_ {1} ^ {t},
$$
and, at the same time, by Bellman equations
$$
\begin{array}{l} \delta_ {h} ^ {t} = \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - Q ^ {\pi_ {h} ^ {t}} (s _ {h} ^ {t}, a _ {h} ^ {t}) = \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) + Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) - Q ^ {\pi^ {t}} (s _ {h} ^ {t}, a _ {h} ^ {t}) \\ = \bar {V} _ {h} ^ {t} \left(s _ {h} ^ {t}\right) - Q _ {h} ^ {\star} \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) + V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {t}\right) - V _ {h} ^ {\pi^ {t}} \left(s _ {h + 1} ^ {t}\right) + \xi_ {h} ^ {t} \\ = \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) + \delta_ {h + 1} ^ {t} - \zeta_ {h + 1} ^ {t} + \xi_ {h} ^ {t}, \\ \end{array}
$$