SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By rolling out the upper bound on regret (8) and using inequality  $(1 + 1 / H)^{H - h} \leq \mathrm{e}$  we have
$$
\mathfrak {R} ^ {T} \leq \mathrm {e} H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {1} \left\{N _ {h} ^ {t} = 0 \right\} + \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \xi_ {h} ^ {t} + \mathrm {e} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t}.
$$
Proof of Theorem 1. First, we notice that the event  $\mathcal{G}'(\delta)$  defined in Lemma 5, holds with probability at least  $1 - \delta$  by Lemma 4 and Proposition 3. Thus, we may assume that  $\mathcal{G}'(\delta)$  holds.
We start from the decomposition given by Lemma 5
$$
\mathfrak {R} ^ {T} \leq \mathrm {e} H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {1} \{k _ {h} ^ {t} (s _ {h} ^ {t}, a _ {h} ^ {t}) = - 1 \} + \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \xi_ {h} ^ {t} + \mathrm {e} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t}.
$$
The first term is upper bounded by  $\mathrm{e}SAH^3$ , since there is no more than  $H$  visits of each state-action-step triple before the update for the first stage. The second term is bounded by  $\widetilde{\mathcal{O}}(\sqrt{H^3T})$  by a definition of the event  $\mathcal{E}(\delta)$  in Lemma 4. To upper bound the last term we have to analyze the following sum
$$
\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \{e _ {k _ {h} ^ {t} (s _ {h} ^ {t} , a _ {h} ^ {t})} > 0 \}}{\sqrt {e _ {k _ {h} ^ {t} (s _ {h} ^ {t} , a _ {h} ^ {t})}}} \leq \sum_ {(s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]} \sum_ {k = 0} ^ {k _ {h} ^ {T + 1} (s, a)} \frac {e _ {k + 1}}{\sqrt {e _ {k}}},
$$
$$
e _ {k} = \left\lfloor \left(1 + \frac {1}{H}\right) ^ {k} H \right\rfloor \Rightarrow \frac {e _ {k + 1}}{\sqrt {e _ {k}}} \leq 2 \sqrt {e _ {k}},
$$
therefore by Cauchy inequality
$$
\sum_ {k = 0} ^ {k _ {h} ^ {T + 1} (s, a)} \frac {e _ {k + 1}}{\sqrt {e _ {k}}} \leq 2 \sum_ {k = 0} ^ {k _ {h} ^ {T + 1} (s, a)} \sqrt {e _ {k}} \leq 2 \sqrt {k _ {h} ^ {T + 1} (s , a)} \sqrt {\sum_ {k = 0} ^ {k _ {h} ^ {T + 1} (s , a)} e _ {k}} \leq 2 \sqrt {\frac {\log (T)}{\log (1 + 1 / H)}} \sqrt {n _ {h} ^ {T + 1} (s , a)},
$$
where we used the definition of the previous stage  $k_h^{T + 1}(s,a)$
$$
N _ {h} ^ {T + 1} (s, a) \geq \sum_ {k = 0} ^ {k _ {h} ^ {T + 1} (s, a)} e ^ {k},
$$
thus by Cauchy inequality and inequality  $\log (1 + 1 / H)\geq 1 / (4H)$  for  $H\geq 1$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \left\{e _ {k _ {h} ^ {t}} \left(s _ {h} ^ {t} , a _ {h} ^ {t}\right) > 0 \right\}}{\sqrt {e _ {k _ {h} ^ {t}} \left(s _ {h} ^ {t} , a _ {h} ^ {t}\right)}} \leq 2 \sqrt {H \log (T)} \sum_ {(s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ]} \sqrt {N _ {h} ^ {T + 1} (s , a) + 1} \\ \leq 4 \sqrt {S A H ^ {2} \log (T)} \sqrt {\sum_ {(s , a , h)} \left(N _ {h} ^ {T + 1} (s , a) + 1\right)} \\ \leq 4 \sqrt {S A H ^ {3} T \log (T)} + 4 S A H ^ {2} \log (T). \\ \end{array}
$$
Using this upper bound, we have
$$
\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t} = \widetilde {\mathcal {O}} \left(H \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \frac {\mathbb {1} \left\{e _ {k _ {h} ^ {t} \left(s _ {h} ^ {t} , a _ {h} ^ {t}\right)} > 0 \right\}}{\sqrt {e _ {k _ {h} ^ {t} \left(s _ {h} ^ {t} , a _ {h} ^ {t}\right)}}}\right) = \widetilde {\mathcal {O}} \left(\sqrt {H ^ {5} S A T} + S A H ^ {3}\right).
$$
Combining this upper bound with the previous ones, we conclude the statement.
E Proofs for Metric algorithm
E.1 Assumptions
In this section we proof Lemma 2 and Lemma 1.
Proof of Lemma 1. By the dual formula for 1-Wasserstein distance (see e.g. Section 6 of Peyre and Cuturi [2019]) we have
$$
\mathcal {W} _ {1} \left(p _ {h} (s, a), p _ {h} \left(s ^ {\prime}, a ^ {\prime}\right)\right) = \sup  _ {f \text {i s} 1 - \text {L i p c h i t z}} \left\{p _ {h} f (s, a) - p _ {h} f \left(s ^ {\prime}, a ^ {\prime}\right) \right\}.
$$
By Assumption 2 we have
$$
p _ {h} f (s, a) - p _ {h} f (s ^ {\prime}, a ^ {\prime}) = \mathbb {E} _ {\xi_ {h}} [ f (F _ {h} (s, a, \xi_ {h})) - f (F _ {h} (s ^ {\prime}, a ^ {\prime}, \xi_ {h})) ] \leq L _ {F} \rho ((s, a), (s ^ {\prime}, a ^ {\prime})).
$$
Proof of Lemma 2. Let us proceed by a backward induction over  $h$ . For  $h = H + 1$  we have  $Q_{H + 1}^{\star}(s, a) = V_{H + 1}^{\star}(s) = 0$ , therefore they are 0-Lipschitz.
Next we assume that for any  $h' > h$  the statement of Lemma 2 holds. Then by Bellman equations
$$
| Q _ {h} ^ {\star} (s, a) - Q _ {h} ^ {\star} (s ^ {\prime}, a ^ {\prime}) | \leq | r _ {h} (s, a) + r _ {h} (s ^ {\prime}, a ^ {\prime}) | + | p _ {h} V _ {h + 1} ^ {\star} (s, a) - p _ {h} V _ {h + 1} ^ {\star} (s ^ {\prime}, a ^ {\prime}) |.
$$
By Assumption 2 we can represent the action of the transition kernel as follows
$$
p _ {h} V _ {h + 1} ^ {\star} (s, a) - p _ {h} V _ {h + 1} ^ {\star} (s ^ {\prime}, a ^ {\prime}) = \mathbb {E} _ {\xi_ {h}} \left[ V _ {h + 1} ^ {\star} (F _ {h} (s, a, \xi_ {h})) - V _ {h + 1} ^ {\star} (F _ {h} (s ^ {\prime}, a ^ {\prime}, \xi_ {h}) \right].
$$
Since by induction hypothesis  $V_{h + 1}^{\star}$  is  $\sum_{h' = h + 1}^{H}L_{F}^{h' - h}L_{r}$ -Lipschitz and  $F_{h}(\cdot ,\xi_{h})$  is  $L_{F}$ -Lipschitz, therefore
$$
\begin{array}{l} \left| Q _ {h} ^ {\star} (s, a) - Q _ {h} ^ {\star} \left(s ^ {\prime}, a ^ {\prime}\right) \right| \leq \left(L _ {r} + L _ {F} \cdot \sum_ {h ^ {\prime} = h + 1} ^ {H} L _ {F} ^ {h ^ {\prime} - h} L _ {r}\right) \rho \left(\left(s, a\right), \left(s ^ {\prime}, a ^ {\prime}\right)\right) \\ \leq \left(\sum_ {h ^ {\prime} = h} ^ {H} L _ {F} ^ {h ^ {\prime} - h} L _ {r}\right) \rho \left(\left(s, a\right), \left(s ^ {\prime}, a ^ {\prime}\right)\right) \\ \end{array}
$$
To show that  $V_h^\star$  is also Lipschitz, we have that there is some action  $a^\star$  equal to  $\pi^\star(s)$  or  $\pi^\star(s')$ , such that
$$
\left| V _ {h} ^ {\star} (s) - V _ {h} ^ {\star} \left(s ^ {\prime}\right) \right| \leq \left| Q _ {h} ^ {\star} \left(s, a ^ {\star}\right) - Q _ {h} ^ {\star} \left(s ^ {\prime}, a ^ {\star}\right) \right| \leq L _ {V, h} \cdot \rho \left(\left(s, a ^ {\star}\right), \left(s ^ {\prime}, a ^ {\star}\right)\right) \leq L _ {V, h} \cdot \rho_ {\mathcal {S}} \left(s, s ^ {\prime}\right),
$$
where in the end we used the sub-additivity assumption on metric over joint space (see Assumption 1).
Next we describe a simple non-adaptive version of our algorithm that works with metric spaces. We assume that for any  $\varepsilon > 0$  we can compute a minimal  $\varepsilon$ -cover of state-action space  $\mathcal{N}_{\varepsilon}$ .<sup>5</sup>
Next we will use the same notation but with state-action pairs replaces with balls from a fixed cover  $\mathcal{N}_{\varepsilon}$ . To unify the notation, we define  $\psi_{\varepsilon} \colon S \times \mathcal{A} \to \mathcal{N}_{\varepsilon}$  that maps any point  $(s, a)$  to any ball from  $\varepsilon$ -cover that contains it.
For any  $t, h$  we define  $B_h^t = \psi_\varepsilon(s_h^t, a_h^t)$ . Next, let  $n_h^t(B)$  be a number of visits of ball  $B$  before the episode  $t$ :  $n_h^t(B) = \sum_{k=1}^{t-1} \mathbb{1}\{B_h^k = B\}$ .
Let  $e_k = \left\lfloor (1 + 1 / H)^k \cdot H \right\rfloor$  be length of each stage for any  $k \geq 0$  and, by convention,  $e_{-1} = 0$ . We will call that in the beginning of episode  $t$  a pair  $(B, h)$  is in  $k$ -th stage if  $n_h^t(B) \in [\sum_{i=0}^{k-1} e_i, \sum_{i=0}^k e_i)$ .
Remark that the greedy algorithm can easily generate  $\varepsilon$ -cover of size  $N_{\varepsilon /2}$ , that will not affect the asymptotic behavior of regret bounds, see Song and Sun [2019].
Algorithm 4 Metric Net-Staged-RandQL
1: Input: inflation coefficient  $\kappa$ ,  $J$  ensemble size, number of prior transitions  $n_0(k)$ , prior reward  $r_0$ , discretization level  $\varepsilon$ .  
2: Initialize:  $\varepsilon$ -net  $\mathcal{N}_{\varepsilon}$ ,  $\overline{Q}_h(B) = \widetilde{Q}_h^j(B) = r_0H$ , initialize counters  $\widetilde{n}_h(B) = 0$  for  $j,h,B \in [J] \times [H] \times \mathcal{N}_{\varepsilon}$ , stage  $q_h(B) = 0$ , quantization map  $\psi_{\varepsilon} \colon S \times A \to \mathcal{N}_{\varepsilon}$ .  
3: for  $t \in [T]$  do  
4: for  $h \in [H]$  do  
5: Play  $a_h \in \arg \max_a \overline{Q}_h(\psi_{\varepsilon}(s_h,a))$  and define  $B_h = \psi_{\varepsilon}(s_h,a_h)$ .  
6: Observe reward and next state  $s_{h+1} \sim p_h(s_h,a_h)$ .  
7: Sample learning rates  $w_j \sim \text{Beta}(1/\kappa, (\widetilde{n} + n_0(q_h(B_h)) / \kappa)$  for  $\widetilde{n} = \widetilde{n}_h(B_h)$ .  
8: Compute value function  $\overline{V}_{h+1}(s_{h+1}) = \max_{a \in A} \overline{Q}_{h+1}(\psi_{\varepsilon}(s_{h+1}, a))$ .  
9: Update temporary  $Q$ -values for all  $j \in [J]$ $\widetilde{Q}_h^j(B) := (1 - w_j) \widetilde{Q}_h^j(B) + w_j(r_h(s_h,a_h) + \overline{V}_{h+1}(s_{h+1}))$ .  
10: Update counter  $\widetilde{n}_h(B_h) := \widetilde{n}_h(B_h) + 1$   
11: if  $\widetilde{n}_h(B_h) = [(1 + 1/H)^q H]$  for  $q = q_h(B_h)$  is the current stage then  
12: Update policy  $Q$ -values  $\overline{Q}_h(B_h) := \max_{j \in [J]} \widetilde{Q}_h^j(B_h)$ .  
13: Reset temporary  $Q$ -values  $\widetilde{Q}_h^j(B_h) := r_0H$ .  
14: Reset counter  $\widetilde{n}_h(B_h) := 0$  and change stage  $k_h(B_h) := k_h(B_h) + 1$ .  
15: end if  
16: end for  
17: end for
Let  $\widetilde{n}_h^t (B)$  be a number of visits of state-action pair during the current stage in the beginning of episode  $t$ . Formally,  $\widetilde{n}_h^t (B) = n_h^t (B) - \sum_{i = 0}^{k - 1}e_i$ , where  $k$  is an index of current stage.
Define  $\kappa > 0$  be a posterior inflation coefficient,  $n_0$  is a number of pseudo-transitions, and  $J$  as a number of temporary  $Q$ -functions. Let  $\widetilde{Q}_h^{t,j}$  be a  $j$ -th temporary Q-value and  $\overline{Q}_h^t$  be a policy Q-value at the beginning of episode  $t$ , defined over the  $\varepsilon$ -cover. We initialize them as follows
$$
\overline {{Q}} _ {h} ^ {1} (B) = r _ {0} H, \quad \widetilde {Q} _ {h} ^ {1, j} (s, a) = r _ {0} H.
$$
Additionally, we define to the value function as follows
$$
\overline {{V}} _ {h} ^ {t} (s) = \max  _ {a \in \mathcal {A}} \overline {{Q}} _ {h} ^ {t} (\psi_ {\varepsilon} (s, a)).
$$
Notice that we cannot precomute it as in the tabular setting, however, it is possible to use its values in lazy fashion.
For each transition we preform the following update of temporary Q-values over balls  $B \in \mathcal{N}_{\varepsilon}$
$$
\widetilde {Q} _ {h} ^ {t + 1 / 2, j} (B) = \left\{ \begin{array}{l l} (1 - w _ {j, \widetilde {n}}) \cdot \widetilde {Q} _ {h} ^ {t, j} (B) + w _ {j, \widetilde {n}} [ r _ {h} (s _ {h} ^ {t}, a _ {h} ^ {t}) + \overline {{V}} _ {h + 1} ^ {t} (s _ {h + 1} ^ {t}) ], & B = B _ {h} ^ {t} \\ \widetilde {Q} _ {h} ^ {t, j} (B) & \text {o t h e r w i s e}, \end{array} \right.
$$
where  $\widetilde{n} = \widetilde{n}_h^t (B)$  is the number of visits of  $(B,h)$  in the beginning of episode  $t$ , and  $w_{j,\tilde{n}}$  is a sequence of independent beta-distribution random variables  $w_{j,\tilde{n}} \sim \mathrm{Beta}(1 / \kappa, (\widetilde{n} + n_0) / \kappa)$ .
$$
\overline {{Q}} _ {h} ^ {t + 1} (B) = \left\{ \begin{array}{l l} \max  _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {t + 1 / 2, j} (B) & \widetilde {n} _ {h} ^ {t} (B) = \lfloor (1 + 1 / H) ^ {k} H \rfloor \\ \overline {{Q}} _ {h} ^ {t} (B) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\widetilde {Q} _ {h} ^ {t + 1, j} (B) = \left\{ \begin{array}{l l} r _ {0} H & n _ {h} ^ {t} (B) \in \widetilde {n} _ {h} ^ {t} (B) = \lfloor (1 + 1 / H) ^ {k} H \rfloor \\ \widetilde {Q} _ {h} ^ {t + 1 / 2, j} (B) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
\overline {{V}} _ {h} ^ {t + 1} (s) = \min  \{r _ {0} (H - h), \max  _ {a \in \mathcal {A}} \overline {{Q}} _ {h} ^ {t + 1} (\psi_ {\varepsilon} (s, a)) \};
$$
$$
\pi_{h}^{t + 1}(s)\in \operatorname *{arg  max}_{a\in \mathcal{A}}\overline{Q}_{h}^{t + 1}(\psi_{\varepsilon}(s,a)),
$$
where  $k$  is the current stage. A detailed description of the algorithm is presented in Algorithm 4.
For episode  $t$  we will call  $k_h^t (B)$  the index of stage where  $\overline{Q}_h^t (B)$  were updated (and  $k_h^t (B) = -1$  if there was no update). For all  $t$  we define  $\tau_h^t (B) \leq t$  as the episode when the stage update happens. In other words, for any  $t$  the following holds
$$
\overline {{Q}} _ {h} ^ {t + 1} (B) = \max _ {j \in [ J ]} \widetilde {Q} _ {h} ^ {\tau_ {h} ^ {t} (B) + 1 / 2, j} (B),
$$
where  $\tau_h^t (B) = 0$  and  $e_k = 0$  if there was no updates. To simplify the notation we will omit dependence on  $(s,a,h)$  where it is deducible from the context.
We notice that in this case we use  $e_k$  samples to compute  $\widetilde{Q}^{\tau_h^t (B) + 1 / 2,j}$  for  $k = k_h^t (s,a)$ . For this  $k$  we define  $\ell_{k,h}^{i}(s,a)$  as the time of  $i$ -th visit of state-action pair  $(s,a)$  during  $k$ -th stage. Then we have the following decomposition
$$
\widetilde {Q} _ {h} ^ {\tau^ {t} + 1 / 2, j} (B) = \sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}} ^ {i} \left(r _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right) + \bar {V} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right)\right), \tag {9}
$$
where we drop dependence on  $k$  and  $(B,h)$  in  $\ell^i$  to simplify notations, using the convention  $r_h(s_h^{\ell^0},a_h^{\ell^0}) = r_0,\overline{V}_{h + 1}^{\ell^0}(s_{h + 1}^{\ell^0}) = r_0(H - 1)$  and the following aggregated weights
E.3 Concentration
Let  $\beta^{\star}\colon (0,1)\times \mathbb{N}\times (0,d_{\mathrm{max}})\to \mathbb{R}_{+}$  and  $\beta^{B},\beta^{\mathrm{conc}},\beta \colon (0,1)\times (0,d_{\mathrm{max}})\to \mathbb{R}_{+}$  be some function defined later on in Lemma 6. We define the following favorable events
$$
\mathcal {E} ^ {\star} (\delta , \varepsilon) \triangleq \left\{\forall t \in \mathbb {N}, \forall h \in [ H ], \forall B \in \mathcal {N} _ {\varepsilon}, k = k _ {h} ^ {t} (B), (s, a) = \operatorname {c e n t e r} (B): \right.
$$
$$
\left. \mathcal {K} _ {\inf } \left(\frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} \delta_ {V _ {h + 1} ^ {\star} \left(F _ {h} (s, a, \xi_ {h + 1} ^ {\ell^ {i}})\right)}, p _ {h} V _ {h + 1} ^ {\star} (s, a)\right) \leq \frac {\beta^ {\star} (\delta , e _ {k} , \varepsilon)}{e _ {k}} \right\},
$$
$$
\begin{array}{l} \mathcal {E} ^ {B} (\delta , \varepsilon) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall B \in \mathcal {N} _ {\varepsilon}, \forall j \in [ J ], k = k _ {h} ^ {t} (B): \right. \\ \left| \sum_ {i = 0} ^ {e _ {k}} \left(W _ {j, e _ {k}, k} ^ {i} - \mathbb {E} \left[ W _ {j, e _ {k}, k} ^ {i} \right]\right) \left(r _ {h} \left(s _ {h} ^ {\ell^ {i}}, a _ {h} ^ {\ell^ {i}}\right) + \overline {{V}} _ {h + 1} ^ {\ell^ {i}} \left(s _ {h + 1} ^ {\ell^ {i}}\right)\right) \right| \\ \leq 6 0 \mathrm {e} ^ {2} \sqrt {\frac {r _ {0} ^ {2} H ^ {2} \kappa \beta^ {B} (\delta , \varepsilon)}{e _ {k} + n _ {0} (k)}} + 1 2 0 0 \mathrm {e} \frac {r _ {0} H \kappa \log (e _ {k} + n _ {0} (k)) (\beta^ {B} (\delta , \varepsilon)) ^ {2}}{e _ {k} + n _ {0} (k)} \Bigg \}, \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {E} ^ {\text {c o n c}} (\delta , \varepsilon) \triangleq \left\{\forall t \in [ T ], \forall h \in [ H ], \forall B \in \mathcal {N} _ {\varepsilon}, k = k _ {h} ^ {t} (B): \right. \\ \left| \frac {1}{e _ {k}} \sum_ {i = 1} ^ {e _ {k}} V _ {h + 1} ^ {\star} \left(s _ {h + 1} ^ {\ell_ {k, h} ^ {i} (B)}\right) - p _ {h} V _ {h + 1} ^ {\star} \left(s _ {h} ^ {\ell_ {k, h} ^ {i} (B)}, a _ {h} ^ {\ell_ {k, h} ^ {i} (B)}\right) \right| \leq \sqrt {\frac {2 r _ {0} ^ {2} H ^ {2} \beta^ {\operatorname {c o n c}} (\delta , \varepsilon)}{e _ {k}}} \Bigg \} \\ \mathcal {E} (\delta) \triangleq \left\{\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 1 / H) ^ {H - h} \left| p _ {h} \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h} ^ {t}, a _ {h} ^ {t}\right) - \left[ V _ {h + 1} ^ {\star} - V _ {h + 1} ^ {\pi_ {t}} \right] \left(s _ {h + 1} ^ {t}\right) \right| \right. \\ \left. \leq 2 \mathrm {e r} _ {0} H \sqrt {2 H T \beta (\delta)}. \right\}. \\ \end{array}
$$
Lemma 6. For any  $\delta \in (0,1)$  and  $\varepsilon \in (0,d_{\mathrm{max}})$  and for the following choices of functions  $\beta$
$$
\beta^ {\star} (\delta , n, \varepsilon) \triangleq \log (8 H / \delta) + \log (N _ {\varepsilon}) + 3 \log (\mathrm {e} \pi (2 n + 1)),
$$
$$
\beta^ {B} (\delta , \varepsilon) \triangleq \log (8 H / \delta) + \log (N _ {\varepsilon}) + \log (T J),
$$
$$
\beta^ {\text {c o n c}} (\delta , \varepsilon) \triangleq \log (8 H / \delta) + \log \left(N _ {\varepsilon}\right) + \log (2 T),
$$
$$
\mathbb {P} \left[ \mathcal {E} ^ {\star} (\delta , \varepsilon) \right] \geq 1 - \delta / 8, \quad \mathbb {P} \left[ \mathcal {E} ^ {B} (\delta , \varepsilon) \right] \geq 1 - \delta / 8,
$$
$$
\mathbb {P} [ \mathcal {E} ^ {\mathrm {c o n c}} (\delta , \varepsilon) ] \geq 1 - \delta / 8, \quad \mathbb {P} [ \mathcal {E} (\delta) ] \geq 1 - \delta / 8.
$$
Proof. Let us describe the changes from the similar statement in Lemma 4.
Regarding event  $\mathcal{E}^{\star}(\delta, \varepsilon)$ , for any fixed ball  $B$  we have exactly the same structure of the problem thanks to Assumption 2 and a sequence of i.i.d. random variables  $\xi_h^{i}$ . Thus, Theorem 4 combined with a union bound over  $B \in \mathcal{N}_{\varepsilon}$  and  $H \in [H]$  concludes  $\mathbb{P}(\mathcal{E}^{\star}(\delta, \varepsilon)) \geq 1 - \delta / 8$ .
The proof for the event  $\mathcal{E}^B (\delta ,\varepsilon)$  remains the almost the same, with two differences: the predictable weights slightly changed but the upper bound for them remain the same, and we have taken a union bound not over all state-action pairs  $(s,a)\in S\times \mathcal{A}$  but all over balls  $B\in \mathcal{N}_{\varepsilon}$ .
To show that  $\mathbb{P}(\mathcal{E}^{\mathrm{conc}}(\delta, \varepsilon)) \geq 1 - \delta/8$ , let us fix  $B \in \mathcal{N}_{\varepsilon}, h \in [H]$  and  $e_k \in [T]$ . Then we can define a filtration  $\mathcal{F}_{t,h} = \sigma\left\{\{(s_{h'}^\ell, a_{h'}^\ell, \pi^\ell), \ell < t, h' \in [H]\} \cup \{(s_{h'}^t, a_{h'}^t, \pi^t), h' \leq h\}\right\}$  and, since  $\ell_{k,h}^{i}(B)$  are stopping times for all  $i = 1, \ldots, e_k$ , we can define the stopped filtration  $\widetilde{\mathcal{F}_i} = \mathcal{F}_{\ell^i,h}$ . Then we notice that  $X_i = V_{h+1}^{\star}(s_{h+1}^{\ell_{k,h}^i(B)}) - p_h V_{h+1}^{\star}(s_{h}^{\ell_{k,h}^i(B)}, a_{h}^{\ell_{k,h}^i(B)})$  forms a martingale-difference sequence with respect to  $\widetilde{\mathcal{F}_{i,h}}$ . Thus, by Azuma-Hoeffding inequality and a union bound we have  $\mathbb{P}(\mathcal{E}^{\mathrm{conc}}(\delta, \varepsilon)) \geq 1 - \delta/8$ .
The proof of  $\mathbb{P}(\mathcal{E}(\delta))\geq 1 - \delta /8$  remains exactly the same as in Lemma 4.
E.4 Optimism
In this section we prove that our estimate of  $Q$ -function  $\overline{Q}_h^t (s,a)$  is optimistic that is the event
$$
\mathcal {E} _ {\mathrm {o p t}} (\varepsilon) \triangleq \left\{\forall t \in [ T ], h \in [ H ], (s, a) \in \mathcal {S} \times \mathcal {A}: \overline {{Q}} _ {h} ^ {t} \left(\psi_ {\varepsilon} (s, a)\right) \geq Q _ {h} ^ {\star} (s, a) \right\}. \tag {10}
$$
holds with high probability on the event  $\mathcal{E}^{\star}(\delta, \varepsilon)$ .
$$
c _ {0} \triangleq \frac {8}{\pi} \left(\frac {4}{\sqrt {\log (1 7 / 1 6)}} + 8 + \frac {4 9 \cdot 4 \sqrt {6}}{9}\right) ^ {2} + 1. \tag {11}
$$
and slightly another constant
$$
\tilde {c} _ {J} \triangleq \frac {1}{\log \left(\frac {4}{3 + \Phi (1)}\right)}, \tag {12}
$$
Proposition 4. Define a constant  $L = L_{r} + L_{V}(1 + L_{F})$ . Assume that  $J = \lceil \widetilde{c}_{J} \cdot (\log(2HT / \delta) + \log(N_{\varepsilon}) \rceil, \kappa = 2\beta^{\star}(\delta, T, \varepsilon), r_{0} = 2$ , and a prior count  $n_0(k) = \lceil \widetilde{n}_0 + \kappa + \frac{\varepsilon L}{H - 1} \cdot (e_k + \widetilde{n}_0 + \kappa) \rceil$  dependent on the stage  $k$ , where  $\widetilde{n}_0 = (c_0 + 1 + \log_{17/16}(T)) \cdot \kappa$ .
Then on event  $\mathcal{E}^{\star}(\delta, \varepsilon)$  the following event
$$
\begin{array}{l} \mathcal {E} _ {\text {a n t i c o n c}} \triangleq \left\{\forall t \in [ T ] \forall h \in [ H ] \forall B \in \mathcal {N} _ {\varepsilon}: \text {f o r} k = k _ {h} ^ {t} (B), (s, a) = \operatorname {c e n t e r} (B): \right. \\ \max _ {j \in [ J ]} \left\{W _ {j, e _ {k}, k} ^ {0} r _ {0} (H - 1) + \sum_ {i = 1} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} V _ {h + 1} ^ {\star} (F _ {h} (s, a, \xi_ {h} ^ {\ell^ {i}})) \right\} \geq p _ {h} V _ {h + 1} ^ {\star} (s, a) + L \varepsilon \Biggr \} \\ \end{array}
$$
holds with probability at least  $1 - \delta / 2$ .
Remark 1. We notice that the obtained result is connected to the theory of Dirichlet processes.
First, let us define the Dirichlet process, following Ferguson [1973]. The stochastic process  $G$ , indexed by elements  $B$  of  $\mathsf{X}$ , is a Dirichlet Process with parameter  $\nu$  ( $G \sim \mathrm{DP}(\nu)$ ) if
$$
G (B _ {1}), \dots , G (B _ {d}) \sim \operatorname {D i r} (\nu (B _ {1}), \dots , \nu (B _ {d})),
$$
for any measurable partition  $(B_{1},\ldots ,B_{d})$  of  $\mathsf{X}$
Let  $\widehat{P}_n = \frac{1}{n}\sum_{i=1}^n\delta_{Z_i}$  be an empirical measure of an i.i.d. sample  $Z_1, \ldots, Z_n \sim P$ . Let  $\nu$  be a finite (not necessarily probability) measure on  $\mathsf{X}$  and  $\widetilde{P}_n \sim \mathrm{DP}(\nu + n\widehat{P}_n)$ . Then we have the following representation for the expectations of a function  $f: \mathsf{X} \to \mathbb{R}$  over a sampled measure  $\widetilde{P}_n$  (see Theorem 14.37 of Ghosal and Van der Vaart [2017] with  $\sigma = 0$  for a proof)
$$
\widetilde {P} _ {n} f = V _ {n} \cdot Q f + (1 - V _ {n}) \sum_ {i = 1} ^ {n} W _ {i} f (Z _ {i}),
$$
where  $V_{n}\sim \mathrm{Beta}(|\nu |,n)$ $Q\sim \mathrm{DP}(\nu)$ , and a vector  $(W_{1},\ldots ,W_{n})$  follows uniform Dirichlet distribution  $\operatorname {Dir}(1,\dots ,1)$ . If we take  $\nu = n_0\cdot \delta_{Z_0}$  for some  $Z_{0}\in \mathsf{X}$  such that  $f(Z_{0}) = r_{0}(H - 1)^{6}$  then by a stick-breaking process representation of the Dirichlet distribution we have
$$
\widetilde {P} _ {n} f = \widetilde {W} _ {0} r _ {0} (H - 1) + \sum_ {i = 1} ^ {n} \widetilde {W} _ {1} f (Z _ {i}), \quad (\widetilde {W} _ {0}, \dots , \widetilde {W} _ {1}) \sim \operatorname {D i r} (n _ {0}, 1, \dots , 1).
$$
By taking an appropriate  $X$  and  $f$  we have that Proposition 4 could be interpreted as a deriving a lower bound on the probability of  $\mathbb{P}[\widetilde{P}_n f \geq Pf + \varepsilon L \mid \{Z_i\}_{i=1}^n]$ .
Proof. First for all, let us fix  $t \in [T], h \in [H]$  and  $B \in \mathcal{N}_{\varepsilon}$  and, consequently,  $k = k_h^t(B)$ . Also, let fix  $j \in [J]$ . To simplify the notation in the sequel, define  $X_0 = r_0(H - 1)$  and  $X_i = V_{h + 1}^\star(F_h(s, a, \xi_h^{\ell^i}))$  for  $i > 0$ . Notice that  $X_i$  for  $i > 0$  is a sequence of i.i.d. random variables supported on  $[0, H - h - 1]$ .
By Lemma 3 we have  $(W_{j,e_k,k}^0,\ldots ,W_{j,e_k,k}^{e_k})\sim \mathrm{Dir}(n_0(k) / \kappa ,1 / \kappa ,\ldots ,1 / \kappa)$ . Then we use the aggregation property of Dirichlet distribution: there is a vector  $(\widetilde{W}_j^{-1},\dots ,\widetilde{W}_j^{e_k})\sim \mathrm{Dir}((n_0(k) - \widetilde{n}_0) / \kappa ,\widetilde{n}_0 / \kappa ,1 / \kappa ,\ldots ,1 / \kappa)$  such that
$$
\sum_ {i = 0} ^ {e _ {k}} W _ {j, e _ {k}, k} ^ {i} X _ {i} = \widetilde {W} _ {j} ^ {- 1} X _ {0} + \sum_ {i = 0} ^ {e _ {k}} \widetilde {W} _ {j} ^ {i} X _ {i}.
$$
Next we are going to represent the Dirichlet random vector  $\widetilde{W}$  by a stick breaking process (or, equivalently, represent via the generalized Dirichlet distribution)