SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\xi_h^t = p_h[V_{h + 1}^\star -V_{h + 1}^{\pi^t}](s_h^t,a_h^t) - [V_{h + 1}^\star -V_{h + 1}^{\pi^t}](s_{h + 1}^t)$  is a martingale-difference sequence. By (23) we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} = \sum_ {t = 1} ^ {T} \overline {{V}} _ {h} ^ {t} (s _ {h} ^ {t}) - Q _ {h} ^ {\star} (s _ {h} ^ {t}, a _ {h} ^ {t}) + \delta_ {h + 1} ^ {t} - \zeta_ {h + 1} ^ {t} + \xi_ {h} ^ {t} \\ \leq \left(1 + \frac {1}{H}\right) \sum_ {t = 1} ^ {T} \frac {1}{e _ {k _ {h} ^ {t}}} \sum_ {i = 1} ^ {e _ {k _ {h} ^ {t}}} \zeta_ {h + 1} ^ {\ell^ {i}} + \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} - \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} \\ + \sum_ {t = 1} ^ {T} \operatorname {c l i p} \left[ H \mathbb {1} \left\{H e _ {k _ {h} ^ {t}} / 2 > k _ {h} ^ {t} + H ^ {2} \right\} + \mathcal {B} _ {h} ^ {t} \left(k _ {h} ^ {t}\right) \mid \frac {\operatorname {g a p} _ {h} \left(B _ {h} ^ {t}\right)}{H + 1} \right] \\ \end{array}
$$
where  $k_h^t = k_h^t (B_h^t)$ . Repeating argument of Lemma 5 and Zhang et al. [2020]
$$
\left(1 + \frac {1}{H}\right) \sum_ {t = 1} ^ {T} \frac {1}{e _ {k _ {h} ^ {t}}} \sum_ {i = 1} ^ {e _ {k _ {h} ^ {t}} ^ {t}} \zeta_ {h + 1} ^ {\ell_ {k _ {h} ^ {t}} ^ {i}} \leq \left(1 + \frac {1}{H}\right) ^ {2} \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t} \leq \left(1 + \frac {3}{H}\right) \sum_ {t = 1} ^ {T} \zeta_ {h + 1} ^ {t}.
$$
Using an upper bound  $\zeta_h^t\leq \delta_h^t$  we have for any  $h\geq 1$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \delta_ {h} ^ {t} \leq \left(1 + \frac {3}{H}\right) \sum_ {t = 1} ^ {T} \delta_ {h + 1} ^ {t} + \sum_ {t = 1} ^ {T} \xi_ {h} ^ {t} \\ + \sum_ {t = 1} ^ {T} \operatorname {c l i p} \left[ H \mathbb {1} \left\{H e _ {k _ {h} ^ {t}} / 2 \leq k _ {h} ^ {t} + H ^ {2} \right\} + \mathcal {B} _ {h} ^ {t} \mid \frac {\operatorname {g a p} _ {h} \left(B _ {h} ^ {t}\right)}{H + 1} \right], \\ \end{array}
$$
and, rolling out starting with  $h = 1$  we have the following regret decomposition
$$
\begin{array}{l} \mathfrak {R} ^ {T} \leq \mathrm {e} ^ {3} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} H \mathbb {1} \left\{H e _ {k _ {h} ^ {t}} / 2 \leq k _ {h} ^ {t} + H ^ {2} \right\} \quad = (\mathbf {A}) \\ + \mathrm {e} ^ {3} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \operatorname {c l i p} \left[ \mathcal {B} _ {h} ^ {t} \mid \frac {\operatorname {g a p} _ {h} \left(B _ {h} ^ {t}\right)}{H + 1} \right] \quad = (\mathbf {B}) \\ + \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (1 + 3 / H) ^ {H - h} \xi_ {h} ^ {t}. \quad = (\mathbf {C}) \\ \end{array}
$$
Term (A) For this term we notice that for any fixed  $h$  the following event
$$
H e _ {k _ {h} ^ {t}} \leq 2 \left(k _ {h} ^ {t} + H ^ {2}\right) \Longleftrightarrow H \left\lfloor H \left(1 + 1 / H\right) ^ {k _ {h} ^ {t}} \right\rfloor \leq 2 \left(k _ {h} ^ {t} + H ^ {2}\right),
$$
that is guaranteed if
$$
\left(1 + 1 / H\right) ^ {k _ {h} ^ {t}} \leq 2 T + 3 \iff k _ {h} ^ {t} \log (1 + 1 / H) \leq \log (2 T / H ^ {2} + 3).
$$
Thus, indicator can be equal to 1 no more than  $H \log (2T + 3)$  times for any  $t \in [T]$ . As a result,
$$
(\mathbf {A}) \leq \mathrm {e} ^ {2} H ^ {3} \log (2 T + 3).
$$
Term (B) Let us rewrite this term using a definition of clipping operator and use the definition of near-optimal set (see Definition 3)
$$
(\mathbf {B}) = \mathrm {e} ^ {3} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t} \mathbb {1} \left\{\left(H + 1\right) \mathcal {B} _ {h} ^ {t} \geq \operatorname {g a p} _ {h} \left(B _ {h} ^ {t}\right) \right\} \leq \mathrm {e} ^ {3} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathcal {B} _ {h} ^ {t} \mathbb {1} \left\{\operatorname {c e n t e r} \left(B _ {h} ^ {t}\right) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \right\}.
$$
Next we consider the summation for a fixed  $h$ . Here we follow Theorem F.3 by Sinclair et al. [2023] and obtain
$$
\sum_ {t = 1} ^ {T} \mathcal {B} _ {h} ^ {t} \mathbb {1} \{\mathrm {c e n t e r} (B _ {h} ^ {t}) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \} = \sum_ {r} \sum_ {B: \mathrm {d i a m} (B) = r} \sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t} \mathbb {1} \{\mathrm {c e n t e r} (B) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \},
$$
where we applied an additional rescaling by a function  $\rho$  defined in (20).
Next we fix a constant  $r_0 > 0$  and break a summation into two parts:  $r \geq r_0$  and  $r \leq r_0$ .
1. Case  $r \leq r_0$ . In this situation we have can apply (20)
$$
\begin{array}{l} \sum_ {r \leq r _ {0}} \sum_ {B: \operatorname {d i a m} (B) = r} \sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t} \mathbb {1} \{\operatorname {c e n t e r} (B) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \} \\ = \mathcal {O} (T r _ {0} \rho (H, \delta , L)). \\ \end{array}
$$
2. Case  $r \geq r_0$ . In this situation we also apply (20) under the indicator function
$$
\begin{array}{l} \sum_ {r \geq r _ {0}} \sum_ {B: \operatorname {d i a m} (B) = r} \sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t} \mathbb {1} \left\{\operatorname {c e n t e r} (B) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \right\} \\ \leq \sum_ {r \geq r _ {0}} \sum_ {B: \operatorname {d i a m} (B) = r \cdot \rho (H, \delta , L)} \mathbb {1} \left\{\operatorname {c e n t e r} (B) \in Z _ {h} ^ {\rho (H, \delta , L) \cdot r} \right\} \sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t}. \\ \end{array}
$$
To upper bound the last sum we repeat the argument of (14) and apply (16), using the fact that  $\mathrm{diam}(B) = r\cdot \rho (H,\delta ,L)$
$$
\begin{array}{l} \sum_ {t: B _ {h} ^ {t} = B} \frac {1}{\sqrt {e _ {k}}} \leq \sum_ {k = 0} ^ {k _ {h} ^ {T} (B)} \frac {e _ {k + 1}}{\sqrt {e _ {k}}} \leq 4 H \sqrt {e ^ {k _ {h} ^ {T} (B) + 1}} \\ \leq 4 \sqrt {H (n _ {h} ^ {T + 1} (B) + 1)} \leq 4 \sqrt {2 H} \cdot \frac {d _ {\max}}{\mathrm {d i a m} (B)} = \frac {\sqrt {3 2 H} \cdot d _ {\max}}{r}. \\ \end{array}
$$
As a result, we have by (19)
$$
\sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t} \leq \frac {\sqrt {3 2 H} \cdot d _ {\mathrm {m a x}}}{r} \cdot \left(2 5 2 2 \mathrm {e} ^ {2} H (\beta^ {\mathrm {m a x}} (\delta , T)) ^ {4} + 5 L d _ {\mathrm {m a x}} / \sqrt {H}\right)
$$
$$
\begin{array}{l} \sum_ {r \geq r _ {0}} \sum_ {B: \operatorname {d i a m} (B) = r} \sum_ {t: B _ {h} ^ {t} = B} \mathcal {B} _ {h} ^ {t} \mathbb {1} \{\operatorname {c e n t e r} (B) \in Z _ {h} ^ {\mathcal {B} _ {h} ^ {t}} \} \\ = \mathcal {O} \left(\sum_ {r \geq r _ {0}} N _ {r} (Z _ {h} ^ {\rho (H, \delta , L) \cdot r}) \cdot \frac {H ^ {3 / 2} d _ {\max} (\beta^ {\max} (\delta , T)) ^ {4} + L d _ {\max} ^ {2}}{r}\right). \\ \end{array}
$$
Finally, by an arbitrary choice of  $r_0$  and a definition of zooming dimension with a scaling  $\rho = \rho(H, \delta, L)$  (Definition 4)
$$
({\bf B}) = \mathcal {O} \left((H ^ {3 / 2} d _ {\mathrm {m a x}} (\beta^ {\mathrm {m a x}} (\delta , T)) ^ {4} + L d _ {\mathrm {m a x}} ^ {2}) \cdot \sum_ {h = 1} ^ {H} \inf _ {\tilde {r} _ {0}} \left\{T r _ {0} + \sum_ {r \geq r _ {0}} \frac {C _ {N , h}}{\tilde {r} ^ {d _ {z , h} + 1}} \right\}\right).
$$
Term (C) For this term we just apply definition of the main event  $\mathcal{G}(\delta)\supseteq \mathcal{E}(\delta)$  and obtain
$$
(\mathbf {C}) = \mathcal {O} \Big (\sqrt {H ^ {3} T \beta^ {\max} (\delta , T)} \Big).
$$
Final regret bound First, we notice that  $\beta^{\max (\delta ,T)} = \widetilde{\mathcal{O}} (d_c)$ , therefore we have
$$
\mathfrak {R} ^ {T} = \widetilde {\mathcal {O}} \left(H ^ {3} d _ {c} + \left(H ^ {3 / 2} d _ {c} ^ {4} + L\right) \sum_ {h = 1} ^ {H} \inf  _ {r _ {0} > 0} \left\{T r _ {0} + \sum_ {r \geq r _ {0}} \frac {C _ {N , h}}{r ^ {d _ {z , h} + 1}} \right\} + \sqrt {H ^ {3} T d _ {c}}\right).
$$
Taking  $r_0 = K^{-d_{z,h} + 1 / 2}$  for each  $h$  and summing the geometric series we conclude the statement.
G Deviation and Anti-Concentration Inequalities
G.1 Deviation inequality for  $\kappa_{\mathrm{inf}}$
For a measure  $\nu \in \mathcal{P}([0,b])$  supported on a segment  $[0,b]$  (equipped with a Borel  $\sigma$ -algebra) and a number  $\mu \in [0,b]$  we recall the definition of the minimum Kullback-Leibler divergence
$$
\mathcal {K} _ {\inf } (\nu , \mu) \triangleq \inf  \left\{\mathrm {K L} (\nu , \eta): \eta \in \mathcal {P} ([ 0, b ]), \nu \ll \eta , \mathbb {E} _ {X \sim \eta} [ X ] \geq \mu \right\}.
$$
As the Kullback-Leibler divergence this quantity admits a variational formula.
Lemma 9 (Lemma 18 by Garivier et al., 2018). For all  $\nu \in \mathcal{P}([0,b]), u\in [0,b)$
$$
\mathcal {K} _ {i n f} (\nu , u) = \max _ {\lambda \in [ 0, 1 ]} \mathbb {E} _ {X \sim \nu} \left[ \log \left(1 - \lambda \frac {X - u}{b - u}\right) \right],
$$
moreover if we denote by  $\lambda^{\star}$  the value at which the above maximum is reached, then
$$
\mathbb {E} _ {X \sim \nu} \left[ \frac {1}{1 - \lambda^ {\star} \frac {X - u}{b - u}} \right] \leq 1.
$$
Remark 2. Contrary to Garivier et al. [2018] we allow that  $u = 0$  but in this case Lemma 9 is trivially true, indeed
$$
\mathcal {K} _ {\inf } (\nu , 0) = 0 = \max  _ {\lambda \in [ 0, 1 ]} \mathbb {E} _ {X \sim \nu} \left[ \log \left(1 - \lambda \frac {X}{b}\right) \right].
$$
Let  $(X_{t})_{t\in \mathbb{N}^{*}}$  be i.i.d. samples from a measure  $\nu$  supported on  $[0,b]$ . We denote by  $\widehat{\nu}_n\in \mathcal{P}([0,b])$  the empirical measure  $\widehat{\nu}_n = \sum_{i = 1}^n\delta_{X_i}$ , where  $\delta_{X_i}$  is a Dirac measure on  $X_{i}\in [0,b]$ .
We are now ready to state the deviation inequality for the  $\kappa_{\mathrm{inf}}$  by Tiapkin et al. [2022b] which is a self-normalized version of Proposition 13 by Garivier et al. [2018]. Notice that this inequality is stated in terms of slightly less general definition of  $\kappa_{\mathrm{inf}}$ , however, the proof remains completely the same.
Theorem 4. For all  $\nu \in \mathcal{P}([0,b])$  and for all  $\delta \in [0,1]$
$$
\mathbb {P} \left(\exists n \in \mathbb {N} ^ {\star}, n \mathcal {K} _ {i n f} \left(\widehat {\nu} _ {n}, \mathbb {E} _ {X \sim \nu} [ X ]\right) > \log (1 / \delta) + 3 \log (e \pi (1 + 2 n))\right) \leq \delta .
$$
G.2 Anti-concentration Inequality for Dirichlet Weighted Sums
In this section we state anti-concentration inequality by Tiapkin et al. [2022a] in terms of slightly different definition of  $\mathcal{K}_{\mathrm{inf}}$ .
$$
c _ {0} (\varepsilon) = \left(\frac {4}{\sqrt {\log (1 7 / 1 6)}} + 8 + \frac {4 9 \cdot 4 \sqrt {6}}{9}\right) ^ {2} \frac {2}{\pi \cdot \varepsilon^ {2}} + \log_ {1 7 / 1 6} \left(\frac {5}{3 2 \cdot \varepsilon^ {2}}\right). \tag {24}
$$
Theorem 5 (Lower bound). For any  $\alpha = (\alpha_0 + 1, \alpha_1, \ldots, \alpha_m) \in \mathbb{R}_{++}^{m+1}$  define  $\overline{p} \in \Delta_m$  such that  $\overline{p}(\ell) = \alpha_\ell / \overline{\alpha}, \ell = 0, \ldots, m$ , where  $\overline{\alpha} = \sum_{j=0}^m \alpha_j$ . Let  $\varepsilon \in (0,1)$ . Assume that  $\alpha_0 \geq c_0(\varepsilon) + \log_{17/16}(\overline{\alpha})$  for  $c_0(\varepsilon)$  defined in (24), and  $\overline{\alpha} \geq 2\alpha_0$ . Then for any  $f: \{0, \ldots, m\} \to [0, b_0]$  such that  $f(0) = b_0$ ,  $f(j) \leq b < b_0 / 2$ ,  $j \in \{1, \ldots, m\}$  and  $\mu \in (\overline{p}f, b_0)$
$$
\mathbb {P} _ {w \sim \operatorname {D i r} (\alpha)} [ w f \geq \mu ] \geq (1 - \varepsilon) \mathbb {P} _ {g \sim \mathcal {N} (0, 1)} \left[ g \geq \sqrt {2 \overline {{\alpha}} \mathcal {K} _ {i n f} \left(\sum_ {i = 0} ^ {m} \bar {p} (i) \cdot \delta_ {f (i)} , \mu\right)} \right].
$$
Next we formulate a simple corollary of Theorem 5, that slightly relaxes assumptions of this theorem under assumption  $\mu < b \leq b_0 / 2$ .
Lemma 10. For any  $\alpha = (\alpha_0 + 1, \alpha_1, \ldots, \alpha_m) \in \mathbb{R}_{++}^{m+1}$  define  $\overline{p} \in \Delta_m$  such that  $\overline{p}(\ell) = \alpha_\ell / \overline{\alpha}, \ell = 0, \ldots, m$ , where  $\overline{\alpha} = \sum_{j=0}^{m} \alpha_j$ . Also define a measure  $\bar{\nu} = \sum_{i=0}^{m} \overline{p}(i) \cdot \delta_{f(i)}$ .
Let  $\varepsilon \in (0,1)$ . Assume that  $\alpha_0 \geq c_0(\varepsilon) + \log_{17/16}(2(\overline{\alpha} - \alpha_0))$  for  $c_0(\varepsilon)$  defined in (24). The set of any  $f \colon \{0, \dots, m\} \to [0, b_0]$  such that  $f(0) = b_0$ ,  $f(j) \leq b \leq b_0 / 2$ ,  $j \in [m]$ , and any  $\mu \in (0, b)$
$$
\mathbb {P} _ {w \sim \operatorname {D i r} (\alpha)} [ w f \geq \mu ] \geq (1 - \varepsilon) \mathbb {P} _ {g \sim \mathcal {N} (0, 1)} \left[ g \geq \sqrt {2 \bar {\alpha} \mathcal {K} _ {i n f} (\bar {\nu} , \mu)} \right].
$$
Proof. Assume that assumption  $\overline{\alpha} \geq 2\alpha_0$  holds.
Then we show that the Theorem 5 also holds for  $\mu \leq \overline{p} f$ . First, we notice that for any  $\gamma > 0$
$$
\mathbb {P} _ {w \sim \operatorname {D i r} (\alpha)} [ w f \geq \mu ] \geq \mathbb {P} _ {w \sim \operatorname {D i r} (\alpha)} [ w f \geq \bar {p} f + \gamma ] \geq (1 - \varepsilon) \mathbb {P} _ {g \sim \mathcal {N} (0, 1)} \left[ g \geq \sqrt {2 \bar {\alpha} \mathcal {K} _ {\inf } (\bar {\nu} , \bar {p} f + \gamma)} \right].
$$
By continuity of  $\mathcal{K}_{\mathrm{inf}}$  in its second argument (see Theorem 7 by Honda and Takemura [2010]) we can tend  $\gamma$  to zero, and then use an equality  $\mathcal{K}_{\mathrm{inf}}(\bar{\nu},\bar{p} f) = \mathcal{K}_{\mathrm{inf}}(\bar{\nu},\mu) = 0$ .
Next, assume  $\overline{\alpha} \leq 2\alpha_0$ . In this case we have  $\overline{p}f \geq b$ , thus for any  $0 \leq \mu \leq b$
$$
\mathbb {P} _ {w \sim \operatorname {D i r} (\alpha)} [ w f \geq \mu ] \geq \mathbb {P} _ {\xi \sim \operatorname {B e t a} \left(\alpha_ {0} + 1, \bar {\alpha} - \alpha_ {0}\right)} [ b _ {0} \xi \geq \mu ] \geq \mathbb {P} _ {\xi \sim \operatorname {B e t a} \left(\alpha_ {0} + 1, \bar {\alpha} - \alpha_ {0}\right)} \left[ \xi \geq \frac {1}{2} \right],
$$
where we first apply a lower bound  $f(j) \geq 0$  for all  $j > 0$  and  $f(0) = b_0$ , and second apply a bound  $\mu \leq b_0 / 2$ . Here we may apply the result of Alfers and Dinges [1984, Theorem 1.2"] and obtain the following lower bound
$$
\mathbb {P} _ {w \sim \mathrm {D i r} (\alpha)} [ w f \geq \mu ] \geq \Phi \Bigl (- \mathrm {s i g n} (\alpha_ {0} / \overline {{\alpha}} - 1 / 2) \cdot \sqrt {2 \overline {{\alpha}} \mathrm {k l} (\alpha_ {0} / \overline {{\alpha}} , 1 / 2)} \Bigr) \geq (1 - \varepsilon) \mathbb {P} _ {g \sim \mathcal {N} (0, 1)} [ g \geq 0 ]
$$
where we used  $\alpha_0 / \overline{\alpha} > 1 / 2$ .
G.3 Rosenthal-type inequality
In this section we state Rosenthal-type inequality for martingale differences by [Pinelis, 1994, Theorem 4.1]. The exact constants could be derived from the proof.
Theorem 6. Let  $X_{1},\ldots ,X_{n}$  be a martingale-difference sequence adapted to a filtration  $\{\mathcal{F}_i\}_{i = 1,\dots ,n}$ :  $\mathbb{E}[X_i|\mathcal{F}_i] = 0$ . Define  $\mathcal{V}_i = \mathbb{E}[X_i^2 |\mathcal{F}_{i - 1}]$ . Then for any  $p\geq 2$  the following holds
$$
\mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} X _ {i} \right| ^ {p} \right] \leq C _ {1} p ^ {1 / 2} \mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} \mathcal {V} _ {i} \right| ^ {p / 2} \right] + 2 C _ {2} p \mathbb {E} ^ {1 / p} \left[ \max  _ {i \in [ n ]} | X _ {i} | ^ {p} \right],
$$
where  $C_1 = 60\mathrm{e}, C_2 = 60$
Additionally, we need some additional lemma to use this inequality in our setting.
Definition 7. A random variable  $X$  is called sub-exponential with parameters  $(\sigma^2, b)$  if the following tail condition holds for any  $t > 0$
$$
\mathbb {P} [ | X - \mathbb {E} [ X ] | \geq t ] \leq 2 \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2} + 2 b t}\right).
$$
By Theorem 1 of Skorski [2023] we have for any  $\xi \in B(\alpha, \beta)$  with  $\beta \geq \alpha$  and any  $t > 0$
$$
\mathbb {P} [ | \xi - \mathbb {E} [ \xi ] | \geq t ] \leq 2 \exp \biggl (- \frac {t ^ {2}}{2 (v + c t / 3)} \biggr),
$$
$$
v = \frac {\alpha \beta}{(\alpha + \beta) ^ {2} (\alpha + \beta + 1)} \leq \frac {\alpha}{(\alpha + \beta) ^ {2}}, \quad c = \frac {2 (\beta - \alpha)}{(\alpha + \beta) (\alpha + \beta + 2)} \leq \frac {2}{\alpha + \beta},
$$
so  $\xi$  is  $(\alpha / (\alpha + \beta)^2, 2 / (3(\alpha + \beta)))$  sub-exponential.
Lemma 11. Let  $X_{1},\ldots ,X_{n}$  be a sequence of centred  $(\sigma^2,b)$  sub-exponential random variables, not necessarily independent. Then for any  $p\geq 2$
$$
\mathbb {E} \bigg [ \max _ {\ell \in [ n ]} | X _ {\ell} | ^ {p} \bigg ] \leq \max \{\sqrt {8 \sigma^ {2} \log n}, 8 b \log n \} ^ {p} + \mathrm {e} (2 \sigma) ^ {p} p ^ {p / 2} + 2 \mathrm {e} (8 b) ^ {p} p ^ {p}.
$$
Proof. By Fubini theorem we have for any  $\eta \geq 0$ :  $\mathbb{E}[\eta^p] = p\int_0^\infty u^{p - 1}\mathbb{P}[\eta \geq u]\mathrm{d}u$ , thus for any  $a > 0$  the following holds
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\ell \in [ n ]} | X _ {\ell} | ^ {p} \right] = p \int_ {0} ^ {\infty} u ^ {p - 1} \mathbb {P} \left[ \max  _ {\ell \in [ n ]} | X _ {\ell} - \mathbb {E} [ X _ {\ell} ] | \geq u \right] \mathrm {d} u \\ \leq a ^ {p} + p \int_ {a} ^ {\infty} u ^ {p - 1} \mathbb {P} [ \exists \ell \in [ n ]: | X _ {\ell} | \geq u ] d u \\ \leq a ^ {p} + 2 p \int_ {a} ^ {\infty} u ^ {p - 1} n \exp \left(- \frac {u ^ {2}}{2 (\sigma^ {2} + b u)}\right) d u. \\ \end{array}
$$
By selecting  $a = \max \{\sqrt{8\sigma^2\log n}, 8b\log n\}$  we have
$$
n \exp \left(- \frac {u ^ {2}}{2 \left(\sigma^ {2} + b u\right)}\right) \leq \exp \left(- \frac {u ^ {2}}{4 \left(\sigma^ {2} + b u\right)}\right) \leq \exp \left(- \frac {u ^ {2}}{8 \sigma^ {2}}\right) + \exp \left(- \frac {u}{8 b}\right)
$$
for any  $u\geq a$  ,thus
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\ell \in [ n ]} | X _ {\ell} | ^ {p} \right] \leq \max  \{\sqrt {8 \sigma^ {2} \log n}, 8 b \log n \} ^ {p} \\ + 2 p \int_ {a} ^ {\infty} u ^ {p - 1} \exp \left(- \frac {u ^ {2}}{8 \sigma^ {2}}\right) d u + 2 p \int_ {a} ^ {\infty} u ^ {p - 1} \exp \left(- \frac {u}{8 b}\right) d u \\ \leq \max \{\sqrt {8 \sigma^ {2} \log n}, 8 b \log n \} ^ {p} + p (2 \sqrt {2} \sigma) ^ {p} \Gamma (p / 2) + 2 p (8 b) ^ {p} \Gamma (p). \\ \end{array}
$$
By the bounds on Gamma-function we have
$$
p \Gamma (p / 2) = \Gamma (p / 2 + 1) \leq (p + 1) ^ {(p + 1) / 2} 2 ^ {- (p + 1) / 2} \mathrm {e} ^ {1 - p / 2} \leq \mathrm {e} p ^ {p / 2} 2 ^ {- p / 2}
$$
and  $p\Gamma(p) = \Gamma(p + 1) \leq (p + 1/2)^{p+1/2} \mathrm{e}^{1-p} \leq \mathrm{e}p^p$  (see Guo et al. [2007]), thus
$$
\mathbb {E} \left[ \max _ {\ell \in [ n ]} | X _ {\ell} | ^ {p} \right] \leq \max \{\sqrt {8 \sigma^ {2} \log n}, 8 b \log n \} ^ {p} + \mathrm {e} (2 \sigma) ^ {p} p ^ {p / 2} + 2 \mathrm {e} (8 b) ^ {p} p ^ {p}.
$$
Proposition 7. Let  $W_{1}, \ldots, W_{n}$  be a sequence of Beta-distributed random variables  $W_{i} \sim \mathrm{Beta}(1 / \kappa, (n - 1) / \kappa)$  for  $\kappa > 0$ . Let  $\{\mathcal{F}_i\}_{i \in [n]}$  be a filtration such that  $W_{i}$  is independent from  $\mathcal{F}_{i-1} : \mathbb{E}[W_i | \mathcal{F}_{i-1}] = \mathbb{E}[W_i]$ , and  $X_{1}, \ldots, X_{n}$  be a sequence of bounded predictable random variables:  $\mathbb{E}[X_i | \mathcal{F}_{i-1}] = X_i, |X_i| \leq B$ .
Then with probability at least  $1 - \delta$  the following holds
$$
\left| \sum_ {i = 1} ^ {n} W _ {i} X _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i} \right| \leq 6 0 \mathrm {e} ^ {2} B \sqrt {\frac {\kappa \log (1 / \delta)}{n}} + 1 2 0 0 \mathrm {e} B \frac {\kappa \log (n) \log^ {2} (1 / \delta)}{n}
$$
Proof. First we notice that  $Z_{i} = (W_{i} - \mathbb{E}[W_{i}])\cdot X_{i}$  forms a martingale-difference sequence:  $\mathbb{E}[Z_i|\mathcal{F}_{i - 1}] = 0$ . Therefore, we can apply Theorem 6
$$
\mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} Z _ {i} \right| ^ {p} \right] \leq 6 0 \mathrm {e} \sqrt {p} \cdot \mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} \mathcal {V} _ {i} \right| ^ {p / 2} \right] + 1 2 0 p \cdot \mathbb {E} ^ {1 / p} \left[ \max _ {i \in [ n ]} | Z _ {i} | ^ {p} \right],
$$
where  $\mathcal{V}_i = \mathbb{E}[Z_i^2 |\mathcal{F}_{i - 1}] = X_i^2\mathrm{Var}(W_i)$ . We can easily upper bound the variance of Beta-distributed random variable and obtain
$$
\mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} \mathcal {V} _ {i} \right| ^ {p / 2} \right] \leq \mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} \frac {\kappa X _ {i} ^ {2}}{n ^ {2}} \right| ^ {p / 2} \right] \leq \sqrt {\frac {\kappa B ^ {2}}{n}}.
$$
For the second term we apply Lemma 11 since  $W_{i}$  are  $(\kappa / n^{2}, 2\kappa / (3n))$ -sub-exponential
$$
\begin{array}{l} \mathbb {E} ^ {1 / p} \left[ \max  _ {i \in [ n ]} | Z _ {i} | ^ {p} \right] \leq B \left(\max  \left\{\sqrt {\frac {8 \kappa \log n}{n ^ {2}}}, \frac {1 6 \kappa \log n}{3 n} \right\} + \mathrm {e} ^ {1 / p} \sqrt {\frac {\kappa}{n ^ {2}}} \sqrt {p} + (2 \mathrm {e}) ^ {1 / p} \frac {1 6 \kappa}{3 n} \cdot p\right) \\ \leq 2 0 B \kappa \cdot p \cdot \frac {\log n}{n}. \\ \end{array}
$$
$$
\mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} Z _ {i} \right| ^ {p} \right] \leq 6 0 \mathrm {e} \cdot p ^ {1 / 2} \sqrt {\frac {\kappa B ^ {2}}{n}} + 1 2 0 0 \cdot p ^ {2} \frac {B \kappa \cdot \log n}{n}.
$$
Next we turn from moments to tails. By Markov inequality with  $p = \log (1 / \delta)$
$$
\begin{array}{l} \mathbb {P} \left[ \left| \sum_ {i = 1} ^ {n} Z _ {i} \right| \geq t \right] \leq \left(\frac {\mathbb {E} ^ {1 / p} \left[ \left| \sum_ {i = 1} ^ {n} Z _ {i} \right| ^ {p} \right]}{t}\right) ^ {p} \\ \leq \left(\frac {6 0 \mathrm {e} B \sqrt {\kappa \log (1 / \delta) / n} + 1 2 0 0 \log^ {2} (1 / \delta) B \kappa \log (n) / n}{t}\right) ^ {\log (1 / \delta)}. \\ \end{array}
$$
Taking  $t = 60\mathrm{e}^2 B\sqrt{\frac{\kappa\log(1 / \delta)}{n}} + 1200\mathrm{e}B\frac{\kappa\log(n)\log^2(1 / \delta)}{n}$  we conclude the statement.
H Technical Lemmas
Lemma 12. Let  $\nu \in \mathcal{P}([0,b])$  be a probability measure over the segment  $[0,b]$  and let  $\bar{\nu} = (1 - \alpha)\delta_{b_0} + \alpha \cdot \nu$  be a mixture between  $\nu$  and a Dirac measure on  $b_0 > b$ . Then for any  $\mu \in (0,b)$
$$
\mathcal {K} _ {i n f} (\bar {\nu}, \mu) \leq (1 - \alpha) \mathcal {K} _ {i n f} (\nu , \mu).
$$
Proof. By a variational formula for  $\kappa_{\mathrm{inf}}$  (see Lemma 9)
$$
\mathcal{K}_{\inf}(\bar{\nu},\mu) = \max_{\lambda \in [0,1 / (b_{0} - \mu)]}\mathbb{E}_{X\sim \bar{\nu}}[\log (1 - \lambda (X - \mu))].
$$
Since  $\bar{\nu}$  is a mixture, we have for any  $\lambda \in [0,1 / (b_0 - \mu)]$
$$
\mathbb {E} _ {X \sim \tilde {\nu}} [ \log (1 - \lambda (X - \mu)) ] = (1 - \alpha) \mathbb {E} _ {X \sim \tilde {\nu}} [ \log (1 - \lambda (X - \mu)) ] + \alpha \log (1 - \lambda (b _ {0} - \mu)).
$$
Notice that  $\max_{\lambda > 0} \log(1 - \lambda (b_0 - \mu)) = 0$ . Thus, maximizing each term separately over  $\lambda$ , we have
$$
\begin{array}{l} \mathcal{K}_{\inf}(\bar{\nu},\mu)\leq \left(1 - \alpha\right)\max_{\lambda \in [0,1 / (b_{0} - \mu)]}\mathbb{E}_{X\sim \bar{\nu}}[\log (1 - \lambda (X - \mu))] \\ \leq \left(1 - \alpha\right) \max _ {\lambda \in [ 0, 1 / (b - \mu) ]} \mathbb {E} _ {X \sim \bar {\nu}} [ \log (1 - \lambda (X - \mu)) ] = (1 - \alpha) \mathcal {K} _ {\inf} (\nu , \mu). \\ \end{array}
$$
In this section we detail the experiments we conducted for tabular and non-tabular environments. For all experiments we used 2 CPUs (Intel Xeon CPU 2.20GHz), and no GPU was used. Each experiment took approximately one hour.
I.1 Tabular experiments
In our initial experiment, we investigated a simple grid-world environment.
Figure 2: Regret curves of RandQL, Staged-RandQL and Sampled-RandQL on a grid-world environment with 100 states and 4 actions for  $H = 50$  an transitions noise 0.2. We show average over 4 seeds.
**Environments** For tabular experiments we use two environments.
The first one is a grid-world environment with 100 states  $(i,j)\in [10]\times [10]$  and 4 actions (left, right, up and down). The horizon is set to  $H = 50$ . When taking an action, the agent moves in the corresponding direction with probability  $1 - \varepsilon$ , and moves to a neighbor state at random with probability  $\varepsilon = 0.2$ . The agent starts at position  $(1,1)$ . The reward equals to 1 at the state  $(10,10)$  and is zero elsewhere.