SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for some  $\gamma_{m} \in [0,1]$ . Based on the GP interpretation of the model,  $\gamma_{m}$  can be understood as the posterior variances of the weights. Let  $D_{0}$  be the smallest  $D$  such that  $\sum_{m = D + 1}^{\infty}\sigma_m\phi_m^2 (z)\leq \epsilon^2 /2$ . From Equation (37), we can see that, for some constant  $C_4$ , only depending on constants  $C_1,C_p,\eta$  and  $\tilde{p}$
$$
D _ {0} \leq C _ {4} \left(\frac {\rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{\bar {p} - 1}}. \tag {50}
$$
For  $\sum_{m=1}^{D_0} \gamma_m \sigma_m \phi_m^2(z)$  on a finite  $D_0$ -dimensional spectrum, as shown in Lemma D.3 of Yang et al. (2020a), an  $\epsilon^2/2$  covering number scales with  $D_0^2$ . Specifically, an  $\epsilon^2/2$  covering number of the space of  $\sum_{m=1}^{D_0} \gamma_m \sigma_m \phi_m^2(z)$  is bounded by
$$
C _ {5} D _ {0} ^ {2} \left(1 + \log \left(\frac {1}{\epsilon}\right)\right). \tag {51}
$$
Combining Equations (50) and (51), we obtain
$$
\begin{array}{l} \mathcal {N} _ {k, \boldsymbol {b} ^ {2}} (\epsilon^ {2}) \leq C _ {5} D _ {0} ^ {2} (1 + \log (\frac {1}{\epsilon})) \\ \leq C _ {5} C _ {4} ^ {2} \left(\frac {\rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {2}{\overline {{p}} - 1}}, \\ \end{array}
$$
D Proof of Theorem 2 (Regret of  $\pi$ -KRVI).
Following the standard analysis of optimisitic LSVI policies, for any  $h \in [H]$ ,  $t \in [T]$ , we define temporal difference error  $\delta_h^t: \mathcal{Z} \to \mathbb{R}$  as
$$
\delta_ {h} ^ {t} (z) = r _ {h} (z) + \left[ P _ {h} V _ {h + 1} ^ {t} \right] (z) - Q _ {h} ^ {t} (z), \forall z \in \mathcal {Z}. \tag {52}
$$
Roughly speaking,  $\{\delta_h^t (z)\}_{h = 1}^H$  quantify how far the  $\{Q_h^t\}_{h = 1}^H$  are from satisfying the Bellman optimality equation.
For any  $h \in [H]$ ,  $t \in [T]$ , we also define
$$
\begin{array}{l} \xi_ {h} ^ {t} = \left(V _ {h} ^ {t} (s _ {h} ^ {t}) - V _ {h} ^ {\pi^ {t}} (s _ {h} ^ {t})\right) - \left(Q _ {h} ^ {t} (z _ {h} ^ {t}) - Q _ {h} ^ {\pi^ {t}} (z _ {h} ^ {t})\right), \\ \zeta_ {h} ^ {t} = \left(\left[ P _ {h} V _ {h + 1} ^ {t} \right] \left(z _ {h} ^ {t}\right) - \left[ P _ {h} V _ {h + 1} ^ {\pi^ {t}} \right] \left(z _ {h} ^ {t}\right)\right) - \left(V _ {h + 1} ^ {t} \left(s _ {h + 1} ^ {t}\right) - V _ {h + 1} ^ {\pi^ {t}} \left(s _ {h + 1} ^ {t}\right)\right). \tag {53} \\ \end{array}
$$
Using the notation defined above, we then have the following regret decomposition into three parts.
$$
\begin{array}{l} \mathcal {R} (T) = \underbrace {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} [ \delta_ {h} ^ {t} (z _ {h}) | s _ {1} = s _ {1} ^ {t} ] - \delta_ {h} ^ {t} (z _ {h} ^ {t})} _ {(i)} + \underbrace {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} (\xi_ {h} ^ {t} + \zeta_ {h} ^ {t})} _ {(i i)} \\ + \underbrace {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\star}} \left[ Q _ {h} ^ {t} \left(s _ {h} , \pi_ {h} ^ {\star} \left(s _ {h}\right)\right) - Q _ {h} ^ {t} \left(s _ {h} , \pi_ {h} ^ {t} \left(s _ {h}\right)\right) \mid s _ {1} = s _ {1} ^ {t} \right]} _ {(i i i)}. \tag {54} \\ \end{array}
$$
The third term is negative, by definition of  $\pi_h^t$  that is the greedy policy with respect to  $Q_h^t$ :
$$
Q _ {h} ^ {t} (s _ {h}, \pi_ {h} ^ {\star} (s _ {h})) - Q _ {h} ^ {t} (s _ {h}, \pi_ {h} ^ {t} (s _ {h})) = Q _ {h} ^ {t} (s _ {h}, \pi_ {h} ^ {\star} (s _ {h})) - \max _ {a \in \mathcal {A}} Q _ {h} ^ {t} (s _ {h}, a) \leq 0,
$$
for all  $s_h \in S$ . The second term is bounded using the following lemma.
Lemma 8 (Lemma 5.3 in Yang et al. (2020a)) For any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , we have
$$
\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\xi_ {h} ^ {t} + \zeta_ {h} ^ {t}\right) \leq 4 \sqrt {T H ^ {3} \log \left(\frac {2}{\delta}\right)}. \tag {55}
$$
Term (i): It turns out that the dominant term and the challenging term to bound is the first term in Lemma 7. We next provide an upper bound on this term.
For step  $h$ , let  $\mathcal{U}_h^T = \bigcup_{t=1}^T S_h^t$  be the union of all cover elements used by  $\pi$ -KRVI over all episodes. The size of  $\mathcal{U}_h^T$  is bounded in the following lemma and is useful in the analysis of Term (i).
Lemma 9 (Lemma 2 in Janz et al. (2020)) The size of  $\mathcal{U}_h^T$  satisfies
$$
\left| \mathcal {U} _ {h} ^ {T} \right| \leq C _ {6} T ^ {\frac {d}{d + \alpha}}, \tag {56}
$$
for some constant  $C_6$ .
The size of  $\mathcal{U}_h^T$  depends on the dimension of the domain and the parameter  $\alpha$  used in the splitting rule in Section 3.1.
Now, consider a cover element  $\mathcal{Z}' \in \mathcal{U}_h^T$ . Using Theorem 1, we have, with probability at least  $1 - \delta$ , for all  $h \in [H], t \in [T], z \in \mathcal{Z}'$ , for some  $\epsilon_h^t \in (0,1)$ ,
$$
\left| r _ {h} (z) + \left[ P _ {h} V _ {h + 1} \right] (z) - \widehat {Q} _ {h} ^ {t} (z) \right| \leq \beta_ {h} ^ {t} (\delta , \epsilon_ {h} ^ {t}) b _ {h} ^ {t} (z) + \epsilon_ {h} ^ {t}, \tag {57}
$$
where  $\beta_h^t (\delta ,\epsilon_h^t)$  is the smallest value satisfying
$$
\beta_ {h} ^ {t} (\delta , \epsilon_ {h} ^ {t}) \geq H + 1 + \frac {H}{\sqrt {2}} \sqrt {\Gamma_ {k , \lambda} (N) + \log \mathcal {N} _ {k , h} (\epsilon_ {h} ^ {t} ; R _ {N} , \beta_ {h} ^ {t} (\delta , \epsilon_ {h} ^ {t})) + 1 + \log \left(\frac {N H}{\delta}\right)} + \frac {3 \sqrt {N} \epsilon_ {h} ^ {t}}{\lambda},
$$
with  $N = N_{h,\mathcal{Z}'}^T$  and  $\epsilon_h^t = \frac{H\sqrt{\log\left(\frac{TH}{\delta}\right)}}{\sqrt{N_{h,\mathcal{Z}'}^T}}$ .
We also note that
$$
\begin{array}{l} \Gamma_ {k, \lambda} \left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right) = \mathcal {O} \left(\left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right) ^ {\frac {1}{p}} \left(\log \left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right)\right) ^ {1 - \frac {1}{p}} \rho_ {\mathcal {Z} ^ {\prime}} ^ {\frac {\alpha}{p}}\right) \\ = \mathcal {O} \left(\left(\rho_ {\mathcal {Z} ^ {\prime}}\right) ^ {\frac {- \alpha}{\bar {p}}} \left(\log \left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right)\right) ^ {1 - \frac {1}{\bar {p}}} \rho_ {\mathcal {Z} ^ {\prime}} ^ {\frac {\alpha}{\bar {p}}}\right) \\ = \mathcal {O} \left(\left(\log \left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right)\right) ^ {1 - \frac {1}{\bar {p}}}\right) \\ = \mathcal {O} (\log (T)), \tag {58} \\ \end{array}
$$
Lemma 7 (Lemma 5.1 in Yang et al. (2020a) on regret decomposition) We have
where the first line is based on Lemma 2, and the second line is by the design of partitioning in  $\pi$ -KRVI. Recall that each hypercube is partitioned when  $\rho_{\mathcal{Z}'}^{-\alpha} < N_{h,\mathcal{Z}'}^t + 1$ , ensuring that  $N_{h,\mathcal{Z}'}^t$  remains at most  $\rho_{\mathcal{Z}'}^{-\alpha}$ .
For the covering number, with the choice of  $\epsilon_h^t = \frac{H\sqrt{\log(\frac{T H}{\delta})}}{\sqrt{N_{h,\mathcal{Z}'}^t}}$ , we have
$$
\begin{array}{l} \log \mathcal {N} _ {k, h} \left(\epsilon_ {h} ^ {t}; R _ {N}, \beta_ {h} ^ {t} (\delta , \epsilon_ {h} ^ {t})\right) \\ { = } { \mathcal { O } \left( \left( \frac { R _ { N } ^ { 2 } \rho _ { \mathcal { Z } ^ { \prime } } ^ { \alpha } } { ( \epsilon _ { h } ^ { t } ) ^ { 2 } } \right) ^ { \frac { 1 } { \overline { { p } } - 1 } } ( 1 + \log ( \frac { R _ { N } } { \epsilon _ { h } ^ { t } } ) ) + \left( \frac { ( \beta _ { h } ^ { t } ( \delta , \epsilon _ { h } ^ { t } ) ) ^ { 2 } \rho _ { \mathcal { Z } ^ { \prime } } ^ { \alpha } } { ( \epsilon _ { h } ^ { t } ) ^ { 2 } } \right) ^ { \frac { 2 } { \overline { { p } } - 1 } } ( 1 + \log ( \frac { \beta _ { h } ^ { t } ( \delta , \epsilon _ { h } ^ { t } ) } { \epsilon _ { h } ^ { t } } ) ) \right) } \\ { = } { \mathcal { O } \left( \left( \frac { R _ { N } ^ { 2 } } { H ^ { 2 } \log ( \frac { H T } { \delta } ) } \right) ^ { \frac { 1 } { \overline { { p } } - 1 } } ( 1 + \log ( \frac { R _ { N } } { \epsilon _ { h } ^ { t } } ) ) + \left( \frac { ( \beta _ { h } ^ { t } ( \delta , \epsilon _ { h } ^ { t } ) ) ^ { 2 } } { H ^ { 2 } \log ( \frac { H T } { \delta } ) } \right) ^ { \frac { 2 } { \overline { { p } } - 1 } } ( 1 + \log ( \frac { \beta _ { h } ^ { t } ( \delta , \epsilon _ { h } ^ { t } ) } { \epsilon _ { h } ^ { t } } ) ) \right) . } \\ \end{array}
$$
We thus see that the choice of  $\beta_h^t (\delta ,\epsilon_h^t) = \Theta (H\sqrt{\log(\frac{T H}{\delta})})$  satisfies the requirement for confidence interval width on  $\mathcal{Z}'$  based on Theorem 1. We now use probability union bound over all  $\mathcal{Z}'\in \mathcal{U}_h^T$  to obtain
$$
\beta_ {T} (\delta) = \Theta (H \sqrt {\log \left(\frac {T H \left| H \mathcal {U} _ {h} ^ {T} \right|}{\delta}\right)}) = \Theta (H \sqrt {\log \left(\frac {T H}{\delta}\right)}). \tag {59}
$$
For this value of  $\beta_T(\delta)$ , we have with probability at least  $1 - \delta$ , for all  $h \in [H], t \in [T], z \in \mathcal{Z}$
$$
\left| r _ {h} (z) + \left[ P _ {h} V _ {h + 1} \right] (z) - \widehat {Q} _ {h} ^ {t} (z) \right| \leq \beta_ {T} (\delta) b _ {h} ^ {t} (z) + \epsilon_ {h} ^ {t}, \tag {60}
$$
where in the above expression  $\epsilon_h^t$  is the parameter of the covering number corresponding to  $\mathcal{Z}'$  when  $z \in \mathcal{Z}'$ .
Therefore, we have, with probability at least  $1 - \delta$
$$
\text {T e r m} (i) \leq \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} - \delta_ {h} ^ {t} \left(z _ {h} ^ {t}\right) \leq 2 \beta_ {T} (\delta) \left(\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} b _ {h} ^ {t} \left(z _ {h} ^ {t}\right)\right) + 2 \epsilon_ {h} ^ {t}, \tag {61}
$$
$$
\epsilon_ {h} ^ {t} = \frac {H \sqrt {\log (\frac {T H}{\delta})}}{\sqrt {N _ {h , \mathcal {Z} \left(z _ {h} ^ {t}\right)} ^ {t}}}. \tag {62}
$$
We bound the total uncertainty in the kernel ridge regression measured by  $\sum_{t=1}^{T} \left(b_h^t(z_h^t)\right)^2$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(b _ {h} ^ {t} \left(z _ {h} ^ {t}\right)\right) ^ {2} = \sum_ {\mathcal {Z} ^ {\prime} \in \mathcal {U} _ {h} ^ {T}} \sum_ {z _ {h} ^ {t} \in \mathcal {Z} ^ {\prime}} \left(b _ {h} ^ {t} \left(z _ {h} ^ {t}\right)\right) ^ {2} \\ \leq \sum_ {\mathcal {Z} ^ {\prime} \in \mathcal {U} _ {h} ^ {T}} \frac {2}{\log (1 + 1 / \lambda^ {2})} \Gamma_ {k, \lambda} \left(N _ {h, \mathcal {Z} ^ {\prime}} ^ {T}\right) \\ = \mathcal {O} \left(\sum_ {\mathcal {Z} ^ {\prime} \in \mathcal {U} _ {h} ^ {T}} \log (T)\right) \\ = \mathcal {O} \left(\left| \mathcal {U} _ {h} ^ {T} \right| \log (T)\right) \\ = \mathcal {O} \left(T ^ {\frac {d}{d + \alpha}} \log (T)\right). \\ \end{array}
$$
The first inequality is commonly used in kernelized bandits. For example see (Srinivas et al., 2010a, Lemma 5.4). The third and fifth lines follow from Equation (58) and Lemma 9, respectively. Also,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\epsilon_ {h} ^ {t}\right) ^ {2} = \sum_ {t = 1} ^ {T} \frac {H ^ {2} \log \left(\frac {T H}{\delta}\right)}{N _ {h , \mathcal {Z} \left(z _ {h} ^ {t}\right)} ^ {t}} \tag {63} \\ \leq \sum_ {\mathcal {Z} ^ {\prime} \in \mathcal {U} _ {h} ^ {T}} \sum_ {z _ {h} ^ {t} \in \mathcal {Z} ^ {\prime}} \frac {H ^ {2} \log (\frac {T H}{\delta})}{N _ {h , \mathcal {Z} ^ {\prime}} ^ {t}} \\ \leq \left| \mathcal {U} _ {h} ^ {T} \right| H ^ {2} \log \left(\frac {T H}{\delta}\right) (\log (T) + 1) \\ \leq \mathcal {O} \left(H ^ {2} T ^ {\frac {d}{d + \alpha}} \log \left(\frac {T H}{\delta}\right) \log (T)\right). \\ \end{array}
$$
We are now ready to bound the
$$
\begin{array}{l} \text {T e r m} (i) \leq 2 \beta_ {T} (\delta) \left(\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} b _ {h} ^ {t} \left(z _ {h} ^ {t}\right)\right) + 2 \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \epsilon_ {h} ^ {t} \tag {64} \\ \leq 2 \beta_ {T} (\delta) \sqrt {T} \sum_ {h = 1} ^ {H} \sqrt {\sum_ {t = 1} ^ {T} \left(b _ {h} ^ {t} \left(z _ {h} ^ {t}\right)\right) ^ {2}} + 2 \sqrt {T} \sum_ {h = 1} ^ {H} \sqrt {\sum_ {t = 1} ^ {T} \left(\epsilon_ {h} ^ {t}\right) ^ {2}} \\ = \mathcal {O} \left(H ^ {2} T ^ {\frac {d + \alpha / 2}{d + \alpha}} \sqrt {\log (T) \log (\frac {T H}{\delta})}\right). \\ \end{array}
$$