SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mercer theorem (Mercer, 1909) provides a representation of the kernel in terms of an infinite dimensional feature map (e.g., see, Christmann and Steinwart, 2008, Theorem 4.49). Let  $\mathcal{Z}$  be a compact metric space and  $\mu$  be a finite Borel measure on  $\mathcal{Z}$  (we consider Lebesgue measure in a Euclidean space). Let  $L_{\mu}^{2}(\mathcal{Z})$  be the set of square-integrable functions on  $\mathcal{Z}$  with respect to  $\mu$ . We further say a kernel is square-integrable if
$$
\int_ {\mathcal {Z}} \int_ {\mathcal {Z}} k ^ {2} (z, z ^ {\prime}) d \mu (z) d \mu (z ^ {\prime}) <   \infty .
$$
Theorem 3 (Mercer Theorem) Let  $\mathcal{Z}$  be a compact metric space and  $\mu$  be a finite Borel measure on  $\mathcal{Z}$ . Let  $k$  be a continuous and square-integrable kernel, inducing an integral operator  $T_{k}: L_{\mu}^{2}(\mathcal{Z}) \to L_{\mu}^{2}(\mathcal{Z})$  defined by
$$
\left(T _ {k} f\right) (\cdot) = \int_ {\mathcal {Z}} k (\cdot , z ^ {\prime}) f \left(z ^ {\prime}\right) d \mu \left(z ^ {\prime}\right),
$$
where  $f \in L_{\mu}^{2}(\mathcal{Z})$ . Then, there exists a sequence of eigenvalue-eigenfeature pairs  $\{(\sigma_m, \phi_m)\}_{m=1}^{\infty}$  such that  $\sigma_m > 0$ , and  $T_k \phi_m = \sigma_m \phi_m$ , for  $m \geq 1$ . Moreover, the kernel function can be represented as
$$
k \left(z, z ^ {\prime}\right) = \sum_ {m = 1} ^ {\infty} \sigma_ {m} \phi_ {m} (z) \phi_ {m} \left(z ^ {\prime}\right),
$$
where the convergence of the series holds uniformly on  $\mathcal{Z}\times \mathcal{Z}$
According to the Mercer representation theorem (e.g., see, Christmann and Steinwart, 2008, Theorem 4.51), the RKHS induced by  $k$  can consequently be represented in terms of  $\{(\sigma_m, \phi_m)\}_{m=1}^{\infty}$ .
Theorem 4 (Mercer Representation Theorem) Let  $\{(\sigma_m,\phi_m)\}_{i = 1}^{\infty}$  be the Mercer eigenvalue eigenfeature pairs. Then, the RKHS of  $k$  is given by
$$
\mathcal {H} _ {k} = \left\{f (\cdot) = \sum_ {m = 1} ^ {\infty} w _ {m} \sigma_ {m} ^ {\frac {1}{2}} \phi_ {m} (\cdot): w _ {m} \in \mathbb {R}, \| f \| _ {\mathcal {H} _ {k}} ^ {2} := \sum_ {m = 1} ^ {\infty} w _ {m} ^ {2} <   \infty \right\}.
$$
Mercer representation theorem indicates that the scaled eigenfeatures  $\{\sqrt{\sigma_m}\phi_m\}_{m = 1}^{\infty}$  form an orthonormal basis for  $\mathcal{H}_k$
B Proof of Theorem 1 (Confidence Interval)
Confidence bounds of the form given in Theorem 1 have been established for a fixed function  $f$  with bounded RKHS norm and sub-Gaussian observation noise in several works including Abbasi-Yadkori (2013); Chowdhury and Gopalan (2017); Vakili et al. (2021a). In the RL setting, however, we apply the confidence interval to  $f = r_h + [P_hV_{h + 1}^t]$ . Although the RKHS norm of this target function is bounded by  $H + 1$ , this is not a fixed function as it depends on  $V_{h + 1}^{t}$ . In addition the observation noise terms  $V_{h + 1}(s_{h + 1}^{t}) - [P_{h}V_{h + 1}^{t}](s_{h}^{t},a_{h}^{t})$  also depend on  $V_{h + 1}^{t}$ . To handle this setting, we prove a confidence interval that holds for all possible  $V_{h + 1}^{t}:S\to [0,H]$ . For this purpose, we use a probability union bound and a covering set argument over the function class of  $V_{h + 1}^{t}$ .
We first recall the confidence interval for a fixed function and noise sequence given in (Chowdhury and Gopalan, 2017, Theorem 2). See also (Abbasi-Yadkori, 2013, Corollary 3.15).
Lemma 4 Let  $\{z^t\in \mathcal{Z}\}_{t = 1}^T$  be a stochastic process predictable with respect to the filtration  $\{\mathcal{F}_t\}_{t = 0}^T$ . Let  $\{\varepsilon^t\}_{t = 1}^T$  be a real valued  $\mathcal{F}_t$  measurable stochastic process with a  $\sigma$  sub-Gaussian distribution conditioned on  $\mathcal{F}_{t - 1}$ . Let  $\mu^{t,f}$  and  $b^{t}$  be the kernel ridge predictor and uncertainty estimate of  $f$  using  $t$  noisy observations of the form  $\{f(z^{\tau}) + \varepsilon^{\tau}\}_{\tau = 1}^{t}$ . Assume  $f\in \mathcal{B}_{k,R}$ . Then with probability at least  $1 - \delta$ , for all  $z\in \mathcal{Z}$  and  $t\geq 1$ ,
$$
| f (z) - \mu^ {t, f} (z) | \leq \beta_ {1} b ^ {t} (z), \tag {22}
$$
where  $\beta_{1} = R + \sigma \sqrt{2(\Gamma_{k,\lambda}(t) + 1 + \log(\frac{1}{\delta}))}$ .
As discussed above, we cannot directly use this confidence interval on  $r_h + [P_h V_{h+1}^t]$  in the RL setting. Instead, we need to prove a new confidence interval that holds true for all possible  $V_{h+1}^t$ . We thus define  $\mathcal{V}$  to be the function class of  $V_{h+1}^t$  as follows.
$$
\mathcal {V} _ {k, h} (R, B) = \{V: V (s) = \max  _ {a \in \mathcal {A}} Q (s, a), \text {f o r s o m e} Q \in \mathcal {Q} _ {k, h} (R, B) \}. \tag {23}
$$
For simplicity of presentation, we specify the parameters  $R$  and  $B$  later.
Let  $\mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$  be the smallest  $\epsilon$ -covering set of  $\mathcal{V}_{k,h}(R, B)$  in terms of  $l_{\infty}$  norm. That is to say for all  $V \in \mathcal{V}_{k,h}(R, B)$ , there exists some  $\overline{V} \in \mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$  such that  $\| V - \overline{V} \|_{l_{\infty}} \leq \epsilon$ . Let  $\mathcal{N}_{k,h}^{\nu}(\epsilon; R, B)$  denote the  $\epsilon$ -covering number of  $\mathcal{V}_{k,h}(R, B)$ . By definition  $\mathcal{N}_{k,h}^{\nu}(\epsilon; R, B) = |\mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)|$ .
We can create a confidence bound for all  $\overline{V} \in \mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$ , using Lemma 4 and a probability union bound over  $\mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$ . Fix  $h \in [H]$  and  $t \in [T]$ . Let us use the notation  $\widehat{\overline{Q}}^t$  for the kernel ridge predictor with  $\overline{V}$ . That is  $\widehat{\overline{Q}}^t(z) = k_{Z^t}(z)(K_{Z^t} + \lambda^2 I)^{-1}\overline{Y}$ , where  $\overline{Y}^\top = [\overline{V}(s_{h+1}^\tau)]_{\tau=1}^t$ , and  $s_{h+1}^\tau$  is the next state drawn randomly from probability distribution  $P_h(\cdot|z_h^\tau)$ . In addition, to simplify the notation, we use  $g = r_h + [P_h\overline{V}]$  and  $\mu^{t,g} = \widehat{\overline{Q}}^t$ . Also, let  $b^t(z) = (k(z,z) - k_{Z^t}^\top(z))(K_{Z^t} + \lambda^2 I)^{-1}k_{Z^t}(z))^{\frac{1}{2}}$ . Then, we have, with probability at least  $1 - \delta$ , for all  $\overline{V} \in \mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$  and for all  $z \in \mathcal{Z}$ ,
$$
\left| g (z) - \mu^ {t, g} (z) \right| \leq \beta_ {2} b ^ {t} (z), \tag {24}
$$
where  $\beta_{2} = H + 1 + \frac{H}{\sqrt{2}}\sqrt{\Gamma_{k,\lambda}(t) + \log\mathcal{N}_{k,h}^{\mathcal{V}}(\epsilon;R,B) + 1 + \log(\frac{1}{\delta})}$ .
Confidence interval (24) is a direct application of Lemma 4 and using a probability union bound over all  $\overline{V} \in \mathcal{C}_{k,h}^{\gamma}(\epsilon; R, B)$ . Note that,  $\|r_h + P_h\overline{V}\|_{\mathcal{H}_k} \leq H + 1$  (Lemma 1). In addition,  $\overline{V}(s_{h+1}^{\tau}) - [P_h\overline{V}](z_h^\tau) \in [0, H]$  for all  $h$  and  $\tau$ . A bounded random variable in  $[0, H]$  is a  $H/2$  sub-Gaussian random variable based on Hoeffding inequality (Hoeffding, 1994).
Now, we extend the uniform confidence interval over all  $\overline{V} \in \mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$  to a uniform confidence interval over all  $V \in \mathcal{V}_{k,h}(R, B)$ . For some  $V \in \mathcal{V}_{k,h}(R, B)$ , define  $f = r_h + [P_h V]$  and  $\mu^{t,f} = \widehat{Q}^t$ , similar to  $g$  and  $\mu^{t,g}$ . By definition of  $\mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$ , there exists  $\overline{V} \in \mathcal{C}_{k,h}^{\nu}(\epsilon; R, B)$ , such that  $\| V - \overline{V} \|_{l_\infty} \leq \epsilon$ . Thus, for all  $z \in \mathcal{Z}$ ,
$$
f (z) - g (z) = \left[ P V \right] (z) - \left[ P \bar {V} \right] (z) \leq \sup  _ {s \in S} | V (s) - \bar {V} (s) | \leq \epsilon . \tag {25}
$$
$$
\begin{array}{l} \left| f (z) - \mu^ {t, f} (z) \right| \leq \left| f (z) - g (z) \right| + \left| g (z) - \mu^ {t, g} (z) \right| + \left| \mu^ {t, g} (z) - \mu^ {t, f} (z) \right| \\ \leq \beta_ {2} b ^ {t} (z) + \epsilon + | \mu^ {t, g} (z) - \mu^ {t, f} (z) |. \tag {26} \\ \end{array}
$$
Next, we prove that  $|\mu^{t,f}(z) - \mu^{t,g}(z)| \leq \frac{3\epsilon\sqrt{t}b^t(z)}{\lambda}$ .
Let us further simplify the notation by introducing  $\alpha_{t}(z) = (K_{Z_{t}} + \lambda^{2}I)^{-1}k_{Z_{t}}(z), F_{t}^{\top} = [f(z_{h}^{\tau})]_{\tau = 1}^{t}$ ,  $E_{t}^{\top} = [\varepsilon^{\tau} = V(s_{h + 1}^{\tau}) - [P_{h}V](z_{h}^{\tau})]_{\tau = 1}^{t}, G_{t}^{\top} = [g(z_{h}^{\tau})]_{\tau = 1}^{t}, \overline{E}_{t}^{\top} = [\bar{\varepsilon}^{\tau} = \overline{V}(s_{h + 1}^{\tau}) - [P_{h}\overline{V}](z_{h}^{\tau})]_{\tau = 1}^{t}$  so that  $\mu^{t,f}(z) = \alpha^{\top}(z)(F_t + E_t)$  and  $\mu^{t,g}(z) = \alpha^{\top}(z)(G_t + \overline{E}_t)$ .
As discussed earlier, the observation noise terms  $\varepsilon^t$  also depend on  $V$ . We have, for all  $t \geq 1$ ,
$$
\begin{array}{l} {| \varepsilon^ {t} - \bar {\varepsilon} ^ {t} |} = {\left| V (s _ {h + 1} ^ {\tau}) - \overline {{V}} (s _ {h + 1} ^ {\tau}) - ([ P _ {h} V ] (z _ {h} ^ {\tau}) - [ P _ {h} \overline {{V}} ] (z _ {h} ^ {\tau}) \right|} \\ \leq 2 \epsilon . \\ \end{array}
$$
Using the difference between  $f$  and  $g$ , as well as the difference between noise terms, we have
$$
\begin{array}{l} \left| \mu^ {t, f} (z) - \mu^ {t, g} (z) \right| = \left| \alpha_ {t} ^ {\top} (z) \left(F _ {t} + E _ {t}\right) - \alpha^ {\top} (z) \left(G _ {t} + \bar {E} _ {t}\right) \right| \\ \leq \| \alpha_ {t} (z) \| \| F _ {t} - G _ {t} + E _ {t} - \bar {E} _ {t} \| \\ \leq 3 \epsilon \sqrt {t} \| \alpha_ {t} (z) \| \\ \leq \frac {3 \epsilon \sqrt {t} b ^ {t} (z)}{\lambda}, \\ \end{array}
$$
where the last inequality follows from  $\| \alpha_t(z)\| \leq \frac{b^t(z)}{\lambda}$  (e.g., see, Vakili et al., 2021a, Proposition 1). The bound on  $|\mu^{t,f}(z) - \mu^{t,g}(z)|$  combined with (26) proves that for a fixed  $t\in [T]$ , fixed  $h\in [H]$ , for all  $z\in \mathcal{Z}$  and for all  $V\in \mathcal{V}_{k,h}(R,B)$ ,
$$
\left| f (z) - \mu^ {t, f} (z) \right| \leq \beta_ {3} b ^ {t} (z) + \epsilon ,
$$
$$
\beta_ {3} = H + 1 + \frac {H}{\sqrt {2}} \sqrt {\Gamma_ {k , \lambda} (t) + \log \mathcal {N} _ {k , h} ^ {\nu} (\epsilon ; R , B) + 1 + \log \left(\frac {1}{\delta}\right)} + \frac {3 \sqrt {t} \epsilon}{\lambda}. \tag {27}
$$
The confidence interval holds uniformly for all  $h \in [H]$  and  $t \in [T]$  using a probability union bound, when  $\beta_{3}$  is replaced with the following
$$
\beta_ {4} = H + 1 + \frac {H}{\sqrt {2}} \sqrt {\Gamma_ {k , \lambda} (t) + \log \mathcal {N} _ {k , h} ^ {\gamma} (\epsilon ; R , B) + 1 + \log \left(\frac {H T}{\delta}\right)} + \frac {3 \epsilon \sqrt {t}}{\lambda}. \tag {28}
$$
To complete the proof, we bound  $\mathcal{N}_{k,h}^{\nu}(\epsilon;R,B)$  in terms of the specific parameters of the problem. Firstly, the  $\epsilon$ -covering number of  $\mathcal{V}_{k,h}(R,B)$  is bounded by that of  $\mathcal{Q}_{k,h}(R,B)$  (Yang et al., 2020a, proof of Lemma D.1). Recall the definition of  $\mathcal{Q}_{k,h}(R,B)$  in (18). We note that  $\| \widehat{Q}_h^t\|_{\mathcal{H}_k}\leq$ $2H\sqrt{\Gamma_{k,\lambda}(t)}$  (Yang et al., 2020a, Lemma C.1). Thus, the theorem follows with  $\beta_h^t (\delta ,\epsilon)$ , where  $\beta_h^t (\delta ,\epsilon)$  is set to some value satisfying
$$
\beta_ {h} ^ {t} (\delta , \epsilon) \geq H + 1 + \frac {H}{\sqrt {2}} \sqrt {\Gamma_ {k , \lambda} (t) + \log \mathcal {N} _ {k , h} (\epsilon ; R _ {t} , \beta_ {h} ^ {t} (\delta , \epsilon)) + 1 + \log \left(\frac {H T}{\delta}\right)} + \frac {3 \epsilon \sqrt {t}}{\lambda}, \tag {29}
$$
with  $R_{t} = 2H\sqrt{\Gamma_{k,\lambda}(t)}$ . That completes the proof of Theorem 1.
C Proof of Lemmas 2 (Maximum Information Gain) and 3 (Covering Number).
We first introduce the projection of the RKHS on a lower dimensional RKHS that is used in the proof of both lemmas. We then present the proofs. Recall the Mercer theorem and the representation of kernel using Mercer eigenvalues and eigenfeatures. Using Mercer representation theorem, any  $f \in \mathcal{B}_R$  can be written as
$$
f = \sum_ {m = 1} ^ {\infty} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m}, \tag {30}
$$
with  $\sum_{m=1}^{\infty} w_m^2 \leq R^2$ . For some  $D \in \mathbb{N}$ , let  $\Pi_D[f]$  denote the projection of  $f$  onto the  $D$ -dimensional RKHS corresponding to the first  $D$  features with the largest eigenvalues
$$
\Pi_ {D} [ f ] = \sum_ {m = 1} ^ {D} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m}. \tag {31}
$$
Let us use the notations  $\boldsymbol{w}_D = [w_1, w_2, \dots, w_D]^\top$  for the  $D$ -dimensional column vector of weights,  $\phi_D(z) = [\phi_1(z), \phi_2(z), \dots, \phi_D(z)]^\top$  for the  $D$ -dimensional column vector of eigenfeatures, and
$\Sigma_{D} = \mathrm{diag}([\sigma_{1},\sigma_{2},\dots ,\sigma_{D}])$  for the diagonal matrix of eigenvalues with  $[\sigma_1,\sigma_2,\dots ,\sigma_D]$  as the diagonal entries. We also use the notations
$$
k ^ {D} (z, z ^ {\prime}) = \phi_ {D} ^ {\top} (z) \Sigma_ {D} \phi_ {D} (z), \tag {32}
$$
to denote the kernel corresponding to the  $D$ -dimensional RKHS, as well as  $k^0(z,z') = k(z,z') - k^D(z,z')$ .
C.1 Proof of Lemma 2 on Maximum Information Gain
Recall the definition of  $\Gamma_{k,\lambda}(t)$ . We have
$$
\begin{array}{l} \frac {1}{2} \log \det  \left(I + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}}\right) = \frac {1}{2} \log \det  \left(I + \frac {1}{\lambda^ {2}} \left(K _ {Z ^ {t}} ^ {D} + K _ {Z ^ {t}} ^ {0}\right)\right) \\ = \underbrace {\frac {1}{2} \log \det  \left(I + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}} ^ {D}\right)} _ {\text {T e r m} (i)} + \underbrace {\frac {1}{2} \log \det  \left(I + \frac {1}{\lambda^ {2}} \left(I + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}} ^ {D}\right) ^ {- 1} K _ {Z ^ {t}} ^ {0}\right)} _ {\text {T e r m} (i i)}. \\ \end{array}
$$
We next bound the two terms on the right hand side.
Term (i): Note that for  $k^D$  corresponding to the  $D$ -dimensional RKHS, we have  $K_{Z^t}^D = \Phi_t\Sigma_D\Phi_t^\top$ , where  $\Phi_t = [\phi_D(z)]_{z\in Z^t}^\top$  is a  $t\times D$  matrix that stacks the feature vectors  $\phi_D(z^\tau)$ ,  $\tau = 1,\dots ,t$ , as it rows. By Weinstein-Aronszajn identity (Pozrikidis, 2014) (a special case of matrix determinant lemma),
$$
\begin{array}{l} \log \det  \left(I ^ {t} + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}} ^ {D}\right) = \log \det  \left(I ^ {t} + \frac {1}{\lambda^ {2}} \boldsymbol {\Phi} _ {t} \Sigma_ {D} \boldsymbol {\Phi} _ {t} ^ {\top}\right) \tag {33} \\ = \log \det  \left(I ^ {D} + \frac {1}{\lambda^ {2}} \Sigma_ {D} ^ {\frac {1}{2}} \Phi_ {t} \Phi_ {t} ^ {\top} \Sigma_ {D} ^ {\frac {1}{2}}\right) \\ \leq D \log \left(\frac {\operatorname {t r} \left(I ^ {D} + \frac {1}{\lambda^ {2}} \Sigma_ {D} ^ {\frac {1}{2}} \Phi_ {t} \Phi_ {t} ^ {\top} \Sigma_ {D} ^ {\frac {1}{2}}\right)}{D}\right) \\ \leq D \log \left(1 + \frac {t}{\lambda^ {2} D}\right). \\ \end{array}
$$
The first inequality follows from the inequality of arithmetic and geometric means on eigenvalues of the argument, and the second inequality follows from  $k^{D} \leq 1$ . For clarity, we used the notations  $I^t$  and  $I^D$  for identity matrices of dimension  $t$  and  $D$ , respectively. Otherwise, we drop the superscript.
Term (ii): Similarly using the inequality of arithmetic and geometric means on eigenvalues, we bound the log det by the log of the trace of the argument. Let us use  $\epsilon_{D}$  to denote an upper bound on  $k^0$ .
$$
\begin{array}{l} \log \det  \left(I + \frac {1}{\lambda^ {2}} \left(I + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}} ^ {D}\right) ^ {- 1} K _ {Z ^ {t}} ^ {0}\right) \leq t \log \left(\frac {\operatorname {t r} \left(I + \frac {1}{\lambda^ {2}} \left(I + \frac {1}{\lambda^ {2}} K _ {Z ^ {t}} ^ {D}\right) ^ {- 1} K _ {Z ^ {t}} ^ {0}\right)}{t}\right) \tag {34} \\ \leq t \log \left(1 + \frac {\epsilon_ {D}}{\lambda^ {2}}\right) \\ \leq \frac {t \epsilon_ {D}}{\lambda^ {2}}. \\ \end{array}
$$
The last inequality uses  $\log (1 + x)\leq x$  which holds for all  $x\in \mathbb{R}$
Combining the bounds on Term  $(i)$  and Term  $(ii)$ , we have
$$
\Gamma_ {k, \lambda} (t) \leq \frac {D}{2} \log \left(1 + \frac {t}{\lambda^ {2} D}\right) + \frac {t \epsilon_ {D}}{2 \lambda^ {2}}. \tag {35}
$$
Now, using the polynomial eigendecay profile given in Definition 2,
$$
\begin{array}{l} k ^ {0} \left(z, z ^ {\prime}\right) = \sum_ {m = D + 1} ^ {\infty} \sigma_ {m} \phi_ {m} (z) \phi_ {m} \left(z ^ {\prime}\right) (36) \\ \leq C _ {1} ^ {2} C _ {p} \rho_ {\mathcal {Z}} ^ {\alpha} \sum_ {m = D + 1} ^ {\infty} m ^ {- p (1 - 2 \eta)} \\ \leq C _ {1} ^ {2} C _ {p} \rho_ {\mathcal {Z}} ^ {\alpha} \int_ {D} ^ {\infty} x ^ {- \tilde {p}} d x \\ \leq \frac {C _ {1} ^ {2} C _ {p} \rho_ {\mathcal {Z}} ^ {\alpha}}{\tilde {p} - 1} D ^ {- \tilde {p} + 1}. (37) \\ \end{array}
$$
The constant  $C_1$  is the uniform bound on  $m^{-p\eta}\phi_m$ , and  $C_p$  is the parameter in Definition 1.
Choosing  $D = Ct^{\frac{1}{p}}\rho_{\mathcal{Z}}^{\frac{\alpha}{p}}(\log (t))^{-\frac{1}{p}}$ , with constant  $C = (\frac{C_1^2C_p}{(\tilde{p} - 1)\lambda^2})^{\frac{1}{p}}$  we obtain
$$
\Gamma_ {k, \lambda} (t) \leq \frac {C}{2} t ^ {\frac {1}{\bar {p}}} \rho_ {\mathcal {Z}} ^ {\frac {\alpha}{\bar {p}}} \left(\log (t) ^ {- \frac {1}{\bar {p}}} \log \left(1 + \frac {t}{\lambda^ {2} D}\right) + (\log (t)) ^ {1 - \frac {1}{\bar {p}}}\right), \tag {38}
$$
that completes the proof.
C.2 Proof of Lemma 3 on Covering Number of State-Action Value Function Class
Recall the definition of the state-action value function class,
$$
\mathcal {Q} _ {k, h} (R, B) = \left\{Q: Q (z) = \min  \left\{Q _ {0} (z) + \beta b (z), H - h + 1 \right\}, \| Q _ {0} \| _ {\mathcal {H} _ {k}} \leq R, \beta \leq B, | Z | \leq T \right\}.
$$
and the notation  $\mathcal{N}_{k,h}(\epsilon; R, B)$  for its  $\epsilon$ -covering number. Let us use the notation  $\mathcal{N}_{k,R}(\epsilon)$  for the  $\epsilon$ -covering number of RKHS ball  $\mathcal{B}_{k,R} = \{f: \| f \|_{\mathcal{H}_k} \leq R\}$ ,  $\mathcal{N}_{[0,B]}(\epsilon)$  for the  $\epsilon$ -covering number of interval  $[0, B]$  with respect to Euclidean distance, and  $\mathcal{N}_{k,\pmb{b}}(\epsilon)$  for the  $\epsilon$ -covering number of class of uncertainty functions  $\pmb{b}_k = \{b(z) = (k(z,z) - k_Z^\top(z))(K_Z + \lambda^2 I)^{-1}k_Z(z))^{\frac{1}{2}}, |Z| \leq T\}$ .
Consider  $Q, \overline{Q} \in \mathcal{Q}_{k,h}(R,B)$  where  $Q(z) = \min \left\{Q_0(z) + \beta b(z), H - h + 1\right\}$  and  $\overline{Q}(z) = \min \left\{\overline{Q}_0(z) + \bar{\beta}\bar{b}(z), H - h + 1\right\}$ . We have
$$
\left| Q (z) - \bar {Q} (z) \right| \leq \left| Q _ {0} (z) - \bar {Q} _ {0} (z) \right| + | \beta - \bar {\beta} | + B | b (z) - \bar {b} (z) |. \tag {39}
$$
$$
\mathcal {N} _ {k, h} (\epsilon ; R, B) \leq \mathcal {N} _ {k, R} \left(\frac {\epsilon}{3}\right) \mathcal {N} _ {[ 0, B ]} \left(\frac {\epsilon}{3}\right) \mathcal {N} _ {k, b} \left(\frac {\epsilon}{3 B}\right). \tag {40}
$$
For the  $\epsilon$ -covering number of the  $[0, B]$  interval, we simply have  $\mathcal{N}_{[0, B]}(\epsilon / 3) \leq 1 + 3B / \epsilon$ . In the next lemmas, we bound the  $\epsilon$ -covering number of the RKHS ball and the class of uncertainty functions.
Lemma 5 (RKHS Covering Number) Consider a positive definite kernel  $k:\mathcal{Z}\times \mathcal{Z}\to \mathbb{R}$ , with polynomial eigendecay on a hypercube with side length  $\rho_{\mathcal{Z}}$ . The  $\epsilon$ -covering number of  $R$ -ball in the RKHS satisfies
$$
\log \mathcal {N} _ {k, R} (\epsilon) = \mathcal {O} \left(\left(\frac {R ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{p - 1}} \log \left(1 + \frac {R}{\epsilon}\right)\right). \tag {41}
$$
Lemma 6 (Uncertainty Class Covering Number) Consider a positive definite kernel  $k: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$ , with polynomial eigendecay on a hypercube with side length  $\rho_{\mathcal{Z}}$ . The  $\epsilon$ -covering number of the class of uncertainty functions satisfies
$$
\log \mathcal {N} _ {k, b} (\epsilon) = \mathcal {O} \left(\left(\frac {\rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {2}{p - 1}} \left(1 + \log \left(\frac {1}{\epsilon}\right)\right)\right) \tag {42}
$$
Combining (40) with Lemmas 5 and 6, we obtain
$$
\log \mathcal {N} _ {k, h} (\epsilon ; R, B) = \mathcal {O} \left(\left(\frac {R ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{\bar {p} - 1}} \left(1 + \log \left(\frac {R}{\epsilon}\right)\right) + \left(\frac {B ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {2}{\bar {p} - 1}} \left(1 + \log \left(\frac {B}{\epsilon}\right)\right)\right), \tag {43}
$$
that completes the proof of Lemma 3. Next, we provide the proof of two lemmas above on the covering numbers of the RKHS ball and the uncertainty function class.
Proof 1 (Proof of Lemma 5) For  $f$  in the RKHS, recall the following representation
$$
f = \sum_ {m = 1} ^ {\infty} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m}, \tag {44}
$$
as well as its projection on the  $D$ -dimensional RKHS
$$
\Pi_ {D} [ f ] = \sum_ {m = 1} ^ {D} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m}. \tag {45}
$$
$$
\begin{array}{l} \| f - \Pi_ {D} [ f ] \| _ {\infty} = \sum_ {m = D + 1} ^ {\infty} w _ {m} \sqrt {\sigma_ {m}} \phi_ {m} \\ \leq C _ {1} C _ {p} ^ {\frac {1}{2}} \rho_ {\mathcal {Z}} ^ {\alpha / 2} \sum_ {m = D + 1} ^ {\infty} | w _ {m} | m ^ {- p (\frac {1}{2} - \eta)} \\ \leq C _ {1} C _ {p} ^ {\frac {1}{2}} \rho_ {\mathcal {Z}} ^ {\alpha / 2} \left(\sum_ {m = D + 1} ^ {\infty} | w _ {m} | ^ {2}\right) ^ {\frac {1}{2}} \left(\sum_ {m = D + 1} ^ {\infty} m ^ {- p (1 - 2 \eta)}\right) ^ {\frac {1}{2}} \\ \leq C _ {1} C _ {\tilde {p}} ^ {\frac {1}{2}} \rho_ {\mathcal {Z}} ^ {\alpha / 2} R \left(\int_ {D} ^ {\infty} x ^ {- \tilde {p}} d x\right) ^ {\frac {1}{2}} \\ { = } { \frac { C _ { 1 } C _ { p } ^ { \frac { 1 } { 2 } } \rho _ { \mathcal { Z } } ^ { \alpha / 2 } R } { \sqrt { \tilde { p } - 1 } } D ^ { \frac { - \tilde { p } + 1 } { 2 } } . } \\ \end{array}
$$
In the expressions above,  $C_1$  is the uniform bound on  $m^{-p\eta}\phi_m$ , and  $C_p$  is the constant specified in Definition 1. The third inequality follows from Cauchy-Schwarz inequality.
Now, let  $D_0$  be the smallest  $D$  such that the right hand side is bounded by  $\frac{\epsilon}{2}$ . There exists a constant  $C_2 > 0$ , only depending on constants  $C_1, C_p, \eta$  and  $\tilde{p}$ , such that
$$
D _ {0} \leq C _ {2} \left(\frac {R ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{\bar {p} - 1}}. \tag {46}
$$
For a  $D$ -dimensional linear model, where the norm of the weights is bounded by  $R$ , the  $\epsilon$ -covering is at most  $C_3 D(1 + \log(\frac{R}{\epsilon})$ , for some constant  $C_3$  (e.g., see, Yang et al., 2020a). Using an  $\epsilon/2$  covering number for the space of  $\Pi_D[f]$  and using the minimum number of dimensions that ensures  $|f - \Pi_D[f]| \leq \epsilon/2$ , we conclude that
$$
\begin{array}{l} \log \mathcal {N} _ {k, R} (\epsilon) \leq C _ {3} D _ {0} \left(1 + \log \left(\frac {R}{\epsilon}\right)\right) \\ \leq C _ {2} C _ {3} \left(\frac {R ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{\bar {p} - 1}} (1 + \log (\frac {R}{\epsilon})), \\ \end{array}
$$
that completes the proof of the lemma.
Proof 2 (Proof of Lemma 6) Let us define  $\pmb{b}_k^2 = \{b^2 : b \in \pmb{b}_k\}$  and  $\mathcal{N}_{k,\pmb{b}^2}(\epsilon)$  to be its  $\epsilon$ -covering number. We note that, for  $b, \bar{b} \in \pmb{b}$ ,
$$
| b (z) - \bar {b} (z) | \leq \sqrt {\left| (b (z)) ^ {2} - (\bar {b} (z)) ^ {2} \right|}. \tag {47}
$$
Thus, an  $\epsilon$ -covering number of  $b$  is bounded by an  $\epsilon^2$ -covering of  $b^2$ :
$$
\mathcal {N} _ {k, \boldsymbol {b}} (\epsilon) \leq \mathcal {N} _ {k, \boldsymbol {b} ^ {2}} \left(\epsilon^ {2}\right). \tag {48}
$$
We next bound  $\mathcal{N}_{k,\pmb{b}^2}(\epsilon^2)$
Using the feature space representation of the kernel, we obtain
$$
(b (z)) ^ {2} = \sum_ {m = 1} ^ {\infty} \gamma_ {m} \sigma_ {m} \phi_ {m} ^ {2} (z), \tag {49}
$$