SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The term  $\widehat{Q}_h^t +\beta b_h^t$  is an upper confidence bound on the state-action value function, that is inspired by the principle of optimism in the face of uncertainty. Since the rewards are assumed to be at most 1, the state-action value function at step  $h$  is also bounded by  $H - h + 1$ . In episode  $t$ , then  $\pi^t$  is the greedy policy with respect to  $Q^{t} = \{Q_{h}^{t}\}_{h = 1}^{H}$ . Under Assumption 1, the estimate  $\widehat{Q}_h^t$ , the parameter  $\beta$  and the exploration bonus  $b_h^t$  can all be designed using kernel ridge regression. Specifically, having the Bellman equation in mind,  $\widehat{Q}_h^t$  is the (kernel ridge) predictor for  $r_h + [P_hV_{h + 1}^t ]$  using (possibly some of) the past  $t - 1$  observations  $\{r_h(z_h^\tau) + V_{h + 1}^t (s_{h + 1}^\tau)\}_{\tau = 1}^{t - 1}$  at points  $\{z_h^\tau \}_{\tau = 1}^{t - 1}$ . Recall that  $\mathbb{E}\left[r_h(z_h^\tau) + V_{h + 1}^t (s_{h + 1}^\tau)\right] = r_h(z_h^\tau) + [P_hV_{h + 1}^t ](z_h^\tau)$ , where the expectation is taken with respect to  $P_{h}(\cdot |z_{h}^{\tau})$ . The observation noise  $V_{h + 1}^{t}(s_{h + 1}^{\tau}) - [P_{h}V_{h + 1}^{t}] (z_{h}^{\tau})$  is due to random transitions and is bounded by  $H - h\leq H$ .
3.1 Domain Partitioning
To overcome the suboptimal performance guarantees rooted in the online confidence intervals in kernel ridge regression, we introduce domain partitioning kernel ridge regression based least-squares value iteration (π-KRVI). The proposed policy partitions the state-action space  $\mathcal{Z}$  into subdomains and builds kernel ridge regression only based on the observations within each subdomain. By doing so, we obtain tighter confidence intervals, ultimately resulting in a tighter regret bound. To formalize this procedure, we consider the state-action space  $\mathcal{Z} \subset [0,1]^d$ . Let  $S_h^t$ ,  $h \in [H]$ ,  $t \in [T]$  be sets of hypercubes overlapping only at edges, covering the entire  $[0,1]^d$ . For any hypercube  $\mathcal{Z}' \in S_h^t$ , we use  $\rho_{\mathcal{Z}'}$  to denote the length of any of its sides, and  $N_h^t(\mathcal{Z}')$  to denote the number of observations at step  $h$  in  $\mathcal{Z}'$  up to episode  $t$ :
$$
N _ {h} ^ {t} \left(\mathcal {Z} ^ {\prime}\right) = \sum_ {\tau = 1} ^ {t} \mathbf {1} \left\{\left(s _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \in \mathcal {Z} ^ {\prime} \right\}. \tag {13}
$$
For all  $h \in [H]$ , we initialize  $S_h^1 = \{[0,1]^d\}$ . At each episode  $t$ , for each step  $h$ , after observing a sample from  $r_h + [P_h V_{h+1}^t]$  at  $(s_h^t, a_h^t)$ , we construct a new cover  $S_h^t$  as follows. We divide every element  $\mathcal{Z}' \in S_h^{t-1}$  that satisfies  $\rho_{\mathcal{Z}'}^{-\alpha} < |N_h^t(\mathcal{Z}')| + 1$ , into two equal halves along each side, generating  $2^d$  hypercubes. The parameter  $\alpha > 0$  in the splitting rule is a constant specified in Definition 1. Subsequently, we define  $S_h^t$  as the set of newly created hypercubes and the elements of  $S_h^{t-1}$  that were not split.
The construction of the cover sets described above ensures the number  $N_h^t(\mathcal{Z}')$  of observations within each cover element  $\mathcal{Z}'$  remains relatively small with respect to the size of  $\mathcal{Z}'$ , while also controlling the total number  $|S_h^t|$  of cover elements. The key parameter managing this tradeoff is  $\alpha$ , which is carefully chosen to achieve an appropriate width for the confidence interval, as shown in Section 4.
3.2  $\pi$ -KRVI
Our proposed policy,  $\pi$ -KRVI, is derived by adopting the precise structure of an optimistic LSVI, as described previously, where the predictor and the exploration bonus are designed based on kernel ridge regression only on cover elements. In particular, for  $z \in \mathcal{Z}$ , let  $\mathcal{Z}_h^t(z) \in \mathcal{S}_h^t$  be the cover element at step  $h$  of episode  $t$  containing  $z$ . Define  $Z_h^t(z) = \{(s_h^\tau, a_h^\tau) \in \mathcal{Z}_h^t(z), \tau < t\}$  to be the set of past observations belonging to the same cover element as  $z$ . We then set
$$
\widehat {Q} _ {h} ^ {t} (z) = k _ {Z _ {h} ^ {t} (z)} ^ {\top} (z) \left(K _ {Z _ {h} ^ {t} (z)} + \lambda^ {2} I\right) ^ {- 1} Y _ {Z _ {h} ^ {t} (z)}, \tag {14}
$$
where  $k_{Z_h^t (z)} = [k(z,z')]_{z'\in Z_h^t (z)}^\top$  is the kernel values between  $z$  and all observations  $z^{\prime}$  in  $Z_h^t (z)$ ,  $K_{Z_h^t (z)} = [k(z',z'')]_{z',z''\in Z_h^t (z)}$  is the kernel matrix for observations in  $Z_h^t (z)$ , and  $Y_{Z_h^t (z)} =$
$[r_h(z') + V_{h+1}^t(s_{h+1}')]_{z' \in Z_h^t(z)}^\top$ , where  $s_{h+1}'$  is drawn from the transition distribution  $P_h(\cdot | z')$ , denotes the observation values for the observation points  $z' \in Z_h^t(z)$ . The vectors  $k_{Z_h^t(z)}$  and  $Y_{Z_h^t(z)}$  are  $N_h^{t-1}(\mathcal{Z}_h^t(z))$  dimensional column vectors, and  $K_{Z_h^t(z)}$  and  $I$  are  $N_h^{t-1}(\mathcal{Z}_h^t(z)) \times N_h^{t-1}(\mathcal{Z}_h^t(z))$  dimensional matrices.
The exploration bonus is determined based on the uncertainty estimate of the kernel ridge regression model on cover elements defined as
$$
b _ {h} ^ {t} (z) = \left(k (z, z) - k _ {Z _ {h} ^ {t} (z)} ^ {\top} (z) \left(K _ {Z _ {h} ^ {t} (z)} + \lambda^ {2} I\right) ^ {- 1} k _ {Z _ {h} ^ {t} (z)} (z)\right) ^ {\frac {1}{2}}. \tag {15}
$$
The policy  $\pi$ -KRVI then is the greedy policy with respect to
$$
Q _ {h} ^ {t} (z) = \min  \left\{\widehat {Q} _ {h} ^ {t} (z) + \beta_ {T} (\delta) b _ {h} ^ {t} (z), H - h + 1 \right\}. \tag {16}
$$
Specifically, at step  $h$  of episode  $t$ , the following action is chosen, after observing  $s_h^t$ ,
$$
a _ {h} ^ {t} = \underset {a \in \mathcal {A}} {\arg \max } Q _ {h} ^ {t} \left(s _ {h} ^ {t}, a\right). \tag {17}
$$
A pseudocode is provided in Algorithm 1.
Algorithm 1 The  $\pi$ -KRVI Policy.
1: Input:  $\lambda, \beta_{T}(\delta), k, M = (\mathcal{S}, \mathcal{A}, H, P, r)$ .  
2: For all  $h \in [H]$ , let  $\mathcal{S}_h^1 = \{[0,1]^d\}$ .  
3: for Episode  $t = 1, 2, \ldots, T$ , do  
4: Receive the initial state  $s_1^t$ .  
5: Set  $V_{H+1}^t(s) = 0$ , for all  $s$ .  
6: for step  $h = H, \ldots, 1$  do  
7: Obtain value functions  $Q_h^t(z)$  as in (16).  
8: end for  
9: for step  $h = 1, 2, \ldots, H$  do  
10: Take action  $a_h^t \gets \arg \max_{a \in \mathcal{A}} Q_h^t(s_h^t, a)$ .  
11: Observe the reward  $r_h(s_h^t, a_h^t)$  and the next state  $s_{h+1}^t$ .  
12: Split any element  $\mathcal{Z}' \in S_h^{t-1}$ , for which  $\rho_{\mathcal{Z}'}^{-\alpha} < |N_h^t(\mathcal{Z}')| + 1$  along the middle of each side, and obtain  $S_h^t$ .  
13: end for  
14: end for
The predictor  $\widehat{Q}_h^t$ , the confidence interval width multiplier  $\beta_T(\delta)$  and the exploration bonus  $b_h^t$  are all designed using kernel ridge regression limited to the observations within cover elements given above. The parameter  $\beta_T(\delta)$ , in particular, is designed in a way that  $Q_h^t$  is a  $1 - \delta$  upper confidence bound on  $r_h + [P_hV_{h + 1}^t]$ . Using Theorem 1 on the confidence intervals, we show that a choice of  $\beta_T(\delta) = \Theta (H\sqrt{\log(\frac{T H}{\delta})})$  satisfies this requirement.
Figure 1 demonstrates the domain partitioning used in  $\pi$ -KRVI on a 2-dimensional domain. The colors represent the value of the target function. The observation points are expected to concentrate around the areas where the target function has a high value. As a result the domain is partitioned to smaller squares in that region.
Runtime complexity. The  $\pi$ -KRVI policy is also runtime efficient with a polynomial runtime complexity. In particular, an upper bound on the runtime of  $\pi$ -KRVI is  $\mathcal{O}(HT^4 + H|\mathcal{A}|T^3)$ , that is similar to KOVI (Yang et al., 2020a). However, analogous to (Janz et al., 2020), we expect an improved runtime for  $\pi$ -KRVI in practice. In addition, the runtime can further improve in terms of  $T$ , utilizing sparse approximations of kernel ridge predictor and uncertainty estimate (e.g., see, Vakili et al., 2022). The dependency of the runtime on  $|\mathcal{A}|$  is due to the step given in Equation (17). If this optimization can be done efficiently over continuous domains,  $\pi$ -KRVI (also KOVI) could handle infinite number of actions. The assumption that the upper confidence bound index can be efficiently optimized over continuous domains is often made in the kernelized bandits (e.g., see, Srinivas et al., 2010a).
Figure 1: A 2-dimensional domain partitioned into smaller squares.
4 Main Results and Regret Analysis
In this section, we present our main results. In Theorem 2, we establish an  $\tilde{\mathcal{O}}(H^2 T^{\frac{d + \alpha / 2}{d + \alpha}})$  regret bound for  $\pi$ -KRVI, for the class of kernels with polynomial eigendecay. We first prove bounds on maximum information gain and covering number of state-action value function class. Those enable us to present our uniform confidence interval for state-action value functions (Theorem 1), and subsequently the regret bound (Theorem 2).
Definition 1 (Polynomial Eigendecay) Consider the Mercer eigenvalues  $\{\sigma_m\}_{m=1}^{\infty}$  of  $k: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$ , given in Equation (7), in a decreasing order, as well as the corresponding eigenfeatures  $\{\phi_m\}_{m=1}^{\infty}$ . Assume  $\mathcal{Z}$  is a  $d$ -dimensional hypercube with side length  $\rho_{\mathcal{Z}}$ . For some  $C_p, \alpha > 0, p > 1$ , the kernel  $k$  is said to have a polynomial eigendecay, if for all  $m \in \mathbb{N}$ ,  $\sigma_m \leq C_p m^{-p} \rho_{\mathcal{Z}}^\alpha$ . In addition, for some  $\eta \geq 0$ ,  $m^{-p\eta} \phi_m(z)$  is uniformly bounded over all  $m$  and  $z$ . We use the notation  $\tilde{p} = p(1 - 2\eta)$ .
The polynomial eigendecay profile encompasses a large class of common kernels, e.g., the Matérn family of kernels. For a Matérn kernel with smoothness parameter  $\nu$ ,  $p = \frac{2\nu + d}{d}$  and  $\alpha = 2\nu$  (e.g., see, Janz et al., 2020). Another example is the NT kernel (Arora et al., 2019). It has been shown that the RKHS of the NT kernel, when the activations are  $s - 1$  times differentiable, is equivalent to the RKHS of a Matérn kernel with smoothness  $\nu = s - \frac{1}{2}$  (Vakili et al., 2021b). For instance, the RKHS of an NT kernel with ReLU activations is equivalent to the RKHS of a Matérn kernel with  $\nu = \frac{1}{2}$  (also known as the Laplace kernel). In this case,  $p = 1 + \frac{1}{d}$  and  $\alpha = 1$ . The hypercube domain assumption is a technical formality that can be relaxed to other regular compact subsets of  $\mathbb{R}^d$ . The uniform boundedness of  $m^{-p\eta}\phi_m(z)$  for some  $\eta > 0$ , also holds for a broad class of kernels, including the Matérn family, as discussed in (Yang et al., 2020a). Several works including (Vakili et al., 2021b; Kassraie and Krause, 2022), have employed an averaging technique over subsets of eigenfeatures, demonstrating that, for the bounds on information gain, the effective value of  $\eta$  can be considered as 0 in the case of Matérn and NT kernels.
4.1 Confidence Intervals for State-Action Value Functions
Confidence intervals are an important building block in the design and analysis of bandit and RL algorithms. For a fixed function  $f$  in the RKHS of a known kernel,  $1 - \delta$  confidence intervals of the form  $|f(z) - \mu^{t,f}(z)| \leq \beta(\delta) b^t(z)$  are established in several works (Srinivas et al., 2010a; Chowdhury and Gopalan, 2017; Abbasi-Yadkori, 2013; Vakili et al., 2021a) under various assumptions. In our setting of interest, however, these confidence intervals cannot be directly applied. This is due to the randomness of the target function itself. Specifically, in our case, the target function is  $r_h + [P_h V_{h+1}^t]$ , which is not a fixed function due to the temporal dependence within an episode. An argument based on the covering number of the state-action value function class was used in Yang et al. (2020a) to establish uniform confidence intervals over all  $z \in \mathcal{Z}$  and all  $f$  in a specific function class. In Theorem 1, we prove a different confidence interval that offers flexibility with respect to setting the parameters of the confidence interval. Our approach leads to a more refined confidence interval,
which, with a proper choice of parameters, contributes to the improved regret bound achieved by our policy.
We first give a formal definition of the two complexity terms: maximum information gain and the covering number of the state-action value function class, which appear in our confidence intervals.
Definition 2 (Maximum Information Gain) In the kernel ridge regression setting described in Section 2.2, the following quantity is referred to as maximum information gain:  $\Gamma_{k,\lambda}(t) = \max_{Z^t\subset \mathcal{Z}}\frac{1}{2}\log det(I + \frac{1}{\lambda^2} K_{Z^t})$
Upper bounds on maximum information gain based on the spectrum of the kernel are established in Janz et al. (2020); Srinivas et al. (2010a); Vakili et al. (2021c). Maximum information gain is closely related to the effective dimension of the kernel. While the feature representation of common kernels is infinite dimensional, with a finite observation set, only a finite number of features have a significant impact on kernel ridge regression, that is referred to as the effective dimension. It has been shown that information gain and effective dimension are the same up to logarithmic factors (Calandriello et al., 2019). This observation offers an intuitive understanding of information gain.
State-action value function class: Let us use  $\mathcal{Q}_{k,h}(R,B)$  to denote the class of state-action value functions. In particular for a set of observations  $Z$ , let  $b_{h}(z)$  be the uncertainty estimate obtained from kernel ridge regression as given in (9). We define
$$
\mathcal {Q} _ {k, h} (R, B) = \left\{Q: Q (z) = \min  \left\{Q _ {0} (z) + \beta b _ {h} (z), H - h + 1 \right\}, \| Q _ {0} \| _ {\mathcal {H} _ {k}} \leq R, \beta \leq B, | Z | \leq T \right\}. \tag {18}
$$
Definition 3 (Covering Set and Number) Consider a function class  $\mathcal{F}$ . For  $\epsilon > 0$ , we define the minimum  $\epsilon$ -covering set  $\mathcal{C}(\epsilon)$  as the smallest subset of  $\mathcal{F}$  that covers it up to an  $\epsilon$  error in  $l_{\infty}$  norm. That is to say, for all  $f \in \mathcal{F}$ , there exists a  $g \in \mathcal{C}(\epsilon)$ , such that  $\| f - g \|_{l_{\infty}} \leq \epsilon$ . We refer to the size of  $\mathcal{C}(\epsilon)$  as the  $\epsilon$ -covering number.
We use the notation  $\mathcal{N}_{k,h}(\epsilon; R, B)$  to denote the  $\epsilon$ -covering number of  $\mathcal{Q}_{k,h}(R, B)$ , that appears in the confidence interval.
In Lemmas 2 and 3, we establish bounds on  $\Gamma_{k,\lambda}(t)$  and  $\mathcal{N}_{k,h}(\epsilon;R,B)$ , respectively.
Lemma 2 (Maximum information gain) Consider a positive definite kernel  $k: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$ , with polynomial eigendecay on a hypercube with side length  $\rho_{\mathcal{Z}}$ . The maximum information gain given in Definition 2 satisfies
$$
\Gamma_ {k, \lambda} (T) = \mathcal {O} \left(T ^ {\frac {1}{p}} (\log (T)) ^ {1 - \frac {1}{p}} \rho_ {\mathcal {Z}} ^ {\frac {\alpha}{p}}\right).
$$
Lemma 3 (Covering Number of  $\mathcal{Q}_{k,h}(R,B)$ ) Recall the class of state-action value functions  $\mathcal{Q}_{k,h}(R,B)$ , where  $k: \mathcal{Z} \times \mathcal{Z} \to \mathbb{R}$  satisfies the polynomial eigendecay on a hypercube with side length  $\rho_{\mathcal{Z}}$ . We have
$$
\log \mathcal {N} _ {k, h} (\epsilon ; R, B) = \mathcal {O} \left(\left(\frac {R ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {1}{\bar {p} - 1}} \left(1 + \log \left(\frac {R}{\epsilon}\right)\right) + \left(\frac {B ^ {2} \rho_ {\mathcal {Z}} ^ {\alpha}}{\epsilon^ {2}}\right) ^ {\frac {2}{\bar {p} - 1}} \left(1 + \log \left(\frac {B}{\epsilon}\right)\right)\right).
$$
Our bound on maximum information gain is stronger than the ones presented in Yang et al. (2020a); Janz et al. (2020); Srinivas et al. (2010a) and is similar to the one given in Vakili et al. (2021c), in terms of dependency on  $T$ . Our bound on function class covering number is similar to the one given in Yang et al. (2020a), in terms of dependency on  $T$ . Both Lemmas 2 and 3 given in this work are, however, novel in terms of dependency on the domain size  $\rho_{\mathcal{Z}}$ , and are required for the analysis of our domain partitioning algorithm.
We next present the confidence interval. Proofs are given in the appendix.
Theorem 1 (Confidence Interval) Let  $\widehat{Q}_h^t$  and  $b_h^t$  denote the kernel ridge predictor and uncertainty estimate of  $r_h + [P_hV_{h + 1}^t]$ , using  $t$  observations  $\{V_{h + 1}^t (s_{h + 1}^\tau)\}_{\tau = 1}^t$  at  $Z_{h}^{t} = \{z_{h}^{\tau}\}_{\tau = 1}^{t}\subset \mathcal{Z}$ , where  $s_{h + 1}^{\tau}$  is the next state drawn from  $P_{h}(\cdot |z_{h}^{\tau})$ . Let  $R_{T} = 2H\sqrt{\Gamma_{k,\lambda}(T)}$ . For  $\epsilon, \delta \in (0,1)$ , with probability, at least  $1 - \delta$ , we have,  $\forall z\in \mathcal{Z}, h\in [H]$  and  $t\in [T]$ ,
$$
| r _ {h} (z) + [ P _ {h} V _ {h + 1} ^ {t} ] (z) - \widehat {Q} _ {h} ^ {t} (z) | \leq \beta_ {h} ^ {t} (\delta , \epsilon) b _ {h} ^ {t} (z) + \epsilon ,
$$
where  $\beta_h^t (\delta ,\epsilon)$  is set to any value satisfying
$$
\beta_ {h} ^ {t} (\delta , \epsilon) \geq H + 1 + \frac {H}{\sqrt {2}} \sqrt {\Gamma_ {k , \lambda} (t) + \log \mathcal {N} _ {k , h} (\epsilon ; R _ {T} , \beta_ {h} ^ {t} (\delta , \epsilon)) + 1 + \log \left(\frac {T H}{\delta}\right)} + \frac {3 \sqrt {t} \epsilon}{\lambda}. \tag {19}
$$
4.2 Regret of  $\pi$ -KRVI
A key step in the analysis of  $\pi$ -KRVI is to apply the confidence interval in Theorem 1 to a subdomain  $\mathcal{Z}' \in S_h^t$ . By design of the splitting rule, we can prove that the maximum information gain corresponding to  $\mathcal{Z}'$  satisfies  $\Gamma_{k,\lambda}(N_h^T(\mathcal{Z}')) = \mathcal{O}(\log(T))$ . In addition, we choose  $\epsilon = \frac{H\sqrt{\log(\frac{T H}{\delta})}}{\sqrt{N_h^t(\mathcal{Z}')}}$ , when applying the confidence interval at step  $h$  of episode  $t$  on this subdomain. That ensures  $\log \mathcal{N}_{k,h}(\epsilon; R_{N_h^T(\mathcal{Z}')}, \beta_h^t(\delta, \epsilon)) = \mathcal{O}(\log(T))$ . From these, and by applying a probability union bound over all subdomains  $\mathcal{Z}'$  created in  $\pi$ -KRVI, we can deduce that the choice of  $\beta_T(\delta) = \Theta(H\sqrt{\log(\frac{T H}{\delta})})$  with a sufficiently large constant, satisfies the requirements for confidence interval widths based on Theorem 1. The details are provided in the proof of Theorem 2 in Appendix D. Then, using standard tools from the analysis of optimistic LSVI algorithms, we arrive at the following regret bound.
Theorem 2 (Regret of  $\pi$ -KRVI) Consider the  $\pi$ -KRVI policy described in Section 3.2, with  $\beta_{T}(\delta) = \Theta(H\sqrt{\log(\frac{TH}{\delta})})$  with a sufficiently large constant implied in the  $\Theta$  notation. Under Assumption 1, for kernels given in Definition 1, with probability at least  $1 - \delta$ , the regret of  $\pi$ -KRVI satisfies
$$
\mathcal {R} (T) = \mathcal {O} \left(H ^ {2} T ^ {\frac {d + \alpha / 2}{d + \alpha}} \log (T) \sqrt {\log \left(\frac {H}{\delta}\right)}\right). \tag {20}
$$
The regret bound of  $\pi$ -KRVI presented in Theorem 2 is sublinear in  $T$  when  $\alpha > 0$ , in contrast to the state of the art regret bound in Yang et al. (2020a). The  $\mathcal{O}$  notation used in the expression above hides constants that depend on  $p, \alpha$  and  $d$ . See Appendix D for more details. When specialized to the Matérn family of kernels, replacing  $p = \frac{2\nu + d}{d}$  and  $\alpha = 2\nu$ , the regret bound becomes
$$
\mathcal {R} (T) = \mathcal {O} \left(H ^ {2} T ^ {\frac {\nu + d}{2 \nu + d}} \log (T) \sqrt {\log \left(\frac {H}{\delta}\right)}\right). \tag {21}
$$
In terms of  $T$  scaling, this matches the lower bound for the special case of kernelized bandits (Scarlett et al., 2017), up to a  $\log(T)$  factor.
The analysis of RL algorithms has predominantly focused on simple settings such as tabular or linear MDPs. Several recent studies have considered more general models, including representing the state-action value functions using RKHSs. Notably, the work in Yang et al. (2020a) derives regret bounds for an optimistic LSVI policy. However, the regret bounds in Yang et al. (2020a) are sublinear only when the eigenvalues of the kernel decay rapidly. In this work, we leveraged a domain partitioning technique, a uniform confidence interval for state-action value functions, and bounds on complexity terms based on the domain size to propose  $\pi$ -KRVI, which attains a sublinear regret bound for a general class of kernels. Moreover, our regret bounds match the lower bound derived for Matérn kernels in the special case of kernelized bandits, up to logarithmic factors. It remains an open problem whether the suboptimal regret bounds in the case of standard optimistic LSVI policies (such as KOVI, Yang et al., 2020a) represent a fundamental shortcoming or an artifact of the proof.
Funding Disclosure
This work was funded by MediaTek Research.
Vakili, S., Scarlett, J., Shiu, D.-s., and Bernacchia, A. (2022). Improved convergence rates for sparse approximation methods in kernel-based learning. In International Conference on Machine Learning, pages 21960-21983. PMLR.
Valko, M., Korda, N., Munos, R., Flaounas, I., and Cristianini, N. (2013). Finite-time analysis of kernelised contextual bandits. In Uncertainty in Artificial Intelligence.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019). Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782):350-354.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning. MIT press, Cambridge, MA.
Yang, L. and Wang, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR.
Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. (2020a). Provably efficient reinforcement learning with kernel and neural function approximations. Advances in Neural Information Processing Systems, 33:13903-13916.
Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. I. (2020b). On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622.
Yang, Z., Jin, C., Wang, Z., Wang, M., and Jordan, M. I. (2020c). On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622.
Yao, H., Szepesvári, C., Pires, B. A., and Zhang, X. (2014). Pseudo-MDPs and factored linear action models. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 1-9. IEEE.
Yeh, S.-Y., Chang, F.-C., Yueh, C.-W., Wu, P.-Y., Bernacchia, A., and Vakili, S. (2023). Sample complexity of kernel-based q-learning. In International Conference on Artificial Intelligence and Statistics, pages 453-469. PMLR.
Zanette, A., Brandfonbrener, D., Brunskill, E., Pirotta, M., and Lazaric, A. (2020a). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics, pages 1954-1964. PMLR.
Zanette, A., Lazaric, A., Kochenderfer, M., and Brunskill, E. (2020b). Learning near optimal policies with low inherent Bellman error. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10978-10989. PMLR.
A Mercer Theorem and the RKHSs