SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof Sketch for Theorem 1. Let  $(\varphi_{n})_{n\geq 1}$  be an orthonormal basis for  $H$ , and, for any  $N\geq 1$ , let  $\pi_N$  denote the projection operator onto  $\overline{H}_N\coloneqq \operatorname {span}\{\varphi_1,\ldots ,\varphi_N\}$ . Note that the projected process  $(\pi_N S_t)_{t\geq 1}$  is an  $H$ -valued martingale with respect to  $(\mathcal{F}_t)_{t\geq 0}$ . Further, note that the projected variance process  $(\pi_N V_t\pi_N^\top)_{t\geq 0}$  satisfies
$$
\pi_ {N} V _ {t} \pi_ {N} ^ {\top} = \sum_ {s = 1} ^ {t} \left(\pi_ {N} f _ {s}\right) \left(\pi_ {N} f _ {s}\right) ^ {\top}.
$$
Since, for any  $N \geq 1$ ,  $H_N$  is a finite-dimensional Hilbert space, it follows from Lemma 1 that the process  $(M_t^{(N)})_{t \geq 0}$  given by
$$
M _ {t} ^ {(N)} := \frac {1}{\sqrt {\det (\mathrm {i d} _ {H} + \rho^ {- 1} \pi_ {N} V _ {t} \pi_ {N} ^ {\top})}} \exp \left\{\frac {1}{2} \left\| (\rho \mathrm {i d} _ {H} + \pi_ {N} V _ {t} \pi_ {N} ^ {\top}) ^ {- 1 / 2} \pi_ {N} S _ {t} \right\| _ {H} ^ {2} \right\},
$$
is a nonnegative supermartingale with respect to  $(\mathcal{F}_t)_{t\geq 0}$ . One can check that, for any  $t\geq 0$ ,  $M_t^{(N)}\xrightarrow[N\to\infty]{N\to\infty}M_t$ . Thus, Fatou's Lemma implies
$$
\begin{array}{l} \mathbb {E} \left(M _ {t} \mid \mathcal {F} _ {t - 1}\right) = \mathbb {E} \left(\lim  _ {N \rightarrow \infty} \inf  M _ {t} ^ {(N)} \mid \mathcal {F} _ {t - 1}\right) \\ \leq \liminf_ {N \rightarrow \infty} \mathbb {E} \left(M _ {t} ^ {(N)} \mid \mathcal {F} _ {t - 1}\right) \\ \leq \liminf_{N\to \infty}M^{(N)}_{t - 1} \\ = M _ {t - 1}, \\ \end{array}
$$
which proves the first part of the claim. The second part of the claim follows from applying Fact 4 to the defined nonnegative supermartingale and rearranging. See Appendix B for details.
The following corollary specializes Theorem 1 (and thus Corollary 3.5 of Abbasi-Yadkori [1]) to the case where  $H$  is a RKHS and  $f_{t} = k(\cdot, X_{t})$ , for all  $t \geq 1$ . In this special case, we can reframe the above theorem in terms familiar Gram matrix  $K_{t}$ , assuming the quantity is invertible. While we prefer the simplicity and elegance of working directly in the RKHS  $H$  in the sequel, the following corollary allows us to present Theorem 1 in a way that is computationally tractable.
Corollary 1. Let us assume the same setup as Theorem 1, and additionally assume that (a)  $(H,\langle \cdot ,\cdot \rangle_{H})$  is a RKHS associated with some kernel  $k$ , and (b) there is some  $\mathcal{X}$ -valued  $(\mathcal{F}_t)_{t\geq 0}$ -predictable process  $(X_{t})_{t\geq 1}$  such that  $(f_{t})_{t\geq 1} = (k(\cdot ,X_{t}))_{t\geq 1}$ . Then, for any  $\rho >0$  and  $\delta \in (0,1)$ , we have that, with probability at least  $1 - \delta$ , simultaneously for all  $t\geq 0$ ,
$$
\left\| \left(V _ {t} + \rho \mathrm {i d} _ {H}\right) ^ {- 1 / 2} S _ {t} \right\| _ {H} \leq \sigma \sqrt {2 \log \left(\sqrt {\frac {1}{\delta} \det  \left(I _ {t} + \rho^ {- 1} K _ {t}\right)}\right)}.
$$
If, in addition, the Gram matrix  $K_{t} = (k(X_{i},X_{j}))_{i,j\in [t]}$  is invertible, we have the equality
$$
\| (I _ {t} + \rho K _ {t} ^ {- 1}) ^ {- 1 / 2} \epsilon_ {1: t} \| _ {2} = \| (\rho \mathrm {i d} _ {H} + V _ {t}) ^ {- 1 / 2} S _ {t} \| _ {H}.
$$
We prove Corollary 1 in Appendix B. With this reframing of Theorem 1, we compare the concentration results of Theorem 1 (and thus Abbasi-Yadkori [1]) to the following, commonly leveraged result from Chowdhury and Gopalan [5].
Fact 6 (Theorem 1 from Chowdhury and Gopalan [5]). Assume the same setup as Fact 5. Let  $\eta > 0$  be arbitrary, and let  $K_{t} := (k(X_{i},X_{j}))_{i,j\in [t]}$  be the Gram matrix corresponding to observations made by time  $t \geq 1$ . Then, with probability at least  $1 - \delta$ , simultaneously for all  $t \geq 1$ , we have
$$
\left\| \left(\left(K _ {t} + \eta I _ {t}\right) ^ {- 1} + I _ {t}\right) ^ {- 1 / 2} \epsilon_ {1: t} \right\| _ {2} \leq \sigma \sqrt {2 \log \left(\frac {1}{\delta} \sqrt {\det \left((1 + \eta) I _ {t} + K _ {t}\right)}\right)}.
$$
To make comparison with this bound clear, we parameterize the bounds in the above fact in terms of  $\eta > 0$  instead of  $\rho > 0$  to emphasize the following difference: both sides of the bound presented in Theorem 1 shrink as  $\rho$  is increased, whereas both sides of the bound in Fact 6 increase as  $\eta$  grows. Thus, increasing  $\rho$  in Theorem 1 should be seen as decreasing  $\eta$  in the bound of Chowdhury and Gopalan [5]. The bounds in Corollary 1 and Fact 6 coincide when  $\rho = 1$  and  $\eta \downarrow 0$  (per Lemma 1 in Chowdhury and Gopalan [5]), but are otherwise not equivalent for other choices of  $\rho$  and  $\eta$ .
We believe Theorem 1 and Corollary 3.5 of Abbasi-Yadkori [1] to be significantly more usable than the result of Chowdhury and Gopalan [5] for several reasons. First, the aforementioned bounds directly extend the method of mixtures (in particular, Fact 5) to potentially infinite-dimensional Hilbert spaces. This similarity in form allows us to leverage existing analysis of Abbasi-Yadkori et al. [2] to prove our regret bounds, with only slight modifications. This is in contrast to the more cumbersome regret analysis that leverages Fact 6, which is not only more difficult to follow, but also obtains inferior, sometimes super-linear regret guarantees.
Second, we note that Theorem 1 provides a bound that has a simple dependence on  $\rho >0$ . In more detail, directly as a byproduct of the simplified bounds, Theorem 2 offers a regret bound that can readily be tuned in terms of  $\rho$ . Due to their use of a "double mixture" technique in proving Fact 6, Chowdhury and Gopalan [5] essentially wind up with a nested, doubly-regularized matrix  $((K_{t} + \eta I_{t})^{-1} + I_{t})^{-1 / 2}$  with which they normalize the residuals  $\epsilon_{1:t}$ . In particular, this more complicated normalization makes it difficult to understand how varying  $\eta$  impacts regret guarantees, which we find to be essential for proving improved regret guarantees.
We note that the central bound discussed in this section does not provide an improvement in dependence on maximum information gain in the sense hypothesized by Vakili et al. [33]. In particular, the authors hypothesized the possibility of shaving a  $\sqrt{\gamma T}$  multiplicative factor off of self-normalized concentration inequalities in RKHS's. This was shown in a recent work (see Lattimore [18]) to be impossible in general. Instead, Theorem 1 and Corollary 3.5 of Abbasi-Yadkori [1] give one access to a family of bounds parameterized by the regularization parameter  $\rho > 0$ . As will be seen in the sequel, by optimizing over this parameter, one can obtain significant improvements in regret.
4 An Improved Regret Analysis of GP-UCB
In this section, we provide the second of our main contributions, which is an improved regret analysis for the GP-UCB algorithm. We provide a description of GP-UCB in Algorithm 1. While we state the algorithm directly in terms of quantities in the RKHS  $H$ , these quantities can be readily converted to those involving Gram matrices or Gaussian processes for those who prefer that perspective [5, 38].
As seen in Section 3, by carefully extending the "method of mixtures" technique (originally by Robbins) of Abbasi-Yadkori et al. [2], Abbasi-Yadkori [1] and de la Peña et al. [7, 8] to Hilbert spaces, we can construct self-normalized concentration inequalities that have simple dependence on the regularization parameter  $\rho$ . These simplified bounds, in conjunction with information about the eigendecay of the kernel  $k$  [32], can be combined to carefully choose  $\rho$  to obtain improved regret. We now present our main result.
Theorem 2. Let  $T > 0$  be a fixed time horizon,  $\rho > 0$  a regularization parameter, and assume Assumptions 2 and 1 hold. Let  $\delta \in (0,1)$ , and for  $t \geq 1$  define
$$
U _ {t} := \sigma \sqrt {2 \log \left(\frac {1}{\delta} \sqrt {\det  (\mathrm {i d} _ {H} + \rho^ {- 1} V _ {t})}\right)} + \rho^ {1 / 2} D.
$$
Algorithm 1 Gaussian Process Upper Confidence Bound (GP-UCB)
Input: Regularization parameter  $\rho >0$  , norm bound  $D$  , confidence bounds  $(U_{t})_{t\geq 1}$  , and time horizon  $T$    
Set  $V_{0}\coloneqq 0,f_{0}\coloneqq 0,\mathcal{E}_{0}\coloneqq \{f\in H:\| f\|_{H}\leq D\}$    
for  $t = 1,\ldots ,T$  do Let  $(X_{t},\widetilde{f_{t}})\coloneqq \arg \max_{x\in \mathcal{X},f\in \mathcal{E}_{t - 1}}\langle f,k(\cdot ,x)\rangle_{H}$  Play action  $X_{t}$  and observe reward  $Y_{t}\coloneqq f^{*}(X_{t}) + \epsilon_{t}$  Set  $V_{t}\coloneqq V_{t - 1} + k(\cdot ,X_{t})k(\cdot ,X_{t})^{\top}$  and  $f_{t}\coloneqq (V_{t} + \rho \mathrm{id}_{H})^{-1}\Phi_{t}^{\top}Y_{1:t}$  Set  $\mathcal{E}_t\coloneqq \left\{f\in H:\left\| (V_t + \rho \mathrm{id}_H)^{1 / 2}(f_t - f)\right\| _H\leq U_t\right\}$
Then, with probability at least  $1 - \delta$ , the regret of Algorithm 1 run with parameters  $\rho, (U_t)_{t \geq 1}$ ,  $D$  satisfies
$$
R _ {T} = O \left(\gamma_ {T} (\rho) \sqrt {T} + \sqrt {\rho \gamma_ {T} (\rho) T}\right),
$$
where in the big-Oh notation above we treat  $\delta, D, \sigma, B,$  and  $L$  as being held constant. If the kernel  $k$  experiences  $(C, \beta)$ -polynomial eigendecay for some  $C > 0$  and  $\beta > 1$ , taking  $\rho = O(T^{\frac{1}{1 + \beta}})$  yields  $R_{T} = \widetilde{O}\left(T^{\frac{3 + \beta}{2 + 2\beta}}\right)^{2}$ , which is always sub-linear in  $T$ .
While we present the above bound with a fixed time-horizon, it can be made anytime by carefully applying a standard doubling argument (see Lattimore and Szepesvári [19], for instance). We specialize the above theorem to the case of the Matérn kernel in the following corollary.
Corollary 2. Definition 3 states that the Matérn kernel with smoothness  $\nu > 1/2$  in dimension  $d$  experiences  $(C, \frac{2\nu + d}{d})$ -eigendecay, for some constnat  $C > 0$ . Thus, GP-UCB obtains a regret rate of  $R_T = \widetilde{O}\left(T^{\frac{\nu + 2d}{2\nu + 2d}}\right)$ .
We note that our regret analysis is the first to show that GP-UCB attains sublinear regret for general kernels experiencing polynomial eigendecay. Of particular import is that Corollary 2 of Theorem 2 yields the first analysis of GP-UCB that implies sublinear regret for the Matérn kernel under general settings of ambient dimension  $d$  and smoothness  $\nu$ . A recent result by Janz [16], using a uniform lengthscale argument, demonstrates that GP-UCB obtains sublinear regret for the specific case of the Matérn family when the parameter  $\nu$  and dimension  $d$  satisfy a uniform boundedness condition independent of scale. Our results are (a) more general, holding for any kernel exhibiting polynomial eigendecay, (b) don't require checking uniform boundedness independent of scale condition, and (c) follow from a simple regularization based argument. In particular, the arguments of Janz [16] require advanced functional analytic and Fourier analytic machinery.
We note that our analysis does not obtain optimal regret, as the theoretically interesting but computationally cumbersome SupKernelUCB algorithm [25, 34] obtains a slightly improved regret bound of  $\widetilde{O}\left(T^{\frac{\beta + 1}{2\beta}}\right)$  for  $(C,\beta)$ -polynomial eigendecay and  $\widetilde{O}\left(T^{\frac{\nu + d}{2\nu + d}}\right)$  for the Matérn kernel with smoothness  $\nu$  in dimension  $d$ . Due to the aforementioned result of Lattimore [18], which shows that improved dependence on maximum information gain cannot be generally obtained in Hilbert space concentration, we believe further improvements on regret analysis for GP-UCB may not possible.
To wrap up this section, we provide a proof sketch for Theorem 2. The entire proof, along with full statements and proofs of the technical lemmas, can be found in Appendix C.
Proof Sketch for Theorem 2. Letting, for any  $t \in [T]$ , the "instantaneous regret" be defined as  $r_t := f^*(x^*) - f^*(X_t)$ , a standard argument yields that, with probability at least  $1 - \delta$ , simultaneously for all  $t \in [T]$ ,
$$
r _ {t} \leq 2 U _ {t - 1} \left\| \left(\rho \mathrm {i d} _ {H} + V _ {t - 1}\right) ^ {- 1 / 2} k (\cdot , X _ {t}) \right\| _ {H}.
$$
2The notation  $\widetilde{O}$  suppresses multiplicative, poly-logarithmic factors in  $T$
A further standard argument using Cauchy-Schwarz and an elliptical potential argument yields
$$
\begin{array}{l} R _ {T} = \sum_ {t = 1} ^ {T} r _ {t} \leq U _ {T} \sqrt {2 T \log \det  (\mathrm {i d} _ {H} + \rho^ {- 1} V _ {T})} \\ = \left(\sigma \sqrt {2 \log \left(\frac {1}{\delta} \sqrt {\det (\mathrm {i d} _ {H} + \rho^ {- 1} V _ {T})}\right)} + \rho^ {1 / 2} D\right) \sqrt {2 T \log \det (\mathrm {i d} _ {H} + \rho^ {- 1} V _ {T})} \\ \leq \left(\sigma \sqrt {2 \log (1 / \delta)} + \sigma \sqrt {2 \gamma_ {T} (\rho)} + \rho^ {1 / 2} D\right) \sqrt {4 T \gamma_ {T} (\rho)} = O \left(\gamma_ {T} (\rho) \sqrt {T} + \sqrt {\rho \gamma_ {T} (\rho) T}\right), \\ \end{array}
$$
which proves the first part of the claim. If, additionally,  $k$  experiences  $(C, \beta)$ -polynomial eigendecay, we know that  $\gamma_T(\rho) = \widetilde{O}\left(\left(\frac{T}{\rho}\right)^{1 / \beta}\right)$  by Fact 2. Setting  $\rho := O(T^{\frac{1}{1 + \beta}})$  thus yields
$$
R _ {T} = O \left(\gamma_ {T} (\rho) \sqrt {T} + \sqrt {\rho \gamma_ {T} (\rho) T}\right) = \widetilde {O} \left(T ^ {\frac {3 + \beta}{2 + 2 \beta}}\right),
$$
proving the second part of the claim.
In this work, we present an improved analysis for the GP-UCB algorithm in the kernelized bandit problem. We provide the first analysis showing that GP-UCB obtains sublinear regret when the underlying kernel  $k$  experiences polynomial eigendecay, which in particular implies sublinear regret rates for the practically relevant Matérn kernel. In particular, we show GP-UCB obtains regret  $\widetilde{O}\left(T^{\frac{3 + \beta}{2 + 2\beta}}\right)$  when  $k$  experiences  $(C,\beta)$ -polynomial eigendecay, and regret  $\widetilde{O}\left(T^{\frac{\nu + 2d}{2\nu + 2d}}\right)$  for the Matérn kernel with smoothness  $\nu$  in dimension  $d$ .
Our contributions are twofold. First, we show the importance of finding the "right" concentration inequality for tackling problems in online learning — in this case the correct bound being a self-normalized inequality originally due to Abbasi-Yadkori [1]. We provide an independent proof of a result equivalent to Corollary 3.5 of Abbasi-Yadkori [1] in Theorem 1, and hope that our simplified, truncation-based analysis will make the result more accessible to researchers working on problems in kernelized learning. Second, we demonstrate the importance of regularization in the kernelized bandit problem. In particular, since the smoothness of the kernel  $k$  governs the hardness of learning, by regularizing in proportion to the rate of eigendecay of  $k$ , one can obtain significantly improved regret bounds.
A shortcoming of our work is that, despite obtaining the first generally sublinear regret bounds for GP-UCB, our rates are not optimal. In particular, there are discretization-based algorithms, such as SupKernelUCB [34], which obtain slightly better regret bounds of  $\widetilde{O}\left(T^{\frac{1 + \beta}{2\beta}}\right)$  for  $(C,\beta)$ -polynomial eigendecay. We hypothesize that the vanilla GP-UCB algorithm, which involves constructing confidence ellipsoids directly in the RKHS  $H$ , cannot obtain this rate.
The common line of reasoning [33] is that because the Lin-UCB (the equivalent algorithm in  $\mathbb{R}^d$ ) obtains the optimal regret rate of  $\widetilde{O}(d\sqrt{T})$  in the linear bandit problem setting, then GP-UCB should attain optimal regret as well. In the linear bandit setting, there is no subtlety between estimating the optimal action and unknown slope vector, as these are one and the same. In the kernel bandit setting, estimating the function and optimal action are not equivalent tasks. In particular, the former serves in essence as a nuisance parameter in estimating the latter: tight estimation of unknown function under the Hilbert space norm implies tight estimation of the optimal action, but not the other way around. Existing optimal algorithms are successful because they discretize the input domain, which has finite metric dimension [26], and make no attempts to estimate the unknown function in RKHS norm. Since compact sets in RKHS's do not, in general, have finite metric dimension [35], this makes estimation of the unknown function a strictly more difficult task. In fact, recent work by Lattimore [18] demonstrate that self-normalized concentration in RKHS's, in general, cannot exhibit improved dependence on maximum information gain. This further supports our hypothesis on the further unimprovability of the regret analysis of GP-UCB past the improvements made in this paper.
AR acknowledges support from NSF DMS-2310718 and NSF IIS-2229881. ZSW and JW were supported in part by the NSF CNS2120667, a CyLab 2021 grant, a Google Faculty Research Award, and a Mozilla Research Grant. JW also acknowledges support from NSF GRFP grants DGE1745016 and DGE2140739.
We also would like to thank Xingyu Zhou and Johannes Kirschner for independently bringing to our attention the result from Abbasi-Yadkori [1] (Corollary 3.5) on self-normalized concentration in Hilbert spaces which is essentially equivalent to Theorem 1. We have rewritten the paper in a way that emphasizes the importance of this result and provides proper attribution to the original author.
[1] Yasin Abbasi-Yadkori. Online learning for linearly parametrized control problems. 2013.
[2] Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems, 24, 2011.
[3] Rajeev Agrawal. The continuum-armed bandit problem. SIAM Journal on Control and Optimization, 33(6):1926–1951, 1995.
[4] Andrew G Barto. Reinforcement learning control. Current Opinion in Neurobiology, 4(6): 888-893, 1994.
[5] Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844–853. PMLR, 2017.
[6] Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of Information Theory, 1(1):279-335, 1991.
[7] Victor de la Peña, Michael J Klass, and Tze Leung Lai. Self-normalized processes: Exponential inequalities, moment bounds and iterated logarithm laws. The Annals of Probability, 32, 07 2004. doi: 10.1214/009117904000000397.
[8] Victor de la Peña, Michael J Klass, and Tze Leung Lai. Pseudo-maximization and self-normalized processes. *Probability Surveys Vol*, 4:172–192, 09 2007. doi: 10.1214/07-PS119.
[9] Victor H de la Peña, Michael J Klass, and Tze Leung Lai. Theory and applications of multivariate self-normalized processes. Stochastic Processes and their Applications, 119(12):4210-4227, 2009.
[10] Audrey Durand, Odalric-Ambrym Maillard, and Joelle Pineau. Streaming kernel regression with provably adaptive mean, variance, and regularization. The Journal of Machine Learning Research, 19(1):650-683, 2018.
[11] Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
[12] Vivek Farias, Ciamac Moallemi, Tianyi Peng, and Andrew Zheng. Synthetically controlled bandits. arXiv preprint arXiv:2202.07079, 2022.
[13] Matthew Hoffman, Eric Brochu, and Nando De Freitas. Portfolio allocation for Bayesian optimization. In UAI, pages 327-336, 2011.
[14] Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales. *Probability Surveys*, 17:257–317, 2020.
[15] Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, non-parametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2), 2021.
[16] David Janz. Sequential decision making with feature-linear models. PhD thesis, 2022.
[17] David Janz, David Burt, and Javier González. Bandit optimisation of functions in the matérn kernel RKHS. In International Conference on Artificial Intelligence and Statistics, pages 2486-2495. PMLR, 2020.