SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 6.1 (Analytic UCB). For all  $\alpha >0$ , we have
$$
\mathrm {U C B} _ {\Theta_ {t}} (a) = \max  _ {\boldsymbol {\theta} \in \Theta_ {t}} \left\{\phi (a) ^ {\top} \boldsymbol {\theta} \right\} \leq \phi (a) ^ {\top} \widehat {\boldsymbol {\theta}} _ {\alpha , t} + R _ {\mathrm {A M M}, t} \sqrt {\phi (a) ^ {\top} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a)}, \tag {8}
$$
where  $\widehat{\pmb{\theta}}_{\alpha ,t} = \left(\Phi_t^\top \Phi_t + \alpha \mathbb{1}\right)^{-1}\Phi_t^\top \pmb {r}_t,$
$$
R _ {\mathrm {A M M}, t} ^ {2} = R _ {\mathrm {M M}, t} ^ {2} + \alpha B ^ {2} - \boldsymbol {r} _ {t} ^ {\top} \boldsymbol {r} _ {t} + \boldsymbol {r} _ {t} ^ {\top} \Phi_ {t} \left(\Phi_ {t} ^ {\top} \Phi_ {t} + \alpha \mathbb {1}\right) ^ {- 1} \Phi_ {t} ^ {\top} \boldsymbol {r} _ {t}.
$$
In App. C.1, we derive this analytic UCB by partial optimisation of the Lagrangian dual function. Using strong duality, one can show that the analytic UCB minimised with respect to  $\alpha$  is equal to  $\mathrm{UCB}_{\Theta_t}(a)$ . Due to the closed-form expression of the analytic UCB in (8), its gradient with respect to  $a$  can be computed using standard automatic differentiation packages.
6.4 Choosing the Mixture Distributions
Both of our algorithms require us to choose mixture distributions  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\pmb{T}_{t})$ . The mixture distributions play a role similar to the priors used in the PAC-Bayes (Shawe-Taylor & Williamson, 1997; McAllester, 1998; Guedj, 2019; Alquier, 2021) and luckiness (Grunwald, 2007, 2023) frameworks. Our confidence sequences and regret bounds are valid for any sequence of adaptive mixture distributions, but (as seen in Eq. (5)) if better/worse mixture distributions are chosen, then our confidence sequences will get smaller/bigger and our regret guarantees will get tighter/looser.
Here, we describe some sensible choices for the mixture distributions. In order for a sequence of Gaussian mixture distributions  $(\mathcal{N}(\pmb{\mu}_t,\pmb{T}_t)|t\in \mathbb{N})$  to be a sequence of adaptive mixture distributions (as defined in Section 5.1), we require: (a)  $\pmb{\mu}_t$  and  $\pmb{T}_t$  can only depend on  $a_1,\dots,a_t$  and  $r_1,\dots,r_{t-1}$ ; (b) the first  $t-1$  elements of  $\pmb{\mu}_t$  must be equal to  $\pmb{\mu}_{t-1}$ ; (c) the upper left  $t-1\times t-1$  block of  $\pmb{T}_t$  must be  $\pmb{T}_{t-1}$ ; (d)  $\pmb{T}_t$  must be positive (semi-)definite. These conditions are all satisfied if we use a mean vector  $\pmb{\mu}_t$  and covariance matrix  $\pmb{T}_t$  of the form
$$
\boldsymbol {\mu} _ {t} = \left[ m \left(a _ {1}\right), m \left(a _ {2}\right), \dots , m \left(a _ {t}\right) \right] ^ {\top}, \quad \boldsymbol {T} _ {t} = \left[ \begin{array}{c c c c} k \left(a _ {1}, a _ {1}\right) & k \left(a _ {1}, a _ {2}\right) & \dots & k \left(a _ {1}, a _ {t}\right) \\ k \left(a _ {2}, a _ {1}\right) & k \left(a _ {2}, a _ {2}\right) & \dots & k \left(a _ {2}, a _ {t}\right) \\ \vdots & \vdots & \ddots & \vdots \\ k \left(a _ {t}, a _ {1}\right) & k \left(a _ {t}, a _ {2}\right) & \dots & k \left(a _ {t}, a _ {t}\right) \end{array} \right], \tag {9}
$$
where  $m:\mathcal{A}\to \mathbb{R}$  is a mean function and  $k:\mathcal{A}\times \mathcal{A}\rightarrow \mathbb{R}$  is a positive-definite kernel function. For the linear bandit problem, it is natural to use a linear mean function  $m(a) = \phi (a)^{\top}\pmb{\theta}_{0}$  and a linear kernel function  $k(a,a^{\prime}) = \phi (a)^{\top}\Sigma_{0}\phi (a^{\prime})$  (where  $\Sigma_0$  is symmetric and positive-definite), since the resulting mixture distribution assigns non-zero probability only to those vectors of function values  $\pmb{f}_t$  that could have been generated by a linear reward function. By direct computation, the Gaussian mixture distribution with this  $m$  and  $k$ , and with  $\pmb{\mu}_t$  and  $\pmb{T}_t$  as in (9), is  $P_{t} = \mathcal{N}(\Phi_{t}\pmb{\theta}_{0},\Phi_{t}\pmb{\Sigma}_{0}\Phi_{t}^{\top})$ . When  $\pmb{\theta}_0 = \mathbf{0}$  and  $\pmb{\Sigma}_0 = \mathbb{1}$ , we recover what we call the standard mixture distributions  $P_{t} = \mathcal{N}(\mathbf{0},\Phi_{t}\Phi_{t}^{\top})$ .
Choosing  $T_{t} = \Phi_{t}\Sigma_{0}\Phi_{t}^{\top}$  has the additional benefit that it allows for cheaper computation of the radius  $R_{\mathrm{MM},t}$ . In App. F, we show that one (only) has to compute the inverse and determinant of a  $d\times d$  matrix instead of the inverse and determinant of the  $t\times t$  matrix  $\mathbb{1} + T_t / \sigma^2$ . Finally, we remark that the requirement that  $(\mathcal{N}(\pmb {\mu}_t,\pmb {T}_t)|t\in \mathbb{N})$  is an adaptive sequence of mixture distributions allows for "more adaptive" choices of  $\pmb{\mu}_{t}$  and  $T_{t}$ . In App E.2, we describe and investigate a method for updating  $\pmb{\mu}_{t}$  and  $T_{t}$  at each round  $t$  based on previously observed actions and rewards.
7 Regret Bounds
In this section, we establish cumulative regret bounds for our CMM-UCB and AMM-UCB algorithms. First, we state a data-dependent regret bound that illustrates how the radius of the analytic UCB from Sec. 6.3 influences the regret of both algorithms. Then, we prove a data-independent regret bound which illustrates the worst-case growth rate of the cumulative regret, with explicit dependence on the feature vector dimension  $d$  and the number of rounds  $T$ . We begin by stating the assumptions (which are standard) under which our regret bounds hold.
Assumption 7.1 (Sub-Gaussian noise). Let  $\mathcal{H}_t$  denote (the  $\sigma$ -algebra generated by)  $(a_1, r_1, \ldots, a_t, r_t, a_{t+1})$ . Each noise variable  $\epsilon_t$  is conditionally zero-mean and  $\sigma$ -sub-Gaussian, which means
$$
\mathbb {E} \left[ \epsilon_ {t} | \mathcal {H} _ {t - 1} \right] = 0, \quad \text {a n d} \quad \forall \lambda \in \mathbb {R}, \mathbb {E} \left[ \exp (\lambda \epsilon_ {t}) | \mathcal {H} _ {t - 1} \right] \leq \exp \left(\lambda^ {2} \sigma^ {2} / 2\right).
$$
Assumption 7.2 (Bounded parameter vector). For some  $B > 0$ ,  $\| \pmb{\theta}^{*}\|_{2} \leq B$
Assumption 7.3 (Bounded feature vectors). For some  $L > 0$ ,  $\| \phi(a) \|_2 \leq L$  for all  $a \in \mathcal{A}$ .
Assumption 7.4 (Bounded expected reward). For some  $C > 0$ ,  $\phi(a)^\top \pmb{\theta}^* \in [-C, C]$  for all  $a \in \mathcal{A}$ .
We remark that to run our algorithms and evaluate the data-dependent regret bound in Thm. 7.5, we only need to know (upper bounds on) the sub-Gaussian parameter  $\sigma$  and the norm bound  $B$ . Assumption 7.2 and Assumption 7.3 together imply that Assumption 7.4 must hold with  $C \leq LB$ . We nevertheless state it as a separate assumption because this leaves open the possibility that a better (than  $LB$ ) value for  $C$  is known.
7.1 Data-Dependent Regret Bounds
Several authors (Dani et al., 2008; Abbasi-Yadkori et al., 2011; Russo & Van Roy, 2013) have shown that the cumulative regret of a UCB algorithm can be upper bounded by the sum of the widths of the confidence sets or confidence bounds that it uses. The width of a confidence set  $\Theta_t$  at the action  $a$  is the difference between the corresponding UCB and the LCB at  $a$  (i.e.,  $\max_{\theta \in \Theta_t} \{\phi(a)^\top \pmb{\theta}\} - \min_{\theta \in \Theta_t} \{\phi(a)^\top \pmb{\theta}\}$ ). In App. D.1, we show that if  $a_1, a_2, \ldots, a_T$  are the actions selected by our CMM-UCB algorithm, then
$$
\sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) \leq \sum_ {t = 1} ^ {T} \max  _ {\theta \in \Theta_ {t - 1}} \left\{\phi \left(a _ {t}\right) ^ {\top} \boldsymbol {\theta} \right\} - \min  _ {\theta \in \Theta_ {t - 1}} \left\{\phi \left(a _ {t}\right) ^ {\top} \boldsymbol {\theta} \right\}. \tag {10}
$$
This gives a data-dependent cumulative regret bound for CMM-UCB. AMM-UCB has a similar data-dependent cumulative regret bound. In App. D.1, we show that if  $a_1, a_2, \ldots, a_T$  are the actions selected by our AMM-UCB algorithm, then
$$
\sum_ {t = 1} ^ {T} \Delta \left(a _ {t}\right) \leq \sum_ {t = 1} ^ {T} \mathrm {A U C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right) - \mathrm {A L C B} _ {\Theta_ {t - 1}} \left(a _ {t}\right), \tag {11}
$$
where  $\mathrm{AUCB}_{\Theta_t}(a)$  is the right-hand-side of (8) and  $\mathrm{ALCB}_{\Theta_t}(a)$  is the equivalent analytic LCB. Since, the analytic UCB/LCB is an upper/lower bound on the numerical UCB/LCB, the bound in Equation (11) also holds for the actions selected by CMM-UCB. By substituting in the expressions for the analytic UCB/LCBs, we obtain the following data-dependent cumulative regret bound for CMM-UCB and AMM-UCB.
Theorem 7.5. Suppose that assumptions 7.1-7.2 hold. For any adaptive sequence of mixture distributions  $P_{t} = \mathcal{N}(\boldsymbol{\mu}_{t},\boldsymbol{T}_{t})$ , any  $\delta \in (0,1)$ , any  $\alpha >0$  and all  $T\geq 1$ , with probability at least  $1 - \delta$ , the cumulative regret of both CMM-UCB and AMM-UCB is bounded by
$$
\Delta_ {1: T} \leq \sum_ {t = 1} ^ {T} 2 R _ {\mathrm {A M M}, t - 1} \sqrt {\phi (a _ {t}) ^ {\top} \left(\Phi_ {t - 1} ^ {\top} \Phi_ {t - 1} + \alpha \mathbb {1}\right) ^ {- 1} \phi (a _ {t})}.
$$
A proof is given in App. D.1. This regret bound tells us that if we choose an adaptive sequence of mixture distributions  $P_{t} = \mathcal{N}(\pmb{\mu}_{t},\pmb{T}_{t})$ , such that the radii  $R_{\mathrm{AMM},t}$  are small, then we can expect to have small cumulative regret.
7.2 Data-Independent Regret Bounds
We now state a data-independent cumulative regret bound for the special case when the adaptive sequence of mixture distributions is  $P_{t} = \mathcal{N}(\mathbf{0}, c\Phi_{t}\Phi_{t}^{\top})$ , and  $\alpha = \sigma^{2}/c$ . These mixture distributions are scaled versions of the standard mixture distributions described in Sec. 6.4.
Theorem 7.6. Suppose that assumptions 7.1-7.4 hold. If for any  $c > 0$ , the sequence of mixture distributions is  $P_{t} = \mathcal{N}(\mathbf{0}, c\Phi_{t}\Phi_{t}^{\top})$ , then for all  $T \geq 1$ , with probability at least  $1 - \delta$ , the cumulative regret of both CMM-UCB and AMM-UCB (with  $\alpha = \sigma^2 / c$ ) is bounded by
$$
\begin{array}{l} \Delta_ {1: T} \leq \frac {2}{\sqrt {\ln 2}} \max  \left\{C, \sigma \sqrt {d \ln \left(1 + \frac {c L ^ {2} T}{\sigma^ {2} d}\right) + \frac {B ^ {2}}{c} + 2 \ln \frac {1}{\delta}} \right\} \sqrt {d T \ln \left(1 + \frac {c L ^ {2} T}{\sigma^ {2} d}\right)} \\ \leq \mathcal {O} (d \sqrt {T} \ln (T)). \\ \end{array}
$$
Proof sketch. Choosing  $P_{t} = \mathcal{N}(\mathbf{0}, c\Phi_{t}\Phi_{t}^{\top})$  and  $\alpha = \sigma^{2}/c$  means that the two quadratic terms in  $R_{\mathrm{AMM},t}^{2}$  cancel out. We then find a data-independent upper bound for the log det term. Following Abbasi-Yadkori et al. (2011), the sum of the norms  $\sqrt{\phi(a_{t})^{\top}(\Phi_{t-1}^{\top}\Phi_{t-1} + \alpha\mathbb{1})^{-1}\phi(a_{t})}$  is upper bounded using an elliptical potential lemma. The result is the data-independent regret bound in the statement of the theorem.
In App. D.2, we give a proof of this special case. In addition, we also treat a more general case when the sequence of mixture distributions is  $P_{t} = \mathcal{N}(\Phi_{t}\pmb{\theta}_{0},\sigma_{0}^{2}\Phi_{t}\Phi_{t}^{\top})$  and  $\alpha$  is any positive number. Using Eq. (5) with  $\pmb{\mu}_t = \Phi_t\pmb{\theta}_0$  and  $T_{t} = \sigma_{0}^{2}\Phi_{t}\Phi_{t}^{\top}$ , one can interpret  $\pmb{\theta}_{0}$  as a guess for  $\pmb{\theta}^{*}$  and  $\sigma_0$  as a guess for the distance between the reward vector  $\boldsymbol{r}_t$  and the prediction  $\Phi_t\pmb{\theta}_0$ .
Focusing on the dependence on  $d$  and  $T$ , this regret bound (and the more general one in App. D.2) is at most  $\mathcal{O}(d\sqrt{T}\ln(T))$ , which matches OFUL and is minimax optimal up to the  $\ln(T)$  factor. If (upper bounds on)  $\sigma^2$ ,  $B$ ,  $L$  and  $C$  are known, then we can evaluate this cumulative regret bound before running the algorithm.
In all our experiments, we set  $\delta = 0.01$ . When using our analytic UCBs (Thm. 6.1), we always choose  $\alpha = \sigma^2$ . Unless stated otherwise, we use the standard mixture distributions  $P_{t} = \mathcal{N}(\mathbf{0}, \Phi_{t}\Phi_{t}^{\top})$ .
Figure 2: Average confidence bound width for different data set sizes  $T$  and feature dimensions  $d$ .
8.1 Upper and Lower Confidence Bounds
Compared Methods. We evaluate the following upper/lower confidence bounds: (a) CMM-UCB: our numerical UCBs/LCBs from Sect. 6.2; (b) AMM-UCB: our analytic UCBs/LCBs from Thm. 6.1; (c) OFUL: the UCBs/LCBs used by the OFUL algorithm (Abbasi-Yadkori et al., 2011); (d) Bayes: a Bayesian credible interval constructed from the Bayesian posterior for linear regression with a Gaussian prior and likelihood (see App. E.1 for details).
Experimental Setup. We conduct experiments on randomly generated linear functions of the form  $f(\pmb{x}) = \phi(\pmb{x})^{\top} \pmb{\theta}^{*}$ , with inputs  $\pmb{x} \in \mathbb{R}^{d_x}$  and  $\pmb{\theta}^{*} \in \mathbb{R}^{d}$ , the latter drawn from a standard Gaussian distribution and if necessary scaled down to  $\| \pmb{\theta}^{*} \|_2 \leq 10 =: B$ . For the feature map  $\phi: \mathbb{R}^{d_x} \to \mathbb{R}^d$ , we use Random Fourier Features (cf. Algorithm 1 of (Rahimi & Recht, 2007)). We investigate the properties of upper and lower confidence bounds constructed from random data sets  $\{(x_t, y_t)\}_{t=1}^T$ , where  $y_t = \phi(x_t)^{\top} \pmb{\theta}^{*} + \epsilon_t, \epsilon_t \sim \mathcal{N}(0, \sigma^2)$  and  $\sigma = 0.1$ .
UCB/LCB Tightness. Fig. 1 shows the data  $\{(x_{t},y_{t})\}_{t = 1}^{T}$  and the UCBs/LCBs of CMM-UCB, AMM-UCB and OFUL for a randomly generated linear function with  $d_{\mathcal{X}} = 1$  and  $d = 20$ . In this example, the confidence bounds of CMM-UCB are slightly tighter than those of AMM-UCB, which are considerably tighter than those of OFUL. Next, we investigate the tightness of the confidence bounds for functions with higher dimensional inputs ( $d_{\mathcal{X}} = 10$ ), a range of data set sizes ( $T\in \{1,2,5,10,20,50,100,200,500,1000\}$ ) and a range feature vector dimensions ( $d\in \{1,2,5,10,20,50,100\}$ ). For each  $T$  and  $d$ , we sample a random feature map  $\phi$  and weight vector  $\theta$  of appropriate size. Then, we sample random training data  $\{(x_{t},y_{t})\}_{t = 1}^{T}$  and random test points  $\{\pmb{x}_t'\}_{t = 1}^{100}$ , where  $\pmb{x}_k$  and  $\pmb{x}_t'$  are drawn uniformly from the  $d_{\mathcal{X}}$ -dimensional unit hypercube. Finally, we use the training data to construct confidence bounds with each method and calculate the average width (UCB minus LCB) at the test points. Fig. 2 shows the average width of the CMM-UCB, AMM-UCB and OFUL confidence bounds with:  $d = 10$  and varying  $T$  (left), and  $T = 100$  and varying  $d$  (right). We observe the same pattern at every  $d$  and  $T$ : CMM-UCB produces the tightest confidence bounds followed by AMM-UCB and then OFUL.
Effect of the Mixture Distributions. Fig. 4 in App. E.1 shows the confidence bounds of CMM-UCB and a Bayesian credible interval for different choices of the mixture distributions/prior  $P_{t}$ . For a well-specified prior, either uninformative or informative, the Bayesian credible interval is slightly tighter than the confidence bounds of CMM-UCB. For a misspecified prior, the confidence bounds of CMM-UCB become looser whereas the Bayesian credible interval becomes wrong (not containing the ground-truth function). Here, misspecification refers solely to Bayesian prior misspecification. In Figs. 5 and 6 in App. E.2 we show that adaptive choices of the mixture distributions  $P_{t}$  can lead to smaller confidence bounds and smaller cumulative regret in linear bandit problems.
8.2 Linear Bandits
Compared Methods. We compare: (a) CMM-UCB: cf. Sec. 6.2; (b) AMM-UCB: cf. Sec. 6.3; (c) OFUL: the OFUL algorithm (Abbasi-Yadkori et al., 2011), with regularisation parameter  $\lambda = \alpha = \sigma^2$ ; (d) IDS: the frequentist Information Directed Sampling (IDS) algorithm (Kirschner & Krause, 2018), specifically the DIDS-F version; (e) Freq-TS: Thompson Sampling with posterior covariance inflation (Agrawal & Goyal, 2013), which we call Frequentist Thompson Sampling.
Table 1: Average test accuracy and maximum test accuracy of our UCB algorithms and OFUL, IDS and Freq-TS in the SVM hyperparameter tuning problems after  $T = 500$  rounds (100 repetitions).
Raisin	Maternal	Banknotes
Mean Acc	Max Acc	Mean Acc	Max Acc	Mean Acc	Max Acc
CMM-UCB (Ours)	0.818 ± 0.018	0.893 ± 0.019	0.744 ± 0.020	0.829 ± 0.023	0.954 ± 0.005	1.000 ± 0.000
AMM-UCB (Ours)	0.800 ± 0.017	0.892 ± 0.020	0.736 ± 0.020	0.829 ± 0.023	0.948 ± 0.005	1.000 ± 0.000
OFUL	0.764 ± 0.019	0.891 ± 0.019	0.722 ± 0.019	0.827 ± 0.022	0.929 ± 0.006	1.000 ± 0.000
IDS	0.706 ± 0.048	0.891 ± 0.020	0.714 ± 0.019	0.827 ± 0.024	0.926 ± 0.007	1.000 ± 0.000
Freq-TS	0.527 ± 0.022	0.884 ± 0.019	0.616 ± 0.018	0.823 ± 0.022	0.808 ± 0.012	1.000 ± 0.000
Figure 3: The smoothed per-round expected reward (test accuracy) of our UCB algorithms compared with OFUL, IDS and Freq-TS in the SVM hyperparameter tuning experiments on three datasets. Shown is the mean reward over 100 runs of each experiment, after Gaussian kernel smoothing.
**Experimental Setup.** We investigate whether our tighter upper confidence bounds translate to better UCB algorithms. We use each linear bandit algorithm to optimise the hyperparameters of a kernel Support Vector Machine (SVM) for three classification data sets from the UCI Machine Learning Repository (Dua & Graff, 2017): Raisin (Cinar et al., 2020), Maternal (Ahmed et al., 2020), and Banknotes. The expected reward function  $f^{*}(a)$  is the average test set accuracy of a kernel SVM trained using an ARD RBF kernel with hyperparameters  $a = (C, \gamma)$ , with  $C$  the regularisation and  $\gamma$  the length-scales. The observed reward  $r_t$  is the validation set accuracy at  $a_t$ . The feature map  $\phi$  is a neural network layer with 20 outputs and random weights. We choose  $\sigma = 0.05$  for the sub-Gaussian parameter and  $B = 10$ , i.e. we assume that  $f^{*}(a) \approx \phi(a)^{\top} \pmb{\theta}^{*}$  for some  $\| \pmb{\theta}^{*} \|_2 \leq 10$ .
Results. Fig. 3 compares the average test accuracy (expected reward) obtained by each bandit algorithm for each data set and at each round  $t = 1, \ldots, 500$ . Our CMM-UCB and AMM-UCB methods outperform all other methods. From the reward curves of CMM-UCB, AMM-UCB and OFUL, we observe that CMM-UCB outperforms AMM-UCB, which outperforms OFUL. Therefore, we can conclude that our tighter confidence bounds (compared to OFUL's) lead to UCB algorithms with improved performance.
In this paper, we developed a novel tail bound for adaptive martingale mixtures and showed that it can be used to construct tighter confidence sequences for linear bandits. We proved that our CMM-UCB and AMM-UCB algorithms match the worst-case cumulative regret of OFUL, and we found that our tighter confidence sequences allowed CMM-UCB and AMM-UCB to achieve greater average and maximum reward in several hyperparameter tuning problems.
A limitation of our algorithms is that they assume a linear expected reward function, which may not always be a realistic assumption for real-world bandit problems. Our general-purpose tail bound in Thm. 5.1 already allows one to derive confidence sequences for non-linear reward functions by simply choosing  $Z_{t}(f_{t}) = (f_{t} - f^{*}(a_{t}))\epsilon_{t}$ , where  $f^{*}$  is the non-linear reward function. In the case where  $f^{*}$  lies in a reproducing kernel Hilbert space, the corresponding UCB is still the solution of a convex program. The main challenge in this setting is that the regret bound must now depend on quantities like the effective dimension or the maximum information gain of the kernel, since the dimension  $d$  of the feature vectors is  $d = \infty$  for interesting kernels.
We believe that further investigation into the degree to which adaptive mixture distributions can lead to improved performance and regret bounds (see App. E.2) is another exciting topic for future work.
Rusmevichientong, P. and Tsitsiklis, J. N. Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395-411, 2010.
Russo, D. and Van Roy, B. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
Salgia, S., Vakili, S., and Zhao, Q. A domain-shrinking based Bayesian optimization algorithm with order-optimal regret performance. Advances in Neural Information Processing Systems, 34: 28836-28847, 2021.
Shawe-Taylor, J. and Williamson, R. C. A PAC analysis of a Bayesian estimator. In Proceedings of the tenth annual conference on Computational learning theory, pp. 2-9, 1997.
Sherman, J. and Morrison, W. J. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 21(1):124-127, 1950.
Srinivas, N., Krause, A., Kakade, S., and Seeger, M. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. International Conference on Machine Learning (ICML), 2010.
Valko, M., Korda, N., Munos, R., Flaounas, I., and Cristianini, N. Finite-Time Analysis of Kernelised Contextual Bandits. In Uncertainty in Artificial Intelligence, 2013.
Ville, J. Etude critique de la notion de collectif. Bull. Amer. Math. Soc, 45(11):824, 1939.
A Proof of the General-Purpose Tail Bound for Adaptive Martingale Mixtures
A.1 Verifying Martingale Properties
First, we recall the definition of  $(M_t(\pmb{f}_t, \pmb{\lambda}_t) | t \in \mathbb{N})$  in Eq. (3). We are given a filtration  $(\mathcal{H}_t | t \in \mathbb{N})$ , a sequence of adapted random functions  $(Z_t : \mathbb{R} \to \mathbb{R} | t \in \mathbb{N})$ , and a sequence of predictable random variables  $(\lambda_t | t \in \mathbb{N})$ .
A filtration is an increasing sequence of  $\sigma$ -algebras  $\mathcal{H}_0 \subseteq \mathcal{H}_1 \subseteq \mathcal{H}_2 \dots$ . Each  $\sigma$ -algebra  $\mathcal{H}_t$  represents the information available at time  $t$ .  $(Z_t : \mathbb{R} \to \mathbb{R}|t \in \mathbb{N})$  being a sequence of adapted (to the filtration  $(\mathcal{H}_t|t \in \mathbb{N})$ ) functions means that, when conditioned on  $\mathcal{H}_t$ ,  $Z_t$  is no longer random.  $(\lambda_t|t \in \mathbb{N})$  being a sequence of predictable random variables means that, when conditioned on  $\mathcal{H}_{t-1}$ ,  $\lambda_t$  is no longer random.
For a sequence of real numbers  $(f_t : t \in \mathbb{N})$ , we define
$$
M _ {t} (\boldsymbol {f} _ {t}, \boldsymbol {\lambda} _ {t}) = \exp \left(\sum_ {k = 1} ^ {t} \lambda_ {k} Z _ {k} (f _ {k}) - \psi_ {k} (f _ {k}, \lambda_ {k})\right),
$$
where  $\psi_t(f_t, \lambda_t)$  is the conditional cumulant generating function
$$
\psi_ {t} \left(f _ {t}, \lambda_ {t}\right) := \ln \left(\mathbb {E} \left[ \exp \left(\lambda_ {t} Z _ {t} (f _ {t})\right) \mid \mathcal {H} _ {t - 1} \right]\right).
$$
Lemma A.1. For any sequence of real numbers  $(f_t|t\in \mathbb{N})$ ,  $(M_t(\pmb {f}_t,\pmb {\lambda}_t)|t\in \mathbb{N})$  is a martingale and  $\mathbb{E}[M_t(\pmb {f}_t,\pmb {\lambda}_t)] = 1$  for all  $t\in \mathbb{N}$ .
$$
\begin{array}{l} \mathbb {E} \left[ M _ {1} \left(\boldsymbol {f} _ {1}, \lambda_ {1}\right) \mid \mathcal {H} _ {0} \right] = \mathbb {E} \left[ \exp \left(\lambda_ {1} Z _ {1} \left(f _ {1}\right) - \psi_ {1} \left(f _ {1}, \lambda_ {1}\right)\right) \mid \mathcal {H} _ {0} \right] \\ = \mathbb {E} \left[ \exp \left(\lambda_ {1} Z _ {1} (f _ {1})\right) \mid \mathcal {H} _ {0} \right] / \exp \left(\psi_ {1} \left(f _ {1}, \lambda_ {1}\right)\right) \\ = \exp \left(\psi_ {1} \left(f _ {1}, \lambda_ {1}\right)\right) / \exp \left(\psi_ {1} \left(f _ {1}, \lambda_ {1}\right)\right) \\ = 1. \\ \end{array}
$$
Using the tower rule of conditional expectation, we also have