SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here,  $A$  is referred to as the log-partition function of the conditional distribution, and  $T(y)$  is the sufficient statistic. The function  $h$  is the base measure, and has little effect on our further developments, as it cancels out in the LR. Examples of commonly used exponential families (Gaussian, Binomial, Poisson or Weibull) with their link functions can be found in Table 1 in App. A.1.
In order to facilitate theoretical analysis for online algorithms, we make the following assumptions about the covariates  $x \in \mathcal{X}$  and the set of plausible parameters  $\Theta$ .
Assumption 1. The covariates are bounded, i.e.,  $\sup_{x\in \mathcal{X}}||x||_2\leq 1$ , and the set  $\Theta$  is contained in an  $\ell_2$ -ball of radius  $B$ . We will also assume that the log-partition function is strongly convex, that is, that there exists  $\mu \coloneqq \inf_{z\in [-B,B]}A''(z)$ , and that  $A$  is  $L$ -smooth, i.e.  $L\coloneqq \sup_{z\in [-B,B]}A''(z)$ .
These assumptions are common in other works addressing the confidence sets of GLMs (Filippi et al., 2010; Faury et al., 2020), who remark that the dependence on  $\mu$  is undesirable. However, in contrast to these works, our confidence sets do not use these assumptions in the construction of the sets. We only require these for our theoretical analysis. As these are worst-case parameters, the practical performance can be much better for our sets.
3.2 Geometry and Concentration
Before stating our results, we need to define a distance notion that the convex negative log-likelihoods induce. For a continuously differentiable convex function  $f$ , we denote the Bregman divergence as  $D_{f}(a,b) \coloneqq f(a) - f(b) - \nabla f(b)^{\top}(a - b)$ . The  $\nu$ -regularized sum of log-partition functions is defined as
$$
Z _ {t} ^ {\nu} (\theta) := \sum_ {s = 1} ^ {t} w _ {s} A \left(x _ {s} ^ {\top} \theta\right) + \frac {\nu}{2} \| \theta \| _ {2} ^ {2}. \tag {8}
$$
This function will capture the geometry of the LR confidence sets. The confidence set size depends mainly on two terms. One refers to a notion of complexity of the space referred to as Bregman information gain:  $\Gamma_t^\nu (\tilde{\theta}_t) = \log \left(\frac{\int_{\mathbb{R}^d}\exp(-\frac{\nu}{2}||\theta||_2^2)\mathrm{d}\theta}{\int_{\mathbb{R}^d}\exp(-D_{Z_t^\nu}(\theta,\tilde{\theta}_t))\mathrm{d}\theta}\right)$ , first defined by Chowdhury et al. (2022) as a generalization of the information gain of Srinivas et al. (2009),  $\gamma_t^\nu = \log \left(\operatorname*{det}\left(\sum_{i=1} \frac{\mu}{\nu} x_i x_i^\top + \mathbf{I}\right)\right)$  for Gaussian likelihoods. We will drop the superscript whenever the regularization is clear from context and simply refer to  $\gamma_t$ . This term appears because one can relate the decay of the likelihood as a function of the Bregman Divergence from  $\theta_\star$  with the performance of a (regularized) maximum likelihood estimator via convex (Fenchel) duality. In particular, if  $\tilde{\theta}_t$  is a regularized MLE,  $\Gamma_t^\nu \coloneqq \Gamma_t^\nu (\tilde{\theta}_t)$  will asymptotically scale as  $\mathcal{O}(d\log t)$  (cf. Chowdhury et al., 2022, for further discussion). For Gaussian likelihoods and  $w_s \equiv 1$ , it coincides with the classical information gain independent of  $\tilde{\theta}_t$ . The second term that affects the size is the regret  $\mathcal{R}_t$  of the online prediction game over  $t$  rounds we introduced previously in (3). These two parts together yield the following result:
Theorem 3. Let  $\nu >0$  and  $\alpha ,\delta \in (0,1)$ . For the level  $1 - \alpha$  confidence set  $\mathcal{C}_t$  defined in (2) under the GLM in (7), with probability  $1 - \delta$ , for all  $t\geq 1$ , any  $\theta \in \mathcal{C}_t$  satisfies
$$
D _ {Z _ {t} ^ {\nu}} (\theta , \theta_ {\star}) \leq \frac {4 L}{\mu} \xi_ {t} + 2 \log \left(\frac {1}{\delta}\right) + 2 \mathcal {R} _ {t}, \tag {9}
$$
where  $\xi_{t} = \left(\log \left(\frac{1}{\alpha}\right) + \nu B^{2} + \Gamma_{t}^{\nu}\right)$  and  $L, \mu$  are defined as above and finally  $\mathcal{R}_t$  is the regret of the game in Eq. (3).
The set defined via the above divergence does not coincide with the LR confidence set. It is slightly larger due to a term involving  $\nu$  (as in Eq. (8)). This is a technical consequence of our proof technique, where the gradient of  $Z_{t}^{\nu}$  needs to be invertible, and regularization is added to this end. We note that this  $\nu > 0$  can be chosen freely. Note that the theorem involves two confidence levels,  $\alpha$  and  $\delta$ :  $\alpha$  is a bound on the Type I error - coverage of the confidence sets - while  $\delta$  upper bounds the probability of a large radius - and is therefore related to the power and Type II error of a corresponding hypothesis test. The proof of the theorem is deferred to App. B.2.
To give more intuition on these quantities, let us instantiate them for the Gaussian likelihood case with  $w_{s} \equiv 1$ . In this scenario,  $Z_{t}^{\nu}(\theta) = \sum_{s=1}^{t} \frac{1}{2\sigma^{2}} ||\theta||_{x_{s}x_{s}^{\top}}^{2} + \frac{\nu}{2} ||\theta||_{2}^{2}$ , and the (in this case symmetric) Bregman divergence is equal to  $D_{Z_{t}^{\nu}}(\theta_{\star}, \theta) = \frac{1}{2} ||\theta - \theta_{\star}||_{\mathbf{V}_{t}^{\sigma-2;\nu}}^{2}$ , where  $\mathbf{V}_{t}^{\mu;\nu} = \sum_{s=1}^{t} \mu x_{s}x_{s}^{\top} + \nu \mathbf{I}$ , which means that our confidence sets are upper bounded by a ball in the same norm as those in the seminal work on linear bandits (Abbasi-Yadkori et al., 2011).
3.3 Online Optimization in GLMs: Follow the Regularized Leader
The size of the confidence sets in Theorem 3 depends on the regret of the online prediction game involving the estimator sequence. We now bound this regret when using the Follow-the-Regularized-Leader (FTRL) algorithm in this setting. This high probability bound is novel to the best of our
knowledge and may be of independent interest. We state in a weight-agnostic manner first, and then with our particular choice. The latter variant uses a specifically chosen regularizer. In this case, we can track the contribution of each time-step towards the regret separately.
Theorem 4. Let  $\psi_t(\theta) = \lambda ||\theta||_2^2$ . Assume Assumption 1, and additionally that  $A$  is  $L$ -smooth everywhere in  $\mathbb{R}^d$ , and let  $w_t \in [0,1]$  be arbitrary. Then, with probability  $1 - \delta$  the regret of FTRL (Alg. 1) satisfies for all  $t \geq 1$
$$
\mathcal {R} _ {t} \leq \lambda B ^ {2} + \frac {L}{\mu} \left(\gamma_ {t} ^ {\lambda} + 2 \log (1 / \delta)\right) + \frac {2 L ^ {2} B ^ {2}}{\mu} \gamma_ {t} ^ {\lambda}. \tag {10}
$$
The regret bounds are optimal in the orders of  $\gamma_t^{\lambda}$ , matching lower bounds of Ouhamma et al. (2021), as for linear models  $\gamma_t = \mathcal{O}(d\log t)$ . Combining results of Thm. 4 with Thm. 3, we get a confidence parameter that scales with  $\mathcal{O}(\sqrt{\gamma_t})$ , for confidence sets of the form  $\| \theta - \theta_{\star} \|_{\mathbf{V}_t}$ , which coincides with the best-known confidence sets in this setting in the worst-case (Abbasi-Yadkori et al., 2012). The requirement of global  $L$ -smoothness can be relaxed to  $L$ -smoothness over  $\Theta$ . With a more elaborate (but less insightful) analysis, we can show that we achieve a  $\tilde{\mathcal{O}}(\gamma_t)$  bound even in this case. The proofs of these results are deferred to App. C.4, App. C.5 and App. C.6 respectively.
Regret, Weighting and Estimation Bias Interestingly, the term in Thm. 4 involving the (crude) proxy to the bias - the bound  $B$  - is not scaled by the same  $L / \mu$  factors as the other terms in the regret bound (10) and in Theorem 3. Namely, the prefactor is  $L^2 /\mu$  instead of  $L / \mu$ . This extra dependence manifests itself in the unnecessary penalization through the estimation bias we introduced in Sec. 2.2, particularly in low-noise settings. We addressed this issue by picking the weights  $\{w_{t}\}$ . While the above theorem holds for any valid weighting, it does not exhibit the possible improvement from using specific weights.
We argued earlier that the error in prediction should not be measured by the likelihood function if there is deterministic error, since initially, we are fully uncertain about the value of  $\theta_{\star}^{\top}(\cdot)$  outside the span of previous observations. Of course, if our goal would be to purely pick weights to minimize  $\mathcal{R}_t$ , then  $w_s = 0$  would lead to zero regret and hence be optimal. However, the likelihood ratio would then be constant, and uninformative. In other words, the associated log-partition Bregman divergence in Theorem 3 would be trivial and not filter out any hypotheses. Clearly, some balance has to be met. With this motivation in mind, we proposed a nonzero weighting that decreases the regret contribution of the bias, namely  $w_t = \frac{1 / L}{1 / L + \mathrm{bias}_{x_t}^2(\hat{\theta}_t)}$ . The advantage of this choice becomes more apparent when we use the regularizer  $\psi_t(\theta) = \lambda ||\theta||^2 + A(x_t^\top \theta)$  to obtain the following result.
Theorem 5. Let  $\psi_s(\theta) = \lambda ||\theta||^2 + A(x_s^\top \theta)$ . Assume Assumption 1, and additionally that  $A$  is  $L$ -smooth everywhere in  $\mathbb{R}^d$ , and choose  $w_s = \frac{1 / L}{1 / L + \mathrm{bias}_{x_s}(\hat{\theta}_s)^2}$ . Additionally, let the sequence of  $x_s$  be such that,  $\sum_{s}(1 - w_s)(f_s(\theta_\star) - f_s(\bar{\theta}_{s+1})) \leq L / \mu \gamma_t^\lambda$ , where  $\bar{\theta}_s$  is the FTRL optimizer with the regularizer  $\lambda ||\theta||_2^2$  from Theorem 4. Then, with probability  $1 - \delta$  the regret of FTRL (Alg. 1) satisfies for all  $t \geq 1$
$$
\mathcal {R} _ {t} \leq \lambda B ^ {2} + \frac {2 L}{\mu} \left(\gamma_ {t} ^ {\lambda} + \log \left(\frac {1}{\delta}\right)\right) + \frac {L}{\mu} \sum_ {s = 1} ^ {t} \frac {B ^ {2}}{1 / L + \mathrm {b i a s} _ {x _ {s}} ^ {2} (\hat {\theta} _ {s})} \Delta \gamma_ {s} ^ {\lambda},
$$
where  $\Delta \gamma_s^\lambda = \gamma_{s + 1}^\lambda -\gamma_s^\lambda$
One can see that for points where the information gain  $\Delta \gamma_{s}$  is large (corresponding to more unexplored regions of the space, where the deterministic source of error is then large), the weighting scheme will make sure that the multiplicative contribution of  $B^{2}$  is mitigated, along with having the correct prefactor  $L / \mu$ . The reader may wonder how this result is useful when we replace  $\mathrm{bias}_{x_s}^2 (\hat{\theta}_s)$  with the upper bound from Thm. 2. While instructive, our bound still only makes the bias proxy  $B^{2}$  appear in front of the information gain  $\Delta \gamma_{t}$ , instead of the more desirable bias itself. In the latter case, we could also directly make use of the upper bound and get an explicit result only using an upper bound on the bias. We leave this for future work.
We point out that this choice of  $\psi_s(\theta)$  in Theorem 5 corresponds to the Vovk-Azoury-Warmuth predictor (Vovk, 2001; Azoury and Warmuth, 1999) in the online learning literature. This choice is helpful in order to track the bias contribution more precisely in our proof.
Note that this assumption was missing in an earlier version.  $\bar{\theta}_{s + 1}$  corresponds to a regularized MLE that did observe the data pair  $(x_{s},y_{s})$
4 Application: Linear and Kernelized Bandits
Our main motivation to construct confidence sets is bandit optimization. A prototypical bandit algorithm – the Upper Confidence Bound (UCB) (Auer, 2002) – sequentially chooses covariates  $x_{s}$  in order to maximize the reward  $\sum_{s=1}^{t} r_{\theta_{\star}}(x_{s})$ , where  $r_{\theta_{\star}}$  is the unknown pay-off function parametrized by  $\theta_{\star}$ . UCB chooses the action  $x_{s}$  which maximizes the optimistic estimate of the reward in each round, namely
$$
x _ {s} = \underset {x \in \mathcal {X}} {\arg \max } \underset {\theta \in \mathcal {C} _ {s - 1}} {\max } r _ {\theta} (x), \tag {11}
$$
where  $\mathcal{C}_{s-1}$  is some confidence set for  $\theta_{\star}$ , and can be constructed with Algorithm 1 from the first  $s-1$  data points. An important special case is when  $r_{\theta_{\star}}$  is linear (Abe and Long, 1999) or modelled by a generalized linear model (Filippi et al., 2010). In that case, the inner optimization problem is convex as long as  $\mathcal{C}_{s-1}$  is convex. The outer optimization is tractable for finite  $\mathcal{X}$ . In the applications we consider, our confidence sets are convex, and we easily solve the UCB oracle using convex optimization toolboxes.
Extension to RKHS We introduced the framework of LR confidence sets only for finite-dimensional Euclidean spaces. However, it can be easily extended to Reproducing Kernel Hilbert Spaces (RKHS) (Cucker and Smale, 2002). The definition of the LR process in (1) is still well-posed, but now the sets are subsets of the RKHS, containing functions  $f \in \mathcal{H}_k$ . An outstanding issue is how to use these sets in downstream applications, and represent them tractably as in Figure 1. Conveniently, even with infinite-dimensional RKHSs, the inner-optimization in (11) admits a Lagrangian formulation, and the generalized representer theorem applies (Schölkopf et al., 2001; Mutny and Krause, 2021). In other words, we can still derive a pointwise upper confidence band as  $\mathrm{ucb}(x) = \max_{f \in \mathcal{H}_k, \| f \|_k \leq B, f \in C_s} \langle f, k(x, \cdot) \rangle$  in terms of  $\{x_j\}_{j=1}^s \cup \{x\}$ , leading to a  $s+1$ -dimensional, tractable optimization problem.
We also point out that the weighting is even more paramount in the RKHS setting, as the bias never vanishes for many infinite dimensional Hilbert spaces (Mutny and Krause, 2022). For this purpose, our weighting is of paramount practical importance, as we can see in Figure 2a), where the gray arrow represents the significant improvement from reweighting.
4.1 Instantiation of the Theory for Linear Bandits
Before going to the experiments, we instantiate our theoretical results from Sec. 3 to the important and well-studied special case of linear payoffs. In that case,  $r_{\theta}(x) = \langle x,\theta \rangle$  and the agent observes  $y_{s} = \langle x_{s},\theta_{\star}\rangle +\eta_{s}$  upon playing action  $x_{s}$ , where  $\eta_s\sim \mathcal{N}(0,\sigma^2)$ . We are interested in minimizing the so-called cumulative pseudo-regret, namely,  $\Re_t = \sum_{s = 1}^t [\langle x_\star ,\theta_\star \rangle -\langle x_s,\theta_\star \rangle ]$ , where  $x_{\star}$  refers to the optimal action. Using the set from (2) along with Theorem 3 and the FTRL result of Theorem 4 we can get a regret bound for the choice  $w_{s}\equiv 1$ .
Theorem 6. Let  $w_{s}\equiv 1$ . For any  $\lambda \geq \frac{1}{\sigma^2}$ , with probability at least  $1 - 3\delta$ , for all  $t\in \mathbb{N}$  we have
$$
\mathfrak {R} _ {t} \leq 6 \sqrt {t \gamma_ {t} ^ {\lambda}} \left(\sigma \sqrt {\log (1 / \delta) + \gamma_ {t} ^ {\lambda}} + \sigma \lambda^ {1 / 2} B + B \sqrt {\gamma_ {t} ^ {\lambda}}\right).
$$
Our results are optimal in both  $d$  and  $t$  up to constant and logarithmic factors. The proof is deferred to App. D, but is an instantiation of the aforementioned theorems, along with a standard analysis. There, we also compare to the seminal result of Abbasi-Yadkori et al. (2011), which does not suffer from the dependence on  $B\sqrt{\gamma_t}$ . We attribute this to the incurred bias in the absence of the reweighting scheme.
For the weighted likelihood ratio, we can obtain a result similar to the above, but multiplied by an upper bound on  $\sup_{s\geq 1}w_s^{-1}$ . This is undesirable, as our experiments will show that the reweighting scheme vastly improves performance. While this could be somewhat mitigated by using the Theorem 5 instead of Theorem 4 to bound the FTRL regret, a better result should be achievable using our weighting scheme that improves upon Theorem 6 and possibly even matches Abbasi-Yadkori et al. (2011) exactly in the worst-case. We leave this for future work.
4.2 Experimental Evaluation
In this subsection, we demonstrate that the practical applicability goes well beyond the Gaussian theoretical result from the previous subsection. In the examples below, we always use the UCB
Figure 2: Bandit experiments: On the  $y$ -axis we report cumulative regret, while the  $x$ -axis shows the number of iterations. In a) and b) we report the results for linear models with different parametric additive noise. In c) we report the results on a survival analysis with a log-Weibull distribution  $(p = 2)$  and in d) we showcase Poisson bandits. See App. E for more details. Heuristic methods are dashed, while provable are solid. Our sets perform the best among all provable methods. Notice in a) the difference in gray and black represents the improvement due to adaptive weighting over  $w_{s} = 1$  for all  $s \in [t]$ . For each experiment we did 10 reruns, median values are plotted.
algorithm but employ different confidence sets. In particular, we compare our LR confidence sets for different likelihood families with alternatives from the literature, notably classical sub-family confidence sets (Abbasi-Yadkori et al., 2011; Mutny and Krause, 2021), and the robust confidence set of Neiswanger and Ramdas (2021). In practice, however, the radius of these confidence sets is often tuned heuristically. We include such sets as a baseline without provable coverage as well. The main take-home message from the experiments is that among all the estimators and confidence sets that enjoy provable coverage, our confidence sets perform the best, on par with successful heuristics. For all our numerical experiments in Figure 2, the true payoff function is assumed to be an infinite dimensional RKHS element. For further details and experiments, please refer to App. E.
Additive Noise Models Suppose that  $r_{\theta_{\star}}$  is linear and we observe  $y_{s} = x_{s}^{\top}\theta_{\star} + \eta_{s}$ , where  $\eta_{s}$  is additive noise, and  $\theta_{\star}$  is an element of a Hilbert space. We consider classical Gaussian noise as well as Laplace noise in Fig. 2[a), b)]. Notice that in both cases our confidence sets yield lower regret than any other provably valid method. In both cases they are performing as good as heuristic confidence sets with confidence parameter  $\beta_{t} \equiv 2\log (1 / \delta)$ . The sub-Gaussian confidence sets of Abbasi-Yadkori et al. (2011) (AY 2011) are invalid for the Laplace distribution as it is not sub-Gaussian but only sub-Exponential. For this reason, we compare also with sub-exponential confidence sets derived similarly to those of (Faury et al., 2020). The confidence sets of (Neiswanger and Ramdas, 2021) (NR 2021) perform similarly on Gaussian likelihood, but are only applicable to this setting, as their generalization to other likelihood families involves intractable posterior inference. We note also the difference between the unweighted LR and the weighted one. The examples in Fig. 2 use the true payoff functions  $r(x) = -(1.4 - 3x)\sin (18x)$ , which we model as an element of a RKHS with squared exponential kernel lengthscale  $\gamma = 6\times 10^{-2}$  on [0, 1.2], which is the baseline function no. 4 in the global optimization benchmark database infinity77 (Gavana, 2021). Additional experiments can be found in App. E.
**Poisson Bandits** A prominent example of generalized linear bandits (GLB) are Poisson bandits, where the linear payoff is scaled by an exponential function. We instantiate our results on a common benchmark problem, and report the results in Fig. 2d). We improve the regret of UCB for GLBs compared to two alternative confidence sets: one that uses a Laplace approximation with a heuristic confidence parameter, and one inspired by considerations in Mutny and Krause (2021) (MK 2021), also with a heuristic confidence parameter. Note that we cannot compare directly to their provable results in their original form as they do not state them in the canonical form of the exponential family.
Survival Analysis Survival analysis is a branch of statistics with a rich history that models the lifespan of a service or product (Breslow, 1975; Cox, 1997; Kleinbaum and Klein, 2010). The classical approach postulates a well-informed likelihood model. Here, we use a specific hazard model, where the survival time  $T$  is distributed with a Weibull distribution, parametrized by  $\lambda$  and  $p$ . The rate  $\lambda_{\theta}(x) = \exp(x^{\top}\theta)$  differs for each configuration  $x$ , and  $p$  - which defines the shape of the survival distribution - is fixed and known. We assume that the unknown part is due to the parameter  $\theta$  which is the quantity we build a confidence set around to use within the UCB Algorithm. In particular, the probability density of the Weibull distribution is  $P(T = t|x) = \lambda_{\theta}(x)pt^{p-1}\exp(-t^{p}\lambda_{\theta}(x))$ . In fact, with  $p = 2$ , the confidence sets are convex and the UCB rule can be implemented efficiently.
Interestingly, this model admits an alternate linear regression formulation. Namely upon using the transformation  $Y = \log T$ , the transformed variables  $Y|x$  follow a Gumbel-type distribution, with the following likelihood that can be obtained by the change of variables  $P(Y = y|x) = \lambda_{\theta}(x)p\exp (y)^{p}\exp (-\exp (y)^{p}\lambda_{\theta}(x))$ . The expectation over  $Y$  allows us to express it as a linear regression problem since  $\mathbb{E}[Y|x] = -(\theta^{\top}x + \gamma) / p$ , where  $\gamma$  is the Euler-Mascheroni constant. More importantly,  $Y|x$  is sub-exponential. Hence, this allows us to use confidence sets for sub-exponential variables constructed with the pseudo-maximization technique inspired by Faury et al. (2020). More details on how these sets are derived can be found in App. E. However, these approaches necessarily employ crude worst-case bounds and as can be seen in Figure 2c) the use of our LR-based confidence sequences substantially reduces the regret of the bandit learner.
5 Related Work and Conclusion
Related Work The adaptive confidence sequences stem from the seminal work of Robbins et al. (1972), who note that these sets have  $\alpha$ -bounded Type I error. The likelihood ratio framework has been recently popularized by Wasserman et al. (2020) for likelihood families without known test statistics under the name universal inference. This approach, although long established, is surprisingly uncommon in sequential decision-making tasks like bandits. This might be due to the absence of an analysis deriving the size of the confidence sets (Mutny and Krause, 2021), a necessary ingredient to obtain regret bounds. We address this gap for generalized linear models. Another reason might be that practitioners might be interested in non-parametric sub-families - a scenario our method does not cover. That being said, many fields such as survival analysis (Cox, 1997) do have well-informed likelihoods. However, most importantly, if used naively, this method tends to fail when one departs from assumptions that our probabilistic model is identifiable (i.e.,  $p_{\theta}(\cdot | x) = p_{\tilde{\theta}}(\cdot | x)$  even if  $\theta \neq \tilde{\theta}$ ). We mitigate this problem by introducing the scaling parameters  $w_{t}$  in Eq. (1) to deal with it.
Prevalent constructions of anytime-valid confidence intervals rely on carefully derived concentration results and for a specific estimator such as the least-squares estimator and noise sub-families such as sub-Gaussian, sub-Bernoulli and sub-Poisson Abbasi-Yadkori et al. (2011); Faury et al. (2020); Mutny and Krause (2021). Their constructions involve bounding the suprema of collections of self-normalized stochastic processes (Faury et al., 2020; Mutny and Krause, 2021; Chowdhury et al., 2022). To facilitate closed-form expressions, worst-case parameters are introduced that prohibitively affect the size of the sets - making them much larger than they need to be.