SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nicolas Emmenegger* ETH Zürich
Mojmir Mutny* ETH Zürich
Andreas Krause ETH Zürich
Certifiable, adaptive uncertainty estimates for unknown quantities are an essential ingredient of sequential decision-making algorithms. Standard approaches rely on problem-dependent concentration results and are limited to a specific combination of parameterization, noise family, and estimator. In this paper, we revisit the likelihood-based inference principle and propose to use likelihood ratios to construct any-time valid confidence sequences without requiring specialized treatment in each application scenario. Our method is especially suitable for problems with well-specified likelihoods, and the resulting sets always maintain the prescribed coverage in a model-agnostic manner. The size of the sets depends on a choice of estimator sequence in the likelihood ratio. We discuss how to provably choose the best sequence of estimators and shed light on connections to online convex optimization with algorithms such as Follow-the-Regularized-Leader. To counteract the initially large bias of the estimators, we propose a reweighting scheme that also opens up deployment in non-parametric settings such as RKHS function classes. We provide a non-asymptotic analysis of the likelihood ratio confidence sets size for generalized linear models, using insights from convex duality and online learning. We showcase the practical strength of our method on generalized linear bandit problems, survival analysis, and bandits with various additive noise distributions.
One of the main issues addressed by machine learning and statistics is the estimation of an unknown model from noisy observations. For example, in supervised learning, this might concern learning the dependence between an input (covariate)  $x$  and a random variable (observation)  $y$ . In many cases, we are not only interested in an estimate  $\hat{\theta}$  of the true model parameter  $\theta_{\star}$ , but instead in a set of plausible values that  $\theta_{\star}$  could take. Such confidence sets are of tremendous importance in sequential decision-making tasks, where uncertainty is used to drive exploration or risk-aversion needs to be implemented, and covariates are iteratively chosen based on previous observations. This setting includes problems such as bandit optimization, reinforcement learning, or active learning. In the former two, the confidence sets are often used to solve the exploration-exploitation dilemma and more generally influence the selection rule (Mukherjee et al., 2022), termination rule (Katz-Samuels and Jamieson, 2020), exploration (Auer, 2002) and/or risk-aversion (Makarova et al., 2021).
When we interact with the environment by gathering data sequentially based on previous confidence sets, we introduce correlations between past noisy observations and future covariates. Data collected in this manner is referred to as adaptively gathered (Wasserman et al., 2020). Constructing estimators, confidence sets, and hypothesis tests for such non-i.i.d. data comes with added difficulty. Accordingly, and also for its importance in light of the reproducibility crisis (Baker, 2016), the task has attracted significant attention in the statistics community in recent years (Ramdas et al., 2022).
(a) Gaussian  $\mathcal{L}$
(b) Laplace  $\mathcal{L}$
(c) Gaussian  $\mathcal{L}$  in RKHS
Figure 1: (a) and (b) show examples of confidence sets defined via level sets of the log-likelihood function in 2D at two dataset sizes, for Gaussian (a) and Laplace (b) likelihoods respectively. The sets inherit the geometry of the likelihood, and are not always ellipsoidal. (c) shows confidence bands on an RKHS function in a bandit game searching for the optimum. We compare prior work on confidence sets (Abbasi-Yadkori et al., 2011), our LR sets, and a common heuristic (orange). Our sets are nearly as small as the commonly used heuristic, but have provable coverage and can vastly improve sequential decision-making tasks such as bandits by quickly eliminating hypotheses.
Instead of deriving explicit concentration inequalities around an online estimator, we construct confidence sets implicitly defined by an inclusion criterion that is easy to evaluate in a computationally efficient manner and requires little statistical knowledge to implement. Roughly speaking, given a model  $p_{\theta}(y \mid x)$  that describes the conditional dependence of the observation  $y$  given the covariate  $x$  under parameter  $\theta$ , we will build sets based on a weighted modification of the sequential likelihood ratio statistic (Robbins et al., 1972; Wasserman et al., 2020)
$$
R _ {t} (\theta) := \frac {\mathcal {L} _ {t} \left(\left\{\hat {\theta} _ {s} \right\} _ {s = 1} ^ {t}\right)}{\mathcal {L} _ {t} (\theta)} := \frac {\prod_ {s = 1} ^ {t} p _ {\hat {\theta} _ {s}} ^ {w _ {s}} \left(y _ {s} \mid x _ {s}\right)}{\prod_ {s = 1} ^ {t} p _ {\theta} ^ {w _ {s}} \left(y _ {s} \mid x _ {s}\right)}, \tag {1}
$$
where  $\{\hat{\theta}_s\}_s$  is a running estimator sequence that we are free to choose, but which may only depend on previously collected data. Parameters  $\theta$  for which this statistic is small, i.e., for which  $R_{t}(\theta) \leq 1 / \alpha$  will be included in the set (and considered plausible). Examples of sets in a parametric and non-parametric setting are shown in Figure 1. The weighting terms  $w_s \in (0,1]$  are crucial for dealing with inherent irregularities of many conditional observation models but can be flexibly chosen. Classically, these are set to  $w_s = 1$ . The full exposition of our method with choice of estimators and weights is given in Section 2. Apart from being easy to use and implement, our approach also comes with performance guarantees. These sets maintain a provable  $1 - \alpha$  coverage – a fact we establish using Ville's inequality for supermartingales (Ville, 1939), which is known to be essentially tight for martingales (see Howard et al., 2018, for a discussion). Therefore, in stark contrast to alternate methods, our confidence sequence is fully data-dependent, making it empirically tighter than competing approaches. Despite the rich history of sequential testing and related confidence sets going back to Wald (1945) and Robbins et al. (1972), these sets have found little use in the interactive machine learning community, which is a gap we fill in the present paper.
Contributions In this work, we revisit the idea of using likelihood ratios to generate anytime-valid confidence sets. The main insight is that whenever the likelihood of the noise process is known, the likelihood ratio confidence sets are fully specified. They inherit their geometry from the likelihood function, and their size depends on the quality of our estimator sequence. We critically evaluate the likelihood ratio confidence sets and, in particular, we shed light on the following aspects: Firstly, for generalized linear models, we theoretically analyze the geometry of the LR confidence sets under mild assumptions. We show their geometry is dictated by Bregman divergences of exponential families (Chowdhury et al., 2022). Secondly, we show that the size of the confidence set is dictated by an online prediction game. The size of these sets depends on a sequence of estimators  $\{\hat{\theta}_s\}_{s=1}^t$  that one uses to estimate the unknown parameter  $\theta_\star$ . We discuss how to pick the estimator sequence in order to yield a provably small radius of the sets, by using the Follow-the-Regularized-Leader algorithm, which implements a regularized maximum-likelihood estimator. We prove that the radius of the confidence sets is nearly-worst-case optimal, and accordingly, they yield nearly-worst-case regret bounds when used in generalized linear bandit applications. However, due to their data-dependent nature, they can be much tighter than this theory suggests. Thirdly, we analyze the limitations of classical (unweighted) LR sets when the underlying conditional observation model is not identifiable. In this case, the resulting (inevitable) estimation bias unnecessarily increases the size of the confidence sets. To mitigate this, we propose an adaptive reweighting scheme that decreases the effect of uninformed early
bias of the estimator sequence on the size of the sets downstream. The reweighting does not affect the coverage guarantees of our sets and utilizes an elegant connection to (robust) powered likelihoods (Wasserman et al., 2020). Finally, thanks to the adaptive reweighting scheme, our sets are very practical as we showcase experimentally. We demonstrate that our method works well with exponential and non-exponential family likelihoods, and in parametric as well as in kernelized settings. We attribute their practical benefits to the fact that they do not depend on (possibly loose) worst-case parameters.
2 The Likelihood Method
The sequential likelihood ratio process (LRP) in (1) is a statistic that compares the likelihood of a given model parameter, with the performance of an adaptively chosen estimator sequence. As noted above, we generalize the traditional definition, which would have  $w_{s} \equiv 1$ , and define a corresponding confidence set as
$$
\mathcal {C} _ {t} = \left\{\theta \mid R _ {t} (\theta) \leq 1 / \alpha \right\}. \tag {2}
$$
The rationale is that the better a parameter  $\theta$  is at explaining the data  $\{(x_s, y_s)\}_s^t$  from the true model  $\theta_{\star}$ , the smaller this statistic will be, thereby increasing its chances to be included in  $\mathcal{C}_t$ . When we construct  $R_t$ , the sequence of  $x_s, w_s$  and  $\hat{\theta}_s$  cannot depend on the noisy observation  $y_s$ . Formally, consider the filtration  $(\mathcal{F}_s)_{s=0}^\infty$  with sub- $\sigma$ -algebras  $\mathcal{F}_s = \sigma(x_1, \ldots, y_1, \ldots, x_s, y_s, x_{s+1})$ . We require that  $\hat{\theta}_s$  and  $w_s$  are  $\mathcal{F}_{s-1}$ -measurable. Under these very mild assumptions and with arbitrary weights  $w_s \in (0, 1]$ , we can show coverage, i.e., our (weighted) confidence sets uniformly track the true parameter with probability  $1 - \alpha$ .
Theorem 1. The stochastic process  $R_{t}(\theta_{\star})$  in (1) is a non-negative supermartingale with respect to the filtration  $(\mathcal{F}_t)$  and satisfies  $R_0(\theta_\star)\equiv 1$ . In addition, the sequence  $C_t$  from (2) satisfies  $\mathbb{P}_{\theta_{\star}}(\exists t:\theta_{\star}\notin \mathcal{C}_t)\leq \alpha$ .
The last statement follows by applying Ville's inequality for super-martingales on  $R_{t}(\theta_{\star})$ . The proof closely follows Wasserman et al. (2020). While coverage is always guaranteed irrespective of the estimator sequence  $\{\hat{\theta}_s\}$ , we would like to make the sets as small as possible at fixed coverage, which we do by picking a well-predicting estimator sequence.
2.1 The Estimator Sequence Game
The specification of the LR process (LRP) allows us to choose an arbitrary estimator sequence  $\{\hat{\theta}_s\}_s$ . To understand the importance of the sequence, let us introduce  $\theta_{\star}$  to the definition of  $R_{t}$  in (1), and divide by  $\mathcal{L}_t(\{\hat{\theta}_s\}_{s=1}^t)$ . This gives the equivalent formulation
$$
\mathcal {C} _ {t} := \left\{\theta \mid \frac {\mathcal {L} _ {t} \left(\theta_ {\star}\right)}{\mathcal {L} _ {t} (\theta)} \leq \frac {1}{\alpha} \frac {\mathcal {L} _ {t} \left(\theta_ {\star}\right)}{\mathcal {L} _ {t} \left(\left\{\hat {\theta} _ {s} \right\} _ {s = 1} ^ {t}\right)} \leftarrow \text {c o n f i d e n c e p a r a m e t e r} \right\}.
$$
We see that the predictor sequence does not influence the geometry of the confidence set, which is fully specified by the likelihood function. We also observe that the ratio on the right-hand side serves as a confidence parameter controlling the size (radius) of the confidence sets measured under the likelihood ratio distance to  $\theta_{\star}$ . If the confidence parameter goes to zero, only  $\theta_{\star}$  is in the set. The better the estimator sequence is at predicting the data, the smaller the inclusion threshold, and hence the smaller the sets will ultimately be. Specifically, taking the log, we would like to minimize
$$
\mathcal {R} _ {t} := \log \frac {\mathcal {L} _ {t} \left(\theta_ {\star}\right)}{\mathcal {L} _ {t} \left(\left\{\hat {\theta} _ {s} \right\} _ {s = 1} ^ {t}\right)} = \sum_ {s = 1} ^ {t} - \log \left(p _ {\hat {\theta} _ {s}} ^ {w _ {s}} \left(y _ {s} \mid x _ {s}\right)\right) - \sum_ {s = 1} ^ {t} - \log \left(p _ {\theta_ {\star}} ^ {w _ {s}} \left(y _ {s} \mid x _ {s}\right)\right). \tag {3}
$$
The quantity  $\mathcal{R}_t$  corresponds to a regret in an online prediction game, as will become apparent below.
Online Prediction Game Online optimization is a mature field in interactive learning (Cesa-Bianchi and Lugosi, 2006; Orabona, 2019). The general goal is to minimize a sequence of loss functions as in Eq. (3) and compete against a baseline, which typically is the best-in-hindsight prediction, or - in our case - given by the performance of the fixed parameter  $\theta_{\star}$ . Specifically, at every timestep  $s$ , iteratively, the agent chooses an action  $\hat{\theta}_s$  based on  $\mathcal{F}_{s-1}$ , and a loss function  $f_s(\theta)$  is revealed. In most of the online optimization literature,  $f_s$  can be chosen adversarially. In our prediction game, we know the whole form of loss function  $f_s(\theta) = -\log(p_\theta^{w_s}(y_s \mid x_s))$ , as can be seen in (3), and not just  $f_s(\hat{\theta}_s)$ . Opposed to traditional assumptions in online prediction, in our case,  $f_s$  are non-adversarial, but have a stochastic component due to  $y_s$ . Also, contrary to most instances of online prediction, we do not compare against the best-in-hindsight predictor, but  $\theta_{\star}$  instead, as this is more meaningful in our setting.
Online Optimization Algorithms Generally, we seek an algorithm that incurs low regret. Here, we focus on Follow-the-Regularized Leader (FTRL), which corresponds exactly to using regularized maximum likelihood estimation, making it a natural and computationally practical choice. The update rule is defined in Alg. 1 (Line 3). While other algorithms could be considered, FTRL enjoys the optimal regret rate for generalized linear regression as we show later, and is easily implemented. In order to run the algorithm, one requires a sequence of strongly convex regularizers. For now, let us think of it as  $\psi_s(\theta) = \lambda ||\theta||_2^2$ , which we use in practice. However, one can derive a tighter analysis for a slightly modified, time-dependent regularization strategy for generalized linear models as we show in Sec. 3.3.
2.2 Adaptive Reweighting: Choosing the Right Loss
There is yet more freedom in the construction of the LR, via the selection of the loss function. Not only do we select the predictor sequence, but also the weights of the losses via  $w_{t}$ . This idea allows controlling the influence of a particular data point  $(x_{t}, y_{t})$  on the cumulative loss based on the value of  $x_{t}$ . For example, if we know a priori that for a given  $x_{t}$  our prediction will be most likely bad, we can opt out of using the pair  $(x_{t}, y_{t})$  by setting  $w_{t} = 0$ . Below we will propose a weighting scheme that depends on a notion of bias, which captures how much of the error in predicting  $y_{t}$  is due to our uncertainty about  $\hat{\theta}_{t}$  (compared to the uncertainty we still would have knowing  $\theta_{\star}$ ). Sometimes this bias is referred to as epistemic uncertainty in the literature, while the residual part of the error is referred to as aleatoric. Putting large weight on a data point heavily affected by this bias might unnecessarily increase the regret of our learner (and hence blow up the size of the confidence set). Note that, conveniently, even if we put low weight (zero) on a data point, nothing stops us from using this sample point to improve the estimator sequence in the next prediction round. As we will show below, our reweighting scheme is crucial in defining a practical algorithm for Reproducing Kernel Hilbert Space (RKHS) models and in high signal-to-noise ratio scenarios. Since we do not know  $\theta_{\star}$ , our strategy is to compute an estimate of the bias of the estimator  $\hat{\theta}_{t}$  and its effect on the value of the likelihood function for a specific  $x$  that we played. We use the value of the bias to rescale the loss via  $w_{t}$  such that its effect is of the same magnitude as the statistical error (see Algorithm 1; we call this step BIAS-WEIGHTING).
Intuition To give a bit more intuition, suppose we have a Gaussian likelihood. Then the negative log-likelihood of  $(x_{t},y_{t})$  with weighting is proportional to  $\frac{w_t}{\sigma^2} (y_t - x_t^\top \hat{\theta}_t)^2$ . Now, if  $x_{t}$  does not lie in the span of the data points  $\{x_s\}_{s = 1}^{t - 1}$  used to compute  $\hat{\theta}_t$ , it is in general unavoidable to incur large error, inversely proportional to  $\sigma^2$ . To see this, let us decompose the projection onto  $x_{t}$  as
$$
x _ {t} ^ {\top} \left(\hat {\theta} _ {t} - \theta_ {\star}\right) = \underbrace {x _ {t} ^ {\top} \left(\hat {\theta} _ {t} - \mathbb {E} \left[ \hat {\theta} _ {t} \right]\right)} _ {\text {s t a t i s t i c a l e r r o r}} + \underbrace {x _ {t} ^ {\top} \left(\mathbb {E} \left[ \hat {\theta} _ {t} \right] - \theta_ {\star}\right)} _ {\text {b i a s} _ {x _ {t}} (\hat {\theta} _ {t})}, \tag {4}
$$
where the first term represents the statistical error up to time  $t$ , while the second, bias, is deterministic, and independent of the actual realization  $y$ , depending only  $\theta_{\star}$ . Estimators with non-zero bias are biased. Plugging this into the likelihood function, we see that in expectation  $\frac{1}{\sigma^2}\mathbb{E}[(y_t - x_t^\top \hat{\theta}_t)^2 |\mathcal{F}_{t - 1}] \lesssim \frac{1}{\sigma^2}\mathrm{bias}_{x_t}^2 (\hat{\theta}_t) + \epsilon^2 +\frac{C}{t}$ , where  $\epsilon^2$  is the unavoidable predictive error in expectation (due to a noisy objective) and is a constant independent of  $\sigma^2$ .  $\frac{C}{t}$  is the statistical error, and  $C$  is independent of  $\sigma^2$ . Note that the bias term scales inversely with the variance, and leads to unnecessarily big confidence parameters for small  $\sigma^2$ .
In fact, the problem is that we use the likelihood to measure the distance between two parameters, but this is only a "good" distance once the deterministic source of the error (bias) vanishes. For this reason, without weighting, the incurred regret blows up severely in low-noise settings. To counter this, we balance the deterministic estimation bias and noise variance via proper selection of  $w_{t}$ . In this case, it turns out that  $w_{t} = \frac{\sigma^{2}}{\sigma^{2} + \mathrm{bias}_{x_{t}}^{2}(\hat{\theta}_{t})}$  ensures that the overall the scaling is independent of  $\sigma^2$ . While the choice of weights  $\{w_s\}_s^t$  influences the geometry of the confidence sets, with a good data collection and estimation strategy the bias asymptotically decreases to zero, and hence the weights converge to 1.
Bias estimation In order to generalize this rule beyond Gaussian likelihoods, we need a proper generalization of the bias. Our generalization is motivated by our analysis of generalized linear models, but the method can be applied more broadly. The role of the squared statistical error (variance) is played by the inverse of the smoothness constant of the negative log-likelihood functions  $f_{s}$ , denoted by  $L$ . This is the usual smoothness, commonly seen in the convex optimization literature.
We consider penalized likelihood estimators with strongly convex regularizers (Alg. 1, line 3). For this estimator class, we define the bias via a hypothetical stochastic-error-free estimate  $\hat{\theta}_t^\times$ , had we access to the expected values of the gradient loss functions (a.k.a. score). We use the first-order optimality conditions and the indicator function of the set  $\Theta$ ,  $i_\Theta$ , to define the error-free-estimate  $\hat{\theta}_t^\times$ , and the bias of the estimator  $\hat{\theta}_t$  as
$$
\operatorname {b i a s} _ {x _ {t}} ^ {2} \left(\hat {\theta} _ {t}\right) = \left(x _ {t} ^ {\top} \left(\theta_ {\star} - \hat {\theta} _ {t} ^ {\times}\right)\right) ^ {2} \quad \text {w i t h} \quad \mathbb {E} \left[ \sum_ {s = 1} ^ {t - 1} \nabla \log p _ {\hat {\theta} _ {t} ^ {\times}} \left(y _ {s} \mid x _ {s}\right) \right] - \nabla \psi_ {t} \left(\hat {\theta} _ {t} ^ {\times}\right) + i _ {\Theta} \left(\hat {\theta} _ {t} ^ {\times}\right) = 0, \tag {5}
$$
where the expectation denotes a sequence of expectations conditioned on the prior filtration. This notion of bias coincides with the definition of bias in Eq. (4) for the Gaussian likelihood. This quantity cannot be evaluated in general, however, we prove a computable upper bound.
Theorem 2 (Bias estimate). Let the negative log-likelihood have the form,  $-\log p_{\theta}(y_s|x_s) = g(x_s^\top \theta)$ , where  $g:\mathbb{R}\to \mathbb{R}$  is  $\mu$  strongly-convex and let the regularizer be  $\psi_t(\theta) = \lambda ||\theta ||_2^2$  making the overall objective strongly convex. Then, defining  $\mathbf{V}_t^{\mu ;\lambda} = \sum_{s = 1}^{t}\mu x_sx_s^\top +\lambda \mathbf{I}$ , we can bound
$$
\left. \operatorname {b i a s} _ {x} ^ {2} \left(\hat {\theta} _ {t}\right) \leq 2 \lambda \left\| \theta_ {\star} \right\| _ {2} ^ {2} x ^ {\top} \left(\mathbf {V} _ {t} ^ {\mu ; \lambda}\right) ^ {- 1} x. \right. \tag {6}
$$
The proof is deferred to App. A.3, and requires elementary convex analysis.
This leads us to propose the weighting scheme  $w_{t} = \frac{1 / L}{\mathrm{bias}_{xt}^{2}(\hat{\theta}_{t}) + 1 / L}$ . We justify that this is a sensible choice by analyzing the confidence set on the GLM class in Section 3, which satisfies the smoothness and strong-convexity conditions. We show that this rule properly balances the stochastic and bias components of the error in the regret as in (3). However, this rule is more broadly applicable beyond the canonical representation of GLM or the GLM family altogether.
Algorithm 1 Constructing the LR Confidence Sequence
1: Input: convex set  $\Theta \subset \mathbb{R}^d$ , confidence level  $\alpha > 0$ , likelihood  $p_{\theta}(y|x)$ , regularizers  $\{\psi_t\}_t$   
2: for  $t \in \mathbb{N}_0$  do  
3:  $\hat{\theta}_t = \arg \min_{\theta \in \Theta} \sum_{s=1}^{t-1} -\log p_{\theta}(y_s \mid x_s) + \psi_t(\theta) \quad \triangleright$  FTRL  
4:  $w_t = \left\{ \begin{array}{ll} \frac{1/L}{1/L + \text{bias}_{x_t}^2(\hat{\theta}_t)} & \text{THIS WORK} \\ 1 & \text{CLASSICAL} \end{array} \right. \quad \triangleright$  BIAS-WEIGHTING bias $_{x_t}(\hat{\theta}_t)$  in Eq. (5) or Eq.(6)  
5:  $\mathcal{C}_t = \left\{ \theta \in \Theta \left| \prod_{s=1}^t \frac{p_{\hat{\theta}_s}^{w_s}(y_s \mid x_s)}{p_{\theta}^{w_s}(y_s \mid x_s)} \leq \frac{1}{\alpha} \right. \right\}$ .  
6: end for
3 Theory: Linear Models
While the coverage (i.e., "correctness") of the likelihood ratio confidence sets is always guaranteed, their worst-case size (affecting the "performance") cannot be easily bounded in general. We analyze the size and the geometry of the LR confidence sequence in the special but versatile case of generalized linear models.
3.1 Generalized Linear Models
We assume knowledge of the conditional probability model  $p_{\theta}(y|x)$ , where the covariates  $x \in \mathcal{X} \subset \mathbb{R}^d$ , and the true underlying model parameter lies in a set  $\Theta \subset \mathbb{R}^d$ . If  $t$  is indexing (discrete) time, then  $x_t$  is acquired sequentially, and the – subsequently observed  $-y_t$  is sampled from an exponential family distribution parametrized as
$$
p _ {\theta} (y \mid x _ {t}) = h (y) \exp \left(T (y) \cdot x _ {t} ^ {\top} \theta - A \left(x _ {t} ^ {\top} \theta\right)\right). \tag {7}
$$