Title: Likelihood Ratio Confidence Sets for Sequential Decision Making
Venue: NeurIPS 2023
Accepted: yes
Keywords: confidence sets

uncertainty quantification

bandits

active learning

testing
Abstract: Certifiable, adaptive uncertainty estimates for unknown quantities are an essential ingredient of sequential decision-making algorithms. Standard approaches rely on problem-dependent concentration results and are limited to a specific combination of parameterization, noise family, and estimator. In this paper, we revisit the likelihood-based inference principle and propose to use \emph{likelihood ratios} to construct \emph{any-time valid} confidence sequences without requiring specialized treatment in each application scenario. Our method is especially suitable for problems with well-specified likelihoods, and the resulting sets always maintain the prescribed coverage in a model-agnostic manner. The size of the sets depends on a choice of estimator sequence in the likelihood ratio. We discuss how to provably choose the best sequence of estimators and shed light on connections to online convex optimization with algorithms such as Follow-the-Regularized-Leader. To counteract the initially large bias of the estimators, we propose a reweighting scheme that also opens up deployment in non-parametric settings such as RKHS function classes. We provide a \emph{non-asymptotic} analysis of the likelihood ratio confidence sets size for generalized linear models, using insights from convex duality and online learning. We showcase the practical strength of our method on generalized linear bandit problems, survival analysis, and bandits with various additive noise distributions.

Likelihood Ratio Confidence Sets for Sequential Decision Making

Abstract

Certifiable, adaptive uncertainty estimates for unknown quantities are an essential ingredient of sequential decision-making algorithms. Standard approaches rely on problem-dependent concentration results and are limited to a specific combination of parameterization, noise family, and estimator. In this paper, we revisit the likelihood-based inference principle and propose to use likelihood ratios to construct any-time valid confidence sequences without requiring specialized treatment in each application scenario. Our method is especially suitable for problems with well-specified likelihoods, and the resulting sets always maintain the prescribed coverage in a model-agnostic manner. The size of the sets depends on a choice of estimator sequence in the likelihood ratio. We discuss how to provably choose the best sequence of estimators and shed light on connections to online convex optimization with algorithms such as Follow-the-Regularized-Leader. To counteract the initially large bias of the estimators, we propose a reweighting scheme that also opens up deployment in non-parametric settings such as RKHS function classes. We provide a non-asymptotic analysis of the likelihood ratio confidence sets size for generalized linear models, using insights from convex duality and online learning. We showcase the practical strength of our method on generalized linear bandit problems, survival analysis, and bandits with various additive noise distributions.

1 Introduction

One of the main issues addressed by machine learning and statistics is the estimation of an unknown model from noisy observations. For example, in supervised learning, this might concern learning the dependence between an input (covariate) x and a random variable (observation) y . In many cases, we are not only interested in an estimate ˆ θ of the true model parameter θ ⋆ , but instead in a set of plausible values that θ ⋆ could take. Such confidence sets are of tremendous importance in sequential decision-making tasks, where uncertainty is used to drive exploration or risk-aversion needs to be implemented, and covariates are iteratively chosen based on previous observations. This setting includes problems such as bandit optimization, reinforcement learning, or active learning. In the former two, the confidence sets are often used to solve the exploration-exploitation dilemma and more generally influence the selection rule (Mukherjee et al., 2022), termination rule (Katz-Samuels and Jamieson, 2020), exploration (Auer, 2002) and/or risk-aversion (Makarova et al., 2021).

When we interact with the environment by gathering data sequentially based on previous confidence sets, we introduce correlations between past noisy observations and future covariates. Data collected in this manner is referred to as adaptively gathered (Wasserman et al., 2020). Constructing estimators, confidence sets, and hypothesis tests for such non-i.i.d. data comes with added difficulty. Accordingly, and also for its importance in light of the reproducibility crisis (Baker, 2016), the task has attracted significant attention in the statistics community in recent years (Ramdas et al., 2022).

L

(b) Laplace

L

L

in RKHS

2

0

-2

truth

β=2log(1/6)

AY (2011)

LR

-0.4

-0.2

0.0

0.2

0.4

Instead of deriving explicit concentration inequalities around an online estimator, we construct confidence sets implicitly defined by an inclusion criterion that is easy to evaluate in a computationally efficient manner and requires little statistical knowledge to implement. Roughly speaking, given a model p θ ( y | x ) that describes the conditional dependence of the observation y given the covariate x under parameter θ , we will build sets based on a weighted modification of the sequential likelihood ratio statistic (Robbins et al., 1972; Wasserman et al., 2020)

where { ˆ θ s } s is a running estimator sequence that we are free to choose, but which may only depend on previously collected data. Parameters θ for which this statistic is small, i.e., for which R t ( θ ) ≤ 1 /α will be included in the set (and considered plausible ). Examples of sets in a parametric and non-parametric setting are shown in Figure 1. The weighting terms w s ∈ (0 , 1] are crucial for dealing with inherent irregularities of many conditional observation models but can be flexibly chosen. Classically, these are set to w s = 1 . The full exposition of our method with choice of estimators and weights is given in Section 2. Apart from being easy to use and implement, our approach also comes with performance guarantees. These sets maintain a provable 1 -α coverage - a fact we establish using Ville's inequality for supermartingales (Ville, 1939), which is known to be essentially tight for martingales (see Howard et al., 2018, for a discussion). Therefore, in stark contrast to alternate methods, our confidence sequence is fully data-dependent , making it empirically tighter than competing approaches. Despite the rich history of sequential testing and related confidence sets going back to Wald (1945) and Robbins et al. (1972), these sets have found little use in the interactive machine learning community, which is a gap we fill in the present paper.

Contributions In this work, we revisit the idea of using likelihood ratios to generate anytime-valid confidence sets. The main insight is that whenever the likelihood of the noise process is known, the likelihood ratio confidence sets are fully specified. They inherit their geometry from the likelihood function, and their size depends on the quality of our estimator sequence. We critically evaluate the likelihood ratio confidence sets and, in particular, we shed light on the following aspects: Firstly , for generalized linear models, we theoretically analyze the geometry of the LR confidence sets under mild assumptions. We show their geometry is dictated by Bregman divergences of exponential families (Chowdhury et al., 2022). Secondly , we show that the size of the confidence set is dictated by an online prediction game. The size of these sets depends on a sequence of estimators { ˆ θ s } t s =1 that one uses to estimate the unknown parameter θ ⋆ . We discuss how to pick the estimator sequence in order to yield a provably small radius of the sets, by using the Follow-the-Regularized-Leader algorithm, which implements a regularized maximum-likelihood estimator. We prove that the radius of the confidence sets is nearly-worst-case optimal, and accordingly, they yield nearly-worst-case regret bounds when used in generalized linear bandit applications. However, due to their data-dependent nature, they can be much tighter than this theory suggests. Thirdly , we analyze the limitations of classical (unweighted) LR sets when the underlying conditional observation model is not identifiable. In this case, the resulting (inevitable) estimation bias unnecessarily increases the size of the confidence sets. To mitigate this, we propose an adaptive reweighting scheme that decreases the effect of uninformed early

bias of the estimator sequence on the size of the sets downstream. The reweighting does not affect the coverage guarantees of our sets and utilizes an elegant connection to (robust) powered likelihoods (Wasserman et al., 2020). Finally , thanks to the adaptive reweighting scheme, our sets are very practical as we showcase experimentally. We demonstrate that our method works well with exponential and non-exponential family likelihoods, and in parametric as well as in kernelized settings. We attribute their practical benefits to the fact that they do not depend on (possibly loose) worst-case parameters .

2 The Likelihood Method

The sequential likelihood ratio process (LRP) in (1) is a statistic that compares the likelihood of a given model parameter, with the performance of an adaptively chosen estimator sequence. As noted above, we generalize the traditional definition, which would have w s ≡ 1 , and define a corresponding confidence set as

The rationale is that the better a parameter θ is at explaining the data { ( x s , y s ) } t s from the true model θ ⋆ , the smaller this statistic will be, thereby increasing its chances to be included in C t . When we construct R t , the sequence of x s , w s and ˆ θ s cannot depend on the noisy observation y s . Formally, consider the filtration ( F s ) ∞ s =0 with subσ -algebras F s = σ ( x 1 , . . . , y 1 , . . . x s , y s , x s +1 ) . We require that ˆ θ s and w s are F s -1 -measurable. Under these very mild assumptions and with arbitrary weights w s ∈ (0 , 1] , we can show coverage, i.e., our (weighted) confidence sets uniformly track the true parameter with probability 1 -α .

Theorem 1. The stochastic process R t ( θ ⋆ ) in (1) is a non-negative supermartingale with respect to the filtration ( F t ) and satisfies R 0 ( θ ⋆ ) ≡ 1 . In addition, the sequence C t from (2) satisfies P θ ⋆ ( ∃ t : θ ⋆ ̸∈ C t ) ≤ α.

The last statement follows by applying Ville's inequality for super-martingales on R t ( θ ⋆ ) . The proof closely follows Wasserman et al. (2020). While coverage is always guaranteed irrespective of the estimator sequence { ˆ θ s } , we would like to make the sets as small as possible at fixed coverage, which we do by picking a well-predicting estimator sequence.

2.1 The Estimator Sequence Game

The specification of the LR process (LRP) allows us to choose an arbitrary estimator sequence { ˆ θ s } s . To understand the importance of the sequence, let us introduce θ ⋆ to the definition of R t in (1), and divide by L t ( { ˆ θ s } t s =1 ) . This gives the equivalent formulation

We see that the predictor sequence does not influence the geometry of the confidence set, which is fully specified by the likelihood function. We also observe that the ratio on the right-hand side serves as a confidence parameter controlling the size (radius) of the confidence sets measured under the likelihood ratio distance to θ ⋆ . If the confidence parameter goes to zero, only θ ⋆ is in the set. The better the estimator sequence is at predicting the data, the smaller the inclusion threshold, and hence the smaller the sets will ultimately be. Specifically, taking the log , we would like to minimize

The quantity R t corresponds to a regret in an online prediction game, as will become apparent below.

Online Prediction Game Online optimization is a mature field in interactive learning (Cesa-Bianchi and Lugosi, 2006; Orabona, 2019). The general goal is to minimize a sequence of loss functions as in Eq. (3) and compete against a baseline, which typically is the best-in-hindsight prediction, or - in our case - given by the performance of the fixed parameter θ ⋆ . Specifically, at every timestep s , iteratively, the agent chooses an action ˆ θ s based on F s -1 , and a loss function f s ( θ ) is revealed. In most of the online optimization literature, f s can be chosen adversarially. In our prediction game, we know the whole form of loss function f s ( θ ) = -log( p w s θ ( y s | x s )) , as can be seen in (3), and not just f s ( ˆ θ s ) . Opposed to traditional assumptions in online prediction, in our case, f s are non-adversarial, but have a stochastic component due to y s . Also, contrary to most instances of online prediction, we do not compare against the best-in-hindsight predictor, but θ ⋆ instead, as this is more meaningful in our setting.

Online Optimization Algorithms Generally, we seek an algorithm that incurs low regret. Here, we focus on Follow-the-Regularized Leader (FTRL) , which corresponds exactly to using regularized maximum likelihood estimation, making it a natural and computationally practical choice. The update rule is defined in Alg. 1 (Line 3). While other algorithms could be considered, FTRL enjoys the optimal regret rate for generalized linear regression as we show later, and is easily implemented. In order to run the algorithm, one requires a sequence of strongly convex regularizers. For now, let us think of it as ψ s ( θ ) = λ || θ || 2 2 , which we use in practice. However, one can derive a tighter analysis for a slightly modified, time-dependent regularization strategy for generalized linear models as we show in Sec. 3.3.

2.2 Adaptive Reweighting: Choosing the Right Loss

There is yet more freedom in the construction of the LR, via the selection of the loss function . Not only do we select the predictor sequence, but also the weights of the losses via w t . This idea allows controlling the influence of a particular data point ( x t , y t ) on the cumulative loss based on the value of x t . For example, if we know a priori that for a given x t our prediction will be most likely bad, we can opt out of using the pair ( x t , y t ) by setting w t = 0 . Below we will propose a weighting scheme that depends on a notion of bias , which captures how much of the error in predicting y t is due to our uncertainty about ˆ θ t (compared to the uncertainty we still would have knowing θ ⋆ ). Sometimes this bias is referred to as epistemic uncertainty in the literature, while the residual part of the error is referred to as aleatoric . Putting large weight on a data point heavily affected by this bias might unnecessarily increase the regret of our learner (and hence blow up the size of the confidence set). Note that, conveniently, even if we put low weight (zero) on a data point, nothing stops us from using this sample point to improve the estimator sequence in the next prediction round. As we will show below, our reweighting scheme is crucial in defining a practical algorithm for Reproducing Kernel Hilbert Space (RKHS) models and in high signal-to-noise ratio scenarios. Since we do not know θ ⋆ , our strategy is to compute an estimate of the bias of the estimator ˆ θ t and its effect on the value of the likelihood function for a specific x that we played. We use the value of the bias to rescale the loss via w t such that its effect is of the same magnitude as the statistical error (see Algorithm 1; we call this step BIAS-WEIGHTING).

Intuition To give a bit more intuition, suppose we have a Gaussian likelihood. Then the negative log-likelihood of ( x t , y t ) with weighting is proportional to w t σ 2 ( y t -x ⊤ t ˆ θ t ) 2 . Now, if x t does not lie in the span of the data points { x s } t -1 s =1 used to compute ˆ θ t , it is in general unavoidable to incur large error, inversely proportional to σ 2 . To see this, let us decompose the projection onto x t as

where the first term represents the statistical error up to time t , while the second, bias, is deterministic, and independent of the actual realization y , depending only θ ⋆ . Estimators with non-zero bias are biased . Plugging this into the likelihood function, we see that in expectation 1 σ 2 E [( y t -x ⊤ t ˆ θ t ) 2 |F t -1 ] ≲ 1 σ 2 bias 2 x t ( ˆ θ t ) + ϵ 2 + C t , where ϵ 2 is the unavoidable predictive error in expectation (due to a noisy objective) and is a constant independent of σ 2 . C t is the statistical error, and C is independent of σ 2 . Note that the bias term scales inversely with the variance, and leads to unnecessarily big confidence parameters for small σ 2 .

In fact, the problem is that we use the likelihood to measure the distance between two parameters, but this is only a 'good' distance once the deterministic source of the error (bias) vanishes. For this reason, without weighting, the incurred regret blows up severely in low-noise settings. To counter this, we balance the deterministic estimation bias and noise variance via proper selection of w t . In this case, it turns out that w t = σ 2 σ 2 +bias 2 x t ( ˆ θ t ) ensures that the overall the scaling is independent of σ 2 . While the choice of weights { w s } t s influences the geometry of the confidence sets, with a good data collection and estimation strategy the bias asymptotically decreases to zero, and hence the weights converge to 1 .

Bias estimation In order to generalize this rule beyond Gaussian likelihoods, we need a proper generalization of the bias. Our generalization is motivated by our analysis of generalized linear models, but the method can be applied more broadly. The role of the squared statistical error (variance) is played by the inverse of the smoothness constant of the negative log-likelihood functions f s , denoted by L . This is the usual smoothness, commonly seen in the convex optimization literature.

We consider penalized likelihood estimators with strongly convex regularizers (Alg. 1, line 3). For this estimator class, we define the bias via a hypothetical stochastic-error-free estimate ˆ θ × t , had we access to the expected values of the gradient loss functions (a.k.a. score). We use the first-order optimality conditions and the indicator function of the set Θ , i Θ , to define the error-free-estimate ˆ θ × t , and the bias of the estimator ˆ θ t as

where the expectation denotes a sequence of expectations conditioned on the prior filtration. This notion of bias coincides with the definition of bias in Eq. (4) for the Gaussian likelihood. This quantity cannot be evaluated in general, however, we prove a computable upper bound.

Theorem 2 (Bias estimate) . Let the negative log-likelihood have the form, -log p θ ( y s | x s ) = g ( x ⊤ s θ ) , where g : R → R is µ strongly-convex and let the regularizer be ψ t ( θ ) = λ || θ || 2 2 making the overall objective strongly convex. Then, defining V µ ; λ t = ∑ t s =1 µx s x ⊤ s + λ I , we can bound

The proof is deferred to App. A.3, and requires elementary convex analysis.

This leads us to propose the weighting scheme w t = 1 /L bias 2 x t ( ˆ θ t )+1 /L . We justify that this is a sensible choice by analyzing the confidence set on the GLM class in Section 3, which satisfies the smoothness and strong-convexity conditions. We show that this rule properly balances the stochastic and bias components of the error in the regret as in (3). However, this rule is more broadly applicable beyond the canonical representation of GLM or the GLM family altogether.

Algorithm 1 Constructing the LR Confidence Sequence

3 Theory: Linear Models

While the coverage (i.e., 'correctness') of the likelihood ratio confidence sets is always guaranteed, their worst-case size (affecting the 'performance') cannot be easily bounded in general. We analyze the size and the geometry of the LR confidence sequence in the special but versatile case of generalized linear models.

3.1 Generalized Linear Models

We assume knowledge of the conditional probability model p θ ( y | x ) , where the covariates x ∈ X ⊂ R d , and the true underlying model parameter lies in a set Θ ⊂ R d . If t is indexing (discrete) time, then x t is acquired sequentially, and the - subsequently observed y t is sampled from an exponential family distribution parametrized as

Here, A is referred to as the log-partition function of the conditional distribution, and T ( y ) is the sufficient statistic. The function h is the base measure, and has little effect on our further developments, as it cancels out in the LR. Examples of commonly used exponential families (Gaussian, Binomial, Poisson or Weibull) with their link functions can be found in Table 1 in App. A.1.

In order to facilitate theoretical analysis for online algorithms, we make the following assumptions about the covariates x ∈ X and the set of plausible parameters Θ .

Assumption 1. The covariates are bounded, i.e., sup x ∈X || x || 2 ≤ 1 , and the set Θ is contained in an ℓ 2 -ball of radius B . We will also assume that the log-partition function is strongly convex, that is, that there exists µ := inf z ∈ [ -B,B ] A ′′ ( z ) , and that A is L-smooth, i.e. L := sup z ∈ [ -B,B ] A ′′ ( z ) .

These assumptions are common in other works addressing the confidence sets of GLMs (Filippi et al., 2010; Faury et al., 2020), who remark that the dependence on µ is undesirable. However, in contrast to these works, our confidence sets do not use these assumptions in the construction of the sets. We only require these for our theoretical analysis. As these are worst-case parameters, the practical performance can be much better for our sets.

3.2 Geometry and Concentration

Before stating our results, we need to define a distance notion that the convex negative log-likelihoods induce. For a continuously differentiable convex function f , we denote the Bregman divergence as D f ( a, b ) := f ( a ) -f ( b ) - ∇ f ( b ) ⊤ ( a -b ) . The ν -regularized sum of log-partition functions is defined as

This function will capture the geometry of the LR confidence sets. The confidence set size depends mainly on two terms. One refers to a notion of complexity of the space referred to as Bregman information gain : Γ ν t ( ˜ θ t ) = log ( ∫ R d exp( -ν 2 || θ || 2 2 )d θ ∫ R d exp( -D Z ν t ( θ, ˜ θ t ))d θ ) , first defined by Chowdhury et al. (2022) as a generalization of the information gain of Srinivas et al. (2009), γ ν t = log ( det( ∑ i =1 µ ν x i x ⊤ i + I ) ) for Gaussian likelihoods. We will drop the superscript whenever the regularization is clear from context and simply refer to γ t . This term appears because one can relate the decay of the likelihood as a function of the Bregman Divergence from θ ⋆ with the performance of a (regularized) maximum likelihood estimator via convex (Fenchel) duality. In particular, if ˜ θ t is a regularized MLE, Γ ν t := Γ ν t ( ˜ θ t ) will asymptotically scale as O ( d log t ) (cf. Chowdhury et al., 2022, for further discussion). For Gaussian likelihoods and w s ≡ 1 , it coincides with the classical information gain independent of ˜ θ t . The second term that affects the size is the regret R t of the online prediction game over t rounds we introduced previously in (3). These two parts together yield the following result:

Theorem 3. Let ν > 0 and α, δ ∈ (0 , 1) . For the level 1 -α confidence set C t defined in (2) under the GLM in (7) , with probability 1 -δ , for all t ≥ 1 , any θ ∈ C t satisfies

where ξ t = ( log ( 1 α ) + νB 2 +Γ ν t ) and L, µ are defined as above and finally R t is the regret of the game in Eq. (3) .

The set defined via the above divergence does not coincide with the LR confidence set. It is slightly larger due to a term involving ν (as in Eq. (8)). This is a technical consequence of our proof technique, where the gradient of Z ν t needs to be invertible, and regularization is added to this end. We note that this ν > 0 can be chosen freely. Note that the theorem involves two confidence levels, α and δ : α is a bound on the Type I error - coverage of the confidence sets - while δ upper bounds the probability of a large radius - and is therefore related to the power and Type II error of a corresponding hypothesis test. The proof of the theorem is deferred to App. B.2.

To give more intuition on these quantities, let us instantiate them for the Gaussian likelihood case with w s ≡ 1 . In this scenario, Z ν t ( θ ) = ∑ t s =1 1 2 σ 2 || θ || 2 x s x ⊤ s + ν 2 || θ || 2 2 , and the (in this case symmetric) Bregman divergence is equal to D Z ν t ( θ ⋆ , θ ) = 1 2 || θ -θ ⋆ || 2 V σ -2 ; ν t , where V µ ; ν t = ∑ t s =1 µx s x ⊤ s + ν I , which means that our confidence sets are upper bounded by a ball in the same norm as those in the seminal work on linear bandits (Abbasi-Yadkori et al., 2011).

3.3 Online Optimization in GLMs: Follow the Regularized Leader

The size of the confidence sets in Theorem 3 depends on the regret of the online prediction game involving the estimator sequence. We now bound this regret when using the Follow-the-RegularizedLeader (FTRL) algorithm in this setting. This high probability bound is novel to the best of our

knowledge and may be of independent interest. We state in a weight-agnostic manner first, and then with our particular choice. The latter variant uses a specifically chosen regularizer. In this case, we can track the contribution of each time-step towards the regret separately.

Theorem 4. Let ψ t ( θ ) = λ || θ || 2 2 . Assume Assumption 1, and additionally that A is L -smooth everywhere in R d , and let w t ∈ [0 , 1] be arbitrary. Then, with probability 1 -δ the regret of FTRL (Alg. 1) satisfies for all t ≥ 1

The regret bounds are optimal in the orders of γ λ t , matching lower bounds of Ouhamma et al. (2021), as for linear models γ t = O ( d log t ) . Combining results of Thm. 4 with Thm. 3, we get a confidence parameter that scales with O ( √ γ t ) , for confidence sets of the form || θ -θ ⋆ || V t , which coincides with the best-known confidence sets in this setting in the worst-case (Abbasi-Yadkori et al., 2012). The requirement of global L -smoothness can be relaxed to L -smoothness over Θ . With a more elaborate (but less insightful) analysis, we can show that we achieve a ˜ O ( γ t ) bound even in this case. The proofs of these results are deferred to App. C.4, App. C.5 and App. C.6 respectively.

Regret, Weighting and Estimation Bias Interestingly, the term in Thm. 4 involving the (crude) proxy to the bias - the bound B - is not scaled by the same L/µ factors as the other terms in the regret bound (10) and in Theorem 3. Namely, the prefactor is L 2 /µ instead of L/µ . This extra dependence manifests itself in the unnecessary penalization through the estimation bias we introduced in Sec. 2.2, particularly in low-noise settings. We addressed this issue by picking the weights { w t } . While the above theorem holds for any valid weighting, it does not exhibit the possible improvement from using specific weights.

We argued earlier that the error in prediction should not be measured by the likelihood function if there is deterministic error, since initially, we are fully uncertain about the value of θ ⋆ ⊤ ( · ) outside the span of previous observations. Of course, if our goal would be to purely pick weights to minimize R t , then w s = 0 would lead to zero regret and hence be optimal. However, the likelihood ratio would then be constant, and uninformative. In other words, the associated log-partition Bregman divergence in Theorem 3 would be trivial and not filter out any hypotheses. Clearly, some balance has to be met. With this motivation in mind, we proposed a nonzero weighting that decreases the regret contribution of the bias, namely w t = 1 /L 1 /L +bias 2 x t ( ˆ θ t ) . The advantage of this choice becomes more apparent when we use the regularizer ψ t ( θ ) = λ || θ || 2 + A ( x ⊤ t θ ) to obtain the following result.

Theorem 5. Let ψ s ( θ ) = λ || θ || 2 + A ( x ⊤ s θ ) . Assume Assumption 1, and additionally that A is L -smooth everywhere in R d , and choose w s = 1 /L 1 /L +bias xs ( ˆ θ s ) 2 . Additionally, let the sequence of x s be such that, ∑ s (1 -w s )( f s ( θ ⋆ ) -f s ( ¯ θ s +1 )) ≤ L/µγ λ t , where ¯ θ s is the FTRL optimizer with the regularizer λ || θ || 2 2 from Theorem 4 2 . Then, with probability 1 -δ the regret of FTRL (Alg. 1) satisfies for all t ≥ 1

One can see that for points where the information gain ∆ γ s is large (corresponding to more unexplored regions of the space, where the deterministic source of error is then large), the weighting scheme will make sure that the multiplicative contribution of B 2 is mitigated, along with having the correct prefactor L/µ . The reader may wonder how this result is useful when we replace bias 2 x s ( ˆ θ s ) with the upper bound from Thm. 2. While instructive, our bound still only makes the bias proxy B 2 appear in front of the information gain ∆ γ t , instead of the more desireable bias itself. In the latter case, we could also directly make use of the upper bound and get an explicit result only using an upper bound on the bias. We leave this for future work.

We point out that this choice of ψ s ( θ ) in Theorem 5 corresponds to the Vovk-Azoury-Warmuth predictor (Vovk, 2001; Azoury and Warmuth, 1999) in the online learning literature. This choice is helpful in order to track the bias contribution more precisely in our proof.

2 Note that this assumption was missing in an earlier version. ¯ θ s +1 corresponds to a regularized MLE that did observe the data pair ( x s , y s ) .

4 Application: Linear and Kernelized Bandits

Our main motivation to construct confidence sets is bandit optimization. A prototypical bandit algorithm - the Upper Confidence Bound (UCB) (Auer, 2002) - sequentially chooses covariates x s in order to maximize the reward ∑ t s =1 r θ ⋆ ( x s ) , where r θ ⋆ is the unknown pay-off function parametrized by θ ⋆ . UCB chooses the action x s which maximizes the optimistic estimate of the reward in each round, namely

where C s -1 is some confidence set for θ ⋆ , and can be constructed with Algorithm 1 from the first s -1 data points. An important special case is when r θ ⋆ is linear (Abe and Long, 1999) or modelled by a generalized linear model (Filippi et al., 2010). In that case, the inner optimization problem is convex as long as C s -1 is convex. The outer optimization is tractable for finite X . In the applications we consider, our confidence sets are convex, and we easily solve the UCB oracle using convex optimization toolboxes.

Extension to RKHS We introduced the framework of LR confidence sets only for finitedimensional Euclidean spaces. However, it can be easily extended to Reproducing Kernel Hilbert Spaces (RKHS) (Cucker and Smale, 2002). The definition of the LR process in (1) is still well-posed, but now the sets are subsets of the RKHS, containing functions f ∈ H k . An outstanding issue is how to use these sets in downstream applications, and represent them tractably as in Figure 1. Conveniently, even with infinite-dimensional RKHSs, the inner-optimization in (11) admits a Lagrangian formulation, and the generalized representer theorem applies (Schölkopf et al., 2001; Mutný and Krause, 2021). In other words, we can still derive a pointwise upper confidence band as ucb( x ) = max f ∈H k , || f || k ≤ B,f ∈ C s ⟨ f, k ( x, · ) ⟩ in terms of { x j } s j =1 ∪ { x } , leading to a s +1 -dimensional, tractable optimization problem.

We also point out that the weighting is even more paramount in the RKHS setting, as the bias never vanishes for many infinite dimensional Hilbert spaces (Mutný and Krause, 2022). For this purpose, our weighting is of paramount practical importance, as we can see in Figure 2a), where the gray arrow represents the significant improvement from reweighting.

4.1 Instantiation of the Theory for Linear Bandits

Before going to the experiments, we instantiate our theoretical results from Sec. 3 to the important and well-studied special case of linear payoffs. In that case, r θ ( x ) = ⟨ x, θ ⟩ and the agent observes y s = ⟨ x s , θ ⋆ ⟩ + η s upon playing action x s , where η s ∼ N (0 , σ 2 ) . We are interested in minimizing the so-called cumulative pseudo-regret, namely, R t = ∑ t s =1 [ ⟨ x ⋆ , θ ⋆ ⟩ - ⟨ x s , θ ⋆ ⟩ ] , where x ⋆ refers to the optimal action. Using the set from (2) along with Theorem 3 and the FTRL result of Theorem 4 we can get a regret bound for the choice w s ≡ 1 .

Theorem 6. Let w s ≡ 1 . For any λ ≥ 1 σ 2 , with probability at least 1 -3 δ , for all t ∈ N we have

Our results are optimal in both d and t up to constant and logarithmic factors. The proof is deferred to App. D, but is an instantiation of the aforementioned theorems, along with a standard analysis. There, we also compare to the seminal result of Abbasi-Yadkori et al. (2011), which does not suffer from the dependence on B √ γ t . We attribute this to the incurred bias in the absence of the reweighting scheme.

For the weighted likelihood ratio, we can obtain a result similar to the above, but multiplied by an upper bound on sup s ≥ 1 w -1 s . This is undesirable, as our experiments will show that the reweighting scheme vastly improves performance. While this could be somewhat mitigated by using the Theorem 5 instead of Theorem 4 to bound the FTRL regret, a better result should be achievable using our weighting scheme that improves upon Theorem 6 and possibly even matches Abbasi-Yadkori et al. (2011) exactly in the worst-case. We leave this for future work.

4.2 Experimental Evaluation

In this subsection, we demonstrate that the practical applicability goes well beyond the Gaussian theoretical result from the previous subsection. In the examples below, we always use the UCB

60 a)

GaussianLikelihood

100 b)

Laplace Likelihood

55

weighting

80

50

Regret

45

60

R

40

35

20

30

25

0

50

100

150

200

250

0

0

50

100

150

200

250

Iterations

Iterations

100 C)

200 d)

PoissonBandits

175

80

150

Regret

60

0100

40

75

50

20

25

0

0

50

100

150

200

250

0

0

50

100

150

200

250

300

Iterations

Iterations

AY (2011)

LR (no weights)

NR (2021)

MK (2021)

LR (Our Alg.1)

Heuristic β = 2log(1/6)

Sub-Exp

Laplace

algorithm but employ different confidence sets. In particular, we compare our LR confidence sets for different likelihood families with alternatives from the literature, notably classical sub-family confidence sets (Abbasi-Yadkori et al., 2011; Mutný and Krause, 2021), and the robust confidence set of Neiswanger and Ramdas (2021). In practice, however, the radius of these confidence sets is often tuned heuristically. We include such sets as a baseline without provable coverage as well. The main take-home message from the experiments is that among all the estimators and confidence sets that enjoy provable coverage, our confidence sets perform the best, on par with successful heuristics. For all our numerical experiments in Figure 2, the true payoff function is assumed to be an infinite dimensional RKHS element. For further details and experiments, please refer to App. E.

Additive Noise Models Suppose that r θ ⋆ is linear and we observe y s = x ⊤ s θ ⋆ + η s , where η s is additive noise, and θ ⋆ is an element of a Hilbert space. We consider classical Gaussian noise as well as Laplace noise in Fig. 2[a), b)]. Notice that in both cases our confidence sets yield lower regret than any other provably valid method. In both cases they are performing as good as heuristic confidence sets with confidence parameter β t ≡ 2 log(1 /δ ) . The sub-Gaussian confidence sets of Abbasi-Yadkori et al. (2011) (AY 2011) are invalid for the Laplace distribution as it is not sub-Gaussian but only subExponential. For this reason, we compare also with sub-exponential confidence sets derived similarly to those of (Faury et al., 2020). The confidence sets of (Neiswanger and Ramdas, 2021) (NR 2021) perform similarly on Gaussian likelihood, but are only applicable to this setting, as their generalization to other likelihood families involves intractable posterior inference. We note also the difference between the unweighted LR and the weighted one. The examples in Fig. 2 use the true payoff functions r ( x ) = -(1 . 4 -3 x ) sin(18 x ) , which we model as an element of a RKHS with squared exponential kernel lengthscale γ = 6 × 10 -2 on [0 , 1 . 2] , which is the baseline function no. 4 in the global optimization benchmark database infinity77 (Gavana, 2021). Additional experiments can be found in App. E.

Poisson Bandits A prominent example of generalized linear bandits (GLB) are Poisson bandits, where the linear payoff is scaled by an exponential function. We instantiate our results on a common benchmark problem, and report the results in Fig. 2d). We improve the regret of UCB for GLBs compared to two alternative confidence sets: one that uses a Laplace approximation with a heuristic confidence parameter, and one inspired by considerations in Mutný and Krause (2021) (MK 2021), also with a heuristic confidence parameter. Note that we cannot compare directly to their provable results in their original form as they do not state them in the canonical form of the exponential family.

Survival Analysis Survival analysis is a branch of statistics with a rich history that models the lifespan of a service or product (Breslow, 1975; Cox, 1997; Kleinbaum and Klein, 2010). The classical approach postulates a well-informed likelihood model. Here, we use a specific hazard model, where the survival time T is distributed with a Weibull distribution, parametrized by λ and p . The rate λ θ ( x ) = exp( x ⊤ θ ) differs for each configuration x , and p - which defines the shape of the survival distribution - is fixed and known. We assume that the unknown part is due to the parameter θ which is the quantity we build a confidence set around to use within the UCB Algorithm. In particular, the probability density of the Weibull distribution is P ( T = t | x ) = λ θ ( x ) pt p -1 exp( -t p λ θ ( x )) . In fact, with p = 2 , the confidence sets are convex and the UCB rule can be implemented efficiently.

Interestingly, this model admits an alternate linear regression formulation. Namely upon using the transformation Y = log T , the transformed variables Y | x follow a Gumbel-type distribution, with the following likelihood that can be obtained by the change of variables P ( Y = y | x ) = λ θ ( x ) p exp( y ) p exp( -exp( y ) p λ θ ( x )) . The expectation over Y allows us to express it as a linear regression problem since E [ Y | x ] = -( θ ⊤ x + γ ) /p , where γ is the Euler-Mascheroni constant. More importantly, Y | x is sub-exponential. Hence, this allows us to use confidence sets for sub-exponential variables constructed with the pseudo-maximization technique inspired by Faury et al. (2020). More details on how these sets are derived can be found in App. E. However, these approaches necessarily employ crude worst-case bounds and as can be seen in Figure 2c) the use of our LR-based confidence sequences substantially reduces the regret of the bandit learner.

5 Related Work and Conclusion

Related Work The adaptive confidence sequences stem from the seminal work of Robbins et al. (1972), who note that these sets have α -bounded Type I error. The likelihood ratio framework has been recently popularized by Wasserman et al. (2020) for likelihood families without known test statistics under the name universal inference . This approach, although long established, is surprisingly uncommon in sequential decision-making tasks like bandits. This might be due to the absence of an analysis deriving the size of the confidence sets (Mutný and Krause, 2021), a necessary ingredient to obtain regret bounds. We address this gap for generalized linear models. Another reason might be that practitioners might be interested in non-parametric subfamilies - a scenario our method does not cover. That being said, many fields such as survival analysis (Cox, 1997) do have well-informed likelihoods. However, most importantly, if used naively, this method tends to fail when one departs from assumptions that our probabilistic model is identifiable (i.e., p θ ( · | x ) = p ˜ θ ( · | x ) even if θ = ˜ θ ). We mitigate this problem by introducing the scaling parameters w t in Eq. (1) to deal with it.

̸

Prevalent constructions of anytime-valid confidence intervals rely on carefully derived concentration results and for a specific estimator such as the least-squares estimator and noise sub-families such as sub-Gaussian, sub-Bernoulli and sub-Poisson Abbasi-Yadkori et al. (2011); Faury et al. (2020); Mutný and Krause (2021). Their constructions involve bounding the suprema of collections of self-normalized stochastic processes (Faury et al., 2020; Mutný and Krause, 2021; Chowdhury et al., 2022). To facilitate closed-form expressions, worst-case parameters are introduced that prohibitively affect the size of the sets - making them much larger than they need to be.

Chowdhury et al. (2022) use the exact form of the likelihood to build confidence sets for parameters of exponential families. However, their approach is restricted to exponential family distributions. They use self-normalization and mixing techniques to explicitly determine the size of the confidence set and do not use an online learning subroutine as we do here. Neiswanger and Ramdas (2021) use likelihood ratios for bandit optimization with possibly misspecified Gaussian processes but is not tractable beyond Gaussian likelihoods. The relation between online convex optimization and confidence sets has been noted in so-called online-to-confidence conversions (Abbasi-Yadkori et al., 2012; Jun et al., 2017; Orabona and Jun, 2021; Zhao et al., 2022), where the existence of a low-regret learner implies a small confidence set. However, these sets still use potentially loose regret bounds to define confidence sets. Our definition is implicit . We do not necessarily need a regret bound to run our method, as the radius will depend on the actual, instance-dependent performance of the learner.

Conclusion In this work, we generalized and analyzed sequential likelihood ratio confidence sets for adaptive inference. We showed that with well-specified likelihoods, this procedure gives small, any-time valid confidence sets with model-agnostic and precise coverage. For generalized linear models, we quantitatively analyzed their size and shape. We invite practitioners to explore and use this very versatile and practical methodology for sequential decision-making tasks.

Acknowledgments and Disclosure of Funding

We thank Wouter Koolen and Aaditya Ramdas for helpful discussions as well as for organizing the SAVI workshop where these discussions took place. NE acknowledges support from the Swiss Study Foundation and the Zeno Karl Schindler Foundation. MM has received funding from the Swiss National Science Foundation through NFP75. This publication was created as part of NCCR Catalysis (grant number 180544), a National Centre of Competence in Research funded by the Swiss National Science Foundation.

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvari, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320.

Abbasi-Yadkori, Y., Pal, D., and Szepesvari, C. (2012). Online-to-confidence-set conversions and application to sparse stochastic bandits. In Lawrence, N. D. and Girolami, M., editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics , volume 22 of Proceedings of Machine Learning Research , pages 1-9, La Palma, Canary Islands. PMLR.

Abe, N. and Long, P. M. (1999). Associative reinforcement learning using linear probabilistic concepts. In ICML , pages 3-11. Citeseer.

Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422.

Azoury, K. S. and Warmuth, M. K. (1999). Relative loss bounds for on-line density estimation with the exponential family of distributions. Machine Learning , 43:211-246.

Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature , 533:452-454.

Breslow, N. E. (1975). Analysis of survival data under the proportional hazards model. International Statistical Review / Revue Internationale de Statistique .

Carpentier, A., Vernade, C., and Abbasi-Yadkori, Y. (2020). The elliptical potential lemma revisited. ArXiv , abs/2010.10182.

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, Learning, and Games . Cambridge University Press, New York, NY, USA.

Chowdhury, S. R., Saux, P., Maillard, O.-A., and Gopalan, A. (2022). Bregman deviations of generic exponential families. arXiv preprint arXiv:2201.07306 .

Cox, D. R. (1997). Some remarks on the analysis of survival data. In Proceedings of the First Seattle Symposium in Biostatistics , pages 1-9. Springer.

Cucker, F. and Smale, S. (2002). On the mathematical foundations of learning. Bulletin of the American mathematical society , 39(1):1-49.

Faury, L., Abeille, M., Calauzènes, C., and Fercoq, O. (2020). Improved optimistic algorithms for logistic bandits. In ICML2020: Proceedings of the 37th International Conference on International Conference on Machine Learning .

Filippi, S., Cappe, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. In Advances in neural information processing systems , pages 586-594.

Gavana, A. (2021). infinity global optimization benchmarks and ampgo. http://infinity77. net/global_optimization/index.html# .

Hazan, E. (2016). Introduction to online convex optimization. Found. Trends Optim. , 2:157-325.

Hazan, E., Agarwal, A., and Kale, S. (2006). Logarithmic regret algorithms for online convex optimization. Machine Learning , 69:169-192.

Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. (2018). Time-uniform chernoff bounds via nonnegative supermartingales. Probability Surveys , 17:257-317.

Jun, K.-S., Bhargava, A., Nowak, R., and Willett, R. (2017). Scalable generalized linear bandits: Online computation and hashing. Advances in Neural Information Processing Systems , 30.

Katz-Samuels, J. and Jamieson, K. (2020). The true sample complexity of identifying good arms. In Chiappa, S. and Calandra, R., editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 1781-1791. PMLR.

Kleinbaum, D. G. and Klein, M. (2010). Survival analysis , volume 3. Springer.

Lattimore, T. and Szepesvári, C. (2020). Bandit Algorithms . Cambridge University Press.

Makarova, A., Usmanova, I., Bogunovic, I., and Krause, A. (2021). Risk-averse heteroscedastic bayesian optimization. In Proc. Neural Information Processing Systems (NeurIPS) .

McCullagh, P. (2018). Generalized linear models. Chapman and Hall.

Mukherjee, S., Tripathy, A. S., and Nowak, R. (2022). Chernoff sampling for active testing and extension to active regression. In International Conference on Artificial Intelligence and Statistics , pages 7384-7432. PMLR.

Mutný, M. and Krause, A. (2021). No-regret algorithms for capturing events in poisson point processes. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 78947904. PMLR.

Mutný, M. and Krause, A. (2022). Experimental design of linear functionals in reproducing kernel hilbert spaces. In Proc. Neural Information Processing Systems (NeurIPS) .

Neiswanger, W. and Ramdas, A. (2021). Uncertainty quantification using martingales for misspecified gaussian processes. In Feldman, V., Ligett, K., and Sabato, S., editors, Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132 of Proceedings of Machine Learning Research , pages 963-982. PMLR.

Orabona, F. (2019). A modern introduction to online learning.

Orabona, F. and Jun, K.-S. (2021). Tight concentrations and confidence sequences from the regret of universal portfolio. IEEE Transactions on Information Theory , 70:436-455.

Ouhamma, R., Maillard, O.-A., and Perchet, V. (2021). Stochastic online linear regression: the forward algorithm to replace ridge. In Neural Information Processing Systems .

Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. (2022). Game-theoretic statistics and safe anytime-valid inference.

Robbins, H., Siegmund, D., et al. (1972). A class of stopping rules for testing parametric hypotheses. In Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume 4: Biology and Health . The Regents of the University of California.

Schölkopf, B., Herbrich, R., and Smola, A. (2001). A generalized representer theorem. In Computational learning theory , pages 416-426. Springer.

Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning .

Ville, J.-L. (1939). Étude critique de la notion de collectif.

Vovk, V. (2001). Competitive on-line statistics. International Statistical Review , 69.

Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint . Cambridge university press.

Wald, A. (1945). Sequential tests of statistical hypotheses. Annals of Mathematical Statistics , 16:256-298.

Wasserman, L. A., Ramdas, A., and Balakrishnan, S. (2020). Universal inference. Proceedings of the National Academy of Sciences , 117:16880 - 16890.

Zhao, H., Zhou, D., He, J., and Gu, Q. (2022). Bandit learning with general function classes: Heteroscedastic noise and variance-dependent regret bounds. ArXiv , abs/2202.13603.

Peer reviews:
comments: SUMMARY:
This paper proposed to use the likelihood ratio approach to provide an any-time valid confidence sequence, which is suitable for problems with well-specified likelihood. It discusses how to provably choose the best sequence of estimators and sheds light on connections to online convex optimization. To counteract the initially large bias of estimators, they propose a reweighing scheme. They also provide a non-asymptotic analysis of the likelihood ratio confidence set size for GLM, and performed numerical simulations.

STRENGTHS:
The idea of using the universal inference method in bandit problems is interesting.

WEAKNESSES:
In bandit problems, the target of interest is regret, and the validity/length of the confidence set is a tool of analysis. One shortcoming of this paper is that there is no regret analysis (of the bandit problem) of the proposed method. One important question is, what is the regret of the proposed method on simple bandit problems such as the K-armed bandit (UCB method gives \sqrt{KT} regret) and the linear bandit (LinUCB gives \sqrt{d^2 T} regret). Since bandit is the focus of this paper, without a concrete regret bound (showing that it matches the best existing method), the benefit of the proposed method is not convincing. 

The method proposed in this paper is similar to the OMLE method [1], which has a provable regret guarantee in many RL problems. One interesting question is to study whether the proposed algorithm has benefit over the OMLE algorithm. 

[1] Optimistic MLE—A Generic Model-based Algorithm for Partially Observable Sequential Decision Making. 

After reading the rebuttal: 
I am convinced that a regret bound can be obtained for this method. I trust the authors will add this in the revised version.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the confidence set of an estimator, defined by a likelihood ratio. The contributions stated within the work, alongside my corresponding queries, are outlined as follows:\n* For generalized linear models, we theoretically analyze the geometry of the LR confidence sets under mild assumptions. We show their geometry is dictated by Bregman divergences of exponential families (Chowdhury et al., 2022). \n*We show that the size of the confidence set is dictated by an online prediction game. The size of these sets depends on a sequence of estimators \\{\\theta_s\\}_{s=1}^t that one uses to estimate the unknown parameter \\theta_*. We discuss how to pick the estimator sequence in order to yield a provably small radius of the sets, by using the Follow-the-Regularized-Leader algorithm, which implements a regularized maximum-likelihood estimator. We prove that the radius of the confidence sets is nearly-worst-case optimal, and accordingly they yield nearly-worst-case regret bounds when used in generalized linear bandit applications. However, due to their data-dependent nature, they can be much tighter than this theory suggests. \n*We analyze limitations of classical (un-weighted) LR sets when the underlying conditional observation model is not identifiable. In this case, the resulting (inevitable) estimation bias unnecessarily increases the size of the confidence sets. To mitigate this, we propose an adaptive reweighting scheme that decreases the effect of uninformed early bias of the estimator sequence on the size of the sets downstream. The reweighting does not affect the coverage guarantees of our sets, and utilizes an elegant connection to (robust) powered likelihoods (Wasserman et al., 2020). \n* Thanks to the adaptive reweighting scheme, our sets are very practical as we showcase experimentally. We demonstrate that our method works well with exponential and non-exponential family likelihoods, and in parametric as well as in kernelized settings. We attribute their practical benefits to the fact that they do not depend on (possibly loose) worst-case parameters.\n\nSTRENGTHS:\nThe research question posed in this paper is notably intriguing and holds significant potential for various applications. Furthermore, the innovative approach adopted by the authors represents a substantial contribution to the field.\n\nWEAKNESSES:\nWhile the paper offers several valuable insights, I noticed that the overall integration of its contributions could be improved. For instance, claimed contributions 1 and 3, which could potentially have been effectively merged, were instead treated separately. This compromise the contribution significantly from my perspective. Furthermore, I have concerns about the clarity and accuracy of some claims. For more specific observations, please refer to the \"Questions\" section. I believe addressing these issues would greatly enhance the coherence and validity of the work.\n\nQUESTIONS:\n* The relationship between LR confidence set geometry and Bregman divergences is briefly discussed on page 6. It would be beneficial to have this connection elaborated upon more thoroughly in the main body of the text.\n\n* Section 2 provides a clear explanation on how to select a sequence of $\\{ \\theta_s \\}_{s=1}^t$ to minimize the radius of the confidence set. However, this connection becomes less clear in Section 3. \n\nMoreover, the paper claims that the radius of the confidence set is nearly-worst-case optimal. Could you please provide further explanation as to why this is the case? While it's clear that the FTRL can lead to nearly-worst-case optimal regret for the online optimization problem, $\\mathcal{R}_t$ defined in $(3)$ appears to be not identical, but smaller than the definition of regret in the online optimization problem. If $\\mathcal{R}_t$ is significantly smaller than the regret of the FTRL algorithm, might our confidence set not be optimal?\n\n* The introduction of a re-weighting scheme is intriguing. However, given that the re-weight $w_t$ is difficult to compute, could you provide any theoretical performance guarantees? Additionally, in Theorem 2, the bias estimate depends on \\norm{\\theta_*}_2^2. Could this potentially conflict with our objective to estimate the confidence set of \\theta_*? Furthermore, in Section 3, we ascertain the geometry of the linear models assuming $w_t = 1$. Is it possible to perform a similar analysis incorporating the re-weighting scheme? Otherwise, the value of the re-weighting scheme might be discounted if we cannot incorporate it into the analysis of simple model\n\n* A query arises from Theorem 3, where \\theta \\in \\mathcal{C}_t satisfies a certain inequality. I am curious about the application of this inequality in estimating the confidence set of \\theta_*, given that the Bregman distance involved depends on $\\theta_*$, which is unknown. Is this result primarily of theoretical interest, or is it of practical use in this context?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use likelihood ratios to construct confidence sequences that facilitate the downstream online decision making under uncertainty. The weighting and corresponding bias estimation are proposed to avoid regret blow-up in low-noise setting. The paper offers theoretical insights for the bias estimation, the geometry of the confidence set for generalized linear model (GLM), and regret in the online optimization in GLM. The paper offers the corresponding empirical results on bandits problems.\n\nSTRENGTHS:\n1. The paper incorporates the likelihood ratios into online decision-making. The LR confidence set reduces the requirement of conventional concentration results and allows existing algorithms to solve more generalized problems as long as the likelihood is well specified. There is room for studying its application in various sequential decision-making tasks.\n2. The adaptive weighting scheme is well justified with practical bias estimation.\n3. The geometry analysis could be helpful to justify the usage of the LR confidence set rather than alternatives.\n\nWEAKNESSES:\n1. In general the paper is clearly organized. However, the scattered yet closely related notations in sections 2 and 3 make it challenging to follow. I would recommend further highlighting important definitions and formulas or trying to concentrate on them to allow easier revisits.\n2. The experiment results shown in Figure 2 lack statistical significance. The fact that the figure shows median values rather than mean values indicates the stability problem of the experiment results.\n3. The mismatch between the theoretical results in section 3 with regards to the online optimization and the empirical study mostly focused on bandit problems in section 4. As only the downstream cumulative regrets are shown.\n\nQUESTIONS:\n1. Minor issue: the product on the denominator of eq (1) seems to be falsely indexed, could the author double-check?\n2. In the bandit problems, how are the confidence levels specified?\n3. Could the author briefly comment on the potential impact of misspecified likelihood (robustness to misspecification when there is a uniform error bound for the misspecification)?\n\nLIMITATIONS:\nDiscussed in the comments above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new construction of confidence sets for a parametric setting, where the likelihood of the noise process is explicitly given.\nThe proposed method is based on a weighted variant of the sequential likelihood ratio (LR) statistics, which was proposed in universal inference (Wasserman et al., 2020).\nThe confidence set is a function of the choice of estimators that construct the sequential LR statistics, and the paper proposes a specific method to construct such estimators by the means of Follow-the-Regularized-Leader (FTRL) algorithm in online learning, by viewing the confidence parameter minimization as a regret minimization problem.\nFurther, the paper proposes a new adaptive reweighting scheme, which uses a powered LR statistics to make the resulting confidence set more “robust”. The proposed reweighting scheme is based on the bias of the used estimators. While the bias may not be directly estimated, they provide a computable upper bound on the bias for a certain form of distributions.\nFor generalized linear models, the size of the confidence set is analyzed under regularity assumptions. To make the bound fully concrete, they also provide a high-probability regret guarantee of the used FTRL estimators (for GLMs).\nThe effectiveness of the proposed method is demonstrated with linear and kernelized bandit experiments.\n\nSTRENGTHS:\nThe paper is very well written. The proposed framework and analysis are a nice and careful combination of several different techniques (universal inference, FTRL, Bregman information gain, …) in statistics and online learning.\nI think this could be indeed useful in bandit problems but it can be also applicable whenever parametric models are available.\n\nWEAKNESSES:\nI do not find a crucial weakness of this paper, except a few questions detailed below.\n\nQUESTIONS:\n- I find the sentence in line 107 a bit confusing. It says “If the confidence parameter goes to zero, only $\\theta_\\star$ is in the set.” This is not the case, as the LHS (of the inequality inside the set notation) in the equation after line 103 is 1 if $\\theta=\\theta_\\star$, and so $\\theta_\\star$ will be excluded if the RHS goes to zero. What should be a proper explanation?\n- I cannot make much sense of the specific reweighting scheme in line 162 right away, though I can understand the intuition thanks to the Gaussian example. Can you elaborate this choice?\n\nA possible typo:\n- Line 153: depending only $\\theta_\\star$\n\nLIMITATIONS:\nLimitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}