Title: Almost Free: Self-concordance in Natural Exponential Families and an Application to Bandits
Venue: NeurIPS 2024
Accepted: yes
Keywords: bandits

generalized linear models

natural exponential families

self-concordance.
Abstract: We prove that single-parameter natural exponential families with subexponential tails are self-concordant with polynomial-sized parameters. For subgaussian natural exponential families we establish an exact characterization of the growth rate of the self-concordance parameter. Applying these findings to bandits allows us to fill gaps in the literature: We show that optimistic algorithms for generalized linear bandits enjoy regret bounds that are both second-order (scale with the variance of the optimal arm's reward distribution) and free of an exponential dependence on the bound of the problem parameter in the leading term. To the best of our knowledge, ours is the first regret bound for generalized linear bandits with subexponential tails, broadening the class of problems to include Poisson, exponential and gamma bandits.

Almost Free: Self-concordance in Natural Exponential Families and an Application to Bandits

Abstract

We prove that single-parameter natural exponential families with subexponential tails are self-concordant with polynomial-sized parameters. For subgaussian natural exponential families we establish an exact characterization of the growth rate of the self-concordance parameter. Applying these findings to bandits allows us to fill gaps in the literature: We show that optimistic algorithms for generalized linear bandits enjoy regret bounds that are both second-order (scale with the variance of the optimal arm's reward distribution) and free of an exponential dependence on the bound of the problem parameter in the leading term. To the best of our knowledge, ours is the first regret bound for generalized linear bandits with subexponential tails, broadening the class of problems to include Poisson, exponential and gamma bandits.

1 Introduction

Single-parameter natural exponential families (NEFs) [Mor82] abound in statistical applications [MN89; Bro86; WJ08; ML09]. In this paper we study properties of NEFs and in doing so we make two main contributions:

We study how tail properties of the base distribution of a NEF impose limits on the NEF: if the base distribution is subexponential (subgaussian), we show that the NEF is selfconcordant with a stretch factor that grows inverse quadratically (respectively, linearly)

In generalized linear bandits whose reward distributions follow a NEF with subexponential base distribution, we show how this new result can be utilized to derive a novel second order regret bound whose leading term is free of exponential dependencies on the problem parameter - the first such result for this setting.

The class of distributions our results extend to includes: normal, Poisson, exponential, gamma and negative binomial distributions. Our findings partially address conjectures on whether generalized linear models with unbounded targets are self-concordant [Bac10; Fau+20]. This significantly generalizes the case when the targets are assumed to be bounded 1 [Mar+19; OB21; Rus+21] and thus extends the applicability of the results therein.

Alex Ayoub

Csaba Szepesvári

Self-concordance in NEFs turns out to be useful for both optimization and statistical estimation. The self-concordance property controls the remainder term, or approximation error, of a NEF's secondorder Taylor expansion. This is useful in designing and analyzing both estimation and optimization methods. Historically the self-concordance property was first found to be useful for optimization [NN89; NT08; ST19] and later for statistical estimation [Bac10; Mar+19; OB21; BFR23]. In this paper, we will employ the self-concordance property in bandit problems [LS20] where it helps with controlling the error terms related to estimation.

Generalized linear bandits (GLBs) [Fil+10] has emerged as a standard framework for studying the role that nonlinear function approximation plays in decision making problems. Earlier works on GLBs (or its special case of of logistic bandits) [Fil+10; LLZ17; SPL23] naively approximates the nonlinear function with a linear first order Taylor expansion. This approach ends up paying a price in the leading term of the regret bound that is exponential in the size of the true underlying parameter. In logistic bandits, [Fau+20] were the first to exploit the self-concordance property of the Bernoulli distribution in order to get regret bounds free of an exponential dependence on the size of the problem parameter in the leading term. [Fau+20] use self-concordance to get a tighter second order Taylor expansion that better captures the curvature of the sigmoid function. Employing improved self-concordant analysis, [AFC21] get second-order regret bounds for logistic bandits and [Jan+24] extend these results to GLBs, under the assumption that the underlying reward distributions are self-concordant. We build upon this line of research and fill a gap in the bandit literature by designing and analyzing algorithms for GLBs with subexponential reward distributions. To the best of the authors' knowledge, our work considers the most general setting in the sense that all the previous works on GLBs considered subgaussian or bounded reward distributions while ours consider subexponential ones. For subgaussian rewards, these works [Fil+10; Jun+21; SPL23; Rus+21] still depend on κ . Note that [Rus+21], also employs self-concordance in GLBs but they assume bounded reward and focuses on addressing non-stationarity of the environment. In addition, their bounds also scale with κ in the leading term, which can be exponentially large for logistic bandits. [Jan+24] consider a similar setting to ours. They assume the moment generating function of the base distribution Q is defined over the entire real line. This implies that Q does not have a tail as heavy as an exponential distribution hence less general than our setting. A concurrent work [LYJ24] develops novel confidence sets for self-concordant GLBs to improve the theoretical bounds while we prove that all GLBs with light-tailed base distribution are self-concordant. An interesting future direction would be applying their techniques to improve the bandit result of our work. [Saw+24] considers GLBs with bounded reward, which is known to be self-concordant, in a regime that only a limited number of decision policy updates is available.

The paper is organized as follows: In Section 2, we introduce single-parameter NEFs and review key properties relevant to our analysis. Section 3 demonstrates the self-concordance property of subexponential (subgaussian) NEFs, with a quadratic (respectively, linear) growth rate of the stretch factor. Additionally, we establish the tightness of the linear growth rate for subgaussian NEFs. In Section 4 we apply these findings to subexponential GLBs and derive novel second-order regret bounds devoid of exponential dependencies on the problem parameter in the leading term. Proofs omitted from the main text are provided in the appendix, except for those of well-known results, which are referenced accordingly.

2 Preliminaries

In this section we first introduce the notation we will use. We then introduce natural exponential families, review some of their basic properties and illustrate the concepts introduced by means of an example.

2.1 Notation

For a real-valued differentiable function f defined over an open interval, we use ˙ f, ¨ f and ... f to denote the first, second and third derivative of f . The set of reals is denoted by R , the set of nonnegative reals by R + . For a set U ⊆ R , we denote its interior by U ◦ . For real numbers a, b , we use a ∧ b and a ∨ b to denote the min { a, b } and max { a, b } , respectively. With ϕ a logical expression, I { ϕ } = 1 if ϕ evaluates to true and I { ϕ } = 0 , otherwise. We use f ≲ g to indicate that g dominates f up to a constant factor over their common domain. For S ⊆ R , x ∈ R , we let S ± x denote the set

{ s ± x : s ∈ S } . A distribution over the reals is centered if it has zero mean. We use P ( ) to denote the probability measure over the probability space that holds our random variables and we let E to denote the expectation corresponding to this measure. By Y ∼ Q we mean that the distribution of Y , a random variable, is Q .

2.2 Single-parameter NEFs

In this section we give our definitions for the NEF. We only consider single-parameter natural exponential families when the base distribution is defined over the reals. We follow the approach of the beautifully written monograph of [Bro86] that the reader is also referred to for any statements made about NEFs with no proofs.

Given a probability distribution Q over R let M Q : R → R + ∪ { + ∞} denote its moment generating function (MGF):

We will find it convenient to also use the logarithm of the moment generating function, which is known as its cumulant generating function (CGF). We denote this by ψ Q : R → R ∪ { + ∞} . Thus, ψ Q ( u ) = log M Q ( u ) .

Let U Q = { u ∈ R : M Q ( u ) < ∞} denote the domain of M Q (and, thus the domain of ψ Q ). As it is well-known, ψ Q is convex and hence U Q is always an interval (which, trivially, always contains 0 ). For a subset of U Q , denoted by U ⊆ U Q , we call Q = ( Q u ) u ∈U a natural exponential family (NEF) generated by Q where for any u ∈ U we have

It follows that Q u is also a probability distribution over the reals for any u ∈ U by definition. An equivalent, useful form for Q u is Q u ( dy ) = exp( uy -ψ Q ( u )) Q ( dy ) . In applications, u denotes an unknown parameter that is to be estimated based on observations from Q u . Thus, U allows one to express extra restrictions on the admissible parameters beyond the limits imposed by Q . We call U the parameter space , and U Q the natural parameter space .

The distributions Q , Q u and parameter u are referred to as the base distribution , the (exponentially) tilted (base) distribution and the tilting parameter . An NEF is said to be regular when U Q is open. It is easy to see that for any u, u 0 ∈ U Q , Q u = ( Q u 0 ) u -u 0 , where the distribution on the right-hand side stands for the tilt of Q u 0 with parameter u -u 0 . As such, up to a constant shift of the parameter space, in a regular NEF, one can always assume that 0 ∈ U ◦ Q . In fact, the same can be assumed for the parameter set, as long as U ◦ is nonempty. If U is an interval then U ◦ = ∅ means that U is a singleton: An uninteresting case if we want to model a host of non-identical distributions.

In a regular family, an equivalent way to parameterize a NEF is using the mean function (cf. Theorem 3.6, page 74, of [Bro86]), µ Q : U Q → R , which is defined as

Since Q 0 = Q , clearly, µ Q (0) is just the mean of Q . To minimize clutter, when Q is clear from the context, we will write µ instead of µ Q . To illustrate the developments so far, we consider the example when the base distribution is an exponential distribution.

Example 1 (Exponential distributions) . For λ > 0 , let Q be an exponential distribution with parameter λ : Q ( dy ) = I { y ≥ 0 } λe -λy dy . As is well known, the MGF of Q takes the form M Q ( u ) = λ λ -u when u < λ and M Q ( u ) = ∞ otherwise. Thus, U Q = { u ∈ R : M Q ( u ) < ∞} = ( -∞ , λ ) . The mean function takes the form of

In what follows, we will need the following proposition to relate the central moments of Q u to the derivatives of ψ q , the CGF.

Proposition 2. Let U ◦ Q be non-empty. Then, ψ Q is infinitely differentiable on U ◦ Q . Furthermore, the first three derivatives of ψ Q at u ∈ U Q give the first moment, second and third central moments of Q u .

In the context of Example 1, Proposition 2 gives that ψ Q ( u ) = log( λ ) -log( λ -u ) when u < λ . Then, ˙ ψ Q ( u ) = 1 λ -u , agreeing with our earlier computation.

3 Self-concordance of NEFs

This section contains the first set of main results of this paper. We start by giving our definition of self-concordance of NEFs, followed by a study of when this property is satisfied. We include a result that shows how self-concordance allows one to derive tail properties of members of the family from that of the base distribution.

In general, if the magnitude of a higher order derivative of a real function can be bounded in terms of a lower order derivative of the function, the function is said to be self-concordant [ST19]. This property is useful for studying how fast the function changes with its argument, as well as for deriving useful bounds on how well the function can be approximated by low order polynomials [NN94; Nes04; ST19]. In the context of single-parameter natural exponential families, we propose the following natural definition:

Definition 1 (Self-concordant NEF) . Let Q = ( Q u ) u ∈U be a NEF with parameter set U ⊂ U ◦ Q and some base distribution Q . We say that Q is self-concordant if there exists a nonnegative valued function Γ : U → R + such that

Any function Γ : U → R + that satisfies Eq. (1) is called a stretch function of the NEF.

This definition takes inspiration from the works of [Bac10; ST19; Fau+20] and [Jan+24] who define an analogous property (cf. Assumption 2 of [Jan+24]). By Proposition 2, we know that ˙ µ ( u ) is the variance of Q u , while ¨ µ ( u ) is the third central moment. Hence, ˙ µ ( u ) is nonnegative, which explains why there is no absolute value on the right-hand side of Eq. (1).

According to our definition, we require that the (absolute value) of the second derivative of µ is bounded by the first derivative, up to the 'stretch factor' Γ( u ) . Clearly, provided that ˙ µ is positive over U , self-concordance is equivalent to stating that Γ Q ( u ) . = | ¨ µ ( u ) | ˙ µ ( u ) is finite valued over U . When ˙ µ ( u ) = 0 for some u ∈ U , one can show that Q u must be a Dirac distribution and hence so does Q ( Q and Q u share their support). In this case, we also have ¨ µ ≡ 0 and hence any nonnegative valued function is a valid stretch-function. In particular, Γ Q ≡ 0 is also a valid stretch function.

It turns out that studying self-concordance property of distributions in detail can turn out much finer bounds than naively bounding ¨ µ and ˙ µ separately.

Example 3 (Avoiding exponential dependencies) . Consider a NEF Q with base distribution Q , a Bernoulli distribution with parameter 1 / 2 , we have U Q = R , µ ( u ) = 1 1+ e -u and thus ˙ µ = µ (1 -µ ) , ¨ µ = ˙ µ (1 -2 µ ) . Hence Q is Γ Q -self-concordant with Γ Q ( u ) ≤ 1 for all u ∈ R . This is an example where naively bounding Γ Q , by bounding the numerator and denominator separately over [ -s, s ] for s > 0 gives a quantity of size e s , which lags far behind the constant we obtained with a direct calculation, or what we can get from the result in Section 3.2, which show a scaling of order O ( s ) .

As opposed to earlier literature where self-concordance is used [NN94; Bac10; ST19], we allow a non-constant stretch-factor Γ . As it turns out, this is necessary if U = U ◦ Q is to be allowed:

Example 4 (Non-constant stretch factor) . Consider a NEF Q with base distribution Q . For Q an exponential distribution with parameter λ > 0 , Q is self-concordant over U = U Q = ( -∞ , λ ) with Γ Q ( u ) = 2 λ -u , u < λ . Indeed, a simple calculation gives that for u < λ , ˙ µ ( u ) = ( λ -u ) -2 and ¨ µ ( u ) = 2( λ -u ) -3 , and so | ¨ µ ( u ) | / ˙ µ ( u ) = 2 / ( λ -u ) .

As was shown above, for the NEF built on the exponential distribution with parameter λ , there is no constant stretch factor that makes the NEF self-concordant over the entirety of the natural parameter space. The main result of the next section shows that a non-constant stretch factor with growth similar

to the exponential always exists provided that the base distribution is subexponential . 2 As it turns out, the growth of the stretch factor plays an important role in applications: Among other things, it allows us to conclude that the tilted distributions are subexponential, as captured by the next result, which is a slightly generalized version of an analogous result of [Jan+24]:

Lemma 5 (From self-concordance to light tails) . Let Q = ( Q u ) u ∈U be a NEF which is selfconcordant with stretch function Γ : U → R + where U is a subinterval of U ◦ Q = ( a, b ) . Then, for any u ∈ U ,

Note that when U is a strict subset of U ◦ Q , 0 ∈ ( a -inf U , b -sup U ) and thus the result is nontrivial as long as K < ∞ . To interpret this result, recall that a centered distribution Q is called subexponential with nonnegative parameters ( ν, α ) if

(cf. Definition 2.7 [Wai19]). A consequence of this is that the mean of n independent random variables drawn from Q , with high probability, will be in an zero-centered interval of length O ( √ ν 2 /n + 1 αn ) . Here, the first term describes a 'subgaussian' behavior, while the second a 'pure subexponential' behavior. Assume for simplicity that a = -b and inf U = -sup U . From Eq. (2) it follows that Q u , when centered, is subexponential with parameters ν 2 = 2 ˙ µ ( u ) , twice the variance of Q u , and α = min(log(2) /K,b -sup U ) . In particular, we see that the growth of Γ impacts the lower-order term in the confidence interval (the term 1 / ( αn ) ), but not the leading term, which is governed by the variance of Q u . It follows that understanding how fast Γ can grow as its argument approaches the boundary of U Q is thus important, although its impact only appears in a low-order term.

We now turn to our first main result that shows that the growth of Γ is at most inverse polynomial provided that the base distribution itself is subexponential.

3.1 Self-concordance with a subexponential base

We start with recalling equivalent definitions of subexponential distributions, which will be useful to interpret our results. In particular, according to Theorem 2.13 of [Wai19], given a zero-mean distribution Q over the reals, the following are equivalent:

(i) Q is subexponential;

(ii) The MGF of Q is defined in an open neighborhood of zero;

(iii) For some positive constants C, c > 0 , P ( | Y | ≥ t ) ≤ C exp( -ct ) where Y ∼ Q .

It will be useful to have a quantitative version of this statement. For this, we separate the left and the right tails (as in some application they have different behaviors, which we may care about). The quantitative result for the right-tail is as follows (the result for the left-tail is omitted; it follows by symmetry):

Proposition 6. Let c 1 , C 1 , c > 0 , Y ∼ Q and assume that E Y = 0 . If

then for any 0 ≤ λ < c 1 , M Q ( λ ) < 1+ C 1 λ 2 c 1 ( c 1 -λ ) . Furthermore, for any c > 0 such that M Q ( c ) < ∞ , P ( Y ≥ t ) ≤ M Q ( c ) e -ct holds for all t ≥ 0 .

The first part is nontrivial; the second follows easily from Chernoff's method. The proof is given in Appendix B.

Let

In words, E right is the class of distributions over the reals whose right-tail displays an exponential decay governed with the rate parameter c 1 > 0 and scaling constant C 1 > 0 . Similarly, we let E left ( c 1 , C 1 ) be the class of distributions Q over the reals such that for X ∼ Q , Y = E X -X satisfies Eq. (3). With this notation, the first part of the previous proposition is equivalent to that if Q ∈ E right ( c 1 , C 1 ) is centered then M Q stays below the function λ ↦→ 1 + C 1 λ 2 c 1 ( c 1 -λ ) over the interval [0 , c 1 ) . In particular, this means that U Q contains [0 , c 1 ) .

With this, we are ready to present our theorem that establishes the self-concordance property of NEFs with subexponential tails.

Theorem 7. Let Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) for some positive constants c i , C i , i = 1 , 2 . Then, the NEF Q = ( Q u ) u ∈U is self-concordant. Moreover, the function Γ : U → R + defined by

is a stretch function for the NEF Q , where G Q ( M 1 , M 2 , m 1 , m 2 ) is a polynomial whose coefficients depend on Q .

The exact expression of G Q can be found in Eq. (8). Let Q be a NEF with base distribution Q when Q is a zero-mean Laplace distribution with variance 2 λ 2 . In this case, we can choose c 1 = c 2 = 1 /λ , C 1 = C 2 = 1 / 2 . The actual stretch function is Γ Q ( u ) = 8 λ 2 | u | (1 /λ -u )(1 /λ + u )(3 λ 4 u 2 + λ 2 ) . Thus, the above theorem gives the correct behavior in that both the stretch function from the theorem and the actual stretch function Γ Q blow up with the inverse of the distance to the boundaries of U Q , except that the actual growth scales linearly with the inverse distance, while the theorem gives a quadratic scaling. It remains an open problem to see whether this quadratic order can be improved.

The following corollary is an immediate result of Lemma 5 and Theorem 7 (Appendix C.3), but can also be proved directly from the definitions (and we include a direct proof as part of the proof of Theorem 7).

Corollary 8 (Distributions in regular NEFs are subexponential) . Let u ∈ U ◦ Q . Then Q u is subexponential both on left and right.

Because of this result, there is essentially no loss in generality in only considering the subexponential case when working with NEFs. In particular, self-concordance in NEFs is 'almost free'.

Proof sketch for Theorem 7 We sketch here the result for µ (0) = 0 , ˙ µ (0) > 0 and u ≥ 0 . The arguments used to extend the result to all cases can be found in Appendix C. By Proposition 2, bounding the stretch function of a NEF amounts to showing

where the division by Var( Q u ) = ∫ ( y -µ ( u )) 2 Q ( dy ) is justified by Lemma 12. We split the proof of that upper bound into two steps: controlling the variance and the absolute third moment.

Step 1: Controlling the variance Since ˙ µ (0) = Var( Q ) > 0 , there exists a Q -dependent constant η > 0 and an interval [ -b, -a ] ⊂ R < 0 s.t. Q ([ -b, -a ]) ≥ η (Lemma 16). With this observation, we can show (Lemma 17):

Thus, the second moment decreases at most exponentially with the parameter u .

Step 2: Controlling the absolute third central moment First, we use a classical result on moments of random variables (see the proof of i ⇒ ii for prop.2.5.2 in [Ver18]):

where Y ∼ Q u and B is a constant to be optimized. It should remain small enough for the first term not to blow up. On the other hand, it should be large enough for the second term divided by the lower bound obtained on ˙ µ ( u ) to remain controlled.

To upper bound the second term in Eq. (5), we start by showing that the right tail of the tilted distribution Q u is also subexponential (Lemma 18):

Following this lemma, we get an upper bound on µ ( u ) (Lemma 20):

In the proof, we bound separately the positive and negative values in the second term of Eq. (5):

We give here the sketch of proof on the bound of ♠ as the proof for bounding ♡ is nearly identical. From plugging in Eq. (6), we get:

By choosing B ≳ ub c 1 -u + ( 1 c 1 -u ) 2 , some algebra gives:

Setting B ≳ ( 1 c 1 -u ) 2 + ub c 2 + u gives a similar bound on ♡ . Chaining the bounds on ♠ and ♡ with Eq. (5) and Lemma 17 finishes the proof. □

3.2 Self-concordance with a subgaussian base

In this section we refine the previous result for NEFs by considering the case when the base distribution is subgaussian. Let σ > 0 . Recall that a centered distribution Q is σ -subgaussian if for all u ∈ R , M Q ( u ) ≤ e σ 2 u 2 / 2 (or, equivalently, ψ Q ( u ) ≤ σ 2 u 2 / 2 for any u ∈ R ). A non-centered distribution is σ -subgaussian, if it is subgaussian after centering. Similarly to the subexponential case, one can show that a centered distribution Q is subgaussian if and only if for some τ, C > 0 , it holds that for any t ≥ 0 , P ( | X | ≥ t ) ≤ C exp( -t 2 / (2 τ 2 )) where X ∼ Q (cf. Proposition 2.5.2 of [Ver18]). 3

Our promised result is as follows:

Theorem 9. Let Q be subgaussian. Then, the NEF Q = ( Q u ) u ∈ R is self-concordant and Γ Q ( u ) = O ( | u | ) , u ∈ R .

As it turns out, the linear growth exhibited in the previous result is tight for NEFs with a subgaussian base distribution:

Theorem 10. There exists a distribution Q that is subgaussian such that lim sup u →∞ Γ Q ( u ) /u > 0 .

Again, this shows that even if we stay with subgaussian distributions, it would be limiting to only consider NEFs that are self-concordant with a bounded (or constant) stretch function over their natural parameter space.

4 Generalized Linear Bandits

In this section we apply the self-concordance property of subexponential NEFs in order to derive novel confidence sets and regret bounds for subexponential generalized linear bandits. To our knowledge, these are the first such results for parametric bandits with subexponential rewards.

4.1 Bandit model

Following Filippi, Cappe, Garivier, and Szepesvári [Fil+10], we consider stochastic generalized linear bandit (GLB) models G specified by a tuple ( X , Θ , Q ) , where X ⊆ R d is a non-empty arm set, Θ ⊆ R d is a non-empty set of potential parameters, both closed for convenience, Q = ( Q u ) u ∈U Q is a NEF with base distribution Q . Without the loss of generality we assume that X is a compact subset of the Euclidean unit ball of R d .

In each round t ∈ N + , the learner selects and plays an arm X t ∈ X . As a response, they receive a reward Y t sampled from the distribution Q X ⊤ t θ ⋆ , where θ ⋆ ∈ R d is a parameter of the bandit environment, which is initially unknown to the learner. The learner's goal is to maximize its total expected reward. The GLB is well-posed when

holds, which we assume from now on. The condition that U ⊆ U Q simply ensures that the reward distributions Q x ⊤ θ are defined regardless of the value of ( x, θ ) ∈ X × Θ . We require the stronger condition U ⊆ U ◦ Q to exclude the boundaries of the interval U Q . This way we avoid pathologies that arise when a parameter reaches the boundary of U Q (e.g., when Q is the exponential distribution with parameter λ , the mean and variance of Q u grow unbounded as u approaches λ from below).

The expected reward in round t given that the learner plays X t is E [ Y t | X t ] = µ ( X ⊤ t θ ⋆ ) . The performance of the learner will be assessed by their pseudo regret R ( T ) , which is the total cumulative shortfall of the mean reward of the arms the learner chose relative to optimal choice:

Here, x ⋆ ∈ arg max x ∈X µ ( x ⊤ θ ⋆ ) is the arm that results in the best possible expected reward in a round. For simplicity, we assume that such an arm exists. We establish guarantees of our algorithm for a subclass of GLBs, captured by the following assumption:

Assumption 1 (Subexponential base) . The base distribution Q is subexponential both on the left and the right. In particular, Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) for some c i , C i ( i = 1 , 2 ) positive numbers. Furthermore, -c 2 < inf U ≤ sup U < c 1 and c 1 , c 2 are known to the learner.

̸

Note that in a well-posed GLB, inf U Q < inf U ≤ sup U < sup U Q . In light of this, the assumption just stated boils down to whether 0 ∈ U ◦ Q , which, as discussed, is free when U ◦ Q = ∅ . Indeed, when 0 ∈ U ◦ Q holds, inf U Q < 0 < sup U Q and one can always find positive values c 1 , c 2 such that inf U Q < -c 2 < inf U ≤ sup U < c 1 < sup U Q . Then, from Proposition 6 and some extra calculation one can conclude that the Q ∈ E right ( c 1 , e -c 1 E X M Q ( c 1 )) ∩ E left ( c 2 , e -c 2 E X M Q ( -c 2 )) . Since it is assumed that the learner has access to Q , we see that Assumption 1 can be satisfied whenever 0 ∈ U ◦ Q , which we think is a rather mild assumption. We will also for the sake of simplicity assume that the learner has access to S 0 = sup {∥ θ ∥ : θ ∈ Θ } , S 2 = inf U , S 1 = sup U . These values will be used in setting the parameters of the algorithm. Note that it is not critical that the learner knows these exact values; appropriate bounds suffice. We also assume that the learner is given access to an upper bound on the worst-case variance over the parameter space U :

Assumption 2 (Bounded Variance) . The learner is given L ≥ 1 such that sup u ∈U ˙ µ ( u ) ≤ L .

Note that since the GLB is well-posed, sup u ∈U ˙ µ ( u ) < ∞ is automatically satisfied. Also, there is no loss of generality in assuming L ≥ 1 . A crude upper bound on sup u ∈U ˙ µ ( u ) is C 1 e/ ( c 1 -S 1 ) 3 ∨ C 2 e/ ( c 2 + S 2 ) 3 , so this assumption is implied by the previous one.

4.2 The OFU-GLB algorithm and its regret

Just like the previous works [Fil+10; LLZ17; Fau+20] which considered special cases of the generalized linear bandit problem, our algorithm follows the 'optimism in the face of uncertainty' principle. In each time step, the algorithm constructs a confidence set C t , based on past information, that contains the unknown parameter θ ⋆ with a controlled probability. Next, the algorithm chooses a parameter θ t , in the confidence set C t , and an underlying action X t ∈ X such that the mean reward underlying X t and θ t is as large as plausibly possible. Since µ = µ Q is guaranteed to be an increasing function (recall that ˙ µ ( u ) is the variance of Q u and is hence nonnegative), it suffices to find the maximizer of x ⊤ θ where ( x, θ ) ∈ X × Θ . We call our algorithm, shown in Algorithm 1, OFU-GLB (Optimism in the Face of Uncertainty in Generalized Linear Bandits). The main novelty here is that our bandit model makes minimal assumptions.

Algorithm 1 The OFU-GLB Algorithm

The confidence set is based on ideas from the work of Janz, Liu, Ayoub, and Szepesvári [Jan+24]. Note that this paper analyzed a randomized method for those GLBs where U Q = R and sup u ∈U Γ Q ( u ) < ∞ . The assumption that U Q = R is restrictive, as it does not allow many common distributions (e.g., the exponential distribution). Thanks to Theorem 7, under our assumptions, sup u ∈U Γ Q ( u ) < ∞ follows. Then, an appropriate confidence set can be constructed based on Lemma 5, which also extended the corresponding result of Janz, Liu, Ayoub, and Szepesvári [Jan+24]. While the confidence set construction is based on the ideas of Janz, Liu, Ayoub, and Szepesvári [Jan+24], the main steps of the analysis are taken from [AFC21] who analyzed logistic bandits, which are 1 -self-concordant. Our result follows by carefully modifying the proof of [Jan+24] and carefully propagating both the effect of replacing their confidence set with a different one, and the effect of sup u ∈U Γ Q ( u ) > 1 . This leads to the main result on GLBs:

Theorem 11 (Regret upper bound of OFU-GLB) . Let δ ∈ (0 , 1] and T a positive integer and consider a well-posed GLB model G = ( X , Θ , Q ) and assume that Assumptions 1 and 2 hold. For θ ⋆ ∈ Θ , let κ ( θ ⋆ ) = 1 ˙ µ ( x ⊤ ⋆ θ ⋆ ) and let Regret( T, θ ⋆ ) stand for the T -round regret of OFU-GLB when it interacts with a GLB specified by θ ⋆ . Then, with an appropriate construction of C t , for any θ ⋆ ∈ Θ , it holds that with probability at least 1 -δ ,

where ˜ O ( · ) hides polylogarithmic factors in T, d, L, 1 /δ and constants that depend on the base distribution Q .

This result extends the class of distributions for which OFU algorithms with parametric models achieve sublinear regret. Previous results in the literature [Fil+10; APS11; LLZ17; Fau+20; Jun+21; Jan+24] all assume that the base distribution is a natural exponential family with subgaussian tail and prove that the OFU algorithm enjoys sublinear regret. Thus, Theorem 11 extends the class of distributions for which optimistic algorithms enjoy sublinear regret to any natural exponential family with subexponential base distribution.

An essential quality of the result is that it makes the dependence of the regret on the instance θ ⋆ explicit. Recalling that in a NEF, ˙ µ ( u ) is the variance of the tilted distribution Q u , we see that the leading term (shown as the first term on the right-hand side of the last display) scales with the variance of the optimal arm's reward distribution. In queuing theory, the service times of agents (actions) in an environment are often modeled as exponentially distributed random variables [GN67]. When aiming to minimize service times (maximize negative reward) with an exponential bandit model (with mean function µ ( x ) = -1 /x ), the variance of the optimal arm's service time lower bounds that of the other arms. Furthermore, the dependence on κ , a term that is inversely proportional to the optimal variance,

is pushed to a second order term. In logistic bandits, κ can be exponentially large in the size of the parameter set S 0 and thus much attention has been focused on mitigating its effect [Fau+20; Jun+21; Jan+24]. Our regret bound also matches the lower bound in logistic bandits given by [AFC21], thus our analysis is tight for this special case.

5 Conclusions and Future Work

The main contribution of this work establishes that all subexponential NEFs are self-concordant with a polynomial-sized stretch factor. We then applied this finding and derived regret bounds for subexponential GLBs that scale with the variance of the optimal arm's reward, which is the smallest variance amongst all arm's rewards in problems such as: minimizing service times [GN67] or minimizing insurance claim severity (dollars lost per claim) [GKT16].

Our findings also have implications when performing maximum likelihood estimation with subexponential NEFs, which includes a rich family of generalized linear models (GLMs). Since the log loss in a NEF is the sum of a linear function and the NEF's CGF, the GLM's loss is self-concordant in the sense of (say) [Bac14] whenever the NEF is self-concordant. While this is outside of the scope of our paper, it follows that this family of GLMs enjoy: ( i ) fast rates of convergence to the minimizer for regularized empirical risk minimization [Mar+19], ( ii ) fast rates for averaged stochastic gradient descent [Bac14] and ( iii ) fast rates for constrained optimization with first-order methods [Dvu+20], without restrictive conditions on bounded responses, which previous works had to assume to achieve these results.

One interesting direction for future work would be either deriving a matching lower bound on the stretch function for subexponential NEFs or tighter analysis that matches the lower bound for subgaussian NEFs. Another potential avenue for future work would be in extending our results to other exponential families, beyond NEFs.

Acknowledgments and Disclosure of Funding

The authors would like to thank Iosif Pinelis for his helpful insights to our work and Samuel Robertson for reviewing and earlier version of this manuscript. Shuai Liu would like to acknowledge Alireza Bakhtiari for helpful discussions. Csaba Szepesvári also gratefully acknowledges funding from the Canada CIFAR AI Chairs Program, Amii and NSERC.

References

[AFC21] Marc Abeille, Louis Faury, and Clément Calauzènes. 'Instance-wise minimax-optimal algorithms for logistic bandits'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . 2021.

[APS11] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. 'Improved algorithms for linear stochastic bandits'. In: Advances in neural information processing systems (NeurIPS) (2011).

[Bac10] Francis Bach. 'Self-concordant analysis for logistic regression'. In: Electronic Journal of Statistics (2010).

[Bac14] Francis Bach. 'Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression'. In: The Journal of Machine Learning Research (JMLR) (2014).

[BFR23] Blair Bilodeau, Dylan J Foster, and Daniel M Roy. 'Minimax rates for conditional density estimation via empirical entropy'. In: The Annals of Statistics (2023).

[BLM13] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities - A Nonasymptotic Theory of Independence . Oxford University Press, 2013.

[Bro86] Lawrence D. Brown. 'Fundamentals of statistical exponential families with applications in statistical decision theory'. In: Lecture Notes-Monograph Series (1986).

[Dvu+20] Pavel Dvurechensky, Petr Ostroukhov, Kamil Safin, Shimrit Shtern, and Mathias Staudigl. 'Self-concordant analysis of Frank-Wolfe algorithms'. In: International Conference on Machine Learning (ICML) . 2020.

[Fau+20] Louis Faury, Marc Abeille, Clement Calauzenes, and Olivier Fercoq. 'Improved optimistic algorithms for logistic bandits'. In: International Conference on Machine Learning (ICML) . 2020.

[Fil+10] Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. 'Parametric bandits: The generalized linear case'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2010.

[GK98] Charles M Goldie and Claudia Klüppelberg. 'Subexponential distributions'. In: A practical guide to heavy tails: Statistical techniques and applications . Birkhauser Boston Inc., Aug. 1998, pp. 435-459.

[GKT16] M. Goldburd, A. Khare, and D. Tevet. Generalized linear models for insurance rating . Casualty Actuarial Society, 2016.

[GN67] William J. Gordon and Gordon F. Newell. 'Closed queuing systems with exponential servers'. In: Operations Research (1967).

[Jan+24] David Janz, Shuai Liu, Alex Ayoub, and Csaba Szepesvári. 'Exploration via linearly perturbed loss minimisation'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . 2024.

[Jun+21] Kwang-Sung Jun, Lalit Jain, Blake Mason, and Houssam Nassif. 'Improved confidence bounds for the linear logistic model and applications to bandits'. In: International Conference on Machine Learning (ICML) . 2021.

[LLZ17] Lihong Li, Yu Lu, and Dengyong Zhou. 'Provably optimal algorithms for generalized linear contextual bandits'. In: International Conference on Machine Learning (ICML) . 2017.

[LS20] Tor Lattimore and Csaba Szepesvári. Bandit Algorithms . Cambridge University Press, 2020.

[LYJ24] Junghyun Lee, Se-Young Yun, and Kwang-Sung Jun. 'A Unified Confidence Sequence for Generalized Linear Models, with Applications to Bandits'. In: Advances in Neural Information Processing Systems (NeurIPS) . 2024.

[Mar+19] Ulysse Marteau-Ferey, Dmitrii Ostrovskii, Francis Bach, and Alessandro Rudi. 'Beyond least-squares: Fast rates for regularized empirical risk minimization through selfconcordance'. In: Conference on Learning Theory (COLT) . 2019.

[ML09] Carl N. Morris and Kari F. Lock. 'Unifying the named natural exponential families and their relatives'. In: The American Statistician (2009).

[MN89] P. McCullagh and J. A. Nelder. Generalized linear models . Chapman & Hall / CRC, 1989.

[Mor82] Carl N. Morris. 'Natural exponential families with quadratic variance functions'. In: The Annals of Statistics (1982).

[Nes04] Yurii Nesterov. Introductory Lectures on Convex Optimization - A Basic Course . Springer, 2004.

[NN89] Yurii. Nesterov and A. Nemirovski. Self-concordant functions and polynomial-time methods in convex programming . USSR Academy of Sciences, Central Economic & Mathematic Institute, 1989.

[NN94] Y. Nesterov and A. Nemirovski. Interior point polynomial algorithms in convex programming . Society for Industrial and Applied Mathematics, 1994.

[NT08] Arkadi S Nemirovski and Michael J Todd. 'Interior-point methods for optimization'. In: Acta Numerica (2008).

[OB21] Dmitrii M. Ostrovskii and Francis Bach. 'Finite-sample analysis of M -estimators using self-concordance'. In: Electronic Journal of Statistics (2021).

[RH23] Philippe Rigollet and Jan-Christian Hütter. High-Dimensional Statistics . 2023. arXiv: 2310.19244 [math.ST] .

[Rus+21] Yoan Russac, Louis Faury, Olivier Cappé, and Aurélien Garivier. 'Self-concordant analysis of generalized linear bandits with forgetting'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . 2021.

[Saw+24] Ayush Sawarni, Nirjhar Das, Siddharth Barman, and Gaurav Sinha. 'Generalized Linear Bandits with Limited Adaptivity'. In: (2024). arXiv: 2404.06831 [cs.LG] .

[SPL23] Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. 'Dueling RL: Reinforcement learning with trajectory preferences'. In: International Conference on Artificial Intelligence and Statistics (AISTATS) . 2023.

[ST19] Tianxiao Sun and Quoc Tran-Dinh. 'Generalized self-concordant functions: A recipe for Newton-type methods'. In: Mathematical Programming (2019).

[Ver18] Roman Vershynin. High-dimensional probability: An introduction with applications in data science . Cambridge university press, 2018.

[Wai19] Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint . Cambridge University Press, 2019.

[WJ08] Martin J Wainwright and Michael I Jordan. Graphical models, exponential families, and variational inference . Now Publishers, Inc., 2008.

A Extra notation

The following extra notation will be used in the appendix: For vector x ∈ R d , we let ∥ x ∥ denote its ℓ 2 -norm and for positive definite matrix M ∈ R d × d , we use ∥ x ∥ M = √ x ⊤ Mx to denote its M -weighted ℓ 2 -norm.

B On subexponential (or 'light tailed') distributions

We first prove Proposition 6, which we repeat for the convenience of the reader:

Proposition 6. Let c 1 , C 1 , c > 0 , Y ∼ Q and assume that E Y = 0 . If

then for any 0 ≤ λ < c 1 , M Q ( λ ) < 1+ C 1 λ 2 c 1 ( c 1 -λ ) . Furthermore, for any c > 0 such that M Q ( c ) < ∞ , P ( Y ≥ t ) ≤ M Q ( c ) e -ct holds for all t ≥ 0 .

We follow the proof of Theorem 2.13 from the book of Wainwright [Wai19].

Proof. We start with the second part. For this let t ≥ 0 , c > 0 . Then, by Chernoff's method, we have

The first part requires more work. Let us start by bounding the p -th moment of the positive part of Y , which we denote by Y + (hence, Y + = max( Y, 0) ). We have

Now let 0 ≤ λ < c 1 . Since Y ≤ Y + , we have M Q ( λ ) = E [ e λY ] ≤ E [ e λY + ] . Hence, by the power-series expansion of the exponential, we get

We note in passing that since λ < c 1 , 1 + C 1 λ c 1 λ c 1 -λ ≤ 1 + C 1 λ c 1 -λ . That Eq. (3) implies that M Q ( λ ) ≤ 1 + C 1 λ c 1 -λ can also be obtained by refining the proof of Theorem 2.13 from the book of Wainwright [Wai19].

C Proof of Theorem 7

For the convenience of the reader we restate the theorem to be proven:

Theorem 7. Let Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) for some positive constants c i , C i , i = 1 , 2 . Then, the NEF Q = ( Q u ) u ∈U is self-concordant. Moreover, the function Γ : U → R + defined by

is a stretch function for the NEF Q , where G Q ( M 1 , M 2 , m 1 , m 2 ) is a polynomial whose coefficients depend on Q .

Note that the function Γ as defined above is non-decreasing on U ∩ R + and non-increasing on U ∩ R -.

The actual form of G Q is as follows: let a, b, η > 0 be such that Q ([ -b + µ (0) , -a + µ (0)]) > η and -a < 0 . Then,

We note in passing that these values are not controlled by the tail behavior of the base distribution Q . This can be seen, for example, by considering Q ( dy ) = (1 -η ) I ( y ≥ 0) e -y dy + ηe -b δ {-b } ( dy ) . Tedious calculation shows that lim u →-∞ Γ Q ( u ) = Ω( b ) as b →∞ . And because Q ∈ E right (1 , 1) ∩ E left (1 , 1) , this shows that the tail behavior of Q is indeed insufficient to control the behavior of Γ Q .

We will prove this result in three parts: (i) Var( Q ) = 0 (ii) Var( Q ) > 0 and U = [0 , c 1 ) , (iii) Var( Q ) > 0 and U = ( -c 2 , 0] . The result follows from combining these cases.

The main work is to prove the result for U = [0 , c 1 ) , which is done in Proposition 13. Case (iii) is handled by 'reflection around the origin' (Corollary 14). Case (i) is handled in Lemma 12 by showing that Γ Q ≡ 0 if Var( Q ) = 0 .

We start with case (i) , the degenerate case when the variance of Q is zero.

Lemma 12. If Var( Q ) = 0 then U Q = R , Q u = Q for all u ∈ R , and Γ Q ≡ 0 . If Var( Q ) > 0 then ˙ µ is strictly positive over the entire set U ◦ Q .

Proof. If Var( Q ) = 0 , then Q is a Dirac on some { v } . Then for all u ∈ R , M Q ( u ) = exp( uv ) < ∞ hence ψ Q = log M Q is supported on R and

Hence Q u = Q and Q = ( Q u ) u ∈ R is trivially self-concordant with the stretch function defined to be Γ Q ≡ 0 .

̸

For the second part, by Proposition 2, we have that ˙ µ ( u ) = ∫ ( y -µ ( u )) 2 Q u ( dy ) ≥ 0 . We will show ˙ µ ( u ) = 0 by contradiction. Assume there exists u 0 ∈ U ◦ Q such that ˙ µ ( u 0 ) = ∫ ( y -µ ( u 0 )) 2 Q u 0 ( dy ) = 0 , then it follows that Q u 0 ( dy ) is a Dirac on { µ ( u 0 ) } , which implies for all A ∈ B ( R )

where B ( R ) denotes the Borel sets on R . Then it follows that Q ( dy ) is also a Dirac on R , which contradicts that Var( Q ) > 0 .

Consider now the case when Var( Q ) > 0 . By the result just stated ˙ µ is bounded away from zero over U ◦ Q and hence it is safe then to define Γ Q with the ratio | ¨ µ ( u ) | ˙ µ ( u ) :

Thus, in order to show our result, it suffices to show Γ Q ≤ Γ with the function Γ as stated in the theorem. Thus, we will study Γ Q . First, notice that for all u ∈ U ◦ Q , by Proposition 2,

Let us now state the results that are concerned with cases (ii) and (iii) mentionned above. For case (ii) , i.e., when U = [0 , c 1 ) we have the following result:

Proposition 13. Let Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) and U = [0 , c 1 ) . Define Γ : U → R + by

where c 0 = C 1 · c 1 · e . Then, for appropriate values of η, a, b > 0 that depend on the base distribution Q , we have Γ Q ≤ Γ over U .

For case (iii) , i.e., when U = ( -c 2 , 0] , we have the following result:

Corollary 14. Let Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) and U = ( -c 2 , 0] . Define Γ : U → R + by

where c 0 = C 2 · c 2 · e . Then, for appropriate values of η, a, b > 0 that depend on the base distribution Q , we have Γ Q ≤ Γ over U .

C.1 Proof of Proposition 13

The key idea is to convert ¨ µ ( · ) and ˙ µ ( · ) to third and second central moments respectively by Proposition 2 and bound the third central moment in terms of the second central moment (variance).

We start by an elementary observation that says that the self-concordance properties of a NEF do not change when the base distribution is shifted by a constant:

Lemma 15. Let Y ∼ Q , c ∈ R and define Q + c to be the distribution of Y + c . Then, U Q = U Q + c , M Q + c ( u ) = e -uc M Q ( u ) for all u ∈ R , and Γ Q = Γ Q + c (here, we take U = U Q = U Q + c ).

Proof. By definition M Q ( u ) = E e uY and M Q + c ( u ) = E e u ( Y + c ) . Hence,

This shows that U Q = U Q + c and that the desired relationship between M Q and M Q + c hold. Now, from the definition that the CGF is the logarithm of the MGF, it follows that ψ Q + c ( u ) = uc + ψ Q ( u ) . Hence, ¨ ψ Q + c = ¨ ψ Q and ˙ ψ Q + c = ˙ ψ Q , which implies that Γ Q = Γ Q + c .

Thanks to this result, from a bound on the self-concordance function of centered distributions, we can deduce a bound on the self-concordance function of the non-centered ones.

We thus first work on establishing the bound when Q is centered.

Since the theorem statement holds trivially when the variance Var( Q ) of Q is zero, we will also assume with no loss of generality in some of the results below that Q has positive variance.

Lemma 16. Take a distribution Q with zero mean and positive variance. Then, there exist η > 0 and 0 < a ≤ b distribution dependent constants such that Q ([ -b, -a ]) ≥ η .

Proof. Since ˙ µ (0) = Var( Q ) > 0 , there exists a > 0 and α > 0 such that Q (( -∞ , -a ])) > α . As lim x → 0 Q (( -∞ , -x ])) = 0 , we can find some b > 0 s.t.

This implies:

The lemma thus holds with a and b described above, and η = α 2 .

Lemma 17. Take a distribution Q with zero mean and positive variance. With η, a, b as in Lemma 16, for all u ∈ U ◦ Q , it holds that

Proof. For a, b described in Lemma 16, we have:

The first inequality holds as ( y -µ ( u )) 2 is non-negative; the second as µ ( u ) increases with u ; the third as -b ≤ -a < µ (0) , µ (0) = 0 and u ≥ 0 ; and the last one by Lemma 16.

For the upper bound, we present lemmas that bound the (upper and lower) tails of Q u and the mean µ ( u ) .

Lemma 18. Take Q ∈ E right ( c 1 , C 1 ) a centered distribution. Then, for all 0 ≤ u < c 1 and t ≥ 0 , we have

Proof. The inequality is trivially satisfied when u = 0 . Indeed, in this case Q = Q 0 , M Q (0) = 1 , which together with Q ∈ E right ( c 1 , C 1 ) implies the inequality.

Let us now assume that 0 < u < c 1 . Let v > 0 be a constant to be chosen later. Then we have that

We choose v = 1 /u > 0 . Then M Q ( u ) Q u (( t, ∞ )) can be upper bounded by

x

where in the last inequality, we used the fact that for all x > 0 , e e x -1 ≤ 1 + 1 x .

Remark 1. For Q a centered exponential distribution Exp( c ) with rate parameter c , the moment generating function is M Q ( u ) = e -u c c c -u . On the other hand, Q u ( t, ∞ ) = e -u c e -( c -u ) t . So Lemma 18 is order tight in its dependency on c -u .

Lemma 19. Take Q ∈ E left ( c 2 , C 2 ) a centered distribution. Then, for all u, t ≥ 0 , we have

Proof. We again separate the u = 0 case. When u = 0 , Q = Q 0 , M Q (0) = 1 and the inequality is equivalent to Q ∈ E left ( c 2 , C 2 ) .

Consider now u > 0 . Then,

Lemma 20. Take Q ∈ E right ( c 1 , C 1 ) with zero mean and positive variance. Define c 0 = c 1 · C 1 · e . Then, for all 0 ≤ u < c 1 , it holds that

Proof. For u = 0 , µ (0) = 0 which satisfies the inequality. We now consider 0 < u < c 1 . We have

By Jensen's inequality, M Q ( u ) = E Q [ e uY ] ≥ e u E Q [ Y ] = 1 , finishing the proof.

Proof of Proposition 13. As noted beforehand, since the statement holds trivially when the variance of Q is zero, we assume it is positive. By our discussion beforehand, we also assume first that Q is centered, so µ (0) = 0 .

Let Y ∼ Q u . Take B > 0 a constant to be optimized later. We have

The following bound holds:

We also have:

Set B = 2 c 0 ( 1 c 1 -u ) 2 + ub c 1 -u + ub u + c 2 and B ′ = 2 c 0 ( 1 c 1 -u ) 2 . Then we can upper bound ( ii, a ) using Lemma 18.

where in the third inequality we used the fact that ( a + b ) 2 ≤ 2 a 2 +2 b 2 for all a, b ∈ R and that B ′ < B -ub c 1 -u . We now bound ( iii, a ) . We have that B ′ ( c 1 -u ) = 2 c 0 ( 1 c 1 -u ) and

Similarly, for ( iii, b ) , we have that

Putting the result together, ( ii, a ) can be upper bounded as

By Lemma 20, B ≥ µ ( u ) + ub c 2 + u . Hence By Lemma 19 we have:

We now focus on

By definition of B , we have that B -µ ( u ) -ub u + c 2 ≥ c 0 ( 1 c 1 -u ) 2 =: B ′′ . We then have that

For ( iv, b ) , note that B ′′ ≥ µ ( u ) by Lemma 20 and we have that

Putting bounds on ( iv, a ) , ( iv, b ) and ( iv, c ) together, we have that

Combining the bounds on ( ii, a ) and ( ii, b ) with Lemma 17 we get:

Chaining the result with the bound on ( i ) together as well as the choice that B = 2 c 0 ( 1 c 1 -u ) 2 + ub c -u + ub u + c , we obtain

For ( iv, c ) ,

̸

Let us now study the case µ Q (0) = 0 . Let a, b, η > 0 be such that Q ([ -b + µ (0) , -a + µ (0)]) > η and -a < 0 . With Q -µ (0) the centered version of Q , this gives Q -µ (0) ([ -b, -a ]) > η . We have just shown that for all u ∈ [0; c 1 ) ,:

By Lemma 15, we have Γ Q -µ (0) = Γ Q , hence:

C.2 Proof of Corollary 14

Lemma 21. Let Y ∼ Q , and Q -let be the distribution of -Y . Then U Q = -U Q -and for any u ∈ U ◦ Q , we have

Proof. Recall that if Q has zero variance, Γ Q ≡ 0 and hence the statement is trivial. Otherwise, for u ∈ U ◦ Q , Γ Q ( u ) = | ... ψ Q ( u ) | / ¨ ψ Q ( u ) . Now, for v ∈ R ,

Hence, U Q = -U -and for any v ∈ U Q , ψ Q ( v ) = ψ -( -v )

which immediately implies the statement, noting that the variance of Q is positive if and only if the variance of Q -is positive.

Proof of Corollary 14. Assume that Q ∈ E right ( c 1 , C 1 ) ∩ E left ( c 2 , C 2 ) . Since the statement holds trivially when Var( Q ) = 0 , assume Var( Q ) > 0 . Then, Q -∈ E right ( c 2 , C 2 ) ∩ E right ( c 1 , C 1 ) . We then get the stated result by applying Proposition 13, combined with Lemma 21. To be more specific, for all u ∈ ( -c 2 , 0] , from these two results it follows that

C.3 Proof of Corollary 8

Let Q be a regular NEF with base distribution Q . By definition, this means that U Q is an open interval. Take any u ∈ U ◦ Q . There exist some ϵ > 0 s.t. M Q ( u -ϵ ) , M Q ( u + ϵ ) < ∞ . We also have:

Similarly, M Q u ( -ϵ ) = M Q ( u -ϵ ) M Q ( u ) . By Proposition 6, this implies Q u ∈ E left ( M Q ( u -ϵ ) M Q ( u ) , ϵ ) ∩ E right ( M Q ( u + ϵ ) M Q ( u ) , ϵ ) .

D Self concordance for subgaussian base distributions

For the convenience of the reader we restate the theorems to be proven.

Theorem 9. Let Q be subgaussian. Then, the NEF Q = ( Q u ) u ∈ R is self-concordant and Γ Q ( u ) = O ( | u | ) , u ∈ R .

Theorem 10. There exists a distribution Q that is subgaussian such that lim sup u →∞ Γ Q ( u ) /u > 0 .

We start by introducing some notations reminiscent of the ones used for subexponential distributions. Recall that a centered distribution Q is subgaussian if and only if for some c, C > 0 , it holds that

where X ∼ Q . Let

This definition is very close to that of E right. In words, G is the class of distributions over the reals whose left and right-tail display a subgaussian decay governed with the rate parameter c > 0 and scaling constant C > 0 .

For the interested reader, we also report here without proof some classical results on the quantitative relation between the MGF and the tail bounds of subgaussian distributions. Details can be found in the textbook [RH23].

Proposition 22 ([RH23]) . If for all u ∈ R , M Q ( u ) ≤ e σ 2 u 2 / 2 , then P ( | X | ≥ t ) ≤ 2 exp( -t 2 / (2 σ 2 ) , and if P ( | X | ≥ t ) ≤ 2 exp( -t 2 / (2 σ 2 )) , then for all u ∈ R , M Q ( u ) ≤ e 4 σ 2 u 2 .

D.1 Proof of Theorem 9

Lemma 12 still holds for Q a subgaussian distribution, hence the theorem holds trivially for ˙ µ (0) = 0 . Lemma 15 can also be applied for Q a subgaussian distribution. Thus, as in the proof for subexponential distributions, the first step is to show the result when µ (0) = 0 and ˙ µ (0) > 0 .

Note that the lower bound on ˙ µ ( u ) of Lemma 17 still holds when ˙ µ (0) > 0 . We turn to upper bounding ∫ | y -µ ( u ) | 3 Q u ( dy ) . The following Lemmas are first steps in that direction.

Lemma 23. Take Q ∈ G ( c, C ) a centered distribution. For all u ≥ 0 and t ≥ ( 4 c +1 ) u + 4 c , we have:

where C 1 = C ( 1 + √ π c ) .

Proof. We have

The first inequality holds as u ≥ 0 , the second by subgaussianity assumption. As t ≥ ( 4 c +1 ) u + 4 c , we have u ≤ c 4 t and:

This implies u ( t +1) ≤ c 4 t 2 on top of u ≤ c 4 t . Then

Reinjecting in Eq. (11), we get:

Lemma 24. Take Q ∈ G ( c, C ) a centered distribution. For all u, t ≥ 0 , we have:

where the last inequality holds by subgaussianity of Q .

Proof. We have that

Lemma 25. Take Q ∈ G ( c, C ) a centered distribution. For all u ≥ 0 the following holds:

where C 3 = √ π √ c C 1 .

Proof. We have:

By Jensen's inequality, M Q ( u ) = E Q [ e uY ] ≥ e u E Q [ Y ] = 1 . Noting that ∫ + ∞ 0 e -cy 2 4 dy ≤ √ π √ c terminates the proof.

Proof of Theorem 9. As previously noted, we first show the result when µ (0) = 0 and ˙ µ (0) > 0 . We start with u > 0 , then by Lemma 21 will extend the bound to u < 0 .

Take u > 0 and B > 0 a constant to be optimized later. We have:

The first equality is a classical result on the relationship between moments and tails of r.v., see for instance Exercise 1.2.3 of [Ver18]. The following bound holds:

We also have:

Set B = µ Q ( u ) + ( 4 c +1 ) u + 4 c + b , where b was defined in Lemma 17. As B ≥ ( 4 c +1 ) u + 4 c , by Lemma 23, we have:

where the first equality holds as for any a > 0 , using integration by part we have that

Denote B ′ = B -µ Q ( u ) = ( 4 c +1 ) u + 4 c + b . By Lemma 24 we have:

where in the second line we use change of variable; third line we use the fact that B ′ > 4 c u + b , hence e -ut ≤ e -B ′ u ≤ e -4 c u 2 -ub for all t ≥ B ′ ; fourth line change of variable and the fact that -( a + b ) 2 ≤ -a 2 -b 2 for all a, b ≥ 0 ; fifth line the fact that ( a + b ) 2 ≤ 2 a 2 +2 b 2 for all a, b ∈ R ; sixth line Eq. (15); and seventh line B ′ > 4 c u .

Combining the bounds on ( ii, a ) and ( ii, b ) with Lemma 17 we get:

As B ≥ 4 c u + b , we have cB 2 4 + ub ≥ 4 c u 2 . This implies

With Equation 14, we get

By Lemma 25, µ Q ( u ) ≤ ( 4 c +1 ) u + 4 c + C 3 , which implies :

For any constants w 1 , w 2 > 0 , we have w 1 u 2 e -w 2 u 2 ≤ w 1 √ w 2 u . This implies that for u > 0 :

Note that if Q ∈ G ( c, C ) , with Q -the distribution of -Y , Y ∼ Q , we have Q -∈ G ( c, C ) . By Lemma 21, that implies:

As, by Lemma 12, Γ Q is constant for ˙ µ (0) = 0 , it remains now only to show the theorem if µ (0) = 0 and ˙ µ (0) > 0 . We have just shown

with Q -µ (0) the centered version of Q . by Lemma 15, we have Γ Q = Γ Q -µ (0) . Hence, for any Q ∈ G ( c, C ) :

Therefore, for u ∈ R :

D.2 Proof for Theorem 10

We construct a distribution Q s.t. for any s > 0 , we can find some u > s with :

Let Q the base measure be with

where C 1 is a normalizing constant. By definition, we have that Q u ( dy ) = ∑ i ≥ 1 q i δ 2 i ( dy ) where

We are going to inspect | E [( Y -µ ( u )) 3 ] | E [( Y -µ ( u )) 2 ] for i a positive even number large enough and u = 2 i +1 . By definition, we have that Q u ( dy ) = ∑ i ≥ 1 q i δ 2 i ( dy ) where

̸

First, we remark a few simple equalities and inequalities that will prove useful in the subsequent computations.

Not that P ( U = 2 i ) = C 1 M Q ( u ) e 4 i , hence:

and, combining with Equation Eq. (19),

From this last equation we get upper bound:

The two next Lemma combined show that for any even i ≥ 4 , 1 . 24 × 2 i ≤ µ ( u ) ≤ 1 . 26 × 2 i

Lemma 26. Let i ≥ 4 be an even number and u = 2 . For random variable U ∼ Q u ( dy ) , it holds that:

. i +1

Proof. Consider j < i , j even. By Eq. (23),

Denote f 1 ( j ) = 2 i + j +1 -3 × 4 j -1 -4 i . The monotonicity of the right hand side, e f 1 ( j ) , is the same as f 1 ( j ) . Take derivative of f 1 ( j ) , we have that

Since i > j we have that f ′ 1 ( j ) ≥ 0 for j ∈ [0 , i ] . Hence the right hand side increases with j , as j ≤ i -1 , we have that

This implies:

This implies:

We now prove the lower bound on the mean.

where the fourth line holds because of Eqs. (20) and (26).

Lemma 27. Let i be a positive even number and u = 2 i +1 . For random variable U ∼ Q u ( dy ) , it holds that:

Proof. By Eq. (23), for any j ≥ i +2 , we have:

This implies:

We turn to bounding ∑ ∞ j =4 2 j e -2 j -1 :

where the second inequality holds as 2 x e -2 x is decreasing for x > 0 . Hence, for any i ≥ 2 :

We are now ready to upper bound µ u :

where we get the third inequality from Eqs. (20), (21), (26) and (29).

Proof for Theorem 10. Let U ∼ Q u where u = 2 i +1 for i ≥ 4 an even number. We first derive an upper bound on the variance E [( U -µ ( u )) 2 ] ≤ E [ U 2 ] .

where the last inequality is obtained from Eqs. (20), (21), (26) and (28).

We inspect the infinite series in the above inequality . Note that 2 2 j e -2 j -1 is decreasing for j ≥ 1 .

For all i ≥ 1 , it follows that

Plugging into Equation 30, we have:

On the other hand, by Lemmas 26 and 27, we have that

Then we can lower bound the third central moment E [ | U -µ ( u ) | 3 ] .

where the third inequality holds by Eqs. (21) and (26). Combining this last bound with Eq. (31), we obtain:

E Bandit algorithm

In this section, we present our analysis of Algorithm 1. The section is organized as follows. In Appendix E.1, we detail how we construct the confidence set. The proof that θ ⋆ lies in the confidence set with high probability will be presented only in Appendix E.4 so that we can continue in Appendix E.2 with the proof of the main result, Theorem 29, bounding the regret. This result differs from Theorem 11 by providing additional detail about the choice of the parameters in the algorithm. The proof presented in Appendix E.2 requires a number of technical lemmas that are presented as the proof develops. The proofs of these are postponed to subsequent sections. Before the proof of these, we devote the next section (Appendix E.3) to technical results on consequences of self-concordance which will be useful for the rest of the proofs. This is followed in Appendix E.4 by the proof that the confidence set constructed indeed has the required coverage. The next section (Appendix E.5) is devoted to proving Lemma 30 ('ellipsoidal diameter bound on the confidence set'), which is one of the two key results required for the proof of the main regret bound (besides the result on the coverage of the confidence set). A self-bounding property of self-concordance functions (Lemma 31), which is the second main ingredient of the regret bound proof is shown in Appendix E.6. Finally, for completeness, we present the (well known) elliptical potential lemma (stated here as Lemma 38) in Appendix E.7.

E.1 Constructing the confidence set

The confidence set construction is based on first obtaining the parameter vector ˆ θ t . This parameter vector is chosen to be the minimizer of the regularized negative log-likelihood function: ˆ θ t = arg min θ ∈ R d L ( θ ; D t ) where D t = (( X i , Y i )) t -1 i =1 is the data available in step t and

where λ > 0 is a tuning parameter (to be chosen later) and q ( y ; u ) = dQ u dQ ( y ) is the density of Q u with respect to Q at y ∈ R . It should be clear from the definitions that q is well-defined. The purpose of regularization is to ensure that the loss function has a unique optimizer even in the data poor regime.

For the construction of the confidence set it will be useful to derive an equivalent expression for the loss L . For this, first note that the density q satisfies q ( y ; u ) = exp( yu -ψ Q ( u )) . Plugging this into the definition of L , we get

As it is well known, ψ is a convex function of its argument (Theorem 1.13 of [Bro86]) and hence θ ↦→L ( θ ; D ) is strictly convex provided that λ > 0 .

For the confidence set construction we will need the non-constant part of the gradient of L ( · ; D t ) , which we denote by g t . We will also need the curvature of L ( · ; D t ) , which we denote by H t . These are

and

The minimizer ˆ θ t = arg max θ ∈ R d L ( θ ; D , λ ) has the property that

This implies that

With this, we can introduce our confidence set construction, which is based on the work of [Jan+24]. For δ ∈ (0 , 1] , we let

where for M to be chosen later (in Lemma 28),

Here, S 0 = sup {∥ θ ∥ : θ ∈ Θ } , as defined in the main body of the paper. In the algorithm we then choose C t to be C δ t ( ˆ θ t ) with a fixed value of δ ∈ [0 , 1] that bounds the failure probability of the algorithm. The following lemma, whose proof is postponed to Appendix E.4, as mentioned beforehand, shows that the confidence sets ∩ t ≥ 1 C δ t ( ˆ θ t ) have coverage 1 -δ :

Lemma 28. Let Assumptions 1 and 2 hold and choose M ≥ max( K/ log(2) , 1 / ( c 1 -S 1 ) , 1 / ( c 2 + S 2 )) in Eqs. (34) and (35) , where Γ is any stretch function for ( Q u ) u ∈ [ S 2 ,S 1 ] and K = sup S 2 ≤ u ≤ S 1 Γ( u ) . Then, for the confidence set defined in Eq. (33) and for all δ ∈ (0 , 1] ,

Note that Theorem 7 and Assumption 2 guarantees the existence of a stretch function Γ mentioned in the theorem.

In the remainder of this section, we will fix Γ to one such stretch function.

In general, here, one wants to use the smallest such stretch function (i.e., Γ = Γ Q ). When Γ Q is not available, in the lack of a better choice for Γ , the choice worked out in Theorem 7 can be used.

E.2 Proof of regret upper bound

Let E δ be the event that E δ = { θ ⋆ ∈ C δ t ( ˆ θ t ) } which by Lemma 28 holds with probability at least 1 -δ . For the next theorem, recall that S 0 = sup θ ∈ Θ ∥ θ ∥ and S 1 = sup U , S 2 = inf U .

Theorem 29. Let δ ∈ (0 , 1] and T a positive integer and consider a well-posed GLB model G = ( X , Θ , Q ) and assume that Assumptions 1 and 2 hold. Then, by setting C t = C δ t ( ˆ θ t ) , for any θ ⋆ ∈ Θ , with probability at least 1 -δ , the regret Regret( T ) of Algorithm 1 when it interacts with the GLB instance specified by θ ⋆ can be upper bounded by,

where c = (1 + 2 K ( S 1 -S 2 )) and

Proof. We first consider the case that the base distribution has 0 variance, which implies that Q is a Dirac. As discussed beforehand, and as it is easy to see it, in this case U Q = R , Q u = Q for any u ∈ R . Hence, all arms have the same payoff and all algorithm incur zero regret. In the rest of this proof, we assume that Var( Q ) > 0 . Since µ ( · ) = ˙ ψ Q ( · ) is infinitely differentiable (Proposition 2), we can perform a second-order Taylor expansion on the regret

where ξ t is between X ⊤ t θ ⋆ and x ⊤ ⋆ θ ⋆ for all t ∈ [ T ] . On event E δ , by definition of X t , θ t (in Algorithm 1), it holds that x ⊤ ⋆ θ ⋆ ≤ X ⊤ t θ t . Observe that γ t ( δ ) (Eq. (35)) is increasing in t , we have that γ t ( δ ) ≤ γ T ( δ ) for all t ∈ [ T ] . Then we can bound R 1 ( T ) as follows

where the last inequality is due to Cauchy-Schwarz. Since θ t and θ ⋆ are all in the confidence set on E δ , we are able to bound ∥ θ t -θ ⋆ ∥ H t ( θ ⋆ ) by the following lemma that exploits the properties of confidence set as well as self-concordant functions. This lemma is a variation of proposition 4 of Abeille, Faury, and Calauzènes [AFC21] where they show the result for logistic function that is 1 -self-concordant.

Lemma 30 ( C δ t ( ˆ θ t ) has small ellipsoidal diameters) . Under Assumptions 1 and 2, for all θ 1 , θ 2 ∈ C δ t ( ˆ θ t ) , it follows that

where K is defined in Eq. (36) .

By Lemma 30, we can upper bound R 1 ( T ) to be

Denote A t = √ ˙ µ ( X ⊤ t θ ⋆ ) X t and we have that H t ( θ ⋆ ) = ∑ t s =1 A t A ⊤ t + λI as well as ∥ A t ∥ ≤ √ L ≤ L where the second inequality is because WLOG we assume L ≥ 1 in Assumption 2. We can

bound R 1 ( T ) in the terms of A t .

where in the last step we use elliptical potential lemma of Abbasi-Yadkori, Pál, and Szepesvári [APS11], which, for easy of reference, we give in Lemma 38. We now start to bound R 2 ( T ) . For convenience, we throw away the factor of 1 / 2 .

where in the last inequality we use Lemma 30. By definition of self-concordant function, we have that ¨ µ ( ξ t ) ≤ Γ( ξ t ) ˙ µ ( ξ t ) . Since ξ t is between x ⊤ ⋆ θ ⋆ and X ⊤ t θ ⋆ . Note that Γ defined in Theorem 7 is increasing on [0 , c 1 ) and decreasing on ( -c 2 , 0) , which gives us Γ( ξ t ) ≤ Γ( X ⊤ t θ ⋆ ) ∨ Γ( x ⊤ ⋆ θ ⋆ ) ≤ K . Let V t = λI + ∑ t i =1 X i X ⊤ i . We hence have

Putting the bound on R 1 ( T ) and R 2 ( T ) together, we have that

We mimic the trick used in Janz, Liu, Ayoub, and Szepesvári [Jan+24] to bound √ ∑ T t =1 ˙ µ ( X ⊤ t θ ⋆ ) which was originally proposed by Abeille, Faury, and Calauzènes [AFC21]. We present the following lemma that is abstracted out from Claim 14 of Janz, Liu, Ayoub, and Szepesvári [Jan+24].

Lemma 31 (Self-bounding property of self-concordance functions) . Let V = [ a, b ] , a closed, nonempty interval over the reals, f a real valued function defined over an interval of the reals that is twice continuously differentiable over V such that for some Γ : V → R + , | ¨ f ( v ) | ≤ Γ( v ) ˙ f ( v ) holds for all v ∈ V . Assume that A = sup v ∈V Γ( v ) < ∞ . Furthermore, assume that either ˙ f is identically zero over V , or ˙ f is positive valued over V . For n a positive integer, let { a t } n t =1 ⊂ V . Then,

We apply this lemma with f = µ , [ a, b ] = [ S 2 , x ⊤ ⋆ θ ⋆ ] ⊂ [ S 2 , S 1 ] and Γ restricted to [ a, b ] . Then, all the conditions of the lemma are satisfied by our choice of Γ . Furthermore, A = sup v ∈ [ S 2 ,x ⊤ ⋆ θ ⋆ ] Γ( v ) ≤ K < + ∞ . Hence, all the conditions of the lemma are verified. Hence,

Plug Eq. (38) into Eq. (37),

Let

we can write out the inequality

Solving it we have that

Plugging in the definition of A and B back,

E.3 Self-concordance control

In this section we provide technical results about self-concordant functions which play important roles in confidence set construction and controlling the regret of Algorithm 1. Specifically, Corollary 33 and Lemma 34 are used to show Lemma 30, one of the key lemmas we use in the proof of Theorem 29. Lemma 5 is used to justify the confidence set contains θ ⋆ with high probability (Lemma 28).

The next lemma shows that for self-concordant NEFs, ˙ µ is a smooth function of its argument. The lemma is essentially the same as Lemma 3 from Janz, Liu, Ayoub, and Szepesvári [Jan+24] (itself based on a result of Sun and Tran-Dinh [ST19]) and is updated only to match our definitions of self-concordance, which is a refinement of that used by Janz, Liu, Ayoub, and Szepesvári [Jan+24]. The proof (based on the proof of a similar result of Sun and Tran-Dinh [ST19]) is included for the convenience of the reader.

Lemma 32 (Self-concordance to smoothness) . Let U be an interval over the reals, µ a real valued function defined over an interval of the reals that is twice continuously differentiable over U such that for some Γ : U → R + , | ¨ µ ( u ) | ≤ Γ( u ) ˙ µ ( u ) holds for all u ∈ U . Assume that K = sup u ∈U Γ( u ) < ∞ . Assume that either ˙ µ is identically zero over U , or ˙ µ is positive valued over U . Then, for any u, u ′ ∈ U ,

An immediate corollary of this lemma is that self-concordance of a NEF implies that the variance function, ˙ µ , of the NEF is smooth:

Corollary 33 (Self-concordance to smoothness) . Let Q = ( Q u ) u ∈U be self-concordant with stretch function Γ : U → R + , where U is an interval and assume K = sup u ∈U Γ( u ) < ∞ . Then for any u, u ′ ∈ U ,

Note that the inequality is well-posed since u, u ′ ∈ U ◦ Q , and µ is known to be differentiable over U ◦ Q and, by the definition of self-concordance, U ⊂ U ◦ Q .

Proof. This result follows from Lemma 32 once we notice that the variance function of a NEF is such that if ˙ µ ( u ) = 0 for any u ∈ U , then ˙ µ is identically zero over U . Indeed, if ˙ µ ( u ) = 0 , then Q u is a Dirac distribution and so is Q v for any v ∈ U .

Proof of Lemma 32. When ˙ µ is identically zero over U , the statement is trivial. Hence, consider now the case when ˙ µ is positive valued over U :

Then, it suffices to show that ln ˙ µ ( u ′ ) ˙ µ ( u ) ≤ K | u -u ′ | . To show this, define ϕ ( t ) = ˙ µ ( u + t ( u ′ -u )) so that ϕ (0) = ˙ µ ( u ) and ϕ (1) = ˙ µ ( u ′ ) . Since U Q is an interval with non-empty interior, ϕ ( t ) is well-defined for all t ∈ [0 , 1] . Furthermore, by Eq. (39) and since U is an interval, we have that ϕ ( t ) > 0 for all t ∈ [0 , 1] . Consider now the map t ↦→ ln ϕ ( t ) where t ∈ [0 , 1] . The derivative of this map exist and is continuous over (0 , 1) , and in particular, d dt ln ϕ ( t ) = ˙ ϕ ( t ) ϕ ( t ) by the chain rule. Indeed, the derivative of ϕ exists and is continuous over (0 , 1) , because the same holds for ˙ µ by the properties of NEFs, and as we just discussed, ϕ is positive over [0 , 1] and is continuous. Now, by the fundamental theorem of calculus applied to t ↦→ d dt ln ϕ ( t ) and by the monotonicity of integrals,

It remains to bound the integrand in the rightmost expression. For this, as discussed earlier we have

To bound the ratio on the right, we again use the chain rule and calculate

where the inequality follows by the definition of K by definition of self-concordant function, we have that for all u ∈ U , | ¨ µ ( u ) | ≤ K ˙ µ ( u ) . Now, the result follows since we have shown that the integrand is upper bounded by K | u -u ′ | and thus ∫ 1 0 ∣ ∣ ∣ d ln ϕ ( t ) dt ∣ ∣ ∣ dt ≤ K | u -u ′ | .

We continue with two results, both of which use the lemma just proved. The first result gives a lower bound for the integral remainder term when Taylor's theorem is used to approximate µ . The second result gives a quadratic upper bound on the CGF of Q u , and will be the basis for constructing our confidence set.

Lemma 34. Let Q = ( Q u ) u ∈U be self-concordant with stretch function Γ : U → R + where U is an interval, and assume K = sup u ∈U Γ( u ) < ∞ . Then for any u, u ′ ∈ U ,

Integrating both sides between 0 and 1 gives us

where the last inequality follows from the elementary inequality (1 -e -x ) /x ≥ 1 / (1 + x ) that holds for all x ≥ 0 .

We are now ready to prove Lemma 5. As noted beforehand, we adopt this lemma from the work of Janz, Liu, Ayoub, and Szepesvári [Jan+24]. In particular, it is an adaptation of their Lemma 1, which was proved for distributions where U Q = R . Here, we deal with the case when U Q is possibly a strict subset of R .

Lemma 5 (From self-concordance to light tails) . Let Q = ( Q u ) u ∈U be a NEF which is selfconcordant with stretch function Γ : U → R + where U is a subinterval of U ◦ Q = ( a, b ) . Then, for any u ∈ U ,

Proof. Let u ∈ U . Hence, by our assumption on U , u ∈ U Q . Now let s ∈ R . Then,

Hence, ψ Q u ( s ) is finite valued whenever u + s ∈ U Q . Assume that this holds and in fact u + s ∈ U ◦ Q .

◦

Since u, u + s ∈ U Q , ψ Q is twice continuously differentiable over an open interval containing u and u + s . Then, by Taylor's theorem there exists ξ in the closed interval between u and u + s such that

Since ˙ ψ Q = µ and ¨ ψ Q = ˙ µ (cf. Proposition 2) we get

Now, by Corollary 33, we have that

where the final inequality follows when | s | ≤ log(2) /K . Putting things together, it follows that if | s | ≤ log(2) /K and s ∈ U ◦ Q -{ u } then

For S ⊂ R , r ∈ R , let S ± r = { s ± r : s ∈ S } . Since u is an arbitrary point in U , the above conditions on s will be satisfied if | s | ≤ log(2) /K and s ∈ Z . = ∩ u ∈U U ◦ Q -u . Now, from U ◦ Q = ( a, b ) , we have Z = ∩ u ∈U ( a -u, b -u ) = (sup u ∈U a -u, inf u ∈U b -u ) = ( a -inf U , b -sup U ) , finishing the proof.

From the calculation at the end of the proof it follows that the statement of the lemma is non-vacuous if for U ◦ Q = ( a, b ) , a -inf U < 0 < b -sup U , which is equivalent to that a < inf U and sup U < b , which is always satisfied when U is a strict subset of U ◦ Q .

E.4 Confidence set construction

We now turn to proving Lemma 28 which is concerned with showing that the confidence sets C δ t contain the true parameter θ ⋆ with probability 1 -δ . The proof is essentially the same as that of Lemma 4 of [Jan+24]. We will need the following result, which is taken verbatim from the paper of Janz, Liu, Ayoub, and Szepesvári [Jan+24].

Proposition 35 (Theorem 2 of [Jan+24]) . Fix λ, M > 0 . Let ( X t ) t ∈ N + be a B d 2 -valued random sequence, ( Y t ) t ∈ N + a real valued random sequence and ( ν t ) t ∈ N be a nonnegative valued random sequence. Let F = ( F t ) t ∈ N be a filtration such that ( i ) ( X t ) t ∈ N + is F -predictable and ( ii ) ( Y t ) t ∈ N + are F -adapted. Let ϵ t = Y t -E [ Y t | F t -1 ] and assume that the following condition holds:

Then, for ˜ H t = ∑ t i =1 ν i -1 X i X ⊤ i + λI and S t = ∑ t i =1 ϵ i X i and δ > 0 ,

We now turn to proving Lemma 28. For the convenience of the reader, we start by recalling the definition of the confidence sets C δ t involved (cf. Eq. (33)). Recall that c 2 < S 2 ≤ x ⊤ θ ≤ S 1 < c 1 for x ∈ X and θ ∈ Θ . For δ ∈ (0 , 1] , we have

where

and recall that S 0 = sup θ ∈ Θ ∥ θ ∥ or an upper bound on this quantity, and M is specified in the next result:

Lemma 28. Let Assumptions 1 and 2 hold and choose M ≥ max( K/ log(2) , 1 / ( c 1 -S 1 ) , 1 / ( c 2 + S 2 )) in Eqs. (34) and (35) , where Γ is any stretch function for ( Q u ) u ∈ [ S 2 ,S 1 ] and K = sup S 2 ≤ u ≤ S 1 Γ( u ) . Then, for the confidence set defined in Eq. (33) and for all δ ∈ (0 , 1] ,

Proof. From definition it follows that θ ⋆ ∈ Θ . Now we prove that with probability at least 1 -δ , it holds that ∥ g t ( θ ⋆ ) -g t ( ˆ θ t ) ∥ ≤ γ T ( δ ) for all t ≥ 1 . The proof goes through by using Proposition 35 and we now match the conditions of Proposition 35. By Assumption 1, we have that ( X t ) t ∈ N + is a B d 2 -valued random sequence. Let F t -1 = σ ( X 1 , Y 1 , ..., X t -1 , Y t -1 , X t ) for t ≥ 1 . Consider the filtration F = ( F t ) t ∈ N . Then by definition, ( X t ) t ∈ N + are F -predictable and ( Y t ) t ∈ N + are F -adapted. Note that µ ( X ⊤ i θ ⋆ ) = E [ Y i |F i -1 ] for all i ∈ [ n ] . Let ε i = Y i -E [ Y i |F i -1 ] . This gives ( ε t ) t ∈ N + are also F -adapted and the following identity follows by definition

where S t = ∑ t i =1 ε i X i . Let

Now we would like to apply Lemma 5 to show that, for | s | ≤ M , it follows that

Applying the definition of ε t , it follows that

where the last equality is because X t is F t -1 -measurable. Since, by definition, the distribution of Y t given F t -1 is Q X ⊤ t θ ⋆ , we have that

where the last inequality is because | s | ≤ M implies that s ∈ [ -log 2 /K, log 2 /K ] ∩ ( U ◦ Q -S 2 ) ∩ ( U ◦ Q -S 1 ) so Lemma 5 is applicable (it is applied with ( Q u ) u ∈ [ S 2 ,S 1 ] and Γ as chosen in the statement). Then, Eq. (42) follows by noting that ν t -1 = ˙ µ ( X ⊤ t θ ⋆ ) .

Lastly as defined above, ˜ H t corresponds to H t ( θ ⋆ ) . Taking the ℓ 2 -norm weighted by H -1 t ( θ ⋆ ) and applying triangle inequality, √

where the last inequality follows by H -1 t ( θ ⋆ ) ⪯ λ -1 I . By Proposition 35, with probability at least 1 -δ , it follows that for all t ≥ 1 , √

We now bound det( H t ( θ ⋆ )) /λ d . Let A i = √ ˙ µ ( X ⊤ i θ ⋆ ) X i for all i ∈ [ t ] , then H t ( θ ⋆ ) can be written as

By Assumption 2, it holds that √ ˙ µ ( X ⊤ t θ ⋆ ) ≤ √ L , thus ∥ A i ∥ 2 ≤ √ L ≤ L for all i ∈ [ t ] . Eq. (20.9) (Note 1 of section 20.2) in [LS20] gives

The stated result follows by chaining all the inequalities together and noting that

E.5 Proof of Lemma 30

The following two lemmas (Lemmas 36 and 37) are variations of Claim 4 and Claim 3 of Janz, Liu, Ayoub, and Szepesvári [Jan+24]. The difference is that Janz, Liu, Ayoub, and Szepesvári [Jan+24] show them for all θ 1 , θ 2 ∈ R d because the MGF M Q therein is finite on R . In our setting, there could be x ∈ X for some θ / ∈ Θ such that M Q ( x ⊤ θ ) = ∞ , hence we show it within the parameter set Θ .

Lemma 36. For all θ 1 , θ 2 ∈ Θ , it follows that

In particular, we have that

Proof. The 'In particular' part follows from definition of ℓ 2 -norm weighted by G -1 t ( θ 1 , θ 2 ) . We now prove g t ( θ 1 ) -g t ( θ 2 ) = G t ( θ 1 , θ 2 )( θ 1 -θ 2 ) . By definition of the difference quotient α ( · , · ) , we have that

Writing out the expression of g t ( θ 1 ) -g t ( θ 2 ) gives

Lemma 37. Under Assumptions 1 and 2, for all θ 1 , θ 2 ∈ Θ , it follows that

where K is defined in Eq. (36) .

Proof. Since { x ⊤ θ : x ∈ X , θ ∈ Θ } ⊂ [ S 2 , S 1 ] by Assumption 1, we have

By Lemma 34, we have that for all x ∈ X ,

Then the following holds

where the last inequality follows by (1 + 2 K · ( S 1 -S 2 )) -1 ≤ 1 . The proof of Eq. (46) follows by substituting θ 1 with θ 2 .

Lemma 30 ( C δ t ( ˆ θ t ) has small ellipsoidal diameters) . Under Assumptions 1 and 2, for all θ 1 , θ 2 ∈ C δ t ( ˆ θ t ) , it follows that

where K is defined in Eq. (36) .

Proof. We first prove the statement for ∥ θ 1 -θ 2 ∥ H t ( θ 1 ) . By Lemma 37, we have that

Note that θ 1 , θ 2 ∈ C δ t ( ˆ θ t ) by hypothesis, then Lemma 37 and the definition of C δ t ( ˆ θ t ) gives that

Chaining all the inequalities together finishes the proof. The proof for the statement for ∥ θ 1 -θ 2 ∥ H t ( θ 2 ) follows similarly by substituting θ 1 with θ 2 .

E.6 Proof of self-bounding property of self-concordance functions: Lemma 31

As mentioned beforehand, the following lemma is abstracted out from Claim 14 of Janz, Liu, Ayoub, and Szepesvári [Jan+24]:

Lemma 31 (Self-bounding property of self-concordance functions) . Let V = [ a, b ] , a closed, nonempty interval over the reals, f a real valued function defined over an interval of the reals that is twice continuously differentiable over V such that for some Γ : V → R + , | ¨ f ( v ) | ≤ Γ( v ) ˙ f ( v ) holds for all v ∈ V . Assume that A = sup v ∈V Γ( v ) < ∞ . Furthermore, assume that either ˙ f is identically zero over V , or ˙ f is positive valued over V . For n a positive integer, let { a t } n t =1 ⊂ V . Then,

Proof. We have

finishing the proof.

E.7 Auxiliary Lemma

Lemma 38 (Elliptical potential lemma) . Fix λ, A > 0 . Let { a t } ∞ t =1 be a sequence in AB d 2 and let V 0 = λI . Define V t +1 = V t + a t +1 a ⊤ t +1 for each t ∈ N . Then, for all n ∈ N + ,

Proof. See, e.g., Lemma 19.4 of Lattimore and Szepesvári [LS20].

F Numerical Simulations

In this section, we report our results on numerical simulations. We run our algorithm on exponential bandits. The setting is as follows. The base distribution is an exponential distribution with parameter λ > 0 where the probability density function can be written as

For each arm x ∈ X , the reward distribution is an exponential distribution with parameter λ -x ⊤ θ ⋆ , where we ensure that sup x ∈X x ⊤ θ ⋆ < λ , as illustrated by Example 1. The number of arms |X| = 20 and X ⊆ R 2 , the max variance of the reward distributions among all arms is 0 . 25 and κ = sup x ∈X 1 / ˙ µ ( x ⊤ θ ⋆ ) ≈ 100 . We run our algorithms with theory suggested parameters for 60 runs where each run has horizon 5000 . To be more specific on the parameters, we list them here:

The failure probability δ is 0 . 05 .

The regularizer is set to be 2 .

The confidence width γ t ( δ, λ ) is set according to Eq. (43).

Here are the results of the experiments. In Fig. 1 we plot the average regret along with standard

400

350

300

250

regret

200

150

100

50

0

0

1000

2000

3000

4000

5000

horizon

6

OFU-GLM

log(x~0.5)+1.5

5

4

log(regret)

3

2

1

0

0

2

4

6

8

log(horizon)

deviation. From the plot we can see that the regret attained seems to be sublinear. In Fig. 2 we display the log-log plot where the x-axis is log( horizon ) and the y-axis is log( regret ) . The slope gradually approaches to 0 . 5 , i.e., the growth rate of regret approaches to √ T which confirms our theoretical bound.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and/or introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We do not have a lower bound.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide detailed assumptions and very detailed proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We do not have experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We do not have experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We do not have experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We do not have experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We do not have experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper discusses Generalized Linear Bandits (GLB) under the self-concordance assumption. The study successfully relaxes the limitations of existing work with an OFU-type algorithm, providing mathematically solid theories for GLB within the self-concordance family. However, the presentation could be significantly improved, and the paper's style does not align well with the venue.

STRENGTHS:
This paper has a sound theoretical foundation with the relaxation of some assumptions in the existing literature.

WEAKNESSES:
The first section should provide a comprehensive overview of the study. The core concept of "self-concordance" is introduced abruptly and is not well-explained. The literature review is incomplete. Specifically, the differences between this study and the existing literature, such as [Rus+21], are not thoroughly discussed. As a result, the contribution of this study is unclear. The paper dedicates the first seven pages to the theoretical foundation and introduces the model and algorithm thereafter. Due to space constraints, these are not fully illustrated, and the paper lacks experimental results. This organization is not well-suited to this venue.

QUESTIONS:
This paper talks about "being free of exponential dependency". I would like to see what the term means in detail. Also, why "being free of exponential dependency" is important in analyses and applications?

LIMITATIONS:
This paper discusses the single-parameter natural exponential family. But, I believe that this family is not comprehensive enough to include most applications in the real world.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors prove that any single-parameter natural exponential family (NEF) with subexponential (subgaussian) base distribution is self-concordant with a stretch factor that grows inverse quadratically (linearly).\n\nSTRENGTHS:\n- Clearly and well-written\n- An important theoretical contribution in establishing that self-concordance for exponential tilting comes for free, which has numerous applications in statistics and bandits (as alluded to in the Conclusion). Surprisingly, the proof consists of some concentration results combined with clever integral/infinite sum manipulations.\n- For the subgaussian, the authors established that the linear growth is tight by establishing a lower bound, which is technically quite interesting.\n- First $d\\sqrt{T/\\kappa_\\star}$ type regret that holds for a wide range of generalized linear bandits, well beyond logistic bandits\n\nWEAKNESSES:\n- The confidence set for the OFU-GLB is nonconvex, and thus, the current algorithm is computationally intractable. There has been some progress on achieving convex confidence sets that are also statistically (regret-wise) tight -- see, e.g., [1,2]. It would be nice to write the confidence set in a convex form, as it is repeatedly alluded to during the authors' proof.\n- With the convex confidence set, it would be nice to see some numerical experiments, especially the exponential bandits.\n- The norm parameters $S_0, S_1, S_2$ were introduced but never kept track of. The dependencies on these norm parameters have been a subject of study on their own [2,3] and are known to perform well [2]. It would be nice to have those dependencies appearing in the main text as well.\n- Most of the discussions regarding the proof of regret analyses are relegated to the Appendix. From my perspective (heavily biased towards bandits community), the techniques used there are arguably a bit more interesting. I would really like to see the technical contributions made in the main text's Section 4, especially the self-concordance control lemmas used (Appendix E.3) and Lemma 31 and more.\n\n\n**Comments**\n- It would be nice to include some discussions regarding [3], which provides the first $d\\sqrt{T/\\kappa_\\star}$-type regret for *bounded* generalized linear bandits but still goes beyond logistic bandits.\n- For each theorem statement, if not already done so, I would like to see a direct (hyper)ref to the Appendix containing its proof\n- Typo: \"Lemma ??\" in pg. 24\n\n[1] https://proceedings.mlr.press/v130/abeille21a.html\n\n[2] https://proceedings.mlr.press/v238/lee24d.html\n\n[3] https://arxiv.org/abs/2404.06831\n\nQUESTIONS:\n- Can the regret analyses be extended to changing arm-set case, i.e., the arm-set $\\mathcal{X}_t$ varies across $t \\in [T]$?\n-\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the self-concordance properties of single-parameter natural exponential families (NEFs) with subexponential tails. It provides two main contributions: first, it demonstrates that NEFs with subexponential tails are self-concordant with polynomial-sized parameters, and second, it applies these findings to generalized linear bandits (GLBs). The authors derive novel second-order regret bounds for GLBs that are free of exponential dependence on the problem parameters. This result extends the applicability of optimistic algorithms for GLBs to reward distributions such as Poisson, exponential, and gamma.\n\nSTRENGTHS:\n- The results on the self-concordance property of NEFs are definitely interesting and general, going beyond its application to GLBs. They can be interesting to many other problem settings (as mentioned by the authors themselves).\n- Not only do the authors provide an algorithm for GLBs, but it guarantees a second-order regret bound. This is particularly desirable to achieve better data adaptivity.\n- The results are presented in a clear manner, with clear definitions and helpful examples.\n\nWEAKNESSES:\n- It appears that the algorithmic results in this work tightly rely on the knowledge of some parameters of the NEF. Parameter-free results are quite important and have become relevant, especially in the past few years.\n- The paper could benefit from a slightly more detailed comparative analysis with existing methods, highlighting the advantages and potential trade-offs of the proposed approach in various scenarios, and how these prior results would fit in the setting considered in this work.\n- The papers could benefit from experimental results, although it is very clear that this is a theoretical paper.\n\nMinor details:\n- Many references have \"et al.\" instead of the full list of authors. This should be fixed.\n\nQUESTIONS:\n- Do the authors believe it possible to lift prior knowledge on parameters such as $S_0$, $L$, $c_1$, and $c_2$ in designing efficient algorithms for GLBs with NEF rewards?\n\nLIMITATIONS:\nThe authors addressed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}