Title: How Does Variance Shape the Regret in Contextual Bandits?
Venue: NeurIPS 2024
Accepted: yes
Keywords: contextual bandit

regret analysis
Abstract: We consider realizable contextual bandits with general function approximation, investigating how small reward variance can lead to better-than-minimax regret bounds.  Unlike in minimax regret bounds, we show that the eluder dimension 
$d_{\text{elu}}$$-$a measure of the complexity of the function class$-$plays a crucial role in variance-dependent bounds. We consider two types of adversary: 

(1) Weak adversary: The adversary sets the reward variance before observing the learner's action. In this setting, we prove that a regret of $\Omega( \sqrt{ \min (A, d_{\text{elu}}) \Lambda } + d_{\text{elu}} )$ is unavoidable when $d_{\text{elu}} \leq \sqrt{A T}$, where $A$ is the number of actions, $T$ is the total number of rounds, and $\Lambda$ is the total variance over $T$ rounds. For the $A\leq d_{\text{elu}}$ regime, we derive a nearly matching upper bound $\tilde{O}( \sqrt{ A\Lambda } + d_{\text{elu} } )$ for the special case where the variance is revealed at the beginning of each round.  

(2) Strong adversary: The adversary sets the reward variance after observing the learner's action. We show that a regret of $\Omega( \sqrt{ d_{\text{elu}} \Lambda } + d_{\text{elu}} )$ is unavoidable when $\sqrt{ d_{\text{elu}} \Lambda } + d_{\text{elu}} \leq \sqrt{A T}$. In this setting, we provide an upper bound of order $\tilde{O}( d_{\text{elu}}\sqrt{ \Lambda } + d_{\text{elu}} )$.

Furthermore, we examine the setting where the function class additionally provides distributional information of the reward, as studied by Wang et al. (2024). We demonstrate that the regret bound 
$\tilde{O}(\sqrt{d_{\text{elu}} \Lambda} + d_{\text{elu}})$ established in their work is unimprovable when $\sqrt{d_{\text{elu}} \Lambda} + d_{\text{elu}}\leq \sqrt{AT}$. However, with a slightly different definition of the total variance and with the assumption that the reward follows a Gaussian distribution, one can achieve a regret of $\tilde{O}(\sqrt{A\Lambda} + d_{\text{elu}})$.

How Does Variance Shape the Regret in Contextual Bandits?

Abstract

We consider realizable contextual bandits with general function approximation, investigating how small reward variance can lead to better-than-minimax regret bounds. Unlike in minimax regret bounds, we show that the eluder dimension d elu-a measure of the complexity of the function class-plays a crucial role in variance-dependent bounds. We consider two types of adversary:

Weak adversary: The adversary sets the reward variance before observing the learner's action. In this setting, we prove that a regret of Ω( √ min { A,d elu } Λ+ d elu ) is unavoidable when d elu ≤ √ AT , where A is the number of actions, T is the total number of rounds, and Λ is the total variance over T rounds. For the A ≤ d elu regime, we derive a nearly matching upper bound ˜ O ( √ A Λ+ d elu ) for the special case where the variance is revealed at the beginning of each round.

Strong adversary: The adversary sets the reward variance after observing the learner's action. We show that a regret of Ω( √ d elu Λ+ d elu ) is unavoidable when √ d elu Λ + d elu ≤ √ AT . In this setting, we provide an upper bound of order ˜ O ( d elu √ Λ+ d elu ) .

Furthermore, we examine the setting where the function class additionally provides distributional information of the reward, as studied by Wang et al. (2024). We demonstrate that the regret bound ˜ O ( √ d elu Λ + d elu ) established in their work is unimprovable when √ d elu Λ + d elu ≤ √ AT . However, with a slightly different definition of the total variance and with the assumption that the reward follows a Gaussian distribution, one can achieve a regret of ˜ O ( √ A Λ+ d elu ) .

1 Introduction

We consider the contextual bandit problem that models repeated interactions between the learner and the environment. In each round, the learner chooses an action based on the received context, and observes the reward of the chosen action. Algorithms designed to achieve minimax regret guarantees under a variety of statistical assumptions and computational models have been extensively studied (Auer et al., 2002; Dudik et al., 2011; Agarwal et al., 2012, 2014; Foster and Rakhlin, 2020; Xu and Zeevi, 2020; Simchi-Levi and Xu, 2022; Zhang, 2022).

However, these algorithms often fail to leverage the potentially benign nature of the environment. In this work, we refine the regret bound by considering the variance of the reward. Such variancedependent regret bounds, also known as second-order regret bounds, have been primarily studied under linear function approximation (Zhang et al., 2021; Kim et al., 2022; Zhao et al., 2023). Notably,

Jian Qian

Chen-Yu Wei

Zhao et al. (2023) first established a near-optimal ˜ O ( d √ Λ+ d ) regret bound for linear contextual bandits, where d represents the feature dimension and Λ the sum of the reward variances.

For contextual bandits with general function approximation, the recent work by Wang et al. (2024) obtained a second-order bound assuming access to a model class containing distributional information about the reward. They showed a regret bound of ˜ O ( √ d elu Λlog |M| + d elu log |M| ) , where |M| is the size of the model class, and d elu = d elu ( M ) is its eluder dimension. As noted in Wang et al. (2024), the dependence on d elu is undesirable, and when the number of actions A is much smaller than d elu, this bound can potentially be improved. This conjecture is supported by Foster and Rakhlin (2020), who showed that the upper bound ˜ O ( √ AT log |F| + A log |F| ) is achievable regardless of the eluder dimension, where T ≥ Λ is the number of rounds, and |F| is the size of the function class containing mean reward information. It is tempting to conjecture that the regret can smoothly scale with Λ , resulting in a bound of ˜ O ( √ A Λlog |F| + A log |F| ) . Such variance-dependent regret bounds that replace the dependence on the number of rounds T by the total variance Λ have been shown in multi-armed bandits (Audibert et al., 2009), linear bandits (Ito and Takemura, 2023), and linear contextual bandits (Zhao et al., 2023).

In this paper, we show, surprisingly, that the aforementioned conjecture is not true in general. Specifically, for any A and any d elu ≤ √ AT , one can construct a problem instance with lower bound Ω( √ min { A,d elu } Λ + d elu ) with log |F| = O (log T ) and d elu ( F ) = d elu . This rules out the possibility of achieving ˜ O ( √ A Λlog |F| + A log |F| ) for all A because we can always make d elu = √ AT , resulting in a lower bound Ω( √ AT ) even with Λ = 0 . Our primary goal is to design algorithms that achieve the near-optimal regret bound ˜ O ( √ min { A,d elu } Λlog |F| + d elu log |F| ) .

The lower bound √ min { A,d elu } Λ+ d elu indicates that the complexity of contextual bandits arises from two parts. The first part accounts for local estimation of the true function, where the complexity is due to the variance of the reward and the local structure of the function set around the ground truth function f ⋆ ∈ F . This results in the term √ min { A,d elu } Λ , with the leading coefficient min { A,d elu } corresponding to the decision-estimation coefficient (Foster et al., 2021). The second part accounts for global search for the true function, in which the complexity is due to a more global structure of the function set and can be quantified by the disagreement among the functions. The complexity of this part scales with d elu, even when A = 2 . The contribution of the global part is usually overshadowed by the local part when only considering regret bounds with constant variance. Our work highlights its role by studying the variance-dependent bound. The fundamental role of disagreement is also discussed in Foster et al. (2020) for gap-dependent bounds. Specifically, they also showed that when trying to obtain the gap-dependent bound that has logarithmic dependence on T , the complexity must scale with some disagreement measure over the function class, instead of just the number of actions.

The previous work by Wei et al. (2020) also derived a set of results for general contextual bandits showing that the tight second-order regret bound is strictly larger than merely replacing the T in the minimax bound by the second-order error. They consider the more general agnostic setting but the tight regret bounds are only established for the |F| = 1 case. Their result for |F| > 1 can be applied to our setting, though it only gives highly sub-optimal bounds. Overall, our work refines theirs in the realizable setting.

When preparing our camera-ready version, the concurrent work of Pacchiano (2024), which studied exactly the same problem as ours, was posted on arXiv. We provide a comparison with their work in Section 3.4. More related works are discussed in Appendix A.

2 Preliminaries

A contextual bandit problem consists of a context space X , an action space A , the total number of rounds T and a class of functions F ⊂ [0 , 1] X×A . At round t , the learner observes a context x t ∈ X , then makes a decision a t ∈ A based on the current context x t and history, and observes a reward r t . We assume that these rewards r t are given by

where f ⋆ : X × A → [0 , 1] is some function unknown to the learner, and ϵ t are independent zeromean random variables with variance σ 2 t such that r t ∈ [0 , 1] . 1 We denote Λ = ∑ T t =1 σ 2 t . The learner

aims to optimize the total expected regret R T , defined as

We make the following realizability assumption:

Assumption 2.1 (Function Realizability) . Assume that f ⋆ in Eq. (1) satisfies f ⋆ ∈ F .

We finish this section with the definition of eluder dimension:

Definition 2.1 (Eluder Dimension (Russo and Van Roy, 2014)) . For function class F defined on space Z , we define the eluder dimension of F at scale α ≥ 0 , denoted by d elu ( F ; α ) , as the length of the longest sequence of tuples ( z 1 , f 1 , f ′ 1 ) , ..., ( z m , f m , f ′ m ) ∈ Z × F × F such that there exists α 0 ≥ α making the following hold for all i = 1 , .., m :

Throughout the paper, if α is not specified, we take the default value α = 1 / T 2 . We also omit the dependence on F when it is clear from the context.

3 Results Overview

We describe our three settings in the following three subsections and summarize the results in Table 1. In the following, F denotes the function class that only contains reward mean information, and M the model class that contains reward distribution information.

3.1 Weak Adversary Case with Variance Revealing (Section 4)

First, we consider the case where the adversary is weak . This means that the variance σ t only depends on the history up to round t -1 , which aligns with the standard 'adaptive adversary' assumption. For this case, we show that for any A and d , one can find an instance of contextual problem problem with |F| ≤ √ AT , such that the regret is at least

For upper bounds in the weak adversary case, we focus on the regime A ≤ d elu ≤ √ AT , where the lower bound can be written as Ω( √ A Λ + d elu ) . 2 While our ultimate goal is to obtain a nearly matching upper bound ˜ O ( √ A Λlog |F| + d elu log |F| ) , we have not achieved it yet in full generality. In Section 4, we provide an algorithm which operates under the assumption that the variance σ t is revealed to the learner at the beginning of round t , and show that it achieves the matching upper bound. An initial attempt to remove this assumption is discussed in Section 7, where we show that when σ t ∈ { 0 , 1 } for all t and σ t is revealed to the learner at the end of round t , the matching upper bound can also be achieved.

3.2 Strong Adversary Case (Section 5)

Next, we consider the case where the adversary is strong . This means the adversary can decide σ t after seeing the action a t chosen by the learner at round t . In this case, the lower bound becomes

The difference with Eq. (2) is that the scaling in front of Λ changes from min { A,d elu } to d elu. This shows the even more crucial role of eluder dimension in the strong adversary case. For this setting, we give an upper bound of ˜ O ( d elu √ Λlog |F| + d elu log |F| ) , which is off from the lower bound by a √ d elu factor along with other logarithmic factors.

Notice : For simplicity, this table only considers the case A ≤ d elu, and all lower bounds should be further taken a minimum with √ AT .

3.3 Learning with a Model Class (Section 6)

In Section 6, we assume that the function class provides information on the distribution of the reward rather than just the mean. Such a function class is usually called a model class . More precisely, the learner is provided with a model class M that includes the true model M ⋆ ∈ M so that M ⋆ ( x, a ) specifies reward distribution for the context-action pair ( x, a ) . Compared to the scenario studied in Section 4, here we do not require variance to be revealed to the learner. This becomes possible because with a model class, the learner can now obtain variance information (though not precise) through the context. Under the assumption that the noise is Gaussian, we provide an ˜ O ( √ A Λ ∞ log |M| + d elu log |M| ) upper bound where Λ ∞ = ∑ t max a σ M ⋆ ( x t , a ) 2 , and a matching lower bound, where σ M ( x, a ) is the reward variance for the context-action pair ( x, a ) predicted by M ∈ M .

The work of Wang et al. (2024) also studied second-order contextual bandits with a model class. They use Λ ◦ = ∑ t σ M ⋆ ( x t , a t ) 2 , i.e., the reward variance of the chosen actions, as the variance measure. They obtain ˜ O ( √ d elu Λ ◦ log |M| + d elu log |M| ) upper bound. We show a nearly matching lower bound Ω(min { √ d elu Λ ◦ + d elu , √ AT } ) , similar to the lower bound for the strong adversary case studied in Section 5. The lower bound indicates that, in general, the bound of Wang et al. (2024) cannot be improved even when A < d elu .

3.4 Comparison with Pacchiano (2024)

The work of Pacchiano (2024) also studied variance-dependent bounds for realizable contextual bandits. They also consider two settings, which can be mapped to those in our Section 4 and Section 5, respectively. For the weak adversary setting with revealed σ t (Section 4), they give an upper bound of ˜ O ( √ d elu Λlog |F| + d elu log |F| ) , 3 which is incomparable to our ˜ O ( √ A Λlog |F| + d elu log |F| ) . However, a full picture of this setting can be obtained by combining their upper bound and our upper bound and the lower bound in Eq. (2). For the strong adversary setting (Section 5), they derive exactly the same upper bound as in our Theorem 5.2. Our work makes additional contribution in the lower bounds and the extension to the distributional setting (Section 6).

4 Weak Adversary Case with Variance Revealing

In this section, we consider cases where the variance σ 2 t at round t is given to the learner at the beginning of round t together with the context x t .

4.1 Lower Bound

The regret lower bound is shown with identical and known variance. The construction is similar to those in Wei et al. (2020). Concretely, we have the following theorem.

Theorem 4.1 (Main lower bound) . For any integer d, A ≥ 2 , any positive real number σ ∈ [0 , 1] , and time T > 0 , there exists a context space X and a contextual bandit problem F ⊂ ( X × A → R ) with eluder dimension d elu (0) ≤ d , action set A with |A| ≤ A , and variance σ t ≤ σ for all t ∈ [ T ] such that any algorithm will suffer a regret at least Ω( √ σ 2 min { A,d } T +min { d, √ AT } ) .

Proof sketch. The full proof is deferred to Appendix C. The two parts in the lower bound came from the following two different hardness: (1) The first part of the lower bound with Ω( √ σ 2 min { A,d } T ) is a natural lower bound with variance σ due to estimation of the mean values. (2) For the second part, we consider the following function class. In this function class, there is a 'good' action that serves as the default choice with a reward of 1/2 for all contexts. For each of the other A -1 'bad' actions, for each context, there is one function that obtains a reward of 1 but obtains 0 for all the other contexts. When d < √ AT , this function class forces the learner to guess for each context which action to choose. So even if the reward is deterministic, i.e., variance σ = 0 , any learner would have to suffer a regret scaling with the number of contexts times the number of actions, which in total coincide with the eluder dimension. When d ≥ √ AT , the learner can simply commit to the 'good' action and suffer √ AT but no better than this.

This lower bound is rather surprising for the following consequences: (1) The most significant implication from this lower bound is that improving the minimax regret bound with the knowledge of the variance is only possible if d < √ AT . (2) Even when d < √ AT , any learner would have to pay for the eluder dimension as a lower-order term. These are non-trivial because the second-order bounds are usually obtained from changing Hoeffding concentration to Bernstein concentration which usually only scales the regret bounds by σ . This lower bound shows that the second-order contextual bandit is not one of the usual cases. In the next section, we will match this lower bound from the upper bound side by combining several algorithmic techniques.

4.2 Upper Bound with Known and Fixed Variance

Motivated by the lower bound in Theorem 4.1, we wonder whether there is an algorithm which can achieve a matching upper bound of ˜ O ( √ σ 2 min { A,d } T + min { d, √ AT } ) , if the learner is provided with information of variance at the beginning of each round. In this subsection, we answer this question affirmly. To begin with, we consider the case when all the variance are identical, i.e. σ 1 = σ 2 = · · · = σ T = σ , and σ is given to the learner. Later (Section 4.3), we will discuss how to generalize this result to the case with nonidentical variances across different rounds.

We assume that r t = f ⋆ ( x t , a t ) + ϵ t ∈ [0 , 1] and Var ( ϵ t ) ≤ σ 2 t for every 1 ≤ t ≤ T . Our results can be easily extended to subgaussian random noise (at the cost of a log T factor) since for such variables, with probability at least 1 -δ , | ϵ t | ≤ C √ log(1 /δ ) for a constant C .

4.2.1 Algorithm and Analysis for Identical Variance

We first consider the case with identical variance, i.e. σ 2 t = σ 2 for all t ∈ [ T ] . We propose Algorithm 1, and show that it has regret upper bound ˜ O ( √ σ 2 AT log |F| + d elu log |F| ) . The algorithm is adapted from SquareCB of Foster and Rakhlin (2020), but additionally maintains a confidence function set, and has mechanisms to learn faster when the functions in the confidence set has larger disagreement. It has the following elements:

Restricting action set (Line 4) At the beginning of round t (Line 4), the learner restricts the action set to A t , which only includes those actions that is the best action of some functions in the function class F t . If we assume that f ⋆ is always in the function class F t , by doing this we remove the unnecessary possibility of choosing actions that can never be the best action.

Checking disagreement (Line 5-Line 7) The next step of the algorithm is to check whether there is an action in A t such that two functions in the function class have large value differences (Line 6). We called such actions 'discriminative actions'. Roughly speaking, we are seeking an action a ∈ A t such that

Algorithm 1 VarCB (Variance-aware Contextual Bandits)

Input:

δ ∈ [0 , 1] , σ ∈ [ 1 / AT , 1] .

1: Let L = log |F| T 2 δσ 2 and ∆ = σ 2 11 √ L and F 1 = F .

2: for t = 1 , 2 . . . , do

3: Receive context x t .

4: Define A t = { a ∈ A : ∃ f ∈ F t , a ∈ arg max a ∈A f ( x t , a ) }

5: Define

6: if max a ∈A t g t ( a ) ≥ ∆ then

Choose action a t = arg max g t ( a ) and receive r t .

8: else

7: a ∈A t

9:

Call online regression oracle (Algorithm 4) with input ( F t , x t ) and obtain f t .

10: Let b t = arg max a ∈A t f t ( x t , a ) (pick an arbitrary maximizer if there are multiple).

11: Draw a t ∼ p t and receive r t , where

̸

12: Define w t = min { 1 σ 2 , 1 g t ( a t ) √ L } and update the confidence set:

If such an action exists, then the learner chooses this action at round t . By selecting such an action that can discriminate disagreed functions, the function set F t can more quickly shrink. To prevent this action to incur overly large regret, it is important to perform Step 1 (Restricting action set). The regret incurred in rounds choosing discriminative actions is of order ˜ O ( d elu log |F| ) .

3. Inverse gap weighting (Line 8-Line 11) At round t , if there is no discriminative action, then the learner performs inverse gap weighting as in the SquareCB algorithm (Foster and Rakhlin (2020)). Inverse gap weighting requires the learner to have access to an online regression oracle that generates online estimations f t and ensures that the estimation error ∑ t ( f t ( x t , a t ) -f ⋆ ( x t , a t )) 2 is small. In the original SquareCB, the requirement for the online regression oracle is

which only allows for a √ AT log |F| regret bound that does not meet our goal. To improve this, we design an online regression oracle that ensures

where T IGW is the set of rounds that we run inverse gap weighting (i.e., entering the else case in Line 8), and ˜ ∆ is an upper bound for max a ∈A t max f,f ′ ∈F t | f ( x t , a ) -f ′ ( x t , a ) | , i.e., the maximum disagreement among the function set F t for the context x t . Thanks to Step 2, we only run inverse gap weighting when ˜ ∆ ≲ ∆ ≈ σ 2 . Thus, with the refined R sq guarantee and standard squareCB arguments, we can get a regret bound of order σ √ AT log |F| for the rounds in T IGW .

The way to achieve '(our condition)' is an interesting part of our algorithm. A standard way to ensure F&R's condition is by aggregating over the function set through exponential weights. Exponential weights ensures R sq = O (log |F| /η ) as long as the functions to be aggregated are η -mixable. Thus, in order to show R sq = O ( σ 2 log |F| ) , we need to argue η = Ω(1 /σ 2 ) . However, because the potential range of r t is [0 , 1] even though the variance σ 2 and and the disagreement ∆ are both much smaller than 1 , the best mixability coefficient η we can show for squared loss is still Θ(1) .

To address this, we resort to the use of the Prod algorithm (Cesa-Bianchi and Lugosi, 2006) with a properly chosen surrogate loss to perform aggregation. This algorithm has a different second-order approximation for the loss compared to the exponential weight algorithm, which is crucial in obtaining the desired bound. The regret analysis is also no longer through mixability. Our online regression oracle is provided in Algorithm 4 in Appendix D. We remark without giving details that in the linear case, such a guarantee can also be obtained through Online Newton Step (Hazan et al., 2007).

4. Updating function set (Line 12) After finishing selecting the action a t for round t , the learner updates the confidence function set F t to prepare for the next round. The construction of the confidence set utilizes the idea of weighted regression that has been widely used in previous varianceaware or corruption-robust contextual bandit or RL algorithms (He et al., 2022; Zhao et al., 2023; Ye et al., 2023; Agarwal et al., 2023). This has the effect of controlling the relative importance of different samples and is crucial in controlling the regret incurred in Step 2.

By putting these building blocks together, we arrive at Algorithm 1. The regret of Algorithm 1 is described in Theorem 4.2, whose proof is deferred to Appendix D.

Theorem 4.2. Algorithm 1 ensures with probability at least 1 -δ ,

Comparison with AdaCB of Foster et al. (2020) Our VarCB (Algorithm 1) shares some similarities with the AdaCB algorithm from Foster et al. (2020), which aims to achieve a ˜ O ( d log |F| GAP ) regret bound. Here, d is a disagreement coefficient of F , which takes the same role as our d elu, and GAP represents the minimal reward gap between the best and second-best decisions. Specifically, both algorithms include a step to remove irrelevant actions (our Step 1). The action selection rule of AdaCB also depends on the amount of disagreement over the function class, which is superficially related to the if-else separation in VarCB . However, we find that the case separations in the two algorithms do not have a clear correspondence to each other, possibly due to the different objectives of the two algorithms. Also, the two algorithms operate under quite different settings: AdaCB works in the setting where the contexts are i.i.d., while VarCB allows for adversarial contexts. On the other hand, AdaCB is parameter-free, but VarCB requires the information of σ . Developing a more unified version for these two better-than-minimax algorithms is an interesting future direction.

4.3 Algorithm and Analysis for Heteroscedastic Noise

Next, we will discuss how to generalize our algorithm to heteroscedastic case, i.e. when the noise of different rounds are different. Based on the values of the variance, we classify each round into the following (log( AT ) + 1) sets: if σ t ∈ [0 , 1 AT ] , we classify t into T 0 , and for σ t ∈ ( 2 i -1 AT , 2 i AT ] , we classify t into T i for 2 ≤ i ≤ log( AT ) , i.e., if σ t falls into the i -th intervals in the following,

̸

we classify t into T i . For each set T i , we maintain an algorithm A i of Algorithm 1 in parallel. At the beginning at round t , when observing that t ∈ T i , only A i is updated, while A j remains the same for j = i . According to Theorem 4.2, we have for any 0 ≤ i ≤ log T ,

we can bound the total regret by

Algorithm 2 Algorithm for Heteroscedastic Noise

Input:

1: Initialize instances A i of VarCB (Algorithm 1) with σ = 2 i AT for 0 ≤ i ≤ log( AT ) .

2: for t = 1 : T do

3: Receive σ t ∈ [0 , 1] , and suppose that σ t ∈ Σ i , where Σ i is defined in Eq. (7).

4: Receive context x t , and inject x t into algorithm A i . to obtain action a t .

5: Play action a t and update algorithm A i .

The formal algorithm for heteroscedastic cases is given in Algorithm 2, and we have the following corollary on the second-order regret bound of Algorithm 2.

Corollary 4.1. The output a t of Algorithm 2 in rount t satisfies that with probability at least 1 -δ ,

The proof of Corollary 4.1 is given in Section E.

5 Strong Adversary Case

In this section, we consider the case where the adversary decides the variance σ t after seeing the action a t chosen by the learner. We provide regret lower and upper bounds matching up to a factor of √ d elu and other logarithmic factors. More importantly, the minimax regret bounds differ with the weak adversary case (Section 4) as discussed in Section 3.2, demonstrating the even more crucial role of eluder dimension in this case.

Regret lower bound In this strong adversary case, we first show that the adversary's power is enhanced in terms of the achievable minimax regret bounds. Concretely, we have the following theorem.

Theorem 5.1. For any integer d, A, T ≥ 2 and any positive real number Λ ∈ [0 , T ] , there exists a context space X , a contextual bandit problem F ⊂ ( X × A → R ) with eluder dimension d elu ( F , 0) = d and action set A = [ A ] and an adversarial sequence of variances σ 2 1 , . . . , σ 2 T with ∑ T t =1 σ 2 t ≤ Λ such that any algorithm will suffer a regret at least Ω(min { √ d Λ+ d, √ AT } ) .

The above theorem shows that the regret is at least Ω(min { √ d Λ + d, √ AT } ) where d = d elu ( F ) even with log |F| = O (log T ) . Recall that the bound in the weak adversary case (Section 4) can be written as Ω(min { √ min { A,d } Λ+ d, √ AT } ) . The power of the strong adversary is exactly the higher complexity d in the Λ term compared to min { A,d } in the weak adversary case. Now, we proceed to provide a matching upper bound up to a factor of √ d .

Regret upper bound For the strong adversary case, we adopt an optimism-based approach. In particular, we generalize the SAVE algorithm by Zhao et al. (2023), which achieves the tight ˜ O ( d √ Λ+ d ) bound for linear contextual bandits. We call the algorithm VarUCB and display it in Algorithm 5 of Appendix F. The algorithm combines the idea of weighted regression and multi-layer structure of SupLinUCB (Chu et al. (2011)) and refined variance-aware confidence set. Since this algorithm is a rather direct extension of Zhao et al. (2023)'s algorithm from the linear case to the non-linear case, we omit the detailed discussion on it and refer the readers to Zhao et al. (2023). Notice that for this algorithm, we do not need σ t to be revealed to the learner as in Section 4. In fact, we do not even need to know Λ . We have the following theorem for its regret guarantee.

Theorem 5.2. When facing the strong adversary, Algorithm 5 guarantees a regret bound of ˜ O ( d elu √ Λlog |F| + d elu log |F| ) with probability at least 1 -δ , where ˜ O ( · ) hides log( T/δ ) factors.

The proof is provided in Appendix F. Notice that when specializing Theorem 5.2 to the linear setting, the bound becomes ˜ O ( √ d 3 Λ+ d 2 ) since log |F| = Θ( d ) , which does not recover the bound of Zhao et al. (2023). Indeed, our analysis deviates from that of Zhao et al. (2023) due to the generality of non-linear function approximation. It is an interesting future direction to see whether our bound can be improved. We mention in passing that the work by Wang et al. (2024) obtained

Algorithm 3 DistVarCB (Distributional Variance-aware Contextual Bandits)

√ d Λlog |M| + d log |M| upper bound where d = d elu ( M ) . However, the algorithm relies on having access to a model class. We study such a setting in our next section.

6 Learning with a Model Class

Distributional setup In this section, we consider the case where the learner is given a model class M⊂ (( X × A ) → ∆( R )) where each model M ∈ M maps any context-action pair to a gaussian distribution, i.e., for any x, a ∈ X × A ,

where f M ( x, a ) and σ M ( x, a ) are the mean and variance of the distribution M ( x, a ) . We assume that all the expected rewards and variances are bounded by [0 , 1] . Recall, at round t , the reward is given by r t = f ⋆ ( x t , a t ) + ϵ t . We further assume throughout this section that ϵ t is Gaussian with variance σ ⋆ ( x t , a t ) (since Gaussian is unbounded, we drop the assumption r t ∈ [0 , 1] that we made in Section 2). Thus, the distribution of r t follows a true model M ⋆ where M ⋆ ( x, a ) = N ( f ⋆ ( x, a ) , σ ⋆ ( x, a )) .

Assumption 6.1 (Model Realizability) . Assume M ⋆ ∈ M .

For this setup, it is useful to consider the Hellinger counterpart of the eluder dimension.

Definition 6.1 (Hellinger Eluder Dimension) . For the model class M defined on the space Z (that is M⊂ ( Z → ∆( R ) ), we define the Hellinger eluder dimension of M at scale α ≥ 0 as d H elu ( α ) be the length of the longest sequence of tuples ( z 1 , M 1 , M ′ 1 ) , ..., ( z m , M m , M ′ m ) and α 0 ≥ α such that for all i = 1 , .., m , functions M i , M ′ i ∈ M ,

Algorithm Similar to Algorithm 1, we present Algorithm 3 tailored for the distributional case. At each round t , upon receiving the context x t , the algorithm first checks if there exists an action a such that two models within the localized model class M t exhibit a significant divergence on the context-action pair x t , a measured by the squared Hellinger distance (Line 4). If such a difference is detected, the learner selects the action associated with the greatest divergence (Line 5). Conversely, if no action causes substantial divergence between models, the learner runs a variant of SquareCB (Foster and Rakhlin, 2020), employing adaptive variances to ensure low regret (Line 7). The major differences between Algorithm 1 and Algorithm 3 is that the latter measures the 'disagreement' in terms of the squared Hellinger distance.

Regret upper bound We obtain the following distributional version regret bound for Algorithm 3.

Theorem 6.1. For d = d H elu ( 1 / √ T ) , the output a t of Algorithm 3 satisfies with probability at least 1 -δ ,

where σ 2 M ⋆ ( x t ) = max a ∈ A σ 2 M ⋆ ( x t , a ) .

A similar upper bound for a more general distributional case is obtained by Wang et al. (2024) in the form of ˜ O ( » d ∑ T t =1 σ 2 M ⋆ ( x t , a t ) · log |M| + d log |M| ) . In the leading term, our bound replaced the dependence of d by the number of actions A which is significantly smaller than A in many cases of interest (e.g. linear, generalized linear). However, as a tradeoff, our bound also suffers a larger cumulative variance term. This tradeoff is necessary as we show in the following lower bound results that both our upper bound and their upper bound are optimal, i.e., matching lower bounds exist. Thus our result is at one end of the Pareto frontier.

Regret lower bounds We present the matching lower bound for our result as follows, which is essentially a rewrite of Theorem 4.1.

Theorem 6.2. For any integer d, A, T ≥ 2 , any positive real number σ ∈ [0 , 1] , there exists a context space X and a contextual bandit gaussian model class M⊂ ( X×A→ ∆( R )) with Hellinger eluder dimension d H elu (0) ≤ d , action set A = [ A ] , and variances σ M ( x, a ) ≤ σ for all M ∈ M , x, a ∈ X × A such that any algorithm will suffer a regret at least Ω( √ σ 2 min { A,d } T +min { d, √ AT } ) .

Now we present the matching lower bound for the upper bound from Wang et al. (2024).

Theorem 6.3. For any integer d, A, T ≥ 2 and any positive real number Λ ∈ [0 , T ] , there exists a context space X , a contextual bandit gaussian model class M⊂ ( X × A → ∆( R )) with Hellinger eluder dimension d H elu (0) ≤ d and action set A = [ A ] and the variances ∑ T t =1 σ M ⋆ ( x t , a t ) 2 ≤ Λ such that any algorithm will suffer a regret at least Ω(min { √ d Λ+ d, √ AT } ) .

The lower bound obtained by Theorem 6.3 is an adaptation from Theorem 5.1 that crucially relies on the fact that the adversary can choose the variance according to the action a t .

7 Open Questions

Removing the revealing σ t assumption in the weak adversary setting The assumption we made in Section 4 that the variance is revealed at the beginning of each round is rather restrictive, and ideally we would like to remove such an assumption. As a first step, we wonder whether the same regret bound ˜ O ( √ A Λ+ d elu ) is achievable if the variance σ t is revealed at the end of round t . We answer this question affirmatively for the special case where σ t ∈ { 0 , 1 } . More details can be found in Appendix H. How to extend this result to general values of σ t is an interesting open question. Handling the case where σ t is never revealed is even more challenging but is the ultimate goal.

Removing the Gaussian noise assumption in the distributional setting Our Theorem 6.1 heavily relies on the assumption that the noise is Gaussian. We wonder whether such assumption can be relaxed or completely lifted. For example, can we obtain the same bound if the noise at round t is just σ M ⋆ ( x t , a t ) -sub-Gaussian? What if it is just a bounded noise with variance σ 2 M ⋆ ( x t , a t ) ?

References

Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics , pages 19-26. PMLR, 2012.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646, 2014.

Alekh Agarwal, Yujia Jin, and Tong Zhang. Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. In The Thirty Sixth Annual Conference on Learning Theory , pages 987-1063. PMLR, 2023.

Zeyuan Allen-Zhu, Sébastien Bubeck, and Yuanzhi Li. Make the minority great again: First-order regret bound for contextual bandits. In International Conference on Machine Learning , pages 186-194. PMLR, 2018.

Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Nicolo Cesa-Bianchi, Yishay Mansour, and Gilles Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning , 66:321-352, 2007.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Yan Dai, Ruosong Wang, and Simon Shaolei Du. Variance-aware sparse linear bandits. In The Eleventh International Conference on Learning Representations , 2023.

Chris Dann, Chen-Yu Wei, and Julian Zimmert. A blackbox approach to best of both worlds in bandits and beyond. In The Thirty Sixth Annual Conference on Learning Theory , pages 5503-5570. PMLR, 2023.

Qiwei Di, Tao Jin, Yue Wu, Heyang Zhao, Farzad Farnoud, and Quanquan Gu. Variance-aware regret bounds for stochastic contextual dueling bandits. In The Twelfth International Conference on Learning Representations , 2024.

Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , pages 169-178, 2011.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan J Foster and Akshay Krishnamurthy. Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. Advances in Neural Information Processing Systems , 34: 18907-18919, 2021.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Dylan J Foster, Yanjun Han, Jian Qian, and Alexander Rakhlin. Online estimation via offline estimation: An information-theoretic framework. Advances in Neural Information Processing Systems , 2024.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Elad Hazan and Satyen Kale. Better algorithms for benign bandits. Journal of Machine Learning Research , 12(4), 2011.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192, 2007.

Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in neural information processing systems , 35:34614-34625, 2022.

Shinji Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Conference on Learning Theory , pages 2552-2583. PMLR, 2021.

Shinji Ito and Kei Takemura. Best-of-three-worlds linear bandit algorithm with variance-adaptive regret bounds. In The Thirty Sixth Annual Conference on Learning Theory , pages 2653-2677. PMLR, 2023.

Yeoneung Kim, Insoon Yang, and Kwang-Sung Jun. Improved regret analysis for variance-adaptive linear bandits and horizon-free linear mixture mdps. Advances in Neural Information Processing Systems , 35:1060-1072, 2022.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Aldo Pacchiano. Second order bounds for contextual bandits with function approximation. arXiv preprint arXiv:2409.16197 , 2024.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 26, 2013.

Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 47(3):1904-1931, 2022.

Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance-aware regret bounds for undiscounted reinforcement learning in mdps. In Algorithmic Learning Theory , pages 770-805. PMLR, 2018.

Vladimir G Vovk. A game of prediction with expert advice. In Proceedings of the eighth annual conference on Computational learning theory , pages 51-60, 1995.

Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in Neural Information Processing Systems , 36, 2023.

Kaiwen Wang, Owen Oertell, Alekh Agarwal, Nathan Kallus, and Wen Sun. More benefits of being distributional: Second-order bounds for reinforcement learning. In International Conference on Machine Learning , 2024.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory , pages 1263-1291. PMLR, 2018.

Chen-Yu Wei, Haipeng Luo, and Alekh Agarwal. Taking a hint: How to leverage loss predictors in contextual bandits? In Conference on Learning Theory , pages 3583-3634. PMLR, 2020.

Yunbei Xu and Assaf Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876 , 2020.

Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning , pages 39834-39863. PMLR, 2023.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Zihan Zhang, Jiaqi Yang, Xiangyang Ji, and Simon S Du. Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems , 34:4342-4355, 2021.

Zihan Zhang, Jason D Lee, Yuxin Chen, and Simon S Du. Horizon-free regret for linear markov decision processes. International Conference on Learning Representations , 2024.

Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In The Thirty Sixth Annual Conference on Learning Theory , pages 4977-5020. PMLR, 2023.

Runlong Zhou, Ruosong Wang, and Simon Shaolei Du. Horizon-free and variance-dependent reinforcement learning for latent markov decision processes. In International Conference on Machine Learning , pages 42698-42723. PMLR, 2023.

Julian Zimmert and Tor Lattimore. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory , pages 3285-3312. PMLR, 2022.

Appendices

A Related Work

In this section, we review the literature in bandits/RL that obtains second-order regret bounds or other data-/instance-dependent regret bounds.

Tabular/linear bandits and MDPs Second-order regret bounds for bandits have been extensively studied in non-contextual (fixed action set) settings, such as stochastic multi-armed bandits (Audibert et al., 2009), adversarial multi-armed bandits (Wei and Luo, 2018; Ito, 2021), and adversarial linear bandits (Hazan and Kale, 2011; Ito and Takemura, 2023). Key techniques in this line of work include replacing Hoeffding-style concentration bounds with Bernstein-style ones in value estimation, and using optimistic mirror descent (Rakhlin and Sridharan, 2013) to achieve variance reduction. Extending beyond non-contextual settings, a series of recent works (Zhang et al., 2021; Kim et al., 2022; Zhao et al., 2023) have focused on obtaining tight variance-dependent bounds for linear contextual bandits. The techniques developed for variance-dependent regret bounds in bandits have been extended to MDPs, leading to either variance-dependent or horizon-free bounds (Talebi and Maillard, 2018; Zanette and Brunskill, 2019; Zhang et al., 2021; Kim et al., 2022; Zhao et al., 2023; Zhou et al., 2023; Zhang et al., 2024). Additionally, other settings such as dueling bandits (Di et al., 2024) and sparse linear bandits (Dai et al., 2023) have been explored.

General contextual bandits For contextual bandits with general policy or function classes, secondorder bounds have been explored in both agnostic settings (Wei et al., 2020) and realizable settings (Wang et al., 2024). Wei et al. (2020) focused on the case where the number of actions is small. They demonstrated that, unlike in multi-armed or linear bandits, the tight second-order bound for general contextual bandits is more complex than simply replacing the zeroth-order term T in the minimax bound with the second-order measure, which aligns with our findings. For instance, even if the second-order error E is O (1) , they showed that the regret could still grow with Ω( T 1 / 4 ) . On the other hand, Wang et al. (2024) focused on the case where the model class that provides distributional information for the reward has small eluder dimension. In contrast to the small action regime, their bounds smoothly scale with the second-order measure.

Other instance-dependent bounds in general contextual bandits Beyond second-order bounds, other works have focused on other data-dependent or instance-dependent bounds. For example, Allen-Zhu et al. (2018), Foster and Krishnamurthy (2021), and Wang et al. (2023) studied first-order (small-loss) bounds for general contextual bandits and MDPs, in which the goal is to make the regret scale with √ L ⋆ , where L ⋆ is the cumulative loss of the best policy. These bounds are generally achievable with specialized algorithms. For general contextual bandits, gap-dependent bounds that exhibit logarithmic dependence on T have been derived by Foster et al. (2020) and Dann et al. (2023) for realizable and agnostic settings, respectively. Notably, the algorithm of Foster et al. (2020) has some similarity with our main algorithm, which we discuss further in the main text.

B Technical Tools

Lemma B.1. For any 0 ≤ σ ≤ 1 / 2 and 0 ≤ ε ≤ σ/ 2 , define

Denote by h σ , h + σ,ϵ , h -σ,ϵ the means of the three distributions respectively. We have

Let V σ , V + σ,ϵ , V -σ,ϵ be the variance of the three distributions respectively. We have

Furthermore,

Proof.

In particular,

Lemma B.2. For any two gaussian distributions N ( µ 1 , σ 1 ) and N ( µ 2 , σ 2 ) , if | log σ 1 -log σ 2 | > 3 , then

Proof of Lemma B.2. Without loss of generality, assume σ 1 > e 3 σ 2 > 8 σ 2 . The Hellinger divergence between two gaussian distributions writes

Lemma B.3 (Lemma 4.2 of Wang et al. (2024)) . For any distribution P and Q on any space X such that D 2 H ( P , Q ) ≤ 1 2 , we have for any function h : X → R ,

Lemma B.4 (Lemma A.11 of Foster et al. (2021)) . For any distribution P and Q on any space X , we have for any function h : X → [0 , R ] ,

Lemma B.5 (Strengthened Freedman's inequality (Theorem 9 of Zimmert and Lattimore (2022))) . Let X 1 , X 2 , . . . , X T be a martingale difference sequence with a filtration F 1 ⊆ F 2 ⊆ · · · such that E [ X t | F t ] = 0 and E [ | X t | | F t ] < ∞ almost surely. Then with probability at least 1 -δ ,

where V T = ∑ T t =1 E [ X 2 t | F t ] and U T = max { 1 , max t ∈ [ T ] | X t |} .

C Omitted Proofs in Section 4.1

Lemma C.1 (Theorem 15.2 of Lattimore and Szepesvári (2020)) . For any integer A ≥ 2 , any positive real number σ ∈ [0 , 1] , and time T > 0 , there exists a context space X and a contextual bandit problem F ⊂ ( X × A → R ) with action set A = [ A ] , d elu ( F ; 0) = A , and variances σ t ≤ σ for all t ∈ [ T ] such that any algorithm will suffer a regret at least Ω( √ σ 2 AT ) .

Proof of Lemma C.1. Without loss of generality assume σ < 1 / 2 . Fix the algorithm. The first lower bound construction follows the standard MAB lower bound. Let the context space X = ∅ and the function class

F = { f i ( · ) = ( σ + ε ) ✶ ( · = i ) + σ ✶ ( · = 1) | i ∈ { 2 , 3 , . . . , A }} ∪ { f 1 ( · ) = ( σ -ε ) ✶ ( · = i ) + σ ✶ ( · = 1) } , where the gap ε will be specified later with the constraint ε ≤ 2 σ . Let P σ , P + σ,ϵ , and P -σ,ϵ be defined as in Lemma B.1. For any function f ∈ F , suppose the environment is such that if f ( i ) = σ, σ + ϵ, σ -ϵ , then the reward distribution is P σ , P + σ,ϵ , P -σ,ϵ respectively. Any environment and algorithm give rise to the distribution of history. Let P i denote the distribution generated by the environment following f i ∈ F together with the algorithm and expectations under P i will be denoted by E i . Let N T ( a ) = ∑ T t =1 ✶ ( a t = a ) where a t is the action taken at time step t for t ∈ [ T ] . Let i ⋆ = arg min j> 1 E 1 [ N T ( j )] .

Since ∑ A i =1 N T ( i ) = T , it holds that E 1 [ N T ( i ⋆ )] ≤ T/ ( A -1) . For the two environments induced by f 1 and f i ⋆ , we have

Then, by Bretagnolle-Huber inequality, we have

Then by the chain rule of KL divergence and Lemma B.1, we have

Thus we have

Then by choosing ε = √ ( A -1) σ 2 / 4 T , we have

Lemma C.2. For any integer N,A,T ≥ 2 , there exists a context space X and a deterministic ( σ t = 0 for all t ∈ [ T ] ) contextual bandit problem F ⊂ ( X × A → R ) with action set A = [ A ] and d elu ( F ; 0) = N ( A -1) such that any algorithm will suffer a regret at least Ω(min { T/N,AN } ) .

Proof of Lemma C.2. Consider the function class F = { f (0) } ∪ { f ( i,j ) } i ∈ [ N ] ,j ∈ [ A -1] with the space of contexts X = { x (1) , . . . , x ( N ) } and the set of actions A = [ A ] . For any i ∈ [ N ] , j ∈ [ A -1] , the function f ( i,j ) is defined as the following: For i ∈ [ N ] and j ∈ [ A -1] ,

̸

Meanwhile

The eluder dimension of this function class is N ( A -1) since f ( i,j ) is uniquely identified by its value on ( x ( i ) , j ) . We assume that x t is uniformly randomly chosen from X , and r t = f ⋆ ( x t , a t ) . That is, σ t = 0 for all t ∈ [ T ] .

Fix any algorithm. Denote by P 0 the probability distribution when f ⋆ = f (0) and E 0 the expectation under P 0 . For any i ∈ [ N ] , j ∈ [ A -1] , denote by P ( i,j ) the probability distribution when f ⋆ = f ( i,j ) and E ( i,j ) the expectation under P ( i,j ) . Let N T ( i, j ) = ∑ T t =1 ✶ [ x t = x ( i ) , a t = j ] . Then the adversary decides f ⋆ based on the following rule: if there exists i ∈ [ N ] , j ∈ [ A -1] such that

then let f ⋆ = f ( i,j ) . If no i, j satisfies this, let f ⋆ = f (0) . If f ⋆ = f (0) , then we have

On the other hand, if f ⋆ = f ( i,j ) , then we have

Then by Lemma D.2 of Foster et al. (2024), we have

Altogether, we can obtain

This in turn implies that

Proof of Theorem 4.1. If A > T and d > T , by Lemma C.2 with N = 1 , we have lower bound Ω( T ) . Below, we assume min { A,d } ≤ T .

In order to prove the lower bound, we only need to show that for any fixed d, A, σ, T such that min { A,d } ≤ T , there exists two classes where one has lower bound Ω( √ σ 2 min { A,d } T ) and the other has lower bound Ω(min { d, √ AT } ) .

If A ≤ d , then we invoke Lemma C.1 to obtain the lower bound of Ω( √ σ 2 AT ) . Else if d ≤ A , we can again invoke Lemma C.1 with the action set [ d ] and then expand the action set with dummy actions with all 0 rewards to obtain the lower bound of Ω( √ σ 2 dT ) . In all, we have shown that there is a lower bound of Ω( √ σ 2 min { A,d } T ) .

If d ≤ A (which implies d ≤ T since we assume min { A,d } ≤ T ), then we invoke Lemma C.2 with N = 1 with action set A be [ d ] plus A -d dummy actions. Then we get a lower bound of Ω(min { T, d } ) = Ω( d ) . Next, consider the case d ≥ A . If d ≤ √ AT , then we invoke Lemma C.2 with N = d/A and obtain a lower bound of Ω(min { AT/d,d } ) = Ω( d ) . Else we have d > √ AT (which implies T ≥ A since we assume min { A,d } ≤ T ). Then we consider the hard case from Lemma C.2 with N = √ T/A then the function class has Hellinger eluder dimension √ AT . Embedding this function class into a more complex model class with a larger Hellinger eluder dimension, we can obtain the lower bound of Ω( √ AT ) . In all, we have shown that there is a lower bound of Ω(min { d, √ AT } ) .

Consequently, we have shown a lower bound of Ω( √ σ 2 min { A,d } T +min { d, √ AT } ) .

Algorithm 4 Prod-based online regression oracle

Input: Parameter η . Contextual Bandit Oracle gives function class F t ⊂ F , action class A t and context x t at round t . Contextual Bandit algorithm which takes online regression oracle and returns an action.

1: Let q 1 ( f ) = 1 / |F 1 | for every function f ∈ F 1 .

2: for t = 1 , 2 , . . . do

3: Generate

4: Output { f t ( x t , a ) : a ∈ A t } , and feed to Contextual Bandit algorithm to receive a t ∈ A t .

5: Call Contextual Bandit oracle to get F t +1 and x t +1 .

6: Calculate

7: and

D Omitted Proofs in Section 4.2

D.1 Analysis of the Online Regression Oracle

Lemma D.1 (Cesa-Bianchi et al. (2007)) . Fix some positive parameter η > 0 . Suppose we have function sets F 1 ⊃ F 2 ⊃ · · · ⊃ F T , and functions ℓ t : F t → R satisfies that η | ℓ t ( f ) | ≤ 1 2 for any f ∈ F t . The prediction rule

ensures for any f ⋆ ∈ F T ,

Proof. Define w t ( f ) = ∏ t τ =1 (1 -ηℓ τ ( f )) and W t = ∑ f ∈F t w t ( f ) .

where in ( i ) we use the definition of q t , in ( ii ) we use the fact that ∑ f ∈F t q t ( f ) = 1 , and in ( iii ) we use the inequality log(1 -x ) ≤ -x for any x < 1 . Therefore,

where in ( i ) we use the fact that W 0 = |F| and W T ≥ w T ( f ⋆ ) for any f ⋆ ∈ F t , and in ( ii ) we use the fact that η | ℓ t ( f ⋆ ) | ≤ 1 2 and also the inequality log(1 -x ) ≥ -x -x 2 for any | x | ≤ 1 2 .

Lemma D.2. Suppose for any x ∈ X , a ∈ A , f ∈ F , we always have f ( x, a ) ∈ [0 , 1] , the reward r t ∈ [0 , 1] and for any f, f ′ ∈ F t , we always have

Then for the output f t according to Algorithm 4, we have with probability at least 1 -δ ,

Proof. We first notice that with our choice of η = 1 4( σ 2 + ˜ ∆) , for any f ∈ F t we have

where ˜ ℓ t is defined in Eq. (10). According to Lemma D.1, we have

By the definition of f t in Eq. (9) and ˜ ℓ t in Eq. (10), we have

Next, we observe that

which implies

We notice that

is a martingale difference sequence, and we further have | ˜ ℓ t ( f ⋆ ) 2 | ≤ 4 ˜ ∆ 2 and

Hence according to Freedman inequality (Freedman, 1975), we have with probability 1 -δ ,

where in the second inequality we use the fact that | f t ( x t , a t ) -f ⋆ ( x t , a t ) | ≤ ˜ ∆ , and in the last inequality we use the AM-GM inequality. Hence, with our choice of η = 1 4( σ 2 + ˜ ∆) , we have

where in the last inequality we use the fact that ˜ ∆ ≤ 1 .

Lemma D.3. Suppose for any x ∈ X , a ∈ A , f ∈ F , we always have f ( x, a ) ∈ [0 , 1] , the reward r t ∈ [0 , 1] and for any f, f ′ ∈ F t , we always have

Then for the output f t according to Algorithm 4, we have with probability at least 1 -δ ,

Proof. We use F t denote the filtration constructed by F t = σ ( x 1: t , a 1: t , r 1: t ) . And we let

Then we have | M t | ≤ 2 ˜ ∆ . According to (Foster and Rakhlin, 2020, Lemma 1, Lemma 4), we have with probability at least 1 -δ/ 2 ,

Next, according to Lemma D.2, we have with probability at least 1 -δ/ 2 ,

Hence we obtain that with probability at least 1 -δ ,

D.2 Proof of Theorem 4.2

For simplicity, we define the following sets based on Algorithm 1:

We first show that with high probability, the true model f ⋆ ∈ F t , where F t is defined in Eq. (5) in Algorithm 1.

Lemma D.4. When the function class F t iteratively defined in Eq. (5) , with probability at least 1 -δ , we have f ⋆ ∈ F t for any 1 ≤ t ≤ T .

Proof. We will prove the result by induction on t . For t = 1 , since F 1 = F and according to Assumption 2.1 we have f ⋆ ∈ F 1 . Next, we will assume that f ⋆ ∈ F t -1 and attempt to prove f ⋆ ∈ F t . Since r τ = f ⋆ ( x τ , a τ ) + ϵ t for any 1 ≤ τ ≤ t -1 , we have

where the last inequality uses the definition that f t is the minimizer of ∑ t -1 τ =1 w τ ( f ( x τ , a τ ) -r τ ) 2 in (6).

We notice that w τ ( f t ( x τ , a τ ) -f ⋆ ( x τ , a τ )) ϵ τ is a martingale difference sequence, and since | w τ | ≤ 1 σ 2 ,

According to the Strengthened Freeman's Inequality (Lemma B.5), and noticing that E τ -1 [ ϵ 2 τ ] = σ 2 for any 1 ≤ τ ≤ t -1 , with probability at least 1 -δ T , for any f ∈ F ,

where in ( i ) we use the fact that w τ ≤ 1 / σ 2 for any τ according to the definition of w τ in Algorithm 1 and using the definition L ≜ |F| T 2 δσ 2 , in ( ii ) we use the fact that

according to the definition of w τ in Algorithm 1 since f τ , f ⋆ ∈ F τ by induction hypothesis, and finally in ( iii ) we use AM-GM inequality. And this proves the induction hypothesis of f ⋆ ∈ F t .

Therefore, we obtain that with probability at least 1 -δ for any 1 ≤ t ≤ T , f ⋆ ∈ F t .

The following lemma is a useful result of Eluder dimension.

Lemma D.5. For any λ ≥ 0 and α ∈ (0 , 1] , we have

Proof. This proof follows Lemma 5.1 of Ye et al. (2023). Create T bins, and call them B 1 , . . . , B T . Each bin is empty at the beginning. Below we will add elements in { 1 , 2 , . . . , T } to the bins.

Suppose that { 1 , 2 , . . . , t -1 } have been assigned to their bins. To assign t to a bin, we find the smallest n ∈ [ T ] such that ' ∃ ϵ ≥ α , ( x t , a t ) is ϵ -independent to the elements in B n with respect to F t .' We let n t to be the n we found, and put element i into bin B n i .

By the procedure above, we can conclude that for each t , ' ∀ ϵ ≥ α , ( x t , a t ) is ϵ -dependent on all of { B 1 , . . . , B n t -1 } with respect to F t .' Next, we show that this necessitates the following: for any f, f ′ ∈ F t ,

This is because if otherwise, then there exists f, f ′ ∈ F t and a bin B n ∈ { B 1 , . . . , B n t -1 } which ( x t , a t ) is ϵ -dependent on ∀ ϵ ≥ α , but

Choose

Clearly, | f ( x τ , a τ ) -f ′ ( x τ , a τ ) | ≤ ξ for all τ ∈ B n , but | f ( x t , a t ) -f ′ ( x t , a t ) | > α , contradicting that ( x t , a t ) is ϵ -dependent on B n ∀ ϵ ≥ 1 / √ T .

We obtain that

Let | B 1 | , . . . , | B T | be the size of the bins after all elements are added. By the eluder dimension definition (Definition 2.1), we know | B n | ≤ d elu ( α ) for all n . Therefore, we have

Lemma D.6. Suppose we have positive number B such that for any t ∈ [ T ] , w t ∈ [0 , B ] . Then for any λ ≥ 0 , we have

Proof. We define set T i = { t ∈ [ T ] : 2 i -1 /T ≤ w t ≤ 2 i /T } for any 1 ≤ i ≤ log( BT ) . and T 0 = { t ∈ [ T ] : w t ∈ [0 , 1 /T ] } . Then we have

Additionally, we notice that for any 1 ≤ i ≤ log( BT ) and t ∈ T i , we have

where in ( i ) we use the fact that for any τ ∈ T i , w τ ∈ [2 i -1 /T, 2 i /T ] , and in ( ii ) we use the fact that 2 i /T ≤ B . According to Lemma D.5, we have for any 1 ≤ i ≤ log( BT ) .

Next we notice that for those t ∈ T 0 , we have w t ≤ 1 /T , which implies that

Next, we present a lemma bounding the single-step regret in terms of g t defined in Eq. (4) in Algorithm 1.

Lemma D.7. Suppose f ⋆ ∈ F t for any 1 ≤ t ≤ T . For any 1 ≤ t ≤ T we have

Proof. In the following, we assume f ⋆ ∈ F t always holds. Let

Then we have

Here in ( i ) we use the fact that f ⋆ ∈ F t , and in ( ii ) we first use the definition of A t in Line 4 of Algorithm 1 that there exists f ′ ∈ F t such that a t = max a ∈A f ′ ( x t , a ) , which implies

In ( iii ) we use the definition of Eq. (5) that for any f, f ′ ∈ F t ,

Our next lemma provides an upper bound to the expectation of regret for rounds falling into T 1 .

Lemma D.8. Suppose f ⋆ ∈ F t for any 1 ≤ t ≤ T . We have

where we use ˜ O to hide constants and factors of log ( T σ ∆ ) .

Proof. We first define M = log 1 ∆ √ T and β = 21 √ L and recall the definition of g t in Eq. (4).

In the following, when with no ambiguity, we write g t = g t ( a t ) . Since for any f ∈ F , we have f ( x, a ) ∈ [0 , 1] , we have | g t | ≤ 1 . Also notice that for any t ∈ T 1 , we have g t ≥ ∆ according to Algorithm 1.

which implies

̸

We further define subsets T 11 h , T 12 h ⊂ [ T ] for every h = ∆ · 2 i with i = 0 , 1 , 2 , · · · , M -1 as T 11 h ≜ T 1 ∩ { t : h ≤ g t ≤ 2 h } ∩ ß t : w t = 1 σ 2 ™ , and T 12 h ≜ T 1 ∩ { t : h ≤ g t ≤ 2 h } ∩ ß t : w t = 1 σ 2 ™ .

Since ∆ ≤ g t ≤ 1 for t ∈ T 1 , we have

Next, we fix h , we analyze T 11 h and T 12 h separately. For those t ∈ T 11 h , we have:

where in ( i ) we use Lemma D.7 and the fact that | f ⋆ ( x, a ) -f ⋆ ( x, a ′ ) | ≤ 1 for any action a, a ′ ∈ A , in ( ii ) we use the definition of g t in Eq. (4) and the simplification g t := g t ( a t ) , and in ( iii ) we use the definition of T 11 h that for any t ∈ T 11 h we always have g t ≤ 2 h . Next, we bound the cardinality of each T 11 h :

where in ( i ) we use the definition of T 11 h , in ( ii ) we merely use the inequality that I { g ≥ h } ≤ 1 h min { g, h } for any g, h ≥ 0 , in ( iii ) we use the the fact that I { w t = 1 /σ 2 } ≤ √ w t σ since w t ≤ 1 /σ 2 always holds, in ( iv ) we use Cauchy-Schwarz inequality, in ( v ) we use the definition of g t in Eq. (4), and finanly in ( vi ) we use Lemma D.6 with λ = h 2 / σ 2 and also w t ∈ [0 , 1 / σ 2 ] . Therefore, we obtain that

Next, we will deal with those t in T 12 h . Similar to the proof for T 11 h , for any fixed h we have

And we can upper bound the cardinality of T 12 h as

̸

̸

̸

where in ( i ) we use the definition of g t in Eq. (4), in ( ii ) we use the definition of w t that when w t = 1 / σ 2 , we always have

and ( iii ) is according to Lemma D.6 and the fact that w t ∈ [0 , 1 / σ 2 ] .

Therefore, according to Eq. (15), we obtain that

Finally, recalling Eq. (14), if we sum the regret obtained above over h , we obtain

(using d elu ≜ d elu ( 1 / T 2 ) from Section 2 and the fact that d elu ( α ) is decreasing in α )

̸

Finally, we provide a lemma which upper bounds the expectation of regret for rounds falling into T 2 . Lemma D.9. Suppose that for any t ∈ [ T ] , we have f ⋆ ∈ F t . With ˜ ∆ = 11∆ √ L , we have with probability at least 1 -δ ,

Proof. We let a ⋆ t = arg max a f ⋆ ( x t , a ) . According to (Foster and Rakhlin, 2020, Lemma 3), we have

Summing this inequality up for all t ∈ T 2 , we obtain

We notice that for any t ∈ T 2 , we have

According to the definition of F t , for any f, f ′ ∈ F t ,

which implies that for any f, f ′ ∈ F t ,

Hence according to Lemma D.2, with probability at least 1 -δ/ 2 , we have

Further noticing that |T 2 | ≤ T , with choice γ = √ AT 4( ˜ ∆+ σ 2 ) log( |F| /δ ) , we have with probability at least 1 -δ/ 2 ,

Finally, since we always have 0 ≤ f ⋆ ( x t , a ⋆ t ) -f ⋆ ( x t , a ) ≤ 1 , according to Bernstein inequality we have with probability at least 1 -δ/ 2 ,

where in the second inequality we use the AM-GM inequality and in the last inequality we use the fact that 0 ≤ f ⋆ ( x t , a ⋆ t ) -f ⋆ ( x t , a ) ≤ 1 . Therefore, we obtain that with probability at least 1 -δ ,

Proof of Theorem 4.2. Lemma D.4 and Lemma D.8 gives that with probability at least 1 -δ/ 2 we have √

Additionally, according to Lemma D.9, we have with probability at least 1 -δ/ 2

Hence when ∆ = σ 2 / (11 √ L ) (so ˜ ∆ = σ 2 ), with probability at least 1 -δ , we have

Then noticing that L = log |F| T 2 2 and that σ ≥ 1

σ δ AT finishes the proof.

E Omitted Proofs in Section 4.3

In this section, we will prove Corollary 4.1.

Proof of Corollary 4.1. Based on the variance σ 2 t at round t , Algorithm 2 classify rounds into T 0 , · · · , T log T . We will bound the regret of rounds in T i separately.

According to Algorithm 2, for t ∈ T i with 0 ≤ i ≤ log( AT ) , we always have 1 / AT ≤ σ t ≤ 2 i / AT . Hence, according to Theorem 4.2, with probability at least 1 -δ/ log( AT ) , we have

We further observe that for 1 ≤ i ≤ log( AT ) , σ t ≥ 2 i -1 / AT , which implies that

And for i = 0 , we have

Therefore, with probability at least 1 -δ , we have

where in ( i ) we use Cauchy-Schwarz inequality.

Algorithm 5 VarUCB

1: Define : L = Θ(log( |F| T/δ )) , F 0 = F .

2: Define : K ≜ ⌈ log T ⌉ . Let Ψ 1 ,k = ∅ for k = 1 , 2 , . . . , K, K +1 .

3: for t = 1 , 2 , . . . do

4: Define confidence set:

where

5: Receive x t , and define A t = { a ∈ A : ∃ f ∈ F t : a ∈ arg max a ′ ∈A f ( x t , a ′ ) } .

6: Define

7: Let k t be the smallest k ∈ [ K ] such that max a ∈A t g t,k ( a ) ≥ 2 -k . (Let k t = K +1 if such k t does not exist)

8: Play a t = ® arg max a ∈A t g t,k t ( a ) if k ≤ K arg max a ∈A t max f ∈F t f ( x t , a ) otherwise , and receive r t .

9: Define g t = g t,k t ( a t ) . Let w t = 2 -k t /g t if k t ≤ K and w t = 1 if k t = K +1 .

̸

10: Update Ψ t +1 ,k t ← Ψ t,k t ∪ { t } and Ψ t +1 ,k ← Ψ t,k for k = k t .

F Algorithm for the Strong Adversary Case and Omitted Proofs in Section 5

F.1 Upper Bound

In this section, we introduce and analyze Algorithm 5, an algorithm that achieves ˜ O ( d elu √ Λlog |F| + d elu log |F| ) regret bound against strong adversary.

Algorithm 5 is an extension of the SA VE algorithm (Zhao et al., 2023) from linear function approximation to general function approximation. The algorithm maintains K +1 = Θ(log T ) bins denoted as { Ψ t,k } , k = 1 , 2 , . . . , K + 1 , which forms a partition of [ t -1] (i.e., every time index τ < t falls into exactly one of these bins). Each bin k can form a confidence function set like in standard LinUCB using samples in Ψ t,k . The overall confidence set F t is the intersection of the confidence sets of individual bins (Line 4).

Upon receiving the context x t , the learner form an active action set that contains plausible actions (Line 5). The next step is to decide which bin t should go to. This is done by leveraging the uncertainty measure g t,k ( a ) for bin k and action a (Line 6), which measures how uncertain the reward of action a is, given prior samples in bin k . The measure g t,k ( a ) corresponds to the quantity ∥ a ∥ Σ -1 t,k usually seen in linear contextual bandits, where Σ t,k is the covariance matrix formed by the samples in Ψ t,k . The algorithm finds the smallest k such that there exists an action with relative large uncertainty g t,k ( a ) ≥ 2 -k (Line 7). The learner would then choose this action in order to gain relatively large shrinking in bin k 's confidence set (Line 8), and put time t in bin k . The sample at time t is assigned a weight w t that is inversely proportional to the uncertainty measure (Line 9). This ensures that the importance of the samples within each bin is more balanced.

Before proving the main theorem Theorem 5.2, we first establish lemmas Lemma F.1-Lemma F.5.

Lemma F.1. Suppose that f ⋆ ∈ F t -1 . Then with probability at least 1 -δ/T , for all k ∈ [ K ] we have

where the last inequality is by the optimality of ˆ f t,k given in Eq. (16). According to the strengthened Freeman's Inequality (Lemma B.5), with L = C log( |F| T/δ ) for some large enough universal constant C (specified in Line 1 of Algorithm 5), the last expression in Eq. (17) can be further bounded by

for all k with probability at least 1 -δ/T by a union bound over k and ˆ f t,k . For any τ ∈ Ψ t,k , by the definition of w τ (Line 9 of Algorithm 5), we have

Combining this and Eq. (18), we get

Solving the inequality yields

Lemma F.2. Suppose that f ⋆ ∈ F t -1 . Then for all k ∈ [ K ] satisfying 2 2 k ≥ 80 L , we have with probability at least 1 -δ/T ,

Proof. With L = C log( |F| T/δ ) for some large enough universal constant C , we have

Rearranging gives the first inequality. For the second inequality, note that we have

Lemma F.3. With probability at least 1 -δ , f ⋆ ∈ F t for all t .

Proof. We prove by induction. Assume that f ⋆ ∈ F t -1 . Then for all k ∈ [ K ] such that 2 2 k ≥ 80 L , by Lemma F.1 and Lemma F.2, with probability at least 1 -δ/T ,

For k ∈ [ K ] such that 2 2 k < 80 L , we bound trivially

In both cases, we have f ⋆ ∈ F t by the definition of F t . By induction, we conclude that with probability at least 1 -δ , f ⋆ ∈ F t for all t .

.

Proof. For t such that k t ∈ [ K ] , by the definition of w t we have

Thus,

Proof.

Here, in ( i ) we use the definition of A t that there exists f ′ ∈ F t such that a t = max a ∈A f ′ ( x t , a ) , which implies

In ( ii ) we use the fact that for f, f ′ ∈ F t , k ≤ K and 2 2 k ≥ 80 L , we have

Proof of Theorem 5.2. Without loss of generality, assume 2 2 K ≥ 80 L , which is equivalent to T 2 ≥ Θ(log( |F| T/δ )) . Notice that [ T ] = ⋃ K +1 k =1 Ψ T +1 ,k .

Bound the regret in Ψ T +1 ,K +1 . Notice that by assumption, for t ∈ Ψ T +1 ,K +1 , we have 2 2( k t -1) = 2 2 K ≥ 80 L . Thus, by Lemma F.5,

Bound the regret in Ψ T +1 ,k with 2 2 k ≥ 80 L . By Lemma F.5,

Bound the regret in Ψ T +1 ,k with 2 2 k < 80 L .

Combining all parts proves the desired bound.

F.2 Lower Bound

Lemma F.6. For any integer A,T ≥ 2 , N ≤ c √ T/A and positive number Λ > AN/c with some c ≤ 1 , there exists a context space X , a contextual bandit problem F ⊂ ( X × A → R ) with eluder dimension d elu ( F , 0) = N ( A -1) and action set A = [ A ] , and adversarially assigned variances σ 2 1 , . . . , σ 2 T that ∑ T t =1 σ 2 t ≤ Λ such that any algorithm will suffer at least Ω(min { √ NA Λ , √ AT } ) .

Proof of Lemma F.6. Let ε 2 ∈ [2 N/T, 1] be a parameter to be decided later. Consider the function class F = { f (0) } ∪ { f ( i,j ) } i ∈ [ N ] ,j ∈ [ A -1] with the space of contexts X = { x (1) , . . . , x ( N ) } and the set of actions A = [ A ] . For any i ∈ [ N ] , j ∈ [ A -1] , the function f ( i,j ) is defined as the following: For i ∈ [ N ] and j ∈ [ A -1] ,

̸

Meanwhile

The eluder dimension of this function class is N ( A -1) since f ( i,j ) is uniquely identified by its value on ( x ( i ) , j ) .

We assume that x t is uniformly randomly chosen from X , and r t = f ⋆ ( x t , a t ) + ϵσ t , where ϵ ∼ N (0 , 1) and σ t is defined in the following way. Fix the algorithm. Let N t ( i, j ) = t ∑ s =1 ✶ ( x s = x ( i ) , a s = j ) for any x ∈ X and the action b . The adversary assigns

σ t = ✶ ( a t = j, N t ( x t , j ) ≤ 1 /ε 2 ) , that is, it assigns variance 1 when the algorithm chooses action b and the context-action pair x t , b are not played for more than 1 /ε 2 times.

Fix any algorithm. Denote by P 0 the probability distribution when f ⋆ = f (0) and E 0 the expectation under P 0 . For any i ∈ [ N ] , j ∈ [ A -1] , denote by P ( i,j ) the probability distribution when f ⋆ = f ( i,j ) and E ( i,j ) the expectation under P ( i,j ) . Then the adversary decides f ⋆ based on the following rule: if there exists i ∈ [ N ] , j ∈ [ A -1] such that

then let f ⋆ = f ( i,j ) . If no i, j satisfies this, let f ⋆ = f (0) .

If f ⋆ = f (0)

, then we have

On the other hand, if f ⋆ = f ( i,j ) , then we have

Then by Lemma D.2 of Foster et al. (2024), we have

Altogether, we can obtain

This in turn, implies that with the choice ε 2 ≥ 2 N/T , that

Thus if √ AT > √ NA Λ , then we set ε = √ NA/ Λ . We have ε ≤ 1 due to the assumption that Λ ≥ AN/c . Then we verify

Furthermore, √ AT > √ NA Λ implies Tε/N > AN/ε , and thus

Otherwise if √ AT < √ NA Λ , then we have T/N ≤ Λ and we set ε = N √ A/T . We have ε ≤ 1 due to the assumption that N ≤ c √ T/A . Then we verify

Finally, we have the lower bounds

Overall, we have proven that

Proof of Theorem 5.1.

We first deal with some corner cases:

Case 1: If A > T and d > T , then by Lemma C.2 with N = 1 , we have a lower bound of Ω( T ) .

Case 2: If A > T , d < T , and d > Λ , then by Lemma C.2 with N = 1 and the number of actions in Lemma C.2 set to d , we have a lower bound of Ω( d ) .

Case 3: If A > T , d < T , and d < Λ , then by Lemma C.1 with the number of actions in Lemma C.1 set to d , we have a lower bound of Ω( √ d Λ) .

Case 4: If A < T , d > T , then by Lemma C.2 with N = √ T/A , we have a lower bound of Ω( √ AT ) .

Case 5: If A < T , d < T , d ≤ A , and d ≥ Λ then by Lemma C.2 with N = 1 and the number of actions in Lemma C.2 set to be d , we have a lower bound of Ω( d ) .

Case 6: If A < T , d < T , d ≤ A , and d > Λ , then by Lemma C.1 with the number of actions in Lemma C.1 set to be d , we have a lower bound of Ω( √ d Λ) .

Now, we consider our main cases where A,d ≤ T and d ≥ A . We consider the following two subcases:

Subcase 1: If d ≥ Ω( √ d Λ) or d ≥ Ω( √ AT ) , then we invoke Lemma C.2 with N = min { d/A, √ T/A } and obtain a lower bound of Ω(min { d, √ AT } ) ≥ Ω(min { √ d Λ+ d, √ AT } ) .

Subcase 2: If d < c √ d Λ and d < c √ AT for a small enough constant 0 < c < 1 . Then we invoke Lemma F.6 with N = d/A and obtain a lower bound of Ω(min { √ d Λ+ d, √ AT } ) .

Thus, we conclude our proof.

G Omitted Proofs in Section 6

We revise the proof of Lemma 3 of Foster and Rakhlin (2020) to show the following guarantee.

Lemma G.1. We have for any t ∈ [ T ] ,

where σ 2 M t ( x t ) = sup a ∈ A σ 2 M t ( x t , a ) .

Proof of Lemma G.1. Fix t ∈ [ T ] . Let q ∈ ∆( A ) be the policy such that

where λ is such that ∑ a ∈A q ( x t , a ) = 1 . We show that such λ exists and λ ∈ (0 , A ] . Let h ( λ ) = ∑ a ∈A 1 λ + γ/σ 2 Mt ( x t ,a ) · (max a ′ ∈A f Mt ( x t ,a ′ ) -f Mt ( x t ,a )) . Then we have h ( λ ) is monotonically decreasing with h (0) = ∞ and h ( A ) < 1 . Thus there exists λ ∈ (0 , A ] such that h ( λ ) = 1 that corresponds to q ( x t , a ) .

We first separate the regret with respect to any fixed M into four parts as the following

where a ⋆ ∈ arg max a ∈A f M ⋆ ( x t , a ) . Firstly, by the definition of q , we have

Secondly, by the AM-GM inequality, we have

Thirdly, again by the AM-GM inequality and the definition of q , we have

Plug the inequality (22), (23), and (24) in the equality (21) to obtain the desired bound of

Lemma G.2. Whenever I t = 2 and M ⋆ ∈ M t , we have for any action a ∈ [ A ] ,

Proof. Since for any M,M ′ ∈ M t and a ∈ A , one has D 2 H ( M ( x t , a ) , M ′ ( x t , a )) ≤ 1 / 2 . Thus by Lemma B.2, we have that σ M ( x t , a ) ≲ σ M ′ ( x t , a ) .

Then since M ⋆ is in M t and M t is a mixture of models in M t , we have by the total variacne law

Lemma G.3. Whenever I t = 2 and M ⋆ ∈ M t , we have for any action a ∈ A

Proof. Since M ⋆ is in M t and M t is a mixture of models in M t , we have for any action a ∈ [ A ] ,

Then by Lemma B.3, we have that

Lemma G.4. The regrets accumalated on rounds where I t = 2 are bounded with probability at least 1 -δ by

where σ 2 M ⋆ ( x t ) = sup a ∈ A σ 2 M ⋆ ( x t , a ) and reg t = E a ∼ p t [max a ′ f M ⋆ ( x t , a ′ ) -f M ⋆ ( x t , a )] .

Proof. Let E t [ · ] := E [ · |H t ] where H t is the history up to time t . By Lemma G.1 and Lemma G.3, whenever I t = 2 and M ⋆ ∈ M t , we have

Then by Lemma A.15 of Foster et al. (2021), we have that with probability at least 1 -δ/ 2 ,

Then by the choice of γ = A T ∑ t =1 σ 2 M t ( x t ) / log(2 |M| /δ ) , we have with probability at least 1 -δ/ 2 ,

where the second inequality is by Lemma G.2. We also have by Lemma A.3 of Foster et al. (2021) that with probability at least 1 -δ/ 4 , for all t ∈ [ T ] ,

By union bound, this implies with probability at least 1 -3 δ/ 4 , for all t ∈ [ T ] , M ⋆ ∈ M t . Again by Lemma A.3 of Foster et al. (2021), we have with probability at least 1 -δ/ 4 , for all t ∈ [ T ]

Lemma G.5. For any λ ≥ 0 and α > 0 , we have

Proof. The proof follows similarly from the proof of Lemma D.5 by replacing the square divergence with squared Hellinger distance.

Lemma G.6. The regrets accumalated on rounds where I t = 1 are bounded by for any α > 0 .

Proof. For any t ∈ [ T ] and M,M ′ ∈ M t , we have

Thus let T ∑ t =1 ✶ ( I t = 1) = T 1 and we have

Thus we have for any α ≥ 0 ,

where the last inequality is by (25). Furthermore, by Cauchy-Schwarz inequality, we have

where the last inequality is by Lemma G.5. Altogether, we have

Reorganizing the above inequality, we obtain the desired bound.

Proof of Theorem 6.1. The proof is straight-forward by combing Lemma G.6 and Lemma G.4, i.e., with probability at least 1 -δ ,

Then, by choosing α = 1 / √ T , we obtain the desired bound.

Proof of Theorem 6.2. The proof of this theorem is essentially the same as that of Theorem 4.1. Recall that in Theorem 4.1 we construct a hard instance for a function class F that only contain reward mean information, but the reward has a fixed variance upper bound σ 2 t ≤ σ 2 . What we do here is simply embed this instance in a model class M where for every model, every context, and every action, the reward distribution is a Gaussian with mean as in F and variance σ 2 . Notice that we can change the distribution P -σ,ϵ and P + σ,ϵ in Lemma B.1 to N ( σ -ϵ, σ 2 ) and N ( σ + ϵ, σ 2 ) , respectively, which still gives us D KL ( P -σ,ϵ ∥ P + σ,ϵ ) ≤ (2 ϵ ) 2 2 σ 2 = 2 ϵ 2 σ 2 . This allows us to prove the same bound as in Lemma C.1. Furthermore, the eluder dimension d H elu ( M , 0) defined through the Hellinger distance between models remain the same as the eluder dimension d elu ( F , 0) defined through the mean difference in the constructions of Lemma C.1 and Lemma C.2. Overall, the lower bounds in Lemma C.1 and Lemma C.2 are still applicable after we change the reward distribution from Bernoulli-style distributions to Gaussian distributions, and change the distance measure from mean difference to Hellinger distance. The arguments in Theorem 4.1 thus allow us to prove the same lower bound in the case here.

Lemma G.7. For any integer A,T ≥ 2 , N ≤ c √ T/A and positive number Λ > AN/c , there exists a context space X , a contextual bandit model class M⊂ ( X × A → ∆( R )) with eluder dimension d H elu ( M , 0) = N ( A -1) and action set A = [ A ] , and adversarially assigned variances σ 2 1 , . . . , σ 2 T that ∑ T t =1 σ 2 t ≤ Λ such that any algorithm will suffer at least Ω(min { √ NA Λ , √ AT } ) .

Proof of Lemma G.7. This lower bound is based on a modification of Lemma F.6. Concretely, we illustrate here how to embed the hard case from Lemma F.6 to an equivalent model class in the distributional case.

One can add information into the context for the hard case constructed in Lemma F.6. Concretely, we don't enlarge the function class, but for each context x , the new context space will have T A corresponding contexts ( x, j 1 , ..., j A ) 1 ≤ j 1 ,...,j A ≤ T , where the second argument will be used to record the number of pulls to each action a under context x . The function value under these contexts will be the same as under the original context. The adversarial thus can choose in the new context space ( x t , N t ( x t , 1) , ..., N t ( x t , A )) to embed the hard case from Lemma F.6. The Hellinger eluder dimension is twice the eluder dimension because there are two types of variances corresponding to each original context. Thus, we obtain the desired bound.

Proof of Theorem 6.3.

This proof follows a similar argument with Theorem 5.1, with only embedding the function classes to model classes. Again, we change the Bernoulli-styled distributions in the construction of Theorem 5.1

to Gaussian distributions. The only other more crucial difference lies in the Subclass 2 at the end. We first deal with some corner cases:

Case 1: If A > T and d > T , then by Lemma C.2 with N = 1 , we have a lower bound of Ω( T ) .

Case 2: If A > T , d < T , and d > Λ , then by Lemma C.2 with N = 1 and the number of actions in Lemma C.2 set to d , we have a lower bound of Ω( d ) .

Case 3: If A > T , d < T , and d < Λ , then by Lemma C.1 with the number of actions in Lemma C.1 set to d , we have a lower bound of Ω( √ d Λ) .

Case 4: If A < T , d > T , then by Lemma C.2 with N = √ T/A , we have a lower bound of Ω( √ AT ) .

Case 5: If A < T , d < T , d ≤ A , and d ≥ Λ then by Lemma C.2 with N = 1 and the number of actions in Lemma C.2 set to be d, we have a lower bound of Ω( d ) .

Case 6: If A < T , d < T , d ≤ A , and d > Λ , then by Lemma C.1 with the number of actions in Lemma C.1 set to be d , we have a lower bound of Ω( √ d Λ) .

Now, we consider our main cases where A,d ≤ T and d ≥ A . We consider the following two subcases:

Subcase 1: If d ≥ Ω( √ d Λ) or d ≥ Ω( √ AT ) , then we invoke Lemma C.2 with N = min { d/A, √ T/A } and obtain a lower bound of Ω(min { d, √ AT } ) ≥ Ω(min { √ d Λ+ d, √ AT } ) .

Subcase 2: If d < c √ d Λ and d < c √ AT for a small enough constant c > 0 . Then we invoke Lemma G.7 and obtain a lower bound of Ω(min { √ d Λ+ d, √ AT } ) .

Thus, we conclude our proof.

H Upper Bound with Zero-One Variance (Section 7)

In this section, we consider the setting where σ t = 0 or 1 , and not reveal to the learner at the beginning of each round. The algorithm is displayed in Algorithm 6. We have the following theorem.

Theorem H.1. With the choice of γ = » 8 A log |F| , we have the upper bound on the expected regret of Algorithm 6

We will prove upper bounds for regret of rounds with σ t = 0 and σ t = 1 separately. First we bound the regret of rounds with σ t = 1 .

Lemma H.1. With γ = » 8 A log |F| , the output actions a t at each round in Algorithm 6 satisfies

Proof. Based on the inverse-gap weighting update rule Eq. (26), similar to (Foster and Rakhlin, 2020, Lemma 3), we have

Algorithm 6 Variance Sensitive SquareCB for Zero-One Noise

Input: γ .

1: Let F 1 ←F .

2: for t = 1 : T do

3: Receive x t , and calculate A t = { a ∈ A : ∃ f ∈ F t , f ( x t , a ) = arg max a ∈A f ( x t , a ) } .

4: Sample action a t ∼ p t and receive σ t and r t , where

5: Calculate L t ( f ) for all f ∈ F as

where for those σ τ = 0 , we define ( f ( x τ ,a τ ) -r τ ) 2 σ 2 = 0 if f ( x τ , a τ ) = r τ and ∞ otherwise.

6: Calculate q t ( f ) to be

7: Calculate f t +1 to be

8: if σ 0 = 0 then

9: Update F t +1 = { f ∈ F t , f ( x t , a t ) = r t } .

10: else

11: Let F t +1 = F t .

For the first term, we have analysis

where the first inequality uses the fact that σ t ∈ { 0 , 1 } and when σ t = 1 we have 1 √ 1+ ∑ t -1 s =1 σ 2 s =

where the last line is according to the analysis of Vovk's aggregating algorithm (Vovk, 1995; CesaBianchi and Lugosi, 2006).

τ

The following is a useful lemma regarding Eluder dimension.

Then we have

Proof. This lemma follows directly according to the definition of Eluder dimension d elu (1 /T ) in Definition 2.1, and the fact that all functions in F t must agree on the previous samples where σ s = 0 .

With this lemma, we are ready to bound the regret of rounds with σ t = 0 .

Lemma H.3. With γ = » 8 A log |F| , the output actions a t at each round in Algorithm 6 satisfies

Proof. In this proof, we focus on the case σ t = 0 . We first notice that for rounds σ t = 0 , we always have f ∗ ( x t , a t ) = r t . Hence f ∗ ∈ F t for every t ∈ [ T ] . We define

At round t , we let b t = arg max a ∈A f t ( x t , a ) , and we define B t ⊂ A t as

For rounds with σ t = 0 , based on x t , f t , we divide such rounds into two cases, which we denote as T 1 and T 2 .

(a) There exists a ∈ B t and f ∈ F t , such that | f ( x t , a ) -f t ( x t , a ) | ≥ 1 / T .

(b) For all f ∈ F t and a ∈ B t , we have | f ( x t , a ) -f t ( x t , a ) | ≤ 1 / T .

Regret in t ∈ T 1 . For those t ∈ T 1 , first we notice that for any a ∈ B t , according to Eq. (26) we have

where in the last inequality we use the fact that

Therefore, according to the definition of T 1 that there exists some a ∈ B t and f ∈ F t such that | f ( x t , a ) -f t ( x t , a ) | ≥ 1 /T , for any t ∈ T 1 with probability at least 1 / 7 A we will sample an action a t such that Z t = 1 ( Z t is defined in Eq. (27)). Therefore, we have

where the last inequality uses Lemma H.2.

Regret in t ∈ T 2 with a t ∈ B t . For those t ∈ T 2 , to facilitate the analysis we define

and

Then we have which implies that

Hence we have

̸

̸

̸

where in ( i ) we use the fact that according to the definition of B t in Eq. (28),

and ( ii ) uses the definition of T 2 that for any a ∈ B t and f ∈ F t we always have | f t ( x t , a ) -f ⋆ ( x t , a ) | ≤ 1 / T , and ( iii ) uses Eq. (30) and also the fact that

Suming this up for every t ∈ T 2 , we obtain that

where in the last inequality we use the definition of Φ t in Eq. (29) that Φ 0 = log |F| and

Regret in t ∈ T 2 with a t / ∈ B t . Next, for a t ̸∈ B t , according to the definition of A t , there exists a function ˜ f ∈ F t such that ˜ f ( x t , a t ) ≥ ˜ f ( x t , a ′ ) for any a ′ ∈ A . This implies that

where in ( i ) we use the definition of B t in Eq. (28), and in ( ii ) we use the definition of T 2 that for any f ∈ F t , | f ( x t , b t ) -f t ( x t , b t ) | ≤ 1 / T . Therefore, in those rounds with σ t = 0 , t ∈ T 2 and a ̸∈ B t , we always have Z t = 1 ( Z t is defined in Eq. (27)), which implies that

where the last inequality uses Lemma H.2.

Finally we combined these these bounds in T 1 and T 2 together, and obtain that

Finally, combining the regret bound for rounds with σ t = 1 and σ t = 0 together, we can prove Theorem H.1.

Proof of Theorem H.1. We decompose the regret

According to Lemma H.1, we have

and according to Lemma H.3, we have

Summing these two together, we obtain that

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: The main claims made in the abstract and introductio accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: The paper discuss the limitations of the work in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper provides detailed assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA] .

Justification: This is a theoretical paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n*** In line 12 there is a comment with acronyms of one of the authors, for the AC's consideration if it is a desk reject ***\n\nThe paper studies the setting of adversarial contextual MAB under the assumption of access to a realizable general function class for approximating the context-dependent rewards.\nThe authors prove both upper and lower regret bounds, that are variance dependent, i.e., dependent on the cumulative data variance rather than on $T$ which is the number of episodes. The authors present the regent bounds in three different settings (1) for strong adversary, (2) for weak adversary, (3) where learning from function class.\nIn all the cases, their bounds are also dependent on the Eluder dimension. They present a matching lower bound for each setting.\nThe presented algorithm is adapted from SquareCB of Foster and Rakhlin (2020), but additionally maintains a confidence function set, to learn faster when the functions in the confidence set have larger disagreement.\n\nSTRENGTHS:\n1.\tVariance-dependent regret bound is an interesting benchmark in RL. An application of it to contextual bandits is nice and worth the community's attention. \n2.\tThe work is extensive – the authors fully analyze three different settings, and prove both upper and lower bounds.\n\nWEAKNESSES:\n1.\tThe related literature review is unclear.  In the presented previous results, there should be a clear separation between variance-dependent bounds and minimax regret bounds. \n\n2.\tThe difference between the three settings is unclear to me. I would expect the authors to use the accepted terms, i.e., oblivious and adaptive adversary. Moreover, to be comparable to previous literature, I think the most interesting case is strong adversary + learning from function class.\n\n3.\tI do not see why the eluder diminution is necessary in all the bounds. It would be appreciated if the authors could provide an intuitive explanation.\n\n4.\tIt seems that the work over a confidence set of functions implies a running time complexity of $|F|$ that is not discussed. \n\n5.\tThe paper seems to be written at the last minute, as there are comments left inside the paper. See for additional example line 185 \"(our contribution)\".\n\n6.\tThe use of the Hellinger-Eluder dimension is given without any justification or an appropriate citation of previous works. The classic Eluder dimension introduced by Russo and Van-Roy is defined with respect to the $\\ell_2$ norm only. If the authors are using other versions of it, they have to cite related work, if such exists, or justify the used guarantees themselves.  In this case, [1] presents a version of the Eluder dimension to bounded metrics, that also holds to the squared Hellinger distance.\n\n\n[1] Eluder-based regret for stochastic contextual MDPs, Levy et al. 2024.\n\nQUESTIONS:\n1.\tPlease provide a specific comparison of your results with the known variance-dependent bounds in previous literature.\n\n2.\tWhat do you mean by $d_{elu}(0)$ ? For my understanding, the Eluder dimension is meaningful only for $\\alpha \\in (0,1] $.\n\n3.\tPlease refer to the weaknesses.\n\nLIMITATIONS:\nMentioned in the \"Weaknesses\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers contextual bandits with function approximation, and studies the influence of the variance information on the regret bound. It studies three regimes: weak adversary where the variance is revealed before each time step; strong adversary where the variance depends on the chosen action at each time step; and learning with a model class where the reward follows a Gaussian distribution with the mean and the variance depending on the context and the arm. By incorporating the variance, both the upper and lower bounds of the regret are refined, subject to some gaps in some scenarios.\n\nSTRENGTHS:\n- The proposed problem is interesting and the result also answers the conjecture that $\\tilde{O}(\\sqrt{A\\Lambda \\log|\\mathcal{F}|}+A\\log |\\mathcal{F}|)$  is not true, when we incorporate the variance into the algorithm design. The eluder dimension plays an important role in this bound and has been used to characterize the bounds.\n- The paper studies three different cases and provides sufficient insights in this line of problems. While there are some gaps between the upper and lower bounds, the discussions are thorough and can lead to further research.\n\nWEAKNESSES:\nSee the questions section\n\nQUESTIONS:\n- Can the authors summarize the main technical novelty used in the paper?\n- Concerning the third case, learning with the model class, the authors indicate that the variance information is not required as input in line 105. Does it mean that the variance, given the model, the context and the action, the variance $\\sigma_{M_t}^2(x_t,a)$ is revealed, as presented in line 7 in Algorithm 3? If we only assume that the variance is unknown but it only depends on the context and the arm, can the algorithm be modified such that it can work with estimating the variance on the fly, like [1]?\n- The  lower bound result Theorem 6.2 is obtained via instances with the same variance. While the lower bound is indeed a minimax lower bound, as this paper is considering the role of the variance, is it possible to derive a lower bound for the case where the variances are heterogeneous?\n\n[1] Jourdan, M., Rémy, D., & Emilie, K. Dealing with unknown variances in best-arm identification. International Conference on Algorithmic Learning Theory. PMLR. 2023.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies contextual bandits with general function approximation, emphasizing the impact of variance information. It proves lower bounds dependent on the Eluder dimension and variance information in both strong and weak adversary settings. It also proposes algorithms for both cases. When the adversary is weak, the proved regret upper bound is optimal, otherwise, there is a $\\sqrt{d}$ gap from optimal. The paper also studies the learning problem when the function class contains the distributional information and provides optimal regret bound.\n\nSTRENGTHS:\n1. The paper studies contextual bandits with general function approximation. The proved lower bound dependent on the Eluder dimension and variance information is very novel.\n2. The paper provides a comprehensive analysis for different settings, e.g, weak/strong adversaries, and distributional settings. The proved results are insightful and optimal in most cases.\n3. The presentation is very clear, including comparison with previous works.\n\nWEAKNESSES:\n1. In the known variance section, this paper only focuses on a regime where $A \\le d_{elu} \\le \\sqrt{AT}$. The case $d_{elu} \\le A$ is also very important, especially when the action set is large and function approximation benefits. In such cases, the proved upper bound and lower bound do not match, so the contribution in this section is a bit overclaimed. (matching upper bound when variance is known)\n2. Some missing related works.\nMore works on variance-dependent results of heteroscedastic bandits:\n\n[1] Zhou et al. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes, Colt 2021\n\n[2] Zhou and Gu. Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPs Neurips 2022\n\n[3] Zhao et al. Optimal Online Generalized Linear Regression with Stochastic Noise and Its Application to Heteroscedastic Bandits\n\n3. This description of \"checking disagreement\" (Line 166) is not very accurate. A similar definition of equation (4) has been used in previous works as weights ([4] [5]), bonus functions ([6][7]), and selection rules ([8][9]). I suggest adding more discussions on this.\n \n[4] Agarwal et al. Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. Colt 2023\n\n[5] Ye et al.  Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. ICML2023\n\n[6] Di et al. Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning, ICLR 2024\n\n[7] Huang et al. Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation AISTATS\n\n[8] Gentile et al. Fast Rates in Pool-Based Batch Active Learning ICML 2024\n\n[9] Zhao et al. A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximation\n\n4. Some minor errors: Line 12 worst case over what? \n\nIn the proof of Lemma E.5, there is an abusive use of $\\epsilon$, it represents both the Gaussian noise and a changeable parameter for $\\sigma$.\n\nQUESTIONS:\n1. Is it possible to prove a variance-dependent upper bound with $d_{elu}$ dependence in the first term of Theorem 4.2?\n2. What difficulty did you meet when trying to recover Zhao et al. (2023)’s result for unknown variance?\n\nLIMITATIONS:\nThe limitations and societal impact of this work have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of obtaining variance-aware regret bounds for realizable contextual bandits with general function approximation. They show that, despite intuition suggested by prior works, there is an unavoidable dependence on the Eluder dimension in getting variance-aware bounds. They investigate several different settings (depending on the strength of the adversary's environment design and the agent's knowledge of distributional assumptions) and show minimax lower bounds each involving the Eluder dimension. They also provide matching regret upper bounds under various assumptions on knowledge of distributions or variances.\n\nSTRENGTHS:\n* The problem is well-motivated, the surrounding literature is well-explained, and the authors take care to explain why the main message is surprising, while also explaining the intuition for the unavoidable dependence on Eluder dimension.\n* The algorithms and needed modifications are well-explained with authors taking care to make simplifying assumptions to enhance presentation and understanding.\n* Along the way, some new technicalities had to be developed such as employing a square regression oracle with variance-based regret employing a more complicated action selection scheme based on checking discerning actions.\n\nWEAKNESSES:\n* The main weakness is that the only upper bound results matching the lower bounds require some knowledge of the variance at every single round or under a stronger model class assumption (which then involves picking up a dependence on the size of the model class of variances).\n* There could be more discussion of other function classes beyond the generic finite regressor class ${\\cal F}$. For instance, how do the lower bounds and results here about variance-based bounds and Eluder dimension compare with the understanding for linear function classes, high-dimensional classes (like those mentioned for SquareCB in Foster and Rakhlin, 2020), neural network, or non-parametric classes? Do the lower bounds hold in greater generality?\n\nQUESTIONS:\nPlease refer to strengths and weaknesses above.\n\nLIMITATIONS:\nNo braoder impact concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}