Title: Sparsity-Agnostic Linear Bandits with Adaptive Adversaries
Venue: NeurIPS 2024
Accepted: yes
Keywords: Regret bounds

online learning

sparse linear regression

model selection
Abstract: We study stochastic linear bandits where, in each round, the learner receives a set of actions (i.e., feature vectors), from which it chooses an element and obtains a stochastic reward. The expected reward is a fixed but unknown linear function of the chosen action. We study \emph{sparse} regret bounds, that depend on the number $S$ of non-zero coefficients in the linear reward function. Previous works focused on the case where $S$ is known, or the action sets satisfy additional assumptions. In this work, we obtain the first sparse regret bounds that hold when $S$ is unknown and the action sets are adversarially generated. Our techniques combine online to confidence set conversions with a novel randomized model selection approach over a hierarchy of nested confidence sets. When $S$ is known, our analysis recovers state-of-the-art bounds for adversarial action sets. We also show that a variant of our approach, using Exp3 to dynamically select the confidence sets, can be used to improve the empirical performance of stochastic linear bandits while enjoying a regret bound with optimal dependence on the time horizon.

Sparsity-Agnostic Linear Bandits with Adaptive Adversaries

Abstract

We study stochastic linear bandits where, in each round, the learner receives a set of actions (i.e., feature vectors), from which it chooses an element and obtains a stochastic reward. The expected reward is a fixed but unknown linear function of the chosen action. We study sparse regret bounds, that depend on the number S of non-zero coefficients in the linear reward function. Previous works focused on the case where S is known, or the action sets satisfy additional assumptions. In this work, we obtain the first sparse regret bounds that hold when S is unknown and the action sets are adversarially generated. Our techniques combine online to confidence set conversions with a novel randomized model selection approach over a hierarchy of nested confidence sets. When S is known, our analysis recovers state-of-the-art bounds for adversarial action sets. We also show that a variant of our approach, using Exp3 to dynamically select the confidence sets, can be used to improve the empirical performance of stochastic linear bandits while enjoying a regret bound with optimal dependence on the time horizon.

1 Introduction

K -armed bandits are a basic model of sequential decision-making in which a learner sequentially chooses which arm to pull in a set of K arms. After each pull, the learner only observes the reward returned by the chosen arm. After T pulls, the learner must obtain a total reward as close as possible to the reward obtained by always pulling the overall best arm. Linear bandits extend K -armed bandits to a setting in which arms belong to a d -dimensional feature space. In each round t of a linear bandit problem, the learner receives an action set A t ⊂ R d from the environment, chooses an arm A t ∈ A t based on the past observations, and then receives a reward X t . In this work, we consider the stochastic setting in which rewards are defined by X t = ⟨ θ ∗ , A t ⟩ + ε t , where θ ∗ ∈ R d is a fixed latent parameter and ε t is zero-mean independent noise. In linear bandits, the learner's goal is to minimize the difference between the total reward obtained by pulling in each round t the arm a ∈ A t maximizing ⟨ θ ∗ , a ⟩ and the total reward obtained by the learner.

In stochastic linear bandits, the regret after T rounds is known to be of order d √ T up to logarithmic factors. The linear dependence on the number d of features implies that the learner is better off by ignoring features corresponding to negligible components of the latent target vector θ ∗ . Hence,

one would like to design algorithms that depend on the number S ≪ d of relevant features without requiring any preliminary knowledge on θ ∗ . This is captured by the setting of sparse linear bandits, where θ ∗ is assumed to have only 0 < S ≤ d nonzero components.

√

̸

In the sparse setting, Lattimore and Szepesvári [17, Section 24.3] show that a regret of Ω ( SdT ) is unavoidable for any algorithm, even with knowledge of S . When S is known, this lower bound is matched (up to log factors) by an algorithm of Abbasi-Yadkori et al. [2] who, under the same assumptions and for the same algorithm, also prove an instance-dependent regret bound of ˜ O ( Sd ∆ ) . Here ∆ is the minimum gap, over all T rounds, between the expected reward of the optimal arm and that of any suboptimal arm. In this work we focus on the sparsity-agnostic setting, i.e., when S is unknown. Fewer results are known for this case, and all of them rely on additional assumptions on the action set, or assumptions on the sparsity structure. For example, if the action set is stochastically generated, Oh et al. [22] prove a ˜ O ( S √ T ) sparsity-agnostic regret bound. More recently, Dai et al. [9] showed a sparsity-agnostic bound ˜ O ( S 2 √ T + S √ dT ) when the action set is fixed and equal to the unit sphere. In a model selection setting, Cutkosky et al. [8] prove a ˜ O ( S 2 √ T ) sparsity-agnostic regret bound for adversarial action sets, but under an additional nestedness assumption: ( θ ∗ ) i = 0 for i = 1 , . . . , S . Surprisingly, no bounds improving on the ˜ O ( d √ T ) regret of the OFUL algorithm [1] in the sparsity-agnostic case are known that avoid additional assumptions on the sparsity structure or on the action set generation.

Main contributions. Here is the summary of our main contributions. All the proofs of our results can be found in the appendix.

We introduce a randomized sparsity-agnostic linear bandit algorithm, SparseLinUCB , achieving regret ˜ O ( S √ dT ) with no assumptions on the sparsity structure (e.g., nestedness) or on the action set (which may be controlled by an adaptive adversary). When S = o ( √ d ) , our bound is strictly better than the OFUL bound ˜ O ( d √ T ) .

Our analysis of SparseLinUCB simultaneously guarantees an instance-dependent regret bound ˜ O ( max { d 2 , S 2 d } / ∆ ) , where ∆ is the smallest suboptimality gap over the T rounds.

If the sparsity level is known, our algorithm recovers the optimal bound ˜ Θ( √ SdT ) .

We also introduce AdaLinUCB , a variant of SparseLinUCB that uses Exp3 to learn the probability distribution over a hierarchy of confidence sets in stochastic linear bandits. Unlike previous works, which only showed a ˜ O ( T 2 / 3 ) regret bound for similar approaches, AdaLinUCB has a ˜ O ( √ T ) regret bound. In experiments on synthetic data, AdaLinUCB performs better than OFUL .

Technical challenges. Recall that the arm chosen in each round by OFUL is

where ̂ θ t is the regularized least-squares estimate of θ ∗ , V t -1 = I + ∑ s<t A s A ⊤ s is the regularized covariance matrix of past actions, and √ γ t is the radius of the confidence set

The squared radius γ t = O ( d ln t ) is such that θ ∗ belongs to (1.2) with high probability simultaneously for all t ≥ 1 . Our approach, instead, builds on the online to confidence set conversion technique of Abbasi-Yadkori et al. [2], where they show how to design a different confidence set for θ ∗ based on the predictions of an arbitrary algorithm for online linear regression, such that the squared radius of the confidence set is roughly equal to the regret bound of the algorithm. Using the algorithm SeqSEW for sparse online linear regression [13], whose regret bound is O ( S log T ) , they obtain the optimal regret ˜ O ( √ SdT ) for sparse linear bandits. Unfortunately, this result requires knowing S to properly set the radius of the confidence set. Our strategy SparseLinUCB (Algorithm 1) bypasses this problem by running the online to confidence set conversion technique over a hierarchy of nested confidence sets with radii α i = 2 i log T for i = 1 , . . . , n = Θ(log d ) . The framework of Abbasi-Yadkori et al. [2] guarantees that, for any sparsity value S ∈ [ d ] , there is a critical radius α o = O ( S log T ) such that, with high probability, θ ∗ lies in the set with radius α i for all i ≥ o . SparseLinUCB randomizes the choice of the index i of the confidence radius α i , used for selecting the action at time t . If

̸

the random index I t is such that I t ≥ o , then we can bound the regret incurred at step t using standard techniques [1, 2]. By choosing P ( I t = i ) proportional to 2 -i , we make sure that larger confidence sets (delivering suboptimal regret bounds) are chosen with exponentially small probability. If I t < o , then θ ∗ is not guaranteed to lie in the confidence set of radius α I t with high probability. Our main technical contribution is to show that the regret summed over these bad rounds is bounded by ˜ O (√ SdT/Q ) , where Q = P ( I t ≥ o ) . The proof of this bound requires showing that the regret in a bad round t (when I t < o ) can be bounded by √ α o ∥ A o t ∥ V -1 t -1 . Proving that ∥ A o t ∥ V -1 t -1 shrinks fast enough uses the fact that det V t grows fast enough due to the exploration in the good rounds t (when I t ≥ o ). This is done by a carefully designed peeling technique that partitions [ T ] in blocks based on the value of det V t .

To extend our analysis of SparseLinUCB and obtain instance-dependent regret bound, we apply the techniques of Abbasi-Yadkori et al. [1] to show that the regret over the good rounds is bounded by ˜ O ( d 2 / ∆ ) . The regret over a bad round t is controlled by ( α o / ∆) ∥ A o t ∥ 2 V -1 t -1 and-using techniques similar to the instance-independent analysis-we bound the regret summed over all bad rounds with ˜ O ( Sd/ ( Q ∆) ) .

Given that SparseLinUCB uses a fixed probability of order 2 -i to choose its confidence radius α i , it is tempting to explore adaptive probability assignments, that increase the probability of a confidence set proportionally to the rewards obtained by the actions that were selected based on that set. Algorithm AdaLinUCB (see Algorithm 2) is a variant of SparseLinUCB using Exp3 [4] to assign probabilities to confidence sets. The analysis of AdaLinUCB combines-in a non-trivial way-the analysis of Exp3 (including a forced exploration term q ) with that of SparseLinUCB . Although the resulting regret bound does not improve on OFUL , our algorithm provides a new principled solution to the problem of tuning the radius in (1.1). Experiments show that SparseLinUCB can perform better than OFUL .

1.1 Additional related work

Sparse linear bandits. With the goal of obtaining sparsity-agnostic regret bounds, different types of assumptions on the action set have been considered in the past. Starting from the ˜ O ( S √ T ) regret upper bound of [18], where the action set is assumed fixed and equal to the hypercube, some works considered stochastic action sets and proved regret bounds depending on spectral parameters of the action distribution, such as the minimum eigenvalue of the covariance matrix [9, 15, 16, 20]. Others assumed a stochastic action set with strong properties, such as compatibility conditions or margin conditions [3, 6, 7, 19, 22]. As far as we know, there has been no research on adaptive adversarial action sets after [2].

Model selection. Sparse linear bandits can be naturally viewed as a bandit model selection problem. For example, Ghosh et al. [14] establish a regret bound of ˜ O ( √ ST + d 2 /α 4 . 65 ) for a fixed action set, where α is the minimum absolute value of the nonzero components of θ ∗ . Quite a bit of work has been devoted to sparse regret bounds in the nested setting. With i.i.d. and fixed-size actions sets, Foster et al. [12] achieve a regret bound of order ˜ O ( S 1 / 3 T 2 / 3 /γ 3 ) in the nested setting, where γ is the smallest eigenvalue of the covariance matrix of A t . Under the same assumption on the action set, Pacchiano et al. [24, 23] obtain a regret bound of ˜ O ( S 2 √ T ) . For adversarial action sets, Cutkosky et al. [8] obtain ˜ O ( S 2 √ T ) in the nested setting. When actions are sampled i.i.d., Cutkosky et al. [8] and Pacchiano et al. [25] obtain a regret bound of ˜ O ( S 2 √ T ) for nested settings. They also obtain simultaneous instance-dependent bounds, in particular, Pacchiano et al. [25] achieve ˜ O ( ( Sd ) 2 / ∆ ) . Compared to the instance-dependent regret bound, our results are more general, as we allow the action set to be adaptively chosen by an adversary and do not require the nested assumption.

Parameter tuning. Although the theoretical anlysis of OFUL only holds for γ t = O ( d log t ) , smaller choices of the radius in (1.2) are known to perform better in practice. Our design of SparseLinUCB and AdaLinUCB borrows ideas from the parameter tuning setting, which is typically addressed using a set of base algorithms and a randomized master algorithm that adaptively changes the probability of selecting each base algorithm [21, 23, 24]. In particular, AdaLinUCB builds on [11], where they show that running Exp3 as the master algorithm over instances of OFUL with different radii has a better empirical performance than Thompson Sampling and UCB. Yet, they only show a regret bound of ˜ O ( T 2 / 3 ) when the action set is drawn i.i.d. in each round (they also prove a bound of order √ T , but only under additional assumptions on the best model). This is consistent with the results of Pacchiano et al. [24], who also obtained a regret of the same order using Exp3 as master algorithm.

2 Problem definition

In linear bandits, a learner and an adversary interact over T rounds. In each round t = 1 , . . . , T :

The adversary chooses an arm set A t ⊂ R d ;

The learner choose an arm A t ∈ A t ;

The learner obtains a reward X t .

We assume the adversary is adaptive, i.e., A t can depend in an arbitrary way on the (possibly randomized) past choices of the learner. The reward in each round t satisfies

Here θ ∗ ∈ R d is a fixed and unknown target vector and { ε t } t ∈ [ T ] are independent conditionally 1 -subgaussian random variables. We also assume ∥ θ ∗ ∥ 2 ≤ 1 and ∥ a ∥ 2 ≤ 1 for all a ∈ A t and all t ∈ [ T ] . 1 The regret of a strategy over T rounds is defined as the difference between the reward obtained by the optimal policy, always choosing the best arm in A t , and the reward obtained by the strategy choosing arm A t ∈ A t for t ∈ [ T ] ,

In the sparse setting, we would like to devise strategies whose regret depends on

̸

corresponding to the number of nonzero components of θ ∗ .

2.1 Online to confidence set conversions

Establishing a confidence set including the target vector with high probability is at the core of linear bandit algorithms, and our approach for designing a sparsity-agnostic algorithm is based on a result by Abbasi-Yadkori et al. [2]. They show how to construct a confidence set for OFUL [1] based on the predictions of a generic algorithm for online linear regression, a sequential decision-making setting defined as follows. For t = 1 , . . . , T :

The adversary privately chooses input A t ∈ R d and outcome X t ∈ R ;

The learner observes A t and chooses prediction ̂ X t ∈ R ;

The adversary reveals X t and the learner suffers loss ( ̂ X t -X t ) 2 .

The learner's goal in online linear regression is to minimize the following notion of regret against any comparator θ ∈ R d

The confidence set proposed by Abbasi-Yadkori et al. [2] is established by the following result.

Lemma 2.1 (Abbasi-Yadkori et al. [2, Corollary 2]) . Let δ ∈ (0 , 1 / 4] and ∥ θ ∗ ∥ 2 ≤ 1 . Assume a sequence { ( A t , X t ) } t ∈ [ T ] , where X t satisfies (2.1) for all t ∈ [ T ] , is fed to an online linear regression algorithm B generating predictions { ̂ X t } t ∈ [ T ] . Then P ( ∃ t ∈ [ T ] : θ ∗ / ∈ C t ) ≤ δ , where

and B T ( θ ∗ ) is an upper bound on the regret ρ T ( θ ∗ ) of B . When understood from the context, we will abuse the notation and denote the best radius in hindsight by γ ( δ ) := γ ( δ, θ ∗ ) and the regret bound by B T .

Gerchinovitz [13] designed an algorithm, SeqSEW , for sparse linear regression that bounds ρ T ( θ ) in terms of ∥ θ ∥ 0 simultaneously for all comparators θ ∈ R d . Below here, we state his bound in the formulation of Lattimore and Szepesvári [17].

Lemma 2.2 (Lattimore and Szepesvári [17, Theorem 23.6]) . Assume max t ∈ [ T ] ∥ A t ∥ 2 ≤ 1 and max t ∈ [ T ] | X t | ≤ 1 . There exists a universal constant c such that algorithm SeqSEW achieves, for any θ ∈ R d ,

where C T = 2 + log 2 log( e + T 1 / 2 ) .

Using the confidence set (2.2) with B set to SeqSEW , Abbasi-Yadkori et al. [2] achieved the minimax optimal regret bound of ˜ O ( √ SdT ) . However, to construct C t , the learner must know γ ( δ ) , which depends on the unknown sparsity level S = ∥ θ ∗ ∥ 0 through B T .

Algorithm 1 SparseLinUCB

1: Input: T ∈ N , n ∈ N and q ∈ ∆ n := { ( q 1 , . . . , q n ) ∈ [0 , 1] n : ∑ n i =1 q i = 1 }

2: Initialization: Let V 0 = I , ̂ θ 0 = (0 , . . . , 0)

3: for t = 1 , 2 , . . . , T do

4: Receive action set A t and draw I t from distribution q

5: Choose action A t = argmax a ∈A t ( ⟨ a, ̂ θ t -1 ⟩ + ∥ a ∥ V -1 t -1 √ 2 I t log T )

6: Receive reward X t

7: V t = V t -1 + A t A ⊤ t

8: Feed ( A t , X t ) to SeqSEW and obtain prediction ̂ X t

9: Compute regularized least squares estimate ̂ θ t = argmin θ ∈ R d ( ∥ θ ∥ 2 2 + t ∑ s =1 ( ̂ X s -⟨ θ, A s ⟩ ) 2 )

10: end for

3 A multi-level sparse linear bandit algorithm

In this section, we introduce our main algorithm, SparseLinUCB , whose pseudo-code is shown in Algorithm 1. The algorithm, which runs SeqSEW as base algorithm B , uses a hierarchy of confidence sets of increasing radius. In each round t = 1 , . . . , T , after receiving the action set A t , the algorithm draws the index I t of the confidence set for time t by sampling from the distribution q ∈ ∆ n := { ( q 1 , . . . , q n ) ∈ [0 , 1] n : ∑ n i =1 q i = 1 } . Then the algorithm plays the action A t using the confidence set C I t t := { θ ∈ R d : ∥ θ -̂ θ t -1 ∥ 2 V t -1 ≤ 2 I t log T } (where a larger I t implies a larger radius, and thus more exploration). Following the online to confidence set approach, upon receiving the reward X t , the algorithm feeds the pair ( A t , X t ) to SeqSEW and uses the prediction ̂ X t to update the regularized least squares estimate ̂ θ t .

Let α i = 2 i log T for all i ∈ N and set n ∈ N as

where γ ( δ, θ ) is defined in Lemma 2.1 for B = SeqSEW .

One can check that n = Θ(log d ) (when ∥ θ ∥ 0 = d ), which gives α n = Θ( d log T ) . Our bounds depend on the following quantity, which defines the index of the smallest 'safe' confidence set (i.e., the smallest i ∈ [ n ] such that θ ∗ ∈ C i t for all t ∈ [ T ] ),

The choice of our confidence set (Line 4 in Algorithm 1) is justified by the following result, which implies that o is safe.

Lemma 3.1. For C t defined in (2.2) , we have that C t ⊆ C o t for all t ∈ [ T ] .

As we use SeqSEW as base algorithm B , α o = O ( S log T ) . Our main result is an upper bound on the regret of SparseLinUCB .

Theorem 3.2. The expected regret of SparseLinUCB run with the number of models n in (3.1) and a distribution q = { q s } s ∈ [ n ] satisfies

If the sparsity level S is indeed known, then o in (3.2) can be computed and we get the following bound, which is tight up to log factors [17].

Corollary 3.3. Assume that the sparsity level S is known and choose the number of models n > o and the distribution { q s } s ∈ [ n ] with q o = 1 , where o is set as in (3.2) . Then, the expected regret of SparseLinUCB is E [ R T ] = O ( √ SdT log T ) .

An instance-dependent bound. SparseLinUCB also enjoys an instance-dependent regret bound comparable to that of OFUL . Let ∆ be the minimum gap between the optimal arm and any suboptimal arms over all rounds,

where A ∗ t = max a ∈A t ⟨ a, θ ∗ ⟩ is the optimal arm for round t .

Theorem 3.4. The expected regret of SparseLinUCB run with the number of models n in (3.1) , a distribution q = { q s } s ∈ [ n ] and using SeqSEW as base algorithm satisfies

where Q = ∑ s ≥ o q s .

Sparsity-agnostic tuning of randomization. Next, we look at a specific choice of q . Fix C ≥ 1 and let where κ > 0 is chosen so to normalize the probabilities. It is easy to verify that for any C ≥ 1 ,

implying that κ can be chosen in [0 , 1] . Combining Theorem 3.2 and 3.4, we obtain the following corollary providing a hybrid distribution-free and distribution-dependent bound.

Corollary 3.5. Pick any C ≥ 1 . Let the number of models n as in (3.1) and q = { q s } s ∈ [ n ] be chosen as in (3.4) . Then the expected regret of SparseLinUCB is

√

For C = 1 the above bound is ˜ O ( S dT ) , which is tight up to the factor S due to the lower bound of Ω( √ SdT ) [17]. However, as mentioned in Lattimore and Szepesvári [17, Section 23.5], no algorithm can enjoy the regret of ˜ O ( √ SdT ) simultaneously for all possible sparsity levels S . While our worst-case regret bound improves with a smaller S , the problem-dependent regret bound scales at least as ( d 2 / ∆)log T , which is independent of S . This raises an interesting question: could the problem-dependent bound also benefit from sparsity? Even with a very small probability p of choosing radius α n , the expected number of steps using α n would be pT . The results in [2] demonstrate that running the OFUL algorithm with α n over pT steps results in a regret of ˜ O ( d 2 / ∆) . One simple way is to decrease the frequency of selecting radius α n . However, selecting α n less than d 2 / ∆ 2 times may prevent the algorithm from obtaining a good enough estimate of θ ∗ in certain settings.

√

Remark 3.6. At first glance, it may seem straightforward to select C in Corollary 3.5, as setting C = √ d yields a regret of ˜ O ( d 2 / ∆) without apparent trade-offs. However, the trade-off lies in balancing the instance-dependent and worst-case regret bounds. Opting for C = d indeed yields an instance-dependent bound of ˜ O ( d 2 / ∆) . However, this comes at the expense of the worst-case bound, which remains ˜ O ( d √ T ) , negating any advantages derived from the sparsity assumption S ≪ d .

If the sparsity level S is indeed known, then o in (3.2) can be computed and we get the following bound.

Corollary 3.7. Assume that the sparsity level S is known and choose { q s } s ∈ [ n ] with q o = 1 , where o is set as in (3.2) . Then, the expected regret of SparseLinUCB is E [ R T ] = ˜ O ( Sd ∆ ) .

We note that by setting q o = 1 in Theorem 3.4, the regret bound becomes ˜ O ( d 2 / ∆) . This result, as detailed in Theorem 3.4, arises from the parameter q n > 0 . However, in this case, q n = 0 , which allows us to achieve a more favorable regret bound.

4 Adaptive model selection for stochastic linear bandits

SparseLinUCB is also designed to handle adaptive adversarial action sets. A crucial parameter of SparseLinUCB is { q s } s ∈ [ n ] , the distribution from which the radius of the confidence set is drawn. It is a natural question whether there exists an algorithm that adaptively updates this distribution based on the observed rewards. In this section we introduce AdaLinUCB (Algorithm 2), which runs Exp3 to dynamically adjust the distribution used by SparseLinUCB .

AdaLinUCB takes as input a forced exploration term q and the learning rate η for Exp3 . Similarly to SparseLinUCB , AdaLinUCB designs confidence sets of various radii, but its selection method differs in two aspects. First, with probability q , the algorithm performs exploration based on the confidence set with the largest radius. With probability 1 -q , the algorithm instead draws the action based on Exp3 . The distribution P t used by Exp3 at round t is based on exponential weights applied to the total estimated loss, denoted by S t (for technical reasons, we translate losses into rewards). The algorithm then draws I i from P t and selects the action A t based on the confidence set with radius 2 I t log T . Finally, reward X t is observed and the pair ( A t , X t ) is fed to SeqSEW . The prediction ̂ X t returned by SeqSEW is used to update the regularized least squares estimate ̂ θ t .

The following theorem states the theoretical regret upper bound of AdaLinUCB .

Theorem 4.1. If the random independent noise ε t in (2.1) satisfies ε t ∈ [ -1 , 1] for all t ∈ [ T ] , then the regret of AdaLinUCB run with η = √ (log n ) / ( Tn ) for n in (3.1) and q ∈ (0 , 1] satisfies

Although AdaLinUCB dynamically adjusts the distribution used by SparseLinUCB and may achieve better empirical performance, its regret bound is no better than that of SparseLinUCB . The issue is that the action chosen by AdaLinUCB in Line 10 does not ensure enough exploration to control the regret. Consequently, the algorithm needs to choose the optimistic action in Line 6 with constant probability q . SparseLinUCB has a similar parameter, Q , that bounds from the above the probability

of choosing the optimistic action. The key difference is that Q can be optimized for an unknown S by carefully selecting the distribution q = { q s } s ≥ 1 , whereas the parameter q does not provide a similar flexibility.

5 Model selection experiments

In this section we describe some experiments we performed on synthetic data to verify whether AdaLinUCB could outperform OFUL in a model selection task. We also test the empirical performance of SparseLinUCB on the same data (additional details on all the algorithms and the experimental setting are in Appendix E). The data for our model selection experiments are generated using targets θ ∗ with different sparsity levels, as we know that sparsity affects the radius of the optimal confidence set. On the other hand, since no efficient implementation of SeqSEW is known [17, Section 23.5], we cannot implement the online to confidence set approach as described in [2] to capture sparsity. Instead, we run SparseLinUCB and AdaLinUCB with ̂ X t = X t for all t ∈ [ T ] , which-due to the form of our confidence sets-amounts to running the algorithms over multiple instances of OFUL with different choices of radius α i for i ∈ [ n ] .

We run SparseLinUCB and AdaLinUCB with α i = α i,t = 2 i log t (a mildly time-dependent choice) for i = 0 , 1 , · · · , log 2 d . We also include α 0 = 0 corresponding to the greedy strategy A t = argmax a ∈A t ⟨ a, ̂ θ t -1 ⟩ . The suffix _ Unif indicates { q s } s ∈ [ n ] set to ( 1 n , · · · , 1 n ) . The suffix _ Theory indicates q s = Θ(2 -s ) for s = 0 , . . . , n as prescribed by (3.4). Finally, we included SparseLinUCB _ Known using q i = 1 { i = o } to test the performance when the optimal index o (for the given S ) is known in advance (see Corollary 3.3). We run our experiments with a fixed set of random actions, A t = A for all t ∈ [ T ] , where |A| = 30 and A is a set of vectors drawn i.i.d. from the unit sphere in R 16 . The target vector θ ∗ is a S -sparse ( S = 1 , 2 , 4 , 8 , 16 ) vector whose non-zero coordinates are drawn from the unit sphere in R S . The noise ε t is drawn i.i.d. from the uniform distribution over [ -1 , 1] . Each curve is an average over 20 repetitions with T = 10 4 where, in each repetition, we draw fresh instances of A and θ ∗ .

As our implementations are not sparsity-aware, we cannot expect the regret to strongly depend on the sparsity level. Indeed, only the regret of SparseLinUCB _ Known (which is tuned to the sparsity S ) is significantly affected by sparsity. The theory-driven choice of { q s } s ∈ [ n ] ( SparseLinUCB _ Theory ) performs better than the uniform assignment ( SparseLinUCB _ Unif ), and is in the same ballpark as OFUL . On the other hand, AdaLinUCB _ Unif and AdaLinUCB _ Theory outperform all the competitors, including OFUL . This provides evidence that using Exp3 for adaptive model selection may significantly boost the empirical performance of stochastic linear bandits.

6 Limitations and open problems

Unlike previous works, we prove sparsity-agnostic regret bounds with no assumptions on the action sets or on θ ∗ (other than boundedness of ∥ θ ∗ ∥ and ∥ a ∥ for a ∈ A t , which are rather standard assumptions). For AdaLinUCB , however, we do require boundedness of the noise ε t (instead of just subgaussianity). We conjecture this requirement could be dropped at the expense of a further log T factor in the regret. Finally, for efficiency reasons our implementations are not designed to capture sparsity. Hence our experiments are limited to testing the impact of model selection.

Our work leaves some open problems:

Proving a lower bound on the regret of sparse linear bandits when the sparsity level is unknown to the learner would be important. Citing again [17, Section 23.5], no algorithm can enjoy regret ˜ O ( √ SdT ) simultaneously for all sparsity levels S . However, we do not know whether the known lower bound Ω( √ SdT ) can be strengthened to Ω( S √ dT ) in the agnostic case.

Our instance-dependent regret bound is of order ˜ O ( max { S 2 , d } d ∆ ) . It would be interesting to prove an upper bound that improves on the factor d 2 / ∆ , or a lower bound showing that d 2 / ∆ cannot be improved on.

Our bound on the regret of AdaLinUCB looks pessimistic due to the presence of the constant exploration probability q . It would be interesting to prove a bound that more closely reflects the good empirical performance of this algorithm.

S=1

S=2

S=4

1600

1600

1600

1400

AL_Theory

1400

AL_Theory

1400

AL_Theory

1200

OFUL

1200

OFUL

0

OFUL

SL_Unif

SL_Unif

SL_Unif

1000

SL_Known

1000

SL_Known

1000

SL_Known

800

SL_Theory

800

SL_Theory

800-

SL_Theory

009

600

600

400

400

400

200

200

200

0

0

0

100

0

2000

4000

6000

8000

0

2000

4000

6000

8000

0

2000

4000

6000

8000

10000

S=8

S=16

1600

1600

1400

AL_Theory

1400

AL_Theory

1200

OFUL

1200

OFUL

SL_Unif

SL_Unif

1000

SL_Known

1000

SL_Known

800

SL_Theory

800

SL_Theory

600

600

400

400

200

200

0

0

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their helpful comments. This research is supported by the MUR PRIN grant 2022EKNE5K (Learning in Markets and Society), the FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme, the EU Horizon CL4-2022-HUMAN-02 research, innovation action under grant agreement 101120237, project ELIAS (European Lighthouse of AI for Sustainability), a Singapore Ministry of Education AcRF Tier 2 grant (A-8000423-00-00), and the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: AISG-PhD/2021-01-004[T]).

In particular, NCB and KJ acknowledge the financial support from the MUR PRIN grant, the FAIR project, and the ELIAS project. TJ is funded by a Singapore Ministry of Education AcRF Tier 2 grant (A-8000423-00-00) and the National Research Foundation, Singapore under its AI Singapore Program (AISG Award No: AISG-PhD/202101-004[T]).

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Online-to-confidence-set conversions and application to sparse stochastic bandits. In Artificial Intelligence and Statistics , pages 1-9. PMLR, 2012.

Kaito Ariu, Kenshi Abe, and Alexandre Proutière. Thresholded lasso bandit. In International Conference on Machine Learning , pages 878-928. PMLR, 2022.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Sunrit Chakraborty, Saptarshi Roy, and Ambuj Tewari. Thompson sampling for highdimensional sparse linear contextual bandits. In International Conference on Machine Learning , pages 3979-4008. PMLR, 2023.

Yi Chen, Yining Wang, Ethan X Fang, Zhaoran Wang, and Runze Li. Nearly dimensionindependent sparse linear bandit over small action spaces via best subset selection. Journal of the American Statistical Association , 119(545):246-258, 2024.

Ashok Cutkosky, Christoph Dann, Abhimanyu Das, Claudio Gentile, Aldo Pacchiano, and Manish Purohit. Dynamic balancing for model selection in bandits and rl. In International Conference on Machine Learning , pages 2276-2285. PMLR, 2021.

Yan Dai, Ruosong Wang, and Simon Shaolei Du. Variance-aware sparse linear bandits. In The Eleventh International Conference on Learning Representations , 2023.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.

Qin Ding, Yue Kang, Yi-Wei Liu, Thomas Chun Man Lee, Cho-Jui Hsieh, and James Sharpnack. Syndicated bandits: A framework for auto tuning hyper-parameters in contextual bandit algorithms. Advances in Neural Information Processing Systems , 35:1170-1181, 2022.

Dylan J Foster, Akshay Krishnamurthy, and Haipeng Luo. Model selection for contextual bandits. Advances in Neural Information Processing Systems , 32, 2019.

Sébastien Gerchinovitz. Sparsity regret bounds for individual sequences in online linear regression. In Proceedings of the 24th Annual Conference on Learning Theory , pages 377-396. JMLR Workshop and Conference Proceedings, 2011.

Avishek Ghosh, Abishek Sankararaman, and Ramchandran Kannan. Problem-complexity adaptive model selection for stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 1396-1404. PMLR, 2021.

Botao Hao, Tor Lattimore, and Mengdi Wang. High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems , 33:10753-10763, 2020.

Kyoungseok Jang, Chicheng Zhang, and Kwang-Sung Jun. Popart: Efficient sparse regression and experimental design for optimal sparse linear bandits. Advances in Neural Information Processing Systems , 35:2102-2114, 2022.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Tor Lattimore, Koby Crammer, and Csaba Szepesvári. Linear multi-resource allocation with semi-bandit feedback. Advances in Neural Information Processing Systems , 28, 2015.

Ke Li, Yun Yang, and Naveen N Narisetty. Regret lower bound and optimal algorithm for high-dimensional contextual linear bandit. Electronic Journal of Statistics , 15(2):5652-5695, 2021.

Wenjie Li, Adarsh Barik, and Jean Honorio. A simple unified framework for high dimensional bandit problems. In International Conference on Machine Learning , pages 12619-12655. PMLR, 2022.

Teodor Vanislavov Marinov and Julian Zimmert. The pareto frontier of model selection for general contextual bandits. Advances in Neural Information Processing Systems , 34:1795617967, 2021.

Min-Hwan Oh, Garud Iyengar, and Assaf Zeevi. Sparsity-agnostic lasso bandit. In International Conference on Machine Learning , pages 8271-8280. PMLR, 2021.

Aldo Pacchiano, Christoph Dann, Claudio Gentile, and Peter Bartlett. Regret bound balancing and elimination for model selection in bandits and RL. arXiv preprint arXiv:2012.13045 , 2020.

Aldo Pacchiano, My Phan, Yasin Abbasi Yadkori, Anup Rao, Julian Zimmert, Tor Lattimore, and Csaba Szepesvari. Model selection in contextual stochastic bandit problems. Advances in Neural Information Processing Systems , 33:10328-10337, 2020.

Aldo Pacchiano, Christoph Dann, and Claudio Gentile. Best of both worlds model selection. Advances in Neural Information Processing Systems , 35:1883-1895, 2022.

Vidyashankar Sivakumar, Steven Wu, and Arindam Banerjee. Structured linear contextual bandits: A sharp and geometric smoothed analysis. In International Conference on Machine Learning , pages 9026-9035. PMLR, 2020.

A Notation

In Table 2 we list the most used notations. Next, we recall some definitions that are used throughout this appendix. We have V t = I + ∑ t s =1 A s A ⊤ s and

where A s ∈ A s is the action chosen by the learner at round t . Recall that α i = 2 i log T . For i ∈ [ n ] ,

where

Finally, recall definition (2.2) with δ = 1 /T ,

and recall that E = ⋂ t ∈ [ T ] { θ ∗ ∈ C t } .

B Analysis of SparseLinUCB

Theorem 3.2. The expected regret of SparseLinUCB run with the number of models n in (3.1) and a distribution q = { q s } s ∈ [ n ] satisfies

where Q = ∑ s ≥ o q s .

Proof. Lemma 3.1 implies C t ⊂ C o t . Hence, if E is true and s < o , then

Therefore,

where in the first and the third inequalities, we use the fact that for any A ∈ R d ,

and for the last inequality,

We can decompose the regret as follows,

Since P ( E c ) ≤ δ ≤ 1 T , the first sum in the above line is easily bounded,

Bounding term ♣ . For each s ≥ 0 , let

Note that det( V t ) is monotone increasing w.r.t t . Define s ′ = ⌈ log 2 det( V T ) /d ⌉ . Then,

By applying Lemma D.5, we obtain

Substituting the bounds of ♠ and ♣ to (B.2), we have

concluding the proof.

Theorem 3.4. The expected regret of SparseLinUCB run with the number of models n in (3.1) , a distribution q = { q s } s ∈ [ n ] and using SeqSEW as base algorithm satisfies

where Q = ∑ s ≥ o q s .

Proof.

Bounding term ♣ . Let s ′ = ⌈ log 2 det( V T ) /d ⌉ . By applying Lemma D.5, we obtain

Bounding term ♠ .

where the first inequality comes from

Therefore,

Case 2: C 2 ∈ [2 o , 2 n ) . For s with C 2 < 2 s , q s = C 2 2 -s . Let o ′ = arg min s>o { C 2 < 2 s } . Then, q o ′ ≥ 1 / 4 . According to Theorem 3.2,

where the last inequality holds because E and I t ≥ o both hold, and so θ ∗ ∈ C t -1 ⊂ C I t t -1 due to Lemma 3.1. The factor 2 is due to an application of the triangular inequality. Substituting the bounds of ♠ and ♣ in (B.2), we have

Corollary B.1. Pick any C ≥ 1 and let { q s } s ∈ [ n ] be chosen as in (3.4) . Then the expected regret of SparseLinUCB is

Proof. We consider the following cases based on the value of C . Case 1: C 2 < 2 o .

q o = C 2 · 2 -o . According to Theorem 3.2,

Besides, according to Theorem 3.4,

Therefore,

Case C 2 ≥ 2 n : q s = 1 /n for all s ∈ [ n ] . It is easy to verify that E [ R T ] = ˜ O ( d √ T ) and E [ R T ] = ˜ O ( d 2 / ∆) . Thus,

Corollary B.2. Assume that the sparsity level S is known and choose a distribution q = { q s } s ∈ [ n ] with q o = 1 , where o is set as in (3.2) . Then, the expected regret of SparseLinUCB is E [ R T ] = ˜ O ( Sd ∆ ) .

Proof. We follow the same steps as in the proof of Theorem 3.4. The only difference lies in the bounding term ♠ in (B.3). We have

where the second equality is because q o = 1 and so A I t t = A o t = A t , and the first inequality comes from

where the last inequality holds because E and I t = o both hold, and so θ ∗ ∈ C t -1 ⊂ C o t -1 due to Lemma 3.1. Substituting the bounds of ♠ and ♣ in (B.2), we have

Besides, according to Theorem 3.4,

C Analysis of AdaLinUCB

Theorem 4.1. If the random independent noise ε t in (2.1) satisfies ε t ∈ [ -1 , 1] for all t ∈ [ T ] , then the regret of AdaLinUCB run with η = √ (log n ) / ( Tn ) for n in (3.1) and q ∈ (0 , 1] satisfies

Proof. Let T 1 be the set of time steps where Z t = 1 in Line 4 of AdaLinUCB and let T 2 = [ T ] \ T 1 . For all t ∈ T 2 the action A t ∈ A t is chosen by Exp3 through an adversarial mapping µ t : [ n ] →A t defined in Lines 9-10. The resulting reward X t ∈ [ -2 , 2] is fed to Exp3 as a [0 , 1] -valued loss ℓ t ( I t ) = (2 -X t ) / 4 , where µ t ( I t ) = A t . Since T 2 is selected through independent coin tosses with fixed bias q , we can apply the Exp3 regret analysis (for losses chosen by a non-oblivious adversary) to bound the regret in T 2 and show that

where we used µ t ( o ) = A o t for all t ∈ [ T ] . We have

Reg Exp3 is bounded in (C.1), hence we only need to bound

Let

Recall E is the event that θ ∗ ∈ C t for all t ∈ [ T ] . Assume E holds. We obtain that for t ∈ [ T ] ,

where the last inequality is because ˜ θ o t ∈ C o t . Then, assuming E holds, we can bound Reg Img as

where in the first inequality, we use the facts ⟨ θ ∗ , A ∗ t -A o t ⟩ ≤ √ α o ∥ A o t ∥ V -1 t -1 and ⟨ θ ∗ , A ∗ t -A o t ⟩ ≤ ⟨ θ ∗ , A ∗ t ⟩ ≤ 2 .

For each s ≥ 0 , let so that

Let s ′ = ⌈ log 2 det( V T ) /d ⌉ . We have

To obtain the final results, we have

where λ 1 , · · · , λ d are the eigenvalues of V T . Therefore, we have s ′ ≤ ⌈ log 2 (1 + TL 2 /d ) ⌉ .

By substituting the bounds on Reg Img and Reg Exp3 into R T 2 , we obtain

where the last inequality is because P ( E c ) ≤ 1 /T from Lemma 2.1 with our choice of δ = 1 /δ .

Finally, we bound R T 1 . Conditioned on event E and using (D.3), ∀ t ∈ [ T ] , θ ∗ ∈ C t ⊆ C 0 t ⊆ C n t . Hence, we can obtain

Hence, conditioned on event E ,

where the last inequality is due to Lemma D.1. Therefore, the expected regret for t ∈ T 1 is

where the first inequality is due to (C.7) and P ( E c ) ≤ 1 /T from Lemma 2.1 and the last inequality is due to the fact that for any t ∈ [ T ] , with probability q , t ∈ T 1 . Finally, we obtain

Note that n = Θ(log d ) and α o = Θ( S log T ) . We obtain

which completes the proof.

D Supporting lemmas

Lemma D.1 (Dani et al. [10]) . Let A 1 , A 2 , . . . , A T ∈ R d and V t = I + ∑ t s =1 A s A ⊤ s for all t ∈ [ T ] . Then

Moreover,

Lemma D.2. For C t defined in (2.2) , we have that C t ⊆ C o t for all t ∈ [ T ] .

Proof. Recall

Note that

Hence, we can express the ellipsoid as

Therefore, for all t ≥ 0 ,

concluding the proof.

Lemma D.3. For any 1 ≤ p ≤ q ≤ n and t ∈ [ T ] ,

Proof. Note that

Lemma D.4.

Since T s 2 ⊆ T s , to show we only need to prove

̸

For contradiction, we assume ∥ A q t ∥ V -1 t -1 < ∥ A p t ∥ V -1 t -1 . Since ∥ A q t ∥ V -1 t -1 < ∥ A p t ∥ V -1 t -1 , ∥ A q t ∥ V -1 t -1 = ∥ A p t ∥ V -1 t -1 . Besides, according to the definition of A p t and A q t , we have

where the first inequality is due to the definition of A p t , the second inequality is due to √ α p < √ α q , and the last inequality is due to the definition of A q t . From the above results, we further have

We obtain a contradiction. Therefore,

Proof. Let det( V t -1 ) = ( q t -1 ) d and det( V t -1 + A n t ( A n t ) ⊤ ) = ( q t -1 + x t ) d . For each s ≥ 0 , let

We let I t = 1 if the coin tosses in the t 's round is Head and 0 otherwise. We divide T s into two disjoint parts T s and T s . Specifically,

From definition of T s and the fact that if I t = 1 , V t = V t -1 + A n t ( A n t ) ⊤ , we have

From Algorithm 2, with probability q , I t = 1 . Therefore, if we let {F t } t ∈ [ T ] be the natural filtration of { A t , I t , X t } t ∈ [ T ] , we have

and therefore E [ ∑ t ∈T s x t ] ≤ 2 s /q . For t ∈ T s , we further have

Therefore,

From definition of T s , if I t = 1 and t ∈ T s , det( V t ) > 2 d ( s +1) . Then, for all τ > t , τ / ∈ T s . Therefore, there is at most one t ∈ T s with I t = 1 . We obtain

where the last inequality is because with probability q , I t = 1 . By combining the bounds for t ∈ T s and t ∈ T s together, lemma follows.

Lemma D.5. Let Q = ∑ s ≥ o q s . Then,

Proof. Let det( V t -1 ) = ( q t -1 ) d . Let x t satisfies det( V t -1 + A o t ( A o t ) ⊤ ) = ( q t -1 + x t ) d . We let I t = 1 { I t ≥ o } . We divide T s into two disjoint parts T s and T s . Specifically,

Note that for I t ≥ o ,

Therefore, if we let det( V t -1 + A I t t ( A I t t ) ⊤ ) = ( q t -1 + x ′ t ) d and I t ≥ o , then x ′ t ≥ x t . Hence,

Note that P ( I t ≥ o ) = ∑ s ≥ o q s = Q . Let {F t } t ∈ [ T ] be the natural filtration of { A t , I t , X t } t ∈ [ T ] . We have

and therefore

For t ∈ T s , we further have

(Law of total expectation) ( x t and 1 { t ∈ T s } are F t -1 -measurable) ( I t is an independent coin-tossing)

Therefore,

From definition of T s , if I t = 1 and t ∈ T s , then I t ≥ o and

Then, for all τ > t , τ / ∈ T s . Therefore, there is at most one t ∈ T s with I t = 1 . We obtain

where the last inequality is because with probability ∑ s ≥ o q s = Q , I t = 1 . By combining the bounds for t ∈ T s and t ∈ T s together, lemma follows.

E Experimental details

The code used in the experiments can be found in the following repository: https://github.com/ jajajang/sparsity_agnostic_model_selection .

E.1 Settings common to all algorithms

Arm set: A t = A for all t ∈ [ T ] . A ⊂ S d -1 (the unit sphere in R d ) is a set of d -dimensional vectors drawn independently and uniformly at random from S d -1 , with d = 16 and |A| = 30 .

θ ∗ is an S -sparse ( S = 1 , 2 , 4 , 8 , 16 ) vector generated as follows: before the game starts, draw ( θ ∗ ) 1 , . . . , ( θ ∗ ) S ∼ S S -1 , and ( θ ∗ ) k = 0 for all k > S .

The noise on rewards: { ε t } t ∈ [ T ] are i.i.d. with ξ t ∼ Unif([ -1 , 1]) .

Number of iterations: T = 10 4 .

Number of models: n = 6 .

Radius of confidence sets: α 0 = 0 , and α i = 2 i log t for i = 1 , · · · , 5 .

Prior distribution { q s } s ∈ [6]

-For _Unif , { q s } s ∈ [6] = ( 1 6 , 1 6 , 1 6 , 1 6 , 1 6 , 1 6 )

-For _Theory , { q s } s ∈ [6] = ( C 2 , C 4 , C 8 , C 16 , C 32 , C 64 ) where C = 63 64 is a normalizing constant.

Each plot is the result of 20 repetitions for each method. The shade represents the 1-standard deviation bound.

Hardware: Lenovo Thinkpad P16s Gen 2 Laptop - Type 21HL

-CPU: 13th Gen Intel(R) Core(TM) i7-1360P 2.20 GHz

-RAM: 32GB

Computation time: total 1338.38 seconds.

E.2 AdaLinUCB details

Since we empirically observed that Exp3 provided enough exploration, we aggressively set the forced exploration parameter q to zero.

The learning rate of Exp3 was set to η t = 2 √ log n nt , see [5].

Given the prior distribution { q s } s ∈ [6] , we set P t as follows:

E.3 OFUL details

We used the log-determinant form of the confidence set based on Abbasi-Yadkori et al. [1, Theorem 2], which gives the choice

for the parameter γ t in (1.1) when λ = 1 and δ = 1 /T .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction accurately represent the contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have added a paragraph in the conclusions section discussing the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and complete proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include our code in our supplementary material and will make the full code public if this work gets accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the details of our experimental setting.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We reported 1-sigma error bars in our plots.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We state all information on the computer resources in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the sparse linear bandit problem without the prior knowledge of sparsity. The studied problem also considers general setup in which the context is chosen by an adaptive adversary and action set is not imposed with additional assumptions. Then, A OFUL based algorithms are proposed and regret bounds are provided.

STRENGTHS:
- Lifting assumptions used in existing works while still maintaining comparable or even better regret bounds.

- The proposed algorithm provides an instance-dependent regret bound and worst-case bound as well.

- As the performance of SparseLinUCB highly depends on distribution q, different choices of q are shown.

- Numerical experiments are provided to support the significance of AdaLinUCB algorithm.

WEAKNESSES:
- The instance dependent regret bound shows no improvement compared with standard OFUL algorithm, and could be even worse than that of OFUL, e.g., S^2>d.

- For known sparsity and adaptive adversary setup, the instance depedent regret bound in Corollary 3.5 is worse than that of [2], which is dS/Delta.

- The regret bound of SparseLinUCB is sensitive to the choice of distribution q. Though authors provide a specific example on the choice of q, the regret bound now highly depends on the choice of constant C. For example, if one chooses C=1, the instance dependent bound is worse than that of OFUL when S^2>d.

- For the action chosen (step 5 in Algorithm 1), it could be difficult to compute A_t since this optimization problem contains non-linear term and the action set is arbitrary.

QUESTIONS:
Can authors provide another distribution selection case which can recover dS/Delta instance-dependent bound in known sparsity and adaptive adversary setup?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes statistically efficient linear bandit algorithms capable of handling cases where prior knowledge of the sparsity level $S$ is not given. The first algorithm, SparseLinUCB, achieves a $\\tilde{O}(S \\sqrt{dT})$ regret bound without any stochastic assumptions on the context vector, covering adversarially given context vectors. The main idea involves sampling the radius of the confidence set for the true reward parameter $\\theta_*$ from a specific distribution and then selecting the optimistic action. It matches the lower bound when sparsity information is provided. The second algorithm, AdaLinUCB, updates the sampling distribution of the confidence radius using an approach (Exp3) that increases the likelihood of selecting a radius providing higher rewards. AdaLinUCB also achieves a $\\tilde{O}(\\sqrt{T})$ regret bound. Various experiments support the theoretical results of the proposed algorithms.\n\nSTRENGTHS:\n- The motivation for the problem addressed in the paper is well explained, and the related work is thoroughly described. Overall, the paper is well-written and easy to understand.\n- The first algorithm, SparseLinUCB, is, to my knowledge, the first sparsity-agnostic linear bandit algorithm for adversarial context vectors. Additionally, it matches the lower bound regret when sparsity information is provided.\n- The second algorithm, which updates the confidence radius distribution at each time step, is also very interesting. Previous works using similar methods achieved loose regret bounds ($\\tilde{O}(T^{2/3})$), whereas the proposed algorithm achieves $\\tilde{O}(\\sqrt{T})$ regret (though I have not rigorously checked this proof).\n- Various numerical experiments support the theory behind the proposed algorithms.\n\nWEAKNESSES:\n- The SparseLinUCB algorithm does not make stochastic assumptions about the action set (context vectors), thus providing theoretical guarantees even in the case of an adaptive adversary. However, AdaLinUCB is described as an algorithm for stochastic linear bandits. It seems that the stochastic assumptions required for AdaLinUCB's regret bound are not explained.\n- The explanation about $n$ in the confidence radius distribution $\\set{ q_s }_{s \\in [n]}$ appears insufficient. Additionally, there seems to be no term for $ n $ in the regret bound. I am curious whether the regret bound is independent of $ n$.\n\nQUESTIONS:\n- (Related to the 1st bullet in Weaknesses) Can AdaLinUCB still achieve the currently presented regret bound if the action set (context vectors) is given by an adaptive adversary? If not, can you briefly explain the issue?\n- (Related to the 2nd bullet in Weaknesses) It seems that how $\\set{ q_s }_{s \\in [n]}$ is determined would impact the regret bound. You introduced a specific distribution in Eq. (3.3). How does the regret bound change if $ n$ is significantly increased or decreased?\n- The motivation for updating the confidence radius distribution at each time step in AdaLinUCB is interesting. However, the current result shows a looser regret bound compared to SparseLinUCB. Despite the computational cost of updating the distribution, there seems to be no statistical gain. In what instances would it be better to use AdaLinUCB over SparseLinUCB? Also, can you briefly explain why AdaLinUCB performs better empirically?\n\nLIMITATIONS:\nThe authors have well-addressed the limitations and further research directions in Section 6. The content discussed in this paper appears to have little to no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper studies the stochastic linear bandits when the action set can be arbitrarily chosen without some additional assumptions. And the authors propose a randomized sparsity-agnostic bandit algorithm using the model selection idea, and show that EXP3 can be used as the master algorithm to obtain a decent regret bound. Experimental results are included in the end to verify the high efficiency of the proposed algorithms.

STRENGTHS:
1. The paper is easy to read, and most parts are pretty clear. E.g. Table 1 helps reader catch up all the existing literature and their differences quickly.
2. This paper studies an interesting and important problem when the arm set is arbitrarily chosen under the sparse linear bandit problem. I didn't check the proof in appendix in detail but all arguments seem reasonable to me.
3. Empirical results are provided to illustrate the high efficiency of the proposed algorithms.

WEAKNESSES:
1. The used techniques are based on the existing literature (e.g. seqsew, exp3 master algorithm). It will be better to show the theory novelty of this work in a separate paragraph. 

2. Lower bounds are not provided, as the authors mention in the limitation parts. So it is a little bit hard for readers to justify how good are the proposed bounds.

QUESTIONS:
1. Why is the boundness of the random noise necessary in your theoretical proof?
2. Is it possible to bound the regret with high probability instead of the expected value? If some existing literature proposed the regret bound with high probability, then it may not be fair to do the direct order comparison in Table 1.
3. Can you report the running time of your method in the experiments?

LIMITATIONS:
No negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies Linear bandits with adversaries when the underlying parameter $\\theta$ is sparse. It combines a previous sparse linear regression algorithm named SeqSEW with LinUCB, proposing an algorithm named SparseLinUCB. It also proposes a variant of the EXP3 algorithm named AdaLinUCB. Regret bounds dependent on the sparsity dimension $S$ are proved. Experiments are conducted on synthetic data.\n\nSTRENGTHS:\n1. The paper proposes algorithms for sparse linear bandits with adversarial action sets. It proved a regret upper bound better than previous when the sparsity dimension $S$ is quite small with no assumptions on the sparsity structure. If the sparsity level is known, the result is optimal.\n2. It also provides an instance-dependent regret bound.\n3. It conducts synthetic experiments to show its better performance.\n\nWEAKNESSES:\n1. The paper misses many very related works, especially those on K-armed bandits and linear bandits.\n\n2. For the main algorithm design, I cannot see the necessity to use an online learning oracle to predict the reward $\\hat X_t$ and then calculate the least squares estimation when having access to the real reward $X_t$. Could you explain it further?\n\n3. The writing is unclear. \n\n(1) In Line 176,  \"the distribution $\\\\{q_i\\\\} _ {i \\in [n]}$\" is not well defined, over what? \n\n(2) The notation of the confidence set in Line 177 is contradictory with the notation in Appendix A, as it uses 2-norm rather than the $V_{t-1}$ norm. \n\n(3) In equation 3.1, $\\gamma$ is a problem-dependent term so you cannot set the exact value of $o$, as is done in Corollary 3.3.\n\n(4) Line 240: in terms of what?\n\n4. Line 249: “the parameter q does not provide a similar flexibility”, so I do not see the use of this probable selection, what is the benefit of the new algorithm design in Section 4?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nThe authors have addressed the limitations and societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}