Title: Anytime Model Selection in Linear Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandits

model selection

online learning
Abstract: Model selection in the context of bandit optimization is a challenging problem, as it requires balancing exploration and exploitation not only for action selection, but also for model selection. One natural approach is to rely on online learning algorithms that treat different models as experts. Existing methods, however, scale poorly ($\mathrm{poly}M$) with the number of models $M$ in terms of their regret.
Our key insight is that, for model selection in linear bandits, we can emulate full-information feedback to the online learner with a favorable bias-variance trade-off. This allows us to develop ALEXP, which has an exponentially improved ($\log M$) dependence on $M$ for its regret.
ALEXP has anytime guarantees on its regret, and neither requires knowledge of the horizon $n$, nor relies on an initial purely exploratory stage.
Our approach utilizes a  novel time-uniform analysis of the Lasso, establishing a new connection between online learning and high-dimensional statistics.

Anytime Model Selection in Linear Bandits

Abstract

Model selection in the context of bandit optimization is a challenging problem, as it requires balancing exploration and exploitation not only for action selection, but also for model selection. One natural approach is to rely on online learning algorithms that treat different models as experts. Existing methods, however, scale poorly ( poly M ) with the number of models M in terms of their regret. Our key insight is that, for model selection in linear bandits, we can emulate full-information feedback to the online learner with a favorable bias-variance trade-off. This allows us to develop ALEXP, which has an exponentially improved ( log M ) dependence on M for its regret. ALEXP has anytime guarantees on its regret, and neither requires knowledge of the horizon n , nor relies on an initial purely exploratory stage. Our approach utilizes a novel time-uniform analysis of the Lasso, establishing a new connection between online learning and high-dimensional statistics.

1 Introduction

When solving bandit problems or performing Bayesian optimization, we need to commit to a reward model a priori , based on which we estimate the reward function and build a policy for selecting the next action. In practice, there are many ways to model the reward by considering different feature maps or hypothesis spaces, e.g., for optimizing gene knockouts [Gonzalez et al., 2015, Pacchiano et al., 2022] or parameter estimation in biological dynamical systems [Ulmasov et al., 2016, Imani et al., 2019]. It is not known a priori which model is going to yield the most sample efficient bandit algorithm, and we can only select the right model as we gather empirical evidence. This leads us to ask, can we perform adaptive model selection, while simultaneously optimizing for reward?

In an idealized setting with no sampling limits, given a model class of size M , we could initialize M bandit algorithms (a.k.a agents) in parallel, each using one of the available reward models. Then, as the algorithms run, at every step we can select the most promising agent, according to the cumulative rewards that are obtained so far. Model selection can then be cast into an online optimization problem on a M -dimensional probability simplex, where the probability of selecting an agent is dependent on its cumulative reward, and the optimizer seeks to find the distribution with the best return in hindsight. This approach is impractical for large model classes, since at every step, it requires drawing M different samples in parallel from the environment so that the reward for each agent is realized.

In realistic applications of bandit optimization, we can only draw one sample at a time, and so we need to design an algorithm which allocates more samples to the agents that are deemed more promising. Prior work [e.g., Maillard and Munos, 2011, Agarwal et al., 2017] propose to run a single meta algorithm which interacts with the environment by first selecting an agent, and then selecting an action according to the suggestion of that agent. The online model selection problem can still be emulated in this setup, however this time the optimizer receives partial feedback, coming from only one agent. Consequently, many agents need to be queried, and the overall regret scales with poly M , again restricting this approach to small model classes. In fact, addressing the limited scope of such algorithms, Agarwal et al. [2017] raise an open problem on the feasibility of obtaining a log M dependency for the regret.

We show that this rate is achievable, in the particular case of linearly parametrizable rewards. We develop a technique to 'hallucinate' the reward for every agent that was not selected, and run the online optimizer with emulated full-information feedback. This allows the optimizer to assess the quality of the agents, without ever having queried them. As a result, our algorithm ALEXP, satisfies a regret of rate O (max { √ n log 3 M,n 3 / 4 √ log M } ) , with high probability, simultaneously for all n ≥ 1 (Theorem 1). Our key idea, leading to log M dependency, is to employ the Lasso as a low-variance online regression oracle, and estimate the reward for the agents that were not chosen. This trick is made possible through our novel time-uniform analysis of online Lasso regression (Theorem 3). Consequently, ALEXP is horizon-independent, and explores adaptively without requiring an initial exploration stage. Empirically we find that ALEXP consistently outperforms prior work across a range of environments.

2 Related Work

Online Model selection (MS) for bandits considers combining a number of agents in a master algorithm, with the goal of performing as well as the best agent [Maillard and Munos, 2011, Agarwal et al., 2017, Pacchiano et al., 2020, Luo et al., 2022]. This literature operates on black-box model classes of size M and uses variants of Online Mirror Descent (OMD) to sequentially select the agents. The optimizer operates on importance-weighted estimates of the agents' rewards, which has high variance ( poly M ) and is non-zero only for the selected agent. Effectively, the optimizer receives partial (a.k.a. bandit) feedback and agents are at risk of starvation, since at every step only the selected agent gets the new data point. These works assume knowledge of the horizon, however, we suspect that this may be lifted with a finer analysis of OMD.

Sparse linear bandits use sparsity-inducing methods, often Lasso [Tibshirani, 1996], for estimating the reward in presence of many features, as an alternative to model selection. Early results are in the data-rich regime where the stopping time n is known and larger than the number of models, and some suffer from poly M regret dependency [e.g., Abbasi-Yadkori et al., 2012]. Recent efforts often consider the contextual case, where at every step only a finite stochastic subset of the action domain is presented to the agent, and that the distribution of these points is i.i.d. and sufficiently diverse [Li et al., 2022, Bastani and Bayati, 2020, Kim and Paik, 2019, Oh et al., 2021, Cella and Pontil, 2021]. We do not rely on such assumptions. Most sparse bandit algorithms either start with a purely exploratory phase [Kim and Paik, 2019, Li et al., 2022, Hao et al., 2020, Jang et al., 2022], or rely on a priori scheduled exploration [Bastani and Bayati, 2020]. The exploration budget is set according to the horizon n . Therefore, such algorithms inherently require the knowledge of n and can be made anytime only via the doubling trick [Auer et al., 1995]. Table 2 presents an in-depth overview.

ALEXP inherits the best of both worlds (Table 1): its regret enjoys the log M dependency of sparse linear bandits even on compact domains, and it has adaptive probabilistic exploration with anytime guarantees. In contrast to prior literature, we perform model selection with an online optimizer (EXP4), which hallucinates full-information feedback using a low-variance Lasso estimator instead of importance-weighted estimates. Moreover, our anytime approach lifts the horizon dependence and the exploration requirement of sparse linear bandits.

Our work is inspired by and contributes to the field of Learning with Expert Advice, which analyzes incorporating the advice of M oblivious (non-adaptive) experts, with bandit or full-information feedback [Haussler et al., 1998, Auer et al., 2002b, McMahan and Streeter, 2009]. The idea of employing an online optimizer for learning stems from this literature, and has been used in various applications of online learning [Foster et al., 2017, Singla et al., 2018, Muthukumar et al., 2019, Karimi et al., 2021, Liu et al., 2022]. In particular, we are inspired by Foster and Rakhlin [2020] and Moradipari et al. [2022], who apply EXP4 to least squares estimates, for arm selection in K -armed

contextual bandits. However, their algorithms are not anytime, and due to the choice of estimator, the corresponding regret scales with O (min( √ M, √ K log M )) .

3 Problem Setting

We consider a bandit problem where a learner interacts with the environment in rounds. At step t the learner selects an action x t ∈ X , where X ⊂ R d 0 is a compact domain and observes a noisy reward y t = r ( x t ) + ε t such that ε t is an i.i.d. zero-mean sub-Gaussian variable with parameter σ 2 . We assume the reward function r : X → R is linearly parametrizable by some unknown feature map, and that the model class { ϕ j : R d 0 → R d , j = 1 , . . . , M } contains the set of plausible feature maps. We consider the setting where M can be very large, and while the set { ϕ j } may include misspecificed feature maps, it contains at least one feature map that represents the reward function, i.e., there exists j ⋆ ∈ [ M ] such that r ( · ) = θ ⊤ j ⋆ ϕ j ⋆ ( · ) . We assume the image of ϕ j spans R d , and no two feature maps are linearly dependent, i.e. for any j, j ′ ∈ [ M ] , there exists no α ∈ R such that ϕ j ( · ) = α ϕ j ′ ( · ) . This assumption, which is satisfied by design in practice, ensures that the features are not ill-posed and we can explore in all relevant directions. We assume that the concatenated feature map ϕ ( x ) := ( ϕ 1 ( x ) , . . . , ϕ M ( x )) is normalized ∥ ϕ ( x ) ∥ ≤ 1 for all x ∈ X and that ∥ θ j ⋆ ∥ ≤ B , which implies | r ( x ) | ≤ B for all x ∈ X .

We will model this problem in the language of model selection where a meta algorithm aims to optimize the unknown reward function by relying on a number of base learners. In order to interact with the environment the meta algorithm selects an agent that in turn selects an action. In our setting we thinking of each of these M feature maps as controlled by a base agent running its own algorithm. Base agent j uses the feature map ϕ j for modeling the reward. At step t of the bandit problem, each agent j is given access to the full history H t -1 := { ( x 1 , y 1 ) , . . . , ( x t -1 , y t -1 ) } , and uses it to locally estimate the reward as ˆ β ⊤ t -1 ,j ϕ j ( · ) , where ˆ β t -1 ,j ∈ R d is the estimated coefficients vector. The agent then uses this estimate to develop its action selection policy p t,j ∈ M ( X ) . Here, M denotes the space of probability measures defined on X . The condition on existence of j ⋆ will ensure that there is at least one agent which is using a correct model for the reward, and thereby can solve the bandit problem if executed in isolation. We refer to agent j ⋆ as the oracle agent.

Our goal is to find a sample-efficient strategy for iterating over the agents, such that their suggested actions maximize the cumulative reward, achieved over any horizon n ≥ 1 . This is equivalent to minimizing the cumulative regret R ( n ) = ∑ n t =1 r ( x ∗ ) -r ( x t ) , where x ∗ is a global maximizer of the reward function. We neither fix n , nor assume knowledge of it.

4 Method

As warm-up, consider an example with deterministic agents, i.e., when p t,j is a Dirac measure on a specific action x t,j . Suppose it was practically feasible to draw the action suggested by every agent and observe the corresponding reward vector r t = ( r t,j ) j M =1 . In this case, model selection becomes a full-information online optimization problem, and we can design a minimax optimal algorithm as follows. We assign a probability distribution q t = ( q t,j ) j M =1 to the models, and update it such that the overall average return ∑ n t =1 q ⊤ t r t is competitive to the best agent's average return ∑ n t =1 r t,j ⋆ . At every step, we update q t +1 ,j ∝ exp( ∑ t s =1 r s,j ) , since such exponential weighting is known to lead to an optimal solution for this classic online learning problem [Cesa-Bianchi and Lugosi, 2006]. In our setting however, the agents are stochastic, and we do not have access to the full r t vector.

We propose the A nytime EXP onential weighting algorithm based on L asso reward estimates (ALEXP), summarized in Algorithm 1. At step t we first sample an agent j t , and then sample an action x t according to the agent's policy p t,j t . Let ∆ M denote the M -dimensional probability simplex. We maintain a probability distribution q t ∈ ∆ M over the agents, and update it sequentially as we accumulate evidence on the performance of each agent. Ideally, we would have adjusted q t,j according to the average return of model j , that is, E x ∼ p t,j r ( x ) . However, since r is unknown, we estimate the average reward with some ˆ r t,j . We then update q t for the next step via,

for all j = 1 , . . . , M , where η t is the learning rate, and controls the sensitivity of the updates. This rule allows us to imitate the full-information example that we mentioned above. By utilizing ˆ r t,j and hallucinating feedback from all agents, we can reduce the probability of selecting a badly performing agent, without ever having sampled them (c.f. Fig. 4). It remains to design the estimator ˆ r t,j . We concatenate all feature maps, and, knowing that many features are redundant, use a sparsity inducing estimator over the resulting coefficients vector. Mainly, let θ = ( θ 1 , . . . , θ M ) ∈ R Md be the concatenated coefficients vector. We then solve

where Φ t = [ ϕ ⊤ ( x s )] s ≤ t ∈ R t × Md is the feature matrix, y t ∈ R t is the concatenated reward vector, and λ t is an adaptive regularization parameter. Problem (1) is the online variant of the group Lasso [Lounici et al., 2011]. The second term is the loss is the mixed (2 , 1) -norm of θ , which can be seen as the ℓ 1 -norm of the vector ( ∥ θ 1 ∥ , . . . , ∥ θ M ∥ ) ∈ R M . This norm induces sparsity at the group level, and therefore, the sub-vectors ˆ θ t,j ∈ R d that correspond to redundant feature maps are expected to be 0 , i.e. the null vector. We then estimate the average return of each model by simply taking an expectation ˆ r t,j = E x ∼ p t +1 ,j [ ˆ θ ⊤ t ϕ ( x )] . This quantity is the average return of the agent's policy p t +1 ,j , according to our Lasso estimator. In Section 5.2 we explain why the particular choice of Lasso is crucial for obtaining a log M rate for the regret.

For action selection, with probability γ t , we sample agent j with probability q t,j and draw x t ∼ p t,j as per suggestion of the agent. With probability 1 -γ t , we choose the action according to some exploratory distribution π ∈ M ( X ) which aims to sample informative actions. This can be any design where supp( π ) = X . We mix p t,j with π , to collect sufficiently diverse data for model selection. We are not restricting the agents' policy, and therefore can not rely on them to explore adequately. In Theorem 1, we choose a decreasing sequence of ( γ t ) t ≥ 1 the probabilities of exploration at step t ≥ 1 , since less exploration will be required as data accumulates. To conclude, the action selection policy of ALEXP is formally described as the mixture

5 Main Results

For the regret guarantee, we consider specific choices of base agents and exploratory distribution. Our analysis may be extended to include other policies, since ALEXP can be wrapped around any bandit agent that is described by some p t,j , and allows for random exploration with any distribution π .

Base Agents. We assume that the oracle agent has either a UCB [Abbasi-Yadkori et al., 2011] or a GREEDY [Auer et al., 2002a] policy, and all other agents are free to choose any arbitrary policy. Similar treatment can be applied to cases where the oracle uses other (sublinear) polices for solving linear or contextual bandits [e.g., Thompson, 1933, Kaufmann et al., 2012, Agarwal et al., 2014]. In either case, agent j ⋆ calculates a ridge estimate of the coefficients vector based on the history H t

Here Φ t,j ⋆ ∈ R t × d is the feature matrix, where each row s is ϕ ⊤ j ⋆ ( x s ) and ˜ λ is the regularization constant. Then at step t , a GREEDY oracle suggests the action which maximizes the reward estimate ˆ β ⊤ t -1 ,j ⋆ ϕ j ⋆ ( x ) , and a UCB oracle queries arg max u t -1 ,j ⋆ ( x ) where u t -1 ,j ⋆ ( · ) is the upper confidence bound that this agent calculates for r ( · ) . Proposition 21 shows that the sequence ( u t,j ⋆ ) t ≥ 1 is in fact an anytime valid upper bound for r over the entire domain.

Exploratory policy. Performance of ALEXP depends on the quality of the samples that π suggests. The eigenvalues of the covariance matrix Σ( π, ϕ ) := E x ∼ π ϕ ( x ) ϕ ⊤ ( x ) reflect how diverse the data is, and thus are a good indicator for data quality. van de Geer and Bühlmann [2009] present a survey on the notions of diversity defined based on these eigenvalues. Let λ min ( A ) denote the minimium eigenvalue of a matrix A . Similar to Hao et al. [2020], we assume that π is the maximizer of the problem below and present our regret bounds in terms of

Algorithm 1 ALEXP

which is greater than zero under the conditions specified in our problem setting. Prior works in the sparse bandit literature all require a similar or stronger assumption of this kind, and Table 2 gives an overview. Alternatively, one can work with an arbitrary π , e.g., Unif( X ) , as long as λ min (Σ( π, ϕ )) is bounded away from zero. Appendix C.1 reviews some configurations of ( ϕ , X , π ) that lead to a nonzero minimum eigenvalue, and Corollary 12 bounds the regret of ALEXP with uniform exploration.

For this choice of agents and exploratory distribution, Theorem 1 presents an informal regret bound. Here, we have used the O notation, and only included the fastest growing terms. The inequality is made exact in Theorem 14, up to constant multiplicative factors.

Theorem 1 (Cumulative Regret of ALEXP, Informal) . Let δ ∈ (0 , 1] and set π to be the maximizer of (2) . Choose learning rate η t = O ( C min t -1 / 2 /C ( M,δ,d )) , exploration probability γ t = O ( t -1 / 4 ) and Lasso regularization parameter λ t = O ( C min t -1 / 2 C ( M,δ,d )) , where

Then ALEXP satisfies the cumulative regret

simultaneously for all n ≥ 1 , with probability greater than 1 -δ .

In this bound, the first term is the regret incurred at exploratory steps (when α t = 1 ), the second term is due to the estimation error of Lasso (i.e., || θ -ˆ θ t || ), and the third term is the regret of the exponential weights sub-algorithm. The fourth term, is the regret bound for the oracle agent j ⋆ , when run within the ALEXP framework. It does not depend on the agent's policy (greedy or optimistic), and is worse than the minimax optimal rate of √ nd log n . This is because the oracle is suggesting actions based on the history H t , which includes uninformative action-reward pairs queried by other, potentially misspecified, agents. In Corollary 12, we provide a regret bound independent of C min , for orthogonal feature maps, and show that the same O (max { √ n log 3 M,n 3 / 4 √ log M } ) rate may be achieved even with the simple choice π = Unif( X ) .

5.1 Proof Sketch

The regret is caused by two sources: selecting a sub-optimal agent, and an agent selecting a suboptimal action. Accordingly, for any j ∈ 1 , . . . , M , we decompose the regret as

The first term shows the cumulative regret of agent j , when run within ALEXP. The second term evaluates the received reward against the cumulative average reward of model j . We bound each term separately.

Virtual Regret. The first term ˜ R j ( n ) := ∑ n t =1 r ( x ⋆ ) -r t,j compares the suggestion of agent j , against the optimal action. We call it the virtual regret since the sequence ( x t,j ) t ≥ 1 of the actions suggested by model j are not necessarily selected by the meta algorithm, unless j t = j . This regret is merely a technical tool, and not actually realized when running ALEXP. The virtual regret of the oracle agent may still be analyzed using standard techniques for linear bandits, e.g., Abbasi-Yadkori et al. [2011], however we need to adapt it to take into account a subtle difference: The confidence sequence of model j ⋆ is constructed according to the true sequence of actions ( x t ) t ≥ 1 , while its virtual regret is calculated based on the virtual sequence ( x t,j ⋆ ) t ≥ 1 , which the model suggests. The two sequences only match at the steps when model j ⋆ is selected. Adapting the analysis of Abbasi-Yadkori et al. [2011] to this subtlety, we obtain in Lemma 15 that with probability greater than 1 -δ , simultaneously for all n ≥ 1 ,

Model Selection Regret. The second term in (3) is the model selection regret, R ( n, j ) := ∑ n t =1 r t,j -r ( x t ) , which evaluates the chosen action by ALEXP against the suggestion of the j -th agent. Our analysis relies on a careful decomposition of R ( n, j ) ,

We bound away each term in a modular manner, until we are left with the regret of the standard exponential weights algorithm. The terms (I) and (III) are controlled by the bias of the Lasso estimator, and are O ( n 3 / 4 C ( M,δ,d )) (Lemma 19). The last term (IV) is zero in expectation, and reflects the deviation of r ( x t ) from its mean. We observe that the summands form a bounded Martingale difference sequence, and their sum grows as O ( √ n ) (Lemma 18). Term (II) is the regret of our online optimizer, which depends on the variance of the Lasso estimator. We bound this term with O ( √ nC ( M,δ,d ) log M ) , by first conducting a standard anytime analysis of exponential weights (Lemma 17), and then incorporating the anytime Lasso variance bound (Lemma 20). We highlight that neither of the above steps require assumptions about the base agents. Combining these steps, Lemma 16 establishes the formal bound on the model selection regret.

Anytime Lasso. We develop novel confidence intervals for Lasso with history-dependent data, which are uniformly valid over an unbounded time horizon. This result may be of independent interest in applications of Lasso for online learning or sequential decision making. Here, we use these confidence intervals to bound the bias and variance terms that appear in our treatment of the model selection regret. The width of the Lasso confidence intervals depends on the quality of feature matrix Φ t , often quantified by the restricted eigenvalue property [Bickel et al., 2009, van de Geer and Bühlmann, 2009, Javanmard and Montanari, 2014]:

Definition 2. For the feature matrix Φ t ∈ R t × dM we define κ (Φ t , s ) for any 1 ≤ s ≤ M as

Our analysis is in terms of this quantity, and Lemma 8 explains the connection between κ (Φ t , s ) and C min as defined in (2), particularly that κ (Φ t , 2) is also positive with a high probability.

Theorem 3 (Anytime Lasso Confidence Sequences) . Consider the data model y t = θ ⊤ ϕ ( x t ) + ε t for all t ≥ 1 , where ε t is i.i.d. zero-mean sub-Gaussian noise, and ( x t ) t ≥ 1 is ( F t ) t ≥ 1 -predictable, where F t := ( x 1 , . . . , x t , ε 1 , . . . , ε t -1 ) . Then the solution of (1) guarantees

if the regularization parameter satisfies for all t ≥ 1

Our confidence bound enjoys the same rate as Lasso with fixed (offline) data, up to log log d factors. We prove this theorem by constructing a self-normalized martingale sequence based on the ℓ 2 -norm of the empirical process error ( Φ ⊤ t ε t ). We then apply the 'stitched' time-uniform boundary of Howard et al. [2021]. Appendix B elaborates on this technique. Previous work on sparse linear bandits also include analysis of Lasso in an online setting, when x t is F t measurable. Cella and Pontil [2021] imitate offline analysis and then apply a union bound over the time steps, which multiplies the width by log n and requires knowledge of the horizon. Bastani and Bayati [2020] also rely on knowledge of n and employ a scalar-valued Bernstein inequality on ℓ ∞ -norm of the empirical process error, which inflates the width of the confidence sets by a factor of √ d . We work directly on the ℓ 2 -norm, and use a curved boundary for sub-Gamma martingale sequences, which according to Howard et al. [2021] is uniformly tighter than a Bernstein bound, especially for small t .

5.2 Discussion

In light of Theorem 1 and Theorem 3, we discuss some properties of ALEXP.

Sparse EXP4. Our approach presents a new connection between online learning and highdimensional statistics. The rule for updating the probabilities in ALEXP is inspired by the exponential weighting for Exploration and Exploitation with Experts (EXP4) algorithm, which was proposed by Auer et al. [2002b] and has been extensively studied in the adversarial bandit and learning with expert advice literature [e.g., McMahan and Streeter, 2009, Beygelzimer et al., 2011]. EXP4 classically uses importance-weighted (IW) or ordinary least squares (LS) estimators to estimate the return r t,j , both of which are unbiased but high-variance choices [Bubeck et al., 2012]. In particular, in our linearly parametrized setting, the variance of IW and LS scales with Md , which will lead to a poly( M ) regret. However, it is known that introducing bias can be useful if it reduces the variance [Zimmert and Lattimore, 2022]. For instance, EXP3-IX [Kocák et al., 2014] and EXP4-IX [Neu, 2015] construct a biased IW estimator. Equivalently, others craft regularizers for the reward of the online optimizer, seeking to improve the bias-variance balance [e.g., Abernethy et al., 2008, Bartlett et al., 2008, Abernethy and Rakhlin, 2009, Bubeck et al., 2017, Lee et al., 2020, Zimmert and Lattimore, 2022]. A key technical observation in this work is that our online Lasso estimator leads EXP4 to achieve sublinear regret which depends logarithmically on M . This is due to the fact that while the estimator itself is Md -dimensional, its bias squared and variance scale with √ d log M . To the best of our knowledge, this work is first to instantiate the EXP4 algorithm with a sparse low-variance estimator.

Adaptive and Anytime. To estimate the reward, prior work on sparse bandits commonly emulate the Lasso analysis on offline data or on a martingale sequence with a known length [Hao et al., 2020, Bastani and Bayati, 2020]. These works require a long enough sequence of exploratory samples, and knowledge of the horizon to plan this sequence. ALEXP removes both of these constraints, and presents a fully adaptive algorithm. Crucially, we employ the elegant martingale bounds of Howard et al. [2020] to present the first time-uniform analysis of the Lasso with history-dependent data (Theorem 3). This way we can use all the data points and explore with a probability which vanishes at a O ( t -1 / 4 ) rate. Our anytime confidence bound for Lasso, together with the horizon-independent analysis of the exponential weights algorithm, also allows ALEXP to be stopped at any time with valid guarantees.

Rate Optimality. For M ≫ n , we obtain a O ( √ n log 3 M ) regret, which matches the rate conjectured by Agarwal et al. [2017]. However, if M is comparable to n or smaller, our regret scales with

0

25

50

75

100

t

0

5

10

15

20

Regret

p

=10

s

=2

Lightly Correlated

0

25

50

75

100

t

0

5

10

15

20

p

=10

s

=8

Highly Correlated

0

25

50

75

100

t

0

5

10

15

20

p

=10

s

=3

Many Models

Oracle UCB

Naive UCB

ETC

ETS

Corral

O ( n 3 / 4 √ log M ) , and while it is still sublinear and scales logarithmically with M , the dependency on n is sub-optimal. This may be due to the conservative nature of our model selection analysis, during which we do not make assumptions about the dynamics of the base agents. Therefore, to ensure sufficiently diverse data for successful model selection, we need to occasionally choose exploratory actions with a vanishing probability of γ t . We conjecture that this is avoidable, if we make more assumptions about the agents, e.g., that a sufficient number of agents can achieve sublinear regret if executed in isolation. Banerjee et al. [2023] show that the data collected by sublinear algorithms organically satisfies a minimum eigenvalue lowerbound, which may also be sufficient for model selection. We leave this as an open question for future work.

6 Experiments

Experiment Setup. We create a synthetic dataset based on our data model (Section 3), and choose the domain to be 1 -dimensional X = [ -1 , 1] . As a natural choice of features, we consider the set of degreep Legendre polynomials, since they form an orthonormal basis for L 2 ( X ) if p grows unboundedly. We construct each feature map, by choosing s different polynomials from this set, and therefore obtaining M = ( p +1 s ) different models. More formally, we let ϕ j ( x ) = ( P j 1 ( x ) , . . . , P j s ( x )) ∈ R s where { j 1 , . . . , j s } ⊂ { 0 , . . . , p } and P j ′ denotes a degree j ′ Legendre polynomial. To construct the reward function, we randomly sample j ⋆ from [ M ] , and draw θ j ⋆ from an i.i.d. standard gaussian distribution. We then normalize || θ j ⋆ || = 1 . When sampling from the reward, we add Gaussian noise with standard deviation σ = 0 . 01 . Figure 5 in the appendix shows how the random reward functions may look. For all experiments we set n = 100 , and plot the cumulative regret R ( n ) averaged over 20 different random seeds, the shaded areas in all figures show the standard error across these runs.

Algorithms. We perform experiments on two UCB algorithms, one with oracle knowledge of j ⋆ , and a naive one which takes into account all M feature maps. We run Explore-then-Commit (ETC) by Hao et al. [2020], which explores for a horizon of n 0 steps, performs Lasso once, and then selects actions greedily for the remaining steps. As another baseline, we introduce Explore-then-Select (ETS) that explores for n 0 steps, performs model selection using the sparsity pattern of the Lasso estimator. For the remaining steps, the policy switches to UCB, calculated based on the selected features. Performance of ETC and ETS depends highly on n 0 , so we tune this hyperparameter separately for each experiment. We also run CORRAL as proposed by Agarwal et al. [2017], with UCB agents similar to ALEXP. We tune the hyper-parameters of CORRAL as well. To initialize ALEXP we set the rates of λ t , γ t and η t according to Theorem 1, and perform a light hyper-parameter tuning to choose the scaling constants. We have included the details and results of our hyper-parameter tuning in Appendix F.1. To solve (1), we use CELER, a fast solver for the group Lasso [Massias et al., 2018]. Every time UCB policy is used, we set the exploration coefficient β t = 2 , and every time exploration is required, we sample according to π = Unif( X ) . Appendix F includes the pseudo-code for all baseline algorithms. 1

Easy vs. Hard Cases. We construct an easy problem instance, where s = 2 , p = 10 , and thus M = 55 . Models are lightly correlated since each two model can have at most one Legendre polynomial in common. We also generate an instance with highly correlated feature maps where s = 8 and p = 10 , which will be a harder problem, since out of the total M = 55 models, there are 36 models which have at least 6 Legendre polynomials in common with the oracle model j ⋆ . Figure 1 shows that not only ALEXP is not affected by the correlations between the models, but also it achieves

0

50

100

t

0.50

0.75

1.00

1.25

1.50

Regret

Oracle UCB

0

50

100

t

0

5

10

15

Naive UCB

0

50

100

t

0

5

10

15

ETC

0

50

100

t

0.5

1.0

1.5

2.0

2.5

0

50

100

t

0

20

40

60

Corral

M

=45

M

=55

M

=66

M

=78

M

=91

0

50

100

150

(a)

0.00

0.02

0.04

0.06

0.08

0.10

qt

=20

MS Distribution

0

50

100

(b)

0.000

0.025

0.050

0.075

0.100

0.125

0.150

Mt

M

Ratio of Visited Agents

0

50

100

(c)

0.00

0.05

0.10

0.15

0.20

0.25

qt

,

j

Prob. of Oracle Agent

a performance competitive to the oracle in both cases, implying that our exponential weights technique for model selection is robust to choice of features. ETC and ETS rely on Lasso for model selection, which performs poorly in the case of correlated features. CORRAL uses log-barrier-OMD with an importance-weighted estimator, which has a significantly high variance. The curve for CORRAL in Figures 1 and 2 is cropped since the regret values get very large. Figure 6 shows the complete results. We construct another hard instance (Fig. 2), where the model class is large ( s = 3 , p = 10 , M = 165 ). ALEXP continues to outperform all baselines with a significant gap. It is clear in the regret curves how explore-then-commit style algorithms are inherently horizon-dependent, and may exhibit linear regret, if stopped at an arbitrary time. This is not an issue with the other algorithms.

Scaling with M . Figure 3 shows how well the algorithms scale as M grows. For this experiment we set s = 2 and change p ∈ { 9 , . . . , 13 } . While increasing M hardly affects ALEXP and Oracle UCB, other baselines become less and less sample efficient.

Learning Dynamics of ALEXP. Figure 4 gives some insight into the dynamics of ALEXP when M = 165 . In particular, it shows how ALEXP can rule out sub-optimal agents without ever having queried them. Figure (a) shows the distribution q t , at t = 20 which is roughly equal to the optimal n 0 for ETC in this configuration. The oracle model j ⋆ is annotated with a star, and has the highest probability of selection. We observe that, already at this time step, more than 80% of the agents are practically ruled out, due to small probability of selection. However, according to Figure (b), which shows M t the total number of visited models, less than 10% of the models are queried at t = 20 . This is the key practical benefit of ALEXP compared to black-box algorithms such as CORRAL. Lastly, Figure (c) shows how q t,j ⋆ the probability of selecting the oracle agent changes with time. While this probability is higher than that of the other agents, Figure (c) shows that q t,j ⋆ is not exceeding 0 . 25 , therefore there is always a probability of greater than 0 . 75 that we sample another agent, making ALEXP robust to hard problem instances where many agents perform efficiently. We conclude that ALEXP seems to rapidly recognize the better performing agents, and select among them with high probability.

7 Conclusion

We proposed ALEXP, an algorithm for simultaneous online model selection and bandit optimization. As a first, our approach leads to anytime valid guarantees for model selection and bandit regret, and does not rely on a priori determined exploration schedule. Further, we showed how the Lasso can be used together with the exponential weights algorithm to construct a low-variance online learner. This new connection between high-dimensional statistics and online learning opens up avenues for future research on high-dimensional online learning. We established empirically that ALEXP has favorable exploration-exploitation dynamics, and outperforms existing baselines. We tackled the open problem of Agarwal et al. [2017], and showed that log M dependency for regret is achievable for linearly parametrizable rewards. This problem remains open for more general, non-parametric reward classes.

Acknowledgments

We thank Jonas Rothfuss and Miles Wang-Henderson for their valuable suggestions regarding the writing. We thank Scott Sussex and Felix Schur for their thorough feedback. This research was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and Innovation Program Grant agreement no. 815943. Nicolas Emmenegger was supported by the Zeno Karl Schindler Foundation and the Swiss Study Foundation. Aldo Pacchiano would like to thank the support of the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This work was supported in part by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems , 2011.

Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Online-to-confidence-set conversions and application to sparse stochastic bandits. In Artificial Intelligence and Statistics . PMLR, 2012.

Jacob Abernethy and Alexander Rakhlin. Beating the adaptive bandit with high probability. In 2009 Information Theory and Applications Workshop . IEEE, 2009.

Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. An efficient algorithm for bandit linear optimization. In Conference on Learning Theory , 2008.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning . PMLR, 2014.

Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory . PMLR, 2017.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Foundations of Computer Science . IEEE, 1995.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 2002a.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 2002b.

Debangshu Banerjee, Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Exploration in linear bandits with rich action sets and its implications for inference. In International Conference on Artificial Intelligence and Statistics . PMLR, 2023.

Peter L Bartlett, Varsha Dani, Thomas P Hayes, Sham M Kakade, Alexander Rakhlin, and Ambuj Tewari. High-probability regret bounds for bandit online linear optimization. In Conference on Learning Theory , 2008.

Hamsa Bastani and Mohsen Bayati. Online decision making with high-dimensional covariates. Operations Research , 2020.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In International Conference on Artificial Intelligence and Statistics , 2011.

Peter J. Bickel, Ya'acov Ritov, and Alexandre B. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics , 2009.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 2012.

Sébastien Bubeck, Yin Tat Lee, and Ronen Eldan. Kernel-based methods for bandit convex optimization. In ACM SIGACT Symposium on Theory of Computing , pages 72-85, 2017.

Peter Bühlmann and Sara Van De Geer. Statistics for high-dimensional data: methods, theory and applications . Springer Science & Business Media, 2011.

Alexandra Carpentier and Rémi Munos. Bandit theory meets compressed sensing for high dimensional stochastic linear bandit. In Artificial Intelligence and Statistics . PMLR, 2012.

Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Conference on Uncertainty in Artificial Intelligence , 2021.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning . PMLR, 2020.

Dylan J Foster, Satyen Kale, Mehryar Mohri, and Karthik Sridharan. Parameter-free online learning via model selection. Advances in Neural Information Processing Systems , 2017.

Dylan J Foster, Akshay Krishnamurthy, and Haipeng Luo. Model selection for contextual bandits. Advances in Neural Information Processing Systems , 32, 2019.

Sébastien Gerchinovitz. Sparsity regret bounds for individual sequences in online linear regression. In Conference on Learning Theory , 2011.

Apostolos Giannopoulos. Notes on isotropic convex bodies, October 2003.

Javier Gonzalez, Joseph Longworth, David C James, and Neil D Lawrence. Bayesian optimization for synthetic gene design. arXiv preprint , 2015.

Botao Hao, Tor Lattimore, and Mengdi Wang. High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems , 2020.

David Haussler, Jyrki Kivinen, and Manfred K Warmuth. Sequential prediction of individual sequences under general loss functions. IEEE Transactions on Information Theory , 1998.

Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform chernoff bounds via nonnegative supermartingales. Probability Surveys , 2020.

Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics , 2021.

Mahdi Imani, Seyede Fatemeh Ghoreishi, Douglas Allaire, and Ulisses M Braga-Neto. Mfbo-ssm: Multi-fidelity bayesian optimization for fast inference in state-space models. In Proceedings of the AAAI Conference on Artificial Intelligence , 2019.

Kyoungseok Jang, Chicheng Zhang, and Kwang-Sung Jun. Popart: Efficient sparse regression and experimental design for optimal sparse linear bandits. In Advances in Neural Information Processing Systems , 2022.

Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for highdimensional regression. The Journal of Machine Learning Research , 2014.

Mohammad Reza Karimi, Nezihe Merve Gürel, Bojan Karlaš, Johannes Rausch, Ce Zhang, and Andreas Krause. Online active model selection for pre-trained classifiers. In International Conference on Artificial Intelligence and Statistics . PMLR, 2021.

Parnian Kassraie, Jonas Rothfuss, and Andreas Krause. Meta-learning hypothesis spaces for sequential decision-making. In International Conference on Machine Learning . PMLR, 2022.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics . PMLR, 2012.

Gi-Soo Kim and Myunghee Cho Paik. Doubly-robust lasso bandit. Advances in Neural Information Processing Systems , 32, 2019.

Tomáš Kocák, Gergely Neu, Michal Valko, and Rémi Munos. Efficient learning by implicit exploration in bandit problems with side observations. Advances in Neural Information Processing Systems , 2014.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in Neural Information Processing Systems , 2020.

Wenjie Li, Adarsh Barik, and Jean Honorio. A simple unified framework for high dimensional bandit problems. In International Conference on Machine Learning . PMLR, 2022.

Xuefeng Liu, Fangfang Xia, Rick L Stevens, and Yuxin Chen. Cost-effective online contextual model selection. arXiv preprint , 2022.

Karim Lounici, Massimiliano Pontil, Sara Van De Geer, and Alexandre B Tsybakov. Oracle inequalities and optimal inference under group sparsity. The annals of statistics , 2011.

Haipeng Luo, Mengxiao Zhang, Peng Zhao, and Zhi-Hua Zhou. Corralling a larger band of bandits: A case study on switching regret for linear bandits. In Conference on Learning Theory . PMLR, 2022.

Odalric-Ambrym Maillard and Rémi Munos. Adaptive bandits: Towards the best history-dependent strategy. In International Conference on Artificial Intelligence and Statistics . JMLR Workshop and Conference Proceedings, 2011.

Mathurin Massias, Alexandre Gramfort, and Joseph Salmon. Celer: a fast solver for the lasso with dual extrapolation. In International Conference on Machine Learning , 2018.

Brendan McMahan and Matthew Streeter. Tighter bounds for multi-armed bandits with expert advice. In Proceedings of the 22nd Annual Conference on Learning Theory , 2009.

Ahmadreza Moradipari, Berkay Turan, Yasin Abbasi-Yadkori, Mahnoosh Alizadeh, and Mohammad Ghavamzadeh. Feature and parameter selection in stochastic linear bandits. In International Conference on Machine Learning . PMLR, 2022.

Vidya Muthukumar, Mitas Ray, Anant Sahai, and Peter Bartlett. Best of many worlds: Robust model selection for online supervised learning. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , 2019.

Gergely Neu. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems , 2015.

Min-hwan Oh, Garud Iyengar, and Assaf Zeevi. Sparsity-agnostic lasso bandit. In International Conference on Machine Learning . PMLR, 2021.

Aldo Pacchiano, My Phan, Yasin Abbasi Yadkori, Anup Rao, Julian Zimmert, Tor Lattimore, and Csaba Szepesvari. Model selection in contextual stochastic bandit problems. Advances in Neural Information Processing Systems , 2020.

Aldo Pacchiano, Drausin Wulsin, Robert A Barton, and Luis Voloch. Neural design for genetic perturbation experiments. arXiv preprint , 2022.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems , 2007.

Felix Schur, Parnian Kassraie, Jonas Rothfuss, and Andreas Krause. Lifelong bandit optimization: No prior and no regret. In Conference on Uncertainty in Artificial Intelligence , 2023.

Adish Singla, Hamed Hassani, and Andreas Krause. Learning to interact with learning agents. In AAAI Conference on Artificial Intelligence , 2018.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 1933.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 1996.

Doniyor Ulmasov, Caroline Baroukh, Benoit Chachuat, Marc Peter Deisenroth, and Ruth Misener. Bayesian optimization with dimension scheduling: Application to biological systems. In Computer Aided Chemical Engineering . Elsevier, 2016.

Sara A. van de Geer and Peter Bühlmann. On the conditions used to prove oracle results for the Lasso. Electronic Journal of Statistics , 2009.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint . Cambridge university press, 2019.

Julian Zimmert and Tor Lattimore. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory . PMLR, 2022.

Contents of Appendix

A Extended Literature Review

The sparse linear bandit literature considers linear reward functions of the form θ ⊤ x , where x ∈ R p , however a sub-vector of size d is sufficient to span the reward function. This can be formulated as model selection among M = ( p d ) different linear parametrizations, where each ϕ j is a d -dimensional feature map. We present the bounds in terms of d and M for coherence with the rest of the text, assuming that M = O ( p ) , which is the case when d ≪ p .

Table 2 compares recent work on sparse linear bandits based on a number of important factors. In this table, the ETC algorithms follow the general format of exploring, performing parameter estimation once at t = n 0 , and then repeatedly suggesting the same action which maximizes ˆ θ ⊤ n 0 ϕ ( x ) . Explorethen-( ϵ )Greedy takes a similar approach, however it does not settle on ˆ θ n 0 , rather it continues to update the parameter estimate and select x t = arg max ˆ θ ⊤ t ϕ ( x ) . The UCB algorithms iteratively update upper confidence bound, and choose actions which maximize them. The regret bounds in Table 2 are simplified to the terms with largest rate of growth, the reader should check the corresponding papers for rigorous results . Some of the mentioned bounds depend on problem-dependent parameters (e.g. c K ), which may not be treated as absolute constants and have complicated forms. To indicate such parameters we use τ in Table 2, following the notation of Hao et al. [2020]. Note that τ varies across the rows of the table, and is just an indicator for existence of other terms.

Abbasi-Yadkori et al. [2012] use the SEQSEW online regression oracle [Gerchinovitz, 2011] for estimating the parameter vector, together with a UCB policy. The regression oracle is an exponential weights algorithm, which runs on the squared error loss. This subroutine, and thereby the algorithm proposed by Abbasi-Yadkori et al. [2012] are not computationally efficient, and this is believed to be unavoidable. This work considers the data-rich regime and shows R ( n ) = O ( √ dMn ) , matching the lower bound of Theorem 24.3 in Lattimore and Szepesvári [2020].

Carpentier and Munos [2012] assume that the action set is a Euclidean ball, and that the noise is directly added to the parameter vector, i.e. y t = x ⊤ ( θ + ε t ) . Roughly put, linear bandits with parameter noise are 'easier' to solve than stochastic linear bandits with reward noise, since the noise is scaled proportionally to the features x i and does less 'damage ' [Chapter 29.3 Lattimore and Szepesvári, 2020]. In this setting, Carpentier and Munos [2012] present a O ( d √ n ) regret bound.

Recent work considers contextual linear bandits, where at every step A t , a stochastic finite subset of size K from A , is presented to the agent. It is commonly assumed that members of A t are i.i.d., and the sampling distribution is diverse and time-independent. The diversity assumption is often in the

form of a restricted eigenvalue condition (Definition 2) on the covariance of the context distribution [e.g. in, Kim and Paik, 2019, Bastani and Bayati, 2020]. Li et al. [2022] require a stronger condition which directly assumes that λ min (Φ t ) the minimum eigenvalue of the empirical covariance matrix is lower bounded. This is generally not true, but may hold with high probability. Hao et al. [2020] assume that the action set spans R dM . We believe that this assumption is the weakest in the literature, and conjecture that it is necessary for model selection. If not met, the agent can not explore in all relevant directions, and may not identify the relevant features. Our diversity assumption is similar to Hao et al. [2020], adapted to our problem setting. Mainly, we consider reward functions which are linearly parametrizable, i.e. θ ⊤ ϕ ( x ) , as oppose to linear rewards, i.e. θ ⊤ x .

A key distinguishing factor between ALEXP and existing work on sparse linear bandit is that ALEXP is horizon-independent and does not rely on a forced exploration schedule. As shown on Table 2, majority of prior work relies either on an initial exploration stage, the length of which is determined according to n [e.g., Carpentier and Munos, 2012, Kim and Paik, 2019, Li et al., 2022, Hao et al., 2020, Jang et al., 2022], or on a hand crafted schedule, which is again designed for a specific horizon [Bastani and Bayati, 2020]. Oh et al. [2021], which analyzes K -armed contextual bandits, does not require explicit exploration, and instead imposes restrictive assumptions on the diversity of context distribution, e.g. relaxed symmetry and balanced covariance. Regardless, the regret bounds hold in expectation, and are not time-uniform.

B Time Uniform Lasso Analysis

We start by showing that the sum of squared sub-gaussian variables is a sub-Gamma process (c.f. Definition 22).

Lemma 4 (Empirical Process is sub-Gamma) . For t ≥ 1 , suppose ξ t are a sequence conditionally standard sub-Gaussians adapted to the filtration F t = σ ( ξ 1 , . . . , ξ t ) . Let v t ∈ R , and Z t := ξ 2 t -1 . Define the processes S t := ∑ t i =1 Z i v i and V t := 4 ∑ t i =1 v 2 i . Then ( S t ) ∞ t =0 is sub-Gamma with variance process ( V t ) ∞ t =0 and scale parameter c = 4max i ≥ 1 v i .

Proof of Lemma 4. By definition [c.f. Definition 1, Howard et al., 2021], S t is sub-Gamma if for each λ ∈ [0 , 1 /c ) , there exists a supermartingale ( M t ( λ )) ∞ t =0 w.r.t. F t , such that E M 0 = 1 and for all t ≥ 1 :

We show the above holds in equality by proving that the left hand side itself, is a supermartingale w.r.t. F t . We define, M t ( λ ) := exp { λS t -λ 2 V t / 2(1 -cλ ) } , therefore,

Note that Z t is F t -1 -measurable, conditionally centered and conditionally sub-exponential with parameters ( ν, α ) = (2 , 4) (c.f. Vershynin [2018, Lemma 2.7.6] and Wainwright [2019, Example 2.8]). Therefore, for λ < 1 /c ,

where the last inequality holds due to the fact that 0 ≤ 1 -cλ < 1 . Therefore,

for λ ∈ [0 , 1 /c )

We now construct a self-normalizing martingale sequence based on ℓ 2 -norm of the empirical process error term, and recognize that it is a sub-gamma process. We then employ our curved Bernstein bound Lemma 25 to control the boundary. This step will allow us to 'ignore' the empirical process error term later in the lasso analysis.

Lemma 5 (Empirical Process is dominated by regularization.) . Let

Then, for any 0 ≤ δ < 1 , the event A = ∩ j M =1 A j happens with probability 1 -δ , if for all t ≥ 1 ,

Proof of Lemma 5. This proof includes a treatment of the empirical process similar to Lemma B.1 in Kassraie et al. [2022], but adapts it to our time-uniform setting. Since ε i are zero-mean sub-gaussian variables, as driven in Lemma 3.1 [Lounici et al., 2011], it holds that

where ξ i are sub-gaussian variables with variance proxy 1 , scalar v i denotes the i -th eigenvalue of Φ t,j Φ ⊤ t,j /t with the concatenated vector v t = ( v 1 , . . . , v t ) , and

We can apply Lemma 25 to control the probability of event A c j by tuning λ . Mainly, for A c j to happen with probability less than δ/M , Lemma 25 states that the following must hold for all t ,

Recall that w.l.o.g. feature maps are bounded everywhere ∥ ϕ j ( · ) ∥ 2 ≤ 1 , and rank(Φ j ) ≤ d which allows for the following matrix inequalities,

Therefore,

For Eq. (4) to hold, is suffices that for all t ≥ 1 ,

Therefore, if λ t are chosen to satisfy the above inequality, each A c j happens with probability less than δ/M . Then by applying union bound, ∪ j M =1 A c j happens with probability less than δ .

Proof of Theorem 3 . The theorem statement requires that the regularization parameter λ t is chosen such that condition of Lemma 5 is met, and therefore event A happens with probability 1 -δ . Throughout this proof, which adapts the analysis of Theorem 3.1. Lounici et al. [2011] to the time-uniform setting, we condition on A happening, and later incorporate the probability.

Step 1. Let ˆ θ t be a minimizer of L and θ be the true coefficients vector, then L ( ˆ θ t ; H t , λ t ) ≤ L ( θ ; H t , λ t ) . Writing out the loss and re-ordering the inequality we obtain,

which is often referred to as the Basic inequality [Bühlmann and Van De Geer, 2011]. By CauchySchwarz and conditioned on event A ,

̸

then adding λ t ∑ M j =1 ∥ ∥ ∥ ˆ θ t,j -θ j ∥ ∥ ∥ 2 to both sides, applying the triangle inequality, and recalling from Section 3 that θ j = 0 for j = j ⋆ gives

Since each term on the left hand side is positive, then each is also individually smaller than the right hand side, and we obtain,

̸

Step 2. Consider a sequence ( c 1 , . . . , c k , . . . ) , where c 1 ≥ · · · ≥ c k . . . , then

Define J 1 = { j ⋆ } and J 2 = { j ⋆ , j ′ } where

̸

For any J ⊂ [ M ] the complementing set is denoted as J c = [ M ] \ J . For simplicity let c j = ∥ ∥ ∥ ˆ θ t,j -θ j ∥ ∥ ∥ 2 , and let π ( k ) denote the index of the k -th largest element of { c j : j ∈ J c 1 } . By definition of J c 2 we have,

which, in turn, gives

Step 3. On the other hand, due to (6), and by definition of J 2 it also holds that

From the theorem assumptions and Definition 2, we know that there exists 0 < κ (Φ t , 2) , therefore by Definition 2, the feature matrix Φ t satisfies,

From here, by applying (8) we get,

If λ t are chosen according to Lemma 5, event A and, in turn, the inequality above hold with probability greater than 1 -δ .

C Results on Exploration

In this section we present lower-bounds on the eigenvalues of the covariance matrix Φ t Φ ⊤ t , as it is later used in our regret analysis. In particular, we show that the feature matrix Φ t satisfies the restricted eigenvalue condition (Definition 2) required for valid Lasso confidence set (Theorem 3), and calculate a lower bound on κ (Φ t , 2) . The lower bound is later used by Lemma 19 and Lemma 20 to develop the model selection regret. We show this bound in three steps.

Equivalent to Definition 2, we write κ (Φ t , s ) = inf b ∈ Ξ s ∥ Φ t b ∥ 2 / √ t where

For simplicity in notation, we further define

Step I. Consider the exploratory steps at which α t = 1 . Let Φ π,t be a sub-matrix of Φ t where only rows from exploratory steps are included. Note that Φ π,t ∈ R t ′ × dM is a random matrix, where the number of rows t ′ are also random. We show that κ 2 (Φ t , s ) is lower bounded by κ 2 (Φ t,π , s ) .

Lemma 6. Suppose Φ π,t has t ′ rows. Then,

Proof of Lemma 6. Let Ψ ( t ) = ϕ ( x t ) ϕ ⊤ ( x t ) ∈ R dM × dM for all t = 1 , . . . , n . Note that Ψ ( t ) is positive semi-definite by construction. We have,

where the set T π contains the indices of the exploratory steps at which the action is selected according to π . Therefore,

where the last inequality holds due to Ψ ( s ) being PSD. Then we have,

While the number of rows of Φ π,t is a random variable, we continue to condition on the event that Φ π,t has t ′ rows, and investigate the distribution of its restricted eigenvalues.

Step II. The restricted eigenvalues of the exploratory submatrix are well bounded away from zero. Lemma 7. Let π be the solution to (2) , and s ∈ N . Suppose Φ π,t has t ′ rows. Then for all δ > 0 ,

where Σ = Σ( π, ϕ ) := E x ∼ π ϕ ( x ) ϕ ⊤ ( x ) and ˜ κ is defined in (10) .

Step III. Remains to combine the two above lemmas and incorporate a high probability bound on t ′ , showing that it is close to ∑ t s =1 γ s .

Lemma 8. There exist absolute constants C 1 , C 2 which satisfy,

if γ t = O ( t -1 / 4 ) . Let π be the solution to (2) , then it further holds that

The regret analysis of Hao et al. [2020] also relies on connecting κ (Φ t , s ) to C min , and for this, they use Theorem 2.4 of Javanmard and Montanari [2014]. This theorem states that there exists a problem-dependent constant C 1 for which κ 2 (Φ t , s ) ≥ C 1 C min with high probability, if t ≥ n 0 and roughly n 0 = O ( 3 √ n 2 log M ) . We highlight that Lemma 8, presents a lower-bound which holds for all t ≥ 1 , however this comes at the cost of getting a looser lower bound than the result of Javanmard and Montanari [2014] for the larger time steps t . In fact, due to the sub-optimal dependency of Lemma 8 on t , we later obtain sub-optimal dependency on the horizon for the case when n ≫ M . It is unclear to us if this rate can be improved without assuming knowledge of n , or that n ≥ n 0 .

For the last lemma in this section we show that the empirical sub-matrices Φ t,j are also bounded away from zero. This will be required later to prove Lemma 15.

Lemma 9 (Base Model λ min Bound) . Assume π is the maximizer of Eq. (2) . Then, with probability greater than 1 -δ , simultaneously for all j = 1 , . . . , M and t ≥ 1 ,

C.1 ALEXP with Uniform Exploration

We presented our main regret bound (Theorem 1) in terms of C min , which only depends on properties of the feature maps and the action domain. We give a lower-bound on C min for a toy scenario which corresponds to the problem of linear feature selection over convex action sets.

Proposition 10 (Invertible Features) . Suppose ϕ ( x ) := A x : R d → R d is an invertible linear map, and X ∈ R d is a convex body. Then,

where T is the transformation which maps X to an isotropic body.

The lower-bound of Proposition 10 is achieved by simply exploring via π = Unif( X ) . Inspired by Schur et al. [2023, Lemma E.13], we show that even for non-convex action domains and orthogonal feature maps, the uniform exploration yields a constant lower-bound on restricted eigenvalues.

Proposition 11 (Orthonormal Features) . Suppose ϕ j : X → R are chosen from an orthogonal basis of L 2 ( X ) , and satisfy ∥ ϕ i ∥ L 2 µ ( X ) / Vol ( X ) ≥ 1 . Then there exist absolute constants C 1 and C 2 for which the exploration distribution π = Unif( X ) satisfies

The d = 1 condition is met without loss of generality, by splitting the higher dimensional feature maps and introducing more base features, which will increase M . Moreover, the orthonormality condition is met by orthogonalizing and re-scaling the feature maps. Basis functions such as Legendre polynomials and Fourier features [Rahimi et al., 2007] satisfy these conditions.

By invoking Proposition 11, instead of Lemma 8 in the proof of Theorem 1, we obtain the regret of ALEXP with uniform exploration.

Corollary 12 (ALEXP with Uniform Exploration) . Let δ ∈ (0 , 1] . Suppose ϕ j : X → R are chosen from an orthogonal basis of L 2 ( X ) , and satisfy ∥ ϕ i ∥ L 2 µ ( X ) / Vol ( X ) ≥ 1 . Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. Choose η t = O (1 / √ tC ( M,δ,d )) and γ t = O ( t -1 / 4 ) and λ t = O ( C ( M,δ,d ) / √ t ) , then ALEXP with uniform exploration π = Unif( X ) attains the regret

with probability greater than 1 -δ , simultaneously for all n ≥ 1 . Here,

C.2 Proof of Results on Exploration

As an intermediate step, we consider the restricted eigenvalue property of the empirical covariance matrix. Given t ′ samples, the empirical estimate of Σ is

where x s are sampled according to π . We show that every entry of ˆ Σ t ′ is close to the corresponding entry in Σ , and later use it in the proofs of eigenvalue lemmas.

Lemma 13 (Anytime Bound for The Entries of Empirical Covariance Matrix) . Let ˆ Σ t ′ be the empirical covariance matrix corresponding to Σ( π, ϕ ) given t ′ samples. Then,

where d ∞ ( A,B ) := max i,j | A i,j -B i,j | .

Proof of Lemma 13. We show the element-wise convergence of Σ to ˆ Σ t for the ( i, j ) entry where i, j = 1 , . . . , dM . Consider the random sequence X s := Σ i,j -ϕ i ( x s ) ϕ j ( x s ) . We show that X 1 , . . . , X n satisfies conditions of Lemma 26. We first observe that

since by definition Σ i,j = E x ∼ π ϕ i ( x ) ⊤ ϕ j ( x ) . Moreover, we have normalized features ∥ ϕ ( · ) ∥ ≤ 1 , therefore, each entry ϕ i ( · ) ϕ j ( · ) is also bounded, yielding | X s | ≤ 2 . Then Lemma 26 implies that for all ˜ δ > 0 ,

Setting ˜ δ = δ/ ( dM ) and taking a union bound over all indices concludes the proof.

We are now ready to present the proofs to the lemmas in Appendix C.

Proof of Lemma 7. By (11) we have ˆ Σ t ′ = Φ ⊤ π,t Φ π,t t ′ , and thereby

Inspired by Lemma 10.1 in van de Geer and Bühlmann [2009], we show that element-wise closeness of matrices Σ and ˆ Σ t ′ (c.f. Lemma 13) implies closeness in ˜ κ :

where the last line holds due to Hölder's. Moreover, since b ∈ Ξ s , for any J ⊂ [ dM ] where | J | ≤ s it additionally holds that ∥ b J ∥ 2 ≤ 1 and

which gives,

′

with probability greater than 1 -δ , simultaneously for all t ≥ 1 .

Proof of Lemma 8. In Lemma 6 we showed that

where t ′ indicates the number of rows in the exploratory sub-matrix of Φ t . Recall that t ′ = ∑ t s =1 α s where α s are i.i.d Bernoulli random variables with success probability of γ s . Due to Lemma 24,

where

Due to Lemma 7, with probability greater than 1 -δ/ 2 the following holds for all t ≥ 1

where the second inequality holds with probability 1 -δ , by incorporating (13) and taking a union bound. For the rest of the proof and to keep the calculations simple, we ignore the values of the absolute constants. We use the notation g ( t ) = o ( f ( t )) to show that f ( t ) grows much faster than g ( t ) . More formally, if for every constant c there exists t 0 , where g ( t ) ≤ c | f ( t ) | for all t ≥ t 0 . Since γ s = O ( s -1 / 4 ) there exists C such that Γ t = Ct 3 / 4 , then it is straightforward to observe that there exists absolute constants ˜ C i which satisfy,

The last inequality holds since t -3 / 2 √ log log t = o ( t -5 / 8 √ log log t ) . The above chain of inequalities imply that there exist absolute constants C 1 , C 2 , for which

If π is chosen according to (2), then ˜ κ (Σ , 2) ≥ C min yielding the lemma's second argument.

Proof of Lemma 9. Fix j ∈ { 1 , . . . , M } , and construct the set

Note that Ξ 1 ,j ⊂ Ξ s . Therefore,

Moreover, by construction of Ξ 1 ,j we have for all b ∈ Ξ 1 ,j that Φ t b = Φ t,j b j , therefore,

From the above equations we conclude that λ min (Φ ⊤ t,j Φ t,j ) ≥ tκ 2 (Φ t , s ) , for all j = 1 , . . . , M . Therefore, using Lemma 8 we obtain that there exists C 1 , C 2 such that

Proof of Proposition 10. Since X is a convex body, then there exists an invertible map T , such that T ( X ) is an isotropic body [e.g. Proposition 1.1.1., Giannopoulos, 2003]. Then by definition, ¯ X ∼ Unif( T ( X )) is an isotropic distribution and Cov( ¯ X ) = I d [e.g., c.f. Chapter 3.3.5 Vershynin, 2018]. Since ϕ is linear and invertible, it may be written is as ϕ ( x ) = A x , where A is an invertible matrix. Therefore,

As for the minimum eigenvalue, suppose v ∈ R d and ∥ v ∥ = 1 , then

̸

Proof of Proposition 11. By the assumption of the proposition, for all i ∈ M

̸

and for all i = j ,

We use Σ = Σ( π, ϕ ) . For any b ∈ R Md where ∥ b ∥ ≤ 1 ,

Which implies,

By Lemma 8, there exist absolute constants C 1 and C 2 for which,

concluding the proof.

D Proof of Regret Bound

Theorem 14 (Anytime Regret, Formal) . Let δ ∈ (0 , 1] and π be the maximizer of (2) . Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. Suppose η t = O ( C min t -1 / 2 C ( M,δ,d )) and γ t = O ( t -1 / 4 ) and λ t = O ( C ( M,δ,d ) t -1 / 2 ) , then exists absolute constants C 1 , . . . , C 6 for which ALEXP attains the regret

with probability greater than 1 -δ , simultaneously for all n ≥ 1 . Here,

Our main regret bound is an immediate corollary of Lemma 15 and Lemma 16, considering the regret decomposition of (3).

Lemma 15 (Virtual Regret of the Oracle) . Let δ ∈ (0 , 1] and ˜ λ > 0 . Assume the oracle agent employs a UCB or a Greedy policy, as laid out in Section 5. If γ t = O ( t 1 / 4 ) , there exists an absolute constant C 1 for which with probability greater than 1 -δ , simultaneously for all n ≥ 1 ,

̸

Lemma 16 (Any-Time Model-Selection Regret, Formal) . Let δ ∈ (0 , 1] and π be the maximizer of (2) . Suppose η t = O ( C min / √ tC ( M,δ,d )) and γ t = O ( t -1 / 4 ) and λ t = O ( C ( M,δ,d ) / √ t ) , then exists absolute constants C i for which ALEXP attains the model selection regret

with probability greater than 1 -δ , simultaneously for all n ≥ 1 . Here,

D.1 Proof of Model Selection Regret

Our technique for bounding the model selection regret relies on a classic horizon-independent analysis of the exponential weights algorithm, presented in Lemma 17.

Lemma 17 (Anytime Exponential Weights Guarantee) . Assume η t ˆ r t,j ≤ 1 for all 1 ≤ j ≤ M and t ≥ 1 . If the sequence ( η t ) t ≥ 1 is non-increasing, then for all n ≥ 1 ,

for any arm k ∈ [ M ] .

Proof of Lemma 17. Define ˆ R t,i := ∑ t s =1 ˆ r s,i to be the expected cumulative reward of agent i after t steps. We rewrite for a fixed k

We focus on a single term in the second sum. For any t , we have

The inner expectation is over j , while the outer one is over i and therefore has no effect. Moreover,

where again, the expectation can be reintroduced to get the last line. Combining (15) and (16),

This transformation is at the core of many exponential weight proofs [Bubeck et al., 2012, Lattimore and Szepesvári, 2020]. We first bound the first term in (17):

where in (I) we use the fact that log( z ) ≤ z -1 and in (II) we use the fact that for x ≤ 1 , we have exp( x ) ≤ 1 + x + x 2 , and hence exp( x ) -1 -x ≤ x 2 . For the second term in (17), we will mirror the potential argument in Bubeck et al. [2012], but with a slightly different potential function. We expand the definition of q t :

where we define J t ( η ) = -1 η log 1 M ∑ M i =1 exp( η ˆ R t,i ) . We also define F t ( η ) = 1 η log 1 M ∑ M i =1 exp( -η ˆ R t,i ) . We observe the relation J ( η ) = F ( -η ) . From this, it follows that for any η , we have J ′ ( η ) = -F ′ ( -η ) ≤ 0 , by the argument in Bubeck et al. [2012, Theorem 3.1] that shows F ′ ( η ) ≥ 0 for any η .

Putting together the pieces Now, we can bound (17) by inputing (18) and (19):

With this, we rewrite (14) as

Potential manipulation We can do an Abel transformation on the sum of potentials in (20), namely obtaining

where we used that J 0 ( η ) = 0 . We know J ′ ( η ) ≤ 0 and so J is decreasing and since η t +1 ≤ η t , we have J ( η t +1 ) ≥ J ( η t ) or ( J t ( η t ) -J t ( η t +1 )) ≤ 0 , so that for any fixed k

where ( ∗ ) follows because exp is positive and -log is decreasing (notice that we drop M -1 terms from the sum). Plugging (21) into (20), we obtain

We expressed in Section 5.2, that the model selection regret of ALEXP, is closely tied to the bias and variance of the reward estimates ˆ r t,j . The following lemma formalizes this claim.

Lemma 18. (Anytime Generic regret bound) If η t is picked such that η t ˆ r t,j ≤ 1 for all 1 ≤ j ≤ M and 1 ≤ t almost surely, then Algorithm 1 satisfies with probability greater than 1 -2 δ/ 3 , that simultaneously for all n ≥ 1

where ω t,i = | r t,i -ˆ r t,i | .

Proof of Lemma 18. Let α t denote the Bernoulli random variable that is equal to 1 if at step t we select actions according to π and 0 otherwise. At each step t with α t = 1 ALEXP accumulates a regret of at most 2 B , since ∥ θ ∥ ∞ ≤ B and ∥ ϕ ( · ) ∥ ≤ 1 . We can decompose the regret as,

For the first term, by Lemma 24, we have

simultaneously for all n ≥ 1 , with probability 1 -δ 1 . Let ˆ r t := ∑ M j =1 q t,j ˆ r t,j . We may re-write the second term of the regret as follows,

We bound the second term on the right hand side, using Lemma 17

As for the third term,

It remains to bound the deviation term. For all t that satisfy α t = 0 , the action/model is selected according to q t,j , therefore the conditional expectation of r t can be written as

The sequence X t := r t -E t -1 r t is a martingale difference sequence adapted to the history H t , since for every t ≥ 1 ,

Since r t ≤ B , then X t ≤ 2 B almost surely, which allows for an application of anytime AzumaHoeffding (Lemma 26):

which, in turn, leads us to

simultaneously for all n ≥ 1 . We set δ 1 = δ 2 = δ/ 3 , take a union bound and put the terms together obtaining,

We upper bound the sum of last two terms to conclude the proof.

The next two lemmas bound the bias and variance terms which appear in Lemma 18.

Lemma 19 (Anytime Bound on the Bias Term) . If the regularization parameter of Lasso is chosen at every step as

and γ t = O ( t -1 / 4 ) , then with probability greater than 1 -δ , simultaneously for all n ≥ 1 ,

where

and C is an absolute constant.

Proof of Lemma 19. By the definition of the expected reward and its estimate,

We highlight that the Cauchy-Schwarz step may be refined. By further assuming that θ j ⋆ is bounded away from zero (also called the beta-min condition [Bühlmann and Van De Geer, 2011]) one can show that θ -ˆ θ t is a 2 -sparse vector. This will then allow one to only rely on boundedness of ∥ ϕ j ∥ rather than ∥ ϕ ∥ to derive the last inequality, and relax our assumption of ∥ ϕ ( · ) ∥ ≤ 1 to ∥ ϕ j ( · ) ∥ ≤ 1 for all j ∈ [ M ] . From Theorem 3, with probability greater than 1 -δ/ 2 simultaneously for all n ≥ 1 ,

where,

From Lemma 8, there exist absolute constants C 1 , C 2 for which,

Using Taylor approximation we observe that, 1 1 -x -1 = 1+ x -1 + o ( x -1 ) = O (1 + x -1 ) . Therefore, these exists absolute constant C 3 , C 4 , for which with probability greater than 1 -δ for all t ≥ 1

Lemma 20 (Anytime Bound on Variance Term) . Suppose λ t is chosen according to Lemma 19, γ t = O ( t -1 / 4 ) and η t = O ( C min t -1 / 2 /C ( M,δ,d )) . Then with probability greater than 1 -δ , the following holds simultaneously for all n ≥ 1 and t ≥ 1

where C i are absolute constants, and C ( M,δ,d ) is as defined in Lemma 19, up to constant factors.

Proof of Lemma 20. We start by upper bounding ˆ r t,j . For all j and t it holds that:

since ∥ θ ∥ 2 ≤ B . To bound the last term, we only need to invoke Theorem 3, which, in turn, will simultaneously bound ˆ r t,j for all j = 1 , . . . , M :

Which implies for all t ≥ 1 ,

For the last term, similar to the proof of Lemma 19 we have,

for some absolute constant C 1 . We treat the squared term similarly,

Note that the last inequality is not tight. This term will not be fastest growing term in the regret, so we have little motivation to bound it tightly. Therefore,

where C i are absolute constants.

Proof of Lemma 16 . We start by conditioning on the event E that η t is picked such that η t ˆ r t,j ≤ 1 for all t ≥ 1 and j = 1 , . . . , M . Then by application of Lemma 18 we get with probability greater than 1 -2 δ/ 3 ,

We invoke Lemma 19 and Lemma 20 with δ → δ/ 3 take a union bound, to bound the variance and ω t,i terms as well. These lemmas require one application of Theorem 3 to hold simultaneously and no additional union bound is required between them, since the randomness comes only from the confidence interval over ˆ θ t .

with probability greater than 1 -δ , conditioned on event E . Assuming that event E happens with probability 1 -2 δ , let B = B ( ˜ δ, M, d, n, B, σ, C min ) denote the right-hand-side of the regret inequality above.

By the chain rule we may write,

It remains to verify that event E is met with probability 1 -2 δ . Recall that η t = O ( C min / √ tC ( M,δ,d )) , and that from Lemma 8 with probability 1 -δ ,

Therefore, from Lemma 20, there exists C η such that η t = C η C min /B √ tC ( M,δ,d ) satisfying,

The proof is then finished by setting δ ← 3 δ (and updating the absolute constants).

D.2 Proof of Virtual Regret

Proposition 21. For any fixed ˜ λ > 0 , there exists an absolute constant C 1 such that

where

Moreover, for u t,j ⋆ ( · ) := ˆ β ⊤ t,j ⋆ ϕ j ⋆ ( · ) + ω ( t, δ, d ) ,

Proof of Proposition 21. Define for convenience V t = Φ ⊤ t,j ⋆ Φ t,j ⋆ + ˜ λ I . We first observe that

We can apply results from Abbasi-Yadkori et al. [2011] to get an anytime-valid confidence set. Their Theorem 2 asserts that with probability 1 -δ , for all t ≥ 1 we have 2

where

Clearly, V t ⪰ λ min ( V t ) I , and therefore with high probability,

uniformly over time. Our assumption is that ∥ ϕ j ( x ) ∥ ≤ 1 , and hence, denoting by ν i the eigenvalues of V t , the geometric-arithmetic mean inequality yields

Given that we can conclude that

We note that

Then due to Lemma 9, there exist absolute constants C 1 and C 2 such that for all t ≥ 1 ,

therefore, there exists C 3 and C 4 such that

with high probability for all t ≥ 1 . Setting

where C 5 is an absolute constant concludes the parametric confidence bound. The upper confidence bound then simply follows: for any x ∈ X

Proof of Lemma 15 . Using Proposition 21 and the Cauchy-Schwarz inequality we obtain,

with probability 1 -δ . If the agent selects actions greedily, then ˆ r t ( x ⋆ ) ≤ ˆ r t (˜ x t,j ) , and

since the feature map is normalized to satisfy ∥ ϕ j ( · ) ∥ ≤ 1 . If the agent selects actions optimistically according to the upper confidence bound of Proposition 21, then

which implies and therefore,

Then due to Proposition 21, with probability greater than 1 -δ , simultaneously for all n ≥ 1 ,

concluding the proof.

E Time-Uniform Concentration Inequalities

We will make use of the elegant concentration results in Howard et al. [2021], which analyzes the boundary of sub-Gamma processes.

Definition 22 (Sub-Gamma process) . Let ( S t ) ∞ t =0 and ( V t ) ∞ t =0 be real-valued processes adapted to ( F t ) ∞ t =1 with S 0 = V 0 = 0 and V t non-negative. We say that S t is sub-Gamma if for λ ∈ [0 , 1 /c ) , there exists a supermartingale ( M t ( λ )) ∞ t =0 w.r.t. F t , such that E M 0 = 1 and for all t ≥ 1 :

The following is a special case of Theorem 1 in Howard et al. [2021]. We have simplified it by making a few straightforward choices for the parameters used originally by Howard et al. [2021], which will yield an easier-to-use bound in our scenario.

Proposition 23 (Curved Boundary of Sub-Gamma Processes) . Let ( S t ) t ≥ 0 be sub-Gamma with scale parameter c and variance process ( V t ) t ≥ 0 . Define the boundary

for v > 0 , where ( x ) + = max(0 , x ) . Then,

Proof of Proposition 23. Suppose ξ ( · ) denotes the Riemann zeta function. Theorem 1 in Howard et al. [2021] states that if ( S t ) t ≥ 0 is a sub-Gamma process with variance process ( V t ) t ≥ 0 then the boundary

satisfies, where

and s, η ≥ 1 . Choosing s = 2 and η = e , we obtain ζ (2) = π 2 / 6 ≤ 2 . Furthermore, we have k 1 ≤ 3 2 and k 2 ≤ 3 2 . Then if v ′ ≥ 1 (which we will enforce by the construction v ′ = max(1 , v ) ), we compute

Therefore, we can upper bound (using our bounds on k 1 , k 2 )

Now, since the boundary is given by S α (max( v, 1)) and v ′ = max( v, 1) ≥ 1 we deduce that

is an any-time valid boundary.

Lemma 24 (Time-Uniform Two-sided Bernoulli) . Let X 1 , . . . , X s , . . . , X t be a martingale sequence of Bernoulli random variables with conditional mean γ s . Then for all δ > 0 ,

Proof of Lemma 24. By Proposition 23, we know that if S t is sub-Gamma with variance process V t and scale parameter c , then

where

By Howard et al. [2020], we know that if ( X t ) ∞ t =1 is a Bernoulli sequence, then S t = ∑ t s =1 ( X s -γ s ) is sub-Gamma with variance process V t = t and scale parameter c = 0 (hence, sub-Gaussian). This implies,

The above arguments also holds for the sequence Z s = -X s . Then taking a union bound and adjusting δ ← δ/ 2 concludes the proof.

Lemma 25 (Time-Uniform Bernstein) . Let ( ξ i ) ∞ i =1 be a sequence of conditionally standard subgaussian variables, where each ξ i is F i -1 = σ ( ξ 1 , . . . , ξ i ) measurable. Then, for v i ∈ R and δ ∈ (0 , 1]

where, v t = ( v 1 , . . . , v t ) ∈ R t and ω δ ( v ) := ( log log(4 ev 2 ) ) + +log(2 /δ ) .

Proof of Lemma 25. From Lemma 4, S t = ∑ t i =1 ( ξ 2 i -1) v i is sub-Gamma with variance process V t = 4 ∑ t i =1 v 2 i and c = 4max i ≥ 1 v i . By Proposition 23, we know that if S t is sub-Gamma with variance process V t and scale parameter c , then

where

Lemma 26 (Time-Uniform Azuma-Hoeffding) . Let X 1 , . . . , X n be a martingale difference sequence such that | X t | ≤ B for all t > 1 almost surely. Then for all δ > 0 ,

Proof of Lemma 26. By Proposition 23, we know that if S t is sub-Gamma with variance process V t and scale parameter c , then

where

By [Howard et al., 2020], we know that if ( X t ) ∞ t =1 is B -bounded martingale difference sequence, then S t = ∑ t s =1 X s is sub-Gamma with variance process V t = tB 2 and scale parameter c = 0 . This implies,

concluding the proof.

F Experiment Details

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

0.6

0.4

0.2

0.0

0.2

0.4

F.1 Hyper-Parameter Tuning Results

We implement 6 algorithms in our experiments, ETC [Algorithm 4, Hao et al., 2020], ETS (Algorithm 5), CORRAL [Algorithm 6, Agarwal et al., 2017], ALEXP (Algorithm 1), and Lastly UCB (Algorithm 3) with the oracle feature map ϕ j⋆ (Oracle), and UCB with the concatenated feature map ϕ (Naive). The Python code is available on github.com/lasgroup/ALEXP. When algorithms require exploration, e.g., in the case of ETC or ALEXP, we simply set π = Unif( X ) . Figure 7 shows the results of our hyperparameter tuning experiment. To ensure that the curves are valid, we run each

0

25

50

75

100

t

0

10

20

30

Regret

p

=10

s

=2

Lightly Correlated

0

25

50

75

100

t

0

10

20

30

p

=10

s

=8

Highly Correlated

0

25

50

75

100

t

0

10

20

30

40

p

=10

s

=3

Many Models

Oracle UCB

Naive UCB

ETC

ETS

Corral

Algorithm 2 GetPosterior

Inputs:

H

t

,

ϕ

,

˜

λ

Algorithm 3

configuration for 20 different random seeds, i.e. on different random environments. The shaded areas in Figure 7 show the standard error.

UCB. For all the experiments, we set the exploration coefficient of UCB to β t = 2 3 and choose the regression regularizer from ˜ λ ∈ { 0 . 01 , 0 . 1 , 0 . 5 } . We use PYTORCH [Paszke et al., 2017] for updating the upper confidence bounds, which requires more regularization for longer feature maps (e.g. when s = 8 , p = 2 ), to be computationally stable.

Lasso. Every time we need to solve Eq. (1), we set λ t according to the rate suggested by Theorem 3. To find a suitable constant scaling coefficient, we perform a hyper-parameter tuning experiment sampling 20 values in [10 -5 , 10 0 ] . We choose λ 0 = 0 . 009 , and scale λ t with it across all experiments.

ALEXP. We set the rates for γ t and η t as prescribed by Theorem 1. For the scaling constants, we perform a hyper-parameter tuning experiment log-uniformly sampling 20 different configurations from γ 0 ∈ [10 -4 , 10 -1 ] and η 0 ∈ [10 0 , 10 2 ] . For each problem instance (i.e. as s and p change) we repeat this process. However we observe that the optimal hyper-parameters work well across all problem instances.

ETC/ETS. For these algorithms, we separately tune n 0 for each problem instance. We set λ 1 ∝ √ log M/n 0 according to Theorem 4.2 of [Hao et al., 2020] and scale it with λ 0 = 0 . 009 , as stated before. We uniformly sample 10 different values where n 0 ∈ [2 , 80] since the horizon is n = 100 . The optimal value often happens around n 0 = 20 .

CORRAL. We set the rates of the parameters as γ = O (1 /n ) and η = O ( √ M/n ) according to Agarwal et al. [2017, Theorem 5,]. Then similar to ALEXP, we tune the scaling constants. The procedure for tuning the constants is identical to ALEXP, as in we use the same search interval, and try 10 different configurations for γ and η .

Algorithm 4 ETC [Hao et al., 2020]

Algorithm 5 ETS

̸

Algorithm 6 CORRAL [Agarwal et al., 2017]

Algorithm 7 LOG-BARRIER-OMD

Inputs:

q

t

,

ℓ

t

,

η

t

0

25

50

75

100

0

10

20

30

40

s

=2

p

=10

ETC

0

25

50

75

100

0

10

20

30

40

ETS

0

25

50

75

100

0

20

40

60

Corral

0

25

50

75

100

0

1

2

3

4

5

0

25

50

75

100

0

10

20

30

40

50

s

=2

p

=11

0

25

50

75

100

0

10

20

30

40

0

25

50

75

100

0

10

20

30

40

50

0

25

50

75

100

0

2

4

6

0

25

50

75

100

0

10

20

30

40

s

=2

p

=12

0

25

50

75

100

0

10

20

30

0

25

50

75

100

0

20

40

60

0

25

50

75

100

0

2

4

6

0

25

50

75

100

0

10

20

30

40

s

=2

p

=13

0

25

50

75

100

0

10

20

30

0

25

50

75

100

0

10

20

30

40

50

0

25

50

75

100

2

4

6

0

25

50

75

100

0

10

20

30

40

s

=3

p

=10

0

25

50

75

100

0

10

20

30

40

0

25

50

75

100

0

20

40

60

0

25

50

75

100

0

2

4

6

0

25

50

75

100

0

10

20

30

40

s

=8

p

=10

0

25

50

75

100

0

10

20

30

40

0

25

50

75

100

0

20

40

60

0

25

50

75

100

1

2

3

4

5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AlExp, a novel algorithm to address the problem of online model selection in the context of bandit optimization. The algorithm interacts with the environment by randomly choosing a learner (linear bandit regret minimizer) each round, then the chosen learner tries to propose the optimal action. The exploration-exploitation trade-off is dealt with both by randomizing the choice of the learner and by the learner's policy itself. The authors provide theoretical guarantees and empirical validation and compare this approach with the existing literature.\n\nSTRENGTHS:\n+ The paper presents an extensive literature review and a detailed comparison between the proposed approach and the existing ones.\n\n+ The algorithm's routine is easy to understand, and its logic is well-commented.\n\n+ The algorithm effectively improves in some dependencies (e.g., the number of learners) w.r.t. to the existing approaches, both theoretically and empirically.\n\n+ The analysis presents technical novelty.\n\nWEAKNESSES:\n- In some points, the presentation is not easy to follow, and it will be nice to have a more intuitive grasp of the theoretical quantities considered.\n\n- It will be nice to have more discussion on the computational aspects of the proposed approach: numerically computing some of the quantities seems to be very expensive and prohibitive when the number of learners scales. This contrasts with the main strength of the approach, i.e., the better regret dependency on the number of learners.\n\n- The algorithm requires the knowledge of theoretical quantities (e.g., expected values). In practice, this can be bypassed by a Monte Carlo sampling procedure. However, when performing finite sampling, there's the need to explain how quickly the estimator converges to the theoretical quantity and explain the computational consequences (see previous point) or eventual corrections to the algorithm to deal with the estimation error (e.g., adding an upper confidence bound on the quantities) if due to computational reasons only a small number of samples can be generated.\n\nQUESTIONS:\n+ Can you please provide an examination of the computational limitations of this approach? E.g., providing the solving complexity of the optimizations involved.\n\n+ Due to the strong dependence (on both algorithm and analysis) on quantity C, can you please provide a more intuitive grasp on the magnitude of this quantity w.r.t. other problem-dependent quantities? Is there an available lower/upper bound on it? I feel that the quality of your theoretical results strongly depends on this quantity, and a characterization (at least qualitative) of its magnitude should be discussed.\n\nLIMITATIONS:\n- I feel that the main limitation of this work relies on its application in practice. However, the authors performed a sufficient experimental campaign, validating their approach. Unfortunately, the code for the experimental campaign has not been released, so reproducibility is another limitation of this work (even if they provide details to reproduce the experiments).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
based on the time-uniform analysis of the Lasso, the anytime exponential weighting algorithm based on Lasso reward estimates with the nature of anytime regret guarantees on model selection linear bandits is developed. The result neither requires knowledge of the horizon n, nor relies on an initial purely exploratory stage.

STRENGTHS:
the anytime exponential weighting algorithm based on Lasso reward estimates are horizon-independent, explores adaptively without requiring an initial exploration stage.

WEAKNESSES:
1 The dimensionality of the proposed algorithm model will be ultra-high, and the complexity of the algorithm's runtime depends on the efficiency of the sparse regression model algorithm.
2 The results of the algorithm still belong to the class of multiplicative weights Update algorithms.

QUESTIONS:
1 What is the relationship between the number of models and their final performance? 
2 During the entire learning process, for two cases, i.e., the number of models is fixed, and the number of models is changed. Can you consider these situations and discuss them?

LIMITATIONS:
the authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper uses tackles model selection for linear bandits with $M$ models. In particular, rewards are estimated from the $M$ models using Lasso and then EXP4 is ran on-top of these estimated rewards to update individual model probabilities. The use of Lasso over ridge regression reduces variance, leading to a $\log M$ dependence rather than usual $\text{rm}poly \, M$. The paper extends the usual martingale mixture analysis from ridge to lasso; this is something I thought would be quite tricky. I look forward to reading it in more detail in the future.

STRENGTHS:
Model selection is a very important problem. Problem is explained and motivated very well. Paper is very well written and polished to the standard of a camera-ready.

WEAKNESSES:
I'm not an expert on model selection in linear bandits or full information learning. As such, I cannot identify any weaknesses in this work.

It took me a couple minutes to see why (1) is indeed (group) Lasso---I was expecting to see a 1-norm. I see that you have a sentence or two explaining this after, but somehow this didn't do the trick for me. Maybe some rewording or an extra comment here could be useful, relating it to the more usual notion of Lasso.

QUESTIONS:
None

LIMITATIONS:
Limitations clearly discussed in text.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of model selection in (lifted) linear bandits. There are M hypothesis models, each of which have a different feature map, and one of these is the true model; it is unknown to the optimizer which of the M models is the correct one. At each timestep, the optimizer chooses one model and obtains instantaneous regret in accordance with the chosen model's selection. The goal is to minimize overall regret across n timesteps.\n\nIf feedback were obtained for all model's chosen query at each timestep, then the multiplicative weights update could be used to update the probability of any model being the true oracle model. And the guarantees from the literature would apply. However, we have limited information and only observe the reward for the chosen model's query.\n\nTo get around this obstacle, the authors consider an aggregated model with all M model's features combined together. And they train this aggregated model on all the data using a sparse LASSO estimator. Then, they use this aggregated model to fill in the missing regrets for the unchosen models. And these imputed values are then used for the MW updates.\n\nThe analysis consists of proving the recovery result on the LASSO, and adjusting the original proof to use the imputed data instead of the original. The authors verify their findings on synthetic data.\n\nSTRENGTHS:\nClarity: \nThe problem is well presented, and the key difficulties of the problem are clearly identified - namely the missing regret data for unchosen models. The exposition of the solution is geared towards addressing the difficulty by imputing the missing values. \n\nThe structure of the proof is well-outlined in the main text. I also appreciated the intuitive explanation around introducing bias to reduce overall error as the reasoning behind the success of the algorithm.\n\nQuality:\nThe arguments appear correct and the correct tools are used to obtain the desired results. \n\nSignificance and Originality:\n\nAs far as I know, this problem has not been considered before, and the improvement in the scaling with M is helpful.\n\nWEAKNESSES:\nMy main concern is Q1 below.\n\nOther weaknesses are mostly minor\n\n1. It would be useful for the authors to highlight more real-world examples where bandits with multiple models would be most applicable.\n2. In line with the above, real-data experiments would be useful to verify the applicability of the algorithm. However, the theoretical contributions are already sufficient so I think this can be deferred to future work.\n3. Figure 3 is missing label on the vertical axis and is difficult to interpret.\n\nQUESTIONS:\n1. If the LASSO performs well already, then what is the need to use the LASSO to perform model selection. Why can we not just use the LASSO directly?\n\n2. While the rewards for the query point of the unchosen model are not available, the predictions on the chosen $x_t$ are available. So, why can we not use the prediction error on these points for all the models to choose the weights? Instead of using the imputed data.\n\n3. What is special about the LASSO that makes it suitable for the choice of the aggregate model?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers linear bandit problem given $M$ models, or sets of feature mappings.\nIn this problem, it is necessary to select the appropriate action as well as the model based on the bandit feedback.\nThis paper provides an algorithm with an anytime regret bound of $O(n^{3/4} \\sqrt{\\log M} + \\sqrt{n \\log^3 M} + n^{5/8}\\sqrt{d \\log n})$,\nwhere $n$ and $d$ represent the time horizon and the dimensionality of each model.\n\nSTRENGTHS:\n- The motivation for the research is well explained.\n- Experimental results support the effectiveness of the proposed method.\n\nWEAKNESSES:\n- The obtained regret bound includes $O(n^{3/4})$-term, which means that the bound is suboptimal if the number of rounds is large.\n- The proposed algorithm requires $O(M)$-time computation in each round. This can be a major computational bottleneck. In fact, in the application of sparse linear bandits, $|M|=\\binom{p}{s}$, which is exponentially large w.r.t. $s$.\nFurther, we need $M = \\Omega(\\exp(n^{1/4}))$ to ensure the regret is of $O(n \\log^3 M)$ (i.e., to ensure $n^{3/4} \\sqrt{\\log M}< \\sqrt{n \\log^3 M}$).\nThis means that the proposed algorithm is effective only when $M$ is exponentially large w.r.t. $n$.\n\nQUESTIONS:\n- Can you add a description of the known regret lower bound and how it compares to the main result?\n- Line 290: $\\{1, ..., p \\}$ <- $\\{ 0, 1, ..., p \\}$ ?\n- Line 274-275: \"the rate conjectured by Agarwal et al. [2017]\" Can you tell me where I can find the corresponding description in Agarwal et al. [2017]?\n- The condition of $\\eta_t = O(..)$ and $\\gamma_t = O(..)$ in Theorem 1: Do the authors mean $\\Theta(..)$? I guess that too small $\\eta_t$ and $\\gamma_t$ do not work well.\n- It is difficult to see what figures 1 and 2 refer to.\n\nLIMITATIONS:\nI have no concerns about the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}