SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The regret is caused by two sources: selecting a sub-optimal agent, and an agent selecting a suboptimal action. Accordingly, for any  $j \in 1, \ldots, M$ , we decompose the regret as
$$
R (n) = \sum_ {t = 1} ^ {n} r \left(\boldsymbol {x} ^ {\star}\right) - r \left(\boldsymbol {x} _ {t}\right) = \left(\sum_ {t = 1} ^ {n} r \left(\boldsymbol {x} ^ {\star}\right) - r _ {t, j}\right) + \left(\sum_ {t = 1} ^ {n} r _ {t, j} - r \left(\boldsymbol {x} _ {t}\right)\right). \tag {3}
$$
The first term shows the cumulative regret of agent  $j$ , when run within ALEXP. The second term evaluates the received reward against the cumulative average reward of model  $j$ . We bound each term separately.
Virtual Regret. The first term  $\tilde{R}_j(n) \coloneqq \sum_{t=1}^n r(\boldsymbol{x}^\star) - r_{t,j}$  compares the suggestion of agent  $j$ , against the optimal action. We call it the virtual regret since the sequence  $(\boldsymbol{x}_{t,j})_{t \geq 1}$  of the actions suggested by model  $j$  are not necessarily selected by the meta algorithm, unless  $j_t = j$ . This regret is merely a technical tool, and not actually realized when running ALEXP. The virtual regret of the oracle agent may still be analyzed using standard techniques for linear bandits, e.g., Abbasi-Yadkori et al. [2011], however we need to adapt it to take into account a subtle difference: The confidence sequence of model  $j^\star$  is constructed according to the true sequence of actions  $(\boldsymbol{x}_t)_{t \geq 1}$ , while its virtual regret is calculated based on the virtual sequence  $(\boldsymbol{x}_{t,j^\star})_{t \geq 1}$ , which the model suggests. The two sequences only match at the steps when model  $j^\star$  is selected. Adapting the analysis of Abbasi-Yadkori et al. [2011] to this subtlety, we obtain in Lemma 15 that with probability greater than  $1 - \delta$ , simultaneously for all  $n \geq 1$ ,
$$
\tilde {R} _ {j ^ {\star}} (n) = \mathcal {O} \left(n ^ {5 / 8} C _ {\min } ^ {- 1 / 2} \sqrt {d \log (n) + \log (1 / \delta)}\right).
$$
Model Selection Regret. The second term in (3) is the model selection regret,  $R(n,j) \coloneqq \sum_{t=1}^{n} r_{t,j} - r(\boldsymbol{x}_t)$ , which evaluates the chosen action by ALEXP against the suggestion of the  $j$ -th agent. Our analysis relies on a careful decomposition of  $R(n,j)$ ,
$$
R (n, j) = \sum_ {t = 1} ^ {n} \left[ \underbrace {r _ {t , j} - \hat {r} _ {t , j}} _ {\mathrm {(I)}} + \underbrace {\hat {r} _ {t , j} - \sum_ {i = 1} ^ {M} q _ {t , i} \hat {r} _ {t , i}} _ {\mathrm {(I I)}} + \underbrace {\sum_ {i = 1} ^ {M} q _ {t , i} (\hat {r} _ {t , i} - r _ {t , i})} _ {\mathrm {(I I I)}} + \underbrace {\sum_ {i = 1} ^ {M} q _ {t , i} r _ {t , i} - r (\boldsymbol {x} _ {t})} _ {\mathrm {(I V)}} \right].
$$
We bound away each term in a modular manner, until we are left with the regret of the standard exponential weights algorithm. The terms (I) and (III) are controlled by the bias of the Lasso estimator, and are  $\mathcal{O}(n^{3/4}C(M,\delta,d))$  (Lemma 19). The last term (IV) is zero in expectation, and reflects the deviation of  $r(\boldsymbol{x}_t)$  from its mean. We observe that the summands form a bounded Martingale difference sequence, and their sum grows as  $\mathcal{O}(\sqrt{n})$  (Lemma 18). Term (II) is the regret of our online optimizer, which depends on the variance of the Lasso estimator. We bound this term with  $\mathcal{O}(\sqrt{n}C(M,\delta,d)\log M)$ , by first conducting a standard anytime analysis of exponential weights (Lemma 17), and then incorporating the anytime Lasso variance bound (Lemma 20). We highlight that neither of the above steps require assumptions about the base agents. Combining these steps, Lemma 16 establishes the formal bound on the model selection regret.
Anytime Lasso. We develop novel confidence intervals for Lasso with history-dependent data, which are uniformly valid over an unbounded time horizon. This result may be of independent interest in applications of Lasso for online learning or sequential decision making. Here, we use these confidence intervals to bound the bias and variance terms that appear in our treatment of the model selection regret. The width of the Lasso confidence intervals depends on the quality of feature matrix  $\Phi_t$ , often quantified by the restricted eigenvalue property [Bickel et al., 2009, van de Geer and Buhlmann, 2009, Javanmard and Montanari, 2014]:
Definition 2. For the feature matrix  $\Phi_t \in \mathbb{R}^{t \times dM}$  we define  $\kappa(\Phi_t, s)$  for any  $1 \leq s \leq M$  as
$$
\begin{array}{l} \kappa (\Phi_ {t}, s) := \inf  _ {(J, \boldsymbol {b})} \frac {1}{\sqrt {t}} \frac {\| \Phi_ {t} \boldsymbol {b} \| _ {2}}{\sqrt {\sum_ {j \in J} \| \boldsymbol {b} _ {j} \| _ {2} ^ {2}}} \\ \text {s . t .} \boldsymbol {b} \in \mathbb {R} ^ {d} \backslash \{0 \}, \sum_ {j \notin J} \| \boldsymbol {b} _ {j} \| _ {2} \leq 3 \sum_ {j \in J} \| \boldsymbol {b} _ {j} \| _ {2}, J \subset \{1, \dots , M \}, | J | \leq s. \\ \end{array}
$$
Our analysis is in terms of this quantity, and Lemma 8 explains the connection between  $\kappa(\Phi_t, s)$  and  $C_{\min}$  as defined in (2), particularly that  $\kappa(\Phi_t, 2)$  is also positive with a high probability.
Theorem 3 (Anytime Lasso Confidence Sequences). Consider the data model  $y_{t} = \pmb{\theta}^{\top}\pmb{\phi}(\pmb{x}_{t}) + \varepsilon_{t}$  for all  $t\geq 1$ , where  $\varepsilon_t$  is i.i.d. zero-mean sub-Gaussian noise, and  $(\pmb {x}_t)_{t\geq 1}$  is  $(\mathcal{F}_t)_{t\geq 1}$ -predictable, where  $\mathcal{F}_t\coloneqq (\pmb {x}_1,\dots ,\pmb {x}_t,\varepsilon_1,\dots ,\varepsilon_{t - 1})$ . Then the solution of (1) guarantees
$$
\mathbb {P} \left(\forall t \geq 1: \left\| \pmb {\theta} - \hat {\pmb {\theta}} _ {t} \right\| _ {2} \leq \frac {4 \sqrt {1 0} \lambda_ {t}}{\kappa^ {2} (\Phi_ {t} , 2)}\right) \geq 1 - \delta
$$
if the regularization parameter satisfies for all  $t \geq 1$
$$
\lambda_ {t} \geq \frac {2 \sigma}{\sqrt {t}} \sqrt {1 + \frac {1 2}{\sqrt {2}} (\log (2 M / \delta) + (\log \log d) _ {+}) + \frac {5}{\sqrt {2}} \sqrt {d (\log (2 M / \delta) + (\log \log d) _ {+})}}.
$$
Our confidence bound enjoys the same rate as Lasso with fixed (offline) data, up to  $\log \log d$  factors. We prove this theorem by constructing a self-normalized martingale sequence based on the  $\ell_2$ -norm of the empirical process error  $(\Phi_t^\top \varepsilon_t)$ . We then apply the "stitched" time-uniform boundary of Howard et al. [2021]. Appendix B elaborates on this technique. Previous work on sparse linear bandits also include analysis of Lasso in an online setting, when  $x_{t}$  is  $\mathcal{F}_t$  measurable. Cella and Pontil [2021] imitate offline analysis and then apply a union bound over the time steps, which multiplies the width by  $\log n$  and requires knowledge of the horizon. Bastani and Bayati [2020] also rely on knowledge of  $n$  and employ a scalar-valued Bernstein inequality on  $\ell_{\infty}$ -norm of the empirical process error, which inflates the width of the confidence sets by a factor of  $\sqrt{d}$ . We work directly on the  $\ell_2$ -norm, and use a curved boundary for sub-Gamma martingale sequences, which according to Howard et al. [2021] is uniformly tighter than a Bernstein bound, especially for small  $t$ .
5.2 Discussion
In light of Theorem 1 and Theorem 3, we discuss some properties of ALEXP.
Sparse EXP4. Our approach presents a new connection between online learning and high-dimensional statistics. The rule for updating the probabilities in ALEXP is inspired by the exponential weighting for Exploration and Exploitation with Experts (EXP4) algorithm, which was proposed by Auer et al. [2002b] and has been extensively studied in the adversarial bandit and learning with expert advice literature [e.g., McMahan and Streeter, 2009, Beygelzimer et al., 2011]. EXP4 classically uses importance-weighted (IW) or ordinary least squares (LS) estimators to estimate the return  $r_{t,j}$ , both of which are unbiased but high-variance choices [Bubeck et al., 2012]. In particular, in our linearly parametrized setting, the variance of IW and LS scales with  $Md$ , which will lead to a poly( $M$ ) regret. However, it is known that introducing bias can be useful if it reduces the variance [Zimmert and Lattimore, 2022]. For instance, EXP3-IX [Kocák et al., 2014] and EXP4-IX [Neu, 2015] construct a biased IW estimator. Equivalently, others craft regularizers for the reward of the online optimizer, seeking to improve the bias-variance balance [e.g., Abernethy et al., 2008, Bartlett et al., 2008, Abernethy and Rakhlin, 2009, Bubeck et al., 2017, Lee et al., 2020, Zimmert and Lattimore, 2022]. A key technical observation in this work is that our online Lasso estimator leads EXP4 to achieve sublinear regret which depends logarithmically on  $M$ . This is due to the fact that while the estimator itself is  $Md$ -dimensional, its bias squared and variance scale with  $\sqrt{d\log M}$ . To the best of our knowledge, this work is first to instantiate the EXP4 algorithm with a sparse low-variance estimator.
Adaptive and Anytime. To estimate the reward, prior work on sparse bandits commonly emulate the Lasso analysis on offline data or on a martingale sequence with a known length [Hao et al., 2020, Bastani and Bayati, 2020]. These works require a long enough sequence of exploratory samples, and knowledge of the horizon to plan this sequence. ALEXP removes both of these constraints, and presents a fully adaptive algorithm. Crucially, we employ the elegant martingale bounds of Howard et al. [2020] to present the first time-uniform analysis of the Lasso with history-dependent data (Theorem 3). This way we can use all the data points and explore with a probability which vanishes at a  $\mathcal{O}(t^{-1/4})$  rate. Our anytime confidence bound for Lasso, together with the horizon-independent analysis of the exponential weights algorithm, also allows ALEXP to be stopped at any time with valid guarantees.
Rate Optimality. For  $M \gg n$ , we obtain a  $\mathcal{O}(\sqrt{n\log^3M})$  regret, which matches the rate conjectured by Agarwal et al. [2017]. However, if  $M$  is comparable to  $n$  or smaller, our regret scales with
Figure 1: ALEXP can model-select in both orthogonal and correlated classes  $(M = 55)$
Figure 2: ALEXP performs well on a large class ( $M = 165$ )
$\mathcal{O}(n^{3/4}\sqrt{\log M})$ , and while it is still sublinear and scales logarithmically with  $M$ , the dependency on  $n$  is sub-optimal. This may be due to the conservative nature of our model selection analysis, during which we do not make assumptions about the dynamics of the base agents. Therefore, to ensure sufficiently diverse data for successful model selection, we need to occasionally choose exploratory actions with a vanishing probability of  $\gamma_t$ . We conjecture that this is avoidable, if we make more assumptions about the agents, e.g., that a sufficient number of agents can achieve sublinear regret if executed in isolation. Banerjee et al. [2023] show that the data collected by sublinear algorithms organically satisfies a minimum eigenvalue lowerbound, which may also be sufficient for model selection. We leave this as an open question for future work.
Experiment Setup. We create a synthetic dataset based on our data model (Section 3), and choose the domain to be 1-dimensional  $\mathcal{X} = [-1,1]$ . As a natural choice of features, we consider the set of degree- $p$  Legendre polynomials, since they form an orthonormal basis for  $L^2(\mathcal{X})$  if  $p$  grows unboundedly. We construct each feature map, by choosing  $s$  different polynomials from this set, and therefore obtaining  $M = \binom{p+1}{s}$  different models. More formally, we let  $\phi_j(x) = (P_{j_1}(x),\ldots,P_{j_s}(x)) \in \mathbb{R}^s$  where  $\{j_1,\dots,j_s\} \subset \{0,\dots,p\}$  and  $P_{j'}$  denotes a degree  $j'$  Legendre polynomial. To construct the reward function, we randomly sample  $j^{\star}$  from  $[M]$ , and draw  $\theta_{j^{\star}}$  from an i.i.d. standard gaussian distribution. We then normalize  $||\theta_{j^{\star}}|| = 1$ . When sampling from the reward, we add Gaussian noise with standard deviation  $\sigma = 0.01$ . Figure 5 in the appendix shows how the random reward functions may look. For all experiments we set  $n = 100$ , and plot the cumulative regret  $R(n)$  averaged over 20 different random seeds, the shaded areas in all figures show the standard error across these runs.
Algorithms. We perform experiments on two UCB algorithms, one with oracle knowledge of  $j^{\star}$ , and a naive one which takes into account all  $M$  feature maps. We run Explore-then-Commit (ETC) by Hao et al. [2020], which explores for a horizon of  $n_0$  steps, performs Lasso once, and then selects actions greedily for the remaining steps. As another baseline, we introduce Explore-then-Select (ETS) that explores for  $n_0$  steps, performs model selection using the sparsity pattern of the Lasso estimator. For the remaining steps, the policy switches to UCB, calculated based on the selected features. Performance of ETC and ETS depends highly on  $n_0$ , so we tune this hyperparameter separately for each experiment. We also run CORRAL as proposed by Agarwal et al. [2017], with UCB agents similar to ALEXP. We tune the hyper-parameters of CORRAL as well. To initialize ALEXP we set the rates of  $\lambda_t, \gamma_t$  and  $\eta_t$  according to Theorem 1, and perform a light hyper-parameter tuning to choose the scaling constants. We have included the details and results of our hyper-parameter tuning in Appendix F.1. To solve (1), we use CELER, a fast solver for the group Lasso [Massias et al., 2018]. Every time UCB policy is used, we set the exploration coefficient  $\beta_t = 2$ , and every time exploration is required, we sample according to  $\pi = \mathrm{Unif}(\mathcal{X})$ . Appendix F includes the pseudo-code for all baseline algorithms.
Easy vs. Hard Cases. We construct an easy problem instance, where  $s = 2$ ,  $p = 10$ , and thus  $M = 55$ . Models are lightly correlated since each two model can have at most one Legendre polynomial in common. We also generate an instance with highly correlated feature maps where  $s = 8$  and  $p = 10$ , which will be a harder problem, since out of the total  $M = 55$  models, there are 36 models which have at least 6 Legendre polynomials in common with the oracle model  $j^{\star}$ . Figure 1 shows that not only ALEXP is not affected by the correlations between the models, but also it achieves
<sup>1</sup>The Python code for reproducing the experiments is accessible on github.com/lasgroup/ALEXP.
Figure 3: ALEXP is hardly affected by increasing the number of models (y-axis have various scales)
(b)
Figure 4: ALEXP can rule out models without ever having queried them ( $M = 165$ )
a performance competitive to the oracle in both cases, implying that our exponential weights technique for model selection is robust to choice of features. ETC and ETS rely on Lasso for model selection, which performs poorly in the case of correlated features. CORRAL uses log-barrier-OMD with an importance-weighted estimator, which has a significantly high variance. The curve for CORRAL in Figures 1 and 2 is cropped since the regret values get very large. Figure 6 shows the complete results. We construct another hard instance (Fig. 2), where the model class is large ( $s = 3$ ,  $p = 10$ ,  $M = 165$ ). ALEXP continues to outperform all baselines with a significant gap. It is clear in the regret curves how explore-then-commit style algorithms are inherently horizon-dependent, and may exhibit linear regret, if stopped at an arbitrary time. This is not an issue with the other algorithms.
Scaling with M. Figure 3 shows how well the algorithms scale as  $M$  grows. For this experiment we set  $s = 2$  and change  $p \in \{9, \dots, 13\}$ . While increasing  $M$  hardly affects ALEXP and Oracle UCB, other baselines become less and less sample efficient.
Learning Dynamics of ALEXP. Figure 4 gives some insight into the dynamics of ALEXP when  $M = 165$ . In particular, it shows how ALEXP can rule out sub-optimal agents without ever having queried them. Figure (a) shows the distribution  $q_{t}$ , at  $t = 20$  which is roughly equal to the optimal  $n_0$  for ETC in this configuration. The oracle model  $j^{\star}$  is annotated with a star, and has the highest probability of selection. We observe that, already at this time step, more than  $80\%$  of the agents are practically ruled out, due to small probability of selection. However, according to Figure (b), which shows  $M_t$  the total number of visited models, less than  $10\%$  of the models are queried at  $t = 20$ . This is the key practical benefit of ALEXP compared to black-box algorithms such as CORRAL. Lastly, Figure (c) shows how  $q_{t,j^{\star}}$  the probability of selecting the oracle agent changes with time. While this probability is higher than that of the other agents, Figure (c) shows that  $q_{t,j^{\star}}$  is not exceeding 0.25, therefore there is always a probability of greater than 0.75 that we sample another agent, making ALEXP robust to hard problem instances where many agents perform efficiently. We conclude that ALEXP seems to rapidly recognize the better performing agents, and select among them with high probability.
We proposed ALEXP, an algorithm for simultaneous online model selection and bandit optimization. As a first, our approach leads to anytime valid guarantees for model selection and bandit regret, and does not rely on a priori determined exploration schedule. Further, we showed how the Lasso can be used together with the exponential weights algorithm to construct a low-variance online learner. This new connection between high-dimensional statistics and online learning opens up avenues for future research on high-dimensional online learning. We established empirically that ALEXP has favorable exploration-exploitation dynamics, and outperforms existing baselines. We tackled the open problem of Agarwal et al. [2017], and showed that  $\log M$  dependency for regret is achievable for linearly parametrizable rewards. This problem remains open for more general, non-parametric reward classes.
We thank Jonas Rothfuss and Miles Wang-Henderson for their valuable suggestions regarding the writing. We thank Scott Sussex and Felix Schur for their thorough feedback. This research was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and Innovation Program Grant agreement no. 815943. Nicolas Emmenegger was supported by the Zeno Karl Schindler Foundation and the Swiss Study Foundation. Aldo Pacchiano would like to thank the support of the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This work was supported in part by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard.
Yasin Abbasi-Yadjkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems, 2011.
Yasin Abbasi-Yadkori, David Pal, and Csaba Szepesvari. Online-to-confidence-set conversions and application to sparse stochastic bandits. In Artificial Intelligence and Statistics. PMLR, 2012.
Jacob Abernethy and Alexander Rakhlin. Beating the adaptive bandit with high probability. In 2009 Information Theory and Applications Workshop. IEEE, 2009.
Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. An efficient algorithm for bandit linear optimization. In Conference on Learning Theory, 2008.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning. PMLR, 2014.
Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory. PMLR, 2017.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Foundations of Computer Science. IEEE, 1995.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 2002a.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 2002b.
Debangshu Banerjee, Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Exploration in linear bandits with rich action sets and its implications for inference. In International Conference on Artificial Intelligence and Statistics. PMLR, 2023.
Peter L Bartlett, Varsha Dani, Thomas P Hayes, Sham M Kakade, Alexander Rakhlin, and Ambuj Tewari. High-probability regret bounds for bandit online linear optimization. In Conference on Learning Theory, 2008.
Hamsa Bastani and Mohsen Bayati. Online decision making with high-dimensional covariates. Operations Research, 2020.
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In International Conference on Artificial Intelligence and Statistics, 2011.
Peter J. Bickel, Ya'acov Ritov, and Alexandre B. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics, 2009.
Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 2012.
Sebastien Bubeck, Yin Tat Lee, and Ronen Eldan. Kernel-based methods for bandit convex optimization. In ACM SIGACT Symposium on Theory of Computing, pages 72-85, 2017.
Adish Singla, Hamed Hassani, and Andreas Krause. Learning to interact with learning agents. In AAAI Conference on Artificial Intelligence, 2018.
William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 1933.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 1996.
Doniyor Ulmasov, Caroline Baroukh, Benoit Chachuat, Marc Peter Deisenroth, and Ruth Misener. Bayesian optimization with dimension scheduling: Application to biological systems. In Computer Aided Chemical Engineering. Elsevier, 2016.
Sara A. van de Geer and Peter Buhlmann. On the conditions used to prove oracle results for the Lasso. Electronic Journal of Statistics, 2009.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge university press, 2019.
Julian Zimmert and Tor Lattimore. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory. PMLR, 2022.
A Extended Literature Review 14
B Time Uniform Lasso Analysis 15
C Results on Exploration 18
C.1 ALEXP with Uniform Exploration 20
C.2 Proof of Results on Exploration 20
D Proof of Regret Bound 23
D.1 Proof of Model Selection Regret 24
D.2 Proof of Virtual Regret 30
E Time-Uniform Concentration Inequalities 32
F Experiment Details 34
F.1 Hyper-Parameter Tuning Results 34
A Extended Literature Review
The sparse linear bandit literature considers linear reward functions of the form  $\pmb{\theta}^{\top}\pmb{x}$ , where  $\pmb{x} \in \mathbb{R}^{p}$  however a sub-vector of size  $d$  is sufficient to span the reward function. This can be formulated as model selection among  $M = \binom{p}{d}$  different linear parametrizations, where each  $\phi_{j}$  is a  $d$ -dimensional feature map. We present the bounds in terms of  $d$  and  $M$  for coherence with the rest of the text, assuming that  $M = \mathcal{O}(p)$ , which is the case when  $d \ll p$ .