Title: High-dimensional Contextual Bandit Problem without Sparsity
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

linear bandits

contextual bandits

overparameterized models

high-dimensional models

online learning
Abstract: In this research, we investigate the high-dimensional linear contextual bandit problem where the number of features $p$ is greater than the budget $T$, or it may even be infinite. Differing from the majority of previous works in this field, we do not impose sparsity on the regression coefficients. Instead, we rely on recent findings on overparameterized models, which enables us to analyze the performance of the minimum-norm interpolating estimator when data distributions have small effective ranks. We propose an explore-then-commit (EtC) algorithm to address this problem and examine its performance. Through our analysis, we derive the optimal rate of the ETC algorithm in terms of $T$ and show that this rate can be achieved by balancing exploration and exploitation. Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. We assess the performance of the proposed algorithms through a series of simulations.

High-dimensional Contextual Bandit Problem without Sparsity

Abstract

In this research, we investigate the high-dimensional linear contextual bandit problem where the number of features 𝑝 is greater than the budget 𝑇 , or it may even be infinite. Differing from the majority of previous works in this field, we do not impose sparsity on the regression coefficients. Instead, we rely on recent findings on overparameterized models, which enables us to analyze the performance of the minimum-norm interpolating estimator when data distributions have small effective ranks. We propose an explore-then-commit (EtC) algorithm to address this problem and examine its performance. Through our analysis, we derive the optimal rate of the ETC algorithm in terms of 𝑇 and show that this rate can be achieved by balancing exploration and exploitation. Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. We assess the performance of the proposed algorithms through a series of simulations.

1 Introduction

The multi-armed bandit problem [Robbins, 1952, Lai and Robbins, 1985] has been widely studied in the field of sequential decision-making problems in uncertain environments, and it can be applied to a variety of real-world scenarios. This problem involves an agent selecting one of 𝐾 arms in each round andreceiving a corresponding reward. The agent aims to maximize the cumulative reward over rounds by using a clever algorithm that balances exploration and exploitation. In particular, a version of this problem called the contextual bandit problem [Abe and Long, 1999, Li et al., 2010] has attracted significant attention in the machine learning community. By observing the contexts associated with the arms, the agent can choose the best arm as a function of the contexts. This extension enables us to model many personalized machine learning scenarios, such as recommendation systems [Li et al., 2010, Wang et al., 2022] and online advertising [Tang et al., 2013], and personalized treatments [Chakraborty and Murphy, 2014].

Most of the papers about stochastic linear bandits assume that the number of features 𝑝 is moderate [Li et al., 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011]. When 𝑝 = 𝑜 ( √ 𝑇 ) to the number of rounds 𝑇 , the model is identifiable, and the agent can choose the best arm for most rounds. However, recent machine learning models desire to utilize an even larger number of features, and the theory of bandit models under the identifiability assumption does not necessarily reflect the modern use of machine learning. Several recent papers have overcome this limitation by considering sparse linear bandit models [Wang et al., 2018, Kim and Paik, 2019, Bastani and Bayati, 2020, Hao et al., 2020, Oh et al., 2021, Li et al., 2022, Jang et al., 2022]. Sparse linear bandit models accept a very large number of features 1 and suppress most of the coefficients by introducing the ℓ 1 regularize.

That said, the sparsity imposed by such models limits the applicability of these models. For example, in the case of recommendation models based on factorization, each user is associated with a dense latent vector [Rendle, 2010, Agarwal et al., 2012, Wang et al., 2022], which implies the sparsity is not unlikely the case. Another possible drawback of sparse models is that it requires the condition number to be close to one (e.g., the restricted isometry property, see Van De Geer and B¨ uhlmann [2009] for review). This implies that the quality of the estimator is compromised by the noise on the features that correspond to small eigenvalues. Furthermore, it is still non-trivial to select a proper value of the penalty coefficient as a hyper-parameter. For example, Hara and Maehara [2017] claims that small changes in the choice of coefficients significantly alter feature selection, and Miolane and Montanari [2021] show a limitation of the conventional theory on the choice of penalty coefficients.

In this paper, We consider an alternative high-dimensional linear bandit problem without sparsity. We allow 𝑝 to be as large as desired, and in fact, we even allow 𝑝 to be infinitely large. Such an overparameterized model has more parameters than the number of training data points. A natural estimator in such a case is an interpolating estimator, which perfectly fits the training data. We adopt recent results that bound the error of the estimator in terms of the effective rank [Koltchinskii and Lounici, 2017, Bartlett et al., 2020] on the covariance of the features. When the eigenvalues of the covariance decay moderately fast, we can obtain a concentration inequality on the squared error of the estimator.

The contributions of this paper are as follows: First, We consider explore-then-commit (EtC) strategy for the stochastic bandit problem based on the minimum-norm interpolating estimator. We derive the optimal rate of exploration that minimizes regret. However, EtC requires model-dependent parameters on the covariance, which limits the practical utility. To address this limitation, we propose an adaptive explore-then-commit (AEtC) strategy, which adaptively estimates these parameters and achieves the optimal rate. We conduct simulations to verify the efficiency of the proposed method.

2 Preliminary

2.1 Notation

For 𝑧 ∈ N , [ 𝑧 ] : = { 1 , 2 , . . . , 𝑧 } . For 𝑥 ∈ R , the notation ⌊ 𝑥 ⌋ here denotes the largest integer that is less than or equal to a scalar 𝑥 . For vectors 𝑋, 𝑋 ′ ∈ R 𝑝 , ⟨ 𝑋, 𝑋 ′ ⟩ : = 𝑋 ⊤ 𝑋 ′ is an inner product, ∥ 𝑋 ∥ 2 2 : = ⟨ 𝑋, 𝑋 ⟩ is an ℓ 2-norm. For a positive-definite matrix 𝐴 ∈ R 𝑝 × R 𝑝 , ∥ 𝑋 ∥ 2 A : = ⟨ 𝑋, A 𝑋 ⟩ is a weighted ℓ 2-norm. ∥ A ∥ op denotes an operator norm of A . 𝑂 (·) , 𝑜 (·) , Ω (·) , 𝜔 (·) and Θ (·) denotes Landau's Big-O, little-o, Big-Omega, little-omega, and Big-Theta notations, respectively. e 𝑂 (·) , e Ω (·) , and e Θ (·) are the notations that ignore polylogarithmic factors.

2.2 Problem Setup

This paper considers a linear contextual bandit problem with 𝐾 arms. We consider the fully stochastic setting, where the contexts, as well as the rewards, are drawn from fixed distributions. For each round 𝑡 ∈ [ 𝑇 ] and arm 𝑖 ∈ [ 𝐾 ] , we define 𝑋 ( 𝑖 ) ( 𝑡 ) as a 𝑝 -dimensional zero-mean sub-Gaussian vector. We assume 𝑋 ( 𝑖 ) ( 𝑡 ) is independent among rounds (i.e., vectors in two different rounds 𝑡, 𝑡 ′ are independent) but allow vectors 𝑋 ( 1 ) ( 𝑡 ) , 𝑋 ( 2 ) ( 𝑡 ) , . . . , 𝑋 ( 𝐾 ) ( 𝑡 ) to be correlated with each other. The forecaster chooses an arm 𝐼 ( 𝑡 ) ∈ [ 𝐾 ] based on the 𝑋 ( 𝑖 ) ( 𝑡 ) values of all the arms, and then observes a reward that follows a linear model as shown in

The unknown true parameters 𝜃 ( 𝑖 ) for each arm 𝑖 ∈ [ 𝐾 ] lie in a parameter space Θ ⊂ R 𝑝 , and the independent noise term 𝜉 ( 𝑡 ) has zero mean and variance 𝜎 2 > 0. We assume that 𝜉 ( 𝑡 ) is sub-Gaussian, and for the sake of simplicity, we assume that it does not depend on the choice of the arm. However, our results can be extended to the case where 𝜉 ( 𝑡 ) varies among arms. We assume that each 𝜃 ( 𝑖 ) are bounded ∥ 𝜃 ( 𝑖 ) ∥ 2 ≤ 𝜃 max. For each 𝑖 ∈ [ 𝐾 ] , we define a covariance matrix Σ ( 𝑖 ) = E [ 𝑋 ( 𝑖 ) ( 𝑡 ) 𝑋 ( 𝑖 ) ( 𝑡 ) ⊤ ] ∈ R 𝑝 × 𝑝 .

We define 𝑖 ∗ ( 𝑡 ) : = argmax 𝑖 ∈[ 𝐾 ] ⟨ 𝑋 ( 𝑖 ) ( 𝑡 ) , 𝜃 ( 𝑖 ) ⟩ as the (ex ante) optimal arm at round 𝑡 . Our goal is to design an algorithm that maximizes the total reward, which is equivalent to minimizing the following

expected regret [Lai and Robbins, 1985, Auer et al., 2002];

where the expectation is taken with respect to the randomness of the contexts and (possibly) on the choice of arm 𝐼 ( 𝑡 ) .

2.3 Theory of Overparametrized Models

The primary focus of this paper is on scenarios where the number of arms 𝐾 is moderate, but the number of features 𝑝 is greater than the budget 𝑇 , possibly to the point of being infinite. The efficiency of linear regression models in such scenarios depends significantly on the covariance matrix, Σ ( 𝑖 ) . Unlike sparsity, the theory of benign overfitting tightly examines errors using the decay of the eigenvalues of the covariance matrix of the context. In particular, if the decay rate of the eigenvalues is at a certain level, the error in linear regression converges to zero, even in high-dimensional spaces. To provide a more rigorous analysis of eigenvalue decay, the concept of effective rank is introduced in Section 2.4.

Remark1. (Dependenceon 𝑇 ) In accordance with Bartlett et al. [2020], we permit the covariance matrix Σ ( 𝑖 ) to depend on 𝑇 . In other words, we consider the sequence of covariances Σ ( 𝑖 ) ( 1 ) , Σ ( 𝑖 ) ( 2 ) , . . . for each 𝑇 = 1 , 2 , . . . . The linear regression is consistent if the effective rank of Σ ( 𝑖 ) ( 𝑇 ) grows sufficiently slow as a function of 𝑇 .

2.4 Effective Ranks of Covariance Matrix

For a covariance matrix Σ ( 𝑖 ) for 𝑖 ∈ [ 𝐾 ] and 𝑘 ∈ [ 𝑝 ] , let 𝜆 ( 𝑖 ) 𝑘 be its 𝑘 -th largest eigenvalue, such that Σ ( 𝑖 ) = ˝ 𝑝 𝑘 = 1 𝜆 ( 𝑖 ) 𝑘 𝑢 ( 𝑖 ) 𝑘 ( 𝑢 ( 𝑖 ) 𝑘 ) ⊤ with order 𝜆 ( 𝑖 ) 1 ≥ 𝜆 ( 𝑖 ) 2 ≥ · · · ≥ 𝜆 ( 𝑖 ) 𝑝 and eigenvectors 𝑢 ( 𝑖 ) 𝑘 . We define the concept of effective rank as

The first quantity 𝑟 𝑘 ( Σ ( 𝑖 ) ) is related to the trace of Σ ( 𝑖 ) , and the second quantity 𝑅 𝑘 ( Σ ( 𝑖 ) ) is related to the decay rate of the eigenvalues.

The effective rank is used as a measure of the intrinsic complexity of high-dimensional data, by rigorously capturing a decay rate of eigenvalues. If the covariance matrix Σ ( 𝑖 ) is an identity matrix of size 𝑝 , then 𝑟 0 ( Σ ( 𝑖 ) ) and 𝑅 0 ( Σ ( 𝑖 ) ) are both equal to 𝑝 (the rank of Σ ( 𝑖 ) ). However, we anticipate that these quantities will be less than 𝑝 , which enables learning with fewer samples. In Figure 1, we plot of the effective rank in 𝑘 with certain cases of eigenvalues { 𝜆 𝑘 } 𝑘 of Σ . Even though Σ is full-rank, the effective rank 𝑅 𝑘 ( Σ ) is sublinear in 𝑘 , which reflects the intrinsic low-complexity of data.

Because of this property, the effective ranks are used in modern high-dimensional statistics. Koltchinskii and Lounici [2017] evaluated concentration inequalities for high-dimensional random matrices using the notion. Bartlett et al. [2020], Tsigler and Bartlett [2020] evaluated the prediction error of a high-dimensional linear regression using the effective ranks and showed the consistency of the prediction.

3 Explore-then-Commit with Interpolation

The Explore-then-Commit (EtC) algorithm is a well-known approach for solving high-dimensional linear bandit problems, and it has been shown to be effective in previous studies such as Hao et al. [2020], Li et al. [2022]. The EtC algorithm operates by first conducting 𝑇 0 = 𝑁𝐾 < 𝑇 rounds of exploration, during which it uniformly explores all available arms to construct an estimator b 𝜃 ( 𝑖 ) for each arm 𝑖 ∈ [ 𝐾 ] . After the exploration phase, the algorithm proceeds with exploitation. Let 𝑁 be the number of the draws of arm 𝑖 , and X ( 𝑖 ) = ( 𝑋 ( 𝑖 ) 1 , ..., 𝑋 ( 𝑖 ) 𝑁 ) ⊤ ∈ R 𝑁 × 𝑝 and Y ( 𝑖 ) = ( 𝑌 ( 𝑖 ) 1 , ..., 𝑌 ( 𝑖 ) 𝑁 ) ⊤ ∈ R 𝑁

Eigenvalue入k

Effective rankrk(E)

Effective rank R()

1.0

50

8

1

Case

0.5

25

4

0.0

0

0

1.0

50

8

2

Case

0.5

25

4

0.0

0

0

0

10

0

10

0

10

k

k

k

be the observed contexts and rewards of arm 𝑖 , where ( 𝑋 ( 𝑖 ) 𝑛 , 𝑌 ( 𝑖 ) 𝑛 ) is the corresponding values on the 𝑛 -th draw of arm 𝑖 . Since we choose 𝐼 ( 𝑡 ) uniformly during the exploration phase, these are independent and identically drawn samples from the corresponding distribution.

For estimating the parameter 𝜃 ( 𝑖 ) , we consider the minimum-norm interpolating estimator that perfectly fits the data, which reveals its advantage in recent results on high-dimensions [Bartlett et al., 2020]. Rigorously, we assume 𝑝 > 𝑁 and and consider the following definition:

This estimator has a simple representation b 𝜃 ( 𝑖 ) = ( X ( 𝑖 ) ) ⊤ ( X ( 𝑖 ) ( X ( 𝑖 ) ) ⊤ ) -1 Y ( 𝑖 ) . The EtC algorithm is presented in Algorithm 1, which utilizes the aforementioned interpolating estimator. In the subsequent sections, we will first discuss the assumptions on the data-generating process, followed by an analysis of the accuracy of the estimator. We then present an upper bounds on the regret of the EtC algorithm.

4 Theory of Explore-then-Commit

This section analyzes the EtC algorithm. If we were to fix the number of samples, this theory would largely align with the existing literature [Bartlett et al., 2020, Tsigler and Bartlett, 2020]. However, the unique challenge in our case arises from the fact that the EtC selects the number of samples used to construct an estimator to balance between exploration and exploitation.

4.1 Assumption and Notion

We consider a spectral decomposition Σ ( 𝑖 ) = 𝑉 ( 𝑖 ) Λ ( 𝑖 ) ( 𝑉 ( 𝑖 ) ) ⊤ with the matrices (operators) 𝑉 ( 𝑖 ) and Λ ( 𝑖 ) for each arm 𝑖 ∈ [ 𝐾 ] , then impose the following assumptions.

Assumption 1 (sub-Gaussianity) . For all 𝑡 ∈ N and 𝑖 ∈ [ 𝐾 ] , the followings hold:

There exists a random vector 𝑍 ( 𝑖 ) ( 𝑡 ) such that 𝑋 ( 𝑖 ) ( 𝑡 ) = 𝑉 ( 𝑖 ) ( Λ ( 𝑖 ) ) 1 / 2 𝑍 ( 𝑖 ) ( 𝑡 ) which is independent elements and sub-Gaussian with a parameter 𝜅 2 𝑥 > 0 , that is, for all 𝜆 ∈ R 𝑝 , we have E [ exp (⟨ 𝜆, 𝑍 ( 𝑖 ) ( 𝑡 )⟩)] ≤ exp ( 𝜅 2 𝑥 ∥ 𝜆 ∥ 2 2 / 2 ) .

Moreover, 𝜉 ( 𝑡 ) is conditionally sub-Gaussian with a parameter 𝜅 2 𝜉 > 0 , that is, for all 𝜆 ∈ R , we have E [ exp ( 𝜆𝜉 ( 𝑡 )) | 𝑋 ( 𝑖 ) ( 𝑡 )] ≤ exp ( 𝜅 2 𝜉 𝜆 2 / 2 ) .

The first bullet point of Assumption 1 requires that the features are multivariate sub-Gaussian and their tail is not too heavy, which is important for concentration inequalities on random variables [Vershynin, 2018]. The second bullet point of Assumption 1, which states that rewards involve sub-Gaussian noise, is a common assumption in contextual bandit problems.

4.2 Benign Covariance

We impose a condition to be benign on the covariance matrix Σ ( 𝑖 ) for all 𝑖 ∈ [ 𝐾 ] . The condition is described using the notion of effective/coherent rank, which is commonly applied to the study of benign overfitting [Bartlett et al., 2020, Tsigler and Bartlett, 2020]. However, unlike those papers that estimate using all 𝑇 samples, we only use a portion of the data for learning.

We first define the coherent rank of Σ ( 𝑖 ) with a number of samples 𝑁 as

where we define min {∅} = +∞ . By using the coherent rank, we decompose the error into two quantities called effective bias/variance denoted as 𝐵 ( 𝑖 ) 𝑁,𝑇 and 𝑉 ( 𝑖 ) 𝑁,𝑇 , based on a budget of 𝑇 and the number of samples 𝑁 used for estimation.

Definition 1 (Benign covariance) . Under Assumption 1, a covariance matrix Σ ( 𝑖 ) is benign , if 𝐵 ( 𝑖 ) 𝑁,𝑇 = 𝑜 ( 1 ) and 𝑉 ( 𝑖 ) 𝑁,𝑇 = 𝑜 ( 1 ) hold as 𝑁,𝑇 →∞ while 𝑁 = Θ ( 𝑇 ) .

To put it differently, Assumption 1 and Definition 1 state that, if we can achieve consistent estimation by using all the data for estimation, then the covariance matrix is considered benign.

Technically, the benign property implies that eigenvalues decay fast enough compared with 𝑇 . In particular, the following examples have been considered in the literature.

Proposition 1 (Example of Benign Covariance, Theorem 31 in Bartlett et al. [2020] 2 ) . Acovariance matrix Σ ( 𝑖 ) with eigenvalues 𝜆 ( 𝑖 ) 1 , 𝜆 ( 𝑖 ) 2 , ... is benign if it satisfies one of the following.

Example 1: Let 𝑎 ∈ ( 0 , 1 ) , 𝑝 = ∞ and 𝜆 ( 𝑖 ) 𝑘 = 𝑘 -( 1 + 1 / 𝑇 𝑎 ) . In this case, we have 𝐵 ( 𝑖 ) 𝑁,𝑇 = 𝑂 (( 𝑇 𝑎 / 𝑁 ) 1 + 1 / 𝑇 𝑎 ) and 𝑉 ( 𝑖 ) 𝑁,𝑇 = 𝑂 ( 𝑇 𝑎 / 𝑁 + 𝑇 -𝑎 ) . For this model to be learnable, 𝑁 ≫ 𝑇 𝑎 is required, and the variance dominates the bias.

Example 2: Let 𝜆 ( 𝑖 ) 𝑘 = 𝑘 -𝑏 and 𝑝 = 𝑝 𝑇 = 𝑂 ( 𝑇 𝑐 ) with 𝑏 ∈ ( 0 , 1 ) and 𝑐 ∈ GLYPH<16> max GLYPH<16> 1 , 2 2 -𝑏 , 1 1 -𝑏 2 GLYPH<17> , 1 1 -𝑏 GLYPH<17> . In this case, we have 𝐵 ( 𝑖 ) 𝑁,𝑇 = 𝑂 ( 𝑇 𝑐 ( 1 -𝑏 ) / 𝑁 ) and 𝑉 ( 𝑖 ) 𝑁,𝑇 = 𝑂 (( 𝑁 / 𝑇 𝑐 ) max ( 1 , 1 -𝑏 𝑏 ) ) . For this model to be learnable, 𝑁 ≫ 𝑇 𝑐 ( 1 -𝑏 ) is required, and the bias dominates the variance.

The first example is when the decay rate is near 𝑘 -1 , but the trace is 𝑂 ( 𝑇 𝑎 ) . The second example is when the decay rate is smaller than 𝑘 -1 , and the trace is 𝑂 ( 𝑇 𝑐 ( 1 -𝑏 ) ) . Note that Bartlett et al. [2020] provided two other examples of the benign covariance matrices. 3 Our analysis mainly focuses on the two examples in Proposition 1, but another example is also empirically tested in Section 6.

4.3 Estimation Error by Exploration

We evaluate an error in the estimator b 𝜃 ( 𝑖 ) by its prediction quality. That is, with a covariance matrix Σ ( 𝑖 ) and an identical copy 𝑋 ( 𝑖 ) ∗ of 𝑋 ( 𝑖 ) 1 , we study

The following result bounds the error of the estimator b 𝜃 ( 𝑖 ) in terms of bias and variance, which is a slight extension of Tsigler and Bartlett [2020]. For 𝑘 ∈ [ 𝑝 ] , we define an empirical submatrix as X ( 𝑖 ) 𝑘 : ∞ ∈ R 𝑁 ×( 𝑝 -𝑘 ) as the 𝑝 -𝑘 columns to the right of X ( 𝑖 ) , and define a Gram sub-matrix 𝐴 ( 𝑖 ) 𝑘 = X ( 𝑖 ) 𝑘 : ∞ ( X ( 𝑖 ) 𝑘 : ∞ ) ⊤ ∈ R 𝑁 × 𝑁 .

Theorem 2. Suppose Assumption 1. If there exists 𝑐 𝑈 > 1 such that 𝑘 ∗ 𝑁 < 𝑁 / 𝑐 𝑈 and a condition number of 𝐴 ( 𝑖 ) 𝑘 is positive with probability at least 1 -𝑐 𝑈 𝑒 -𝑁 / 𝑐 𝑈 , then we have

with some constant 𝐶 𝑈 > 0 and probability at least 1 -2 𝑐 𝑈 𝑒 -𝑁 / 𝑐 𝑈 .

Theorem 2 implies that the estimation error converges to zero if Σ ( 𝑖 ) has the benign property. The assumption on the condition number of 𝐴 ( 𝑖 ) 𝑘 has a sufficient condition, which is provided in Lemma 3 in Tsigler and Bartlett [2020].

Moreover, the following lemma implies the tightness of the analysis in Theorem 2.

Lemma 3 (Lower bound of estimation error, Theorem 10 in Tsigler and Bartlett [2020]) . Suppose Assumption 1. These exist some constants 𝑐 𝐿 , 𝐶 𝐿 > 0 such that, with probability at least 1 -𝑐 𝐿 𝑒 -𝑁 / 𝑐 𝐿 , we have

In other words, the upper bound in Theorem 2 is optimal up to a constant. 4

4.4 Regret Bound of Explore-then-Commit

This section analyzes the EtC algorithm. We introduce an error function E ( 𝑁,𝑇 ) that characterizes the rate of regret, which can be obtained by considering ( 𝐵 ( 𝑖 ) 𝑁,𝑇 + 𝑉 ( 𝑖 ) 𝑁,𝑇 ) 1 / 2 .

Assumption 2. (Error function 5 ) There exists a continuous function E : R 2 → R + that is decreasing in 𝑁 such that ∥ b 𝜃 ( 𝑖 ) -𝜃 ( 𝑖 ) ∥ Σ ( 𝑖 ) = e Θ ( E ( 𝑁,𝑇 )) as 𝑁,𝑇 →∞ .

Assumption 2 is satisfied in Examples 1 and 2. In particular, for Example 1 in Proposition 1, the error function is given as E ( 𝑁,𝑇 ) = √︁ 𝑇 𝑎 / 𝑁 + 𝑇 -𝑎 , while for Example 2 in Proposition 1, it is given as E ( 𝑁,𝑇 ) = √︁ 𝑇 𝑐 ( 1 -𝑏 ) / 𝑁 .

Theorem 4. Suppose Assumptions 1-2. Suppose that we run the EtC algorithm (Algorithm 1). Then, regret (2) satisfies

as 𝑇 →∞ with some 𝛼 > 0 . where 𝐿 ( 𝑇, 𝐾 ) is such that

Specifically, for Example 1 in Proposition 1, we have 𝐿 ( 𝑇, 𝐾 ) = max ( 𝑇 ( 2 + 𝑎 )/ 3 𝐾 2 / 3 , 𝑇 1 -𝑎 / 2 ) , whereas for Example 2 in Proposition 1, we have 𝐿 ( 𝑇, 𝐾 ) = 𝑇 ( 2 + 𝑐 ( 1 -𝑏 ) )/ 3 𝐾 2 / 3 .

Proof sketch of Theorem 4. Weshow that the regret-per-round is 𝑂 ( 1 ) during the exploration phase. Moreover, regret-per-round is e 𝑂 ( E ( 𝑇 0 / 𝐾,𝑇 )) during the exploitation phase. The total regret is 𝑇 0 + e 𝑂 ( E ( 𝑇 0 / 𝐾,𝑇 )) 𝑇 , and optimizing 𝑇 0 by using the decreasing property of E ( 𝑁,𝑇 ) in 𝑁 yields the desired result. □

4.5 Matching Lower Bound

We show that this choice of 𝑇 0 as a function of 𝑇 is indeed optimal.

Theorem 5. Suppose Assumptions 1-2. Assume that we run the EtC algorithm (Algorithm 1). For any choice of 𝑇 0 , the following event occurs with strictly positive probability as 𝑇 →∞ :

Proof sketch of Theorem 5. We explicitly construct a model with 𝐾 = 3. Let 𝜖 𝑇 = E ( 𝑇 0 / 𝐾,𝑇 ) . In the model, the Σ ( 1 ) , Σ ( 2 ) , Σ ( 3 ) are identical, and the only difference is that the first coefficients 𝜃 ( 1 ) 1 = 1 , 𝜃 ( 2 ) 1 = Θ ( 𝜖 𝑇 ) , 𝜃 ( 3 ) 1 = 0. All other coefficients are set to zero. Roughly speaking, the gap is

.

The regret-per-round during the exploration phase is Θ ( 1 ) due to a misidentification of the best arm between arm 1 and arms { 2 , 3 } . The regret-per-round during the exploitation phase is Θ ( 𝜖 𝑇 ) due to a misidentification of the best arm between arm 2 and arm 3. □

5 Adaptive Explore-then-Commit (AEtC) Algorithm

In the prior section, we demonstrated that the optimal way to minimize EtC's regret is by selecting 𝑇 0, with 𝑇 0 balancing the exploration and the exploitation. However, this requires knowledge of the covariance matrix's spectrum, which can be difficult to obtain in advance in certain scenarios. This section explores the way to adaptively determines the extent of exploration required.

5.1 Estimator

We assume that Σ ( 𝑖 ) follows the data-generating process of Example 1 or 2 in Proposition 1. We use 𝛽 𝑇 to denote that

with some constant 𝐶 𝜆 > 0. We have 𝛽 𝑇 = 1 + 1 / 𝑇 𝑎 for Example 1, and 𝛽 𝑇 = 𝑏 ( < 1 ) for Example 2.

Balancing exploration and exploitation in an overparameterized model is challenging for the following reasons. First, the number of features 𝑝 is very large or even infinite 6 . Second, the trace is heavy-tail

because the decay is not very fast. As a result, a naive use of a traditional method does not work. We address this issue by utilizing two estimators. The first estimator is on the trace tr ( b Σ ( 𝑖 ) ) that we extracted from overparameterization theory [Bartlett et al., 2020], whereas the second estimator is on the estimated decay rate b 𝛽 𝑇 that derives from Bosq [2000], Koltchinskii and Lounici [2017].

For an arm 𝑖 ∈ [ 𝐾 ] with 𝑁 = 𝑇 0 / 𝐾 draws, we define an estimated eigenvalues b 𝜆 ( 𝑖 ) 1 , ..., b 𝜆 ( 𝑖 ) 𝑁 > 0 from an empirical covariance matrix b Σ ( 𝑖 ) : = 𝑁 -1 ˝ 𝑁 𝑗 = 1 𝑋 ( 𝑖 ) 𝑗 ( 𝑡 ) 𝑋 ( 𝑖 ) 𝑗 ( 𝑡 ) ⊤ . We define the empirical trace to be tr ( b Σ ( 𝑖 ) ) = ˝ 𝑗 b 𝜆 ( 𝑖 ) 𝑗 . The following lemma states the consistency of the estimated trace under very mild conditions.

Lemma 6. (Error bound of empirical trace) Suppose Assumption 1. Suppose the data generating process (DGP) of Example 1 or Example 2 in Proposition 1. Then, for any 𝐶 poly > 0 , with probability at least 1 -1 / 𝑇 𝐶 poly , we have the following as 𝑇 →∞ :

Moreover, the following bounds the estimation error of each eigenvalue.

Lemma 7. Suppose Assumption 1. For any 𝛿 ∈ ( 0 , 1 ) , for any 𝐶 poly > 0 , with probability at least 1 -1 / 𝑇 𝐶 poly , we have the following for any 𝑗 = 1 , ..., 𝑝 as 𝑇 →∞ :

Lemma 7 implies the estimator of each eigenvalue is 𝑜 ( 1 ) if we choose 𝑁 ≫ tr ( Σ ( 𝑖 ) ) . However, even if we choose a large 𝑁 , the error is still non-negligible for the tail of eigenvalues where | b 𝜆 ( 𝑖 ) 𝑗 -𝜆 ( 𝑖 ) 𝑗 | is very small 7 . Consequently, b 𝜆 ( 𝑖 ) 𝑗 for large 𝑗 are not necessarily consistent, so as to the effective rank and the coherent rank estimated from them. To address this, we estimate the decay rate 𝛽 𝑇 , and then estimate the subsequent statistics.

Let the estimated decay rate be b 𝛽 𝑇 = ( 1 / 𝜏 ) ˝ 𝜏 𝑘 = 1 log ( b 𝜆 𝑘 / b 𝜆 𝑘 + 1 )/ log (( 𝑘 + 1 )/ 𝑘 ) . Theoretically, b 𝛽 𝑇 is consistent for any constant 𝜏 > 1. In practice, we can use a reasonable constant 𝜏 , such as 𝜏 = 10, for robustness. To estimate the effective ranks, we use the following form

Namely, we consider empirical analogues of the effective rank for 𝑘 as

We also define an estimator for the coherent rank as b 𝑘 𝑁 : = min { 𝑘 ∈ N ∪ { 0 } | b 𝑟 𝑘 ( Σ ( 𝑖 ) ) ≥ 𝑁 } . Further, we define estimators of 𝐵 ( 𝑖 ) 𝑁,𝑇 , 𝑉 ( 𝑖 ) 𝑁,𝑇 of Eq. (4) as

Note that the estimators b 𝑟 𝑘 and b 𝑅 𝑘 use the trace tr ( b Σ ( 𝑖 ) ) , which is a sum of eigenvalues, instead of the partial sum of the eigenvalues that constitutes the effective rank of Eq. (3). Despite this change, this does not affect 8 asymptotic consistency of the coherent rank estimator b 𝑘 𝑁 .

The following lemma states that small 𝜏 suffices to assure the quality of b 𝛽 ( 𝑖 ) . Westudy a convergence rate of b 𝛽 ( 𝑖 ) and the other estimators as follows.

Algorithm 2 Adaptive Explore-then-Commit (AEtC)

as 𝑇 →∞ for all 𝑘 ∈ [ 𝜏 ] . Moreover, it implies | 𝛽 𝑇 -b 𝛽 𝑇 | = 𝑜 ( 1 ) and

Eq. (11) states that the estimated rate of error is accurate as 𝑇 → ∞ . To see this, for Example 1 in Proposition 1, the estimated error rate is 𝑇 𝑜 ( 1 ) ( √︁ 𝑇 𝑎 / 𝑁 + 𝑇 -𝑎 ) , whose exponent approaches to √︁ 𝑇 𝑎 / 𝑁 + 𝑇 -𝑎 as 𝑇 →∞ .

5.2 Adaptive Explore-then-Commit (AEtC)

The Adaptive Explore-then-Commit (AEtC) algorithm is described in Algorithm 2. Unlike EtC, the amount of exploration in AEtC is adaptively determined based on the stopping condition:

and Stop = ∩ 𝑖 ∈[ 𝐾 ] Stop ( 𝑖 ) , where 𝐶 𝑇 = 𝐶 𝑇 ( 𝑇 ) is a function of 𝑇 that slowly diverges to +∞ . The first condition 𝑁 > 𝐶 𝑇 tr ( b Σ ( 𝑖 ) ) ensures that 𝑁 is large enough to have consistency on the estimators, whereas the second condition 𝑁𝐾 ≥ 𝑇 √︃ b 𝐵 ( 𝑖 ) 𝑁,𝑇 + b 𝑉 ( 𝑖 ) 𝑁,𝑇 balances the exploration and exploitation. The following theorem bounds the regret of AEtC.

Theorem 9. Suppose Assumption 1. Suppose DGP of Example 1 or Example 2 in Proposition 1. Then, for any 𝑐 > 0 , the regret (2) when we run the AEtC algorithm (Algorithm 2) with a sufficiently slowly diverging 𝐶 𝑇 is bounded as follows as 𝑇 →∞ :

where 𝐿 ( 𝑇, 𝐾 ) is the function defined in Theorem 4.

Theorem 9 states that, we can choose arbitrarily small 𝑐 > 0 and the exponent of the regret of AEtC approaches that of EtC (Theorem 4). Note that the condition 𝑁 ≥ 𝐶 𝑇 tr ( b Σ ( 𝑖 ) ) should not dominate the balance between exploration and exploitation: For example, 𝐶 𝑇 = 𝐶 log 𝑇 for any 𝐶 > 0 suffices. In practice, a constant 𝐶 𝑇 works.

Proof sketch of Theorem 9. We can derive that Eqs. (9), (10), and (11) for all 𝑁 ∈ [ 𝑇 ] that hold with probability at least 1 -𝑂 ( 1 / 𝑇 ) by setting 𝐶 poly = 2 and taking a union bound over 𝑁 , and thus

 « ‹  holds. Under this event, the stopping time of AEtC and EtC are at most 𝑇 𝑜 ( 1 ) times different. Given this, the proof of the theorem is easily modified from the proof of Theorem 4. □

T=500 p=200 K=2 DGP:1

T=500p=200K=2DGP:2

T=500 p=200K=2 DGP:3

T=500 p=200 K=2 DGP:4

Accumrated Regret

10

12

proposal

ESTC

Accumrated Regret

3.5

proposal

ESTC

Accumrated Regret

4

5

proposal

ESTC

Accumrated Regret

353.

proposal

ESTC

8

DRLasso

2.5

DRLasso

DRLasso

25

DRLasso

6

LinUCB

20

LinUCB

3

LinUCB

2015

LinUCB

1.5

1.0

105

2

0.5

0

0.0

0

0

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

Roundt

Roundt

Round t

Round t

T=1000 p=200K=10DGP:3

T=1000 p=200K=10 DGP:4

OL

200

Accumrated Regret

8

proposal

ESTC

Accumrated Regret

15.0

17.5

proposal

ESTC

Accumrated Regret

25

30

ESTC

proposal

Accumrated Regret

175

150

proposal

ESTC

40

DRLasso

LinUCB

12.5

10.0

DRLasso

LinUCB

20

LinUCB

DRLasso

100

125

DRLasso

LinUCB

30

7.5

2

5.0

10

SL

50

10

25

25

0

0.0

0

0

0

200

Roundt

400600

800

1000

0

200

Round t

400600

800

1000

0

200

Roundt

400600

800

1000

0

200

Roundt

400600

800

1000

6 Simulation

We consider two setups: ( 𝐾, 𝑝, 𝑇 ) = ( 2 , 200 , 500 ) and ( 𝐾, 𝑝, 𝑇 ) = ( 10 , 200 , 1000 ) . For each setup, we consider a covariance matrix Σ ( 𝑖 ) = 𝑐 ( 𝑖 ) ¯ Σ where 𝑐 ( 𝑖 ) ∼ Uni ([ 0 . 5 , 1 . 5 ]) and a base covariance ¯ Σ for each 𝑖 ∈ [ 𝐾 ] , which represents a heterogeneous covariance among the arms. The base covariance ¯ Σ follows the following configurations: DGP 1: ¯ Σ = diag ( 𝜆 1 , ..., 𝜆 𝑝 ) , 𝜆 𝑘 = 𝑘 -0 . 5 , DGP 2: ¯ Σ = diag ( 𝜆 1 , ..., 𝜆 𝑝 ) , 𝜆 𝑘 = exp (-𝑘 ) + 𝑇 exp (-𝑇 )/ 𝑝 , DGP 3: ¯ Σ = diag ( 𝜆 1 , ..., 𝜆 𝑝 ) , 𝜆 𝑘 = 𝑘 -1 + 1 / 𝑇 , and DGP 4: ¯ Σ 𝑖, 𝑗 = 0 . 3 and ¯ Σ 𝑖,𝑖 = 0 . 7 for 𝑗 ≠ 𝑖 ∈ [ 𝑝 ] . Note that the DGPs 1 and 3 correspond to Examples 2 and 1 in Proposition 1. DGP 3 is benign as well, while DGP 4 is not. We generate 𝑋 ( 𝑖 ) ( 𝑡 ) from a centered 𝑝 -dimensional Gaussian with covariance Σ ( 𝑖 ) , 𝜃 ( 𝑖 ) from a standard normal distribution which yields non-sparse parameter vectors, and 𝑌 ( 𝑖 ) ( 𝑡 ) by the linear model (1) with the noise with variance 𝜎 2 = 0 . 01. We compare proposal (AEtC, 𝐶 𝑇 = 2) to ESTC [Hao et al., 2020], LinUCB [Li et al., 2010, Abbasi-Yadkori et al., 2011], and DR Lasso Bandit [Kim and Paik, 2019].

Figure 2 illustrates the accumulated regret 𝑅 ( 𝑡 ) in relation to the round 𝑡 ∈ [ 𝑇 ] . In the first three benign DGPs, the AEtC consistently outperforms the other methods. In DGP 2, the advantage of AEtC is insignificant because the regret of any method is already small. This is because in DGP 2, a model with exponentially decaying behavior, only a small fraction of eigenvalues are important. However, AEtC performs significantly better than its competitors in DGP 1 and DGP 3, where the eigenvalues have a heavy-tail distribution. In a non-benign example of DGP 4, the proposed method is still comparable to LinUCB, which demonstrates the utility of an interpolating estimator.

References

Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011. 1, 10, 13

Naoki Abe and Philip M. Long. Associative reinforcement learning using linear probabilistic concepts. In Proceedings of the Sixteenth International Conference on Machine Learning , ICML, page 3-11. Morgan Kaufmann Publishers Inc., 1999. 1

Deepak K. Agarwal, Bee-Chung Chen, Pradheep Elango, and Xuanhui Wang. Personalized click shaping through lagrangian duality for online recommendation. In SIGIR '12 , 2012. 2

Peter Auer, Nicol` o Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn. , 47(2-3):235-256, 2002. doi: 10.1023/A:1013689704352. 3

Peter L Bartlett, Philip M Long, G´ abor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020. 2, 3, 4, 5, 6, 8, 13, 19

Hamsa Bastani and Mohsen Bayati. Online decision making with high-dimensional covariates. Oper. Res. , 68(1):276-294, 2020. 1

Denis Bosq. Linear processes in function spaces: theory and applications , volume 149. Springer Science & Business Media, 2000. 8, 19

Bibhas Chakraborty and Susan A. Murphy. Dynamic treatment regimes. Annual Review of Statistics and Its Application , 1(1):447-464, 2014. 1

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Geoffrey Gordon, David Dunson, and Miroslav Dud´ ık, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 208-214. PMLR, 11-13 Apr 2011. 1

Dylan J. Foster, Akshay Krishnamurthy, and Haipeng Luo. Model selection for contextual bandits. In Advances in Neural Information Processing Systems 32 , pages 14714-14725, 2019. 13

Botao Hao, Tor Lattimore, and Mengdi Wang. High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems , 33:10753-10763, 2020. 1, 3, 10

Satoshi Hara and Takanori Maehara. Enumerate lasso solutions for feature selection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017. 2

Kyoungseok Jang, Chicheng Zhang, and Kwang-Sung Jun. Popart: Efficient sparse regression and experimental design for optimal sparse linear bandits. In NeurIPS , 2022. 1

Gi-Soo Kim and Myunghee Cho Paik. Doubly-robust lasso bandit. Advances in Neural Information Processing Systems , 32, 2019. 1, 10

Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli , 23(1):110-133, 2017. 2, 3, 8, 19

T.L Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics , 6(1):4-22, 1985. ISSN 0196-8858. 1, 3

Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web , WWW'10, page 661-670. Association for Computing Machinery, 2010. 1, 10

Wenjie Li, Adarsh Barik, and Jean Honorio. A simple unified framework for high dimensional bandit problems. In International Conference on Machine Learning , pages 12619-12655. PMLR, 2022. 1, 3

L´ eo Miolane and Andrea Montanari. The distribution of the lasso: Uniform control over sparse balls and adaptive parameter tuning. The Annals of Statistics , 49(4):2313-2335, 2021. 2

Shogo Nakakita and Masaaki Imaizumi. Benign overfitting in time series linear model with overparameterization. arXiv preprint arXiv:2204.08369 , 2022. 13

Min-hwan Oh, Garud Iyengar, and Assaf Zeevi. Sparsity-agnostic lasso bandit. In International Conference on Machine Learning , pages 8271-8280. PMLR, 2021. 1

Steffen Rendle. Factorization machines. In Geoffrey I. Webb, Bing Liu, Chengqi Zhang, Dimitrios Gunopulos, and Xindong Wu, editors, ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17 December 2010 , pages 995-1000. IEEE Computer Society, 2010. 2

Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527 - 535, 1952. 1

Liang Tang, R´ omer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13 , pages 1587-1594. ACM, 2013. 1

Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14286 , 2020. 3, 4, 5, 6, 13, 14

Sara A Van De Geer and Peter B¨ uhlmann. On the conditions used to prove oracle results for the lasso. Electronic Journal of Statistics , 3:1360-1392, 2009. 2

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018. 5

Xue Wang, Mingcheng Wei, and Tao Yao. Minimax concave penalized multi-armed bandit model with high-dimensional covariates. In International Conference on Machine Learning , pages 5200-5208. PMLR, 2018. 1

Yuyan Wang, Long Tao, and Xian Xing Zhang. Recommending for a multi-sided marketplace with heterogeneous contents. In Proceedings of the 16th ACM Conference on Recommender Systems , RecSys '22, page 456-459. Association for Computing Machinery, 2022. 1, 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of high-dimensional linear contextual bandits where the number of features is greater than the budget or even infinite. Unlike previous works, the paper does not impose sparsity on the regression coefficients but instead leverages over parameterized models. The authors propose an explore-then-commit (EtC) algorithm and an adaptive explore-then-commit (AEtC) algorithm to tackle this problem. The performance of the proposed algorithms is evaluated through simulations, and the optimal rate of the ETC algorithm is derived, showing that it can be achieved by balancing exploration and exploitation.\n\nSTRENGTHS:\nThe paper studies a new problem in high-dimensional bandits when there is no sparsity in parameter vector. EtC is first proposed based on the minimum-norm interpolating estimator with the optimal exploration rate, and then AEtC is introduced to deal with the model-dependent parameters on the covariance.\n\nWEAKNESSES:\n1. Lower bound of the problem is not discussed in the paper. Since this is the first work on the high-dimensional bandit without sparsity, a lower bound can help people understand how good your algorithm is in theory. I acknowledge that authors mention this point as limitation in the appendix, but I believe this is an important issue.\n\n2. Assumptions are strong and more explanations will be better. Some assumptions are given in the paper, e.g. Assumption 1, which may not be realistic in practice. For example, I can hardly understand the first part of Assumption 1. What is the intuition behind it? Why you used spectral decomposition to construct $X^{(i)}(t)$? Some examples on which these assumptions hold should be given.\n\n3. More explanation on definitions should be presented. Authors use some definitions in this work that are not commonly used in the bandit community, e.g. effective bias, effective variance, coherent rank, effective rank, benign covariance. Some more explanations on these definitions and maybe correlation between them should be given in the paper.\n\n4. The regret bound in Theorem 4 depends on some $\\alpha > 0$, but this value is not clearly explained in Theorem 4. I think when authors write a formal Theorem in the paper, it is better to explain every term in detail (e.g. how to formulate $\\alpha$ in general).\n\n5. (minor): in the first page, the footnote 1 is under the sentence \"submitted to 37th Conference on .......\", which seems unusual. Maybe the authors change the margin of the template a little bit?\n\nQUESTIONS:\nBesides the questions in the above Weakness section, I have some questions here:\n\n1. For your experiments, in several cases (DGP 1, DGP 2) it seems that the cumulative regret of your algorithm and other existing algorithms is linearly increasing according to the plots. Could authors explain why sublinear pattern is not observed?\n\n2. Can your algorithm still work well when $p$ is greater than $T$ in practice? It might be more comprehensive if such case could be considered in your simulation studies.\n\nLIMITATIONS:\nNo negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a high-dimensional contextual linear bandit setting and uses recent results on \nthe risk of minumum\nnorm linear interpolators to evaluate the regret of two variants of an explore-then-commit algorithm. \nThe first variant makes use of the risk results to balance regret in the \"explore\" vs \"commit\"\nphases, assuming oracle knowledge of the population covariance of the contexts (which \ndetermine the risk, hence regret, of the linear interpolator and derived arms). The second variant\nuses simple sample-covariance-based estimators that do not require such oracle knowledge to inform\nthe stopping of the explore phase, making the algorithm adaptive to the instance.\n\nSTRENGTHS:\nThe paper's ideas are straightforward, and relatively easy to understand once one is familiar\nwith the work analyzing the risk of interpolating linear estimators. The paper also lays\nout the results and proof ideas well. The algorithms are\nalso easy to motivate from that context, and amenable to quick implementation. The setting\nof high-dimensional linear contextual bandits without sparsity is also interesting, pointing towards potentially\ninfinite-dimensional contexts.\n\nWEAKNESSES:\nIn my view, the papers main limitations are related to its reliance on\nthe interpolating estimator work:\n1. It is unclear to me why the paper is focused on covariance structures of the type\nin Proposition 1. Presumably this is because these are the covariances for which the \nminimum norm interpolators have non-trivial risk guarantees, but that seems like\na weak reason to focus solely on them; e.g. one could consider ridge estimators, lasso\nestimators (notwithstanding the lack of sparsity, given it can still have good prediction error), etc.\n2. The authors provide a lower bound for explore then commit algorithms showing that \ntheir upper bound is tight, restricted to this class of algorithms. However, it is not obvious\nthat ucb or thompson sampling algorithms would face the same obstacle in performance. An information-theoretic\nlower bound would be convincing on this score.\n\nQUESTIONS:\n1. Can you relate/situate this work with e.g. the paper \"Model selection for Contextual Bandits\" by Foster et al. and the references therein? \nThe settings are similar, but different in crucial ways (e.g. in their work context covariances are assumed bounded below).\n2. Can you elaborate on the regret dependence in terms of $\\theta_\\max$? Presumably knowledge of this affects the optimal choice of $T_0$?\n3. Can the class of covariance structures considered \"benign\" be expanded if one does not use interpolating estimators?\n4. Can the example of Theorem 5 provide a \"hard\" example for more than explore-then-commit algorithms?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors studied the high-dimensional linear contextual bandit problem where the number of features 𝑝 is greater than the budget 𝑇, or it may even be infinite. \n\nProblem setting: They assumed each arm has an individual hidden parameter, and the contexts are drawn from nature for each round. \n\nThey argue that only by using the notion of effective dimension, in some cases when data distributions have a small effective rank, explore-then-commit algorithm can address this problem.\n\nSTRENGTHS:\nThey adopted the new techniques from [Bartlett et al., 2020] to design an explore-then-commit algorithm with this 'tough' environment - where the dimension $p$ is large compared to the number of samples. Seems like this observation is original.\n\nPlus, they proved that their regret bound is tight by proving several lower bounds. If their arguments are all correct, these results can be significant.\n\nWEAKNESSES:\nI am unsure whether this paper is clear enough.\n\n1) It's not straightforward whether they assumed 'benign covariance matrix' in proposition 1 or their assumption 1 implies benign covariance matrix.\n\n2) I cannot understand how one can take an inverse matrix in line 117 when $p>n$.\n\n3) Eventually, seems like they made only two major assumptions - Assumption 1 with sub-gaussianity, and Assumption 2 which is about the error rate of the OLS. Especially, for AEtC, they only used Assumption 1. I am strongly doubtful that they achieve a meaningful regret bound without any additional assumption on high-dimensional space - As in the first paragraph of Chapter 7, 'High-dimensional statistics' by Martin Wainwright, if the model lacks any additional structure, then there is no hope of getting a consistent estimator in the high-dimensional environment.\n\nQUESTIONS:\n0) In line 117, you mentioned that $\\hat{\\theta}$ can be achieved by simple OLS (ordinary least square) operation. However, you mentioned that $p>n$. How can you define $(XX^\\top)^{-1}$ in this case? The authors also mentioned that their analysis can be executable even when $p=\\infty$ (the first sentence of the abstract.) Please help me to understand this. \n\n\n1) The authors use the word 'optimal balance' when they describe AEtC. Here's the sentence:\n\n> Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. \n\nI want to ask what the word 'optimal' means. I cannot see any lower-bound proof of the exploration in this paper. Did you prove that none of the EtC algorithms can beat the AEtC algorithm? If not, I believe that one should be careful of using the word 'optimal.'\n\n\n2) According to my understanding, AEtC is an algorithm that relieves the assumption on the knowledge of $\\Sigma^{(i)}$ (and naturally Assumption 2) from EtC, not the algorithm that improves the order of regret, am I right?\n\n\n3) I want to know whether you 'assumed' the covariance matrix to be benign, or Assumption 1 (or 2) implies the covariance matrix to be benign. \n\n\n4) In your proposition 1, I want to check that your $\\lambda_k^{(i)}$ is exactly $k^{-1 + 1/T^a}$ or $k^b$, not the orderwise notation, am I right?\n\n\n5) Maybe a silly question, but suppose that we regard your problem as a K-armed bandit problem, without considering the context. Then, isn't this mean that we already have a regret bound of $O(\\sqrt{KT})$? Even for EtC, we have $O(T^{2/3})$ regret bound for K-armed bandit with EtC. What is the main factor that separates your problem from the naive K-armed bandit problem?\n\nLIMITATIONS:\nThere's no negative societal impact of their work. \nI also think they didn't state their limitations or future works properly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the linear contextual bandit problem in high dimensions where the number of features is greater than the budget.  While the recent works impose an assumption about sparsity on the regression coefficients, this paper uses the minimum-norm interpolating estimator which is inspired by recent results on overparameterized models when data distributions have small effective ranks. The authors propose an explore-then-commit algorithm and derive an optimal regret for the proposed algorithm under their setting. This algorithm requires model-dependent parameters on the covariance, which can limit the applications in practice. Therefore, they further propose\nan adaptive explore-then-commit algorithm by estimating these parameters and show that their adaptive version also achieves the optimal rate. Finally,  they provide some simulations to verify the efficiency of the proposed method.\n\nSTRENGTHS:\n- Unlike previous works, the paper solves the high-dimensional linear contextual bandit problem without sparsity. This seems to be the first work in the field of high-dimensional contextual bandits. This paper also proposes new algorithms in the form of the Explore-then-Commit strategy in their setting and derives the optimal regrets for proposed algorithms.\n\n- The organization and the presentation are almost clear to follow.\n\nWEAKNESSES:\n- Although the paper introduces a new direction for the high-dimensional linear contextual bandits, the authors seem not to analyze deeply the difference between their work and the recent works using sparsity (e.g., Kim and Paik, 2019, Bastani and Bayati, 2020, Hao et al., 2022). Without sparsity, their work needs to impose a condition to be benign on the covariance matrix as well as Assumption 2 in the paper. I am wondering if this assumption is weaker than the assumption on the sparsity of the recent works. Thus, I think that a section of the related works and a comparison of assumptions with recent works are needed. \n- The regret bounds in Theorem 4 and Theorem 9 are in terms of $\\alpha$. However, the definition of $\\alpha$ is not clear at all. For examples 1 and 2 in the paper, $\\alpha$ can be understood but in general, what is $\\alpha$?\n-  Although the regret bounds are provided for the proposed algorithms, I think that a comparison of regret bounds with the recent works (e.g., Hao et al., 2022) is also needed. \n-  The used techniques are adapted from the recent work of Bartlett et al., 2020.\n\nQUESTIONS:\n- In Theorem 4, they impose Assumptions 1 and 2. Thus, where is the assumption about the benign covariance?\n- Also see my questions in the Weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper analyzes the Explore-then-commit algorithm for the linear contextual bandit problem in high dimensions, when the number of features is higher than the number of rounds T. The authors use recent studies on overparametrized models for linear regression to analyze the algorithm. Since this algorithm requires the knowledge of the underlying covariance matrices of the contexts, the authors provide an alternate algorithm where they adaptively determine the extent of exploration required.

STRENGTHS:
The paper presents a thorough analysis of the high-dimensional linear contextual bandit problem by providing upper and matching lower bounds on the estimation error. The Adaptive algorithm presented in the paper further adds to the paper's strengths and provides a practical way to solve the problem. The simulations provided demonstrate the the adaptive algorithm works practically in both the benign and non-benign cases.

WEAKNESSES:
The authors could improve the writing of the paper in the following ways:
1. The introduction of the term benign covariance is a bit abrupt and very technical very quickly. The authors could provide more intuition about what it means before defining the bias and variance terms. 

2. The effective rank definitions used in this paper vary slightly from the standard definitions in literature. The authors could maybe visualize the spectra of a matrix and illustrate what these quantities correspond to.

QUESTIONS:
1. Could the authors define the conditional number of a matrix? If they are referring to the standard condition number, how can this number be negative?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12