SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junpei Komiyama
New York University junpei@komiyama.info
Masaaki Imaizumi
The University of Tokyo / RIKEN Center for AIP imaizumi@g.ecc.u-tokyo.ac.jp
In this research, we investigate the high-dimensional linear contextual bandit problem where the number of features  $p$  is greater than the budget  $T$ , or it may even be infinite. Differing from the majority of previous works in this field, we do not impose sparsity on the regression coefficients. Instead, we rely on recent findings on overparameterized models, which enables us to analyze the performance of the minimum-norm interpolating estimator when data distributions have small effective ranks. We propose an explore-then-commit (EtC) algorithm to address this problem and examine its performance. Through our analysis, we derive the optimal rate of the ETC algorithm in terms of  $T$  and show that this rate can be achieved by balancing exploration and exploitation. Moreover, we introduce an adaptive explore-then-commit (AEtC) algorithm that adaptively finds the optimal balance. We assess the performance of the proposed algorithms through a series of simulations.
The multi-armed bandit problem [Robbins, 1952, Lai and Robbins, 1985] has been widely studied in the field of sequential decision-making problems in uncertain environments, and it can be applied to a variety of real-world scenarios. This problem involves an agent selecting one of  $K$  arms in each round and receiving a corresponding reward. The agent aims to maximize the cumulative reward over rounds by using a clever algorithm that balances exploration and exploitation. In particular, a version of this problem called the contextual bandit problem [Abe and Long, 1999, Li et al., 2010] has attracted significant attention in the machine learning community. By observing the contexts associated with the arms, the agent can choose the best arm as a function of the contexts. This extension enables us to model many personalized machine learning scenarios, such as recommendation systems [Li et al., 2010, Wang et al., 2022] and online advertising [Tang et al., 2013], and personalized treatments [Chakraborty and Murphy, 2014].
Most of the papers about stochastic linear bandits assume that the number of features  $p$  is moderate [Li et al., 2010, Chu et al., 2011, Abbasi-Yadkori et al., 2011]. When  $p = o(\sqrt{T})$  to the number of rounds  $T$ , the model is identifiable, and the agent can choose the best arm for most rounds. However, recent machine learning models desire to utilize an even larger number of features, and the theory of bandit models under the identifiability assumption does not necessarily reflect the modern use of machine learning. Several recent papers have overcome this limitation by considering sparse linear bandit models [Wang et al., 2018, Kim and Paik, 2019, Bastani and Bayati, 2020, Hao et al., 2020, Oh et al., 2021, Li et al., 2022, Jang et al., 2022]. Sparse linear bandit models accept a very large number of features<sup>1</sup> and suppress most of the coefficients by introducing the  $\ell 1$  regularize.
Typically, the number of feature  $p$  can be exponential to the number of datapoints  $T$ .
That said, the sparsity imposed by such models limits the applicability of these models. For example, in the case of recommendation models based on factorization, each user is associated with a dense latent vector [Rendle, 2010, Agarwal et al., 2012, Wang et al., 2022], which implies the sparsity is not unlikely the case. Another possible drawback of sparse models is that it requires the condition number to be close to one (e.g., the restricted isometry property, see Van De Geer and Buhlmann [2009] for review). This implies that the quality of the estimator is compromised by the noise on the features that correspond to small eigenvalues. Furthermore, it is still non-trivial to select a proper value of the penalty coefficient as a hyper-parameter. For example, Hara and Maehara [2017] claims that small changes in the choice of coefficients significantly alter feature selection, and Miolane and Montanari [2021] show a limitation of the conventional theory on the choice of penalty coefficients.
In this paper, We consider an alternative high-dimensional linear bandit problem without sparsity. We allow  $p$  to be as large as desired, and in fact, we even allow  $p$  to be infinitely large. Such an overparameterized model has more parameters than the number of training data points. A natural estimator in such a case is an interpolating estimator, which perfectly fits the training data. We adopt recent results that bound the error of the estimator in terms of the effective rank [Koltchinskii and Lounici, 2017, Bartlett et al., 2020] on the covariance of the features. When the eigenvalues of the covariance decay moderately fast, we can obtain a concentration inequality on the squared error of the estimator.
The contributions of this paper are as follows: First, We consider explore-then-commit (EtC) strategy for the stochastic bandit problem based on the minimum-norm interpolating estimator. We derive the optimal rate of exploration that minimizes regret. However, EtC requires model-dependent parameters on the covariance, which limits the practical utility. To address this limitation, we propose an adaptive explore-then-commit (AEtC) strategy, which adaptively estimates these parameters and achieves the optimal rate. We conduct simulations to verify the efficiency of the proposed method.
For  $z \in \mathbb{N}$ ,  $[z] := \{1,2,\ldots,z\}$ . For  $x \in \mathbb{R}$ , the notation  $\lfloor x \rfloor$  here denotes the largest integer that is less than or equal to a scalar  $x$ . For vectors  $X, X' \in \mathbb{R}^p$ ,  $\langle X, X' \rangle := X^\top X'$  is an inner product,  $\|X\|_2^2 := \langle X, X \rangle$  is an  $\ell 2$ -norm. For a positive-definite matrix  $A \in \mathbb{R}^p \times \mathbb{R}^p$ ,  $\|X\|_{\mathbf{A}}^2 := \langle X, \mathbf{A}X \rangle$  is a weighted  $\ell 2$ -norm.  $\|\mathbf{A}\|_{\mathrm{op}}$  denotes an operator norm of  $\mathbf{A}$ .  $O(\cdot), o(\cdot), \Omega(\cdot), \omega(\cdot)$  and  $\Theta(\cdot)$  denote Landau's Big-O, little-o, Big-Omega, little-omega, and Big-Theta notations, respectively.  $\widetilde{O}(\cdot), \widetilde{\Omega}(\cdot)$ , and  $\widetilde{\Theta}(\cdot)$  are the notations that ignore polylogarithmic factors.
2.2 Problem Setup
This paper considers a linear contextual bandit problem with  $K$  arms. We consider the fully stochastic setting, where the contexts, as well as the rewards, are drawn from fixed distributions. For each round  $t \in [T]$  and arm  $i \in [K]$ , we define  $X^{(i)}(t)$  as a  $p$ -dimensional zero-mean sub-Gaussian vector. We assume  $X^{(i)}(t)$  is independent among rounds (i.e., vectors in two different rounds  $t, t'$  are independent) but allow vectors  $X^{(1)}(t), X^{(2)}(t), \ldots, X^{(K)}(t)$  to be correlated with each other. The forecaster chooses an arm  $I(t) \in [K]$  based on the  $X^{(i)}(t)$  values of all the arms, and then observes a reward that follows a linear model as shown in
$$
Y ^ {(I (t))} (t) = \left\langle X ^ {(I (t))} (t), \theta^ {(I (t))} \right\rangle + \xi (t). \tag {1}
$$
The unknown true parameters  $\theta^{(i)}$  for each arm  $i\in [K]$  lie in a parameter space  $\Theta \subset \mathbb{R}^p$ , and the independent noise term  $\xi (t)$  has zero mean and variance  $\sigma^2 >0$ . We assume that  $\xi (t)$  is sub-Gaussian, and for the sake of simplicity, we assume that it does not depend on the choice of the arm. However, our results can be extended to the case where  $\xi (t)$  varies among arms. We assume that each  $\theta^{(i)}$  are bounded  $\| \theta^{(i)}\| _2\leq \theta_{\mathrm{max}}$ . For each  $i\in [K]$ , we define a covariance matrix  $\Sigma^{(i)} = \mathbb{E}[X^{(i)}(t)X^{(i)}(t)^{\top}]\in \mathbb{R}^{p\times p}$ .
We define  $i^{*}(t) \coloneqq \operatorname{argmax}_{i \in [K]} \langle X^{(i)}(t), \theta^{(i)} \rangle$  as the (ex ante) optimal arm at round  $t$ . Our goal is to design an algorithm that maximizes the total reward, which is equivalent to minimizing the following
expected regret [Lai and Robbins, 1985, Auer et al., 2002];
$$
R (T) := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\langle X ^ {(i ^ {*} (t))} (t), \theta^ {(i ^ {*} (t))} \rangle - \langle X ^ {(I (t))} (t), \theta^ {(I (t))} \rangle\right) \right], \tag {2}
$$
where the expectation is taken with respect to the randomness of the contexts and (possibly) on the choice of arm  $I(t)$ .
2.3 Theory of Overparametrized Models
The primary focus of this paper is on scenarios where the number of arms  $K$  is moderate, but the number of features  $p$  is greater than the budget  $T$ , possibly to the point of being infinite. The efficiency of linear regression models in such scenarios depends significantly on the covariance matrix,  $\Sigma^{(i)}$ . Unlike sparsity, the theory of benign overfitting tightly examines errors using the decay of the eigenvalues of the covariance matrix of the context. In particular, if the decay rate of the eigenvalues is at a certain level, the error in linear regression converges to zero, even in high-dimensional spaces. To provide a more rigorous analysis of eigenvalue decay, the concept of effective rank is introduced in Section 2.4.
Remark 1. (Dependence on  $T$ ) In accordance with Bartlett et al. [2020], we permit the covariance matrix  $\Sigma^{(i)}$  to depend on  $T$ . In other words, we consider the sequence of covariances  $\Sigma^{(i)}(1), \Sigma^{(i)}(2), \ldots$  for each  $T = 1, 2, \ldots$ . The linear regression is consistent if the effective rank of  $\Sigma^{(i)}(T)$  grows sufficiently slow as a function of  $T$ .
2.4 Effective Ranks of Covariance Matrix
For a covariance matrix  $\Sigma^{(i)}$  for  $i\in [K]$  and  $k\in [p]$ , let  $\lambda_k^{(i)}$  be its  $k$ -th largest eigenvalue, such that  $\Sigma^{(i)} = \sum_{k = 1}^{p}\lambda_k^{(i)}u_k^{(i)}(u_k^{(i)})^\top$  with order  $\lambda_1^{(i)}\geq \lambda_2^{(i)}\geq \dots \geq \lambda_p^{(i)}$  and eigenvectors  $u_{k}^{(i)}$ . We define the concept of effective rank as
$$
r _ {k} \left(\Sigma^ {(i)}\right) := \frac {\sum_ {j > k} \lambda_ {j} ^ {(i)}}{\lambda_ {k + 1} ^ {(i)}}, \text {a n d} R _ {k} \left(\Sigma^ {(i)}\right) := \frac {\left(\sum_ {j > k} \lambda_ {j} ^ {(i)}\right) ^ {2}}{\sum_ {j > k} \left(\lambda_ {j} ^ {(i)}\right) ^ {2}}. \tag {3}
$$
The first quantity  $r_k(\Sigma^{(i)})$  is related to the trace of  $\Sigma^{(i)}$ , and the second quantity  $R_k(\Sigma^{(i)})$  is related to the decay rate of the eigenvalues.
The effective rank is used as a measure of the intrinsic complexity of high-dimensional data, by rigorously capturing a decay rate of eigenvalues. If the covariance matrix  $\Sigma^{(i)}$  is an identity matrix of size  $p$ , then  $r_0(\Sigma^{(i)})$  and  $R_0(\Sigma^{(i)})$  are both equal to  $p$  (the rank of  $\Sigma^{(i)}$ ). However, we anticipate that these quantities will be less than  $p$ , which enables learning with fewer samples. In Figure 1, we plot of the effective rank in  $k$  with certain cases of eigenvalues  $\{\lambda_k\}_k$  of  $\Sigma$ . Even though  $\Sigma$  is full-rank, the effective rank  $R_{k}(\Sigma)$  is sublinear in  $k$ , which reflects the intrinsic low-complexity of data.
Because of this property, the effective ranks are used in modern high-dimensional statistics. Koltchinskii and Lounici [2017] evaluated concentration inequalities for high-dimensional random matrices using the notion. Bartlett et al. [2020], Tsigler and Bartlett [2020] evaluated the prediction error of a high-dimensional linear regression using the effective ranks and showed the consistency of the prediction.
3 Explore-then-Commit with Interpolation
The Explore-then-Commit (EtC) algorithm is a well-known approach for solving high-dimensional linear bandit problems, and it has been shown to be effective in previous studies such as Hao et al. [2020], Li et al. [2022]. The EtC algorithm operates by first conducting  $T_0 = NK < T$  rounds of exploration, during which it uniformly explores all available arms to construct an estimator  $\widehat{\theta}^{(i)}$  for each arm  $i \in [K]$ . After the exploration phase, the algorithm proceeds with exploitation. Let  $N$  be the number of the draws of arm  $i$ , and  $\mathbf{X}^{(i)} = (X_1^{(i)},\dots,X_N^{(i)})^\top \in \mathbb{R}^{N\times p}$  and  $\mathbf{Y}^{(i)} = (Y_1^{(i)},\dots,Y_N^{(i)})^\top \in \mathbb{R}^N$ .
Figure 1: Eigenvalues  $\lambda_{k}$  (left), the effective rank  $r_k(\Sigma)$  (middle), and the effective rank  $R_{k}(\Sigma)$  (right) with  $k = 1,\ldots ,15$  and  $T = 3$ ; Case 1:  $\lambda_{k} = Ck^{-1 + 1 / T^{0.99}}$ , and Case 2:  $\lambda_{k} = C(\exp (-k) + T\exp (-T) / p)$ . Case 1 considers a decay that is slightly faster than  $k^{-1}$ , whereas Case 2 considers an exponentially fast decay. The slower increase of  $r_k(\Sigma)$  and  $R_{k}(\Sigma)$  in Case 2 reflects the impact of the eigenvalues' faster decay.
be the observed contexts and rewards of arm  $i$ , where  $(X_{n}^{(i)},Y_{n}^{(i)})$  is the corresponding values on the  $n$ -th draw of arm  $i$ . Since we choose  $I(t)$  uniformly during the exploration phase, these are independent and identically drawn samples from the corresponding distribution.
For estimating the parameter  $\theta^{(i)}$ , we consider the minimum-norm interpolating estimator that perfectly fits the data, which reveals its advantage in recent results on high-dimensions [Bartlett et al., 2020]. Rigorously, we assume  $p > N$  and and consider the following definition:
$$
\widehat{\theta}^{(i)}:= \operatorname *{argmin}\left\{\| \theta \|_{2}\bigg|\theta \in \mathbb{R}^{p},0 = \sum_{\substack{(Y_{n}^{(i)},X_{n}^{(i)}):n\leq N}}(Y_{n}^{(i)} - \langle \theta ,X_{n}^{(i)}\rangle)^{2}\right\} .
$$
This estimator has a simple representation  $\widehat{\theta}^{(i)} = (\mathbf{X}^{(i)})^{\top}(\mathbf{X}^{(i)}(\mathbf{X}^{(i)})^{\top})^{-1}\mathbf{Y}^{(i)}$ . The EtC algorithm is presented in Algorithm 1, which utilizes the aforementioned interpolating estimator. In the subsequent sections, we will first discuss the assumptions on the data-generating process, followed by an analysis of the accuracy of the estimator. We then present an upper bounds on the regret of the EtC algorithm.
4 Theory of Explore-then-Commit
This section analyzes the EtC algorithm. If we were to fix the number of samples, this theory would largely align with the existing literature [Bartlett et al., 2020, Tsigler and Bartlett, 2020]. However, the unique challenge in our case arises from the fact that the EtC selects the number of samples used to construct an estimator to balance between exploration and exploitation.
4.1 Assumption and Notion
We consider a spectral decomposition  $\Sigma^{(i)} = V^{(i)}\Lambda^{(i)}(V^{(i)})^{\top}$  with the matrices (operators)  $V^{(i)}$  and  $\Lambda^{(i)}$  for each arm  $i\in [K]$ , then impose the following assumptions.
Assumption 1 (sub-Gaussianity). For all  $t \in \mathbb{N}$  and  $i \in [K]$ , the followings hold:
Algorithm 1 Explore-then-Commit (EtC)
Require: Exploration duration  $T_0$    
for  $t = 1,\dots,T_0$  do Observe  $X^{(i)}(t)$  for all  $i\in [K]$  Choose  $I(t) = t - K\lfloor t / K\rfloor$  Receive a reward  $Y^{(I(t))}(t)$    
end for   
for  $i\in [K]$  do  $\widehat{\theta}^{(i)}\gets (\mathbf{X}^{(i)})^{\top}(\mathbf{X}^{(i)}(\mathbf{X}^{(i)})^{\top})^{-1}\mathbf{Y}^{(i)}$    
end for   
for  $t = T_0 + 1,\ldots ,T$  do Observe  $X^{(i)}(t)$  for all  $i\in [K]$  Choose  $I(t) = \mathrm{argmax}_{i\in [K]}\langle X^{(i)}(t),\widehat{\theta}^{(i)}\rangle .$  Receive a reward  $Y^{(I(t))}(t)$    
end for
- There exists a random vector  $Z^{(i)}(t)$  such that  $X^{(i)}(t) = V^{(i)}(\Lambda^{(i)})^{1/2}Z^{(i)}(t)$  which is independent elements and sub-Gaussian with a parameter  $\kappa_x^2 > 0$ , that is, for all  $\lambda \in \mathbb{R}^p$ , we have  $\mathbb{E}[\exp(\langle \lambda, Z^{(i)}(t) \rangle)] \leq \exp(\kappa_x^2\|\lambda\|_2^2/2)$ .
- Moreover,  $\xi(t)$  is conditionally sub-Gaussian with a parameter  $\kappa_{\xi}^{2} > 0$ , that is, for all  $\lambda \in \mathbb{R}$ , we have  $\mathbb{E}[\exp(\lambda \xi(t)) \mid X^{(i)}(t)] \leq \exp(\kappa_{\xi}^{2} \lambda^{2}/2)$ .
The first bullet point of Assumption 1 requires that the features are multivariate sub-Gaussian and their tail is not too heavy, which is important for concentration inequalities on random variables [Vershynin, 2018]. The second bullet point of Assumption 1, which states that rewards involve sub-Gaussian noise, is a common assumption in contextual bandit problems.
4.2 Benign Covariance
We impose a condition to be benign on the covariance matrix  $\Sigma^{(i)}$  for all  $i\in [K]$ . The condition is described using the notion of effective/coherent rank, which is commonly applied to the study of benign overfitting [Bartlett et al., 2020, Tsigler and Bartlett, 2020]. However, unlike those papers that estimate using all  $T$  samples, we only use a portion of the data for learning.
We first define the coherent rank of  $\Sigma^{(i)}$  with a number of samples  $N$  as
$$
k _ {N} ^ {*} = k _ {N} ^ {*} \left(\Sigma^ {(i)}, N\right) := \min  \{k \in \mathbb {N} \cup \{0 \} \mid r _ {k} \left(\Sigma^ {(i)}\right) \geq N \},
$$
where we define  $\min\{\emptyset\} = +\infty$ . By using the coherent rank, we decompose the error into two quantities called effective bias/variance denoted as  $B_{N,T}^{(i)}$  and  $V_{N,T}^{(i)}$ , based on a budget of  $T$  and the number of samples  $N$  used for estimation.
$$
B _ {N, T} ^ {(i)} := \lambda_ {k _ {N} ^ {*}} ^ {(i)}, \text {a n d} V _ {N, T} ^ {(i)} := \left(\frac {k _ {N} ^ {*}}{N} + \frac {N}{R _ {k _ {N} ^ {*}} ^ {*} (\Sigma^ {(i)})}\right). \tag {4}
$$
Definition 1 (Benign covariance). Under Assumption 1, a covariance matrix  $\Sigma^{(i)}$  is benign, if  $B_{N,T}^{(i)} = o(1)$  and  $V_{N,T}^{(i)} = o(1)$  hold as  $N, T \to \infty$  while  $N = \Theta(T)$ .
To put it differently, Assumption 1 and Definition 1 state that, if we can achieve consistent estimation by using all the data for estimation, then the covariance matrix is considered benign.
Technically, the benign property implies that eigenvalues decay fast enough compared with  $T$ . In particular, the following examples have been considered in the literature.
Proposition 1 (Example of Benign Covariance, Theorem 31 in Bartlett et al. [2020]). A covariance matrix  $\Sigma^{(i)}$  with eigenvalues  $\lambda_1^{(i)},\lambda_2^{(i)},\ldots$  is benign if it satisfies one of the following.
It should be noted that Bartlett et al. [2020] only provided the variance term. Later on, Tsigler and Bartlett [2020] described both the variance and bias terms, which we follow.
- Example 1: Let  $a \in (0,1)$ ,  $p = \infty$  and  $\lambda_k^{(i)} = k^{-(1 + 1 / T^a)}$ . In this case, we have  $B_{N,T}^{(i)} = O((T^a / N)^{1 + 1 / T^a})$  and  $V_{N,T}^{(i)} = O(T^a / N + T^{-a})$ . For this model to be learnable,  $N \gg T^a$  is required, and the variance dominates the bias.
Example 2: Let  $\lambda_k^{(i)} = k^{-b}$  and  $p = p_T = O(T^c)$  with  $b \in (0,1)$  and  $c \in \left(\max\left(1, \frac{2}{2-b}, \frac{1}{1-b^2}\right), \frac{1}{1-b}\right)$ . In this case, we have  $B_{N,T}^{(i)} = O(T^{c(1-b)} / N)$  and  $V_{N,T}^{(i)} = O((N / T^c)^{\max\left(1, \frac{1-b}{b}\right)})$ . For this model to be learnable,  $N \gg T^{c(1-b)}$  is required, and the bias dominates the variance.
The first example is when the decay rate is near  $k^{-1}$ , but the trace is  $O(T^a)$ . The second example is when the decay rate is smaller than  $k^{-1}$ , and the trace is  $O(T^{c(1 - b)})$ . Note that Bartlett et al. [2020] provided two other examples of the benign covariance matrices. Our analysis mainly focuses on the two examples in Proposition 1, but another example is also empirically tested in Section 6.
4.3 Estimation Error by Exploration
We evaluate an error in the estimator  $\widehat{\theta}^{(i)}$  by its prediction quality. That is, with a covariance matrix  $\Sigma^{(i)}$  and an identical copy  $X_{*}^{(i)}$  of  $X_{1}^{(i)}$ , we study
$$
\| \theta^ {(i)} - \widehat {\theta} ^ {(i)} \| _ {\Sigma^ {(i)}} ^ {2} = \mathbb {E} _ {X _ {*} ^ {(i)}} \left[ (\langle \theta^ {(i)}, X _ {*} ^ {(i)} \rangle - \langle \widehat {\theta} ^ {(i)}, X _ {*} ^ {(i)} \rangle) ^ {2} \right],
$$
The following result bounds the error of the estimator  $\widehat{\theta}^{(i)}$  in terms of bias and variance, which is a slight extension of Tsigler and Bartlett [2020]. For  $k\in [p]$ , we define an empirical submatrix as  $\mathbf{X}_{k:\infty}^{(i)}\in \mathbb{R}^{N\times (p - k)}$  as the  $p - k$  columns to the right of  $\mathbf{X}^{(i)}$ , and define a Gram sub-matrix  $A_{k}^{(i)} = \mathbf{X}_{k:\infty}^{(i)}(\mathbf{X}_{k:\infty}^{(i)})^{\top}\in \mathbb{R}^{N\times N}$ .
Theorem 2. Suppose Assumption 1. If there exists  $c_{U} > 1$  such that  $k_{N}^{*} < N / c_{U}$  and a condition number of  $A_{k}^{(i)}$  is positive with probability at least  $1 - c_{U}e^{-N / c_{U}}$ , then we have
$$
\left\| \widehat {\theta} ^ {(i)} - \theta^ {(i)} \right\| _ {\Sigma^ {(i)}} ^ {2} \leq C _ {U} \left(B _ {N, T} ^ {(i)} + V _ {N, T} ^ {(i)}\right), \tag {5}
$$
with some constant  $C_U > 0$  and probability at least  $1 - 2c_{U}e^{-N / c_{U}}$ .
Theorem 2 implies that the estimation error converges to zero if  $\Sigma^{(i)}$  has the benign property. The assumption on the condition number of  $A_{k}^{(i)}$  has a sufficient condition, which is provided in Lemma 3 in Tsigler and Bartlett [2020].
Moreover, the following lemma implies the tightness of the analysis in Theorem 2.
Lemma 3 (Lower bound of estimation error, Theorem 10 in Tsigler and Bartlett [2020]). Suppose Assumption 1. These exist some constants  $c_{L}, C_{L} > 0$  such that, with probability at least  $1 - c_{L}e^{-N / c_{L}}$ , we have
$$
\left\| \widehat {\theta} ^ {(i)} - \theta^ {(i)} \right\| _ {\Sigma (i)} ^ {2} \geq C _ {L} \left(B _ {N, T} ^ {(i)} + V _ {N, T} ^ {(i)}\right). \tag {6}
$$
In other words, the upper bound in Theorem 2 is optimal up to a constant.4
4.4 Regret Bound of Explore-then-Commit
This section analyzes the EtC algorithm. We introduce an error function  $\operatorname{E}(N, T)$  that characterizes the rate of regret, which can be obtained by considering  $(B_{N, T}^{(i)} + V_{N, T}^{(i)})^{1/2}$ .
Assumption 2. (Error function<sup>5</sup>) There exists a continuous function  $\mathbf{E} : \mathbb{R}^2 \to \mathbb{R}^+$  that is decreasing in  $N$  such that  $\| \widehat{\theta}^{(i)} - \theta^{(i)} \|_{\Sigma^{(i)}} = \widetilde{\Theta}(\mathbf{E}(N, T))$  as  $N, T \to \infty$ .
3One of the omitted examples is the case where eigenvalues decay slightly slower than Example 1. The other example is the case where eigenvalues decay exponentially but has a noise term.
4Note that the constant here can depend on model parameters.
The coherent rank  $k_N^*$  as well as  $B_{N,T}^{(i)},V_{N,T}^{(i)}$  are discrete in  $N,T$ , and the error function  $\operatorname {E}(N,T)$  is introduced to circumvent the issues related to this discreteness.
Assumption 2 is satisfied in Examples 1 and 2. In particular, for Example 1 in Proposition 1, the error function is given as  $\operatorname{E}(N,T) = \sqrt{T^a / N + T^{-a}}$ , while for Example 2 in Proposition 1, it is given as  $\operatorname{E}(N,T) = \sqrt{T^{c(1 - b)} / N}$ .
Theorem 4. Suppose Assumptions 1-2. Suppose that we run the EtC algorithm (Algorithm 1). Then, regret (2) satisfies
$$
R (T) = \widetilde {O} (L (T, K)),
$$
as  $T\to \infty$  with some  $\alpha >0$  . where  $L(T,K)$  is such that
$$
N ^ {*} = \min  _ {N} \{N: N \geq T \mathrm {E} (N / K, T) \}. \tag {7}
$$
Specifically, for Example 1 in Proposition 1, we have  $L(T,K) = \max (T^{(2 + a) / 3}K^{2 / 3},T^{1 - a / 2})$ , whereas for Example 2 in Proposition 1, we have  $L(T,K) = T^{(2 + c(1 - b)) / 3}K^{2 / 3}$ .