SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof sketch of Theorem 4. We show that the regret-per-round is  $O(1)$  during the exploration phase. Moreover, regret-per-round is  $\widetilde{O}(\mathrm{E}(T_0 / K, T))$  during the exploitation phase. The total regret is  $T_0 + \widetilde{O}(\mathrm{E}(T_0 / K, T))T$ , and optimizing  $T_0$  by using the decreasing property of  $\mathrm{E}(N, T)$  in  $N$  yields the desired result.
4.5 Matching Lower Bound
We show that this choice of  $T_0$  as a function of  $T$  is indeed optimal.
Theorem 5. Suppose Assumptions 1-2. Assume that we run the EtC algorithm (Algorithm 1). For any choice of  $T_0$ , the following event occurs with strictly positive probability as  $T \to \infty$ :
$$
R (T) = \widetilde {\Omega} (L (T, 3)).
$$
Proof sketch of Theorem 5. We explicitly construct a model with  $K = 3$ . Let  $\epsilon_{T} = \mathrm{E}(T_{0} / K, T)$ . In the model, the  $\Sigma^{(1)}, \Sigma^{(2)}, \Sigma^{(3)}$  are identical, and the only difference is that the first coefficients  $\theta_{1}^{(1)} = 1, \theta_{1}^{(2)} = \Theta(\epsilon_{T}), \theta_{1}^{(3)} = 0$ . All other coefficients are set to zero. Roughly speaking, the gap is
$$
\left| \langle X ^ {(i)} (t), \widehat {\theta} ^ {(i)} \rangle - \langle X ^ {(j)} (t), \widehat {\theta} ^ {(j)} \rangle \right| = \left\{ \begin{array}{l l} \Theta (1) & (i = 1, j = 2, 3) \\ \Theta (\epsilon_ {T}) & (i = 2, j = 3) \end{array} \right..
$$
The regret-per-round during the exploration phase is  $\Theta(1)$  due to a misidentification of the best arm between arm 1 and arms  $\{2,3\}$ . The regret-per-round during the exploitation phase is  $\Theta(\epsilon_T)$  due to a misidentification of the best arm between arm 2 and arm 3.
5 Adaptive Explore-then-Commit (AEtC) Algorithm
In the prior section, we demonstrated that the optimal way to minimize EtC's regret is by selecting  $T_0$ , with  $T_0$  balancing the exploration and the exploitation. However, this requires knowledge of the covariance matrix's spectrum, which can be difficult to obtain in advance in certain scenarios. This section explores the way to adaptively determines the extent of exploration required.
5.1 Estimator
We assume that  $\Sigma^{(i)}$  follows the data-generating process of Example 1 or 2 in Proposition 1. We use  $\beta_{T}$  to denote that
$$
\lambda_ {j} ^ {(i)} = C _ {\lambda} j ^ {- \beta_ {T}}, \tag {8}
$$
with some constant  $C_{\lambda} > 0$ . We have  $\beta_{T} = 1 + 1 / T^{a}$  for Example 1, and  $\beta_{T} = b(< 1)$  for Example 2.
Balancing exploration and exploitation in an overparameterized model is challenging for the following reasons. First, the number of features  $p$  is very large or even infinite<sup>6</sup>. Second, the trace is heavy-tail
6Namely,  $p = \infty$  for for Proposition 1 (1) or  $p = T^c$  in for Proposition 1 (2).
because the decay is not very fast. As a result, a naive use of a traditional method does not work. We address this issue by utilizing two estimators. The first estimator is on the trace  $\mathrm{tr}(\widehat{\Sigma}^{(i)})$  that we extracted from overparameterization theory [Bartlett et al., 2020], whereas the second estimator is on the estimated decay rate  $\widehat{\beta}_T$  that derives from Bosq [2000], Koltchinskii and Lounici [2017].
For an arm  $i \in [K]$  with  $N = T_0 / K$  draws, we define an estimated eigenvalues  $\widehat{\lambda}_1^{(i)}, \dots, \widehat{\lambda}_N^{(i)} > 0$  from an empirical covariance matrix  $\widehat{\Sigma}^{(i)} \coloneqq N^{-1} \sum_{j=1}^{N} X_j^{(i)}(t) X_j^{(i)}(t)^\top$ . We define the empirical trace to be  $\operatorname{tr}(\widehat{\Sigma}^{(i)}) = \sum_j \widehat{\lambda}_j^{(i)}$ . The following lemma states the consistency of the estimated trace under very mild conditions.
Lemma 6. (Error bound of empirical trace) Suppose Assumption 1. Suppose the data generating process (DGP) of Example 1 or Example 2 in Proposition 1. Then, for any  $C_{\mathrm{poly}} > 0$ , with probability at least  $1 - 1 / T^{C_{\mathrm{poly}}}$ , we have the following as  $T \to \infty$ :
$$
\frac {\left| \operatorname {t r} \left(\Sigma^ {(i)}\right) - \operatorname {t r} \left(\widehat {\Sigma} ^ {(i)}\right) \right|}{\operatorname {t r} \left(\Sigma^ {(i)}\right)} = \widetilde {O} \left(\frac {\sqrt {\sum_ {j} \left(\lambda_ {j} ^ {(i)}\right) ^ {2}}}{\sum_ {j} \lambda_ {j} ^ {(i)}}\right) = o (1). \tag {9}
$$
Moreover, the following bounds the estimation error of each eigenvalue.
Lemma 7. Suppose Assumption 1. For any  $\delta \in (0,1)$ , for any  $C_{\mathrm{poly}} > 0$ , with probability at least  $1 - 1 / T^{C_{\mathrm{poly}}}$ , we have the following for any  $j = 1,\dots,p$  as  $T\to \infty$ :
$$
\left| \widehat {\lambda} _ {j} ^ {(i)} - \lambda_ {j} ^ {(i)} \right| = O \left(\sqrt {\frac {\operatorname {t r} \left(\Sigma^ {(i)}\right)}{N}}\right). \tag {10}
$$
Lemma 7 implies the estimator of each eigenvalue is  $o(1)$  if we choose  $N \gg \operatorname{tr}(\Sigma^{(i)})$ . However, even if we choose a large  $N$ , the error is still non-negligible for the tail of eigenvalues where  $|\widehat{\lambda}_j^{(i)} - \lambda_j^{(i)}|$  is very small<sup>7</sup>. Consequently,  $\widehat{\lambda}_j^{(i)}$  for large  $j$  are not necessarily consistent, so as to the effective rank and the coherent rank estimated from them. To address this, we estimate the decay rate  $\beta_T$ , and then estimate the subsequent statistics.
Let the estimated decay rate be  $\widehat{\beta}_T = (1 / \tau)\sum_{k = 1}^{\tau}\log (\widehat{\lambda}_k / \widehat{\lambda}_{k + 1}) / \log ((k + 1) / k)$ . Theoretically,  $\widehat{\beta}_T$  is consistent for any constant  $\tau >1$ . In practice, we can use a reasonable constant  $\tau$ , such as  $\tau = 10$ , for robustness. To estimate the effective ranks, we use the following form
$$
\widetilde {\lambda} _ {k} ^ {(i)} = \widehat {\lambda} _ {1} ^ {(i)} k ^ {- \widehat {\beta} _ {T}}.
$$
Namely, we consider empirical analogues of the effective rank for  $k$  as
$$
\widehat {r} _ {k} (\Sigma^ {(i)}) := \frac {\operatorname {t r} (\widehat {\Sigma} ^ {(i)})}{\widetilde {\lambda} _ {k + 1} ^ {(i)}}, \text {a n d} \widehat {R} _ {k} (\Sigma^ {(i)}) := \frac {(\operatorname {t r} (\widehat {\Sigma} ^ {(i)})) ^ {2}}{\sum_ {j > k} (\widetilde {\lambda} _ {j} ^ {(i)}) ^ {2}}.
$$
We also define an estimator for the coherent rank as  $\widehat{k}_N\coloneqq \min \{k\in \mathbb{N}\cup \{0\} \mid \widehat{r}_k(\Sigma^{(i)})\geq N\}$ . Further, we define estimators of  $B_{N,T}^{(i)}$ ,  $V_{N,T}^{(i)}$  of Eq. (4) as
$$
\widehat {B} _ {N, T} ^ {(i)} := \widetilde {\lambda} _ {\widehat {k} _ {N}} ^ {(i)}, \mathrm {a n d} \widehat {V} _ {N, T} ^ {(i)} := \left(\frac {\widehat {k} _ {N}}{N} + \frac {N}{\widehat {R} _ {\widehat {k} _ {N}} (\Sigma^ {(i)})}\right),
$$
Note that the estimators  $\widehat{r}_k$  and  $\widehat{R}_k$  use the trace  $\mathrm{tr}(\widehat{\Sigma}^{(i)})$ , which is a sum of eigenvalues, instead of the partial sum of the eigenvalues that constitutes the effective rank of Eq. (3). Despite this change, this does not affect<sup>8</sup> asymptotic consistency of the coherent rank estimator  $\widehat{k}_N$ .
The following lemma states that small  $\tau$  suffices to assure the quality of  $\widehat{\beta}^{(i)}$ . We study a convergence rate of  $\widehat{\beta}^{(i)}$  and the other estimators as follows.
7Remember that we consider  $\Sigma^{(i)} = \Sigma^{(i)}(T)$  that depends on  $T$ . Tail of eigenvalues are  $o(1)$  to  $T$  as well.
This is because  $\sum_{j} j^{-\beta_T}$  with  $\beta_T < 1$  or  $\beta_T \approx 1$  is tail-heavy.
Algorithm 2 Adaptive Explore-then-Commit (AEtC)
while  $t = 1,2,3,\ldots$  do Observe  $X^{(i)}(t)$  for all  $i\in [K]$  Choose  $I(t) = t - K\lfloor t / K\rfloor$  Receive a reward  $Y^{(I(t))}(t)$  if  $t\in \{\lfloor e^N\rfloor \mid N\in \mathbb{N},N\geq \log T\}$  then if  $\mathrm{Stop}(t / K)$  then  $T_0\gets t$  Break. end if end if end while   
for  $i\in [K]$  do  $\widehat{\theta}^{(i)}\leftarrow (\mathbf{X}^{(i)})^\top (\mathbf{X}^{(i)}(\mathbf{X}^{(i)})^\top)^{-1}\mathbf{Y}^{(i)}$    
end for   
for  $t = T_0 + 1,\dots,T$  do Observe  $X^{(i)}(t)$  for all  $i\in [K]$  Choose  $I(t) = \operatorname {argmax}_{i\in [K]}\langle X^{(i)}(t),\widehat{\theta}^{(i)}\rangle .$  Receive a reward  $Y^{(I(t))}(t)$    
end for
Lemma 8. Suppose Assumption 1. Suppose DGP of Example 1 or Example 2 in Proposition 1. Assume that we choose  $N = N(T)$  such that  $\mathrm{tr}(\Sigma^{(i)}) / N = o(1)$  for all  $i$ . Then, for any  $C_{\mathrm{poly}} > 0$  with probability at least  $1 - 1 / T^{C_{\mathrm{poly}}}$ , we have
$$
\frac {| \widetilde {\lambda} _ {k} ^ {(i)} - \lambda_ {k} ^ {(i)} |}{\lambda_ {k} ^ {(i)}} = o (1),
$$
as  $T\to \infty$  for all  $k\in [\tau ]$  . Moreover, it implies  $|\beta_T - \widehat{\beta}_T| = o(1)$  and
$$
\max  \left\{\frac {\widehat {B} _ {N , T} ^ {(i)}}{B _ {N , T} ^ {(i)}}, \frac {\widehat {V} _ {N , T} ^ {(i)}}{V _ {N , T} ^ {(i)}} \right\} = T ^ {o (1)}. \tag {11}
$$
Eq. (11) states that the estimated rate of error is accurate as  $T \to \infty$ . To see this, for Example 1 in Proposition 1, the estimated error rate is  $T^{o(1)}(\sqrt{T^a / N + T^{-a}})$ , whose exponent approaches to  $\sqrt{T^a / N + T^{-a}}$  as  $T \to \infty$ .
5.2 Adaptive Explore-then-Commit (AEtC)
The Adaptive Explore-then-Commit (AEtC) algorithm is described in Algorithm 2. Unlike EtC, the amount of exploration in AEtC is adaptively determined based on the stopping condition:
$$
\operatorname {S t o p} ^ {(i)} (N) = \left\{N > C _ {T} \operatorname {t r} (\widehat {\Sigma} ^ {(i)}) \cap N K \geq T \sqrt {\widehat {B} _ {N , T} ^ {(i)} + \widehat {V} _ {N , T} ^ {(i)}} \right\},
$$
and  $\mathrm{Stop} = \cap_{i\in [K]}\mathrm{Stop}^{(i)}$ , where  $C_T = C_T(T)$  is a function of  $T$  that slowly diverges to  $+\infty$ . The first condition  $N > C_T\operatorname {tr}(\widehat{\Sigma}^{(i)})$  ensures that  $N$  is large enough to have consistency on the estimators, whereas the second condition  $NK\geq T\sqrt{\widehat{B}_{N,T}^{(i)} + \widehat{V}_{N,T}^{(i)}}$  balances the exploration and exploitation. The following theorem bounds the regret of AEtC.
Theorem 9. Suppose Assumption 1. Suppose DGP of Example 1 or Example 2 in Proposition 1. Then, for any  $c > 0$ , the regret (2) when we run the AEtC algorithm (Algorithm 2) with a sufficiently slowly diverging  $C_T$  is bounded as follows as  $T \to \infty$ :
$$
R (T) = \widetilde {O} (L (T, K) T ^ {c}),
$$
where  $L(T,K)$  is the function defined in Theorem 4.
Note that DGP of Example 1 or 2 implies the existence of the error function of Assumption 2.
Theorem 9 states that, we can choose arbitrarily small  $c > 0$  and the exponent of the regret of AEtC approaches that of EtC (Theorem 4). Note that the condition  $N \geq C_T \mathrm{tr}(\widehat{\Sigma}^{(i)})$  should not dominate the balance between exploration and exploitation: For example,  $C_T = C \log T$  for any  $C > 0$  suffices. In practice, a constant  $C_T$  works.
Proof sketch of Theorem 9. We can derive that Eqs. (9), (10), and (11) for all  $N \in [T]$  that hold with probability at least  $1 - O(1 / T)$  by setting  $C_{\mathrm{poly}} = 2$  and taking a union bound over  $N$ , and thus
$$
\bigcap_ {i \in [ K ]} \bigcap_ {N \geq C _ {T} \operatorname {t r} (\widehat {\Sigma} ^ {(i)})} \left\{\log \left(\frac {\sqrt {\widehat {B} _ {N , T} ^ {(i)} + \widehat {V} _ {N , T} ^ {(i)}}}{\sqrt {B _ {N , T} ^ {(i)} + V _ {N , T} ^ {(i)}}}\right) = \log \left(T ^ {o (1)}\right) \right\} \tag {12}
$$
holds. Under this event, the stopping time of AEtC and EtC are at most  $T^{o(1)}$  times different. Given this, the proof of the theorem is easily modified from the proof of Theorem 4.
Figure 2: Comparison of methods on four DGPs. Smaller regret indicates better performance. The solid lines are the mean of 10 repetitions and the bands represent the standard deviation.
6 Simulation
We consider two setups:  $(K, p, T) = (2, 200, 500)$  and  $(K, p, T) = (10, 200, 1000)$ . For each setup, we consider a covariance matrix  $\Sigma^{(i)} = c^{(i)}\bar{\Sigma}$  where  $c^{(i)} \sim \mathrm{Uni}([0.5, 1.5])$  and a base covariance  $\bar{\Sigma}$  for each  $i \in [K]$ , which represents a heterogeneous covariance among the arms. The base covariance  $\bar{\Sigma}$  follows the following configurations: DGP 1:  $\bar{\Sigma} = \mathrm{diag}(\lambda_1, \dots, \lambda_p)$ ,  $\lambda_k = k^{-0.5}$ , DGP 2:  $\bar{\Sigma} = \mathrm{diag}(\lambda_1, \dots, \lambda_p)$ ,  $\lambda_k = \exp(-k) + T\exp(-T)/p$ , DGP 3:  $\bar{\Sigma} = \mathrm{diag}(\lambda_1, \dots, \lambda_p)$ ,  $\lambda_k = k^{-1+1/T}$ , and DGP 4:  $\bar{\Sigma}_{i,j} = 0.3$  and  $\bar{\Sigma}_{i,i} = 0.7$  for  $j \neq i \in [p]$ . Note that the DGPs 1 and 3 correspond to Examples 2 and 1 in Proposition 1. DGP 3 is benign as well, while DGP 4 is not. We generate  $X^{(i)}(t)$  from a centered  $p$ -dimensional Gaussian with covariance  $\Sigma^{(i)}$ ,  $\theta^{(i)}$  from a standard normal distribution which yields non-sparse parameter vectors, and  $Y^{(i)}(t)$  by the linear model (1) with the noise with variance  $\sigma^2 = 0.01$ . We compare proposal (AEtC,  $C_T = 2$ ) to ESTC [Hao et al., 2020], LinUCB [Li et al., 2010, Abbasi-Yadkori et al., 2011], and DR Lasso Bandit [Kim and Paik, 2019].
Figure 2 illustrates the accumulated regret  $R(t)$  in relation to the round  $t \in [T]$ . In the first three benign DGPs, the AEtC consistently outperforms the other methods. In DGP 2, the advantage of AEtC is insignificant because the regret of any method is already small. This is because in DGP 2, a model with exponentially decaying behavior, only a small fraction of eigenvalues are important. However, AEtC performs significantly better than its competitors in DGP 1 and DGP 3, where the eigenvalues have a heavy-tail distribution. In a non-benign example of DGP 4, the proposed method is still comparable to LinUCB, which demonstrates the utility of an interpolating estimator.
Shogo Nakakita and Masaaki Imaizumi. Benign overfitting in time series linear model with overparameterization. arXiv preprint arXiv:2204.08369, 2022. 13
Min-hwan Oh, Garud Iyengar, and Assaf Zeevi. Sparsity-agnostic lasso bandit. In International Conference on Machine Learning, pages 8271–8280. PMLR, 2021. 1
Steffen Rendle. Factorization machines. In Geoffrey I. Webb, Bing Liu, Chengqi Zhang, Dimitrios Gunopoulos, and Xindong Wu, editors, ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17 December 2010, pages 995-1000. IEEE Computer Society, 2010. 2
Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527 - 535, 1952. 1
Liang Tang, Rómer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In 22nd ACM International Conference on Information and Knowledge Management, CIKM'13, pages 1587-1594. ACM, 2013. 1
Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009.14286, 2020. 3, 4, 5, 6, 13, 14
Sara A Van De Geer and Peter Buhlmann. On the conditions used to prove oracle results for the lasso. Electronic Journal of Statistics, 3:1360-1392, 2009. 2
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018. 5
Xue Wang, Mingcheng Wei, and Tao Yao. Minimax concave penalized multi-armed bandit model with high-dimensional covariates. In International Conference on Machine Learning, pages 5200-5208. PMLR, 2018. 1
Yuyan Wang, Long Tao, and Xian Xing Zhang. Recommending for a multi-sided marketplace with heterogeneous contents. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, page 456-459. Association for Computing Machinery, 2022. 1, 2