SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mufang Ying
Rutgers University - New Brunswick
my426@scarletmail.rutgers.edu
Koulik Khamaru
k1241@stat.rutgers.edu
Cun-Hui Zhang
czhang@stat.rutgers.edu
Sequential data collection has emerged as a widely adopted technique for enhancing the efficiency of data gathering processes. Despite its advantages, such data collection mechanism often introduces complexities to the statistical inference procedure. For instance, the ordinary least squares (OLS) estimator in an adaptive linear regression model can exhibit non-normal asymptotic behavior, posing challenges for accurate inference and interpretation. In this paper, we propose a general method for constructing unbiased estimator which remedies this issue. It makes use of the idea of adaptive linear estimating equations, and we establish theoretical guarantees of asymptotic normality, supplemented by discussions on achieving near-optimal asymptotic variance. A salient feature of our estimator is that in the context of multi-armed bandits, our estimator retains the non-asymptotic performance of the least squares estimator while obtaining asymptotic normality property. Consequently, this work helps connect two fruitful paradigms of adaptive inference: a) non-asymptotic inference using concentration inequalities and b) asymptotic inference via asymptotic normality.
Adaptive data collection arises as a common practice in various scenarios, with a notable example being the use of (contextual) bandit algorithms. Algorithms like these aid in striking a balance between exploration and exploitation trade-offs within decision-making processes, encompassing domains such as personalized healthcare and web-based services [35, 24, 3, 22]. For instance, in personalized healthcare, the primary objective is to choose the most effective treatment for each patient based on their individual characteristics, such as medical history, genetic profile, and living environment. Bandit algorithms can be used to allocate treatments based on observed response, and the algorithm updates its probability distribution to incorporate new information as patients receive treatment and their response is observed. Over time, the algorithm can learn which treatments are the most effective for different types of patients.
Although the adaptivity in data collection improves the quality of data, the sequential nature (non-iid) of the data makes the inference procedure quite challenging [34, 26, 5, 28, 27, 10, 30, 29]. There is a lengthy literature on the problem of parameter estimation in the adaptive design setting. In a series of work [15, 19, 17], the authors studied the consistency of the least squares estimator for an adaptive
linear model. In a later work, Lai [14] studied the consistency of the least squares estimator in a nonlinear regression model. The collective wisdom of these papers is that, for adaptive data collection methods, standard estimators are consistent under a mild condition on the maximum and minimum eigenvalues of the covariance matrix [19, 14]. In a more recent line of work [1, 2], the authors provide a high probability upper bound on the  $\ell_2$ -error of the least squares estimator for a linear model. We point out that, while the high probability bounds provide a quantitative understanding of OLS, these results assume a stronger sub-Gaussian assumption on the noise variables.
The problem of inference, i.e. constructing valid confidence intervals, with adaptively collected data is much more delicate. Lai and Wei [19] demonstrated that for a unit root autoregressive model, which is an example of adaptive linear regression models, the least squares estimator doesn't achieve asymptotic normality. Furthermore, the authors showed that for a linear regression model, the least squares estimator is asymptotically normal when the data collection procedure satisfies a stability condition. Concretely, letting  $\boldsymbol{x}_i$  denote the covariate associated with  $i$ -th sample, the authors require
$$
\mathbf {B} _ {n} ^ {- 1} \mathbf {S} _ {n} \xrightarrow {p} \mathbf {I} \tag {1}
$$
where  $\mathbf{S}_n = \sum_{i=1}^n \boldsymbol{x}_i \boldsymbol{x}_i^\top$  and  $\{\mathbf{B}_n\}_{n \geq 1}$  is a sequence of non-random positive definite matrices. Unfortunately, in many scenarios, the stability condition (1) is violated [38, 19]. Moreover, in practice, it might be difficult to verify whether the stability condition (1) holds or not. In another line of research [10, 36, 37, 4, 9, 28, 31, 25, 38], the authors assume knowledge of the underlying data collection algorithm and provide asymptotically valid confidence intervals. While this approach offers intervals under a much weaker assumption on the underlying model, full knowledge of the data collection algorithm is often unavailable in practice.
Online debiasing based methods: In order to produce valid statistical inference when the stability condition (1) does not hold, some authors [8, 7, 13] utilize the idea of online debiasing. At a high level, the online debiased estimator reduces bias from an initial estimate (usually the least squares estimate) by adding some correction terms, and the online debiasing procedure does not require the knowledge of the data generating process. Although this procedure guarantees asymptotic reduction of bias to zero, the bias term's convergence rate can be quite slow.
In this work, we consider estimating the unknown parameter in an adaptive linear model by using a set of adaptive linear estimating equations (ALEE). We show that our proposed ALEE estimator achieves asymptotic normality without knowing the exact data collection algorithm while addressing the slowly decaying bias problem in online debiasing procedure.
2 Background and problem set-up
In this section, we provide the background for our problem and set up a few notations. We begin by defining the adaptive data collection mechanism for linear models.
2.1 Adaptive linear model
Suppose a scalar response variable  $y_{t}$  is linked to a covariate vector  $\pmb{x}_t \in \mathbb{R}^d$  at time  $t$  via the linear model:
$$
y _ {t} = \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\theta} ^ {*} + \epsilon_ {t} \quad \text {f o r} t \in [ n ], \tag {2}
$$
where  $\pmb{\theta}^{*}\in \mathbb{R}^{d}$  is the unknown parameter of interest.
In an adaptive linear model, the regressor  $\pmb{x}_t$  at time  $t$  is assumed to be a (unknown) function of the prior data point  $\{x_1, y_1, \dots, x_{t-1}, y_{t-1}\}$  as well as additional sources of randomness that may be present in the data collection process. Formally, we assume there is an increasing sequence of  $\sigma$ -fields  $\{\mathcal{F}_t\}_{t \geq 0}$  such that
$$
\sigma \left(\boldsymbol {x} _ {1}, y _ {1}, \dots , \boldsymbol {x} _ {t - 1}, y _ {t - 1}, \boldsymbol {x} _ {t}\right) \in \mathcal {F} _ {t - 1} \quad \text {f o r} t \in [ n ].
$$
For the noise variables  $\{\epsilon_t\}_{t\geq 1}$  appearing in equation (2), we impose the following conditions
$$
\mathbb {E} \left[ \epsilon_ {t} \mid \mathcal {F} _ {t - 1} \right] = 0, \quad \mathbb {E} \left[ \epsilon_ {t} ^ {2} \mid \mathcal {F} _ {t - 1} \right] = \sigma^ {2}, \quad \text {a n d} \quad \sup  _ {t \geq 1} \mathbb {E} \left[ \left| \epsilon_ {t} / \sigma \right| ^ {2 + \delta} \mid \mathcal {F} _ {t - 1} \right] <   \infty , \tag {3}
$$
for some  $\delta > 0$ . The above condition is relatively mild compared to a sub-Gaussian condition.
Examples of adaptive linear model arise in various problems, including multi-armed and contextual bandit problems, dynamical input-output systems, adaptive approximation schemes and time series models. For instance, in the context of the multi-armed bandit problem, the design vector  $\boldsymbol{x}_t$  is one of the basis vectors  $\{e_k\}_{k \in [d]}$ , representing an arm being pulled, while  $\theta^*, y_t$  represent the true mean reward vector and reward at time  $t$ , respectively.
2.2 Adaptive linear estimating equations
As we mentioned earlier, the OLS estimator can fail to achieve asymptotic normality due to the instability of the covariance matrix with adaptively collected data. To get around this issue, we consider a different approach ALEE (adaptive linear estimating equations). Namely, we obtain an estimate by solving a system of linear estimating equations with adaptive weights,
$$
\text {A L E E :} \quad \sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \left(y _ {t} - \boldsymbol {x} _ {t} ^ {\top} \widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}}\right) = \mathbf {0}. \tag {4}
$$
Here the weight  $\boldsymbol{w}_t \in \mathbb{R}^d$  is chosen in a way that  $\boldsymbol{w}_t \in \mathcal{F}_{t-1}$  for  $t \in [n]$ . Let us now try to gain some intuition behind the construction of ALEE. Rewriting equation (4), we have
$$
\left\{\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {x} _ {t} \right\} \cdot \left(\widehat {\boldsymbol {\theta}} _ {\text {A L E E}} - \boldsymbol {\theta} ^ {*}\right) = \sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \epsilon_ {t}. \tag {5}
$$
Notably, the choice of  $\pmb{w}_t\in \mathcal{F}_{t - 1}$  makes  $\sum_{t = 1}^{n}\pmb{w}_{t}\epsilon_{t}$  the sum of a martingale difference sequence. Our first theorem postulates conditions on the weight vectors  $\{\pmb {w}_t\}_{t\geq 1}$  such that the right-hand side of (5) converges to normal distribution asymptotically. Throughout the paper, we use the shorthand  $\mathbf{W}_t = (\pmb {w}_1,\dots ,\pmb {w}_t)^\top \in \mathbb{R}^{t\times d},\mathbf{X}_t = (\pmb {x}_1,\dots ,\pmb {x}_t)^\top \in \mathbb{R}^{t\times d}$ .
Proposition 2.1. Suppose condition (3) holds and the predictable sequence  $\{\pmb{w}_t\}_{1\leq t\leq n}$  satisfies
$$
\max  _ {1 \leq t \leq n} \| \boldsymbol {w} _ {t} \| _ {2} = o _ {p} (1) \quad a n d \quad \left\| \mathbf {I} _ {d} - \mathbf {W} _ {n} ^ {\top} \mathbf {W} _ {n} \right\| _ {\mathrm {o p}} = o _ {p} (1). \tag {6}
$$
Let  $\mathbf{A}_w = \mathbf{V}_w\mathbf{U}_w^\top \mathbf{X}_n$  with  $\mathbf{W}_n = \mathbf{U}_w\boldsymbol{\Lambda}_w\mathbf{V}_w^\top$  being the SVD of  $\mathbf{W}_n$ . Then,
$$
\mathbf {A} _ {w} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}} - \boldsymbol {\theta} ^ {*}\right) / \widehat {\sigma} \xrightarrow {d} \mathcal {N} \left(\mathbf {0}, \mathbf {I} _ {d}\right), \tag {7}
$$
where  $\widehat{\sigma}$  is any consistent estimator for  $\sigma$ .
Proof. Invoking the second part of the condition (6), we have that  $\Lambda_w$  is invertible for large  $n$ , and  $\| \mathbf{V}_w \boldsymbol{\Lambda}_w^{-1} \mathbf{V}_w^\top - \mathbf{I}_d \|_{\mathrm{op}} = o_p(1)$ . Utilizing the expression (5), we have
$$
\mathbf {A} _ {w} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}} - \boldsymbol {\theta} ^ {*}\right) / \sigma = \mathbf {V} _ {w} \boldsymbol {\Lambda} _ {w} ^ {- 1} \mathbf {V} _ {w} ^ {\top} \mathbf {W} _ {n} ^ {\top} \mathbf {X} _ {n} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}} - \boldsymbol {\theta} ^ {*}\right) / \sigma = \mathbf {V} _ {w} \boldsymbol {\Lambda} _ {w} ^ {- 1} \mathbf {V} _ {w} ^ {\top} \sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \epsilon_ {t} / \sigma .
$$
Invoking the stability condition on the weights  $\{\pmb{w}_t\}$  and using the fact that  $\sum_{t=1}^{n} \pmb{w}_t \epsilon_t$  is a martingale difference sequence, we conclude from martingale central limit theorem [11, Theorem 2.1] that
$$
\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \epsilon_ {t} / \sigma \xrightarrow {d} \mathcal {N} \left(\mathbf {0}, \mathbf {I} _ {d}\right).
$$
Combining the last equation with  $\| \mathbf{V}_w\pmb{\Lambda}_w^{-1}\mathbf{V}_w^\top -\mathbf{I}_d\|_{\mathrm{op}} = o_p(1)$  and using Slutsky's theorem yield
$$
\mathbf {A} _ {w} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}} - \boldsymbol {\theta} ^ {*}\right) / \sigma \xrightarrow {d} \mathcal {N} \left(\mathbf {0}, \mathbf {I} _ {d}\right).
$$
The claim of Proposition 2.1 now follows from Slutsky's theorem.
A few comments regarding the Proposition 2.1 are in order. Straightforward calculation shows
$$
\mathbf {A} _ {w} ^ {\top} \mathbf {A} _ {w} = \mathbf {X} _ {n} ^ {\top} \mathbf {P} _ {w} \mathbf {X} _ {n} \preceq \mathbf {S} _ {n}, \quad \text {w h e r e} \quad \mathbf {P} _ {w} = \mathbf {W} _ {n} \left(\mathbf {W} _ {n} ^ {\top} \mathbf {W} _ {n}\right) ^ {- 1} \mathbf {W} _ {n} ^ {\top}. \tag {8}
$$
In words, the volume of the confidence region based on (7) is always larger than the confidence region generated by the least squares estimate. Therefore, the ALEE-based inference, which is consistently valid, exhibits a reduced efficiency in cases where both types of confidence regions are valid. Compared with the confidence regions based on OLS, the advantage of the ALEE approach is to provide flexibility in the choice of weights to guarantee the validity of the CLT conditions (6).
Next, note that the matrix  $\mathbf{A}_w$  is asymptotically equivalent to the matrix  $\mathbf{W}_n^\top \mathbf{X}_n$  (see equation (5)) under the stability condition (6). The benefit of this reformulation is that it helps us better understand
efficiency of ALEE compared with the OLS. This has led us to define a notion of affinity between the weights  $\{\pmb{w}_t\}_{t\geq 1}$  and covariates  $\{\pmb{x}_t\}_{t\geq 1}$  for better understanding of the efficiency of ALEE and ways to design nearly optimal weights, as it will be clear in the next section.
Finally, it is straightforward to obtain a consistent estimate for  $\sigma$ . For instance, assuming  $\log (\lambda_{\max}(\mathbf{X}_n^\top \mathbf{X}_n)) / n \xrightarrow{a.s.} 0$  and the noise condition (3), we have
$$
\widehat {\sigma} ^ {2} := \frac {1}{n} \sum_ {t = 1} ^ {n} \left(y _ {t} - \boldsymbol {x} _ {t} ^ {\top} \widehat {\boldsymbol {\theta}} _ {\mathrm {L S}}\right) ^ {2} \xrightarrow {a . s .} \sigma^ {2}. \tag {9}
$$
Here,  $\widehat{\theta}_{\mathrm{LS}}$  refers to the least squares estimate. See [19, Lemma 3] for a detailed proof of equation (9).
3 Main results
In this section, we propose methods to construct weights  $\{\pmb{w}_t\}_{t\geq 1}$  which satisfy the stability property (6), and study the resulting ALEE. Section 3.1 is devoted to the multi-armed bandit case, Section 3.2 to an autoregressive model, and Section 3.3 to the contextual bandit case. Before delving into details, let us try to understand intuitively how to construct weights that have desirable properties.
The expression (8) reveals that the efficiency of ALEE depends on the projection of the data matrix  $\mathbf{X}_n$  on  $\mathbf{W}_n$ . Thus, the efficiency of the ALEE approach can be measured by the principal angles between the random projections  $\mathbf{P}_w$  in (8) and  $\mathbf{P}_x = \mathbf{X}_n\mathbf{S}_n^{-1}\mathbf{X}_n^\top$ . Accordingly, we define the affinity  $\mathcal{A}(\mathbf{W}_n,\mathbf{X}_n)$  of the weights  $\{\pmb{w}_t\}_{t\geq 1}$  as the cosine of the largest principle angle, or equivalently
$$
\mathcal {A} \left(\mathbf {W} _ {n}, \mathbf {X} _ {n}\right) = \sigma_ {d} \left(\mathbf {P} _ {w} \mathbf {P} _ {x}\right) = \sigma_ {d} \left(\mathbf {U} _ {w} ^ {\top} \mathbf {X} _ {n} \mathbf {S} _ {n} ^ {- 1 / 2}\right) \tag {10}
$$
as the  $d$ -th largest singular value of  $\mathbf{P}_w\mathbf{P}_x$ . Formally, the above definition captures the cosine of the angle between the two subspaces spanned by the columns of  $\mathbf{X}_n$  and  $\mathbf{W}_n$ , respectively [12]. Good weights  $\{\pmb{w}_t\}_{t\geq 1}$  are those with relatively large affinity or
$$
\mathbf {U} _ {w} \propto \mathbf {X} _ {n} \mathbf {S} _ {n} ^ {- 1 / 2} \quad (\text {a p p r o x i m a t e l y}). \tag {11}
$$
3.1 Multi-armed bandits
In the context of the  $K$ -arm bandit problem, the Gram matrix has a diagonal structure, which means that we can focus on constructing weights  $\{\pmb{w}_t\}_{t\geq 1}$  for each coordinate independently. For an arm  $k\in [K]$  and round  $t\geq 1$ , define
$$
s _ {t, k} = s _ {0} + \sum_ {i = 1} ^ {t} x _ {i, k} ^ {2} \quad \text {f o r s o m e p o s i t i v e} s _ {0} \in \mathcal {F} _ {0}. \tag {12}
$$
Define the  $k$ -th coordinate of the weight  $\boldsymbol{w}_t$  as
$$
w _ {t, k} = f \left(\frac {s _ {t , k}}{s _ {0}}\right) \cdot \frac {x _ {t , k}}{\sqrt {s _ {0}}} \quad \text {w i t h} \quad f (x) = \sqrt {\frac {\log 2}{x \cdot \log \left(e ^ {2} x\right) \cdot (\log \log \left(e ^ {2} x\right)) ^ {2}}}. \tag {13}
$$
The intuition behind the above construction is as follows. The discussion at near equation (11) indicates that the  $k$ -th coordinate of  $\boldsymbol{w}_t$  should be proportional to  $x_{t,k} / (\sum_{i\leq n}x_{i,k}^2)^{1 / 2}$ . However, the weight  $\boldsymbol{w}_t$  is required to be predictable, which can only depend on the data points up to time  $t$ . Consequently, we approximate the sum  $\sum_{i\leq n}x_{i,k}^2$  by the partial sum  $s_{t,k}$  in (12). Finally, note that
$$
w _ {t, k} = f \left(\frac {s _ {t , k}}{s _ {0}}\right) \cdot \frac {x _ {t , k}}{\sqrt {s _ {0}}} \approx \frac {x _ {t , k}}{\sqrt {s _ {t , k}}}. \tag {14}
$$
The logarithmic factors in (13) ensure that the stability conditions (6) hold. In the following theorem, we generalize the above method as a general strategy for constructing weights  $\{\pmb{w}_t\}_{t\geq 1}$  satisfying the stability condition (6).
<sup>1</sup> Note that  $\pmb{x}_{t,k} \in \mathcal{F}_{t-1}$  can be used to construct  $\pmb{w}_t$ .
Distribution of errors in AR(1)
Distribution of errors with  $\varepsilon$ -greedy
Figure 1: Empirical distribution of the standardized estimation errors from OLS and ALEE approach. Results are obtained with a dataset of size  $n = 1000$  and 3000 independent replications. Left: AR(1) model  $y_{t} = y_{t - 1} + \epsilon_{t}$  with independent errors  $\epsilon_t\sim \mathcal{N}(0,1)$ . Right: Two-armed bandit problem with equal arm mean  $\theta_1^* = \theta_2^* = 0.3$  and independent noise  $\epsilon_t\sim \mathcal{N}(0,1)$ . Figure 4 in Section C.3 considers the same setting with centered Poisson noise, which is not sub-Gaussian.
3.1.1 Stable weight construction strategy
Consider a positive decreasing function  $f(x)$  on the interval  $[1, \infty)$  with an increasing derivative  $f'(x)$ . Let  $f$  satisfy the conditions:  $f' / f$  is increasing,
$$
\int_ {1} ^ {\infty} f ^ {2} (x) d x = 1, \quad \text {a n d} \quad \int_ {1} ^ {\infty} f (x) d x = \infty . \tag {15}
$$
With  $s_0 \in \mathcal{F}_0$ , we define weight  $w_{t,k}$  as
$$
w _ {t, k} = f \left(\frac {s _ {t , k}}{s _ {0}}\right) \frac {x _ {t , k}}{\sqrt {s _ {0}}} \quad \text {w i t h} \quad s _ {t, k} = s _ {0} + \sum_ {i = 1} ^ {t} x _ {i, k} ^ {2}. \tag {16}
$$
A key condition that ensures the weights  $\{w_{t,k}\}_{t\geq 1}$  satisfy the desirable stability property (6) is
$$
\max  _ {1 \leq t \leq n} f ^ {2} \left(\frac {s _ {t , k}}{s _ {0}}\right) \frac {x _ {t , k} ^ {2}}{s _ {0}} + \max  _ {1 \leq t \leq n} \left(1 - \frac {f \left(s _ {t , k} / s _ {0}\right)}{f \left(s _ {t - 1 , k} / s _ {0}\right)}\right) + \int_ {s _ {n, k} / s _ {0}} ^ {\infty} f ^ {2} (x) d x = o _ {p} (1). \tag {17}
$$
For multi-armed bandits, this condition is automatically satisfied when both quantities  $1 / s_0$  and  $s_0 / s_{n,k}$  converge to zero in probability. Putting together the pieces, we have the following result for multi-armed bandits.
Theorem 3.1. Suppose condition (3) holds and  $1 / s_0 + s_0 / s_{n,k} = o_p(1)$  for some  $k \in [K]$ . Then, the  $k$ -th coordinate  $\widehat{\theta}_{\mathrm{ALEE},k}$ , obtained using weights from equation (16), satisfies
$$
\left(\widehat {\theta} _ {\mathrm {A L E E}, k} - \theta_ {k} ^ {*}\right) \cdot \int_ {1} ^ {s _ {n, k} / s _ {0}} \frac {\sqrt {s _ {0}}}{\widehat {\sigma}} f (x) d x \xrightarrow {d} \mathcal {N} (0, 1), \tag {18}
$$
where  $\widehat{\sigma}$  is a consistent estimate of  $\sigma$ . Equivalently,
$$
\frac {\left(\widehat {\theta} _ {\mathrm {A L E E} , k} - \theta_ {k} ^ {*}\right)}{\widehat {\sigma} \sqrt {\sum_ {1 \leq t \leq n} w _ {t , k} ^ {2}}} \cdot \left(\sum_ {t = 1} ^ {n} w _ {t, k} x _ {t, k}\right) \xrightarrow {d} \mathcal {N} (0, 1). \tag {19}
$$
The proof of Theorem 3.1 can be found in Section A.1 of the Appendix. A few comments regarding Theorem 3.1 are in order.
First, the above theorem enables us to construct valid CI in the estimation of the mean  $\theta_{k}^{*}$  for a sub-optimal arm  $k$  when employing an asymptotically optimal allocation rule to achieve the optimal regret in [18] with sample size  $\sum_{t\leq n}x_{t,k}\asymp \log n$ , or when using a sub-optimal rule to achieve polylog(n). On the other hand, the classical martingale CLT is applicable to the optimal arm (if unique) under such asymptotically optimal or sub-optimal allocation rules. Consequently, one may obtain a valid CI for the optimal arm from the standard OLS estimate [19]. However, it is important to note that such CIs are not guaranteed for sub-optimal arms.
Next, while Theorem 3.1 holds for any  $s_0$  diverging to infinity but of smaller order than  $s_{n,k}$  (which may depend on  $k$ ), the convergence rate of  $\sum_{1\leq t\leq n}w_{t,k}\epsilon_t$  to normality is enhanced by choosing a large value for  $s_0$ . In practical terms, it is advisable to choose an  $s_0$  that is slightly smaller than the best-known lower bound for  $s_{n,k}$ .
Finally, the choice of function  $f$  determines the efficiency of ALEE estimator. For instance, taking function  $f(x) = 1 / x$ , we obtain an estimator with asymptotic variance of order  $1 / \{s_0\log^2 (s_{n,k} / s_0)\}$ , which is only better than what one would get using stopping time results by a logarithmic factor. In the next Corollary, an improved choice of  $f$  yields near optimal variance up to logarithmic terms.
Corollary 3.2. Consider the same set of assumptions as stated in Theorem 3.1. The ALEE estimator  $\widehat{\theta}_{\mathrm{ALEE},k}$ , obtained by using  $f(x) = (\beta \log^{\beta}2)^{1 / 2}\{x(\log e^{2}x)(\log \log e^{2}x)^{1 + \beta}\}^{-1 / 2}$  for any  $\beta >0$  satisfies
$$
\sqrt {\frac {4 \beta (\log 2) ^ {\beta}}{\log (s _ {n , k} / s _ {0}) \{\log \log (s _ {n , k} / s _ {0}) \} ^ {1 + \beta}}} \cdot \frac {\sqrt {s _ {n , k} (\widehat {\theta} _ {\mathrm {A L E E} , k} - \theta_ {k} ^ {*})}}{\widehat {\sigma}} \xrightarrow {d} \mathcal {N} (0, 1).
$$
The proof of this corollary follows directly from Theorem 3.1. For  $s_0 = \log n / \log \log n$  in multiarmed bandits with asymptotically optimal allocations,  $\log (s_{n,k} / s_0) = (1 + o(1))\log \log s_{n,k}$ .
3.1.2 Finite sample bounds for ALEE estimators
One may also construct finite sample confidence intervals for each arm via applying concentration bounds. Indeed, for any arm  $k \in K$ , we have
$$
\left\{\sum_ {t = 1} ^ {n} w _ {t, k} x _ {t, k} \right\} \cdot \left(\widehat {\theta} _ {\mathrm {A L E E}, k} - \theta_ {k} ^ {*}\right) = \sum_ {t = 1} ^ {n} w _ {t, k} \epsilon_ {t}. \tag {20}
$$
Following the construction of  $w_{t,k} \in \mathcal{F}_{t-1}$ , the term  $\sum_{t=1}^{n} w_{t,k} \epsilon_t$  is amenable to concentration inequalities if we assume that the noise  $\epsilon_t$  is sub-Gaussian conditioned on  $\mathcal{F}_{t-1}$ , i.e.
$$
\forall \lambda \in \mathbb {R} \quad \mathbb {E} \left[ e ^ {\lambda \epsilon_ {t}} \mid \mathcal {F} _ {t - 1} \right] \leq e ^ {\sigma_ {g} ^ {2} \lambda^ {2} / 2}. \tag {21}
$$
Corollary 3.3 (Theorem 1 in [1]). Suppose the sub-Gaussian noise condition (21) is in force. Then for any  $\delta >0$  and  $\lambda_0 > 0$ , the following bound holds with probability at least  $1 - \delta$
$$
\left| \sum_ {t = 1} ^ {n} w _ {t, k} x _ {t, k} \right| \cdot \left| \widehat {\theta} _ {\mathrm {A L E E}, k} - \theta_ {k} ^ {*} \right| \leq \sigma_ {g} \sqrt {\left(\lambda_ {0} + \sum_ {t = 1} ^ {n} w _ {t , k} ^ {2}\right) \cdot \log \left(\frac {\lambda_ {0} + \sum_ {t = 1} ^ {n} w _ {t , k} ^ {2}}{\delta^ {2} \lambda_ {0}}\right)}. \tag {22}
$$
Remark 3.4. In the context of multi-armed bandit, by considering the function  $f$  in Corollary 3.2 with  $\beta = 1$  and Corollary 3.3 with  $\lambda_0 = 1$ , we derive that with probability at least  $1 - \delta$