SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \max _ {1 \leq t \leq n + m _ {n}} \| \boldsymbol {w} _ {t} \| _ {2} ^ {(i)} \stackrel {{(i)}} {{\leq}} \max _ {1 \leq t \leq n + m _ {n}} \left(\sqrt {1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}} \cdot \| \mathbf {V} _ {t} \| \mathrm {o p} \cdot \| \boldsymbol {z} _ {t} \| _ {2}\right) \\ \stackrel {(i i)} {\leq} \max  _ {1 \leq t \leq n + m _ {n}} \sqrt {2} \| \boldsymbol {z} _ {t} \| _ {2} \\ \stackrel {(i i i)} {\leq} \sqrt {2} \| \boldsymbol {\Sigma} _ {0} ^ {- 1 / 2} \| \mathrm {o p}. \\ \end{array}
$$
Inequality (i) follows from the definition of  $\mathbf{w}_t$ . In inequality (ii), we use the assumption that  $\mathbf{\Sigma}_0 \succeq \mathbf{I}_d$  and the fact that  $\| \mathbf{z}_t\|_2 \leq 1$  and  $\| \mathbf{V}_t\|_{\mathrm{op}} \leq 1$ . The last inequality (iii) follows from the definition of  $\mathbf{z}_t$  and the condition that  $\| \mathbf{\Sigma}_0^{-1}\|_{\mathrm{op}} = o_p(1)$ . Hence, we can see that
$$
\max  _ {1 \leq t \leq n + m _ {n}} \| \boldsymbol {w} _ {t} \| _ {2} \xrightarrow {p} 0. \tag {77}
$$
$$
\max  _ {1 \leq t \leq n + m _ {n}} | \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} | \xrightarrow {p} 0 \quad \text {a n d} \quad \sum_ {t = 1} ^ {n + m _ {n}} \operatorname {V a r} [ \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} \epsilon_ {t} \mid \mathcal {F} _ {t - 1} ] \xrightarrow {p} \sigma^ {2}. \tag {78}
$$
Note that condition (70a) holds because  $\{\pmb{v}^{\top}\pmb{w}_{k}\epsilon_{k}\}_{k\geq 1}$  is a martingale difference sequence by construction. Condition (70b) follows from statement (78). It remains to verify condition (70c). Observe that
$$
\begin{array}{l} \sum_ {t = 1} ^ {n + m _ {n}} \mathbb {E} \left[ \left| \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} \epsilon_ {t} \right| ^ {2 + \delta} \mid \mathcal {F} _ {t - 1} \right] = \sum_ {t = 1} ^ {n + m _ {n}} \left| \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} \right| ^ {2 + \delta} \mathbb {E} \left[ \left| \epsilon_ {t} \right| ^ {2 + \delta} \mid \mathcal {F} _ {t - 1} \right] \\ \leq \left(\max  _ {1 \leq t \leq n + m _ {n}} | \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} | ^ {\delta}\right) \cdot \left(\sup  _ {t \geq 1} \mathbb {E} [ | \epsilon_ {t} | ^ {2 + \delta} \mid \mathcal {F} _ {t - 1} ]\right) \cdot \max  \left\{\frac {1}{\sigma^ {2}}, \frac {1}{\widehat {\sigma} ^ {2}} \right\} \sum_ {t = 1} ^ {n + m _ {n}} \operatorname {V a r} [ \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} \epsilon_ {t} \mid \mathcal {F} _ {t - 1} ] \\ \stackrel {(i v)} {=} o _ {p} (1) \cdot O _ {p} (1) \cdot O _ {p} (1) = o _ {p} (1). \\ \end{array}
$$
Equation (iv) follows from condition (3), equation (78) and the fact that  $\hat{\sigma}^2$  is a consistent estimator. Lastly, by applying Slutsky's theorem, we prove that
$$
\frac {1}{\widehat {\sigma}} \sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {x} _ {t} ^ {\top} \left(\widehat {\boldsymbol {\theta}} _ {\text {A L E E}} - \theta^ {*}\right) \xrightarrow {d} \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}). \tag {79}
$$
C Simulation
In this section, we provide additional comparisons among the ALEE method, the OLS, the W-decorrelation [8], and the concentration inequality based bounds [1]. The code can be found at https://github.com/mufangying/ALEE.
C.1 Simulation details
Throughout our experiments, we utilize  $\widehat{\sigma}^2$  from equation (9) as an (consistent) estimate of  $\sigma^2$  [19].
OLS: When data are i.i.d, the least squares estimator satisfies the following condition
$$
\frac {1}{\sigma^ {2}} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {L S}} - \boldsymbol {\theta} ^ {*}\right) ^ {\top} \mathbf {S} _ {n} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {L S}} - \boldsymbol {\theta} ^ {*}\right) \xrightarrow {d} \chi_ {d} ^ {2}.
$$
Therefore, we consider  $1 - \alpha$  confidence region to be
$$
\mathbf {C} _ {\mathrm {L S}} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d}: \frac {1}{\widehat {\sigma} ^ {2}} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {L S}} - \boldsymbol {\theta}\right) ^ {\top} \mathbf {S} _ {n} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {L S}} - \boldsymbol {\theta}\right) \leq \chi_ {d, 1 - \alpha} ^ {2} \right\}. \tag {80}
$$
We point out that the above confidence region is not guaranteed to be accurate when the data is collected in an adaptive manner, as will also be highlighted in our experiments.
W-decorrelation: The W-decorrelation method is borrowed from Algorithm 1 in [8]. Specifically, the estimator takes the form
$$
\widehat {\boldsymbol {\theta}} _ {\mathrm {W}} = \widehat {\boldsymbol {\theta}} _ {\mathrm {L S}} + \sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \left(y _ {t} - \boldsymbol {x} _ {t} ^ {\top} \widehat {\boldsymbol {\theta}} _ {\mathrm {L S}}\right). \tag {81}
$$
Given a parameter  $\lambda$ , weights  $\{\pmb{w}_t\}_{1\leq t\leq n}$  are set as follows
$$
\boldsymbol {w} _ {t} = \left(\mathbf {I} _ {d} - \sum_ {i = 1} ^ {t - 1} \boldsymbol {w} _ {t} \boldsymbol {x} _ {t} ^ {\top}\right) \boldsymbol {x} _ {t} / (\lambda + \| \boldsymbol {x} _ {t} \| _ {2} ^ {2}). \tag {82}
$$
Following the recommendations from the paper [8], in order to set  $\lambda$  appropriately, we first run the bandit algorithm or time series with  $N$  replications and record the corresponding minimum eigenvalues  $\{\lambda_{\min}(\mathbf{S}_n^{(1)}),\dots ,\lambda_{\min}(\mathbf{S}_n^{(N)})\}$ . We choose  $\lambda$  to be the 0.1-quantile of  $\{\lambda_{\min}(\mathbf{S}_n^{(1)}),\dots ,\lambda_{\min}(\mathbf{S}_n^{(N)})\}$ . Finally, we obtain a  $1 - \alpha$  confidence region for  $\pmb{\theta}^*$  as
$$
\mathbf {C} _ {\mathrm {W}} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d}: \frac {1}{\widehat {\sigma} ^ {2}} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {W}} - \boldsymbol {\theta}\right) ^ {\top} \mathbf {W} ^ {\top} \mathbf {W} \left(\widehat {\boldsymbol {\theta}} _ {\mathrm {W}} - \boldsymbol {\theta}\right) \leq \chi_ {d, 1 - \alpha} ^ {2} \right\}, \tag {83}
$$
where  $\mathbf{W} = (\pmb {w}_1,\dots ,\pmb {w}_n)^\top$
Concentration based on self-normalized martingales: We consider [1, Theorem 1] for a single coordinate in two-armed bandit problem and AR(1) model. For contextual bandits, we apply [1, Theorem 2]. Applying concentration bounds requires a sub-Gaussian parameter, for which we use  $\widehat{\sigma}$  from equation (9) as an estimate. We point out that this estimate of the sub-Gaussian parameter is conservative, as the sub-Gaussian parameter of a sub-Gaussian random variable is always lower bounded by its variance [33, Chapter 2]. This variance estimate is accurate for Gaussian noise random variables.
- For one dimensional examples, we have that for any  $\lambda > 0$ , with probability at least  $1 - \alpha$ :
$$
\left| \widehat {\theta} _ {\mathrm {L S}} - \theta^ {*} \right| \leq \frac {\widehat {\sigma} \sqrt {\lambda + \sum_ {t = 1} ^ {n} x _ {t} ^ {2}}}{\sum_ {t = 1} ^ {n} x _ {t} ^ {2}} \sqrt {\log \left(\frac {\lambda + \sum_ {t = 1} ^ {n} x _ {t} ^ {2}}{\lambda \alpha^ {2}}\right)}. \tag {84}
$$
In two-armed bandit problem,  $x_{t}$  is simply  $x_{t,1}$  for  $\theta_1^*$  or  $x_{t,2}$  for  $\theta_2^*$ . Here we consider  $\lambda = 1$ .
- For the contextual bandit examples, we apply Theorem 2 from [1], and set  $S = \sqrt{d}$ ; we set a small value of  $\lambda = 0.01$  to mimic the performance of an OLS estimators. Specifically, we utilize the following  $1 - \alpha$  confidence region
$$
\mathbf {C} _ {\text {c o n}} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d}: \left(\widehat {\boldsymbol {\theta}} _ {r} - \boldsymbol {\theta}\right) ^ {\top} \left(\lambda \mathbf {I} _ {d} + \mathbf {S} _ {n}\right) \left(\widehat {\boldsymbol {\theta}} _ {r} - \boldsymbol {\theta}\right) \leq \left(\widehat {\sigma} \sqrt {\log \left(\frac {\operatorname* {d e t} \left(\lambda \mathbf {I} _ {d} + \mathbf {S} _ {n}\right)}{\lambda^ {d} \alpha^ {2}}\right)} + \lambda^ {\frac {1}{2}} S\right) ^ {2} \right\}, \tag {85}
$$
where  $\widehat{\pmb{\theta}}_r = (\mathbf{X}_n^\top \mathbf{X}_n + \lambda \mathbf{I}_d)^{-1}\mathbf{X}_n^\top \mathbf{Y}_n$  and  $\mathbf{Y}_n = (y_1,\dots ,y_n)^\top$ .
C.2 Tables for contextual bandits
In all the contextual bandit simulations, we consider noises that are generated from a centered Poisson distribution (i.e.  $\text{Poisson}(1) - 1$ ). We would like to highlight that the centered Poisson random variable is not sub-Gaussian. Therefore, it is important to note that concentration inequality-based bounds [1] may not be guaranteed to work. In the simulations of this section, we set the number of samples as  $n = 1000$ , and the tables below show results over 1000 replications. The tables below clearly show that the average log-volume of the confidence regions are smallest for ALEE among methods which yield valid confidence regions (empirical coverage is more than the target coverage). The volume of the confidence region obtained from the OLS estimate is the smallest, but they under-cover the true parameter. The confidence regions for ALEE are obtained from Theorem B.4 with  $\mathbf{\Sigma_0} = \log(n) \cdot \mathbf{I}_d$  and  $\kappa_n = d\log(n)$ .
Table 2: Contextual bandit:  $\mathrm{d} = {10}$
Method	Level of confidence
0.8	0.85	0.9
Avg. Coverage	Avg. log(Volume)	Avg. Coverage	Avg. log(Volume)	Avg. Coverage	Avg. log(Volume)
ALEE	0.819 (± 0.385)	-2.761 (± 0.263)	0.872 (± 0.334)	-2.370 (± 0.263)	0.920 (± 0.271)	-1.894 (± 0.263)
OLS	0.807 (± 0.395)	-7.306 (± 0.262)	0.863 (± 0.344)	-6.915 (± 0.262)	0.905 (± 0.293)	-6.439 (± 0.262)
W-Decorrelation	0.785 (± 0.411)	8.382 (± 0.252)	0.827 (± 0.378)	8.773 (± 0.252)	0.868 (± 0.338)	9.249 (± 0.252)
Concentration	1.000 (± 0.000)	2.517 (± 0.252)	1.000 (± 0.000)	2.548 (± 0.252)	1.000 (± 0.000)	2.591 (± 0.252)
Table 3: Contextual bandit:  $\mathrm{d} = {50}$
Method	Level of confidence
0.8	0.85	0.9
Avg. Coverage	Avg. log(Volume)	Avg. Coverage	Avg. log(Volume)	Avg. Coverage	Avg. log(Volume)
ALEE	0.744 (± 0.436)	72.759 (± 1.403)	0.809 (± 0.393)	73.680 (± 1.403)	0.875 (± 0.331)	74.822 (± 1.403)
OLS	0.730 (± 0.444)	44.640 (± 1.370)	0.791 (± 0.407)	45.560 (± 1.370)	0.847 (± 0.360)	46.703 (± 1.370)
W-Decorrelation	0.192 (± 0.394)	97.559 (± 1.337)	0.225 (± 0.418)	98.479 (± 1.337)	0.276 (± 0.447)	99.622 (± 1.337)
Concentration	1.000 (± 0.000)	90.964 (± 1.312)	1.000 (± 0.000)	91.004 (± 1.312)	1.000 (± 0.000)	91.060 (± 1.312)
C.3 Asymptotic normality with centered Poisson noise variables
Distribution of errors in AR(1)
Figure 4: Same setting as Figure 1 but with noise variables  $\{\epsilon_t\}$  distributed as centered Poisson(1). We set  $n = 3000$  and the number of replications is set to 1000. The simulations show that the asymptotic distribution of ALEE is in good accordance with the asymptotic normality proved in Corollary 3.5 and Theorem 3.1.
Distribution of errors with  $\varepsilon$ -greedy