SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb{P}\left(\max_{1\leq t\leq \lfloor n^{2 / 3}\rfloor}y_{t - 1}^{2}\geq \epsilon\right) \\ = \mathbb {P} \left(\max  \left\{\max  _ {1 \leq t \leq \lfloor n ^ {2 / 3} \rfloor} y _ {t - 1}, \max  _ {1 \leq t \leq \lfloor n ^ {2 / 3} \rfloor} - y _ {t - 1} \right\} \geq \sqrt {\epsilon}\right) (44) \\ \leq \frac {\mathbb {E} \left[ \max  \left\{\max  _ {1 \leq t \leq \lfloor n ^ {2 / 3} \rfloor} y _ {t - 1} , \max  _ {1 \leq t \leq \lfloor n ^ {2 / 3} \rfloor} - y _ {t - 1} \right\} \right]}{\sqrt {\epsilon}} (44) \\ \stackrel {(i)} {\leq} \sqrt {\frac {2 n ^ {2 / 3} \sigma_ {g} ^ {2} \log (2 n ^ {2 / 3})}{\epsilon}}, \\ \end{array}
$$
where inequality is derived from [33, Exercise 2.12] and the fact that  $y_{i}$  is sub-Gaussian with sub-Gaussian parameter  $\sigma_g^2 n^{2 / 3}$  for  $i\leq \lfloor n^{2 / 3}\rfloor$ . Therefore, we conclude that
$$
\max_{1\leq t\leq \lfloor n^{2 / 3}\rfloor}y_{t - 1}^{2} = O_{p}(n^{2 / 3}\log n).
$$
Consequently, we have
$$
\max  _ {1 \leq t \leq \lfloor n ^ {2 / 3} \rfloor} \frac {y _ {t - 1} ^ {2}}{s _ {0}} = \frac {n ^ {2 / 3} \log n}{n / \log \log n} \cdot O _ {p} (1) = o _ {p} (1). \tag {45}
$$
By applying the same trick to  $\max_{\lfloor n^{2 / 3}\rfloor +1\leq t\leq n}y_{t - 1}^2$  , we can derive
$$
\max _ {\lfloor n ^ {2 / 3} \rfloor + 1 \leq t \leq n} y _ {t - 1} ^ {2} = O _ {p} (n \log n).
$$
Hence we have
$$
\max  _ {\lfloor n ^ {2 / 3} \rfloor + 1 \leq t \leq n} \frac {y _ {t - 1} ^ {2}}{s _ {\lfloor n ^ {2 / 3} \rfloor} - s _ {0}} = \frac {O _ {p} (n \log n)}{n ^ {4 / 3} / \log \log n ^ {2 / 3}} \cdot \frac {n ^ {4 / 3} / \log \log n ^ {2 / 3}}{s _ {\lfloor n ^ {2 / 3} \rfloor} - s _ {0}} \stackrel {(i i)} {=} o _ {p} (1) \cdot O _ {p} (1) = o _ {p} (1). \tag {46}
$$
Equality (ii) makes use of equation (42). Combining equation (44) with equations (45) and (46), we conclude that  $T_{1} = o_{p}(1)$ .
$T_{1}$  with  $\theta^{*} = -1$ : When  $\theta^{*} = -1$ , the proof is essentially the same as the case when  $\theta^{*} = -1$ . The only difference lies in the order of  $\sum_{1 \leq i \leq n} y_{i-1}^{2}$ . However, by pairing  $\epsilon_{2t-1}$  with  $\epsilon_{2t}$  for  $t \in \mathbb{N}^{+}$ , we can arrive at the same result. Specifically, for  $t \in \mathbb{N}^{+}$ , we let  $\epsilon_{t}' = \epsilon_{2t} - \epsilon_{2t-1}$  and define
$$
y _ {t} ^ {\prime} = \sum_ {k = 1} ^ {t} \epsilon_ {k} ^ {\prime}
$$
where  $y_0^\prime \triangleq 0$  and  $\{\epsilon_t^{\prime}\}_{t\geq 1}$  are random variables with mean zero, variance  $2\sigma^2$  and sub-Gaussian parameter  $2\sigma_g^2$ . Therefore, applying equation (44) yields
$$
\lim  _ {n \rightarrow \infty} n ^ {- 2} (\log \log n) \sum_ {t = 1} ^ {n} \left(y _ {t - 1} ^ {\prime}\right) ^ {2} = \sigma^ {2}. \tag {47}
$$
Setting  $n_0 = \lfloor (\lfloor n^{2 / 3}\rfloor -1) / 2\rfloor$  , we have
$$
s _ {\lfloor n ^ {2 / 3} \rfloor} - s _ {0} = \sum_ {t = 1} ^ {\lfloor n ^ {2 / 3} \rfloor - 1} y _ {t} ^ {2} \geq \sum_ {t = 1} ^ {n _ {0}} \left(y _ {t} ^ {\prime}\right) ^ {2} = \sum_ {t = 1} ^ {n _ {0} + 1} \left(y _ {t - 1} ^ {\prime}\right) ^ {2}. \tag {48}
$$
According to equation (47) and equation (48), we have
$$
\begin{array}{l} \max _ {\lfloor n ^ {2 / 3} \rfloor + 1 \leq t \leq n} \frac {y _ {t - 1} ^ {2}}{s _ {\lfloor n ^ {2 / 3} \rfloor} - s _ {0}} \leq \frac {\max _ {\lfloor n ^ {2 / 3} \rfloor + 1 \leq t \leq n} y _ {t - 1} ^ {2}}{\sum_ {1 \leq t \leq n _ {0} + 1} (y _ {t - 1} ^ {\prime}) ^ {2}} \\ = \frac {\operatorname* {m a x} _ {[ n ^ {2 / 3} ] + 1 \leq t \leq n} y _ {t - 1} ^ {2}}{(n _ {0} + 1) ^ {2} / \log \log (n _ {0} + 1)} \cdot \frac {(n _ {0} + 1) ^ {2} / \log \log (n _ {0} + 1)}{\sum_ {1 \leq t \leq n _ {0} + 1} (y _ {t - 1} ^ {\prime}) ^ {2}} \tag {49} \\ = o _ {p} (1) \cdot O _ {p} (1) = o _ {p} (1), \\ \end{array}
$$
which completes the proof of  $T_{1} = o_{p}(1)$  for the case when  $\theta^{*} = -1$ .
$T_{1}$  with  $\theta^{*} \in (-1, 1)$ : Given  $\theta^{*} \in (-1, 1)$ , we observe that  $y_{t}$  is a sub-Gaussian random variable with sub-Gaussian parameter  $\frac{\sigma_g^2}{1 - (\theta^*)^2}$  for any  $t \in \mathbb{N}^+$ . Therefore, following equation (43), we have
$$
T _ {1} \leq \frac {1}{2 (\log 2) ^ {1 + \beta}} \max  _ {1 \leq t \leq n} \frac {y _ {\ell - 1} ^ {2}}{s _ {0}} \tag {50}
$$
where in the above inequality we use  $s_0$  as a lower bound for  $s_t$ . By applying [33, Exercise 2.12], we have
$$
\max  _ {1 \leq t \leq n} y _ {t - 1} ^ {2} = O _ {p} (\log n), \tag {51}
$$
leading to the conclusion that  $T_{1} = o_{p}(1)$ .
$T_{2}$  with  $\theta^{*} \in [-1, 1]$ : Similar to equation (33), we have
$$
\begin{array}{l} T _ {2} = \max  _ {1 \leq t \leq n} \left(1 - \frac {f \left(s _ {t} / s _ {0}\right)}{f \left(s _ {t - 1} / s _ {0}\right)}\right) = \max  _ {1 \leq t \leq n} \frac {f \left(s _ {t - 1} / s _ {0}\right) - f \left(s _ {t} / s _ {0}\right)}{f \left(s _ {t - 1} / s _ {0}\right)} \\ \leq \max _ {1 \leq t \leq n} \frac {- f ^ {\prime} (s _ {t - 1} / s _ {0})}{f (s _ {t - 1} / s _ {0})} \frac {y _ {t - 1} ^ {2}}{s _ {0}}. \\ \end{array}
$$
Define  $g(x) = -f'(x) / f(x)$  and we can compute that
$$
\int g (x) d x = - \int \frac {f ^ {\prime} (x)}{f (x)} d x = - \int \frac {1}{f} d f = - \log f + C,
$$
where  $C$  is some constant. Doing some calculation yields
$$
\begin{array}{l} g (x) = \frac {d}{d x} - \log f = \frac {d}{d x} \left\{\frac {1}{2} \left(\log (x) + \log \log \left(e ^ {2} x\right)\right) + (1 + \beta) \log \log \log \left(e ^ {2} x\right) \right\} \\ = \frac {1}{2 x} \left\{1 + \frac {1}{\log (e ^ {2} x)} + \frac {1 + \beta}{\log (e ^ {2} x)} \cdot \frac {1}{\log \log (e ^ {2} x)} \right\}. \\ \end{array}
$$
$$
\begin{array}{l} T _ {2} \leq \max  _ {1 \leq t \leq n} \frac {- f ^ {\prime} \left(s _ {t - 1} / s _ {0}\right)}{f \left(s _ {t - 1} / s _ {0}\right)} \frac {y _ {t - 1} ^ {2}}{s _ {0}} = \max  _ {1 \leq t \leq n} g \left(s _ {t - 1} / s _ {0}\right) \frac {y _ {t - 1} ^ {2}}{s _ {0}} \\ \leq \frac {1}{2} \left(\frac {3}{2} + \frac {1 + \beta}{2 \log 2}\right) \max  _ {1 \leq t \leq n} \frac {y _ {t - 1} ^ {2}}{s _ {t - 1}}. \\ \end{array}
$$
We note that demonstrating  $\max_{1\leq t\leq n}y_{t - 1}^2 /s_{t - 1} = o_p(1)$  follows the same approach as the proof of  $\max_{1\leq t\leq n}y_{t - 1}^2 /s_t = o_p(1)$ . Hence, we omit it. To conclude, we show that  $T_{2} = o_{p}(1)$  for  $\theta^{*}\in [-1,1]$
$T_{3}$  with  $\theta^{*} \in [-1, 1]$ : To prove  $T_{3} = o_{p}(1)$ , it suffices to verify that
$$
\frac {s _ {0}}{\sum_ {1 \leq t \leq n} y _ {t} ^ {2}} = o _ {p} (1). \tag {52}
$$
For convenience, in equation (52) we use  $y_{t}$  instead of  $y_{t - 1}$ . Note that when  $\theta^{*} = 1$  or  $\theta^{*} = -1$ , we have provided almost sure lower bounds for  $\sum_{1\leq t\leq n}y_t^2$  in the proof of  $T_{1} = o_{p}(1)$ . Therefore, equation (52) follows from these lower bounds. To prove equation (52) when  $\theta^{*}\in (-1,1)$ , we begin by rewriting  $\sum_{1\leq t\leq n}y_t^2$  in quadratic form. Without confusion and loss of generality, we replace  $\theta^{*}$  by  $\theta$ , consider  $\mathrm{Var}(\epsilon_t) = 1$ , and set  $\varepsilon_{n} = (\epsilon_{1},\epsilon_{2},\dots ,\epsilon_{n})^{\top}$ . For  $t\in [n]$ , we have
$$
y _ {t} = \sum_ {k = 1} ^ {t} \theta^ {t - k} \epsilon_ {k} = \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {\varepsilon} _ {n},
$$
where  $\pmb{a}_t\in \mathbb{R}^n$  and  $a_{t,j} = \theta^{t - j}$  for  $j\leq t$  and  $a_{t,j} = 0$  for  $j > t$ . Therefore,  $\sum_{1\leq t\leq n}y_t^2$  can be written as
$$
\sum_ {1 \leq t \leq n} y _ {t} ^ {2} = \sum_ {1 \leq t \leq n} \varepsilon_ {n} ^ {\top} \boldsymbol {a} _ {t} \boldsymbol {a} _ {t} ^ {\top} \varepsilon_ {n} = \varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n}, \tag {53}
$$
where  $\mathbf{A} = \sum_{1\leq t\leq n}\pmb{a}_{t}\pmb{a}_{t}^{\top}$ . Applying Hanson-Wright inequality (e.g. see [32]), we have
$$
\mathbb {P} \left(\left| \boldsymbol {\varepsilon} _ {n} ^ {\top} \mathbf {A} \boldsymbol {\varepsilon} _ {n} - \mathbb {E} \boldsymbol {\varepsilon} _ {n} ^ {\top} \mathbf {A} \boldsymbol {\varepsilon} _ {n} \right| > t\right) \leq 2 \exp \left[ - c \min  \left(\frac {t ^ {2}}{K ^ {4} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2}}, \frac {t}{K ^ {2} \| \mathbf {A} \| _ {\mathrm {F}}}\right) \right], \tag {54}
$$
where  $c$  and  $K$  are some universal constants. Observe that
$$
\begin{array}{l} \mathbb {E} \boldsymbol {\varepsilon} _ {n} ^ {\top} \mathbf {A} \boldsymbol {\varepsilon} _ {n} = \operatorname {t r a c e} (A) = \operatorname {t r a c e} (\sum_ {1 \leq t \leq n} \boldsymbol {a} _ {t} \boldsymbol {a} _ {t} ^ {\top}) = \operatorname {t r a c e} (\sum_ {1 \leq t \leq n} \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} _ {t}) \\ = \sum_ {1 \leq t \leq n} \left(1 + \theta^ {2} + \dots + \theta^ {2 (t - 1)}\right) \\ = \sum_ {1 \leq t \leq n} \frac {1 - \theta^ {2 t}}{1 - \theta^ {2}} \\ = \frac {n}{1 - \theta^ {2}} - \frac {\theta^ {2} (1 - \theta^ {2 n})}{(1 - \theta^ {2}) ^ {2}}. \\ \end{array}
$$
$$
\begin{array}{l} \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} = \operatorname {t r a c e} (\mathbf {A} ^ {\top} \mathbf {A}) = \operatorname {t r a c e} \left(\sum_ {1 \leq i \leq n} \boldsymbol {a} _ {i} \boldsymbol {a} _ {i} ^ {\top} \cdot \sum_ {1 \leq j \leq n} \boldsymbol {a} _ {j} \boldsymbol {a} _ {j} ^ {\top}\right) \\ = \sum_ {1 \leq i \leq n} \sum_ {1 \leq j \leq n} \left(\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {a} _ {j}\right) ^ {2} \tag {55} \\ = \sum_ {1 \leq i \leq n} \| \boldsymbol {a} _ {i} \| _ {2} ^ {4} + 2 \sum_ {1 \leq i <   j \leq n} \| \boldsymbol {a} _ {i} \| _ {2} ^ {4} \cdot \theta^ {2 (j - i)}. \\ \end{array}
$$
Subsequently, we have
$$
\sum_ {1 \leq i \leq n} \| \boldsymbol {a} _ {i} \| _ {2} ^ {4} \leq \| \mathbf {A} \| _ {\mathrm {F}} ^ {2} \leq \left(1 + \frac {2}{1 - \theta^ {2}}\right) \sum_ {1 \leq i \leq n} \| \boldsymbol {a} _ {i} \| _ {2} ^ {4}, \tag {56}
$$
$$
\sum_ {1 \leq i \leq n} \| \boldsymbol {a} _ {i} \| _ {2} ^ {4} = \frac {n}{(1 - \theta^ {2}) ^ {2}} - \frac {2 \theta^ {2} (1 - \theta^ {2 n})}{(1 - \theta^ {2}) ^ {3}} + \frac {\theta^ {4} (1 - \theta^ {4 n})}{(1 - \theta^ {2}) ^ {2} (1 - \theta^ {4})}.
$$
Assuming  $\delta \leq 2e^{-c}$  and  $t = \frac{1}{c} K^2\| \mathbf{A}\|_{\mathrm{F}}\log \left(\frac{2}{\delta}\right)$ , we have with probability at least  $1 - \delta$
$$
\varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n} \geq \mathbb {E} \varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n} - \frac {1}{c} K ^ {2} \| \mathbf {A} \| _ {\mathrm {F}} \log \left(\frac {2}{\delta}\right). \tag {57}
$$
We note that the term on the right hand side of equation (57) has order  $n$ . For any  $\epsilon > 0$ , consider the following probability
$$
\begin{array}{l} \limsup_{n\to \infty}\mathbb{P}\left(\frac{s_{0}}{\varepsilon_{n}^{\top}\mathbf{A}\varepsilon_{n}} >\epsilon\right)\leq \limsup_{n\to \infty}\mathbb{P}\left(\frac{s_{0}}{\varepsilon_{n}^{\top}\mathbf{A}\varepsilon_{n}} >\epsilon ,\varepsilon_{n}^{\top}\mathbf{A}\varepsilon_{n}\geq \mathbb{E}\varepsilon_{n}^{\top}\mathbf{A}\varepsilon_{n} - \frac{1}{c} K^{2}\| \mathbf{A}\|_{F}\log \left(\frac{2}{\delta}\right)\right) \\ + \lim  _ {n \rightarrow \infty} \mathbb {P} \left(\varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n} <   \mathbb {E} \varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n} - \frac {1}{c} K ^ {2} \| \mathbf {A} \| _ {\mathrm {F}} \log (\frac {2}{\delta})\right) \tag {58} \\ \leq \limsup_{n\to \infty}\mathbb{P}\left(\frac{s_{0}}{\mathbb{E}\varepsilon_{n}^{\top}\mathbf{A}\varepsilon_{n} - \frac{1}{c}K^{2}\| \mathbf{A}\|_{F}\log\left(\frac{2}{\delta}\right)} >\epsilon\right) + \delta . \\ \end{array}
$$
By fixing  $\delta$  and comparing the order of  $s_0$  with the order of  $\varepsilon_n^\top \mathbf{A}\varepsilon_n - \frac{1}{c} K^2\| \mathbf{A}\|_{\mathrm{F}}\log \left(\frac{2}{\delta}\right)$ , we have
$$
\limsup_{n\to \infty}\mathbb{P}\left(\frac{s_{0}}{\mathbb{E}\boldsymbol{\varepsilon}_{n}^{\top}\mathbf{A}\boldsymbol{\varepsilon}_{n} - \frac{1}{c}K^{2}\| \mathbf{A}\|_{\mathrm{F}}\log (\frac{2}{\delta})} >\epsilon\right) = 0.
$$
Since  $\delta$  can be arbitrarily small, we conclude that
$$
\frac {s _ {0}}{\varepsilon_ {n} ^ {\top} \mathbf {A} \varepsilon_ {n}} = o _ {p} (1), \tag {59}
$$
which completes the proof of  $T_{3} = o_{p}(1)$ .
A.4 Proof of Theorem 3.6
Note that for any  $t \geq 1$ , we have
$$
\left\| \mathbf {V} _ {t} \right\| \mathrm {o p} \leq 1 \quad \text {a n d} \quad \mathbf {V} _ {t} = \mathbf {V} _ {t - 1} - \mathbf {V} _ {t - 1} z _ {t} z _ {t} ^ {\top} \mathbf {V} _ {t - 1} / \left(1 + z _ {t} ^ {\top} \mathbf {V} _ {t - 1} z _ {t}\right). \tag {60}
$$
The second part of equation (60) follows from the Sherman-Morrison formula. Let  $\pmb{u}_t = \mathbf{V}_t\pmb{z}_t$  and we adopt the notation  $\mathbf{V}_0 = \mathbf{I}_d$ . By multiplying  $\pmb{z}_t$  on the right hand side of  $\mathbf{V}_t$ , we have
$$
\begin{array}{l} \mathbf {V} _ {t} \boldsymbol {z} _ {t} = \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} - \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} / (1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}) \\ = \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} \left(1 - \frac {\boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}}{1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}}\right) = \frac {\mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}}{1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}}. \tag {61} \\ \end{array}
$$
Therefore, following the definition of  $\pmb{u}_t$ , we have  $(1 + \pmb{z}_t^\top \mathbf{V}_{t - 1}\pmb{z}_t)\pmb{u}_t = \mathbf{V}_{t - 1}\pmb{z}_t$ . Consequently,
$$
\sum_ {t = 1} ^ {n} \left(1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}\right) \boldsymbol {u} _ {t} \boldsymbol {u} _ {t} ^ {\top} = \sum_ {t = 1} ^ {n} \mathbf {V} _ {t - 1} \left(\mathbf {V} _ {t} ^ {- 1} - \mathbf {V} _ {t - 1} ^ {- 1}\right) \mathbf {V} _ {t} = \mathbf {I} _ {d} - \mathbf {V} _ {n}. \tag {62}
$$
By recognizing  $\pmb{w}_t = \sqrt{1 + \pmb{z}_t^\top\mathbf{V}_{t - 1}\pmb{z}_t} \cdot \pmb{u}_t$ , we come to
$$
\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top} = \sum_ {t = 1} ^ {n} \mathbf {V} _ {t - 1} \left(\mathbf {V} _ {t} ^ {- 1} - \mathbf {V} _ {t - 1} ^ {- 1}\right) \mathbf {V} _ {t} = \mathbf {I} _ {d} - \mathbf {V} _ {n}.
$$
What remains now is to verify conditions in (6). Notably, assumption  $\| \mathbf{V}_n\|_{\mathrm{op}} = o_p(1)$  implies
$$
\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top} \xrightarrow {p} \mathbf {I} _ {d}. \tag {63}
$$
Since  $\| \pmb{\Sigma}_0^{-1} \|_{\mathrm{op}} = o_p(1)$ ,  $\| \mathbf{V}_t \|_{\mathrm{op}} \leq 1$  and  $\| \pmb{x}_t \|_2 \leq 1$ , we can show
$$
\max  _ {1 \leq t \leq n} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t} \boldsymbol {z} _ {t} = \max  _ {1 \leq t \leq n} \boldsymbol {x} _ {t} ^ {\top} \boldsymbol {\Sigma} _ {t - 1} ^ {- \frac {1}{2}} \mathbf {V} _ {t} \boldsymbol {\Sigma} _ {t - 1} ^ {- \frac {1}{2}} \boldsymbol {x} _ {t} = o _ {p} (1). \tag {64}
$$
Besides, equation (61) together with equation (64) implies
$$
\max  _ {1 \leq t \leq n} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} = \max  _ {1 \leq t \leq n} \frac {\boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t} \boldsymbol {z} _ {t}}{1 - \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t} \boldsymbol {z} _ {t}} = o _ {p} (1). \tag {65}
$$
Thus, it follows that
$$
\begin{array}{l} \max  _ {1 \leq t \leq n} \| \boldsymbol {w} _ {t} \| _ {2} = \max  _ {1 \leq t \leq n} \left\| \sqrt {1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}} \cdot \mathbf {V} _ {t} \boldsymbol {z} _ {t} \right\| _ {2} \\ \leq \max  _ {1 \leq t \leq n} \left(\sqrt {1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}} \cdot \| \mathbf {V} _ {t} ^ {\frac {1}{2}} \| \mathrm {o p} \cdot \| \mathbf {V} _ {t} ^ {\frac {1}{2}} \boldsymbol {z} _ {t} \| _ {2}\right) \tag {66} \\ \leq \sqrt {\left(1 + \max  _ {1 \leq t \leq n} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}\right) \cdot \max  _ {1 \leq t \leq n} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t} \boldsymbol {z} _ {t}} = o _ {p} (1). \\ \end{array}
$$
Combining equations (66) and (63) yields (6). Hence we complete the proof by applying Proposition 2.1.
Remark A.3. The detailed proof of Lemma 3.7 can be found in the proof of Theorem 3.6.
B Generalized Theorem 3.6
In Theorem 3.6, we impose the following condition (67) so that the ALEE estimator with weights specified in equation (28) achieves asymptotic normality:
$$
\left\| \mathbf {V} _ {n} \right\| \mathbf {o p} = o _ {p} (1). \tag {67}
$$
However, it is typically difficult to directly verify the above condition in practice. To tackle this problem, in this section, we provide a modified version of ALEE estimator which achieves asymptotic normality without requiring condition (67). In this section, we use the same notations  $\pmb{\Sigma}_t$ ,  $\pmb{z}_t$ ,  $\mathbf{V}_t$ , and  $\pmb{w}_t$  as defined in equations (26), (27) and (28), respectively. Furthermore, we let  $\lambda_1 \geq \dots \geq \lambda_n$  be the eigenvalues of the matrix  $\mathbf{V}_n^{-1}$  and  $a_1, \ldots, a_n$  be the corresponding eigenvectors.
At a high level, we construct additional  $m_{n}$  vectors  $\{\pmb{z}_t\}_{n + 1\leq t\leq n + m_n}$  so that the minimum eigenvalue of the resulting matrix  $\mathbf{V}_{n + m_n}^{-1}$  is greater than a pre-specified constant  $\kappa_{n}$ , which satisfies  $\lim_{n\to \infty}\kappa_n = \infty$ . It is easy to see that by construction (see Algorithm 1), the matrix  $\mathbf{V}_{n + m_n}$  satisfies
$$
\left\| \mathbf {V} _ {n + m _ {n}} \right\| \mathrm {o p} \leq \frac {1}{\kappa_ {n}} \xrightarrow {p} 0 \quad \text {w h e r e} \quad m _ {n} = \sum_ {k = 1} ^ {d} n _ {k}. \tag {69}
$$
Remark B.1. Parameter  $\kappa_{n}$  is set to ensure condition (69) holds. In practice, we set  $\kappa_{n} = d\log (n)$ .
Remark B.2. It's worth mentioning that the number of extra  $\{\pmb{z}_t\}_{t > n}$  is a random variable. Therefore, in order to prove a similar asymptotic normality theorem to Theorem 3.6, we have to apply martingale central limit theorem with stopping times [11, Theorem 2.1].
Theorem B.3 (Theorem 2.1 in [11]). Let  $\{\xi_{n,k}\}_{k\geq 1,n\geq 1}$  be an array of random variables defined on a probability space  $(\Omega ,\mathcal{F},P)$  and let  $\{\mathcal{F}_{n,k}\}_{n\geq 1,k\geq 0}$  be an array of  $\sigma$  -fields such that  $\xi_{n,k}$  is  $\mathcal{F}_{n,k}$  -measurable and  $\mathcal{F}_{n,k - 1}\subset \mathcal{F}_{n,k}\subset \mathcal{F}$  for each  $n$  and  $k\geqslant 1$  . For each  $n$  , let  $k_{n}$  be a stopping time with respect to  $\{\mathcal{F}_{n,k}\}_{k\geq 0}$  . Suppose that
Algorithm 1: Modified ALEE estimate
1: Input:  $\{(x_{t},y_{t})\}_{t = 1}^{n}$  and tuning parameter  $\kappa_{n}$    
2: Compute  $\{(z_t,\mathbf{V}_t,w_t)\}_{t = 1}^n$ $(\lambda_k,a_k)\}_{k = 1}^d$  , and obtain a consistent estimate  $\widehat{\sigma}^2$  of  $\sigma^2$    
3: Initiate  $t = n$  and set  $\tau_{n} = 1 / \| \Sigma_{0}^{-1 / 2}\|_{\mathrm{op}}$    
4: for  $k = 1,\dots ,d$  do   
5: Compute  $n_k = \lceil \max \{\kappa_n - \lambda_k,0\} \cdot \tau_n\rceil$    
6: if  $n_k > 0$  then   
7: for  $i = 1,\ldots ,n_k$  do   
8: Set  $t = t + 1$    
9: Simulate  $\epsilon_t\sim \mathcal{N}(0,\widehat{\sigma}^2)$    
10: Define  $z_{t} = a_{k} / \tau_{n}$    
11: Compute
$$
\mathbf {V} _ {t} = \mathbf {V} _ {t - 1} - \frac {\mathbf {V} _ {t - 1} \boldsymbol {z} _ {t} \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1}}{1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}} \quad \text {a n d} \quad \boldsymbol {w} _ {t} = \sqrt {1 + \boldsymbol {z} _ {t} ^ {\top} \mathbf {V} _ {t - 1} \boldsymbol {z} _ {t}} \cdot \mathbf {V} _ {t} \boldsymbol {z} _ {t}
$$
12: end for
13: end if
14: end for
15: Obtain  $\widehat{\theta}_{\mathrm{ALEE}}$  from equation
$$
\sum_ {i = 1} ^ {n} \boldsymbol {w} _ {i} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}}\right) + \sum_ {i = n + 1} ^ {t} \boldsymbol {w} _ {i} \epsilon_ {i} = 0 \tag {68}
$$
16: Output:  $\widehat{\theta}_{\mathrm{ALEE}}$
$$
\sum_ {k = 1} ^ {k _ {n}} \mathbb {E} \left[ \xi_ {n, k} \mid \mathcal {F} _ {n, k - 1} \right] \xrightarrow {p} 0, \tag {70a}
$$
$$
\sum_ {k = 1} ^ {k _ {n}} \operatorname {V a r} \left[ \xi_ {n, k} \mid \mathcal {F} _ {n, k - 1} \right] \xrightarrow {p} 1, \tag {70b}
$$
$$
\sum_ {k = 1} ^ {k _ {n}} \mathbb {E} \left[ | \xi_ {n, k} | ^ {2 + \delta} \mid \mathcal {F} _ {n, k - 1} \right] \xrightarrow {p} 0 \quad f o r s o m e \delta > 0, \tag {70c}
$$
then  $\sum_{k=1}^{k_n} \xi_{n,k} \xrightarrow{d} \mathcal{N}(0,1)$ .
With this setup, we are now ready to prove the asymptotic normality of  $\widehat{\pmb{\theta}}_{\mathrm{ALEE}}$  from (68).
Theorem B.4 (Generalized Theorem 3.6). Suppose condition (3) holds. Then, for any tuning parameters  $\Sigma_0$  and  $\kappa_{n}$  that satisfy  $\| \pmb{\Sigma}_0^{-1}\|_{\mathrm{op}} = o_p(1)$  and  $\lim_{n\to \infty}\kappa_n = \infty$ , the ALEE estimator  $\widehat{\pmb{\theta}}_{\mathrm{ALEE}}$  obtained from equation (68) satisfies
$$
\left(\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {x} _ {t}\right) \cdot \frac {\widehat {\boldsymbol {\theta}} _ {\mathrm {A L E E}} - \boldsymbol {\theta} ^ {*}}{\widehat {\boldsymbol {\sigma}}} \xrightarrow {d} \mathcal {N} (\mathbf {0}, \mathbf {I} _ {d}),
$$
where  $\widehat{\sigma}$  is a consistent estimator of  $\sigma$ .
Remark B.5. We would like to reiterate that the asymptotic variance of the modified ALEE estimator obtained from (68) is the same as the one mentioned in Theorem 3.6. Additionally, this modified version does not require the condition  $\| \mathbf{V}_n\|_{\mathrm{op}} = o_p(1)$  hold and hence is more applicable in practice with theoretical guarantee.
Proof. Rewriting equation (68), we have
$$
\sum_ {t = 1} ^ {n} \boldsymbol {w} _ {t} \boldsymbol {x} _ {t} ^ {\top} \left(\widehat {\boldsymbol {\theta}} _ {\text {A L E E}} - \boldsymbol {\theta} ^ {*}\right) = \sum_ {t = 1} ^ {n + m _ {n}} \boldsymbol {w} _ {t} \epsilon_ {t}. \tag {71}
$$
Therefore, by Cramér-Wold theorem, it suffices to show that for any unit vector  $\pmb{v}$
$$
\sum_ {t = 1} ^ {n + m _ {n}} \boldsymbol {v} ^ {\top} \boldsymbol {w} _ {t} \epsilon_ {t} \xrightarrow {d} \mathcal {N} (0, \sigma^ {2}). \tag {72}
$$
The proof now follows by verifying the conditions (70a)-(70c) of Theorem B.3 with  $\xi_{n,k} = \pmb{v}^{\top}\pmb{w}_{k}\epsilon_{k}$ . We begin by verifying conditions (70a)-(70c). By Lemma 3.7, we have
$$
\sum_ {t = 1} ^ {n + m _ {n}} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top} = \mathbf {I} _ {d} - \mathbf {V} _ {n + m _ {n}}. \tag {73}
$$
$$
\sum_ {t = 1} ^ {n + m _ {n}} \operatorname {V a r} \left[ \boldsymbol {w} _ {t} \epsilon_ {t} \mid \mathcal {F} _ {t - 1} \right] = \sum_ {t = 1} ^ {n + m _ {n}} \sigma^ {2} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top} + \sigma^ {2} \left(\frac {\widehat {\sigma} ^ {2}}{\sigma^ {2}} - 1\right) \sum_ {t = n + 1} ^ {n + m _ {n}} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top}. \tag {74}
$$
By equation (73) and the fact that  $\widehat{\sigma}^2$  is consistent, we have
$$
\sum_ {t = n + 1} ^ {n + m _ {n}} \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top} \preceq \mathbf {I} _ {d} \quad \text {a n d} \quad \frac {\hat {\sigma} ^ {2}}{\sigma^ {2}} - 1 \xrightarrow {p} 0. \tag {75}
$$
Combining equations (69), (73), (74) and (75), we conclude
$$
\sum_ {t = 1} ^ {n + m _ {n}} \operatorname {V a r} \left[ \boldsymbol {w} _ {t} \epsilon_ {t} \mid \mathcal {F} _ {t - 1} \right] \xrightarrow {p} \sigma^ {2} \mathbf {I} _ {d}. \tag {76}
$$
On the other hand, we have