SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem C.1. Let assumptions 2.1, 2.2, 2.3, and 2.4 hold. Let  $\sigma_{x} = \frac{c_{\lambda}\sigma}{(1 - \lambda_{\star})}$ , where  $\lambda_{\star} = \max_{i,n}|\lambda_{ni}|$  and  $c_{\lambda} = \max_{n}\sum_{i = 1}^{p_n}\left|\prod_{1\leq j\leq p,j\neq i}\left(1 - \frac{\lambda_{nj}}{\lambda_{ni}}\right)^{-1}\right|$ . Then, the HSVT estimate  $\widehat{\mathbf{Z}}_f$  with parameter  $k$  is such that
$$
\max  _ {j \in [ q ]} \frac {1}{L} \left\| \left[ \widehat {\mathbf {Z}} _ {f} \right] _ {\cdot j} - \left[ \mathbf {Z} _ {f} \right] _ {\cdot j} \right\| _ {2} ^ {2} \leq \frac {C \sigma_ {x} ^ {2} (N T) ^ {2}}{L ^ {3} \sigma_ {k} \left(\mathbf {Z} _ {f}\right) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\operatorname* {m a x}} ^ {2}\right) + \frac {C \sigma_ {x} ^ {2} k \log (N T / L)}{L}, \tag {25}
$$
with probability of at least  $1 - \frac{c}{(NT)^{11}}$ .
Proof. We start by identifying certain high probability events. Subsequently, using these events and (24), we will conclude the proof.
High Probability Events. Let  $q \coloneqq NT / L$  be the number of columns in the stacked page matrix  $\widehat{\mathbf{Z}}_f$ , and let  $C > 0$  be some positive absolute constant. Define
$$
E _ {1} := \left\{\| \mathbf {Z} _ {x} \| _ {2} \leq C \sigma_ {x} \sqrt {q} \right\},
$$
$$
E _ {2} := \left\{\| \mathbf {Z} _ {x} \| _ {\infty , 2}, \| \mathbf {Z} _ {x} \| _ {2, \infty} \leq C \sigma_ {x} \sqrt {q} \right\},
$$
$$
E _ {3} := \left\{\max _ {j \in [ q ]} \| \mathbf {U} _ {k} \mathbf {U} _ {k} ^ {\top} [ \mathbf {Z} _ {x} ] _ {\cdot j} \| _ {2} ^ {2} \leq C \sigma_ {x} ^ {2} k \log (q) \right\},
$$
Lemma C.2. For some positive constant  $c_1 > 0$  and  $C > 0$  large enough in definitions of  $E_1, E_2,$  and  $E_3$ ,
$$
\mathbb {P} (E _ {1}) \geq 1 - 2 e ^ {- c _ {1} q},
$$
$$
\mathbb {P} \left(E _ {2}\right) \geq 1 - 2 e ^ {- c _ {1} q},
$$
$$
\mathbb {P} (E _ {3}) \geq 1 - \frac {c _ {1}}{(N T) ^ {1 1}}.
$$
Proof. We bound the probability of events above below.
Bounding  $E_{1}$ . This is an immediate consequence of Lemma A.2.
Bounding  $E_{2}$ . Recall that we assume  $L \leq q$ . Observe that for any matrix  $A \in \mathbb{R}^{L \times q}$ ,  $\| A \|_{\infty, 2}$ ,  $\| A \|_{2, \infty} \leq \| A \|_{2}$ . Thus using the argument to bound  $E_{1}$ , concludes the proof.
Bounding  $E_3$ . Let  $u_{1}, \ldots, u_{k}$  be the orthonormal basis for  $\mathbf{U}_k$ . Then, consider for  $j \in [q]$ ,
$$
\left\| \mathbf {U} _ {k} \mathbf {U} _ {k} ^ {\top} [ \mathbf {Z} _ {x} ] _ {. j} \right\| _ {2} ^ {2} = \sum_ {i = 1} ^ {k} \left\| u _ {i} u _ {i} ^ {\top} [ \mathbf {Z} _ {x} ] _ {. j} \right\| _ {2} ^ {2} \leq \sum_ {i = 1} ^ {k} \left(u _ {i} ^ {\top} [ \mathbf {Z} _ {x} ] _ {. j}\right) ^ {2} = \sum_ {i = 1} ^ {k} Z _ {i} ^ {2},
$$
where  $Z_{i} = u_{i}^{\top}[\mathbf{Z}_{x}]_{\cdot j}$ . As shown in Lemma A.1,  $[\mathbf{Z}_x]_{\cdot j}$  is a sub-gaussian vector with variance proxy  $\sigma_x^2$ . Using Lemma H.4, we have
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {k} Z _ {i} ^ {2} > t\right) \leq c \exp \left(- \frac {t}{1 6 k \sigma_ {x} ^ {2}}\right).
$$
Therefore, for choice of  $t = C\sigma_x^2 k\log (q)$  with large enough constant  $C > 368$  we have,
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {k} Z _ {i} ^ {2} > C \sigma_ {x} ^ {2} k \log (q)\right) \leq \frac {c _ {1}}{q ^ {2 3}}.
$$
Recalling that  $L \leq q$ , and taking a union bound over all  $j \in [q]$ , we have that
$$
\mathbb {P} \left(E _ {4} ^ {c}\right) \leq \frac {c _ {1}}{(q L) ^ {1 1}}.
$$
The following is an immediate corollary of the above stated bounds.
Corollary C.1. Let  $E \coloneqq E_1 \cap E_2 \cap E_3$ . Then,
$$
\mathbb {P} \left(E ^ {c}\right) \leq \frac {C _ {1}}{\left(N T\right) ^ {1 1}}, \tag {26}
$$
where  $C_1$  is an absolute positive constant.
Thus, under event  $E$ , and using (24), we have, with probability  $1 - \frac{c}{(NT)^{10}}$ ,
$$
\begin{array}{l} \max  _ {j \in [ q ]} \left\| \left[ \widehat {\mathbf {Z}} _ {f} \right]. j - \left[ \mathbf {Z} _ {f} \right]. j \right\| _ {2} ^ {2} \leq \frac {C \sigma_ {x} ^ {2} q}{\sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(q \sigma_ {x} ^ {2} + q f _ {\max } ^ {2}\right) \\ + C \sigma_ {x} ^ {2} k \log (q) \\ \end{array}
$$
Recall that  $q = NT / L$  then,
$$
\max  _ {j \in [ q ]} \frac {1}{L} \left\| \left[ \widehat {\mathbf {Z}} _ {f} \right] _ {. j} - \left[ \mathbf {Z} _ {f} \right] _ {. j} \right\| _ {2} ^ {2} \leq \frac {C \sigma_ {x} ^ {2} (N T) ^ {2}}{L ^ {3} \sigma_ {k} \left(\mathbf {Z} _ {f}\right) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max } ^ {2}\right) + \frac {C \sigma_ {x} ^ {2} k \log (N T / L)}{L} \tag {27}
$$
This completes the proof of Theorem C.1. Finally, now we are ready to bound  $\operatorname{EstErr}(N, T)$ . First, let  $\Omega_n = \left\{\frac{(n-1)T}{L} + 1, \dots, \frac{nT}{L}\right\}$  be the set of columns in the stacked Page matrix  $\mathbf{Z}_f$  that belongs to the  $n$ -th time series  $f_n(t)$ . Note that
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\widehat {f} _ {n} (t) - f _ {n} (t)\right) ^ {2} = \frac {1}{T} \sum_ {j \in \Omega_ {n}} \left\| \left[ \widehat {\mathbf {Z}} _ {f} \right] _ {\cdot j} - \left[ \mathbf {Z} _ {f} \right] _ {\cdot j} \right\| _ {2} ^ {2} \\ \leq \frac {1}{T} \sum_ {j \in \Omega_ {n}} \frac {C \sigma_ {x} ^ {2} (N T) ^ {2}}{L ^ {2} \sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max } ^ {2}\right) + C \sigma_ {x} ^ {2} k \log (N T / L) \\ \leq \frac {C \sigma_ {x} ^ {2} (N T) ^ {2}}{L ^ {3} \sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\mathrm {m a x}} ^ {2}\right) + \frac {C \sigma_ {x} ^ {2} k \log (N T / L)}{L}. \\ \end{array}
$$
Choosing  $L = \sqrt{NT}$ , and using  $\sigma_k(\mathbf{Z}_f)^2 \geq \frac{\gamma^2 NT}{k}$ , we get,
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \left(\widehat {f} _ {n} (t) - f _ {n} (t)\right) ^ {2} \leq \frac {C f _ {\max } ^ {2} \gamma^ {2} \sigma_ {x} ^ {4} k \log (N T)}{\sqrt {N T}}.
$$
Finally and setting  $k = RG$  concludes the proof.
In this section, we prove Theorem 4.2, which bounds the estimation error  $\| \widehat{\alpha}_n - \alpha_n\|_2 \forall n \in [N]$ . First, recall the OLS estimate  $\widehat{\alpha}_n = [\widehat{\alpha}_{n,1}, \widehat{\alpha}_{n,2}, \ldots, \widehat{\alpha}_{n,p}]$  defined as
$$
\widehat {\alpha} _ {n} = \operatorname {a r g m i n} _ {\alpha \in \mathbb {R} ^ {p}} \sum_ {t = p} ^ {T - 1} \left(\widehat {x} _ {n} (t + 1) - \alpha^ {\top} \widehat {X} _ {n} (t)\right) ^ {2}, \tag {28}
$$
where  $\widehat{X}_n(t) \coloneqq [\widehat{x}_n(t), \dots, \widehat{x}_n(t - p + 1)]$ . Alternatively, the OLS estimate can be defined as,
$$
\widehat {\alpha} _ {n} = \left(\widehat {\boldsymbol {X}} _ {n} ^ {\top} \widehat {\boldsymbol {X}} _ {n}\right) ^ {- 1} \widehat {\boldsymbol {X}} _ {n} ^ {\top} \widehat {Y} _ {n}, \tag {29}
$$
where  $\widehat{\pmb{X}}_n\in \mathbb{R}^{(T - p)\times p}$  is the row-wise concatenation of  $\widehat{X}_n(t)$  for  $t\in \{p,p + 1,\ldots ,T - 1\}$ , and  $\widehat{Y}_n = [\widehat{x}_n(p + 1),\dots ,\widehat{x}_n(T)]$ . Further, consider the OLS estimate with access to the true AR  $(p)$  series  $x_{n}(\cdot)$ . Precisely, let  $\bar{\alpha}_{n}$  defined as
$$
\bar {\alpha} _ {n} = \left(\boldsymbol {X} _ {n} ^ {\top} \boldsymbol {X} _ {n}\right) ^ {- 1} \boldsymbol {X} _ {n} ^ {\top} Y _ {n}, \tag {30}
$$
where  $\mathbf{X}_n \in \mathbb{R}^{(T - p) \times p}$  is the row-wise concatenation of  $X_n(t) \coloneqq [x_n(t), \ldots, x_n(t - p + 1)]$  for  $t \in \{p, p + 1, \ldots, T - 1\}$ , and  $Y_n = [x_n(p + 1), \ldots, x_n(T)]$ .
In what follows, as we do the analysis for one process  $x_{n}$ , we drop the subscript  $n$ . We note that the analysis holds for all  $n \in [N]$ . First, note that,
$$
\left\| \widehat {\alpha} - \alpha \right\| _ {2} ^ {2} \leq 2 \left\| \widehat {\alpha} - \bar {\alpha} \right\| _ {2} ^ {2} + 2 \left\| \bar {\alpha} - \alpha \right\| _ {2} ^ {2} \tag {31}
$$
To bound  $\| \bar{\alpha} - \alpha \|_2^2$ , we use Corollary E.2 which states that with probability  $1 - \frac{c}{T^{11}}$
$$
\left\| \bar {\alpha} - \alpha \right\| _ {2} ^ {2} \leq C \frac {p ^ {2}}{T \lambda_ {\min } (\Psi)} \log \left(\frac {T \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right). \tag {32}
$$
Next, to bound  $\| \widehat{\alpha} - \bar{\alpha} \|_2^2$ , let  $M^\dagger \coloneqq (M^\top M)^{-1} M^\top$  denote the Moore-Penrose inverse of a matrix  $M$ . Further, let  $\delta(t) = x(t) - \widehat{x}(t) = \widehat{f}(t) - f(t)$ , and  $\Delta = X - \widehat{X}$ , i.e.,  $\Delta$  is the row-wise concatenation of  $\Delta(t) \coloneqq [\delta(t), \ldots, \delta(t - p + 1)]^\top$  for  $t \in \{p, p + 1, \ldots, T - 1\}$ . Then, with  $\Delta_Y \coloneqq [\delta(p + 1), \ldots, \delta(T)]$ , we have
$$
\begin{array}{l} \left\| \widehat {\alpha} - \bar {\alpha} \right\| _ {2} = \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \widehat {Y} - \boldsymbol {X} ^ {\dagger} Y \right\| _ {2} \\ = \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \Delta_ {Y} + (\boldsymbol {X} ^ {\dagger} - \widehat {\boldsymbol {X}} ^ {\dagger}) Y \right\| _ {2} \\ \leq \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \Delta_ {Y} \right\| _ {2} + \left\| (\boldsymbol {X} ^ {\dagger} - \widehat {\boldsymbol {X}} ^ {\dagger}) Y \right\| _ {2} \\ \leq \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \right\| _ {2} \| \Delta_ {Y} \| _ {2} + \left\| \boldsymbol {X} ^ {\dagger} - \widehat {\boldsymbol {X}} ^ {\dagger} \right\| _ {2} \| Y \| _ {2} \\ \leq \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \right\| _ {2} \| \Delta_ {Y} \| _ {2} + 2 \max  \left\{\left\| \boldsymbol {X} ^ {\dagger} \right\| _ {2} ^ {2}, \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \right\| _ {2} ^ {2} \right\} \| \boldsymbol {\Delta} \| _ {2} \| Y \| _ {2} \tag {33} \\ \end{array}
$$
Where Lemma H.6 is used in the last inequality. Note that,
$$
\left\| \boldsymbol {X} ^ {\dagger} \right\| _ {2} ^ {- 1} = \inf  _ {v \in \mathbb {R} ^ {p}: \| v \| _ {2} = 1} \| \boldsymbol {X} v \| _ {2} = \sqrt {\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right)} \tag {34}
$$
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {X}} ^ {\dagger} \right\| _ {2} ^ {- 1} = \inf _ {v \in \mathbb {R} ^ {p}: \| v \| _ {2} = 1} \left\| \widehat {\boldsymbol {X}} v \right\| _ {2} \\ = \inf  _ {v \in \mathbb {R} ^ {p}: \| v \| _ {2} = 1} \| (\boldsymbol {X} - \boldsymbol {\Delta}) v \| _ {2} \\ \geq \sqrt {\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right)} - \| \boldsymbol {\Delta} \| _ {2} \tag {35} \\ \end{array}
$$
Using (33), (34), (35), and the theorem conditions, we get
$$
\left\| \widehat {\alpha} - \bar {\alpha} \right\| _ {2} \leq \frac {\left\| \Delta_ {Y} \right\| _ {2}}{\sqrt {\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right)} - \left\| \boldsymbol {\Delta} \right\| _ {2}} + \frac {2 \left\| \boldsymbol {\Delta} \right\| _ {2} \left\| Y \right\| _ {2}}{\min  \left\{\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) , \left(\sqrt {\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right)} - \left\| \boldsymbol {\Delta} \right\| _ {2}\right) ^ {2} \right\}}. \tag {36}
$$
Using Corollary E.1 we get that with probability  $1 - \frac{c}{T^{11}}$
$$
\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \geq \frac {1}{2} \sigma^ {2} (T - p) \lambda_ {\min } (\Psi). \tag {37}
$$
Now, note that  $\| \Delta_Y\| _2^2 = \sum_{t = p + 1}^T (\widehat{f} (t) - f(t))^2$  , and
$$
\left\| \boldsymbol {\Delta} \right\| _ {2} ^ {2} \leq \left\| \boldsymbol {\Delta} \right\| _ {F} ^ {2} \leq p \sum_ {t = 1} ^ {T} (\widehat {f} (t) - f (t)) ^ {2}. \tag {38}
$$
Recall that  $\sum_{t=1}^{T} (\widehat{f}(t) - f(t))^2 = T\mathrm{EstErr}(N, T, n)$ , which we will refer to herein as  $\mathrm{EstErr}(N, T)$  as we drop the dependence on  $n$ . Therefore, we have,
$$
\left\| \boldsymbol {\Delta} \right\| _ {2} ^ {2} \leq p T E s t E r r (N, T) \tag {39}
$$
$$
\left\| \Delta_ {Y} \right\| _ {2} ^ {2} \leq T \operatorname {E s t E r r} (N, T). \tag {40}
$$
Finally, using Lemma H.4, with probability at least  $1 - \frac{c}{T^{11}}$  it holds for an absolute constant  $C > 4$ ,
$$
\left\| Y \right\| _ {2} \leq C \sigma_ {x} \sqrt {T \log (T)}. \tag {41}
$$
First, note that, using (37) and (39), we have
$$
\begin{array}{l} \sqrt {\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right)} - \| \boldsymbol {\Delta} \| _ {2} \geq \frac {\sigma \sqrt {(T - p) \lambda_ {\min } (\Psi)}}{\sqrt {2}} - \sqrt {p T \operatorname {E s t E r r} (N , T)} \\ \geq C \sigma \sqrt {T \lambda_ {\min } (\Psi)}, \tag {42} \\ \end{array}
$$
for sufficiently small  $\operatorname{EstErr}(N, T)$  such that  $\operatorname{EstErr}(N, T) \leq \frac{\sigma^2 \lambda_{\min}(\Psi)}{6p}$ . Now using (36), (37), (39), (40), (41), and (42) we have,
$$
\| \widehat {\alpha} - \bar {\alpha} \| _ {2} \leq \frac {C \sigma_ {x} \sqrt {p \operatorname {E s t E r r} (N , T) \log (T)}}{\min \left\{\sigma^ {2} \lambda_ {\min } (\Psi) , \sigma \sqrt {\lambda_ {\min} (\Psi)} \right\}}
$$
$$
\left\| \widehat {\alpha} - \bar {\alpha} \right\| _ {2} ^ {2} \leq C \frac {p \log (T)}{\lambda_ {\min } (\Psi)} \frac {\sigma_ {x} ^ {2}}{\sigma^ {2}} \operatorname {E s t E r r} (N, T) \max  \left\{1, \frac {1}{\sigma \lambda_ {\min } (\Psi)} \right\} \tag {43}
$$
finally, using (31), (32), and (43), we get with probability  $1 - \frac{c}{T^{11}}$
$$
\left\| \widehat {\alpha} - \alpha \right\| _ {2} ^ {2} \leq C \frac {p \log (T)}{\lambda_ {\min } (\Psi)} \frac {\sigma_ {x} ^ {2}}{\sigma^ {2}} \operatorname {E s t E r r} (N, T) \max  \left\{1, \frac {1}{\sigma^ {2} \lambda_ {\min } (\Psi)} \right\} + \frac {C p ^ {2}}{T \lambda_ {\min } (\Psi)} \log \left(\frac {T \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right), \tag {44}
$$
which completes the proof of Theorem 4.2.
E Identification of AR processes
In this section, we state key theorems for the identification of AR processes without the perturbation we consider in this paper. The theorems in this section and their proofs follow closely the ones in [17], we modify them to accommodate general sub-gaussian noise and state them fully for completeness.
First, consider the stationary  $\mathrm{AR}(p)$  process defined by the transition matrix
$$
\boldsymbol {A} = \left[ \begin{array}{c c} \alpha_ {1: p - 1} ^ {\top} & \alpha_ {p} \\ I _ {p - 1} & 0 _ {p - 1} \end{array} \right], \tag {45}
$$
where  $0_{p - 1}$  is the column vector of  $p - 1$  zeroes, and input vector
$$
B = \left[ \begin{array}{c} 1 \\ 0 _ {p - 1} \end{array} \right]. \tag {46}
$$
Then, with  $X(t) \coloneqq [x(t), \ldots, x(t - p + 1)]$ , the AR process is described as
$$
X (t) = \boldsymbol {A} X (t - 1) + B \eta_ {t}. \tag {47}
$$
With  $\eta_t$  being a sub-gaussian random variable with variance proxy  $\sigma^2$ , With this setup, we define two important matrices,
$$
\Gamma = \sum_ {t = 0} ^ {\infty} \boldsymbol {A} ^ {t} \left(\boldsymbol {A} ^ {\top}\right) ^ {t} \tag {48}
$$
$$
\Psi = \sum_ {t = 0} ^ {\infty} \boldsymbol {A} ^ {t} B B ^ {\top} \left(\boldsymbol {A} ^ {\top}\right) ^ {t}, \tag {49}
$$
where  $\Psi$  is known as the controllability Gramian.
Theorem E.1. Consider the  $\mathrm{AR}(p)$  process described in (47),  $\{\eta_t\}$  is an i.i.d. mean-zero subgaussian noise with variance  $\sigma^2$ . Given  $0 < \epsilon \leq 1$ , define the following quantities:
$$
V _ {\ell} := \sigma^ {2} (T - p) (\Psi - \epsilon \Gamma)
$$
$$
V _ {u} := \sigma^ {2} (T - p) (\Psi + \epsilon \Gamma)
$$
$$
\delta (\epsilon , T) := c \exp \left(- \tilde {c} \frac {\epsilon \sqrt {T}}{p}\right)
$$
$$
T _ {0} (\epsilon) := \frac {7 5 ^ {2} \sigma_ {x} ^ {4} p ^ {2} \log (p) ^ {2}}{\sigma^ {4} \epsilon^ {2}} \max  \{1, \| \alpha \| _ {2} ^ {4} \}
$$
where  $c$  and  $\tilde{c}$  are constants that depends only on the process coefficients  $\alpha_{1},\ldots ,\alpha_{p}$ . Then, for  $T > T_0(\epsilon)$  and all valid values of  $\epsilon$  such that  $V_{\ell}\succ 0$ , we have
$$
\mathbb {P} \left(V _ {\ell} \preceq \mathbf {X} ^ {\top} \mathbf {X} \preceq V _ {u}\right) \geq 1 - \delta (\epsilon , T).
$$
Remark E.1. To get  $V_{\ell} \succ 0$ , one must choose a sufficiently small  $\epsilon$  such that  $(\Psi - \epsilon \Gamma) \succ 0$ . Using Weyl's inequality, we have
$$
\lambda_ {\min } (\Psi - \epsilon \Gamma) \geq \lambda_ {\min } (\Psi) - \epsilon \lambda_ {\max } (\Gamma). \tag {50}
$$
This suggests that  $\epsilon < \frac{\lambda_{\min}(\Psi)}{\lambda_{\max}(\Gamma)}$  yields  $V_{\ell} \succ 0$ . For example, a choice of  $\epsilon = \frac{\lambda_{\min}(\Psi)}{2\lambda_{\max}(\Gamma)}$  will yield  $V_{\ell} \succeq \frac{1}{2}\sigma^{2}(T - p)\lambda_{\min}(\Psi)I_{p}$ . Note that this ensures  $\epsilon \leq 1/2$ , since  $\epsilon \leq \frac{\lambda_{\max}(\Psi)}{\lambda_{\max}(\Gamma)}$  and  $\Gamma \succeq \Psi$  as can be seen from
$$
\begin{array}{l} z _ {i} ^ {\top} (\Gamma - \Psi) z = \sum_ {t = 0} ^ {\infty} z _ {i} ^ {\top} A ^ {t} (A ^ {\top}) ^ {t} z - z _ {i} ^ {\top} A ^ {t} B B ^ {\top} (A ^ {\top}) ^ {t} z \\ = \sum_ {t = 0} ^ {\infty} \left\| \left(A ^ {\top}\right) ^ {t} z \right\| _ {2} ^ {2} - \left(B ^ {\top} \left(A ^ {\top}\right) ^ {t} z\right) ^ {2} \\ \geq \sum_ {t = 0} ^ {\infty} \left\| \left(A ^ {\top}\right) ^ {t} z \right\| _ {2} ^ {2} - \underbrace {\left\| B \right\| _ {2} ^ {2}} _ {1} \left\| \left(A ^ {\top}\right) ^ {t} z \right\| _ {2} ^ {2} \\ \geq 0. \\ \end{array}
$$
Proof. First, write
$$
\mathbf {X} ^ {\top} \mathbf {X} = \sum_ {t = p} ^ {T - 1} X _ {t} X _ {t} ^ {\top}.
$$
Expanding the summand using the process dynamics yields
$$
\begin{array}{l} X _ {t + 1} X _ {t + 1} ^ {\top} = \left(A X _ {t} + B \eta_ {t + 1}\right) \left(A X _ {t} + B \eta_ {t + 1}\right) ^ {\top} \\ = A X _ {t} X _ {t} ^ {\top} A ^ {\top} + \eta_ {t + 1} A X _ {t} B ^ {\top} + \eta_ {t + 1} B X _ {t} ^ {\top} A ^ {\top} + \eta_ {t + 1} ^ {2} B B ^ {\top}. \\ \end{array}
$$
Now, defining  $V_{T} = \mathbf{X}^{\top}\mathbf{X} / (T - p)$ , we can sum the expression above over  $t = p - 1,\dots ,T - 2$  to get
$$
V _ {T} = A V _ {T} A ^ {\top} + \underbrace {\frac {1}{T - p} \left(A (X _ {p - 1} X _ {p - 1} ^ {\top} - X _ {T - 1} X _ {T - 1} ^ {\top}) A ^ {\top} + \sum_ {t = p - 1} ^ {T - 2} (\eta_ {t + 1} A X _ {t} B ^ {\top} + \eta_ {t + 1} B X _ {t} ^ {\top} A ^ {\top} + \eta_ {t + 1} ^ {2} B B ^ {\top})\right)} _ {E _ {T}}.
$$
This is called a Lyapunov equation, and due to the stability of the process it has solution  $V_{T} = \sum_{t=0}^{\infty} A^{t} E_{T}(A^{\top})^{t}$ . The key step is to argue that  $E_{T}$  converges to  $\sigma^{2} B B^{\top}$  with high probability, which will allow us to show that  $V_{T}$  converges to  $\sigma^{2} \Psi$ . To do this, we will split the terms in  $E_{T}$  into three groups, handling them separately. For some  $\epsilon > 0$ , we will bound the probability of the following events:
$$
\mathcal {E} _ {1} := \left\{\rho \left(A \left(X _ {p - 1} X _ {p - 1} ^ {\top} - X _ {T - 1} X _ {T - 1} ^ {\top}\right) A ^ {\top}\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \right\},
$$
$$
\mathcal {E} _ {2} := \left\{\rho \left(\sum_ {t = p - 1} ^ {T - 2} \left(\eta_ {t + 1} ^ {2} B B ^ {\top} - \sigma^ {2} B B ^ {\top}\right)\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \right\},
$$
$$
\mathcal {E} _ {3} := \left\{\rho \left(\sum_ {t = p - 1} ^ {T - 2} \eta_ {t + 1} A X _ {t} B ^ {\top} + \eta_ {t + 1} B X _ {t} ^ {\top} A ^ {\top}\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \right\},
$$
where  $\rho (\cdot)$  denotes the spectral radius of a matrix.
Lemma E.1. With the setup above, it holds that
$$
\mathbb {P} \left(\mathcal {E} _ {1}\right) \geq 1 - c \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{3 \sigma_ {x} ^ {2} p (\| \alpha \| _ {2} ^ {2} + 1)}\right)
$$
for some absolute constant  $c > 0$ .
$$
\begin{array}{l} \rho \left(A \left(X _ {p - 1} X _ {p - 1} ^ {\top} - X _ {T - 1} X _ {T - 1} ^ {\top}\right) A ^ {\top}\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \\ \Longleftarrow \rho \left(A \left(X _ {p - 1} X _ {p - 1} ^ {\top} + X _ {T - 1} X _ {T - 1} ^ {\top}\right) A ^ {\top}\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \\ \Longleftarrow \| A X _ {p - 1} \| _ {2} ^ {2} + \| A X _ {T - 1} \| _ {2} ^ {2} \leq \epsilon \sigma^ {2} (T - p) / 3 \\ \Longleftarrow \| X _ {p - 1} \| _ {2} ^ {2} + \| X _ {T - 1} \| _ {2} ^ {2} \leq \frac {\epsilon \sigma^ {2} (T - p)}{3 \| A \| _ {2}} \\ \Longleftarrow \| X _ {p - 1} \| _ {2} ^ {2} \vee \| X _ {T - 1} \| _ {2} ^ {2} \leq \frac {\epsilon \sigma^ {2} (T - p)}{6 \| A \| _ {2}}, \\ \end{array}
$$
so, considering the fact that  $X_{p - 1}$  and  $X_{T - 1}$  are identically distributed by stationarity, it suffices to upper bound the probability of
$$
\mathcal {E} ^ {\prime} = \left\{\| X _ {p - 1} \| _ {2} ^ {2} > \frac {\epsilon \sigma^ {2} (T - p)}{6 \| A \| _ {2}} \right\}.
$$
First, recall that  $X_{p - 1}$  is a sub-gaussian vector with variance proxy  $\frac{\sigma^2\gamma^2}{(1 - \lambda^*)^2} \coloneqq \sigma_x^2$  (see Lemma A.2). Thus, using Lemma H.2, for
$$
\begin{array}{l} \mathbb {P} \left(\| X _ {p - 1} \| _ {2} ^ {2} \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \| A \| _ {2}}\right) \leq c \exp \left(- \frac {\epsilon \sigma^ {2} (T - p)}{9 6 \sigma_ {x} ^ {2} p \| A \| _ {2}}\right) \\ \leq c \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{3 \sigma_ {x} ^ {2} \| A \| _ {2}}\right), \\ \end{array}
$$
where in the last inequality, we used the assumption  $T > T_0(\epsilon) > 75^2 p^2$ . Finally, note that  $\| A\| _2\leq \mathrm{Tr}(A) = \| \alpha \| _2^2 +p - 1\leq p(\| \alpha \| _2^2 +1)$ . Recalling that  $\mathbb{P}(\mathcal{E}_1) > 1 - 2\mathbb{P}(\mathcal{E}')$  concludes the proof.
Lemma E.2. Continuing with the setup above, it holds that
$$
\mathbb {P} \left(\mathcal {E} _ {2}\right) \geq 1 - c \exp \left(\frac {- \epsilon \sqrt {T - p}}{4 8}\right).
$$
Proof. Since the spectral radius of  $BB^{\top}$  is unity,  $\mathcal{E}_2$  is equivalent to the complement of the event
$$
\mathcal {E} ^ {\prime} = \left\{\left| (T - p) ^ {- 1} \sum_ {t = p - 1} ^ {T - 2} \left(\eta_ {t + 1} ^ {2} - \sigma^ {2}\right) \right| > \epsilon \sigma^ {2} / 3 \right\}.
$$
Since  $\eta_{t + 1}$  is sub-gaussian with variance proxy  $\sigma^2$ ,  $\eta_{t + 1}^2 - \sigma^2$  is sub-exponential with parameter  $16\sigma^2$ . Thus, Lemma H.5 applies, and therefore,
$$
\mathbb {P} \left(\mathcal {E} ^ {\prime}\right) \leq c \exp \left(\frac {- \epsilon \sqrt {T - p}}{4 8}\right). \tag {51}
$$
Lemma E.3. With the same setup, we have
$$
\mathbb {P} \left(\mathcal {E} _ {3}\right) \geq 1 - c \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{7 5 p \sigma_ {x} ^ {2} \left(2 + \| \alpha \| _ {2} ^ {2}\right)}\right)
$$
$$
T \geq T _ {0} (\epsilon) := \frac {7 5 ^ {2} \sigma_ {x} ^ {4} p ^ {2} \log (p) ^ {2}}{\sigma^ {4} \epsilon^ {2}} \left(2 + \| \alpha \| _ {2} ^ {2}\right) ^ {2}.
$$
Proof. Recall the definition
$$
\mathcal {E} _ {3} := \left\{\rho \left(\sum_ {t = p - 1} ^ {T - 2} \eta_ {t + 1} A X _ {t} B ^ {\top} + \eta_ {t + 1} B X _ {t} ^ {\top} A ^ {\top}\right) \leq \epsilon \sigma^ {2} (T - p) / 3 \right\}.
$$
For the sake of a variational analysis of the spectral radius, consider any vector  $q = [q_{1} \tilde{q}^{\top}]^{\top} \in \mathbb{R}^{p}$  such that  $\| q \|_2 = 1$ . For notational simplicity. Then
$$
q ^ {\top} \underbrace {\left(\sum_ {t = p - 1} ^ {T - 2} \eta_ {t + 1} A X _ {t} B ^ {\top} + \eta_ {t + 1} B X _ {t} ^ {\top} A ^ {\top}\right)} _ {M + M ^ {\top}} q \leq 2 \| M q \| _ {2}.
$$
Next, we aim to bound  $\| Mq\| _2$ . First note that only the first column of  $M$  is nonzero which evaluates to
$$
M _ {., 1} = \eta_ {t + 1} \cdot \left[ \begin{array}{c} \sum_ {t = p - 1} ^ {T - 2} \alpha^ {\top} X _ {t} \\ \sum_ {t = p - 1} ^ {T - 2} x _ {t} \\ \vdots \\ \sum_ {t = p - 1} ^ {T - 2} x _ {t - p + 1} \end{array} \right] \tag {52}
$$
Therefore, we have  $Mq = q_{1}M_{\cdot ,1}$ , which implies that for any unit vector  $q$
$$
q ^ {\top} \left(M + M ^ {\top}\right) q \leq 2 \sqrt {\sum_ {i = 1} ^ {p} M _ {i , 1} ^ {2}}.
$$
Therefore, we will bound  $\mathbb{P}(\mathcal{E}_3^c)$  by