SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {P} \left(\mathcal {E} _ {3} ^ {c}\right) \leq \mathbb {P} \left(4 \sum_ {i = 1} ^ {p} M _ {i, 1} ^ {2} \geq \epsilon^ {2} \sigma^ {4} (T - p) ^ {2} / 9\right) \\ \leq \sum_ {i = 1} ^ {p} \mathbb {P} \left(| M _ {i, 1} | \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \sqrt {p}}\right) \tag {53} \\ \end{array}
$$
First note that each term  $\mathbb{P}\left(|M_{i,1}| \geq \frac{\epsilon\sigma^2(T - p)}{6\sqrt{p}}\right)$  can be upper bounded by  $2\mathbb{P}\left(\sum_{t = p}^{T - 1} z_i^\top X_{t - 1}\eta_t \geq \frac{\epsilon\sigma^2(T - p)}{6\sqrt{p}}\right)$  where  $z_1 = \alpha$  and  $z_i = e_{i - 1}$  for  $i > 1$  where  $e_i$  is the  $i$ -th standard basis vector. Using Lemma H.7 and choosing  $Z_t = z_i^\top X_{t - 1}$  and  $W_t = \eta_t$ , we have that
$$
\mathbb {P} \left(\left\{\sum_ {t = p} ^ {T - 1} \eta_ {t} z _ {i} ^ {\top} X _ {t - 1} \geq \kappa \right\} \cap \left\{\sum_ {t = p} ^ {T - 1} (z _ {i} ^ {\top} X _ {t - 1}) ^ {2} \leq \lambda \right\}\right) \leq \exp \left(- \frac {\kappa^ {2}}{2 \sigma^ {2} \lambda}\right),
$$
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {t = p} ^ {T - 1} \eta_ {t} z _ {i} ^ {\top} X _ {t - 1} \geq \kappa\right) \leq \exp \left(- \frac {\kappa^ {2}}{2 \sigma^ {2} \lambda}\right) + \mathbb {P} \left(\sum_ {t = p} ^ {T - 1} \left(z _ {i} ^ {\top} X _ {t - 1}\right) ^ {2} > \lambda\right) \\ \leq \exp \left(- \frac {\kappa^ {2}}{2 \sigma^ {2} \lambda}\right) + \mathbb {P} \left(\sum_ {t = p} ^ {T - 1} \| X _ {t - 1} \| _ {2} ^ {2} > \lambda / \| z _ {i} \| _ {2} ^ {2}\right). \\ \end{array}
$$
What is left is to bound  $\mathbb{P}\left(\sum_{t=0}^{T-2} x_t^2 > \frac{\lambda}{p\|z\|_2^2}\right)$ . Using Lemma H.2, we get,
$$
\mathbb {P} \left(\sum_ {t = 0} ^ {T - 2} x _ {t} ^ {2} > \frac {\lambda}{p \| z \| _ {2} ^ {2}}\right) \leq c \exp \left(- \frac {\lambda}{1 6 p T \sigma_ {x} ^ {2} \| z \| _ {2} ^ {2}}\right) \tag {54}
$$
Choosing  $\kappa = \epsilon \sigma^2 (T - p) / (6\sqrt{p})$  and  $\lambda = \| z\| _2^2\epsilon \sigma^2 T^{3 / 2}$  yields
$$
\mathbb {P} \left(\sum_ {t = p} ^ {T - 1} \eta_ {t} z _ {i} ^ {\top} X _ {t - 1} \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \sqrt {p}}\right) \leq \exp \left(- \frac {\epsilon (T - p) ^ {2}}{7 2 p \| z \| _ {2} ^ {2} T ^ {3 / 2}}\right) + c \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{1 6 p \sigma_ {x} ^ {2}}\right).
$$
Which implies, that for  $T > 50p$ , we have,
$$
\mathbb {P} \left(\sum_ {t = p} ^ {T - 1} \eta_ {t} z _ {i} ^ {\top} X _ {t - 1} \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \sqrt {p}}\right) \leq c \exp \left(- \frac {\epsilon \sqrt {T}}{p} \min  \left\{\frac {1}{7 5 \| z \| _ {2} ^ {2}}, \frac {\sigma^ {2}}{1 6 \sigma_ {x} ^ {2}} \right\}\right), \tag {55}
$$
for some absolute constants  $c, C > 0$ . Now, note that
$$
\begin{array}{l} \min  \left\{\frac {1}{7 5 \| z \| _ {2} ^ {2}}, \frac {\sigma^ {2}}{1 6 \sigma_ {x} ^ {2}} \right\} \geq \frac {1}{7 5} \min  \left\{\frac {1}{\| z _ {i} \| _ {2} ^ {2}}, \frac {\sigma^ {2}}{\sigma_ {x} ^ {2}} \right\} \\ \geq \frac {1}{7 5} \frac {\sigma^ {2}}{\sigma_ {x} ^ {2}} \min \left\{\frac {1}{\| z _ {i} \| _ {2} ^ {2}}, 1 \right\} \\ \geq \frac {1}{7 5} \frac {\sigma^ {2}}{\sigma_ {x} ^ {2}} \frac {1}{1 + \| z _ {i} \| _ {2} ^ {2}} \\ \geq \frac {1}{7 5} \frac {\sigma^ {2}}{\sigma_ {x} ^ {2}} \frac {1}{2 + \| \alpha \| _ {2} ^ {2}}. \\ \end{array}
$$
Thus, using (55) and (53), we have
$$
\begin{array}{l} \mathbb {P} \left(\mathcal {E} _ {3} ^ {c}\right) \leq \sum_ {i = 1} ^ {p} \mathbb {P} \left(| M _ {i, 1} | \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \sqrt {p}}\right) (56) \\ \leq \sum_ {i = 1} ^ {p} 2 \mathbb {P} \left(\sum_ {t = p} ^ {T - 1} z _ {i} ^ {\top} X _ {t - 1} \eta_ {t} \geq \frac {\epsilon \sigma^ {2} (T - p)}{6 \sqrt {p}}\right) (57) \\ \leq c p \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{7 5 p \sigma_ {x} ^ {2} \left(2 + \| \alpha \| _ {2} ^ {2}\right)}\right). (58) \\ \end{array}
$$
Using the condition  $T > T_{0}(\epsilon)$  concludes the proof.
Now we argue that the sum of the probabilities deduced from lemmas (E.1), (E.2), and (E.3) can be expressed in the simple form of  $\delta (\epsilon ,T)$  as defined above. First, from lemma (E.1) we have
$$
\mathbb {P} \left(\mathcal {E} _ {1} ^ {\prime}\right) \leq c \exp \left(- \frac {1}{3 \left(\| \alpha \| _ {2} ^ {2} + 1\right)} \frac {\sigma^ {2}}{\sigma_ {x} ^ {2}} \frac {\epsilon \sqrt {T}}{p}\right).
$$
From lemma (E.2), and  $T > T_{0}(\epsilon) > 50p$  we get
$$
\mathbb {P} \left(\mathcal {E} _ {2} ^ {\prime}\right) \leq c \exp \left(\frac {- \epsilon \sqrt {T - p}}{4 8}\right) \leq c \exp \left(\frac {- \epsilon \sqrt {T}}{4 9}\right).
$$
Lastly, lemma (E.3) says, for  $T \geq T_0(\epsilon)$ ,
$$
\mathbb {P} \left(\mathcal {E} _ {3} ^ {\prime}\right) \leq c \exp \left(- \frac {\epsilon \sigma^ {2} \sqrt {T}}{7 5 p \sigma_ {x} ^ {2} \left(2 + \| \alpha \| _ {2} ^ {2}\right)}\right)
$$
Thus we can write
$$
\mathbb {P} \left(\mathcal {E} _ {1} ^ {\prime} \cup \mathcal {E} _ {2} ^ {\prime} \cup \mathcal {E} _ {3} ^ {\prime}\right) \leq c \exp \left(- \frac {1}{7 5 \left(2 + \| \alpha \| _ {2} ^ {2}\right)} \frac {\epsilon \sigma^ {2} \sqrt {T}}{p \sigma_ {x} ^ {2}}\right) := \delta (\epsilon , T).
$$
Where in the statement we set  $\tilde{c} \coloneqq \frac{1}{75\left(2 + \|\alpha\|_2^2\right)}\frac{\sigma_x^2}{\sigma_x^2}$ . Using lemmas (E.1), (E.2), and (E.3) and the subadditivity of the spectral radii of symmetric matrices, we get with probability at least  $1 - \delta(\epsilon, T)$  that
$$
\begin{array}{l} \mathcal {E} _ {1} \cap \mathcal {E} _ {2} \cap \mathcal {E} _ {3} \Longrightarrow \rho (E _ {T} - \sigma^ {2} B B ^ {\top}) \leq \epsilon \sigma^ {2} \\ \Rightarrow \sigma^ {2} (B B ^ {\top} - \epsilon I) \preceq E _ {T} \preceq \sigma^ {2} (B B ^ {\top} + \epsilon I) \\ \Longrightarrow \sigma^ {2} (T - p) (\Psi - \epsilon \Gamma) \preceq \mathbf {X} ^ {\top} \mathbf {X} \preceq \sigma^ {2} (T - p) (\Psi + \epsilon \Gamma). \\ \end{array}
$$
Finally, we establish the following corollary, which is an immediate consequence of Theorem E.1.
Corollary E.1. Consider the  $\mathrm{AR}(p)$  process  $x_{t}$  described in (A.1). Let  $\mathbf{X} \in \mathbb{R}^{(T - p) \times p}$  such that  $[X]_{ij} = x_{i + j - 1}$ . Then, for sufficiently large  $T$  such that  $\frac{\log(T)}{\sqrt{T}} < \frac{\tilde{c}}{22p}\left(\frac{\lambda_{\min}(\Psi)}{\lambda_{\max}(\Gamma)}\right)$ , we have that with probability  $1 - \frac{c}{T^{11}}$ ,
$$
\lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \geq \frac {1}{2} \sigma^ {2} (T - p) \lambda_ {\min } (\Psi), \tag {59}
$$
$$
\lambda_ {\max } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \leq \frac {3}{2} \sigma^ {2} (T - p) \lambda_ {\max } (\Psi), \tag {60}
$$
where  $c, C$  are constants and  $\tilde{c} \coloneqq \frac{1}{75(2 + \| \alpha \|_2^2)} \frac{\sigma_x^2}{\sigma_x^2}$ ,  $\psi$  (the controllability Gramian) and  $\Gamma$  are defined in (14).
Proof. Note that according to Theorem E.1, Weyl's inequality (see Lemma H.3), and using  $\epsilon = \frac{11p\log(T)}{\tilde{c}\sqrt{T}}$  we have, with probability  $1 - \frac{c}{T^D}$
$$
\begin{array}{l} \lambda_ {\min } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \geq \sigma^ {2} (T - p) (\lambda_ {\min } (\Psi) - \epsilon \lambda_ {\max } (\Gamma)) (61) \\ \geq \sigma^ {2} (T - p) \left(\lambda_ {\min } (\Psi) - 1 1 \frac {p \log (T)}{\tilde {c} \sqrt {T}} \lambda_ {\max } (\Gamma)\right) (62) \\ \geq \frac {1}{2} \sigma^ {2} (T - p) (\lambda_ {\min } (\Psi)). (63) \\ \end{array}
$$
$$
\begin{array}{l} \lambda_ {\max } \left(\boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \leq \sigma^ {2} (T - p) (\lambda_ {\min } (\Psi) + \epsilon \lambda_ {\max } (\Gamma)) (64) \\ \leq \frac {3}{2} \sigma^ {2} (T - p) (\lambda_ {\min } (\Psi)). (65) \\ \end{array}
$$
What is left is to check the condition on  $T$  and  $\epsilon$ . Note that Theorem E.1 requires  $T > \frac{75^2\sigma_x^4p^2\log(p)^2}{\sigma^4\epsilon^2}\max \{1,\| \alpha \| _2^4\}$ , which with our choice of  $\epsilon$  implies  $T > p$ . To see that the choice of  $\epsilon$  is valid, note that  $\frac{11p\log(T)}{\tilde{c}\sqrt{T}}\leq \frac{\lambda_{\min}(\Psi)}{2\lambda_{\max}(\Gamma)}$ , which validates the choice of  $\epsilon$ .
Theorem E.2. Consider the  $\mathrm{AR}(p)$  process described in (47). where  $\{\eta_t\}$  is an i.i.d. mean-zero sub-gaussian noise with variance  $\sigma^2$ . Let  $\alpha \coloneqq [\alpha_1, \dots, \alpha_p]^\top$  and  $\bar{\alpha}$  be its least squares estimator. If  $T \geq T_0(\epsilon)$ , then for any unit vector  $w \in S_p$ ,
$$
\mathbb {P} \left(\left| w ^ {\top} (\bar {\alpha} - \alpha) \right| > 2 \sigma \left\| w ^ {\top} V _ {\ell} ^ {- 1 / 2} \right\| _ {2} \sqrt {\log \left(\frac {\det \left(V _ {u} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}}{\delta (\epsilon , T)}\right)}\right) \leq 2 \delta (\epsilon , T),
$$
where  $V_{\ell}, V_{u}T_{0}(\epsilon)$ , and  $\delta(\epsilon, T)$  are as described in Theorem E.1.
Proof. Write  $E = [\eta_{p + 1}, \dots, \eta_T]^\top$ . Using Cauchy-Schwarz, we have
$$
\begin{array}{l} \left| w ^ {\top} \left(\hat {\alpha} (T) - \alpha\right) \right| = \left| w ^ {\top} \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1} \mathbf {X} ^ {\top} E \right| \\ \leq \left\| w ^ {\top} (\mathbf {X} ^ {\top} \mathbf {X}) ^ {- 1 / 2} \right\| _ {2} \left\| (\mathbf {X} ^ {\top} \mathbf {X}) ^ {- 1 / 2} \mathbf {X} ^ {\top} E \right\| _ {2}. \\ \end{array}
$$
Now we will need to introduce a lemma from [1]:
Lemma E.4. Let  $\{\mathcal{F}_t\}_{t=0}^{\infty}$  be a filtration. Let  $\{\eta_t\}_{t=1}^{\infty}$  be a real-valued stochastic process such that  $\eta_t$  is  $\mathcal{F}_t$ -measurable and is conditionally sub-gaussian with variance proxy  $\sigma^2 > 0$ . Further, let  $\{X_t\}_{t=0}^{\infty}$  be an  $\mathbb{R}^d$ -valued stochastic process such that  $X_t$  is  $\mathcal{F}_{t-1}$ -measurable. Consider any positive definite matrix  $V \in \mathbb{R}^{d \times d}$ . For any  $t \geq 1$ , define
$$
\bar {V} _ {t} = V + \sum_ {s = 1} ^ {t} X _ {s} X _ {s} ^ {\top}, \quad S _ {t} = \sum_ {s = 1} ^ {t} \eta_ {s} X _ {s}.
$$
Then for any  $\delta > 0$ , with probability at least  $1 - \delta$  for all  $t \geq 1$ ,
$$
\| S _ {t} \| _ {\bar {V} _ {t} ^ {- 1}} ^ {2} \leq 2 \sigma^ {2} \left(\frac {\det  (\bar {V} _ {t}) ^ {1 / 2} \det  (V) ^ {- 1 / 2}}{\delta}\right).
$$
We will apply this result to bound the second term above. Choosing  $V = V_{\ell}$ ,  $\sum_{s=1}^{t} X_{s} X_{s}^{\top} = \mathbf{X}^{\top} \mathbf{X}$ , and  $S_{t} = \mathbf{X}^{\top} E$ , we have with probability at least  $1 - \delta(\epsilon, T)$
$$
\| \mathbf {X} ^ {\top} E \| _ {(\mathbf {X} ^ {\top} \mathbf {X} + V _ {\ell}) ^ {- 1}} \leq \sqrt {2 \sigma^ {2} \log \left(\frac {\operatorname* {d e t} (\mathbf {X} ^ {\top} \mathbf {X} + V _ {\ell}) ^ {1 / 2} \operatorname* {d e t} (V _ {\ell}) ^ {- 1 / 2}}{\delta (\epsilon , T)}\right)}.
$$
Now further condition on the event of Theorem E.1, which occur together with the condition above with probability at least  $1 - 2\delta(\epsilon, T)$ . Recall, then, that  $\mathbf{X}^{\top}\mathbf{X} + V_{\ell} \preceq 2\mathbf{X}^{\top}\mathbf{X}$ , and so  $(\mathbf{X}^{\top}\mathbf{X} + V_{\ell})^{-1} \succeq \frac{1}{2} (\mathbf{X}^{\top}\mathbf{X})^{-1}$ . This means that
$$
\begin{array}{l} \left\| \left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1 / 2} \mathbf {X} ^ {\top} E \right\| _ {2} = \| \mathbf {X} ^ {\top} E \| _ {\left(\mathbf {X} ^ {\top} \mathbf {X}\right) ^ {- 1}} \\ \leq \sqrt {2} \| \mathbf {X} ^ {\top} E \| _ {(\mathbf {X} ^ {\top} \mathbf {X} + V _ {\ell}) ^ {- 1}} \\ \leq 2 \sigma \sqrt {\log \left(\frac {\operatorname* {d e t} \left(\mathbf {X} ^ {\top} \mathbf {X} + V _ {\ell}\right) ^ {1 / 2} \operatorname* {d e t} \left(V _ {\ell}\right) ^ {- 1 / 2}}{\delta (\epsilon , T)}\right)} \\ = 2 \sigma \sqrt {\log \left(\frac {\operatorname* {d e t} \left(\mathbf {X} ^ {\top} \mathbf {X} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}}{\delta (\epsilon , T)}\right)} \\ \leq 2 \sigma \sqrt {\log \left(\frac {\operatorname* {d e t} \left(V _ {u} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}}{\delta (\epsilon , T)}\right)}. \\ \end{array}
$$
Finally, it is clear that under the conditions in Theorem E.1 we have  $\left\| w^{\top}(\mathbf{X}^{\top}\mathbf{X})^{-1 / 2}\right\|_{2}\leq$ $\left\| w^{\top}V_{\ell}^{-1 / 2}\right\|_{2}.$
Corollary E.2. Let  $\epsilon = 12\frac{p\log(T)}{\tilde{c}\sqrt{T}}$ , where  $\tilde{c}$  is as defined in Theorem (E.1). Then, for  $T$  such that  $\frac{\log(T)}{\sqrt{T}} < \frac{\tilde{c}}{24p}\left(\frac{\lambda_{\min}(\Psi)}{\lambda_{\max}(\Gamma)}\right)$  we have
$$
\mathbb {P} \left(\| \bar {\alpha} - \alpha \| _ {2} > C p \sqrt {\frac {1}{T \lambda_ {\min} (\Psi)} \log \left(\frac {T \lambda_ {\max} (\Psi)}{\lambda_ {\min} (\Psi)}\right)}\right) \leq \frac {c}{T ^ {1 1}},
$$
Proof. First, note that with the choice of  $\epsilon = 12\frac{p\log(T)}{\tilde{c}\sqrt{T}}$  we have,
$$
\begin{array}{l} \delta (\epsilon , T) = c \exp (- \log (T ^ {1 2})) \\ = \frac {c}{T ^ {1 2}} \tag {66} \\ \end{array}
$$
Hence,  $-\log (\delta (\epsilon ,T)) = C\log (T)$  .Further,
$$
\begin{array}{l} \left\| w ^ {\top} V _ {\ell} ^ {- 1 / 2} \right\| _ {2} \leq \left\| V _ {\ell} ^ {- 1 / 2} \right\| _ {2} \\ = \sqrt {\lambda_ {\max} (V _ {\ell} ^ {- 1})} \\ = \frac {1}{\sqrt {\lambda_ {\operatorname* {m i n}} (V _ {\ell})}} \\ \leq \frac {1}{\sigma \sqrt {(T - p) (\lambda_ {\operatorname* {m i n}} (\Psi) - \epsilon \lambda_ {\operatorname* {m a x}} (\Gamma))}} \\ \leq \frac {2}{\sigma \sqrt {(T - p) \lambda_ {\operatorname* {m i n}} (\Psi)}}, \tag {67} \\ \end{array}
$$
where the last inequality holds with probability  $1 - \delta(\epsilon, T)$  using Theorem E.1. Finally, consider
$$
\begin{array}{l} \log \left(\det  \left(V _ {u} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}\right) = \frac {1}{2} \log \left(\frac {\det  \left(V _ {u} + V _ {\ell}\right)}{\det  \left(V _ {\ell}\right)}\right) \\ = \frac {1}{2} \log \left(\frac {2 ^ {p} \sigma^ {2 p} (T - p) ^ {p} \det (\Psi)}{\det \left(V _ {\ell}\right)}\right) \tag {68} \\ \end{array}
$$
Note that, again by our assumed condition on  $\epsilon$ ,
$$
\begin{array}{l} \det  \left(V _ {\ell}\right) \geq \frac {1}{2} \det  (\Psi) \\ \geq \frac {1}{2} \sigma^ {2 p} (T - p) ^ {p} \lambda_ {\min } (\Psi) ^ {p} \tag {69} \\ \end{array}
$$
Using (69) and (68), we have,
$$
\begin{array}{l} \log \left(\det  \left(V _ {u} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}\right) \leq \frac {1}{2} \log \left(\frac {2 ^ {p + 1} \det  (\Psi)}{\lambda_ {\min } (\Psi) ^ {p}}\right) \\ \leq p \log \left(\frac {4 \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right) \tag {70} \\ \end{array}
$$
From (70), (67) and (66), we have,
$$
\begin{array}{l} 2 \sigma \left\| w ^ {\top} V _ {\ell} ^ {- 1 / 2} \right\| _ {2} \sqrt {\log \left(\frac {\det \left(V _ {u} V _ {\ell} ^ {- 1} + I _ {p}\right) ^ {1 / 2}}{\delta (\epsilon , T)}\right)} \\ \leq \sqrt {\frac {C p}{(T - p) \lambda_ {\operatorname* {m i n}} (\Psi)}} \sqrt {\log \left(\frac {4 \lambda_ {\operatorname* {m a x}} (\Psi)}{\lambda_ {\operatorname* {m i n}} (\Psi)}\right) + \log (T)}. \tag {71} \\ \end{array}
$$
Using (71) and Theorem E.2, we get that with probability  $1 - \frac{2c}{T^{12}}$
$$
\left| \bar {\alpha} _ {i} (T) - \alpha_ {i} \right| \leq \sqrt {\frac {C p}{(T - p) \lambda_ {\min } (\Psi)}} \sqrt {\log \left(\frac {4 \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right) + \log (T)}, \tag {72}
$$
For any  $i\in [p]$  . This implies,
$$
\left\| \bar {\alpha} - \alpha \right\| _ {2} \leq \sqrt {\sum_ {i = 1} ^ {p} \left| \hat {\alpha} _ {i} (T) - \alpha_ {i} \right| ^ {2}} \leq \sqrt {\frac {C p ^ {2}}{(T - p) \lambda_ {\min } (\Psi)}} \sqrt {\log \left(\frac {4 \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right) + \log (T)}, \tag {73}
$$
with probability  $1 - \frac{c}{T^{11}}$  which completes the proof.
F.1 Setup
Recall  $\beta^{*}$  as defined in Proposition 2.2, and its estimate  $\widehat{\beta}$  defined as
$$
\widehat {\beta} = \operatorname {a r g m i n} _ {\beta \in \mathbb {R} ^ {L - 1}} \quad \sum_ {m = 1} ^ {N \times T / L} \left(y _ {m} - \beta^ {\top} \widehat {F} _ {m}\right) ^ {2}. \tag {74}
$$
To establish this bound, we first define (and recall) the following notations,
- Recall that  $\mathbf{Z}_y = [\mathbf{Z}(y_1, L, T, 1) \quad \mathbf{Z}(y_2, L, T, 1) \quad \dots \quad \mathbf{Z}(y_n, L, T, 1)]$  is the stacked Page matrix of  $y_1(t), \ldots, y_n(t) \forall t \in [T]$ .
- Let  $\mathbf{Z}_x$  be the stacked Page matrix of  $x_1(t),\ldots ,x_n(t)\forall t\in [T]$
- Let  $\mathbf{Z}_f$  be the stacked Page matrix of  $f_1(t),\ldots ,f_n(t)\forall t\in [T]$ . That is,  $\mathbf{Z}_y = \mathbf{Z}_f + \mathbf{Z}_x$ .
- Let  $\mathbf{Z}_y' \in \mathbb{R}^{(L - 1) \times (NT / L)}$ , be the sub-matrix obtained by dropping the  $L$ -th row from the stacked Page matrix  $\mathbf{Z}_y$ . Define  $\mathbf{Z}_f'$  and  $\mathbf{Z}_x'$  analogously.
- Let  $\mathbf{U}\pmb{\Sigma}\mathbf{V}^{\top}$  denote the SVD of  $\mathbf{Z}_f^\prime$
- let  $\mathbf{V}^{\perp}$  and  $\mathbf{U}^{\perp}$  be matrices of orthonormal basis vectors that span the null space of  $\mathbf{Z}_f'$  and  $\mathbf{Z}_{f}^{\prime \top}$ , respectively.
- Let  $\widetilde{\mathbf{U}}\widetilde{\boldsymbol{\Sigma}}\widetilde{\mathbf{V}}^{\top}$  denote the top  $k$  singular components of the SVD of  $\mathbf{Z}_y^\prime$ , while  $\widetilde{\mathbf{U}}^{\perp}\widetilde{\boldsymbol{\Sigma}}^{\perp}(\widetilde{\mathbf{V}}^{\perp})^{\top}$  denote the remaining  $L - k - 1$  components such that  $\mathbf{Z}_y^\prime = \widetilde{\mathbf{U}}\widetilde{\boldsymbol{\Sigma}}\widetilde{\mathbf{V}}^{\top} + \widetilde{\mathbf{U}}^{\perp}\widetilde{\boldsymbol{\Sigma}}^{\perp}(\widetilde{\mathbf{V}}^{\perp})^{\top}$ .
- Let  $\widehat{\mathbf{Z}}_f^\prime$  be the HSVT estimate of  $\mathbf{Z}_f^\prime$  with parameter  $k$ . That is  $\widehat{\mathbf{Z}}_f^\prime = \widetilde{\mathbf{U}}\widetilde{\boldsymbol{\Sigma}}\widetilde{\mathbf{V}}^\top$ .
F.2 Deterministic Bound
First, given the notation above, the solution for (74), can be written as,
$$
\widehat {\beta} = \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {\top , \dagger} \left[ \mathbf {Z} _ {y} \right] _ {L}. = \widetilde {\mathbf {U}} \widetilde {\boldsymbol {\Sigma}} ^ {- 1} \widetilde {\mathbf {V}} ^ {\top} \left[ \mathbf {Z} _ {y} \right] _ {L}. \tag {75}
$$
Then, let's consider the following expansion for  $\| \widehat{\beta} -\beta^{*}\|_{2}$
$$
\begin{array}{l} \left\| \widehat {\beta} - \beta^ {*} \right\| _ {2} ^ {2} = \left\| \widetilde {\mathbf {U}} ^ {\perp} \left(\widetilde {\mathbf {U}} ^ {\perp}\right) ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) + \widetilde {\mathbf {U}} \left(\widetilde {\mathbf {U}}\right) ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} \\ = \| \widetilde {\mathbf {U}} ^ {\perp} (\widetilde {\mathbf {U}} ^ {\perp}) ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} + \| \widetilde {\mathbf {U}} (\widetilde {\mathbf {U}}) ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} \\ = \| \widetilde {\mathbf {U}} ^ {\perp} (\widetilde {\mathbf {U}} ^ {\perp}) ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} + \| \widetilde {\mathbf {U}} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} \\ = \| \widetilde {\mathbf {U}} ^ {\perp} (\widetilde {\mathbf {U}} ^ {\perp}) ^ {\top} \beta^ {*} \| _ {2} ^ {2} + \| \widetilde {\mathbf {U}} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2}, \tag {76} \\ \end{array}
$$
where the last equality follows from the fact that  $(\widetilde{\mathbf{U}}^{\perp})^{\top}\widehat{\beta} = \left((\widetilde{\mathbf{U}}^{\perp})^{\top}\widetilde{\mathbf{U}}\right)\widetilde{\mathbf{\Sigma}}^{-1}\widetilde{\mathbf{V}}^{\top}[\mathbf{Z}_{y}]_{L.} = 0$ . Now, consider the first term in (76), and note that  $\beta^{*} = \mathbf{UU}^{\top}\beta^{*}$
$$
\begin{array}{l} \left\| \tilde {\mathbf {U}} ^ {\perp} \left(\tilde {\mathbf {U}} ^ {\perp}\right) ^ {\top} \beta^ {*} \right\| _ {2} = \left\| \mathbf {U} ^ {\perp} \left(\mathbf {U} ^ {\perp}\right) ^ {\top} \mathbf {U} \mathbf {U} ^ {\top} \beta^ {*} + \left(\tilde {\mathbf {U}} ^ {\perp} \left(\tilde {\mathbf {U}} ^ {\perp}\right) ^ {\top} - \mathbf {U} ^ {\perp} \left(\mathbf {U} ^ {\perp}\right) ^ {\top}\right) \beta^ {*} \right\| _ {2} \\ \leq \left\| \left(\widetilde {\mathbf {U}} ^ {\perp} (\widetilde {\mathbf {U}} ^ {\perp}) ^ {\top} - \mathbf {U} ^ {\perp} (\mathbf {U} ^ {\perp}) ^ {\top}\right) \beta^ {*} \right\| _ {2} \\ \leq \left\| \widetilde {\mathbf {U}} ^ {\perp} (\widetilde {\mathbf {U}} ^ {\perp}) ^ {\top} - \mathbf {U} ^ {\perp} (\mathbf {U} ^ {\perp}) ^ {\top} \right\| _ {2} \| \boldsymbol {\beta} ^ {*} \| _ {2} \\ = \left\| \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} - \mathbf {U} \mathbf {U} ^ {\top} \right\| _ {2} \| \beta^ {*} \| _ {2}. \tag {77} \\ \end{array}
$$
Next, by Wedin sin  $\Theta$  Theorem (see [10, 32]) we have:
$$
\begin{array}{l} \left\| \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} - \mathbf {U} \mathbf {U} ^ {\top} \right\| _ {2} \| \beta^ {*} \| _ {2} \leq \frac {\| \mathbf {Z} _ {y} ^ {\prime} - \mathbf {Z} _ {f} ^ {\prime} \| _ {2}}{\sigma_ {k} (\mathbf {Z} _ {f} ^ {\prime})} \| \beta^ {*} \| _ {2} \\ = \frac {\left\| \mathbf {Z} _ {x} ^ {\prime} \right\| _ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right)} \| \beta^ {*} \| _ {2}. \tag {78} \\ \end{array}
$$
$$
\left\| \widetilde {\mathbf {U}} ^ {\perp} \left(\widetilde {\mathbf {U}} ^ {\perp}\right) ^ {\top} \beta^ {*} \right\| _ {2} \leq \frac {\left\| \mathbf {Z} _ {x} ^ {\prime} \right\| _ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right)} \| \beta^ {*} \| _ {2}. \tag {79}
$$
What is left is bounding  $\| \widetilde{\mathbf{U}}^{\top}(\widehat{\beta} -\beta^{*})\|_{2}^{2}$ . To that end, first consider
$$
\begin{array}{l} \| (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} \leq 2 \| (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {\top} \widehat {\beta} - (\mathbf {Z} _ {f} ^ {\prime}) ^ {\top} \beta^ {*} \| _ {2} ^ {2} + 2 \| (\mathbf {Z} _ {f} ^ {\prime}) ^ {\top} \beta^ {*} - (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {\top} \beta^ {*} \| _ {2} ^ {2} \\ \leq 2 \| \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {\top} \beta^ {*} \| _ {2} ^ {2} + 2 \| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2}. \tag {80} \\ \end{array}
$$
Also, consider
$$
\begin{array}{l} \| \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} = \left(\widehat {\beta} - \beta^ {*}\right) ^ {\top} \widetilde {\mathbf {U}} \widetilde {\boldsymbol {\Sigma}} ^ {2} \widetilde {\mathbf {U}} ^ {\top} (\widehat {\beta} - \beta^ {*}) \\ \geq \sigma_ {k} \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {2} \| \widetilde {\mathbf {U}} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \| _ {2} ^ {2}. \tag {81} \\ \end{array}
$$
From (81) and (80) we get,
$$
\left\| \widetilde {\mathbf {U}} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} \leq \frac {2}{\sigma_ {k} \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {2}} \left(\left\| \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} + \left\| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \right\| _ {2, \infty} ^ {2} \left\| \beta^ {*} \right\| _ {1} ^ {2}\right). \tag {82}
$$
First, recall that  $[\mathbf{Z}_y]_{L^{\cdot}} = [\mathbf{Z}_f]_{L^{\cdot}} + [\mathbf{Z}_x]_{L^{\cdot}}$ , and that by definition,  $(\mathbf{Z}_f')^\top \beta^* = [\mathbf{Z}_f]_{L^{\cdot}}$ . Then the term  $\| (\widehat{\mathbf{Z}}_f')^\top \widehat{\beta} - (\mathbf{Z}_f')^\top \beta^* \|_2^2$  can be bounded as follows. First, consider