SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \widetilde {\mathbf {U}} _ {o u t} \widetilde {\mathbf {U}} _ {o u t} ^ {\top} - \mathbf {U} _ {o u t} \mathbf {U} _ {o u t} ^ {\top} \right\| _ {2} \| \beta^ {*} \| _ {2} \leq \frac {\left\| \mathbf {Z} _ {x , o u t} \right\| _ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f , o u t}\right)} \| \beta^ {*} \| _ {2} \tag {94}
$$
Using this definition, (91), (92), (93), and (94) we have
$$
\left\| \left(\mathbf {Z} _ {y, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t}\right) ^ {\top} \widehat {\beta} \right\| _ {2} ^ {2} \leq 2 \| \mathbf {Z} _ {x, o u t} ^ {\prime} \| _ {2} ^ {2} \left(\frac {\| \mathbf {Z} _ {x , o u t} \| _ {2} ^ {2} \| \beta^ {*} \| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f , o u t}\right) ^ {2}} + \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2}\right). \tag {95}
$$
Second term:  $\left\| \widehat{\mathbf{Z}}_{f,out}^{\top}\widehat{\beta} - (\mathbf{Z}_{f,out}^{\prime})^{\top}\beta^{*}\right\|_{2}$ . To bound the second term, we follow a similar proof to that shown in [5].
$$
\begin{array}{l} \left\| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} = \left\| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \widehat {\beta} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \beta^ {*} + \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \beta^ {*} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} \\ \leq 2 \| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} + 2 \| \left(\mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t}\right)) ^ {\top} \beta^ {*} \| _ {2} ^ {2} \\ \leq 2 \| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \| _ {2} ^ {2} + 2 \| \mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\prime} \| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2}. \tag {96} \\ \end{array}
$$
Next, we bound the term  $\| \widehat{\mathbf{Z}}_{f,out}^{\top}(\widehat{\beta} -\beta^{*})\| _2^2$
$$
\begin{array}{l} \left\| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} \leq \left\| \left(\widehat {\mathbf {Z}} _ {f, o u t} - \mathbf {Z} _ {f, o u t} ^ {\prime} + \mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} (97) \\ \leq 2 \left\| \left(\widehat {\mathbf {Z}} _ {f, o u t} - \mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} + 2 \left\| \mathbf {Z} _ {f, o u t} ^ {\prime} \right. ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \| _ {2} ^ {2} (98) \\ \leq 2 \left\| \mathbf {Z} _ {x, o u t} ^ {\prime} \right\| _ {2} ^ {2} \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} + 2 \left\| \mathbf {Z} _ {f, o u t} ^ {\prime} \right. ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2}. (99) \\ \end{array}
$$
Next, we bound  $\| \mathbf{Z}_{f,out}^{\prime}^{\top}(\widehat{\beta} -\beta^{*})\|_{2}^{2}$ . Recall that  $\mathbf{U}$  spans the column space of  $\mathbf{Z}_{f,out}^{\prime}$ . Thus  $\mathbf{Z}_{f,out}^{\prime \top} = \mathbf{Z}_{f,out}^{\prime \top}\mathbf{U}\mathbf{U}^{\top}$ , therefore,
$$
\begin{array}{l} \| \mathbf {Z} _ {f, o u t} ^ {-} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} = \| \mathbf {Z} _ {f, o u t} ^ {\prime} ^ {-} \mathbf {U U} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} \\ \leq \left\| \mathbf {Z} _ {f, o u t} ^ {\prime} \right\| _ {2} ^ {2} \left\| \mathbf {U} \mathbf {U} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2}. \tag {100} \\ \end{array}
$$
Now recall that we denote the top  $k$  left singular vectors of  $\mathbf{Z}_y^\prime$  by  $\widetilde{\mathbf{U}}$ , then consider
$$
\begin{array}{l} \left\| \mathbf {U} \mathbf {U} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} = \left\| \left(\mathbf {U} \mathbf {U} ^ {\top} + \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} - \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top}\right) \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} \\ \leq 2 \| \mathbf {U} \mathbf {U} ^ {\top} - \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} \| _ {2} ^ {2} \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} + 2 \| \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2}. \tag {101} \\ \end{array}
$$
Recall that, as we prove in (85) in Appendix F,
$$
\left. \left\| \widetilde {\mathbf {U}} \widetilde {\mathbf {U}} ^ {\top} \left(\widehat {\beta} - \beta^ {*}\right) \right\| _ {2} ^ {2} \leq \frac {8}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}} \left(\left\| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \right\| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2} + \frac {\left\| \mathbf {Z} _ {x} ^ {\prime} \right\| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}} \| [ \mathbf {Z} _ {x} ] _ {L}. \| _ {2} ^ {2} + \left\| [ \mathbf {Z} _ {x} ] _ {L}. ^ {\top} \mathbf {V} \right\| _ {2} ^ {2}\right). \right. \tag {102}
$$
Using (101), (102) and Wedin  $\sin \Theta$  Theorem, we obtain,
$$
\begin{array}{l} \left\| \mathbf {U} \mathbf {U} ^ {\top} (\widehat {\beta} - \beta^ {*}) \right\| _ {2} ^ {2} \leq 2 \frac {\left\| \mathbf {Z} _ {x} ^ {\prime} \right\| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}} \left(\| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} + \frac {8 \left\| [ \mathbf {Z} _ {x} ] _ {L \cdot} \right\| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}}\right) \\ + \frac {8}{\sigma_ {k} \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {2}} \left(\| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2} + \left\| [ \mathbf {Z} _ {x} ] _ {L ^ {.}} ^ {\top} \mathbf {V} \right\| _ {2} ^ {2}\right). \tag {103} \\ \end{array}
$$
Therefore, using (96), (100), and (103), we have
$$
\begin{array}{l} \| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \| _ {2} ^ {2} \\ \leq 2 \| \widehat {\mathbf {Z}} _ {f, o u t} ^ {\top} (\widehat {\beta} - \beta^ {*}) \| _ {2} ^ {2} + 2 \| \mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\prime} \| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2} \\ \leq 4 \| \mathbf {Z} _ {x, o u t} ^ {\prime} \| _ {2} ^ {2} \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} \\ + 4 \| \mathbf {Z} _ {f, o u t} ^ {\prime} \| _ {2} ^ {2} \frac {\| \mathbf {Z} _ {x} ^ {\prime} \| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}} \| \left(\| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} + \frac {8 \| [ \mathbf {Z} _ {x} ] _ {L ^ {\cdot}} \| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}}\right) \\ + 1 6 \left\| \mathbf {Z} _ {f, o u t} ^ {\prime} \right\| _ {2} ^ {2} \frac {1}{\sigma_ {k} \left(\widehat {\mathbf {Z}} _ {f} ^ {\prime}\right) ^ {2}} \left(2 \left\| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \right\| _ {2, \infty} ^ {2} \left\| \beta^ {*} \right\| _ {1} ^ {2} + \left\| [ \mathbf {Z} _ {x} ] _ {L ^ {.}} ^ {\top} \mathbf {V} \right\| _ {2} ^ {2}\right) \\ + 2 \left\| \mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\prime} \right\| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2}. \tag {104} \\ \end{array}
$$
Combining. Incorporating the two bounds in (95) and (104) yields,
$$
\begin{array}{l} \left\| \left(\mathbf {Z} _ {y, o u t} ^ {\prime}\right) ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} \leq c \| \mathbf {Z} _ {x, o u t} ^ {\prime} \| _ {2} ^ {2} \left(\frac {\| \mathbf {Z} _ {x , o u t} \| _ {2} ^ {2} \| \beta^ {*} \| _ {2} ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f , o u t}) ^ {2}} + \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2}\right) \\ + c \| \mathbf {Z} _ {f, o u t} ^ {\prime} \| _ {2} ^ {2} \left(\frac {\| \mathbf {Z} _ {x} ^ {\prime} \| _ {2} ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f} ^ {\prime}) ^ {2}} \| \widehat {\beta} - \beta^ {*} \| _ {2} ^ {2} + \frac {\| \mathbf {Z} _ {x} ^ {\prime} \| _ {2} ^ {2} \| [ \mathbf {Z} _ {x} ] L \cdot \| _ {2} ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f} ^ {\prime}) ^ {4}}\right) \\ + c \frac {\| \mathbf {Z} _ {f , o u t} ^ {\prime} \| _ {2} ^ {2}}{\sigma_ {k} (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {2}} \left(2 \| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \| _ {2, \infty} ^ {2} \| \boldsymbol {\beta} ^ {*} \| _ {1} ^ {2} + \left\| [ \mathbf {Z} _ {x} ] _ {L ^ {.}} ^ {\top} \mathbf {V} \right\| _ {2} ^ {2}\right). \\ + c \| \mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\prime} \| _ {2, \infty} ^ {2} \| \beta^ {*} \| _ {1} ^ {2}. \tag {105} \\ \end{array}
$$
For some absolute constant  $c > 0$
High probability bound. With our choice  $L = \sqrt{NT}$ , Let  $q \coloneqq \sqrt{NT}$  be the number of columns in the stacked page matrix  $\mathbf{Z}_y$  and subsequently  $q + 1$  to be the number of columns in the stacked page matrix  $\mathbf{Z}_{y,out}$ . Let  $C > 0$  be some positive absolute constant, and  $C(f_{\max},\gamma)$  be a constant that
depends only on model parameters  $f_{\mathrm{max}}$  and  $\gamma$ . Define
$$
\bar {E} _ {1} := \left\{\| \mathbf {Z} _ {x} ^ {\prime} \| _ {2} \leq C \sigma_ {x} \sqrt {q} \right\}, \tag {106}
$$
$$
\bar {E} _ {2} := \left\{\left\| \mathbf {Z} _ {x, o u t} ^ {\prime} \right\| _ {2} \leq C \sigma_ {x} \sqrt {q} \right\}, \tag {107}
$$
$$
\bar {E} _ {3} := \left\{\left\| \widehat {\beta} - \beta^ {*} \right\| _ {2} \leq C \left(f _ {\max }, \gamma\right) \left(\frac {\sigma_ {x} ^ {4} G ^ {2} R ^ {2} \log (q)}{q}\right) \max  \left\{\left\| \beta^ {*} \right\| _ {1} ^ {2}, 1 \right\} \right\}, \tag {108}
$$
$$
\bar {E} _ {4} := \left\{\| \mathbf {Z} _ {f} ^ {\prime} - \widehat {\mathbf {Z}} _ {f} ^ {\prime} \| _ {2, \infty} ^ {2} \leq \frac {C \sigma_ {x} ^ {2} q ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max } ^ {2}\right) + C \sigma_ {x} ^ {2} k \log (q) \right\} \tag {109}
$$
$$
\bar {E} _ {5} := \left\{\| \mathbf {Z} _ {f, o u t} ^ {\prime} - \widehat {\mathbf {Z}} _ {f, o u t} ^ {\prime} \| _ {2, \infty} ^ {2} \leq \frac {C \sigma_ {x} ^ {2} q ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max }\right) + C \sigma_ {x} ^ {2} k \log (q) \right\} \tag {110}
$$
$$
\bar {E} _ {6} := \left\{\left\| [ \mathbf {Z} _ {x} ] _ {L} ^ {\top} \mathbf {V} \right\| _ {2} \leq C \sigma_ {x} \sqrt {k \log (q)} \right\}, \tag {111}
$$
$$
\bar {E} _ {7} := \left\{\| [ \mathbf {Z} _ {x} ] _ {L ^ {.}} \| _ {2} \leq C \sigma_ {x} \sqrt {q} \right\}, \tag {112}
$$
First, with  $c_{1} > 0$  is an absolute constant, recall from Lemma F.1,
$$
\mathbb {P} \left(\bar {E} _ {1}\right) \geq 1 - 2 e ^ {- c _ {1} q},
$$
$$
\mathbb {P} (\bar {E} _ {4}) \geq 1 - \frac {c _ {1}}{(N T) ^ {1 1}},
$$
$$
\mathbb {P} (\bar {E} _ {6}) \geq 1 - \frac {c _ {1}}{(N T) ^ {1 1}},
$$
$$
\mathbb {P} \left(\bar {E} _ {7}\right) \geq 1 - 2 e ^ {- c _ {1} q},
$$
Further, note that  $\mathbf{Z}_{x,out}^{\prime}$  is a sub-matrix of  $\mathbf{Z}_{x,out}$ , hence its operator norm is bounded by that of  $\mathbf{Z}_{x,out}$ . Therefore, as an immediate consequence of Lemma A.2,
$$
\mathbb {P} (\bar {E} _ {2}) \geq 1 - 2 e ^ {- c _ {1} q}.
$$
The probability of event  $\bar{E}_3$  is bounded by Theorem 4.3 as
$$
\mathbb {P} (\bar {E} _ {3}) \geq 1 - \frac {c _ {1}}{(N T) ^ {1 1}}.
$$
Finally, as an immediate result of Corollary C.1 and its consequence in (27), we have
$$
\mathbb {P} (\bar {E} _ {5}) \geq 1 - \frac {c _ {1}}{(N T) ^ {1 1}}.
$$
Now consider the event  $\bar{E} := \bar{E}_1 \cap \bar{E}_2 \cap \bar{E}_3 \cap \bar{E}_4 \cap \bar{E}_5 \cap \bar{E}_6 \cap \bar{E}_7$ . Then, under event  $\bar{E}$ , and using (105), we have, with probability  $1 - \frac{c}{(NT)^{11}}$ ,
$$
\begin{array}{l} \left\| \left(\mathbf {Z} _ {y, o u t} ^ {\prime}\right) ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} \leq C (f _ {\max }, \gamma) \sigma_ {x} ^ {2} \left(\frac {\sigma_ {x} ^ {2} q ^ {2} \| \beta^ {*} \| _ {2} ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f , o u t}) ^ {2}} + \left(\sigma_ {x} ^ {4} G ^ {2} R ^ {2} \log (q)\right) \max  \{\| \beta^ {*} \| _ {1} ^ {2}, 1 \}\right) \\ + C \left( \right.f _ {\max }, \gamma) \| \mathbf {Z} _ {f, o u t} ^ {\prime} \| _ {2} ^ {2} \left(\frac {\sigma_ {x} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f} ^ {\prime}\right) ^ {2}} \left(\sigma_ {x} ^ {4} G ^ {2} R ^ {2} \log (q)\right) \max  \left\{\| \beta^ {*} \| _ {1} ^ {2}, 1 \right\}\right) \\ + c \| \mathbf {Z} _ {f, o u t} ^ {\prime} \| _ {2} ^ {2} \frac {\sigma_ {x} ^ {4} q ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f} ^ {\prime}) ^ {4}} \\ + c \frac {\| \mathbf {Z} _ {f , o u t} ^ {\prime} \| _ {2} ^ {2}}{\sigma_ {k} (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {2}} \| \beta^ {*} \| _ {1} ^ {2} \left(\frac {C \sigma_ {x} ^ {2} q ^ {2}}{\sigma_ {k} (\mathbf {Z} _ {f}) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max } ^ {2}\right) + C \sigma_ {x} ^ {2} k \log (q)\right) \\ + c \frac {\| \mathbf {Z} _ {f , o u t} ^ {\prime} \| _ {2} ^ {2}}{\sigma_ {k} (\widehat {\mathbf {Z}} _ {f} ^ {\prime}) ^ {2}} \sigma_ {x} ^ {2} k \log (q) \\ + c \frac {\sigma_ {x} ^ {2} q ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f}\right) ^ {2}} \left(\sigma_ {x} ^ {2} + f _ {\max } ^ {2}\right) \| \beta^ {*} \| _ {1} ^ {2} (113) \\ + c \sigma_ {x} ^ {2} k \log (q) \| \beta^ {*} \| _ {1} ^ {2}. (114) \\ \end{array}
$$
Recalling  $q = \sqrt{NT}$ ,  $k = RG$ , and Assumption 4.1, we get,
$$
\left\| \left(\mathbf {Z} _ {y, o u t} ^ {\prime}\right) ^ {\top} \widehat {\beta} - \left(\mathbf {Z} _ {f, o u t} ^ {\prime}\right) ^ {\top} \beta^ {*} \right\| _ {2} ^ {2} \leq C \left(f _ {\max }, \gamma\right) R ^ {3} G ^ {3} \sigma_ {x} ^ {6} \max  \left\{\| \beta^ {*} \| _ {1} ^ {2}, 1 \right\} \log (N T). \tag {115}
$$
Therefore, with probability  $1 - c / (NT)^{11}$
$$
\begin{array}{l} \frac {L}{N T} \sum_ {n = 1} ^ {N} \sum_ {m = 0} ^ {T / L - 1} \left(\widehat {f} _ {n} (T + m L + 1) - f _ {n} (T + m L + 1)\right) ^ {2} (116) \\ \leq C \left(f _ {\max }, \gamma\right) R ^ {3} G ^ {3} \sigma_ {x} ^ {6} \left(\frac {\operatorname* {m a x} \left\{\| \beta^ {*} \| _ {1} ^ {2} , 1 \right\} \log (N T)}{\sqrt {N T}}\right). (117) \\ \end{array}
$$
Further, note that the same argument can be used to bound the sum  $\frac{L}{NT} \sum_{n=1}^{N} \sum_{m=0}^{T/L-1} \left( \widehat{f}_n(T + mL + \ell) - f_n(T + mL + \ell) \right)^2$  for all  $\ell \in [L]$ . Thus, with probability  $1 - cL/(NT)^{11} > 1 - c/(NT)^{10}$ , we have,
$$
\frac {2}{N T} \sum_ {n = 1} ^ {N} \sum_ {t = T + 1} ^ {2 T} \left(\widehat {f} _ {n} (t) - f _ {n} (t)\right) ^ {2} \leq C \left(f _ {\max }, \gamma\right) R ^ {3} G ^ {3} \sigma_ {x} ^ {6} \left(\frac {\operatorname* {m a x} \left\{\| \beta^ {*} \| _ {1} ^ {2} , 1 \right\} \log (N T)}{\sqrt {N T}}\right). \tag {118}
$$
G.2 Bounding Forecasting Error of  $x$
Now, we bound the error of forecasting  $x_{n}(t)$ . First, recall  $\widetilde{x}_n(t) = y_n(t) - \widehat{f}_n(t)$ , and let  $\widetilde{X}_n(t) \coloneqq [\widetilde{x}_n(t),\ldots ,\widetilde{x}_n(t - p + 1)]$  and  $X_{n}(t)\coloneqq [x_{n}(t),\dots,x_{n}(t - p + 1)]$ . Now, let  $\mathbf{X}_n$ ,  $\widetilde{\mathbf{X}}_n$  and  $\Delta_{n}$ , all in  $\mathbb{R}^{(T - p)\times p}$ , be the row-wise concatenations of  $X_{n}(t)$ ,  $\widetilde{X}_n(t)$ , and  $\Delta_{n}(t)\coloneqq X_{n}(t) - \widehat{X}_{n}(t)$  for  $t\in \{T + p,T + L,\ldots ,2T + p - 1\}$ .
Then, we have, where  $\bar{x}_n(t) = \mathbb{E}\left[x_n(t)|x_n(1),\dots ,x_n(t - 1)\right]$
$$
\begin{array}{l} \sum_ {t = T + 1} ^ {2 T} \left(\hat {x} _ {n} (t) - \bar {x} _ {n} (t)\right) ^ {2} = \left\| \widehat {\alpha} _ {n} ^ {\top} \widetilde {\boldsymbol {X}} _ {n} ^ {\top} - \alpha_ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2} (119) \\ = \left\| \widehat {\alpha} _ {n} ^ {\top} \widetilde {\boldsymbol {X}} _ {n} ^ {\top} - \widehat {\alpha} _ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} + \widehat {\alpha} _ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} - \alpha_ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2} (120) \\ \leq 2 \left(\left\| \hat {\alpha} _ {n} ^ {\top} \widetilde {\boldsymbol {X}} _ {n} ^ {\top} - \hat {\alpha} _ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2} + \left\| \hat {\alpha} _ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} - \alpha_ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2}\right) (121) \\ \leq 2 \left(\left\| \widehat {\alpha} _ {n} ^ {\top} \widetilde {\boldsymbol {X}} _ {n} ^ {\top} - \widehat {\alpha} _ {n} ^ {\top} \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2} + \left\| \widehat {\alpha} _ {n} ^ {\top} - \alpha_ {n} ^ {\top} \right\| _ {2} ^ {2} \left\| \boldsymbol {X} _ {n} ^ {\top} \right\| _ {2} ^ {2}\right) (122) \\ \leq 4 \left(\| \boldsymbol {\Delta} _ {n} \| _ {2} ^ {2} \left(\| \alpha_ {n} \| _ {2} ^ {2} + \| \widehat {\alpha} _ {n} - \alpha_ {n} \| _ {2} ^ {2}\right) + \| \boldsymbol {X} _ {n} \| _ {2} ^ {2} \| \widehat {\alpha} _ {n} - \alpha_ {n} \| _ {2} ^ {2}\right). (123) \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {n = 1} ^ {N} \left\| \boldsymbol {\Delta} _ {n} \right\| _ {2} ^ {2} \leq \sum_ {n = 1} ^ {N} \left\| \boldsymbol {\Delta} _ {n} \right\| _ {F} ^ {2} (124) \\ \leq p \sum_ {n = 1} ^ {N} \sum_ {t = T - p + 1} ^ {2 T - 1} \left(x _ {n} (t) - \widetilde {x} _ {n} (t)\right) ^ {2} (125) \\ = p \sum_ {n = 1} ^ {N} \sum_ {t = T - p + 1} ^ {2 T - 1} \left(f _ {n} (t) - \widehat {f} _ {n} (t)\right) ^ {2}. (126) \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {n = 1} ^ {N} \sum_ {t = T + 1} ^ {2 T} \left(\widehat {x} _ {n} (t) - \bar {x} _ {n} (t)\right) ^ {2} \leq 4 p \max  _ {n} \left(\left\| \alpha_ {n} \right\| _ {2} ^ {2} + \left\| \widehat {\alpha} _ {n} - \alpha_ {n} \right\| _ {2} ^ {2}\right) \sum_ {n = 1} ^ {N} \sum_ {t = T - p + 1} ^ {2 T - 1} \left(f _ {n} (t) - \widehat {f} _ {n} (t)\right) ^ {2} (127) \\ + \sum_ {n = 1} ^ {N} \| \boldsymbol {X} _ {n} \| _ {2} ^ {2} \| \widehat {\alpha} _ {n} - \alpha_ {n} \| _ {2} ^ {2} (128) \\ \end{array}
$$
High probability bound. For the high probability bound, consider the event  $\tilde{E} = \tilde{E}_1 \cap \tilde{E}_2 \cap \tilde{E}_3$  where
$$
\tilde {E} _ {1} := \bigcap_ {n = 1} ^ {N} \left\{\| \widehat {\alpha} _ {n} - \alpha_ {n} \| _ {2} ^ {2} \leq C (f _ {\max }, \gamma) \frac {G R p}{\lambda_ {\min } (\Psi)} \frac {\sigma_ {x} ^ {6}}{\sigma^ {2}} \frac {\log (N T) ^ {2}}{\sqrt {N T}} \max  \left\{1, \frac {1}{\sigma^ {2} \lambda_ {\min } (\Psi)} \right\} + \frac {C p ^ {2}}{T \lambda_ {\min } (\Psi)} \log \left(\frac {T \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right) \right\}
$$
$$
\tilde {E} _ {2} := \left\{\sum_ {n = 1} ^ {N} \sum_ {t = T - p + 1} ^ {2 T - 1} \left(f _ {n} (t) - \widehat {f} _ {n} (t)\right) ^ {2} \leq C \left(f _ {\max }, \gamma\right) R ^ {3} G ^ {3} \sigma_ {x} ^ {6} \left(\max  \left\{\| \beta^ {*} \| _ {1} ^ {2}, 1 \right\} \sqrt {N T} \log (N T)\right) \right. \tag {130}
$$
$$
\tilde {E} _ {3} := \bigcap_ {n = 1} ^ {N} \left\{\| \boldsymbol {X} _ {n} \| _ {2} ^ {2} \leq \frac {3}{2} \sigma^ {2} (T - p) \lambda_ {\max } (\Psi) \right\}, \tag {131}
$$
Note that  $\mathbb{P}(\tilde{E}_1) \geq 1 - \frac{CN}{T^{11}}$  using Theorems 4.2, 4.1 and the union bound. Further, note that  $\mathbb{P}(\tilde{E}_2) \geq 1 - \frac{C}{(NT)^{10}}$  using both Lemma G.1 and Theorem 4.1. Further, note that according to Corollary E.1, with probability  $1 - \frac{c}{T^{11}}$ , we have
$$
\left\| \boldsymbol {X} _ {n} \right\| _ {2} ^ {2} = \lambda_ {\max } \left(\boldsymbol {X} _ {n} ^ {\top} \boldsymbol {X} _ {n}\right) \leq \frac {3}{2} \sigma^ {2} (T - p) \lambda_ {\max } (\Psi). \tag {132}
$$
Therefore, under  $\tilde{E}$ , and recalling  $N < T$ , we have with probability  $1 - \frac{C}{T^{10}}$ ,
$$
\begin{array}{l} \frac {1}{N T} \sum_ {n = 1} ^ {N} \sum_ {t = T + L} ^ {2 T + L} \left(\widehat {x} _ {n} (t) - x _ {n} (t)\right) ^ {2} (133) \\ \leq C \left(f _ {\max }, \gamma\right) G ^ {3} R ^ {3} p ^ {2} \frac {\lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)} \sigma_ {x} ^ {6} \max  \left\{\| \beta^ {*} \| _ {1} ^ {2}, 1 \right\} \left(\frac {p \sigma^ {2}}{T} \log \left(\frac {T \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right)\right. (134) \\ \left. + \frac {G R \alpha_ {\operatorname* {m a x}} ^ {2}}{\min  \{1 , \sigma^ {2} \lambda_ {\min } (\Psi) \}} \frac {\sigma_ {x} ^ {6}}{\sigma^ {2}} \frac {\log (N T) ^ {2}}{\sqrt {N T}}\right). (135) \\ \end{array}
$$
where  $\alpha_{\mathrm{max}} = \max_n\| \alpha_n\| _2$  therefore, with probability  $1 - \frac{C}{T^{10}}$ , we have
$$
\begin{array}{l} \frac {2}{N T} \sum_ {n = 1} ^ {N} \sum_ {t = T + 1} ^ {2 T} \left(\widehat {y} _ {n} (t) - y _ {n} (t)\right) ^ {2} (136) \\ \leq C \left(f _ {\max }, \gamma\right) G ^ {3} R ^ {3} p ^ {2} \frac {\lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)} \sigma_ {x} ^ {6} \max  \left\{\| \beta^ {*} \| _ {1} ^ {2}, 1 \right\} \left(\frac {p \sigma^ {2}}{T} \log \left(\frac {T \lambda_ {\max } (\Psi)}{\lambda_ {\min } (\Psi)}\right)\right. (137) \\ \left. + \frac {G R \left\| \alpha_ {n} \right\| _ {2} ^ {2}}{\min  \left\{1 , \sigma^ {2} \lambda_ {\min } (\Psi) \right\}} \frac {\sigma_ {x} ^ {6}}{\sigma^ {2}} \frac {\log (N T) ^ {2}}{\sqrt {N T}}\right) (138) \\ \end{array}
$$
H Helper Lemmas and Definitions
Definition H.1 (Sub-gaussian random variable [30, 31]). A zero-mean random variable  $X \in \mathbb{R}$  is said to be sub-gaussian with variance proxy  $\sigma^2$  (denoted as  $X \sim \text{subG}(\sigma^2)$ ) if its moment generating function satisfies
$$
\mathbb {E} [ \exp (s X) ] \leq \exp \left(\frac {\sigma^ {2} s ^ {2}}{2}\right) \forall s \in \mathbb {R}.
$$
Definition H.2 (Sub-exponential random variable [30, 31]). A zero-mean random variable  $X \in \mathbb{R}$  is said to be sub-exponential with parameter  $\nu$  (denoted as  $X \sim \text{subE}(\nu)$ ) if its moment generating function satisfies
$$
\mathbb {E} [ \exp (s X) ] \leq \exp \left(\frac {\nu^ {2} s ^ {2}}{2}\right) \forall | s | \leq \frac {1}{\nu}
$$
Lemma H.1 ([31, 30, 6]). Let  $X \sim \text{subG}(\sigma_s^2)$  and  $Z \sim N(0, \sigma_g^2)$ . For  $i \in [N]$ , let  $Y_i \sim \text{subE}(\nu_i)$ . Then:
1.  $Z\sim subG(\sigma_g^2)$
2.  $Z\sim subE(\sigma_g)$
3.  $X^2 -\mathbb{E}[X^2 ]\sim subE(16\sigma_s^2)$
4.  $\sum_{i=1}^{N} Y_i \sim \text{subE}(\sum_{i=1}^{N} \nu_i)$
If  $Y_{i}$  are independent:
5.  $\sum_{i=1}^{N} Y_i \sim \text{subE}((\sum_{i=1}^{N} \nu_i^2)^{1/2})$
Definition H.3 (Sub-gaussian vector). A random vector  $X \in \mathbb{R}^d$  is a sub-gaussian random vector with parameter  $\sigma^2$  if
$$
v ^ {T} X \sim \operatorname {s u b} G \left(\sigma_ {s} ^ {2}\right), \forall v \in \mathbb {S} ^ {d - 1}
$$
where  $\mathbb{S}^{d - 1} = \{x\in \mathbb{R}^d:\| x\| = 1\}$
Lemma H.2 (Norm of sub-gaussian vector). Let  $X \in \mathbb{R}^d$  be a sub-gaussian random vector with parameter  $\sigma^2$ . Then, with probability at least  $1 - \delta$  for  $\delta \in (0,1)$ :
$$
\left\| X \right\| _ {2} \leq 4 \sigma \sqrt {d} + 2 \sigma \sqrt {\log \left(\frac {1}{\delta}\right)}.
$$
This implies that for any  $t > 2\sigma$
$$
\left\| X \right\| _ {2} \leq t,
$$
with probability at least  $1 - \exp \left(-\frac{t^2}{16d\sigma^2}\right)$ .
Definition H.4 (Stationary process [26]). A process  $x(t) \in \mathbb{R}$  is said to be stationary if its expectation does not depend on  $t$  and its autocovariance function depends only on the time difference. That is,
$$
\mathbb {E} [ x (t) ] = \mu
$$
$$
\mathbb {E} ([ x (t) - \mu) (x (t + j) - \mu) ] = \gamma (j)
$$
$\forall t$
$\forall t$  and any  $j$