SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $y_{m}$  is the  $m$ -th component of  $[y_1(L), y_1(2 \times L), \ldots, y_1(T), y_2(L), \ldots, y_2(T), \ldots, y_N(T)] \in \mathbb{R}^{N \times T / L}$ , and  $\widehat{F}_m \in \mathbb{R}^{L - 1}$  corresponds to the vector formed by the entries of the first  $L - 1$  rows in the  $m$ th column of  $\widehat{\mathbf{Z}}_f^8$  for  $m \in [N \times T / L]$ . We then use  $\widehat{\beta}$  to produce  $\widehat{f}_n(t) = \widehat{\beta}^\top Y_n(t - 1)$ , where again  $Y_{n}(t - 1)$  is the vector of the  $L - 1$  lags of  $y_{n}(t - 1)$ . That is  $Y_{n}(t - 1) = [y_{n}(t - 1)) \ldots y_{n}(t - L)]$ .
Then, for each  $n \in [N]$ , we estimate  $\alpha_{ni} \forall i \in [p_n]$ , the parameters for the  $n$ -th  $\mathrm{AR}(p_n)$  process. Let  $\widehat{\alpha}_n$  denote the OLS estimate defined as
$$
\widehat {\alpha} _ {n} = \operatorname {a r g m i n} _ {\alpha \in \mathbb {R} ^ {p _ {n}}} \quad \sum_ {t = p _ {n}} ^ {T - 1} \left(\widehat {x} _ {n} (t + 1) - \alpha^ {\top} \widehat {X} _ {n} (t)\right) ^ {2}. \tag {7}
$$
Then, produce a forecast for  $x_{n}$  as  $\widehat{x}_n(t) = \widehat{\alpha}_n^\top \widetilde{X}_n(t - 1)$ , where again  $\widetilde{X}_n(t - 1) = [\widetilde{x}_n(t - 1),\ldots ,\widetilde{x}_n(t - p_n)]$ . Finally, produce the forecast for  $y_{n}$  as  $\widehat{y}_n(t) = \widehat{f}_n(t) + \widehat{x}_n(t)$ . For a visual depiction of the algorithm, refer to Figure 1.
In this section, we provide finite-sample high probability bounds on the following quantities:
1. Estimation error of non-stationary component. First, we give an upper bound for the estimation error of each one of  $f_{1}(t), \ldots, f_{N}(t)$  for  $t \in [T]$ . Specifically, we upper bound the following metric
$$
\operatorname {E s t E r r} (N, T, n) = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\widehat {f} _ {n} (t) - f _ {n} (t)\right) ^ {2}, \tag {8}
$$
To be precise,  $\widehat{\mathbf{Z}}_f$  here is the truncated SVD of a sub-matrix of  $\mathbf{Z}_y$  which consist of its first  $L - 1$  rows.
where  $\widehat{f}_n(t) n \in [N], t \in [T]$  are the estimates produced by the algorithm we proposed in Section 3.
2. Identification of AR parameters. Second, we analyze the accuracy of the estimates  $\widehat{\alpha}_n\forall n\in [N]$  produced by the proposed algorithm. Specifically, we upper bound the following metrics
$$
\left\| \widehat {\alpha} _ {n} - \alpha_ {n} \right\| _ {2} \quad \forall n \in [ N ], \tag {9}
$$
where  $\alpha_{n} = [\alpha_{n1},\dots ,\alpha_{np_{n}}]$
3. Out-of-sample forecasting error. Finally, we provide an upper bound for the forecasting error of  $y_{1}(t), \ldots, y_{N}(t)$  for  $t \in \{T + 1, \ldots, 2T\}$ . Specifically, we upper bound the following metric
$$
\operatorname {F o r E r r} (N, T) = \frac {1}{N T} \sum_ {n = 1} ^ {N} \sum_ {t = T + 1} ^ {2 T} \left(\widehat {y} _ {n} (t) - \mathbb {E} \left[ y _ {n} (t) \mid y _ {n} (t - 1), \dots , y _ {n} (1) \right]\right) ^ {2}, \tag {10}
$$
where  $\widehat{y}_n(\cdot) \forall n \in [N], t \in \{T + 1, \dots, 2T\}$  are the forecasts produced by the algorithm we propose in Section 3. Before stating the main results, we state key additional assumptions.
Assumption 4.1 (Balanced spectra). Let  $\mathbf{Z}_f(t_0)$  denote the  $L\times NT / L$  stacked Page matrix associated with all  $N$  time series  $f_{1}(\cdot),\ldots ,f_{N}(\cdot)$  for their  $T$  consecutive entries starting from  $t_0$ . Let  $k = \mathrm{rank}(\mathbf{Z}_f(t_0))$ , and let  $\sigma_k(\mathbf{Z}_f(t_0))$  denote the  $k$ -th singular value for  $\mathbf{Z}_f(t_0)$ . Then, for any  $t_0 > 0$ ,  $\mathbf{Z}_f(t_0)$  is such that  $\sigma_k(\mathbf{Z}_f(t_0))\geq \gamma \sqrt{NT} /\sqrt{k}$  for some absolute constant  $\gamma >0$ .
This assumption holds whenever the non-zero singular values are "well-balanced", a standard assumption in the matrix/tensor estimation literature [3, 6]. Note that this assumption, as stated, ensures balanced spectra for the stacked Page matrix of any set of  $T$  consecutive entries of  $f_{1}(\cdot),\ldots ,f_{N}(\cdot)$ .
Finally, we will impose an additional necessary restriction on the complexity of the  $N$  time series  $f_{1}(t),\ldots ,f_{N}(t)$  for  $t > T$  (as is done in [3, 5]). Let  $\mathbf{Z}_f^\prime$  denote the  $(L - 1)\times (NT / L)$  matrix formed using the top  $L - 1$  rows of  $\mathbf{Z}_f$ . Further, for any  $t_0\in [T + 1]$ , let  $\mathbf{Z}_f^{\prime}(t_0)$  denote the  $(L - 1)\times (NT / L)$  matrix formed using the top  $L - 1$  rows of  $\mathbf{Z}_f(t_0)$ . For any matrix  $M$ , let  $\operatorname {colspan}(M)$  denote the subspace spanned by the its columns. We assume the following property.
Assumption 4.2 (Subspace inclusion). For any  $t_0 \in [T + 1]$ ,  $\operatorname{colspan}(\mathbf{Z}_f'(t_0)) \subseteq \operatorname{colspan}(\mathbf{Z}_f')$ .
This assumption is necessary as it requires the stacked Page matrix of the out-of-sample time series  $\mathbf{Z}_f'(t_0)$  to be only as "rich" as that of the stacked Page matrix of the "in-sample" time series  $\mathbf{Z}_f'$ .
First, recall that  $R$  is defined in Assumption 2.1,  $G$  in Assumption 2.2,  $k$  is in Proposition 2.1, while  $\gamma$  is defined in Assumption 4.1. Further, recall that  $\lambda_{ni}$  for  $i \in [p_n]$  are the roots of the characteristic polynomial of the  $n$ -th AR process, as defined in Assumption 2.3. Throughout, let  $c$  and  $C$  be absolute constants,  $f_{\max} \coloneqq \max_{n,t \leq 2T} |f_n(t)|$ ,  $p \coloneqq \max_{n} p_n$ ,  $\alpha_{\max} = \max_{n} \| \alpha_n \|_2$ , and  $C(f_{\max}, \gamma)$  denote a constant that depends only (polynomially) on model parameters  $f_{\max}$  and  $\gamma$ . Last but not least, define the key quantity
$$
\sigma_ {x} := \frac {c _ {\lambda} \sigma}{(1 - \lambda_ {\star})}
$$
where  $\lambda_{\star} = \max_{i,n}|\lambda_{ni}|$  and  $c_{\lambda}$  is a constant that depends only on  $\lambda_{ni}^9$ . Note that  $\sigma_x^2$  is a key quantity that we use to bound important spectral properties of AR processes, as Lemma A.2 establishes.
4.1.1 Estimation Error of Non-Stationary Component
Theorem 4.1 (Estimation error of  $f$ ). Assume access to the observations  $y_{1}(t),\dots ,y_{N}(t)$  for  $t\in [T]$  as defined in (1). Let assumptions 2.1, 2.2, 2.3, 2.4 and 4.1 hold. Let  $L = \sqrt{NT}$  and  $\hat{k} = k$ , then, for any  $n\in [N]$ , with probability of at least  $1 - \frac{c}{(NT)^{10}}$
$$
\operatorname {E s t E r r} (N, T, n) \leq C \left(f _ {\max }, \gamma\right) \left(\frac {\sigma_ {x} ^ {4} G R \log (N T)}{\sqrt {N T}}\right). \tag {11}
$$
See Appendix A for an explicit expression of  $c_{\lambda}$
This theorem implies that the mean squared error of estimating  $f$  scales as  $\tilde{O}\left(\frac{1}{\sqrt{NT}}\right)^{10}$  with high probability. The proof of Theorem 4.1 is in Appendix C.
4.1.2 Identification of AR Processes
Theorem 4.2 (AR Identification). Let the conditions of Theorem 4.1 hold. Let  $\widehat{\alpha}_n$  be as defined in (7), and  $\alpha_{n} = [\alpha_{n1},\dots ,\alpha_{np_{n}}]$  as defined in (2). Then, for a sufficiently large  $T$  such that  $\frac{\log(T)}{\sqrt{T}} < \frac{C\sigma^2}{p\sigma_x^2\alpha_{\max}^2}\left(\frac{\lambda_{\min}(\Psi)}{\lambda_{\max}(\Gamma)}\right)$ , and for any  $n\in [N]$  where  $\operatorname{EstErr}(N,T,n)\leq \frac{\sigma^2\lambda_{\min}(\Psi)}{6p}$ , we have with probability of at least  $1 - \frac{c}{T^{10}}$ ,
$$
\left\| \widehat {\alpha} _ {n} - \alpha_ {n} \right\| _ {2} ^ {2} \leq \frac {C p}{\lambda_ {\operatorname* {m i n}} (\Psi)} \left(\frac {p}{T} \log \left(\frac {T \lambda_ {\operatorname* {m a x}} (\Psi)}{\lambda_ {\operatorname* {m i n}} (\Psi)}\right) + \frac {\sigma_ {x} ^ {2} \operatorname {E s t E r r} (N , T , n) \log (T)}{\sigma^ {2} \operatorname* {m i n} \{1 , \sigma^ {2} \lambda_ {\operatorname* {m i n}} (\Psi) \}}\right). \tag {12}
$$
Recall that  $\widehat{\alpha}_n$  is estimated using  $\widehat{x}_n(\cdot)$ , a perturbed version of the true AR process  $x_n(\cdot)$ . This perturbation comes from the estimation error of  $x_n(\cdot)$ , which is a consequence of  $\operatorname{EstErr}(N, T, n)$ . Hence, it is not surprising that  $\operatorname{EstErr}(N, T, n)$  shows up in the upper bound. Indeed, the upper bound we provide here consists of two terms: the first, which scales as  $\tilde{O}\left(\frac{1}{T}\right)$  is the bound one would get with access to the true AR process  $x_n(\cdot)$ , as shown in [17]. The second term characterizes the contribution of the perturbation caused by the estimation error, and it scales as  $O\left(\operatorname{EstErr}(N, T, n)\right)$  which we show is  $\tilde{O}\left(\frac{1}{\sqrt{NT}}\right)$  with high probability in Theorem 4.1. Also, note that the theorem holds when the estimation error is sufficiently small ( $\operatorname{EstErr}(N, T, n) \leq \frac{\sigma^2 \lambda_{\min}(\Psi)}{6p}$ ). This condition ensures that the perturbation arising from the estimation error remains below the lower bound for the minimum eigenvalue of the sample covariance matrix associated with the autoregressive processes. Note that  $\lambda_{\max}(\Psi), \lambda_{\min}(\Psi)$  and  $\lambda_{\max}(\Gamma)$  are quantities that relate to the parameters of the  $\operatorname{AR}(p)$  processes and their "controllability Gramian" as we detail in Appendix A.1. The proof of Theorem 4.2 is in Appendix D.
4.1.3 Out-of-sample Forecasting Error
We first establish an upper bound on the error of our estimate  $\widehat{\beta}$ .
Theorem 4.3 (Model Identification  $(\beta^{*})$ ). Let the conditions of Theorem 4.1 hold. Let  $\widehat{\beta}$  be defined as in (6), and  $\beta^{*}$  as defined in Proposition 2.2. Then, with probability of at least  $1 - \frac{c}{(NT)^{10}}$
$$
\left\| \widehat {\beta} - \beta^ {*} \right\| _ {2} ^ {2} \leq C \left(f _ {\max }, \gamma\right) \left(\frac {\sigma_ {x} ^ {4} G ^ {2} R ^ {2} \log (N T)}{\sqrt {N T}}\right) \max  \left\{\left\| \beta^ {*} \right\| _ {1} ^ {2}, 1 \right\}. \tag {13}
$$
This shows that the error of estimating  $\beta^{*}$  (in squared euclidean norm) scales as  $\tilde{O}\left(\frac{1}{\sqrt{NT}}\right)$ .
Theorem 4.4. Let the conditions of Theorem 4.2 and Assumption 4.2 hold. Then, with probability of at least  $1 - \frac{c}{T^{10}}$
$$
\mathsf {F o r E r r} (N, T) \leq \tilde {C} G ^ {3} R ^ {3} p ^ {2} \sigma_ {x} ^ {6} \bigg (\frac {p \sigma^ {2} \log {(T)}}{T} + \frac {G R \sigma_ {x} ^ {6}}{\min \left\{\sigma^ {2} , \sigma^ {4} \right\}} \frac {\log {(N T) ^ {2}}}{\sqrt {N T}} \bigg),
$$
where  $c$  is an absolute constant, and  $\tilde{C}$  denotes a constant that depends only (polynomially) on  $f_{\max}, \gamma, \lambda_{\min}(\Psi), \lambda_{\max}(\Psi), \beta^*$  and  $\alpha_{\max}$ .
This theorem establishes that the forecasting error for the next  $T$  time steps scales as  $\tilde{O}\left(\frac{1}{T} +\frac{1}{\sqrt{NT}}\right)$  with high probability. Note that the  $\tilde{O}\left(\frac{1}{T}\right)$  term is a function of  $T$  only, as it is a consequence of the error incurred when we estimate each  $\alpha_{n}$ . Recall that we estimate  $\alpha_{n}$  separately for  $n\in [N]$  using the available (perturbed)  $T$  observation of each process. The  $\tilde{O}\left(\frac{1}{\sqrt{NT}}\right)$  term on the other hand is a consequence of the error incurred when learning and forecasting  $f_{1},\ldots ,f_{n}$ , which is done collectively across the  $N$  time series. Finally, Theorem 4.4 implies that when  $N = \Theta (T)$ , the forecasting error scales as  $\tilde{O}\left(\frac{1}{T}\right)$ . The proof of Theorems 4.3 and 4.4 are in Appendix F and G, respectively.
10The  $\tilde{O} (\cdot)$  notation is analogous to the standard  $O(\cdot)$  while ignoring log dependencies.
Table 1: Performance of algorithms on various datasets, measured by mean  $R^2$ .
Traffic	Electricity	Exchange	Synthetic
SAMoSSA	0.776	0.829	0.731	0.476
mSSA	0.747	0.605	0.674	0.366
ARIMA	0.723	<-10	0.756	0.305
Prophet	0.462	0.197	<-10	-0.445
DeepAR	0.824	0.764	0.579	0.323
LSTM	0.821	-1.261	-1.825	0.381
In this section, we support our theoretical results through several experiments using synthetic and real-world data. In particular, we draw the following conclusions:
1. Our results align with numerical simulations concerning the estimation of non-stationary components under AR stationary noise and the accuracy of AR parameter estimation (Section 5.1).
2. Modeling and learning the autoregressive process in SAMoSSA led to a consistent improvement over mSSA. The improvements range from  $5\%$  to  $37\%$  across standard datasets (Section 5.2).
5.1 Model Estimation
Setup. We generate a synthetic multivariate time series  $(N = 10)$  that is a mixture of both harmonics and an AR noise process. The AR process is stationary, and its parameter is chosen such that  $\lambda^{*} = \max_{n,i}|\lambda_{ni}|$  is one of three values  $\{0.3,0.6,0.95\}$ . Refer to Appendix B.1 for more details about the generating process. We then evaluate the estimation error EstErr  $(N,T,1)$  and the AR parameter estimation error  $\| \alpha_{1} - \widehat{\alpha}_{1}\|_{2}$  as we increase  $T$  from 200 to 500000.
Results. Figure 2a visualizes the mean squared estimation error of  $f_{1}$ , while Figure 2b shows the AR parameter estimation error. The solid lines in both figures indicate the mean across ten trials, whereas the shaded areas cover the minimum and maximum error across the ten trials. We find that, as the theory suggests, the estimation error for both the non-stationary component and the AR parameter decay to zero as  $NT$  increases. We also see that the estimation error is inversely proportional to  $(1 - \lambda^{*})$ , which is being reflected in the AR parameter estimation error as well.
(b)
Figure 2: The error in SAMoSSA's estimation of the non-stationary components and the AR parameters decays to zero as  $NT$  increases as the theorem suggests.
5.2 Forecasting
We showcase SAMoSSA's forecasting performance relative to standard algorithms. Notably, we compare SAMoSSA to the mSSA variant in [3] to highlight the value of learning the autoregressive process.
Setup. The forecasting ability of SAMoSSA was compared against (i) mSSA [3], (ii) ARIMA, a very popular classical time series prediction algorithm, (iii) Prophet [28], (iv) DeepAR [23] and (v) LSTM [14] on three real-life datasets and one synthetic dataset containing both deterministic trend/seasonality and a stationary noise process (see details in Appendix B). Each dataset was split into train, validation, and test sets (see Appendix B.1). Each dataset has multiple time series, so we
aggregate the performance of each algorithm using the mean  $R^2$  score. We use the  $R^2$  score since it is invariant to scaling and gives a reasonable baseline: a negative value indicates performance inferior to simply predicting the mean.
Results. In Table 1 we report the mean  $R^2$  for each method on each dataset. We highlight that, across all datasets, SAMoSSA consistently performs the best or is otherwise very competitive with the best. The results also underscore the significance of modeling and learning the autoregressive process in real-world datasets. Notably, learning the autoregressive model in SAMoSSA consistently led to an improvement over mSSA, with the increase in  $R^2$  values ranging from  $5\%$  to  $37\%$ . We note that this improvement is due to the fact that mSSA, as described in [3], overlooks any potential structure in the stochastic processes  $x_{1}(\cdot), \ldots, x_{N}(\cdot)$  and assumes i.i.d.mean-zero noise process. While in SAMoSSA, we attempt to capture the structure of  $x_{1}(\cdot), \ldots, x_{N}(\cdot)$  through the learned AR process.
We presented SAMoSSA, a two-stage procedure that effectively handles mixtures of deterministic nonstationary and stationary AR processes with minimal model assumptions. We analyze SAMoSSA's ability to estimate non-stationary components under stationary AR noise, the error rate of AR system identification via OLS under observation errors, and a finite-sample forecast error analysis.
We note that our results can be readily adapted to accommodate (i) approximate low-rank settings (as in the model by Agarwal et al [3]); and (ii) scenarios with incomplete data. We do not discuss these settings to focus on our core contributions, but they represent valuable directions for future studies.
Our analysis reveals some limitations, providing avenues for future research. One limitation of our model is that it only considers stationary stochastic processes. Consequently, processes with non-stationary stochastic trends, such as a random walk, are not incorporated. Investigating the inclusion of such models and their interplay with the SSA literature is a worthy direction for future work. Second, our model assumes non-interaction between the  $N$  stationary processes  $x_{1},\ldots ,x_{N}$ . Yet, it might be plausible to posit the existence of interactions among them, possibly through a vector AR model (VAR). Examining this setting represents another compelling direction for future work.
[1] Y. Abbasi-yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
[2] A. Agarwal, A. Alomar, and D. Shah. tspdb: Time series predict db. In NeurIPS 2020 Competition and Demonstration Track, pages 27-56. PMLR, 2021.
[3] A. Agarwal, A. Alomar, and D. Shah. On multivariate singular spectrum analysis and its variants. ACM SIGMETRICS Performance Evaluation Review, 50(1):79-80, 2022.
[4] A. Agarwal, M. J. Amjad, D. Shah, and D. Shen. Model agnostic time series analysis via matrix estimation. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2(3):1-39, 2018.
[5] A. Agarwal, D. Shah, and D. Shen. On principal component regression in a high-dimensional error-in-variables setting. arXiv preprint arXiv:2010.14449, 2020.
[6] A. Alanqary, A. Alomar, and D. Shah. Change point detection via multivariate singular spectrum analysis. Advances in Neural Information Processing Systems, 34:23218-23230, 2021.
[7] A. Alexandrov, K. Benidis, M. Bohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D. C. Maddix, S. Rangapuram, D. Salinas, J. Schulz, et al. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv:1906.05264, 2019.
[8] T. W. Anderson. The statistical analysis of time series. John Wiley & Sons, 2011.
[9] F. Chollet. keras. https://github.com/fchollet/keras, 2015. Online; accessed 25 February 2020.
[32] P.-Å. Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics, 12(1):99-111, 1972.
[33] P.-Å. Wedin. Perturbation theory for pseudo-inverses. BIT Numerical Mathematics, 13:217-232, 1973.
A Concentration inequalities for AR processes
In this section, we present two important lemmas that are central to the main results. Before stating the two lemmas, we start with key important quantities of AR Processes.
A.1 Controllability Gramian of AR Processes
In this section, we explain the notation of key quantities used in the main results. First, note that the process in (2) can be written in matrix-vector form by defining a transition matrix and input vector
$$
\boldsymbol {A} _ {n} = \left[ \begin{array}{c c} \alpha_ {n 1: n (p _ {n} - 1)} ^ {\top} & \alpha_ {n p _ {n}} \\ I _ {p _ {n} - 1} & 0 _ {p _ {n} - 1} \end{array} \right], \qquad B _ {n} = \left[ \begin{array}{c} 1 \\ 0 _ {p _ {n} - 1} \end{array} \right],
$$
where  $0_{p_n - 1}$  is the column vector of  $p_n - 1$  zeroes. Then, with  $X_{n}(t)\coloneqq [x_{n}(t),\ldots ,x_{n}(t - p_{n} + 1)]$
$$
X _ {n} (t) = \boldsymbol {A} _ {n} X _ {n} (t - 1) + B _ {n} \eta_ {n t}.
$$
With this setup, we define two important matrices,
$$
\Gamma_ {n} = \sum_ {t = 0} ^ {\infty} \boldsymbol {A} _ {n} ^ {t} \left(\boldsymbol {A} _ {n} ^ {\top}\right) ^ {t}, \quad \Psi_ {n} = \sum_ {t = 0} ^ {\infty} \boldsymbol {A} _ {n} ^ {t} B _ {n} B _ {n} ^ {\top} \left(\boldsymbol {A} _ {n} ^ {\top}\right) ^ {t}, \tag {14}
$$
where  $\Psi_{n}$  is known as the controllability Gramian of the  $n$ -th  $\mathrm{AR}(p_n)$  process  $x_{n}$ . For a positive semi-definite matrix  $M$ , let  $\lambda_{\max}(M) \geq 0$  and  $\lambda_{\min}(M) \geq 0$  be its maximum and minimum eigenvalues, respectively. We define  $\lambda_{\max}(\Psi) \coloneqq \max_{n} \lambda_{\max}(\Psi_{n})$  and  $\lambda_{\min}(\Psi) \coloneqq \min_{n} \lambda_{\min}(\Psi_{n})$ . We also define  $\lambda_{\max}(\Gamma)$  and  $\lambda_{\min}(\Gamma)$  analogously for  $\Gamma$ .
A.2 Key Lemmas
Now, we present two important lemmas that are central to the main results. The first lemma states that a vector of consecutive entries of a stationary autoregressive process is a sub-gaussian vector with variance proxy  $\sigma_x^2$ .
Lemma A.1. Let assumptions 2.3 and 2.4 hold, then, for any  $n \in [N]$ ,  $t \geq 1$ ,  $K \geq 1$ , the vector  $[x_{n}(t), \ldots, x_{n}(t + K)]$  is a sub-gaussian vector with variance proxy  $\sigma_{x}^{2}$ .
This lemma, along with the well known bound on the norm of sub-gaussian vectors, is key to the bounds presented in Section 4.1. Further, this Lemma, along with an  $\varepsilon$ -net argument, allows us to bound the operator norm of the stacked Page matrix of the AR processes  $x_{1}(t),\ldots ,x_{N}(t)$ . Specifically, we establish next that its operator norm grows as  $O(\sqrt{NT / L})$  with high probability.
Lemma A.2. For any  $T > 1$ ,  $1 \leq L \leq \sqrt{NT}$ , and  $t_0 > 0$ , let
$$
\mathbf {Z} _ {x} (t _ {0}) = \left[ \begin{array}{c c c c} \mathbf {Z} (x _ {1}, L, T, t _ {0}) & \mathbf {Z} (x _ {2}, L, T, t _ {0}) & \dots & \mathbf {Z} (x _ {n}, L, T, t _ {0}) \end{array} \right]
$$
be the  $L \times NT / L$  stacked Page matrix of the AR processes  $x_{1}(t),\ldots ,x_{N}(t)$ , as defined in (2). Let Assumptions 2.3 and 2.4 hold. Then, each row  $[\mathbf{Z}_x(t_0)]_i$  is a sub-gaussian vector with variance proxy  $\sigma_x^2$ . Further, with probability  $1 - \exp (-cNT / L)$
$$
\left\| \mathbf {Z} _ {x} (t _ {0}) \right\| _ {2} \leq 2 c \sigma_ {x} \sqrt {N T / L}.
$$
Next, we prove Lemma A.1 and Lemma A.2. Before stating the proofs, we start with the important helper Lemma A.3.
Lemma A.3 (MA(∞) representation of an AR(p) process). Let  $x(t)$  be a stationary AR(p) process defined as
$$
x (t) = \sum_ {i = 1} ^ {p} \alpha_ {i} x (t - i) + \eta (t).
$$
Assume that the roots  $(\lambda_1, \dots, \lambda_p)$  of its characteristic polynomial  $g(z) \coloneqq z^p - \sum_{i=1}^{p} \alpha_i z^{p-i}$  are distinct. Then,  $x(t)$  has the  $MA(\infty)$  representation
$$
x (t) = \sum_ {k = 0} ^ {\infty} \beta_ {k} \eta (t - k),
$$
with  $\beta_{k} := \sum_{i=1}^{p} a_{i} \lambda_{i}^{k}$ ,  $a_{i} := \prod_{1 \leq j \leq p, j \neq i} \left(1 - \frac{\lambda_{j}}{\lambda_{i}}\right)^{-1}$ .
Proof. First, note that we can write  $x(t)$  in lag operator form as
$$
f (L) x (t) \triangleq \left(1 - \sum_ {i = 1} ^ {p} \alpha_ {i} L ^ {i}\right) x (t) = \eta (t). \tag {15}
$$
Here,  $L$  is a lag operator such that  $Lx(t) = x(t - 1)$ . Note that since we have assumed the AR process to be stationary, all the roots of  $g(z)$  lie inside the unit circle [26]. Using the roots of  $g(z)$ , (15) can be written as,
$$
\prod_ {i = 1} ^ {p} (1 - \lambda_ {i} L) x (t) = \eta (t).
$$
Further, using the partial fraction decomposition we can write the process as
$$
\begin{array}{l} x _ {t} = \frac {1}{\prod_ {i = 1} ^ {p} (1 - \lambda_ {i} L)} \eta (t) \\ = \sum_ {i = 1} ^ {p} \frac {a _ {i}}{1 - \lambda_ {i} L} \eta (t) \\ \end{array}
$$
$$
a _ {i} = \prod_ {1 \leq j \leq p, j \neq i} \left(1 - \frac {\lambda_ {j}}{\lambda_ {i}}\right) ^ {- 1}
$$
Recall that we can expand each fraction as
$$
\frac {a _ {i}}{1 - \lambda_ {i} L} = a _ {i} \left(\sum_ {k = 0} ^ {\infty} \lambda_ {i} ^ {k} L ^ {k}\right).
$$
Thus, finally, we can write  $x(t)$  as,
with  $\beta_{k}\coloneqq \sum_{i = 1}^{p}a_{i}\lambda_{i}^{k}$