SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. First, let  $X_{n}(t,K) = [x_{n}(t),\ldots ,x_{n}(T + K)]$  be the vector of interest. Let  $E_{n}(t,K) = [\eta_{n}(t),\dots ,\eta_{n}(T + K)]$ , then using Lemma A.3, we can write  $X_{n}(t,K)$  as
$$
X _ {n} (t, K) = \sum_ {k = 0} ^ {\infty} \beta_ {n k} E _ {n} (t - k, K)
$$
Note that  $\beta_{nk}E_n(t - k,K)$  is clearly a sub-gaussian vector with variance proxy  $\beta_{nk}^2\sigma^2$ . Further note that the sum of two sub-gaussian random vectors, not necessarily independent, is also a sub-gaussian random vector. To see this, suppose  $\mathbf{x}$  and  $\mathbf{y}$  are both sub-gaussian vectors with variance proxies  $\sigma_x^2$  and  $\sigma_y^2$ . Then for any  $\mathbf{v}\in S^K$ ,
$$
\begin{array}{l} \mathbf {v} ^ {T} (\mathbf {x} + \mathbf {y}) = \mathbf {v} ^ {T} \mathbf {x} + \mathbf {v} ^ {T} \mathbf {y} \\ \sim \operatorname {s u b G} \left(\left(\sigma_ {x} + \sigma_ {y}\right) ^ {2}\right). \\ \end{array}
$$
Thus,  $X_{n}(t,K)$ , for any  $n$ , is also a sub-gaussian random vector with variance proxy
$$
\left(\sum_ {k = 0} ^ {\infty} \left| \beta_ {n k} \right| \sigma\right) ^ {2} \leq \sigma^ {2} \left(\sum_ {k = 0} ^ {\infty} \max  _ {n} \left| \beta_ {n k} \right|\right) ^ {2}.
$$
Now we are interested in bounding  $\sum_{k=0}^{\infty} \max_n |\beta_{nk}|$ . To do so, first recall that  $\beta_{nk} := \sum_{i=1}^{p_n} a_{ni} \lambda_{ni}^k$  where  $\lambda_{ni} \in \mathbb{C}$ ,  $i \in [p_n]$  denote the roots of  $g_n(z)$  and  $a_i = \prod_{1 \leq j \leq p_n, j \neq i} \left(1 - \frac{\lambda_{nj}}{\lambda_{ni}}\right)^{-1}$ . Let  $\lambda_\star = \max_{i,n} |\lambda_{ni}|$  and  $c_{\lambda_n} = \sum_{i=1}^{p_n} \left| \prod_{1 \leq j \leq p, j \neq i} \left(1 - \frac{\lambda_{nj}}{\lambda_{ni}}\right)^{-1} \right|$ . Then, we can bound  $\beta_{nk}$  as
$$
\begin{array}{l} | \beta_ {n k} | \leq \lambda_ {\star} ^ {k} \sum_ {i = 1} ^ {p _ {n}} | a _ {i} | \\ \leq c _ {\lambda_ {n}} \lambda_ {\star} ^ {k}. \\ \end{array}
$$
Thus, with  $c_{\lambda} = \max_{n} c_{\lambda_n}$
$$
\sum_ {k = 0} ^ {\infty} \max  _ {n} | \beta_ {n k} | \leq \frac {c _ {\lambda}}{1 - \lambda_ {\star}}. \tag {16}
$$
Hence,  $X_{n}(t,K)$  is a sub-gaussian random vector with variance proxy  $\sigma_x^2\coloneqq \frac{c_\lambda^2\sigma^2}{(1 - \lambda_\star)^2}$ .
A.4 Proof of Lemma A.2
Proof. Let  $q = NT / L$ . We will find an upper bound on the operator norm by following two steps: (i) we will represent each  $x_{n}(t)\forall n\in [N]$  as an  $\mathrm{MA}(\infty)$  process; then, (ii) we will bound the operator norm of the stacked Page matrix of this  $\mathrm{MA}(\infty)$  processes.
Step 1:  $\mathbf{MA}(\infty)$  representation for  $\operatorname {AR}(p)$
As a direct application of Lemma A.3, we can write  $x_{n}(t)$  as,
$$
x _ {n} (t) = \sum_ {k = 0} ^ {\infty} \beta_ {n k} \eta_ {n} (t - k),
$$
with  $\beta_{nk} := \sum_{i=1}^{p} a_{ni} \lambda_{ni}^{k}$ ,  $a_{ni} := \prod_{1 \leq j \leq p, j \neq i} \left(1 - \frac{\lambda_{nj}}{\lambda_{ni}}\right)^{-1}$
Step 2: Bound the Page matrix for the  $\mathbf{MA}(\infty)$  Page matrix.
First, let's consider, without loss of generality, the case when  $t_0 = 1$ . That is, let's consider  $\mathbf{Z}_x \coloneqq \mathbf{Z}_x(1)$ . To get the desired bound, we will first obtain a high probability bound on the Euclidean norm of  $\mathbf{Z}_x^\top \mathbf{v}$  for any vector  $\mathbf{v} \in S^{L-1} \coloneqq \{\mathbf{v} \in \mathbb{R}^L : \| \mathbf{v} \|_2 = 1\}$ . Then, we will bound the operator norm using this and an  $\varepsilon$ -net argument.
Note that by step (1), we have
$$
x _ {n} (t) = \sum_ {k = 0} ^ {\infty} \beta_ {n k} \eta_ {n} (t - k).
$$
Now define  $\pmb{E}_i$  as the analogous stacked Page matrix for  $\eta_1(t), \dots, \eta_N(t)$  for  $t \in \{i, \dots, i + T - 1\}$ . Observe that  $\pmb{E}_i$  are simply identically distributed (but not independent) matrices of i.i.d. sub-gaussian random variables. Then we can write
$$
\mathbf {Z} _ {x} = \sum_ {k = 0} ^ {\infty} E _ {1 - k} B _ {k}, \tag {17}
$$
where  $B_{k} \in \mathbb{R}^{q \times q}$  is a diagonal matrix with the  $i$ -th diagonal  $[B_k]_{ii} = \beta_{\lceil \frac{iN}{q}\rceil k}$ . Then
$$
\left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} \right\| _ {2} = \left\| \sum_ {k = 0} ^ {\infty} \boldsymbol {B} _ {k} \boldsymbol {E} _ {1 - k} ^ {\top} \mathbf {v} \right\| _ {2}.
$$
It is easy to verify that  $\pmb{E}_i^\top \mathbf{v}$  is a sub-gaussian random vector (see Definition H.3) with variance proxy  $\sigma^2$ , because for any  $\mathbf{u} \in S^{q-1}$  we have (using  $\pmb{E}$  as a shorthand for  $\pmb{E}_i$ )
$$
\begin{array}{l} \mathbf {v} ^ {T} \boldsymbol {E} \mathbf {u} = \sum_ {i = 1} ^ {L} \sum_ {j = 1} ^ {q} v _ {i} u _ {j} E _ {i j} \\ \sim \operatorname {s u b G} \left(\sum_ {i = 1} ^ {L} \sum_ {j = 1} ^ {q} v _ {i} ^ {2} u _ {j} ^ {2} \sigma^ {2}\right) \\ \sim \operatorname {s u b G} \left(\sigma^ {2}\right). \\ \end{array}
$$
Using the above,  $\pmb{B}_k\pmb{E}_i^\top \mathbf{v}$  is  $\sim \mathrm{subG}(\max_n|\beta_{nk}|^2\sigma^2)$ . Since  $\mathbf{Z}_x\mathbf{v}$  is a sum of sub-gaussian random vectors, it is also a sub-gaussian random vector with variance proxy  $\sigma_x^2$  as we showed in the proof of Lemma A.1.
Using Lemma H.2, we get, with probability at least  $1 - \delta$
$$
\left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} \right\| _ {2} \leq 4 \sigma_ {x} \sqrt {q} + 2 \sigma_ {x} \sqrt {\log \left(\frac {1}{\delta}\right)}
$$
Equivalently, for  $t \geq 0$ , it holds that
$$
\mathbb {P} \left(\| \mathbf {Z} _ {x} \mathbf {v} \| _ {2} - 4 \sigma_ {x} \sqrt {q} > t\right) \leq \exp \left(- \frac {t ^ {2}}{4 \sigma_ {x} ^ {2}}\right).
$$
Now we have shown that for any fixed  $\mathbf{v} \in S^{L-1}$ , the quantity  $\| \mathbf{Z}_x \mathbf{v} \|_2$  grows as  $O\left(\sqrt{q}\right)$  with high probability. What remains is to apply the union bound over an  $\varepsilon$ -net to extend this to the maximum over  $S^{L-1}$ , i.e. the operator norm of  $\mathbf{Z}_x$ .
Let  $\Sigma$  be a maximal  $1/2$ -net of  $S^{L-1}$ , i.e. a set of points in  $S^{L-1}$  spaced at least  $1/2$  from each other such that no other point can be added without violating this property. Now consider  $\mathbf{v}_{\star}$  such that  $\left\| \mathbf{Z}_x^\top \mathbf{v}_{\star} \right\|_2 = \left\| \mathbf{Z}_x \right\|_2$ . Since  $\Sigma$  is maximal, there is some  $\mathbf{v} \in \Sigma$  within  $1/2$  of  $\mathbf{v}_{\star}$ . Since
$$
\left\| \mathbf {Z} _ {x} ^ {\top} (\mathbf {v} - \mathbf {v} _ {\star}) \right\| _ {2} \leq \| \mathbf {Z} _ {x} \| _ {2} \| \mathbf {v} - \mathbf {v} _ {\star} \| _ {2} \leq \| \mathbf {Z} _ {x} \| _ {2} / 2,
$$
$$
\left\| \mathbf {Z} _ {x} ^ {\top} (\mathbf {v} - \mathbf {v} _ {\star}) \right\| _ {2} \geq \left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} _ {\star} \right\| _ {2} - \left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} \right\| _ {2} = \| \mathbf {Z} _ {x} \| _ {2} - \left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} \right\| _ {2},
$$
$$
\left\| \mathbf {Z} _ {x} ^ {\top} \mathbf {v} \right\| _ {2} \geq \left\| \mathbf {Z} _ {x} \right\| _ {2} / 2,
$$
i.e. taking the maximum over  $\Sigma$  is equivalent to doing so over  $S^{L - 1}$  up to a factor of two. Using the fact that  $|\Sigma |\leq C^L$  for some absolute constant  $C$ , the fact that  $L < q$ , and using a sufficiently large absolute constant  $c > 0$  we have
$$
\begin{array}{l} \mathbb {P} \left(\| \mathbf {Z} _ {x} \| _ {2} > 2 c \sigma_ {x} \sqrt {q}\right) \leq \sum_ {\mathbf {v} \in \Sigma} \mathbb {P} \left(\| \mathbf {Z} _ {x} \mathbf {v} \| _ {2} > c \sigma_ {x} \sqrt {q}\right) \\ \leq 2 C ^ {L} \exp (- c ^ {\prime} q) \\ = \exp (- c ^ {\prime \prime} q). \\ \end{array}
$$
where  $c'' > 0$  and  $c' > 0$  are absolute constants. Recalling that  $q = NT / L$  concludes the proof for the operator norm.
Finally, we show that each row  $[\mathbf{Z}_x(t_0)]_i$ : in the stacked page matrix is a sub-gaussian vector with variance proxy  $\sigma_x^2$ . Consider, without loss of generality, the first row  $[\mathbf{Z}_x]_1$ . Using (17), we have for any  $\mathbf{v} \in S^{L-1}$
$$
\mathbf {v} ^ {\top} \left[ \mathbf {Z} _ {x} \right] _ {1.} = \sum_ {k = 0} ^ {\infty} \mathbf {v} ^ {\top} \left[ \boldsymbol {E} _ {1 - k} \boldsymbol {B} _ {k} \right] _ {1..},
$$
note that  $\mathbf{v}^{\top}[\pmb{E}_{1 - k}\pmb{B}_k]_1$ . is  $\sim$  subG  $(\max_{n}\beta_{nk}^{2}\sigma^{2})$ , and hence, using the same argument for the sum of dependent sub-gaussian random variables used in the proof above, we have  $\mathbf{v}^{\top}[\mathbf{Z}_x]_1$ .  $\sim$  subG  $(\sigma_x^2)$ .
In Appendix B.1, we detail the datasets used. In Appendix B.2, we explain the implementations and hyperparameter choices for our method and benchmark algorithms. In Appendix B.3, we describe our hyperparameter tuning strategy.
In the model estimation experiments, we generate a synthetic multivariate time series  $(N = 10)$  that is a mixture of both harmonics and an AR(2) process. The harmonics are generated as follow: we first generate  $R = 3$  fundamental time series of the form  $g_{k}(t) = \sin (\omega_{k}t + \phi_{k})$  for  $k\in [R]$ , where  $\omega_{k}$  and  $\phi_{k}$  are uniformly randomly sampled on the ranges  $[2\pi /100,2\pi /50],[0,2\pi ]$  respectively. We then sample the  $N = 10$  series using  $f_{k}(t) = \sum_{i = 1}^{R}c_{ki}g_{k}(t)$  for  $k\in [N]$ , where the mixture coefficients  $c_{ki}$  are independent standard normal random variables. The AR process is stationary, and its parameter is chosen such that  $\lambda^{*} = \max_{n,i}|\lambda_{ni}|$  is set to one of  $\{0.3,0.6,0.95\}$ . The noise of the process has a variance  $\sigma^2 = 0.2$ .
In the forecasting experiments, we evaluate our method and benchmark algorithms on three real-world datasets and one synthetic dataset. The preprocessing steps and setup of each dataset are described below.
Traffic Dataset. This public dataset obtained from the UCI repository shows the occupancy rate of traffic lanes in San Francisco. The data is sampled every 15 minutes but to be consistent with previous work, we aggregate the data into hourly data and use the first 10248 time-points for training, the next 48 points for validation, and another 48 points for testing in the forecasting experiments. Specifically, in our testing period, we do 1-hour ahead forecasts for the next 48 hours.
Electricity Dataset. This is a public dataset obtained from the UCI repository which shows the 15-minutes electricity load of 370 households. We aggregate the data into hourly intervals and use the first 25824 time-points for training, the next 48 points for validation, and another 48 points for testing in the forecasting experiments. Specifically, in our testing period, we do 1-hour ahead forecasts for the next 48 hours.
Exchange Dataset. This is a dataset containing the daily exchange rates of eight foreign currencies, including those of Australia, the UK, Canada, Switzerland, China, Japan, New Zealand and Singapore, between 1990 and 2016. We standardize each time series to have zero mean and unit variance since the range of typical exchange rates varies greatly across the currencies. We use the first 7528 timepoints for training, the next 30 for validation, and the final 30 for testing. All forecasts are made 1-day ahead.
Synthetic Dataset. We generate  $R$  fundamental time series of the form  $g_{k}(t) = \sin (\omega_{k}t + \phi_{k}) + m_{k}t$  for  $k\in [R]$ , where  $\omega_{k},\phi_{k}$ , and  $m_{k}$  are uniformly randomly sampled on the ranges  $[2\pi /100,2\pi /10]$ ,  $[0,2\pi ]$ , and  $[-5\times 10^{-4},5\times 10^{-4}]$  respectively. We then sample  $N$  mixtures of the form  $f_{k}(t) = \sum_{i = 1}^{R}c_{ki}g_{k}(t)$  for  $k\in [N]$ , where the mixture coefficients  $c_{ki}$  are independent standard normals. We then inject AR(1) autoregressive noise to each series to produce  $y_{k}(t) = f_{k}(t) + x_{k}(t)$  for  $k\in [N]$ , where  $\{x_{k}(t)\}_{k\in [N]}$  are independent processes, each with a fixed autoregressive coefficient  $\alpha = -0.5$  and noise variance  $\sigma^2 = 1$ . We fix the coefficient to ensure that each time series has significantly autocorrelated noise that can be exploited to improve predictions.
Table 2: Dataset and training/validation/test split details.
Dataset	No. time series	Training period	Validation period	Test period
Traffic	963	1 to 10248	10249 to 10296	10296 to 10344
Electricity	370	1 to 25824	25825 to 25872	25872 to 25919
Exchange	8	1 to 7528	7529 to 7558	7559 to 7588
Synthetic	25	1 to 10000	10001 to 10025	10026 to 10050
B.2 Algorithms
In this section, we discuss the implementations used and hyperparameter tuning details for each algorithm that was evaluated.
SAMoSSA & mSSA. Since mSSA is a special case of SAMoSSA where the second stage – fitting autoregressive models – is skipped, we use one in-house implementation to evaluate both algorithms. The relevant hyperparameters are as follows:
1. The number of retained singular values,  $k$ . This hyperparameter is relevant for both algorithms. We select  $k$  using one of three methods: (i) the data-driven procedure suggested in [13] which computes a threshold based on the shape and median singular value of the page matrix; (ii) picking  $k$  as the minimum number of singular values required to capture 90% of the spectral energy of the page matrix; (iii) picking a fixed low rank, specifically  $k = 5$ .
2. The shape parameter of the stacked Page matrix. This is the ratio between the number of columns and rows of the stacked Page matrix used to carry out the matrix estimation procedure. While our theoretical analysis fixes this value to 1 (recall that we set  $L = \sqrt{NT}$ ), in practice we have found that a slightly wider Page matrix can improve performance. This parameter is chosen from  $\{1,3,5\}$ .
3. The number of autoregressive lag coefficients,  $p$ . This hyperparameter is only relevant for SAMoSSA. While each time series does not necessarily need to use the same value, we enforce this to be the case for computational efficiency. We choose  $p \in \{0, 1, 2, 3\}$ , i.e., the model is allowed to not fit a residual model if it brings no apparent benefit, corresponding to the case of  $p = 0$ .
Prophet. We use Prophet's Python library with the parameters selected using a grid search of the following parameters as suggested in [12]:
1. Changepoint prior scale. This parameter determines how much the trend changes at the detected trend changepoints. We choose this parameter from  $\{0.001, 0.05, 0.2\}$ .
2. Seasonality prior scale. This parameter controls the magnitude of the seasonality. We choose this parameter from  $\{0.01, 10\}$ .
3. Seasonality Mode. We choose between an "additive" and "multiplicative" seasonality term.
The parameters for each time series are chosen independently.
ARIMA. We used the ARIMA implementation of the Python library statsmodels [25]. The hyperparameters are grid searched independently for each time series, and are as follows:
1. Autoregressive order. This is the number of autoregressive lag coefficients fitted, chosen from  $\{1,2,3\}$ .
2. Differencing order. This denotes the number of times the time series is differenced before a model is fitted, and we choose between 0 (no differencing) and 1.
3. Moving average order. This is the number of moving average lag coefficients, chosen from  $\{1,2,3\}$ .
LSTM. We use Keras implementation [9]. We perform a grid search on the number of layers  $\{2,3,4\}$ .
DeepAR. We use the implementation provided by the GluonTS package [7]. We use the default parameters.
B.3 Parameter Selection
We tune hyperparameters for all algorithms evaluated using rolling cross validation. During the validation step, each model is fitted only once on the training set; then, it repeatedly makes one-step-ahead forecasts and is afterwards provided with the realized value, in order to use the updated data to make the next forecast. During test time, the model is fitted on the training and validation sets, and similarly repeatedly makes one-step-ahead forecasts with the fitted parameters. For the models that choose one set of hyperparameters for all  $N$  time series, the mean  $R^2$  score over all time series on the validation set predictions is the metric of choice.
In this section, we provide the proof for Theorem 4.1. While the analysis generally follows the argument for the imputation error in [3], we adapt it for our different assumptions and metric of interest. Metric. First, recall that our metric of interest is
$$
\operatorname {E s t E r r} (N, T, n) = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(\widehat {f} _ {n} (t) - f _ {n} (t)\right) ^ {2}. \tag {18}
$$
Recall that  $\widehat{\mathbf{Z}}_f$  is the stacked Page matrix of  $\widehat{f}_1, \ldots, \widehat{f}_N$ , and  $\mathbf{Z}_f$  is the stacked Page matrix of  $f_1, \ldots, f_N$ . To bound this error, we first establish a deterministic bound through a general lemma for Hard Singular Value Thresholding (HSVT).
C.1 Deterministic Bound
We state the following result, a more general version of which is stated in [3].
Lemma C.1 ([3]). For  $k \geq 1$ , let  $\mathbf{Y} = \mathbf{M} + \mathbf{E} \in \mathbb{R}^{q \times p}$  with  $\text{rank}(\mathbf{M}) = k$ . Let  $\mathbf{U}_k \coloneqq \mathbf{\Sigma}_k \mathbf{V}_k^\top$  and  $\widehat{\mathbf{U}}_k \widehat{\mathbf{\Sigma}}_k \widehat{\mathbf{V}}_k^\top$  denote the top  $k$  singular components of the SVD of  $\mathbf{M}$  and  $\mathbf{Y}$  respectively. Then, the HSVT estimate  $\widehat{\mathbf{M}} = \widehat{\mathbf{U}}_k \widehat{\mathbf{\Sigma}}_k \widehat{\mathbf{V}}_k^\top$  is such that for all  $j \in [q]$ ,
$$
\| \widehat {\boldsymbol {M}} _ {j \cdot} ^ {\top} - \boldsymbol {M} _ {j \cdot} ^ {\top} \| _ {2} ^ {2} \leq 2 \frac {\| \boldsymbol {E} \| _ {2} ^ {2}}{\sigma_ {k} (\boldsymbol {M}) ^ {2}} \left(\left\| \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} + \left\| \boldsymbol {M} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2}\right) + 2 \left\| \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2},
$$
Proof. First, note that
$$
\widehat {\boldsymbol {M}} _ {j \cdot} ^ {\top} - \boldsymbol {M} _ {j \cdot} ^ {\top} = \left(\widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {Y} _ {j \cdot} ^ {\top} - \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {M} _ {j \cdot} ^ {\top}\right) + \left(\widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {M} _ {j \cdot} ^ {\top} - \boldsymbol {M} _ {j \cdot} ^ {\top}\right),
$$
where  $\widehat{M}_{j\cdot}^{\top} = \widehat{\mathbf{V}}_k\widehat{\mathbf{V}}_k^{\top}\mathbf{Y}_{j\cdot}^{\top}$  by definition. Note that the vector  $\left(\widehat{\mathbf{V}}_k\widehat{\mathbf{V}}_k^{\top}M_{j\cdot}^{\top} - M_{j\cdot}^{\top}\right)$  is in the span of a subspace orthogonal to  $\widehat{\mathbf{V}}_k\widehat{\mathbf{V}}_k^{\top}$ , and hence we have by the Pythagorean theorem,
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {M}} _ {j} ^ {\top} - \boldsymbol {M} _ {j} ^ {\top} \right\| _ {2} ^ {2} = \left\| \widehat {\boldsymbol {V}} _ {k} \widehat {\boldsymbol {V}} _ {k} ^ {\top} \boldsymbol {Y} _ {j} ^ {\top} - \widehat {\boldsymbol {V}} _ {k} \widehat {\boldsymbol {V}} _ {k} ^ {\top} \boldsymbol {M} _ {j} ^ {\top} \right\| _ {2} ^ {2} + \left\| \widehat {\boldsymbol {V}} _ {k} \widehat {\boldsymbol {V}} _ {k} ^ {\top} \boldsymbol {M} _ {j} ^ {\top} - \boldsymbol {M} _ {j} ^ {\top} \right\| _ {2} ^ {2} \\ = \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} + \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {M} _ {j.} ^ {\top} - \boldsymbol {M} _ {j.} ^ {\top} \right\| _ {2} ^ {2}. \tag {19} \\ \end{array}
$$
The first term can be further decomposed as
$$
\begin{array}{l} \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} \leq 2 \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} - \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} + 2 \left\| \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} \\ \leq 2 \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} - \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \right\| _ {2} ^ {2} \left\| \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} + 2 \left\| \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j.} ^ {\top} \right\| _ {2} ^ {2} \tag {20} \\ \end{array}
$$
Next, we bound the first term on the right hand side of (20). To that end, by Wedin sin  $\Theta$  Theorem (see [10, 32]) and recalling  $\mathrm{rank}(M) = k$ ,
$$
\left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} - \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \right\| _ {2} \leq \frac {\| \boldsymbol {E} \| _ {2}}{\sigma_ {k} (\boldsymbol {M})}
$$
$$
\left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} \leq 2 \frac {\| \boldsymbol {E} \| _ {2} ^ {2}}{\sigma_ {k} (\boldsymbol {M}) ^ {2}} \left\| \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} + 2 \left\| \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} \tag {21}
$$
Then, we turn to bounding the second term from (19).
$$
\begin{array}{l} \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} M _ {j \cdot} ^ {\top} - M _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} = \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} M _ {j \cdot} ^ {\top} - \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} M _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} \\ \leq \left\| \widehat {\mathbf {V}} _ {k} \widehat {\mathbf {V}} _ {k} ^ {\top} - \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \right\| _ {2} ^ {2} \left\| \boldsymbol {M} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} \\ \leq \frac {\left\| \boldsymbol {E} \right\| _ {2} ^ {2}}{\sigma_ {k} (\boldsymbol {M}) ^ {2}} \left\| \boldsymbol {M} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2}. \tag {22} \\ \end{array}
$$
Using (19), (21), and (22), we have,
$$
\left\| \widehat {\boldsymbol {M}} _ {j \cdot} ^ {\top} - \boldsymbol {M} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} \leq 2 \frac {\left\| \boldsymbol {E} \right\| _ {2} ^ {2}}{\sigma_ {k} (\boldsymbol {M}) ^ {2}} \left(\left\| \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2} + \left\| \boldsymbol {M} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2}\right) + 2 \left\| \mathbf {V} _ {k} \mathbf {V} _ {k} ^ {\top} \boldsymbol {E} _ {j \cdot} ^ {\top} \right\| _ {2} ^ {2}, \tag {23}
$$
Now, Let's apply Lemma C.1 to our setting. Let  $\mathbf{Z}_x$  be the stacked page matrix for  $x_1(t),\ldots ,x_n(t)$ . Using Lemma C.1, and setting  $\mathbf{Y} = \mathbf{Z}_y^\top$ ,  $M = \mathbf{Z}_f^\top$ ,  $\mathbf{E} = \mathbf{Z}_x^\top$ , and reusing  $\mathbf{U}_k\Sigma_k\mathbf{V}_k^\top$  to represent the top  $k$  singular components of the SVD of  $\mathbf{Z}_f$ , results in the following deterministic bound for HSVT with rank set to  $k$ ,
$$
\left\| \left[ \widehat {\mathbf {Z}} _ {f} \right] _ {\cdot j} - \left[ \mathbf {Z} _ {f} \right] _ {\cdot j} \right\| _ {2} ^ {2} \leq 2 \frac {\left\| \mathbf {Z} _ {x} \right\| _ {2} ^ {2}}{\sigma_ {k} \left(\mathbf {Z} _ {f}\right) ^ {2}} \left(\left\| \left[ \mathbf {Z} _ {x} \right] _ {\cdot j} \right\| _ {2} ^ {2} + \left\| \left[ \mathbf {Z} _ {f} \right] _ {\cdot j} \right\| _ {2} ^ {2}\right) + 2 \left\| \mathbf {U} _ {k} \mathbf {U} _ {k} ^ {\top} \left[ \mathbf {Z} _ {x} \right] _ {\cdot j} \right\| _ {2} ^ {2}, \tag {24}
$$
C.2 Deterministic To High-Probability
Next, we convert the bound in (24) to a bound in expectation (as well as one in high-probability) for  $\| \widehat{\mathbf{Z}}_f - \mathbf{Z}_f\|_{2,\infty}$ . In particular, we establish