Title: SAMoSSA:  Multivariate Singular Spectrum Analysis with Stochastic Autoregressive Noise
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time series

System Identification

Singular Spectrum Analysis
Abstract: The well-established practice of time series analysis involves estimating deterministic, non-stationary trend and seasonality components followed by learning the residual stochastic, stationary components. Recently, it has been shown that one can learn the deterministic non-stationary components accurately using  multivariate Singular Spectrum Analysis (mSSA) in the absence of a correlated stationary component; meanwhile, in the absence of deterministic non-stationary components, the Autoregressive (AR) stationary component can also be learnt readily, e.g. via Ordinary Least Squares (OLS). However, a theoretical underpinning of multi-stage learning algorithms involving both deterministic and stationary components has been absent in the literature despite its pervasiveness. We resolve this open question by establishing desirable theoretical guarantees for a natural two-stage algorithm, where mSSA is first applied to estimate the non-stationary components despite the presence of a correlated stationary AR component, which is subsequently learned from the residual time series. We provide a finite-sample forecasting consistency bound for the proposed algorithm, SAMoSSA, which is data-driven and thus requires minimal parameter tuning. To establish theoretical guarantees, we overcome three hurdles: (i) we characterize the spectra of Page matrices of stable AR processes, thus extending the analysis of mSSA; (ii) we extend the analysis of AR process identification in the presence of arbitrary bounded perturbations; (iii) we characterize the out-of-sample or forecasting error, as opposed to solely considering model identification. Through representative empirical studies, we validate the superior performance of SAMoSSA compared to existing baselines. Notably, SAMoSSA's ability to account for AR noise structure yields improvements ranging from 5% to 37% across various benchmark datasets.

SAMoSSA: Multivariate Singular Spectrum Analysis with Stochastic Autoregressive Noise

Abstract

The well-established practice of time series analysis involves estimating deterministic, non-stationary trend and seasonality components followed by learning the residual stochastic, stationary components. Recently, it has been shown that one can learn the deterministic non-stationary components accurately using multivariate Singular Spectrum Analysis (mSSA) in the absence of a correlated stationary component; meanwhile, in the absence of deterministic non-stationary components, the Autoregressive (AR) stationary component can also be learnt readily, e.g. via Ordinary Least Squares (OLS). However, a theoretical underpinning of multi-stage learning algorithms involving both deterministic and stationary components has been absent in the literature despite its pervasiveness. We resolve this open question by establishing desirable theoretical guarantees for a natural two-stage algorithm, where mSSA is first applied to estimate the non-stationary components despite the presence of a correlated stationary AR component, which is subsequently learned from the residual time series. We provide a finite-sample forecasting consistency bound for the proposed algorithm, SAMoSSA, which is data-driven and thus requires minimal parameter tuning. To establish theoretical guarantees, we overcome three hurdles: (i) we characterize the spectra of Page matrices of stable AR processes, thus extending the analysis of mSSA; (ii) we extend the analysis of AR process identification in the presence of arbitrary bounded perturbations; (iii) we characterize the out-of-sample or forecasting error, as opposed to solely considering model identification. Through representative empirical studies, we validate the superior performance of SAMoSSA compared to existing baselines. Notably, SAMoSSA's ability to account for AR noise structure yields improvements ranging from 5% to 37% across various benchmark datasets.

1 Introduction

Background. Multivariate time series have often been modeled as mixtures of stationary stochastic processes (e.g. AR process) and deterministic non-stationary components (e.g. polynomial and harmonics). To handle such mixtures, classical time series forecasting algorithms first attempt to estimate and then remove the non-stationary components. For example, before fitting an Autoregressive Moving-average (ARMA) model, polynomial trend and seasonal components must be estimated and then removed from the time series. Once the non-stationary components have been eliminated 1 , the ARMA model is learned. This estimation procedure often relies on domain knowledge and/or fine-tuning, and theoretical analysis for such multiple-stage algorithms is limited in the literature.

Prior work presented a solution for estimating non-stationary deterministic components without domain knowledge or fine-tuning using mSSA [3]. This framework systematically models a wide

Devavrat Shah MIT

class of (deterministic) non-stationary multivariate time series as linear recurrent formulae (LRF), encompassing a wide class of spatio-temporal factor models that includes harmonics, exponentials, and polynomials. However, mSSA, both algorithmically and theoretically, does not handle additive correlated stationary noise, an important noise structure in time series analysis. Indeed, all theoretical results of mSSA are established in the noiseless setting [16] or under the assumption of independent and identically distributed (i.i.d.) noise [4, 3].

On the other hand, every stable stationary process can be approximated as a finite order AR process [8, 26]. The classical OLS procedure has been shown to accurately learn finite order AR processes [17]. However, the feasibility of identifying AR processes in the presence of a non-stationary deterministic component has not been addressed.

In summary, despite the pervasive practice of first estimating non-stationary deterministic components and then learning the stationary residual component, neither an elegant unified algorithm nor associated theoretical analyses have been put forward in the literature.

A step towards resolving these challenges is to answer the following questions: (i) Can mSSA consistently estimate non-stationary deterministic components in the presence of correlated stationary noise? (ii) Can the AR model be accurately identified using the residual time series, after removing the non-stationary deterministic components estimated by mSSA, potentially with errors? (iii) Can the out-of-sample or forecasting error of such a multi-stage algorithm be analyzed?

In this paper, we resolve all three questions in the affirmative: we present SAMoSSA, a two-stage procedure, where in the first stage, we apply mSSA on the observations to extract the non-stationary components; and in the second stage, the stationary AR process is learned using the residuals.

Setup. We consider the discrete time setting where we observe a multivariate time series Y ( t ) := [ y 1 ( t ) , . . . , y N ( t )] ∈ R N at each time index t ∈ [ T ] := { 1 , . . . , T } where T ≥ N 2 . For each n ∈ [ N ] , and each timestep t ∈ [ T ] , the observations take the form

where f n : Z + → R denotes the non-stationary deterministic component, and x n ( t ) is a stationary AR noise process of order p n ( AR( p n ) ). Specifically, each x n ( t ) is governed by

where η n ( t ) refers to the per-step noise, modeled as mean-zero i.i.d. random variables, and α ni ∈ R ∀ i ∈ [ p n ] are the parameters of the n -th AR( p n ) process.

Goal. For each n ∈ [ N ] , our objective is threefold. The first is estimating f n ( t ) from the noisy observations y n ( t ) for all t ∈ [ T ] . The second is identifying the AR process' parameters α ni ∀ i ∈ [ p n ] . The third is out-of-sample forecasting of y n ( t ) for t > T .

Contributions. The main contributions of this work is SAMoSSA, an elegant two-stage algorithm, which manages to learn both non-stationary deterministic and stationary stochastic components of the underlying time series. A detailed summary of the contributions is as follows.

(a) Estimating non-stationary component with mSSA under AR noise. The prior theoretical results for mSSA are established in the deterministic setting [16] or under the assumption of i.i.d. noise [4, 3]. In Theorem 4.1, we establish that the mean squared estimation error scales as ∼ 1 / √ NT under AR noise - the same rate was achieved by [3] under i.i.d. noise. Key to this result is establishing spectral properties of the 'Page' matrix of AR processes, which may be of interest in its own right (see Lemma A.2).

(b) Estimating AR model parameters under bounded perturbation. We bound the estimation error for the OLS estimator of AR model parameters under arbitrary and bounded observation noise, which could be of independent interest. Our results build upon the recent work of [17] which derives similar results but without any observation noise. In that sense, ours can be viewed as a robust generalization of [17].

(c) Out-of-sample finite-sample consistency of SAMoSSA. We provide a finite-sample analysis for the forecasting error of SAMoSSA- such analysis of two-stage procedures in this setup is nascent despite its ubiquitous use in practice. Particularly, we establish in Theorem 4.4 that the out-ofsample forecasting error for SAMoSSA for the T time-steps ahead scales as ∼ 1 T + 1 √ NT with high probability.

(d) Empirical results. We demonstrate superior performance of SAMoSSA using both real-world and synthetic datasets. We show that accounting for the AR noise structure, as implemented in SAMoSSA, consistently enhances the forecasting performance compared to the baseline presented in [3]. Specifically, these enhancements range from 5% to 37% across benchmark datasets.

Related Work. Time series analysis is a well-developed field. We focus on two pertinent topics.

mSSA. Singular Spectrum Analysis (SSA), and its multivariate extension mSSA, are well-studied methods for time series analysis which have been used heavily in a wide array of problems including imputation [4, 3], forecasting [18, 2], and change point detection [22, 6]. Refer to [16, 15] for a good overview of the SSA literature. The classical SSA method consists of the following steps: (1) construct a Hankel matrix of the time series of interest; (2) apply Singular Value Decomposition (SVD) on the constructed Hankel matrix, (3) group the singular triplets to separate the different components of the time series; (4) learn a linear model for each component to forecast. mSSA is an extension of SSA which handles multiple time series simultaneously and attempts to exploit the shared structure between them [18]. The only difference between mSSA and SSA is in the first step, where Hankel matrices of individual series are 'stacked' together to create a single stacked Hankel matrix. Despite its empirical success, mSSA's classical analysis has mostly focused on identifying which time series have a low-rank Hankel representation, and defining sufficient asymptotic conditions for signal extraction, i.e., when the various time series components are separable. All of the classical analysis mostly focus on the deterministic case, where no observation noise is present.

Recently, a variant of mSSA was introduced for the tasks of forecasting and imputation [3]. This variant, which we extend in this paper, uses the Page matrix representation instead of the Hankel matrix, which enables the authors to establish finite-sample bounds on the imputation and forecasting error. However, their work assumes the observation noise to be i.i.d., and does not accommodate correlated noise structure, which is often assumed in the time series literature [26]. Our work extends the analysis in Agarwal et al. [3] to observations under AR noise structure. We also extend the analysis of the forecasting error by studying how well we can learn (and forecast) the AR noise process with perturbed observations.

Estimating AR parameters. AR processes are ubiquitous and of interest in many fields, including time series analysis, control theory, and machine learning. In these fields, it is often the goal to estimate the parameters of an AR process from a sample trajectory. Estimation is often carried out through OLS, which is asymptotically optimal [11]. The asymptotic analysis of the OLS estimator is well established, and recently, given the newly developed statistical tools of high dimensional probability, cf. [31, 29], various recent works have tackled its finite-time analysis as well. For example, several works established results for the finite-time identification for general first order vector AR systems [20, 19, 27, 24, 21]. Further, González et al. [17] provides a finite-time bound on the deviation of the OLS estimate for general AR( p ) processes. To accommodate our setting, we extend the results of [17] in two ways. First, we extend the analysis to accommodate any sub-gaussian noise instead of assuming gaussianity; Second, and more importantly, we extend it to handle arbitrary bounded observation errors in the sampled trajectory, which can be of independent interest.

2 Model

Deterministic Non-Stationary Component. We adopt the spatio-temporal factor model of [3, 6] described next. The spatio-temporal factor model holds when two assumptions are satisfied: the first assumption concerns the spatial structure, i.e., the structure across the N time series f 1 , . . . , f N ; the second assumption pertains to the 'temporal' structure. Before we describe the assumptions, we first define a key time series representation: the Page matrix.

Definition 2.1 (Page Matrix) . Given a time series f : Z + → R , and an initial time index t 0 > 0 , the Page matrix representation over the T entries f ( t 0 ) , . . . , f ( t 0 + T -1) with parameter 1 ≤ L ≤ T is given by the matrix Z ( f, L, T, t 0 ) ∈ R L ×⌊ T/L ⌋ with Z ( f, L, T, t 0 ) ij = f ( t 0 + i -1 + ( j -1) × L ) for i ∈ [ L ] , j ∈ [ ⌊ T/L ⌋ ] .

In words, to construct the L ×⌊ T/L ⌋ Page matrix of the entries f ( t 0 ) , . . . , f ( t 0 + T -1) , partition them into ⌊ T/L ⌋ segments of L contiguous entries and concatenate these segments column-wise (see Figure 1). Now we introduce the main assumptions for f 1 , . . . , f N .

Assumption 2.1 (Spatial structure) . For each n ∈ [ N ] , f n ( t ) = ∑ R r =1 u nr w r ( t ) for some u nr ∈ R and w r : Z + → R .

Assumption 2.1 posits that each time series f n ( · ) ∀ n ∈ [ N ] can be described as a linear combination of R 'fundamental' time series. The second assumption relates to the temporal structure of the fundamental time series w r ( · ) ∀ r ∈ [ R ] , which we describe next.

Assumption 2.2 (Temporal structure) . For each r ∈ [ R ] and for any T > 1 , 1 ≤ L ≤ T, t 0 > 0 , rank ( Z ( w r , L, T, t 0 )) ≤ G .

Assumption 2.2 posits that the Page matrix of each 'fundamental' time series w r has finite rank. While this imposed temporal structure may seem restrictive at first, it has been shown that many standard functions that model time series dynamics satisfy this property [4, 3]. These include any finite sum of products of harmonics, low-degree polynomials, and exponential functions (refer to Proposition 2.1 in [3]).

Stochastic Stationary Component. We adopt the following two assumptions about the AR processes x n ∀ n ∈ [ N ] . We first assume that these AR processes are stationary (see Definition H.4 in Appendix H).

Assumption 2.3 (Stationarity and distinct roots) . x n ( t ) is a stationary AR( p n ) process ∀ n ∈ [ N ] . That is, let λ ni ∈ C , i ∈ [ p n ] denote the roots of g n ( z ) := z p n -∑ p n i =1 α ni z p n -i . Then, | λ ni | < 1 ∀ i ∈ [ p n ] . Further, ∀ n ∈ [ N ] , the roots of g n ( z ) are distinct.

Further, we assume the per-step noise η n ( t ) are i.i.d. sub-gaussian random variables (see Definition H.1 in Appendix H).

Assumption 2.4 (Sub-gaussian noise) . For n ∈ [ N ] , t ∈ [ T ] , η n ( t ) are zero-mean i.i.d. sub-gaussian random variables with variance σ 2 .

Model Implications. In this section, we state two important implications of the model stated above. The first is that the stacked Page matrix defined as

is low-rank. Precisely, we recall the following Proposition stated in [3].

Proposition 2.1 (Proposition 2 in [3] ) . Let Assumptions 2.1 and 2.2 hold. Then for any L ≤ ⌊ √ T ⌋ with any T ≥ 1 , t 0 > 0 , the rank of the Page matrix Z ( f n , L, T, t 0 ) for n ∈ [ N ] is at most R × G . Further, the rank of the stacked Page matrix Z f ( L, T, t 0 ) is k ≤ R × G .

Throughout, we will use the shorthand Z f := Z f ( L, T, 1) and Z f ( t 0 ) := Z f ( L, T, t 0 ) 3 . The second implication is that there exists a linear relationship between the last row of Z f and its top L -1 rows. The following proposition establishes this relationship. First, Let [ Z f ] L · denote the L -th row of Z f and let Z ′ f ∈ R ( L -1) × ( N ⌊ T/L ⌋ ) denote the sub-matrix that consist of the top L -1 rows of Z f .

Proposition 2.2 (Proposition 3 in [3] ) . Let Assumptions 2.1 and 2.2 hold. Then, for L > RG , there exists β ∗ ∈ R L -1 such that [ Z f ] ⊤ L · = Z ′ f ⊤ β ∗ . Further, ∥ β ∗ ∥ 0 ≤ RG .

3 Algorithm

The proposed algorithm provides two main functionalities. The first one is decomposing the observations y n ( t ) into an estimate of the non-stationary and stationary components for t ≤ T . The second is forecasting y n ( t ) for t > T , which involves learning a forecasting model for both f n ( t ) and x n ( t ) .

Univariate Case . For ease of exposition, we will first describe the algorithm for the univariate case ( N = 1 ). The algorithm has the following parameters: 1 < L ≤ √ T , and 1 ≤ ˆ k ≤ L (refer

to Appendix B.2 for how to choose these parameters). For clarity, we will assume, without loss of generality, that L is chosen such that T/L is an integer 4 . In the first step of the algorithm, we transform the observations y 1 ( t ) , t ∈ [ T ] into the L × T/L Page matrix Z ( y 1 , L, T, 1) . We will use the shorthand Z y 1 := Z ( y 1 , L, T, 1) henceforth.

Decomposition . We compute the SVD Z y 1 = ∑ L ℓ =1 s ℓ u ℓ v ⊤ ℓ , where s 1 ≥ s 2 · · · ≥ s L ≥ 0 denote its ordered singular values, and u ℓ ∈ R L , v ℓ ∈ R T/L denote its left and right singular vectors, respectively, for ℓ ∈ [ L ] . Then, we obtain ̂ Z f 1 by retaining the top ˆ k singular components of Z y 1 (i.e., by applying Hard Singular Value Thresholding (HSVT) with threshold ˆ k ). That is, ̂ Z f 1 = ∑ ˆ k ℓ =1 s ℓ u ℓ v ⊤ ℓ .

We denote by ̂ f 1 ( t ) and ̂ x 1 ( t ) the estimates of f 1 ( t ) and x 1 ( t ) respectively. We read off the estimates ̂ f 1 ( t ) directly from the matrix ̂ Z f 1 using the entry that corresponds to t ∈ [ T ] . More precisely, for t ∈ [ T ] , ̂ f 1 ( t ) equals the entry of ̂ Z f 1 in row ( t -1 mod L ) + 1 and column ⌈ t/L ⌉ , while ̂ x 1 ( t ) = y 1 ( t ) -̂ f 1 ( t ) .

Forecasting. To forecast y 1 ( t ) for t > T , we produce a forecast for both ̂ f 1 ( t ) and ̂ x 1 ( t ) . Both forecasts are performed through linear models ( ̂ β for ̂ f 1 ( t ) and ̂ α 1 for ̂ x 1 ( t ) .) For ̂ f 1 ( t ) , we first learn a linear model ̂ β defined as

where ̂ F m = [ ̂ f 1 ( L ( m -1) + 1) , . . . , ̂ f 1 ( L × m -1)] for m ∈ [ T/L ] 5 We then use ̂ β and the L -1 lagged observations to produce ̂ f 1 ( t ) . That is ̂ f 1 ( t ) = ̂ β ⊤ Y 1 ( t -1) , where Y 1 ( t -1) = [ y 1 ( t -1) , . . . , y 1 ( t -L )] .

For ̂ x 1 ( t ) , we first estimate the parameters for the AR( p 1 ) process. Specifically, define the p 1 -dimensional vectors ̂ X 1 ( t ) := [ ̂ x 1 ( t ) , . . . , ̂ x 1 ( t -p 1 +1)] 6 . Then, define the OLS estimate as,

Then, let ˜ x 1 ( t ′ ) = y 1 ( t ′ ) -̂ f 1 ( t ′ ) for any t ′ < t , 7 . We then use ̂ α 1 and the p 1 lagged entries of ˜ x 1 ( · ) to produce ̂ x 1 ( t ) . That is ̂ x 1 ( t ) = ̂ α ⊤ 1 ˜ X 1 ( t -1) , where ˜ X 1 ( t -1) = [ ˜ x 1 ( t -1) , . . . , ˜ x 1 ( t -p 1 )] . Finally, produce the forecast ̂ y 1 ( t ) = ̂ f 1 ( t ) + ̂ x 1 ( t ) .

Multivariate Case. The key change for the case N > 1 is the use of the stacked Page matrix Z y , which is the column-wise concatenation of the Page matrices induced by individual time series. Specifically, consider the stacked Page matrix Z y ∈ R L × NT/L defined as

Decomposition . The procedure for learning the non-stationary component f 1 , . . . , f N for t ≤ T is similar to that of the univariate case. Specifically, we perform HSVT with threshold ˆ k on Z y to produce ̂ Z y . Then, we read off the estimate of f n ( t ) from ̂ Z y . Specifically, for t ∈ [ T ] , and n ∈ [ N ] , let ̂ Z f n refer to sub-matrix of ̂ Z f induced by selecting only its [( n -1) × ( T/L ) + 1 , . . . , n × T/L ]

.

T

Il. Learn β

y(t)

y(1）y(2)）y(3）….y(T)

(1)

...

f2(1)

*..

f(1)

y2()

yv(t)

y(1）y(2) yv(3）..y(T)

y2(1）y(2））

：

y2(3)

..y(T)

：

f(L-1)

f(2)

：

...

f(T-1)

f(L-1)

f(2)

...

f2(T-1)

*..

...

...

...

fN(L- 1)

fN(2)

：

...

...

fM(T-1)

：

1.StackedPage

y(L)

y(T)

y2(L)

...

yN(L)

yM(T)

Matrixtransformation

IV. Learn α,n

y(1）y(L+1)

...

y2(1)

...

...

yv(1)

...

x(1)

x(2)

x(T-p)

y(2)

：

...

...

：

y2(2)

.*.

yv(2)

：

...

an

x,(2)

：

x(3)

：

：:

x(T-p+2)

：

y(L)

...

y(T)y2(L)

...

...

yN(L)

...

yN(T)

x(p-1)

x(P)

...

x(T-1)

Z

II. HSVT

x(P)

x(p+1)

...

.

V.Forecastfort>T

1

K

n()=BT(-1)

+a

x（t-1)

众(（1)

(2)

：

...

...

x（1)

：

...

...

".

X(1)

X(2)

...

...

...

...

：

f(1)

f;(2)

：

...

：

...

..

fe(1)

f2(2)

：

...

..

：

.

fM(2)

fM(1)

：

'..

：

y(t-L)

：

：

x（t-Pn）

：

x(L)

...

,(T)x(L)

...

...

...

XT)

(L

f(T)f(L)

..

f(T)

...

fN(L)...

fM(T)

Z,-Z

Zf

columns. Then for t ∈ [ T ] , ̂ f n ( t ) equals the entry of ̂ Z f n in row ( t -1 mod L ) + 1 and column ⌈ t/L ⌉ . We then produce an estimate for x n ( t ) as ̂ x n ( t ) = y n ( t ) -̂ f n ( t ) .

Forecasting. To forecast y n ( t ) for t > T , as in the univariate case, we learn a linear model ̂ β defined as

where y m is the m -th component of [ y 1 ( L ) , y 1 (2 × L ) , . . . , y 1 ( T ) , y 2 ( L ) , . . . , y 2 ( T ) , . . . , y N ( T )] ∈ R N × T/L , and ̂ F m ∈ R L -1 corresponds to the vector formed by the entries of the first L -1 rows in the m th column of ̂ Z f 8 for m ∈ [ N × T/L ] . We then use ̂ β to produce ̂ f n ( t ) = ̂ β ⊤ Y n ( t -1) , where again Y n ( t -1) is the vector of the L -1 lags of y n ( t -1) . That is Y n ( t -1) = [ y n ( t -1)) . . . y n ( t -L )] .

Then, for each n ∈ [ N ] , we estimate α ni ∀ i ∈ [ p n ] , the parameters for the n -th AR( p n ) process. Let ̂ α n denote the OLS estimate defined as

Then, produce a forecast for x n as ̂ x n ( t ) = ̂ α ⊤ n ˜ X n ( t -1) , where again ˜ X n ( t -1) = [ ˜ x n ( t -1) , . . . , ˜ x n ( t -p n )] . Finally, produce the forecast for y n as ̂ y n ( t ) = ̂ f n ( t ) + ̂ x n ( t ) . For a visual depiction of the algorithm, refer to Figure 1.

4 Results

In this section, we provide finite-sample high probability bounds on the following quantities:

Estimation error of non-stationary component . First, we give an upper bound for the estimation error of each one of f 1 ( t ) , . . . , f N ( t ) for t ∈ [ T ] . Specifically, we upper bound the following metric

where ̂ f n ( t ) n ∈ [ N ] , t ∈ [ T ] are the estimates produced by the algorithm we proposed in Section 3.

2. Identification of AR parameters. Second, we analyze the accuracy of the estimates ̂ α n ∀ n ∈ [ N ] produced by the proposed algorithm. Specifically, we upper bound the following metrics

where α n = [ α n 1 , . . . , α np ] .

3. Out-of-sample forecasting error. Finally, we provide an upper bound for the forecasting error of y 1 ( t ) , . . . , y N ( t ) for t ∈ { T +1 , . . . , 2 T } . Specifically, we upper bound the following metric

where ̂ y n ( · ) ∀ n ∈ [ N ] , t ∈ { T +1 , . . . , 2 T } are the forecasts produced by the algorithm we propose in Section 3. Before stating the main results, we state key additional assumptions.

Assumption 4.1 (Balanced spectra) . Let Z f ( t 0 ) denote the L × NT/L stacked Page matrix associated with all N time series f 1 ( · ) , . . . , f N ( · ) for their T consecutive entries starting from t 0 . Let k = rank ( Z f ( t 0 )) , and let σ k ( Z f ( t 0 )) denote the k -th singular value for Z f ( t 0 ) . Then, for any t 0 > 0 , Z f ( t 0 ) is such that σ k ( Z f ( t 0 )) ≥ γ √ NT/ √ k for some absolute constant γ > 0 .

This assumption holds whenever the the non-zero singular values are 'well-balanced', a standard assumption in the matrix/tensor estimation literature [3, 6]. Note that this assumption, as stated, ensures balanced spectra for the stacked Page matrix of any set of T consecutive entries of f 1 ( · ) , . . . , f N ( · ) .

Finally, we will impose an additional necessary restriction on the complexity of the N time series f 1 ( t ) , . . . , f N ( t ) for t > T (as is done in [3, 5]). Let Z ′ f denote the ( L -1) × ( NT/L ) matrix formed using the top L -1 rows of Z f . Further, for any t 0 ∈ [ T +1] , let Z ′ f ( t 0 ) denote the ( L -1) × ( NT/L ) matrix formed using the top L -1 rows of Z f ( t 0 ) . For any matrix M , let colspan ( M ) denote the subspace spanned by the its columns. We assume the following property.

Assumption 4.2 (Subspace inclusion) . For any t 0 ∈ [ T +1] , colspan ( Z ′ f ( t 0 )) ⊆ colspan ( Z ′ f ) .

This assumption is necessary as it requires the stacked Page matrix of the out-of-sample time series Z ′ f ( t 0 ) to be only as 'rich' as that of the stacked Page matrix of the 'in-sample' time series Z ′ f .

4.1 Main Results

First, recall that R is defined in Assumption 2.1, G in Assumption 2.2, k is in Proposition 2.1, while γ is defined in Assumption 4.1. Further, recall that λ ni for i ∈ [ p n ] are the roots of the characteristic polynomial of the n -th AR process, as defined in Assumption 2.3. Throughout, let c and C be absolute constants, f max := max n,t ≤ 2 T | f n ( t ) | , p := max n p n , α max = max n ∥ α n ∥ 2 , and C ( f max , γ ) denote a constant that depends only (polynomially) on model parameters f max and γ . Last but not least, define the key quantity

where λ ⋆ = max i,n | λ ni | and c λ is a constant that depends only on λ ni 9 . Note that σ 2 x is a key quantity that we use to bound important spectral properties of AR processes, as Lemma A.2 establishes.

4.1.1 Estimation Error of Non-Stationary Component

Theorem 4.1 (Estimation error of f ) . Assume access to the observations y 1 ( t ) , · · · , y N ( t ) for t ∈ [ T ] as defined in (1) . Let assumptions 2.1, 2.2, 2.3, 2.4 and 4.1 hold. Let L = √ NT and ˆ k = k , then, for any n ∈ [ N ] , with probability of at least 1 -c ( NT ) 10

This theorem implies that the mean squared error of estimating f scales as ˜ O ( 1 √ NT ) 10 with high probability. The proof of Theorem 4.1 is in Appendix C.

4.1.2 Identification of AR Processes

Theorem 4.2 (AR Identification) . Let the conditions of Theorem 4.1 hold. Let ̂ α n be as defined in (7) , and α n = [ α n 1 , . . . , α np n ] as defined in (2) . Then, for a sufficiently large T such that log( T ) √ T < Cσ 2 pσ 2 x α 2 max ( λ min (Ψ) λ max (Γ) ) , and for any n ∈ [ N ] where EstErr ( N,T,n ) ≤ σ 2 λ min (Ψ) 6 p , we have with probability of at least 1 -c T 10 ,

Recall that ̂ α n is estimated using ̂ x n ( · ) , a perturbed version of the true AR process x n ( · ) . This perturbation comes from the estimation error of x n ( · ) , which is a consequence of EstErr ( N,T,n ) . Hence, it is not surprising that EstErr ( N,T,n ) shows up in the upper bound. Indeed, the upper bound we provide here consists of two terms: the first, which scales as ˜ O ( 1 T ) is the bound one would get with access to the true AR process x n ( · ) , as shown in [17]. The second term characterizes the contribution of the perturbation caused by the estimation error, and it scales as O ( EstErr ( N,T,n )) , which we show is ˜ O ( 1 √ NT ) with high probability in Theorem 4.1. Also, note that the theorem holds when the estimation error is sufficiently small ( EstErr ( N,T,n ) ≤ σ 2 λ min (Ψ) 6 p ). This condition ensures that the perturbation arising from the estimation error remains below the lower bound for the minimum eigenvalue of the sample covariance matrix associated with the autoregressive processes. Note that λ max (Ψ) , λ min (Ψ) and λ max (Γ) are quantities that relate to the parameters of the AR( p ) processes and their 'controllability Gramian' as we detail in Appendix A.1. The proof of Theorem 4.2 is in Appendix D.

4.1.3 Out-of-sample Forecasting Error

We first establish an upper bound on the error of our estimate ̂ β .

Theorem 4.3 (Model Identification ( β ∗ )) . Let the conditions of Theorem 4.1 hold. Let ̂ β be defined as in (6) , and β ∗ as defined in Proposition 2.2. Then, with probability of at least 1 -c ( NT ) 10

This shows that the error of estimating β ∗ (in squared euclidean norm) scales as ˜ O ( 1 √ NT ) .

Theorem 4.4. Let the conditions of Theorem 4.2 and Assumption 4.2 hold. Then, with probability of at least 1 -c T 10

where c is an absolute constant, and ˜ C denotes a constant that depends only (polynomially) on f max , γ, λ min (Ψ) , λ max (Ψ) , β ∗ and α max .

This theorem establishes that the forecasting error for the next T time steps scales as ˜ O ( 1 T + 1 √ NT ) with high probability. Note that the ˜ O ( 1 T ) term is a function of T only, as it is a consequence of the error incurred when we estimate each α n . Recall that we estimate α n separately for n ∈ [ N ] using the available (perturbed) T observation of each process. The ˜ O ( 1 √ NT ) term on the other hand is a consequence of the error incurred when learning and forecasting f 1 , . . . , f n , which is done collectively across the N time series. Finally, Theorem 4.4 implies that when N = Θ( T ) , the forecasting error scales as ˜ O ( 1 T ) . The proof of Theorems 4.3 and 4.4 are in Appendix F and G, respectively.

5 Experiments

In this section, we support our theoretical results through several experiments using synthetic and real-world data. In particular, we draw the following conclusions:

Our results align with numerical simulations concerning the estimation of non-stationary components under AR stationary noise and the accuracy of AR parameter estimation (Section 5.1).

Modeling and learning the autoregressive process in SAMoSSA led to a consistent improvement over mSSA. The improvements range from 5% to 37% across standard datasets (Section 5.2).

5.1 Model Estimation

Setup. We generate a synthetic multivariate time series ( N = 10 ) that is a mixture of both harmonics and an AR noise process. The AR process is stationary, and its parameter is chosen such that λ ∗ = max n,i | λ ni | is one of three values { 0 . 3 , 0 . 6 , 0 . 95 } . Refer to Appendix B.1 for more details about the generating process. We then evaluate the estimation error EstErr ( N,T, 1) and the AR parameter estimation error ∥ α 1 -̂ α 1 ∥ 2 as we increase T from 200 to 500000 .

Results. Figure 2a visualizes the mean squared estimation error of f 1 , while Figure 2b shows the AR parameter estimation error. The solid lines in both figures indicate the mean across ten trials, whereas the shaded areas cover the minimum and maximum error across the ten trials. We find that, as the theory suggests, the estimation error for both the non-stationary component and the AR parameter decay to zero as NT increases. We also see that the estimation error is inversely proportional to (1 -λ ∗ ) , which is being reflected in the AR parameter estimation error as well.

10

2

10

3

10

-

3

10

-

2

10

-

1

10

0

√

NT

EstErr(

N,T,

1

)

λ

∗

= 0

.

3

λ

∗

= 0

.

6

λ

∗

= 0

.

95

(a)

10

2

10

3

0

0

.

5

1

√

NT

∥

α

1

-

̂

α

1

∥

λ

∗

= 0

.

3

λ

∗

= 0

.

6

λ

∗

= 0

.

95

(b)

5.2 Forecasting

We showcase SAMoSSA's forecasting performance relative to standard algorithms. Notably, we compare SAMoSSA to the mSSA variant in [3] to highlight the value of learning the autoregressive process.

Setup. The forecasting ability of SAMoSSA was compared against (i) mSSA [3], (ii) ARIMA, a very popular classical time series prediction algorithm, (iii) Prophet [28], (iv) DeepAR [23] and (v) LSTM [14]on three real-life datasets and one synthetic dataset containing both deterministic trend/seasonality and a stationary noise process (see details in Appendix B). Each dataset was split into train, validation, and test sets (see Appendix B.1). Each dataset has multiple time series, so we

aggregate the performance of each algorithm using the mean R 2 score. We use the R 2 score since it is invariant to scaling and gives a reasonable baseline: a negative value indicates performance inferior to simply predicting the mean.

Results. In Table 1 we report the mean R 2 for each method on each dataset. We highlight that, across all datasets, SAMoSSA consistently performs the best or is otherwise very competitive with the best. The results also underscore the significance of modeling and learning the autoregressive process in real-world datasets. Notably, learning the autoregressive model in SAMoSSA consistently led to an improvement over mSSA, with the increase in R 2 values ranging from 5% to 37%. We note that this improvement is due to the fact that mSSA, as described in [3], overlooks any potential structure in the stochastic processes x 1 ( · ) , . . . , x N ( · ) and assumes i.i.d.mean-zero noise process. While in SAMoSSA, we attempt to capture the structure of x 1 ( · ) , . . . , x N ( · ) through the learned AR process.

6 Discussion and Limitations

Wepresented SAMoSSA, a two-stage procedure that effectively handles mixtures of deterministic nonstationary and stationary AR processes with minimal model assumptions. We analyze SAMoSSA's ability to estimate non-stationary components under stationary AR noise, the error rate of AR system identification via OLS under observation errors, and a finite-sample forecast error analysis.

We note that our results can be readily adapted to accommodate (i) approximate low-rank settings (as in the model by Agarwal et al [3]); and (ii) scenarios with incomplete data. We do not discuss these settings to focus on our core contributions, but they represent valuable directions for future studies.

Our analysis reveals some limitations, providing avenues for future research. One limitation of our model is that it only considers stationary stochastic processes. Consequently, processes with non-stationary stochastic trends, such as a random walk, are not incorporated. Investigating the inclusion of such models and their interplay with the SSA literature is a worthy direction for future work. Second, our model assume non-interaction between the N stationary processes x 1 , . . . , x N . Yet, it might be plausible to posit the existence of interactions among them, possibly through a vector AR model (VAR). Examining this setting represents another compelling direction for future work.

References

Y. Abbasi-yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 24. Curran Associates, Inc., 2011.

A. Agarwal, A. Alomar, and D. Shah. tspdb: Time series predict db. In NeurIPS 2020 Competition and Demonstration Track , pages 27-56. PMLR, 2021.

A. Agarwal, A. Alomar, and D. Shah. On multivariate singular spectrum analysis and its variants. ACM SIGMETRICS Performance Evaluation Review , 50(1):79-80, 2022.

A. Agarwal, M. J. Amjad, D. Shah, and D. Shen. Model agnostic time series analysis via matrix estimation. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 2(3):1-39, 2018.

A. Agarwal, D. Shah, and D. Shen. On principal component regression in a high-dimensional error-in-variables setting. arXiv preprint arXiv:2010.14449 , 2020.

A. Alanqary, A. Alomar, and D. Shah. Change point detection via multivariate singular spectrum analysis. Advances in Neural Information Processing Systems , 34:23218-23230, 2021.

A. Alexandrov, K. Benidis, M. Bohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D. C. Maddix, S. Rangapuram, D. Salinas, J. Schulz, et al. Gluonts: Probabilistic time series models in python. arXiv preprint arXiv:1906.05264 , 2019.

T. W. Anderson. The statistical analysis of time series . John Wiley & Sons, 2011.

F. Chollet. keras. https://github.com/fchollet/keras , 2015. Online; accessed 25 February 2020.

C. Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis , 7(1):1-46, 1970.

J. Durbin. Estimation of parameters in time-series regression models. Journal of the royal statistical society: Series B (Methodological) , 22(1):139-153, 1960.

Facebook. Prophet. https://github.com/facebook/prophet , 2020.

M. Gavish and D. L. Donoho. The optimal hard threshold for singular values is 4 / √ 3 . IEEE Transactions on Information Theory , 60(8):5040-5053, 2014.

F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: Continual prediction with lstm. 1999.

N. Golyandina, A. Korobeynikov, and A. Zhigljavsky. Singular spectrum analysis with R . Springer, 2018.

N. Golyandina, V. Nekrutkin, and A. A. Zhigljavsky. Analysis of time series structure: SSA and related techniques . Chapman and Hall/CRC, 2001.

R. A. González and C. R. Rojas. A finite-sample deviation bound for stable autoregressive processes. In Learning for Dynamics and Control , pages 191-200. PMLR, 2020.

H. Hassani and R. Mahmoudvand. Multivariate singular spectrum analysis: A general view and new vector forecasting approach. International Journal of Energy and Statistics , 1(01):55-83, 2013.

Y. Jedra and A. Proutiere. Sample complexity lower bounds for linear system identification. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 2676-2681. IEEE, 2019.

Y. Jedra and A. Proutiere. Finite-time identification of stable linear systems optimality of the least-squares estimator. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 996-1001. IEEE, 2020.

Y. Jedra and A. Proutiere. Finite-time identification of linear systems: Fundamental limits and optimal algorithms. IEEE Transactions on Automatic Control , 2022.

Microsoft. nimbusml package, ssa class, 2020. https://docs.microsoft.com/enus/python/api/nimbusml/nimbusml.timeseries.ssachangepointdetector.

D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 2019.

T. Sarkar and A. Rakhlin. Near optimal finite time identification of arbitrary linear dynamical systems. In International Conference on Machine Learning , pages 5610-5618. PMLR, 2019.

S. Seabold and J. Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010.

R. H. Shumway, D. S. Stoffer, and D. S. Stoffer. Time series analysis and its applications , volume 3. Springer, 2000.

M. Simchowitz, H. Mania, S. Tu, M. I. Jordan, and B. Recht. Learning without mixing: Towards a sharp analysis of linear system identification, 2018.

S. J. Taylor and B. Letham. Forecasting at scale. The American Statistician , 72(1):37-45, 2018.

R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

P.-Å. Wedin. Perturbation bounds in connection with singular value decomposition. BIT Numerical Mathematics , 12(1):99-111, 1972.

P.-Å. Wedin. Perturbation theory for pseudo-inverses. BIT Numerical Mathematics , 13:217-232, 1973.

A Concentration inequalities for AR processes

In this section, we present two important lemmas that are central to the main results. Before stating the two lemmas, we start with key important quantities of AR Processes.

A.1 Controllability Gramian of AR Processes

In this section, we explain the notation of key quantities used in the main results. First, note that the process in (2) can be written in matrix-vector form by defining a transition matrix and input vector

where 0 p n -1 is the column vector of p n -1 zeroes. Then, with X n ( t ) := [ x n ( t ) , . . . , x n ( t -p n +1)] ,

With this setup, we define two important matrices,

where Ψ n is known as the controllability Gramian of the n -th AR( p n ) process x n . For a positive semi-definite matrix M , let λ max ( M ) ≥ 0 and λ min ( M ) ≥ 0 be its maximum and minimum eigenvalues, respectively. We define λ max (Ψ) := max n λ max (Ψ n ) and λ min (Ψ) := min n λ min (Ψ n ) . We also define λ max (Γ) and λ min (Γ) analogously for Γ .

A.2 Key Lemmas

Now, we present two important lemmas that are central to the main results. The first lemma states that a vector of consecutive entries of a stationary autoregressive process is a sub-gaussian vector with variance proxy σ 2 x .

Lemma A.1. Let assumptions 2.3 and 2.4 hold, then, for any n ∈ [ N ] , t ≥ 1 , K ≥ 1 , the vector [ x n ( t ) , . . . , x n ( t + K )] is a sub-gaussian vector with variance proxy σ 2 x .

This lemma, along with the well known bound on the norm of sub-gaussian vectors, is key to the bounds presented in Section 4.1. Further, this Lemma, along with an ε -net argument, allows us to bound the operator norm of the stacked Page matrix of the AR processes x 1 ( t ) , . . . , x N ( t ) . Specifically, we establish next that its operator norm grows as O ( √ NT/L ) with high probability. √

be the L × NT/L stacked Page matrix of the AR processes x 1 ( t ) , . . . , x N ( t ) , as defined in (2) . Let Assumptions 2.3 and 2.4 hold. Then, each row [ Z x ( t 0 )] i · is a sub-gaussian vector with variance proxy σ 2 x . Further, With probability 1 -exp( -cNT/L ) ,

Next, we prove Lemma A.1 and Lemma A.2. Before stating the proofs, we start with the important helper Lemma A.3.

Lemma A.3 (MA ( ∞ ) representation of an AR( p ) process) . Let x ( t ) be a stationary AR( p ) process defined as

Assume that the roots ( λ 1 , . . . , λ p ) of its characteristic polynomial g ( z ) := z p -∑ p i =1 α i z p -i are distinct. Then, x ( t ) has the MA ( ∞ ) representation

̸

Proof. First, note that that we can write x ( t ) in lag operator form as

Here, L is a lag operator such that Lx ( t ) = x ( t -1) . Note that since we have assumed the AR process to be stationary, all the roots of g ( z ) lie inside the unit circle [26]. Using the roots of g ( z ) , (15) can be written as,

Further, using the partial fraction decomposition we can write the process as

̸

Recall that we can expand each fraction as

where,

Thus, finally, we can write x ( t ) as, with β k := ∑ p i =1 a i λ k i .

A.3 Proof of Lemma A.1

Proof. First, let X n ( t, K ) = [ x n ( t ) , . . . , x n ( T + K )] be the vector of interest. Let E n ( t, K ) = [ η n ( t ) , . . . , η n ( T + K )] , then using Lemma A.3, we can write X n ( t, K ) as

Note that β nk E n ( t -k, K ) is clearly a sub-gaussian vector with variance proxy β 2 nk σ 2 . Further note that the sum of two sub-gaussian random vectors, not necessarily independent, is also a sub-gaussian random vector. To see this, suppose x and y are both sub-gaussian vectors with variance proxies σ 2 x and σ 2 y . Then for any v ∈ S K ,

Thus, X n ( t, K ) , for any n , is also a sub-gaussian random vector with variance proxy

Thus, with c λ = max n c λ n

Now define E i as the analogous stacked Page matrix for η 1 ( t ) , . . . , η N ( t ) for t ∈ { i, . . . , i + T -1 } . Observe that E i are simply identically distributed (but not independent) matrices of i.i.d. sub-gaussian random variables. Then we can write

where B k ∈ R q x q is a diagonal matrix with the i -th diagonal [ B k ] ii = β ⌈ iN q ⌉ k . Then

̸

Now we are interested in bounding ∑ ∞ k =0 max n | β nk | . To do so, first recall that β nk := ∑ p n i =1 a ni λ k ni where λ ni ∈ C , i ∈ [ p n ] denote the roots of g n ( z ) and a i = ∏ 1 ≤ j ≤ p n , j = i ( 1 -λ nj λ ni ) -1 . Let

̸

λ ⋆ = max i,n | λ ni | and c λ n = ∑ p n i =1 ∣ ∣ ∣ ∣ ∏ 1 ≤ j ≤ p, j = i ( 1 -λ nj λ ni ) -1 ∣ ∣ ∣ ∣ . Then, we can bound β nk as

Hence, X n ( t, K ) is a sub-gaussian random vector with variance proxy σ 2 x := c 2 λ σ 2 (1 -λ ⋆ ) 2 .

A.4 Proof of Lemma A.2

Proof. Let q = NT/L . We will find an upper bound on the operator norm by following two steps: (i) we will represent each x n ( t ) ∀ n ∈ [ N ] as an MA ( ∞ ) process; then, (ii) we will bound the operator norm of the stacked Page matrix of this MA ( ∞ ) processes.

Step 1: MA ( ∞ ) representation for AR( p ) .

As a direct application of Lemma A.3, we can write x n ( t ) as,

̸

with β nk := ∑ p i =1 a ni λ k ni , a ni := ∏ 1 ≤ j ≤ p, j = i ( 1 -λ nj λ ni ) -1

Step 2: Bound the Page matrix for the MA ( ∞ ) Page matrix.

First, let's consider, without loss of generality, the case when t 0 = 1 . That is, let's consider Z x := Z x (1) . To get the desired bound, we will first obtain a high probability bound on the Euclidean norm of Z ⊤ x v for any vector v ∈ S L -1 := { v ∈ R L : ∥ v ∥ 2 = 1 } . Then, we will bound the operator norm using this and an ε -net argument.

Note that by step (1), we have

It is easy to verify that E ⊤ i v is a sub-gaussian random vector (see Definition H.3) with variance proxy σ 2 , because for any u ∈ S q -1 we have (using E as a shorthand for E i )

Using the above, B k E ⊤ i v is ∼ subG (max n | β nk | 2 σ 2 ) . Since Z x v is a sum of sub-gaussian random vectors, it is also a sub-gaussian random vector with variance proxy σ 2 x as we showed in the proof of Lemma A.1.

Using Lemma H.2, we get, with probability at least 1 -δ ,

Equivalently, for t ≥ 0 , it holds that

Now we have shown that for any fixed v ∈ S L -1 , the quantity ∥ Z x v ∥ 2 grows as O ( √ q ) with high probability. What remains is to apply the union bound over an ε -net to extend this to the maximum over S L -1 , i.e. the operator norm of Z x .

Let Σ be a maximal 1 / 2 -net of S L -1 , i.e. a set of points in S L -1 spaced at least 1 / 2 from each other such that no other point can be added without violating this property. Now consider v ⋆ such that ∥ ∥ Z ⊤ x v ⋆ ∥ ∥ 2 = ∥ Z x ∥ 2 . Since Σ is maximal, there is some v ∈ Σ within 1 / 2 of v ⋆ . Since and

we have that

i.e. taking the maximum over Σ is equivalent to doing so over S L -1 up to a factor of two. Using the fact that | Σ | ≤ C L for some absolute constant C , the fact that L < q , and using a sufficiently large absolute constant c > 0 we have

where c ′′ > 0 and c ′ > 0 are absolute constants. Recalling that q = NT/L concludes the proof for the operator norm.

Finally, we show that each row [ Z x ( t 0 )] i · in the stacked page matrix is a sub-gaussian vector with variance proxy σ 2 x . Consider, without loss of generality, the first row [ Z x ] 1 · . Using (17), we have for any v ∈ S L -1

note that v ⊤ [ E 1 -k B k ] 1 · is ∼ subG(max n β 2 nk σ 2 ) , and hence, using the same argument for the sum of dependent sub-gaussian random variables used in the proof above, we have v ⊤ [ Z x ] 1 · ∼ subG( σ 2 x ) .

B Experiment details

In Appendix B.1, we detail the datasets used. In Appendix B.2, we explain the implementations and hyperparameter choices for our method and benchmark algorithms. In Appendix B.3, we describe our hyperparameter tuning strategy.

B.1 Datasets

In the model estimation experiments, we generate a synthetic multivariate time series ( N = 10 ) that is a mixture of both harmonics and an AR(2) process. The harmonics are generated as follow: we first generate R = 3 fundamental time series of the form g k ( t ) = sin ( ω k t + ϕ k ) for k ∈ [ R ] , where ω k and ϕ k are uniformly randomly sampled on the ranges [2 π/ 100 , 2 π/ 50] , [0 , 2 π ] respectively. We then sample the N = 10 series using f k ( t ) = ∑ R i =1 c ki g k ( t ) for k ∈ [ N ] , where the mixture coefficients c ki are independent standard normal random variables. The AR process is stationary, and its parameter is chosen such that λ ∗ = max n,i | λ ni | is set to one of { 0 . 3 , 0 . 6 , 0 . 95 } . The noise of the process has a variance σ 2 = 0 . 2 .

In the forecasting experiments, we evaluate our method and benchmark algorithms on three real-world datasets and one synthetic dataset. The preprocessing steps and setup of each dataset are described below.

Traffic Dataset. This public dataset obtained from the UCI repository shows the occupancy rate of traffic lanes in San Francisco. The data is sampled every 15 minutes but to be consistent with previous work, we aggregate the data into hourly data and use the first 10248 time-points for training, the next 48 points for validation, and another 48 points for testing in the forecasting experiments. Specifically, in our testing period, we do 1-hour ahead forecasts for the next 48 hours.

Electricity Dataset. This is a public dataset obtained from the UCI repository which shows the 15-minutes electricity load of 370 households. We aggregate the data into hourly intervals and use the first 25824 time-points for training, the next 48 points for validation, and another 48 points for testing in the forecasting experiments. Specifically, in our testing period, we do 1-hour ahead forecasts for the next 48 hours.

Exchange Dataset. This is a dataset containing the daily exchange rates of eight foreign currencies, including those of Australia, the UK, Canada, Switzerland, China, Japan, New Zealand and Singapore, between 1990 and 2016. We standardize each time series to have zero mean and unit variance since the range of typical exchange rates varies greatly across the currencies. We use the first 7528 timepoints for training, the next 30 for validation, and the final 30 for testing. All forecasts are made 1-day ahead.

Synthetic Dataset. Wegenerate R fundamental time series of the form g k ( t ) = sin ( ω k t + ϕ k )+ m k t for k ∈ [ R ] , where ω k , ϕ k , and m k are uniformly randomly sampled on the ranges [2 π/ 100 , 2 π/ 10] , [0 , 2 π ] , and [ -5 × 10 -4 , 5 × 10 -4 ] respectively. We then sample N mixtures of the form f k ( t ) = ∑ R i =1 c ki g k ( t ) for k ∈ [ N ] , where the mixture coefficients c ki are independent standard normals. We then inject AR(1) autoregressive noise to each series to produce y k ( t ) = f k ( t ) + x k ( t ) for k ∈ [ N ] , where { x k ( t ) } k ∈ [ N ] are independent processes, each with a fixed autoregressive coefficient α = -0 . 5 and noise variance σ 2 = 1 . We fix the coefficient to ensure that each time series has significantly autocorrelated noise that can be exploited to improve predictions.

B.2 Algorithms

In this section, we discuss the implementations used and hyperparameter tuning details for each algorithm that was evaluated.

SAMoSSA & mSSA. Since mSSA is a special case of SAMoSSA where the second stage - fitting autoregressive models - is skipped, we use one in-house implementation to evaluate both algorithms. The relevant hyperparameters are as follows:

The number of retained singular values, k . This hyperparameter is relevant for both algorithms. We select k using one of three methods: (i) the data-driven procedure suggested in [13] which computes a threshold based on the shape and median singular value of the page matrix; (ii) picking k as the minimum number of singular values required to capture 90% of the spectral energy of the page matrix; (iii) picking a fixed low rank, specifically k = 5 .

The shape parameter of the stacked Page matrix. This is the ratio between the number of columns and rows of the stacked Page matrix used to carry out the matrix estimation procedure. While our theoretical analysis fixes this value to 1 (recall that we set L = √ NT ), in practice we have found that a slightly wider Page matrix can improve performance. This parameter is chosen from { 1 , 3 , 5 } .

The number of autoregressive lag coefficients, p . This hyperparameter is only relevant for SAMoSSA. While each time series does not necessarily need to use the same value, we enforce this to be the case for computational efficiency. We choose p ∈ { 0 , 1 , 2 , 3 } , i.e., the model is allowed to not fit a residual model if it brings no apparent benefit, corresponding to the case of p = 0 .

Prophet. We use Prophet's Python library with the parameters selected using a grid search of the following parameters as suggested in [12]:

Changepoint prior scale. This parameter determines how much the trend changes at the detected trend changepoints. We choose this parameter from { 0 . 001 , 0 . 05 , 0 . 2 } .

Seasonality prior scale. This parameter controls the magnitude of the seasonality. We choose this parameter from { 0 . 01 , 10 } .

Seasonality Mode. We choose between an 'additive' and 'multiplicative' seasonality term.

The parameters for each time series are chosen independently.

ARIMA. We used the ARIMA implementation of the Python library statsmodels [25]. The hyperparameters are grid searched independently for each time series, and are as follows:

Autoregressive order. This is the number of autoregressive lag coefficients fitted, chosen from { 1 , 2 , 3 } .

Differencing order. This denotes the number of times the time series is differenced before a model is fitted, and we choose between 0 (no differencing) and 1.

Moving average order. This is the number of moving average lag coefficients, chosen from { 1 , 2 , 3 } .

LSTM. We use Keras implementation [9]. We perform a grid search on the number of layers { 2 , 3 , 4 } .

DeepAR. We use the implementation provided by the GluonTS package [7]. We use the default parameters.

B.3 Parameter Selection

We tune hyperparameters for all algorithms evaluated using rolling cross validation. During the validation step, each model is fitted only once on the training set; then, it repeatedly makes one-stepahead forecasts and is afterwards provided with the realized value, in order to use the updated data to make the next forecast. During test time, the model is fitted on the training and validation sets, and similarly repeatedly makes one-step-ahead forecasts with the fitted parameters. For the models that choose one set of hyperparameters for all N time series, the mean R 2 score over all time series on the validation set predictions is the metric of choice.

C Proof of Theorem 4.1

In this section, we provide the proof for Theorem 4.1. While the analysis generally follows the argument for the imputation error in [3], we adapt it for our different assumptions and metric of interest. Metric . First, recall that our metric of interest is

Recall that ̂ Z f is the stacked Page matrix of ̂ f 1 , . . . , ̂ f N , and Z f is the stacked Page matrix of f 1 , . . . , f N . To bound this error, we first establish a deterministic bound through a general lemma for Hard Singular Value Thresholding (HSVT).

C.1 Deterministic Bound

We state the following result, a more general version of which is stated in [3].

Lemma C.1 ([3]) . For k ≥ 1 , let Y = M + E ∈ R q x p with rank ( M ) = k . Let U k Σ k V k ⊤ and ̂ U k ̂ Σ k ̂ V ⊤ k denote the top k singular components of the SVD of M and Y respectively. Then, the HSVT estimate ̂ M = ̂ U k ̂ Σ k ̂ V ⊤ k is such that for all j ∈ [ q ] ,

Proof. First, note that

where ̂ M ⊤ j · = ̂ V k ̂ V ⊤ k Y ⊤ j · by definition. Note that the vector ( ̂ V k ̂ V ⊤ k M ⊤ j · -M ⊤ j · ) is in the span of a subspace orthogonal to ̂ V k ̂ V ⊤ k , and hence we have by the Pythagorean theorem,

The first term can be futher decomposed as

Next, we bound the first term on the right hand side of (20). To that end, by Wedin sin Θ Theorem (see [10, 32]) and recalling rank ( M ) = k ,

Then it follows that

Then, we turn to bounding the second term from (19).

Using (19), (21), and (22), we have,

which completes the proof.

Now, Let's apply Lemma C.1 to our setting. Let Z x be the stacked page matrix for x 1 ( t ) , . . . , x n ( t ) . Using Lemma C.1, and setting Y = Z ⊤ y , M = Z ⊤ f , E = Z ⊤ x , and reusing U k Σ k V k ⊤ to represent the top k singular components of the SVD of Z f , results in the following deterministic bound for HSVT with rank set to k ,

C.2 Deterministic To High-Probability

Next, we convert the bound in (24) to a bound in expectation (as well as one in high-probability) for ∥ ̂ Z f -Z f ∥ 2 , ∞ . In particular, we establish

̸

Theorem C.1. Let assumptions 2.1, 2.2, 2.3, and 2.4 hold. Let σ x = c λ σ (1 -λ ⋆ ) , where λ ⋆ = max i,n | λ ni | and c λ = max n ∑ p n i =1 ∣ ∣ ∣ ∣ ∏ 1 ≤ j ≤ p, j = i ( 1 -λ nj λ ni ) -1 ∣ ∣ ∣ ∣ . Then, the HSVT estimate ̂ Z f with parameter k is such that

with probability of at least 1 -c ( NT ) 11 .

Proof. We start by identifying certain high probability events. Subsequently, using these events and (24), we will conclude the proof.

High Probability Events. Let q := NT/L be the number of columns in the stacked page matrix ̂ Z f , and let C > 0 be some positive absolute constant. Define

Lemma C.2. For some positive constant c 1 > 0 and C > 0 large enough in definitions of E 1 , E 2 , and E 3 ,

Proof. We bound the probability of events above below.

Bounding E 1 . This is an immediate consequence of Lemma A.2.

Bounding E 2 . Recall that we assume L ≤ q . Observe that for any matrix A ∈ R L × q , ∥ A ∥ ∞ , 2 , ∥ A ∥ 2 , ∞ ≤ ∥ A ∥ 2 . Thus using the argument to bound E 1 , concludes the proof.

Bounding E 3 . Let u 1 , . . . , u k be the orthonormal basis for U k . Then, consider for j ∈ [ q ] ,

where Z i = u ⊤ i [ Z x ] · j . As shown in Lemma A.1, [ Z x ] · j is a sub-gaussian vector with variance proxy σ 2 x . Using Lemma H.4, we have

Therefore, for choice of t = Cσ 2 x k log( q ) with large enough constant C > 368 , we have,

Recalling that L ≤ q , and taking a union bound over all j ∈ [ q ] , we have that

The following is an immediate corollary of the above stated bounds.

Corollary C.1. Let E := E 1 ∩ E 2 ∩ E 3 . Then,

where C 1 is an absolute positive constant.

Thus, under event E , and using (24), we have, with probability 1 -c ( NT ) 10 ,

Recall that q = NT/L then,

This completes the proof of Theorem C.1. Finally, now we are ready to bound EstErr ( N,T ) . First, let Ω n = { ( n -1) T L +1 , . . . , nT L } be the set of columns in the stacked Page matrix Z f that belongs to the n -th time series f n ( t ) . Note that

Choosing L = √ NT , and using σ k ( Z f ) 2 ≥ γ 2 NT k , we get,

Finally and setting k = RG concludes the proof.

D Proof of Theorem 4.2

In this section, we prove Theorem 4.2, which bounds the estimation error ∥ ̂ α n -α n ∥ 2 ∀ n ∈ [ N ] . First, recall the OLS estimate ̂ α n = [ ̂ α n, 1 , ̂ α n, 2 , . . . ̂ α n,p ] defined as

where ̂ X n ( t ) := [ ̂ x n ( t ) , . . . , ̂ x n ( t -p +1)] . Alternatively, the OLS estimate can be defined as,

where ̂ X n ∈ R ( T -p ) × p is the row-wise concatenation of ̂ X n ( t ) for t ∈ { p, p +1 , . . . , T -1 } , and ̂ Y n = [ ̂ x n ( p +1) , . . . , ̂ x n ( T )] . Further, consider the OLS estimate with access to the true AR( p ) series x n ( · ) . Precisely, let ¯ α n defined as

where X n ∈ R ( T -p ) × p is the row-wise concatenation of X n ( t ) := [ x n ( t ) , . . . , x n ( t -p +1)] for t ∈ { p, p +1 , . . . , T -1 } , and Y n = [ x n ( p +1) , . . . , x n ( T )] .

In what follows, as we do the analysis for one process x n , we drop the subscript n . We note that the analysis holds for all n ∈ [ N ] . First, note that,

To bound ∥ ¯ α -α ∥ 2 2 , we use Corollary E.2 which states that with probability 1 -c T 11 ,

Next, to bound ∥ ̂ α -¯ α ∥ 2 2 , let M † := ( M ⊤ M ) -1 M ⊤ denote the Moore-Penrose inverse of a matrix M . Further, let δ ( t ) = x ( t ) -̂ x ( t ) = ̂ f ( t ) -f ( t ) , and ∆ = X -̂ X , i.e., ∆ is the row-wise concatenation of ∆( t ) := [ δ ( t ) , . . . , δ ( t -p + 1)] ⊤ for t ∈ { p, p + 1 , . . . , T -1 } . Then, with ∆ Y := [ δ ( p +1) , . . . , δ ( T )] ,we have

Where Lemma H.6 is used in the last inequality. Note that,

Using (33), (34), (35), and the theorem conditions, we get

Using Corollary E.1 we get that with probability 1 -c T 11

Now, note that ∥ ∆ Y ∥ 2 2 = ∑ T t = p +1 ( ̂ f ( t ) -f ( t )) 2 , and

Recall that ∑ T t =1 ( ̂ f ( t ) -f ( t )) 2 = T EstErr ( N,T,n ) , which we will refer to herein as EstErr ( N,T ) as we drop the dependence on n . Therefore, we have,

Finally, using Lemma H.4, with probability at least 1 -c T 11 it holds for an absolute constant C > 4 ,

First, note that, using (37) and (39), we have

for sufficiently small EstErr ( N,T ) such that EstErr ( N,T ) ≤ σ 2 λ min (Ψ) 6 p . Now using (36), (37), (39), (40), (41), and (42) we have,

Therefore,

finally, using (31), (32), and (43), we get with probability 1 -c T 11 ,

which completes the proof of Theorem 4.2.

E Identification of AR processes

In this section, we state key theorems for the identification of AR processes without the perturbation we consider in this paper. The theorems in this section and their proofs follow closely the ones in [17], we modify them to accommodate general sub-gaussian noise and state them fully for completeness.

First, consider the stationary AR( p ) process defined by the transition matrix

where 0 p -1 is the column vector of p -1 zeroes, and input vector

Then, with X ( t ) := [ x ( t ) , . . . , x ( t -p +1)] , the AR process is described as

With η t being a sub-gaussian random variable with variance proxy σ 2 With this setup, we define two important matrices,

where Ψ is known as the controllability Gramian .

Theorem E.1. Consider the AR( p ) process described in (47) , { η t } is an i.i.d. mean-zero subgaussian noise with variance σ 2 . Given 0 < ϵ ≤ 1 , define the following quantities:

where c and ˜ c are constants that depends only on the process coefficients α 1 , . . . , α p . Then, for T > T 0 ( ϵ ) and all valid values of ϵ such that V ℓ ≻ 0 , we have

Remark E.1. To get V ℓ ≻ 0 , one must choose a sufficiently small ϵ such that (Ψ -ϵ Γ) ≻ 0 . Using Weyl's inequality, we have

This suggest that ϵ < λ min (Ψ) λ max (Γ) yields V ℓ ≻ 0 . For example, a choice of ϵ = λ min (Ψ) 2 λ max (Γ) will yield V ℓ ⪰ 1 2 σ 2 ( T -p ) λ min (Ψ) I p . Note that this ensures ϵ ≤ 1 / 2 , since ϵ ≤ λ max (Ψ) λ max (Γ) and Γ ⪰ Ψ as can be seen from

Proof. First, write

Expanding the summand using the process dynamics yields

Now, defining V T = X ⊤ X / ( T -p ) , we can sum the expression above over t = p -1 , . . . , T -2 to get

This is called a Lyapunov equation, and due to the stability of the process it has solution V T = ∑ ∞ t =0 A t E T ( A ⊤ ) t . The key step is to argue that E T converges to σ 2 BB ⊤ with high probability, which will allow us to show that V T converges to σ 2 Ψ . To do this, we will split the terms in E T into three groups, handling them separately. For some ϵ > 0 , we will bound the probability of the following events:

where ρ ( · ) denotes the spectral radius of a matrix.

Lemma E.1. With the setup above, it holds that

for some absolute constant c > 0 .

Proof. Note that

so, considering the fact that X p -1 and X T -1 are identically distributed by stationarity, it suffices to upper bound the probability of

First, recall that X p -1 is a sub-gaussian vector with variance proxy σ 2 γ 2 (1 -λ ∗ ) 2 := σ 2 x (see Lemma A.2). Thus, using Lemma H.2, for

as long as

where in the last inequality, we used the assumption T > T 0 ( ϵ ) > 75 2 p 2 . Finally, note that ∥ A ∥ 2 ≤ Tr( A ) = ∥ α ∥ 2 2 + p -1 ≤ p ( ∥ α ∥ 2 2 +1) . Recalling that P ( E 1 ) > 1 -2 P ( E ′ ) concludes the proof.

Lemma E.2. Continuing with the setup above, it holds that

Proof. Since the spectral radius of BB ⊤ is unity, E 2 is equivalent to the complement of the event

Since η t +1 is sub-gaussian with variance proxy σ 2 , η 2 t +1 -σ 2 is sub-exponential with parameter 16 σ 2 . Thus, Lemma H.5 applies, and therefore,

Lemma E.3. With the same setup, we have

Proof. Recall the definition

For the sake of a variational analysis of the spectral radius, consider any vector q = [ q 1 ˜ q ⊤ ] ⊤ ∈ R p such that ∥ q ∥ 2 = 1 . For notational simplicity. Then

Next, we aim to bound ∥ Mq ∥ 2 . First note that only the first column of M is nonzero which evaluates to

Therefore, we have Mq = q 1 M · , 1 , which implies that for any unit vector q ,

Therefore, we will bound P ( E c 3 ) by

First note that each term P ( | M i, 1 | ≥ ϵσ 2 ( T -p ) 6 √ p ) can be upper bounded by 2 P ( ∑ T -1 t = p z ⊤ i X t -1 η t ≥ ϵσ 2 ( T -p ) 6 √ p ) where z 1 = α and z i = e i -1 for i > 1 where e i is the i -th standard basis vector. Using Lemma H.7 and choosing Z t = z ⊤ i X t -1 and W t = η t , we have that

and thus

What is left is to bound P ( ∑ T -2 t =0 x 2 t > λ p ∥ z ∥ 2 2 ) . Using Lemma H.2, we get,

Choosing κ = ϵσ 2 ( T -p ) / (6 √ p ) and λ = ∥ z ∥ 2 2 ϵσ 2 T 3 / 2 yields

Which implies, that for T > 50 p , we have,

for some absolute constants c, C > 0 . Now, note that

Thus, using (55) and (53), we have

Using the condition T > T 0 ( ϵ ) concludes the proof.

Now we argue that the sum of the probabilities deduced from lemmas (E.1), (E.2), and (E.3) can be expressed in the simple form of δ ( ϵ, T ) as defined above. First, from lemma (E.1) we have

From lemma (E.2), and T > T 0 ( ϵ ) > 50 p we get

Lastly, lemma (E.3) says, for T ≥ T 0 ( ϵ ) ,

Thus we can write

Where in the statement we set ˜ c := 1 75 ( 2+ ∥ α ∥ 2 2 ) σ 2 σ 2 x . Using lemmas (E.1), (E.2), and (E.3) and the subadditivity of the spectral radii of symmetric matrices, we get with probability at least 1 -δ ( ϵ, T ) that

Finally, we establish the following corollary, which is an immediate consequence of Theorem E.1. Corollary E.1. Consider the AR( p ) process x t described in (A.1) . Let X ∈ R ( T -p ) × p such that [ X ] ij = x i + j -1 . Then, for sufficiently large T such that log( T ) √ T < ˜ c 22 p ( λ min (Ψ) λ max (Γ) ) , we have that with probability 1 -c T 11 ,

where c , C are constants and ˜ c := 1 75 ( 2+ ∥ α ∥ 2 2 ) σ 2 σ 2 x , ψ (the controllability Gramian) and Γ are defined in (14) .

Proof. Note that according to Theorem E.1, Weyl's inequality (see Lemma H.3), and using ϵ = 11 p log( T ) ˜ c √ T we have, with probability 1 -c T D ,

What is left is to check the condition on T and ϵ . Note that Theorem E.1 requires T > 75 2 σ 4 x p 2 log( p ) 2 σ 4 ϵ 2 max { 1 , ∥ α ∥ 4 2 } , which with our choice of ϵ implies T > p . To see that the choice of ϵ is valid, note that 11 p log( T ) ˜ c √ T ≤ λ min (Ψ) 2 λ max (Γ) , which validates the choice of ϵ .

Also,

Theorem E.2. Consider the AR( p ) process described in (47) . where { η t } is an i.i.d. mean-zero sub-gaussian noise with variance σ 2 . Let α := [ α 1 , . . . , α p ] ⊤ and ¯ α be its least squares estimator. If T ≥ T 0 ( ϵ ) , then for any unit vector w ∈ S p ,

where V ℓ , V u T 0 ( ϵ ) , and δ ( ϵ, T ) are as described in Theorem E.1.

Proof. Write E = [ η p +1 , . . . , η T ] ⊤ . Using Cauchy-Schwarz, we have

Now we will need to introduce a lemma from [1]:

Lemma E.4. Let {F t } ∞ t =0 be a filtration. Let { η t } ∞ t =1 be a real-valued stochastic process such that η t is F t -measurable and is conditionally sub-gaussian with variance proxy σ 2 > 0 . Further, let { X t } ∞ t =0 be an R d -valued stochastic process such that X t is F t -1 -measurable. Consider any positive definite matrix V ∈ R d × d . For any t ≥ 1 , define

Then for any δ > 0 , with probability at least 1 -δ for all t ≥ 1 ,

We will apply this result to bound the second term above. Choosing V = V ℓ , ∑ t s =1 X s X ⊤ s = X ⊤ X , and S t = X ⊤ E , we have with probability at least 1 -δ ( ϵ, T )

Now further condition on the event of Theorem E.1, which occur together with the condition above with probability at least 1 -2 δ ( ϵ, T ) . Recall, then, that X ⊤ X + V ℓ ⪯ 2 X ⊤ X , and so ( X ⊤ X + V ℓ ) -1 ⪰ 1 2 ( X ⊤ X ) -1 . This means that

Finally, it is clear that under the conditions in Theorem E.1 we have ∥ ∥ w ⊤ ( X ⊤ X ) -1 / 2 ∥ ∥ 2 ≤ ∥ ∥ ∥ w ⊤ V -1 / 2 ℓ ∥ ∥ ∥ 2 .

Corollary E.2. Let ϵ = 12 p log( T ) ˜ c √ T , where ˜ c is as defined in Theorem (E.1) . Then, for T such that log( T ) √ T < ˜ c 24 p ( λ min (Ψ) λ max (Γ) ) we have

Proof. First, note that with the choice of ϵ = 12 p log( T ) ˜ c √ T we have,

Hence, where the last inequality holds with probability 1 -δ ( ϵ, T ) using Theorem E.1. Finally, consider

Note that, again by our assumed condition on ϵ ,

Using (69) and (68), we have,

From (70), (67) and (66), we have,

Using (71) and Theorem E.2, we get that with probability 1 -2 c T 12

For any i ∈ [ p ] . This implies,

with probability 1 -c T 11 which completes the proof.

F Proof of Theorem 4.3

F.1 Setup

Recall β ∗ as defined in Proposition 2.2, and its estimate ̂ β defined as

To establish this bound, we first define (and recall) the following notations,

Recall that Z y = [ Z ( y 1 , L, T, 1) Z ( y 2 , L, T, 1) . . . Z ( y n , L, T, 1)] is the stacked Page matrix of y 1 ( t ) , . . . , y n ( t ) ∀ t ∈ [ T ] .

Let Z x be the stacked Page matrix of x 1 ( t ) , . . . , x n ( t ) ∀ t ∈ [ T ] .

Let Z f be the stacked Page matrix of f 1 ( t ) , . . . , f n ( t ) ∀ t ∈ [ T ] . That is, Z y = Z f + Z x .

Let Z ′ y ∈ R ( L -1) x ( NT/L ) , be the sub-matrix obtained by dropping the L -th row from the stacked Page matrix Z y . Define Z ′ f and Z ′ x analogously.

Let UΣV ⊤ denote the SVD of Z ′ f .

let V ⊥ and U ⊥ be matrices of orthonormal basis vectors that span the null space of Z ′ f and Z ′ ⊤ f , respectively.

Let ˜ U ˜ Σ ˜ V ⊤ denote the top k singular components of the SVD of Z ′ y , while ˜ U ⊥ ˜ Σ ⊥ ( ˜ V ⊥ ) ⊤ denote the remaining L -k -1 components such that Z ′ y = ˜ U ˜ Σ ˜ V ⊤ + ˜ U ⊥ ˜ Σ ⊥ ( ˜ V ⊥ ) ⊤ .

Let ̂ Z ′ f be the HSVT estimate of Z ′ f with parameter k . That is ̂ Z ′ f = ˜ U ˜ Σ ˜ V ⊤ .

F.2 Deterministic Bound

First, given the notation above, the solution for (74), can be written as,

Then, let's consider the following expansion for ∥ ̂ β -β ∗ ∥ 2 ,

where the last equality follow from the fact that ( ˜ U ⊥ ) ⊤ ̂ β = ( ( ˜ U ⊥ ) ⊤ ˜ U ) ˜ Σ -1 ˜ V ⊤ [ Z y ] L · = 0 . Now, consider the first term in (76), and note that β ∗ = UU ⊤ β ∗ ,

Next, by Wedin sin Θ Theorem (see [10, 32]) we have:

What is left is bounding ∥ ˜ U ⊤ ( ̂ β -β ∗ ) ∥ 2 2 . To that end, first consider

Also, consider

That is,

From (81) and (80) we get,

First, recall that [ Z y ] L · = [ Z f ] L · +[ Z x ] L · , and that by definition, ( Z ′ f ) ⊤ β ∗ = [ Z f ] L · . Then the term ∥ ( Z ′ ) ⊤ β -( Z ′ ) ⊤ β ∗ ∥ 2 can be bounded as follows. First, consider

Note that ( ̂ Z ′ f ) ⊤ ̂ β = ˜ V ˜ Σ ˜ U ⊤ ̂ β = ˜ V ˜ Σ ˜ U ⊤ ˜ U ˜ Σ † ˜ V ⊤ [ Z y ] L · = ˜ V ˜ V ⊤ [ Z y ] L · , and ( ̂ Z ′ f ) ⊤ β ∗ = ˜ V ˜ Σ ˜ U ⊤ β ∗ , thus,

Finally, from (82) and (83) we get

The term ∥ ∥ ∥ [ Z x ] ⊤ L · ˜ V ∥ ∥ ∥ 2 2 can be further decomposed as

Using Wedin sin Θ Theorem,

Therefore,

From (76), (79), and (85) we have

F.3 High probability bound

Let q := NT/L be the number of columns in the stacked page matrix ̂ Z ′ f , and let C > 0 be some positive absolute constant. Define

Lemma F.1. For some positive constant c 1 > 0 and C > 0 large enough in definitions of E ′ 1 , E ′ 2 , E ′ 3 and E ′ 4 ,

Proof. We bound the probability of events above below.

Bounding E ′ 1 and E ′ 2 . Note that Z ′ x is a sub-matrix of Z x , and its operator norm is bounded by that of Z x . Hence, as the probability is an immediate consequence of Lemma A.2. The second event probability follow from the fact that ∥ [ Z x ] L · ∥ 2 ≤ ∥ Z x ∥ 2 .

Bounding E ′ 3 . This is immediate from Corollary C.1 and its consequence in (27).

Bounding E ′ 4 . First, recall that [ Z x ] L · is a q -dimensional sub-gaussian vector with variance proxy σ 2 x . That is, for any unit vector v ∈ S q -1 , [ Z x ] L · v ∼ subG( σ 2 x ) . Now, recall that V ∈ R q × k is a set of k orthonormal vectors V i , i ∈ [ k ] . Let Z i = [ Z x ] L · V i . Clearly, Z i ∼ subG( σ 2 x ) . Therefore, [ Z x ] L · V is a k -dimensional vector with dependent sub-gaussian entries with variance proxy σ 2 x . Using Lemma H.4, we have

Therefore, for choice of t = Cσ x √ k log( q ) with large enough constant C ≥ 19 , we have,

Recalling that q > L , and q = NT/L , concludes the proof.

Let E ′ := E ′ 1 ∩ E ′ 2 ∩ E ′ 3 ∩ E ′ 4 . Then, under event E , and using (86), we have, with probability 1 -c ( NT ) 11 ,

Choosing L = √ NT , and k = RG , recalling Assumption 4.1, and using Weyl's lemma (H.3) to bound | σ k ( ̂ Z ′ f ) 2 -σ k ( Z ′ f ) 2 | we get,

which concludes the proof of Theorem 4.3.

G Proof of Theorem 4.4

In this section, we prove an upper bound on the forecasting error,

where ¯ y n ( t ) = E [ y n ( t ) | y n (1) , . . . , y n ( t -1)] = f n ( t ) + ¯ x n ( t ) , where ¯ x n ( t ) := E [ x n ( t ) | x n (1) , . . . , x n ( t -1)] . First, recall that for t > T ,

where Y n ( t -1) = [ y n ( t -( L -1)) , . . . , y n ( t -1)] , and ˜ X n ( t -1) = [ ˜ x n ( t -p ) , . . . , ˜ x n ( t -1)] . Then, we have,

Now, we will bound each error term separately.

G.1 Bounding Forecasting Error of f

To bound the forecasting error of f , we use the following Lemma.

Lemma G.1. Let the conditions of Theorem 4.1 hold. Then, with probability 1 -c ( NT ) 11

where c is an absolute constant, and C ( f max , γ ) denotes a constant that depends only (polynomially) on model parameters f max , γ .

Proof. The proof of this lemma is similar to that in [3], but we adapt it for our different settings. First, note that, where we assume that NT/L is an integer,

In what follows, we provide an upper bound for the inner sum L NT ∑ N n =1 ∑ T/L -1 m =0 ( ̂ f n ( T + mL +

ℓ ) -f n ( T + mL + ℓ ) ) 2 for all ℓ ∈ [ L ] . Next, we show, without loss of generality, the case for ℓ = 1 . To establish this bound, we first define (and recall) the following notations,

Recall that Z y = [ Z ( y 1 , L, T ) Z ( y 2 , L, T ) . . . Z ( y n , L, T )] is the stacked Page matrix of y 1 ( t ) , . . . , y n ( t ) for t ∈ [ T ] .

Similarly, let Z y,out denote the stacked Page matrix of the out-of-sample observations y 1 ( t ) , . . . , y n ( t ) for t ∈ { T -L +2 , . . . , 2 T -L +1 } , i.e., Z y,out ∈ R L × NT/L .

Let Z x be the stacked Page matrix of x 1 ( t ) , . . . , x n ( t ) for t ∈ [ T ] , and Z x,out be the stacked Page matrix of x 1 ( t ) , . . . , x n ( t ) for t ∈ { T -L +2 , . . . , 2 T -L +1 } .

Similarly, let Z f and Z f,out be defined similarly but for f 1 ( t ) , . . . , f n ( t ) .

Let Z ′ y ∈ R ( L -1) × ( NT/L ) , be the a sub-matrix obtained by dropping the L -th row from the stacked Page matrix Z y . Define Z ′ y,out , Z ′ f , Z ′ f,out , Z ′ x , and Z ′ x,out analogously.

Let UΣV ⊤ and U out Σ out V out ⊤ denote the SVD of Z ′ f and Z ′ f,out respectively.

let V ⊥ and U ⊥ be matrices of orthonormal basis vectors that span the null space of Z ′ f and Z ′ ⊤ f , respectively. Define V out ⊥ and U ⊥ out similarly for the matrix Z ′ f,out .

Let ˜ U ˜ Σ ˜ V ⊤ denote the top k singular components of the SVD of Z ′ y , while ˜ U ⊥ ˜ Σ ⊥ ( ˜ V ⊥ ) ⊤ denote the remaining L -k -1 components such that Z ′ y = ˜ U ˜ Σ ˜ V ⊤ + ˜ U ⊥ ˜ Σ ⊥ ( ˜ V ⊥ ) ⊤ .

Similarly, Let ˜ U out ˜ Σ out ˜ V ⊤ out denote the top k singular components of the SVD of Z ′ y,out , while ˜ U ⊥ out ˜ Σ ⊥ out ( ˜ V ⊥ out ) ⊤ denote the remaining L -k -1 components such that Z ′ y,out = ˜ U out ˜ Σ out ˜ V ⊤ out + ˜ U ⊥ out ˜ Σ ⊥ out ( ˜ V ⊥ out ) ⊤ .

Let ̂ Z f = ˜ U ˜ Σ ˜ V ⊤ and ̂ Z f,out = ˜ U out ˜ Σ out ˜ V ⊤ out .

Recall that we are interested in a high-probability bound for the following out-of-sample prediction error:

Recall that ̂ f n ( t ) = ̂ β ⊤ Y n ( t -1) and hence we can rewrite (87) as

Next, we derive a deterministic upper found for the expression above.

Deterministic Bound. Through adding and subtracting ̂ Z ⊤ f,out ̂ β and triangle inequality, we have

Next, we proceed to bound each of the two terms on the right hand side.

First term: ∥ ∥ ∥ ( Z ′ y,out -̂ Z f,out ) ⊤ ̂ β ∥ ∥ ∥ 2

Note that ∥ ˜ Σ ⊥ out ∥ 2 equals the ( k + 1) -th singular value of Z ′ y,out . Using Weyl's inequality (see Lemma H.3), and recalling that Z ′ y,out = Z ′ f,out + Z ′ x,out , and that the rank of Z ′ f,out is k , we can bound the ( k +1) -th singular value of Z ′ y,out by the operator norm of Z ′ x,out . That is,

Next, we bound the term ∥ ( ˜ U ⊥ out ) ⊤ ̂ β ∥ 2 2 .

Further expanding the first term,

Where in the first equality we use the fact that β ∗ = U out ( U out ) ⊤ β ∗ , i.e., β ∗ lives in the column space of Z ′ f,out (Assumption 4.2).

Next, by Wedin sin Θ Theorem (see [10, 32]) we bound ∥ ∥ ∥ ˜ U out ˜ U ⊤ out -U out U out ⊤ ∥ ∥ ∥ 2 as follows: ∥ ∥ ∥ ˜ U out ˜ U ⊤ out -U out U out ⊤ ∥ ∥ ∥ 2 ∥ β ∗ ∥ 2 ≤ ∥ Z x,out ∥ 2 σ k ( Z f,out ) ∥ β ∗ ∥ 2 (94)

Using this definition, (91), (92), (93), and (94) we have

Second term: ∥ ∥ ∥ ̂ Z ⊤ f,out ̂ β -( Z ′ f,out ) ⊤ β ∗ ∥ ∥ ∥ 2 . To bound the second term, we follow a similar proof to that shown in [5].

Next, we bound the term ∥ ̂ Z ⊤ f,out ( ̂ β -β ∗ ) ∥ 2 2 .

Next, we bound ∥ Z ′ f,out ⊤ ( ̂ β -β ∗ ) ∥ 2 2 . Recall that U spans the column space of Z ′ f,out . Thus Z ′ f,out ⊤ = Z ′ f,out ⊤ UU ⊤ , therefore,

Now recall that we denote the top k left singular vectors of Z ′ y by ˜ U , then consider

Recall that, as we prove in (85) in Appendix F,

Using (101), (102) and Wedin sin Θ Theorem, we obtain,

Therefore, using (96), (100), and (103), we have

Combining . Incorporating the two bounds in (95) and (104) yields,

For some absolute constant c > 0 .

High probability bound. With our choice L = √ NT , Let q := √ NT be the number of columns in the stacked page matrix Z y and subsequently q +1 to be the number of columns in the stacked page matrix Z y,out . Let C > 0 be some positive absolute constant, and C ( f max , γ ) be a constant that

depends only on model parameters f max and γ . Define

First, with c 1 > 0 is an absolute constant, recall from Lemma F.1,

Further, note that Z ′ x,out is a sub-matrix of Z x,out , hence its operator norm is bounded by that of Z x,out . Therefore, as an immediate consequence of Lemma A.2,

The probability of event ¯ E 3 is bounded by Theorem 4.3 as

Finally, as an immediate results of Corollary C.1 and its consequence in (27), we have

Now consider the event ¯ E := ¯ E 1 ∩ ¯ E 2 ∩ ¯ E 3 ∩ ¯ E 4 ∩ ¯ E 5 ∩ ¯ E 6 ∩ ¯ E 7 . Then, under event ¯ E , and using (105), we have, with probability 1 -c ( NT ) 11 ,

Recalling q = √ NT , k = RG , and Assumption 4.1, we get,

Therefore, with probability 1 -c/ ( NT ) 11 ,

Further, note that the same argument can be used to bound the sum L NT ∑ N n =1 ∑ T/L -1 m =0 ( ̂ f n ( T + mL + ℓ ) -f n ( T + mL + ℓ ) ) 2 for all ℓ ∈ [ L ] . Thus, with probability 1 -cL/ ( NT ) 11 > 1 -c/ ( NT ) 10 , we have,

G.2 Bounding Forecasting Error of x

Now, we bound the error of forecasting x n ( t ) . First, recall ˜ x n ( t ) = y n ( t ) -̂ f n ( t ) , and let ˜ X n ( t ) := [ ˜ x n ( t ) , . . . , ˜ x n ( t -p +1)] and X n ( t ) := [ x n ( t ) , . . . , x n ( t -p +1)] . Now, let X n , ˜ X n and ∆ n , all in R ( T -p ) × p , be the row-wise concatenations of X n ( t ) , ˜ X n ( t ) , and ∆ n ( t ) := X n ( t ) -̂ X n ( t ) for t ∈ { T + p, T + L, . . . , 2 T + p -1 } .

Then, we have, where ¯ x n ( t ) = E [ x n ( t ) | x n (1) , . . . , x n ( t -1)]

First note that

Therefore,

High probability bound. For the high probability bound, consider the event ˜ E = ˜ E 1 ∩ ˜ E 2 ∩ ˜ E 3 where

Note that P ( ˜ E 1 ) ≥ 1 -CN T 11 using Theorems 4.2, 4.1 and the the union bound. Further, note that P ( ˜ E 2 ) ≥ 1 -C ( NT ) 10 using both Lemma G.1 and Theorem 4.1. Further, note that according to Corollary E.1, with probability 1 -c T 11 , we have

Therefore, under ˜ E , and recalling N < T , we have with probability 1 -C T 10 ,

where α max = max n ∥ α n ∥ 2 therefore, with probability 1 -C T 10 , we have

which completes the proof.

H Helper Lemmas and Definitions

Definition H.1 (Sub-gaussian random variable [30, 31]) . A zero-mean random variable X ∈ R is said to be sub-gaussian with variance proxy σ 2 (denoted as X ∼ subG ( σ 2 ) ) if its moment generating function satisfies

Definition H.2 (Sub-exponential random variable [30, 31]) . A zero-mean random variable X ∈ R is said to be sub-exponential with parameter ν (denoted as X ∼ subE ( ν ) ) if its moment generating function satisfies

Lemma H.1 ([31, 30, 6]) . Let X ∼ subG ( σ 2 s ) and Z ∼ N (0 , σ 2 g ) . For i ∈ [ N ] , let Y i ∼ subE ( ν i ) . Then:

If Y i are independent:

Definition H.3 (Sub-gaussian vector) . A random vector X ∈ R d is a sub-gaussian random vector with parameter σ 2 if

Lemma H.2 (Norm of sub-gaussian vector) . Let X ∈ R d be a sub-gaussian random vector with parameter σ 2 . Then, with probability at least 1 -δ for δ ∈ (0 , 1) :

This implies that for any t > 2 σ ,

with probability at least 1 -exp( -t 2 16 dσ 2 ) .

Definition H.4 (Stationary process [26]) . A process x ( t ) ∈ R is said to be stationary if its expectation does not depend on t and its autocovariance function depend only on the time difference. That is,

Lemma H.3 ( Weyl's inequality). Given A , B ∈ R m × n , let σ i and ̂ σ i be the i -th singular values of A and B , respectively, in decreasing order and repeated by multiplicities. Then for all i ∈ [ m ∧ n ] ,

Lemma H.4. Let X = [ X 1 , . . . , X n ] where each X i , i ∈ [ n ] is a sub-gaussian (not necessarily independent) random variable with variance proxy σ 2 and E [ X 2 i ] = γ . Then with probability at least 1 -c exp ( -t 2 16 nσ 2 ) ,

Proof. From Lemma H.1, we can see that ∥ X ∥ 2 2 -nγ = ∑ n i =1 x 2 i -nγ is sub-exponential with parameter 16 nσ 2 . Let d := ∑ n i =1 x 2 i -nγ , then

Choosing t ′ = t 2 -nγ , ν = 16 nσ 2 , and s = 1 16 nσ 2 , we get,

where the last inequality follows from the fact the second moment of x 1 is bounded by its variance proxy up to a multiplicative constant.

Lemma H.5. Let X 1 , . . . , X n be i.i.d. sub-exponential random variables with parameter ν . Let S = ∑ n i =1 X i . Then

Proof. Recall that the sum of n i.i.d. sub-exponential random variables is also a x sub-exponential random variable with parameter ν √ n (see Lemma H.1). Thus, using the definition of sub-exponential random variable, we have

Choosing λ = 1 √ nν , we get,

Applying the same inequality to -S

completes the proof.

Lemma H.6. [[33]] Let A ∈ R m × n and B = A + E . Then

where M † := ( M ⊤ M ) -1 M ⊤ denote the Moore-Penrose inverse of M .

Lemma H.7. [Lemma 4.2 of [27]] Let {F t } t ≥ 0 be a filtration, and { Z t } t ≥ 1 and { W t } t ≥ 1 be realvalued processes adapted to F t and F t +1 respectively. Moreover, assume W t |F t is mean zero and σ 2 -sub-gaussian. Then, for any positive real numbers α, β we have

Peer reviews:
comments: SUMMARY:
The authors propose SAMoSSA, an algorithm that combines deterministic trend estimation via mSSA with estimation of an autoregressive component of a time series.  They provide error rates for trend estimation, estimation of the AR coefficients, as well as the prediction error.   In addition, they consider real and simulated examples and show that the proposed method offers very competitive performance.

STRENGTHS:
- While the time series literature is vast and well-developed, it does appear that theoretical results that analyze the effect of trend estimation on AR estimates and the prediction error were quite limited.  Detrending a time series and then modeling using ARMA or VAR is a common routine so this is a practically important problem.  
- The derived error bounds appear to be novel.  
- The paper is well-written.  Assumptions and implications of theorems are for the most part explained well.

WEAKNESSES:
- The authors could do a more thorough literature review when it comes to inference for trends.  For example, there is a statistics literature on testing for trends. See Chen and Wu (2019): Testing for Trends in High-Dimensional Time Series and references therein.  
- While it is a nice result, the implications for real-world data are questionable due to the conditions on the trend as well as the assumed autoregressive structure.

QUESTIONS:
The more general ARMA model is first discussed in the introduction.  What challenges do estimating the MA components pose for the analysis?

LIMITATIONS:
The authors do acknowledge some limitations with the studied framework, including assuming stationarity of the stochastic component.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper proposed SAMoSSA, a two-stage procedure that effectively handles mixtures of deterministic nonstationary and stationary AR processes with minimal model assumptions. The authors analyze SAMoSSA’s ability to estimate non-stationary components under stationary AR noise, the error rate of AR system identiﬁcation via OLS under observation errors, and a ﬁnite-sample forecast error analysis.

STRENGTHS:
1. This paper is well-written and easy to follow.
2. The theoretical results are solid.

WEAKNESSES:
1. The theoretical contributions are somewhat incremental.
2. The experiment are insufficient.

QUESTIONS:
1. The biggest difference between this paper and [3] lies in the different noise setting, i.e., this paper replaces the i.i.d noise with stationary AR noise. However, the theoretical derivation in this article is almost the same as it in [3]. What technical difficulties did the authors encounter during the proof process? 

2. How limited are  Assumptions 2.1 and 4.2 in practice?

3. In experiment, more baselines are needed. 

4. How to verify that the dataset (especially the real dataset) satisfies the assumptions?

LIMITATIONS:
Not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage approach based on multivariate Singular Spectrum Analysis (mSSA) to  estimate the non-stationary components in a time series in the presence of a correlated stationary AR noise, which is subsequently estimated from the residual time series. Theoretical results on the performance of the algorithm in this novel setting, are established along with a finite-sample forecasting consistency bound. Empirical results demonstrate significant improvements in forecasting performance due to identification of the AR noise structure, across various benchmark datasets.\n\nSTRENGTHS:\nmSSA allows estimation of non-stationary deterministic components without domain knowledge or fine-tuning, however, cannot handle additive correlated stationary noise. The paper deals with the important problem of estimating non-stationary deterministic components in the presence of correlated stationary noise, through a unified approach via mSSA. Theoretical results for the mSSA are established beyond the i.i.d noise setting (specifically with AR noise). The paper also provides theoretical results on out-of-sample forecasting error for the proposed two-step algorithm. Overall, I found the contribution to be sufficiently novel and of high quality; addressing a crucial gap in the literature.\n\nWEAKNESSES:\nOverall, the paper is well-written but is difficult to follow in some parts, and is lacking in conveying the key ideas. For example, presentation of the algorithm (Section 3): The details of the algorithm are presented well however, the main idea behind the approach is missing. A few sentences to convey the big picture behind the steps would have been very helpful.\n\nQUESTIONS:\nThe abbreviation SAMoSSA is used in the abstract and introduction without being explicitly defined (I can guess what it is through the title but not sure why the order is reversed). can you please state this explicitly?\n\nAssumption 2.1: Please provide some intuitive reasoning behind this assumption and what you mean by `fundamental' time series. \n\nIn Assumption 2.2, L<=T. Why does it become L<=sqrt(T) in line 168 on p.4?\n\nSection 3: The paper mentions that the proposed algorithm builds on the work of A. Agarwal, et al. [3]. It is not clear how the algorithm in Section 3 is different from what exists in the literature -it would be helpful to state this clearly.\n\nLIMITATIONS:\nYes, limitations of the approach are included in the final section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a comprehensive work on a new extended variant of multivariate Singular Spectrum Analysis (mSSA), which manages to handle time series of deterministic trend/seasonality with AR stationary components, with rigorous theoretical guarantees. The algorithm is a natural extension of the variant of mSSA using the Page matrix. It further provides guarantees on the bound of estimation error of non-stationary components, the bound of AR parameter estimation error, and the bound of out-of-sample forecasting error. And its superior performance is shown in experiments on four datasets.\n\nSTRENGTHS:\nThe extension of the algorithm naturally follows by adding the second step of AR estimation. It is impressive that this paper manages to show all these three bounds on estimation/forecasting errors. What is surprising that, for such a two-stage approach, the bound of out-of-sample forecasting error could be attained.\n\nWEAKNESSES:\nThe numerical experiments could be richer to show the performance boundary of the proposed method, considering it is dealing with a general signal as the non-stationary plus the AR components. The types of non-stationary components may affects the performance; in particular, instead of the complicated one, the simple deterministic components that could be also be captured by (close to marginal stable) AR parameters may raise issues. And the signal-noise-ratio, the intensity of AR process compared to the deterministic, etc. may also be tested in experiments. All these simulation may help us to further understand the strengths and weaknesses of the proposed method. The reason why the reviewer expects more numerical testing comes from the concerns on the power of truncated SVD of Page matrix for approximately splitting the deterministic/non-stationary and the AR components. If this step fails completely, the whole would fail. We like to know its performance boundary and reliability. \n\nAnother concern may be taken into account by area chair. This paper is more typical in the fields of econometrics or statistics. The reviewer is not sure if it fits NeurIPS well.\n\nQUESTIONS:\nBesides the questions in the \"Weaknesses\" part, there could be more deserving the authors' attentions.\n\n1. Is it correct that we consider such a simplified multivariate time series modelling problem that each channel/variable (i.e. y_n) is driven an independent AR process (x_n)? Together with Assumption 2.1, we actually can model each channel signal y_n separately. This also allows the stack of Page matrix for an extension from the univariate to the multivariate.\n\n2. Following the previous question, the analysis for SAMOSSA for multivariate case has nothing essentially different from the univariate case, right? Maybe we missed some technical challenges.\n\n3. In the proof of out-of-sample forecasting error bound, what is the particular issue that our problem has compared to the analysis in [3]?\n\n4. For time-series forecasting, there has been plenty of SOTA methods using deep learning (like Informer, PatchTST, etc.). We are curious about the comparative results. Indeed yours may be less accurate than DL SOTA, while yours has additional explainability. The performance gap might tell the cost for explainable modelling.\n\nLIMITATIONS:\nWe do not see any potential negative societal impact of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends previous work on multivariate Singular Spectrum Analysis (mSSA) to observations with autoregressive (AR) noise. The method constructs a sliding window representation of the target univariate or multivariate time series called the Page matrix and learns the deterministic non-stationary component (with a linear model) from the singular value decomposition of this representation. The model then produces forecasts by first estimating the parameters of a linear model for the deterministic component followed by the AR parameter estimates of the AR model in the second step. The authors provide theoretical bounds for the error estimates under AR noise of this method, the AR model parameters under perturbation, and the out-of-sample forecasting error. Finally, the authors compare the performance on their method with other time series models (including the mSSA method that is the basis for the presented algorithm.\n\nSTRENGTHS:\nOriginality\n\nFrom an algorithmic point of view, the proposed method is a straightforward extension of the previously developed mSSA model to incorporate autoregressive noise and use a two-step model to learn it’s parameters (and the parameters of the deterministic part). Of course, I think the main contribution lies in the theoretical error bound results but I am not familiar enough with the theory to judge the originality on these and will leave this to other reviewers.\n\n\nQuality\n\nAgain, I am not an expert on the underlying theory but so I cannot judge on the correctness of the theoretical results. The presented method is an interesting approach to time series forecasting and the theoretical guarantees are appealing. The empirical evaluation on the synthetic results confirm the established error bounds on that synthetic example and the paper also contains a (albeit brief) empirical evaluation with different baselines. \n\nClarity\n\nThe paper and its contribution are clearly written. However, the paper is very much written from the perspective Singular Spectrum Analysis point-of-view. I would have appreciated more background and discussion of related approaches in classical time series literature (which I will go into detail in Weaknesses). \n\nSignificance\n\nTime series methods that perform well in practice and also have provable error bounds are not very common in the recent ML literature that has focused mostly on deep learning models that lack these guarantees. As such, the method presented in this paper is significant, particularly for high-stakes applications where provable error bounds are required.\n\nWEAKNESSES:\nThe paper is brief on related work, especially on related approaches in classical time series forecasting (STL decomposition, simple exponential smoothing (SES) with drift). I think it would be useful to the reader to contrast the presented method to these approaches. Another class of related models are matrix factorization models that are not discussed either. Again, I think it would be useful to the reader to discuss this class of methods. \n\nI understand that the main focus on the paper is on the derivation of the theoretical results and I appreciate the effort here. However, the quantitative evaluation is very brief and does not give much insight into the performance of the method relative to either closely related models (other than mSSA) or modern deep learning methods. \n\nI would appreciate if the authors would at least compare their method that is conceptually or algorithmically similar. To this end, I would propose to compare against SES with drift and at least one matrix factorization method for forecasting (TRMF for example: https://dl.acm.org/doi/abs/10.5555/3157096.3157191). A comparison with deep learning baselines such as DeepAR or Fedformer (https://arxiv.org/abs/2201.12740) would also be interesting to understand how the method compares to currently used deep learning models. Simple baselines are also missing (seasonal naive for Traffic/Electricity; the naive method proposed in Bergmeir et al., KDD 2022 (https://link.springer.com/article/10.1007/s10618-022-00894-5) for Exchange).\n\nI would also appreciate more insight on the non-stationary deterministic part learned by SAMoSSA. I’m actually wondering on the added benefit of that part on the Electricity and Traffic datasets, which are quite stationary (but this might be up to debate).\n\nQUESTIONS:\nWhy is there such a large improvement from SAMoSSA over mSSA for Electricity? I would appreciate any insight on why the proposed method performs much better here.\n\nLIMITATIONS:\nLimitations are briefly discussed (and the discussion is sufficient in my opinion).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses a two-stage algorithm for time series analysis, which involves estimating deterministic, non-stationary trend and seasonality components, followed by learning the residual stochastic, stationary components. The first stage involves using multivariate Singular Spectrum Analysis (mSSA) to estimate the non-stationary components, even in the presence of a correlated stationary Autoregressive (AR) component. The AR component is then learned from the residual time series in the second stage. The authors provide a finite-sample forecasting consistency bound for SAMoSSA, which is data-driven and requires minimal parameter tuning. The paper also presents empirical studies that validate the superior performance of SAMoSSA compared to existing baselines. Notably, SAMoSSA's ability to account for AR noise structure yields improvements ranging from 5% to 37% across various benchmark datasets. The authors also provide a detailed explanation of the model and assumptions used in their analysis, including the spatial and temporal structure of the time series and the properties of the AR processes. They then describe the algorithm for the univariate case and explain how it decomposes the observations into estimates of the non-stationary and stationary components.\n\nSTRENGTHS:\n1.\tThis paper assesses the importance and novelty of the proposed SAMoSSA methodology and discusses the paper's originality in the use of mSSA and AR processes to jointly learn the deterministic non-stationary and stationary stochastic components of time series.\n2.\tThe paper addresses a gap in the literature, namely the theoretical underpinning of multi-stage learning algorithms involving deterministic and stationary components.\n3.\tThe results demonstrate significant performance improvements over existing baseline approaches.\n\nWEAKNESSES:\n1.\tThe complexity of the paper could be a potential barrier, making it difficult for people without theoretical interests in the area of time series analysis to understand. The paper could provide clearer intuitions and simplifications alongside the complex definitions and theorems to make it more approachable.\n2.\tThe empirical evaluation could be slightly limited. The variety of benchmark baselines used to demonstrate the efficacy of the method could be extended. The analysis of the results should be discussed more carefully, providing insight into why SAMoSSA outperforms other models. It is unclear whether the assumptions made in the paper hold in some of the typical practical scenarios like the electricity and health datasets.\n3.\tThe paper should discuss in more depth where these assumptions come from and under what conditions they might not hold. It should also explore how the method could perform if some of the assumptions are violated. The method proposed seems tailored for a specific kind of problem and it is unclear how generalizable it is.\n\nQUESTIONS:\nThe authors should provide simpler explanations or visual aids alongside the more complex mathematical definitions and proofs to make the paper more accessible. More extensive empirical evaluation should be carried out with a broader array of datasets to prove the efficacy of the method. The impact of the violation of some assumptions should be discussed, indicating potential limitations of the proposed method. The authors should provide more discussion on how this method could be generalized or modified to tackle different types of time series.\n\nLIMITATIONS:\nThe authors discuss but do not highlight the potential limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}