Change Point Detection via Multivariate Singular Spectrum Analysis

1 Introduction . The task of change point detection ( CPD ) is concerned with detecting significant changes in the temporal evolution of a system from noisy observations . This task has attracted considerable attention in the statistics and machine learning communities as it has a broad range of applications including quality control [ 5 ] , climate research [ 35 ] , and speech recognition [ 36 ] . This interest resulted in many methods and algorithmswe refer to [ 3 ] for a thorough review . Despite the relatively simple setup of the CPD problem , different variations of the problem have been studied and analyzed . We contrast two general settings characterized by the assumptions on the data generating process.The first is the classical view that aims to detect distribution changes from independent and identically distributed ( i.i.d . ) observations , cf . [ 34 ] and more recently [ 7 , 23 ] . The second deals with the more intricate time series dynamics by assuming the data to be dependent and following a specific temporal structure . Therein , the aim is to detect changes in this temporal structure of the observations . Some of the time series models considered include auto-regressive models [ 4 , 46 ] and state-space models [ 20 ] . An important model comes from the singular spectrum analysis ( SSA ) literature , where the data is assumed to follow an underlying function associated with a low-dimensional subspace . This time series model has inspired algorithms for different inference tasks in time series , including imputation , forecasting , and CPD [ 13 ] . The key idea behind using SSA for CPD is that the subspace spanned by the columns of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Hankel matrix constructed from an initial subsequence of the time series is approximately equivalent to the one spanned by the later subsequences of the time series . The theoretical analysis of this algorithm focused primarily on the probability of error in the asymptotic setting where the signal and noise are assumed to be separable . However , previous work on SSA for CPD does not provide finite-sample analysis on any metric on interest in CPD . Further , its extension to the case of multivariate time series ( multiple SSA or mSSA ) lacks theoretical understanding , despite being empirically effective [ 11 ] . In this work , we consider a variant of mSSA proposed and analyzed in [ 1 ] for the tasks of imputation and forecasting . We extend the use of this mSSA variant to devise an algorithm for CPD . Another important distinction between CPD methods lies in whether the method is online versus offline . In the offline setting , the dataset is fixed and the detection is made retrospectively based on the knowledge of the entire dataset . In such settings , only the accuracy of the detection is of concern . In contrast , in the online setting , the data arrives sequentially and a decision must be made as quickly as possible . In addition to the detection accuracy , the delay in making the detection ( i.e . the number of data points observed after the change until it is detected ) is another important metric in the online setting . A common overarching framework for the online CPD problem extends from sequential hypothesis testing , where the problem is formulated as a binary hypothesis test that is applied sequentially as new data comes in [ 40 ] . This framework provides a way to evaluate the performance of online CPD methods in terms of the trade-off between false alarm rate and detection delay . The classical setting of this framework uses parametric models and forms likelihood ratio-based test statistics . In more recent work , this framework has been utilized with different test statistics . For example , tests using statistics based on the kernel Fisher discriminant ratio [ 16 ] , maximum mean discrepancy [ 25 ] , and subspace distance [ 19 ] were proposed . However , rigorously characterizing the trade-off between false alarm rate and detection delay for time series is absent in the literature . In this work , we are motivated to fill the gap in literature by providing an online method for CPD for the setting of multivariate time series data ( not i.i.d . ) to detect change points with provable performance guarantees and desirable empirical performance . Summary of contributions . Below , we summarize the main contributions of this work . 1 . We introduce an expressive spatio-temporal model as described in Section 2.1 to model the dynamics and posit the question of detection as a change in these dynamics . 2 . We develop a variant of the SSA algorithm for online change point detection in multivariate time series . In particular , we propose an algorithm that utilizes the low-dimensional structure of the time series to construct a cumulative sum ( CUSUM ) statistic ( a la [ 34 ] ) based on subspace distance . We refer to Section 3 for a full description of the proposed algorithm . 3 . We analyze the performance of this algorithm in terms of the average running length ( ARL ) , a common metric used in CUSUM tests to measure both the delay in detection and the average running time until a false alarm . Our main result shows that in the case of no change , the expected running length is exponential in the selected threshold ( Theorem 4.1 ) . While in the presence of change , the expected detection delay is linear in the threshold ( Theorem 4.2 ) . The results are consistent with the behavior of the CUSUM tests in the i.i.d . case . 4 . We conduct comparative empirical experiments against state-of-the-art change point detection methods using benchmark datasets . In Table 1 , we provide a summary that suggests that our method is at least as good , if not better than , state-of-art methods ( SOTA ) across datasets . Related work . Due to space constraints , we provide an overview of related topics in Appendix A . 2 Problem Setup . We consider the discrete time setting where for each time index t2 [ T ] : = { 1 , ... T } , we observe a multivariate time series X ( t ) : = [ X1 ( t ) , ... , XN ( t ) ] 2RN . For each n2 [ N ] , we denote the n-th latent time series by fn : Z+ ! R , such that at each timestep t2 [ T ] , the observations take the formXn ( t ) =fn ( t ) +en ( t ) , where en ( t ) is the per-step noise modeled as a mean-zero i.i.d . random variable . We define the latent multivariate time series as f : Z+ ! RN such that f ( t ) : = [ f1 ( t ) , ... , fN ( t ) ] . That is , the observation X ( t ) takes the form X ( t ) =f ( t ) +e ( t ) , ( 1 ) where e ( t ) : = [ e1 ( t ) , ... , en ( t ) ] . We note that although the additive noise en ( t ) is i.i.d. , the latent time series f ( t ) can have a complex dependence structure across both t and n , see Section 2.1 . Our objective is to detect changes in the latent time series f ( t ) as quickly as possible . Precisely , our goal is to detect a change point ⌧ such that X ( t ) = ⇢ f0 ( t ) +e ( t ) if t < ⌧ f1 ( t ) +e ( t ) if t ⌧ . where f0 , f1 : Z+ ! RN and f0 ( t ) 6= f1 ( t ) . In this setting , we treat the change point ⌧ as an unknown deterministic quantity . We frame this CPD problem as a sequential hypothesis test where at each time t one of two hypotheses is accepted H0 : IE [ X ( t ) ] =f0 ( t ) ( 2 ) H1 : IE [ X ( t ) ] =f1 ( t ) . Let H ( t ) : Z+ ! { 0,1 } be the indicator random variable that indicates whether or not the null hypothesis is rejected at time t. We then estimate the change point ⌧ as ⌧̂=inf { t|H ( t ) =1 } . ( 3 ) Ideally , we wish to have ⌧̂ ⌧ , but also ⌧̂ ⌧ to be small . 2.1 Time Series Model . There are multiple settings for the CPD problem characterized by the probabilistic structure of the observations . In this work , we consider the spatio-temporal factor model for multivariate time series introduced in [ 1 ] , which captures a wide variety of time series dynamics . Next , we describe the model in detail . The spatio-temporal factor model requires the underlying latent multivariate time series to satisfy two properties . The first property captures the “ spatial ” structure , i.e. , the structure across the N time series ( Property 2.1 ) ; and the second property captures the “ temporal ” structure ( Property 2.2 ) . Property 2.1 ( Spatial structure ) . There exist R 2 N , where 1R⌧min ( N , T ) , Wr : Z+ ! R , and ↵nr2R 8r2 [ R ] , n2 [ N ] , such that for any n2 [ N ] , t2 [ T ] , fn ( t ) = PR r=1↵nr Wr ( t ) , where |↵nr| ↵ , |Wr ( t ) | W for constants ↵ , W > 0 . Property 2.1 implies that there are R “ fundamental ” time series , and each latent time series fn ( · ) 8n2 [ N ] can be obtained through a weighted combination of these R times series . To capture the temporal structure , additional assumptions are imposed on these fundamental time series Wr ( · ) 8r2 [ R ] . To do so , we first introduce the Page matrix representation of a time series . Definition 2.1 ( Page Matrix ) . Given a time series f : Z+ ! R , its Page matrix representation over T observations with parameter 1LT is given by the matrix Z2RL⇥bT/Lc with Zij=f ( i+ ( j 1 ) ⇥L ) for i2 [ L ] , j2 [ bT/Lc ] . Property 2.2 ( Temporal structure ) . For each r2 [ R ] and for any T > 1 , 1LT , let Z ( r ) W be RL⇥bT/Lc Page Matrix associated with time series Wr ( t ) , t2 [ T ] . Then , the rank of Z ( r ) W is at most G for any choice of 1LT . Property 2.2 imposes a particular temporal structure in the time series Wr ( · ) ,8r2 [ R ] . While seemingly restrictive at first , it has been shown in the SSA literature that many standard functions that model time series dynamics satisfy this property . In particular , the Page matrix representation of any finite sum of products of harmonics , low-degree polynomials , and exponential functions is low-rank and satisfies Property 2.2 ( refer to Proposition 2.1 in [ 1 ] ) . These functions can capture a rich class of time series dynamics which are typically modeled by three components : stationarity , periodicity , and trend . Periodicity is modeled as harmonics and trend as polynomials ; the mixture of both are instances of this model . The inclusion of stationarity in this model can be understood by considering the spectral representation of stationary processes ( refer to Property 4.1 in [ 39 ] ) , which states that any sample-path of a stationary process can be decomposed into a sum of harmonics . Therefore , a finite sum of harmonics provides a good model representation for stationary processes . For a given 1LT , let Z ( n ) f 2RL⇥bT/Lc denote the Page matrix induced by the n-th latent time series fn ( t ) 8n2 [ N ] . Further , consider the stacked Page matrix Zf 2RL⇥ ( NbT/Lc ) , obtained by a column-wise concatenation of the Page matrices Z ( 1 ) f , ... , Z ( N ) f ( See Figure 1 ) . Specifically , Zf = h Z ( 1 ) f Z ( 2 ) f ... Z ( N ) f i . ( 4 ) Proposition 2.1 ( Proposition 2.2 in [ 1 ] ) establishes that under the spatio-temporal factor model satisfying Properties 2.1 and 2.2 , the stacked Page matrix is low-rank . We refer to [ 1 ] for the proof of this property . Proposition 2.1 ( Proposition 2.2 in [ 1 ] ) . Let Properties 2.1 and 2.2 hold . Then for any 1LT , the rank of Z ( n ) f for n2 [ N ] is at most RxG . Moreover , the rank of the stacked Page matrix Zf is also at most RxG .

Summary: The paper studies change-point detection in time-series using a combination multi-variate singular spectrum analysis (SSA) and CUSUM.   While multi-variate SSA and CUSUM are established techniques, the contribution of the paper is to (a) analyze the online CPD setting with a dependent (non-iid) timeseries. (b) Conduct rigorous analysis of the tradeoff of detection delay vs. false-alarm within the MSSA model.  (c) The exact version of MSSA is a bit different from the classical one.