Detecting Change in Seasonal Pattern via Autoencoder and Temporal Regularization

1 INTRODUCTION . Time series data are sequences of measurements over time describing the behavior of systems . Time series analysis has become increasingly important in monitoring systems health and performance . As the system behavior changes over time due to external events and/or internal modifications , the problem of identifying the locations of these changes , referred to as Change Point Detection ( CPD ) has quickly drawn researchers attention . The CPD problem has been widely researched during the last three decades ( 4 ; 18 ; 13 ; 6 ; 16 ; 9 ) and it has been applied to several fields such as financial market analysis , medicine , climate science as well as system monitoring . The first methods found in the literature for CPD compared probability distributions between two consecutive intervals in a time-series , and alarmed if the difference became significant . Among them we find the cumulative sum algorithm ( 4 ) or the change finder for auto-regressive processes ( 18 ) . Another line of research focuses on subspace identification ( 12 ) , where the time series is modeled using a linear state-space and a change is identified using the model parameters . Since all these methods make strong assumptions on the distributions , a need arises for more generic solutions and non parametric algorithms , such as direct density estimation methods . Unfortunately , these methods suffer from the curse of dimensionality ( 17 ) and are not applicable to real life problems . To overcome this challenge , one possible solution is to estimate the ratio of densities between two successive window without computing the densities themselves . This is achieved by going through the estimation of a probability divergence metric such as Kullback-Leibler in ( 16 ) or the Person divergence ( 13 ) . Such methods proved to be quite successful . Another line of research focuses on Kernel two sample test ( 8 ) , where the kernel trick is used to evaluate mean discrepancy of two samples in a Reproducing Kernel Hilbert Space . For example , Harchaoui et al . ( 10 ) introduced a test statistic using the maximum kernel fisher discriminant ratio . More recently , Chang et al . ( 6 ) proposed a way to learn an optimal kernel representation for CPD by using an auxiliary generative model . Nonetheless , although more general , these models still assume that the process is time independent . However , very often , a time-series follows a seasonal behavior . Seasonality is defined as the tendency of a time-series to exhibit behavior that repeats itself every fixed period of time . The term season is used to represent the period of time before behavior begins to repeat itself . Detecting change in the seasonal pattern of a time-series is critical for many applications such as service mon- itoring or climate change detection ( 5 ; 15 ) . In some cases , it requires a totally different approach than regular CPD solutions . For example , Figure 1 shows the CPU utilization metric values recorded over a period of six days for some server machine . An event occurs every day at 10AM , representing some background process that is important for the system . A forecasting system should adapt its predictions to take into consideration this event . If for some reason , this process was moved to 4PM , the forecasting system is expected to detect this change and adjust the forecast values accordingly . Some information about the location of the peak in each period has to be taken into account in order to detect this kind of change . The regular algorithms presented in this section would fail to detect such kind of change points . Indeed : ( a ) Existing parametric models for CPD compare the distribution between two windows by looking at some statistics . Here , we will not observe a persistent statistical change , as the spike still occurs . ( b ) Kernel based methods assume that the samples are generated i.i.d ( 10 ; 13 ; 6 ) . Hence , they will consider every seasonal spike as anomaly instead of including them to the model . In this paper , we introduce a CPD variation for seasonal time series . The issue of seasonal CPD has also been addressed by Lund et al . ( 14 ) where they developed a test for periodic and auto correlated time series . However , this test is based on predefined statistics and focuses on median ( level ) change of the metric . We present ATR-CSPD , a method which uses an Autencoder with Temporal Regularization for detecting Changes in Seasonal Pattern . Our contributions are the following : • In section 2 , we introduce a variation of the Change Point Detection ( CPD ) problem called Change in Seasonal Pattern Detection ( CSPD ) for seasonal time series , and explain why current solutions to CPD are inefficient at detecting them . • In section 3 we describe ATR-CSPD , an unsupervised approach that is designed to capture essential information of each period in a seasonal time series . This is achieved by applying a time dependant regularization to the model ’ s loss . We explain then how to use this model for CPD . • Sections 4 and 5 present an extensive evaluation on both generated and real-life data . This benchmark exhibits how ATR-CSPD manages to detect new types of change points , undetected by regular algorithms . 2 PROBLEM FORMULATION . In this section , we start by presenting the simplified formulation of the Change Point Detection problem . Then we will introduce a more elaborated formulation for the detection of change points seasonal metrics , and illustrate the difference with an example . The simplified formulation assumes a single generative function behind the time-series values , and independence between the observations . Many common methods for detecting change points are applied based on this simplified version . For example , kernel density methods are based on the analysis of the single underlying function that generates the observations before and after the change ( 13 ; 6 ) . Definition 1 . Change Point Detection ( CPD ) : Given a sequence of 1-dimensional observations { x1 , ... , xt , ... xN } for which ∀i , xi ∈ R. A change point t0 is a point such that { x1 , ... , xt0−1 } are sampled i.i.d from a distribution F1 , and { xt0 , ... , xN } are sampled i.i.d from the distribution F2 where F1 6= F2 . However , for a seasonal time-series the assumption that up to the change-point all samples are i.i.d does not hold , since seasonality creates dependencies between the samples and their index in the period ( i.e their phase ) . In the extreme , it might be that each observation within a period is drawn from an entirely different distribution . However , as we are targeting metrics that are generated by one origin system , a more plausible assumption would be that each data point xi is generated by a combination of a generative function of the time series and another generative function of its phase . For a seasonal time series with period size p , we define each point xi as xjk where j is its period number , k is its phase number , and i = j · p+ k. We denote a single seasonal window of observations of size p by wi where wi = xi1 , ... xip . We can represent the original time series by grouping all its seasonal windows : { w1 , ... , wt , ... } , wi ∈ Rp . Denote F as the generative function of the time series , { Sk } p1 as the phase-wise generative functions , and xjk ∼ Gj , whereGj = Sj⊗F and⊗ can be an additive or a multiplicative factor ( 7 ) . In additive seasonal models , the metric is explained by a weighted sum of the seasonal components { Sk } p1 and the generating function F . In multiplicative model , the sum is replaced by a multiplication . In our analysis and results we don ’ t differentiate the two . Definition 2 . Change in Seasonal Pattern Detection ( CSPD ) : A change point in a seasonal pattern is a period number j0 such that ∀ ( k , j < j0 ) , xjk ∼ Gk , and ∃k , xj0k ∼ G ′ k , where Gk 6= G ′ k. The difference between the two formulations can be demonstrated by the example displayed in Figure 1 . It is clear that the time-series displayed in the chart represents a system that has altered its behavior . However , when considering the classical CPD problem formulation , the chart does not fall under the definition of a time series containing a change point as the overall distribution did not change . Considering the CSPD formulation , the change of the seasonal distribution component of both S10AM and S4PM distribution functions are identified as a change point in the time series . This observation suggests the CSPD is a generalization of the CPD problem . While CPD is centered around the cumulative values distribution parameters such as median and variance of the time series values , CSPD also focuses on the shape and proportions between the values observed in each period cycle . Remark : Both problems can be extended to multi-dimensional time series and we are only presenting it in 1-dimension for simplicity . 3 AUTOENCODER WITH TEMPORAL REGULARIZATION . We introduce an algorithm that is able to detect changes in the seasonal pattern of a time-series . We start by using an autoencoder to capture the main pattern of each period in the time-series . The aim is to have close encoding ( in terms of euclidean distance ) for two periods that behaves similar , and different ones if there is an abrupt change between them . By having such a representation we can detect if there has been a change point by examining the euclidean distance between two adjacent encoded periods in the time series . 3.1 THE AUTOENCODER MODEL . Autoencoders are neural networks that attempt to copy its input to its output . It consists of an encoder function , that maps the input to an encoded version , and a decoder that performs the reconstruction . Here we are training an autoencoder to reconstruct fix-size windows of a time-series . Let xi ∈ Rd×p , be the ith window of size p in a d-dimensional time series , fθ1 : Rd×p → Rd×q our encoder function , gθ2 : Rd×q → Rd×p the decoder function and n the total number of windows . In general autoencoder model , want to minimize : min θ1 , θ2 n∑ i=1 ||xi − gθ2 ( fθ1 ( xi ) ) ||2 + λ ( ||θ1||2 + ||θ2||2 ) ( 1 ) θ1 and θ2 are a set of parameters that can be learn by gradient descent using back-propagation and λ ∈ R allows us to control the L2-norm of θ1 and θ2 . In ATR-CSPD a regularization term is added to this loss function , which will contribute to the detection of change points . 3.2 TEMPORAL REGULARIZATION . In order to encourage the network to generate similar low-dimensional representations , we introduce a new term in equation ( 1 ) . The idea is to penalize the network for a difference between the encoding of two consecutive periods . The resulting loss function is given in ( 2 ) . min θ1 , θ2 n∑ i=1 ||xi − gθ2 ( fθ1 ( xi ) ) ||2 + λ ( ||θ1||2 + ||θ2||2 ) + γ n−1∑ i=1 ||hi+1 − hi||1 ( 2 ) We refer to the last term as temporal regularization as it applies to neighbours period in the timeseries . Here γ ∈ R allows us to control the strength of this regularization and hi = fθ1 ( xi ) . To understand its effect we take the following examples . Figure 2a shows a one-dimensional , weekly seasonal , time-series . On it we run two autoencoder models , the first one optimizing equation ( 1 ) and the second one equation ( 2 ) . We set p = 288 meaning that we consider the period as daily ( the metric is recorded every 5 minutes ) , and expect to observe changes in the weekends . The additional parameters are set to λ = 0.00005 , γ = 0.001 and q = 18 . We observe that , although the regular autoencoder doesn ’ t get rid of the major anomalies in the data ( Figure 2b ) , the temporal regularized model creates a generic pattern for the weekdays and for the weekends ( Figure 2c ) . According to this result , we can infer that the low dimensional representations are similar within the weekdays and within the days of the weekends , and we can use clustering to identify breakpoints . In Appendix C , we show the results of a Principal Components Analysis ( PCA ) applied to the encodings generated by both algorithms , which confirms this hypothesis .

Summary: The paper raises an alarm that state-of-the art change-point detection methods in the ML literature do not handle important practical aspects arising in time-series modeling, namely seasonality. Indeed, methods designed to detect changing distribution under an i.i.d. setting can fail dramatically when the assumption is violated, when the change happens in the seasonal component.  The paper proposes to use an auto-encoder to find the "main pattern" within each seasonal window, and to use total variation penalty (l1-norm on the change) of the hidden state in the auto-encoder to encourage a smooth state-sequence which allow breaks.  They use k-means clustering to partition data-points, and detect a change-point if two consequent hidden states don't end up in the same cluster.