Detecting Change in Seasonal Pattern via Autoencoder and Temporal Regularization

1 INTRODUCTION . Time series data are sequences of measurements over time describing the behavior of systems . Time series analysis has become increasingly important in monitoring systems health and performance . As the system behavior changes over time due to external events and/or internal modifications , the problem of identifying the locations of these changes , referred to as Change Point Detection ( CPD ) has quickly drawn researchers attention . The CPD problem has been widely researched during the last three decades ( 4 ; 18 ; 13 ; 6 ; 16 ; 9 ) and it has been applied to several fields such as financial market analysis , medicine , climate science as well as system monitoring . The first methods found in the literature for CPD compared probability distributions between two consecutive intervals in a time-series , and alarmed if the difference became significant . Among them we find the cumulative sum algorithm ( 4 ) or the change finder for auto-regressive processes ( 18 ) . Another line of research focuses on subspace identification ( 12 ) , where the time series is modeled using a linear state-space and a change is identified using the model parameters . Since all these methods make strong assumptions on the distributions , a need arises for more generic solutions and non parametric algorithms , such as direct density estimation methods . Unfortunately , these methods suffer from the curse of dimensionality ( 17 ) and are not applicable to real life problems . To overcome this challenge , one possible solution is to estimate the ratio of densities between two successive window without computing the densities themselves . This is achieved by going through the estimation of a probability divergence metric such as Kullback-Leibler in ( 16 ) or the Person divergence ( 13 ) . Such methods proved to be quite successful . Another line of research focuses on Kernel two sample test ( 8 ) , where the kernel trick is used to evaluate mean discrepancy of two samples in a Reproducing Kernel Hilbert Space . For example , Harchaoui et al . ( 10 ) introduced a test statistic using the maximum kernel fisher discriminant ratio . More recently , Chang et al . ( 6 ) proposed a way to learn an optimal kernel representation for CPD by using an auxiliary generative model . Nonetheless , although more general , these models still assume that the process is time independent . However , very often , a time-series follows a seasonal behavior . Seasonality is defined as the tendency of a time-series to exhibit behavior that repeats itself every fixed period of time . The term season is used to represent the period of time before behavior begins to repeat itself . Detecting change in the seasonal pattern of a time-series is critical for many applications such as service mon- itoring or climate change detection ( 5 ; 15 ) . In some cases , it requires a totally different approach than regular CPD solutions . For example , Figure 1 shows the CPU utilization metric values recorded over a period of six days for some server machine . An event occurs every day at 10AM , representing some background process that is important for the system . A forecasting system should adapt its predictions to take into consideration this event . If for some reason , this process was moved to 4PM , the forecasting system is expected to detect this change and adjust the forecast values accordingly . Some information about the location of the peak in each period has to be taken into account in order to detect this kind of change . The regular algorithms presented in this section would fail to detect such kind of change points . Indeed : ( a ) Existing parametric models for CPD compare the distribution between two windows by looking at some statistics . Here , we will not observe a persistent statistical change , as the spike still occurs . ( b ) Kernel based methods assume that the samples are generated i.i.d ( 10 ; 13 ; 6 ) . Hence , they will consider every seasonal spike as anomaly instead of including them to the model . In this paper , we introduce a CPD variation for seasonal time series . The issue of seasonal CPD has also been addressed by Lund et al . ( 14 ) where they developed a test for periodic and auto correlated time series . However , this test is based on predefined statistics and focuses on median ( level ) change of the metric . We present ATR-CSPD , a method which uses an Autencoder with Temporal Regularization for detecting Changes in Seasonal Pattern . Our contributions are the following : • In section 2 , we introduce a variation of the Change Point Detection ( CPD ) problem called Change in Seasonal Pattern Detection ( CSPD ) for seasonal time series , and explain why current solutions to CPD are inefficient at detecting them . • In section 3 we describe ATR-CSPD , an unsupervised approach that is designed to capture essential information of each period in a seasonal time series . This is achieved by applying a time dependant regularization to the model ’ s loss . We explain then how to use this model for CPD . • Sections 4 and 5 present an extensive evaluation on both generated and real-life data . This benchmark exhibits how ATR-CSPD manages to detect new types of change points , undetected by regular algorithms . 2 PROBLEM FORMULATION . In this section , we start by presenting the simplified formulation of the Change Point Detection problem . Then we will introduce a more elaborated formulation for the detection of change points seasonal metrics , and illustrate the difference with an example . The simplified formulation assumes a single generative function behind the time-series values , and independence between the observations . Many common methods for detecting change points are applied based on this simplified version . For example , kernel density methods are based on the analysis of the single underlying function that generates the observations before and after the change ( 13 ; 6 ) . Definition 1 . Change Point Detection ( CPD ) : Given a sequence of 1-dimensional observations { x1 , ... , xt , ... xN } for which ∀i , xi ∈ R. A change point t0 is a point such that { x1 , ... , xt0−1 } are sampled i.i.d from a distribution F1 , and { xt0 , ... , xN } are sampled i.i.d from the distribution F2 where F1 6= F2 . However , for a seasonal time-series the assumption that up to the change-point all samples are i.i.d does not hold , since seasonality creates dependencies between the samples and their index in the period ( i.e their phase ) . In the extreme , it might be that each observation within a period is drawn from an entirely different distribution . However , as we are targeting metrics that are generated by one origin system , a more plausible assumption would be that each data point xi is generated by a combination of a generative function of the time series and another generative function of its phase . For a seasonal time series with period size p , we define each point xi as xjk where j is its period number , k is its phase number , and i = j · p+ k. We denote a single seasonal window of observations of size p by wi where wi = xi1 , ... xip . We can represent the original time series by grouping all its seasonal windows : { w1 , ... , wt , ... } , wi ∈ Rp . Denote F as the generative function of the time series , { Sk } p1 as the phase-wise generative functions , and xjk ∼ Gj , whereGj = Sj⊗F and⊗ can be an additive or a multiplicative factor ( 7 ) . In additive seasonal models , the metric is explained by a weighted sum of the seasonal components { Sk } p1 and the generating function F . In multiplicative model , the sum is replaced by a multiplication . In our analysis and results we don ’ t differentiate the two . Definition 2 . Change in Seasonal Pattern Detection ( CSPD ) : A change point in a seasonal pattern is a period number j0 such that ∀ ( k , j < j0 ) , xjk ∼ Gk , and ∃k , xj0k ∼ G ′ k , where Gk 6= G ′ k. The difference between the two formulations can be demonstrated by the example displayed in Figure 1 . It is clear that the time-series displayed in the chart represents a system that has altered its behavior . However , when considering the classical CPD problem formulation , the chart does not fall under the definition of a time series containing a change point as the overall distribution did not change . Considering the CSPD formulation , the change of the seasonal distribution component of both S10AM and S4PM distribution functions are identified as a change point in the time series . This observation suggests the CSPD is a generalization of the CPD problem . While CPD is centered around the cumulative values distribution parameters such as median and variance of the time series values , CSPD also focuses on the shape and proportions between the values observed in each period cycle . Remark : Both problems can be extended to multi-dimensional time series and we are only presenting it in 1-dimension for simplicity . 3 AUTOENCODER WITH TEMPORAL REGULARIZATION . We introduce an algorithm that is able to detect changes in the seasonal pattern of a time-series . We start by using an autoencoder to capture the main pattern of each period in the time-series . The aim is to have close encoding ( in terms of euclidean distance ) for two periods that behaves similar , and different ones if there is an abrupt change between them . By having such a representation we can detect if there has been a change point by examining the euclidean distance between two adjacent encoded periods in the time series . 3.1 THE AUTOENCODER MODEL . Autoencoders are neural networks that attempt to copy its input to its output . It consists of an encoder function , that maps the input to an encoded version , and a decoder that performs the reconstruction . Here we are training an autoencoder to reconstruct fix-size windows of a time-series . Let xi ∈ Rd×p , be the ith window of size p in a d-dimensional time series , fθ1 : Rd×p → Rd×q our encoder function , gθ2 : Rd×q → Rd×p the decoder function and n the total number of windows . In general autoencoder model , want to minimize : min θ1 , θ2 n∑ i=1 ||xi − gθ2 ( fθ1 ( xi ) ) ||2 + λ ( ||θ1||2 + ||θ2||2 ) ( 1 ) θ1 and θ2 are a set of parameters that can be learn by gradient descent using back-propagation and λ ∈ R allows us to control the L2-norm of θ1 and θ2 . In ATR-CSPD a regularization term is added to this loss function , which will contribute to the detection of change points . 3.2 TEMPORAL REGULARIZATION . In order to encourage the network to generate similar low-dimensional representations , we introduce a new term in equation ( 1 ) . The idea is to penalize the network for a difference between the encoding of two consecutive periods . The resulting loss function is given in ( 2 ) . min θ1 , θ2 n∑ i=1 ||xi − gθ2 ( fθ1 ( xi ) ) ||2 + λ ( ||θ1||2 + ||θ2||2 ) + γ n−1∑ i=1 ||hi+1 − hi||1 ( 2 ) We refer to the last term as temporal regularization as it applies to neighbours period in the timeseries . Here γ ∈ R allows us to control the strength of this regularization and hi = fθ1 ( xi ) . To understand its effect we take the following examples . Figure 2a shows a one-dimensional , weekly seasonal , time-series . On it we run two autoencoder models , the first one optimizing equation ( 1 ) and the second one equation ( 2 ) . We set p = 288 meaning that we consider the period as daily ( the metric is recorded every 5 minutes ) , and expect to observe changes in the weekends . The additional parameters are set to λ = 0.00005 , γ = 0.001 and q = 18 . We observe that , although the regular autoencoder doesn ’ t get rid of the major anomalies in the data ( Figure 2b ) , the temporal regularized model creates a generic pattern for the weekdays and for the weekends ( Figure 2c ) . According to this result , we can infer that the low dimensional representations are similar within the weekdays and within the days of the weekends , and we can use clustering to identify breakpoints . In Appendix C , we show the results of a Principal Components Analysis ( PCA ) applied to the encodings generated by both algorithms , which confirms this hypothesis .

Summary: This paper proposed a new model for change point detection, using autoencoders with temporal regularization, in order to impose temporal smoothness in the latent codes. To motivate this new model, the authors also provided a toy example to show how the abnormality in a time series is removed in the reconstructed signal using this additional regularization term. Experimental results were provided to support the proposed new model.