STRIC: Stacked Residuals of Interpretable Components for Time Series Anomaly Detection

We present a residual-style architecture for interpretable forecasting and anomaly detection in multivariate time series . Our architecture is composed of stacked residual blocks designed to separate components of the signal such as trends , seasonality , and linear dynamics . These are followed by a Temporal Convolutional Network ( TCN ) that can freely model the remaining components and can aggregate global statistics from different time series as context for the local predictions of each time series . The architecture can be trained end-to-end and automatically adapts to the time scale of the signals . After modeling the signals , we use an anomaly detection system based on the classic CUMSUM algorithm and a variational approximation of the f -divergence to detect both isolated point anomalies and change-points in statistics of the signals . Our method outperforms state-ofthe-art robust statistical methods on typical time series benchmarks where deep networks usually underperform . To further illustrate the general applicability of our method , we show that it can be successfully employed on complex data such as text embeddings of newspaper articles . 1 INTRODUCTION . Time series data is being generated in increasing volumes from industrial , medical , commercial and scientific applications . Such growth is fueling demand for anomaly detection algorithms that are general enough to be applicable across domains , yet reliable enough to operate on real-world time series data ( Munir et al. , 2019 ; Geiger et al. , 2020 ; Su et al. , 2019 ) . While recent developments have focused on deep neural networks ( DNNs ) , simple linear models still outperform DNNs in applications that require robustness ( Braei & Wagner , 2020 ) and interpretable failure modes ( Geiger et al. , 2020 ; Su et al. , 2019 ) . To harvest the flexibility and interpretability of engineered modules while enabling end-to-end differentiable training , we introduce STRIC : Stacked Residuals of Interpretable Components . We follow standard practice and consider a two stage anomaly detection pipeline comprising a model of the normal time series and an anomaly detector based on the prediction residuals ( Munir et al. , 2019 ) . In particular , STRIC is composed of three modules : An interpretable local predictor , a non-linear predictor and a novel non-parametric anomaly detector . More specifically , STRIC uses a parametric model implemented by a sequence of residuals blocks with each layer capturing the prediction residual of previous layers . The first layer models trends , the second layer models quasiperiodicity/seasonality at multiple temporal scales , the third layer is a general linear predictor , and the last is a general non-linear model in the form of a Temporal Convolution Network ( TCN ) . While the first three layers are local ( i.e . applied to each component of the time series separately ) , the last integrates global statistics from additional time series ( covariates ) . Thanks to the residual structure the interpretable linear blocks do not reduce the representative power of our non-linear architecture : any component of the input time-series which can not be modeled by the interpretable blocks is processed deeper into our architecture by the non-linear module ( see Section 4 ) . The model is trained end-toend with a prediction loss and we automatically select its complexity using an upper bound of the marginal likelihood which , to the best of our knowledge , has never been applied to TCNs before . Anomalies are detected by checking for time instants in which the prediction residual is not stationary . To avoid any unrealistic assumption on the prediction residuals distribution , we use a likelihood ratio test that we derive from a variational upper bound of f -divergences and that can be computed directly from the data points . To summarize , our contributions are : 1 . We introduce STRIC , a stacked residual model that explicitly isolates interpretable factors such as slow trends , quasi-periodicity , and linearly predictable statistics ( Oreshkin et al. , 2019 ; Cleveland et al. , 1990 ) , and incorporates statistics from other time series as context/side information . 2 . We introduce a novel regularization that is added to the prediction loss and is used for automatic model complexity selection according to the Empirical Bayes framework ( Rasmussen & Williams , 2006 ) . 3 . We introduce a non-parametric extesion of the CUMSUM algorithm which entails a tunable parameter corresponding to the length of observation and enables anomaly detection in the absence of knowledge about the pre- and post-distributions . 4 . We test our method on standard anomaly detection benchamrks and show it merges both the advantages of simple and interpretable linear models and the flexibility of non-linear ones while discounting their major drawbacks : lack of flexibility of linear models and lack of interpretability and overfitting of non-linear ones . 2 RELATED WORK . A time series is an ordered sequence of data points . We focus on discrete and regularly spaced time indices , and thus we do not include literature specific to asynchronous time processes in our review . Different methods for time series anomaly detection ( TSAD ) can be taxonomized by their choice of ( i ) discriminant function , ( ii ) continuity criterion , and ( iii ) optimization method to determine the tolerance threshold . It is common to use statistics of the prediction error as the discriminant ( Braei & Wagner , 2020 ) , and the likelihood ratio between the distribution of the prediction error before and after a given time instant as the continuity criterion ( Yashchin , 1993 ) . Recent methods compute the discriminant using deep neural network architectures and euclidean distance as continuity criterion ( Munir et al. , 2019 ; Geiger et al. , 2020 ; Su et al. , 2019 ; Bashar & Nayak , 2020 ) . Our method follows a similar line but introduces novel elements both in ( i ) and ( ii ) : ( i ) the discriminant function is the prediction error residual of a novel regularized stacked residual architecture ; ( ii ) the decision function is based on a novel non-parametric extension of the CUMSUM algorithm . The resulting method , STRIC , has the advantage of separating interpretable components due to trends and seasonality , without reducing the representative power of our architecture . At initialization , the system is approximately equivalent to a multi-scale SARIMA model ( Adhikari & Agrawal , 2013 ) , which can be reliably applied out-of-the-box on most time series . However , as more data is acquired , any part of the system can be further fine-tuned in an unsupervised end-to-end fashion . Munir et al . ( 2019 ) argue that anomaly detection can be solved by exploiting a flexible model provided a proper inductive bias is introduced ( e.g . TCN ) . In Appendix A.7.2 we show that TCN alone might overfit simple time series . We therefore take their direction a step further , and , differently from previous works ( Bai et al. , 2018 ; Munir et al. , 2019 ; Sen et al. , 2019 ; Geiger et al. , 2020 ) , we provide our temporal model with an interpretable structure , similar to Oreshkin et al . ( 2019 ) . Moreover , unlike previous works on interpretability of DNNs ( Tsang et al. , 2018 ; Guen et al. , 2020 ) , our architecture explicitly imposes both an inductive bias and a regularization which are designed to expose the user both a STL-like decomposition ( Cleveland et al. , 1990 ) and the relevant time scale of the signals . Since TCNs tend to overfit if not properly regularized ( Appendix A.7.2 ) , we constrain our TCN ’ s representational power by enforcing fading memory ( Zancato & Chiuso , 2021 ) while retaining what is needed to predict future values . Our method outperforms both classical statistical methods ( Braei & Wagner , 2020 ) and deep networks ( Munir et al. , 2019 ; Geiger et al. , 2020 ; Su et al. , 2019 ; Bergman & Hoshen , 2020 ; Bashar & Nayak , 2020 ) on different anomaly detection benchmarks ( Laptev & Amizadeh , 2020 ; Lavin & Ahmad , 2015 ) ( Section 6 ) . Moreover , we show it can be employed to detect anomalous patterns on complex data such as text embeddings of newspaper articles ( Figure 4 ) . 3 NOTATION . We denote vectors with lower case and matrices with upper case . In particular y is multi-variate time series { y ( t ) } t∈Z , y ( t ) ∈ Rn ; we stack observations from time t to t+ k − 1 and denote the resulting matrix as Y t+k−1t : = [ y ( t ) , y ( t+ 1 ) , ... , y ( t+ k − 1 ) ] ∈ Rn×k . The row index refers to the dimension of the time series while the column index refers to the temporal dimension . We denote the i-th component of the time series y as yi and its evaluation at time t ad as yi ( t ) ∈ R. We refer to { y ( s ) , s > t } as the test/future and to { y ( s ) , s ≤ t } as the reference/past intervals . At time t , sub-sequences containing the np past samples up to time t − np + 1 are given by Y tt−np+1 ( note that we include the Ytt−np+1 t ̂y ( t + 1 ) t − np + 1 t + 1 past future n ⋮ Linear X̂LIN XLIN ∈ ℝn×np Seasonal X̂SEAS XSEAS ∈ ℝn×np ̂ySEAS ( t + 1 ) ∈ ℝ ̂yLIN ( t + 1 ) ∈ ℝ ̂yTCN ( t + 1 ) ∈ ℝ X̂TREND XTREND ∈ ℝn×np TCNTrend ̂yTREND ( t + 1 ) ∈ ℝ ( a ) STRIC predictor architecture . Z = z1 z2 ⋮ zn Z ∈ { XTREND , XSEAS , XLIN } Φ = φ1 φ2 ⋮ φn Φ ∈ { 𝒦TREND , 𝒦SEAS , 𝒦LIN } Layer inputs Layer filters banks Layer features selectors A = aT1 aT2 ⋮ aTn A ∈ { 𝒜TREND , 𝒜SEAS , 𝒜LIN } Time Features Future Predictor Gi aTi Gi aTi Gibi ℝl×np∈ φ1 * zi φ2 * zi ⋮ φl * zi zii = 1 , ... , n For each B = bT1 bT2 ⋮ bTn Layer predictors B ∈ { ℬTREND , ℬSEAS , ℬLIN } ∈ ∈ ℝ1×np ℝ ( b ) Interpretable blocks structure . Time features are extracted independently for each time series ( see Appendix A.1 for more details ) . present data into the past data ) , while future samples up to time t + nf are Y t+nf t+1 . We will use past data to predict future ones , where the length of past and future intervals is an hyper-parameter that is up to the user to design . 4 TEMPORAL RESIDUAL ARCHITECTURE . Our architecture is depicted in Figure 1a . Its basic building blocks are causal convolutions ( Bai et al. , 2018 ) , with a fixed-size 1-D kernel with input elements from time t and earlier . Rather than initializing the convolutional filter randomly , as commonly done in deep learning , we initialize the weights so that each layer is biased to attend at different components of the signal , as explained in the following . Linear module . The first ( linear ) module is interpretable and captures local statistics of a given time series by means of a cascade of learnable linear filters . Its first layer models and removes slow-varying components in the input data . We initialize the filters to mimic a causal Hodrick Prescott ( HP ) filter ( Ravn & Uhlig , 2002 ) . The second layer models and removes periodic components : it is initialized to have a periodic impulse response . Finally , the third layer implements a linear stationary filter bank . We treat the impulse responses parameters of the linear filters as trainable parameters . To allow our model to work on a wide variety of time scales , we initialize the trend layer with different HP smoothness degrees , while we initialize the periodic and linear-stationary layers with randomly chosen poles ( Farahmand et al. , 2017 ) both on the unit circle and within the unit circle , thus allowing to capture different periodicities . Non-linear module . The second ( non-linear ) module aggregates global statistics from different time series using a TCN model ( Sen et al. , 2019 ) . It takes as input the prediction residual of the linear module and outputs a matrix G ( Y tt−np+1 ) ∈ R l×np where l is the number of output features extracted by the TCN model . The columnG ( Y tt−np+1 ) j with j = 1 , ... , np of the non-linear features is computed using data up to time t−np + j ( due to the internal structure of a TCN network ( Bai et al. , 2018 ) ) . We build a linear predictor on top of G ( Y tt−np+1 ) for each single time series independenty : the predictor for the i-th time series is given by : ŷTCN ( t+ 1 ) i : = aTi G ( Y t t−np+1 ) bi where ai ∈ R l and bi ∈ Rnp . Since ai combines features ( uniformly in time ) we can interpret it as a feature selector . While bi aggregates relevant features across time indices to build the one-step ahead predictor ( see Appendix A.1 ) . Note that the third layer of the linear module is a superset of preceding ones , and the non-linear module is a superset of the whole linear module . While this makes the model redundant , we show that this design , coupled with proper initialization and regularization , improves the reliability and intepretability of the final model . We improve filters optimization by sharing their kernel parameters among different time series so that global information ( e.g. , common trend shapes , periodicities ) can be extracted . In Appendix A.1 , we describe each component of the model in detail .

Summary: This paper aims to boost the performance of deep neural networks (DNNs) for time series applications by focusing on the characteristics of interpretable forecasting and anomaly detection which are important for real-world time series data. The authors propose an end-to-end trainable DNN architecture which is composed of stacked residual blocks to separate signal components including slow trends, quasi-periodicity, and linear dynamics, followed by a temporal convolutional network (TCN) to model other components. Although previous studies showed that conventional simple linear models often outperformed DNN models on typical time series benchmarks that require robustness and interpretability, this paper shows that the proposed DNN model outperforms state-of-the-art robust statistical methods in some datasets, kind of demonstrating the best of both worlds.