Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

1 INTRODUCTION . Real-world systems always work in a continuous way , which can generate several successive measurements monitored by multi-sensors , such as industrial equipment , space probe , etc . Discovering the malfunctions from large-scale system monitoring data can be reduced to detecting the abnormal time points from time series , which is quite meaningful for ensuring security and avoiding financial loss . But anomalies are usually rare and hidden by vast normal points , making the data labeling hard and expensive . Thus , we focus on time series anomaly detection under the unsupervised setting . Unsupervised time series anomaly detection is extremely challenging in practice . The model should learn informative representations from complex temporal dynamics through unsupervised tasks . Still , it should also derive a distinguishable criterion that can detect the rare anomalies from plenty of normal time points . Various classic anomaly detection methods have provided many unsupervised paradigms , such as the density-estimation methods proposed in local outlier factor ( LOF , Breunig et al . ( 2000 ) ) , clustering-based methods presented in one-class SVM ( OC-SVM , Schölkopf et al . ( 2001 ) ) and SVDD ( Tax & Duin , 2004 ) . These classic methods do not consider the temporal information and are difficult to generalize to unseen real scenarios . Benefiting from the representation learning capability of neural networks , recent deep models ( Su et al. , 2019 ; Shen et al. , 2020 ; Li et al. , 2021 ) have achieved superior performance . A major category of methods focus on learning pointwise representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task . Here , a natural and practical anomaly criterion is the pointwise reconstruction or prediction error . However , due to the rarity of anomalies , the pointwise representation is less informative for complex temporal patterns and can be dominated by normal time points , making anomalies less distinguishable . Also , the reconstruction or prediction error is calculated point by point , which can not provide a comprehensive description of the temporal context . ∗Equal Contribution Another major category of methods detect anomalies based on explicit association modeling . The vector autoregression and state space models fall into this category . The graph was also used to capture the association explicitly , through representing time series with different time points as vertices and detecting anomalies by random walk ( Cheng et al. , 2008 ; 2009 ) . In general , it is hard for these classic methods to learn informative representations and model fine-grained associations . Recently , graph neural network ( GNN ) has been applied to learn the dynamic graph among multiple variables in multivariate time series ( Zhao et al. , 2020 ; Deng & Hooi , 2021 ) . While being more expressive , the learned graph is still limited to a single time point , which is insufficient for complex temporal patterns . Besides , subsequence-based methods detect anomalies by calculating the similarity among subsequences ( Boniol & Palpanas , 2020 ) . While exploring wider temporal context , these methods can not capture the fine-grained temporal association between each time point and the whole series . In this paper , we adapt Transfomers ( Vaswani et al. , 2017 ) to time series anomaly detection in the unsupervised regime . Transformers have achieved great progress in various areas , including natural language processing ( Brown et al. , 2020 ) , machine vision ( Liu et al. , 2021 ) and time series ( Zhou et al. , 2021 ) . This success is attributed to its great power in unified modeling of global representation and long-range relation . Applying Transformers to time series , we find that the temporal association of each time point can be obtained from the self-attention map , which presents as a distribution of its association weights to all the time points along the temporal dimension . The association distribution of each time point can provide a more informative description for the temporal context , indicating dynamic patterns , such as the period or trend of time series . We name the above association distribution as the series-association , which can be discovered from the raw series by Transformers Further , we observe that due to the rarity of anomalies and the dominance of normal patterns , it is harder for anomalies to build strong associations with the whole series . The associations of anomalies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity . Such an adjacent-concentration inductive bias is referred to as the prior-association . In contrast , the dominating normal time points can discover informative associations with the whole series , not limiting to the adjacent area . Based on this observation , we try to utilize the inherent normal-abnormal distinguishability of the association distribution . This leads to a new anomaly criterion for each time point , quantified by the distance between each time point ’ s prior-association and its series-association , named as Association Discrepancy . As aforementioned , because the associations of anomalies are more likely to be adjacent-concentrating , anomalies will present a smaller association discrepancy than normal time points . Go beyond previous methods , we introduce Transformers to the unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning . To compute the Association Discrepancy , we renovate the self-attention mechanism to the Anomaly-Attention , which contains a two-branch structure to model the prior-association and series-association of each time point respectively . The prior-association employs the learnable Gaussian kernel to present the adjacentconcentration inductive bias of each time point , while the series-association corresponds to the selfattention weights learned from raw series . Besides , a minimax strategy is applied between the two branches , which can amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based criterion . Anomaly Transformer achieves strong results on six benchmarks , covering three real applications . The contributions are summarized as follows : • Based on the key observation of Association Discrepancy , we propose the Anomaly Transformer with an Anomaly-Attention mechanism , which can model the prior-association and series-association simultaneously to embody the Association Discrepancy . • We propose a minimax strategy to amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based detection criterion . • Anomaly Transformer achieves the state-of-the-art anomaly detection results on six benchmarks for three real applications . Extensive ablations and insightful case studies are given . 2 RELATED WORK . 2.1 UNSUPERVISED TIME SERIES ANOMALY DETECTION . As a vital real-world problem , unsupervised time series anomaly detection has been widely explored . Categorizing by anomaly determination criterion , the paradigms roughly include the densityestimation , clustering-based , reconstruction-based and autoregression-based methods . As for the density-estimation methods , the classic methods local outlier factor ( LOF , Breunig et al . ( 2000 ) ) and connectivity outlier factor ( COF , Tang et al . ( 2002 ) ) calculates the local density and local connectivity for outlier determination respectively . DAGMM ( Zong et al. , 2018 ) and MPPCACD ( Yairi et al. , 2017 ) integrate the Gaussian Mixture Model to estimate the density of representations . In clustering-based methods , the anomaly score is always formalized as the distance to cluster center . SVDD ( Tax & Duin , 2004 ) and Deep SVDD ( Ruff et al. , 2018 ) gather the representations from normal data to a compact cluster . THOC ( Shen et al. , 2020 ) fuses the multi-scale temporal features from intermediate layers by a hierarchical clustering mechanism and detects the anomalies by the multi-layer distances . ITAD ( Shin et al. , 2020 ) conducts the clustering on decomposed tensors . The reconstruction-based models attempt to detect the anomalies by the reconstruction error . Park et al . ( 2018 ) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder ( VAE ) for reconstruction . OmniAnomaly proposed by Su et al . ( 2019 ) further extends the LSTM-VAE model with a normalizing flow and uses the reconstruction probabilities for detection . InterFusion from Li et al . ( 2021 ) renovates the backbone to a hierarchical VAE to model the inter- and intra- dependency among multiple series simultaneously . GANs ( Goodfellow et al. , 2014 ) are also used for reconstruction-based anomaly detection ( Schlegl et al. , 2019 ; Li et al. , 2019a ; Zhou et al. , 2019 ) and perform as an adversarial regularization . The autoregression-based models detect the anomalies by the prediction error . VAR extends ARIMA ( Anderson & Kendall , 1976 ) and predicts the future based on the lag-dependent covariance . The autoregressive model can also be replaced by LSTMs ( Hundman et al. , 2018 ; Tariq et al. , 2019 ) . This paper is characterized by a new association-based criterion . Different from the random walk and subsequence-based methods ( Cheng et al. , 2008 ; Boniol & Palpanas , 2020 ) , our criterion is embodied by a co-design of the temporal models for learning more informative time-point associations . 2.2 TRANSFORMERS FOR TIME SERIES ANALYSIS . Recently , Transformers ( Vaswani et al. , 2017 ) have shown great power in sequential data processing , such as natural language processing ( Devlin et al. , 2019 ; Brown et al. , 2020 ) , audio processing ( Huang et al. , 2019 ) and computer vision ( Dosovitskiy et al. , 2021 ; Liu et al. , 2021 ) . For time series analysis , benefiting from the advantage of the self-attention mechanism , Transformers are used to discover the reliable long-range temporal dependencies ( Kitaev et al. , 2020 ; Li et al. , 2019b ; Zhou et al. , 2021 ; Wu et al. , 2021 ) . Especially for time series anomaly detection , GTA proposed by Chen et al . ( 2021 ) employs the graph structure to learn the relationship among multiple IoT sensors , as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detection . Unlike the previous usage of Transformers , Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy . 3 METHOD . Suppose monitoring a successive system of dmeasurements and recording the equally spaced observations over time . The observed time series X is denoted by a set of time points { x1 , x2 , · · · , xN } , where xt ∈ Rd represents the observation of time t. The unsupervised time series anomaly detection problem is to determine whether xt is anomalous or not without labels . As aforementioned , we highlight the key to unsupervised time series anomaly detection as learning informative representations and finding distinguishable criterion . We propose the Anomaly Transformer to discover more informative associations and tackle this problem by learning the Association Discrepancy , which is inherently normal-abnormal distinguishable . Technically , we propose the Anomaly-Attention to embody the prior-association and series-associations , along with a minimax optimization strategy to obtain a more distinguishable association discrepancy . Co-designed with the architecture , we derive an association-based criterion based on the learned association discrepancy . 3.1 ANOMALY TRANSFORMER . Given the limitation of Transformers ( Vaswani et al. , 2017 ) for anomaly detection , we renovate the vanilla architecture to the Anomaly Transformer ( Figure 1 ) with an Anomaly-Attention mechanism . Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attention blocks and feed-forward layers alternately . This stacking structure is conducive to learning underlying associations from deep multi-level features . Suppose the model contains L layers with length-N Published as a conference paper at ICLR 2022 Sigma Linear Q K V MatMul Scale SoftMax Global Relation Local Relation Diagonal Filling MatMul KL Distance Min KL Max KL Z KL Distance Stop gradient Sigma Q K C o n cat Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Anomaly Attention Global-Local KL Distance Reconstruction Loss Sigma Linear Q K V MatMul Scale SoftMax Global-Series Association Adjacent-Series Association Diagonalize MatMul KL Distance Min KL Max KL X KL Distance Stop gradient Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop Grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 ( , 21 ) ( , 22 ) ( , 2 ) 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 … Scale Grad Grad Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 ( , 21 ) ( , 22 ) ( , 2 ) 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 … Scale grad grad Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop Grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x … Rescale Grad Grad 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) G ( |j − 2| ; σ2 ) G ( |j − N | ; σN ) 参考文献 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) ( |j 2| ; σ2 ) | | ; 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) G ( |j − 2| ; σ2 ) G ( |j − N | ; σN ) Figure 1 : Anomaly Transformer architecture . Anomaly-Attention ( left ) models the prior-association and series-association simultaneously . In addition to the reconstruction loss , our model is also optimized by the minimax strategy with a specially-designed stop-gradient mechanism ( gray arrows ) to constrain the prior- and series- associations for more distinguishable association discrepancy . input time series X ∈ RN×d . The overall equations of the l-th layer are formalized as : Z l = Layer-Norm ( Anomaly-Attention ( X l−1 ) + X l−1 ) X l = Layer-Norm ( Feed-Forward ( Z l ) + Z l ) , ( 1 ) where X l ∈ RN×dmodel , l ∈ { 1 , · · · , L } denotes the output of the l-th layer with dmodel channels . The initial input X 0 = Embedding ( X ) represents the embedded raw series . Z l ∈ RN×dmodel is the l-th layer ’ s hidden representation . Anomaly-Attention ( · ) is to compute the association discrepancy . Anomaly-Attention Note that the single-branch self-attention mechanism ( Vaswani et al. , 2017 ) can not model the prior-association and series-association simultaneously . We propose the AnomalyAttention with a two-branch structure ( Figure 1 ) . For the prior-association , we adopt a learnable Gaussian kernel to calculate the prior with respect to the relative temporal distance . Benefiting from the unimodal property of the Gaussian kernel , this design can pay more attention to the adjacent horizon constitutionally . We also use a learnable scale parameter σ for the Gaussian kernel , making the prior-associations adapt to the various time series patterns , such as different lengths of anomaly segments . The series-association branch is to learn the associations from raw series , which can find the most effective associations adaptively . Note that these two forms maintain the temporal dependencies of each time point , which are more informative than point-wise representation . They also reflect the adjacent-concentration prior and the learned real associations respectively , whose discrepancy shall be normal-abnormal distinguishable . The Anomaly-Attention in the l-th layer is : Initialization : Q , K , V , σ = X l−1W lQ , X l−1W lK , X l−1W lV , X l−1W lσ Prior-Association : P l = Rescale ( [ 1√ 2πσi exp ( −|j − i| 2 2σ2i ) ] i , j∈ { 1 , ··· , N } ) Series-Association : Sl = Softmax ( QKT√ dmodel ) Reconstruction : Ẑ l = SlV , ( 2 ) where Q , K , V ∈ RN×dmodel , σ ∈ RN×1 represent the query , key , value of self-attention and the learned scale respectively . W lQ , W l K , W l V ∈ Rdmodel×dmodel , W lσ ∈ Rdmodel×1 represent the parameter matrices for Q , K , V , σ in the l-th layer respectively . Prior-association P l ∈ RN×N is generated based on the learned scale σ ∈ RN×1 and the i-th element σi corresponds to the i-th time point . Concretely , for the i-th time point , its association weight to the j-th point is calculate by the Gaussian kernel G ( |j− i| ; σi ) = 1√2πσi exp ( − |j−i|2 2σ2i ) w.r.t . the distance |j− i| . Further , we use Rescale ( · ) to transform the association weights to discrete distributions P l by dividing the row sum . Sl ∈ RN×N denotes the series-associations . Softmax ( · ) normalizes the attention map along the last dimension . 4 Published as a conference paper at ICLR 2022 Association Discrepancy b . Association Discrepancy d. Min KL Max KL a. Min KL Max KL c. Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Normal Time PointsAbnormal Time Points Maximize PhaseMinimize Phase Association Discrepancy Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Adjacet-series Association Global-series Association Constrained by Reconstruction Global-series Association Adjacet-series Association Within the Gaussian Family Time series Time series Optimization Direction Global-series Association Adjacet-series Association Time series Adjacet-series Association Maximize PhaseMinimize Phase Association Discrepancy Prior-Association Series-AssociationGaussian Family Reconstruction Loss Constrained Space PriorAssociation SeriesAssociation Constrained by Reconstruction SeriesAssociation PriorAssociation Within the Gaussian Family Time series Time series Optimization Direction Association Discrepancy Thus , each row of Sl forms a discrete distribution . Ẑ l ∈ RN×dmodel is the hidden representation after the Anomaly-Attention in the l-th layer . We use Anomaly-Attention ( · ) to summarize Equation 2 . In the multi-head version that we use , the learned scale is σ ∈ RN×h for h heads . Qm , Km , Vm ∈ RN× dmodel h denote the query , key and value of the m-th head respectively . The block concatenates the outputs { Ẑ lm ∈ RN× dmodel h } 1≤m≤h from multiple heads and gets the final result Ẑ l ∈ RN×dmodel . Association Discrepancy We formalize the Association Discrepancy as the symmetrized KL divergence between prior- and series- associations , which represents the information gain between these two distributions ( Neal , 2007 ) . We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as : AssDis ( P , S ; X ) = [ 1 L L∑ l=1 ( KL ( P li , :‖Sli , : ) + KL ( Sli , :‖P li , : ) ) ] i=1 , ··· , N ( 3 ) where KL ( ·‖· ) is the KL divergence computed between two discrete distributions corresponding to every row of P l and Sl . AssDis ( P , S ; X ) ∈ RN×1 is the point-wise association discrepancy of X with respect to prior-association P and series-association S from multiple layers . The i-th element of results corresponds to the i-th time point ofX . From previous observation , anomalies will present smaller AssDis ( P , S ; X ) than normal time points , which makes AssDis inherently distinguishable .

Summary: This paper introduces an Anomaly Transformer for detecting anomalies in time series with association discrepancy. This paper introduces two discrepancies: the series association (e.g., period and trend) and the prior association (e.g., the local smoothness or continuity). For abnormal points, these two associations have a small discrepancy and for normal points, there is a large discrepancy between these two associations. This is because abnormal points have a strong local association while the normal points have a global association. In the model, a two-branch strategy is adopted to model the two associations separately. The model is trained by minimizing the reconstruction loss and maximizing the discrepancy. The experimental results demonstrate the effectiveness of the proposed model.