Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

1 INTRODUCTION . Real-world systems always work in a continuous way , which can generate several successive measurements monitored by multi-sensors , such as industrial equipment , space probe , etc . Discovering the malfunctions from large-scale system monitoring data can be reduced to detecting the abnormal time points from time series , which is quite meaningful for ensuring security and avoiding financial loss . But anomalies are usually rare and hidden by vast normal points , making the data labeling hard and expensive . Thus , we focus on time series anomaly detection under the unsupervised setting . Unsupervised time series anomaly detection is extremely challenging in practice . The model should learn informative representations from complex temporal dynamics through unsupervised tasks . Still , it should also derive a distinguishable criterion that can detect the rare anomalies from plenty of normal time points . Various classic anomaly detection methods have provided many unsupervised paradigms , such as the density-estimation methods proposed in local outlier factor ( LOF , Breunig et al . ( 2000 ) ) , clustering-based methods presented in one-class SVM ( OC-SVM , Schölkopf et al . ( 2001 ) ) and SVDD ( Tax & Duin , 2004 ) . These classic methods do not consider the temporal information and are difficult to generalize to unseen real scenarios . Benefiting from the representation learning capability of neural networks , recent deep models ( Su et al. , 2019 ; Shen et al. , 2020 ; Li et al. , 2021 ) have achieved superior performance . A major category of methods focus on learning pointwise representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task . Here , a natural and practical anomaly criterion is the pointwise reconstruction or prediction error . However , due to the rarity of anomalies , the pointwise representation is less informative for complex temporal patterns and can be dominated by normal time points , making anomalies less distinguishable . Also , the reconstruction or prediction error is calculated point by point , which can not provide a comprehensive description of the temporal context . ∗Equal Contribution Another major category of methods detect anomalies based on explicit association modeling . The vector autoregression and state space models fall into this category . The graph was also used to capture the association explicitly , through representing time series with different time points as vertices and detecting anomalies by random walk ( Cheng et al. , 2008 ; 2009 ) . In general , it is hard for these classic methods to learn informative representations and model fine-grained associations . Recently , graph neural network ( GNN ) has been applied to learn the dynamic graph among multiple variables in multivariate time series ( Zhao et al. , 2020 ; Deng & Hooi , 2021 ) . While being more expressive , the learned graph is still limited to a single time point , which is insufficient for complex temporal patterns . Besides , subsequence-based methods detect anomalies by calculating the similarity among subsequences ( Boniol & Palpanas , 2020 ) . While exploring wider temporal context , these methods can not capture the fine-grained temporal association between each time point and the whole series . In this paper , we adapt Transfomers ( Vaswani et al. , 2017 ) to time series anomaly detection in the unsupervised regime . Transformers have achieved great progress in various areas , including natural language processing ( Brown et al. , 2020 ) , machine vision ( Liu et al. , 2021 ) and time series ( Zhou et al. , 2021 ) . This success is attributed to its great power in unified modeling of global representation and long-range relation . Applying Transformers to time series , we find that the temporal association of each time point can be obtained from the self-attention map , which presents as a distribution of its association weights to all the time points along the temporal dimension . The association distribution of each time point can provide a more informative description for the temporal context , indicating dynamic patterns , such as the period or trend of time series . We name the above association distribution as the series-association , which can be discovered from the raw series by Transformers Further , we observe that due to the rarity of anomalies and the dominance of normal patterns , it is harder for anomalies to build strong associations with the whole series . The associations of anomalies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity . Such an adjacent-concentration inductive bias is referred to as the prior-association . In contrast , the dominating normal time points can discover informative associations with the whole series , not limiting to the adjacent area . Based on this observation , we try to utilize the inherent normal-abnormal distinguishability of the association distribution . This leads to a new anomaly criterion for each time point , quantified by the distance between each time point ’ s prior-association and its series-association , named as Association Discrepancy . As aforementioned , because the associations of anomalies are more likely to be adjacent-concentrating , anomalies will present a smaller association discrepancy than normal time points . Go beyond previous methods , we introduce Transformers to the unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning . To compute the Association Discrepancy , we renovate the self-attention mechanism to the Anomaly-Attention , which contains a two-branch structure to model the prior-association and series-association of each time point respectively . The prior-association employs the learnable Gaussian kernel to present the adjacentconcentration inductive bias of each time point , while the series-association corresponds to the selfattention weights learned from raw series . Besides , a minimax strategy is applied between the two branches , which can amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based criterion . Anomaly Transformer achieves strong results on six benchmarks , covering three real applications . The contributions are summarized as follows : • Based on the key observation of Association Discrepancy , we propose the Anomaly Transformer with an Anomaly-Attention mechanism , which can model the prior-association and series-association simultaneously to embody the Association Discrepancy . • We propose a minimax strategy to amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based detection criterion . • Anomaly Transformer achieves the state-of-the-art anomaly detection results on six benchmarks for three real applications . Extensive ablations and insightful case studies are given . 2 RELATED WORK . 2.1 UNSUPERVISED TIME SERIES ANOMALY DETECTION . As a vital real-world problem , unsupervised time series anomaly detection has been widely explored . Categorizing by anomaly determination criterion , the paradigms roughly include the densityestimation , clustering-based , reconstruction-based and autoregression-based methods . As for the density-estimation methods , the classic methods local outlier factor ( LOF , Breunig et al . ( 2000 ) ) and connectivity outlier factor ( COF , Tang et al . ( 2002 ) ) calculates the local density and local connectivity for outlier determination respectively . DAGMM ( Zong et al. , 2018 ) and MPPCACD ( Yairi et al. , 2017 ) integrate the Gaussian Mixture Model to estimate the density of representations . In clustering-based methods , the anomaly score is always formalized as the distance to cluster center . SVDD ( Tax & Duin , 2004 ) and Deep SVDD ( Ruff et al. , 2018 ) gather the representations from normal data to a compact cluster . THOC ( Shen et al. , 2020 ) fuses the multi-scale temporal features from intermediate layers by a hierarchical clustering mechanism and detects the anomalies by the multi-layer distances . ITAD ( Shin et al. , 2020 ) conducts the clustering on decomposed tensors . The reconstruction-based models attempt to detect the anomalies by the reconstruction error . Park et al . ( 2018 ) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder ( VAE ) for reconstruction . OmniAnomaly proposed by Su et al . ( 2019 ) further extends the LSTM-VAE model with a normalizing flow and uses the reconstruction probabilities for detection . InterFusion from Li et al . ( 2021 ) renovates the backbone to a hierarchical VAE to model the inter- and intra- dependency among multiple series simultaneously . GANs ( Goodfellow et al. , 2014 ) are also used for reconstruction-based anomaly detection ( Schlegl et al. , 2019 ; Li et al. , 2019a ; Zhou et al. , 2019 ) and perform as an adversarial regularization . The autoregression-based models detect the anomalies by the prediction error . VAR extends ARIMA ( Anderson & Kendall , 1976 ) and predicts the future based on the lag-dependent covariance . The autoregressive model can also be replaced by LSTMs ( Hundman et al. , 2018 ; Tariq et al. , 2019 ) . This paper is characterized by a new association-based criterion . Different from the random walk and subsequence-based methods ( Cheng et al. , 2008 ; Boniol & Palpanas , 2020 ) , our criterion is embodied by a co-design of the temporal models for learning more informative time-point associations . 2.2 TRANSFORMERS FOR TIME SERIES ANALYSIS . Recently , Transformers ( Vaswani et al. , 2017 ) have shown great power in sequential data processing , such as natural language processing ( Devlin et al. , 2019 ; Brown et al. , 2020 ) , audio processing ( Huang et al. , 2019 ) and computer vision ( Dosovitskiy et al. , 2021 ; Liu et al. , 2021 ) . For time series analysis , benefiting from the advantage of the self-attention mechanism , Transformers are used to discover the reliable long-range temporal dependencies ( Kitaev et al. , 2020 ; Li et al. , 2019b ; Zhou et al. , 2021 ; Wu et al. , 2021 ) . Especially for time series anomaly detection , GTA proposed by Chen et al . ( 2021 ) employs the graph structure to learn the relationship among multiple IoT sensors , as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detection . Unlike the previous usage of Transformers , Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy . 3 METHOD . Suppose monitoring a successive system of dmeasurements and recording the equally spaced observations over time . The observed time series X is denoted by a set of time points { x1 , x2 , · · · , xN } , where xt ∈ Rd represents the observation of time t. The unsupervised time series anomaly detection problem is to determine whether xt is anomalous or not without labels . As aforementioned , we highlight the key to unsupervised time series anomaly detection as learning informative representations and finding distinguishable criterion . We propose the Anomaly Transformer to discover more informative associations and tackle this problem by learning the Association Discrepancy , which is inherently normal-abnormal distinguishable . Technically , we propose the Anomaly-Attention to embody the prior-association and series-associations , along with a minimax optimization strategy to obtain a more distinguishable association discrepancy . Co-designed with the architecture , we derive an association-based criterion based on the learned association discrepancy . 3.1 ANOMALY TRANSFORMER . Given the limitation of Transformers ( Vaswani et al. , 2017 ) for anomaly detection , we renovate the vanilla architecture to the Anomaly Transformer ( Figure 1 ) with an Anomaly-Attention mechanism . Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attention blocks and feed-forward layers alternately . This stacking structure is conducive to learning underlying associations from deep multi-level features . Suppose the model contains L layers with length-N Published as a conference paper at ICLR 2022 Sigma Linear Q K V MatMul Scale SoftMax Global Relation Local Relation Diagonal Filling MatMul KL Distance Min KL Max KL Z KL Distance Stop gradient Sigma Q K C o n cat Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Anomaly Attention Global-Local KL Distance Reconstruction Loss Sigma Linear Q K V MatMul Scale SoftMax Global-Series Association Adjacent-Series Association Diagonalize MatMul KL Distance Min KL Max KL X KL Distance Stop gradient Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop Grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 ( , 21 ) ( , 22 ) ( , 2 ) 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 … Scale Grad Grad Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 ( , 21 ) ( , 22 ) ( , 2 ) 2 N ( 1 , 21 ) N ( 2 , 22 ) N ( N , 2N ) 2 … Scale grad grad Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop Grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x … Rescale Grad Grad 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) G ( |j − 2| ; σ2 ) G ( |j − N | ; σN ) 参考文献 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) ( |j 2| ; σ2 ) | | ; 作业要求 本次作业为单人作业（需独立完成）。 如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。 由于作业 较晚，本次作业《最终报告与代码》不允许迟交。 在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。 G ( |j − 1| ; σ1 ) G ( |j − 2| ; σ2 ) G ( |j − N | ; σN ) Figure 1 : Anomaly Transformer architecture . Anomaly-Attention ( left ) models the prior-association and series-association simultaneously . In addition to the reconstruction loss , our model is also optimized by the minimax strategy with a specially-designed stop-gradient mechanism ( gray arrows ) to constrain the prior- and series- associations for more distinguishable association discrepancy . input time series X ∈ RN×d . The overall equations of the l-th layer are formalized as : Z l = Layer-Norm ( Anomaly-Attention ( X l−1 ) + X l−1 ) X l = Layer-Norm ( Feed-Forward ( Z l ) + Z l ) , ( 1 ) where X l ∈ RN×dmodel , l ∈ { 1 , · · · , L } denotes the output of the l-th layer with dmodel channels . The initial input X 0 = Embedding ( X ) represents the embedded raw series . Z l ∈ RN×dmodel is the l-th layer ’ s hidden representation . Anomaly-Attention ( · ) is to compute the association discrepancy . Anomaly-Attention Note that the single-branch self-attention mechanism ( Vaswani et al. , 2017 ) can not model the prior-association and series-association simultaneously . We propose the AnomalyAttention with a two-branch structure ( Figure 1 ) . For the prior-association , we adopt a learnable Gaussian kernel to calculate the prior with respect to the relative temporal distance . Benefiting from the unimodal property of the Gaussian kernel , this design can pay more attention to the adjacent horizon constitutionally . We also use a learnable scale parameter σ for the Gaussian kernel , making the prior-associations adapt to the various time series patterns , such as different lengths of anomaly segments . The series-association branch is to learn the associations from raw series , which can find the most effective associations adaptively . Note that these two forms maintain the temporal dependencies of each time point , which are more informative than point-wise representation . They also reflect the adjacent-concentration prior and the learned real associations respectively , whose discrepancy shall be normal-abnormal distinguishable . The Anomaly-Attention in the l-th layer is : Initialization : Q , K , V , σ = X l−1W lQ , X l−1W lK , X l−1W lV , X l−1W lσ Prior-Association : P l = Rescale ( [ 1√ 2πσi exp ( −|j − i| 2 2σ2i ) ] i , j∈ { 1 , ··· , N } ) Series-Association : Sl = Softmax ( QKT√ dmodel ) Reconstruction : Ẑ l = SlV , ( 2 ) where Q , K , V ∈ RN×dmodel , σ ∈ RN×1 represent the query , key , value of self-attention and the learned scale respectively . W lQ , W l K , W l V ∈ Rdmodel×dmodel , W lσ ∈ Rdmodel×1 represent the parameter matrices for Q , K , V , σ in the l-th layer respectively . Prior-association P l ∈ RN×N is generated based on the learned scale σ ∈ RN×1 and the i-th element σi corresponds to the i-th time point . Concretely , for the i-th time point , its association weight to the j-th point is calculate by the Gaussian kernel G ( |j− i| ; σi ) = 1√2πσi exp ( − |j−i|2 2σ2i ) w.r.t . the distance |j− i| . Further , we use Rescale ( · ) to transform the association weights to discrete distributions P l by dividing the row sum . Sl ∈ RN×N denotes the series-associations . Softmax ( · ) normalizes the attention map along the last dimension . 4 Published as a conference paper at ICLR 2022 Association Discrepancy b . Association Discrepancy d. Min KL Max KL a. Min KL Max KL c. Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Normal Time PointsAbnormal Time Points Maximize PhaseMinimize Phase Association Discrepancy Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Adjacet-series Association Global-series Association Constrained by Reconstruction Global-series Association Adjacet-series Association Within the Gaussian Family Time series Time series Optimization Direction Global-series Association Adjacet-series Association Time series Adjacet-series Association Maximize PhaseMinimize Phase Association Discrepancy Prior-Association Series-AssociationGaussian Family Reconstruction Loss Constrained Space PriorAssociation SeriesAssociation Constrained by Reconstruction SeriesAssociation PriorAssociation Within the Gaussian Family Time series Time series Optimization Direction Association Discrepancy Thus , each row of Sl forms a discrete distribution . Ẑ l ∈ RN×dmodel is the hidden representation after the Anomaly-Attention in the l-th layer . We use Anomaly-Attention ( · ) to summarize Equation 2 . In the multi-head version that we use , the learned scale is σ ∈ RN×h for h heads . Qm , Km , Vm ∈ RN× dmodel h denote the query , key and value of the m-th head respectively . The block concatenates the outputs { Ẑ lm ∈ RN× dmodel h } 1≤m≤h from multiple heads and gets the final result Ẑ l ∈ RN×dmodel . Association Discrepancy We formalize the Association Discrepancy as the symmetrized KL divergence between prior- and series- associations , which represents the information gain between these two distributions ( Neal , 2007 ) . We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as : AssDis ( P , S ; X ) = [ 1 L L∑ l=1 ( KL ( P li , :‖Sli , : ) + KL ( Sli , :‖P li , : ) ) ] i=1 , ··· , N ( 3 ) where KL ( ·‖· ) is the KL divergence computed between two discrete distributions corresponding to every row of P l and Sl . AssDis ( P , S ; X ) ∈ RN×1 is the point-wise association discrepancy of X with respect to prior-association P and series-association S from multiple layers . The i-th element of results corresponds to the i-th time point ofX . From previous observation , anomalies will present smaller AssDis ( P , S ; X ) than normal time points , which makes AssDis inherently distinguishable .

Summary: This paper proposes a new anomaly detection approach based on a Transformer architecture. The main idea is to leverage self-attention to capture the temporal dependency structure of the observations in a sliding window as a measure of anomaly. Since anomalies can be defined as a sequence that is inconsistent with regular ones, the attention matrix is expected to reflect some major aspects of anomalies.   The authors propose a two-branch attention architecture to handle multi-dimensional real-valued time-series data, where prior- and series-attention matrices are computed and utilized in a certain min-max competitive learning framework.   The final anomaly score is defined as the product between the re-construction error and the KL distance between the two attention matrices.