ANOMALY TRANSFORMER: TIME SERIES ANOMALY DETECTION WITH ASSOCIATION DISCREPANCY
Year: 2022

Abstract:
Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to derive a distinguishable criterion. Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufficient to reason about the intricate dynamics. Recently, Transformers have shown great power in unified modeling of pointwise representation and pairwise association, and we find that the self-attention weight distribution of each time point can embody rich association with the whole series. Our key observation is that due to the rarity of anomalies, it is extremely difficult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies’ associations shall mainly concentrate on their adjacent time points. This adjacent-concentration bias implies an association-based criterion inherently distinguishable between normal and abnormal points, which we highlight through the Association Discrepancy. Technically, we propose the Anomaly Transformer with a new Anomaly-Attention mechanism to compute the association discrepancy. A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy. The Anomaly Transformer achieves state-of-theart results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space & earth exploration, and water treatment.

heading: 1 INTRODUCTION
text: Real-world systems always work in a continuous way, which can generate several successive measurements monitored by multi-sensors, such as industrial equipment, space probe, etc. Discovering the malfunctions from large-scale system monitoring data can be reduced to detecting the abnormal time points from time series, which is quite meaningful for ensuring security and avoiding financial loss. But anomalies are usually rare and hidden by vast normal points, making the data labeling hard and expensive. Thus, we focus on time series anomaly detection under the unsupervised setting.
Unsupervised time series anomaly detection is extremely challenging in practice. The model should learn informative representations from complex temporal dynamics through unsupervised tasks. Still, it should also derive a distinguishable criterion that can detect the rare anomalies from plenty of normal time points. Various classic anomaly detection methods have provided many unsupervised paradigms, such as the density-estimation methods proposed in local outlier factor (LOF, Breunig et al. (2000)), clustering-based methods presented in one-class SVM (OC-SVM, Schölkopf et al. (2001)) and SVDD (Tax & Duin, 2004). These classic methods do not consider the temporal information and are difficult to generalize to unseen real scenarios. Benefiting from the representation learning capability of neural networks, recent deep models (Su et al., 2019; Shen et al., 2020; Li et al., 2021) have achieved superior performance. A major category of methods focus on learning pointwise representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task. Here, a natural and practical anomaly criterion is the pointwise reconstruction or prediction error. However, due to the rarity of anomalies, the pointwise representation is less informative for complex temporal patterns and can be dominated by normal time points, making anomalies less distinguishable. Also, the reconstruction or prediction error is calculated point by point, which cannot provide a comprehensive description of the temporal context.
∗Equal Contribution
Another major category of methods detect anomalies based on explicit association modeling. The vector autoregression and state space models fall into this category. The graph was also used to capture the association explicitly, through representing time series with different time points as vertices and detecting anomalies by random walk (Cheng et al., 2008; 2009). In general, it is hard for these classic methods to learn informative representations and model fine-grained associations. Recently, graph neural network (GNN) has been applied to learn the dynamic graph among multiple variables in multivariate time series (Zhao et al., 2020; Deng & Hooi, 2021). While being more expressive, the learned graph is still limited to a single time point, which is insufficient for complex temporal patterns. Besides, subsequence-based methods detect anomalies by calculating the similarity among subsequences (Boniol & Palpanas, 2020). While exploring wider temporal context, these methods cannot capture the fine-grained temporal association between each time point and the whole series.
In this paper, we adapt Transfomers (Vaswani et al., 2017) to time series anomaly detection in the unsupervised regime. Transformers have achieved great progress in various areas, including natural language processing (Brown et al., 2020), machine vision (Liu et al., 2021) and time series (Zhou et al., 2021). This success is attributed to its great power in unified modeling of global representation and long-range relation. Applying Transformers to time series, we find that the temporal association of each time point can be obtained from the self-attention map, which presents as a distribution of its association weights to all the time points along the temporal dimension. The association distribution of each time point can provide a more informative description for the temporal context, indicating dynamic patterns, such as the period or trend of time series. We name the above association distribution as the series-association, which can be discovered from the raw series by Transformers
Further, we observe that due to the rarity of anomalies and the dominance of normal patterns, it is harder for anomalies to build strong associations with the whole series. The associations of anomalies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity. Such an adjacent-concentration inductive bias is referred to as the prior-association. In contrast, the dominating normal time points can discover informative associations with the whole series, not limiting to the adjacent area. Based on this observation, we try to utilize the inherent normal-abnormal distinguishability of the association distribution. This leads to a new anomaly criterion for each time point, quantified by the distance between each time point’s prior-association and its series-association, named as Association Discrepancy. As aforementioned, because the associations of anomalies are more likely to be adjacent-concentrating, anomalies will present a smaller association discrepancy than normal time points.
Go beyond previous methods, we introduce Transformers to the unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning. To compute the Association Discrepancy, we renovate the self-attention mechanism to the Anomaly-Attention, which contains a two-branch structure to model the prior-association and series-association of each time point respectively. The prior-association employs the learnable Gaussian kernel to present the adjacentconcentration inductive bias of each time point, while the series-association corresponds to the selfattention weights learned from raw series. Besides, a minimax strategy is applied between the two branches, which can amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based criterion. Anomaly Transformer achieves strong results on six benchmarks, covering three real applications. The contributions are summarized as follows:
• Based on the key observation of Association Discrepancy, we propose the Anomaly Transformer with an Anomaly-Attention mechanism, which can model the prior-association and series-association simultaneously to embody the Association Discrepancy.
• We propose a minimax strategy to amplify the normal-abnormal distinguishability of the Association Discrepancy and further derive a new association-based detection criterion.
• Anomaly Transformer achieves the state-of-the-art anomaly detection results on six benchmarks for three real applications. Extensive ablations and insightful case studies are given.

heading: 2 RELATED WORK

heading: 2.1 UNSUPERVISED TIME SERIES ANOMALY DETECTION
text: As a vital real-world problem, unsupervised time series anomaly detection has been widely explored. Categorizing by anomaly determination criterion, the paradigms roughly include the densityestimation, clustering-based, reconstruction-based and autoregression-based methods.
As for the density-estimation methods, the classic methods local outlier factor (LOF, Breunig et al. (2000)) and connectivity outlier factor (COF, Tang et al. (2002)) calculates the local density and local connectivity for outlier determination respectively. DAGMM (Zong et al., 2018) and MPPCACD (Yairi et al., 2017) integrate the Gaussian Mixture Model to estimate the density of representations.
In clustering-based methods, the anomaly score is always formalized as the distance to cluster center. SVDD (Tax & Duin, 2004) and Deep SVDD (Ruff et al., 2018) gather the representations from normal data to a compact cluster. THOC (Shen et al., 2020) fuses the multi-scale temporal features from intermediate layers by a hierarchical clustering mechanism and detects the anomalies by the multi-layer distances. ITAD (Shin et al., 2020) conducts the clustering on decomposed tensors.
The reconstruction-based models attempt to detect the anomalies by the reconstruction error. Park et al. (2018) presented the LSTM-VAE model that employed the LSTM backbone for temporal modeling and the Variational AutoEncoder (VAE) for reconstruction. OmniAnomaly proposed by Su et al. (2019) further extends the LSTM-VAE model with a normalizing flow and uses the reconstruction probabilities for detection. InterFusion from Li et al. (2021) renovates the backbone to a hierarchical VAE to model the inter- and intra- dependency among multiple series simultaneously. GANs (Goodfellow et al., 2014) are also used for reconstruction-based anomaly detection (Schlegl et al., 2019; Li et al., 2019a; Zhou et al., 2019) and perform as an adversarial regularization.
The autoregression-based models detect the anomalies by the prediction error. VAR extends ARIMA (Anderson & Kendall, 1976) and predicts the future based on the lag-dependent covariance. The autoregressive model can also be replaced by LSTMs (Hundman et al., 2018; Tariq et al., 2019).
This paper is characterized by a new association-based criterion. Different from the random walk and subsequence-based methods (Cheng et al., 2008; Boniol & Palpanas, 2020), our criterion is embodied by a co-design of the temporal models for learning more informative time-point associations.

heading: 2.2 TRANSFORMERS FOR TIME SERIES ANALYSIS
text: Recently, Transformers (Vaswani et al., 2017) have shown great power in sequential data processing, such as natural language processing (Devlin et al., 2019; Brown et al., 2020), audio processing (Huang et al., 2019) and computer vision (Dosovitskiy et al., 2021; Liu et al., 2021). For time series analysis, benefiting from the advantage of the self-attention mechanism, Transformers are used to discover the reliable long-range temporal dependencies (Kitaev et al., 2020; Li et al., 2019b; Zhou et al., 2021; Wu et al., 2021). Especially for time series anomaly detection, GTA proposed by Chen et al. (2021) employs the graph structure to learn the relationship among multiple IoT sensors, as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detection. Unlike the previous usage of Transformers, Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy.

heading: 3 METHOD
text: Suppose monitoring a successive system of dmeasurements and recording the equally spaced observations over time. The observed time series X is denoted by a set of time points {x1, x2, · · · , xN}, where xt ∈ Rd represents the observation of time t. The unsupervised time series anomaly detection problem is to determine whether xt is anomalous or not without labels.
As aforementioned, we highlight the key to unsupervised time series anomaly detection as learning informative representations and finding distinguishable criterion. We propose the Anomaly Transformer to discover more informative associations and tackle this problem by learning the Association Discrepancy, which is inherently normal-abnormal distinguishable. Technically, we propose the Anomaly-Attention to embody the prior-association and series-associations, along with a minimax optimization strategy to obtain a more distinguishable association discrepancy. Co-designed with the architecture, we derive an association-based criterion based on the learned association discrepancy.

heading: 3.1 ANOMALY TRANSFORMER
text: Given the limitation of Transformers (Vaswani et al., 2017) for anomaly detection, we renovate the vanilla architecture to the Anomaly Transformer (Figure 1) with an Anomaly-Attention mechanism.
Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attention blocks and feed-forward layers alternately. This stacking structure is conducive to learning underlying associations from deep multi-level features. Suppose the model contains L layers with length-N
Published as a conference paper at ICLR 2022
Sigma Linear Q K V MatMul Scale SoftMax Global Relation Local Relation Diagonal Filling MatMul
KL Distance Min KL Max KL Z KL Distance Stop gradient Sigma Q K
C o n cat
Lin ear
Anomaly Attention Layer Norm Feed Forward Layer Norm + + Anomaly Attention
Global-Local KL Distance Reconstruction Loss Sigma Linear Q K V MatMul Scale SoftMax Global-Series Association Adjacent-Series Association Diagonalize MatMul KL Distance Min KL Max KL X KL Distance Stop gradient Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x Linear Q K V MatMul Scale SoftMax SeriesAssociation PriorAssociation MatMul Minimize Discrepancy X Maximize Discrepancy Stop Grad Lin ear Anomaly Attention Layer Norm Feed Forward Layer Norm + + Reconstruction L x
2
N (1, 21) N (2, 22) N (N, 2N )
2
( , 21) ( , 22) ( , 2 )
2
N (1, 21) N (2, 22) N (N, 2N )
2
… Scale
Grad Grad
Linear
Q
K
V
MatMul Scale SoftMax SeriesAssociation
PriorAssociation
MatMul
Minimize Discrepancy
X
Maximize Discrepancy
Stop grad
Lin ear
Anomaly Attention
Layer Norm
Feed Forward Layer Norm
+
+ Reconstruction
L x
2
N (1, 21) N (2, 22) N (N, 2N )
2
( , 21) ( , 22) ( , 2 )
2
N (1, 21) N (2, 22) N (N, 2N )
2
… Scale
grad
grad
Linear
Q
K
V
MatMul Scale SoftMax SeriesAssociation
PriorAssociation
MatMul
Minimize Discrepancy
X
Maximize Discrepancy
Stop Grad
Lin ear
Anomaly Attention
Layer Norm
Feed Forward
Layer Norm
+
+
Reconstruction
L x
…
Rescale
Grad
Grad
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
G(|j − 1|; σ1) G(|j − 2|; σ2) G(|j − N |; σN )
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
G(|j − 1|; σ1) (|j 2|; σ2) | |;
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
G(|j − 1|; σ1) G(|j − 2|; σ2) G(|j − N |; σN )
Figure 1: Anomaly Transformer architecture. Anomaly-Attention (left) models the prior-association and series-association simultaneously. In addition to the reconstruction loss, our model is also optimized by the minimax strategy with a specially-designed stop-gradient mechanism (gray arrows) to constrain the prior- and series- associations for more distinguishable association discrepancy.
input time series X ∈ RN×d. The overall equations of the l-th layer are formalized as: Z l = Layer-Norm ( Anomaly-Attention(X l−1) + X l−1 ) X l = Layer-Norm ( Feed-Forward(Z l) + Z l ) ,
(1)
where X l ∈ RN×dmodel , l ∈ {1, · · · , L} denotes the output of the l-th layer with dmodel channels. The initial input X 0 = Embedding(X ) represents the embedded raw series. Z l ∈ RN×dmodel is the l-th layer’s hidden representation. Anomaly-Attention(·) is to compute the association discrepancy. Anomaly-Attention Note that the single-branch self-attention mechanism (Vaswani et al., 2017) cannot model the prior-association and series-association simultaneously. We propose the AnomalyAttention with a two-branch structure (Figure 1). For the prior-association, we adopt a learnable Gaussian kernel to calculate the prior with respect to the relative temporal distance. Benefiting from the unimodal property of the Gaussian kernel, this design can pay more attention to the adjacent horizon constitutionally. We also use a learnable scale parameter σ for the Gaussian kernel, making the prior-associations adapt to the various time series patterns, such as different lengths of anomaly segments. The series-association branch is to learn the associations from raw series, which can find the most effective associations adaptively. Note that these two forms maintain the temporal dependencies of each time point, which are more informative than point-wise representation. They also reflect the adjacent-concentration prior and the learned real associations respectively, whose discrepancy shall be normal-abnormal distinguishable. The Anomaly-Attention in the l-th layer is:
Initialization: Q,K,V, σ = X l−1W lQ,X l−1W lK,X l−1W lV ,X l−1W lσ Prior-Association: P l = Rescale ([
1√ 2πσi exp
( −|j − i| 2
2σ2i )] i,j∈{1,··· ,N} )
Series-Association: Sl = Softmax ( QKT√
dmodel ) Reconstruction: Ẑ l = SlV,
(2)
where Q,K,V ∈ RN×dmodel , σ ∈ RN×1 represent the query, key, value of self-attention and the learned scale respectively. W lQ,W l K,W l V ∈ Rdmodel×dmodel ,W lσ ∈ Rdmodel×1 represent the parameter matrices for Q,K,V, σ in the l-th layer respectively. Prior-association P l ∈ RN×N is generated based on the learned scale σ ∈ RN×1 and the i-th element σi corresponds to the i-th time point. Concretely, for the i-th time point, its association weight to the j-th point is calculate by the Gaussian kernel G(|j− i|;σi) = 1√2πσi exp(− |j−i|2 2σ2i
) w.r.t. the distance |j− i|. Further, we use Rescale(·) to transform the association weights to discrete distributions P l by dividing the row sum. Sl ∈ RN×N denotes the series-associations. Softmax(·) normalizes the attention map along the last dimension.
4
Published as a conference paper at ICLR 2022
Association Discrepancy b.
Association Discrepancy d. Min KL Max KL a. Min KL Max KL c. Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Normal Time PointsAbnormal Time Points Maximize PhaseMinimize Phase Association Discrepancy Association Space Adjacet-series Association Global-series Association Gaussian Family Reconstruction Loss Constrained Space Adjacet-series Association Global-series Association Constrained by Reconstruction Global-series Association Adjacet-series Association Within the Gaussian Family Time series Time series Optimization Direction
Global-series Association
Adjacet-series Association
Time series
Adjacet-series Association
Maximize PhaseMinimize Phase Association Discrepancy Prior-Association Series-AssociationGaussian Family Reconstruction Loss Constrained Space PriorAssociation SeriesAssociation
Constrained by Reconstruction SeriesAssociation PriorAssociation Within the Gaussian Family Time series Time series Optimization Direction Association Discrepancy
Thus, each row of Sl forms a discrete distribution. Ẑ l ∈ RN×dmodel is the hidden representation after the Anomaly-Attention in the l-th layer. We use Anomaly-Attention(·) to summarize Equation 2. In the multi-head version that we use, the learned scale is σ ∈ RN×h for h heads. Qm,Km,Vm ∈ RN× dmodel h denote the query, key and value of the m-th head respectively. The block concatenates the outputs {Ẑ lm ∈ RN× dmodel h }1≤m≤h from multiple heads and gets the final result Ẑ l ∈ RN×dmodel .
Association Discrepancy We formalize the Association Discrepancy as the symmetrized KL divergence between prior- and series- associations, which represents the information gain between these two distributions (Neal, 2007). We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as:
AssDis(P,S;X ) = [ 1
L L∑ l=1 ( KL(P li,:‖Sli,:) + KL(Sli,:‖P li,:) )] i=1,··· ,N
(3)
where KL(·‖·) is the KL divergence computed between two discrete distributions corresponding to every row of P l and Sl. AssDis(P,S;X ) ∈ RN×1 is the point-wise association discrepancy of X with respect to prior-association P and series-association S from multiple layers. The i-th element of results corresponds to the i-th time point ofX . From previous observation, anomalies will present smaller AssDis(P,S;X ) than normal time points, which makes AssDis inherently distinguishable.

heading: 3.2 MINIMAX ASSOCIATION LEARNING
text: As an unsupervised task, we employ the reconstruction loss for optimizing our model. The reconstruction loss will guide the series-association to find the most informative associations. To further amplify the difference between normal and abnormal time points, we also use an additional loss to enlarge the association discrepancy. Due to the unimodal property of the prior-association, the discrepancy loss will guide the series-association to pay more attention to the non-adjacent area, which makes the reconstruction of anomalies harder and makes anomalies more identifiable. The loss function for input series X ∈ RN×d is formalized as:
LTotal(X̂ ,P,S, λ;X ) = ‖X − X̂‖2F − λ× ‖AssDis(P,S;X )‖1 (4) where X̂ ∈ RN×d denotes the reconstruction ofX . ‖·‖F, ‖·‖k indicate the Frobenius and k-norm. λ is to trade off the loss terms. When λ > 0, the optimization is to enlarge the association discrepancy. A minimax strategy is proposed to make the association discrepancy more distinguishable.
Minimax Strategy Note that directly maximizing the association discrepancy will extremely reduce the scale parameter of the Gaussian kernel (Neal, 2007), making the prior-association meaningless. Towards a better control of association learning, we propose a minimax strategy (Figure 2). Concretely, for the minimize phase, we drive the prior-association P l to approximate the seriesassociation Sl that is learned from raw series. This process will make the prior-association adapt to various temporal patterns. For the maximize phase, we optimize the series-association to enlarge the association discrepancy. This process forces the series-association to pay more attention to the non-adjacent horizon. Thus, integrating the reconstruction loss, the loss functions of two phases are:
Minimize Phase: LTotal(X̂ ,P,Sdetach,−λ;X ) Maximize Phase: LTotal(X̂ ,Pdetach,S, λ;X ),
(5)
where λ > 0 and ∗detach means to stop the gradient backpropagation of the association (Figure 1). As P approximates Sdetach in the minimize phase, the maximize phase will conduct a stronger constraint to the series-association, forcing the time points to pay more attention to the non-adjacent area. Under the reconstruction loss, this is much harder for anomalies to achieve than normal time points, thereby amplifying the normal-abnormal distinguishability of the association discrepancy.
Association-based Anomaly Criterion We incorporate the normalized association discrepancy to the reconstruction criterion, which will take the benefits of both temporal representation and the distinguishable association discrepancy. The final anomaly score ofX ∈ RN×d is shown as follows:
AnomalyScore(X ) = Softmax ( −AssDis(P,S;X ) ) [ ‖Xi,: − X̂i,:‖22 ] i=1,··· ,N
(6)
where is the element-wise multiplication. AnomalyScore(X ) ∈ RN×1 denotes the point-wise anomaly criterion of X . Towards a better reconstruction, anomalies usually decrease the association discrepancy, which will still derive a higher anomaly score. Thus, this design can make the reconstruction error and the association discrepancy collaborate to improve detection performance.

heading: 4 EXPERIMENTS
text: We extensively evaluate Anomaly Transformer on six benchmarks for three practical applications.
Datasets Here is a description of the six experiment datasets: (1) SMD (Server Machine Dataset, Su et al. (2019)) is a 5-week-long dataset that is collected from a large Internet company with 38 dimensions. (2) PSM (Pooled Server Metrics, Abdulaal et al. (2021)) is collected internally from multiple application server nodes at eBay with 26 dimensions. (3) Both MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) are public datasets from NASA (Hundman et al., 2018) with 55 and 25 dimensions respectively, which contain the telemetry anomaly data derived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems. (4) SWaT (Secure Water Treatment, Mathur & Tippenhauer (2016)) is obtained from 51 sensors of the critical infrastructure system under continuous operations. (5) NeurIPS-TS (NeurIPS 2021 Time Series Benchmark) is a dataset proposed by Lai et al. (2021) and includes five time series anomaly scenarios categorized by behavior-driven taxonomy as point-global, pattern-contextual, pattern-shapelet, pattern-seasonal and pattern-trend. The statistical details are summarized in Table 13 of Appendix.
Implementation details Following the well-established protocol in Shen et al. (2020), we adopt a non-overlapped sliding window to obtain a set of sub-series. The sliding window is with a fixed size of 100 for all datasets. We label the time points as anomalies if their anomaly scores (Equation 6) are larger than a certain threshold δ. The threshold δ is determined to make r proportion data of the validation dataset labeled as anomalies. For the main results, we set r = 0.1% for SWaT, 0.5% for SMD and 1% for other datasets. We adopt the widely-used adjustment strategy (Xu et al., 2018; Su et al., 2019; Shen et al., 2020): if a time point in a certain successive abnormal segment is detected, all anomalies in this abnormal segment are viewed to be correctly detected. This strategy is justified from the observation that an abnormal time point will cause an alert and further make the whole segment noticed in real-world applications. Anomaly Transformer contains 3 layers. We set the channel number of hidden states dmodel as 512 and the number of heads h as 8. The hyperparameter λ (Equation 4) is set as 3 for all datasets to trade-off two parts of the loss function. We use the ADAM (Kingma & Ba, 2015) optimizer with an initial learning rate of 10−4. The training process is early stopped within 10 epochs with the batch size of 32. All the experiments are implemented in Pytorch (Paszke et al., 2019) with a single NVIDIA TITAN RTX 24GB GPU.
Baselines We extensively compare our model with 18 baselines, including the reconstructionbased models: InterFusion (2021), BeatGAN (2019), OmniAnomaly (2019), LSTM-VAE (2018); the density-estimation models: DAGMM (2018), MPPCACD (2017), LOF (2000); the clusteringbased methods: ITAD (2020), THOC (2020), Deep-SVDD (2018); the autoregression-based models: CL-MPPCA (2019), LSTM (2018), VAR (1976); the classic methods: OC-SVM (2004), IsolationForest (2008). Another 3 baselines from change point detection and time series segmentation are deferred to Appendix I. InterFusion (2021) and THOC (2020) are the state-of-the-art deep models.

heading: 4.1 MAIN RESULTS
text: Real-world datasets We extensively evaluate our model on five real-world datasets with ten competitive baselines. As shown in Table 1, Anomaly Transformer achieves the consistent state-of-theart on all benchmarks. We observe that deep models that consider the temporal information outperform the general anomaly detection model, such as Deep-SVDD (Ruff et al., 2018) and DAGMM (Zong et al., 2018), which verifies the effectiveness of temporal modeling. Our proposed Anomaly Transformer goes beyond the point-wise representation learned by RNNs and models the more informative associations. The results in Table 1 are persuasive for the advantage of association learning in time series anomaly detection. In addition, we plot the ROC curve in Figure 3 for a complete comparison. Anomaly Transformer has the highest AUC values on all five datasets. It means that our model performs well in the false-positive and true-positive rates under various pre-selected thresholds, which is important for real-world applications.
Ano form
er Om niAn oma ly Dee p-S VDD THO C LST M-V AE DAG MM Bea tGA N
50
55
60
65
70
75
71.31
60.14
56.51
62.52
58.39
51.77
67.45
50
55
60
65
70
75
An om
aly
Tr an sfo rm er Om
niAn om aly De ep - SV DD TH OC LS TM -V AE DA GM M Be atG AN
F1 -S
co re
(% )
Figure 4: Results for NeurIPS-TS.
NeurIPS-TS benchmark This benchmark is generated from well-designed rules proposed by Lai et al. (2021), which completely includes all types of anomalies, covering both the pointwise and pattern-wise anomalies. As shown in Figure 4, Anomaly Transformer can still achieve state-of-the-art performance. This verifies the effectiveness of our model on various anomalies.
Ablation study As shown in Table 2, we further investigate the effect of each part in our model. Our association-based criterion outperforms the widely-used reconstruction criterion consistently.
Specifically, the association-based criterion brings a remarkable 18.76% (76.20→94.96) averaged absolute F1-score promotion. Also, directly taking the association discrepancy as the criterion still achieves a good performance (F1-score: 91.55%) and surpasses the previous state-of-the-art model
THOC (F1-score: 88.01% calculated from Table 1). Besides, the learnable prior-association (corresponding to σ in Equation 2) and the minimax strategy can further improve our model and get 8.43% (79.05→87.48) and 7.48% (87.48→94.96) averaged absolute promotions respectively. Finally, our proposed Anomaly Transformer surpasses the pure Transformer by 18.34% (76.62→94.96) absolute improvement. These verify that each module of our design is effective and necessary. More ablations of association discrepancy can be found in Appendix D.

heading: 4.2 MODEL ANALYSIS
text: To explain how our model works intuitively, we provide the visualization and statistical results for our three key designs: anomaly criterion, learnable prior-association and optimization strategy.
0 20 40 60 80 100 length
0
0.05
0.1
0.15
va lu e
60 100 Time
0.1
.
0
F in
a l L o ss
0 20 40 60 80 100 length
0
2
4
6
8
10
va lu e R ec o n st
ru ct
io n L
o ss
Time 0 20 40 60 80 100 length
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
va lu e
00 Time
0
0.2
0.4
1
R ec
o n
st ru
ct io
n L
o ss
0 20 40 60 80 100 length
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
va lu e
Time 800 Time
0
.
.
R ec
o n
st ru
ct io
n L
o ss
0 20 40 60 80 100 length
0
0.5
1
1.5
2
2.5
va lu e
100 Time
R ec
o n st
ru ct
io n L
o ss
0
1
.
2
.
0 20 40 60 80 100 length
0
0.5
1
1.5
va lu e
80 0 Time
R ec
o n
st ru
ct io
n L
o ss
0
0.5
1.5
2
0 20 40 60 80 100 length
-1
0
1
2
3
4
va lu e
Time
V al ue
0 20 40 60 80 100 length
-1.5
-1
-0.5
0
0.5
1
1.5
va lu e
.5
1
0
.5
-1
.5
V al ue
0 Time
0 20 40 60 80 100 length
-1
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4 -0.3 va lu e
Time
-1
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4 -0.3 V al ue
0 20 40 60 80 100 length
-1.5
-1
-0.5
0
0.5
1
1.5
va lu e
Time
V al ue
- .
-1
- .
0
1
0 20 40 60 80 100 length
-1.5
-1
-0.5
0
0.5
1
1.5
va lu e
0 2 8 10 Time
V al ue
-1.5
-1
-0.5
0
0.
1
1.
0 20 40 60 80 100 0
2
4
6
8
10
12
14
16
va lu e
0 20 40 60 80 100 length
0
0.05
0.1
0.15
0.2
0.25
0.3
va lu e Fi n al
L o ss
0
.
0.1
0.2
.
0.3
Time 0 20 40 60 80 100 length
0
0.02
0.04
0.06
0.08
va lu e
1
Time
Fi n
al L
o ss
0.02
0.04
0.06
0.08
0
0 20 40 60 80 100 length
0
0.02
0.04
0.06
va lu e
0 20 40 60 80 100 Time
F in
a l L o ss
0.02
0.04
0.06
0
Point
Global Contextual Shapelet Seasonal Trend
Pattern Time Series Anomaly
0 0
0
200 40 60 80 100 200 40 60 80 100 200 40 60 80 100 i e Time Time Time Ti e
i e i Time Time Time
Ti e Time Time Time
2
0
-1
1
2
0
0.3
0.2
0.1
0
.
- .
.
0.7
0.3
0.1
0.03
0.02
0.01
.
-0.5
- .
1.6
1.
0.
0.
0
0. 6
. 3
.
.
- .
-
1.5
- .
- .
- .
1.0
0.5
0.08
.06
.
In p u t
Ti m
e Se
ri es
R ec
o n st
ru ct
io n
C ri
te ri
o n
A ss
o ci
at io
n -b
as ed
C ri
te ri
o n
Detection Error
Detection Error
0.04
0
Detection Error Detection
Error
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
作业要求
本次作业为单人作业（需独立完成）。
如与他人交流或使用开源代码、论文，请在报告中说明或给出引用。
由于作业 较晚，本次作业《最终报告与代码》不允许迟交。
在项目中遇到任何困难，如选题、数据集挑选或者技术方案设计，请及时和助教联系。
δ = 0.03
δ = 0.5
参考文献
Figure 5: Visualization of different anomaly categories (Lai et al., 2021). We plot the raw series (first row) from NeurIPS-TS dataset, as well as their corresponding reconstruction (second row) and association-based criteria (third row). The point-wise anomalies are marked by red circles and the pattern-wise anomalies are in red segments. The wrongly detected cases are bounded by red boxes.
Anomaly criterion visualization To get more intuitive cases about how association-based criterion works, we provide some visualization in Figure 5 and explore the criterion performance under different types of anomalies, where the taxonomy is from Lai et al. (2021). We can find that our proposed association-based criterion is more distinguishable in general. Concretely, the associationbased criterion can obtain the consistent smaller values for the normal part, which is quite contrasting
in point-contextual and pattern-seasonal cases (Figure 5). In contrast, the jitter curves of the reconstruction criterion make the detection process confused and fail in the aforementioned two cases. This verifies that our criterion can highlight the anomalies and provide distinct values for normal and abnormal points, making the detection precise and reducing the false-positive rate.
Prior-association visualization During the minimax optimization, the prior-association is learned to get close to the series-association. Thus, the learned σ can reflect the adjacent-concentrating degree of time series. As shown in Figure 6, we find that σ changes to adapt to various data patterns of time series. Especially, the prior-association of anomalies generally has a smaller σ than normal time points, which matches our adjacent-concentration inductive bias of anomalies.
Optimization strategy analysis Only with the reconstruction loss, the abnormal and normal time points present similar performance in the association weights to adjacent time points, corresponding to a contrast value closed to 1 (Table 3). Maximizing the association discrepancy will force the series-associations to pay more attention to the non-adjacent area. However, to obtain a better reconstruction, the anomalies have to maintain much larger adjacent association weights than normal time points, corresponding to a larger contrast value. But direct maximization will cause the optimization problem of Gaussian kernel, cannot strongly amplify the difference between normal and abnormal time points as expected (SMD:1.15→1.27). The minimax strategy optimizes the prior-association to provide a stronger constraint to series-association. Thus, the minimax strategy obtains more distinguishable contrast values than direct maximization (SMD:1.27→2.39) and thereby performs better.

heading: 5 CONCLUSION AND FUTURE WORK
text: This paper studies the unsupervised time series anomaly detection problem. Unlike previous methods, we learn the more informative time-point associations by Transformers. Based on the key observation of association discrepancy, we propose the Anomaly Transformer, including an AnomalyAttention with the two-branch structure to embody the association discrepancy. A minimax strategy is adopted to further amplify the difference between normal and abnormal time points. By introducing the association discrepancy, we propose the association-based criterion, which makes the reconstruction performance and association discrepancy collaborate. Anomaly Transformer achieves the state-of-the-art results on an exhaustive set of empirical studies. Future work includes theoretical study of Anomoly Transformer in light of classic analysis for autoregression and state space models.

heading: A PARAMETER SENSITIVITY
text: We set the window size as 100 throughout the main text, which considers the temporal information, memory and computation efficiency. And we set the loss weight λ based on the convergence property of the training curve.
Furthermore, Figure 7 provides the model performance under different choices of the window size and the loss weight. We present that our model is stable to the window size over extensive datasets (Figure 7 left). Note that a larger window size indicates a larger memory cost and a smaller sliding number. Especially, only considering the performance, its relationship to the window size can be determined by the data pattern. For example, our model performs better when the window size is 50 for the SMD dataset. Besides, we adopt the loss weight λ in Equation 5 to trade off the reconstruction loss and the association part. We find that λ is stable and easy to tune in the range of 2 to 4. The above results verify the sensitivity of our model, which is essential for applications.
1 2 3 4 5 6 factor
85
90
95 100 f-s co re
SMD MSL SMAP SWaT PSM S
100
90
85
Fsc
o re
50 100 150 200 2 0 300
Wi dow Size
1 2 3 4 5 6 factor
90
92
94
96
98
100
f-s co re
SMD MSL SMAP SWaT PSM S MSL S AP S a PS
90
92
94
96
98
Fsc o re
!
F1 -s
co re
( %
)
F1 -s
co re
( %
)
1 2 3 4 5 6
Windo Size Loss Weight
B IMPLEMENTATION DETAILS
We present the pseudo-code of Anomaly-Attention in Algorithm 1.
Algorithm 1 Anomaly-Attention Mechanism (multi-head version). Input: X ∈ RN×dmodel : input; D = ( (j − i)2 ) i,j∈{1,··· ,N}
∈ RN×N : relative distance matrix Layer params: MLPinput: linear projector for input; MLPoutput: linear projector for output
1: Q,K,V, σ = Split ( MLPinput(X ),dim=1 ) . Q,K,V ∈ RN×dmodel , σ ∈ RN×h
2: for (Qm,Km,Vm, σm) in (Q,K,V, σ): . Qm,Km,Vm ∈ RN× dmodel h , σm ∈ RN×1 3: for σm = Broadcast(σm,dim=1) . σm ∈ RN×N 4: for Pm = 1√2πσmexp ( − D2σ2m ) . Pm ∈ RN×N
5: for Pm = Pm/Broadcast ( Sum(Pm,dim=1) ) . Rescaled Pm ∈ RN×N
6: for Sm = Softmax (√
h dmodel QmKTm
) . Sm ∈ RN×N
7: for Ẑm = SmVm . Ẑm ∈ RN× dmodel h 8: Ẑ = MLPoutput ( Concat([Ẑ1, · · · , Ẑh],dim=1) ) . Ẑ ∈ RN×dmodel
9: Return Ẑ . Keep the Pm and Sm, m = 1, · · · , h

heading: C MORE SHOWCASES
text: To obtain an intuitive comparison of main results (Table 1), we visualize the criterion of various baselines. Anomaly Transformer can present the most distinguishable criterion (Figure 8). Besides,
for the real-world dataset, Anomaly Transformer can also detect the anomalies correctly. Especially for the SWaT dataset (Figure 9(d)), our model can detect the anomalies in the early stage, which is meaningful for real-world applications, such as the early warning of malfunctions.

heading: D ABLATION OF ASSOCIATION DISCREPANCY
text: We present the pseudo code of the calculation in Algorithm 2.
D.1 ABLATION OF MULTI-LEVEL QUANTIFICATION
We average the association discrepancy from multiple layers for the final results (Equation 6). We further investigate the model performance under the single-layer usage. As shown in Table 4, the multiple-layer design achieves the best, which verifies the effectiveness of multi-level quantification.
D.2 ABLATION OF STATISTICAL DISTANCE
We select the following widely-used statistical distances to calculate the association discrepancy:
• Symmetrized Kullback–Leibler Divergence (Ours). • Jensen–Shannon Divergence (JSD). • Wasserstein Distance (Wasserstein). • Cross-Entropy (CE). • L2 Distance (L2).
As shown in Table 5, our proposed definition of association discrepancy still achieves the best performance. We find that both the CE and JSD can provide fairly good results, which are close to our definition in principle and can be used to represent the information gain. The L2 distance is not suitable for the discrepancy, which overlooks the property of discrete distribution. The Wasserstein distance also fails in some datasets. The reason is that the prior-association and series-association are exactly matched in the position indexes. Still, the Wasserstein distance is not calculated point by point and considers the distribution offset, which may bring noises to the optimization and detection.
Algorithm 2 Association Discrepancy AssDis(P,S;X ) Calculation (multi-head version). Input: time series length N ; layers number L; heads number h; prior-association Pall ∈
RL×h×N×N ; series-association Sall ∈ RL×h×N×N ; 1: P ′ = Mean(P,dim=1) . P ′ ∈ RL×N×N
2: S ′ = Mean(S,dim=1) . S ′ ∈ RL×N×N 3: R′ = KL ( (P ′,S ′),dim=-1 ) + KL ( (S ′,P ′),dim=-1 ) .R′ ∈ RL×N
4: R = Mean(R′,dim=0) .R ∈ RN×1
5: ReturnR . Represent the association discrepancy of each time point
D.3 ABLATION OF PRIOR-ASSOCIATION
In addition to the Gaussian kernel with a learnable scale parameter, we also try to use the power-law kernel P (x;α) = x−α with a learnable power parameter α for prior-association, which is also a unimodal distribution. As shown in Table 6, power-law kernel can achieve a good performance in most of the datasets. However, because the scale parameter is easier to optimize than the parameter of power, Gaussian kernel still surpasses the power-law kernel consistently.

heading: E ABLATION OF ASSOCIATION-BASED CRITERION
text: We present the pseudo-code of association-based criterion in Algorithm 3.
Algorithm 3 Association-based Criterion AnomalyScore(X ) Calculation Input: time series length N ; input time series X ∈ RN×d; reconstruction time series X̂ ∈ RN×d;
association discrepancy AssDis(P,S;X ) ∈ RN×1; 1: CAD = Softmax(−AssDis(P,S;X ),dim=0) . CAD ∈ RN×1
2: CRecon = Mean ( (X − X̂ )2,dim=1 ) . CRecon ∈ RN×1
3: C = CAD × CRecon . C ∈ RN×1
4: Return C . Anomaly score for each time point
E.2 ABLATION OF CRITERION DEFINITION
We explore the model performance under different definitions of anomaly criterion, including the pure association discrepancy, pure reconstruction performance and different combination methods for association discrepancy and reconstruction performance: addition and multiplication.
Association Discrepancy: AnomalyScore(X ) = Softmax ( −AssDis(P,S;X ) ) ,
Reconstruction: AnomalyScore(X ) = [ ‖Xi,: − X̂i,:‖22 ] i=1,··· ,N ,
Addition: AnomalyScore(X ) = Softmax ( −AssDis(P,S;X ) ) + [ ‖Xi,: − X̂i,:‖22 ] i=1,··· ,N ,
Multiplication (Ours): AnomalyScore(X ) = Softmax ( −AssDis(P,S;X ) ) [ ‖Xi,: − X̂i,:‖22 ] i=1,··· ,N .
(7)
From Table 7, we find that directly using our proposed association discrepancy can also achieve a good performance, which surpasses the competitive baseline THOC (Shen et al., 2020) consistently. Besides, the multiplication combination that we used in Equation 6 performs the best, which can bring a better collaboration to the reconstruction performance and association discrepancy.

heading: F CONVERGENCE OF MINIMAX OPTIMIZATION
text: The total loss of our model (Equation 4) contains two parts: the reconstruction loss and the association discrepancy. Towards a better control of association learning, we adopt a minimax strategy for optimization (Equation 5). During the minimization phase, the optimization trends to minimize the association discrepancy and the reconstruction error. During the maximization phase, the optimization trends to maximize the association discrepancy and minimize the reconstruction error.
We plot the change curve of the above two parts during the training procedure. As shown in Figures 10 and 11, both parts of the total loss can converge within limited iterations on all the five real-world datasets. This nice convergence property is essential for the optimization of our model.
14
12
10
8
14
12
10
8
14
12
10
8
14
12
10
8
2000 100 300 400 0 400 800 0 400 800 0 400 800 0 400 800
14
12
10
8
iterations iterations iterations iterations iterations
(a) SMD (b) MSL (c) SMAP (d) SWaT (e) PSM
A ss
D is
l o ss
A ss
D is
l o ss
A ss
D is
l o ss
A ss
D is
l o ss
A ss
D is
l o ss
16
14
12
10
8
16
14
12
10
8
16
14
12
10
8
16
14
12
10
8
2000 100 300 400 0 400 800 0 400 800 0 400 800 0 400 800
16
14
12
10
8
0.2
0.5
0
0
0
2000 100 300 400 4000 200 600 4000 200 600 800 4000 200 600 4000 200 600 iterations iterations iterations iterations iterations
iterations iterations iterations iterations iterations
(a) SMD (b) MSL (c) SMAP (d) SWaT (e) PSM
(a) SMD (b) MSL (c) SMAP (d) SWaT (e) PSM
A ss
D is
l o
ss
A ss
D is
l o
ss
A ss
D is
l o
ss
A ss
D is
l o
ss
A ss
D is
l o
ss
Figure 11: Change curve of association discrepancy ‖AssDis(P,S;X )‖1 in real-world datasets during the training process.

heading: G MODEL PARAMETER SENSITIVITY
text: In this paper, we set the hyper-parameters L and dmodel following the convention of Transformers (Vaswani et al., 2017; Zhou et al., 2021).
Furthermore, to evaluate model parameter sensitivity, we investigate the performance and efficiency under different choices for the number of layers L and hidden channels dmodel. Generally, increasing the model size can obtain better results but with larger memory and computation costs.

heading: L = 2 89.26 94.33 91.72 91.89 94.73 93.29 93.79 98.91 96.28 92.37 94.59 93.47 97.22 98.23 97.72

heading: H PROTOCOL OF THRESHOLD SELECTION
text: Our paper focuses on unsupervised time series anomaly detection. Experimentally, each dataset includes training, validation and testing subsets. Anomalies are only labeled in the testing subset. Thus, we select the hyper-parameters following the Gap Statistic method (Tibshirani et al., 2001) in K-Means. Here is the selection procedure:
• After the training phase, we apply the model to the validation subset (without label) and obtain the anomaly scores (Equation 6) of all time points.
• We count the frequency of the anomaly scores in the validation subset. It is observed that the distribution of anomaly scores is separated into two clusters. We find that the cluster with a larger anomaly score contains r time points. And for our model, r is closed to 0.1%, 0.5%, 1% for SWaT, SMD and other datasets respectively (Table 10).
• Due to the size of the test subset being still inaccessible in real-world applications, we have to fix the threshold as a fixed value δ, which can gaurantee that the anomaly scores of r time points in the validation set are larger than δ and thus detected as anomalies.
Note that, directly setting the δ is also feasible. According to the intervals in Table 10, we can fix the δ as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets, which yield a quite close performance to setting r.
In real-world applications, the number of selected anomalies is always decided up to human resources. Under this consideration, setting the number of detected anomalies by the ratio r is more practical and easier to decide according to the available resources.

heading: I MORE BASELINES
text: In addition to the time series anomaly detection methods, the methods for change point detection and time series segmentation can also perform as valuable baselines. Thus, we also include the BOCPD (Adams & MacKay, 2007) and TS-CP2 (Deldari et al., 2021) from change point detection and UTime (Perslev et al., 2019) from time series segmentation for comparison. Anomaly Transformer still achieves the best performance.

heading: J LIMITATIONS AND FUTURE WORK
text: Window size As shown in the Figure 7 of Appendix A, the model may fail if the window size is too small for association learning. But the Transformers is with quadratic complexity w.r.t. the window size. The trade-off is needed for real-world applications.
Theoretical analysis As a well-established deep model, the performance of Transformers has been explored in previous works. But it is still under-exploring for the theory of complex deep models. In the future, we will explore the theorem of Anomaly Transformer for better justifications in light of classic analysis for autoregression and state space models.

heading: K DATASET
text: Here is the statistical details of experiment datasets.

heading: L UCR DATASET
text: UCR Dataset is a very challenging and comprehensive dataset provided by the Multi-dataset Time Series Anomaly Detection Competition of KDD2021 (Keogh et al., Competition of International Conference on Knowledge Discovery & Data Mining 2021). The whole dataset contains 250 subdatasets, covering various real-world scenarios. Each sub-dataset of UCR has only one anomaly segment and only has one dimension. These sub-datasets range in length from 6,684 to 900,000 and are pre-divided into training and test sets.
We also experiment on the UCR dataset for a wide evaluation. As show in Table 14, our Anomaly Transformer still achieves the state-of-the-art in this challenging benchmark.

Review summaries:
Summary: This paper is looking at anomalies in time series data. In particular they define anomalies as ones that lie outside some learned Gaussian-like distribution. They propose a minimax objective function that tries to minimize and maximize the discrepancy between a Gaussian distribution with a learned variance parameter and an “empirical” one learned directly through self-attention on the data. They measure discrepancy using the symmetric KL divergence between the two distributions and also use this for their anomaly score. 

This paper proposes a new anomaly detection approach based on a Transformer architecture. The main idea is to leverage self-attention to capture the temporal dependency structure of the observations in a sliding window as a measure of anomaly. Since anomalies can be defined as a sequence that is inconsistent with regular ones, the attention matrix is expected to reflect some major aspects of anomalies.   The authors propose a two-branch attention architecture to handle multi-dimensional real-valued time-series data, where prior- and series-attention matrices are computed and utilized in a certain min-max competitive learning framework.   The final anomaly score is defined as the product between the re-construction error and the KL distance between the two attention matrices.  

This paper introduces an Anomaly Transformer for detecting anomalies in time series with association discrepancy. This paper introduces two discrepancies: the series association (e.g., period and trend) and the prior association (e.g., the local smoothness or continuity). For abnormal points, these two associations have a small discrepancy and for normal points, there is a large discrepancy between these two associations. This is because abnormal points have a strong local association while the normal points have a global association. In the model, a two-branch strategy is adopted to model the two associations separately. The model is trained by minimizing the reconstruction loss and maximizing the discrepancy. The experimental results demonstrate the effectiveness of the proposed model.