Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy
Authors: Jiehui Xu
Haixu Wu
Jianmin Wang
Mingsheng Long
Venue: ICLR 2022 Spotlight (2022)
Keywords: Time series anomaly detection
Transformers
Anomaly attention
Association discrepancy

Abstract: Unsupervised detection of anomaly points in time series is a challenging problem, which requires the model to derive a distinguishable criterion. Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufficient to reason about the intricate dynamics. Recently, Transformers have shown great power in unified modeling of pointwise representation and pairwise association, and we find that the self-attention weight distribution of each time point can embody rich association with the whole series. Our key observation is that due to the rarity of anomalies, it is extremely difficult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies' associations shall mainly concentrate on their adjacent time points. This adjacent-concentration bias implies an association-based criterion inherently distinguishable between normal and abnormal points, which we highlight through the Association Discrepancy. Technically, we propose the Anomaly Transformer with a new Anomaly-Attention mechanism to compute the association discrepancy. A minimax strategy is devised to amplify the normal-abnormal distinguishability of the association discrepancy. The Anomaly Transformer achieves state-of-the-art results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space & earth exploration, and water treatment.

Published as a conference paper at ICLR 2022
ANOMALY TRANSFORMER: TIME SERIES ANOMALY DETECTION WITH ASSOCIATION DISCREPANCY
Jiehui Xu," Haixu Wu, Jianmin Wang, Mingsheng Long School of Software, BNRist, Tsinghua University, Beijing 100084, China
{x fx3,20, whx20) Ema s... t& nghua. eduaaaeddu.ccom edu.cn, {j mimang mi Sts
ABSTRACT
Unsupervised detection of anomaly points in time series a challenging problem. which requires the model to derive a distinguishable criterion. Previous methods tackle the problem mainly through learning pointwise representation or pairwise association, however, neither is sufficient to reason about the intricate dynamics. Recently, Transformers have shown great power in unified modeling of pointwise representation and pairwise association, and we find that the self-attention weight distribution of each time point can embody rich association with the whole series, Our key observation is that due to the rarity of anomalies, it is extremely difficult to build nontrivial associations from abnormal points to the whole series, thereby, the anomalies' associations shall mainly concentrate their adjacent time points. This adjacent-concentration bias implies an association-bare distinguishable between normal and abnormal points, which we highlight through the Association Discrepancy. Technically, we propose the Anomaly Transformer with a new Anomaly-Artentn mechanism to compute the association discrepancy. A minimax strategy is devised to amplify the normal- of the association discrepancy. The Anomaly Transformer achieves state-of ofae--ofuul the art results on six unsupervised time series anomaly detection benchmarks of three applications: service monitoring, space earth exploration, and water treatment.
criterion inherently
distinguishability
INTRODUCTION
Real-world systems always work in continuous way, which can generate several successive mea- surements monitored by multi-sensors, such as industrial equipment, space probe, etc. Discovering the malfunctions from large-scale system monitoring data can be reduced to detecting the abnormal time points from time series, which is quite meaningful for ensuring security and avoiding financial loss. But anomalies are usually rare and hidden by vast normal points, making the data labeling hard and expensive. Thus, we focus on time series anomaly detection under the unsupervised setting.
Unsupervised time series anomaly detection is extremely challenging in practice. The model should learn informative representations from complex temporal dynamics through unsupervised tasks. Still, it should also derive a distinguishable criterion that can detect the rare anomalies from plenty of normal time points. Various classic anomaly detection methods have provided many unsupervised paradigms, such as the density-estimation methods proposed in local outlier factor (LOF, (Breunig et al. 2000)), clustering- based methods presented in one-class SVM (OC-SVM, (Schölkopf et al.) 200T and SVDD (Tax & Duin 2004). These classic methods do not consider the temporal infor- mation and are difficult to generalize to unseen real scenarios. Benefiting from the representation learning capability of neural networks, recent deep models (Su et al.] 2019 Shen et al.) 2020)] al. 2021) have achieved superior performance. A major category methods focus learning pointwise representations through well-designed recurrent networks and are self-supervised by the reconstruction or autoregressive task. Here, natural and practical anomaly criterion the pointwise reconstruction or prediction error. However, due to the rarity of anomalies, the pointwise representa- tion is less informative for complex temporal patterns and can be dominated by normal time points, making anomalies less distinguishable. Also, the reconstruction or prediction error is calculated point by point, which cannot provide comprehensive description of the temporal context.
"Equal Contribution

---

Published as a conference paper at ICLR 2022
Another major category of methods detect anomalies based explicit association modeling. The vector autoregression and state space models fall into this category. The graph was also used to cap- ture the association explicitly, through representing time series with different time points vertices and detecting anomalies by random walk (Cheng et al. 2008: 2009). In general, it is hard for these classic methods to learn informative representations and model fine-grained associations. Recently, graph neural network (GNN) has been applied to learn the dynamic graph among multiple variables in multivariate time series (Zhao et al. 2020 Deng & Hooi 2021). While being more expressive, the learned graph is still limited to single time point, which is insufficient for complex temporal patterns. Besides, subsequence based methods detect anomalies by calculating the similarity among subsequences (Boniol & Palpanas 2020). While exploring wider temporal context, these methods cannot capture the fine-grained temporal association between each time point and the whole series.
In this paper, we adapt Transfomers (Vaswani et al., 2017) to time series anomaly detection in the unsupervised regime. Transformers have achieved great progress in various areas, including natural language processing (Brown et al., 2020), machine vision (Liu et al., 202II and time series (Zhou ct al. (02T). This success is attributed to its great power in unified modeling of global representation and long-range relation. Applying Transformers to time series, find that the temporal association of each time point can be obtained from the self-attention map, which presents as a distribution of its association weights all the time points along the temporal dimension. The association distribution of each time point can provide a more informative description for the temporal context, indicat- ing dynamic patterns, such as the period or trend of time series. We name the above association distribution as the series-association, which can be discovered from the raw series by Transformers.
Further, we observe that due to the rarity of anomalies and the dominance of normal patterns, is harder for anomalies to build strong associations with the whole series. The associations of anoma- lies shall concentrate on the adjacent time points that are more likely to contain similar abnormal patterns due to the continuity. Such an conceenttooncnetattion inductive bias is referred to as the prior-association. In contrast, the dominating normal time points can discover informative associ- ations with the whole series, not limiting to the adjacent area. Based on this observation, we try to utilize the inherent abboomll distinguishability of the association distribution. This leads to anomaly criterion for each time point, quantified by the distance between each time point's s and its aarris-association, named as Association Discrepancy. As aforementioned, because the associations of anomalies more likely to be udjacent-cooncetratin anomalies will present a smaller association discrepancy than normal time points.
Going beyond previous methods, introduce Transformers to unsupervised time series anomaly detection and propose the Anomaly Transformer for association learning. To compute the Associ- ation Discrepancy, we renovate the self-attention mechanism to the Anomaly-Aftentinn tains a two-branch structure to model the prior-association and series-association of each time point respectively. The prior-association employs the learnable Gaussian kernel to present the adjacent- concentration inductive bias of each time point, while the attention weights learned from raw series. Besides, a minimax strategy is applied between the two branches, which can amplify the normal-abnornal distinguishability of the Association Discrepancy and further derive a new association-base criterion. Anomaly Transformer achieves strong results on six benchmarks, covering three real applications. The contributions are summarized as follows:
which con-
corresponds to the self-
Based on the key observation of Association Discrepancy, we propose the Anomaly Trans- former with an Anomaly- Attention mechanism, which can model the prior-association and series-association simultaneously to embody the Association Discrepancy.
We propose a minimax strategy to amplify the normal-abnorne distinguishability of the Association Discrepancy and further derive a new association-basse detection criterion.
Anomaly Transformer achieves the state-of-the-art anomaly detection results on six bench- marks for three real applications, justified by extensive ablations and insightful case studies.
2 RELATED WORK
2.1
UNSUPERVISED TIME SERIES ANOMALY DETECTION
As important real-world problem, unsupervised time series anomaly detection has been widely explored. Categorizing by the anomaly determination criterion, the paradigms roughly include the density-estimatiin, clustering-based, reconstructton--based and autoregression-based methods.
2

---

Published as a conference paper at ICLR 2022
In density-estimation methods, the classic methods such local outlier factor (LOF, (Breunig al. ) and connectivity outlier factor (COF, (Tang et al. 2002)) respectively calculate local density and local connectivity for outlier determination. DAGMM ZZng et al., 2018) and MPPCACD Yairi et al. 2017) integrate the Gaussian Mixture Model to estimate the density of representations.
In clustering-baasee methods, the anomaly score is always formalized as the distance to cluster center. SVDD (Tax & Duin 2004) and Deep SVDD (Ruff et al. 2018) gather the representations from normal data to compact cluster. THOC (Shen al. 2020 fuses the multi-scale temporal features from intermediate layers by hierarchical clustering mechanism and detects the anomalies by the multi-layer distances. ITAD (Shin al. 2020) conducts the clustering on decomposed tensors.
The reconstruction-baseee models attempt to detect the anomalies by the reconstruction error. Park al. (2018) presented the LSTM-VAE model that employs the LSTM backbone for temporal mod- eling and the Variational AutoEncoder (VAE) for reconstruction. OmniAnomaly proposed by |Su et al. (2019) further extends the LSTM-VA VAE model with normalizing flow and uses the recon- struction probabilities for detection. InterFusion from Li et al. (2021) renovates the backbone to hierarchical VAE to model the inter- and intra-dependency among multiple series simultaneously. GANs (Goodfellow et al., 2014) are also used for reconstruction-bet -ased anomaly detection (Schlegl
ct al. 2019 Li et al. 2019a Zhou et al. 2019) and perform as an adversarial regularization.
The autoregression based models detect the anomalies by the prediction error. VAR extends ARIMA (Anderson & Kendall 1976) and predicts the future based on the lag-dependent covariance. The
autoregressive model can also be replaced by LSTMs (Hundman et al. 2018 Tariq et al., 2019).
This paper is characterized by new association based criterion. Different from the random walk and subsequence-basee methods (Cheng al. 2008: Boniol & Palpanas| 2020). our criterion is em- bodied by a co-design of the temporal models for learning more informative time-point associations.
2.2 TRANSFORMERS FOR TIME SERIES ANALYSIS
Recently, Transformers (Vaswani et al. 2017) have shown great power in sequential data process- ing, such as natural language processing (Devlin et al. 2019, Brown et al. 2020), audio processing (Huang et al., 2019) and computer vision (Dosovitskiy al. 2021 Liu al. [022]. For time series analysis, benefiting from the advantage of the self-attention mechanism. Transformers are used to
discover the reliable long-range temporal dependencies (Kitaev al., 2020 Li et al. 2019b Zhou et al. 2021 Wu et al. 202T). Especially for time series anomaly detection, GTA proposed by Chen al. (2021) employs the graph structure to learn the relationship among multiple IoT sensors, as well as the Transformer for temporal modeling and the reconstruction criterion for anomaly detec- tion. Unlike the previous usage of Transformers, Anomaly Transformer renovates the self-attention mechanism to the Anomaly-Attention based on the key observation of association discrepancy.
METHOD
Suppose monitoring a successive system of d measurements and recording the equally spaced obser- vations over time. The observed time series X' is denoted by set of time points {2, ... ,d, ). where I3 E Ed represents the observation of time t. The unsupervised time series anomaly detection problem is to determine whether x, is anomalous or not without labels.
As aforementioned, we highlight the key to unsupervised time series anomaly detection as learning informative representations and finding distinguishable criterion. We propose the Anomaly Trans- former to discover more informative associations and tackle this problem by learning the Associa- tion Discrepancy, which is inherently normal-abnorma distinguishable. Technically, we propose the Anomaly-Aftentin to embody the prior-association and series-associationn along with minimax optimization strategy to obtain a more distinguishable association discrepancy. Co-designed with the architecture, we derive an association-bass criterion based on the learned association discrepancy.
3.1 ANOMALY TRANSFORMER
Given the limitation of Transformers (Vaswani et al. 2017) for anomaly detection, we renovate the vanilla architecture to the Anomaly Transformer (Figure L with an Anomaly-Attention mechanism.
Overall Architecture Anomaly Transformer is characterized by stacking the Anomaly-Attentin blocks and feed-forward layers alternately. This stacking structure is conducive to learning underly-
3

---

Published as a conference paper at ICLR 2022
Reconstruction
Layer Norm
G(li 11:01)
G[ 2/)
Rescale
Prior- Association
Grad
Minimize
:
Discrepancy
G(U N).oal
Feed
Forward
Grad
MatMul
Scale
SoftMax
Series- Association
Maximize
Layer Norm
Discrepancy
Linear
MatMul
Anomaly Attention
Figure I: Anomaly Transformer. mnomaly-Attentiin (left) models the prior-association and series- association simultaneously. In addition to the reconstruction loss, our model is also optimized by the minimax strategy with specially-designed stop-gradient mechanism (gray arrows) to constrain the prior- and series- associaissoissionns for more distinguishable association discrepancy.
ing associations from deep multi-level features. Suppose the model contains layers with length-N input time series X E The overall equations of the I-th layer are formalized as:
z' = Layer-Norm Anomaly-Attention/ (1=11) + x²-1)
x = Layer-Norm Feed-Forwarl (2') 2
where X' E IR N Xduass , (1,...,. L} denotes the output the /-th layer with dmnddal channels. The initial input X Embbdding((tt) represents the embedded raw series. 2 TK. X2Nxduoke is the 1-th layer's hidden representation. Anomaly-Attention() is to compute the association discrepancy.
Note that the single-branch self-attention mechanism (Vaswani et al. 2017) cannot model the prior-association and aarres-aasociation simultaneously. We propose the Anomaly- Attention with a two-branch structure (Figure T]. For the prior-association, we adopt a learnable Gaussian kernel to calculate the prior with respect the relative temporal distance. Benefiting from the unimodal property of the Gaussian kernel, this design can pay more attention to the adjacent horizon constitutionally. We also use learnable scale parameter for the Gaussian kernel, making the prior-associas asoocisoosst adapt to the various time series patterns, such different lengths of anomaly segments. The series-association branch is to learn the associations from raw series, which can find the most effective associations adaptively. Note that these two forms maintain the temporal depen- dencies each time point, which are more informative than point-wise representation. They also prior and the learned associations respectively, whose discrepancy
Anomaly-Attentin
reflect the shall be normal-abnormal distinguishable. The Anomaly- Attention the /-th layer is: Anomaly-Attention
Initialization: O, V.O =x'-' x x' ' 2 lw!
( exp
(_ 1 112
Prior-Association: p' Rescale
L OKT Vdmodel
20
(JE(L.-.
(2)
Series-Association: S Softmax
Reconstruction:
21 ='vs'v,
where Q KC, V E N E Nx represent the query, key, value of self-attention and the learned scale respectively. W! Wi W E R Wi E Rdlasaxi represent the parame- ter matrices for Q, K,, V, in the (-th layer respectively. Prior-asxociation Pl E NxN is generated based on the learned scale E gx x1 and the i-th element o, corresponds the i-th time point. Con- cretely, for the i-th time point, its association weight to the j-th point is calculated by the Gaussian Rescale(-) to transform the association weights discrete distributions pl by dividing the sum. S N
kernel G(Ij (1) /ेाग.. exp(- lا=1ا² w.r.t. the distance I -i) Further,
20

---

Published as a conference paper at ICLR 2022
Minimize Phase
Time seriés
Maximize Phase
Time series
Association Discrepancy
Association Discrepancy
wmma
Series-
Series-
Constrained by Reconstruction
Assuciation
Association
Prior-
Constrained by Gaussian Kernel
Prior Association
Association
Distribution Family
Reconstruction loss Optimization
Prion-Asocciaion
O Sssois-Assoiiation
From Gaussian Kernel
Constrained Space
Direction
Figure 2: Minimax association learning. At the minimize phase, the prior-association minimizes the Association Discrepancy within the distribution family derived by Gaussian kernel. At the maximize phase, the series-asson associatiooiiii maximizes the Association Discrepancy under the reconstruction loss.
denotes the sesies-associations.. Softmax(() normalizes the attention map along the last dimension, and each row of S forms a discrete distribution. Z gN xdxsm is the hidden representation after the Anomaly-Attention in the 1-th layer. We use Anomaly-Attention
to summarize Equation
In the multi-head version, the learned scale is E RN for / heads. 2m, Km,, Vm E R Smake denote the query, key and value of the m-th head respectively. The block concatenates the outputs (2), E TR.
Ampli Iimsa from multiple heads and gets the final result RRN Ndmaas
Association Discrepancy We formalize the Association Discrepancy the symmetrized KL di- vergence between prior- and scries-associations which represents the information gain between these distributions (Neal) (007). We average the association discrepancy from multiple layers to combine the associations from multi-level features into a more informative measure as:
L
SP: =
IL(P KL(S).IPPl.)
where KL((II-) is the KL divergence computed between two discrete distributions corresponding to every row of Pl and St. Ass, X)) E Nx1 is the point-wise association discrepancy of X with respect to prior-association P and series-association S from multiple layers. The i-th element of AssDis corresponds to the i-th time point of X. From previous observation, anomalies will present smaller AssDis(P, X) than normal time points, which makes AssDis inherently distinguishable.
3.2 MINIMAX ASSOCIATION LEARNING
As unsupervised task, employ the reconstruction loss for optimizing our model. The recon- struction loss will guide the ausecsaasociation to find the most informative associations. To further amplify the difference between normal and abnormal time points, also use additional loss to enlarge the association discrepancy. Due to the unimodal property of the prior-association, the discrepancy loss will guide the series-association which makes the reconstruction of anomalies harder and makes anomalies more identifiable. The loss function for input series X' R.N xd formalized as:
pay more attention to the non-adjacent area,
Toual X P,S,A;
|| P1β X [Ass S:211
(4)
where & E N denotes the reconstruction of X. |1.1|r/11 11-11e indicate the Frobenius and k-norm. is to trade off the loss terms. When À > O. the optimization is to enlarge the association discrepancy. minimax strategy is proposed make the association discrepancy more distinguishable.
Minimax Strategy Note that directly maximizing the association discrepancy will extremely re- duce the scale parameter of the Gaussian kernel (Neal 2007), making the prior-association mean- ingless. Towards better control of association learning, propose minimax strategy (Figure Concretely, for the minimize phase, we drive the prior-association P' to approximate the series- association St that is leamed from raw series. This process will make the prior- associasooi adapt to various temporal patterns. For the maximize phase, we optimize the associaaoooiitionn to enlarge the association discrepancy. This process forces the aesies-assooiatiion pay attention the
5

---

Published as a conference paper at ICLR 2022
non-adjacent horizon. Thus, integrating the reconstruction loss, the loss functions of two phases are:
Minimize Phase: Lreal (xi, x P, Sdetach, -x;
(5)
Maximize Phase: L (2,, ( Pdetach: S,A;
where À > O and *detach means to stop the gradient backpropagation of the association (Figure D. As P approximates Stench in the minimize phase, the maximize phase will conduct a stronger constraint to the area. Under the reconstruction loss, this much harder for anomalies achieve than normal time points, thereby amplifying the normal-abnoree distinguishability of the association discrepancy.
forcing the time points to pay more attention to the non-adjacent
Association-base Anomaly Criterion We incorporate the normalized association discrepancy to the reconstruction criterion, which will take the benefits of both temporal representation and the distinguishable association discrepancy. The final anomaly score of X' E RNxx is shown as follows:
A AnomalyScore(2)) Softmax Ass (;
|| 12.111 ii
(6)
where @ is the wliment-wise multiplication. AnomalyScore(22) E R denotes the point-wise anomaly criterion of X. Towards better reconstruction, anomalies usually decrease the associa- tion discrepancy, which will still derive a higher anomaly score. Thus, this design can make the reconstruction error and the association discrepancy collaborate improve detection performance.
EXPERIMENTS
We extensively evaluate Anomaly Transformer on six benchmarks for three practical applications.
Datasets Here a description of the six experiment datasets: (1) SMD (Server Machine Dataset, Su et al. ) is 5-week-long dataset collected from a large Internet company with 38 dimen- sions. (2) PSM (Pooled Server Metrics, Abdulaal al. (2021) is collected internally from multi- ple application server nodes at eBay with 26 dimensions.
(3) Both MSL (Mars Science Laboratory rover) and SMAP (Soil Moisture Active Passive satellite) are public datasets from NASA (Hundman) et al. 2018) with 55 and 25 dimensions respectively, which contain the telemetry anomaly data de- rived from the Incident Surprise Anomaly (ISA) reports of spacecraft monitoring systems. (4) SWaT (Secure Water Treatment, Mathur & Tippenhauer (2016)) is obtained from 51 sensors of the criti- cal infrastructure system under continuous operations. (5) NeurIPS-TS (NeurIPS 2021 Time Series Benchmark) is dataset proposed by Lai et al. (2021) and includes five time series anomaly scenar- ios categorized by behavior-driven taxonomy as point-global, pattern-contextual, pattern-shapele pattern-seasonal and pattern-trend. The statistical details are summarized in Table 13 Appendix.
Implementation details Following the well-established protocol in Shen et al. 2020), J we adopt non-overlapped sliding window to obtain a set of sub-series. The sliding window is with fixed size of 100 for all datasets. We label the time points as anomalies their anomaly scores (Equation (6) are larger than certain threshold 8. The threshold 5 is determined to make proportion of time points of the validation dataset labeled as anomalies. For the main results, we set r = 0.1% for SWaT, 0.5% for SMD and 1% for other datasets. We adopt the widely-used adjustment strategy (Xu et al. 2018 Su et al. 2019;; Shen et al., 2020): if a time point in a certain successive abnormal segment is detected, all anomalies in this abnormal segment are viewed to be correctly detected. This strategy is justified from the observation that an abnormal time point will cause an alert and further make the whole segment noticed in real-world applications. Anomaly Transformer contains 3 layers. We set the channel number of hidden states dmodel as 512 and the number of heads h as 8. The hyperparameter X (Equation 4) is set as 3 for all datasets to trade-off two parts of the loss function. We use the ADAM (Kingma & Ba 2015) optimizer with initial learning rate of 10-4. The training process is early stopped within 10 epochs with the batch size of 32. All the experiments are implemented in Pytorch (Paszke et al. 2019) with a single NVIDIA TITAN RTX 24GB GPU.
Baselines We extensively compare our model with 18 baselines, including the reconstruction- based models: InterFusion (202T). BeatGAN (2019), OmniAnomaly (011), LSTM LSTM-VAE (2018); the density-estimation models: DAGMM (2018), MPPCACD (2017). LOF 2000); the clustering- mod- els: CL-MPPCA (2019). LSTM (2018). VAR (1976); the classic methods: OC-SVM (2004), Isola- tionForest (2008). Another baselines from change point detection and time series segmentation are
based methods: ITAD (2020). THOC (2000). Deep-SVDD (2018): the autoregression-basee
deferred to Appendix InterFusion (2021) and THOC (2020) are the state-of-the the--hterrtttr deep models.
6

---

Published as a conference paper at ICLR 2022
0.
0.7
2-4
Bast 2i-.-attin at-ear
02
Dup--tDC 0.648 02
CR00-9000-AA 2.8754
Sap-500DiAu 8.84
DeppxVD GENE
03000
000 1011)
SIM 9778)
0.0
Figure 3: ROC curves (horizontal-axis: false-positive rate; vertical-axis: true-positive rate) for the five datasets. A higher AUC value (area under the ROC curve) indicates better performance. The predefined threshold proportion r is in (0.5%, 1.0%, 1.5%, 2.0%, 10%, 20%, 30%).
Table 1: Quantitative results for Anomaly Transformer (Ours) in the five datasets. The P. R and Fl represent the precision, recall and Fi-score (as %) respectively. F1-score is the harmonic mean of precision and recall. For these three metrics, a higher value indicates a better performance.
Dataset
SMD
MSI.
SMAP
SWaT
PSM
Metric
P
R
FI
R
FI
FI
P
P
R
FI
OCSVM
44.34
76.72
56.19
59.78
86.87
70.82
53.85
59.07
56.34
45.39
49.22
47 23
62.75
80.89
70.67
ForeationForest
42.31
73.29
53.64
53.94
86.54
66.45
52.39
59.07
55.53
49.29
44.95
47.02
76.09
92.45
83.48
LOF
56.34
39.86
46.68
47.72
85.25
61.18
58.93
56.33
57.60
72.15
65.43
68.62
57.89
90.49 70.61
Deep-SVDD
78.54
79.67
79.10
91.92
76.63
83.58
89.93
56.02
69.04
80.42
84,45
82.39
95.41
86.49 90.73
DAGMM
67.30
49.89 57.30
89.60
63.93
74.62
86.45
56.73
68.51
89.92
57.84
70.40
93.49
70.03 80.08
MMPCACD
71.20
79.28 75.02
81.42
61.31 69.95
88.61
75.84
81.73
82.52
68.29 74.73
76.26
78.35 77.29
VAR
78.35
70.26 74.08
74.68
81.42
77.90
81.38
53.88
64.83
81.59
60.29 69.34
90.71 83.82 87.13
LSTM
78.55 85.28 81.78
85
82.50
83.95
89.41
78.13 83.39
86.15
83.27 84.69
76.93 89.64 82.80
CL-MPPCA
82.36 76.07 79.09
73.71
88.54
80.44
86.13
63.16 72.88
76.78
81.50 79.07
56.02 99.93 71.80
ITAD
86.22 73.71 79.48
69.44
84.09 76.07
82.42
66.89 73.85
63.13
52.08 57.08
72.80 64.02 68.13
LAT
75.76 90.08 82.30
85.49
79.94 82.62
92.20
67.75 18.10
76.00
89.50 82.20
73.62
89.92 80.96
BeatGAN
72.90 84.09 78.10
89.75
85.42 87.53
92.38
55.85 69.61
64.01
87.46 73.92
90.30
93.84 92.04
OmniAnomaly
83.68 86.82 85.22
89.02
86.37 87.67
92.49
81.99 86.92
81.42
84,30 82.83
88.39
74.46 80.83
InterFuusion
87.02 85,43 86.22
81.28
92.70 86.62
89.77
88.52 89.14
80.59
85,58 83,01
83,61 83,45 83,52
THOC
79.76 90.95 84.99
88.45
90.97
89.69
92.06
89.34
90.68
83.94
86.36
85.13
88.14
90.99 89.54
Ours
89.40 95.45 92.33
92.09
95.15
93.59
94.
99.40
96.69
91.55
96.73
94.07
96.91
98.90 97.89
4.1 MAIN RESULTS
Real-world datasets We extensively evaluate model on five real-world datasets with ten com- petitive baselines. As shown in Table Anomaly Transformer achieves the consistent state-of-the art on all benchmarks. We observe that deep models that consider the temporal information outper- form the general anomaly detection model, such as Deep-SVDD (Ruff et al., 2018) and DAGMM (Zong al.] 2018). which verifies the effectiveness of temporal modeling. Our proposed Anomaly Transformer goes beyond the point-wise representation learned by RNNs and models the more infor- mative associations. The results in Table T are persuasive for the advantage of association learning in time series anomaly detection. In addition, we plot the ROC curve in Figure 3 for complete comparison. Anomaly Transformer has the highest AUC values on all five datasets. means that model performs well in the false-positive and true-positive rates under various preset thresholds, which is important for real-world applications.
IeSTISS-TS benchmark This benchmark is generated from well-designed rules proposed by Lai et al. (202I), including all types of anomalies and covering both the point-wise and pattern- wise anomalies. As shown in Figure Anomaly Transformer can still achieve state-of- the-art performance. This verifies the effec- tiveness of our model on various anomalies.
71.11
67545
F1-Score
60.35 53
&
nex intusss
Ablation study As shown in Table 2 we further investigate the effect of each part in model. Our association-basel criterion outperforms the widely-used reconstruction criterion consistently. Specifically, the association-bel based criterion brings remarkable 18.76%
Figure 4: Results for NSS..PS-TS.
averaged absolute Fi-score promotion. Also, directly taking the association discrepancy as the criterion still achieves a good performance (F1-score: 91.55% and surpasses the previous state-of-the-an model
7

---

Published as a conference paper at ICLR 2022
THOC (Fl-score: 88.01 calculated from Table Besides, the learnable prior- asoocissoiitiion (corre- sponding to a in Equation 2) and the minimax strategy can further improve our model and get 8.43% (79.05-487.48) +87.48) and 7.48% (87.48 -99496)94.96) averaged absolute promotions respectively. Finally, our proposed Anomaly Transformer surpasses the pure Transformer by 18.34% lute improvement. These verify that each module of our design is effective and necessary. More ablations of association discrepancy can be found in Appendix D
abso-
Ablation results (FI-score) in anomaly criterion, prior-association and optimization strat- egy. Recon, AssDis and Assoc mean the pure reconstruction performance, pure association dis- crepancy and our proposed association- based criterion respectively. Fix is to fix Learnable scale parameter a of prior-association as 1.0. Max and Minimax refer to the strategies for association discrepancy in the maximization (Equation and minimax (Equation way respectively.
Table 2:
Anomaly Criterion
Prior- Association
Optimization Strategy
Avg (as %)
Architecture
SMD
MSL
SMAP
SWaT
PSM
Transformer
Recon
x
79.72
76.64
73.74
74.56
78.43
76.62
Recon
Learnable
Minmax
71.35
78.61
69.12
81.53
80.40
76.20
Anomaly
AssDis
Learnable
Minmax
87.57
90.50
90.98
93.21
95.47
91.55
Transformer
Assoc
Fix
Max 83.95
82,17
70.65
79.46
79.04
79.05
Assoc
Learnable
Max 88.88
85.20
87.84
81.65
93.83
87.48
*final
Assoc
Learnable
Minmax
92.33
93.59
96.90
94.07
97.89
94.96
4.2
MODEL ANALYSIS
To explain how model works intuitively, we provide the visualization and statistical results for our three key designs: anomaly criterion, learnable prior-association and optimization strategy.
Time Series Anomaly
Point
Pattern
Global
Contextual
Shapelet
Seasonal
Trend
Input Time Series
Dataction
Detection
Reconstruction Criterion
6.00
Error
Detectton Enor
Association- based
Criterion
000
010
2.04
C-1103
010
Figure 5: Visualization of different anomaly categories (Li et al. 2021). We plot the raw series (first row) from NeurIPS-TS dataset, well as their corresponding reconstruction (second row) and association- based criteria (third row). The point-wise anomalies are marked by red circles and the pattern-wise anomalies are in red segments. The wrongly detected cases are bounded by red boxes.
Anomaly criterion visualization To get more intuitive cases about how association-based crite- rion works, we provide some visualization Figure 5 and explore the criterion performance under different types of anomalies, where the taxonomy is from Lai et al. (202I). We can find that our proposed association criterion is more distinguishable in general. Concretely, the association- based criterion can obtain the consistent smaller values for the normal part, which is quite contrasting
8

---

Published as a conference paper at ICLR 2022
in point-contextual and pattern-seasoaal cases (Figure [ In contrast, the jitter curves of the recon- struction criterion make the detection process confused and fail in the aforementioned two cases. This verifies that our criterion can highlight the anomalies and provide distinct values for normal and abnormal points, making the detection precise and reducing the false-positive rate.
Input Time Series
Tee
Time
NW
1.
(a) Point-Global
(c) Shater-Shapelet
Pattern
(e)
Figure 6: Learned scale parameter o for different types of anomalies (highlight in red).
Prior-association visualization During the minimax optimization, the prior-associatee is learned to get close to the series-association. Thus, the learned can reflect the adjacent concentttcnn degree of time series. As shown Figure we find that a changes to adapt to various data patterns of time series. Especially, the prior- ansocaasooittiinn of anomalies generally has smaller than normal time points, which matches our adjacent-concentee concentration inductive bias of anomalies.
Optimization strategy analysis Only with the reconstruction loss, the abnormal and normal time points present similar behavior in the association weights to adjacent time points, corresponding to a contrast value closed associations to pay more attention the non-adjacent area. However, obtain a better reconstruc- tion, the anomalies must maintain much larger adjacent association weights than normal time points, corresponding to larger contrast value. But direct maximization will cause optimization difficulty of Gaussian kernel, and cannot strongly amplify the difference between normal and abnormal time points expected (SMD:1.15 1.277. The minimax strategy optimizes the prior-association to provide a stronger constraint to series-association, thereby obtaining more distinguishable contrast values and better performance than the direct maximization (SMD:1.27-2.333
(Table()). Maximizing the association discrepancy will force the series-
Table 3: Results of adjacent association weights for Abnormal and Normal time points respectively. Recon, Max and Minimax represent the association learning process that is supervised by reconstruc- tion loss, direct maximization and minimax strategy respectively. A higher contrast value ( (bmormal) indicates stronger distinguishability between normal and abnormal time points.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Optimization
Recon Max Ours Recon Max Ours Recon Max Ours Recon Max Ours Recon Max Ours
Abnormal (%)
1.08 0.95
0.86
1.01
0.65
0.35
1.29
1.18
0.70
1.27
0.89
0.37
1.02
0.56
0.29 0.54 0.11
Normal (%)
0.94
0.75
0.36
1.00
0.59
0.22
1.23
1.09
0.49
1.18
0.78
0.21
0.99
Contrast Abnormal Normal
| 1.15
1.27 2.39
1.01
1.10 1.59
1.05
1.08
1.43
1.08
1.14
1.76
1.03
1.04 2.64
CONCLUSION AND FUTURE WORK
This paper studies the unsupervised time series anomaly detection problem. Unlike previous works, we learn the more informative time-point associations by Transformers. Based on the key obser- vation of association discrepancy, we propose the Anomaly Transformer, including an Anomaly- Attention with the two-branch structure to embody the association discrepancy. A minimax strategy is adopted to further amplify the difference between normal and abnormal time points. By introduc- ing the association discrepancy, we propose the association-baxt criterion, which makes the recon- struction performance and association discrepancy collaborate. Anomaly Transformer achieves the state-of-the the-the-rr results on an exhaustive set of empirical studies. Future work includes theoretical study of Anomoly Transformer in light of classic analysis for autoregression and state space models.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported by the National Megaproject for New Generation AI (2020AAA0109201) National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (220110000882044), and BNRist Innovation Fund IBNR2021RCO1000
REFERENCES
Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. KDD, 2021.
Ryan Prescott Adams and David J. C. MacKay. Bayesian online changepoint detection. arXiv preprint arXiv:0710. 2007.
O. Anderson and M. Kendall. Time-series, 2nd edn. J. R. Stat. Soc. (Series D), 1976.
Paul Boniol and Themis Palpanas. Series2graph: Graph-based subsequence anomaly detection for time series. Proc. VLDB Endow. 2020.
Markus M. Breunig. Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander. LOF: identifying density-based local outliers. In SIGMOD, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Zekai Chen, Dingshuo Chen, Zixuan Yuan, Xiuzhen Cheng, and Xiao Zhang. Learning graph structures with transformer for multivariate time series anomaly detection in iot. ArXiv, abs/2104.03466. 2021.
Haibin Cheng, Pang-Ning Tan, Christopher Potter, and Steven A. Klooster: robust graph-based algorithm for detection and characterization of anomalies in noisy multivariate time series. ICDM Workshops, 2008.
Haibin Cheng, Pang-Ning Tan, Christopher Potter, and Steven A. Klooster. Detection and charac- terization of anomalies in multivariate time series. SDM, 2009.
Shohreh Deldari, Daniel V. Smith, Hao Xue, and Flora D. Salim. Time series change point detection with self-supervised contrastive predictive coding. In WwW. 2021.
Ailin Deng and Bryan Hooi. Graph neural network-basee anomaly detection in multivariate time series. AAAI, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
I. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In ICLR, 2019.
Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Söderström. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. KDD, 2018.
10

---

Published as a conference paper at ICLR 2022
Eamonn J. Keogh, Taposh Roy, Naik U, and Agrawal A. Multi-dataset time-series anomaly detection competition, Competition of International Conference on Knowledge Discovery & Data Mining 2021. URLtttps:: compete.eeeeeee/eeeo hwwagon- wttp://wwmmmomo////eee/emeeee httpae/tiiiieeemmmtttoto/oo////s
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
Kwei-Herng Lai, D. Zha, Junjie Xu, and Yue Zhao. Revisiting time series outlier detection: Defini- tions and benchmarks. In NeurIPS Dataset and Benchmark Track, 2021.
Dan Li, Dacheng Chen, Lei Shi, Baihong Jin, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multi- variate anomaly detection for time series data with generative adversarial networks. In ICANN, 2019a.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurlPS, 2019b.
Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time ries anomaly detection and interpretation using hierarchical inter- metric and temporal embedding. KDD, 2021.
F. Liu, K. Ting, and Z. Zhou. Isolation forest. ICDM, 2008
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Ching-Feng Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021.
AdityaP. P. Mathur and Nils Ole Tippenhauer. Swat: water treatment testbed for research and training on ICS security. In CySWATER, 2016.
Radford M. Neal. Pattern recognition and machine learning. Technometrics, 2007.
Dachyung Park, Yuuna Hoshi, and Charles C. Kemp. A multimodal anomaly detector for robot-
assisted feeding using an lstm-based variational autoencoder. RA-L, 2018.
Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L.. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jø rgen Jennum, and Christian Igel. U-time: A fully convolutional network for time series segmentation applied to sleep staging. In NeurIPS. 2019.
Lukas Ruff, Nico Görnitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Robert A. Vandermeulen, Alexander Binder, Emmanuel Müller, and M. Kloft. Deep one-class classification. In ICML, 2018.
T. Schlegl, Philipp Seeböck, S. Waldstein, G. Langs, and U. Schmidt-Erfurth f-anogan: Fast unsu- pervised anomaly detection with generative adversarial networks. Med. Image Anal., 2019.
B. Schölkopf, John C. Platt, J. Shawe- Shawe-Taylor, Alex Smola, and R. C. Williamson. Estimating the support of high-dimensional distribution. Neural Comput., 2001.
Lifeng Shen, Zhuocong Li, and James T. Kwok. Timeseries anomaly detection using temporal hicrarchical one-class network. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), NeurIPS, 2020.
Youjin Shin. Sangyup Lee, Shahroz Tariq. Myeong Shin Lee, Okchul Jung, Daewon Chung, and Si- mon S. Woo. Itad: Integrative tensor- rensor-based anomaly detection system for reducing false positives of satellite systems. CIKM, 2020.
11

---

Published as a conference paper at ICLR 2022
Ya Su. Y. Zhao, Chenhao Niu, Rong Liu, W. Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. KDD, 2019.
Jian Tang, Zhixiang Chen, A. Fu, and D. Cheung. Enhancing effectiveness of outlier detections for low density patterns. In PAKDD, 2002.
Shahroz Tariq, Sangyup Lee, Youjin Shin, Myeong Shin Lee, Okchul Jung, Daewon Chung, and Simon S. Woo. Detecting anomalies in space using multivariate convolutional Istm with mixtures of probabilistic pea. KDD, 2019.
D. Tax and R. Duin. Support vector data description. Mach. Learn., 2004.
Robert Tibshirani, Guenther Walther, and Trevor Hastie. Estimating the number of clusters a dataset via the gap statistic. J. R. Stat. Soc. (Series B), 2001.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with Auto-Correlation for long-term series forecasting, In NeurIPS, 2021.
Haowen Xu, Wenxiao Chen, Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Y. Liu, Y. Zhao, Dan Pei, Yang Feng, Jian Jhen Chen, Zhaogang Wang, and Honglin Qiao. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. WWW, 2018.
Takehisa Yairi, Naoya Takeishi, Tetsuo Oda, Yuta Nakajima, Naoki Nishimura, and Noboru Takata. A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction. IEEE Trans. Aerosp. Electron. Syst. 2017.
Hang Zhao, Yujing Wang, Juanyong Duan, Congrui Huang, Defu Cao, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Multivariate time-series anomaly detection via graph attention network. ICDM, 2020.
Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI, 2019.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2021.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dae-ki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2022
A PARAMETER SENSITIVITY
We set the window size as 100 throughout the main text, which considers the temporal informa- tion, memory and computation efficiency. And set the loss weight À based on the convergence property the training curve.
Furthermore, Figure 7 provides the model performance under different choices of the window size and the loss weight. We present that our model is stable to the window size over extensive datasets (Figure left). Note that a larger window size indicates larger memory cost and smaller sliding number. Especially, only considering the performance, its relationship to the window size can be determined by the data pattern. For example, our model performs better when the window size is 50 for the SMD dataset. Besides, we adopt the loss weight À in Equation trade off the reconstruction loss and the association part. We find that stable and easy tune the range of The above results verify the sensitivity of our model, which is essential for applications.
100
95
95
F1-score
F1-score
85
SMD
MSL
MSL
75
SWaT
SWaT
PSM
85
65
2
Window Size
Loss Weight
Figure 7: Parameter sensitivity for sliding window size (left) and loss weight (right). The model
with X = O still adopts the association-based criterion but only supervised by reconstruction loss.
B IMPLEMENTATION DETAILS
We present the pscudo-code of Anomaly-Attention in Algorithm
Algorithm Anomaly- Mechanism (multi-head version).
Input: X E R Nxdudet; input; D = (i ((-1)² (...
R NxN, relative distance matrix
Layer params: input: linear projector for input; MLP o linear projector for output
1: Q,K.V Split MLP input (2t), dim=1
Q,K, V. R Nx X
E Nxh
2: for (Qm. Km, Vm, mm) in KC, V,
> 2m, Kmny Vm E Nx Smnte
0m E N
Fm Broadcast (ams dim=1)
on NNN
4: Pm V2kam exp
D
D Pm NNNN
2
5: Pm Pm Pmm/Broadcast (Sum( dim=1)
Rescaled Pm E RNNN
Sm =Softmax
( drinke -m KT
Sm NNN
2m SmnV
2m R Lmpks
7:
8: 2 MLP output (Conctt(2,).... En), dim=1)
> 2 E B. Xduiial
9: Return
Keep the Pm, and Sm, = 1,,...
C MORE SHOWCASES
To obtain an intuitive comparison main results (Table [T.. we visualize the criterion various baselines. Anomaly Transformer present the most distinguishable criterion (Figure 8. Besides,
13

---

Published as a conference paper at ICLR 2022
for the real-world dataset, Anomaly Transformer can also detect the anomalies correctly. Especially for the SWaT dataset (Figure d)), our model can detect the anomalies in the early stage, which is meaningful for real-world applications, such as the early warning of malfunctions.
Point-Global
Point-Contextual
Nattern-Shapelee
Pattern-Scasonal
Pattern Trend
Inputt Time Series
BeatGAN
Deep-SYDD
E3H WWW www.0/0000
LSTM-VAE
Transforier
Anomaly
0.0
2.00
3.00
Figure 8: Visualization of learned criterion for the NeurlPS-TS dataset. Anomalies are labeled by
red circles and red segments (first row). The failure cases of the baselines are bounded by red boxes.
Input Tine Series
150
48
47
Assciation based Criterion
Early Stage Detection
15
(a) SMD
(u) MSL
(c) SMAP
(d) SWaT
(e) PSM
Figure 9: Visualization of the model learned criterion in real-world datasets. We select one dimen- sion of the data for visualization. These showcases are from the test set of corresponding datasets.
D ABLATION OF ASSOCIATION DISCREPANCY
We present the pseudo code of the calculation in Algorithm
14

---

Published as a conference paper at ICLR 2022
D.1 ABLATION OF MULTI-LEVEL QUANTIFICATION
We average the association discrepancy from multiple layers for the final results (Equation 6 further investigate the model performance under the single-layer usage. As shown in Table the
We
multiple-layer design achieves the best, which verifies the effectiveness of multi-level quantification.
Table Model performance under difference selection model layers for association discrepancy.
Dataset
SMD
MSL.
SMAP
SWaT
PSM
Metric
P
R
F1
P
R
F1
P
R
F1
P
R
F1
R
FI
layer 1 layer 2 layer
87.15 92.87
89.92
90.36
94.11
92.19
93.65
99.03
96.26
92.61
91.92
92.27
97.20
97.50
97.35
87.22 95.17
91.02
90.82
92.1
91.60
93.69
98.75
96.15
92.48
92.50
92.49
96.12
98.62
97.35
87.27
93.89
90.46
91.61
88.81
90.19
93.40
96.04
88.75
91.22
89.96
77.25
94.53
85.02
Multiple-layer| 89.40 95.45 92.33
92.09
95.15
93.59
94.13
99.40
96.69
91.55
96.73
94.07
96.91
98.90
97.89
D.2 ABLATION OF STATISTICAL DISTANCE
We select the following widely-used statistical distances to calculate the association discrepancy:
Symmetrized Kullback-Leibler Divergence (Ours).
Bensen-Shanme Divergence (JSD)).
Wasserstein Distance (Wasserstein).
Cross-Entropy (CE).
L2 Distance (L2).
Table 5: Model performance under different definitions of association discrepancy.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Metric
F1
R
F1
R
P
85.26
74.80
79.69
85.58
81.30
83.39
91.25
56.77
70.00
79.90
87.45
83.51
70.24
96.34
81.24
CE
88.23
81.85
84.92
90.07
86.44
88.22
92.37
64.08
75.67
62.78
81.50
70.93
70.71
94.68
80.96
Wasserstein
78.80
71.86
75.17
60.77
36.47
45.58
90.46
57.62
70.40
92.00
71.63
80.55
68.25
92.18
78.43
ISD
85.33
90.09
87.64
91.19
92.42
91.80
94.83
95.14
94.98
83.75
96.75
89.78
95.33
98.58
96.93
Ours
89.40 95.45 92.33
92.09
95.15 93.59
94.13
99.40 96.69
91.55
96.73
94.07
96.91
98.90
97.89
As shown in Table our proposed definition of association discrepancy still achieves the best per- formance. We find that both the CE and JSD can provide fairly good results, which are close to our definition in principle and can be used to represent the information gain. The L, distance is not suitable for the discrepancy, which overlooks the property of discrete distribution. The Wasserstein distance also fails in some datasets. The reason is that the prior-association and series-association are exactly matched in the position indexes. Still, the Wasserstein distance is not calculated point by point and considers the distribution offset, which may bring noises to the optimization and detection.
Algorithm 2 Association Discrepancy AssDis(P, X)) Calculation (multi-head version).
Input: time series length N; layers number L; heads number h; prior-association Pan xxxxxXxN, series-association San E BRLxxxNNN
1: P' Mean(P, dimm=1)
D P' E RLNNN S' R R' ELEN
2: S' Mean(S, dim=1)
3: R' KL (P)' dim=-1) (S)),'),
4: R = Mean dim=0)
B R
5: Return R
Represent the association discrepancy of each time point
15

---

Published as a conference paper at ICLR 2022
D.3 ABLATION OF PRIOR-AST CASOR-ASSCIIATION
In addition to the Gaussian kernel with learnable scale parameter, we also try to use the power-law kernel P(z; ai) = x with learnable power parameter a for prior-association, which also a unimodal distribution. As shown in Table [ power-law kernel can achieve good performance in most of the datasets. However, because the scale parameter is easier to optimize than the parameter of power, Gaussian kernel still surpasses the power-law kernel consistently.
Table 6: Model performance under different definitions of prior-association. Our Anomaly Trans- former adopts the Gaussian kernel as the prior, Power-law refers the power-law kernel.
Dataset
SMD
MSL R
SMAP
SWaT
PSM
Metric
F1
R
Power-law
89.41
92.46
90.91
90.95
85.87
88.34
91.95
58.24
71.31
92.52
93.29
92.90
96.46
97.30 98.90 97.89
98.15
Ours
89.40
95.45 92.33
92.09
95.15
93.59
94.13
99.40
96.69
91.55
96.73
94.07
96.91
E ABLATION OF ASSOCIATION-BASER CRITERION
E.I
CALCULATION
We present the pseudo-code of association-baser criterion in Algorithm
Algorithm Association-based Criterion AnomalyScore(( Calculation
Input: time series length N; input time series A' E NR,d, reconstruction time series & E R
association discrepancy S;; Nx1;
CAD 1ofttax((-AssDs((2 (,),, dim=0)
CAD NxI
2: CRecon = Mean (X dim=i)
b CReon E Nx1
3: C= =CAD x CRecon
D D C Nx1
4: Return C
b Anomaly score for each time point
E.2 ABLATION OF CRITERION DEFINITION
We explore the model performance under different definitions of anomaly criterion, including the pure association discrepancy, pure reconstruction performance and different combination methods for association discrepancy and reconstruction performance: addition and multiplication.
Association Discrepancy: AnomalyScore(2) = Softmax
AssDDssPP,s:1)
Reconstruction: AnomalyScore(21) = ||26 - 8...11
i=
Addition: AnomalyScore(2)) = Softmax
Ass +
|| - -Pi. i=
Multiplication (Ours): AnomalyScore(x) Softmax
AssDis
| 22.
1
(7)
From Table we find that directly using our proposed association discrepancy can also achieve a good performance, which surpasses the competitive baseline THOC (Shen et al. 2020) consistently. Besides. the multiplication combination that we used Equation 6 performs the best, which can bring better collaboration the reconstruction performance and association discrepancy.
16

---

Published as a conference paper at ICLR 2022
Table 7: Ablation criterion definition. We also include the state-of-the-ar deep model THOC (hen et al., 2020) for comparison. AssDis and Recon represent the pure association discrepancy and the pure reconstruction performance respectively. Ours refers to our proposed association- criterion with the multiplication combination.
Dataset
SMD
MSL.
SMAP
SWaT
PSM
Avg
Metric
P
R
F1
P
R
FI
R
FI
P
R
F1
P
R
F1
FI(%)
THOC
79.76 90.95
84.99
88.45
90.97
89.69
92.06
89.34 90.68
83.94
86.36
85.13
88.14
90.99
89.54
88.01
Recon
78.63
65.29
71.35
79.15
78.07
78.61
89.38
56.35
69.12
76.81
86.89
81.53
69.84
94.73
80.40
76.20
AssDis
86.74
88.42
87.57
91.20
89.81 90.50
91.56
90.41
90.98
97.27
89.48
93.21
97.80
93.25
95.47
91.55
Addition
77.16 70.58
73.73
88.08
87.37
87.72
91.28
55.97 69.39
84.34
81.98
83.14
97.60
97.61
97.61
82.32
Ours
89.40 95.45 92.33
92.09
95.15
93.59
94.13
99.40
96.69
91.55
96.73
94.07
96.91
98.90
97.89
94.96
F CONVERGENCE OF MINIMAX OPTIMIZATION
The total loss of our model (Equation 4) contains two parts: the reconstruction loss and the associa- tion discrepancy. Towards better control of association learning, we adopt minimax strategy for optimization (Equation 5). During the minimization phase, the optimization trends to minimize the association discrepancy and the reconstruction error. During the maximization phase, the optimiza- tion trends to maximize the association discrepancy and minimize the reconstruction error.
We plot the change curve of the above two parts during the training procedure. As shown in Figures 10 II both parts of the total loss can converge within limited iterations on all the five real-world datasets. This nice convergence property is essential for the optimization of our model.
200
401
601
200
MSL
Figure 10: Change curve of reconstruction loss X - 21 in real-world datasets during training.
Tanabon
Bessdion
SMD
MSL
SMAP
SWAT
(e) PSM
Figure 11: Change curve of association discrepancy (Ass in real-world datasets during the training process.
G MODEL PARAMETER SENSITIVITY
In this paper, set the hyper-parametee parameters L and dpodd following the convention of Transformers Vaswaani et al. 2017 Zhou et al. 2021).
Furthermore, evaluate model parameter sensitivity, investigate the performance and efficiency under different choices for the number of layers and hidden channels d Generally, increasing the model size obtain better results but with larger memory and computation costs.
17

---

Published as a conference paper at ICLR 2022
Table 8: Model performance under different choices of the number of layers L.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Metric
P
R
F1
P
R
F1
R
F1
L
89.2 93.73
91.43
91.99
97.59 94.71
93.58
99.35 96.38
91.57
95.33
93.42
96.74
98.09
97.41
L = 2 89. 94.33
91.72
91.89
94.73 93.29
93.79
98.91 96.28
92.37
94.59
93,47
97.22
98.23 97.72
L 89.40 95.45
92.33
92.09
95.15
94.13
99.40 96.69
91.55
94.07
96.91 98.90 97.89
L = 4 89.59 95.76 92.58
91.88
95.40
93.61
93.75
99.13 96.37
93.37
93.45 93.41 97.30 97.58 97.44
Table 9: Model performance under different choices of the number of hidden channels dmoodl- Mem means the averaged GPU memory cost. Time is the averaged running time of 100 iterations during the training process.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Mem
Time
Metric
P
R
FI
P
R
FI
P
R
FI
P
R
FI
P
R
F1
(GB)
(s)
dmoda 256 88.83
91.82 90.30
91.96
97.60 94.70 93.74 99.47 96.52 93,91 93.99 93.95
97.38 98. 97.77
4.9
0.12
dmoiee 512 89.40
95.45 92.33
92.09
95.15
94.13
99.40 96.69 91.55 96.73 94.07
96.91 98.90 97.89
5.5
0.15
dmedll = 1024
89..44 96.33
92.76
92.76
91.80 94.99
93.37
93.58 99,47 96.43 92.02
95.01
93.49
95.78
98.12 96.94
6.6 0.27
H PROTOCOL OF THRESHOLD SELECTION
Our paper focuses unsupervised time series anomaly detection. Experimentally, each dataset includes training. validation and testing subsets. Anomalies are only labeled in the testing subset. Thus, select the hyper- papeer-rrameeerss following the Gap Statistic method (Tibshirani et al., 2001 in K-Means. Here the selection procedure:
After the training phase, we apply the model to the validation subset (without label) and obtain the anomaly scores (Equation of all time points.
We count the frequency of the anomaly scores in the validation subset. is observed that the distribution of anomaly scores is separated into two clusters. We find that the cluster with a larger anomaly score contains time points. And for our model, r is closed to 0. %, 0.5%, 1% for SWaT, SMD and other datasets respectively (Table
Due the size of the test subset being still inaccessible in real-world applications, we have to fix the threshold as fixed value 8, which can gaurantee that the anomaly scores r time points in the validation set are larger than and thus detected anomalies.
Table 10: Statistical results of anomaly score distribution on the validation set. We count the number of time points with corresponding values in several intervals.
(a) SMD, MSL. and SWaT datasets.
(b) SMAP and PSM datasets.
Anomaly Score Interval
SMD
MSL
SWaT
Anomaly Score Interval
SMAP
PSM
(0, +co]
141681
11664
99000
(0, +00]
27037
26497
[0, 10
140925
11537
98849
[0, 10-11
26732
26223
210-2.0.1 0.1]
8
17
(10 10
0
5
(00
754
119
134
(10-2 +o0]
305
269
Ratio of (0.1, +00
0.53%
1.02%
0.14%
Ratio of (10-2, +ool
1.12%
1.01
Note that, directly setting the 8 is also feasible. According to the intervals in Table 10 fix the 8 as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets, which yield quite close performance to setting
18

---

Published as a conference paper at ICLR 2022
Table 11: Model performance, Choose by 5 means that we fix as 0.1 for the SMD, MSL and SWaT datasets, 0.01 for the SMAP and PSM datasets. Choose means that we select 0.1% for SWaT, 0.5% for SMD and 1% for the other datasets.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Metric
P
R
FI
P
R
FI
P
R
FI
P
R
FI
P
R
FI
Choose by 88.65 97.17 92.71 91.86 95.15 93.47 97.69 98.24 97.96 86.02
95.01 90.29
97.69 98.24 97.96
Choose by
89.40 95.45 92.33 92.09 95.15 93.59 94.13 99.40 96.69 91.55 96.73 94.07
96.91 98.90 97.89
In real-world applications, the number of selected anomalies always decided up human re- sources. Under this consideration, setting the number of detected anomalies by the ratio is more practical and easier to decide according to the available resources.
MORE BASELINES
In addition the time series anomaly detection methods, the methods for change point detection and time series segmentation can also perform as valuable baselines. Thus, we also include the BOCPD
(Adams & MacKay 2007) and TS- TS-CP2 (Deldari et al.) 202T) from change point detection and U-
Time (Perslev et al. 2019) from time series segmentation for comparison. Anomaly Transformer still achieves the best performance.
Table 12: Additional quantitative results for Anomaly Transformer (Ours) in five real-world datasets.
The P, R and F/ represent the precision, recall and Fi-score (as %) respectively. F1-score is the har- monic mean of precision and recall. For these metrics, higher value indicates better performance.
Dataset
SMD
MSL
SMAP
SWaT
PSM
Metrie
P
F1
F1
BOCPD
70.90
82.04
76.07
80.32
87.20
83.62
84.65
85.85
85.24
89.46
70.75
80.22
75.33
77.70
TS-CP2
87.42
66.25
75.38
86.45
68.48
76.42
87.65
83.18
85.36
81.23
74.10
77.50
82.67
78.16
80.35
U-Time
65.95
74.75
70.07
57.20
71.66
63.62
49.71
56.18
52.75
46.20
87.94
60.58
82.85
79.34
81.06
Ours
89.40
95.45
92.33
92.09
95.15
93.59
94.13
99.40
96.69
91.55
96.73
94.07
96.91
98.90
97.89
J LIMITATIONS AND FUTURE WORK
Window size As shown in the Figure 7 of Appendix A too small for association learning. But the Transformers is with quadratic complexity w.r.t. the window size. The trade-off is needed for real-world applications.
the model may fail if the window size
Theoretical analysis As eetllsstabllssed deep model, the performance of Transformers has been explored in previous works. But it is still under-exploring for the theory of complex deep models. In the future, we will explore the theorem of Anomaly Transformer for better justifications light of classic analysis for autoregression and state space models.
K DATASET
Here the statistical details of experiment datasets.
Table Details of benchmarks. AR represents the truth abnormal proportion of the whole dataset.
Benchmarks
Applications
Dimension
Window
#Training
#Validation
#Test (labeled)
AR (Truth)
SMD
Server
38
100
566,724
141,681
708,420
0.042
PSM
Server
25
100
105,984
26,497
87,841
0.278
MSL.
Space
55
100
46.653
11,664
73,729
0.105
SMAP
Space
25
100
108. 146
27.037
427.617
0.128
SWaT
Water
51
100
396.000
99,000
449,919
0.121
NeurIPS-TS
Various Anomalies
100
20,000
10,000
20,000
0.018
19

---

Published as a conference paper at ICLR 2022
L UCR DATASET
UCR Dataset is very challenging and comprehensive dataset provided by the Multi-dataset Time Series Anomaly Detection Competition of KDD2021 (Keogh et al., Competition of International Conference on Knowledge Discovery & Data Mining 2021). The whole dataset contains 250 sub- datasets, covering various real-world scenarios. Each sub-dataset of UCR has only one anomaly segment and only has one dimension. These sub-datasets range in length from 6,684 to 900,000 and are pre-divided into training and test sets.
We also experiment on the UCR dataset for wide evaluation. As show in Table 14 our Anomaly Transformer still achieves the state-of-the-ard this challenging benchmark.
Table 14: Quantitative results in UCR Dataset. IF refers to the IsolationForest 2008 Ours is our Anomaly Transformer. P, R and FI represent the precison, recall and Fl-score (%) respectively,
Metric
LSTM-VAE
InterFusion
OmniAnomaly
THOC
Deep-SVDD
BeatGAN
LOF
OC-SVM
IF
Ours
62.08
60.74
64.21
54.61
47.08
45.20
41,47
41.14
40.77
72.80
R
97.60
95.20
86.93
80.83
88.91
88.42
98.80
94.00
93.60
99.60
F1
75,89
74.16
73.86
65.19
61.56
59.82
58.42
57.23
56.80
84.12
20


Reviews:
[{"review_id": "f1gFpWzYwxU", "reviewer": "Reviewer_bfbm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength:\n1) The paper is well written and easy to follow\n2) Figure 1 and 2 are very intuitive and easy to understand.\n3) Detailed empirical analysis.\n\n### Weakness:\n1) If I am not wrong then the SMAP and MSL dataset are from [a] but the author cite Su et al. 2019b.\n2) There are repetitive ve entries in references. For example Su et al. 2019a and Su et al. 2019b. I suggest that the authors recheck all entries in bibliography carefully.\n3) I think it is important to at least provide the reader with different types of methods used for anomaly detection. The authors have done a good job at it but in my humble opinion literature review is still missing some important papers. For example, LSTM based method and SMAP & MSL dataset [a], Dimensionality reduction & clustering method [b], Spatiotemporal method using Convolutional LSTM  [c], Tensor decomposition based method [d].\n4) I have several question/concerns/suggestions about the experiment section.\n\n    - How the threshold δ is set?\n    - Can you add more details and provide a solid reason on why r=0.5% for SMD, 0.1% for SWAT, 1% for other dataset.\n    - The author should discuss the false-positive rate in more details inside the main text as minimizing false-alarm is really important in practical scenarios.\n    - What is the reason behind selecting 3 layers for Anomaly Transformer?\n    - channel states of hidden state $d_{model}$ is set to 512. Can you provide a reason for selecting this number and also can you discuss the impact of increasing and reducing this number on performance, efficiency, memory etc.\n    - In my humble opinion, the term robustness is very loosely used in the paper. I suggest the author to tone down the sentences about robustness. \n    -  Building on the previous point, the robustness of Anomaly Transformer is not evaluated against adversarial attacks. Latest research have shown that almost all anomaly detectors such as MSCRED fails against simple FGSM and PGD attack. It would be interesting to see how the robustness of proposed method in those scenarios. \n    - Figure 5 and 6, are hard to understand. It suggest that the author add some background shading for anomaly regions and also add the threshold line so that the reader can easily understand how your method is outperforming other methods. Also, Figure 6 need more context and detail in the main text.  \n\n  5) I was unable to locate the link to the code repository. It is critical to validate the paper's claims, and one of the simplest ways to do so is to access and run the code. I believe that the authors should consider making the code for their method and empirical experiments available to the reviewers and later release it publicly.\n\n\n[a] Hundman, Kyle, et al. \"Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding.\" Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.\n\n[b] Yairi, Takehisa, et al. \"A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction.\" IEEE Transactions on Aerospace and Electronic Systems 53.3 (2017): 1384-1401.\n\n[c] Tariq, Shahroz, et al. \"Detecting anomalies in space using multivariate convolutional LSTM with mixtures of probabilistic PCA.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.\n\n[d] Shin, Youjin, et al. \"ITAD: Integrative Tensor-based Anomaly Detection System for Reducing False Positives of Satellite Systems.\" Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.", "summary_of_the_paper": "The authors begin by discussing the discrepancy in association between normal and anomalous points and then suggest an anomaly transformer based on an anomaly-attention mechanism that is further improved using a minimax technique. On six different datasets, empirical analysis demonstrates that the proposed method outperforms state-of-the-art anomaly detection methods.", "summary_of_the_review": "Well written. Good empirical Analysis. But in some places the paper lacks the justifications and proper reasoning. Experiment/results section need some better explanations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RHRyEfUqIbX", "reviewer": "Reviewer_BKSk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Update after the discussion with the authors.\n\nThe authors have addressed all of my immediate concerns. The paper now looks very strong. I recommend acceptance. \n\n----\nThe key idea of using Tranformer's self-attention as a measure of anomalousness sounds novel. It is an excellent idea. The proposed architecture featuring a two-branch attention mechanism also looks novel. The novelty seems undisputable, to the best of my knowledge.\n\nThe problem is, however, that the paper mostly ignores almost all the existing anomaly detection approaches for **time series**. It obviously does not make sense to use point-wise anomaly detection methods such as SVDD to capture sequential anomalies. Many researchers in this domain may agree that the baseline model in the present context can be the vector autoregressive model, which naturally realizes a particular type of self-attention in the form of the lag-dependent covariance matrices. You can find many works that leverage a dependency graph for anomaly or change detection. I'd also suggest looking at the literature on time-series segmentation. \n\nAnother issue is that the paper lacks sound justifications for the proposed approach. Given much theoretical/empirical work with VAR or state-space models (or their neural extensions), we expect a much more understandable derivation of the proposed model.  Many \"functions\" such as AssDiss lack a proper definition. \n (For example, I didn't understand how the probability distributions had been defined from P^l and S^l --- just writing SoftMax or just showing Eq.(2) does not mean you have defined a distribution for a **matrix**. The Gaussian distribution in Eq.(2) is defined in the entire real domain. But probably, you are on a regular time grid. Apparently, some mathematical inconsistency exists.)\n\nThat's not a problem if this were part of API documentation of a software library. But as a technical paper submitted to a top machine learning conference, there may be different expectations. \n\nI think the main idea deserves much more careful and deep thoughts. I encourage the authors to re-do the text to perfect it. I'm sure that the new version will be a great piece of work in the community.  \n", "summary_of_the_paper": "This paper proposes a new anomaly detection approach based on a Transformer architecture. The main idea is to leverage self-attention to capture the temporal dependency structure of the observations in a sliding window as a measure of anomaly. Since anomalies can be defined as a sequence that is inconsistent with regular ones, the attention matrix is expected to reflect some major aspects of anomalies. \n\nThe authors propose a two-branch attention architecture to handle multi-dimensional real-valued time-series data, where prior- and series-attention matrices are computed and utilized in a certain min-max competitive learning framework. \n\nThe final anomaly score is defined as the product between the re-construction error and the KL distance between the two attention matrices.  ", "summary_of_the_review": "Updated after the discussion\n- Novel idea of using the degree of self-attention as a metric of anomalousness.\n- New approach of positional encoding based on Gaussian kernels\n- Comprehensive empirical comparison with alternative methods\n\n----- Original summary -----\n- Novel idea of using the degree of self-attention as a metric of anomalousness.\n- Unclear and unjustified descriptions.\n- Lack of the critical baseline (major issue in this domain). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CUkXPP0F2Vf", "reviewer": "Reviewer_88ie", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Comments: \n- Is this approach a generalization / extension of change point detection? That is not mentioned at all.\n\n- The prior association is learned as one that minimized the difference with the series association and the series association is one that is learned to maximize the difference with the prior association? This will promote the series association to learn as non-Gaussian-like of a distribution as possible, while the prior association is finding the closest Gaussian distribution to the series association? Will this minimax strategy not lead to a degeneration where both distributions becomes extremely wide and almost uniform like?\n\n- “The reconstruction loss will guide the series-association to find the most informative associations, such as the adjacent time points of anomalies.” Why are adjacent time points of anomalies most informative? Is this because this “window” of time can be altogether be considered anomalies as opposed to noise from a single out of distribution point?\n\n- The anomaly score just indicated that there is anomalies or not within the N time point window? In order to narrow down where the anomalies are with the time points, do you need to then test smaller windows of time? Wouldn’t smaller windows cause there to be a shortage of data samples needed for learning?\n\n- KL divergence is not very good at measuring differences in the tails of distributions (it does not put enough importance on that area). But shouldn’t the anomalies be in the tails of the distribution as they are rare? \n\nMinor Comments:\n\nSome of the sentences are grammatically strange in the abstract and introduction sections. ", "summary_of_the_paper": "Summary:\nThis paper is looking at anomalies in time series data. In particular they define anomalies as ones that lie outside some learned Gaussian-like distribution. They propose a minimax objective function that tries to minimize and maximize the discrepancy between a Gaussian distribution with a learned variance parameter and an “empirical” one learned directly through self-attention on the data. They measure discrepancy using the symmetric KL divergence between the two distributions and also use this for their anomaly score. ", "summary_of_the_review": "Overall the paper has goods results and seems sufficiently interesting and novel.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AuZBgAdWdBl", "reviewer": "Reviewer_nXE3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The observation that the abnormal points have a strong local association (or prior association) and a weak global association is interesting. \n2. To better model the association discrepancy, the paper proposes a novel min-max association learning strategy to avoid the Gaussian prior reduction problem.\n3. Comprehensive evaluation on a variety of anomaly detection datasets demonstrate the effectiveness of the proposed association discrepancy learning strategy.\n\nWeaknesses:\n1. What's the convergence property of the min-max strategy? \n2. Some details are not very clear. \n- For example, in equation (2), what's the shape of $W^l$, and what does the operation $*$ mean?\n- Equation (6) is ambigous, $||\\mathcal{X}-\\hat{\\mathcal{X}}||^2_2$ is a scalar, while the AssDis score after Softmax is a N-by-d matrix.\n- In the implementation details, for the $r$, why not set $r$ similar to AR of the datasets as shown in Table 1?\n- In table 3, for the \"Recon\" of Anomaly Transformer, if you only use the reconstruction error, then there should be no optimization strategy for the association discrepancy. Why the optimization strategy is \"minmax\"? \n- In figure 5, what are the labels for the y-axis? For the reconstruction and association criteria, are they the values of loss?", "summary_of_the_paper": "This paper introduces an Anomaly Transformer for detecting anomalies in time series with association discrepancy. This paper introduces two discrepancies: the series association (e.g., period and trend) and the prior association (e.g., the local smoothness or continuity). For abnormal points, these two associations have a small discrepancy and for normal points, there is a large discrepancy between these two associations. This is because abnormal points have a strong local association while the normal points have a global association. In the model, a two-branch strategy is adopted to model the two associations separately. The model is trained by minimizing the reconstruction loss and maximizing the discrepancy. The experimental results demonstrate the effectiveness of the proposed model.", "summary_of_the_review": "In general, the observation that abnormal points have a large discrepancy between the prior association and the series association is very interesting. A novel Transformer based model is proposed to model this discrepancy. Experiments demonstrate the effectiveness of the proposed method. However, it is unclear about the convergence property for the proposed min-max strategy. Besides, there are many ambiguous details that hinder the readability of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a novel approach that leverages the discrepancies between the (global) series association and the (local) prior association for detecting anomalies in time series. The authors provided detailed empirical support to motivate the above detection criterion, and introduced a two-branch attention architecture for modeling the discrepancies and establishing an anomaly score.

All reviewers acknowledge the technical novelty of this work (including the key insight of modeling anomalousness with Transformer’s self-attention and concrete training mechanism via a minimax optimization process) as well as the comprehensiveness of the empirical study. 

Meanwhile, there were some concerns in the positioning of the work, in particular in the clarity in connection to related work, and some reviews concern the clarity of the presentation (e.g. missing some details in experimental results), and the clarity of the exposition of the training process. The authors provided effective feedback during the discussion phase, which helped clarify many of the above concerns. All reviewers agree that the revision makes a solid paper and unanimously recommend acceptance of this work. 

The authors are strongly encouraged to take into account the feedback from the discussion phase to further improve the clarity concerning the technical details as well as the reproducibility of the results.
Decision: Accept (Spotlight)