AnoFormer: Time Series Anomaly Detection using Transformer-based GAN with Two-Step Masking
Authors: Ah-Hyung Shin
Seong Tae Kim
Gyeong-Moon Park
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Anomaly detection
Transformer
Masking
Time series
Entropy

Abstract: Time series anomaly detection is a task that determines whether an unseen signal is normal or abnormal, and it is a crucial function in various real-world applications. Typical approach is to learn normal data representation using generative models, like Generative Adversarial Network (GAN), to discriminate between normal and abnormal signals. Recently, a few studies actively adopt transformer to model time series data, but there is no transformer-based GAN framework for time series anomaly detection. As a pioneer work, we propose a new transformer-based GAN framework, called AnoFormer, and its effective training strategy for better representation learning. Specifically, we improve the detection ability of our model by introducing two-step masking strategies. The first step is \textit{Random masking}: we design a random mask pool to hide parts of the signal randomly. This allows our model to learn the representation of normal data. The second step is \textit{Exclusive and Entropy-based Re-masking}: we propose a novel refinement step to provide feedback to accurately model the exclusive and uncertain parts in the first step. We empirically demonstrate the effectiveness of re-masking step that our model generates more normal-like signals robustly. Extensive experiments on various datasets show that AnoFormer significantly outperforms the state-of-the-art methods in time series anomaly detection.

AnoFormer: Time Series Anomaly Detection using Transformer-based GAN with Two-Step Masking
Anonymous Author(s)
Affiliation Address email
Abstract
Time series anomaly detection is a task that determines whether an unseen signal is normal or abnormal, and is a crucial function various real-world applications. Typical approach is to learn normal data representation using generative models, like Generative Adversarial Network (GAN), to discriminate between normal and abnormal signals. Recently, a few studies actively adopt transformer to model time series data, but there is no transformer-bass series anomaly detection. As pioneer work, we propose new transformer- based GAN framework, called AnoFormer, and its effective training strategy for better representation learning. Specifically, we improve the detection ability of our model by introducing two-step masking strategies. The first step is Random masking: we design random mask pool to hide parts of the signal randomly. This allows our model to learn the representation Exclusive and Entropy-based Re-masking: to provide feedback to accurately model the exclusive and uncertain parts the first step. We empirically demonstrate the effectiveness of re-mat masking step that our model generates more normal-like signals robustly. Extensive experiments on various datasets show that AnoFormer significantly outperforms the state-of-thee methods in time series anomaly detection.
GAN framework for time
10
11
12
normal data. The second step is propose novel refinement step
14
16
18
Introduction
20
Time series anomaly detection a crucial technology to prevent potential risks and financial losses in variety of areas, such detecting anomalies sensor data of large-scale plants [1. ECG monitoring [2). and the network traffic analysis [3] To deal with this task, from the classic methods
21
22
4
5
6
to the recent deep learning-basee based methods 2 7 8 10 12 13] many studies have
24
focused on unsupervised learning methods due to the lack of labeled anomalies and highly nonlinear
25
temporal dependencies.
26
One of major deep learning-basee approaches is a reconstruction-basee method. It typically uses an autoencoder (AE) Generative Adversarial Network (GAN) to learn the representation of normal data and to reconstruct a normal- signal from an input always. As a backbone network, the existing studies widely utilize CNN (Convolutional Neural Networks) 2 or RNN (Recurrent Neural Networks) S 10]. More recently, there have been attempts to apply transformer [14] to time series anomaly detection, and it shows remarkable performances [1T]. In this work, also adopt transformer to embed time series representation, but design an adversarial framework for anomaly
28
30
31
32
38
detection.
34 we devise GAN using transformer encoder, we expect that the model learns normal time series data and eventually generates real normal-like signals. However, there is major issue. Unlike the 36 AE structure, a pure transformer encoder- mnsder-bbased generator does not have a compressed latent space,
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37
i.e., it makes the model find the trivial solution, just copying an input and pasting to the output for the reconstruction. Therefore, we need a new training method for the generator learn the distribution of normal time series data. To address this issue, we introduce a novel two-step masking strategy. From this approach, the next question is where to mask an input signal to detect anomalies effectively. Understandably, in order to make the normal-like output, the best masking positions are abnormal points in the input signal. It is a challenging to mask the abnormal areas selectively because we do not know where the abnormal parts are in advance.
38
39
40
41
42
43
44 In this paper, we propose AnoFrrmrr which is a novel transformer- based GAN utilizing a pure transformer encoder only. To learn data representation effectively, we adopt masking strategy. We first train transformer-basee based GAN with random masking (Step 1) for representation learning of the normal time series data. While filling the randomly masked parts of the input at Step 1, the model learns the distribution of normal data effectively. In Step alone, all parts of the input signal cannot
45
46
47
48
49 be considered, and this randomness is big problem in anomaly detection. Therefore, we solve this problem by re-masking the exclusive parts of Step 1. Also, to find the best masking positions, we calculate entropy from the attention maps of transformer blocks and re-mask the parts with high entropy that is likely to be abnormal points with high uncertainty. This exclusive and entropy-basee re-masking (Step 2) provides feedback for better representation learning, eventually improving the anomaly detection performance. We experimentally prove that the proposed two-step masking is essential for AnoFormer to solve anomaly detection problem successfully.
50
51
52
53
54
56
Our contributions can be summarized as follows:
We propose a simple yet effective transformer-based GAN framework having generator and a discriminator for unsupervised time series anomaly detection, called AnoFormer. Moreover, present pre- processscs and embedding methods for our framework to deal with time series data effectively.
58
59
60
81
We introduce a new two-step masking method to encode the distribution of normal time series data. A newly proposed entropy-based re-masking helps our model to provide the feedback to the uncertain parts based on entropy. From the extensive ablations, we empirically verify that our two-step masking makes our model robust and successfully embed the representation of normal time series data.
69
65
66
AnoFormer achieves new state-of-the-art results with significant improvements on various unsupervised time series anomaly detection datasets: NeurIPS-TS, MIT-BIH, 2D-gesture, and Power-demand.
67
68
2
Related Work
69
70
Generative models using transformer have been proposed and applied to diverse domains, c.g..
71
computer vision [1 I6
17 18). natural language processing [19 20). and sequence modeling 22). In particular, these models are used to solve various tasks in the image domain, such as scene generation [15] 16, 23), saliency prediction [18], semantic segmentation [24], and sketch
72
21
73
74 synthesis (25]. Moreover, transformer presented solve graph-to-sequenc transduction task using graph neural network (26). text generation task (27), and time series forecasting task with the modified self-attention mechanism (28). We also utilize transformer to construct a generative framework, i.e., having both a generator and a discriminator. In this framework, we propose an appropriate embedding method and loss form to effectively solve the anomaly detection problems.
75
76
77
78
79
Many studies have used masking to the transformer architecture for effective representation learning. Including BERT 29]. which proposes the Masked Language Model (MLM) technique to pretrain 81 the language representation, many studies also adopt the masking methods, like 30 for action recognition, [30] for text classification task, [3I) for text log anomaly detection, 33] for visual representation learning. In [34], the CNN-based model learns the semantic context features by using a multi-scale mask across the whole image with different scales for anomaly detection in image
80
82
83
84
85 domain. We also use masking in our transformer-bate
GAN for time series anomaly detection, but 86 unlike the above studies, we propose the two-step masking strategy for training and test to provide
feedback that boosts the model to generate the uncertain parts successfully.
2

---

Rendoen Masking
Tranforna based Geuerator
Oulput
Pool
N: Lengh Sequenee Nuntaar blkess
Original Itput
Radka Seartis
Xi
Sch-rppu
Fnteiding at"h
(hee & Quali K4
Uutputt
Braadding y
Pre Proceesdd X
Exchedie& & Eatrupy-basee based Re-masking
Eaihedding
Invers Tarheddinng
Oulput
Transfermer-Joe lase Critic
Tereeriien dddition
Enttray
Tep 13h
Ang.
Input.X Ouigui R
Enabeliing
Tancoaa block
Limer Clastter
Catic Scores
Giee
Cousa Seukech
Radx
Figure I: Overview of the proposed AnoFormer. For simplicity, this figure shows the univariate case. In Step (random masking), a pre-processed input X masked with a randomly selected mask from predefined mask pool. After passing the masked input X" to the generator, X generated as the output by passing through embedding, transformer encoder, and inverse embedding layers. In Step 2 (exclusive and entropy- re- m-maasinnn), based on the entropy calculated from attention maps of all layers in Step 1, X re-man re-masked and X2 is generated again from the generator. Final output X is constructed via the combination of the masked parts of Step and Step With aid a critic, the generator able to generate more normal-like signals. Here, Ladv is the adversarial loss, including Ladv and La Note that the critic used only for the train time.
3 AnoFormer
89 In this section, we propose AnoFormer for unsupervised time series anomaly detection. We first define the target task including algorithm procedure briefly in Section 3.1 We then describe how 91 to construct a transformer- based GAN framework based on transformer encoder in Section B.2 Next, we introduce two different masking steps for our model to encode time series data effectively 88 in Section 3.3 Finally, we present the whole training scheme of AnoFormer in Section 3.4 Figure shows the overall architecture of AnoFormer.
3.1 Problem Definition
96 Let X = (x1, T2, ... , E 7 an input signal of T lengths, where 2t vector of dimension n. Since it is easier to get nor-
97 {wr xn. ...... , ]r] E Ih" time step mal time series data compared to abnormal ones, we train generator G and a discriminator D using only normal data without any label in an unsupervised manner. After training, for each unseen signal X, which can be normal or abnormal, the generator G generates normal- signal X. From the determine whether the observed signal X is normal not based on the reconstruction errors between the given signal X and the generated signal X.
98
99
100
generated signal,
3.2 Transformer-based GAN for Time Series Data
104
Pre- Proceesccssng To deal with an input signal for a transformer encoder, we need pre-processing step that makes the input signal discrete tokens. To this end, normalize each time series input X between -1 and by using the min-max scaling. Then, we quantize the normalized real value within a specific range [0, K), where the integer K is hyperparameter controlling the quantization resolution, and use the corresponding integer value as token. Let X lg''xa be the pre-processed signal. We set K 400 for all the experiments, in which the pre- pre-processed signal X looks almost like the input X. In total of K tokens (quantization levels), we add [MASK] token to utilize it for both training and test. To sum up, the input signal X is pre-processed to be X by applying scaling and quantization sequentially.
106
108
109
110
112
113
Embedding, An embedding step embeds discrete tokens into the embedding vectors. Here we use token embedding layer to map each token to the corresponding entry in embedding weight W e (k+))x We denote z' Txxx as the output token embedding, where d is embedding
3

---

116
dimension. We add a sinusoidal positional embedding p' to the token embedding x' to allow the model attend relative positions as follows:
117
2 z' + p'.
(1)
Transformer Encoder. A transformer encoder uses the embedding = E RTxd as the input, and outputs & E Rx'' Each block of the transformer encoder contains multi-head self-attention layer and a feed-forward network, followed by a residual connection and a layer normalization. Through the self-attention mechanism, it is possible to attend the relevant information of each time step at once, while multiple attention heads can consider different periodicities in time series data [35]
118
120
122
Inverse Embedding. We need to invert the output into the original form of time series, X RT
125
124
To this end, we introduce an inverse embedding layer to our model. We calculate the cosine similarity between the output embedding and the embedding weight W taken from the token embedding layer, and apply the softmax operation as follows:
125
2. WWWT
p- somr
|| WeT
From the above equation, we obtain the probability distribution P E R where Pt, means the probability that k will be selected in the range of [0, K) except the [MASK] token at the position t.
127
128
129
We then extract an index é; of the maximum probability for each time step t [1,2,- ... ,T] using the soft-argmax operation as follows:
130
f = soft soft-argmar(p)) = -Σ Σラー PP.
131
where 3 is a sufficiently large value, such 1000. Then, the indices in all time steps are concatenated to reconstruct the quantized output X as follows:
132
X = { 22. 3T}.
(3)
Transformer-based GAN Framework. To enhance the generation quality of X, we design an adversarial framework using transformer encoders. Following the notation of WGAN-GP (36), from now use the term critic C' instead of the discriminator D. Same as the generator G, construct the critic C' using the transformer encoder, but in the critic C, [CLS] token is added in front of the input tokens for classification. After passing through the transformer encoder, the linear classifier outputs the critic score using only the [CLS] token. While classifying the real input X and the fake output X, the critic C guides the generator G reconstruct more normal-like signal X. As result, our model can distinguish X whether it is normal abnormal according the difference between the input signal X and the reconstructed signal X from the generator G at test time.
134
136
137
139
140
141
142
3.3
Two-Step Masking for Time Series Encoding
143 In the previous section, we introduce the transformer- based GAN framework for time series data. However, we empirically find that the representation learning of the proposed transformer-basee GAN is not possible because the generator G just copies the input as the output always. Inspired by recent studies 35
145
32 33
that effectively learn the representation through masking in transformer,
147
we propose two different masking steps during training and test time: 1) random masking and 2) exclusive and entropy- based re- masking. We experimentally demonstrate that the proposed two-step masking is essential for our framework to learn the distribution of normal time series data successfully. In the following content, we describe how to mask the input effectively in each step with details.
149
151
Step 1: Random Masking, As the first step, we partially hide the input signal X using randomly selected mask from a mask pool. To construct the mask pool, we design a single mask in which the mask and the non-mask sections alternately appear. We then generate multiple masks by applying sliding window to the single mask, and group them as the mask pool. The composition of the mask
152
154
pool depends on length in of a single mask section, ratio I'm of all mask parts, and a stride 8m for the sliding window. The number of masks R.m in the predefined mask pool is determined as follows:
156
4m
T0m = 2 x
(4)
8m

---

157
Using the above equation, we generate the enough number of masks in the pool to cover all sections of the signal. During the train and test time, the mask is randomly selected in the predefined mask pool per each signal, and the generator G reconstructs X; from the masked input X
158
Step 2: Exclusive and Entropy-based Re-Masking. After X is generated from Step I, again mask the exclusive parts that are not covered Step for our model to consider all parts of the input. To avoid the error accumulation, here we re-mask the input X, instead of the first output X1- In addition, provide feedback to our model by re-masking the parts that the model considers uncertain during Step 1. To this end, we get an attention map from each layer of the generator as follows:
161
162
184
Q" K Vd
A softmar
11
A' A
where / E [1, ... 2j and A' is the attention map in the 1-th layer, calculated by the average of all attention maps for individual heads, A This layer-wise wise attention map determines how much
186
167
168
specific time step focuses the other parts of the input per signal. In this context, the uniformly distributed attention means that the model does not know which connections are valuable (37] iee.
169
170
the prediction is uncertain. To quantify the uncertainty, we calculate an entropy H x, of the masked
input X as follows:
(t) - ト A log Alj.
Hx, (H(),, (2), H(T))
172 To provide feedback on the high entropy parts, we re-mask 50% of the parts already masked in Step Then the generator G re-generates the second output X₂ from the masked signal X2. Finally, we combine the masked parts generated from Step 1 and the ones from Step 2 to construct the final output X. If there are overlapped parts between Step I and Step 2, the parts of Step 2 are used. From this re-masking step, we experimentally prove that our model becomes robust to unexplored and uncertain parts within a fixed model size. We also use the same random masking and re-masking strategies at test time.
173
174
176
178
179
3.4
Training AnoFormer
To train AnoFormer, we apply the cross-entropy loss to reconstruct the same input X from the final output X as follows:
180
181
Cree
X i log(p..))
(5)
i=1==1
182 where X denotes the one-hot label vector from the input and P.,, denotes the probability distribu-
tion of the final output X. Using X from the generator G during two-step masking, the critic C' tries
183
184 to minimize the following loss function:
X +x
(6)
LC = -I
[c X
(7)
185
where is randomly chosen between zero and one. The first term measures the Wasserstein distance and the second term is the gradient penalty, where X' is random sample from Px to enforce the Lipschitz constraint. The coefficient a harmonic parameter to balance the Wasserstein distance and the gradient penalty, where we use the value of 10. The loss function of the generator G is as follows:
187
188
189
C = -B [c (x)]
(8)
5

---

Table I: Quantitative comparisons in four datasets. For all of the metrics, higher value indicates better
performance.
NeurIPS-TS
Metric
Method
MIT-DIII
2D-gestune
Power-demand
Architecture
Glohal
Contestual
Shapelet
Sensonal
Trend
Averge
CNN
BeaGAN
0,9753
0.6188
0,7398
0,9742
1,0000
0.8372
0,9425
0,7256
0.5796
RNN
To0GAN
1,6000
04285
0,9834
0,9744
0.9327
0.9726
0.8256
0,5291
0.8438
AUROC
ensemble
0,9318
0.7246
0.8138
0.6587
RAMED
0,5365
0.9325
0,9084
0,9628
0,7259
08112
0,7839
0.6787
Transfemnrr
Anomaly Transformer
0,9931
0.624
0,7407
0,9352
0.9976
0.8400
0.8108
0,7858
0.7739
AnoFormer (Ours)
1,0000
0.9758
0.9900
0.9085
0.098
0
0.9532
0.8407
0.5667
CNN
BeHGAN
0.9555
0:7051
0.0817
0.9748
1,0000
0.9634
09747
0,4952
0.1228
RNN
TodGAN
7,0000
0.3603
0,9365
0,9751
0.873
0,9806
0.4621
0,4367
0 0.3098
RAE ensemble
0,0453
0,8297
0,8159
0,9191
0.1378
0.5496
0,5287
0.1400
ALP
RAMED
0.0443
0.223
0.6873
0,9109
0.1291
0.5188
0.5331
0. 1627
Tranformerr
Anomaly Transformer
0.6957
0.9978
0.9639
0,1967
AnoFormer (Ours)
1,0000
0,884
0,9901
0,9985
0,9987
0.9982
0.9187
0,6142
0.5584
CNN
BeaGAN
0,9345
0.7348
0.613
0.9487
1,0000
0.9008
0,8015
0,4944
0.2266
RNN
TadGAN
1,0000
0.3590
0.933
0,98M
0.8170
0,9389
0,5289
0.4138
0.5714
ensemble
0,333
0,7750
0.9181
0.3539
0.6003
F1 score
RAMED
0,0838
0.8272
0,6203
0,8782
0,4010
0.5627
0,5633
0 0.294
Transfemmer
Anomaly Transfommer
0
0.7258
0,6115
0.8730
0,9958
0.9011
0,5646
0.5
0.6033
AnoFormer (Ours)
1,0000
0.9400
0.9696
0.9913
0.9974
0.9798
0,8410
0.6667
0.6226
which makes the critic C not be able to classify the generated X. To sum up. the proposed AnoFormer is trained via the following loss functions for the generator G and the critic C:
190
LG = Areehree + Audo adv:
(9) (10)
Lc = Ludus
192
where we Xrec and dads as
4
Experiments
193
194
Datasets. We evaluated AnoFormer on four real-world benchmarks: 1) MIT-BIH contains 48 ECG records of test subjects from Beth Israel Hospital, 2) 2D-gesture contains time series of and coordinates of an actor's right hand, 3) Power-demand is dataset measuring the power comsumption for the Dutch research facility, and 4) NeurIPS-TS 2 38 is a synthetic dataset including five different time series anomaly scenarios as point-global, point-contextual, pattern-shapelet, pattern-seasonal, and pattern-trend More details on each dataset are summarized in Appendix A.
195
197
199
200
We compared our model with various baselines, including CNN, RNN, and transformer- based reconstruction models. BeatGAN (2) and TadGAN [ are CNN and LSTM-based GAN models, respectively. RAB-ensemble 9 is an ensemble of RNNs with sparse skip connections in
Baselines
201
202
203
autoencoder. RAMED 10 additionally uses the multiresolution decoding based on RAE-ensemble.
204
Anomaly Transformer
develops the transformer architecture to utilize association information.
205
Implementation Details. For both the generator G and the critic C, we utilized the basic transformer encoders with 9 and 6 layers for MIT-BIH, and 4 and 2 layers for other datasets, respectively. The embedding dimension and the number of heads are 128 and 8, respectively. The mask length 4m is
206
207
208
about 10% of the sequence length T, and the mask stride sm is about half of the mask length 4m-
209
We used Adam optimizer with initial learning rate, momentum B1, and 32 as 0.0001, 0.5, and 0.999, respectively, We implemented our model using PyTorch and trained on a NVIDIA RTX 3090 GPU.
210
211
4.1
Quantitative Results
212
Table shows the anomaly detection performances of each baseline on three different real-world datasets (i.e., MIT-BIH, 2D-gesture, and Power-demand), and a synthetic dataset (i.e., NeurIPS-TS
213
214
models showed high performances except MIT- MIH.BIH. In MIT-BIH, BeatGAN showed the second-bes, performances among all the benchmarks. In case of
38). Overall, RNN or transformer-bas
215
216
the proposed AnoFormer, this model outperformed all the baselines in four different datasets. In

---

ropu
0.5
0.25
0.23
3.00
0.00
0.00
-.25
0.2
4.25
0.50
4.50
-030
0.75
0.75
0.75
20
60
20
40
60
80
20
40
90
50
(a)
(b)
(c)
Input Output
0.20
0.20
0.00
0.00
0.00
4.20
-0.20
(d)
(c)
(f)
Figure 2: Output visualization in Point-Contextual (NeurIPS-TS) and MIT-BIH datasets. Left: visualization abnormal input and the normal-like output. Middle: reconstruction results of random masking (blue). Right: reconstruction results of exclusive (green) and entropy- based (red) re-masking. Vest viewed in color.
particular, model performed well NeurIPS-TS containing five types of outliers, and it means AnoFormer is robust to the various types outliers. AnoFormer achieved the state-of-the-arr results from small datasets (e.g., 2D-gesture and Power-demand) with about 1,000 training sets to large datasets (e.g., NeurIPS-TSs and MIT-BIH) with about tens of thousands training sets, and from univariate to multivariate cases. The experimental results demonstrate that the proposed transformer-
218
220
222
based GAN framework with the two-step masking strategy effective to reconstruct normal time series data for anomaly detection.
4.2
Qualitative Results
Figure shows the qualitative examples of AnoFormer. First column (Figure 2[a) and Figure d)) shows the abnormal examples of point-contextual of NeurIPS-TS and MIT-BIH datasets, respectively. The other columns show that the proposed madom masking and re-masking strategies actually provide feedback to our framework. For example, the incorrectly copied parts in Step I was refined by the entropy-basee re-masking (please see the black circles the figure). As shown the figure, when the abnormal inputs were received, the model generated the normal-like outputs. Therefore, AnoFormer can detect the abnormal points through the difference between the input and the output.
225
227
229
231
232
Ablation Study
233
We conducted various ablation studies to analyze the effectiveness of the proposed transformer- based GAN framework and two-step masking. All of the ablation studies were performed on the Point-Contene Cont-Contextull dataset of NeurIPS-TS, since it is the most difficult task to detect the anomalies out of the five types of outliers. Figure shows an example of Point-Contextus dataset, which has the small glitches the outliers. In Appendix B, we additionally examined the sensitivity of each hyperparameter newly adopted in our model.
234
236
238
4.3.1 Transformer-based GAN Framework
We first investigated the effectiveness of the transformer-bast adversarial framework in our model. In this experiment, we used BeatGAN as a CNN-based baseline. Table[]ssowws the ablation results when the generator and the critic use different backbone networks, such as CNN, and transformer. As shown in the table, the transformer-based generator showed higher performances on with large margins than the CNN- generator. Interestingly, we empirically found that there was
241
242
243
metrics
245 no synergy when using CNN-based critic with the transformer-base generator. means, not
helpful for the transformer-be based generator construct the critic with an inappropriate baseline. On

---

Table 2: Ablation study of the proposed transformer-based GAN.
Generator
Critic
AUROC
AUPRC
Fl score
CNN
CNN
0.6128
0.7051
0.7348
Transformer
0.9572
0,9735
0.9093
Transformer
CNN
0.9510
0,9675
0.9026
Transformer
Transformer
0.9758
0.9854
0.9400
Table Ablation study proposed two-step masking.
Step 1
Step 2
AUROC
AUPRC
Fl score
0.5000
0,3602
0.2386
Random
0,8557
0.7959
0.7548
Mask pool
0,9109
0.8651
0.8200
Mask pool
Mask pool (50%)
0,9277
0.8590
0,7808
Mask pool
Exclusive (50%)
0,9489
0.9622
0,9057
Mask pool
Exclusive Random (75%)
0,9709
0,9466
0,9004
Mask pool
Exclusive Anomaly score (75%)
0.9747
0.9533
0.9119
Mask pool
Exclusive + Entropy (75%)
0.9758
0.9854
0,9400
the other hand, the transformer-be beaded critic showed better performances than the baseline without the critic, which means it encourages the generated output to be close to the normal signal. From this result, we demonstrate that transformer-based GAN framework trained with the proposed masking strategy is effective to reconstruct normal time series data for anomaly detection.
248
249
250
251
4.3.2
Two-Step Masking
252
As shown in Table 3 we investigated the effect of two-step masking in our model. The first row means naive form of the transformer-batt GAN without any masking. The result was 0.5 of AUROC, which means the naive transformer-ban based GAN cannot distinguish between normal and abnormal signals at all. To overcome this critical issue, we adopted various masking strategies. First, we investigated the masking for Step 1. Here, Random means a fully random masking without any predefined mask pool. Mask Pool means our predefined mask pool defined Section 3.3 The results showed that regardless of the masking strategy. masking itself during training and test enabled the transformer-basee GAN to effectively learn the distribution of normal time series data. Moreover, we confirmed that Mask Pool is much better than Random masking, because each mask in the mask pool definitely covers the different parts from each other, providing a complementary effect.
253
254
255
256
257
259
261
262
Next, we conducted in-depth experiments to evaluate and compare different re- re-masking strategies in Step 2. Mask pool method in Step can be also used for re-masking. Exclusive method re-mar re-masks the exclusive parts of the random mask selected in Step 1. From the results, we found that re- re-masking
263
264
265
improved the detection ability of our model, and especially, exclusive masking strategy was really effective. This is because the model can consider the characteristic of whole signal during two-step masking. To provide more feedback to our model, we additionally re-masked the masked parts in Step 1. We experimented the following three cases: 1) Random method re-masks the signal randomly, 2) Anomaly score method re-masks the parts with high anomaly scores, and 3) Entropy method masks the parts with high entropy values. The results showed that masking the uncertain parts provided the proper feedback to our model, resulting in the highest scores among all the baselines. Therefore, we confirmed that the entropy- based re-masking is more effective than the other additional masking methods.
257
289
271
273
5
Discussion
We further conducted analysis to demonstrate the effectiveness of the proposed two-step masking strategy. To confirm the importance Step 2, we compared our method with the absence of Step Here, we also used Point-Contextudl dataset in NeurlPS-TT for analysis.
276

---

Step tet atnermal Input:
L Step 1 nomell trudt
20
Step z0pp abnomma Insust
Anramaay Score 1951
Anomay Score
02
12
0.6
0.5
Step
Step
40
60
80
a5-
Law Entrayy (+50%)
High Grenpy ra20%)
Lan Entrady Ic33%
High (rtrey
Eoech
Entroev (b)
Entropy
(a)
(c)
Figure 3: (a) AUROC and std of all masks in the predefined mask pool. (b) Analysis on the entropy antropy-based -based re-masking strategy (normal-case). (c) Analysis on the entropy-based re-masking strategy (abnormal-case).
278
The Effectiveness of Re-Masking. First, we investigated the validity re-masking step. We reported the average AUROC and std for all masks in the predefined mask pool in Figure [ In Step 1, there was a problem that the standard deviation (std) was too high because both training and test time had randomness in selecting the mask parts. By re-masking through Step 2, the randomness of the masked parts was eliminated and the std was significantly reduced. The performance of anomaly detection also increased with large margin by referring to the entire signal.
279
280
281
282
284
Analysis on Entropy-based Re-Masking. To understand the effectiveness of entropy-based re- masking intuitively, we visualized the relation between entropy and anomaly score in Figure 3)) and Figure (]) for both cases of normal and abnormal. Since the entropy-basee method re-masked the parts selected in Step I, we measured the anomaly score only in the parts corresponding to Step 1. We found two meaningful insights through the analysis. First, the higher the entropy, the more incorrect signal the model generates. In training phase, the high anomaly score means that the model generates output incorrectly, because only normal data is used. From the result of Figure 3))), the entropy was also high in the parts with high anomaly score, which means that the model did not generate signals well in the parts with high entropy during Step 1. This is because in the high entropy the attention uniformly distributed and the meaningful connection is not learned. By re-masking these parts in Step 2, anomaly score was significantly reduces, which means that the model reconstructed the normal data well in the training process. Second, the entropy-bacd based masking improves the discriminative ability between normal and abnormal in test time. As shown in Figure c), likewise in the case of normal, the higher the entropy, the higher the anomaly score in abnormal case. However, there was
285
286
287
288
290
292
294
296
298
also a part with a high entropy and a low anomaly score. These parts mean that the model copied the abnormal input it was without making it normal. It is possible provide feedback in both cases with entropy-basee based re- re-masking. By re- masking the parts with high entropy, anomaly score was consideaaby increased, and it means the parts that were not well generated due to copying in Step 1 were well re-generated. This makes it possible to further discriminate between normal and abnormal through anomaly score. In fact, large anomaly score does not mean getting close normal data. We
300
302
303
304
used a NeurIPS-Tr TS dataset to see if the output gets closer to normal data through re-masking. Since we synthesized abnormal datasets by injecting sporadic outliers in an additive manner, we could casily get the original normal version of the abnormal data. From this, we confirmed that the output was correctly getting closer the original normal through Step We further experimented about which layer's 's entropy information should be used? From the results in Appendix C, we calculated entropy from all layers and averaged them.
305
308
307
308
309
Conclusion
310
In this paper, we introduce AnoFormer, novel transformer-basee GAN for time series anomaly detection. To learn time series data directly with our model, we propose propessess and embedding methods suitable for time series data. A new training scheme based on two-step masking enables AnoFormer to embed the representation of normal signals. Especially, the exclusive and entropy- several
312
314
based re-masking method significantly improves the anomaly detection performances benchmark datasets. From the extensive experiments, we empirically demonstrate that our model is really effective solve time series anomaly detection. As future work, plan to study novel techniques for shorter inference time, and deal with time series data longer than an hour or a day.
316
318

---

References
319
320 [1] Ailin Deng and Bryan Hooi. Graph neural seawork-basede anomaly detection in multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4027-4035, 2021.
322
[2] Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI, pages 4433-4439, 2019.
324
325
[3] Zekai Chen, Dingshuo Chen, Xiao Zhang, Zixuan Yuan, and Xiuzhen Cheng. Learning graph structures with transformer for multivariate time series anomaly detection in iot. IEEE Internet of Things Journal, 2021.
328
327
328
[4] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
329
330
331
[5] Bernhard Schölkopf, John C Platt, John Shawe- Shawe-Taylor, Alex Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, 13(7):1443- 1471, 2001.
333
334
[6] David MJ Tax and Robert PW Duin. Support vector data description. Machine learning. 54(1):45-66 66, 2004.
335
336
Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems, 33:13016- 13026. 2020.
337
338
88] Alexander Geiger, Dongyu Liu, Sarah Alnegheimish, Alfredo Cuesta-Infamt and Kalyan Vecramachaneni. Tadgan: Time series anomaly detection using generative adversarial networks. In 2020 IEEE International Conference on Big Data (Big Data), pages 33-43. IEEE, 2020.
340
342
99] Tung Kieu, Bin Yang, Chenjuan Guo, and Christian Jensen. Outlier detection for time series with recurrent autoencoder ensembles. In IJCAI, pages 2725-2732, 2019.
344
[10] Lifeng Shen, Zhongzhong Yu, Qianli Ma, and James T Kwok. Time series anomaly detection with multiresolution ensemble decoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9567-9575. 2021.
345
346
347
Jiehui Xu, Haixu Wu, Jianmin Wang. and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations, 2022.
348
349
[12] Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for unsupervised anomaly detection and diagnosis in multivariate time series data. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 1409-1416, 2019.
351
353
354
[13] Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. Usad: unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3995-3404, 2020.
355
356
357
358
[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
359
380
[15] Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfsiion: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Processing Systems, 34, 2021.
362
363
364
[16] Dor Arad Hudson and Larry Zitnick, Compositional transformers for scene generation, Advances in Neural Information Processing Systems, 34, 2021.
10

---

386
[17]
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
367
[18] Jing Zhang. Jianwen Xie, Nick Barnes, and Ping Li. Learning generative vision transformer with energy-based latent space for saliency prediction. Advances in Neural Information Processing Systems, 34, 2021.
370
[19] Yufei Wang, Can Xu, Huang Hu, Chongyang Tao, Stephen Wan, Mark Dras, Mark Johnson, and Daxin Jiang. Neural uue-exeection tracking machine for transformer-basee -based text generation. Advances in Neural Information Processing Systems, 34, 2021.
373
[20] Marcella Comia,, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587, 2020.
376
377
378
[21] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
379
380
381
[22] Jiehui Xu, Jianmin Wang, Mingsheng Long, et al. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
382
383
384
[23] Dor Arad Hudson and Larry Zitnick. Compositional transformers for scene generation. Advances in Neural Information Processing Systems, 34, 2021.
[24] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
387
[25] Mingrui Zhu, Changcheng Liang. Nannan Wang, Xiaoyu Wang, Zhifeng Li, and Xinbo Gao. A kketch-transformen network for face photo-sketch synthesis. In International Joint Conference on Artificial Intelligence, 2021.
390
391
392
[26]
Deng Cai and Wai Lam. Graph transformer for graph-to-sequence learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7464-7471, 2020.
393
394
[27] Zhidong Liu, Junhui Li, and Muhua Zhu. Improving text generation with dynamic masking and recovering. IJCAI, 2021.
385
396
288] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
398
399
[29]
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
401
403
404
[30] Yi-Bin Cheng, Xipeng Chen, Dongyu Zhang, and Liang Lin. Motion-transformerr: self- supervised pre-training for skeleton-based action recognition. In Proceedings of the 2nd ACM International Conference on Multimedia in Asia, pages 1-6, 2021.
405
406
407
[31] Haixuan Guo, Shuhan Yuan, and Xintao Wu. Logbert: Log anomaly detection via bert. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021.
408
409
[32] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34, 2021.
410
411
412
[33] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
11

---

414
[34] Xudong Yan, Huaidong Zhang, Xuemiao Xu, Xiaowei Hu, and Pheng-ANe Ann Heng. Learning semantic context from normal samples for unsupervised anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3110-3118, 2021.
415
[35] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer- based framework for multivariate time series representation learning. In Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2114-2124, 2021.
418
420
421
[36]
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
423
425
[37] Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry. Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 606-613, 2020.
428
427
[38] Kwei-Herng Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang, and Xia Hu. Revisiting time series outlier detection: Definitions and benchmarks. Thirty-fiih fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
430
12


Reviews:
[{"review_id": "qqBt_yrIuyU", "reviewer": "Reviewer_ZSRp", "summary": "The paper proposed a transformer-based generative model, named anoformer, to detect anomaly for time series in an unsupervised manner. Specifically, anoformer combined the Transformer model and the GAN model by adopting transformer-blocks as the generator and discriminator module of GAN. Besides, a two-step masking mechanism was designed to improve the model robustness. Through extensive experiments on 4 public datasets, including real-world and synthetic datasets, the superiority of anoformer was verified. ", "questions": "please clarify the question below\n* One sample in this paper is the entire time series or the subsequence of the time series? If it is subsequence, how do you do the sub-sampling. \n\n* Whether the Anoformer can provide the anomaly detection in a real-time manner ? Please states the computing complexity and execution speed information. \n\n* The formulation of the critic C is missing in the section 3.4, please further clarify it. ", "limitations": "We have concerns on the following points:\n\n* The experiment section only introduces the F1-based metrices, while F1-score highly depends on the threshold. Other metrics, e.g., AUC, the complexity, the running time, should be added.\n\n* It is not very clear Anoformer provides the anomaly detection service on the data-point level or the time-series level? It will be great differences on performance and execution complexity. \n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper is easy to follow with some well-designed figure to visualize the key modules of the Anoformer framework. In particular, each key module of Anoformer is elaborated on in following sections.\n+ The introduction and the related work section provide a clear and brief introduction of the transformer-based generative models and the difference between the Anoformer and SOTA models.\n+ The two-step mask is designed to avoid the generator just copying the input as the output. Specifically, the mask pool in the random mask step provides different views of the input; the exclusive mask in the re-mask step makes the model consider all parts of the input; the entropy-base mask in the re-mask step makes the model consider the uncertain part.   \n+ The extensive ablation experiment on different datasets verify the contribution of each key module of the Anoformer. \n\n\nWeakness:\n- Anoformer detects anomaly based on the entire input or the subsequence of the input. The length of the input time series impacts the delay of the anomaly detection service.\n- Too many hyper-parameters are introduced in different sections. A summary table of these parameters should be added for a better review experience.\n- The referred paper [30] was cited as both action recognition and text classification reference in line 81-82, while [30] only relates to action recognition. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DZt_mKW91xA", "reviewer": "Reviewer_73J5", "summary": "This paper tackles the task of detecting and localizing anomalies in time series data. It follows a transformer approach where time series are automatically masked, then the transformer attempt to reconstruct the masked region. The reconstruction error is used as an anomaly score. A few technical tricks are incorporated such as using a GAN loss in training and an adaptive masking strategy. The method is evaluated on a few datasets and achieves better results than a set of deep learning approaches.  ", "questions": "* How were the benchmark datasets chosen?\n* How do the classical benchmarks for NeurIPS-TS/[2] etc. perform?\n* How well does the method scale to multivariate TS (e.g. 100 dimensions)?\n\n", "limitations": "Some were discussed in the appendix. A few more were highlighted in the review. A more extensive discussion would have been helpful.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The main premise of using inpainting as a task for reconstruction-based anomaly detection in not new (e.g. [1] but there are many). Nor is using a GAN discriminator (also found in [1]). The main difference here is using a transformer, the particular masking approach and the particular hyperparameter choices. This is an original contribution but a modest one. \n\nQuality: The method appears sound as far as it goes. The GAN ablation is somewhat worrisome as it appears to be useful only in a particular configuration and could be due to randomness in the data or a particular hyperparameter choice. However, the evaluation is not very convincing. The NeurIPS-TS datasets were generated by the authors, and are really easy. Note the original paper had classical baselines as doing much better than deep learning methods - but the original datasets were much smaller than the ones generated here. Also the other 3 benchmarks are not standard. It is not clear to me how they were selected. Also simple/classical baselines were not attempted. Several papers (e.g. NeurIPS-TS cited in this paper or [2]) have recently show that deep learning methods do not outperform classical baselines in many cases. Another issue I did not not see highlighted - the transformer + discretization approach might find it hard scale to multivariate time series (as the number of token to be predicted might expand by two orders of magnitude, in some cases). Although one of the benchmarks had 2 dimensions, it is not clear to me how this method would scale to standard benchmarks used by the deep learning time series AD community (SWaT, MSL, WADI etc.). \n\nClarity: The paper is well written, was a pleasure to read.\n\nSignificance: Due to the concerns mentioned above and the limited originality, I think this work is at risk of having low significance.   \n\n[1] Yan, Xudong, et al. \"Learning semantic context from normal samples for unsupervised anomaly detection.\" AAAI'21\n[2] Kim, Siwon, et al. \"Towards a rigorous evaluation of time-series anomaly detection.\" AAAI'22.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "APADY-yKZzw", "reviewer": "Reviewer_W6VQ", "summary": "The paper proposes a novel transformer-based GAN framework called AnoFormer for time series anomaly detection. In addition, the proposed model is enhanced by a two-step masking strategy consisting of 1) random masking and 2) entropy-based re-masking. The experimental results show the proposed AnoFormer achieves state-of-the-art performance on several benchmark datasets on time series anomaly detection.", "questions": "Why do we make the input signal discrete instead of continuous?\n\nWhy do we build the mask pool like this?\n\nWhy do we re-mask 50% of parts masked in Step 1?\n\nWhy the hyper-parameter $\\epsilon$ is set to random? What is the distribution $P_{X'}$ in Line 186?\n\nThe performance of the paper is attractive, however, considering the novelty and the unclear elaboration, I vote for borderline reject. ", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The paper is easy to follow and well-organized.\n+ The transformer-based GAN framework and the two-step masking strategy work well.\n+ The performance gains are significant.\n\nWeaknesses\n\n- Lack of motivation. The paper only talks about how it is designed, while not elaborating clearly on why it is designed like this. E.g. Why do we make the input signal discrete instead of continuous? Why do we build the mask pool like this? Why do we re-mask 50% of parts masked in Step 1? Why the hyper-parameter $\\epsilon$ is set to random?\n- Lack of novelty and related work. Actually, the transformer-based GAN framework for time series is not novel and has been proposed in [1], which is not cited and discussed in this paper. The related work section should be more specific (e.g. time series related or time series anomaly detection related literature). The baselines, such as BeatGAN[2], TadGAN[3], RAMED[4], and Anomaly Transformer[5], should be discussed in this section.\n- Some notations and presentations are confusing. E.g. $\\epsilon$ in Line 185, how it is randomly sampled, with uniform random or normal random? And what is $P_{X'}$ in Line 186?\n- The performance is good but codes are not provided.\n\n[1] Adversarial Sparse Transformer for Time Series Forecasting. NeurIPS 2020\n\n[2] BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series. IJCAI 2019\n\n[3] TadGAN: Time Series Anomaly Detection Using Generative Adversarial Networks. IEEE International Conference on Big Data 2020\n\n[4] Time Series Anomaly Detection with Multiresolution Ensemble Decoding. AAAI 2021\n\n[5] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy.  ICLR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject