MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection
Authors: Junho Song
Keonwoo Kim
Jeonglyul Oh
Sungzoon Cho
Venue: NeurIPS 2023 poster (2023)
Keywords: Multivariate time series
Anomaly detection
TL;DR: We introduce an unsupervised memory-guided Transformer model for multivariate time series anomaly detection.

Abstract: Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, reconstruction-based deep models have been widely used to solve the problem. However, these methods still suffer from an over-generalization issue and fail to deliver consistently high performance. To address this issue, we propose the MEMTO, a memory-guided Transformer using a reconstruction-based approach. It is designed to incorporate a novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, we use a two-phase training paradigm which involves using K-means clustering for initializing memory items. Additionally, we introduce a bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method on five real-world datasets from diverse domains, and it achieves an average anomaly detection F1-score of 95.74%, significantly outperforming the previous state-of-the-art methods. We also conduct extensive experiments to empirically validate the effectiveness of our proposed model's key components.

MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection
Junho Song
Keonwoo Kiml.
1.2+
Jeonglyul Oh
Sungzoon Cho
Scoul National University
VRRRRW Inc.
{jhsong, keonwookim, jamesch0813)@bdai snu, ac.kr
zoon@snu, ac. kr
Abstract
Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, ientttuuctioobbaaed deep models have been widely used to solve the problem. However, these methods still suffer from an wvergenneallization issue and fail deliver consistently high performance. To address this issue, propose the approach.
MEMTO, memory-guicee -gidedd Transformer using a reconstruction-bast It is designed to incorporate novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, involves using K-means clustering for initializing memory items. Additionally, we introduce bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method five real-world datasets from diverse domains, and achieves an average anomaly detection F1-score of 95.74%, significantly outperforming the previous state-of-the-an methods. We also conduct extensive experiments empirically validate the effectiveness of our proposed model's 's key components.
two-phase training paradigm which
1 Introduction
As cyber-physical systems advance, a vast amount of time series data is continuously collected from numerous sensors. Anomalies resulting from malfunctions critical infrastructures, such as water treatment facilities and space probes, can incur fatal property loss. The task of multivariate time series anomaly detection involves identifying whether each timestamp of the multivariate time series is normal or abnormal. Anomaly detection in real-world scenarios is challenging due to severe data imbalance and the prevalence unlabeled anomalies. We have formulated the problem as an unsupervised learning task to tackle these challenges. The underlying assumption of this approach is
that the training data solely consists of normal samples [25] 421.
Traditional unsupervised learning methods such as one-class SVM (OC-SVM) 28 support vector data description (SVDD) [35], isolation forest (20), and local outlier factor (LOF) 5 have been widely used for anomaly detection tasks. Recently, density-
methods combined with deep representation learning, such as DAGMM (45 and MPPCACD [4], have also been presented. For the clustering- based method, Deep SVDD 25 finds the smallest hyper-sphere enclosing most normal samples the feature space trained using deep neural networks. However, they exhibit poor performance in the time series domain due to their inability to capture dynamic nonlinear temporal
indicates equal contributions indicates corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

dependencies and complex inter-variable correlations. Deep models tailored for sequential data have been introduced to solve these inherent challenges 30
THOC (2 uses a differentiable hierarchical clustering mechanism to combine temporal features of varying scales from different resolutions. Multiple hyper-spheres are used to represent each normal pattern at various resolutions, improving the representational capacity to capture intricate temporal features of time series data.
29
One of the primary types of recent deep methods is icconstructionn-assed method, which uses an encoder-decoder architecture trained by a self-supervised pretext task of reconstructing input. This approach expects accurate reconstruction for normal samples and high reconstruction errors for anomalies. Early methods include LSTM --SeM-
-ecoodr model [22] and LSTM- VAE (23). OmniAnomaly [31] and InterFusion [19] are other stochastic recurrent neural network- waaserk--eedddd-d models extended from LSTM- VAE. Another branch of the reconstruction- based method is deep generative models. MAD-GAN 17 and BeatGAN
42 are generative adversarial network [9] variants tailored for time series anomaly detection. The most recently proposed Anomaly Transformer [40] introduces the mhomaly-Attention mechanism to simultaneously model prior- and series- associations. However, to the best of our knowledge, existing reconstruction-bent based methods may suffer from over- generalization problem, where abnormal inputs are reconstructed too well [8] 24]. This can occur if the encoder extracts unique features of an anomaly or if the decoder has excessive decoding capabilities for abnormal encoding vectors.
Our paper presents a new reconstruction- based method, memory-guided Transformer, for multivari- ate time series anomaly detection (MEMTO). One of the key components in MEMTO is the Gated memory module, which includes items representing the prototypical features of normal patterns in the data. We employ an incremental approach to train the individual items in the Gated memory module. It determines the degree to which each existing item should be updated based on the input data. This approach enables MEMTO to adjust to diverse normal patterns in a more data-driven manner. Recon- structing abnormal samples using the stored features of normal patterns in the memory items can lead to reconstruction outputs that resemble normal samples. Under this condition, MEMTO finds difficult to reconstruct anomalies, thereby releasing the over-generalizzation issue. Moreover, we have noticed that updating memory items incrementally can result in training instability if the items are initialized randomly. Therefore, we propose two-phase training paradigm to ensure stable training by injecting inductive bias of normal prototypical patterns into memory items through applying K-means clustering. Furthermore, our observation reveals that the latent representations of anomalies are considerably more distant from memory items than those of normal time points since each item encompasses a prototype of normal patterns. Consequently, while calculating anomaly scores, attempt to fully employ the memory items' nature of storing prototypical features of normal patterns. We introduce bi-dimensional deviation-based detection criterion that comprehensively considers both the input and latent space. Our approach has been tested on five standard benchmarks, which include real-wor wolldwrldd applications. Experimental results indicate that our approach is effective and achieves fatt--ofthheaasd art anomaly detection results compared to existing methods. The contributions of this paper are fourfold:
Our proposed MEMTO is the first multivariate time series anomaly detection method that uses the Gated memory module, which adjusts to diverse normal patterns in a data-driven manner.
We propose a two-phase training paradigm, an universally applicable approach designed to enhance the stability and robustness of memory module-based models.
We propose bi- bi-dimensional deviation based detection criterion in the online detection process. It considers both latent and input space to aggregate information in data compre- hensively.
MEMTO achieves state-of-the-arr results on five real-world benchmark datasets. Extensive ablation studies demonstrate the effectiveness of key components in MEMTO.
2
Related work
In this section, we provide a brief explanation of the memory network. [ 38 are early studies of external memory that can be retrieved and written. 34 proposes the improved version of Memory Networks (8), which uses attention mechanism training supervision due to end-to-end training manner. Recently, several attempts have been
3 to retrieve memory entries, thus requiring less

---

772
X
X
45
46
Transformce Faender
Mi:
4'
Weak Decoder
Aeele Map
95
Uplenl
01:
leput
9C
aud
Money
Chenee
Setioeel nomory
Query updare stage
memory splate stage
Illustration of proposed MEMTO. Green >4 depicts the update gate y in the Gated memory update stage, which controls the degree of updating items. Red 4 represents concatenation of queries q and retrieved memory items qfc 'Weak Decoder' represents two fully-connected layers.
Figure 1:
made apply memory network in various domains. Natural language processing includes question 34 In computer vision, there exist video representation learning [12], one-
answering [16
39
IT
shot learning [4 6 27 and text-to-image synthesis 44 Moreover, several works have also adopted
memory network for anomaly detection in computer vision g 24 21 Although MemAF [ is the first integrate memory network into autoencoder architecture, it lacks explicit memory updating process. MNAD [24] has suggested a distinct strategy for updating memory, which involves explicitly storing diverse normal patterns of normal data in memory items. Nonetheless, this approach simply adds a weighted sum of the relevant queries to each memory item without controlling the degree of new information being injected into existing memory items. However, our Gated memory module in MEMTO is the first to adapt to diverse normal patterns in data-driven way by learning how intensively each existing memory item should be updated in response to new normal patterns.
3 Method
3.1 Overall description of MEMTO
We define the raw time series D as a set of sub-series D {X) (X.,...,,*"" X.",, where N represents the total number of sub-series. Each sub-series X* E R is sequence of timestamps [ai...., .r,]]. L, n, and x E R" represent sub-series length, input dimension, and timestamp at time t, respectively. q² .g.....,99) .7. E I L.xC refers to the encoder output feature (i.e., query sequence) corresponding to X", where C denotes latent dimension. of E CRC 1,.., L.) is an individual query vector (i.c., a query) at time t.
Figure iillustrates the overall architecture of MEMTO, which mainly consists of encoder-decoder architecture with the Gated memory module. The input sub-series X first fed the encoder, and the encoder output features are then used queries to retrieve relevant items or update items in the Gated memory module. The inputs the decoder are the updated queries 9 R (t 1.... L). which are the combined features of the queries of and the retrieved memory items of E RC, The decoder maps the updated query sequence d [ ..f,.....90 ,.. back to input space, and outputs the reconstructed input sub-series X E RLxn, Algorithm shows the overall mechanism for our proposed model in Appendix B
3.1.1 Encoder and decoder
Transformer well known to capture long-term complex temporal patterns time series data [43] MEMTO uses Transformer encoder to project the input sub-series into a latent space.

---

In contrast, we adopt weak decoder consisting of two fully-connected layers. It is essential that the decoder should not be overly powerful, as this can lead situation where its performance is independent of the encoder's s ability to encode input time series. In the extreme case, strong decoder comprised of multiple deep layers can generate the input data accurately even from random noise that contains no information about input data [37]
3.1.2 Gated memory module
In order tackle the problem of vver-generalization in reconstruction based models, we introduce new memory module mechanism that adjusts to diverse normal patterns in data-drives manner. In this approach, each item stored in the memory module represents the prototypical features of normal data. Our proposed two-stage iterative update process allows to extract latent representations of input sub-series using the prototypical patterns of normal data stored in the memory module, which acts as regularization to mitigate ver-generalization. This mechanism is designed to limit the encoder's ss ability to capture the unique properties of anomalies, thereby making it more challenging to reconstruct abnormal data.
Gated memory update stage Memory item m, E RC (i= M), where M denotes the number of memory items, is trained to contain prototypical normal patterns of the input time series [24]. We expect memory items to contain prototypes all queries of corresponding to normal timestamps. Thus, we adopt an incremental approach for updating the memory items by defining query-conditioned memory attention ur, is calculated by softmax on dot product between each memory item and queries follows:
exp (m,, gi) /T)
vf, P(m4 + qi) Σー expp (m,, .g)) /T)
(1)
where 7 denotes a temperature hyperparameter. We have proposed an update gate và in our memory module mechanism to train each memory item flexibly according to various normal patterns. This gate controls the extent to which the newly acquired normal patterns from queries are infused into the existing prototypical normal patterns stored in memory items. allows our model to learn to what degree each existing memory item should be updated in data-driven way. Equations for our memory update mechanism are:
[vmm (U + W
1/ =
Σ
(2)
1=1
m4 13) m4 + th
(3)
where Up and We denote linear projections, and a and o denotes sigmoid activation and element-wise multiplication, respectively. The memory update stage is only executed in the training phase.
Query update stage In query update stage, we generate updated queries of which are then fed to decoder inputs. Similar to the Gated memory update stage, we define nemory-conditione query-attention w calculated by softmax on dot product between each query and memory items as follows:
exp((mi, qt) /T)
W P(g mi)
ΣM exp((m gE) /T)
Then, retrieved memory item of is obtained by weighted sum of the items using wi, as its corresponding weight as follows:
M
u
m
(5)

---

Query qi and retrieved memory item q are concatenated along feature dimension to constitute updated query q. The updated queries qi (t it...... L) are the new robust inputs for the decoder since unique properties of an anomaly in can be offset by the relevant normal patterns in the memory items. Reconstruction outputs of anomalies often appear similar to normal samples, making it more challenging to reconstruct anomalies. Such intensified difficulty can help distinguish between normal and abnormal data more effectively and prevent wwer-generalization.
3.2 Training
For the pretext task for self-supervision, we minimize reconstruction loss while training. The reconstruction loss Lree is defined as L2 loss between X x
N X
Lieec
X
2
Dense W' enables some anomalies to have the probability of being well reconstructed [ where W' denotes the matrix form w (I. -,...., [.) and (i 1i .M) in (4). Hence, to guarantee
retrieving only restricted number of closely relevant normal prototypes in memory, introduce
entropy loss Lentr as our auxiliary loss for sparsity regularization on W*:
M
1
Lentr
wi,,log(wi,,)
t=
The objective function is to minimize the combination loss term and 7 as follows:
Tiree Alientrs
(8)
where x denotes a weighting coefficient.
3.3
Two-phase training paradigm
Algorithm Memory module initialization with K K-means clustering
Input: X R. randomly sampled sub-series
fe: trained encoder of MEMTO after first phase E MCC randomly initialized memory items (Nrand: number of sub-series, M: number of memory items, C: latent dimension)
1: q raand (X)
\\ rand E RTC where Nrand x
2: K-means (q" rand)
Get M clustering centroids e MxC
3: for i 1,..., M do
Initialize memory items m with clustering centroids
4: m; = C2
5: end for
6: return m
Initializing memory items with K-means clustering Since we update the memory items incre- mentally, there is a risk of instability during training if the items are randomly initialized. We suggest novel two-phase training paradigm that uses clustering method to set the initial value of memory items to the approximate normal prototypical pattern the data.
In the first phase, MEMTO is trained by supervisd task of reconstructing inputs, and the trained encoder of MEMTO generates queries for the randomly sampled 10% of the training data, We then apply K K-menns clustering algorithm to cluster the queries and designate each centroid as the initial value of a memory item. In the second phase, MEMTO is trained with these well-initialized items on anomaly detection tasks. Algorithm | outlines the memory module initialization with K K-means clustering. In all of our experiments, we use K K-means clustering as a baseline clustering method. Our claim is not that K K-means clustering is the optimal choice among various clustering
5

---

methods, but rather the effectiveness of two-phase training paradigm that allows for the selection of an appropriate clustering algorithm depending on the type of dataset.
3.4 Anomaly criterion
We introduce a bi-dimensional deviation-based detection criterion that comprchensively considers input and latent space. We define Latent Space Deviation (LSD) LSD(g) m) at time point t as distance between each query oi and its nearest memory item mi 6,000 latent space (9) LSD of anomalies would be larger than those of normal time points because each memory item contains prototype of normal patterns. Additionally, at time as distance between input X E R" and reconstructed input E, E IR" in input space 100
define Input Space Deviation (ISD) ISD(X;,.. X
LSD(g m) = 1192 -m,
4.pos
(9)
X ||
IXD
1. X
X
(10)
We multiply normalized LSD with ISD, using LSD as weights for amplifying the normal-abnormal gap in ISD:
A(X*) softmaz (LLSD(ff m) IISD(X X(,))
(11)
where element-wise multiplication and A(X*) E R+ is anomaly score at each time point. Leveraging normal-abnormal distinguishing criteria in latent space (i.e., Latent Space Deviation) and input space (i.e., Input Space Deviation) leads to better detection performance.
4 Experiment
Table 1: Precision(P). rec,ll(R), FI-score(FI) results (as %) on five real-world datasets. 'A.T.' and 'avg' indicate Anomaly Transformer and average. We reproduce the MEMTO and Anomaly Transformer results while adopting the reported performance from [40] for the other baselines.
SMD
MSL
SMAP
SWaT
PSM
avg.
R
F1
P
R
F1
F1
LOF
56.34
39.86
46.68
47.72
85.25
61.18
58.93
56.33
$7.60
72.15
65.43
68.62
57.89
90.49
70.61
60.94
OC-SVM
44.34
76.72
56.19
59.78
86.87
70.82
53.85
59.07
56.34
45.39
49.22
47.23
62.75
80.89
70.67
60.25
Foresi
42.31
73.29
53.64
53.94
86.54
66.45
52.39
59.07
55.53
49.29
44.95
47.02
76.09
92.45
83.48
61.22
MMPCACD
71.20
79.28
75 02
81.42
61.31
69.95
88 61
75.84
81.73
82.52
68.29
74.73
76.25
78.35
77.29
75.74
DAGMM
67.30
49.89
57.30
89.60
63.93
74.62
86.45
56.73
68.51
89.92
57.84
70.40
93.49
70.03
80.08
70.18
Deep-SVDD
78.54
79.67
79.10
91.92
76.63
83.58
89.93
56.02
69.04
80.42
84.45
82.39
95.41
86.49
90.73
80.97
90.99
89.54
$8.01
LSTM-VAL VAE
75.76
90.08
82.30
49
79.94
82.62
92.20
67.75
78.10
76.00
89.50
82.20
73.62
89.92
80.96
81.24
BeaGAN
72.90
84.09
78.10
89.75
85.42
87.53
64.01
87.46
73.92
90.30
93.84
92.04
80.24
OmmiAnomaly
83.68
86.82
85 22
89.02
86.37
87.67
92.49
99
86.92
81.42
84.30
82.83
88.39
74.46
80.83
84.69
InterFusion
80.59
85.58
83.01
83.45
83.52
T.T.
87.96
94.68
91 20
91.13
90.12
90.63
93.96
98.45
96.15
85.85
100.00
92.39
96.81
98.63
97.71
93.62
MEMTO
89.13
98.40
93.54
92.07
96.76
94.36
93.76
99.63
96.61
94.18
97.54
95.83
97.46
99.23
98.34
95.74
4.1 Experimental setup
Dataset We evaluate MEMTO on five real-world multivariate time series datasets. (i) Server Machine Dataset (SMD 33 is a large-scale dataset collected over five weeks with 38 dimensions released from a large Internet company. (ii & iii) Mars Science Laboratory rover (MSL) and Soil Moisture Active Passive satellite (SMAP) are public data released from NASA 13] with 55 and 38 dimensions, respectively. (iv) Secure Water Treatment (SWaT (18) consists of a six-stage infrastructure process with 51 sensors under 11 days of continuous operation.
(v) Pooled Server Metrics (PSM I]) consists of data of diverse application server nodes from eBay with dimensions. Additional details the datasets can be found in Appendix A

---

Implementation details We generate sub-series by applying non-overlapped sliding window with length of 100 to obtain fixed-length imputs for each dataset. We split the training data into 80% for training and 20% for validation. More detailed information on hyperparameter settings can be found in Appendix [ The standard evaluation metries for binary classification, including precision, recall, and Fl-score, fail to account for the sequential properties of time series data, making them insufficient in assessing contextual and collective anomalies [10, 4]. Therefore, we use adjusted versions of these metrics, which have become widely used evaluation metrics in time series anomaly detection [32] 2 Here point adjustment method is used, where if single timestamp is detected as abnormal, every time point in the anomalous segment containing that timestamp is considered correctly detected.
4.2 Main results
In the main experiment, we evaluate the performance of MEMTO on multivariate time series anomaly detection tasks by comparing it with 12 models. The traditional machine learning baselines are LOF [5), OC-SVM (28). and Isolation Forest [20]. We also compare with several recent deep models, including density-estimation models (MPPCAD [4]] and DAGMM (45),, clustering-base models (Deep-SVDD (26] and THOC 29 ), and reconstruction baaed models LLSTM-VAE 23
BeatGAN 42). OmniAnomaly [II InterFusion [1]], and Anomaly Transformer [40]).
T shows the evaluation results five multivariate time series anomaly detection tasks. Overall, models using Transformers such as MEMTO and Anomaly Transformer consistently show high performance with an F1-score of over 90% all datasets. However, MEMTO substantially improves the average F1-score on benchmarks compared to the previous state-of-the-art model, Anomaly Transformer, from 93.62% 95.74%. MEMTO effectively detects anomalies in complex time series data by adjusting to diverse normal patterns present in the data and creating prototypes of these patterns using a data-driven approach, thus achieving new state-of-the-art results.
4.3 Ablation study
In this section, we comprehensively analyze the proposed model, MEMTO, through a series of ablation studies on three key components: the anomaly criterion, the memory module, and the training paradigm. Appendix C shows a more detailed analysis of the ablation studies.
Table 2: Effectiveness of ISD and LSD.
Anomaly Criterion
F1-score
ISD
LSD
SMD
MSL
PSM
SMAP
SWaT
avg.
X
77.54
87.22
79.25
70.99
31.17
69.23
X
V
72.78
80.33
80.15
67.55
0.00
60.16
/
93.54
94.36
98.34
96.61
95.83
95.73
Table 3: Ablation results of memory module and training paradigm. 'MM' denotes memory module.
F1-score
K E-mans
SMD
MSL
PSM
SMAP
SWaT
avg.
w/o MM
77.40
87.47
79.83
71.20
31.42
69.46
89.12
90.42
86.70
70.16
61.12
79.50
MemAE
V
91.39
92.50
98.00
96.13
93.06
94.21
65.56
85,41
93.43
69.45
82.31
79.23
MNAD
V
92.39
92.13
98.13
96.04
94.52
94.64
86.77
91.54
97.06
70.07
91.24
87.33
MEMTO
93.54
94.36
98.34
96.61
95.83
95.73
Anomaly criterion Unlike other time series anomaly detection models, MEMTO considers both the input and latent space to derive anomaly scores. presens the performance of MEMTO
7

---

using different anomaly detection criteria. Using only one of the two anomaly criteria, ISD or LSD, the average Fl-score is low, by 69.23% and 60.16%, respectively. In particular, they exhibit extremely low performance on SWaT,, indicating high variance in performance across datasets. In contrast, our proposed method, which uses both ISD and L.SD, consistently shows the highest performance over all datasets with relatively low variance compared to others.
Memory module In this study, we assess the capability of our novel Gated memory module in anomaly detection tasks. Table3 shows that removing the Gated memory module results in significant decrease of the average F1-score by 32.56p% and a remarkable decrease of the Fi-score specifically on SWaT less than third. To further validate the effectiveness of our approach, we also compare it with two existing memory module mechanisms (MemAE [8] and MNAD [24]) by replacing the memory module in MEMTO while using the same experimental settings. Our findings suggest that simply adding an explicit memory update process does not necessarily improve the performance, evidenced by the comparable performance between MemAE and MÑAD. However, significant performance improvements are observed when we apply our proposed memory module that integrates the update gate into the memory update process. Our approach demonstrates its superiority in anomaly detection tasks on diverse domains of datasets.
Training paradigm In order to enhance the stability of memory item updates during training, we K-means clustering initialize the values of memory items instead of random initialization. Our experimental results provide strong evidence that supports the effectiveness of the two-phase training paradígm of MEMTO. Table demonstrates that not applying K-means clustering results in an 8.4p% decrease in the average FI- -score of MEMTO compared to when it is applied. For such cases MemAE and MNAD, scores decrease by 14.7p% and 15, 15.4p% respectively, which indicates that our proposed training paradigm can be universally applicable to memory module-base models.
4.4 Discussion and analysis
0
Dacaeet
Ours
MemAA
MNAD
Figure 2: Distribution of LSD values for normal and abnormal samples.
Table 4: Statistical analysis results of LSD. Values in the table are calculated as follows: Egp Normal (g m)/ Aboormal LSD(q ,,), where qt and q denote query of normal and abnormal sample respectively.
SMD
MSL
PSM
SMAP
SWaT
MemAE
0.9674
0.9719
0.9796
1.0059
1.0321
MNAD
1.0052
1.0043
0.9960
1.0004
0,9750
Ours
0.9014
0.9484
0.9368
1.0098
0.6240
Statistical analysis of LSD We further explore the Gated memory module's superior ability to capture prototypical normal patterns data. Figure 2 demonstrates box plots depicting the distribution of LSD values for normal and abnormal samples across three memory modules: ours,

---

MemAE, and MNAD. We calculate mean LSD values for normal and abnormal samples as follows: Eup NNomall (q m) and E.z EAbnormal Lq m.), where m denotes fixed memory items. Mean LSD values for each type memory module are in Appendix D
We calculate the ratio between the mean LSD values for normal and abnormal samples in the test dataset. presents the ratio values for each dataset while using three different memory modules. Our proposed approach consistently shows smaller ratio values than those of other methods across the majority of datasets. This suggests that the items stored in the Gated memory module are more distant from queries of abnormal samples than those of normal samples, and the gap between them is greater than that in other types of memory modules. It supports our assertion that the Gated memory module better encodes normal patterns in the data than other compared methods.
MEMTO (Ours)
MEMTO (LSD)
MEMTO (ISD)
Anomaly transformer
Visualization of anomaly scores on SMD. Each graph displays the anomaly scores on the y-axis and time the x-axis. The yellow-shaded area indicates the ground truth of abnormal labels. while the blue line represents the anomaly scores predicted by the models, and the red dashed line represents the threshold values. Additional plots for different datasets are included in Appendix D
Figure 3:
Anomaly score We experiment randomly extracted time segments of length 150 from all datasets to demonstrate the effectiveness the bi-dimensional deviation-basse based detection criterion. The baselines used in this experiment include MEMTO leveraging either LSD or ISD detection criterion and the previous state-of-the-arr model, Anomaly Transformer. Figure illuttrates the anomaly score and ground truth label associated with each timestamp within segment. The results show that the bi-dimensional deviation-based criterion robustly detects anomalies compared to the baselines. Compared to Anomaly Transformer, which uses association-bast dimensional deviation-baat either latent input space aspects are removed, MEMTO shows unstable performance. While MEMTO using only LSD as a detection criterion exhibits a similar pattern in anomaly scores, scores in abnormal time points fail exceed the threshold. This suggests that considering both factors is crucial for robust performance since it amplifies the gap between normal and abnormal time points based on anomaly scores.
detection criteria, the bi- criterion shows precise detection with reduced false alarms. Furthermore,
Table 5: Time (in seconds) measured for each stage. Experiments are done on SWaT dataset.
time (sec)
First phase
K-means
Second phase
Total training
Inference
MEMTO
45.17
16.07
60.44
121.7
9,360
w/o Gated memory module w/o two-phase training paradigm
39.04
39.04
9.230
49.62
49.62
9.360
Anomaly Transformer
46.87
46.87
10.53
Computational efficiency We measure the training time with and without the use the Gated memory module and the two-phase training paradigm, respectively. The results are presented in T When training MEMTO without the Gated memory module, there reduction in both the number of parameters and training time. However, this results in 26.27%p decline in performance, as in indicated in Table Also, the two-phase training paradigm increases the training duration by approximately 2.45 times compared to its absence, though the the inference time remains unaffected.

---

Remarkably, the two-phase training paradigm enhances performance by 10.4%p, justifying its adoption even with the extended training duration.
We further compare the computational efficiency between MEMTO and the previous state-of-the-art model, Anomaly Transformer. The Anomaly Transformer computes and retains crris-assooiaatinn and prior-association per encoder layer, then averages the association discrepancies across layers. Performing this process during inference is computationally demanding, while MEMTO, requiring only dot product operations between each query and a few memory items, is simpler and results in shorter inference time. As presented in Table while MEMTO requires over twice the training time of Anomaly Transformer due to the two-phase training paradigm. it is noteworthy that when considering the critical inference time for real-world applications, MEMTO is faster by 1.17 seconds.
1.000
Flarom
SND
VSL
PMM
SMAP
0,800
memony
Figure 4: Performance across different numbers of memory items in the Gated memory module
Furthermore, given that the computational cost increases as the number of memory items within MEMTO's Gated memory module grows, we conduct experiments to analyze the optimal number of memory items. Figure 4 illustrates the relationship between the performance of MEMTO's performance and the number of memory items used. Results show that MEMTO's performance robust to the number of memory items, as the performance variance across datasets is small. Hence, we designate ten memory items as the default value after weighing performance and computational complexity. Our study highlights the effectiveness of employing a restricted number of memory items to extract prototypical features of normal patterns in time series data. Unlike computer vision, which may require thousands of memory items (8), we demonstrate that only ten memory items necessary for a task in the time series domain.
Conclusion
We introduce MEMTO, an unsupervised reconstruction-basel model for multivariate time series anomaly detection. The Gated memory module MEMTO adaptively captures the normal patterns in response to the input data, and can be trained robustly using two-phase training paradigm. Our proposed anomaly criterion, which comprehensively considers bi-dimensional space, enhances the performance of MEMTO. Extensive experiments on real-world multivariate time series bench- marks validate that our proposed model achieves ffte-of-the-ar performance compared to existing competitive models.
Limitations While two update stages in the Gated memory module and bi-dimensional deviation- criterion have led to enhanced performance, a thorough theoretical proof for its efficacy remains be established. Also, acknowledge the limitation of our omission of visually inspecting the prototypical normal patterns stored memory items. In the future, we will explore these issues further as a part of our research.
Broader impacts MEMTO is tailored for detecting anomalies in multivariate time series data and can be applied to various complex cyber-physical systems, such as smart factories, power grids, data centers, and vehicles. However, we strongly discourage its use activities related to financial crimes or other applications that could have negative societal consequences.
10

---

Acknowledgements
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea goVe))men((MSIT. (No. 202IRIA2C20977855)
References
[1] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2485-24454 2494, 2021.
[2] Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. Usad: Unsupervised anomaly detection multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3395-3404, 2020.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1xXiv:1409.0473: 2014.
[4] Ane Blázquez-Garcfa, Angel Conde, Usue Mori, and Jose A Lozano. A review on out- lier/anomaly detection time series data. ACM Computing Surveys (CSUR), 54(3):1-33, 2021.
[5] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based based local outliers. Proceedings of the 2000 ACM SIGMÓD international conference on Management of data, pages 93 93-104, 2000.
[6] Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4080-4088, 2018
[7] Yongliang Cheng, Yan Xu, Hong Zhong, and Yi Liu. Hs-ten: A semi-supervised hierarchical stacking temporal convolutional network for anomaly detection in iot. In 2019 IEEE 38th International Performance Computing and Communications Conference (IPCCC), pages 1-7. IEEE, 2019.
[8] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton den Hengel. Memorizing normality to detect anomaly: Memory- augmented deep autoencoder for unsupervised anomaly detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705-1714, 2019.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farler Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139 (5(11):139-14, 2020.
[10] Mononito Goswami, Cristian Challu, Laurent Callot, Lenon Minories, and Andrey Kan. Unsu- pervised model selection for time-series anomaly detection. arXiv preprint arXiv:2210.01078. 2022.
[11] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5400 2014.
[12] Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 111 16, pages 312-329. Springer, 2020.
[13) Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using Istms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 387-395 2018.
[14] tukasz Kaiser, Ofrr Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint arXin:1703.03122, 2017. 2017.

---

[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[16] Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning. PMLR, 2016.
pages
[17] Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In -MCANN 2019: Text and Time Series: 28th
Artificial Neural Networks and Machine International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV, pages 703-716. Springer, 2019.
[18] Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In Artificial Neural Networks and Machine
-nCnnnng-CCANN--- 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV, pages 703-716. Springer, 2019.
[19] Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hicrarchical inter-metric and temporal embedding. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 3220 3230, 2021.
[20] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.
[21] Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13588-13597, 2021.
[22] Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv: 1607:0007000440 2016.
[23] Dachyung Park, Yuuna Hoshi, and Charles C Kemp. A multimodal anomaly detector for robot- assisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(4):15441155)) 2018
[24] Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14372- 14381, 2020.
[25] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference machine learning, pages 4393-4402. PMLR, 2018.
[26] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference machine learning, pages 4393 4402-4402. PMLR, 2018.
[27] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
(28) Bernhard Schölkopf, John C Platt, John Shawe-Taylon Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-
1471, 2001.
[29] Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. Advances Neural Information Processing Systems, 33: 13016-13026, 2020.
12

---

[30] Lifeng Shen, Zhongzhong Yu, Qianli Ma. and James T Kwok. Time series anomaly detection with multiresolution ensemble decoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9567-9575, 2021.
[31] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & duta mining, pages 2828-2833 2019.
[32] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2828 2837-2877722 2019.
[33]
Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pages 2828 2837, 2019.
[34] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
[35] David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54: 45-66, 2004.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[37] Xixuan Wang, Dechang Pi, Xiangyan Zhang, Hao Liu, and Chang Guo. Variational transformer- based anomaly detection approach for multivariate time series. Measurement, 191:110791, 2022.
[38] Jason Weston, Sumit Chopra, and Antoine Bordes. arXiv:1410.3916. 2014.
Memory networks. arXiv preprint
[39] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning, pages 2397-2406. PMLR, 2016.
[40] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:21 2021.
[41] Takehisa Yairi, Naoya Takeishi, Tetsuo Oda, Yuta Nakajima, Naoki Nishimura, and Noboru Takata. A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction. IEEE Transactions on Aerospace and Electronic Systems, 53(3):1384 1101-144141111- 2017.
[42] Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI, volume 2019, pages 4433 4439, 2019.
[43] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[44] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 5802-5810, 5810, 2019.
[45] Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations, 2018.
13

---

A Training details
We use memory items for our MEMTO model, corresponding the number of clusters in our K- X-means clustering. To determine anomalies, we set the threshold the top-p% of the combined results of the anomaly scores from both the training and validation data, with specified values for each dataset outlined in Table 6 following [40]. We set À in the objective function to 0.01, use Adam optimizer [15] with a learning rate of 5e-5, and employ early stopping with the patience of 10 epochs against the validation loss during training. Our experiments are conducted using the Pytorch framework on four NVIDIA GTX 1080 12GB GPUs. Furthermore, during the execution of our experiment, make partial references to the code of [40].
A.1 Hyperparameter settings
Important hyperparameters MEMTO were determined through grid search, while others were set to commonly used default values based on empirical observations. We performed a grid search to determine the values of each hyperparameter within the following range:
À e [lc+0, 5e-1, le-1, 5e-2, 1e-2, Se-3, le-3)
Ir [le-4, 3e-4, 5e-4, 1e-5, 5e-5)
E (0.1, 0.3. 0.5, 0.7, 0.9)
.M E (5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100)
where ir, T, and M denote the learning rate, the temperature in the softmax function, and the number of clusters, respectively. Since set the centroids of clusters as memory items, the number of memory items and that of clusters are the same. We set the optimal hyperparameters as follows: le-2, ir 5e-5, 0.1, and M 10. All experiments in this paper conducted using the same hyperparameters regardless of the dataset.
A.2 Dataset
Table 6: Details in five benchmarks. The number of samples in the training, validation, and test sets represented in the columns labeled 'Train,' 'Valid,' and "Test,' respectively. The "p% column indicates the anomaly ratio used in the experiment. The 'Dim' column shows the dimension size of data for each dataset.
Train
Valid
Test
p( )
Dim
SMD
566,724
141,681
708,420
0.5
38
MSL
46,653
11,664
73,729
1.0
PSM 105,984
26,497
87,841
1.0
26
SMAP 108,146
27,037
427,617
SWaT
396,000
99,000
449,919
0.1
53
Table Table the statistical details of datasets used in experiments. We obtained SWaT by submitting a request through attps:://ttee sutd. edu. wedu.sg/ittee sg/sirusttttt....r labs__abaa__ ltt-laaabaa_asseaeeeeaaaaaaaaaae datasets/
14

---

B Algorithm for MEMTO
Algorithm 2 Proposed Method MEMTO
Input X E R/xx; input sub-series
Training params fo: encoder, fa: decoder, U., Wi E RCC, linear projection matrices \\ feed-forward encoder, q" RLxC
q² fo (X*)
2: r sof aoftmar (m (g))
\\ Gated memory update start, TRM v Rxx
3: d sigmoid (mUv ( We)
\1 t RMxc
4: m (1 - +) oom + No (1"g")
Gated memory update end
5: d = sof otmax qP (m))
|1 Query update start, w E LxL
6. q w" m
q' E RLxc 7: q concat (1 (") dim 1)
11 Query update end, 9" R2x2C
8: X La (4*)
\\ feed -forward decoder, X E RLxn
9: return X
reconstructed sub-series
Algorithm provides an overall mechanism for our model. demonstrates the matrix operation version of the forward process when single input sub-series X is fed to MEMTO.
C
Additional experiments
Table 7: The ablation results (F1-score) in anomaly criterion and objective function. Lyee and Lentr signify Reconstruction Loss and Entropy Loss, respectively.
Loss
Anomaly Criterion
F1-score
ISD
LSD
SMD
MSL
PSM
SMAP
SWaT
avg.
79.63
86.23
82.15
71.18
31.29
70.09
Lrec
69.73
72.63
93.07
67.69
82.50
77.12
93.19
92.66
98.05
96.48
93.34
94.74
X
75.71
88.39
87.47
69.28
79.28
80.02
Lentr
12.53
84.34
76.49
68.17
83.52
65.01
88.43
93.40
97.97
96.22
92.77
93.75
X
77.54
87.22
79.25
70.99
31.17
69.23
Lyve + ALentr
72.78
80.33
80.15
67.55
0.00
60.16
93.54
94.36
98.34
96.61
95.83
95.73
Table
We report mean and standard deviation over 10 runs for A.T (Anomaly Transformer) and MEMTO, respectively, We conduct t-test (p 0.05) to indicate statistical signficance.
SMD
MSL
SMAP
SWaT
PSM
A.T
mean
91.34
92.60
95.83
92.86
97.57
std
0.6942
1.103
1.088
0.9455
0.1230
mean 93.05
94.07
96.49
95.23
98.15
MEMTO
std
0.3762
0.5185
0.07987
1.016
0.1666
p-value
0.0000
0.0301
0.0360
0.0002
0.0000
C.1 Objective function and anomaly criterion
In this experiment, we investigate the impact of loss terms, specifically the reconstruction loss Tirec and the entropy loss Lentr, on the performance of our proposed framework, MEMTO, We remove
15

---

one of the two terms one by one from the objective function and evaluate the resulting performance. Table 7 demonstrates the significance of incorporating both Lyee and Lentr terms the objective function. Applying bi-dimensional deviation-basee criterion to the MEMTO variants that only use Treee or Lentr as the loss function shows competitive performance compared to ours in terms of average F1-score. This demonstrates the robustness of MEMTO to loss terms. Additionally, both cases show significant performance drop when using only ISD or LSD as the anomaly criterion, emphasizing the importance of combining ISD and LSD for achieving optimal performance.
C.2 Statistical significance test
We also perform statistical tests comparing our results the latest state-of-the-are art model, Anomaly Transformer. We conduct t-test to prove a significant performance difference between MEMTO and Anomaly Transformer. The results in Table show a p-value less than 0.05 across all datasets, confirming a significant performance difference between the two models.
C.3 Number of decoder layers
1.00
SMD
SHIP
0.58
313333
302
af decoder lnyers
Figure 5: Fl-score and number of parameters, according the number of decoder layers. The right y-axis represents the values of the blue line graph in million units, while the left y-axis represents the values of the bar graph.
Figure provides the performance of MEMTO under different numbers of decoder layers. As shown Figure 5 decoder that is too shallow (e.g., decoder with single layer) performs worse because it lacks sufficient capacity to reconstruct the input data accurately. On the other hand, if the decoder is too large (e.g., decoder with ten layers), it can become overly expressive and reconstruct even anomalies regardless of the encoding ability of the encoder. Therefore, it can lead to over-generalization problem, which can ultimately decrease the performance of anomaly detection by reconstructing anomalies too accurately. Furthermore, larger decoder layer with more parameters can increase computational and memory costs. We empirically find that considering the balance between performance and resource cost, a decoder with two layers is most suitable for anomaly detection tasks presented our paper.
D
Additional details for discussion
D.1
LSD values
Ishows mean LSD values of normal and abnormal samples across various domains datasets while using different memory module mechanisms. In most datasets, our proposed Gated memory module consistently exhibits a lower mean LSD value for normal samples than for abnormal samples. Furthermore, the relative difference between these values is more significant than other memory module mechanisms. These results demonstrate the efficacy of our memory module mechanism in capturing prototypical features of normal patterns in data.
16

---

Table 9:
The mean LSD values corresponding to test data.
MSL
PSM
SMAP
SWaT
Normal
Abnormal
Normal
Abnormal
Normal
Abnoonal
Normal
Abnormal
Normal
Abmormal
MemAE 814.7836 842.2023
622.5195
6400.4954
766.2473 782. 1895 710,4929
706.3115 795.7227 770.9069
MNAD 259.3635 258.0175
791
788, 2654
292.3340
293.4836
301.3480
301.2153
303.1933
310.9818
Ours 297.5692
330.1162
249.8632
263,4332
340.7552
363.7520
237.0070
234.7110
450.0926
721.3093
D.2 Anomaly score
PSM
SMAP
SWaT
Ours
ISD
Anomaly transformer
Figure 6: Visualization of anomaly scores for MSL, PSM, SMAP, and SWaT datasets.
Figure 6 visually represents the anomaly scores for benchmark datasets not discussed in Section 4.4 We randomly sampled data of length 150 from MSI., PSM, SMAP, and SWaTT test datasets and plotted the anomaly scores for each segment. Compared to other baselines, proposed method consistently detects anomalies precisely with low false positive rate from the perspective of the point adjustment method.
17


Reviews:
[{"review_id": "Gp2apOvVKM", "reviewer": "Reviewer_pZZS", "summary": "The authors of this paper focused on tackling the problem of over-generalization in reconstruction-based deep models and made a contribution to the field of multivariate time series anomaly detection. The main challenge they encountered was dealing with complex dependencies and inter-variable correlations within the data. To address this, they proposed a novel gated memory module approach. This module learns how much each memory item should be updated based on the input data, effectively capturing the prototypical features of normal patterns in the data. By doing so, it aims to overcome the inability of existing models to capture dynamic nonlinear temporal dependencies and complex inter-variable correlations.", "questions": "• How does the proposed method compare to existing approaches in terms of computational efficiency? Were any considerations or optimizations implemented to ensure its feasibility in real- time or resource-constrained environments?\n• What led the authors to choose K-means clustering for initializing memory items? Is K-means considered the most effective approach for this task, particularly in terms of computational efficiency?\n• How does the model behavior change if sub-series are generated with the help of overlapping sliding window?\n• Since K-means clustering creates prototypes based on the seen data items, how does the model handle unseen cases? If the proposed MEMTO couldn't consider the degree of new information being injected into existing memory items, what is the contribution of the MEMTO compared to the MNAD?", "limitations": "Please refer to the Weaknesses and Questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "• The authors explained the importance of multivariate time series anomaly detection with the proper real-world examples\n• Comprehensive comparison of their approach with traditional methods (OC-SVM, Deep-SVDD, LSTM-VAE, Anomaly Transformer etc.)\n• Thorough explanation of the training hyper-parameters, dataset setting, algorithms and code\n• Achieved state-of-the-art results on five real-world benchmark datasets", "weaknesses": "• Lack of theoretical proof: two-phase training approach and bi-dimensional deviation-based criterion need a detailed theoretical proof\n• Limited discussion of computational efficiency: how gated memory module and two-phase train- ing affect the training time\n• I couldn’t find any explicit explanations that how the LSD and ISD help the training. In the descriptions, LSD and ISD only appears in Table2 (and the sec3.4). If LSD+ ISD just help the anomaly score but doesn't help the training, this should be clearly justified. \n• Need more quantitative experiment about the improvement from gated memory seems limit. (Table 3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your dedicated review of our work. Your critical feedback have helped us to extend and refine the paper. We provide a detailed response to your comments.\n\n$\\textbf{Weakness}$\n\n- $\\textbf{Lack of theoretical proof}$: We agree that we lack theoretical proof, and we have mentioned this in Conclusion(line 286). We plan to address this as a topic of our future research.\n- $\\textbf{Limited discussion of computational efficiency}$: As you suggested, we measured the training time with and without using Gated memory module and the two-phase training paradigm, respectively. The results are presented in Table 4 of the PDF, in the section on global response. Training MEMTO without the Gated memory module reduces both the number of parameters and training time, but it leads to decreased performance by 26.27%p. Also, while the two-phase training paradigm requires approximately 2.45 times more time for training compared to its absence, the inference time remains the same. As shown in Table 3 in our paper, using a two-phase training paradigm improves the performance by 10.4%p, leading us to apply it at the cost of increase in training time.\n- $\\textbf{Explanation of LSD and ISD}$: As you can see in Section 3.4, LSD and ISD are used in our proposed anomaly score computation scheme, which reflects how anomalous a given timestamp is. As conventionally done in anomaly detection task, anomaly score is only for inference and is completely irrelevant to the training process. Table 2 in our paper represents an ablation study where we experimented with varying methods of calculating the anomaly score while performing inference on the test dataset with MEMTO. \n- $\\textbf{Quantitative experiment of Gated memory module}$: The 'Statistical analysis of LSD' experiment conducted in Section 4.4 is related to the Gated memory module (See line 251). To verify if the Gated memory module effectively captures the prototypical normal patterns of the data, we evaluated the distances from the memory items within the Gated memory module to queries of normal timestamps and to queries of abnormal timestamps, respectively.\n\n$\\textbf{Questions}$\n\n- $\\textbf{Computational efficiency of MEMTO / Considerations for real-world applications}$: We compare the computational efficiency between MEMTO and the previous state-of-the-art model, Anomaly Transformer. The Anomaly Transformer computes and retains series-association and prior-association per encoder layer, then averages the association discrepancies across layers. Performing this process during inference is computationally demanding, while MEMTO, requiring only dot product operations between each query and a few memory items, is simpler and results in shorter inference time. We conducted an additional experiment to verify the effectiveness of MEMTO compared to the previous state-of-the-art model in the aspect of inference time. The results can be seen in Table 4 of the PDF, in the section on global response. The resources used to train MEMTO only require four GTX 1080 GPUs, and the training time took about 2 minutes on SWaT dataset, which can be considered as low computation resources and short training time compared to recent deep learning models. To ensure fast and effective operation in real world applications, we utilized only ten memory items within the Gated memory module and deployed just two MLP layers in the decoder, which makes the little number of parameters possible.\n- $\\textbf{Reasons to use K-means clustering for initializing memory items}$: Since our objective for initializing memory items was to quickly acquire rough normal prototypical patterns rather than precisely, we chose K-means clustering, which has relatively low computational cost. Given $n$ as the number of data points, the time complexity of K-means clustering is $O(n)$, whereas that of Hierarchical clustering or Expectation Maximization clustering is both $O(n^3)$. Considering computational efficiency, we ultimately opted for K-means clustering.\n- $\\textbf{Overlapping sliding window vs non-overlapping sliding window}$: We did not employ an overlapping sliding window when constructing sub-series due to computational inefficiency during the model's training and inference phase. Using an overlapping sliding window increases the size of the training dataset, which in turn augments the computational cost required to train the model. Also, when performing inference on the test dataset, the computational cost increases as the anomaly score for overlapping timestamps needs to be calculated multiple times and averaged. \nWe observe the following changes in performance when using an overlapping sliding window (see Table 3 of the PDF, in the section on global response).\n- $\\textbf{How does the model handle unseen cases during memory module initialization with K-means clustering?}$: For K-means clustering, we randomly sampled 10% of the training data under the assumption of iid (independent and identically distributed). Given this assumption, we believe the distribution of the remaining 90% of the training dataset, not used for clustering, is similar to that of the sampled 10%. \n- $\\textbf{The contribution of MEMTO compared to MNAD}$: We agree that the structure of the Gated memory module in MEMTO is similar to the memory module in MNAD, if we set aside the ability to control the degree of injecting new information into existing memory items. However, MEMTO simplifies the anomaly score computation process with bi-dimensional deviation-based detection criterion, which consists of multiplication of normalized LSD and ISD, where LSD functions as weights for amplifying the normal-abnormal gap in ISD. More detailed explanations regarding this question are elaborated on global response. "}, {"review_id": "tukNW7jz3t", "reviewer": "Reviewer_C8uM", "summary": "The paper used memory network to capture frequently present normal pattern in date set. While training, the memory are built in latent space in the autoencoder framework.  Along with reconstruction error, the distance between a representation with the closest memory unit is used to calculate anomaly score. If a data does not have representation close to memory then that is an anomaly. \n", "questions": "Most cases  the F1 score for the proposed method is close to the A.T benchmark. Did you do a statistical significance test?\nDo you run your experiment for different random train text split? In that case how is the  variance of accuracy ?\nThe F1 score for only LSD and only ISD is quite low. Did you note down if there are more FP or FN in both cases?\nHow to decide the required number of memory unit ?\n", "limitations": "Number of majority patterm in normal part need to be known before which is not possible during unsupervised task. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Idea of keeping more than one memory for common patterns from normal data is Novel and makes more sense than existing methods of keeping memory of anomalies.  Experimental results were produced on a large number  of data sets.  ", "weaknesses": "Literature surveys missed graph based anomaly detection techniques for multivariate time series like Deng, Ailin, and Bryan Hooi. \"Graph neural network-based anomaly detection in multivariate time series.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 5, pp. 4027-4035. 2021.\n\n\nIt is required to have an ablation study in the main paper to bring our significance of memory gate and Kmeans clustering. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful reading of our paper, and your feedback. Your insightful feedback has guided us in expanding and improving our paper.\n\n$\\textbf{Weaknesses}$\n\n$\\textbf{Literature surveys missed graph based anomaly detection techniques for multivariate time series}$\n\nWe will also include graph-based anomaly detection methods for multivariate time series such as the paper (1), (2), and (3) in the literature review.\n\n(1)   Deng, Ailin, and Bryan Hooi. \"Graph neural network-based anomaly detection in multivariate time series.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 5, pp. 4027-4035. 2021.\n\n(2)   Zhang, Weiqi, Chen Zhang, and Fugee Tsung. \"Grelen: Multivariate time series anomaly detection from the perspective of graph relational learning.\" Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. Vol. 7. 2022.\n\n(3)   Zhao, Hang, et al. \"Multivariate time-series anomaly detection via graph attention network.\" 2020 IEEE International Conference on Data Mining (ICDM). IEEE, 2020.\n\n$\\textbf{It is required to have an ablation study in the main paper to bring our significance of memory gate and Kmeans clustering.}$\nThe key aspect of the Gated memory module is the update gate, which is the most significant point of differentiation compared to MNAD and MemAE. Therefore, to assess the capability of our proposed Gated memory module mechanism, we conduct an ablation study comparing MEMTO with other memory modules in Section 4.3 (line 232-242) and present the following results in Table 3. Moreover, as for ablation study on memory module initialization with K-means clustering, experimental results in Table 3 provide strong evidence of its effectiveness (see section 4.3 Ablation study line 243-249).\n\n\n$\\textbf{Questions}$\n\n$\\textbf{Most cases the F1 score for the proposed method is close to the A.T benchmark. Did you do a statistical significance test?}$\nWe had not performed statistical significance tests in the manuscript since it was not generally done in the previous works. However, as you requested, we conduct additional experiments with varying random seeds to verify the mean and standard deviation of the performance  (see Table 1 of the PDF, in the section on global response). We also perform statistical tests comparing our results to the latest state-of-the-art model, Anomaly Transformer. We conduct a t-test to prove a significant performance difference between MEMTO and Anomaly Transformer. The results show a p-value less than 0.05 across all datasets, confirming a significant performance difference between the two models (see Table 1 of the PDF, in the section on global response).\n\n$\\textbf{Do you run your experiment for different random train text split? In that case how is the variance of accuracy?}$\nWe did not run the experiment for different random train-test splits. The reason is that the train dataset and test dataset from the benchmarks are pre-split and designated when downloaded from the archive, where the anomaly labels exist only in the test dataset. \n\n$\\textbf{The F1 score for only LSD and only ISD is quite low. Did you note down if there are more FP or FN in both cases?}$\nIn Table 2 in our paper, we provided F1 score as the only performance metric. Nonetheless, at our experiments, we also record precision and recall values. The results we obtained can be seen in Table 2 of the PDF, in the section on global response. Both precision and recall values are lower in the \"Only LSD\" or \"Only ISD\" cases compared to the \"LSD+ISD\" case. This indicates that in both cases, we have more false positives (FP) and false negatives (FN) compared to the combined approach (LSD+ISD).\n\n$\\textbf{How to decide the required number of memory unit?}$\nIn the Supplementary material, Appendix C.2 (line 478), there is an experiment result conducted on the number of memory items. Through this experiment, we discovered that MEMTO is robust to the number of memory items and works effectively even with a small number of items. Considering computational efficiency, we chose to use only 10 memory items as default value in our experiments.\n\n\n\n\n$\\textbf{Limitations}$\n\n$\\textbf{Number of majority patterm in normal part need to be known before which is not possible during unsupervised task.}$\nThe number of memory items, identical to the number of prototypical patterns of normal data, is not a known prior value but a hyperparameter. We experimented by varying this hyperparameter in Appendix C.2 and found that MEMTO shows robust performance regardless of its value."}, {"review_id": "hrvKPLnjiH", "reviewer": "Reviewer_EFum", "summary": "The authors propose a memory-guided transformer with a reconstruction paradigm for multivariate time series anomaly detection. The time-series encoder, memory parameters of normal patterns, and projection heads are updated during training time in a two-step fashion. On test time, input queries are projected on the normal patterns learned base with a learned sparse linear projection. As a result, reconstructed anomalies tend to look like normal samples, amplifying the reconstruction loss and enabling detection. Experiments are run over common multivariate time-series datasets following previous works practices.", "questions": "Please see the weaknesses section.", "limitations": "The authors have discussed potential limitations and shortcomings of their algorithm and no special negative societal impact exclusive of their work is evident.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The proposed gated memory module seems simple, original, and efficient for detecting anomalies on multivariate time series based on transformer encodings and input sample reconstruction. \n\n* The authors provide thorough ablation studies and some insights into the importance of each part of their algorithm.\n\n* The manuscript is overall easy to follow and well-written. Code is provided.\n\n## Suggestions\n\nThe typical memory size is quite small, which I see as a positive point of this method. So, I suggest this could be emphasized in the main manuscript. I read the entire paper wondering what the memory size was and whether the trade-off memory-performance was good, just to be positively surprised in the appendix.", "weaknesses": "## Regarding novelty\n\n1. Overall this work seems to be an incremental modification of [8], [24], and [40]. In particular, contribution #3 (line 77) does not seem very novel. I would like the authors to please discuss Equation (11) in this paper versus (6) in [40] in more detail.\n2. How does the memory update stage is different from other memory gates, such as GRU, MRU, etc?\n\n## Regarding state-of-the-art claim\n\n1. Why not compare to MNDA [24], memAE [8], USAD[2] in Table 1? Why are A.T. numbers in Table 1 different from [40]? I would suggest adding error bars at least to these methods and yours to make the comparison fair. I know this was not done in these previous works, but claiming state-of-the-art without comparing confidence intervals in an empirical field is not a good practice.\n\n## Regarding methodology\n\n4. Not clear why not just have a memory on the input space and retrieve the closest samples instead of the proposed mechanism on the latent space?\n5. Why is it so sensitive to k-means initialization? Is it computed with enough iterations of the K-means algorithm?\n6. How does algorithm 1 outlines the two-phase training paradigm (line 175)? The first and second phase seems to be omitted.\n\n## Other less important questions\n\n7. How are the thresholds computed in figure 3?\n8. What is the computation overhead when compared to A.T.? MEMTO seems simpler computationally. Could the authors discuss this?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for your detailed evaluation and insight on our research paper. We will make sure to incorporate the parts that you suggested for clarity and reflect their feedback on paper.\n\n$\\textbf{Suggestion}$\nAs you suggested, we will emphasize in the main manuscript that our proposed MEMTO is robust to the number of memory items and even performs well with small memory size (low resource setting), in our revised version. \n\n$\\textbf{Regarding novelty}$\n1. Drawing inspiration from [8], [24], and [40] like you mentioned, MEMTO stands out from prior works with the following distinctive features. Due to the limit on response length, we answered the follwoing question on the global response. \n\n2. As you mentioned, both our proposed Gated memory module and other memory gates, such as the GRU and MRU, have the ability to control how much new information is injected. However, other memory gates, such as GRU and MRU, are often inadequately compartmentalized to retain precise knowledge from the past, as the information gets compressed into dense vectors. Therefore, it is impossible to have them compartmentalized and memorize diverse prototypical features of normal patterns in the data. In contrast, Gated memory update stage in our Gated memory module applies attention-based operation (query-conditioned memory attention) while writing new information into each memory item. Since each memory item can retrieve information from only relevant queries, this mechanism enables each memory item to be compartmentalized and retain prototypical features of normal patterns in the data.\n\n$\\textbf{Regarding state-of-the-art claim}$\n\n3. The reason we did not add MNAD [24] and memAE [8] as baselines in Table 1 in our paper is that they are anomaly detection models for the computer vision domain. The performance of MNAD and MemAE is shown in Table3 in our paper as the result of an ablation study on the memory module. In the experiment, only the memory module is modified while keeping all other experimental conditions, including the pipeline and architecture, identical to those used in MEMTO. Therefore, they cannot be considered as independent models for multivariate time series anomaly detection tasks, as the only difference lies in the memory module itself. Also, while USAD computes the anomaly score on a window level to determine if each window is anomalous, we compute the anomaly score on a timestamp level to ascertain the anomalousness of each time point. Therefore, we only used models that compute the anomaly score on a timestamp level as our baselines. Moreover, we agree that adding error bars is a valid way to properly demonstrate the state-of-the-art results and provide a more comprehensive understanding of the experimental outcomes. We provide you with some additional experiments comparing with the latest state-of-the-art model, Anomaly Transformer (A.T), on five real-world benchmark datasets.The results can be seen in Table 1 of the PDF, in the section on global response.\n\n\n$\\textbf{Regarding methodology}$\n\n4. The latent representations obtained from the transformer encoder for each timestamp consider the temporal dependency information between timestamps. On the other hand, the raw timestamp values in the input space do not inherently account for temporal dependencies with other timestamps, making them unsuitable for accurately learning the normal prototypical pattern of the time series. As you pointed out, not explicitly mentioning this in the main script could lead to ambiguity. We will clearly note this in our revised paper.\n\n5. As mentioned in line 243~247, we empirically observed that updating memory items incrementally can result in training instability if the items are initialized randomly. Memory module initialization with K-means clustering enables injecting inductive bias of normal prototypical patterns into memory items. By providing the memory items with informative initial values (rough prototypical feature of the normal pattern), the model starts with a better understanding of the normal pattern at the second phase of two-phase training paradigm, which can lead to more stable and efficient learning compared to simply using random initialization. When experimented with multiple random seeds(using a two-phase training paradigm), MEMTO consistently demonstrates stable and robust detection performance. This result indicates that the iteration value, which is the hyper-parameter of the K-means algorithm, is large enough for our model to have reliable performance.\n\n6. Thank you for careful checking of our paper. As you rightly pointed out, we will correct it as ‘Algorithm 1 outlines the memory module initialization with K-means clustering’ in our revised version.\n\n$\\textbf{Other less important questions}$\n\n7. In the case of threshold selection, you can refer to Appendix A Training details (line 440~442).\n\n8. The computation analysis between MEMTO and Anomaly Transformer is explained in the response on the first question (Regarding novelty). To quantitatively verify the computational efficiency in MEMTO, we conduct additional experiments for inference time of MEMTO compared to Anomaly Transformer on five real-world benchmark datasets. Results in Table 4 of the PDF in the section on global response, show that whereas training time for MEMTO is longer than that of Anomaly Transformer, its inference time in real time detection process is shorter.\nSince we agree that computation overhead is crucial measurement for real-world application, we will include this content in the revised version.\n"}, {"review_id": "bi7w61MogF", "reviewer": "Reviewer_UfyT", "summary": "This paper presents a reconstruction-based method for multivariate time series anomaly detection. It is a memory-guided Transformer which contains the gated memory module. Because of the training instability in updating memory items incrementally when the items are\ninitialized randomly, the authors propose a two-phase training paradigm to ensure stable training. The method calculates anomaly scores considering the input and latent space. The method achieves state-of-the-art results on five benchmark datasets.", "questions": "Please address the points mentioned in Weakness.", "limitations": "Yes, the authors have addressed the potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors make a good explorations on leveraging memory module to multivariate time series anomaly detection task, and the proposed method can be effective solutions for the given task.\n2. This paper contains enough experiment results and ablation study on five datasets to support its claims.\n3. The proposed methodology is well presented, and most of the paper is well written.", "weaknesses": "1. The paper writing can still be improved. For example, the grammar error on line 88 \"there exist video representation learning\" -> \"there exists video representation learning\". Typo on line 234 \"32.56p%\". The citation in related work can be extended with author's name + 'et al.' rather than just a number. \n2. The font size in the figures (in introduction and experiment) is pretty small.\n3. I saw that you split the training data into 80% for training and 20% for validation. Do other methods for comparison in the experiments share the same data split(ratio and files)？ How many times you repeated your experiments? Since I saw the results you reproduced for Anomaly Transformer are quite different from their own paper, is it fair to list all these methods to compare with your model?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your meticulous review and feedback on our paper. Your perceptive observations have steered us towards enhancing and refining our work.\n\n\n$\\textbf{Weaknesses}$\n\n1. We appreciate your keen attention to detail and will proceed to correct the grammar error on line 88, the typo on line 234, and the citation format as you pointed out.\n\n2. We agree with your observation regarding the small font size in our figures. We will ensure to increase it for better readability as per your suggestion.\n\n3. In our paper, all the methods including MEMTO employed the same data division ratio of 80% for training and 20% for validation. The main experiment in Table 1 in our paper was conducted once, but we carried out additional trials to calculate statistical measures such as averages and standard deviations, yielding robust scores over multiple runs (see Table 1 of the PDF, in the section on global response). Moreover, we noticed a discrepancy between the performance of the Anomaly Transformer reported in its original paper and our findings. While the original paper stated that they used 80% of the data for training and 20% for validation, a review of the official source code revealed a different setting. They used the entirety of the training data (100%) for training, inclusive of the 20% initially set aside for validation. Since this deviates from the standard measurement procedure, we decided to adjust the data settings for the Anomaly Transformer to align with that of MEMTO, by excluding the validation data from training. This adjustment ensures a fair comparison of performances.\n"}]

Author rebuttal:
We are very grateful to all the reviewers for your careful reading of our paper and helpful feedback. We will make sure to incorporate the parts that you suggested for clarity and reflect your feedback on the revised paper. We have compiled the results of additional experiments related to the reviewers' questions and comments in the attached PDF, and kindly ask you to review them. The following responses address common questions raised by several reviewers.

$\textbf{Comparing MEMTO with the Anomaly Transformer [40] in the aspect of anomaly criterion}$

Eq (11) in our paper and eq (6) in [40] are the equations for calculating anomaly scores for real-time detection process (inference). Since both belong to reconstruction-based models, they use reconstruction error as the main criterion for distinguishing between normal and abnormal samples. However, the LSD in MEMTO and the Association Discrepancy in Anomaly Transformer represent fundamentally different concepts. LSD, the distance between each time point’s query and its nearest memory item in latent space, employs the memory items’ nature of storing prototypical features of normal patterns in the data. However, Association Discrepancy, quantified by the distance between each time point’s prior-association and its series-association, utilizes the inherent normal-abnormal distinguishability of the association distribution. The Anomaly Transformer calculates and stores series-association and prior-association in memory for each encoder layer, and then averages the association discrepancies across multiple layers. Performing this process during inference is computationally demanding, while MEMTO, requiring only dot product operations between each query and a few memory items, is simpler, resulting in shorter inference time.

$\textbf{Comparing MEMTO with MemAE [8] and MNAD [24] in the aspect of overall architecture}$

Previous works [8] and [24] are both for the anomaly detection task in the computer vision domain. We proposed MEMTO, an algorithm that takes the characteristics of time series data into account and appropriately applies a memory module to the anomaly detection task in the time series domain for the first time. You can refer to Section 2 for comparison with [8] and [24] in the aspect of memory module mechanism. In particular, our proposed Gated memory module’s main distinction from [8] and [24] is written in line 93~97. Also, to further alleviate the over-generalization problem in reconstruction-based models, we use a weak decoder which requires a small number of parameters. While existing models, including both [8] and [24], typically use a symmetrical encoder-decoder architecture, we deviate from this norm by adopting an asymmetric model structure with a weak decoder. Leveraging a decoder with less parameters also enables our MEMTO to be much more computationally efficient.


Meta-review: This paper addresses a method for overcoming the over-generalization which typically occurs in the reconstruction-based methods for anomaly detection. The approach taken in this paper is to employ the memory module where each memory is updated in response to the input data in such a way that the over-generalization is alleviated. Two main contributions were claimed, including the gated memory module and the two-phase training with k-means clustering for initialization. In fact, there are a few memory-augmented methods, but this work might be the first attempt for applying such an idea to time series, which is valuable. Most of reviewers agree that the method in this paper is sound. Most of concerns raised by reviewers were well responded. However, one reviewer had some concerns remained even after the rebuttal. He/she was not convinced with the k-means clustering for initialization. I think using k-means to initialize prototype vectors is a minor issue, which anyone can think of as a baseline technique.  If authors claims that it is the one of main contributions in this paper, I doubt it. Despite some arguments, the paper has some interesting contributions that can be presented at the conference.
Decision: Accept (poster)