SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table1 shows the evaluation results on five multivariate time series anomaly detection tasks. Overall, models using Transformers such as MEMTO and Anomaly Transformer consistently show high performance with an F1-score of over  $90\%$  on all datasets. However, MEMTO substantially improves the average F1-score on benchmarks compared to the previous state-of-the-art model, Anomaly Transformer, from  $93.62\%$  to  $95.74\%$ . MEMTO effectively detects anomalies in complex time series data by adjusting to diverse normal patterns present in the data and creating prototypes of these patterns using a data-driven approach, thus achieving new state-of-the-art results.
In this section, we comprehensively analyze the proposed model, MEMTO, through a series of ablation studies on three key components: the anomaly criterion, the memory module, and the training paradigm. Appendix C shows a more detailed analysis of the ablation studies.
Table 2: Effectiveness of ISD and LSD.
Anomaly Criterion	F1-score
ISD	LSD	SMD	MSL	PSM	SMAP	SWaT	avg.
✓	×	77.54	87.22	79.25	70.99	31.17	69.23
×	✓	72.78	80.33	80.15	67.55	0.00	60.16
✓	✓	93.54	94.36	98.34	96.61	95.83	95.73
Table 3: Ablation results of memory module and training paradigm. 'MM' denotes memory module.
F1-score
K-means	SMD	MSL	PSM	SMAP	SWaT	avg.
w/o MM	-	77.40	87.47	79.83	71.20	31.42	69.46
MemAE	×	89.12	90.42	86.70	70.16	61.12	79.50
✓	91.39	92.50	98.00	96.13	93.06	94.21
MNAD	×	65.56	85.41	93.43	69.45	82.31	79.23
✓	92.39	92.13	98.13	96.04	94.52	94.64
MEMTO	×	86.77	91.54	97.06	70.07	91.24	87.33
✓	93.54	94.36	98.34	96.61	95.83	95.73
Anomaly criterion Unlike other time series anomaly detection models, MEMTO considers both the input and latent space to derive anomaly scores. Table2 presents the performance of MEMTO
using different anomaly detection criteria. Using only one of the two anomaly criteria, ISD or LSD, the average F1-score is low, by  $69.23\%$  and  $60.16\%$ , respectively. In particular, they exhibit extremely low performance on SWaT, indicating a high variance in performance across datasets. In contrast, our proposed method, which uses both ISD and LSD, consistently shows the highest performance over all datasets with relatively low variance compared to others.
Memory module In this study, we assess the capability of our novel Gated memory module in anomaly detection tasks. Table3 shows that removing the Gated memory module results in a significant decrease of the average F1-score by  $32.56\%$  and a remarkable decrease of the F1-score specifically on SWaT to less than a third. To further validate the effectiveness of our approach, we also compare it with two existing memory module mechanisms (MemAE [8] and MNAD [24]) by replacing the memory module in MEMTO while using the same experimental settings. Our findings suggest that simply adding an explicit memory update process does not necessarily improve the performance, as evidenced by the comparable performance between MemAE and MNAD. However, significant performance improvements are observed when we apply our proposed memory module that integrates the update gate into the memory update process. Our approach demonstrates its superiority in anomaly detection tasks on diverse domains of datasets.
Training paradigm In order to enhance the stability of memory item updates during training, we use  $K$ -means clustering to initialize the values of memory items instead of random initialization. Our experimental results provide strong evidence that supports the effectiveness of the two-phase training paradigm of MEMTO. Table 3 demonstrates that not applying  $K$ -means clustering results in an  $8.4\mathrm{p}\%$  decrease in the average F1-score of MEMTO compared to when it is applied. For such cases in MemAE and MNAD, scores decrease by  $14.7\mathrm{p}\%$  and  $15.4\mathrm{p}\%$ , respectively, which indicates that our proposed training paradigm can be universally applicable to memory module-based models.
4.4 Discussion and analysis
MemAE
MNAD
Figure 2: Distribution of LSD values for normal and abnormal samples.
Table 4: Statistical analysis results of LSD. Values in the table are calculated as follows:  $\mathbb{E}_{q^{+}\in Normal}LSD(q^{+},m) / \mathbb{E}_{q^{-}\in Abnormal}LSD(q^{-},m)$ , where  $q^{+}$  and  $q^{-}$  denote query of normal and abnormal sample respectively.
SMD	MSL	PSM	SMAP	SWaT
MemAE	0.9674	0.9719	0.9796	1.0059	1.0321
MNAD	1.0052	1.0043	0.9960	1.0004	0.9750
Ours	0.9014	0.9484	0.9368	1.0098	0.6240
Statistical analysis of LSD We further explore the Gated memory module's superior ability to capture prototypical normal patterns in data. Figure 2 demonstrates box plots depicting the distribution of LSD values for normal and abnormal samples across three memory modules: ours,
MemAE, and MNAD. We calculate mean LSD values for normal and abnormal samples as follows:  $\mathbb{E}_{q^{+}\in Normal}LSD(q^{+},m)$  and  $\mathbb{E}_{q^{-}\in Abnormal}LSD(q^{-},m)$ , where  $m$  denotes fixed memory items. Mean LSD values for each type of memory module are in Appendix D.
We calculate the ratio between the mean LSD values for normal and abnormal samples in the test dataset. Table 4 presents the ratio values for each dataset while using three different memory modules. Our proposed approach consistently shows smaller ratio values than those of other methods across the majority of datasets. This suggests that the items stored in the Gated memory module are more distant from queries of abnormal samples than those of normal samples, and the gap between them is greater than that in other types of memory modules. It supports our assertion that the Gated memory module better encodes normal patterns in the data than other compared methods.
Figure 3: Visualization of anomaly scores on SMD. Each graph displays the anomaly scores on the y-axis and time on the x-axis. The yellow-shaded area indicates the ground truth of abnormal labels, while the blue line represents the anomaly scores predicted by the models, and the red dashed line represents the threshold values. Additional plots for different datasets are included in Appendix D.
Anomaly score We experiment on randomly extracted time segments of length 150 from all datasets to demonstrate the effectiveness of the bi-dimensional deviation-based detection criterion. The baselines used in this experiment include MEMTO leveraging either LSD or ISD as a detection criterion and the previous state-of-the-art model, Anomaly Transformer. Figure 3 illustrates the anomaly score and ground truth label associated with each timestamp within a segment. The results show that the bi-dimensional deviation-based criterion robustly detects anomalies compared to the baselines. Compared to Anomaly Transformer, which uses association-based detection criteria, the bi-dimensional deviation-based criterion shows precise detection with reduced false alarms. Furthermore, if either latent or input space aspects are removed, MEMTO shows unstable performance. While MEMTO using only LSD as a detection criterion exhibits a similar pattern in anomaly scores, scores in abnormal time points fail to exceed the threshold. This suggests that considering both factors is crucial for robust performance since it amplifies the gap between normal and abnormal time points based on anomaly scores.
Table 5: Time (in seconds) measured for each stage. Experiments are done on SWaT dataset.
time (sec)
First phase	K-means	Second phase	Total training	Inference
MEMTO	45.17	16.07	60.44	121.7	9.360
w/o Gated memory module	39.04	-	-	39.04	9.230
w/o two-phase training paradigm	49.62	-	-	49.62	9.360
Anomaly Transformer	46.87	-	-	46.87	10.53
Computational efficiency We measure the training time with and without the use of the Gated memory module and the two-phase training paradigm, respectively. The results are presented in Table 5. When training MEMTO without the Gated memory module, there is a reduction in both the number of parameters and training time. However, this results in a  $26.27\%$  p decline in performance, as in indicated in Table 3. Also, the two-phase training paradigm increases the training duration by approximately 2.45 times compared to its absence, though the inference time remains unaffected.
Remarkably, the two-phase training paradigm enhances performance by  $10.4\%$  p, justifying its adoption even with the extended training duration.
We further compare the computational efficiency between MEMTO and the previous state-of-the-art model, Anomaly Transformer. The Anomaly Transformer computes and retains series-association and prior-association per encoder layer, then averages the association discrepancies across layers. Performing this process during inference is computationally demanding, while MEMTO, requiring only dot product operations between each query and a few memory items, is simpler and results in shorter inference time. As presented in Table 5, while MEMTO requires over twice the training time of Anomaly Transformer due to the two-phase training paradigm, it is noteworthy that when considering the critical inference time for real-world applications, MEMTO is faster by 1.17 seconds.
Figure 4: Performance across different numbers of memory items in the Gated memory module
Furthermore, given that the computational cost increases as the number of memory items within MEMTO's Gated memory module grows, we conduct experiments to analyze the optimal number of memory items. Figure 4 illustrates the relationship between the performance of MEMTO's performance and the number of memory items used. Results show that MEMTO's performance is robust to the number of memory items, as the performance variance across datasets is small. Hence, we designate ten memory items as the default value after weighing performance and computational complexity. Our study highlights the effectiveness of employing a restricted number of memory items to extract prototypical features of normal patterns in time series data. Unlike computer vision, which may require thousands of memory items [8], we demonstrate that only ten memory items are necessary for a task in the time series domain.
We introduce MEMTO, an unsupervised reconstruction-based model for multivariate time series anomaly detection. The Gated memory module in MEMTO adaptively captures the normal patterns in response to the input data, and it can be trained robustly using a two-phase training paradigm. Our proposed anomaly criterion, which comprehensively considers bi-dimensional space, enhances the performance of MEMTO. Extensive experiments on real-world multivariate time series benchmarks validate that our proposed model achieves state-of-the-art performance compared to existing competitive models.
Limitations While our two update stages in the Gated memory module and bi-dimensional deviation-based criterion have led to enhanced performance, a thorough theoretical proof for its efficacy remains to be established. Also, we acknowledge the limitation of our omission of visually inspecting the prototypical normal patterns stored in memory items. In the future, we will explore these issues further as a part of our research.
Broader impacts MEMTO is tailored for detecting anomalies in multivariate time series data and can be applied to various complex cyber-physical systems, such as smart factories, power grids, data centers, and vehicles. However, we strongly discourage its use in activities related to financial crimes or other applications that could have negative societal consequences.
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT). (No. 2021R1A2C2093785)
[1] Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 2485-2494, 2021.
[2] Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3395-3404, 2020.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
[4] Ane Blázquez-García, Angel Conde, Usue Mori, and Jose A Lozano. A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR), 54(3):1-33, 2021.
[5] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jorg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[6] Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4080-4088, 2018.
[7] Yongliang Cheng, Yan Xu, Hong Zhong, and Yi Liu. Hs-tn: A semi-supervised hierarchical stacking temporal convolutional network for anomaly detection in iot. In 2019 IEEE 38th International Performance Computing and Communications Conference (IPCCC), pages 1-7. IEEE, 2019.
[8] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705-1714, 2019.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
[10] Mononito Goswami, Cristian Challu, Laurent Callot, Lenon Minorics, and Andrey Kan. Unsupervised model selection for time-series anomaly detection. arXiv preprint arXiv:2210.01078, 2022.
[11] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural tuning machines. arXiv preprint arXiv:1410.5401, 2014.
[12] Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 312-329. Springer, 2020.
[13] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 387-395, 2018.
[14] Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint arXiv:1703.03129, 2017.
We use ten memory items for our MEMTO model, corresponding to the number of clusters in our  $K$ -means clustering. To determine anomalies, we set the threshold as the top- $p\%$  of the combined results of the anomaly scores from both the training and validation data, with specified values of  $p$  for each dataset outlined in Table 6, following [40]. We set  $\lambda$  in the objective function to 0.01, use Adam optimizer [15] with a learning rate of 5e-5, and employ early stopping with the patience of 10 epochs against the validation loss during training. Our experiments are conducted using the Pytorch framework on four NVIDIA GTX 1080 Ti 12GB GPUs. Furthermore, during the execution of our experiment, we make partial references to the code of [40].
A.1 Hyperparameter settings
Important hyperparameters of MEMTO were determined through grid search, while others were set to commonly used default values based on empirical observations. We performed a grid search to determine the values of each hyperparameter within the following range:
-  $\lambda \in \{1\mathrm{e} + 0,5\mathrm{e} - 1,1\mathrm{e} - 1,5\mathrm{e} - 2,1\mathrm{e} - 2,5\mathrm{e} - 3,1\mathrm{e} - 3\}$
-  $lr \in \{1\mathrm{e} - 4,3\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 5,3\mathrm{e} - 5,5\mathrm{e} - 5\}$
-  $\tau \in \{0.1, 0.3, 0.5, 0.7, 0.9\}$
-  $M \in \{5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100\}$
, where  $lr$ ,  $\tau$ , and  $M$  denote the learning rate, the temperature in the softmax function, and the number of clusters, respectively. Since we set the centroids of clusters as memory items, the number of memory items and that of clusters are the same. We set the optimal hyperparameters as follows:  $\lambda$  as 1e-2,  $lr$  as 5e-5,  $\tau$  as 0.1, and  $M$  as 10. All experiments in this paper are conducted using the same hyperparameters regardless of the dataset.
A.2 Dataset
Table 6: Details in five benchmarks. The number of samples in the training, validation, and test sets is represented in the columns labeled 'Train,' 'Valid,' and 'Test,' respectively. The  $p\%$  column indicates the anomaly ratio used in the experiment. The 'Dim' column shows the dimension size of the data for each dataset.
Train	Valid	Test	p(%)	Dim
SMD	566,724	141,681	708,420	0.5	38
MSL	46,653	11,664	73,729	1.0	55
PSM	105,984	26,497	87,841	1.0	26
SMAP	108,146	27,037	427,617	1.0	25
SWaT	396,000	99,000	449,919	0.1	53
Table 6 shows the statistical details of datasets used in experiments. We obtained SWaT by submitting a request through https://itrust.sutd.edu.sg/itrust-labs_datasets/.
B Algorithm for MEMTO
Algorithm 2 Proposed Method MEMTO
Input  $X^{s}\in \mathbb{R}^{L\times n}$  : input sub-series   
Training params  $f_{e}$  encoder,  $f_{d}$  decoder,  $U_{\psi},W_{\psi}\in \mathbb{R}^{C\times C}$  linear projection matrices   
1:  $q^s = f_e(X^s)$  \feed-forward encoder,  $q^{s}\in \mathbb{R}^{L\times C}$    
2:  $v^{s} =$  softmax  $\left(m(q^s)^T\right)$  Gated memory update start,  $m\in \mathbb{R}^{M\times C}$ $v^{s}\in \mathbb{R}^{M\times L}$    
3:  $\psi =$  sigmoid  $(mU_{\psi} + (v^{s}q^{s})W_{\psi})$ $\backslash \backslash$ $\psi \in \mathbb{R}^{M\times C}$    
4:  $m = (1 - \psi)\circ m + \psi \circ (v^{s}q^{s})$  Gated memory update end   
5:  $w^{s} =$  softmax  $\left(q^{s}(m)^{T}\right)$ $\backslash \backslash$  Query update start,  $w^{s}\in \mathbb{R}^{L\times M}$    
6:  $\tilde{q}^s = w^s m$ $\backslash \backslash$ $\tilde{q}^s\in \mathbb{R}^{L\times C}$    
7:  $\hat{q}^s =$  concat  $[q^s,\tilde{q}^s ],dim = 1)$ $\backslash \backslash$  Query update end,  $\hat{q}^s\in \mathbb{R}^{L\times 2C}$    
8:  $\hat{X}^s = f_d(\hat{q}^s)$  \feed-forward decoder,  $\hat{X}^s\in \mathbb{R}^{L\times n}$    
9: return  $\hat{X}^s$  \reconstructed sub-series
Algorithm 2 provides an overall mechanism for our model. It demonstrates the matrix operation version of the forward process when a single input sub-series  $X^{s}$  is fed to MEMTO.
Table 7: The ablation results (F1-score) in anomaly criterion and objective function.  $L_{rec}$  and  $L_{entr}$  signify Reconstruction Loss and Entropy Loss, respectively.
Loss	Anomaly Criterion	F1-score
ISD	LSD	SMD	MSL	PSM	SMAP	SWaT	avg.
\( L_{rec} \)	✓	×	79.63	86.23	82.15	71.18	31.29	70.09
×	✓	69.73	72.63	93.07	67.69	82.50	77.12
✓	✓	93.19	92.66	98.05	96.48	93.34	94.74
\( L_{entr} \)	✓	×	75.71	88.39	87.47	69.28	79.28	80.02
×	✓	12.53	84.34	76.49	68.17	83.52	65.01
✓	✓	88.43	93.40	97.97	96.22	92.77	93.75
\( L_{rec} + \lambda L_{entr} \)	✓	×	77.54	87.22	79.25	70.99	31.17	69.23
×	✓	72.78	80.33	80.15	67.55	0.00	60.16
✓	✓	93.54	94.36	98.34	96.61	95.83	95.73
Table 8: We report mean and standard deviation over 10 runs for A.T (Anomaly Transformer) and MEMTO, respectively. We conduct t-test  $(\mathrm{p} < 0.05)$  to indicate statistical significance.
SMD	MSL	SMAP	SWaT	PSM
A.T	mean	91.34	92.60	95.83	92.86	97.57
std	0.6942	1.103	1.088	0.9455	0.1230
MEMTO	mean	93.05	94.07	96.49	95.23	98.15
std	0.3762	0.5185	0.07987	1.016	0.1666
p-value	0.0000	0.0301	0.0360	0.0002	0.0000
C.1 Objective function and anomaly criterion
In this experiment, we investigate the impact of loss terms, specifically the reconstruction loss  $L_{rec}$  and the entropy loss  $L_{entr}$ , on the performance of our proposed framework, MEMTO. We remove
one of the two terms one by one from the objective function and evaluate the resulting performance. Table 7 demonstrates the significance of incorporating both  $L_{rec}$  and  $L_{entr}$  terms in the objective function. Applying bi-dimensional deviation-based criterion to the MEMTO variants that only use  $L_{rec}$  or  $L_{entr}$  as the loss function shows competitive performance compared to ours in terms of average F1-score. This demonstrates the robustness of MEMTO to loss terms. Additionally, both cases show a significant performance drop when using only ISD or LSD as the anomaly criterion, emphasizing the importance of combining ISD and LSD for achieving optimal performance.
C.2 Statistical significance test
We also perform statistical tests comparing our results to the latest state-of-the-art model, Anomaly Transformer. We conduct a t-test to prove a significant performance difference between MEMTO and Anomaly Transformer. The results in Table 8 show a p-value less than 0.05 across all datasets, confirming a significant performance difference between the two models.
C.3 Number of decoder layers
Figure 5: F1-score and number of parameters, according to the number of decoder layers. The right y-axis represents the values of the blue line graph in million units, while the left y-axis represents the values of the bar graph.
Figure 5 provides the performance of MEMTO under different numbers of decoder layers. As shown in Figure 5, a decoder that is too shallow (e.g., a decoder with a single layer) performs worse because it lacks sufficient capacity to reconstruct the input data accurately. On the other hand, if the decoder is too large (e.g., decoder with ten layers), it can become overly expressive and reconstruct even anomalies regardless of the encoding ability of the encoder. Therefore, it can lead to an over-generalization problem, which can ultimately decrease the performance of anomaly detection by reconstructing anomalies too accurately. Furthermore, a larger decoder layer with more parameters can increase computational and memory costs. We empirically find that considering the balance between performance and resource cost, a decoder with two layers is most suitable for anomaly detection tasks presented in our paper.
D Additional details for discussion
D.1 LSD values
Table 9 shows mean LSD values of normal and abnormal samples across various domains of datasets while using different memory module mechanisms. In most datasets, our proposed Gated memory module consistently exhibits a lower mean LSD value for normal samples than for abnormal samples. Furthermore, the relative difference between these values is more significant than other memory module mechanisms. These results demonstrate the efficacy of our memory module mechanism in capturing prototypical features of normal patterns in data.
Table 9: The mean LSD values corresponding to test data.
SMD	MSL	PSM	SMAP	SWaT
Normal	Abnormal	Normal	Abnormal	Normal	Abnormal	Normal	Abnormal	Normal	Abnormal
MemAE	814.7836	842.2023	622.5195	640.4954	766.2473	782.1895	710.4929	706.3115	795.7227	770.9069
MNAD	259.3633	258.0175	791.6371	788.2654	292.3340	293.4836	301.3480	301.2153	303.1933	310.9818
Ours	297.5692	330.1162	249.8632	263.4532	340.7552	363.7520	237.0070	234.7110	450.0926	721.3093
D.2 Anomaly score
Figure 6: Visualization of anomaly scores for MSL, PSM, SMAP, and SWaT datasets.
Figure 6 visually represents the anomaly scores for benchmark datasets not discussed in Section 4.4. We randomly sampled data of length 150 from MSL, PSM, SMAP, and SWaT test datasets and plotted the anomaly scores for each segment. Compared to other baselines, our proposed method consistently detects anomalies precisely with a low false positive rate from the perspective of the point adjustment method.