Title: MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multivariate time series

Anomaly detection
Abstract: Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, reconstruction-based deep models have been widely used to solve the problem. However, these methods still suffer from an over-generalization issue and fail to deliver consistently high performance. To address this issue, we propose the MEMTO, a memory-guided Transformer using a reconstruction-based approach. It is designed to incorporate a novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, we use a two-phase training paradigm which involves using K-means clustering for initializing memory items. Additionally, we introduce a bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method on five real-world datasets from diverse domains, and it achieves an average anomaly detection F1-score of 95.74%, significantly outperforming the previous state-of-the-art methods. We also conduct extensive experiments to empirically validate the effectiveness of our proposed model's key components.

MEMTO: Memory-guided Transformer for Multivariate Time Series Anomaly Detection

Abstract

Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, reconstruction-based deep models have been widely used to solve the problem. However, these methods still suffer from an over-generalization issue and fail to deliver consistently high performance. To address this issue, we propose the MEMTO, a memory-guided Transformer using a reconstruction-based approach. It is designed to incorporate a novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, we use a two-phase training paradigm which involves using K -means clustering for initializing memory items. Additionally, we introduce a bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method on five real-world datasets from diverse domains, and it achieves an average anomaly detection F1-score of 95.74%, significantly outperforming the previous state-of-the-art methods. We also conduct extensive experiments to empirically validate the effectiveness of our proposed model's key components.

1 Introduction

As cyber-physical systems advance, a vast amount of time series data is continuously collected from numerous sensors. Anomalies resulting from malfunctions in critical infrastructures, such as water treatment facilities and space probes, can incur fatal property loss. The task of multivariate time series anomaly detection involves identifying whether each timestamp of the multivariate time series is normal or abnormal. Anomaly detection in real-world scenarios is challenging due to severe data imbalance and the prevalence of unlabeled anomalies. We have formulated the problem as an unsupervised learning task to tackle these challenges. The underlying assumption of this approach is that the training data solely consists of normal samples [25, 42].

Traditional unsupervised learning methods such as one-class SVM (OC-SVM) [28], support vector data description (SVDD) [35], isolation forest [20], and local outlier factor (LOF) [5] have been widely used for anomaly detection tasks. Recently, density-estimation methods combined with deep representation learning, such as DAGMM [45] and MPPCACD [41], have also been presented. For the clustering-based method, Deep SVDD [25] finds the smallest hyper-sphere enclosing most normal samples in the feature space trained using deep neural networks. However, they exhibit poor performance in the time series domain due to their inability to capture dynamic nonlinear temporal

dependencies and complex inter-variable correlations. Deep models tailored for sequential data have been introduced to solve these inherent challenges [30, 7, 29]. THOC [29] uses a differentiable hierarchical clustering mechanism to combine temporal features of varying scales from different resolutions. Multiple hyper-spheres are used to represent each normal pattern at various resolutions, improving the representational capacity to capture intricate temporal features of time series data.

One of the primary types of recent deep methods is a reconstruction-based method, which uses an encoder-decoder architecture trained by a self-supervised pretext task of reconstructing input. This approach expects accurate reconstruction for normal samples and high reconstruction errors for anomalies. Early methods include LSTM-based encoder-decoder model [22] and LSTM-VAE [23]. OmniAnomaly [31] and InterFusion [19] are other stochastic recurrent neural network-based models extended from LSTM-VAE. Another branch of the reconstruction-based method is deep generative models. MAD-GAN [17] and BeatGAN [42] are generative adversarial network [9] variants tailored for time series anomaly detection. The most recently proposed Anomaly Transformer [40] introduces the Anomaly-Attention mechanism to simultaneously model prior- and series- associations. However, to the best of our knowledge, existing reconstruction-based methods may suffer from an overgeneralization problem, where abnormal inputs are reconstructed too well [8, 24]. This can occur if the encoder extracts unique features of an anomaly or if the decoder has excessive decoding capabilities for abnormal encoding vectors.

Our paper presents a new reconstruction-based method, a memory-guided Transformer, for multivariate time series anomaly detection ( MEMTO ). One of the key components in MEMTO is the Gated memory module, which includes items representing the prototypical features of normal patterns in the data. We employ an incremental approach to train the individual items in the Gated memory module. It determines the degree to which each existing item should be updated based on the input data. This approach enables MEMTO to adjust to diverse normal patterns in a more data-driven manner. Reconstructing abnormal samples using the stored features of normal patterns in the memory items can lead to reconstruction outputs that resemble normal samples. Under this condition, MEMTO finds it difficult to reconstruct anomalies, thereby releasing the over-generalization issue. Moreover, we have noticed that updating memory items incrementally can result in training instability if the items are initialized randomly. Therefore, we propose a two-phase training paradigm to ensure stable training by injecting inductive bias of normal prototypical patterns into memory items through applying K -means clustering. Furthermore, our observation reveals that the latent representations of anomalies are considerably more distant from memory items than those of normal time points since each item encompasses a prototype of normal patterns. Consequently, while calculating anomaly scores, we attempt to fully employ the memory items' nature of storing prototypical features of normal patterns. We introduce a bi-dimensional deviation-based detection criterion that comprehensively considers both the input and latent space. Our approach has been tested on five standard benchmarks, which include real-world applications. Experimental results indicate that our approach is effective and achieves state-of-the-art anomaly detection results compared to existing methods. The contributions of this paper are fourfold:

Our proposed MEMTO is the first multivariate time series anomaly detection method that uses the Gated memory module, which adjusts to diverse normal patterns in a data-driven manner.

We propose a two-phase training paradigm, an universally applicable approach designed to enhance the stability and robustness of memory module-based models.

We propose a bi-dimensional deviation-based detection criterion in the online detection process. It considers both latent and input space to aggregate information in data comprehensively.

MEMTO achieves state-of-the-art results on five real-world benchmark datasets. Extensive ablation studies demonstrate the effectiveness of key components in MEMTO.

2 Related work

In this section, we provide a brief explanation of the memory network. [11, 38] are early studies of external memory that can be retrieved and written. [34] proposes the improved version of Memory Networks [38], which uses attention mechanism [3] to retrieve memory entries, thus requiring less training supervision due to an end-to-end training manner. Recently, several attempts have been

m

Xs

m

Transformer

2

m2

2

Weak

Encoder

+

Updated

Decoder

：

m;

queries

Input sub-series

：

Reconstructed

sub-series

Memory items

Queries

Retrieved memory items

Query update stage

.→Memory-conditioned query attention

→

Update process

Gated memory update stage

.> Query-condioned memory atetion

→Update process

made to apply memory network in various domains. Natural language processing includes question answering [16, 39, 11, 34]. In computer vision, there exist video representation learning [12], oneshot learning [14, 6, 27], and text-to-image synthesis [44]. Moreover, several works have also adopted memory network for anomaly detection in computer vision [8, 24, 21]. Although MemAE [8] is the first to integrate a memory network into autoencoder architecture, it lacks an explicit memory updating process. MNAD [24] has suggested a distinct strategy for updating memory, which involves explicitly storing diverse normal patterns of normal data in memory items. Nonetheless, this approach simply adds a weighted sum of the relevant queries to each memory item without controlling the degree of new information being injected into existing memory items. However, our Gated memory module in MEMTO is the first to adapt to diverse normal patterns in a data-driven way by learning how intensively each existing memory item should be updated in response to new normal patterns.

3 Method

3.1 Overall description of MEMTO

We define the raw time series D as a set of sub-series D = { X 1 , . . . , X N } , where N represents the total number of sub-series. Each sub-series X s ∈ R L × n is a sequence of timestamps [ x s 1 , . . . , x s L ] . L , n , and x s t ∈ R n represent sub-series length, input dimension, and timestamp at time t , respectively. q s = [ q s 1 , . . . , q s L ] ∈ R L × C refers to the encoder output feature (i.e., a query sequence) corresponding to X s , where C denotes a latent dimension. q s t ∈ R C ( t = 1 , . . . , L ) is an individual query vector (i.e., a query) at time t .

Figure 1 illustrates the overall architecture of MEMTO, which mainly consists of encoder-decoder architecture with the Gated memory module. The input sub-series X s is first fed to the encoder, and the encoder output features are then used as queries to retrieve relevant items or update items in the Gated memory module. The inputs of the decoder are the updated queries ˆ q s t ∈ R 2 C ( t = 1 , . . . , L ) , which are the combined features of the queries q s t and the retrieved memory items ˜ q s t ∈ R C . The decoder maps the updated query sequence ˆ q s = [ˆ q s 1 , . . . , ˆ q s L ] back to the input space, and outputs the reconstructed input sub-series ˆ X s ∈ R L × n . Algorithm 2 shows the overall mechanism for our proposed model in Appendix B.

3.1.1 Encoder and decoder

Transformer [36] is well known to capture long-term complex temporal patterns in time series data [43]. MEMTO uses a Transformer encoder to project the input sub-series into a latent space.

In contrast, we adopt a weak decoder consisting of two fully-connected layers. It is essential that the decoder should not be overly powerful, as this can lead to a situation where its performance is independent of the encoder's ability to encode input time series. In the extreme case, a strong decoder comprised of multiple deep layers can generate the input data accurately even from random noise that contains no information about input data [37].

3.1.2 Gated memory module

In order to tackle the problem of over-generalization in reconstruction-based models, we introduce a new memory module mechanism that adjusts to diverse normal patterns in a data-driven manner. In this approach, each item stored in the memory module represents the prototypical features of normal data. Our proposed two-stage iterative update process allows us to extract latent representations of input sub-series using the prototypical patterns of normal data stored in the memory module, which acts as regularization to mitigate over-generalization. This mechanism is designed to limit the encoder's ability to capture the unique properties of anomalies, thereby making it more challenging to reconstruct abnormal data.

Gated memory update stage Memory item m i ∈ R C ( i = 1 , . . . , M ) , where M denotes the number of memory items, is trained to contain prototypical normal patterns of the input time series [24]. We expect memory items to contain prototypes of all queries q s t corresponding to normal timestamps. Thus, we adopt an incremental approach for updating the memory items by defining query-conditioned memory attention v s i,t . It is calculated by softmax on dot product between each memory item and queries as follows:

where τ denotes a temperature hyperparameter. We have proposed an update gate ψ in our memory module mechanism to train each memory item flexibly according to various normal patterns. This gate controls the extent to which the newly acquired normal patterns from queries are infused into the existing prototypical normal patterns stored in memory items. It allows our model to learn to what degree each existing memory item should be updated in a data-driven way. Equations for our memory update mechanism are:

where U ψ and W ψ denote linear projections, and σ and ◦ denotes sigmoid activation and element-wise multiplication, respectively. The memory update stage is only executed in the training phase.

Query update stage In query update stage, we generate updated queries ˆ q s t which are then fed to decoder as inputs. Similar to the Gated memory update stage, we define memory-conditioned query-attention w s t,i , calculated by softmax on dot product between each query and memory items as follows:

Then, retrieved memory item ˜ q s t is obtained by weighted sum of the items m i , using w s t,i as its corresponding weight as follows:

Query q s t and retrieved memory item ˜ q s t are concatenated along feature dimension to constitute updated query ˆ q s t . The updated queries ˆ q s t ( t = 1 , . . . , L ) are the new robust inputs for the decoder since unique properties of an anomaly in q s t can be offset by the relevant normal patterns in the memory items. Reconstruction outputs of anomalies often appear similar to normal samples, making it more challenging to reconstruct anomalies. Such intensified difficulty can help distinguish between normal and abnormal data more effectively and prevent over-generalization.

3.2 Training

For the pretext task for self-supervision, we minimize reconstruction loss while training. The reconstruction loss L rec is defined as L 2 loss between X s and ˆ X s :

Dense W s enables some anomalies to have the probability of being well reconstructed [8], where W s denotes the matrix form of w s t,i ( t = 1 , . . . , L ) and ( i = 1 , . . . , M ) in (4). Hence, to guarantee retrieving only a restricted number of closely relevant normal prototypes in memory, we introduce entropy loss L entr as our auxiliary loss for sparsity regularization on W s :

The objective function L is to minimize the combination of loss term (6) and (7) as follows:

where λ denotes a weighting coefficient.

3.3 Two-phase training paradigm

Initializing memory items with K -means clustering Since we update the memory items incrementally, there is a risk of instability during training if the items are randomly initialized. We suggest a novel two-phase training paradigm that uses a clustering method to set the initial value of memory items to the approximate normal prototypical pattern of the data.

In the first phase, MEMTO is trained by a self-supervised task of reconstructing inputs, and the trained encoder of MEMTO generates queries for the randomly sampled 10% of the training data. We then apply K -means clustering algorithm to cluster the queries and designate each centroid as the initial value of a memory item. In the second phase, MEMTO is trained with these well-initialized items on anomaly detection tasks. Algorithm 1 outlines the memory module initialization with K -means clustering. In all of our experiments, we use K -means clustering as a baseline clustering method. Our claim is not that K -means clustering is the optimal choice among various clustering

methods, but rather the effectiveness of a two-phase training paradigm that allows for the selection of an appropriate clustering algorithm depending on the type of dataset.

3.4 Anomaly criterion

We introduce a bi-dimensional deviation-based detection criterion that comprehensively considers input and latent space. We define Latent Space Deviation (LSD) LSD ( q s t , m ) at time point t as distance between each query q s t and its nearest memory item m s,pos t in latent space (9). LSD of anomalies would be larger than those of normal time points because each memory item contains a prototype of normal patterns. Additionally, we define Input Space Deviation (ISD) ISD ( X s t, : , ˆ X s t, : ) at time t as distance between input X s t, : ∈ R n and reconstructed input ˆ X s t, : ∈ R n in input space (10).

We multiply normalized LSD with ISD, using LSD as weights for amplifying the normal-abnormal gap in ISD:

where ◦ is element-wise multiplication and A ( X s ) ∈ R L is anomaly score at each time point. Leveraging normal-abnormal distinguishing criteria in latent space (i.e., Latent Space Deviation) and input space (i.e., Input Space Deviation) leads to better detection performance.

4 Experiment

4.1 Experimental setup

Dataset We evaluate MEMTO on five real-world multivariate time series datasets. (i) Server Machine Dataset (SMD [33]) is a large-scale dataset collected over five weeks with 38 dimensions released from a large Internet company. (ii & iii) Mars Science Laboratory rover (MSL) and Soil Moisture Active Passive satellite (SMAP) are public data released from NASA [13], with 55 and 38 dimensions, respectively. (iv) Secure Water Treatment (SWaT [18]) consists of a six-stage infrastructure process with 51 sensors under 11 days of continuous operation. (v) Pooled Server Metrics (PSM [1]) consists of data of diverse application server nodes from eBay with 26 dimensions. Additional details on the datasets can be found in Appendix A.

Implementation details We generate sub-series by applying a non-overlapped sliding window with a length of 100 to obtain fixed-length inputs for each dataset. We split the training data into 80% for training and 20% for validation. More detailed information on hyperparameter settings can be found in Appendix A. The standard evaluation metrics for binary classification, including precision, recall, and F1-score, fail to account for the sequential properties of time series data, making them insufficient in assessing contextual and collective anomalies [10, 4]. Therefore, we use adjusted versions of these metrics, which have become widely used evaluation metrics in time series anomaly detection [32, 2]. Here point adjustment method is used, where if a single timestamp is detected as abnormal, every time point in the anomalous segment containing that timestamp is considered correctly detected.

4.2 Main results

In the main experiment, we evaluate the performance of MEMTO on multivariate time series anomaly detection tasks by comparing it with 12 models. The traditional machine learning baselines are LOF [5], OC-SVM [28], and Isolation Forest [20]. We also compare with several recent deep models, including density-estimation models (MPPCAD [41] and DAGMM [45]), clustering-based models (Deep-SVDD [26] and THOC [29]), and reconstruction-based models (LSTM-VAE [23], BeatGAN [42], OmniAnomaly [31], InterFusion [19], and Anomaly Transformer [40]).

Table1 shows the evaluation results on five multivariate time series anomaly detection tasks. Overall, models using Transformers such as MEMTO and Anomaly Transformer consistently show high performance with an F1-score of over 90% on all datasets. However, MEMTO substantially improves the average F1-score on benchmarks compared to the previous state-of-the-art model, Anomaly Transformer, from 93.62% to 95.74%. MEMTO effectively detects anomalies in complex time series data by adjusting to diverse normal patterns present in the data and creating prototypes of these patterns using a data-driven approach, thus achieving new state-of-the-art results.

4.3 Ablation study

In this section, we comprehensively analyze the proposed model, MEMTO, through a series of ablation studies on three key components: the anomaly criterion, the memory module, and the training paradigm. Appendix C shows a more detailed analysis of the ablation studies.

Anomaly criterion Unlike other time series anomaly detection models, MEMTO considers both the input and latent space to derive anomaly scores. Table2 presents the performance of MEMTO

using different anomaly detection criteria. Using only one of the two anomaly criteria, ISD or LSD, the average F1-score is low, by 69.23% and 60.16%, respectively. In particular, they exhibit extremely low performance on SWaT, indicating a high variance in performance across datasets. In contrast, our proposed method, which uses both ISD and LSD, consistently shows the highest performance over all datasets with relatively low variance compared to others.

Memory module In this study, we assess the capability of our novel Gated memory module in anomaly detection tasks. Table3 shows that removing the Gated memory module results in a significant decrease of the average F1-score by 32.56p% and a remarkable decrease of the F1-score specifically on SWaT to less than a third. To further validate the effectiveness of our approach, we also compare it with two existing memory module mechanisms (MemAE [8] and MNAD [24]) by replacing the memory module in MEMTO while using the same experimental settings. Our findings suggest that simply adding an explicit memory update process does not necessarily improve the performance, as evidenced by the comparable performance between MemAE and MNAD. However, significant performance improvements are observed when we apply our proposed memory module that integrates the update gate into the memory update process. Our approach demonstrates its superiority in anomaly detection tasks on diverse domains of datasets.

Training paradigm In order to enhance the stability of memory item updates during training, we use K -means clustering to initialize the values of memory items instead of random initialization. Our experimental results provide strong evidence that supports the effectiveness of the two-phase training paradigm of MEMTO. Table 3 demonstrates that not applying K -means clustering results in an 8.4p% decrease in the average F1-score of MEMTO compared to when it is applied. For such cases in MemAE and MNAD, scores decrease by 14.7p% and 15.4p%, respectively, which indicates that our proposed training paradigm can be universally applicable to memory module-based models.

4.4 Discussion and analysis

900

normal

type

abnormal

1000

type

normal

abnormal

800

type

normal

abnormal

800

006

700

700

LSD

600

LSD

800

LSD

600

500

700

500

400 -

400

300

600

300

200

500

SMD

MSL

Dataset

PSM

SMAP

SWaT

SMD

MSL

Dataset

PSM

SMAP

SWaT

SMD

MSL

Dataset

PSM

SMAP

SWaT

Ours

MemAE

MNAD

Statistical analysis of LSD We further explore the Gated memory module's superior ability to capture prototypical normal patterns in data. Figure 2 demonstrates box plots depicting the distribution of LSD values for normal and abnormal samples across three memory modules: ours,

MemAE, and MNAD. We calculate mean LSD values for normal and abnormal samples as follows: E q + ∈ Normal LSD ( q + , m ) and E q -∈ Abnormal LSD ( q -, m ) , where m denotes fixed memory items. Mean LSD values for each type of memory module are in Appendix D.

We calculate the ratio between the mean LSD values for normal and abnormal samples in the test dataset. Table 4 presents the ratio values for each dataset while using three different memory modules. Our proposed approach consistently shows smaller ratio values than those of other methods across the majority of datasets. This suggests that the items stored in the Gated memory module are more distant from queries of abnormal samples than those of normal samples, and the gap between them is greater than that in other types of memory modules. It supports our assertion that the Gated memory module better encodes normal patterns in the data than other compared methods.

0.5

1.0

9.5

0.5

0.4

80

0.4

0.4

auoos:

Ajeu

0.3

0.6

EO

Ajeu

0.3

0.2

0.2

0.2

0.1

0.2

0.1

0.1

0.0

0.0

0.0

0

20

40

60

Time

80

100

120

140

20

40

60

Time

80

100

120

140

0

26

40

60

Tme

80

100

120

140

20

40

60

Time

80

100

120

140

MEMTO (Ours)

MEMTO (LSD)

MEMTO (ISD)

Anomalytransformer

Anomaly score We experiment on randomly extracted time segments of length 150 from all datasets to demonstrate the effectiveness of the bi-dimensional deviation-based detection criterion. The baselines used in this experiment include MEMTO leveraging either LSD or ISD as a detection criterion and the previous state-of-the-art model, Anomaly Transformer. Figure 3 illustrates the anomaly score and ground truth label associated with each timestamp within a segment. The results show that the bi-dimensional deviation-based criterion robustly detects anomalies compared to the baselines. Compared to Anomaly Transformer, which uses association-based detection criteria, the bidimensional deviation-based criterion shows precise detection with reduced false alarms. Furthermore, if either latent or input space aspects are removed, MEMTO shows unstable performance. While MEMTO using only LSD as a detection criterion exhibits a similar pattern in anomaly scores, scores in abnormal time points fail to exceed the threshold. This suggests that considering both factors is crucial for robust performance since it amplifies the gap between normal and abnormal time points based on anomaly scores.

Computational efficiency We measure the training time with and without the use of the Gated memory module and the two-phase training paradigm, respectively. The results are presented in Table 5. When training MEMTO without the Gated memory module, there is a reduction in both the number of parameters and training time. However, this results in a 26.27%p decline in performance, as in indicated in Table 3. Also, the two-phase training paradigm increases the training duration by approximately 2.45 times compared to its absence, though the the inference time remains unaffected.

Remarkably, the two-phase training paradigm enhances performance by 10.4%p, justifying its adoption even with the extended training duration.

We further compare the computational efficiency between MEMTO and the previous state-of-the-art model, Anomaly Transformer. The Anomaly Transformer computes and retains series-association and prior-association per encoder layer, then averages the association discrepancies across layers. Performing this process during inference is computationally demanding, while MEMTO, requiring only dot product operations between each query and a few memory items, is simpler and results in shorter inference time. As presented in Table 5, while MEMTO requires over twice the training time of Anomaly Transformer due to the two-phase training paradigm, it is noteworthy that when considering the critical inference time for real-world applications, MEMTO is faster by 1.17 seconds.

1.000

0.975

0.950

0.925

F1-score

0.900

0.875

SMD

0.850

MSL

PSM

0.825

SMAP

SWaT

0.800

20

40

60

80

100

#of memomy items

Furthermore, given that the computational cost increases as the number of memory items within MEMTO's Gated memory module grows, we conduct experiments to analyze the optimal number of memory items. Figure 4 illustrates the relationship between the performance of MEMTO's performance and the number of memory items used. Results show that MEMTO's performance is robust to the number of memory items, as the performance variance across datasets is small. Hence, we designate ten memory items as the default value after weighing performance and computational complexity. Our study highlights the effectiveness of employing a restricted number of memory items to extract prototypical features of normal patterns in time series data. Unlike computer vision, which may require thousands of memory items [8], we demonstrate that only ten memory items are necessary for a task in the time series domain.

5 Conclusion

We introduce MEMTO, an unsupervised reconstruction-based model for multivariate time series anomaly detection. The Gated memory module in MEMTO adaptively captures the normal patterns in response to the input data, and it can be trained robustly using a two-phase training paradigm. Our proposed anomaly criterion, which comprehensively considers bi-dimensional space, enhances the performance of MEMTO. Extensive experiments on real-world multivariate time series benchmarks validate that our proposed model achieves state-of-the-art performance compared to existing competitive models.

Limitations While our two update stages in the Gated memory module and bi-dimensional deviation-based criterion have led to enhanced performance, a thorough theoretical proof for its efficacy remains to be established. Also, we acknowledge the limitation of our omission of visually inspecting the prototypical normal patterns stored in memory items. In the future, we will explore these issues further as a part of our research.

Broader impacts MEMTO is tailored for detecting anomalies in multivariate time series data and can be applied to various complex cyber-physical systems, such as smart factories, power grids, data centers, and vehicles. However, we strongly discourage its use in activities related to financial crimes or other applications that could have negative societal consequences.

References

Ahmed Abdulaal, Zhuanghua Liu, and Tomer Lancewicki. Practical approach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2485-2494, 2021.

Julien Audibert, Pietro Michiardi, Frédéric Guyard, Sébastien Marti, and Maria A Zuluaga. Usad: Unsupervised anomaly detection on multivariate time series. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3395-3404, 2020.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

Ane Blázquez-García, Angel Conde, Usue Mori, and Jose A Lozano. A review on outlier/anomaly detection in time series data. ACM Computing Surveys (CSUR) , 54(3):1-33, 2021.

Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages 93-104, 2000.

Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4080-4088, 2018.

Yongliang Cheng, Yan Xu, Hong Zhong, and Yi Liu. Hs-tcn: A semi-supervised hierarchical stacking temporal convolutional network for anomaly detection in iot. In 2019 IEEE 38th International Performance Computing and Communications Conference (IPCCC) , pages 1-7. IEEE, 2019.

Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memoryaugmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1705-1714, 2019.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Mononito Goswami, Cristian Challu, Laurent Callot, Lenon Minorics, and Andrey Kan. Unsupervised model selection for time-series anomaly detection. arXiv preprint arXiv:2210.01078 , 2022.

Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401 , 2014.

Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmented dense predictive coding for video representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 312-329. Springer, 2020.

Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 387-395, 2018.

Łukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint arXiv:1703.03129 , 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In International conference on machine learning , pages 1378-1387. PMLR, 2016.

Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In Artificial Neural Networks and Machine Learning-ICANN 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV , pages 703-716. Springer, 2019.

Dan Li, Dacheng Chen, Baihong Jin, Lei Shi, Jonathan Goh, and See-Kiong Ng. Mad-gan: Multivariate anomaly detection for time series data with generative adversarial networks. In Artificial Neural Networks and Machine Learning-ICANN 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17-19, 2019, Proceedings, Part IV , pages 703-716. Springer, 2019.

Zhihan Li, Youjian Zhao, Jiaqi Han, Ya Su, Rui Jiao, Xidao Wen, and Dan Pei. Multivariate time series anomaly detection and interpretation using hierarchical inter-metric and temporal embedding. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 3220-3230, 2021.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining , pages 413-422. IEEE, 2008.

Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13588-13597, 2021.

Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148 , 2016.

Daehyung Park, Yuuna Hoshi, and Charles C Kemp. A multimodal anomaly detector for robotassisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Letters , 3(3):1544-1551, 2018.

Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided normality for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14372-14381, 2020.

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning , pages 4393-4402. PMLR, 2018.

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning , pages 4393-4402. PMLR, 2018.

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850. PMLR, 2016.

Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation , 13(7):14431471, 2001.

Lifeng Shen, Zhuocong Li, and James Kwok. Timeseries anomaly detection using temporal hierarchical one-class network. Advances in Neural Information Processing Systems , 33: 13016-13026, 2020.

Lifeng Shen, Zhongzhong Yu, Qianli Ma, and James T Kwok. Time series anomaly detection with multiresolution ensemble decoding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9567-9575, 2021.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2828-2837, 2019.

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems , 28, 2015.

David MJ Tax and Robert PW Duin. Support vector data description. Machine learning , 54: 45-66, 2004.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Xixuan Wang, Dechang Pi, Xiangyan Zhang, Hao Liu, and Chang Guo. Variational transformerbased anomaly detection approach for multivariate time series. Measurement , 191:110791, 2022.

Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. arXiv preprint arXiv:1410.3916 , 2014.

Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning , pages 2397-2406. PMLR, 2016.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642 , 2021.

Takehisa Yairi, Naoya Takeishi, Tetsuo Oda, Yuta Nakajima, Naoki Nishimura, and Noboru Takata. A data-driven health monitoring method for satellite housekeeping data based on probabilistic clustering and dimensionality reduction. IEEE Transactions on Aerospace and Electronic Systems , 53(3):1384-1401, 2017.

Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. Beatgan: Anomalous rhythm detection using adversarially generated time series. In IJCAI , volume 2019, pages 4433-4439, 2019.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5802-5810, 2019.

Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations , 2018.

A Training details

We use ten memory items for our MEMTO model, corresponding to the number of clusters in our K -means clustering. To determine anomalies, we set the threshold as the topp %of the combined results of the anomaly scores from both the training and validation data, with specified values of p for each dataset outlined in Table 6, following [40]. We set λ in the objective function to 0.01, use Adam optimizer [15] with a learning rate of 5e-5, and employ early stopping with the patience of 10 epochs against the validation loss during training. Our experiments are conducted using the Pytorch framework on four NVIDIA GTX 1080 Ti 12GB GPUs. Furthermore, during the execution of our experiment, we make partial references to the code of [40].

A.1 Hyperparameter settings

Important hyperparameters of MEMTO were determined through grid search, while others were set to commonly used default values based on empirical observations. We performed a grid search to determine the values of each hyperparameter within the following range:

λ ∈ {1e+0, 5e-1, 1e-1, 5e-2, 1e-2, 5e-3, 1e-3}

lr ∈ {1e-4, 3e-4, 5e-4, 1e-5, 3e-5, 5e-5}

τ ∈ {0.1, 0.3, 0.5, 0.7, 0.9}

M ∈ {5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100}

, where lr , τ , and M denote the learning rate, the temperature in the softmax function, and the number of clusters, respectively. Since we set the centroids of clusters as memory items, the number of memory items and that of clusters are the same. We set the optimal hyperparameters as follows: λ as 1e-2, lr as 5e-5, τ as 0.1, and M as 10. All experiments in this paper are conducted using the same hyperparameters regardless of the dataset.

A.2 Dataset

Table 6: Details in five benchmarks. The number of samples in the training, validation, and test sets is represented in the columns labeled 'Train,' 'Valid,' and 'Test,' respectively. The ' p %' column indicates the anomaly ratio used in the experiment. The 'Dim' column shows the dimension size of the data for each dataset.

Table 6 shows the statistical details of datasets used in experiments. We obtained SWaT by submitting a request through https://itrust.sutd.edu.sg/itrust-labs_datasets/ .

B Algorithm for MEMTO

Algorithm 2 Proposed Method MEMTO

Algorithm 2 provides an overall mechanism for our model. It demonstrates the matrix operation version of the forward process when a single input sub-series X s is fed to MEMTO.

C Additional experiments

C.1 Objective function and anomaly criterion

In this experiment, we investigate the impact of loss terms, specifically the reconstruction loss L rec and the entropy loss L entr , on the performance of our proposed framework, MEMTO. We remove

one of the two terms one by one from the objective function and evaluate the resulting performance. Table 7 demonstrates the significance of incorporating both L rec and L entr terms in the objective function. Applying bi-dimensional deviation-based criterion to the MEMTO variants that only use L rec or L entr as the loss function shows competitive performance compared to ours in terms of average F1-score. This demonstrates the robustness of MEMTO to loss terms. Additionally, both cases show a significant performance drop when using only ISD or LSD as the anomaly criterion, emphasizing the importance of combining ISD and LSD for achieving optimal performance.

C.2 Statistical significance test

We also perform statistical tests comparing our results to the latest state-of-the-art model, Anomaly Transformer. We conduct a t-test to prove a significant performance difference between MEMTO and Anomaly Transformer. The results in Table 8 show a p-value less than 0.05 across all datasets, confirming a significant performance difference between the two models.

C.3 Number of decoder layers

1.00

SMD

8.0

MSL

PSM

SMAP

0.98

SWaT

7.5

#of parameters (M)

0.96

7.0

F1-score

6.5

0.94

6.0

0.92

5.5

0.90

2

5

10

#of decoder layers

Figure 5 provides the performance of MEMTO under different numbers of decoder layers. As shown in Figure 5, a decoder that is too shallow (e.g., a decoder with a single layer) performs worse because it lacks sufficient capacity to reconstruct the input data accurately. On the other hand, if the decoder is too large (e.g., decoder with ten layers), it can become overly expressive and reconstruct even anomalies regardless of the encoding ability of the encoder. Therefore, it can lead to an over-generalization problem, which can ultimately decrease the performance of anomaly detection by reconstructing anomalies too accurately. Furthermore, a larger decoder layer with more parameters can increase computational and memory costs. We empirically find that considering the balance between performance and resource cost, a decoder with two layers is most suitable for anomaly detection tasks presented in our paper.

D Additional details for discussion

D.1 LSD values

Table 9 shows mean LSD values of normal and abnormal samples across various domains of datasets while using different memory module mechanisms. In most datasets, our proposed Gated memory module consistently exhibits a lower mean LSD value for normal samples than for abnormal samples. Furthermore, the relative difference between these values is more significant than other memory module mechanisms. These results demonstrate the efficacy of our memory module mechanism in capturing prototypical features of normal patterns in data.

D.2 Anomaly score

MSL

”

17

100

00t

9100

0.014

0.012

0.150

PSM

0.004

200

SMAP

0.75

SWaT

Ours

LSD

ISD

Anomaly transformer

Figure 6 visually represents the anomaly scores for benchmark datasets not discussed in Section 4.4. We randomly sampled data of length 150 from MSL, PSM, SMAP, and SWaT test datasets and plotted the anomaly scores for each segment. Compared to other baselines, our proposed method consistently detects anomalies precisely with a low false positive rate from the perspective of the point adjustment method.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a reconstruction-based method for multivariate time series anomaly detection. It is a memory-guided Transformer which contains the gated memory module. Because of the training instability in updating memory items incrementally when the items are\ninitialized randomly, the authors propose a two-phase training paradigm to ensure stable training. The method calculates anomaly scores considering the input and latent space. The method achieves state-of-the-art results on five benchmark datasets.\n\nSTRENGTHS:\n1. The authors make a good explorations on leveraging memory module to multivariate time series anomaly detection task, and the proposed method can be effective solutions for the given task.\n2. This paper contains enough experiment results and ablation study on five datasets to support its claims.\n3. The proposed methodology is well presented, and most of the paper is well written.\n\nWEAKNESSES:\n1. The paper writing can still be improved. For example, the grammar error on line 88 \"there exist video representation learning\" -> \"there exists video representation learning\". Typo on line 234 \"32.56p%\". The citation in related work can be extended with author's name + 'et al.' rather than just a number. \n2. The font size in the figures (in introduction and experiment) is pretty small.\n3. I saw that you split the training data into 80% for training and 20% for validation. Do other methods for comparison in the experiments share the same data split(ratio and files)？ How many times you repeated your experiments? Since I saw the results you reproduced for Anomaly Transformer are quite different from their own paper, is it fair to list all these methods to compare with your model?\n\nQUESTIONS:\nPlease address the points mentioned in Weakness.\n\nLIMITATIONS:\nYes, the authors have addressed the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a memory-guided transformer with a reconstruction paradigm for multivariate time series anomaly detection. The time-series encoder, memory parameters of normal patterns, and projection heads are updated during training time in a two-step fashion. On test time, input queries are projected on the normal patterns learned base with a learned sparse linear projection. As a result, reconstructed anomalies tend to look like normal samples, amplifying the reconstruction loss and enabling detection. Experiments are run over common multivariate time-series datasets following previous works practices.\n\nSTRENGTHS:\n* The proposed gated memory module seems simple, original, and efficient for detecting anomalies on multivariate time series based on transformer encodings and input sample reconstruction. \n\n* The authors provide thorough ablation studies and some insights into the importance of each part of their algorithm.\n\n* The manuscript is overall easy to follow and well-written. Code is provided.\n\n## Suggestions\n\nThe typical memory size is quite small, which I see as a positive point of this method. So, I suggest this could be emphasized in the main manuscript. I read the entire paper wondering what the memory size was and whether the trade-off memory-performance was good, just to be positively surprised in the appendix.\n\nWEAKNESSES:\n## Regarding novelty\n\n1. Overall this work seems to be an incremental modification of [8], [24], and [40]. In particular, contribution #3 (line 77) does not seem very novel. I would like the authors to please discuss Equation (11) in this paper versus (6) in [40] in more detail.\n2. How does the memory update stage is different from other memory gates, such as GRU, MRU, etc?\n\n## Regarding state-of-the-art claim\n\n1. Why not compare to MNDA [24], memAE [8], USAD[2] in Table 1? Why are A.T. numbers in Table 1 different from [40]? I would suggest adding error bars at least to these methods and yours to make the comparison fair. I know this was not done in these previous works, but claiming state-of-the-art without comparing confidence intervals in an empirical field is not a good practice.\n\n## Regarding methodology\n\n4. Not clear why not just have a memory on the input space and retrieve the closest samples instead of the proposed mechanism on the latent space?\n5. Why is it so sensitive to k-means initialization? Is it computed with enough iterations of the K-means algorithm?\n6. How does algorithm 1 outlines the two-phase training paradigm (line 175)? The first and second phase seems to be omitted.\n\n## Other less important questions\n\n7. How are the thresholds computed in figure 3?\n8. What is the computation overhead when compared to A.T.? MEMTO seems simpler computationally. Could the authors discuss this?\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThe authors have discussed potential limitations and shortcomings of their algorithm and no special negative societal impact exclusive of their work is evident.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper used memory network to capture frequently present normal pattern in date set. While training, the memory are built in latent space in the autoencoder framework.  Along with reconstruction error, the distance between a representation with the closest memory unit is used to calculate anomaly score. If a data does not have representation close to memory then that is an anomaly.

STRENGTHS:
Idea of keeping more than one memory for common patterns from normal data is Novel and makes more sense than existing methods of keeping memory of anomalies.  Experimental results were produced on a large number  of data sets.

WEAKNESSES:
Literature surveys missed graph based anomaly detection techniques for multivariate time series like Deng, Ailin, and Bryan Hooi. "Graph neural network-based anomaly detection in multivariate time series." In Proceedings of the AAAI conference on artificial intelligence, vol. 35, no. 5, pp. 4027-4035. 2021.


It is required to have an ablation study in the main paper to bring our significance of memory gate and Kmeans clustering.

QUESTIONS:
Most cases  the F1 score for the proposed method is close to the A.T benchmark. Did you do a statistical significance test?
Do you run your experiment for different random train text split? In that case how is the  variance of accuracy ?
The F1 score for only LSD and only ISD is quite low. Did you note down if there are more FP or FN in both cases?
How to decide the required number of memory unit ?

LIMITATIONS:
Number of majority patterm in normal part need to be known before which is not possible during unsupervised task.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper focused on tackling the problem of over-generalization in reconstruction-based deep models and made a contribution to the field of multivariate time series anomaly detection. The main challenge they encountered was dealing with complex dependencies and inter-variable correlations within the data. To address this, they proposed a novel gated memory module approach. This module learns how much each memory item should be updated based on the input data, effectively capturing the prototypical features of normal patterns in the data. By doing so, it aims to overcome the inability of existing models to capture dynamic nonlinear temporal dependencies and complex inter-variable correlations.\n\nSTRENGTHS:\n• The authors explained the importance of multivariate time series anomaly detection with the proper real-world examples\n• Comprehensive comparison of their approach with traditional methods (OC-SVM, Deep-SVDD, LSTM-VAE, Anomaly Transformer etc.)\n• Thorough explanation of the training hyper-parameters, dataset setting, algorithms and code\n• Achieved state-of-the-art results on five real-world benchmark datasets\n\nWEAKNESSES:\n• Lack of theoretical proof: two-phase training approach and bi-dimensional deviation-based criterion need a detailed theoretical proof\n• Limited discussion of computational efficiency: how gated memory module and two-phase train- ing affect the training time\n• I couldn’t find any explicit explanations that how the LSD and ISD help the training. In the descriptions, LSD and ISD only appears in Table2 (and the sec3.4). If LSD+ ISD just help the anomaly score but doesn't help the training, this should be clearly justified. \n• Need more quantitative experiment about the improvement from gated memory seems limit. (Table 3)\n\nQUESTIONS:\n• How does the proposed method compare to existing approaches in terms of computational efficiency? Were any considerations or optimizations implemented to ensure its feasibility in real- time or resource-constrained environments?\n• What led the authors to choose K-means clustering for initializing memory items? Is K-means considered the most effective approach for this task, particularly in terms of computational efficiency?\n• How does the model behavior change if sub-series are generated with the help of overlapping sliding window?\n• Since K-means clustering creates prototypes based on the seen data items, how does the model handle unseen cases? If the proposed MEMTO couldn't consider the degree of new information being injected into existing memory items, what is the contribution of the MEMTO compared to the MNAD?\n\nLIMITATIONS:\nPlease refer to the Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}