SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junho Song $^{1*}$  Keonwoo Kim $^{1,2*}$  Jeonglyul Oh $^{1}$  Sungzoon Cho $^{1\dagger}$
$^{1}$ Seoul National University
$^{2}$ VRCREW Inc.
{jhsong, keonwookim, jamesoh0813}@b dai.snu.ac.kr
zoon@snu.ac.kr
Detecting anomalies in real-world multivariate time series data is challenging due to complex temporal dependencies and inter-variable correlations. Recently, reconstruction-based deep models have been widely used to solve the problem. However, these methods still suffer from an over-generalization issue and fail to deliver consistently high performance. To address this issue, we propose the MEMTO, a memory-guided Transformer using a reconstruction-based approach. It is designed to incorporate a novel memory module that can learn the degree to which each memory item should be updated in response to the input data. To stabilize the training procedure, we use a two-phase training paradigm which involves using  $K$ -means clustering for initializing memory items. Additionally, we introduce a bi-dimensional deviation-based detection criterion that calculates anomaly scores considering both input space and latent space. We evaluate our proposed method on five real-world datasets from diverse domains, and it achieves an average anomaly detection F1-score of  $95.74\%$ , significantly outperforming the previous state-of-the-art methods. We also conduct extensive experiments to empirically validate the effectiveness of our proposed model's key components.
As cyber-physical systems advance, a vast amount of time series data is continuously collected from numerous sensors. Anomalies resulting from malfunctions in critical infrastructures, such as water treatment facilities and space probes, can incur fatal property loss. The task of multivariate time series anomaly detection involves identifying whether each timestamp of the multivariate time series is normal or abnormal. Anomaly detection in real-world scenarios is challenging due to severe data imbalance and the prevalence of unlabeled anomalies. We have formulated the problem as an unsupervised learning task to tackle these challenges. The underlying assumption of this approach is that the training data solely consists of normal samples [25, 42].
Traditional unsupervised learning methods such as one-class SVM (OC-SVM) [28], support vector data description (SVDD) [35], isolation forest [20], and local outlier factor (LOF) [5] have been widely used for anomaly detection tasks. Recently, density-estimation methods combined with deep representation learning, such as DAGMM [45] and MPPCACD [41], have also been presented. For the clustering-based method, Deep SVDD [25] finds the smallest hyper-sphere enclosing most normal samples in the feature space trained using deep neural networks. However, they exhibit poor performance in the time series domain due to their inability to capture dynamic nonlinear temporal
*indicates equal contributions
$\dagger$  indicates corresponding author
dependencies and complex inter-variable correlations. Deep models tailored for sequential data have been introduced to solve these inherent challenges [30, 7, 29]. THOC [29] uses a differentiable hierarchical clustering mechanism to combine temporal features of varying scales from different resolutions. Multiple hyper-spheres are used to represent each normal pattern at various resolutions, improving the representational capacity to capture intricate temporal features of time series data.
One of the primary types of recent deep methods is a reconstruction-based method, which uses an encoder-decoder architecture trained by a self-supervised pretext task of reconstructing input. This approach expects accurate reconstruction for normal samples and high reconstruction errors for anomalies. Early methods include LSTM-based encoder-decoder model [22] and LSTM-VAE [23]. OmniAnomaly [31] and InterFusion [19] are other stochastic recurrent neural network-based models extended from LSTM-VAE. Another branch of the reconstruction-based method is deep generative models. MAD-GAN [17] and BeatGAN [42] are generative adversarial network [9] variants tailored for time series anomaly detection. The most recently proposed Anomaly Transformer [40] introduces the Anomaly-Attention mechanism to simultaneously model prior- and series- associations. However, to the best of our knowledge, existing reconstruction-based methods may suffer from an overgeneralization problem, where abnormal inputs are reconstructed too well [8, 24]. This can occur if the encoder extracts unique features of an anomaly or if the decoder has excessive decoding capabilities for abnormal encoding vectors.
Our paper presents a new reconstruction-based method, a memory-guided Transformer, for multivariate time series anomaly detection (MEMTO). One of the key components in MEMTO is the Gated memory module, which includes items representing the prototypical features of normal patterns in the data. We employ an incremental approach to train the individual items in the Gated memory module. It determines the degree to which each existing item should be updated based on the input data. This approach enables MEMTO to adjust to diverse normal patterns in a more data-driven manner. Reconstructing abnormal samples using the stored features of normal patterns in the memory items can lead to reconstruction outputs that resemble normal samples. Under this condition, MEMTO finds it difficult to reconstruct anomalies, thereby releasing the over-generalization issue. Moreover, we have noticed that updating memory items incrementally can result in training instability if the items are initialized randomly. Therefore, we propose a two-phase training paradigm to ensure stable training by injecting inductive bias of normal prototypical patterns into memory items through applying  $K$ -means clustering. Furthermore, our observation reveals that the latent representations of anomalies are considerably more distant from memory items than those of normal time points since each item encompasses a prototype of normal patterns. Consequently, while calculating anomaly scores, we attempt to fully employ the memory items' nature of storing prototypical features of normal patterns. We introduce a bi-dimensional deviation-based detection criterion that comprehensively considers both the input and latent space. Our approach has been tested on five standard benchmarks, which include real-world applications. Experimental results indicate that our approach is effective and achieves state-of-the-art anomaly detection results compared to existing methods. The contributions of this paper are fourfold:
- Our proposed MEMTO is the first multivariate time series anomaly detection method that uses the Gated memory module, which adjusts to diverse normal patterns in a data-driven manner.
- We propose a two-phase training paradigm, an universally applicable approach designed to enhance the stability and robustness of memory module-based models.
- We propose a bi-dimensional deviation-based detection criterion in the online detection process. It considers both latent and input space to aggregate information in data comprehensively.
- MEMTO achieves state-of-the-art results on five real-world benchmark datasets. Extensive ablation studies demonstrate the effectiveness of key components in MEMTO.
In this section, we provide a brief explanation of the memory network. [11, 38] are early studies of external memory that can be retrieved and written. [34] proposes the improved version of Memory Networks [38], which uses attention mechanism [3] to retrieve memory entries, thus requiring less training supervision due to an end-to-end training manner. Recently, several attempts have been
Figure 1: Illustration of proposed MEMTO. Green  $\bowtie$  depicts the update gate  $\psi$  in the Gated memory update stage, which controls the degree of updating items. Red  $\oplus$  represents concatenation of queries  $q_{t}^{s}$  and retrieved memory items  $\tilde{q}_{t}^{s}$ . 'Weak Decoder' represents two fully-connected layers.
made to apply memory network in various domains. Natural language processing includes question answering [16, 39, 11, 34]. In computer vision, there exist video representation learning [12], one-shot learning [14, 6, 27], and text-to-image synthesis [44]. Moreover, several works have also adopted memory network for anomaly detection in computer vision [8, 24, 21]. Although MemAE [8] is the first to integrate a memory network into autoencoder architecture, it lacks an explicit memory updating process. MNAD [24] has suggested a distinct strategy for updating memory, which involves explicitly storing diverse normal patterns of normal data in memory items. Nonetheless, this approach simply adds a weighted sum of the relevant queries to each memory item without controlling the degree of new information being injected into existing memory items. However, our Gated memory module in MEMTO is the first to adapt to diverse normal patterns in a data-driven way by learning how intensively each existing memory item should be updated in response to new normal patterns.
3.1 Overall description of MEMTO
We define the raw time series  $D$  as a set of sub-series  $D = \{X^1, \ldots, X^N\}$ , where  $N$  represents the total number of sub-series. Each sub-series  $X^s \in \mathbb{R}^{L \times n}$  is a sequence of timestamps  $[x_1^s, \ldots, x_L^s]$ .  $L$ ,  $n$ , and  $x_t^s \in \mathbb{R}^n$  represent sub-series length, input dimension, and timestamp at time  $t$ , respectively.  $q^s = [q_1^s, \ldots, q_L^s] \in \mathbb{R}^{L \times C}$  refers to the encoder output feature (i.e., a query sequence) corresponding to  $X^s$ , where  $C$  denotes a latent dimension.  $q_t^s \in \mathbb{R}^C$  ( $t = 1, \ldots, L$ ) is an individual query vector (i.e., a query) at time  $t$ .
Figure 1 illustrates the overall architecture of MEMTO, which mainly consists of encoder-decoder architecture with the Gated memory module. The input sub-series  $X^{s}$  is first fed to the encoder, and the encoder output features are then used as queries to retrieve relevant items or update items in the Gated memory module. The inputs of the decoder are the updated queries  $\hat{q}_t^s \in \mathbb{R}^{2C} (t = 1,\dots ,L)$ , which are the combined features of the queries  $q_{t}^{s}$  and the retrieved memory items  $\tilde{q}_t^s \in \mathbb{R}^C$ . The decoder maps the updated query sequence  $\hat{q}^s = [\hat{q}_1^s,\dots ,\hat{q}_L^s]$  back to the input space, and outputs the reconstructed input sub-series  $\hat{X}^s \in \mathbb{R}^{L\times n}$ . Algorithm 2 shows the overall mechanism for our proposed model in Appendix B.
3.1.1 Encoder and decoder
Transformer [36] is well known to capture long-term complex temporal patterns in time series data [43]. MEMTO uses a Transformer encoder to project the input sub-series into a latent space.
In contrast, we adopt a weak decoder consisting of two fully-connected layers. It is essential that the decoder should not be overly powerful, as this can lead to a situation where its performance is independent of the encoder's ability to encode input time series. In the extreme case, a strong decoder comprised of multiple deep layers can generate the input data accurately even from random noise that contains no information about input data [37].
3.1.2 Gated memory module
In order to tackle the problem of over-generalization in reconstruction-based models, we introduce a new memory module mechanism that adjusts to diverse normal patterns in a data-driven manner. In this approach, each item stored in the memory module represents the prototypical features of normal data. Our proposed two-stage iterative update process allows us to extract latent representations of input sub-series using the prototypical patterns of normal data stored in the memory module, which acts as regularization to mitigate over-generalization. This mechanism is designed to limit the encoder's ability to capture the unique properties of anomalies, thereby making it more challenging to reconstruct abnormal data.
Gated memory update stage Memory item  $m_{i}\in \mathbb{R}^{C}$ $(i = 1,\dots ,M)$ , where  $M$  denotes the number of memory items, is trained to contain prototypical normal patterns of the input time series [24]. We expect memory items to contain prototypes of all queries  $q_{t}^{s}$  corresponding to normal timestamps. Thus, we adopt an incremental approach for updating the memory items by defining query-conditioned memory attention  $v_{i,t}^{s}$ . It is calculated by softmax on dot product between each memory item and queries as follows:
$$
v _ {i, t} ^ {s} = P \left(m _ {i} \rightarrow q _ {t} ^ {s}\right) = \frac {\exp \left(\left\langle m _ {i} , q _ {t} ^ {s} \right\rangle / \tau\right)}{\sum_ {p = 1} ^ {L} \exp \left(\left\langle m _ {i} , q _ {p} ^ {s} \right\rangle / \tau\right)}, \tag {1}
$$
where  $\tau$  denotes a temperature hyperparameter. We have proposed an update gate  $\psi$  in our memory module mechanism to train each memory item flexibly according to various normal patterns. This gate controls the extent to which the newly acquired normal patterns from queries are infused into the existing prototypical normal patterns stored in memory items. It allows our model to learn to what degree each existing memory item should be updated in a data-driven way. Equations for our memory update mechanism are:
$$
\psi = \sigma \left(U _ {\psi} m _ {i} + W _ {\psi} \sum_ {t = 1} ^ {L} v _ {i, t} ^ {s} q _ {t} ^ {s}\right), \tag {2}
$$
$$
m _ {i} \leftarrow (1 - \psi) \circ m _ {i} + \psi \circ \sum_ {t = 1} ^ {L} v _ {i, t} ^ {s} q _ {t} ^ {s}, \tag {3}
$$
where  $U_{\psi}$  and  $W_{\psi}$  denote linear projections, and  $\sigma$  and  $\circ$  denotes sigmoid activation and element-wise multiplication, respectively. The memory update stage is only executed in the training phase.
Query update stage In query update stage, we generate updated queries  $\hat{q}_t^s$  which are then fed to decoder as inputs. Similar to the Gated memory update stage, we define memory-conditioned query-attention  $w_{t,i}^{s}$ , calculated by softmax on dot product between each query and memory items as follows:
$$
w _ {t, i} ^ {s} = P \left(q _ {t} ^ {s} \rightarrow m _ {i}\right) = \frac {\exp \left(\left\langle m _ {i} , q _ {t} ^ {s} \right\rangle / \tau\right)}{\sum_ {j = 1} ^ {M} \exp \left(\left\langle m _ {j} , q _ {t} ^ {s} \right\rangle / \tau\right)}, \tag {4}
$$
Then, retrieved memory item  $\tilde{q}_t^s$  is obtained by weighted sum of the items  $m_{i}$ , using  $w_{t,i}^{s}$  as its corresponding weight as follows:
$$
\tilde {q} _ {t} ^ {s} = \sum_ {i ^ {\prime} = 1} ^ {M} w _ {t, i ^ {\prime}} ^ {s} m _ {i ^ {\prime}}. \tag {5}
$$
Query  $q_{t}^{s}$  and retrieved memory item  $\tilde{q}_{t}^{s}$  are concatenated along feature dimension to constitute updated query  $\hat{q}_{t}^{s}$ . The updated queries  $\hat{q}_{t}^{s} (t = 1, \dots, L)$  are the new robust inputs for the decoder since unique properties of an anomaly in  $q_{t}^{s}$  can be offset by the relevant normal patterns in the memory items. Reconstruction outputs of anomalies often appear similar to normal samples, making it more challenging to reconstruct anomalies. Such intensified difficulty can help distinguish between normal and abnormal data more effectively and prevent over-generalization.
For the pretext task for self-supervision, we minimize reconstruction loss while training. The reconstruction loss  $L_{rec}$  is defined as  $L2$  loss between  $X^s$  and  $\hat{X}^s$ :
$$
L _ {r e c} = \frac {1}{N} \sum_ {s = 1} ^ {N} \left\| X ^ {s} - \hat {X} ^ {s} \right\| _ {2} ^ {2}. \tag {6}
$$
Dense  $W^{s}$  enables some anomalies to have the probability of being well reconstructed [8], where  $W^{s}$  denotes the matrix form of  $w_{t,i}^{s}$  ( $t = 1, \dots, L$ ) and ( $i = 1, \dots, M$ ) in (4). Hence, to guarantee retrieving only a restricted number of closely relevant normal prototypes in memory, we introduce entropy loss  $L_{entr}$  as our auxiliary loss for sparsity regularization on  $W^{s}$ :
$$
L _ {e n t r} = \frac {1}{N} \sum_ {s = 1} ^ {N} \sum_ {t = 1} ^ {L} \sum_ {i = 1} ^ {M} - w _ {t, i} ^ {s} \log \left(w _ {t, i} ^ {s}\right). \tag {7}
$$
The objective function  $L$  is to minimize the combination of loss term (6) and (7) as follows:
$$
L = L _ {r e c} + \lambda L _ {e n t r}, \tag {8}
$$
where  $\lambda$  denotes a weighting coefficient.
3.3 Two-phase training paradigm
Algorithm 1 Memory module initialization with  $K$ -means clustering
Input:  $X\in \mathbb{R}^{N_{rand}\times L\times n}$  : randomly sampled sub-series  $\tilde{f}_e$  : trained encoder of MEMTO after first phase  $m\in \mathbb{R}^{M\times C}$  : randomly initialized memory items  $(N_{rand}$  : number of sub-series,  $M$  : number of memory items,  $C$  : latent dimension)  
1:  $q^{rand} = \tilde{f}_e(X)$  \\  $\backslash q^{rand}\in \mathbb{R}^{T\times C}$ , where  $T = N_{rand}\times L$   
2:  $c = K$ -means  $(q^{rand})$  \\  $\backslash$  Get  $M$  clustering centroids  $c\in \mathbb{R}^{M\times C}$   
3: for  $i = 1,\dots ,M$  do \\  $\backslash$  Initialize memory items  $m$  with clustering centroids  $c$   
4:  $m_i = c_i$   
5: end for  
6: return  $m$
Initializing memory items with  $K$ -means clustering Since we update the memory items incrementally, there is a risk of instability during training if the items are randomly initialized. We suggest a novel two-phase training paradigm that uses a clustering method to set the initial value of memory items to the approximate normal prototypical pattern of the data.
In the first phase, MEMTO is trained by a self-supervised task of reconstructing inputs, and the trained encoder of MEMTO generates queries for the randomly sampled  $10\%$  of the training data. We then apply  $K$ -means clustering algorithm to cluster the queries and designate each centroid as the initial value of a memory item. In the second phase, MEMTO is trained with these well-initialized items on anomaly detection tasks. Algorithm 1 outlines the memory module initialization with  $K$ -means clustering. In all of our experiments, we use  $K$ -means clustering as a baseline clustering method. Our claim is not that  $K$ -means clustering is the optimal choice among various clustering
methods, but rather the effectiveness of a two-phase training paradigm that allows for the selection of an appropriate clustering algorithm depending on the type of dataset.
3.4 Anomaly criterion
We introduce a bi-dimensional deviation-based detection criterion that comprehensively considers input and latent space. We define Latent Space Deviation (LSD)  $LSD(q_{t}^{s},m)$  at time point  $t$  as distance between each query  $q_{t}^{s}$  and its nearest memory item  $m_{t}^{s,pos}$  in latent space (9). LSD of anomalies would be larger than those of normal time points because each memory item contains a prototype of normal patterns. Additionally, we define Input Space Deviation (ISD)  $ISD(X_{t,:}^{s},\hat{X}_{t,:}^{s})$  at time  $t$  as distance between input  $X_{t,:}^{s}\in \mathbb{R}^{n}$  and reconstructed input  $\hat{X}_{t,:}^{s}\in \mathbb{R}^{n}$  in input space (10).
$$
L S D \left(q _ {t} ^ {s}, m\right) = \left\| q _ {t} ^ {s} - m _ {t} ^ {s, p o s} \right\| _ {2} ^ {2} \tag {9}
$$
$$
I S D \left(X _ {t,:} ^ {s}, \hat {X} _ {t,:} ^ {s}\right) = \left\| X _ {t,:} ^ {s} - \hat {X} _ {t,:} ^ {s} \right\| _ {2} ^ {2} \tag {10}
$$
We multiply normalized LSD with ISD, using LSD as weights for amplifying the normal-abnormal gap in ISD:
$$
A \left(X ^ {s}\right) = \operatorname {s o f t m a x} \left(\left[ L S D \left(q _ {t} ^ {s}, m\right) \right] _ {t = 1, \dots , L}\right) \circ \left[ I S D \left(X _ {t,:} ^ {s}, \hat {X} _ {t,:} ^ {s}\right) \right] _ {t = 1, \dots , L}, \tag {11}
$$
where  $\circ$  is element-wise multiplication and  $A(X^s) \in \mathbb{R}^L$  is anomaly score at each time point. Leveraging normal-abnormal distinguishing criteria in latent space (i.e., Latent Space Deviation) and input space (i.e., Input Space Deviation) leads to better detection performance.
Table 1: Precision(P), recall(R), F1-score(F1) results (as %) on five real-world datasets. 'A.T.' and 'avg' indicate Anomaly Transformer and average. We reproduce the MEMTO and Anomaly Transformer results while adopting the reported performance from [40] for the other baselines.
SMD	MSL	SMAP	SWaT	PSM	avg.
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	F1
LOF	56.34	39.86	46.68	47.72	85.25	61.18	58.93	56.33	57.60	72.15	65.43	68.62	57.89	90.49	70.61	60.94
OC-SVM	44.34	76.72	56.19	59.78	86.87	70.82	53.85	59.07	56.34	45.39	49.22	47.23	62.75	80.89	70.67	60.25
IsolationForest	42.31	73.29	53.64	53.94	86.54	66.45	52.39	59.07	55.53	49.29	44.95	47.02	76.09	92.45	83.48	61.22
MMPCACD	71.20	79.28	75.02	81.42	61.31	69.95	88.61	75.84	81.73	82.52	68.29	74.73	76.25	78.35	77.29	75.74
DAGMM	67.30	49.89	57.30	89.60	63.93	74.62	86.45	56.73	68.51	89.92	57.84	70.40	93.49	70.03	80.08	70.18
Deep-SVDD	78.54	79.67	79.10	91.92	76.63	83.58	89.93	56.02	69.04	80.42	84.45	82.39	95.41	86.49	90.73	80.97
THOC	79.76	90.95	84.99	88.45	90.97	89.69	92.06	89.34	90.68	83.94	86.36	85.13	88.14	90.99	89.54	88.01
LSTM-VAE	75.76	90.08	82.30	85.49	79.94	82.62	92.20	67.75	78.10	76.00	89.50	82.20	73.62	89.92	80.96	81.24
BeatGAN	72.90	84.09	78.10	89.75	85.42	87.53	92.38	55.85	69.61	64.01	87.46	73.92	90.30	93.84	92.04	80.24
OmniAnomaly	83.68	86.82	85.22	89.02	86.37	87.67	92.49	81.99	86.92	81.42	84.30	82.83	88.39	74.46	80.83	84.69
InterFusion	87.02	85.43	86.22	81.28	92.70	86.62	89.77	88.52	89.14	80.59	85.58	83.01	83.61	83.45	83.52	85.70
A.T.	87.96	94.68	91.20	91.13	90.12	90.63	93.96	98.45	96.15	85.85	100.00	92.39	96.81	98.63	97.71	93.62
MEMTO	89.13	98.40	93.54	92.07	96.76	94.36	93.76	99.63	96.61	94.18	97.54	95.83	97.46	99.23	98.34	95.74
Dataset We evaluate MEMTO on five real-world multivariate time series datasets. (i) Server Machine Dataset (SMD [33]) is a large-scale dataset collected over five weeks with 38 dimensions released from a large Internet company. (ii & iii) Mars Science Laboratory rover (MSL) and Soil Moisture Active Passive satellite (SMAP) are public data released from NASA [13], with 55 and 38 dimensions, respectively. (iv) Secure Water Treatment (SWaT [18]) consists of a six-stage infrastructure process with 51 sensors under 11 days of continuous operation. (v) Pooled Server Metrics (PSM [1]) consists of data of diverse application server nodes from eBay with 26 dimensions. Additional details on the datasets can be found in Appendix A.
Implementation details We generate sub-series by applying a non-overlapped sliding window with a length of 100 to obtain fixed-length inputs for each dataset. We split the training data into  $80\%$  for training and  $20\%$  for validation. More detailed information on hyperparameter settings can be found in Appendix A. The standard evaluation metrics for binary classification, including precision, recall, and F1-score, fail to account for the sequential properties of time series data, making them insufficient in assessing contextual and collective anomalies [10, 4]. Therefore, we use adjusted versions of these metrics, which have become widely used evaluation metrics in time series anomaly detection [32, 2]. Here point adjustment method is used, where if a single timestamp is detected as abnormal, every time point in the anomalous segment containing that timestamp is considered correctly detected.
In the main experiment, we evaluate the performance of MEMTO on multivariate time series anomaly detection tasks by comparing it with 12 models. The traditional machine learning baselines are LOF [5], OC-SVM [28], and Isolation Forest [20]. We also compare with several recent deep models, including density-estimation models (MPPCAD [41] and DAGMM [45]), clustering-based models (Deep-SVDD [26] and THOC [29]), and reconstruction-based models (LSTM-VAE [23], BeatGAN [42], OmniAnomaly [31], InterFusion [19], and Anomaly Transformer [40]).