1

Adaptive Memory Networks with Self-supervised
Learning for Unsupervised Anomaly Detection

arXiv:2201.00464v1 [cs.LG] 3 Jan 2022

Yuxin Zhang, Jindong Wang, Yiqiang Chen, Han Yu, Tao Qin
Abstract—Unsupervised anomaly detection aims to build models to effectively detect unseen anomalies by only training on the normal
data. Although previous reconstruction-based methods have made fruitful progress, their generalization ability is limited due to two
critical challenges. First, the training dataset only contains normal patterns, which limits the model generalization ability. Second, thefeature representations learned by existing models often lack representativeness which hampers the ability to preserve the diversity ofnormal patterns. In this paper, we propose a novel approach called Adaptive Memory Network with Self-supervised Learning (AMSL)
to address these challenges and enhance the generalization ability in unsupervised anomaly detection. Based on the convolutional
autoencoder structure, AMSL incorporates a self-supervised learning module to learn general normal patterns and an adaptive
memory fusion module to learn rich feature representations. Experiments on four public multivariate time series datasets demonstratethat AMSL significantly improves the performance compared to other state-of-the-art methods. Specifically, on the largest CAP sleep
stage detection dataset with 900 million samples, AMSL outperforms the second-best baseline by 4%+ in both accuracy and F1 score.
Apart from the enhanced generalization ability, AMSL is also more robust against input noise.
Index Terms—Unsupervised anomaly detection, Time series, Self-supervised learning, Memory network.

F

1

I NTRODUCTIONW

ITH the prevalence of Internet of Things (IoT) devices
in many applications (e.g., health care, human activity recognition and industrial control systems), there is
growing interest in anomaly detection [1], [2], [3]. However,
it is prohibitively expensive to acquire large amounts oflabeled anomaly data required to train machine learningmodels following the current approaches. For instance,
the collection of sleep apnea data [4] is extremely timeconsuming and restricted by the environment, while thedata for the normal states are much easier to obtain. For such
scenarios, unsupervised anomaly detection is a promisingparadigm that aims to detect anomalies by training fromonly normal samples, which is our main focus in this paper,and specifically, for multivariate time series.
The key challenge to unsupervised anomaly detection
is to learn generalized patterns from the normal training
data such that it can achieve good performance on theunseen anomalies. Over the years, there have been a lineof works on this topic. Autoencoder (AE) is a powerfulunsupervised learning technique that has been widely used

•

•

•
•
•

Y. Zhang is with Global Energy Interconnection Development and Cooperation Organization, Xicheng District, Beijing, China, and Beijing
Key Laboratory of Mobile Computing and Pervasive Device, Instituteof Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences, Beijing, China. E-mail: yuxinzhang@geidco.org.
Y. Chen are with Beijing Key Laboratory of Mobile Computing andPervasive Device, Institute of Computing Technology, Chinese Academy of
Sciences, Beijing, China and University of Chinese Academy of Sciences,
Beijing, China. Y. Chen is also with Peng cheng Laboratory (PCL). E-mail:yqchen@ict.ac.cn.
J. Wang and T. Qin are with Microsoft Research, Beijing, China. E-mail:{jindong.wang, taoqin}@microsoft.com.
H. Yu is with the School of Computer Science and Engineering, Nanyang
Technological University, Singapore. E-mail: han.yu@ntu.egu.sg.
Corresponding author: Yiqiang Chen and Jindong Wang

Manuscript received April 19, 2005; revised August 26, 2015.

for unsupervised anomaly detection [5], [6], [7]. AE is usually trained by minimizing the reconstruction error on thenormal data, and then using such errors as indicators or
thresholds of anomalies. Based on the AE structure, LSTMAE [8], Convolutional AE (CAE) [6], and ConvLSTM-AE [9]have emerged as promising anomaly detection approaches.
Despite the progress, the generalization ability of unsupervised anomaly detection methods is still limited due totwo critical challenges. The first challenge is limited normal
data. The normal training data are rather limited comparedto the unseen test data, making the model prone to overfitting. As shown in Fig. 1, when the normal (Fig. 1(a)) andabnormal samples (Fig. 1(b)) are similar, existing methods
can overfit to the anomalies. The second challenge is limited
feature representations. For complex time series data, thefeature representations learned by existing models lack representativeness required to preserve the diversity of normal
patterns. This is shown in Fig. 1, when the testing data is
becoming diverse to the normal samples (Fig. 1(c)), existing
methods fail to capture the diverse patterns. Therefore, since
the normal training data and feature representations areboth limited, the generalization performance of the existingmodels is poor when applied to unseen normal and abnormal data.
In this paper, we propose a novel Adaptive Memory
Network with Self-supervised Learning (AMSL) to increase
the generalization ability of unsupervised anomaly detection by tackling the above two challenges. First, to copewith the limited normal training data, AMSL incorporates
a self-supervised learning module to learn general patterns
from the normal data. Second, to cope with the limited
feature representations, AMSL introduces an adaptive memory fusion network to learn common and specific features
via the global and local memory modules, respectively.
Then, AMSL adopts an adaptive fusion module to fuse the2Others:

Ours:

Ours:

Ours:

Acceleration

Acceleration

Others:Acceleration

Others:

Time

Time

NormalTime

Abnormal
SimilarNormalDiverseFig. 1: Illustration of AMSL. Even with similar inputs ((a)
and (b)), existing methods can overfit to the anomalies; On
the other hand, when the inputs are diverse ((a) and (c)),
our method remains accurate while existing methods canfail. Axis represents three dimensional signals on dataset.
global and local representations into the final representation,
which is used for reconstruction. Based on the convolutional
autoencoder framework, AMSL can be easily trained in an
end-to-end manner. As shown in Fig. 1, while other methods
fail in face of the two challenges, our AMSL can correctly
perform anomaly detection.
This work makes the following three contributions:
1)

2)

3)

2

We propose AMSL to tackle the limited normal data
and feature representation challenges by adopting
self-supervised learning and memory networks, respectively.
We propose to learn both the global and local memories to enhance the representation ability and further
propose an adaptive memory fusion module to fuse
the global and local memories for final representations.
Extensive experiments on four public datasets
demonstrate the effectiveness of AMSL. Specificallyon the largest CAP dataset [4] with 900M+ instances,
AMSL significantly outperforms the best comparison approach by 4%+ in both accuracy and F1 score.
Moreover, AMSL is also robust against noise andremain both time and memory efficient.

R ELATED W ORK

Unsupervised anomaly detection has been studied for
decades. Traditional approaches include reconstruction
methods (e.g., PCA, Kernel PCA [10], [11]), clustering methods (e.g., GMM, K-means [12], [13]), and one-class learning
methods (e.g., OCSVM, SVDD [14], [15]). Deep learning
methods are also popular. They can be categorized into thereconstruction-based and the prediction-based methods.
Reconstruction-based methods focus on reducing the
expected reconstruction errors. For instance, Autoencoders(AEs) [5] are often utilized for anomaly detection by learning to reconstruct a given input. As the model is trained
exclusively on normal data, whenever it is not able toreconstruct a given input with equal quality compared to
the reconstruction of normal data, the instance is treatedas an anomaly. The LSTM Encoder-Decoder model [8] is
proposed to learn temporal representations of the timeseries via the LSTM networks and use reconstruction errors
to detect anomalies. Despite its effectiveness, LSTM does nottake spatial correlation into consideration. Convolutional
Autoencoder (CAE) [6], [7] is an important method for video
anomaly detection. It is able to capture the 2D image structure since its weights are shared among all locations in the
input image. Furthermore, since the Convolutional LSTM
(ConvLSTM) can model spatio-temporal correlations by using the convolutional layers instead of the fully connected
layers, later works [9], [16] add the ConvLSTM layers into
the autoencoder, which encodes the change of appearance
for normal data more effectiveness. Others like Variational
Autoenocders (VAEs) [17], [18], De-noising AutoEncoders(DAEs) [19], Deep belief networks (DBNs) [20], Replicatingneural network [21] and Robust Deep Autoencoder (RDA)[22] have also reported promising performance.
Prediction-based methods aim to predict one or more
continuous values. In anomaly detection, researchers [23],[24], [25] propose RNN-based forecasting models to predictvalues of the next time period, and minimize the meansquared error between predicted and future values as thecriterion for identifying anomalies. There have also been attempts to propose a forecasting model, which uses CNN and
RNN, namely LSTNet [26], to extract short-term local dependency patterns and long-term patterns for multivariate
time series analysis for anomaly detection. The GAN-based
method [27] adopts U-Net as the generator to predict thenext frame in a video, and leverages adversarial training todistinguish if the predicted result fake (i.e. abnormal events)
compared to the ground truth. However, these methods lacka reliable mechanism to learn representations of normal data
at a fine-grained level.
Feature representation learning is one important aspectof deep learning and machine learning, and good representations of input data are essential for the generalization
ability, interpretability and robustness of methods. Some recent works [28], [29] use image context and spatial-temporal
relationships to well capture feature correlations for video
object detection. Self-supervised learning (SSL) [30] is a
type of unsupervised learning paradigm that uses the data
itself to generate supervision to learn good representations.
For instance, in an image classification problem where mostof the images are unlabeled, we can still rotate it by different angles (e.g., 0◦ , 90◦ , 180◦ , 270◦ ), and then use theseangles as their labels. In this way, all samples are withlabels. By learning on this multi-class classification problem(the auxiliary task), the model can learn general featuresfrom these images that can be used for classification (themain task) later. There are self-supervised techniques acrosscomputer vision [31], [32], natural language processing [33],and speech recognition tasks [34]. In anomaly detection,[35], [36] use self-supervised visual representation learning
to learn the features of in-distribution (normal) samples.
The vanilla memory network is for question answering [37]. Generally, the RNNs or LSTM-based models capture long-term structure within sequences through local
memory cells. Since the memory in these models is unstable
over time, memory networks are proposed by using a global
memory with shared read and write functions. Consideringthat memory can record information stably, several works
adopt memory networks such as one-shot learning [38],31)[39], neural machine translation [40], anomaly detection[41], [42], [43]. In anomaly detection, the memory moduleaims to record various patterns of normal data compared toitems in the memory, thus distinguishing between normal
and abnormal data.

3

P ROPOSED A PPROACH

3.1

Problem statement

3)

Definition 1 (Multivariate time series). A multivariate timeseries can be represented as X = (x1 , x2 , · · · , xN )> ∈ RN ×V ,where xi ∈ RV is a signal with length V and N is the total
number of signals.12 y ∈ Y denotes the corresponding labels,
where Y = {1, · · · , K} represents normal data with K classes.
Definition 2 (Anomaly). A sample (xa , ya ) is called ananomaly if its label ya does not belong to any predefined classes,
i.e. ya ∈/ Y .3In this paper, we deal with the most challenging unsupervised anomaly detection problem, where only unlabeled
normal samples are available during training. This is a more
realistic setting since large-scale collection of anomalous
samples is often not practical.
3.2

Overview

We employ a convolutional autoencoder (CAE) as the basenetwork, which is widely used in existing anomaly detection literature [5], [6], [7]. An autoencoder (AE) is anunsupervised neural network combining an encoder fe anda decoder fd parameterized by θe and θd , respectively. The
encoder maps the high-dimensional input x ∈ RV into alatent representation z ∈ RF , where F  V . Then, thedecoder fd reconstructs the original input as x0 by mappingz back into the input space. The latent representation z andthe reconstructed input x0 can be respectively computed as:z = fe (x; θe ), x0 = fd (z; θd ).

(1)

The reconstruction error is calculated using the Mean
Squared Error (MSE):

LM SE = kx − x0 k22 ,2)

4)

Details of AMSL are presented in the following sections.
3.3

Self-Supervised learning

In this section, we introduce the self-supervised learning
module of AMSL that enables generalized feature representation learning for normal data. Compared to the potentiallylarge number of unseen anomalies that could be in any
forms of representations, the number of the normal training
data is relatively limited. Hence, anomaly detection models
trained on such limited normal samples tend to overfit.
However, it is prohibitively expensive to collect all theinfinite training data. To address this issue, we propose touse self-supervised learning to increase the model’s generalization ability.
Assuming that the instances are consistent before andafter basic data augmentations (i.e., normal data and abnormal data are still distinguishable after the same augmentations) [30], we design feature transformations on theoriginal data for self-supervision. Then, we train the model
to recognize the transformation type of a sample as its
auxiliary task. In the following experiments, as shown in
Fig. 3(a), we can observe that the instances after the same
augmentations can still identify normal and abnormal data.
Specifically, we propose to utilize six signal transformations
inspired by [45], which are described as follows:1)(2)

where k·k22 is the l2 -norm.
In this paper, we propose a novel Adaptive Memory
Network with Self-supervised Learning (AMSL) for unsupervised anomaly detection. AMSL consists of four novelcomponents as shown in Fig. 2: 1) a self-supervised learning
module, 2) a global memory module, 3) a local memory
module and 4) an adaptive fusion module. AMSL works
in the following four steps.
1. Each sensor signal may have different lengths due to different
protocols. In this case, they can still be processed into the same length
V by down-sampling or up-sampling.
2. Remark on the data dimension: In real applications, we often
employ a sliding window technique to construct samples in time series,
thus, x is a tensor with >2 dimensions. For simplicity, we denote its
flattened dimension as V .
3. Theoretically speaking, there is currently no unified definition ofan anomaly. In this paper, we follow the common definitions in existing
literature [9], [42], [44].

First, the encoder maps the raw time series signal
and its six transformations into a latent feature
space.
Then, for self-supervised learning, a multi-classclassifier is built to classify these feature types inorder to learn generalized feature representations.
Meanwhile, the features are also fed into the global
and local memory network modules to learn bothcommon and specific features.
Finally, the adaptive fusion module fuses these features to obtain new representation which is used forreconstruction.

2)

3)

4)

5)

6)

Noise: As noisy sensor signal may exist in the realworld, adding noise to signal can help model learn
more robust features against the noise. Here, thetransformation with Gaussian noise is implemented.
Reverse: This transformation reverses the samples
along the temporal dimension, resulting in the samples with the opposite time direction.
Permute: This transformation randomly perturbs
signal along the time dimension by slicing andswapping different time windows to generate new
samples. It aims to enhance the permutation invariant property of the resulting model.
Scale: Scaling changes the magnitude of signals
within a time window by multiplying a random
scalar. Here, we select [0.5, 0.8, 1.5, 2] as scalar values. The addition of scaled signals can help themodel learn scale-invariant patterns.
Negate: This transformation is a special type of scaling transformations. It is scaled by -1, resulting in a
mirror image of the input signals.
Smooth: This transformation applies the SavitzkyGolay (SG) method to smooth the signals. The

4

Self-Supervised Learning module

𝐿"#

FC
Logits

Input

×𝑅

𝒘
𝒛

Adaptive fusion module
𝒛 0 ×𝑅

Global memory module

×𝑅

Encoder~..
.

Output×𝑅..
.

Decoder

𝒛,

×𝑅

𝐿./#

𝐿&'()
×𝑅

Local memory module
𝒘
𝒛

~

..
.

×𝑅
×𝑅
..
.

New representations𝒛-: Forward operation: Loss operation~ : Similarity calculation: Matrix multiplication: Concatenation: Adaptive fusion

Fig. 2: The structure of the proposed AMSL. It consists of four components: self-supervised learning, global memory, local
memory and adaptive fusion. The notation “×R” denotes R copies where each one corresponds to one transformation.
Savitzky-Golay filter is a particular type of low-passfilters, well adapted for noisy signal smoothing.
To learn general feature representations from these transformations, the model is enforced to distinguish their transformation types by using the Cross-Entropy loss function:

LCE = −

RX

yi log (pi ) ,Given a query z ∈ RF (i.e. encoding), where F denotes thefilter size in the last layer of the encoder, the memory-guided
module outputs ẑ , which represents a weighted averageof memory items mi , by matching probabilities w ∈ RC
between the query z and memory matrix M as follows:ẑ = wM =(3)

wi mi ,

(4)

i=1

i=1

where R denotes the number of self-supervised learningclasses (R = 7 in this work by including all six transformations and the original signal). yi and pi are the pseudolabel and the predicted probability, respectively. A Softmax
activation function is applied to the probability before thecross-entropy loss.

C
X

where wi denotes the i-th entry of w. The weight vector w iscomputed based on the normalized similarity between thequery z and memory entry mi :exp (Score (z, mi )).
wi = PC
j=1 exp (Score (z, mj ))(5)

The Score function is implemented as the cosine similar3.4

Adaptive Memory Fusion Module

Traditional AE is negatively affected by noisy or unknown
training data, so it can also consistently reconstruct abnormal inputs too well [42], [46]. Therefore, the model fails
to learn representative features. To tackle this challenge,
we propose a adaptive memory fusion module to enhance theability of the model in distinguishing between normal andabnormal data via recording the prototypical patterns. In
the following, we will first introduce the memory network.
Then, we present our adaptive memory fusion module indetail.
3.4.1 Memory module
The memory module [41], [42] consists of a memory representation to represent the encoded patterns, and a memory
updating part to update the memory items based on thesimilarity of the memory items and the input z . Specifically,
the memory is instantiated as a matrix M ∈ RC×F , storing
C vectors of dimension F . Let the row vector mi , ∀i ∈ [C]denote the i-th row of M , where [C] denotes the set of
integers from 1 to N . Each mi denotes a memory item.

ity:

Score (z, mi ) =z · m>
i.
kzkkmi k(6)

In the training phase, the memory matrix can be updated
by the reconstruction loss function, thus forcing it to record
the normal characteristics. In the testing phase, the memory network outputs a representation with a combinationof all items, taking into account the multiple patterns ofnormal characteristics. Therefore, the normal instances canbe reconstructed well. The anomalies reconstructed using
the retrieved normal patterns in the memory module will
result in higher reconstruction errors.
3.4.2

Adaptive fusion module

We further propose an adaptive memory fusion network tolearn both the common and specific representations fromall the feature augmentations. Specifically, we propose theglobal memory module to learn the common representations
contained in all transformations, and the local memory
module to learn augmentation-specific representations foreach transformation. Finally, we propose a adaptive fusion

5

module to fuse these two levels of features into the final
representations that will be used for reconstruction. The
motivation is that we can capture the common patterns
for the normal data and its specific information (i.e. eachdifferent transformation) that is useful for the normal data
patterns, thus improving feature representations of normal
data at a fine-grained level.
We build the global memory module with a sharedmemory matrix. By using the encoded representation as aquery, the global memory module can record general itemsin the memory matrix. Through a shared memory module,
the outputs are obtained as:

zig = fg (fe (xi ; θe ); θg ), i ∈ [R],(7)

where fg (·) is the function of the global memory module.
[R] = {1, 2, · · · , R} and θg denote the shared parameters of
one global memory module.
We build R local memory modules for the raw data
and the six transformations. Each memory matrix records
the corresponding normal characteristics of transformation.
These outputs are obtained by the local memory modulesas:zil = fl (fe (xi ; θe ); θli ), i ∈ [R],(8)

where fl (·) is the function of the local memory module, andθli denotes the parameters of seven local memory modules.
It is intuitive that the common and specific features are
not equally important in representing a given instance. To
adaptively fuse these features, we use a feed-forward layerthat takes the features and a free variable r ∈ R as inputs togenerate the fused representations by weights α ∈ R2R (2weights for local and global memories, with R transformations in total). Note that we use Batch Normalization andsigmoid activation to normalize the weights and control
their values to be within the range of (0, 1). r is used
to increase randomness. Then, we get the adaptive fused
representations:
 g
zi
 g

z̃i = αi αil   , i ∈ [R],
(9)
zil
g

where αi , αil ∈ R denote the weights of common (global)
and specific (local) features, respectively.
The decoders concatenate z (encoding output) and z̃
(adaptive fusion output) as inputs to reconstruct the original
inputs. The reconstruction loss is defined by minimizing thel2 distance between the decoder output and the original
input as follows:

LM SE =RX

kfd (concat(z̃i , zi ); θdi ) − xi k22 .

(10)

i=1

To constrain the sparsity of the memory weight w to
avoid over-reconstruction of anomalies by the complex
combination of memory items, we adopt sparse loss byminimizing the entropy of w:

Lspar =

CX
i=1

−wi · log(wi ).

(11)

Algorithm 1 Learning procedure of AMSL
Input: Normal Dataset X = {X1 , X2 , · · · , XH }, hyperparameters λ1 , λ2 , variable r.
Output: Decision threshold µ and model parameter θ.
1: Transform each sample into X = {x1 , ..., xR } ∈RR×V ×N , where R = 7 including raw data and 6transformations;
2: Initialize all parameters;
3: while not converge do
4:
Z ← Encoder(X);5:p ← F C(Z);6:
Z l , w ← LM em(Z);7:
Z g , w ← GM em(Z);8:
Learn adaptive fuse weights α;
9:Z̃ ← F usion(Z l , Z g ; α);10:X 0 ← Decoder(Concat(
Z̃, Z))
;
PR
1 PH
0
11:
J(θ)
=
−
i=1 ( j=1 L(Xij , Xij )H
P2R PCPR
λ1 j=1 yij log (pij )−λ2 j=1 k=1 wijk ·log(wijk ));//Implementation of Eq. (12) for multivariate timeseries12:
Update model parameter θ using Eq. (12);13: end while
14: Calculate the decision threshold µ by the training samples;
15: return Optimal µ and θ .

3.5

Training and Inference

3.5.1 Training
By integrating the reconstruction loss in Eq. (10), the sparseloss in Eq. (11), and the self-supervised loss in Eq. (3) withtradeoff parameters λ1 , λ2 , we obtain the overall training
objective for AMSL which can be optimized in an end-toend manner:

J(θ) = LM SE + λ1 LCE + λ2 Lspar .

(12)

The training procedure of AMSL on multivariate timeseries is shown in Algorithm 1. Note that we need to takeof the multiple dimensions of the inputs in real implementation.
3.5.2 Inference
For the autoencoder-based models, it is generally assumed that the compression is different for instances ofdifferent categories. That is, if the training dataset only
contains normal instances, the reconstruction error becomes
higher for abnormal instances. Therefore, we can classify
these instances into “abnormal” or “normal” by their reconstruction error during inference phase.
Given normal samples as the training dataset X ={X1 , · · · , XH }. First, we need to construct the selfsupervised transformations. Let the corresponding decisionthreshold µ be the 99th percentile of Err(Xi ) on this training
set, where Err(Xi ) is the value of reconstruction loss function LM SE for Xi . In inference (or detection) process, thedecision rule is that if Err(Xi ) > µ, the testing sample Xi
in a sequence will be classified as “abnormal”; otherwise,
it will be classified as “normal”. Here, LMem and GMem
denote local memory module and global memory module,

6

Algorithm 2 Inference procedure of AMSL
Input: Testing dataset X = {X1 , X2 , · · · , XH }, X ∈RV ×N , threshold µ, model parameter θ, hyperparameters λ1 , λ2 .
Output: Label yi of Xi .
1: Transform each sample into X = {x1 , ..., xR } ∈RR×V ×N , where R = 7 including raw data and 6transformations;
2: Load the model trained parameters θ ;3: Z ← Encoder(X);
4: Z l ← LM em(Z);5: Z g ← GM em(Z);
6: Load α;7: Z̃ ← F usion(Z l , Z g ; α);8: X 0 ← Decoder(Concat(Z̃, Z));
9: Calculate
reconstruction
loss
Err(Xi )=PR0
j=1 L(Xij , Xij );10: if Err(Xi ) > µ then11:yi = “Abnormal”;
12: else
13:
yi = “Normal”;
14: end if
15: return yi .

respectively. The inference procedure of AMSL is shown in
Algorithm 2.

4

E XPERIMENTAL E VALUATION

4.1

Datasets

In the experiments, we adopt the following four datasets forevaluation as shown in TABLE 1.
TABLE 1: The detailed statistics of the four datasets
DatasetDSADS [47]
PAMAP2 [48]
WESAD [49]
CAP [4]

Application
Activity Recognition
Activity Recognition
Stress & Affect Detection
Sleep Stage Detection#Instance
1,140,000
2,844,868
63,000,000
921,700,000

#Dim
45
36
14
21

#Class19
18
3
8

TABLE 2: The selection of normal and abnormal classes on
DSADS and PAMAP2 datasets(a) DSADS
Activity
sitting, standing, lying, moving,walking, cycling, exercising, rowing
running, ascending stairs, descending stairs,rope jumping, playing basketballabnormal(b) PAMAP2
Activity
lying, sitting, standing, walking, cycling,
Nordic walking, vacuum cleaning, ironing
running, ascending stairs, descending stairs,rope jumping

Class
normalabnormalanomaly classes, including running, ascending stairs, descending stairs and rope jumping. The remaining categories
are used as the normal classes in experiments.
WESAD [49] is a public dataset for wearable stress andaffect detection, which contains physiological and motion
data of 15 subjects. We use sensor modalities of chest-worndevice including electrocardiogram, electrodermal activity,
electromyogram, respiration, body temperature, and threeaxis acceleration. In our experiments, we chose normal
emotional states (neutral, amusement) as normal classes andstress states as anomaly class.
CAP Sleep Database [4], which stands for the Cyclic
Alternating Pattern (CAP) database. It is a clinical dataset
from PhysioNet repository [55]. It is characterized by periodic physiological signals occurring during wake, S1-S4
and REM sleep stages. The signals include EEG, EOG,
EMG, EKG and respiration. There are 16 healthy subjects
and 92 patients in the database. In this task, we extracted
7 valid channels (ROC-LOC, C4-P4, C4-A1, F4-C4, P4-O2,
ECG1-ECG2 and EMG1-EMG2). For detecting sleep apnea
events, we chose healthy subjects as the normal class andthe patients with sleep-disordered breathing as the anomalyclass.
4.2

DSADS [47] consists of motion sensor data (i.e. accelerometer, gyroscope, magnetometer) of 19 activities ofdaily living and sports activities performed by 8 subjects.
To simulate normal and abnormal classes, we chose running, ascending stairs, descending stairs, rope jumping andplaying basketball as anomaly classes, while the remaining
categories are regarded as normal classes. These designatedanomaly activities are relatively intense and rare comparedto other activities. In TABLE 2(a), we describe the selection of normal and abnormal classes on DSADS datasets.
We chose these activities as the abnormal classes that are
relatively intense and rare activities compared to otheractivities, while the rest categories are defined as the normalclasses.
PAMAP2 [48] is a mobile dataset containing data of 18
different physical activities performed by 9 subjects wearing
3 inertial measurement units comprising an accelerator, agyroscope and a magnetometer. As shown in TABLE 2(b),we treat the classes with relatively small samples as theClassnormalComparison Methods

We compare AMSL against four popular traditional
anomaly detection methods, which can be applied afterfeature extraction:
•

•

•

•

KPCA [11], which is a non-linear extension of PCA
commonly used for anomaly detection. We adopt
Gaussian kernel in our experiments.
ABOD [50], which uses k nearest neighbors to approximate the complexity reduction. For an observation, the variance of its weighted cosine scores to allneighbors could be viewed as the abnormal score.
OCSVM [15], which adopts PCA for dimension reduction and employs the Gaussian kernel with abandwidth of 0.1.
HMM [51], which is applied after extracting features
and then it calculates the anomaly probability fromthe state sequence generated by the model.

We also compare it against seven deep unsupervised
methods:

7

TABLE 3: The comparison of mean precision, recall, F1 and accuracy of AMSL and other baselines. The best andsecond-best results are bold and underlined, respectively. We can see that AMSL significantly outperforms other methods.
DSADS dataset

MethodKernel PCA [11]ABOD [50]OCSVM [15]
HMM [51]
CNN-LSTM [52]
LSTM-AE [8]MSCRED [9]ConvLSTM-AE [16]ConvLSTM-COMP [16]BeatGAN [53]
MNAD-P [43]MNAD-R [43]GDN [54]UODA [23]AMSL (Ours)
Improvement

•

•

•

•

•

•

•

•

PAMAP2 dataset

WESAD dataset

CAP dataset

mPre

mRec

mF1

Acc

mPre

mRec

mF1

Acc

mPre

mRec

mF1

Acc

mPre

mRec

mF1

Acc

0.61840.68800.76080.69590.68450.84710.75400.81640.82290.95170.58160.83370.87060.86790.94077.28%

0.61820.65100.72770.69170.62700.77290.56020.69510.73790.56630.57830.76940.81510.82810.929810.17%

0.61830.66900.74390.69370.65450.80830.64280.75090.77810.71000.57990.80030.84190.84750.93528.77%

0.61860.65540.73120.69010.64250.78520.61920.71210.75180.78180.57210.78110.82510.83650.93329.67%

0.72360.86530.76000.69500.66800.86190.69970.73590.8844
0.79810.81980.83500.81290.89570.97889.44%

0.65790.90220.72040.65530.53920.79970.73010.73610.88420.74200.81760.83550.81040.85130.97138.71%

0.68920.88340.73970.67450.59680.82960.7146
0.73600.88430.76910.81860.83530.81160.87300.97509.07%

0.56450.89850.76790.57250.61310.84260.75170.73410.88440.83690.81350.83340.81230.88230.97709.26%

0.54960.87820.60920.61230.58830.23530.88500.97330.96260.75860.76000.74260.75200.76230.99532.20%

0.54950.8786
0.56310.60600.54400.47620.81240.96980.96290.50000.69380.66770.5434
0.6314
0.99492.51%

0.54950.87840.58520.60970.56530.31500.8471
0.9716
0.96270.60270.72540.70310.63090.69070.99512.35%

0.54860.87830.55180.60180.53180.45990.84200.97090.96190.51720.68490.65790.55900.61910.99512.42%

0.76030.78670.92670.82380.61590.71470.64100.81500.83670.52510.77420.81890.68310.75570.9771
5.04%

0.5847
0.63650.92590.80780.52170.62530.57840.81940.83770.50020.74890.82350.62370.5124
0.97364.77%

0.66110.70370.92630.81570.56490.66710.60810.81720.83720.51230.76130.82120.65200.61070.97534.90%

0.5892
0.63260.92570.80900.57620.6286
0.58190.81650.83940.84370.69600.78710.65690.51730.9756
4.99%

CNN-LSTM [52], which is a prediction-based method
built by firstly defining a network comprised of
Conv2D and MaxPooling2D layers ordered into astack of the required depth. The result is then fed
into the LSTM and FC layers as prediction.
LSTM-AE [8], which is a reconstruction-based
method using a single-layer LSTM on both encoderand decoder.
MSCRED [9], which is an encoder-decoder modelwith multi-scale matrices as inputs for multivariate
time series analysis.
ConvLSTM-COMPOSITE [16], which is a composite
encoder-decoder model with reconstruction and prediction task. We choose the “conditional” version
to build a single model called ConvLSTM-AE byremoving the forecasting decoder.
BeatGAN [53], which is a reconstruction-based
method with an adversarial generation approach as
regularization.
MNAD [43], which is an encoder-decoder model
based on a memory module for video anomaly detection. It has two variants: one with the prediction
task (MNAD-P), another with the reconstruction task
(MNAD-R).
GDN [54], which is a Graph-based neural network
to learn a graph of the dependence relationships
between sensors for anomaly detection.
UODA [23], which is a RNN-based network
for anomaly detection. We re-implemented it bycustomizing the number of layers and hyperparameters.

We re-implement the comparison methods based on
several open-source repositories4 as well as our own implementations. As for ours, there is a data pre-processing
stage where data is normalized, split into windows withlength V and transformed. The encoder runs with Conv2D→ M axpool → Conv2D → M axpool, i.e., Conv1- Conv2
with 32 and 64 kernels of size 4 × 4, and Maxpooling withsize 2 × 2. In adaptive memory fusion module, F = 64 andC = 800 are the best choice. Besides, we use a variable r as
the initial weight of adaptive fusion network with F C(2R)
4. https://pyod.readthedocs.io/en/stable/,https://github.com/
7fantasysz/MSCRED,https://github.com/cvlab-yonsei/MNAD,https://github.com/Vniex/BeatGAN, https://github.com/d-ailin/GDN→ BN → Sigmoid → M ultiply . These weights are multiplied into local and global feature representations by Eq. (9).
As the input of the decoder is by concatenating the output
of encoder and the output of memory module, the decoder
is asymmetric to the encoder that uses 4 Conv2DT ranspose
modules, with {128, 64, 32, 1} kernels of size 4 × 4 ineach layer, respectively. To compute classification error, theoutput of encoder is also incorporated into classification
network with Conv2D → F latten → F C(128) → Dropout→ F C(R), i.e. Conv with 1 kernel of size 4 × 4. AMSL
is trained in an end-to-end fashion using Keras [56] on
a TITAN XP GPU. The Adam optimizer is used to trainmodel for about 100 epochs with 32 or 64 batch size. The
learning rate is 0.001. And we set the hyper-parameters:
λ1 = 1 and λ2 = 0.0002 while parameter sensitivity analysis
is presented in later sections.
In practice, it is difficult to know the ground truth, andanomalous data points are rare. Hence, the semi-supervised
setting is a commonly used as evaluation method [42], [46],[57], the training set only contains the normal samples andhas no overlapping with the testing set. For each dataset, we
split the normal samples into training, validation, and testing sets with the ratio of 5 : 1 : 4. The model selection criterion, i.e., hyperparameters, used for tuning is the validation
error on the validation set. Besides, since most datasets havemore normal samples than anomalies, accuracy along is
insufficient for evaluation. Thus, for comprehensive evaluation, we adopt four evaluation metrics: mean precision, recall,
F1 score, and accuracy following existing literature [23], [46],
[57].
4.3

Results and Analysis

TABLE 3 reports the overall performance results on thesepublic datasets. It can be observed that the proposed AMSL
method achieves significantly superior performance overthe baseline methods in all the datasets. Specifically, compared with other methods, AMSL significantly improves
the F1 score by 9.07% on PAMAP2 dataset, 4.90% on CAP
dataset, 8.77% on DSADS dataset and 2.35% on WESAD
dataset. The same pattern goes for precision and recall.
Especially for the largest CAP dataset with over 900 Millon
samples, AMSL dramatically outperforms the second-bestbaseline (OCSVM) with an F1 score of 4.90%, indicating its
effectiveness.

8

For DSADS, PAMAP2 and CAP datasets (WESAD isrelatively easier to train than the other datasets), we findthat as the datasets become larger, the improvements tend
to decrease. This means that self-supervision is more usefulon small-scale datasets where it is hard to learn generalized
representations. This is in accordance with existing machinelearning conclusions. In addition, even if the samples are
relatively fewer in DSADS and PAMAP2 datasets but withmore categories, our AMSL still significantly outperforms
other methods by large margins, which indicate its abilityof dealing with the diversities in the limited training data.
Traditional methods perform differently on different
datasets since they are limited by the feature extraction
methods and the curse of dimensionality. In deep learning
methods, CNN-LSTM achieves the lowest F1 score, whichmeans CNN-LSTM alone is not enough to capture general
patterns and more regularizations are needed to improve its
performance. For the reconstruction-based methods (LSTMAE, MSCRED, and ConvLSTM-AE), their performances arelimited by the noise in the training data, which is likely
to mislead the model by making it hard to distinguishbetween the normal, abnormal, and noisy data. MNAD and
ConvLSTM are proposed for video data, which may not besuitable for multivariate time series. For BeatGAN, it haspoor performance on CAP and WESAD datasets since it
is prone to mode collapse and convergence problems onlarge-scale datasets. UODA performs reasonably well on
the PAMAP2 and DSADS datasets that relies on pre-training
denoising autoencoder (DAEs) and deep recurrent networks(RNNs) before fine-tuning. In fact, its performance is still
not optimal because RNN does not have the capability tomemorize long-term time series. GDN has a good performance on PAMAP2 and DSADS datasets. However, with
the increase of data, the running speed of the model slows
down and the accuracy also decreases affected by the graphstructure. For ConvLSTM-COMPOSITE model, it performs
better than most baselines. However, its efficiency may belimited due to the existence of two decoders in its structure.
Besides, we make confusion matrices to error analysis
for our proposed method AMSL. As shown in Fig. 4, we
find that the proportion of misclassification of normal data
is lower than that of abnormal data for most datasets. The
F1-scores of normal class and abnormal class are 93.99% and92.48% on DSADS dataset, the F1-scores of normal class andabnormal class are 98.23% and 96.74% on PAMAP2 dataset,the F1-scores of normal class and abnormal class are 99.49%
and 99.52% on WESAD dataset, the F1-scores of normalclass and abnormal class are 97.85% and 97.18% on CAP
dataset. The proposed AMSL method achieves significantly
superior performance in all the datasets, with an F1 score ofat least 93%.
4.4

Ablation StudyIn this section, we conduct ablation study to show theeffectiveness of the self-supervised learning, memory andadaptive fusion modules of AMSL. All experiments were
performed using the PAMAP2 dataset. TABLE 4 reports theperformance of different variants of AMSL. These results
show that the proposed self-supervised learning moduleand memory module can both increase the performance of

TABLE 4: Ablation results on the PAMAP2 dataset. “SSL”
denotes self-supervised learning module.
Variant
CAE
CAE + MemCAE + SSL
CAE + SSL + Mem
CAE + SSL + Ada Mem
CAE + Mem w/o Spar
CAE + SSL w/o SSL loss

mPre
0.87280.88080.93930.96160.97880.87130.8231mRec
0.87110.88190.93930.94200.97130.87230.8175mF1
0.87190.88130.93930.95170.97500.87190.8203

Acc
0.86760.88090.93640.9554
0.97700.87180.8205the base convolutional autoencoder (CAE) module. After
combining these two modules, it can be observed thatAMSL can achieve better performance than that without
the adaptive fusion module. Here, “Mem” denotes memorygfusion module. This means that zi and zil are fused in a
proportion of 1 : 1.
AMSL also consists of other components, such as selfsupervised loss and sparse loss. To demonstrate the effectiveness of these components, we conduct other ablation
studies shown in TABLE4 (last two rows). It is worth
noting that CAE + SSL without self-supervised loss (i.e.,
only using different augmentations) obtain a lower F1 scorethan other variants of the approach. This is because theaugmented data (via different transformations) complicatesthe distribution of normal data, thus leading to the underfitting problem. Therefore, it is necessary to classify thesetransformations using pseudo labels. Similarly, we observethat adding sparse loss further improves the performance.
4.5

Detailed Analysis

4.5.1 Self-Supervised Learning
Self-supervised learning helps the network learn generaland diverse features for the normal data, thus increasing
the generalization ability of the model to recognize the unseen normal and abnormal instances. In Fig. 3(a), we show
comparative performance analysis of each self-supervised
data transformation. This assessment helps us understandwhether model performance via jointly learning augmented
data is better than learning individual data. The experiments
were performed using the PAMAP2 dataset. The results
show that the overall performance is competitive except
for the noisy signals, thus it is beneficial to combine alltransformations for better generalization. In Section 4.10,
we discard the poorly performing transformations such as“Noise” and “Scale”. We observe that the F1 score and accuracy of AMSL decrease as transformations R decrease. This
demonstrate that it’s helpful to use different self-supervised
data transformations to increase the model’s generalization
ability.
4.5.2 Adaptive Memory Fusion module
In Fig. 3(b), we show the performance of CAE, GMSL, LMSL
and AMSL. GMSL and LMSL are global memory networkand local memory network with SSL. The experimental
results demonstrate that adaptive memory fusion network
achieves better performance than using single memory network (i.e. global and local memory module). AMSL achieves
a 1.86% higher F1-score than GMSL, and a 3.12% higher F1score than LMSL. The same goes for the precision and recall.

9

1

0.5

mPre mRec mF1

Weight0.95

NoisedNegated

0.9

Reversed

l
1
g
1
l
2
g
2

0.4

Raw data

0.3
0.2

Permuted

Smooth

0

All

20

0.8
CAE

(a) SSL transformations

l
5
g
5
l
6
g
6

l
7
g
7

100

120

0.1

0.85

Scale

l
3
g
3
l
4
g
4

GMSL LMSL AMSL(b) Adaptive memory fusion

40

60
80
Epoch

(c) Learned weights

Fig. 3: Analysis of AMSL from several key aspects on PAMAP2 dataset. (a) The effectiveness of each data transformationin self-supervised learning. (b) Comparison between local, global, and our adaptive memory fusion modules. (c) The
weights learned by our adaptive memory fusion module.
TABLE 5: Ablation results on all these datasets.
(a) DSADS
Variant
GMSL
LMSL
AMSL

mPre
0.88900.89150.9407mRec
0.85510.85230.9298mF1
0.87170.8716
0.9352

Acc
0.86240.86050.9332

mF1
0.9564
0.94380.9750Acc
0.95530.94330.9770mF1
0.9856
0.98570.9951

Acc
0.98510.98550.9951mF1
0.95970.95750.9753Acc
0.95980.95750.9756(b) PAMAP2(a) DSADS

Variant
GMSL
LMSL
AMSL

(b) PAMAP2

mPre
0.95580.9431
0.9788mRec
0.9571
0.9444
0.9713(c) WESAD

Variant
GMSL
LMSL
AMSL

mPre
0.98630.9864
0.9953

Variant
GMSL
LMSL
AMSL

mPre
0.96380.96200.9771mRec
0.98490.98500.9949(d) CAP(c) WESAD(d) CAPmRec
0.9556
0.9531
0.9736

Fig. 4: The confusion matrices on four datasets.
More results on different datasets are shown in TABLE 5.
Our method AMSL is always superior than GMSL and
LMSL. As mentioned above, AMSL finds the best weight
αl and αg values automatically. To observe the changes ofadaptive weights at the training stage, this experiment was
conducted using the PAMAP2 dataset. As shown in Fig. 3(c),it occurred at around the 70-th epoch when the valuesof adaptive weights stabilized. Here, α1 − α7 denote theweights of raw data and 6 transformations corresponding
to transformations in Fig. 3(a).
4.6

robustness of different methods to noisy data on PAMAP2
dataset. We manually control the noisy data ratio in the
training data. We inject Gaussian noise (µ = 0, σ = 0.3) ina random selection of samples with a ratio varying between
1% to 30%. We compare the performance of three methods:
UODA, ConvLSTM-Composite, and AMSL in Fig. 6(c). As
the noise increases, the performance of all methods decreases. Among them, AMSL remains significantly superior
to others, demonstrating its robustness to noisy data. The
result of precision and recall are presented in Fig. 6(a) and
Fig. 6(b).

Robustness to Noisy Data

In real-world applications, the collection of multivariate
time series data can be easily polluted with noise due tochanges in the environment or the data collection devices.
The noisy data bring critical challenges to the unsupervised
anomaly detection methods. In this section, we evaluate the

4.7

Percentage of Anomaly

Generally, the percentage of anomaly is significantly lower
than the normal range. Hence, we conduct experiments
on CAP dataset when the percentage of anomaly on thetesting set is 1%, 5%, 10%, 15%, 20%, 25% and 30%. In10Acceleration

Acceleration

Acceleration

Acceleration

AbnormalAbnormalAbnormalAcceleration

NormalNormal

Acceleration

Normal

Time

Time

Time

Time

Time

Time

MNAD:

MNAD:

MNAD:

MNAD:

MNAD:

MNAD:

UODA:

UODA:

UODA:

UODA:

UODA:

UODA:

Ours:

Ours:

Ours:

Ours:

Ours:

Ours:

Fig. 5: The visualization for samples in normal and abnormal class. Each column is the instance whether is correctly
detected by our method AMSL, MNAD and UODA. Axis1,2,3 represents three dimensional signals on DSADS dataset.

mRec

mPre

0.9
0.8
0.7
0.6
0

UODA
ConvLSTM-COMP
AMSL

1

1

0.9

0.9

0.8

0.8

0.7
0.6

0.01 0.02 0.05 0.1
Noise factor

0.5
0

0.3

(a) Precision

mF1

1

UODA
ConvLSTM-COMP
AMSL

0.7
0.6

0.01 0.02 0.05 0.1
Noise factor

0.3

0.5
0

UODA
ConvLSTM-COMP
AMSL

0.01 0.02 0.05 0.1
Noise factor(b) Recall

0.3

in two situations: (1) when a normal sample is not similar to
the majority of normal samples (i.e., overfitting) or (2) whenan abnormal sample is very similar to the normal sample(i.e., less powerful representations). This demonstrate that
our AMSL can effectively handle the problem about thediversity of samples.

(c) F1

4.9

Fig. 6: Robustness to noise.

Parameter sensitivity analysis

We consider three key parameters: 1) window length V ,2) size of memory matrix M , and 3) filter size F in thelast layer of encoder. These parameters are chosen from:V ∈ {64, 128, 236}, C ∈ {50, 200, 500, 800}, and F ∈{16, 32, 64}. Fig. 8 shows that AMSL is robust to different
parameter choices on PAMAP2 dataset.

1
0.8

F1

0.6
0.4
OCSVM
ConvLSTM-COMP
MNAD-R
AMSL

0.2
0
1

5

10
15
20
Anomaly(%)

25

30

Fig. 7: F1 score on the abnormal class with different
percentage of anomaly.
Fig. 7, we show the F1 score of the abnormal class using
different methods. We compare the performance of four
methods: OCSVM, ConvLSTM-COMPOSITE, MNAD-R and
AMSL. These methods have good performance for CAP
dataset in the above experiments. We can observe that as
the percentage of anomaly decreases, the F1 score of othermethods have decreased significantly, while our method still
remain stable. This demonstrates that our method achieves
high precision and recall on the abnormal class even if
the percentage of anomaly is very low on the testing set.
Therefore, we can come to a conclusion that our proposed
method AMSL has good stability even facing dataset imbalance problem.
4.8

Case Study

We present case studies of MNAD, UODA and our method
AMSL as shown in Fig. 5 by visualizing some normal
and abnormal classes. We choose three dimensional signals
on DSADS dataset. It can be shown that our AMSL cancorrectly classify these samples, while other methods fail1

mPre AMSL
mRec

mF1

0.975

mPre AMSL
mRec

0.96620.91450.95

mPre AMSL
mRec

mF1

1

0.9545

0.95

0.975

mF1

1
0.975

0.93580.96470.9277

0.9416

0.9

0.9

0.85

0.95

0.85

0.8

0.8

0.9

64
128
256
Window length50
200 500 800
Size of memory matrix

16
32
64
Filter size in the last layer of encoder(a) Window length V(b) Memory size C(c) Filter size F

Fig. 8: Parameter sensitivity analysis on PAMAP2 dataset.
We also provide the sensitivity analysis of LMSL and
GMSL. Fig. 9 (a-b) present the results of LMSL and GMSL
with different window length {64, 128, 236}. We observethat the choice of window length is critical to the method,which window length of 128 can achieve the best performance for PAMAP2 dataset. The second factor is thesize of memory matrix M ∈ RC×F that is set as C ∈{50, 200, 500, 800} and F = 64. Note that the dimensionof F is equal to the filter size in the last layer of encoder. As
shown Fig. 9 (c-d), they indicate that by increasing the sizeof C , the performance improves until C reaches nearly 800.
Fig. 9 (e-f) show the results for F ∈ {16, 32, 64}. The filter
size in the last layer of encoder obtains the best performance
when F = 64. We use encoder to store more informationand contribute it to reconstruct the latent features.
Moreover, we adjust the hyperparameters λ1 and λ2 inthe loss function (Eq. (12)). As shown in Fig. 9 (g-h), we
observe that λ1 = 1 and λ2 = 0.0002 are the best choice inour experiments.
The selection of the threshold µ in Section 3.5.2 is also
an important issue for which we conduct experiments to

11
LMSL

GMSL

1

1
mPre

mRec

mPre

mF1

0.04

LMSL

mRec

Reconstruction loss
Self-supervised loss

1

mF1

mPre

mRec

mF1

0.9564
0.94380.95

0.95

0.94290.95

0.94380.89070.9

0.9
0.86450.846

0.85070.8394

0.85

0.85

0.85

0.8

0.8

0.8

64
128
256
Window length

64
128
256
Window length50
200 500 800
Size of memory matrix(a) LMSL(b) GMSL(c) LMSL

GMSL
1

mPre

LMSL

mRec

0.9252

mPre

mRec

mF1

mPre

mRec

MSCRED

20

40
Epoch

24.37

MNAD

4.51

AMSL

2.89

60

0

(a) Convergence

mF1

0.9564

29.57

0.02

0

1

46.87

UODA

0.01

GMSL

1

mF1

0.9538

0.95

Loss

0.9

Test time (ms)
COMP

0.03

0.9207

10

20

30

40

50

(b) Inference time

0.92930.9564

0.9

0.95

0.95

0.94380.93920.9455

Fig. 10: Convergence and inference time of AMSL.

0.936
0.9283

0.85
0.8

0.9

0.9

50
200 500 800
Size of memory matrix

16
32
64
Filter size in the last layer of encoder

16
32
64
Filter size in the last layer of encoder(d) GMSL(e) LMSL(f) GMSLAMSL
1Method
MNAD
MSCREDUODAAMSL(R=7)
AMSL(R=6)
AMSL(R=5)
AMSL(R=4)
AMSL(R=3)

AMSL
1

0.975
0.9247

0.94110.975

0.9605
0.935

0.9

0.9
0.80350.8262

0.8

0.8
mPre

mRec

mF1

0.7
0.1

0.5

1

2

(g) λ1

mPre

mRec

mF1

0.7
10-2

10-3

10-4 2x10-4(h) λ2

Fig. 9: Other parameter sensitivity analysis on PAMAP2
dataset.
TABLE 6: Experimental evaluation of threshold selection
for AMSL.
Decision Rules

mPre

mRec

mF1

Acc

THR90th percentile
THR95th percentile
THR99th percentile

0.9164
0.94860.97880.94010.95970.97130.9281
0.95410.97500.92840.95680.9770compare different thresholds for α-percentiles: 90, 95 and
99. As shown in TABLE 6, we find that the 99th percentile
can estimate the optimal threshold. Therefore, we set theanomaly detection threshold to the 99 percentile.
4.10

TABLE 7: Comparison of model parameters.

Convergence, Time and Space Complexity

Fig. 10(a) shows the convergence of reconstruction loss withmemory modules along with the self-supervised loss. In
conclusion, AMSL can be applied more effectively with afast and stable convergence performance.
We also evaluate the inference time of AMSL and otherstrong baselines on DSADS dataset. Here, “COMP” means
ConvLSTM-COMPOSITE. As shown in Fig. 10(b), in addition to achieving the best performance, our method requires
shorter running time than most other methods.
Besides, according to TABLE 7 evaluated on DSADS
dataset, the number of parameters and model size of AMSL
are relatively smaller than most other methods. We also
show that by reducing the model parameters through controlling the self-supervised data transformations R. We discard the poorly performing transformations in TABLE 3(a).
AMSL(R=6) discards the poorly performing transformation“Noise”, AMSL(R=5) discards “Noise” and “Scale” transformations, AMSL(R=4) discards “Noise”, ”Scale” and “Permuted” transformations, and AMSL(R=3) discards “Noise”,“Scale”, “Permuted” and “Reversed” transformations. We

F1
0.80030.64280.84750.93520.92980.92420.91380.9099ACC
0.78110.61920.83650.93320.92730.92330.91350.9096#Params
3.8M
3.6M
1.9M
3.3M
2.8M
2.4M
1.9M
1.5M

Model size
14.7MB
13.8MB
7.3MB
12.7MB
10.9MB
9.2MB
7.4MB
5.7MB

observe that our AMSL still achieves the best F1 and accuracy scores. On other datasets, the conclusions are alsosimilar. This makes our method flexible in real applications.

5

C ONCLUSIONS AND F UTURE W ORK

In this paper, we proposed an Adaptive Memory Networkwith Self-supervised Learning (AMSL) for unsupervised
anomaly detection on multivariate time series signals. To
enhance the model’s generalization ability towards unseen
anomalies, we proposed to use the self-supervised learning
module to learn diverse normal patterns and an adaptive
memory fusion network to learn rich feature representations
by the global and local memory modules. Experiments onfour public datasets demonstrate that our methods significantly outperforms existing approaches in terms of accuracy,
generalization, and robustness.
In the future, we plan to extend AMSL to other modalities such as images and videos for unsupervised anomaly
detection. In addition, we also plan to develop more efficienttraining algorithms and pursue the theoretical analysis of
our method.

ACKNOWLEDGMENT
This work is supported by the National Key Researchand Development Plan of China (No. 2020YFC2007104),Natural Science Foundation of China (No. 61972383, No.
61902377, No. 61902379), Science and Technology Service
Network Initiative, Chinese Academy of Sciences (No. KFJSTS-QYZD-2021-11-001), the National Research Foundation,
Singapore under its AI Singapore Programme (AISG Award
No. AISG2-RP-2020-019), the RIE 2020 Advanced Manufacturing and Engineering (AME) Programmatic Fund (No.
A20G8b0102), Singapore, and the Nanyang Assistant Professorship (NAP).

12

R EFERENCES[1]

B. Kiran, D. Thomas, and R. Parakkal, “An overview of deeplearning based methods for unsupervised and semi-supervised
anomaly detection in videos,” Journal of Imaging, vol. 4, no. 2, p. 36,
2018.
[2] Y. Mirsky, T. Doitshman, Y. Elovici, and A. Shabtai, “Kitsune: anensemble of autoencoders for online network intrusion detection,”
arXiv preprint arXiv:1802.09089, 2018.
[3] M. C. Chuah and F. Fu, “Ecg anomaly detection via time series
analysis,” in International Symposium on Parallel and Distributed
Processing and Applications. Springer, 2007, pp. 123–135.
[4] M. G. Terzano, L. Parrino, A. Smerieri, R. Chervin, S. Chokroverty,
C. Guilleminault, M. Hirshkowitz, M. Mahowald, H. Moldofsky,
A. Rosa et al., “Atlas, rules, and recording techniques for thescoring of cyclic alternating pattern (cap) in human sleep,” Sleep
medicine, vol. 3, no. 2, pp. 187–199, 2002.
[5] M. Sakurada and T. Yairi, “Anomaly detection using autoencoders
with nonlinear dimensionality reduction,” in Proceedings of the
MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data
Analysis. ACM, 2014, p. 4.
[6] M. Gutoski, N. M. R. Aquino, M. Ribeiro, E. Lazzaretti, and
S. Lopes, “Detection of video anomalies using convolutional
autoencoders and one-class support vector machines,” in XIII
Brazilian Congress on Computational Intelligence, 2017, 2017.
[7] M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S.
Davis, “Learning temporal regularity in video sequences,” in
CVPR, 2016, pp. 733–742.
[8] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and
G. Shroff, “Lstm-based encoder-decoder for multi-sensor anomaly
detection,” arXiv preprint arXiv:1607.00148, 2016.
[9] C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng,
J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network
for unsupervised anomaly detection and diagnosis in multivariate
time series data,” in Proceedings AAAI, vol. 33, 2019, pp. 1409–1416.
[10] R. Paffenroth, P. Du Toit, R. Nong, L. Scharf, A. P. Jayasumana, and
V. Bandara, “Space-time signal processing for distributed patterndetection in sensor networks,” IEEE Journal of Selected Topics in
Signal Processing, vol. 7, no. 1, pp. 38–49, 2013.
[11] H. Hoffmann, “Kernel pca for novelty detection,” Pattern recognition, vol. 40, no. 3, pp. 863–874, 2007.
[12] R. Laxhammar, G. Falkman, and E. Sviestins, “Anomaly detection
in sea traffic-a comparison of the gaussian mixture model and thekernel density estimator,” in 2009 12th International Conference on
Information Fusion. IEEE, 2009, pp. 756–763.
[13] L. J. Latecki, A. Lazarevic, and D. Pokrajac, “Outlier detection withkernel density functions,” in International Workshop on MLDM.
Springer, 2007, pp. 61–75.
[14] A. Banerjee, P. Burlina, and C. Diehl, “A support vector method foranomaly detection in hyperspectral imagery,” IEEE Transactions on
Geoscience and Remote Sensing, vol. 44, no. 8, pp. 2282–2291, 2006.
[15] J. Ma and S. Perkins, “Time-series novelty detection using oneclass support vector machines,” in Proceedings of the International
Joint Conference on Neural Networks, 2003., vol. 3. IEEE, 2003, pp.
1741–1745.
[16] J. R. Medel and A. Savakis, “Anomaly detection in video using predictive convolutional long short-term memory networks,”
arXiv preprint arXiv:1612.00390, 2016.
[17] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special Lecture on IE, vol. 2,
no. 1, 2015.
[18] H. Xu, W. Chen, N. Zhao, Z. Li, J. Bu, Z. Li, Y. Liu, Y. Zhao, D. Pei,
Y. Feng et al., “Unsupervised anomaly detection via variational
auto-encoder for seasonal kpis in web applications,” in Proceedingsof the 2018 World Wide Web Conference. International World Wide
Web Conferences Steering Committee, 2018, pp. 187–196.
[19] D. Xu, Y. Yan, E. Ricci, and N. Sebe, “Detecting anomalous eventsin videos by learning deep representations of appearance andmotion,” Computer Vision and Image Understanding, vol. 156, pp.
117–127, 2017.
[20] D. Wulsin, J. Blanco, R. Mani, and B. Litt, “Semi-supervised
anomaly detection for eeg waveforms using deep belief nets,”
in 2010 Ninth International Conference on Machine Learning andApplications. IEEE, 2010, pp. 436–441.
[21] S. Hawkins, H. He, G. Williams, and R. Baxter, “Outlier detection
using replicator neural networks,” in International Conference on
Data Warehousing and Knowledge Discovery. Springer, 2002, pp.
170–180.

[22] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust
deep autoencoders,” in KDD. ACM, 2017, pp. 665–674.
[23] W. Lu, Y. Cheng, C. Xiao, S. Chang, S. Huang, B. Liang, andT. Huang, “Unsupervised sequential outlier detection with deeparchitectures,” IEEE transactions on image processing, vol. 26, no. 9,
pp. 4321–4330, 2017.
[24] P. Filonov, A. Lavrentyev, and A. Vorontsov, “Multivariate industrial time series with cyber-attack simulation: Fault detection using an lstm-based predictive data model,” arXiv preprint
arXiv:1612.06676, 2016.
[25] T. Ergen, A. H. Mirza, and S. S. Kozat, “Unsupervised and semisupervised anomaly detection with lstm neural networks,” arXiv
preprint arXiv:1710.09207, 2017.
[26] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-andshort-term temporal patterns with deep neural networks,” in The
41st International ACM SIGIR Conference on Research & Development
in Information Retrieval. ACM, 2018, pp. 95–104.
[27] W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction for
anomaly detection–a new baseline,” in CVPR, 2018, pp. 6536–6545.
[28] Z. Li, J. Tang, and T. Mei, “Deep collaborative embedding forsocial image understanding,” IEEE transactions on pattern analysis
and machine intelligence, vol. 41, no. 9, pp. 2070–2083, 2018.
[29] Z. Zhu and Z. Li, “Online video object detection via local and midrange feature propagation,” in Proceedings of the 1st International
Workshop on Human-centric Multimedia Analysis, 2020, pp. 73–82.
[30] L. Jing and Y. Tian, “Self-supervised visual feature learning withdeep neural networks: A survey,” IEEE Transactions on PatternAnalysis and Machine Intelligence, 2020.
[31] M. Yang, Y. Li, Z. Huang, Z. Liu, P. Hu, and X. Peng, “Partially view-aligned representation learning with noise-robust contrastive loss,” in Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, 2021, pp. 1134–1143.
[32] Y. Lin, Y. Gou, Z. Liu, B. Li, J. Lv, and X. Peng, “Completer:
Incomplete multi-view clustering via contrastive prediction,” inProceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2021, pp. 11 174–11 183.
[33] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart:
Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” arXiv preprint
arXiv:1910.13461, 2019.
[34] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro,
J. Trmal, and Y. Bengio, “Multi-task self-supervised learning forrobust speech recognition,” in ICASSP 2020-2020 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2020, pp. 6989–6993.
[35] R. Ali, M. U. K. Khan, and C. M. Kyung, “Self-supervised representation learning for visual anomaly detection,” arXiv preprint
arXiv:2006.09654, 2020.
[36] S. Wang, Y. Zeng, X. Liu, E. Zhu, J. Yin, C. Xu, and M. Kloft,“Effective end-to-end unsupervised outlier detection via inlier priority of discriminative network,” in Advances in Neural Information
Processing Systems, 2019, pp. 5962–5975.
[37] S. Sukhbaatar, J. Weston, R. Fergus et al., “End-to-end memory
networks,” in Advances in neural information processing systems,
2015, pp. 2440–2448.
[38] Q. Cai, Y. Pan, T. Yao, C. Yan, and T. Mei, “Memory matching
networks for one-shot image recognition,” in CVPR, 2018, pp.
4080–4088.
[39] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap,“Meta-learning with memory-augmented neural networks,” inInternational conference on machine learning, 2016, pp. 1842–1850.
[40] M. Wang, Z. Lu, H. Li, and Q. Liu, “Memory-enhanced decoder
for neural machine translation,” arXiv preprint arXiv:1606.02003,
2016.
[41] C. Zhang, Y. Wang, X. Zhao, Y. Guo, G. Xie, C. Lv, and B. Lv,“Memory-augmented anomaly generative adversarial network
for retinal oct images screening,” in 2020 IEEE 17th International
Symposium on Biomedical Imaging (ISBI). IEEE, 2020, pp. 1971–1974.
[42] D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh,
and A. v. d. Hengel, “Memorizing normality to detect anomaly:
Memory-augmented deep autoencoder for unsupervised anomaly
detection,” in Proceedings of the IEEE International Conference on
Computer Vision, 2019, pp. 1705–1714.
[43] H. Park, J. Noh, and B. Ham, “Learning memory-guided normality
for anomaly detection,” in CVPR, 2020, pp. 14 372–14 381.

13[44] L. Shen, Z. Li, and J. Kwok, “Timeseries anomaly detection using temporal hierarchical one-class network,” Advances in Neural
Information Processing Systems, vol. 33, 2020.
[45] A. Saeed, T. Ozcelebi, and J. Lukkien, “Multi-task self-supervised
learning for human activity detection,” Proceedings of the ACM on
Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 3, no. 2,
pp. 1–30, 2019.
[46] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho,and H. Chen, “Deep autoencoding gaussian mixture model for
unsupervised anomaly detection,” in ICLR, 2018.
[47] K. Altun, B. Barshan, and O. Tunçel, “Comparative study onclassifying human activities with miniature inertial and magnetic
sensors,” Pattern Recognition, vol. 43, no. 10, pp. 3605–3620, 2010.
[48] A. Reiss and D. Stricker, “Introducing a new benchmarked dataset
for activity monitoring,” in 2012 16th International Symposium on
Wearable Computers. IEEE, 2012, pp. 108–109.
[49] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven, “Introducing wesad, a multimodal dataset for wearable
stress and affect detection,” in Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018, pp. 400–408.
[50] H.-P. Kriegel, M. Schubert, and A. Zimek, “Angle-based outlier
detection in high-dimensional data,” in SIGKDD. ACM, 2008,
pp. 444–452.
[51] S. S. Joshi and V. V. Phoha, “Investigating hidden markov models
capabilities in anomaly detection,” in Proceedings of the 43rd annual
Southeast regional conference-Volume 1. ACM, 2005, pp. 98–103.
[52] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent
convolutional networks for visual recognition and description,” in
CVPR, 2015, pp. 2625–2634.
[53] B. Zhou, S. Liu, B. Hooi, X. Cheng, and J. Ye, “Beatgan: Anomalous
rhythm detection using adversarially generated time series.” inIJCAI, 2019, pp. 4433–4439.
[54] A. Deng and B. Hooi, “Graph neural network-based anomaly
detection in multivariate time series,” in Proceedings of the AAAI
Conference on Artificial Intelligence, vol. 35, no. 5, 2021, pp. 4027–
4035.
[55] “Cap sleep database,” 2012. [Online]. Available: https://physionet.org/content/capslpdb/1.0.0/[56] N. Ketkar, “Introduction to keras,” in Deep learning with Python.
Springer, 2017, pp. 97–111.
[57] S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured
energy based models for anomaly detection,” arXiv preprint
arXiv:1605.07717, 2016.