1

Unsupervised Deep Anomaly Detection for
Multi-Sensor Time-Series Signals

arXiv:2107.12626v2 [cs.AI] 2 Aug 2021

Yuxin Zhang, Yiqiang Chen, Senior Member, IEEE, Jindong Wang, and Zhiwen Pan, Member, IEEE
Abstract—Nowadays, multi-sensor technologies are applied in many fields, e.g., Health Care (HC), Human Activity Recognition(HAR), and Industrial Control System (ICS). These sensors can generate a substantial amount of multivariate time-series data.
Unsupervised anomaly detection on multi-sensor time-series data has been proven critical in machine learning researches. The key
challenge is to discover generalized normal patterns by capturing spatial-temporal correlation in multi-sensor data. Beyond thischallenge, the noisy data is often intertwined with the training data, which is likely to mislead the model by making it hard to distinguishbetween the normal, abnormal, and noisy data. Few of previous researches can jointly address these two challenges. In this paper, we
propose a novel deep learning-based anomaly detection algorithm called Deep Convolutional Autoencoding Memory network(CAE-M). We first build a Deep Convolutional Autoencoder to characterize spatial dependence of multi-sensor data with a Maximum
Mean Discrepancy (MMD) to better distinguish between the noisy, normal, and abnormal data. Then, we construct a Memory Network
consisting of linear (Autoregressive Model) and non-linear predictions (Bidirectional LSTM with Attention) to capture temporaldependence from time-series data. Finally, CAE-M jointly optimizes these two subnetworks. We empirically compare the proposed
approach with several state-of-the-art anomaly detection methods on HAR and HC datasets. Experimental results demonstrate thatour proposed model outperforms these existing methods.
Index Terms—Unsupervised anomaly detection, Multi-sensor time series, Convolutional autoencoder, Attention based BiLSTM.

F1I NTRODUCTIONANOMALY detection has been one of the core researchareas in machine learning for decades, with wide applications such as cyber-intrusion detection [1], medical care[2], sensor networks [3], video anomaly detection [4] and soon. Anomaly detection seems to be a simple two-categoryclassification, i.e., we can learn to classify the normal or
abnormal data. However, it is also faced with the following
challenges. First, training data is highly imbalanced since
the anomalies are often extremely rare in a dataset comparedto the normal instances. Standard classifiers try to maximize
accuracy in classification, so it often falls into the trap ofoverlapping problem, which means that the model classifies
the overlapping region as belonging to the majority classwhile assuming the minority class as noise. Second, there isno easy way for users to manually label each training data,
especially the anomalies. In many cases, it is prohibitively
hard to represent all types of anomalous behaviors. Due toabove challenges, there is a growing trend to use unsupervised learning approaches for anomaly detection comparedwith semi-supervised and supervised learning approaches
since unsupervised methods can handle the imbalanced and

•

•

•

Y. Zhang is with Global Energy Interconnection Development and Cooperation Organization, Xicheng District, Beijing, China, and Beijing
Key Laboratory of Mobile Computing and Pervasive Device, Instituteof Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences, Beijing, China. E-mail: yuxinzhang@geidco.org.
Y. Chen and Z. Pan are with Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese
Academy of Sciences and University of Chinese Academy of Sciences,
Beijing, China. Y. Chen is also with Peng cheng Laboratory (PCL). Email: {yqchen, pzw}@ict.ac.cn (Corresponding author: Yiqiang Chen).
J. Wang is with Microsoft Research Asia, Beijing, China, also correspondence. E-mail: Jindong.Wang@microsoft.com.

Manuscript received April 19, 2005; revised August 26, 2015.

unlabeled data in a more principled way [5]–[9].
Nowadays, the prevalence of sensors in machine learning and pervasive computing research areas such as HealthCare (HC) [10], [11] and Human Activity Recognition(HAR) [12], [13] generate a substantial amount of multivariate time-series data. These learning algorithms based on
multi-sensor time-series signals give priority to dealing withspatial-temporal correlation of multi-sensor data. Many approaches for spatial-temporal dependency amongst multiple sensors [14]–[16] have been studied. It seems intuitive toapply previous unsupervised anomaly detection methods
on multi-sensor time-series data. Unfortunately, there arestill several challenges.
First, anomaly detection in spatial-temporal domain
becomes more complicated due to the temporal component in time-series data. Conventional anomaly detection
techniques such as PCA [17], k-means [18], OCSVM [19]and Autoencoder [20] are unable to deal with multivariate time-series signals since they cannot simultaneously
capture the spatial and temporal dependencies. Second,
these reconstruction-based models such as Convolutional
AutoEncoders (CAEs) [21] and Denoising AutoEncoders(DAEs) [22] are usually used for anomaly detection. It
is generally assumed that the compression of anomalous
samples is different from that on normal samples, and thereconstruction error becomes higher for these anomalous
samples. In reality, being influenced by the high complexityof model and the noise of data, the reconstruction error
for the abnormal input could also be fit so well by thetraining model [23], [24]. That is, the model is robust to
noise and anomalies. Third, in order to reduce the dimensionality of multi-sensor data and detect anomalies, twostep approaches are widely adopted. As for the drawbackof some works [25], [26], the joint performance of two

2baseline models can easily get stuck in local optima, since
two models are trained separately.
In order to solve the above three challenges, this paper
presents a novel unsupervised deep learning based anomaly
detection approach for multi-sensor time-series data called
Deep Convolutional Autoencoding Memory network (CAEM). The CAE-M network composes of two main subnetworks: characterization network and memory network.
Specifically, we employ deep convolutional autoencoder
as feature extraction module, with attention-based Bidirectional LSTMs and Autoregressive model as forecasting module. By simultaneously minimizing reconstruction error andprediction error, the CAE-M model can be jointly optimized.
During the training phase, the CAE-M model is trained to
explicitly describe the normal pattern of multi-sensor timeseries data. During the detection phase, the CAE-M model
calculate the compound objective function for each captured
testing data. Through combining these errors as a composite
anomaly score, a fine-grained anomaly detection decision
can be made. To summarize, the main contributions of thispaper are four-fold:
1) The proposed composite model is designed to characterize complex spatial-temporal patterns by concurrently
performing the reconstruction and prediction analysis. In
reconstruction analysis, we build Deep Convolutional Autoencoder to fuse and extract low-dimensional spatial features from multi-sensor signals. In prediction analysis,
we build Attention-based Bidirectional LSTM to capturecomplex temporal dependencies. Moreover, we incorporate
Auto-regressive linear model in parallel to improve therobust and adapt for different use cases and domains.
2) To reduce the influence of noisy data, we improve
Deep Convolutional Autoencoder with a Maximum Mean
Discrepancy (MMD) penalty. MMD is used to encourage
the distribution of the low-dimensional representation toapproximate some target distribution. It aims to make the
distribution of noisy data close to the distribution of normal training data, thereby reducing the risk of overfitting.
Experiments demonstrate that it is effective to enhance therobustness and generalization ability of our method.
3) The CAE-M is an end-to-end learning model that
two sub-networks can co-optimize by a compound objective function with weight coefficients. This single-stage
approach can not only streamline the learning procedure foranomaly detection, but also avoid the model getting stuck
in local minimum through joint optimization.
4) Experiments on three multi-sensor time-series datasets
demonstrate that CAE-M model has superior performanceover state-of-the-art techniques. In order to further verify
the effect of our proposed model, fine-grained analysis,
effectiveness evaluation, parameter sensitivity analysis andconvergence analysis show that all the components of CAEM together leads to the robust performance on all datasets.
The rest of the paper is organized as follows. Section
2 provides an overview of existing methods for anomaly
detection. Our proposed methodology and detailed framework is described in Section 3. Performance evaluation andanalysis of experiment is followed in Section 4. Finally,
Section 5 concludes the paper and sketches directions forpossible future work.

2

R ELATED W ORK

Anomaly detection has been studied for decades. Based onwhether the labels are used in the training process, they
are grouped into supervised, semi-supervised and unsupervised anomaly detection. Our main focus is the unsupervised setting. In this section, we demonstrate various typesof existing approaches for unsupervised anomaly detection,
which can be categorized into traditional anomaly detection
and deep anomaly detection.
2.1

Traditional anomaly detection

Conventional methods can be divided into three categories.
1) Reconstruction-based methods are proposed to represent
and reconstruct accurately normal data by a model, forexample, PCA [17], Kernel PCA [27], [28] and Robust PCA[29]. Specifically, RPCA is used to identify a low rank representation including random noise and outliers by using
a convex relaxation of the rank operator; 2) Clustering
analysis is used for anomaly detection, such as Gaussian
Mixture Models (GMM) [30], k-means [18] and Kernel
Density Estimator (KDE) [31]. They cluster different data
samples and find anomalies via a predefined outlierness
score; 3) the methods of one-class learning model are also
widely used for anomaly detection. For instance, One-Class
Support Vector Machine (OCSVM) [19] and Support Vector
Data Description (SVDD) [32] seek to learn a discriminative
hypersphere surrounding the normal samples and thenclassify new data as normal or abnormal.
It is notable that these conventional methods foranomaly detection are designed for static data. To capture
the temporal dependencies appropriately, Autoregression(AR) [33], Autoregressive Moving Average (ARMA) [34]and Autoregressive Integrated Moving Average (ARIMA)
model [35] are widely used. These models represent timeseries that are generated by passing the input through alinear or nonlinear filter which produces the output at any
time using the previous output values. Once we have theforecast, we can use it to detect anomalies and compare withgroundtruth. Nevertheless, AR model and its variants are
rarely used in multi-sensor multivariate time series due totheir high computational cost.
2.2

Deep anomaly detection

In deep learning-based anomaly detection, the reconstruction models, forecasting models as well as composite models
will be discussed.
2.2.1 Reconstruction models
The reconstruction model focuses on reducing the expectedreconstruction error by different methods. For instance,
Autoencoders [20] are often utilized for anomaly detection
by learning to reconstruct a given input. The model istrained exclusively on normal data. Once it is not able toreconstruct the input with equal quality compared to thereconstruction of normal data, the input sequence is treatedas anomalous data. LSTM Encoder-Decoder model [36] is
proposed to learn temporal representation of the input timeseries by LSTM networks and use reconstruction error todetect anomalies. Despite its effectiveness, LSTM does not3take spatial correlation into consideration. Convolutional
Autoencoders (CAEs) [21] are an important method of video
anomaly detection, which are able of capturing the 2D
image structure since the weights are shared among all locations in the input image. Furthermore, since Convolutional
long short-term memory (ConvLSTM) can model spatialtemporal correlations by using convolutional layers insteadof fully connected layers, some researchers [15], [37] add
ConvLSTM layers to autoencoder, which better encodes thechange of appearance for normal data.
Variational Autoenocders (VAEs) are a special form ofautoencoder that models the relationship between two random variables, latent variable z and visible variable x. A
prior for z is usually multivariate unit Gaussian N (0, I). For
anomaly detection, authors [38] define the reconstruction
probability that is the average probability of the original
data generating from the distribution. Data points withhigh reconstruction probability is classified as anomalies,
vice versa. Others like Denoising AutoEncoders (DAEs)[22], Deep Belief Networks (DBNs) [39] and Robust Deep
Autoencoder (RDA) [40] have also been reported good
performance for anomaly detection.
2.2.2 Forecasting models
The forecasting model can also be used for anomaly detection. It aims to predict one or more continuous values, e.g.
forecasting the current output values xt for the past p values[xt−p , ..., xt−2 , xt−1 ]. RNN and LSTM is the standard model
for sequence prediction. In the work [41], [42], authors
perform anomaly detection by using RNN-based forecastingmodels to predict values for the next time period andminimize the mean squared error (MSE) between predicted
and future values. Recently, there have also been attemptedto perform anomaly detection using other feed-forward
networks. For instance, Shalyga et al. [43] develop Neural
Network (NN) based forecasting approach to early anomaly
detection. Kravchik and Shabtai [44] apply different variantsof convolutional and recurrent networks to perform forecasting model. And the results show that 1D convolutional
networks obtain the best accuracy for anomaly detection inindustrial control systems. In another work [45], Lai et al.
propose a forecasting model, which uses CNN and RNN,
namely LSTNet, to extract short-term local dependency
pattern and long-term pattern for multivariate time series,
and incorporates Linear SVR model in the LSTNet model.
Besides, other efforts have been performed in [46] using
GAN-based anomaly detection. The model adopts U-Net as
generator to predict next frame in video and leverages theadversarial training to discriminate whether the prediction
is real or fake, thus abnormal events can be easily identified
by comparing the prediction and ground truth.
2.2.3 Composite models
Besides single model, composite model for unsupervised
anomaly detection has gained a lot attention recently. Zonget al. [23] utilize a deep autoencoder to generate a lowdimensional representation and reconstruction error, which
is further fed into a Gaussian Mixture Model to model
density distribution of multi-dimensional feature. However,they cannot consider the spatial-temporal dependency for
multivariate time series data. Different from this work, theComposite LSTM model [47] uses single encoder LSTM andmultiple decoder LSTMs to perform different tasks such asreconstructing the input sequence and predicting the future
sequence. In [48], the authors use ConvLSTM model as aunit within the composite LSTM model following a branch
for reconstruction and another for prediction. This type ofcomposite model is currently used to extract features fromvideo data for the tasks of action recognition. Similarly, authors in [49] propose Spatial-Temporal AutoEncoder (STAE)
for video anomaly detection, which utilizes 3D convolutional architecture to capture the spatial-temporal changes.
The architecture of the network is an encoder followed by
two branches of decoder for reconstructing past sequenceand predicting future sequence respectively.
As mentioned above, unsupervised anomaly detection
techniques have still many deficiencies. For traditional
anomaly detection, it is hard to learn representations ofspatial-temporal patterns in multi-sensor time-series signals. For a reconstruction model, a single task could makethe model suffer from the tendency to store information only
about the inputs that are memorized by the AE. And for theforecasting model, this task could suffer from only storing
the last few values that are most important for predicting
the future [47], [48]. Hence, their performance will be limited
since model only learn trivial representations. For composite
model, these researchers design their models for different
purposes. Zong et al. [23] could solve problem that themodel is robust to noise and anomalies through performing density estimation in a low-dimensional space. Zhao
et al. [49] could consider the spatial-temporal dependency
through 3D convolutional reconstructing and forecastingarchitectures. However, few studies could address theseissues simultaneously.
Different from these works, our research makes the following contributions: 1) The proposed model is designed tocharacterize complex spatial-temporal dependencies, thus
discovering generalized pattern of multi-sensor data; 2)
Adding a Maximum Mean Discrepancy (MMD) penalty
could avoid the model generalizing so well for noisy
data and anomalies; 3) Combining Attention-based Bidirectional LSTM (BiLSTM) and traditional Auto-regressive
linear model could boost the model’s performance fromdifferent time scale; 4) The composite baseline model isgenerated based on end-to-end training which means all
the components within the model are jointly trained withcompound objective function.
Besides, some learning algorithms based on time-series
data have been studied for decades. [50] propose Unsupervised Salient Subsequence Learning to extract subsequence
as new representations of the time series data. Due to theinternally sequential relationship, many neural networkbased models can be applied to time series in an unsupervised learning manner. For example, some 1D-CNN models[51], [52] have been proposed to solve time series tasks witha very simple structure and the sota performance. Moreover,the multiple time series signal usually has some kinds ofco-relations, [53] propose a method to learn the relationgraph on multiple time series. Some anomaly detection
based on multiple time series applications are available forwastewater treatment [54], for ICU [55], and for sensors [56].

4

𝑧𝑟

𝑧1

𝑦ℎ

𝑌ℎ

𝑧2

𝑦2

𝑧ℎ−1

𝑦ℎ−1

𝑧ℎ

𝑧ℎ
𝑧𝑓

Fig. 1: The overview of the proposed CAE-M model.

3

T HE P ROPOSED M ETHOD

3.1

Notation

In a multi-sensor time series anomaly detection problem, weare given a dataset generated by n sensors (n > 1). Without
loss of generality, we assume each sensor generates m signals (e.g., an accelerometer often generates 3-axis signals).
Denote S the signal set, we have N = |S| = nm signals in
total. For each signal xi ∈ S , xi ∈ Rti ×1 , where ti denotes
the length of signal xi . Note that even each sensor signal
may have different length, we are often interested in theirintersections, i.e., all sensors are having the same length T ,i.e., X = (x1 , · · · , xN )T ∈ RN ×T denotes an input sample
containing all sensors.
Definition 1 (Unsupervised anomaly detection). It is nontrivial to formally define an anomaly. In this paper, we areinterested in detecting anomalies in a classification problem.
Let Y = {1, 2, · · · , K} be the classification label set, and K
the total number of classes, then the dataset D = (Xi , yi )N
i=1 .
Eventually, our goal is to detect whether an input sample
Xa belongs to one of the K predefined classes with ahigh confidence. If not, then we call Xa an anomaly. Notethat in this paper, we are dealing with an unsupervised
anomaly detection problem, where the labels are unseen
during training, which is more obviously challenging.
3.2

Overview

There are some existing works [19], [21], [25] attempting
to resolve the unsupervised anomaly detection problem.
Unfortunately, they may face several critical challenges.
First, conventional anomaly detection techniques such as
PCA [17], k-means [18] and OCSVM [19] are unable tocapture the temporal dependencies appropriately because
they cannot deliver temporal memory states. Second, since
the normal samples might contain noise and anomalies,
using deep anomaly detection approaches such as standard
Autoencoders [20], [21] is likely to affect the generalization
capability. Third, the multi-stage approaches, i.e., feature

extraction and predictive model building are separated [25],[26], can easily get stuck in local optima.
In this paper, we present a novel approach called Convolutional Autoencoding Memory network (CAE-M) to tackle
the above challenges. Fig. 1 gives an overview of the proposed method. In a nutshell, CAE-M is built upon a convolutional autoencoder, which is then fed into a predictive
network. Concretely, we encode the spatial information in
multi-sensor time-series signals into the low-dimensional
representation via Deep Convolutional Autoencoder (CAE).
In order to reduce the effect of noisy data, some existing
works have tried to add Memory module [24] or Gaussian
Mixture Model (GMM) [23]. In our proposed method, we
simplify these modules into penalty item, which called Maximum Mean Discrepancy (MMD) penalty. Adding a MMD
term can encourage the distribution of training data toapproximate the same distribution such as Gaussian distribution, thus reducing the risk of overfitting caused by noise
and anomalies in training data [23]. And then we feed therepresentation and reconstruction error to the subsequentprediction network based on Bidirectional LSTM (Bi-LSTM)
with Attention mechanism and Auto-regressive model (AR)which could predict future feature values by modeling the
temporal information. Through the composite model, thespatial-temporal dependencies of multi-sensor time-series
signals can be captured. Finally, we propose a compound
objective function with weight coefficients to guide endto-end training. For normal data, the reconstructed value
generated by data coding is similar to the original input
sequence and the predicted value is similar to the future
value of time series, while the reconstructed value and thepredicted value generated by abnormal data change greatly.
Therefore, in inference process, we can detect anomalies precisely by computing the loss function in composite model.
3.3

Characterization Network

In the characterization network, we perform representative
learning by fusing multivariate signals in multiple sensors.

5

The low-dimensional representation contains two components: (1) the features which are abstracted from the multivariate signals; (2) the reconstruction error over the distancemetrics such as Euclidean distance and Minkowski distance.
To avoid the autoencoder generalizing so well for abnormal
inputs, optimization function combines reconstruction loss
by measuring how close the reconstructed input is to theoriginal input and the regularization term by measuring thesimilarity between the two distributions (i.e., the distribution of low-dimensional features and Gaussian distribution).
3.3.1 Deep feature extraction
We employ a deep convolutional autoencoder to learn the
low-dimensional features. Specifically, given N time series
with length T , we pack into a matrix x ∈ RN ×T with multisensor time-series data. The matrix is then fed to deep convolutional autoencoder (CAE). The CAE model is composedof two parts, an encoder and a decoder as in Eq. (1) andEq. (2). Assuming that x0 denotes the reconstruction of thesame shape as x, the model is to compute low-dimensional
representation zf , as follows:

zf = Encode(x),(1)

x0 = Decode(zf ).

(2)

The encoder in Eq. (1) maps an input matrix x to a hidden representation zf by many convolutional and poolinglayers. Each convolutional layer will be followed by a maxpooling layer to reduce the dimensions of the layers. A
max-pooling layer pools features by taking the maximum
value for each patch of the feature maps and produce theoutput feature map with reduced size according to the sizeof pooling kernel.
The decoder in Eq. (2) maps the hidden representation zf
back to the original input space as a reconstruction. In particular, a decoding operation needs to convert from a narrow
representation to a wide reconstructed matrix, therefore thetransposed convolution layers are used to increase the widthand height of the layers. They work almost exactly the same
as convolutional layers, but in reverse.
The difference between the original input vector x andthe reconstruction x0 is called the reconstruction error zr .
The error typically used in the autoencoder is Mean Squared
Error (MSE), which measures how close the reconstructed
input x0 is to the original input x, as follows in Eq. (3).

LM SE = kx − x0 k22 ,(3)

where k·k22 is the l2 -norm.
3.3.2 Handling noisy data
To reduce the influence of noisy data, we need to observe
the changes in low-dimensional features and the changesof distribution over the samples in a more granular way,thus distinguishing between normal and abnormal data
obviously.
Inspired by [23], in order to avoid the autoencoder generalizing so well for noisy data and abnormal data, we hope
to detect ”lurking” anomalies that reside in low-density
areas in the reduced low-dimensional space. Our proposed
method is conceptually similar to Gaussian Mixture Model(GMM) as target distributions. The loss function is complemented by MMD as a regularization term that encourages
the distribution of the low-dimensional representation to be
similar to a target distribution. It aims to make the distribution of noisy data close to the distribution of normal training
data, thereby reducing the risk of overfitting. Specifically,
Maximum Mean Discrepancy (MMD) [57] is a distancemeasure between the samples of the distributions. Given(1)(h)
the latent representation za = {zf , ..., zf } ∈ Rh×d , whered is a latent space (usually d < N × T ) and h denotes all of
the time steps at one iteration. For CAE with MMD penalty,
the Gaussian distribution Pz in reproduction kernel Hilbert
space H is chosen as the target distribution. We compute theKernel MMD as the follows:
h
h
1X
1X
(i)
LM M D (Z, Pz ) = ||φ(zf ) −φ(z (i) )||2H . (4)
h i=1
h i=1
Here we have the distribution Z of the low-dimensional(i)representation zf and the target distribution z (i) ∼ Pz overa set X . The MMD is defined by a feature map φ : X → H
where H is a reproducing kernel Hilbert space (RKHS).
During the training process, we could apply the kernel
trick to compute the MMD. And it turns out that many
kernels, including the Gaussian kernel, lead to the MMD
being zero if and only the distributions are identical. Lettingk(x, y) = hφ(x), φ(y)iH , we yield an alternative characterization of the MMD as follows:

1 X
1 X(i) (j)k(zf , zf ) + 2
k(z (i) , z (j) )LM M D (Z, Pz ) =|| 2
h i6=jh i6=j2 X(i)− 2k(zf , z (j) )||H .
h i,j
(5)
||u−v||2
Here the kernel is defined as k(u, v) = exp(− 2σ2 ).
The latent representation with Gaussian distribution Pz isperformed by sampling from Pz and approximating by
averaging the kernel k(·, ·) evaluated at all pairs of samples.
Note that we usually do batch training for neural network training. It means that the model is trained using a
subsample of data at one iteration. In this work, we need tocompute the MMD over a set of X at one iteration, where thenumber of X is equal to batchsize×timestep. That is, the la(1)(l)
tent representation is denoted as za = {zf , ..., zf } ∈ Rl×d ,where l = batchsize × h.
3.4

Memory Network

To simultaneously capture the spatial and temporal dependencies, our proposed model is designed to characterize
complex spatial-temporal patterns by concurrently performing the reconstruction analysis and prediction analysis. Considering the importance of temporal component in time series, we propose non-linear prediction and linear prediction
to detect anomalies by comparing the future prediction andthe next value appearance in the feature space.
The characterization network generates feature representations, which include reconstruction error and reduced
low-dimensional features learned by the CAE at h time
steps. Denote input features as zh for h = 1, ..., H :zh = [zf , zr ]h , h ∈ [1, H].

(6)

6

Our goal is to predict the current value zh for the
past values [z1 , z2 , ..., zh−1 ]. The memory network combines
non-linear function based predictor and linear function
based predictor to tackle temporal dependency problem.
3.4.1 Non-linear prediction
Non-linear predictor function has different types such as Recurrent neural networks (RNNs), Long Short-Term Memory(LSTM) [58] and Gated Recurrent Unit (GRU) [59]. Original
RNNs fall short of learning long-term dependencies. In thiswork, we adopt a Bidirectional LSTM with attention mechanism [60] which could consider the whole/local context
while calculating the relevant hidden states. Specifically,
the Bidirectional LSTM (BiLSTM) runs the input in twoways, one LSTM from past to future and one LSTM fromfuture to past. Different from unidirectional, the two hidden
states combined are able in any point in time to preserve
information from both past and future. A BiLSTM unit
consists of four components: input gate ih , forget gate fh ,output gate oh and cell activation vector ch . The hidden
state yh given input zh is computed as follows:

ih = σ(Wzi zh + Wyi yh−1 + Wci ch−1 + bi ),(7)

fh = σ(Wzf zh + Wyf yh−1 + Wcf ch−1 + bf ),(8)oh = σ(Wzo zh + Wyo yh−1 + Wco ch−1 + bo ),(9)ceh = tanh(Wzc zh + Wyc hh−1 + Wcc ch−1 + bc ),(10)

ch = fh ⊗ ch−1 + ih ⊗ ceh ,(11)yh = oh ⊗ tanh(ch ),(12)ŷh = [yh1 ; yh2 ],(13)

where ih , fh , oh , ch represent the value of i, f, o, c at the
moment h respectively, W and b denote the weight matrix
and bias vector, σ(·) and tanh(·) are activation function, theoperator ⊗ denotes element-wise multiplication, the current
cell state ch consists of two components, namely previous
memory ch−1 and modulated new memory ceh , the outputŷh combines the forward yh1 and backward yh2 pass outputs.
Note that the merge mode by which outputs of the forward
and backward are combined has different types, e.g. sum,multiply, concatenate, average. In this work, we use themode “sum” to obtain the output ŷh .
Attention mechanism for processing sequential data that
could focus on the features of the keywords to reducethe impact of non-key temporal context. Hence, we adopt
temporal attention mechanism to produce a weight vector
and merge raw features from each time step into a segmentlevel feature vector, by multiplying the weight vector. The
work process of attention mechanism is following detailed.

Mh = tanh(Wh ŷh + bh ),(14)

Eh = σ(Wa Mh + ba ),(15)

Ah = sof tmax(Eh ),X
Yh =Ah ∗ ŷh .

(16)
(17)

h

Here W and b are represented as the weight and bias. A
weighted sum of the ŷh based on the weight Ah is computed
as the context representation Yh . The context representationis considered as the predicted value of zh for temporalfeatures [z1 , z2 , ..., zh−1 ].

3.4.2 Linear prediction
Autoregressive (AR) model is a regression model that uses
the dependencies between an observation and a number
if lagged observations. Non-linear Recurrent Networks aretheoretically more expressive and powerful than AR models. In fact, AR models also yield good results in forecasting
short term modeling. In specific real datasets, such infinitehorizon memory isn’t always effective. Therefore, we incorporate AR model in parallel to the non-linear memory
network part.
The AR model is formulated as follows:ẑh = ch−1
Xwh−i +

i=1

h−1
X

wh−i ∗ zh−i ,

(18)

i=1

where w1 , ..., wh−1 are the weights of the AR model, c
is a constant, ẑh represents the predicted value for past
temporal value [z1 , z2 , ..., zh−1 ]. We implement this modelusing Dense layer of network to combine the weights anddata.
In the output layer, the prediction error is obtained bycomputing the difference between the output of predictor
model and true value zh . The final prediction error integrates the output of non-linear prediction model and linear
prediction model. The following equation is written as:
X
Lpredict =( ||Yh − zh ||2F + ||ẑh − zh ||2F ),
|
{z
}
|
{z
} (19)h∈Ωbatch

Attention-based BiLSTM

Autoregressive

where Ωbatch is a subsample of training data, || · ||F is the
Frobenius norm.
3.5

Joint optimization

As for multi-step approach, it can easily get stuck in local
optima, since models are trained separately. Therefore, we
propose an end-to-end hybrid model by minimizing compound objective function.
The CAE-M objective has four components, MSE (reconstruction error) term, MMD (regularization) term, prediction error (non-linear forecasting task) term and prediction error (linear forecasting task) term. Given X samples{x1 , x2 , ..., xD }, xi ∈ RN ×T , the objective function is constructed as:J(θ) = LM SE + λ1 · LM M D + λ2 · Llp + λ3 · Lnp
M
1 X=
L(xi , x0i ) + λ1 LM M D (Z, PZ )
M i=1
+

(20)

M
X

1
(i)
(i)
(i)
(i)
[λ2 ||Yh − zh ||2F + λ3 ||ẑh − zh ||2F ],M i=1

where M is batch size used for training, h is current time
step, λ1 ,λ2 and λ3 are the meta parameters controlling theimportance of the loss function.
Restating our goals more formally, we would like to:•

Minimize the reconstruction error in the characterization network, that is, minimize the error in reconstructing x0 from x at all time step h. We need tocompute the average error at each time step of sample. The purpose is to obtain better low-dimensional
representation for multi-sensor data.

7•

Minimize the MMD loss that encourages the distribution Z of the low-dimensional representation tobe similar to a target distribution Pz . It can makeanomalies deviate from normal data in the reduced
dimensions.

•

Minimize the prediction error by integrating nonlinear predictor and linear predictor. We split theset {z1 , z2 , ..., zh } obtained by characterization network into the current value zh and the past values[z1 , z2 , ..., zh−1 ]. And then the predicted values Yhand ẑh are obtained by minimizing prediction errors.
The purpose is to accurately express the informationof the next temporal slice using different predictor,
thus updating low-dimensional feature and reconstruction error.

•

3.6

λ1 , λ2 and λ3 are the meta parameters in CAE-M.
In practice, λ1 = e − 04, λ2 = 0.5, and λ3 = 0.5
usually achieve desirable results. Here MMD is complemented as a regularization term. The parameterselection is performed in Section 4.8.1.
Inference

Given samples as training dataset X = {x1 , x2 , ..., xD }, xi ∈RN ×T , we are able to compute the corresponding decisionthreshold (THR):
v
u
D
D
u1 X
1 X
THR =
Err(xi ) + t(Err(xi ) − µ)2 , (21)
D i=1
D i=1
where we denote Err(xi ) as the sum of loss function for xi ,and µ is the average value of Err(xi ) for i = 1, ..., D. The
setting is similar to the normal training distribution N (µ, σ)following with 1 standard deviation σ of the mean µ.
In inference process, the decision rule is that if Err(xi ) >THR, the testing sample in a sequence can be predicted to
be “abnormal”, otherwise “normal”.
The complete training and inference procedure of CAEM is shown in Algorithm 1.

4

Calculate MMD between za and Gaussian distribution
Pz ; // Eq. (5)6:Combine zf and zr into zh = [zf , zr ]h for eachsample; // Eq. (6)7:Predict the current value zh for the past values[z1 , z2 , ..., zh−1 ] by Attention-based BiLSTM and AR
model; // Eq. (7-18)
8:Update w by minimizing the compound objectivefunction; // Eq. (20)9: end while
10: Calculate the decision threshold THR by the training
samples; // Eq.(21)
11: return Optimal w and THR.
Inference process
Input: Normal and Anomalous dataset X = {x1 , x2 , ...,
xD }, threshold THR, model parameter w, hyperparameters λ1 , λ2 and λ3 .
Output: Label of all xi .
1: for all xi do
2:
Calculate the loss Err(xi ) = f (xi ; w); //f (·) denotes
CAE-M
3:
if Err(xi ) > THR then4:xi = “anomaly”;
5:
else
6:
xi = “normal”;
7:
end if
8: end for
9: return Label of all xi .
5:

E XPERIMENTS

In this section, we conduct extensive experiments to evaluate the performance of our proposed CAE-M approach foranomaly detection on several real-world datasets.
4.1

Algorithm 1 Training and Inference procedure of CAE-M
Training process
Input: Normal Dataset X = {x1 , x2 , ..., xD }, time steps h,batch size M and hyperparameters λ1 , λ2 , λ3 .
Output: Anomaly decision threshold THR and model parameter w.
1: Transform each sample x ∈ RN ×T into x ∈ Rh×N ×t in
the time axis;
2: Randomly initialize parameter w ;3: while not converge do
4:
Calculate low-dimensional representation zf and reconstruction error zr at each time step; // Eq. (1) (3)Datasets

We adopt two large publicly-available datasets and a private
dataset: PAMAP2, CAP and Mental fatigue dataset. These
datasets are exploiting multi-sensor time series for activity
recognition, sleep state detection, and mental fatigue detection, respectively. Therefore, they are ideal testbeds forevaluating anomaly detection algorithms.
PAMAP2 [61] dataset is a mobile dataset with respect
to actions or activities from UCI repository, containing dataof 18 different physical activities performed by 9 subjects
wearing 3 inertial measurement units, e.g. accelerator, gyroscope and magnetometer. There are 18 activity categories intotal. For experiments, we treat these classes with relatively

smaller samples as the anomaly classes (including running,
ascending stairs, descending stairs and rope jumping), while
the rest categories are combined to form the normal classes.
CAP Sleep Database [62], which stands for the Cyclic
Alternating Pattern (CAP) database, is a clinical dataset
from PhysioNet repository. It is characterized by periodic
physiological signals occurring during wake, S1-S4 sleep
stages and REM sleep. The waveforms include at least 3
EEG channels, 2 EOG channels, EMG signal, respiration
signal and EKG signal. There are 16 healthy subjects and92 patients in the database. The pathological recordings
include the patients diagnosed with bruxism, insomnia,
narcolepsy, nocturnal frontal lobe epilepsy, periodic leg
movements, REM behavior disorder and sleep-disorderedbreathing. In this task, we extracted 7 valid channels of all
the channels like ROC-LOC, C4-P4, C4-A1, F4-C4, P4-O2,
ECG1-ECG2, EMG1-EMG2 etc. For detecting sleep apnea

8

TABLE 1: The detailed statistics of three datasets
DatasetPAMAP2 [61]
CAP [62]
Mental Fatigue Dataset [2]DomainActivity Recognition
Sleep Stage Detection
Fatigue Detection

events, we chose healthy subjects as normal class and thepatients with sleep-disordered breathing as anomaly class.
Mental Fatigue Dataset [2] is a real world health-caredataset. Aiming to detect mental fatigue in the healthygroup, we collected the physiological signals (e.g., GSR,
HR, R-R intervals and skin temperature) using wearable
device. There are 6 healthy young subjects participated in
the mental fatigue experiments. In this task, non-fatigue
data samples are labeled as normal class and fatigue data
samples are labeled as anomaly class. Fatigue data accounts
for a fifth of the total.
The detailed information of the datasets is shown in
TABLE 1.
4.2

Baseline Methods

In order to extensively evaluate the performance of the
proposed CAE-M approach, we compare it with several
traditional and deep anomaly detection methods:(1) KPCA (Kernel principal component analysis) [28],which is a non-linear extension of PCA commonly used foranomaly detection. (2) ABOD (Angle-based outlier detection) [63], which is a probabilistic model that well suitedfor high dimensional data. (3) OCSVM (One-class support
vector machine) [64], which is the one-class learning methodthat classifies new data as similar or different to the training
set. (4) HMM (Hidden Markov Model) [65] is a finite setof states, each of which is associated with a probability
distribution. In a particular state an observation can be generated, according to the associated probability distribution.
(5) CNN-LSTM [66], which is a forecasting model composedof convolutional and LSTM networks. It can obtain the forecast by estimating the current data, and detect anomalies oncomparing the forecasting value with actuals. (6) LSTM-AE(LSTM based autoencoder) [36], which is an unsupervised
detection technique used in time series that can induce a
representation by learning an approximation of the identity
function of data. (7) ConvLSTM-COMPOSITE [48], which
utilizes a composite structure that is able to encoder the
input, reconstruct it, and predict its near future. To simplify the name, “ConvLSTM-COMP” denotes ConvLSTMCOMPOSITE. We choose the “conditional” version to build
a single model called ConvLSTM-AE by removing theforecasting decoder. (8) UODA (Unsupervised sequential
outlier detection with deep architecture) [67], which utilizes
autoencoders to capture the intrinsic difference between
normal and abnormal samples, and then integrates themodel to RNNs that perform fine-tuning to update the parameters in DAE. (9) MSCRED (Multi-scale convolutional
recurrent encoder-decoder) [15], which is a reconstructionbased anomaly detection and diagnosis method.
4.3

Implementation details

For traditional anomaly detection, we scale the sequential
data into segments and extract the features from each

Instances
1,140,000
921,700,000
1,458,648

Dimensions
27
21
4

Classes18
82Permissions
PublicPublicPrivatesegment. In PAMAP2 dataset, multiple sensors are wornon three different position (wrist, chest, ankle). Hence, we
extract 324 features including time and frequency domainfeatures. In CAP Sleep dataset, we first pass through the
Hanning window low pass filter for removing the highfrequency components of signals. And then we extract 91
features for EEG, EMG and ECG signals [68]–[70]; In Mental Fatigue dataset, we preprocess physiological signals byinterpolation and filtering algorithm. Then we extract 23
features for Galvanic Skin Response (GSR), Heart Rate (HR),R-R intervals and skin temperature sensors [2].
For Deep Anomaly Detection (DAD) method, we filter
multi-sensor signals and then pack these signals into matrix
as input to construct the deep model.
We reimplement these methods based on open-source
repositories1 or our own implementations. For KPCA, weemploy Gaussian kernel with a bandwidth of 600, 500,
0.5, respectively for PAMAP2, CAP, and Mental Fatigue
datasets. For ABOD, we use k nearest neighbors to approximate the complexity reduction. For an observation, thevariance of its weighted cosine scores to all neighbors could
be viewed as the abnormal score. For OCSVM, we adopt
PCA for OCSVM as a dimension reduction tool and employ
the Gaussian kernel with a bandwidth of 0.1. For HMM,
we build a Markov model after extracting features andcalculate the anomaly probability from the state sequence
generated by the model. For CNN-LSTM, we define a CNNLSTM model in Keras by first defining 2D convolutional
network as comprised of Conv2D and MaxPooling2D layers
ordered into a stack of the required depth, wrapping themin a TimeDistributed layer and then defining the LSTM
and output layers. For LSTM-AE, we use single-layer LSTM
on both encoder and decoder in the task. For ConvLSTMCOMPOSITE, we choose ”conditional” version and adapt
this technique to anomaly detection in multivariate timeseries. Here we also build a single model called ConvLSTMAE by removing forecasting decoder. For UODA, we reimplement this algorithm by customizing the number of layers
and hyper-parameters. For MSCRED, we first constructmulti-scale matrices for multi-sensor data, and then fed it
into MSCRED model and evaluate the performance.
For our own CAE-M, we use library Hyperopt [71]to select the best hyper-parameters (i.e., time window,
the number of neurons, learning rate, activation function,
optimization criteria and iterations). The characterization
network runs with Conv2D → M axpooling → Conv2D →M axpooling → Conv2DT ranspose → Conv2DT ranspose→ Conv2DT ranspose, i.e., Conv1-Conv5 with 32 kernelsof size 4 × 4, 64 kernels of size 4 × 4, 64 kernels of size
4 × 4, 32 kernels of size 4 × 4, 1 kernels of size 4 × 4,
and Maxpooling with size 2 × 2. We use Rectified Linear
Unit (ReLU) as the activation function of convolutional
1. https://pyod.readthedocs.io/en/stable/,
7fantasysz/MSCRED

https://github.com/

9

TABLE 2: The mean precision, recall and F1 score of baselines and our proposed method, * p-value = 0.0077.
PAMAP2

MethodmPre

mRec

CAP dataset
mF1

mPre

mRec

mF1

mPre

mRec

mF1

KPCA

0.72360.65790.68920.76030.5847

0.66110.53410.50140.5173ABOD

0.86530.90220.8834

0.7867

0.63650.70370.66790.61450.6401

OCSVM

0.76000.72040.73970.92670.92590.92630.56050.57100.5290HMM

0.69500.65530.67450.82380.80780.81570.60660.60760.6071

CNN-LSTM

0.66800.53920.59680.61590.52170.56490.57800.5042

0.5386

LSTM-AE0.86190.79970.82960.71470.62530.66710.71400.68200.6870UODA

0.89570.85130.87300.75570.51240.61070.82800.77700.8017MSCRED0.69970.73010.7146

0.64100.57840.60810.80160.68020.7359ConvLSTM-AE

0.73590.73610.73600.81500.81940.81720.90100.93460.9175ConvLSTM-COMP

0.88440.88420.88430.83670.83770.83720.93730.9316

0.9344

CAE-M (Ours)

0.96080.96700.96390.99390.99520.9961

0.99620.99590.9960

Improvement

7.64%

6.48%

7.96%

6.72%

6.93%

6.98%

5.89%

6.13%

6.16%

layers. The memory network contains non-linear prediction
and linear prediction, where the non-linear network runs
with BiLST M (512) → Attention(h − 1) → Dropout(0.2)→ F C(1000, linear), and the linear network runs withF C(1000, linear). The CAE-M model is trained in an endto-end fashion using Keras [72]. The optimization algorithm is Adam and the batch size is set as 32. And we set
parameters of compound objective function λ1 = e − 04,
λ2 = 0.5 and λ3 = 0.5. The time step h usually gives
desirable results as h = 5 or h = 10.
Note that in addition to the complete CAE-M approach,
we further evaluate its several variants as baselines to justify
the effectiveness of each component:•

Fatigue dataset

CAE-Mw/oPre. The CAE-M model removes the linear
and non-linear prediction. That is, this variant only
adopts the characterization network with reconstruction loss and MMD loss. (i.e., λ1 = e − 04, λ2 =0, λ3 = 0)•

CAE-Mw/oRec+MMD. The CAE-M model removes
the reconstruction error and MMD. Different from
CNN-LSTM model, the characterization network is
still performed as the deep convolutional autoencoder. We put the latent representation without reconstruction error into the memory network. (i.e.,
λ1 = 0, λ2 = 0.5, λ3 = 0.5)•

CAE-Mw/oATTENTION. The CAE-M model without
Attention component is implemented. (i.e., λ1 = e −04, λ2 = 0.5, λ3 = 0.5)•

CAE-Mw/oAR. The CAE-M model without AR component is implemented. (i.e., λ1 = e − 04, λ2 =0.5, λ3 = 0)•

CAE-Mw/oMMD. The CAE-M model without MMD
component is implemented. (i.e., λ1 = 0, λ2 =0.5, λ3 = 0.5)

Note that anomaly detection problems are often withhighly-imbalanced classes, hence accuracy is not suitable as
the evaluation metric. In order to thoroughly evaluate theperformance of our proposed method, we follow existing
works [15], [23], [73] to adopt the mean precision, recall, andF1 score as the evaluation metrics. The mean precision means
the average precision of normal and abnormal class. The
same pattern goes for mean recall, F1 score.
In the experiments, the train-validation-test sets are split
by following existing works [15], [67]. Concretely speaking,for each dataset, we split normal samples into training, validation, and test with the ratio of 5 : 1 : 4, where the training
and validation set only contain normal samples and haveno overlapping with testing set. The anomalous samples are
only used in the testing set. The model selection criterion,
i.e., hyperparameters, used for tuning is the validation error
on the validation set.
4.4

Results and Analysis

As shown in TABLE 2, we compare our proposed methodwith traditional and deep anomaly detection methods using
the mean precision, recall and F1 score. We can see that
our method outperforms most of the existing methods,
which demonstrates the effectiveness of our method. From
TABLE 2, we can observe the following results.
For the PAMAP2 dataset, the CAE-M achieves the highest precision and recall compared by 10 popular methods. Traditional methods perform differently on PAMAP2
dataset since they are limited by the feature extraction and feature selection methods. In deep learning
method, CNN-LSTM has a lowest F1 score. This meansthat more constraints such as data preprocessing method
and anomaly evaluation strategy need to be added forprediction-based anomaly detection. For LSTM-AE, MSCRED and ConvLSTM-AE, they both are reconstructionbased anomaly detection methods. Their performance islimited by the “noisy data” problem, resulting in reconstruction error for the abnormal input could be fit so well.
For UODA, it performs reasonably well on the PAMAP2
dataset, but it is not end-to-end training, which is neededby pre-training denoising autoencoder (DAEs) and deep
recurrent networks (RNNs), and then fine-tuning the UODA
model composing of the DAE and RNN. For ConvLSTMCOMPOSITE model, it performs better than other baselinemodels. The model consists of a single encoder, two decoders of reconstruction branch and prediction branch. In

10TABLE 3: Detection performance in different sleep stages of baselines and our proposed method, *p-value = 0.0074.
Method
KPCA

WAKE
mPre

mRec

S1
mF1

mPre

S2
mRec

S3
mF1

mPre

mRec

S4
mF1

mPre

mRec

REM

mRec

mF1

mPre

mF1

mPre

mRec

0.9162 0.8213 0.8662 0.8267 0.7598

0.79180.9257 0.9353 0.9305 0.9039 0.8689 0.8861 0.9402 0.9604 0.9502 0.9536 0.9614 0.9575

mF1

0.9389 0.6550 0.7716 0.8489 0.6184 0.7155 0.6749 0.6448 0.6595 0.5915 0.5909 0.5912

ABOD

0.9872 0.8686 0.9242 0.9347 0.5522

0.6942

OCSVM

0.9784 0.9492 0.9636 0.9655 0.9504

0.95790.9395 0.9448 0.9421 0.9714 0.9499 0.9605 0.8701 0.9488 0.9077 0.9784 0.9492 0.9636

HMM

0.8417 0.8406 0.8411 0.8790 0.8856

0.88230.8967 0.8887 0.8927 0.6880 0.6747 0.6813 0.7279 0.7286 0.7282 0.8024 0.8649 0.8325

LSTM-AE

0.6990 0.7178 0.7082 0.6517 0.6492

0.65040.7430 0.7331 0.7380 0.7689 0.7828 0.7758 0.7274 0.7569 0.7418 0.6590 0.6887 0.6735

UODA

0.6159 0.6326 0.6241 0.6762 0.6762

0.67620.7290 0.5223 0.6086 0.5716 0.5766 0.5741 0.6626 0.8498 0.6807 0.5626 0.6116 0.5861

ConvLSTM0.9889 0.9772 0.9830 0.9755 0.9850
COMP

0.98640.9250 0.9127 0.9188 0.9401 0.9041 0.9217 0.8647 0.8866 0.9023 0.9675 0.9949 0.9810

CAE-M

0.9974 0.9949 0.9961 0.9958 0.9950 ß0.9954 0.9950 0.9950 0.9950 0.9294 0.8842 0.9063 0.9842 0.9950 0.9895 0.9681 0.9950 0.9813

fact, since its efficiency is influenced by reconstruction error
and prediction error respectively, its performance could be
limited by one of encoder-decoder models.
For the CAP dataset, most of methods show a low F1score. As CAP dataset contains different sleep stages of subjects, some methods are limited by high complexity of data.
For OCSVM and HMM, they achieve better performance
because of dimensionality reduction from 36 dimensionsof PAMAP2 dataset to 7 dimensions. For MSCRED, due
to batch size =1 for the training model in the open source
code, the loss function couldn’t converge during training
model and the training speed is slow. Our proposed method
achieves about 7% improvement at F1 score, compared withthe existing methods.
For Fatigue dataset, it is difficult to label fatigue andnon-fatigue data manually. Therefore, it may be a lot ofnoise or misclassification patterns in the data, so that mostof methods fail to solve this problem. For UODA, MSCRED
and ConvLSTM, they have ability to overcome noise andmisclassification of training data. Our proposed method also
solves this problem successfully and achieves at least 6%
improvement at F1 score.
Besides, in order to indicate significant differences fromour proposed method and other baselines, we use Wilcoxon
signed rank test [74] to analyze these results in TABLE 2. We
compute average p-value of CAE-M compared with otherbaselines. A p-value = 0.0077 indicates that the performanceof our proposed method differs from other methods. Thisp-value is also computed in TABLE 3.
4.5

Fine-grained Analysis

In addition to the anomaly detection of different classes oneach dataset, we conduct a fine-grained analysis to evaluate
the performance of each method within each class. Considering intra-class diversity, we conduct a group of experiments to detect anomalies in different sleep stages. In fact,
these physiological signals in different sleep stages havesignificant differences. We choose 4 traditional methods and3 deep methods with good performance in global domain ascomparison methods. As shown in TABLE 3, we can observethat our architecture is most robust across same experimentsettings. Several observations from these results are worthhighlighting. For ABOD, the testing performance is unstablein local domain, which the highest F1 score is 0.92 in WAKEand the lowest F1 score is 0.59 in REM. For KPCA andTABLE 4: The evaluation results on LOSO cross validation
approach, including the best, the worst and the mean F1score of 8 subjects.
MethodWorst mF1

Best mF1

Mean mF1

ABOD
ConvLSTM-COMP
UODA
CAE-M

0.60930.70330.59380.80090.85070.92240.9336
0.94330.7706
0.84930.79840.8616

ConvLSTM-COMPOSITE, the testing performance in local
domain far exceeds the performance in global domain. This
demonstrates that the two model can achieve better performance when intra-class data have similar distribution or
regular pattern. For other methods, the testing performanceis consistent in local and global domain. For our proposed
method, the best testing performance can be achieved nomatter in local domain or global domain. This study clearlyjustifies the superior representational capacity of our architecture to solve intra-class diversity.
4.6

Effectiveness Evaluation

4.6.1 Leave One Subject Out
In this section, we measure the generalization ability ofmodels using Leave One Subject Out (LOSO). The fact isthat when training and testing datasets contain the same
subject, the model is likely to know more about the current subject which may be biased towards a new one.
Therefore, LOSO could help to evaluate the generalization
ability. We choose the PAMAP2 dataset to conduct subjectindependent experiments which contain 8 subjects. As canbe seen in Fig. 2(a), we evaluate our proposed method andthree methods with relatively high F1 score. By examining the results, one can easily notice that deep learningbased methods obtain better performance than traditional
methods. However, complex models such as deep neuralnetworks are prone to overfitting because of their flexibility
in memorizing the idiosyncratic patterns in the training set,instead of generalizing to unseen data.
TABLE 4 shows the best, the worst and average performance among 8 subjects. We can observe that UODA
and ConvLSTM-COMPOSITE model perform well in somespecific subjects, but they fail to reduce the effects of overfitting to each test subject, even drop to 0.70 and 0.59

11(a) Leave One Subject Out Evaluation(b) Ablation Study

Fig. 2: Effectiveness evaluation using LOSO method and ablation study.

100

100

80

Method

mPre

mRec

mF1

ABOD

0.6240±0.000

0.5946±0.000

0.6090±0.000

60

ConvLSTM-COMP

0.8953±0.029

0.8081±0.038

0.8488±0.019

50
0

UODA

0.8155±0.063

0.7464±0.027

0.7782±0.031

CAE-M

0.9437±0.024

0.8191±0.003

0.8770±0.012

100
90

90
mPre(%)

mF1(%)

90

70

0.01

80

70

UODA
ConvLSTM-COMP
CAE-M

0.02
0.05
Noise factor

0.1

0.3

mRec(%)TABLE 5: The repeated measures analysis of variance on
LOSO cross validation approach of one subject.

600(a) mF1

UODA
ConvLSTM-COMP
CAE-M

0.01

0.02
0.05
Noise factor

80
70
60

0.1

0.3

(b) mPre

50
0

UODA
ConvLSTM-COMP
CAE-M

0.01

0.02
0.05
Noise factor

0.1

0.3

(c) mRec

Fig. 3: Robustness to noisy data.
for some subjects. Compared to these methods, CAE-M
can generalize well on testing subjects it hasn’t appeared
before, which reach the average F1 score of 0.86. Besides,
we perform an analysis of variance on repeated measures
within subject 1 (corresponding to numbers in Fig. 2(a)). As
shown in TABLE 5, we observe that CAE-M remains a morestable performance on repeated measurements. In summary,the above demonstrates that our model can be motivated to
improve the generalization ability.
4.6.2 Ablation Study
The proposed CAE-M approach consists of several components such as CAE, MMD, Attention mechanism, BiLSTM
and Auto-regressive. To demonstrate the effectiveness ofeach component, we conduct ablation studies in this section.
The ablation study is shown in Fig. 2(b). These ID numbers
represent CAE-M without non-linear and linear prediction,
CAE-M without reconstruction error and MMD, CAE-M
without attention module, CAE-M without AR, CAE-M
without MMD and CAE-M, respectively. The experimental
results indicate that for the removal of different component
above, there is corresponding performance drop at F1 score.
We can observe that CAE-M model without prediction orreconstruction error achieves a low F1 score relatively. This
demonstrates that our composite model is effective andnecessary for anomaly detection in multi-sensor time-series
data. Compared to original CAE-M model, removing the
AR component (in CAE-Mw/o AR ) from the full model causes
significant performance drops on most of the datasets. Thisshows the critical role of the AR component in general.
Moreover, attention and MMD components can also cause
big performance rises on all the datasets. More details areshown in TABLE 6. Here, these ID numbers are corresponding to numbers in Fig. 2(b).
4.7Robustness to Noisy Data

In real-world applications, the collection of multi-sensor
time-series data can be easily polluted with noise due tochanges in the environment or the data collection devices.
The noisy data bring critical challenges to the unsupervised
anomaly detection methods. In this section, we evaluate therobustness of different methods to noisy data. We manually control the noisy data ratio in the training data. We
inject Gaussian noise (µ=0, σ =0.3) in a random selectionof samples with a ratio varying between 1% to 30%. We
compare the performance of three methods on PAMAP2
dataset: UODA, ConvLSTM-COMPOSITE, and CAE-M in
Fig. 3. These methods have good stability in the above
experiments. As the noise increases, the performance of all
methods decreases. For CAE-M, the F1 score, precision andrecall have no significant decline. Among them, our model
remains significantly superior to others, demonstrating its
robustness to noisy data.
4.8
4.8.1

Further Analysis
Parameter Sensitivity Analysis

In this section, we evaluate the parameter sensitivity of
CAE-M model. It is worth noting that CAE-M achieves
the best performance by adjusting weight coefficient ofcompound objective function. We apply control variate reduction technique [75] to empirically evaluate the sensitivity

12

TABLE 6: The mean precision, recall and F1 score from variants.
PAMAP2

Method

mRec

mF1

mPre

mRec

mF1

mPre

mRec

mF1

CAE-Mw/o Pre

0.81030.80230.80630.82990.81010.81990.60050.60960.60502

CAE-Mw/o Rec+MMD

0.56930.54400.55630.8896

0.77840.83030.70500.68140.69303

CAE-Mw/o ATTENTION

0.91510.9276

0.92130.92510.92910.92710.96050.9551

0.95784

CAE-Mw/o AR

0.90600.86910.88720.96340.93810.95060.90460.90480.9047

5

CAE-Mw/o MMD

0.94370.95500.94930.92930.92130.92530.94070.92880.9347

6

CAE-M

0.96080.96700.96390.99390.99520.9961

0.99620.99590.9960

100

90

90

90

80

80

70

PAMAP2
CAP
Fatigue60
e-04

mF1(%)

100

e-05

e-06

e-07

0.5

80

70

PAMAP2
CAP
Fatigue

60
0.1

1.0

5.0

10.0

50.0

5

PAMAP2
CAP
Fatigue

60
0.1

0.5

1.0

5.0

10.0

50.0

λ1

λ2

λ3

(a) MMD loss(b) Bi-LSTM loss(c) Autoregressive loss

Fig. 4: Parameter sensitivity analysis of the proposed
CAE-M approach.
of parameter λ1 , λ2 , λ3 with a wide range. The results areshown in Fig. 4. As the value of MMD loss is greater thanothers, we select its weight coefficient within e-04 ∼ e-07
and other weight coefficients within [0.1, 0.5, 1, 5, 10, 50]. We
adjust one of λ while fixing the other respective λ to keep
the optimal value (λ1 = e − 04, λ2 = 0.5, and λ3 = 0.5).
When weight coefficient is increased, we observe that F1score tends to decline. The optimal parameter is λ1 = e−04,
λ2 = 0.5, and λ3 = 0.5. It can be seen that the performanceof CAE-M stays robust within a wide range of parameterchoice.
0.03

5
Total
Reconstruction
MMD
Bi-LSTM
Autoregressive

0.02

Total
Reconstruction
MMD
Bi-LSTM
Autoregressive

0.02

Total
Reconstruction
MMD
Bi-LSTM
Autoregressive

Loss

3

Loss

Loss

×10 -3

4

0.015
0.01

2

0.01
0.005
0

Fatigue dataset

mPre

100

70

CAP dataset

1

mF1(%)

mF1(%)

ID

20

40

60

80

100

Epoch

(a) PAMAP2 dataset

0

1

20

40

60

80

Epoch

(b) CAP dataset

100

0

20

40

60

80

C ONCLUSION AND F UTURE W ORK

In this paper, we introduced a Deep Convolutional Autoencoding Memory network named CAE-M to detect
anomalies. The CAE-M model uses a composite framework
to model generalized pattern of normal data by capturing spatial-temporal correlation in multi-sensor time-series
data. We first build Deep Convolutional Autoencoder witha Maximum Mean Discrepancy (MMD) penalty to characterize multi-sensor time-series signals and reduce the riskof overfitting caused by noise and anomalies in training
data. To better represent temporal dependency of sequential
data, we use non-linear Bidirectional LSTM with Attention
and linear Auto-regressive model for prediction. Extensive
empirical studies on HAR and HC datasets demonstratethat CAE-M performs better than other baseline methods.
In the future work, we will focus on the point-basedfine-grained anomaly detection approach and further improve our method for multi-sensor data by designing proper
sparse operations.

ACKNOWLEDGMENT
This work was supported by Key-Area Researchand Development Program of Guangdong Province(No.2019B010109001), Science and Technology Service
Network Initiative, Chinese Academy of Sciences (No. KFJSTS-QYZD-2021-11-001), and Natural Science Foundationof China (No.61972383, No.61902377, No.61902379).

100

Epoch

(c) Fatigue dataset

Fig. 5: Convergence analysis of the proposed CAE-M
approach on different datasets.

4.8.2 Convergence Analysis
Since CAE-M involves several components, it is natural to
ask whether and how quickly it can converge. In this section,
we analyze the convergence to answer this question. We
extensively show the results of each component on threedatasets in Fig. 5. These results demonstrate that even if the
proposed CAE-M approach involves several components,
it could reach a steady performance within fewer than 40
iterations. Therefore, in real applications, CAE-M can beapplied more easily with a fast and steady convergence
performance.

R EFERENCES
[1]
[2]
[3]

[4]

[5]

[6]

R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” arXiv preprint arXiv:1901.03407, 2019.
Y. Zhang, Y. Chen, and Z. Pan, “A deep temporal model for mental
fatigue detection,” in 2018 IEEE International Conference on Systems,
Man, and Cybernetics (SMC). IEEE, 2018, pp. 1879–1884.
J. E. Ball, D. T. Anderson, and C. S. Chan, “Comprehensive surveyof deep learning in remote sensing: theories, tools, and challenges
for the community,” Journal of Applied Remote Sensing, vol. 11, no. 4,
p. 042609, 2017.
B. Kiran, D. Thomas, and R. Parakkal, “An overview of deeplearning based methods for unsupervised and semi-supervised
anomaly detection in videos,” Journal of Imaging, vol. 4, no. 2, p. 36,
2018.
A. Ramchandran and A. K. Sangaiah, “Unsupervised anomaly
detection for high dimensional data—an exploratory analysis,” inComputational Intelligence for Multimedia Big Data on the Cloud withEngineering Applications. Elsevier, 2018, pp. 233–251.
G. Pang, C. Shen, L. Cao, and A. V. D. Hengel, “Deep learning for
anomaly detection: A review,” ACM Computing Surveys (CSUR),vol. 54, no. 2, pp. 1–38, 2021.

13

[7]

L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon,
W. Samek, M. Kloft, T. G. Dietterich, and K. Müller, “A unifying
review of deep and shallow anomaly detection,” 2020.
[8] Y. Zhang, Y. Chen, and C. Gao, “Deep unsupervised multi-modal
fusion network for detecting driver distraction,” Neurocomputing,vol. 421, pp. 26–38, 2021.
[9] Y. Chen, J. Wang, M. Huang, and H. Yu, “Cross-position activity
recognition with stratified transfer learning,” Pervasive and Mobile
Computing, vol. 57, pp. 1–13, 2019.
[10] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao, “Fedhealth: Afederated transfer learning framework for wearable healthcare,”IEEE Intelligent Systems, vol. 35, no. 4, pp. 83–93, 2020.
[11] J. Wang, V. W. Zheng, Y. Chen, and M. Huang, “Deep transfer
learning for cross-domain activity recognition,” in proceedings ofthe 3rd International Conference on Crowd Science and Engineering,
2018, pp. 1–8.
[12] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, “Deep learning forsensor-based activity recognition: A survey,” Pattern Recognition
Letters, vol. 119, pp. 3–11, 2019.
[13] J. Wang, Y. Chen, L. Hu, X. Peng, and S. Y. Philip, “Stratified transfer learning for cross-domain activity recognition,” in 2018 IEEE
International Conference on Pervasive Computing and Communications(PerCom). IEEE, 2018, pp. 1–10.
[14] D. Li, D. Chen, B. Jin, L. Shi, J. Goh, and S.-K. Ng, “Mad-gan:
Multivariate anomaly detection for time series data with generative adversarial networks,” in International Conference on Artificial
Neural Networks. Springer, 2019, pp. 703–716.
[15] C. Zhang, D. Song, Y. Chen, X. Feng, C. Lumezanu, W. Cheng,
J. Ni, B. Zong, H. Chen, and N. V. Chawla, “A deep neural network
for unsupervised anomaly detection and diagnosis in multivariate
time series data,” in Proceedings of the AAAI Conference on Artificial
Intelligence, vol. 33, 2019, pp. 1409–1416.
[16] V. Patraucean, A. Handa, and R. Cipolla, “Spatio-temporal
video autoencoder with differentiable memory,” arXiv preprint
arXiv:1511.06309, 2015.
[17] R. Paffenroth, P. Du Toit, R. Nong, L. Scharf, A. P. Jayasumana, and
V. Bandara, “Space-time signal processing for distributed patterndetection in sensor networks,” IEEE Journal of Selected Topics in
Signal Processing, vol. 7, no. 1, pp. 38–49, 2013.
[18] L. J. Latecki, A. Lazarevic, and D. Pokrajac, “Outlier detection withkernel density functions,” in International Workshop on Machine
Learning and Data Mining in Pattern Recognition. Springer, 2007,
pp. 61–75.
[19] B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C.
Williamson, “Estimating the support of a high-dimensional distribution,” Neural computation, vol. 13, no. 7, pp. 1443–1471, 2001.
[20] M. Sakurada and T. Yairi, “Anomaly detection using autoencoders
with nonlinear dimensionality reduction,” in Proceedings of the
MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data
Analysis. ACM, 2014, p. 4.
[21] M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S.
Davis, “Learning temporal regularity in video sequences,” inProceedings of the IEEE conference on computer vision and patternrecognition, 2016, pp. 733–742.
[22] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,”
in Proceedings of the 25th international conference on Machine learning.
ACM, 2008, pp. 1096–1103.
[23] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho,and H. Chen, “Deep autoencoding gaussian mixture model for
unsupervised anomaly detection,” 2018.
[24] D. Gong, L. Liu, V. Le, B. Saha, M. R. Mansour, S. Venkatesh,
and A. v. d. Hengel, “Memorizing normality to detect anomaly:
Memory-augmented deep autoencoder for unsupervised anomaly
detection,” arXiv preprint arXiv:1904.02639, 2019.
[25] D. Li, D. Chen, J. Goh, and S.-k. Ng, “Anomaly detection withgenerative adversarial networks for multivariate time series,”
arXiv preprint arXiv:1809.04758, 2018.
[26] R. Aliakbarisani, A. Ghasemi, and S. F. Wu, “A data-driven metriclearning-based scheme for unsupervised network anomaly detection,” Computers & Electrical Engineering, vol. 73, pp. 71–83, 2019.
[27] B. Schölkopf, A. Smola, and K.-R. Müller, “Nonlinear component
analysis as a kernel eigenvalue problem,” Neural computation,
vol. 10, no. 5, pp. 1299–1319, 1998.
[28] H. Hoffmann, “Kernel pca for novelty detection,” Pattern recognition, vol. 40, no. 3, pp. 863–874, 2007.

[29] R. Paffenroth, K. Kay, and L. Servi, “Robust pca for anomaly
detection in cyber networks,” arXiv preprint arXiv:1801.01571, 2018.
[30] R. Laxhammar, G. Falkman, and E. Sviestins, “Anomaly detection
in sea traffic-a comparison of the gaussian mixture model and thekernel density estimator,” in 2009 12th International Conference on
Information Fusion. IEEE, 2009, pp. 756–763.
[31] J. Kim and C. D. Scott, “Robust kernel density estimation,” Journalof Machine Learning Research, vol. 13, no. Sep, pp. 2529–2565, 2012.
[32] D. M. Tax and R. P. Duin, “Support vector data description,”
Machine learning, vol. 54, no. 1, pp. 45–66, 2004.
[33] N. Günnemann, S. Günnemann, and C. Faloutsos, “Robust multivariate autoregression for anomaly detection in dynamic productratings,” in Proceedings of the 23rd international conference on World
wide web. ACM, 2014, pp. 361–372.
[34] J. D. Hamilton, Time series analysis. Princeton university press
Princeton, NJ, 1994, vol. 2.
[35] H. Z. Moayedi and M. Masnadi-Shirazi, “Arima model for network traffic prediction and anomaly detection,” in 2008 International Symposium on Information Technology, vol. 4. IEEE, 2008, pp.
1–6.
[36] P. Malhotra, A. Ramakrishnan, G. Anand, L. Vig, P. Agarwal, and
G. Shroff, “Lstm-based encoder-decoder for multi-sensor anomaly
detection,” arXiv preprint arXiv:1607.00148, 2016.
[37] W. Luo, W. Liu, and S. Gao, “Remembering history with convolutional lstm for anomaly detection,” in 2017 IEEE International
Conference on Multimedia and Expo (ICME). IEEE, 2017, pp. 439–
444.
[38] J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special Lecture on IE, vol. 2,
no. 1, 2015.
[39] D. Wulsin, J. Blanco, R. Mani, and B. Litt, “Semi-supervised
anomaly detection for eeg waveforms using deep belief nets,”
in 2010 Ninth International Conference on Machine Learning andApplications. IEEE, 2010, pp. 436–441.
[40] C. Zhou and R. C. Paffenroth, “Anomaly detection with robust
deep autoencoders,” in Proceedings of the 23rd ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining.
ACM, 2017, pp. 665–674.
[41] P. Filonov, F. Kitashov, and A. Lavrentyev, “Rnn-based early cyberattack detection for the tennessee eastman process,” arXiv preprint
arXiv:1709.02232, 2017.
[42] T. Ergen, A. H. Mirza, and S. S. Kozat, “Unsupervised and semisupervised anomaly detection with lstm neural networks,” arXiv
preprint arXiv:1710.09207, 2017.
[43] D. Shalyga, P. Filonov, and A. Lavrentyev, “Anomaly detection forwater treatment system based on neural network with automaticarchitecture optimization,” arXiv preprint arXiv:1807.07282, 2018.
[44] M. Kravchik and A. Shabtai, “Detecting cyber attacks in industrial
control systems using convolutional neural networks,” in Proceedings of the 2018 Workshop on Cyber-Physical Systems Security andPrivaCy. ACM, 2018, pp. 72–83.
[45] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, “Modeling long-andshort-term temporal patterns with deep neural networks,” in The
41st International ACM SIGIR Conference on Research & Development
in Information Retrieval. ACM, 2018, pp. 95–104.
[46] W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction
for anomaly detection–a new baseline,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2018, pp.
6536–6545.
[47] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised
learning of video representations using lstms,” in International
conference on machine learning, 2015, pp. 843–852.
[48] J. R. Medel and A. Savakis, “Anomaly detection in video using predictive convolutional long short-term memory networks,”
arXiv preprint arXiv:1612.00390, 2016.
[49] Y. Zhao, B. Deng, C. Shen, Y. Liu, H. Lu, and X.-S. Hua, “Spatiotemporal autoencoder for video anomaly detection,” in Proceedingsof the 25th ACM international conference on Multimedia. ACM, 2017,
pp. 1933–1941.
[50] Q. Zhang, J. Wu, P. Zhang, G. Long, and C. Zhang, “Salient
subsequence learning for time series clustering,” IEEE transactions
on pattern analysis and machine intelligence, vol. 41, no. 9, pp. 2193–
2207, 2018.
[51] H. I. Fawaz, B. Lucas, G. Forestier, C. Pelletier, D. F. Schmidt,
J. Weber, G. I. Webb, L. Idoumghar, P.-A. Muller, and F. Petitjean,“Inceptiontime: Finding alexnet for time series classification,” Data
Mining and Knowledge Discovery, vol. 34, no. 6, pp. 1936–1962, 2020.

14[52] W. Tang, G. Long, L. Liu, T. Zhou, J. Jiang, and M. Blumenstein,“Rethinking 1d-cnn for time series classification: A stronger baseline,” arXiv preprint arXiv:2002.10061, 2020.
[53] Z. Wu, S. Pan, G. Long, J. Jiang, X. Chang, and C. Zhang, “Connecting the dots: Multivariate time series forecasting with graph neuralnetworks,” in Proceedings of the 26th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining, 2020, pp. 753–
763.
[54] B. Jun, “Fault detection using dynamic time warping (dtw) algorithm and discriminant analysis for swine wastewater treatment,”
Journal of hazardous materials, vol. 185, no. 1, pp. 262–268, 2011.
[55] W. Chen, S. Wang, G. Long, L. Yao, Q. Z. Sheng, and X. Li, “Dynamic illness severity prediction via multi-task rnns for intensive
care unit,” in 2018 IEEE International Conference on Data Mining(ICDM). IEEE, 2018, pp. 917–922.
[56] X. Zhang, L. Yao, C. Huang, S. Wang, M. Tan, G. Long, andC. Wang, “Multi-modality sensor data classification with selective
attention,” arXiv preprint arXiv:1804.05493, 2018.
[57] A. Smola, A. Gretton, L. Song, and B. Schölkopf, “A hilbert
space embedding for distributions,” in International Conference on
Algorithmic Learning Theory. Springer, 2007, pp. 13–31.
[58] K. Greff, R. K. Srivastava, J. Koutnı́k, B. R. Steunebrink, and
J. Schmidhuber, “Lstm: A search space odyssey,” IEEE transactions
on neural networks and learning systems, vol. 28, no. 10, pp. 2222–
2232, 2017.
[59] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,”
arXiv preprint arXiv:1412.3555, 2014.
[60] G. Liu and J. Guo, “Bidirectional lstm with attention mechanism
and convolutional layer for text classification,” Neurocomputing,vol. 337, pp. 325–338, 2019.
[61] A. Reiss and D. Stricker, “Introducing a new benchmarked dataset
for activity monitoring,” in 2012 16th International Symposium on
Wearable Computers. IEEE, 2012, pp. 108–109.
[62] M. G. Terzano, L. Parrino, A. Smerieri, R. Chervin, S. Chokroverty,
C. Guilleminault, M. Hirshkowitz, M. Mahowald, H. Moldofsky,
A. Rosa et al., “Atlas, rules, and recording techniques for thescoring of cyclic alternating pattern (cap) in human sleep,” Sleep
medicine, vol. 3, no. 2, pp. 187–199, 2002.
[63] H.-P. Kriegel, M. Schubert, and A. Zimek, “Angle-based outlier
detection in high-dimensional data,” in Proceedings of the 14th ACM
SIGKDD international conference on Knowledge discovery and data
mining. ACM, 2008, pp. 444–452.
[64] J. Ma and S. Perkins, “Time-series novelty detection using oneclass support vector machines,” in Proceedings of the International
Joint Conference on Neural Networks, 2003., vol. 3. IEEE, 2003, pp.
1741–1745.
[65] S. S. Joshi and V. V. Phoha, “Investigating hidden markov models
capabilities in anomaly detection,” in Proceedings of the 43rd annual
Southeast regional conference-Volume 1. ACM, 2005, pp. 98–103.
[66] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent
convolutional networks for visual recognition and description,”in Proceedings of the IEEE conference on computer vision and patternrecognition, 2015, pp. 2625–2634.
[67] W. Lu, Y. Cheng, C. Xiao, S. Chang, S. Huang, B. Liang, andT. Huang, “Unsupervised sequential outlier detection with deeparchitectures,” IEEE transactions on image processing, vol. 26, no. 9,
pp. 4321–4330, 2017.
[68] D. P. Subha, P. K. Joseph, R. Acharya, and C. M. Lim, “Eeg signal
analysis: a survey,” Journal of medical systems, vol. 34, no. 2, pp.
195–212, 2010.
[69] A. Phinyomark, C. Limsakul, and P. Phukpattaranont, “A novel
feature extraction for robust emg pattern recognition,” arXiv
preprint arXiv:0912.3973, 2009.
[70] M. K. Gautama and V. K. Giri, “An overview of feature extraction
techniques of ecg,” American-Eurasian Journal of Scientific Research,
vol. 12, no. 1, pp. 54–60, 2017.
[71] J. Bergstra, D. Yamins, and D. D. Cox, “Hyperopt: A python
library for optimizing the hyperparameters of machine learningalgorithms,” in Proceedings of the 12th Python in science conference.
Citeseer, 2013, pp. 13–20.
[72] N. Ketkar, “Introduction to keras,” in Deep learning with Python.
Springer, 2017, pp. 97–111.
[73] Y. Guo, W. Liao, Q. Wang, L. Yu, T. Ji, and P. Li, “Multidimensionaltime series anomaly detection: A gru-based gaussian mixture

variational autoencoder approach,” in Asian Conference on Machine
Learning, 2018, pp. 97–112.
[74] D. Rey and M. Neuhäuser, “Wilcoxon-signed-rank test,” Springer
Berlin Heidelberg.
[75] S. Kucherenko, B. Delpuech, B. Iooss, and S. Tarantola, “Application of the control variate technique to estimation of total
sensitivity indices,” Reliability Engineering & System Safety, vol. 134,
pp. 251–259, 2015.