Prediction Based Deep Autoencoding Model for Anomaly Detection
Zhanzhong Pang1, Xiaoyi Yu1, Jun Sun1 and Inakoshi Hiroya2
1 Fujitsu R&D Center Co., Ltd, Beijing, China;
2 Fujitsu Laboratories Ltd, Kawasaki, Japan{pangzhanzhong, yuxiaoyi, sunjun}@cn.fujitsu.com, inakoshi.hiroya@jp.fujitsu.com
Prediction mechanism(MLP)

Abstract:
Latent variables and reconstruction error are the two importantfeatures generated from an auto encoder. We propose a method
combining these two features together for anomaly detection. The
proposed architecture comprises of two networks. To compress andrebuild an input, a deep auto encoder is utilized where low
dimensional latent variables and reconstruction error can be obtained,
and compactness loss is introduced to maintain a low intra-variance inlatent variables for normal class. Meanwhile multi-layer perceptron(MLP) network which takes the generated latent variables as input isestablished aiming at predicting its corresponding reconstruction error.
By introducing MLP network, anomalies sharing similar reconstruction
error yet different distribution of latent variables to normal data or viceversa can be further separated. The prediction error form MLP
network is used as final score for anomaly detection. Experiments on
several benchmarks including image and multivariable datasets
demonstrate the effectiveness and practicability of this new approach
when comparing with several up-to-data algorithms.

◆ Connections between latent variables and reconstruction error only innormal data will be learned.
Latent variable

𝑝𝑛 → 𝑒𝑛1 𝑎𝑛𝑑 𝑒𝑛2

𝑒𝑛1

𝑒𝑛2

𝑒𝑎1

𝑒𝑎2

Normal 𝑝𝑛

▲★

★

----

Abnormal 𝑝𝑎--

--

▲★

★

𝑝𝑎 → 𝑒𝑎1 𝑎𝑛𝑑 𝑒𝑎2▲- separable only using reconstruction error.
★- separable using proposed method

Reconstruction error

Fig.2. Distributions of reconstruction error and latent variables on MNIST dataset: (a) construction
error distribution. 𝑒𝑛 for reconstruction error from normal data, 𝑒𝑎 for anomalies. The reconstruction
error can be further divided into two groups for each class according to its corresponding value. (b)latent variables distribution of samples in the bounding box of (a) (denoted as 𝑝)

Background Problem

EvaluationAnomaly detection without anomalies for training◆ Minimize the cost function:

 Traditional solutions

𝐽 𝜃𝑒 , 𝜃𝑑 , 𝜃𝑚 = 𝐽𝐷𝑒𝐴𝐸 𝜃𝑒 , 𝜃𝑑 + λ3 𝐽𝑀𝐿𝑃 𝜃𝑚

Self-reconstruction model and statistical analysis

1=𝑛𝑛𝑖=1𝑥𝑖 − 𝑥′𝑖 2 + 𝜆12𝑙𝜃𝑖=1 𝑖 + 𝜆2 𝐿C ,𝐽𝑀𝐿𝑃 𝜃𝑚

1=𝑛𝑛𝑖=1𝑒𝑖 − 𝑒′𝑖 2

 The proposed method

𝐽𝐷𝑒𝐴𝐸 𝜃𝑒 , 𝜃𝑑Auto encoder with prediction mechanism by jointly utilizing latentvariables and reconstruction error.

𝜃𝑒 , 𝜃𝑑 , 𝜃𝑖 and 𝜃𝑚 are network parameters in encoder, decoder, fully connected layers ofDeAE, and MLP. 𝑒𝑖 and 𝑒𝑖′ are reconstruction error and the predicted one from MLP.

◆ Prediction error 𝑠 from MLP for anomaly detection.

 Example application

𝑠𝑖 =Performance evaluation and event recognition𝑒𝑖 − 𝑒𝑖′ 2

𝑇𝑎𝑟𝑔𝑒𝑡 𝑐𝑙𝑎𝑠𝑠prediction =𝐴𝑛𝑜𝑚𝑎𝑙𝑦

Flowchart

𝑖𝑓 𝑠 < 𝜏𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

𝜏 is the predefined threshold.

◆ Average precision, recall and F1 score.

Experimental results
Test on two kinds of datasets◆ On image datasets: MNIST and CIFAR-10
Randomly choose one class as normal data. Treat other 9 classes as anomalies.
Fig.1. Overview of the proposed structure

Step 1 Low dimensional latent variables by encoder with constraint fromcompactness loss.
Step 2 Reestablishment of the input by decoder with reconstruction error.
Step 3 Prediction mechanism by MLP for connecting latent variables andreconstruction error.
◆ Latent variables as input.

Fig.3. F1 scores on two images datasets with different anomaly ratio◆ Reconstruction error as ground truth for guidance.

◆ On multi-variable datasets: KDD, Thyroid and Arrhythmia

Step 4 Prediction error for anomaly detection

Treat one class or combination of several classes as anomalies as same as [2].
Table.1. Average scores on three datasets with different models

The Proposed Method

MethodDeep auto encoder(DeAE)◆ Denoising auto encoder / convolutional denoising auto encoder◆ Compactness loss
1𝐿𝐶 =𝑛𝑘𝑛

𝑑𝑖𝑇 𝑑𝑖 , 𝒘𝒉𝒆𝒓𝒆 𝑑𝑖 =𝑖=1

1𝑧𝑖 − 𝑚𝑖 2 , 𝜃𝑚 =𝑛𝑛𝑧𝑖
𝑖=1

In a batch with size 𝑛, the compactness loss 𝐿𝐶 is defined as the average distance ofthe latent variables 𝑧𝑖 ∈ 𝑅𝑘 .

KDD

Thyroid

Arrhythmia

PrecRecF1PrecRecF1

Prec

Rec

F1OCSVM

0.74570.85230.79540.36390.42390.38870.62510.45450.5263

DSEBM-e[1]0.86190.64460.73990.68110.50540.58020.60540.52940.5650DAGMM[2]0.9711

0.94140.95590.65730.50530.57140.65690.46970.5487

AE

0.94950.88970.91850.61970.47310.5366

0.61110.50120.5493

Proposed

0.97790.95820.96790.67600.51610.58540.67270.5606

0.6115[1] Zhai, S.: “Deep Structured Energy Based Model for Anomaly Detection,” International Conference onMachine Learning, pp. 1100-1109, 2016.
[2] Zong, B.: “Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection,” 6-thInternational Conference on Learning Representations, 2018.