Deep End-to-end Unsupervised Anomaly Detection 

1 INTRODUCTION . Anomaly detection refers to the identification of patterns that do not conform to expected normal behavior ( Chandola et al . ( 2009 ) ) . It is a critical task in diverse application domains such as fraud detection ( Phua et al . ( 2010 ) ) , intrusion detection ( Lazarevic et al . ( 2003 ) ) and surveillance video profiling ( Xiang & Gong ( 2008 ) ; Saligrama et al . ( 2010 ) ) . While the concept of an anomaly is intuitively easy for humans to understand , it is hard to define mathematically . Fundamentally , an anomaly is something with insufficient similarity to the rest of the data . This similarity can be computed on the basis of some feature difference . However , what makes an ideal feature representation for the data depends on what constitutes an anomaly . This forces anomaly detection into a chicken-or-egg problem in which there are a pair of problems , neither of which can be solved before the other . To date , a number of works have attempted this problem by training an autoencoder to create lowdimensional representations for anomaly detection ( Chalapathy et al . ( 2018 ) ; Zhai et al . ( 2016 ) ; Zong et al . ( 2018 ) ) . The anomalies are rejected and the autoencoder retrained ( Mishne & Cohen ( 2017 ) ; Wang et al . ( 2017 ) ) . While this gives reasonable results , it is fundamentally dependent on how well the first iteration solves the problem . We propose a solution in which anomalies can be defined using approximately correct features . This is achieved through an observation . Given a feature , anomalies approximately correspond to instances of high variance distributions . Such instances can be identified using a distribution-clustering ( Lin et al . ( 2018 ) ) framework . This hypothesis provides a reliable starting point for normal data selection . We train an autoencoder from the normal data subset , and iterate between hypothesizing normal candidate subset and representation learning . The reconstruction error serves as a scoring function to assess the normality of the data . The proposed framework does not rely on any training labels . Instead , it iteratively distills out anomalous data and improves the learned representation of normal data by incorporating clustering techniques into the process . We extensively assess the broad applicability of the proposed model on network intrusion , image and video data . Empirical results show that the proposed method outperforms the existing state-of-art approaches in terms of both accuracy and robustness to the percentage of anomalous data . 2 RELATED WORKS . Existing anomaly detection methods can be grouped into three categories . Reconstruction-based method These methods assume that anomalies are incompressible and thus can not be effectively reconstructed from low-dimensional projections . Classical methods like Principle Component Analysis ( PCA ) ( I.T.Jolliffe ( 1986 ) ) and Robust-PCA ( Candès et al . ( 2011 ) ) are motivated by this assumption . In recent works , different forms of deep autoencoder are proposed to analyze the reconstruction error . Xia et al . ( Xia et al . ( 2015 ) ) show that by introducing a regularizing term to a convolutional autoencoder , the anomalies tend to produce a bigger reconstruction error . Variational Autoencoder ( VAE ) ( An & Cho ( 2015 ) ) and Generative Adversarial Networks ( GANs ) ( Schlegl et al . ( 2019 ) ) have also been introduced to perform reconstruction-based anomaly detection . These methods demonstrate promising results when the anomaly ratio is fairly low . Although the reconstruction of anomalous samples , based on a reconstruction scheme optimized for normal data , tends to generate a higher error , a significant amount of anomalous samples could mislead the autoencoders to learn the correlations in the anomalous data instead . Density estimation and clustering Motivated by the assumption that anomalies occur less frequently , these algorithms treat anomalies as low-density regions in some feature space . Clustering analysis , such as Robust-KDE ( JooSeuk Kim & Scott ( 2008 ) ) , is often used for density estimation and anomaly detection . Unfortunately , due to the curse of dimensionality , these methods are less applicable to analysing high-dimensional data , where density estimation is a challenge in itself . A two-step approach is normally adopted to counter this issue , where dimensionality reduction is conducted first , followed by clustering analysis as a separate step . One drawback of this approach is that dimensionality reduction is trained without the guidance from the subsequent clustering analysis ; hence the key information for clustering analysis could be lost during dimensionality reduction . Recent works jointly learn dimensionality reduction and clustering components based on deep autoencoder ( Zhai et al . ( 2016 ) ; Zong et al . ( 2018 ) ) . Notably , DAGMM ( Zong et al . ( 2018 ) ) utilizes an autoencoder to generate a low-dimensional representation and its reconstruction error , which is further fed into an estimation network based on Gaussian Mixture Model ( GMM ) . However , as its autoencoder was trained on the whole dataset , it is vulnerable to a high percentage of anomalous samples and may learn wrong correlations . In contrast , our proposed method addresses this issue by first finding a normal candidate subset to train an autoencoder and then iterating between representation learning and refinement of the normal candidate . One-class classification One-class SVM ( Erfani et al . ; Chalapathy et al . ( 2018 ) ) is also widely used . Under this framework , a discriminative boundary surrounding the normal instances is learned by algorithms . However , when dimensionality goes higher , such techniques often suffer from suboptimal performance due to the curse of dimensionality . OCNN ( Chalapathy et al . ( 2018 ) ) attempted to circumvent this problem by using an autoencoder for dimensionality reduction . However , OCNN requires training data with relatively low anomaly ratio , in order to obtain an optimized NN model to differentiate anomalies from single-class normal data . 3 PROBLEM FORMULATION . Let X = { xi } , i = 1 , . . . , N , x ∈ Rk be the set of input data points that contains a certain percentage of anomaly . The goal of anomaly detection is to learn a scoring function h ( x ) , h : Rk 7→ R , to classify samples xi based on some threshold λ : yi = { 0 , if h ( xi ) < λ 1 , if h ( xi ) ≥ λ ( 1 ) where yi are the labels . yi = 0 indicates xi is normal and yi = 1 indicates anomalous . An overview of the proposed end-to-end anomaly detection system is presented in Fig . 1 . The major component of this system is an autoencoder that learns a low-dimensional representation of the input data that are often of high dimensions , to enable simplified modeling of the underlying distribution of the data . Under a fully unsupervised setting , the only information we are given is the set of input data X , without any label information . As an initialization , we leverage the latest clustering technique for high-dimensional data ( Lin et al . ( 2018 ) ) to provide soft supervisory signals . Since our input data is unlabelled , we derive a “ training ” set Strain , where Strain ⊂ X based on the following : Strain = C ( X , p0 ) ( 2 ) where C represents a selection process based on clustering output , and p0 represents the percentage of anomaly , it controls which are the clusters to be accepted into the “ training ” set . In our experiments , we compute the threshold as the ( 100− p0 ) th percentile of cluster variance , and accept clusters with variance smaller than this threshold . The assumption here is that clusters with large variance are likely to contain anomalous members . 3.1 SCORING FUNCTION LEARNING . The autoencoder network provides two sources of features : ( 1 ) a low-dimensional representation of the original input data ; and ( 2 ) the reconstruction error by comparing the input with its decoded counter-part . Using the training set Strain = { s1 , s2 , · · · , sM } , the autoencoder learns the encoding function fen : zc = fen ( s ; Θen ) , ∀s ∈ Strain , zc ∈ Rkbn ( 3 ) where Θen are the learned parameters for the encoder . zc are known as the bottle-neck features of dimension kbn . Similarly , for the decoding part , we have : x′ = fde ( zc ; Θde ) , ( 4 ) where Θde are the learned parameters for decoding . x′ are the reconstructed features . Upon training , we have a learned autoencoder with optimized parameters { Θen , Θde } . We run the entire input set X forward through the encoder network and produce a set of new features Z = { z1 , z2 , · · · , zN } , by concatenating the bottle-neck feature with the reconstruction error : z = [ zc ; zr ] , z ∈ Rkbn+1 , ( 5 ) where the reconstruction error zr is measured in terms of cosine similarity between x and its decoded counter-part : zr = d ( x , x ′ ) = cos−1 ( xTx′ ‖ x ‖‖ x′ ‖ ) ( 6 ) Note that Z is now of a much lower dimension than the input data X . Hence , traditional clustering techniques such as Gaussian Mixture Model would suffice for subsequent training set selection . To ensure the initial training set can capture most of the normal samples , we adopt a more conservative cluster variance threshold . With the new encoding scheme , the entire input set X is now represented as Z . We can “ re-label ” the training set X by using Z as a proxy . Similar to the initial training set selection , we select members that belong to low-variance clusters in Z . The process of Training set selection → Autoencoder training→ New feature computation is performed iteratively . The training set is updated as follows : Zt+1train = C ( Z t , p ) , ( 7 ) St+1train = { xj : ∀zj ∈ Z t+1 train } , ( 8 ) where the superscript t here refers to the tth iteration . Finally , the training process terminates when there is no change in the set of selected normal samples between two successive iterations . After the last iteration , t = tF , we obtain the autoencoder parameters { ΘtFen , Θ tF de } , and use it to construct the scoring function : h ( x ) = d ( x , x′ ) = d ( x , fde ( fen ( x ; Θ tF en ) ; Θ tF de ) ) , ( 9 ) where x′ is the result of going through the encoding-decoding process according to the trained autoencoder . 3.2 ALGORITHM . The proposed framework is summarized in Algorithm 1 . We obtain an initial split of the data into normal and abnormal subsets through clustering ( i.e . GMM for KDDCUP data and Distribution Clustering ( Lin et al . ( 2018 ) ) for image and video data ) . Selected outputs for CIFAR-10 and MNIST datasets using Distribution Clustering are presented in Fig . 4 in the appendix . We observe that as cluster variance increases , the samples ’ appearance become more anomalous . Algorithm 1 Deep end-to-end Unsupervised Anomaly Detection Input : X = { xi } , i = 1 , 2 ... , N : set of normal and anomalous input examples . r : number of epochs required for re-evaluation of the membership of the entire input set X. p0 and p : thresholds for initial and subsequent training set selection , respectively Output : Reconstruction-based anomaly score function h ( x ) and trained autoencoder { ΘtFen , Θ tF de } , 1 : procedure GET DECISION SCORE ( X , r , p , fen , fde ) 2 : Strain ← C ( X , p0 ) . Run clustering , select instances from low-variance clusters 3 : L = { k : ∀xk ∈ Strain } . L is the set of indices of selected normal training samples 4 : Lold : = ∅ 5 : while setdiff ( Lold , L ) 6= ∅ do 6 : for each epoch do 7 : if ( ( CurrentEpoch+ 1 ) mod r ) == 0 then . Re-evaluate normality every r epochs 8 : Zc ← fen ( X , Θen ) . Bottle-neck features 9 : X′ ← fde ( Zc , Θde ) 10 : Zr ← d ( X , X′ ) . Reconstruction error 11 : Z← [ Zc ; Zr ] 12 : Strain ← C ( Z , p ) . Get new training set according to threshold p 13 : Lold : = L 14 : L← Strain . Update set of indices for training samples 15 : else 16 : Train fen , fde on Strain to obtain { Θen , Θde } 17 : end for 18 : end while 19 : ΘtFen = Θen , Θ tF de = Θde 20 : Output h ( x ) according to finalized autoencoder { ΘtFen , Θ tF de } base on Eq . ( 9 ) 21 : end procedure Convergence Assuming a p % anomaly ratio , our algorithm starts with a tight cut-off , accepting clusters with variances below ( 100− p0 ) th percentile as an initial training set , where p0 > p. This ensures the initial training set is as pure as possible . Our assumption is that given partial normal data , the autoencoder would be able to learn a representation and generalize well on the “ unseen ” normal data that was discarded , and progressively recover them as iterations go on . Empirically , we plotted the AUROC , AUPRC and F-score for 20 iterations for the KDDCUP experiment , presented in Fig . 5 of the appendix . It demonstrates the convergence as iteration progresses . The same behavior was observed throughout our experiments on other datasets .

Summary: The paper proposed an unsupervised anomaly detection method for the scenarios where the training data not only includes normal data but also a lot of anomaly data. The basic idea of this paper is to iteratively refine the normal data subset, selected from the whole training data set. Specifically, the paper first train an auto-encoder (AE) and determine which are the normal data samples according to the reconstruction errors. Then, using the normal data to retrain the AE again, and re-select the normal samples. Repeat the above two steps until convergence.