Deep End-to-end Unsupervised Anomaly Detection 

1 INTRODUCTION . Anomaly detection refers to the identification of patterns that do not conform to expected normal behavior ( Chandola et al . ( 2009 ) ) . It is a critical task in diverse application domains such as fraud detection ( Phua et al . ( 2010 ) ) , intrusion detection ( Lazarevic et al . ( 2003 ) ) and surveillance video profiling ( Xiang & Gong ( 2008 ) ; Saligrama et al . ( 2010 ) ) . While the concept of an anomaly is intuitively easy for humans to understand , it is hard to define mathematically . Fundamentally , an anomaly is something with insufficient similarity to the rest of the data . This similarity can be computed on the basis of some feature difference . However , what makes an ideal feature representation for the data depends on what constitutes an anomaly . This forces anomaly detection into a chicken-or-egg problem in which there are a pair of problems , neither of which can be solved before the other . To date , a number of works have attempted this problem by training an autoencoder to create lowdimensional representations for anomaly detection ( Chalapathy et al . ( 2018 ) ; Zhai et al . ( 2016 ) ; Zong et al . ( 2018 ) ) . The anomalies are rejected and the autoencoder retrained ( Mishne & Cohen ( 2017 ) ; Wang et al . ( 2017 ) ) . While this gives reasonable results , it is fundamentally dependent on how well the first iteration solves the problem . We propose a solution in which anomalies can be defined using approximately correct features . This is achieved through an observation . Given a feature , anomalies approximately correspond to instances of high variance distributions . Such instances can be identified using a distribution-clustering ( Lin et al . ( 2018 ) ) framework . This hypothesis provides a reliable starting point for normal data selection . We train an autoencoder from the normal data subset , and iterate between hypothesizing normal candidate subset and representation learning . The reconstruction error serves as a scoring function to assess the normality of the data . The proposed framework does not rely on any training labels . Instead , it iteratively distills out anomalous data and improves the learned representation of normal data by incorporating clustering techniques into the process . We extensively assess the broad applicability of the proposed model on network intrusion , image and video data . Empirical results show that the proposed method outperforms the existing state-of-art approaches in terms of both accuracy and robustness to the percentage of anomalous data . 2 RELATED WORKS . Existing anomaly detection methods can be grouped into three categories . Reconstruction-based method These methods assume that anomalies are incompressible and thus can not be effectively reconstructed from low-dimensional projections . Classical methods like Principle Component Analysis ( PCA ) ( I.T.Jolliffe ( 1986 ) ) and Robust-PCA ( Candès et al . ( 2011 ) ) are motivated by this assumption . In recent works , different forms of deep autoencoder are proposed to analyze the reconstruction error . Xia et al . ( Xia et al . ( 2015 ) ) show that by introducing a regularizing term to a convolutional autoencoder , the anomalies tend to produce a bigger reconstruction error . Variational Autoencoder ( VAE ) ( An & Cho ( 2015 ) ) and Generative Adversarial Networks ( GANs ) ( Schlegl et al . ( 2019 ) ) have also been introduced to perform reconstruction-based anomaly detection . These methods demonstrate promising results when the anomaly ratio is fairly low . Although the reconstruction of anomalous samples , based on a reconstruction scheme optimized for normal data , tends to generate a higher error , a significant amount of anomalous samples could mislead the autoencoders to learn the correlations in the anomalous data instead . Density estimation and clustering Motivated by the assumption that anomalies occur less frequently , these algorithms treat anomalies as low-density regions in some feature space . Clustering analysis , such as Robust-KDE ( JooSeuk Kim & Scott ( 2008 ) ) , is often used for density estimation and anomaly detection . Unfortunately , due to the curse of dimensionality , these methods are less applicable to analysing high-dimensional data , where density estimation is a challenge in itself . A two-step approach is normally adopted to counter this issue , where dimensionality reduction is conducted first , followed by clustering analysis as a separate step . One drawback of this approach is that dimensionality reduction is trained without the guidance from the subsequent clustering analysis ; hence the key information for clustering analysis could be lost during dimensionality reduction . Recent works jointly learn dimensionality reduction and clustering components based on deep autoencoder ( Zhai et al . ( 2016 ) ; Zong et al . ( 2018 ) ) . Notably , DAGMM ( Zong et al . ( 2018 ) ) utilizes an autoencoder to generate a low-dimensional representation and its reconstruction error , which is further fed into an estimation network based on Gaussian Mixture Model ( GMM ) . However , as its autoencoder was trained on the whole dataset , it is vulnerable to a high percentage of anomalous samples and may learn wrong correlations . In contrast , our proposed method addresses this issue by first finding a normal candidate subset to train an autoencoder and then iterating between representation learning and refinement of the normal candidate . One-class classification One-class SVM ( Erfani et al . ; Chalapathy et al . ( 2018 ) ) is also widely used . Under this framework , a discriminative boundary surrounding the normal instances is learned by algorithms . However , when dimensionality goes higher , such techniques often suffer from suboptimal performance due to the curse of dimensionality . OCNN ( Chalapathy et al . ( 2018 ) ) attempted to circumvent this problem by using an autoencoder for dimensionality reduction . However , OCNN requires training data with relatively low anomaly ratio , in order to obtain an optimized NN model to differentiate anomalies from single-class normal data . 3 PROBLEM FORMULATION . Let X = { xi } , i = 1 , . . . , N , x ∈ Rk be the set of input data points that contains a certain percentage of anomaly . The goal of anomaly detection is to learn a scoring function h ( x ) , h : Rk 7→ R , to classify samples xi based on some threshold λ : yi = { 0 , if h ( xi ) < λ 1 , if h ( xi ) ≥ λ ( 1 ) where yi are the labels . yi = 0 indicates xi is normal and yi = 1 indicates anomalous . An overview of the proposed end-to-end anomaly detection system is presented in Fig . 1 . The major component of this system is an autoencoder that learns a low-dimensional representation of the input data that are often of high dimensions , to enable simplified modeling of the underlying distribution of the data . Under a fully unsupervised setting , the only information we are given is the set of input data X , without any label information . As an initialization , we leverage the latest clustering technique for high-dimensional data ( Lin et al . ( 2018 ) ) to provide soft supervisory signals . Since our input data is unlabelled , we derive a “ training ” set Strain , where Strain ⊂ X based on the following : Strain = C ( X , p0 ) ( 2 ) where C represents a selection process based on clustering output , and p0 represents the percentage of anomaly , it controls which are the clusters to be accepted into the “ training ” set . In our experiments , we compute the threshold as the ( 100− p0 ) th percentile of cluster variance , and accept clusters with variance smaller than this threshold . The assumption here is that clusters with large variance are likely to contain anomalous members . 3.1 SCORING FUNCTION LEARNING . The autoencoder network provides two sources of features : ( 1 ) a low-dimensional representation of the original input data ; and ( 2 ) the reconstruction error by comparing the input with its decoded counter-part . Using the training set Strain = { s1 , s2 , · · · , sM } , the autoencoder learns the encoding function fen : zc = fen ( s ; Θen ) , ∀s ∈ Strain , zc ∈ Rkbn ( 3 ) where Θen are the learned parameters for the encoder . zc are known as the bottle-neck features of dimension kbn . Similarly , for the decoding part , we have : x′ = fde ( zc ; Θde ) , ( 4 ) where Θde are the learned parameters for decoding . x′ are the reconstructed features . Upon training , we have a learned autoencoder with optimized parameters { Θen , Θde } . We run the entire input set X forward through the encoder network and produce a set of new features Z = { z1 , z2 , · · · , zN } , by concatenating the bottle-neck feature with the reconstruction error : z = [ zc ; zr ] , z ∈ Rkbn+1 , ( 5 ) where the reconstruction error zr is measured in terms of cosine similarity between x and its decoded counter-part : zr = d ( x , x ′ ) = cos−1 ( xTx′ ‖ x ‖‖ x′ ‖ ) ( 6 ) Note that Z is now of a much lower dimension than the input data X . Hence , traditional clustering techniques such as Gaussian Mixture Model would suffice for subsequent training set selection . To ensure the initial training set can capture most of the normal samples , we adopt a more conservative cluster variance threshold . With the new encoding scheme , the entire input set X is now represented as Z . We can “ re-label ” the training set X by using Z as a proxy . Similar to the initial training set selection , we select members that belong to low-variance clusters in Z . The process of Training set selection → Autoencoder training→ New feature computation is performed iteratively . The training set is updated as follows : Zt+1train = C ( Z t , p ) , ( 7 ) St+1train = { xj : ∀zj ∈ Z t+1 train } , ( 8 ) where the superscript t here refers to the tth iteration . Finally , the training process terminates when there is no change in the set of selected normal samples between two successive iterations . After the last iteration , t = tF , we obtain the autoencoder parameters { ΘtFen , Θ tF de } , and use it to construct the scoring function : h ( x ) = d ( x , x′ ) = d ( x , fde ( fen ( x ; Θ tF en ) ; Θ tF de ) ) , ( 9 ) where x′ is the result of going through the encoding-decoding process according to the trained autoencoder . 3.2 ALGORITHM . The proposed framework is summarized in Algorithm 1 . We obtain an initial split of the data into normal and abnormal subsets through clustering ( i.e . GMM for KDDCUP data and Distribution Clustering ( Lin et al . ( 2018 ) ) for image and video data ) . Selected outputs for CIFAR-10 and MNIST datasets using Distribution Clustering are presented in Fig . 4 in the appendix . We observe that as cluster variance increases , the samples ’ appearance become more anomalous . Algorithm 1 Deep end-to-end Unsupervised Anomaly Detection Input : X = { xi } , i = 1 , 2 ... , N : set of normal and anomalous input examples . r : number of epochs required for re-evaluation of the membership of the entire input set X. p0 and p : thresholds for initial and subsequent training set selection , respectively Output : Reconstruction-based anomaly score function h ( x ) and trained autoencoder { ΘtFen , Θ tF de } , 1 : procedure GET DECISION SCORE ( X , r , p , fen , fde ) 2 : Strain ← C ( X , p0 ) . Run clustering , select instances from low-variance clusters 3 : L = { k : ∀xk ∈ Strain } . L is the set of indices of selected normal training samples 4 : Lold : = ∅ 5 : while setdiff ( Lold , L ) 6= ∅ do 6 : for each epoch do 7 : if ( ( CurrentEpoch+ 1 ) mod r ) == 0 then . Re-evaluate normality every r epochs 8 : Zc ← fen ( X , Θen ) . Bottle-neck features 9 : X′ ← fde ( Zc , Θde ) 10 : Zr ← d ( X , X′ ) . Reconstruction error 11 : Z← [ Zc ; Zr ] 12 : Strain ← C ( Z , p ) . Get new training set according to threshold p 13 : Lold : = L 14 : L← Strain . Update set of indices for training samples 15 : else 16 : Train fen , fde on Strain to obtain { Θen , Θde } 17 : end for 18 : end while 19 : ΘtFen = Θen , Θ tF de = Θde 20 : Output h ( x ) according to finalized autoencoder { ΘtFen , Θ tF de } base on Eq . ( 9 ) 21 : end procedure Convergence Assuming a p % anomaly ratio , our algorithm starts with a tight cut-off , accepting clusters with variances below ( 100− p0 ) th percentile as an initial training set , where p0 > p. This ensures the initial training set is as pure as possible . Our assumption is that given partial normal data , the autoencoder would be able to learn a representation and generalize well on the “ unseen ” normal data that was discarded , and progressively recover them as iterations go on . Empirically , we plotted the AUROC , AUPRC and F-score for 20 iterations for the KDDCUP experiment , presented in Fig . 5 of the appendix . It demonstrates the convergence as iteration progresses . The same behavior was observed throughout our experiments on other datasets .

Summary: In the article, the authors solve the problem of anomaly detection using a fully unsupervised approach. They try to deal with the main challenge of anomaly detection: a lack of certainty on what defines normal data and anomaly one. For this purpose, the authors iteratively use: 1) autoencoders to learn the representation of the data; 2) applying in the latent space clustering to get a new training set and retrain autoencoders. The experimental results show that the author’s method performed better results than such a baseline model as one-class SVM and one-class NN.