Versatile Anomaly Detection with Outlier Preserving Distribution Mapping Autoencoders

1 INTRODUCTION . Background and Motivation . Outlier detection , the task of discovering rare or abnormal instances in a dataset , is critical for many applications from fraud detection , error identification in measurements , to fault detection in systems ( 2 ) . Which points are outliers is often unknown in practice , and thus labeling for which points are outliers versus inliers is scarce or even completely unavailable . For this reason , in this work we focus on unsupervised methods for outlier detection ( 1 ) . In the ideal case in which inliers follow a simple unimodal distribution and outliers occur “ far away ” from the mean compared to the inliers , outlier detection has a simple solution . In this special case , outliers can be found by simply hardcoding some fixed distance as the cutoff between which points are classified as inlier or as outlier , such as is done by EllipticEnvelope ( 7 ) . Unfortunately , in practice data is often not distributed in such a convenient manner ( 13 ) . Rather , inliers are distributed in a complex , unknown fashion potentially with multiple clusters of inliers in the feature space ( 6 ) . Other methods with not quite as strict assumptions on the distribution of the data can work on arbitrary distributions , such as One-Class methods ( 4 ) and IsolationForest ( 3 ) to name just a few . These methods typically use a density-based approach ( 3 ; 4 ; 23 ) . However , these methods are known to become infeasible in high dimensional data sets , because in sufficiently high dimensional spaces , all data points are known to be roughly equidistant and the space around all points will be sparse ( 5 ) . State-of-Art Deep Learning Methods for Outlier Detection . Deep learning methods for outlier detection not only overcome the limitations of density and distance based methods , but they can also leverage the inherent featurization capabilities of deep networks . Recently , deep learning methods use the reconstruction error of an autoencoder-based model for outlier detection , either by considering the reconstruction error itself as an outlier score ( 18 ; 20 ) or by leveraging the reconstruction error in a more sophisticated way ( 10 ; 19 ; 9 ) . However , it has been shown that the usage of the mere reconstruction error can be a suboptimal choice for an outlier score because outliers tend to converge to an average reconstruction error that is indistinguishable from that of inliers as the network converges ( 19 ) . This is a greater issue for the case where outliers occur in clusters as opposed to uniform noise , as shown in Figure 1 . The cluster outlier case is is likely to arise when the outliers represent a small minority class or when they are caused by a systematic error . To go beyond this problem of the pure reconstruction error , recent works have leveraged the distribution-mapping capabilities of generative models such as Variational Autoencdoers ( VAEs ) that encourage the data in the latent space to follow a prior distribution ( 19 ; 21 ) . These methods assume that due to their anomalous nature outlier points will not be mapped to the encouraged prior distribution while the inliers will be , and consequently outliers will occur in low-probability regions of the prior distribution ( 19 ; 21 ) [ Figure 2 ( b ) ] . However , it was observed that this widely held assumption is faulty in practice ( 21 ) , with the distribution mapping continuing to map both inliers and outliers to high probability regions of the prior distribution for a wide variety of applications [ Figure 2 ( a ) ] . OP-DMA : Our Approach . In this work , we propose a method which leverages the distribution mapping capabilities of generative models , but unlike other distribution mapping outlier detection methods we no longer conveniently assume that outliers are not well mapped to the enforced prior distribution . Our core idea instead is to design a novel prior probability-weighted loss function that actively encourages outliers to be mapped to low probability regions in the latent space . Our method is an autoencoder architecture that we henceforth refer to as the Outlier Preserving Distribution Mapping Autoencoder ( OP-DMA ) . The network is trained to minimize both the Maximum Mean Discrepancy ( MMD ) distance between the distribution of the data in the latent space and a multivariate Gaussian distribution , and what we call a prior-weighted L2 distance between the input and the reconstruction of the input . This prior-weighted distance weights each input datapoint point by the corresponding value of the multivariate Gaussian PDF evaluated at that point ’ s latent representation . The prior-weighted L2 distance encourages points with a high reconstruction error , as we assume outliers have , to be placed in low-probability regions in the latent space . This corresponds to the important observation that outlier will be mapped far from the mean of the data . The beauty of this mapping is that it thus allows us to use a simple off-the-shelf distance-based metric to identify outliers post-transformation . As the original distribution of the data in the feature space is arbitrary and no distance or density assumptions are placed on the data , OP-DMA thus corresponds to a highly versatile outlier detection strategy . Our key contributions are summarized as follows : 1 . Introduce OP-DMA , a novel distribution-mapping autoencoder that preserves ourliers in the latent space without making asumptions on the distribution of the data in the feature space . 2 . Prove that the optimal solution for OP-DMA places outliers further than√∫ 1−α 0 t−n/2−1e 1 2t 2 n 2 Γ ( n2 ) dt according to the Mahalanobis distance , where α is the proba- bility that draw from a given dataset is an outlier . This allows us to use a distance-based outlier detection classifier 3 . Show experimentally that OP-DMA routinely outperforms standard distribution-mapping autoencoders for outlier detection on a variety of real-world benchmark datasets 2 RELATED WORK . Outlier Detection Using Autoencoders Coupled with Classical Outlier Detectors . Erfani et al . ( 11 ) used autoencoders for dimension reduction before classifying the projected data with a One Class SVM . Raghavendra et al . ( 12 ) introduce a One Class Neural Network ( OC-NN ) , a feed-forward neural network inspired by One Class SVMs for the outlier-detection . The OC-NN method first pretrains an autoencoder before introducing a OC-NN layer after the encoder . Unlike our method ( OP-DMA ) these methods do not encourage outliers in the feature space to remain outliers in the latent space , making the strong and in practice often incorrect assumption that outliers in the feature space remain outliers in the latent space . Recreation Error Methods . Zhou and Paffenroth ( 10 ) introduced Robust Deep Autoencoders ( RDA ) , a deep network version of Robust Principal Component Analysis . It separates a given data matrix D into a low-rank matrix L and a sparse matrix S , where anomalous datapoints are isolated in S. This forces the network to reconstruct the instances in L , and punishes the network for putting instances into S. Chen et al . ( 9 ) introduced RandNet , an ensemble of autoencoders for outlier detection with outlier score corresponding to the average recreation error among the networks . Sabokrou et . al . introduced Adversarially Learned One-Class Classifiers ( ALOCC ) for Novelty Detection ( 27 ) . ALOCC consists of a generator networkR which is trained to reconstruct noise-contaminated images by adversarially trainingR with D , a discriminator network which distinguishes between real images and reconstructed images . They show that the reconstruction error for R as well as the output of discriminator D are higher for outliers as opposed to inliers . Xia et . al . ( 28 ) used an autoencoder for outlier detection by adaptively grouping unlabeled input data into either an outlier or normal class , such that the network is trained to minimimize variance of reconstruction error within a class while increasing variance overall . However , as shown in ( 19 ) , for autoencoders the reconstruction error of outliers often converges to that of inliers . This negatively impacts the performance of such reconstruction error methods . Generative Outlier Detection Methods . Perera et . al . ( 21 ) proposed adversarial autoencoders with two discriminator networks for outlier detection . The first discriminator encourages the representation of the data in the latent space to be uniform , while the second encourages the output of the decoder conditioned on random samples from the uniform distribution to match real data samples . Vu et . al . ( 20 ) used an adversarial autoencoder to map the latent representation of their data to a standard normal distribution . They then combine the outlier score from a One Class SVM on the latent data with the One Class score on the reconstructed data as the final outlier score . Liu et . al . proposed MO-GAAL ( 26 ) , a deep generative model which uses multiple generator networks to create outlier datapoints near subsets of the input data . A dsicriminator network is used to identify boundaries around the subsets of real data such that the generated outliers are rejected . These distribution mapping methods rely on the assumption that unlike the inliers , the outliers will be mapped to low-orobability regions of the prior distribution . However , as we discussed in Section 1 , in practice both inliers and outliers are often well mapped to the prior distribution . This is because outliers that are mapped to low-probability regions will generally incur a higher cost from the divergence term which matches the latent distribution to the prior , and unlike in our OP-DMA approach there is not a corresponding term to counteract this cost . 3 PROPOSED METHOD : OUTLIER PRESERVING DISTRIBUTION MAPPING . The goal of OP-DMA is to find a latent representation of a dataset X such that outlier in X can be easily isolated from inliers in the latent representation of X . That is , we find a transformation f that we can apply to a given dataset X with a distribution PX such that f ( X ) ∼ PZ , where PX is an arbitrary and unknown distribution of our input dataset X and PZ is the target distribution . Further , if a ∈ X is an outlier , then f ( a ) should be an outlier in f ( X ) as well . That is , pZ ( f ( a ) ) should be small . For the purpose of outlier detection , we chose PZ to be a Gaussian distribution as low probability regions would then simply be values far from the mean . We chose an autoencoder architecture to accomplish this distribution mapping , where f is selected to be the encoder Q parameterized by weights θ that are learned through stochastic backpropagation . OP-DMA accomplishes the goal of mapping outliers to low probability regions on PZ by weighing the autoencoder ’ s reconstruction error by the likelihood of the latent data in the latent space with distribution PZ . In order to accomplish this , we need a one-to-one correspondence between each original datapoint , the latent representation of that datapoint , and the reconstructed point . Additionally , while we want the distribution of the latent data to follow PZ on the whole , we do not want all points to be mapped to high-probability regions of the prior . Rather , we must develop a solution that encourages some datapoints ( i.e . outliers ) to have a low likelihood in the prior distribution . For these reasons , a Wasserstein Autoencoder ( WAE ) is a more appropriate choice than the more common Variational Autoencoder ( VAE ) . We chose WAEs , instead of the more commonly used VAEs , because the WAEs encourages the latent representations as a whole to match the prior , whereas the loss function of a VAE encourages each individual latent point to represent a distribution that matches the prior ( 15 ) . This is counter to the goal of OP-DMA , which is to have some points ( i.e. , the outliers ) mapped away from the prior distribution . Additionally , by formulating the reconstruction process as a Wassertein distance we can leverage the distribution mapping capabilities of the generative model while still operating with both a deterministic encoder and decoder . This is important as this allows us to maintain a one-to-one correspondence between a given point in the original space and its corresponding mapped point in the latent space so we can weight the reconstruction error by the value of the prior PDF evaluated in the latent space .

Summary: The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution.