Outlier Preserving Distribution Mapping Autoencoders 

1 INTRODUCTION . Background . Outlier detection , the task of discovering abnormal instances in a dataset , is critical for applications from fraud detection , error measurement identification to system fault detection ( Singh & Upadhyaya , 2012 ) . Given outliers are by definition rare , it is often infeasible to get enough labeled outlier examples that are represetnative of all the forms the outliers could take . Consequently , unsupervised outlier detection methods that do not require prior labeling of inliers or outliers are frequently adopted ( Chandola et al. , 2009 ) . State-of-Art Deep Learning Methods for Outlier Detection . Deep learning methods for outlier detection commonly utilize the reconstruction error of an autoencoder model as an outlier score for outlier detection ( Sakurada & Yairi , 2014 ; Vu et al. , 2019 ) . However , directly using the reconstruction error as the outlier score has a major flaw . As the learning process converges , both outliers and inliers tend to converge to the average reconstruction error ( to the same outlier score ) – making them indistinguishable ( Beggel et al. , 2019 ) . This is demonstrated in Figure 1a , which shows that the ratio of average reconstruction error for outliers converges to that of the inliers . To overcome this shortcoming , recent work ( Beggel et al. , 2019 ; Perera et al. , 2019 ) utilizes the distribution-mapping capabilities of generative models that encourage data to follow a prior distribution in the latent space . These cutting-edge methods assume that while the mapping of inlier points will follow the target prior distribution , outliers will not due to their anomalous nature . Instead , outliers will be mapped to low-probability regions of the prior distribution , making it easy to detect them as outliers ( Beggel et al. , 2019 ; Perera et al. , 2019 ) . However , this widely held assumption has been shown to not hold in practice ( Perera et al. , 2019 ) . Unfortunately , as shown in Figure 1b , both inliers and outliers are still mapped to the same high probability regions of the target prior distribution , making them difficult to distinguish . Problem Definition . Given a given dataset X ∈ RM of multivariate observations , let f : RM → RN , N ≤M , be a function from the multivariate feature space of X to a latent space f ( x ) ∈ RN such that f ( X ) ∼ PZ , where PZ is a known and tractable prior probability density function . The dataset X ∈ RM is composed as X = XO + XI , where XO and XI are a set of outlier and inlier points , respectively . During training , it is unknown whether any given point x ∈ X is an outlier or an inlier . Intuitively , our goal is to find a function f that maps instances of a dataset X into a latent space S with a known distribution , such that outliers are mapped to low probability regions and inliers to high probability regions . More formally , we define unsupervised distribution-mapping outlier detection as the problem of finding a function f∗ with the aforementioned properties of f such that we maximize the number of outliers xo ∈XO and inliers xi ∈XI for which PZ ( f∗ ( xo ) ) < PZ ( f ∗ ( xi ) ) holds . Challenges . To address the open problem defined above , the following challenges exist : 1 . Overpowering divergence penalty . Intuitively , distribution mapping methods utilize a divergence penalty to achieve a latent space mapping of input data that has a high probability of following a target prior distribution . While the data overall should follow this prior distribution , a solution must be found to instead maps outliers to low-probability regions of the prior . Having the data match the prior overall , while having outliers mapped to low probability regions of the prior creates a conflict , as the two tasks are diametrically opposed . To achieve such a mapping requires overpowering the divergence penalty in order to map outliers to low probability regions in the latent space . 2 . Unknown outlier status . In unsupervised outlier detection , during training points do not have any labels indicating whether they are outliers or inliers . This unsupervised scenario , while common in practice ( Singh & Upadhyaya , 2012 ) , makes it challenging to design strategies that explicitly coerce outliers to be mapped to low-probability regions . Our OP-DMA Approach . In this work , we propose the Outlier Preserving Distribution Mapping Autoencoder ( OP-DMA ) . Our core idea is to propose a novel Prior Weighted Loss ( PWL ) function that solves the two conflicting tasks of mapping the input data to a prior distribution while encouraging outliers to be mapped to low probability regions of that prior . This PWL directly addresses the shortcomings of the existing distribution mapping outlier detection methods ( Vu et al. , 2019 ; Perera et al. , 2019 ) , and to the best of our knowledge is the first unsupervised cost function that explicitly encourages outliers to be mapped to low probability regions . We assume that outliers will have a high reconstruction error during the initial stages of training , which causes the PWL to place them in low-probability ( low PDF ) regions in the latent space . This way , PWL overcomes the challenge of overpowering the divergence penalty . It succeeds in mapping outliers to low-probability regions ( far from the mean of the latent distribution ) even though each input point ’ s outlier status is unknown . Our OP-DMA framework is pluggable , meaning off-theshelf distance-based outlier methods can be flexibly plugged in post-transformation . Our key contributions are as follows : 1 . Propose OP-DMA , a novel distribution-mapping autoencoder that effectively separates outliers from inliers in the latent space without knowing nor making assumptions on the original distribution of the data in the feature space . 2 . Design the Prior-Weighted Loss ( PWL ) , which when coupled with a divergence penalty encourages outliers to be mapped to low-probability regions while inliers are mapped to high-probability regions of the latent space of an autoencoder . 3 . Provide rigorous theoretical proof that the optimal solution for OP-DMA places outliers further than inliers from the mean of the distribution of the data in the latent space . 4 . Demonstrate experimentally that OP-DMA consistently outperforms other state-of-art outlier detection methods on a rich variety of real-world benchmark outlier datasets . Significance : OP-DMA is a versatile outlier detection strategy as it can handle input data that has arbitrary distributions in the feature space , while not making any distance or density assumptions on the data . To the best of our knowledge , we are the first to propose a loss function that explicitly encourages outliers to be mapped to low-probability regions while inliers are mapped to high probability regions . Our PWL approach is pluggable , and can easily be incorporated into alternate outlier detectors . Our ideas could also spur further research into various prior weighted loss functions . 2 RELATED WORK . State-of-the-art deep outlier detection methods fall into one of three categories : 1 ) Autoencoders coupled with classic outlier detectors ( Erfani et al. , 2016 ; Chalapathy et al. , 2018 ) , 2 ) Reconstruction error-based outlier detection methods ( Zhou & Paffenroth , 2017 ; Chen et al. , 2017 ; Sabokrou et al. , 2018 ; Xia et al. , 2015 ) , or 3 ) Generative outlier detection methods ( Perera et al. , 2019 ; Vu et al. , 2019 ; Liu et al. , 2019 ) . 1 ) Autoencoders coupled with classic outlier detectors project data into a lower dimensional latent space before performing outlier detection on that latent representation . These methods make the strict assumption that outliers in the original space will remain outliers in the latent space . Further , they fail to explicitly encourage this in the mapping function . 2 ) Reconstruction error-based outlier detection methods utilize the reconstruction error of an autoencoder network to identify outliers . They typically use the reconstruction error directly as the anomaly score ( An & Cho , 2015 ) . In more recent work , they try to separate outliers into a separate low-rank matrix analogous to RPCA ( Zhou & Paffenroth , 2017 ) or they introduce a separate discriminator network ( Sabokrou et al. , 2018 ) . However , as shown in ( Beggel et al. , 2019 ) , for autoencoders the reconstruction error of outliers often converges to that of inliers . This negatively impacts the performance of such reconstruction error methods . 3 ) Generative outlier detection methods leverage deep generative models ( Goodfellow et al. , 2014 ; Kingma & Welling , 2013 ) to generate the latent space such that the distribution of the latent space is encouraged to match a known prior so that thereafter an appropriate outlier method for the prior can be applied ( Vu et al. , 2019 ) to the latent space , or a discriminator can identify outliers in the latent space ( Vu et al. , 2019 ) or both the latent space and reconstructed space ( Perera et al. , 2019 ) . However , as discussed in Section 1 , in practice both inliers and outliers are both mapped to the prior distribution as outliers that are mapped to low-probability regions will generally incur a high cost from the divergence term which matches the latent distribution to the prior . OP-DMA shares characteristics with each of these three categories . However , unlike the other methods in these categories , OP-DMA actively encourages outlier to be mapped to low-probability regions instead of just assuming that this will be the case . OP-DMA is is a generative outlier method that uses the reconstruction error to encourage outliers to be mapped to low-probability regions . Further , it can flexibly be paired with nearly any classic outlier detector after distribution mapping . 3 PROPOSED APPROACH : OP-DMA . Overview of approach . OP-DMA consists of three main components : 1 . A distribution mapping autoencoder ( DMA ) that OP-DMA utilizes to map a datasetX from the feature space RM into a lower dimensional latent space RN , such that the distribution of the encoded data in the lower dimensional latent space has a known probability distribution PZ . This step is crucial as it makes it easy for OP-DMA to easily identify low probability regions of the latent space ( outliers should be mapped here ) . This can be done because after the distribution mapping , we can explicitly calculate the Probability Density Function ( PDF ) of the latent space so long as we selected a prior distribution with a known PDF . 2 . A novel Probability-Weighted Loss ( PWL ) function for distribution mapping that encourages outliers to be mapped to low-probability regions of the latent space , solving both the challenges of overpowering divergence penalty and unknown outlier status . 3 . An traditional outlier detection method is used to identify outliers in the transformed latent space . The choice of outlier detection method is flexible as long as it is amenable to the prior distribution PZ selected in step 1 of OP-DMA . For instance , when a Gaussian distribution is used for the prior , then OP-DMA utilizes a classical distance-based outlier detection method for step 3 . These steps are described in the following subsections and illustrated in Figure 2 . 3.1 DISTRIBUTION MAPPING AUTOENCODER ( DMA ) . In order to use prior-weighting to map outliers to low-probability regions of a known PDF in a latent space , our distribution mapping method must meet two design requirements : 1 . A one-to-one mapping between each original data point , its latent representation and the reconstructed data point must be established so that each data point ’ s reconstructed data point is unique and can be determined , and vice versa . 2 . The divergence term must impose a cost based on how well a batch of latent data points match the prior overall , rather than requiring individual data points to have a high probability of being a draw from the prior . To meet these requirements , we select the Wasserstein AutoEncoder ( WAE ) ( Tolstikhin et al. , 2017 ) as the foundation for our distribution mapping . WAEs are distribution-mapping autoencoders that minimize the Wasserstein distance between the original data and its reconstruction , while mapping the input data to a latent space with a known prior distribution . To see why we base our distributionmapping technique on this method , consider the WAE objective function for encoder networkQ and decoder network G : Wλc ( X , Y ) = Reconstruction Error︷ ︸︸ ︷ inf Q EPXEQ ( Z|X ) [ c ( X , G ( Z ) ) ] + Divergence Penalty︷ ︸︸ ︷ λD ( PQ , PZ ) . ( 1 ) The first term on the right hand side of Equation 1 corresponds to the reconstruction error between the input data and reconstructed data for cost function c. The second term D is a divergence penalty between the distribution of the latent space and the prior distribution , with λ a constant weight term that determines how much that divergence is penalized . Let us deterministically produce the latent representation Q ( X ) and output G ( Q ( X ) |X ) ( by using Q ( X ) = δµ ( X ) , where µ is some function mapping input data set X to Q ( X ) , for instance ) . It is now clear why Wasserstein autoencoders are an appropriate choice to model our distribution mapping method , as the reconstruction error term EPXEQ ( Z|X ) [ c ( X , G ( Z ) ) ] in Equation 1 represents a one-to-one correspondence between input data , its latent representation and the reconstructed output ( meeting requirement 1 ) . Additionally , D is a batch-level cost term that would be incurred if the latent representation of a batch doesn ’ t match the prior distribution but doesn ’ t require individual points to be mapped to a high probability region of the prior ( meeting requirement 2 ) . However , we note that WAEs unfortunately do not encourage outliers in the feature space to remain outliers in the latent space . Consider D to be a discriminator network . Then D is likely to learn a boundary around the high probability region of the prior distribution . Thus the encoder network Q will be penalized for mapping an outlier to a low probability region outside of the boundary found by D as the discriminator D would correctly identify it as a generated point .

Summary: The paper proposes an autoencoder-based outlier detection system. The main idea of the paper is to ensure that outlier points are mapped to areas distant from the inliers in the embedding space. To this end, a novel cost function is introduced, which weighs the reconstruction error based on a prior distribution for the embedding space. This cost function hopes to force inliers to be mapped to the high probability region of the prior distribution and push outliers to low probability regions. Combined with a normal/multivariate prior distribution, this then enables the use of simple distance-based outlier detection methods.