Year: 2019

Abstract:
State-of-the-art deep learning methods for outlier detection make the assumption that outliers will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming, we introduce a novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result in an overall penalty if they are mapped to low-probability regions. We show that if the global minimum of our newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than δ, and outliers to regions past this δ, δ being the inverse Chi Squared CDF evaluated at 1−α with α the percentage of outliers in the dataset. Our experiments confirm that OP-DMA consistently outperforms the state-of-art methods on a rich variety of outlier detection benchmark datasets.

heading: 1 INTRODUCTION
text: Background and Motivation. Outlier detection, the task of discovering rare or abnormal instances in a dataset, is critical for many applications from fraud detection, error identification in measurements, to fault detection in systems (2). Which points are outliers is often unknown in practice, and thus labeling for which points are outliers versus inliers is scarce or even completely unavailable. For this reason, in this work we focus on unsupervised methods for outlier detection (1).
In the ideal case in which inliers follow a simple unimodal distribution and outliers occur “far away” from the mean compared to the inliers, outlier detection has a simple solution. In this special case, outliers can be found by simply hardcoding some fixed distance as the cutoff between which points are classified as inlier or as outlier, such as is done by EllipticEnvelope (7). Unfortunately, in practice data is often not distributed in such a convenient manner (13). Rather, inliers are distributed in a complex, unknown fashion potentially with multiple clusters of inliers in the feature space (6).
Other methods with not quite as strict assumptions on the distribution of the data can work on arbitrary distributions, such as One-Class methods (4) and IsolationForest (3) to name just a few. These methods typically use a density-based approach (3; 4; 23). However, these methods are known to become infeasible in high dimensional data sets, because in sufficiently high dimensional spaces, all data points are known to be roughly equidistant and the space around all points will be sparse (5).
State-of-Art Deep Learning Methods for Outlier Detection. Deep learning methods for outlier detection not only overcome the limitations of density and distance based methods, but they can also leverage the inherent featurization capabilities of deep networks. Recently, deep learning methods use the reconstruction error of an autoencoder-based model for outlier detection, either by considering the reconstruction error itself as an outlier score (18; 20) or by leveraging the reconstruction error in a more sophisticated way (10; 19; 9). However, it has been shown that the usage of the mere reconstruction error can be a suboptimal choice for an outlier score because outliers tend to converge
to an average reconstruction error that is indistinguishable from that of inliers as the network converges (19). This is a greater issue for the case where outliers occur in clusters as opposed to uniform noise, as shown in Figure 1. The cluster outlier case is is likely to arise when the outliers represent a small minority class or when they are caused by a systematic error.
To go beyond this problem of the pure reconstruction error, recent works have leveraged the distribution-mapping capabilities of generative models such as Variational Autoencdoers (VAEs) that encourage the data in the latent space to follow a prior distribution (19; 21). These methods assume that due to their anomalous nature outlier points will not be mapped to the encouraged prior distribution while the inliers will be, and consequently outliers will occur in low-probability regions of the prior distribution (19; 21) [Figure 2 (b)]. However, it was observed that this widely held assumption is faulty in practice (21), with the distribution mapping continuing to map both inliers and outliers to high probability regions of the prior distribution for a wide variety of applications [Figure 2 (a)].
OP-DMA: Our Approach. In this work, we propose a method which leverages the distribution mapping capabilities of generative models, but unlike other distribution mapping outlier detection methods we no longer conveniently assume that outliers are not well mapped to the enforced prior distribution. Our core idea instead is to design a novel prior probability-weighted loss function that actively encourages outliers to be mapped to low probability regions in the latent space. Our method is an autoencoder architecture that we henceforth refer to as the Outlier Preserving Distribution Mapping Autoencoder (OP-DMA). The network is trained to minimize both the Maximum Mean Discrepancy (MMD) distance between the distribution of the data in the latent space and a multivariate Gaussian distribution, and what we call a prior-weighted L2 distance between the input and the reconstruction of the input. This prior-weighted distance weights each input datapoint point by the corresponding value of the multivariate Gaussian PDF evaluated at that point’s latent representation. The prior-weighted L2 distance encourages points with a high reconstruction error, as we assume outliers have, to be placed in low-probability regions in the latent space. This corresponds to the important observation that outlier will be mapped far from the mean of the data. The beauty of this mapping is that it thus allows us to use a simple off-the-shelf distance-based metric to identify outliers post-transformation. As the original distribution of the data in the feature space is arbitrary and no distance or density assumptions are placed on the data, OP-DMA thus corresponds to a highly versatile outlier detection strategy. Our key contributions are summarized as follows:
1. Introduce OP-DMA, a novel distribution-mapping autoencoder that preserves ourliers in the latent space without making asumptions on the distribution of the data in the feature space.
2. Prove that the optimal solution for OP-DMA places outliers further than√∫ 1−α 0 t−n/2−1e 1 2t
2 n 2 Γ( n2 )
dt according to the Mahalanobis distance, where α is the proba-
bility that draw from a given dataset is an outlier. This allows us to use a distance-based outlier detection classifier
3. Show experimentally that OP-DMA routinely outperforms standard distribution-mapping autoencoders for outlier detection on a variety of real-world benchmark datasets

heading: 2 RELATED WORK
text: Outlier Detection Using Autoencoders Coupled with Classical Outlier Detectors. Erfani et al. (11) used autoencoders for dimension reduction before classifying the projected data with a One Class SVM. Raghavendra et al. (12) introduce a One Class Neural Network (OC-NN), a feed-forward neural network inspired by One Class SVMs for the outlier-detection. The OC-NN method first pretrains an autoencoder before introducing a OC-NN layer after the encoder. Unlike our method (OP-DMA) these methods do not encourage outliers in the feature space to remain outliers in the latent space, making the strong and in practice often incorrect assumption that outliers in the feature space remain outliers in the latent space.
Recreation Error Methods. Zhou and Paffenroth (10) introduced Robust Deep Autoencoders (RDA), a deep network version of Robust Principal Component Analysis. It separates a given data matrix D into a low-rank matrix L and a sparse matrix S, where anomalous datapoints are isolated in S. This forces the network to reconstruct the instances in L, and punishes the network for putting instances into S. Chen et al. (9) introduced RandNet, an ensemble of autoencoders for outlier detection with outlier score corresponding to the average recreation error among the networks. Sabokrou et. al. introduced Adversarially Learned One-Class Classifiers (ALOCC) for Novelty Detection (27). ALOCC consists of a generator networkR which is trained to reconstruct noise-contaminated images by adversarially trainingR with D, a discriminator network which distinguishes between real images and reconstructed images. They show that the reconstruction error for R as well as the output of discriminator D are higher for outliers as opposed to inliers. Xia et. al. (28) used an autoencoder for outlier detection by adaptively grouping unlabeled input data into either an outlier or normal class, such that the network is trained to minimimize variance of reconstruction error within a class while increasing variance overall. However, as shown in (19), for autoencoders the reconstruction error of outliers often converges to that of inliers. This negatively impacts the performance of such reconstruction error methods.
Generative Outlier Detection Methods. Perera et. al. (21) proposed adversarial autoencoders with two discriminator networks for outlier detection. The first discriminator encourages the representation of the data in the latent space to be uniform, while the second encourages the output of the decoder conditioned on random samples from the uniform distribution to match real data samples. Vu et. al. (20) used an adversarial autoencoder to map the latent representation of their data to a standard normal distribution. They then combine the outlier score from a One Class SVM on the latent data with the One Class score on the reconstructed data as the final outlier score. Liu et. al. proposed MO-GAAL (26), a deep generative model which uses multiple generator networks to create outlier datapoints near subsets of the input data. A dsicriminator network is used to identify boundaries around the subsets of real data such that the generated outliers are rejected. These distribution mapping methods rely on the assumption that unlike the inliers, the outliers will be mapped to low-orobability regions of the prior distribution. However, as we discussed in Section 1, in practice both inliers and outliers are often well mapped to the prior distribution. This is because outliers that are mapped to low-probability regions will generally incur a higher cost from the divergence term which matches the latent distribution to the prior, and unlike in our OP-DMA approach there is not a corresponding term to counteract this cost.

heading: 3 PROPOSED METHOD: OUTLIER PRESERVING DISTRIBUTION MAPPING
text: The goal of OP-DMA is to find a latent representation of a dataset X such that outlier in X can be easily isolated from inliers in the latent representation of X. That is, we find a transformation f that we can apply to a given dataset X with a distribution PX such that f(X) ∼ PZ , where PX is an arbitrary
and unknown distribution of our input dataset X and PZ is the target distribution. Further, if a ∈ X is an outlier, then f(a) should be an outlier in f(X) as well. That is, pZ(f(a)) should be small. For the purpose of outlier detection, we chose PZ to be a Gaussian distribution as low probability regions would then simply be values far from the mean. We chose an autoencoder architecture to accomplish this distribution mapping, where f is selected to be the encoder Q parameterized by weights θ that are learned through stochastic backpropagation.
OP-DMA accomplishes the goal of mapping outliers to low probability regions on PZ by weighing the autoencoder’s reconstruction error by the likelihood of the latent data in the latent space with distribution PZ . In order to accomplish this, we need a one-to-one correspondence between each original datapoint, the latent representation of that datapoint, and the reconstructed point. Additionally, while we want the distribution of the latent data to follow PZ on the whole, we do not want all points to be mapped to high-probability regions of the prior. Rather, we must develop a solution that encourages some datapoints (i.e. outliers) to have a low likelihood in the prior distribution. For these reasons, a Wasserstein Autoencoder (WAE) is a more appropriate choice than the more common Variational Autoencoder (VAE). We chose WAEs, instead of the more commonly used VAEs, because the WAEs encourages the latent representations as a whole to match the prior, whereas the loss function of a VAE encourages each individual latent point to represent a distribution that matches the prior (15). This is counter to the goal of OP-DMA, which is to have some points (i.e., the outliers) mapped away from the prior distribution. Additionally, by formulating the reconstruction process as a Wassertein distance we can leverage the distribution mapping capabilities of the generative model while still operating with both a deterministic encoder and decoder. This is important as this allows us to maintain a one-to-one correspondence between a given point in the original space and its corresponding mapped point in the latent space so we can weight the reconstruction error by the value of the prior PDF evaluated in the latent space.

heading: 3.1 WASSERSTEIN AUTOENCODERS
text: As OP-DMA is an extension of WAE, we first briefly describe the structure of WAEs. WAEs are distribution-mapping autoencoders which minimize the Wasserstein distance between original data and its reconstruction. For original data X and reconstruction Y, it is defined as:
Wc(PX , PY ) := inf γ∈Γ
∫ ∫ c(x, y)γ(x, y)dxdy (1)
where Γ denotes the set of all joint distributions of X and Y , such that the marginal distributions Γ(X|Y ) and Γ(Y |X) are distributed according to PX and PY respectively, and c is a Wassersteindivergence cost function. That is, the c-Wasserstein distance is the expectation of the cost function c taken with respect to the joint distribution γ with marginals equal to the two priors, such that γ is a joint distribution that produces the minimum expectation of Equation 1.
Simply finding the Wasserstein distance, which is the distance that a WAE is trained to minimize, is itself an optimization problem. However, it is shown in (17) that when the encoded dataQ(X) ∼ PZ , where Q is the encoder network and PZ is the prior, then the search over all joint distributions previously required to compute the Wasserstein distance can be replaced with a search over all random encoders, infγ∈Γ E(X,Y )∼γ [c(X,G(Z))] = infQ:PQ=PZ EPXEQ(Z|X)[c(X,G(Z))]. This means we can minimize the c-Wasserstein distance by finding the weights of the encoder that produce the minimum expectation of cost c. As is done in (16) where WAE is defined, we can relax the constraint that PQ ∼ PZ by adding a penalty term D(PQ, PZ), where D is some function that measures the divergence between PQ and PZ . Thus,
Wλc (X,Y ) = inf Q EPXEQ(Z|X)[c(X,G(Z))] + λD(PQ, PZ), (2)
where λ is a constant weight term that determines how much the divergence of PQ from the prior PZ is penalized. In the case of the optimal solution for Q, Wλc (X,Y ) = Wc(X,Y ). In that case the Wasserstein objective will equal the true c-Wasserstein distance. While the WAE objective function encourages the distribution of the latent data as a whole to match the prior distribution, WAEs do not encourage outliers in the feature space to remain outliers in the latent space. Even though the
divergence term does not explicitly encourage every point to occur in a high probability region, in practice this is what happens. Consider when D is a discriminator network. The discriminator is likely to learn a boundary around the high probably region of the prior distribution, and the encoder network will be penalized for mapping an outlier to a low probablity region outside of the boundary as the discriminator would correctly identify it as a generated point. On the other hand, as described below, our proposed Outlier-Perserving Distribution Mapping Autoencoders actively encourages outliers to be mapped to low probability region in the latent space.

heading: 3.2 OUTLIER-PRESERVING DISTRIBUTION MAPPING AUTOENCODERS (OP-DMA)
text: So far, we have formulated the distribution mapping process in terms of random encoders and decoders Q and G. However, we can deterministically produce Q(X) and G(Q(X)|X) by requiring Q(X) = δµ(X), where µ is some function mapping X to Q(X). This allows us to have a one-to-one correspondence between input points X and output G(Q(X)) points, so that EPXEQ(X)[c(X,G(Q(X)))] can define a proper reconstruction error between each point x ∈ X and its corresponding output y ∈ Y . This reconstruction error can be used to determine which points should be mapped to low probability regions in the latent space. Let us define c′ := c(x,G(Q(x))) · pZ(Q(X)). We can thus define the OP-DMA objective function, Wλc′ , as:
Wλc′ = inf Q:PQ=PZ EPXEQ(Z|X)[c′(X,G(Z))] + λD(PQ, PZ) (3)
We can think of pZ(Q(X)) as a weight on the reconstruction term. Thus, high-reconstruction error points that are mapped to high-probability regions will be penalized more than high-reconstruction error points which are mapped to low probability regions. Since outliers are assumed to result in a high reconstruction error (at least during early epochs of training), by lessening the penalty to the network for poorly reconstructed points that have been mapped to low-probability regions of the prior we encourage the network to map outlier to these low-probability regions. However, the encoder isn’t encouraged to map every point (that is, both inliers and outliers) to low probability regions, as the D term makes it so that the distribution of all the encoded points on the whole should well match the prior.
Although we have made a significant modification to the Wasserstein loss function, as Theorem 1 states minimizing this cost function also corresponds to minimizing a lower bound on the nonprior-weighted Wasserstein divergence:. The proofs of all theorems below are provided in the appendix for space reasons. Theorem 1. Let PZ = N (µ,Σ). Then Wc′(X,G(Q(X))) ≤Wc(X,G(Q(X))) 1(2πn/2)|Σ|1/2 . Additionally, if |Σ| ≤ 1(4πn) then W λ c′(X,G(Q(X))) ≤Wλc (X,G(Q(X))) 1(2πn/2)|Σ|1/2 .
Additionally, not only is the cost function of OP-DMA a lower bound on the c-Wasserstein divergence, but also itself is a Wasserstein divergence (the c′-Wasserstein divergence): Theorem 2. Let Wc be a Wasserstein divergence. Then Wc′ is a Wasserstein divergence, with c’ the prior-weighted c.
As stated previously, our ideal mapping would place all inliers within regions where the probability was greater than some value, and all outliers into some regions where the probability is less than that value. Theorem 3 shows that this scenario is the optimal solution for the loss function of OP-DMA. Theorem 3. Let Q be an encoder such that D(PQ, PZ ,F) = 0, where D(A,B,F) is the Maximum Mean Discrepancy between A an B, F is the set of unbounded continuous functions and PZ = N (0,Σ). Let X : Ω → Rn be a centered random variable, X ∼ PX . Let X(A), A ⊂ Ω, be outliers and let H = Ω − A be the inliers, where ∫ X(A)
pX(x)dx = α. Furthermore, let c′(a,G(Q(a)) > c′(h,G(Q(h)) ∀ a ∈ X(A), h ∈ X(H). Then, the optimal solution of OP-DMA is
to map such that ‖Q(X(A))‖m ≥ δ and ‖Q(X(H))‖m < δ, where δ = √∫ 1−α 0 t−n/2−1e 1 2t
2 n 2 Γ( n2 )
dt and
‖ · ‖m is the Mahalanobis distance.
Thus, the optimal solution for OP-DMA’s cost function is one that maps outliers to regions with a larger Mahalanobis distance than that of inliers. This has the important implication that after

heading: Calculate
text: transformation with OP-DMA, outliers can be separated from inliers with a simple distance metric. This motivates out outlier detection scheme outlined in Section 3.3, where due to Theorem 1 we use the EllipticEnvelope method to identify outliers.

heading: 3.3 UNSUPERVISED OUTLIER DETECTION WITH OP-DMA
text: Lastly, we describe our end-to-end solution of outlier detection using OP-DMA. First, we transform the distribution of the dataset X to match a Gaussian distribution using OP-DMA. That is, the input dataset is transformed to match a prior distribution in the latent space of our autoencoder by a divergence term in the network’s loss function that penalizes the divergence between the latent distribution and the prior distribution. For the purpose of outlier detection, we chose this prior to be a multivariate Gaussian distribution with 0 mean and identity covariance. Then, EllipticEnvelope (7) is used to determine outliers by fitting a Gaussian to the transformed data. Theorem 3 tells us that EllipticEnvelope is an appropriate choice for the outlier detection step, as it determines outliers based on Mahalanobis distance. However, other outlier detection methods such as IsolationForest (3) or OneClassSVM (4) could equally be leveraged. We use the empirical rather than the theoretical mean and covariance for the EllipticEnvelope as it is unrealistic to expect any such deep distribution mapping model to exactly match the prior. The outlier detection process is outlined in Algorithm 1 and in Figure 2. As seen in Algorithm 1, we use the unbiased estimator of the MMD from Gretton et. al. (22). For the kernel k of the MMD we use the inverse multiquadratics kernel as is used in (16), and Mean Squared Error (MSE) for c. A high-level overview of the training and outlier detection process of OP-DMA is given in Figure 3.

heading: 4 EXPERIMENTAL EVALUATION

heading: 4.1 EXPERIMENTAL METHODOLOGY
text: Synthetic Data Sets. We determine possible failure cases for our method OP-DMA versus a standard autoencoder (AE). We use two synthetic datasets: (D1) Multimodal distribution in R4 with one cluster mean at (0,0,0,0) and another at (5,5,5,5) both with standard covariance and uniform noise added as outliers, and (D2) same multimodal distribution for inliers with a small cluster of outliers with a uniform probability in a small region between (-5,-5,-5,-5) and (-3,-3, -3, -3). In both cases, inliers account for 2.4% of the data.
Benchmark Data Sets. We compared OP-DMA to state-of-the-art methods from the ODDs database (8), a commonly used benchmark database of outlier detection datasets. Since we perform unsupervised outlier detection, we do not split each dataset into a training set and testing set. Instead,
Algorithm 1: Unsupervised Outlier Detection with OP-DMA Require :Regularization coefficient λ Contamination parameter α Initialized encoder network QΦ and decoder network GΘ with random weights Φ and Θ Dataset X while Θ, Φ not converged do
Sample {x1, x1, ..., xn} from X Sample {z1, z1, ..., zn} from N (0, I) Sample {z̃1, z̃1, ..., z̃n} from QΦ(Z|X) Update weights Φ and Θ by descending
1
n n∑ i=1 c(xi, GΘ(z̃i))·λ·pz(z̃i)+ 1 n(n− 1) ∑ h 6=j k(zh, zj)+ 1 n(n− 1) ∑ h6=j k(z̃h, z̃j)− 2 n2 ∑ h,j k(zh, z̃j)
end Find Dmin = {QΦ(xi), QΦ(xj), ..., QΦ(xk)}, ‖Dmin‖ = (1− α)‖D‖ with Minimum Covariance Determinant estimator, infΣ̃Det{Σ̃}. Find estimated mean µ̃ from Dmin return ‖QΦ(xi)‖m = (QΦ(xi)− µ̃)′Σ̃(QΦ(xi)− µ̃) for xi ∈ D as outlier scores
outlier detection is performed in an unsupervised manner on the entire dataset. In each dataset all points are labeled as either inlier or outlier. We use these labels to calculate the weighted F1 score, but no method we test is trained on the labels. The datasets were chosen such that we could test our methods on datasets with a wide range of dimensionality in the feature space and with a wide range of outlier contamination percentage. Table 3 (in Appendix B) breaks doen the number of featurs, datapoints, and percentage of outliers in each dataset.
Alternative Methods. In order to determine the validity of our OP-DMA, we compare it to several state-of-the-art distribution mapping outlier detection methods. These include methods that perform outlier detection on the latent space of a Wasserstein, a Variational Autoencoder, and an Adversarial Autoencoder (25) all with a Gaussian prior, here denoted as WAE, VAE, and AAE respectively. Additionally, we test against MO-GAAL (26) and ALOCC (27), two state-of-the-art deep generative outlier detection models. We also test against LOF (23) and OC-SVM (4), two state-of-the-art non-deep learning outlier detection method.
Parameter Configurations of Methods. The encoders and decoders of all methods consist of 3- layer neural networks, where the decoder in each encoder-decoder pair mirrors the structure of the encoder. The number of nodes in the hidden layer of each network is a hyperparameter chosen from {5, 6, 9, 15, 18, 100}. The number of nodes in the output/latent layer is chosen from {2, 3, 6, 9, 15}. The value of the regularization parameter λ is chosen such that the reconstruction error is on the same order of magnitude as the MMD error for the first epoch. We use the standard parameters of MO-GAAL from the authors’ publically released code. Likewise, we use the standard configuration of ALOC from the authors code, although we add an additional dense layer at the beginning of each subnetwork. We do this as ALOCC is a convolutional method and assumes input to be images of a certain shape. The additional dense layer we add to the model transforms the inut data from its original dimensionality into the required shape. We use the standard parameters from Scikit-Learn for LOF and OC-SVM.

heading: 4.2 EXPERIMENTAL RESULTS
text: Synthetic Experiments. Figure 4 A shows the ratio of the recreation error for outliers/inliers for OP-DMA versus an autoencoder (AE) for each synthetic dataset. For uniform noise (D1), both OP-DMA and AE feature a high ratio for the error of outlier/inliers. In this case, a standard AE would be adequate to find the outliers. However, for data where outliers are a small cluster (D2), the ratio between outliers to inliers goes quickly to 1 for AE whereas it levels off to ~10 for OP-DMA. In this case, our OP-DMA succeeds to identify the outliers, while AE cannot. We also verify that OP-DMA maps outliers to low-probability regions of the latent space. Figure 4 B shows the latent space of
OP-DMA for the uniform outlier dataset. The outliers occur far from the mean in low-probability regions. The average probability in the prior for outliers after transformation was 0.02, whereas the average probability of the inliers according to the prior was 0.08
Outlier Detection Accuracy. The results reported in Table 2 are the maximum weighted F1-scores of each method over all contamination rates from 0.1 to 1.0 with a step size of 0.1.
Running Time. In order to determine the feasibility of using OP-DMA instead of one of the leading methods in practice, we ran an experiment to determine the running times of each method. This consisted of performing 5 runs of OP-DMA, WAE, VAE, and AAE on the Satellite dataset (24). The average running time of each method over the 5 iterations is shown in the following table. Each method was run on a machine with 32GB of DDR4 RAM and a Intel Xeon Platinum 8160 CPU.

heading: 5 DISCUSSION
text: OP-DMA outperforms the state-of-the-art methods on the majority of the outlier detection datasets. This is because unlike the state-of-the-art methods, OP-DMA actively encourages outliers to be mapped to low probability regions instead of relying on outliers to be poorly mapped to the prior. While in this work, we focused on unsupervised outlier detection, OP-DMA could be extended to supervised outlier detection by making the following change to the objective function:
L = inf Q:PQ=PZ EPXinEQ(Z|X)[c(Xin, G(Q(Xin)))]
+ EPXoutEQ(Z|X)[c(Xin, G(Q(Xout))) · pZ(Q(Xout))] + λD(PQ, PZ),
where Xin corresponds to the set of labeled inlier points and Xout is the set of outliers. OP-DMA can also be modified for semisupervised OP-DMA by pretraining on labeled inliers without weighting the reconstruction error by the likelihood, and then performing OP-DMA as is done in the unsupervised version on the remaining unlabeled datapoints.

heading: 6 CONCLUSION
text: We have introduced OP-DMA, an autoencoder-based distribution mapping method for outlier detection that maps outliers in the feature space to low probability regions in the latent space in which a multivariate standard normal Gaussian prior distribution is enforced. Outliers are consequently

heading: Dataset OP-DMA WAE VAE AAE MO-GAAL ALOCC LOF OC-SVM
text: easily identifiable in the latent space. Our experimental study comparing OP-DMA to state-of-the-art methods on a collection of different benchmark outlier detection datasets shows that it outperforms WAE, VAE, and AAE on the majority of the datasets. We have also demonstrated that there is not a significant increase in running time between our method and state-of-the-art methods.

heading: Appendices
text: A

heading: PROOFS
text: Theorem 1. Let PZ = N (µ,Σ). Then Wc′(X,G(Q(X))) ≤Wc(X,G(Q(X))) 1(2πn/2)|Σ|1/2 . Additionally, if |Σ| ≤ 1(4πn) then W λ c′(X,G(Q(X))) ≤Wλc (X,G(Q(X))) 1(2πn/2)|Σ|1/2 .
Proof. We will first show that Wc′(X,G(Q(X))) ≤Wc(X,G(Q(X))) 1(2πn/2)|Σ|1/2 , and then show that Wλc′(X,G(Q(X))) ≤Wλc (X,G(Q(X))) 1(2πn/2)|Σ|1/2 .
The maximum value of pZ(z) will occur at z = µ. Then, sup pZ = 1(2πn/2)|Σ|1/2 . Thus
Wc′(X,G(Q(X))) = inf Q:PQ=PZ EPXEQ(Z|X)[c(X,G(Q(X))) · pZ(Q(X))] ≤ ( 1
(2πn/2)|Σ|1/2 ) ( inf Q:PQ=PZ EPXEQ(Z|X)[c(X,G(Q(X)))])
= 1
(2πn/2)|Σ|1/2 WC(X,G(Q(X)))
Similarly, to show that the prior-weighted c-Wasserstein divergence is a constant multiple of the true Wasserstein divergence:
Wλc′(X,G(Q(X))) = inf Q:PQ=PZ EPXEQ(Z|X)[c(X,G(Q(X))) · pZ(Q(X))] + λD(PQ, PZ) ≤ ( 1
(2πn/2)|Σ|1/2 ) ( inf Q:PQ=PZ EPXEQ(Z|X)[c(X,G(Q(X)))] + λD(PQ, PZ))
= 1
(2πn/2)|Σ|1/2 WλC(X,G(Q(X)))
Theorem 2. Let Wc be a Wasserstein divergence. Then Wc′ is a Wasserstein divergence, with c’ the prior-weighted c.
Proof. Since c is a Wasserstein divergence, we know that c(x1, x2) ≥ 0 ∀ x1, x2 ∈ supp(P ), c(x, x) = 0 ∀ x ∈ supp(P ), and Eγ [c(x1, x2)] ≥ 0 ∀ γ ∈ Γ[P, Pz]. Since PZ(z) ≥ 0 ∀ z, c′ will also fulfill the three aforementioned properties of c. Thus, Wc′ is a Wasserstein divergence.
Theorem 3. Let Q be an encoder such that D(PQ, PZ ,F) = 0, where D(A,B,F) is the Maximum Mean Discrepancy between A an B, F is the set of unbounded continuous functions and PZ = N (0,Σ). Let X : Ω → Rn be a centered random variable, X ∼ PX . Let X(A), A ⊂ Ω, be outliers and let H = Ω − A be the inliers, where ∫ X(A)
pX(x)dx = α. Furthermore, let c′(a,G(Q(a)) > c′(h,G(Q(h)) ∀ a ∈ X(A), h ∈ X(H). Then, the optimal solution of OP-DMA is
to map such that ‖Q(X(A))‖m ≥ δ and ‖Q(X(H))‖m < δ, where δ = √∫ 1−α 0 t−n/2−1e 1 2t
2 n 2 Γ( n2 )
dt and
‖ · ‖m is the Mahalanobis distance.
Proof. The Mahalanobis distance of Q(X) can itself be expressed as a random variable, δ =√ Q(X)Σ−1Q(X)T . Let Φδ be the CDF of δ. Then, Φδ(1 − α) = P (δ ≤ 1 − α) = P (δ2 ≤ (1− α)2) = Φδ2((1− α)2). Let Y = Q(X)M−1, where MTM = Σ is the Choleski decomposition of the covariance Σ. Since D(PQ, PZ ,F) = 0, andD(A,B,F) = 0 iffA = B, we thus know thatQ(X) ∼ N (0,Σ). Thus since
Q(X) is normally distributed and centered, Y thus is normally distributed with identity covariance. Since δ2 = Q(X)Σ−1Q(X)T = Y Y T , Φδ2 is the CDF of of the sum of squares of n normally distributed variables with mean 0 and σ = 1. Thus, Φδ2 is the Chi Squared distribution. The inverse Chi Squared CDF will therefore give us the distance δ such that 1−α percent of the points are within δ:
δ = √∫ 1−α 0 t−n/2−1e 1 2t 2 n 2 Γ(n2 ) dt
Now, let us assume that for some parameter choice Θ′ for Q that αP (Q(X(A)|Θ′) ≤ δ) = β, β > 0. Consequently, (1 − α)P (Q(X(H)|Θ′) > δ) = β, since P (Q(X) > δ) = α and∫ X(A)
pX(x)dx = α. Conversely, let us assume that there is a parameter configuration Θ such that αP (Q(X(A)|Θ) ≤ δ) = 0 and so (1− α)P (Q(X(H)|Θ) > δ) = 0. Since PZ ∼ N (0,Σ), pZ(d1) < pZ(d2) for ‖d1‖m > ‖d2‖m. Thus, since we assume c(a,G(Q(a)) > c(h,G(Q(h)) ∀ a ∈ X(A), h ∈ X(H), then EPXEQ(Z|X)c′(xp, G(Q(xp|Θ′))) = EPXEQ(Z|X)c(xp, G(Q(xp|Θ′)))pz(xp) > EPXEQ(Z|X)c(xp, G(Q(xp|Θ)))pz(xp) = EPXEQ(Z|X)c′(xp, G(Q(xp|Θ))). Thus, a mapping that places outliers δ away from the mean will have a lower loss than one that places some percentage of outliers less than δ from the mean.
B

heading: DATASETS
text: C

heading: SENSITIVITY TO CONTAMINATION PARAMETER
text: While the contamination parameter α is not used during the training of OP-DMA, it is used to fit the Elliptic Envelope on the encoded data after training. We test the sensitivity of this method to the value of the contamination parameter by evaluating the F1 score of outlier detection on the Satellite dataset after encoding the data with OP-DMA. The results are shown in the figure below.
This analysis shows that as long as the contamination parameter is not significantly less than the true contamination percentage the F1-score is robust to different values of the contamination parameter.

Review summaries:
The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution.