Versatile Anomaly Detection with Outlier Preserving Distribution Mapping Autoencoders
Authors: Walter Gerych
Elke Rundensteiner
Emmanuel Agu
Venue:  (2020)
Keywords: Anomaly detection
outliers
deep learning
distribution mapping
wasserstein autoencoders

Abstract:  State-of-the-art deep learning methods for outlier detection make the assumption that anomalies will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, because the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming,  we introduce a novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of an autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preserving distribution mapping through weighting the reconstruction error of individual points by the value of a multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result overall penalty if they are mapped to low-probability regions. We show that if the  global minimum of our newly proposed loss function is achieved, 
 then our OP-DMA  maps inliers to regions with a Mahalanobis distance less than delta, and outliers to regions past this delta, delta being the inverse Chi Squared CDF evaluated at (1-alpha) with alpha the percentage of outliers in the dataset. Our experiments confirm that OP-DMA  consistently outperforms  the state-of-art  methods on a rich variety of outlier detection benchmark datasets.

Under review as a conference paper at ICLR 2020
VERSATILE OUTLIER DETECTION WITH OUTLIER PRE- SERVING DISTRIBUTION MAPPING AUTOENCODERS
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-arr deep learning methods for outlier detection make the assumption that outliers will appear far away from inlier data in the latent space produced by distribution mapping deep networks. However, this assumption fails in practice, be- cause the divergence penalty adopted for this purpose encourages mapping outliers into the same high-probability regions as inliers. To overcome this shortcoming, we introduce novel deep learning outlier detection method, called Outlier Preserving Distribution Mapping Autoencoder (OP-DMA), which succeeds to map outliers to low probability regions in the latent space of autoencoder. For this we leverage the insight that outliers are likely to have a higher reconstruction error than inliers. We thus achieve outlier-preservee reconstruction error of individual points by the value of multivariate Gaussian probability density function evaluated at those points. This weighting implies that outliers will result in an overall penalty if they are mapped to low low-probability regions. We show that if the global minimum of newly proposed loss function is achieved, then our OP-DMA maps inliers to regions with a Mahalanobis distance less than ö, and outliers regions past this ö, 8 being the inverse Chi Squared CDF evaluated at 1 - a with a the percentage of outliers in the dataset. Our experiments confirm that OP-DMA consistently outperforms the state-of-art methods on rich variety of outlier detection benchmark datasets.
distribution mapping through weighting the
INTRODUCTION
Background and Motivation. Outlier detection, the task of discovering rare or abnormal instances in dataset, is critical for many applications from fraud detection, error identification in measurements, to fault detection in systems (2). Which points are outliers is often unknown in practice, and thus labeling for which points are outliers versus inliers is scarce
completely unavailable. For this reason, in this work we focus on unsupervised methods for outlier detection (I).
In the ideal case in which inliers follow simple unimodal distribution and outliers occur "far away" from the mean compared to the inliers, outlier detection has a simple solution. In this special case, outliers can be found by simply hardcoding some fixed distance as the cutoff between which points are classified as inlier or as outlier, such as is done by Enveppic (7). Unfortunately. in practice data is often not distributed in such a convenient manner (13). Rather, inliers are distributed in complex, unknown fashion potentially with multiple clusters of inliers in the feature space (6).
Other methods with not quite as strict assumptions on the distribution of the data can work on arbitrary distributions, such as One-Clas methods (4) and IsolationForest (3) to name just few. These methods typically use a density- based approach ( 4 23). However, these methods are known to become infeasible in high dimensional data sets, because in sufficiently high dimensional spaces, all data points are known be roughly equidistant and the space around all points will be sparse (5).
State-of-Art Deep Learning Methods for Outlier Detection. Deep learning methods for outlier detection not only overcome the limitations of density and distance based methods, but they can also leverage the inherent featurization capabilities of deep networks. Recently, deep learning methods use the reconstruction error of an autoencoder-bate the reconstruction eтrоr itself as outlier score (88 20 or by leveraging the reconstruction error in a more sophisticated way (IO) 19 9). However, it has been shown that the usage the mere reconstruction error be suboptimal choice for outlier score because outliers tend to converge
model for outlier detection, either by considering

---

Under review as a conference paper at ICLR 2020
of Recorstructon Im..In.
Recreation Error Cluster Outliers
Ratio of Reconstrution Out.inn.
AE Retreation Error Noise Outliers
10-
15
a
b
5-
spochs
epochs
Figure 1:
Ratio of average reconstruction error of outliers/inliers for an autoencoder trained on Gaussian data with cluster outliers (a) uniform noise outliers (b)
to an average reconstruction error that is indistinguishable from that of inliers as the network converges (19). This is greater issue for the case where outliers oceur in clusters as opposed to uniform noise, as shown in Figure [ The cluster outlier case is is likely to arise when the outliers represent a small minority class or when they are caused by a systematic error.
b)
Legend
Outker
Feature Space
Latent Space
Feature Space
Latent Space
Figure 2: a) Standard deep distribution mapping methods that enforce certain prior distribution on the latent space (VAEs, WAEs) in practice map both outliers and inliers to high probability regions. b) The erroneous assumption of many methods is that outliers are mapped to regions far from the high probability region of the prior distribution.
To go beyond this problem of the pure reconstruction error, recent works have leveraged the distribution-mapping capabilities of generative models such as Variational Autoenedoers (VAEs) that encourage the data in the latent space to follow prior distribution (19) 21). These methods assume that due to their anomalous nature outlier points will not be mapped to the encouraged prior distribution while the inliers will be, and consequently outliers will occur in low-prohability regions of the prior distribution 19. 21) [Figure 2 (b)). However, was observed that this widely held assumption is faulty in practice (21), with the distribution mapping continuing to map both inliers and outliers to high probability regions of the prior distribution for a wide variety of applications [Figure (a)].
Our Approach. In this work, we propose method which leverages the distribution mapping capabilities of generative models, but unlike other distribution mapping outlier detection methods we no longer conveniently assume that outliers are not well mapped to the enforced prior distribution. Our core idea instead to design a novel prior actively encourages outliers to be mapped to low probability regions in the latent space. Our method an autoencoder architecture that we henceforth refer to as the Outlier Preserving Distribution Mapping Autoencoder (OP-DMA). The network is trained to minimize both the Maximum Mean Discrepancy (MMD) distance between the distribution of the data in the latent space and a multivariate Gaussian distribution, and what call prior- weighted L2 distance between the input and the reconstruction of the input. This prightwe distance weights each input datapoint point by the corresponding value of the multivariate Gaussian PDF evaluated at that point's latent representation. The prigh-weighted 1.2 distance encourages points with a high reconstruction error, as we assume outliers have, be placed in low-probability regions the latent space. This corresponds to the important observation that outlier will be mapped far from the mean of the data. The beauty of this mapping is that it thus allows us to use a simple off off-the-shelf distance-baa -ased metric to identify outliers post- cost-ransformation. As the original distribution of the data in the feature space is arbitrary and no distance or density assumptions are placed on the data, OP-DMA thus corresponds to a highly versatile outlier detection strategy. Our key contributions are summarized as follows:
OP-DMA:
loss funetion that
1. Introduce DMA,,, a novel distribution distrbution-mmaping autoencoder that preserves ourliers the latent space without making asumptions on the distribution of the data in the feature space.
2

---

Under review as a conference paper at ICLR 2020
2. Prove that the optimal solution for OP-DMA places outliers further than
n/2 L dt according to the Mahalanobis distance, where is the proba- 2+) bility that draw from a given dataset is an outlier. This allows us to use distance-basee outlier detection classifier
V .o Jo-a 1--22-1/210
3. Show experimentally that OP-DMA routinely outperforms standard distribution-mapping autoencoders for outlier detection on a variety of real-worke benchmark datasets
RELATED WORK
Outlier Detection Using Autoencoders Coupled with Classical Outlier Detectors. Erfani et al. (IT) used autoencoders for dimension reduction before classifying the projected data with a One Class SVM. Raghavendra et al. (12) introduce a One Class Neural Network (OC-NN), feed-forward neural network inspired by One Class SVMs for the outlier-detectio
The OC-NN method first pretrains an autoencoder before introducing OC-NN layer after the encoder. Unlike our method (OP-DMA) these methods do not encourage outliers in the feature space to remain outliers in the latent space, making the strong and in practice often incorrect assumption that outliers in the feature space remain outliers in the latent space.
Recreation Error Methods. Zhou and Paffenroth (10) introduced Robust Deep Autoencoders (RDA), deep network version of Robust Principal Component Analysis. It separates given data matrix D into low-rank matrix L and a sparse matrix S, where anomalous datapoints are isolated in S. This forces the network reconstruct the instances in L, and punishes the network for putting instances into S. Chen et al. (9) introduced RandNet, an ensemble of autoencoders for outlier detection with outlier score corresponding to the average recreation error among the networks. Sabokrou et. al. introduced Adversarially Learned One-Class Classifiers (ALOCC) for Novelty Detection (27). ALOCC consists of generator network R which is trained reconstruct noise-contaminated images by adversarially training R with D, a discriminator network which distinguishes between real images and reconstructed images. They show that the reconstruction eтror for R as well as the output of discriminator 'D are higher for outliers as opposed to inliers. Xia et. al. (28) used an autoencoder for outlier detection by adaptively grouping unlabeled input data into either an outlier or normal class, such that the network is trained to minimimize variance of reconstruction error within class while increasing variance overall. However, as shown in (1),, for autoencoders the reconstruction error of outliers often converges to that of inliers. This negatively impacts the performance of such reconstruction error methods.
Generative Outlier Detection Methods. Perera et. al. (21) proposed adversarial autoencoders with two discriminator networks for outlier detection. The first discriminator encourages the representation of the data in the latent space to be uniform, while the second encourages the output of the decoder conditioned on random samples from the uniform distribution to match real data samples. Vu et. al. (20) used an adversarial autoencoder map the latent representation of their data to a standard normal distribution. They then combine the outlier score from a One Class SVM on the latent data with the One Class score on the reconstructed data the final outlier score. Liu et. al. proposed MO-GAAL (26). deep generative model which uses multiple generator networks to create outlier datapoints near subsets of the input data. A dsieriminator network is used identify boundaries around the subsets of real data such that the generated outliers are rejected. These distribution mapping methods rely on the assumption that unlike the inliers, the outliers will be mapped to low-orobability regions of the prior distribution. However, as we discussed in Section 1, in practice both inliers and outliers are often well mapped to the prior distribution. This is because outliers that mapped to low- oww-robabilitty regions will generally incur higher cost from the divergence term which matches the latent distribution to the prior, and unlike in our OP-AMA approach there is not a corresponding term to counteract this cost.
PROPOSED METHOD: OUTLIER PRESERVING DISTRIBUTION MAPPING
The goal of OP- is to find latent representation of dataset X such that outlier in X can be easily isolated from inliers in the latent representation of X. That is, we find a transformation of that we apply given dataset X with distribution Px such that f(X) Pz, where Px is arbitrary
3

---

Under review as a conference paper at ICLR 2020
and unknown distribution of our imput dataset X and P2 is the target distribution. Further, if a X is an outlier, then f(a) should be an outlier in fX) ) as well. That is, (f()))) should be small. For chose Pz to be Gaussian distribution as low probability regions would then simply be values far from the mean. We chose an autoencoder architecture to accomplish this distribution mapping, where is selected to be the encoder Q parameterized by weights 0 that are learned through stochastic backpropagation.
the purpose of outlier detection,
OP-DMA accomplishes the goal of mapping outliers low probability regions on Pz by weighing the autoencoder's ss reconstruction error by the likelihood of the latent data in the latent space with distribution Pz. In order to accomplish this, we need one-to-one correspondence between each original datapoint, the latent representation of that datapoint, and the reconstructed point. Additionally, while we want the distribution of the latent data to follow Pz on the whole, we do not want all points to be mapped to -robbabbabyy regions of the prior. Rather, we must develop a solution that encourages some datapoints (i.e. outliers) to have a low likelihood in the prior distribution. For these reasons, a Wasserstein Autoencoder (WAE) is a more appropriate choice than the more common Variational Autoencoder (VAE). We chose WAEs, instead of the more commonly used VAEs, because the WAEs encourages the latent representations function of a VAE encourages each individual latent point to represent a distribution that matches the prior (5). This is counter the goal of OP-DMA, which is to have some points (i.e., the outliers) mapped away from the prior distribution. Additionally, by formulating the reconstruction process as a Wassertein distance we can leverage the distribution mapping capabilities of the generative model while still operating with both a deterministic encoder and decoder. This is important as this allows us maintain one-to-one correspondence between given point in the original space and its corresponding mapped point in the latent space so we can weight the reconstruction error by the value of the prior PDF evaluated in the latent space.
whole to match the prior, whereas the loss
3.1 WASSERSTEIN AUTOENCODERS
As OP-DMA is an extension of WAE, we first briefly describe the structure WAEs. WAEs are distribution-mapple autoencoders which minimize the Wasserstein distance between original data and its reconstruction. For original data X and reconstruction Y, is defined as:
f vdddy yer
inf /
(T(Px,P))=
where I' denotes the set of all joint distributions of X and Y, such that the marginal distributions T(X and X) are distributed according to Px and Py respectively, and c is a Wasserstein- divergence cost function. That is, the c-Wasserttein distance is the expectation of the cost function c taken with respect to the joint distribution with marginals equal to the two priors, such that y is joint distribution that produces the minimum expectation of Equation
Simply finding the Wasserstein distance, which is the distance that a WAE is trained to minimize, is itself an optimization problem. However, it is shown in (17 that when the encoded data QX) Pz. where Q is the encoder network and Pz is the prior, then the search over all joint distributions previously required to compute the Wasserstein distance can be replaced with a search over all random encoders, in [xx,y)>>(((X, G(Z)) = infq:Pa- P2 Epx (Q(2)x)[((X) G(2)) This means we can minimize the e- Wasserstein distance by finding the weights of the encoder that produce the minimum expectation of cost c. As is done in (16) where WAE is defined, constraint that Po ~ Pz by adding a penalty term D(Pq. Pz). where D is some function that measures the divergence between PQ and Pz. Thus,
relax the
= Epp = +
(2)
where À is a constant weight term that determines how much the divergence of Pq from the prior Pz is penalized. In the case of the optimal solution for Q, (X, = WX(X, Y). In that case the Wasserstein objective will equal the true c- Wasserstein distance. While the WAE objective function encourages the distribution of the latent data as a whole to match the prior distribution. WAEs do not encourage outliers in the feature space remain outliers in the latent space. Even though the
4

---

Under review as a conference paper at ICLR 2020
divergence term does not explicitly encourage every point to occur in high probability region. in practice this is what happens. Consider when D is a discriminator network. The discriminator is likely to learn boundary around the high probably region of the prior distribution, and the encoder network will be penalized for mapping an outlier to a low probablity region outside of the boundary as the discriminator would correctly identify it as a generated point. On the other hand, as described below, our proposed OutlierPPrrerrvnn Distribution Mapping Autoencoders actively encourages outliers to be mapped low probability region in the latent space.
3.2 PRESERING DISTRIBUTION MAPPING AUTOENCODERS (OP-DMA)
So far, we have formulated the distribution mapping process in terms of random encoders and decoders Q and G. However, we can deterministically produce Q(X) and G(Q(X) by requiring Q(x) = 5y(x), where f is some function mapping X to (X).. This allows us to have one-to-one correspondence between input points X and output G(Q(X)) points, so that Ep, (ox)xxx) G(Q(X))) can define a proper reconstruction error between each point E X and its corresponding output y E Y. This reconstruction error can be used to deter- mine which points should be mapped to low probability regions in the latent space. Let us define e' == c(z ((Q()))) pz pz()(()). We can thus define the OP-DMA objective function, W as:
W = inf Epy Exxzxrzxx Q:Pa=Px
XD(PQ, Pz)
We can think of pz (x)) as a weight on the reconstruction term. Thus, high-rrcoosttruction error points that are mapped to high-probability regions will be penalized more than nigh-reconstruction error points which are mapped to low probability regions. Since outliers are assumed to result in a high reconstruction error (at least during early epochs of training), by lessening the penalty to the network for poorly reconstructed points that have been mapped to low- low-probability regions of the prior we encourage the network map outlier to these low-probability regions. However, the encoder isn't encouraged to map every point (that is, both inliers and outliers) to low probability regions, as the D term makes so that the distribution of all the encoded points on the whole should well match the prior.
Although we have made significant modification to the Wasserstein loss function, as Theorem states minimizing this cost function also corresponds to minimizing lower bound on the non- prior-weighted Wasserstein divergence:. The proofs of all theorems below are provided in the appendix for space reasons.
[
Theorem 1.
Let Pz = N(, Σ). Then (xX & Wx(X, G(Q(X))) (217/1)87172 Ad- ditionally, if [ S ((n=) then WX(X, G(Q(X)) (XX (2m"/1)|21172
Additionally, not only is the cost function of OP-DMA a lower bound on the e- --asssesttein divergence, but also itself is Wasserstein divergence (the e'Wasserstein divergence):
Theorem 2. Let We be a Wasserstein divergence. Then Wa is a Wasserstein divergence, with e' the prior-weighted
As stated previously, our ideal mapping would place all inliers within regions where the probability was greater than some value, and all outliers into some regions where the probability is less than that value. Theorem shows that this scenario is the optimal solution for the loss function of OP-DMA.
Theorem 3. Let Q be an encoder such that D(Pq. F2. P2. = O. where D(A, B, F,F) is the Maximum Mean Discrepancy between A an B, F is the set of unbounded continuous functions and Pz N E). Let X Ω - T be a centered random variable, X Px. Let X(A), A C Ω. be outliers and let H Ω A be the inliers, where Jx(a) px(x))d = Furthermore, let daa G(Q(a)) > c(Ch, ((h,) a (A), h E (H). Then, the optimal solution of OP-DMA is
to map such that [IQ(X 2 and IIQ(((H)) < 8, where 11-a n/2 1 dt and 2Tr(3)
|| Il is the Mahalanobis distance.
Thus, the optimal solution for OP-DMA's cost function is one that maps outliers to regions with larger Mahalanobis distance than that of inliers. This has the important implication that after
5

---

Under review as a conference paper at ICLR 2020
Loss
OP-DMA Training Process
Latent Dist.
Prior Dist.
Prtor- Weghitd Racinsmution Emor
Divergence
Approximate Latent Dist
b)
Evaluate prior PDF each point
a)
Recinsmmction
Enar
Original
Encoder
Latent Data
Decoder
Reconstructed
Original
Data
Data
Data
Outlier
Outlier
Detection
d)
Distance Based
Isolated
Inlier
Detection
Outliers
Procedure
Input/Culp.u
An overview OP-DMA, a) Input data is encoded into latent space. b) The divergence between the latent distribution Po and the prior distribution Pz is calculated and added the loss term. c) The latent data is reconstructed, and the reconstruction error of each datapoint is weighted by the likelihood of observing the latent point according the prior distribution Pz. After training the network, outlier detection is performed by calculating the Mahalanobis distance of each point in the latent space
Figure 3:
transformation with OP-DMA, outliers can be separated from inliers with simple distance metric. This motivates out outlier detection scheme outlined in Section 3.3, where due to Theorem 1 the EllipticEnvelope method to identify outliers.
3.3 UNSUPERVISED OUTLIER DETECTION WITH OP-DMA
Lastly, we describe our end-to-end solution of outlier detection using OP-DMA. First, we transform the distribution of the dataset X to match Gaussian distribution using OP-DMA. That is, the input dataset is transformed to match a prior distribution the latent space of our autoencoder by divergence term in the network's loss function that penalizes the divergence between the latent distribution and the prior distribution. For the purpose of outlier detection, we chose this prior to be a multivariate Gaussian distribution with mean and identity covariance. Then, Elvilpti (7) used to determine outliers by fitting a Gaussian to the transformed data. Theorem tells us that EllippiiEnvelope an appropriate choice for the outlier detection step, as it determines outliers based on Mahalanobis distance. However, other outlier detection methods such as IsolationForest (3) or OneClassS
could equally be leveraged. We use the empirical rather than the theoretical mean and covariance for the EllipticEnvelope as it is unrealistic to expect any such deep distribution mapping model to exactly match the prior. The outlier detection process is outlined Algorithm and in Figure 2. As seen in Algorithm [ we use the unbiased estimator of the MMD from Gretton et. al. (22). For the kernel and Mean Squared Error (MSE) for A high-level overview of the training and outlier detection process of OP-DMA is given in Figure
the MMD we use the inverse multiquadratics kernel is used in (16),
4 EXPERIMENTAL EVALUATION
4.1 EXPERIMENTAL METHODOLOGY
Synthetic Data Sets. We determine possible failure cases for our method OP-DMA versus standard autoencoder (AE). We use two synthetic datasets: (DI) Multimodal distribution R4 with one cluster mean at (0,0,0.0) and another (5,5.5.5) both with standard covariance and uniform noise added as outliers, and (D2) same multimodal distribution for inliers with a small cluster outliers with uniform probability in small region between (-5,-5,55--5) and (-3,-3, -3). In both cases, inliers account for 2.4% the data.
Benchmark Data Sets. We compared OP-DMA to state-of-the-arr methods from the ODDs database 8), a commonly used benchmark database outlier detection datasets. Since we perform unsupervised outlier detection, do not split each dataset into training and testing set. Instead,

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Unsupervised Outlier Detection with OP-DMA
Require: : Regularization coefficient
Contamination parameter o: Initialized encoder network Qx and decoder network Gg with random weights + and +
Dataset X
while e, & not converged do
Sample { 21, ... J from X
Sample { [21 21, 2n from (0,
Sample {21. 81, .... 2n) from Q (ZlX)
Update weights and θ by descending
daa Go(E)))
1 n(n 10-1]
Σ htj
k(zh: zg)+
1
Σ
k(āns (5)
2 -5)ー n²
Σ
k(zu: Zz)
n(n -
hgj
end
Find Dmin = (Q+();), Qo(z)) Qo(())) [|miaal] a)]|D|| with Minimum
Covariance Determinant estimator, inf (et{E).
Find estimated mean j from Dmin
return (I,))l (Qx(z,) p)E((Q+(x)) (t) for 24 E D as outlier scores
outlier detection is performed in an unsupervised manner on the entire dataset. In each dataset all points are labeled as either inlier or outlier. We use these labels to calculate the weighted Fl score, no method we test trained on the labels. The datasets were chosen such that we could test our methods on datasets with wide range of dimensionality in the feature space and with wide range of outlier contamination percentage. Table (in Appendix B) breaks doen the number of featurs, datapoints, and percentage of outliers in each dataset.
Alternative Methods. In order to determine the validity of our OP-DMA, we compare to several state-of-the oftth--rrt distribution mapping outlier detection methods. These include methods that perform outlier detection on the latent space of a Wasserstein, a Variational Autoencoder, and an Adversarial Autoencoder (25) all with Gaussian prior, here denoted as WAE, VAE, and AAE respectively. Additionally, we test against MO-GAAL (26) and ALOCC (27), two state-of-the-ar deep generative outlier detection models. We also test against LOF (2 and OC-SVM (4), two state-of-the-art non-deep learning outlier detection method.
Parameter Configurations of Methods. The encoders and decoders of all methods consist of 3- layer neural networks, where the decoder in each encoder-decoder pair mirrors the structure of the encoder. The number of nodes in the hidden layer of each network hyperparameter chosen from (5, 6, 9, 15, 18, 100). The number of nodes in the output/latent layer is chosen from (2, 6, 9, 15). The value of the regularization parameter À is chosen such that the reconstruction error is the same order of magnitude as the MMD error for the first epoch. We use the standard parameters of MO-GAAL from the authors' publically released code. Likewise, we use the standard configuration of ALOC from the authors code, although we add an additional dense layer at the beginning of each subnetwork. We do this as ALOCC is a convolutional method and assumes input be images of certain shape. The additional dense layer we add to the model transforms the inut data from its original dimensionality into the required shape. We use the standard parameters from Scikit-Learn for LOF and OC-SVM.
4.2 EXPERIMENTAL. RESULTS
Synthetic Experiments. Figure 4 A shows the ratio of the recreation error for outliers/inliers for OP-DMA versus an autoencoder (AE) for each synthetic dataset. For uniform noise (D1), both OP-DMA and AE feature a high ratio for the error of outlier/inliers. In this case, standard AE would be adequate to find the outliers. However, for data where outliers are a small cluster (D2), the ratio between outliers to inliers goes quickly
for AE whereas it levels off to -10 for OP-DMA. In this case, our OP-DMA succeeds to identify the outliers, while AE cannot. We also verify that OP-DMA maps outliers to low-probability regions of the latent space. Figure 4 B shows the latent space of
7

---

Under review as a conference paper at ICLR 2020
OP-DMA for the uniform outlier dataset. The outliers oceur far from the mean in low- bow-probayility regions. The average probability the prior for outliers after transformation was 0.02, whereas the average probability of the inliers according to the prior was 0.08
B
CP DUI Secration
Secoatin
Data in the Latent Dimension
OP-DMA and can identify outlers D1
D2. Dataset With Cluster Outtiers
OP DMG Recestee Enar
Secosdion
OP DM- identifles outlers, AE idertiyy culliers for D2
Figure 4: A: Reconstruction error vs. epoch for OP-DMA and a standard autoencoder for both noise outliers and cluster outliers. B: Latent representation of data in OP-DMA for uniform outliers.
Outlier Detection Accuracy. The results reported Table 2 are the maximum weighted F1-scores of each method over all contamination rates from 0.1 to 1.0 with step size of 0.1.
OP-DMA
WAE
VAE
AAE
Running Time (seconds)
827.14
723.51
857,46
971.51
Table 1: Running time for methods tested
Running Time. order to determine the feasibility using OP-DMA instead one the leading methods in practice, we ran experiment to determine the running times of each method. This consisted performing runs of OP-DMA, WAE, VAE, and AAE on the Satellite dataset (24). The average running time of each method over the 5 iterations is shown in the following table. Each method was run on a machine with 32GB of DDR4 RAM and a Intel Xeon Platinum 8160 CPU.
DISCUSSION
OP-DMA outperforms the state-of-the-art methods the majority of the outlier detection datasets. This is because unlike the state-of-the-art methods, OP-DMA actively encourages outliers to be mapped to low probability regions instead of relying on outliers While in this work, we focused on unsupervised outlier detection, OP-DMA could be extended to supervised outlier detection by making the following change the objective function:
poorly mapped to the prior.
L = inf Exx.. E Eqzzzx) G(Q((,))))
0:Pa Pz
+ Epxon (c(X,,) G(Q((our))) pz (V(((N))))) XD(Po, Pz),
where Xún corresponds to the of labeled inlier points and Xout the set of outliers. OP-DMA also be modified for semisupervised OP-DMA by pretraining on labeled inliers without weighting the reconstruction error by the likelihood, and then performing OP-DMA done in the unsupervised version on the remaining unlabeled datapoints.
CONCLUSION
We have introduced OP-DMA- autoencoder-basse distribution mapping method for outlier detec- tion that maps outliers in the feature space to low probability regions the latent space in which mulrivariate standard normal Gaussian prior distribution is enforced. Outliers are consequently
8

---

Under review as a conference paper at ICLR 2020
Dataset Satellite
OP-DMA
WAE
VAE
AAE
MO-GAAL
ALOCC
LOF
OC-SVM
0.735
0.554
0,480
0,481
0.706
0.413
0.417
10.012
10.009
10.007
10.008
10.001
10.008
Pima
0.625
0.520
0.23
0.497
0.518
0.517
0.456
0.440
10.018
+0.020
+0.019
10.00
10.002
10.010
WBC
0.590
0.448
0,268
0.468
0.529
0.480
0.199
10.011
+0.013
+0.011
000018
+0.060
+0.007
Arrythmia
0.531
0.601
0,201
0.294
0.518
0,457
0.464
0.254
+0.017
=0.015
+0.010
00.010
+0.011
#0.020
Breastw
0.951
0.950
0,368
0.479
0.944
0.863
0.292
0.824
10.014
10.011
+0.009
00000
+0.014
#0.062
Letter
0.182
0.091
0.048
0.10
0.159
0.165
0.488
0.208
+00001
+0.003
+-0.005
+00000
+0.003
++0.001
Cardio
0.590
0.290
0.221
0.204
0.542
0.432
0.208
0.323
#0.013
#0.012
+0.008
0.000
+0.032
#0.071
Lympho
0.585
0.443
0.341
0.310
0.719
0.958
0.833
0.150
10.012
10.008
10.011
00.018
10.119
10.083
Musk
0.32
0.330
0.243
0.228
0.394
0.199
0.069
0.101
+0.007
+0.010
+0.009
+00025
+0.090
+0.160
Thyroid
0.29
0.173
0.130
0.170
0.225
0.084
0.200
0.090
+0.019
+0.021
+0.019
+0.023
+0.027
+0.026
Satimage-2
0.860
0.176
0.148
0.535
0.814
0.818
0.122
0.019
+0.039
10.013
+0.007
10.015
+0.049
+0.026
Table 2: Weighted Fl scores for OP-DMA state-of-the-art methods datasets from the ODDs outlier detection database with 95% confidence interval.
easily identifiable in the latent space. Our experimental study comparing OP-DMA to sfae-oo-thee-nd methods on a collection different benchmark outlier detection datasets shows that it outperforms WAE, VAE, and AAE on the majority of the datasets. We have also demonstrated that there is not significant increase in running time between our method and stat--oftthe-ae methods.
REFERENCES
[1] Chandola, Varun, Arindam Banerjee, and Vipin Kumar. "Anomaly detection: A survey." ACM computing surveys (CSUR) (2009)
[2] Singh. Karanjit, and Shuchita Upadhyaya. "Outlier detection: applications and techniques." International Journal of Computer Science Issues (IJCSI) (2012)
[3] T. Liu, K. M. Ting, and Z. H. Zhou. "Isolation forest", In Proceedings of the 8th IBEE International Conference on Data Mining 2008.
[4] Scholkopf, J.C. Platt, J.Shawe- Taylor, A.J. Smola, and R.C. Williamson. "Estimating the Support of a High- Dimeensoaal Distribution", Technical report, Microsoft Research, 1999.
[5] C. C. Aggarwal. High-Dimensional Outlier Detection: The Subspace Method, Springer New York, New York, NY, 2013
[6] Sugiyama, Masashi. "Dimensionality Reduction of Multimodal Labeled Data by Local Fisher Discriminant Analysis." " Journal of machine learning research 8.May (2007)
[7] Rousseeuw, P.J., Van Driessen, K. "A Fast Algorithm for the Minimum Covariance Determinant Estimator" Technometries (1999)
[8] Shebuti Rayana (2016). ODDS Library http://loddd...ssstoyybooo.kkk. Stony Brook, NY: Stony Brook University, Department of Computer Science.
[9] Chen, Jinghui, al. "Outlier Detection with Autoencoder Ensembles." Proceedings of the 2017 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2017.

---

Under review as a conference paper at ICLR 2020
[10] Zhou, Chong, and Randy C. Paffenroth. "Anomaly Detection With Robust Deep Autoencoders." Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.
[11]
Erfani, Sarah M., al. "High-Dimennionall And Large-Scale anomaly detection using linear one-class SVM with deep Learning." Pattern Recognition 58 (2016)
[12] Chalapathy, Raghavendra, Aditya Krishna Menon, and Sanjay Chawla. "Anomaly Detection using One-Class Neural Networks." " arXiv preprint
802.06360 (2018).
[13] Limpert, Eckhard, and Werner A. Stahel. "Problems with Using the Normal Distribution-an Ways to Improve Quality and Efficiency of Data Analysis." PLoS One 6.7 (2011
[14] Goodfellow, Ian, et al. "Generative Adversarial Nets." Advances in neural information process- ing systems. 2014.
[15] Kingma, Diederik P., and Max Welling. "Auto-Encoding Variational Bayes." " arXiv preprint arXiv:1312.614 (2013).
[16] Tolstikhin, Ilya, et al. "Wasserstein Auto-Encoders. " (2018).
[17] O. Bousquet, S. Gelly, Tolstikhin, C. J. Simon-Gabriel,, and B. Scholkopf. From optimal transport generative modeling: the VEGAN cookbook, (2017).
[18] Sakurada, Mayu, and Takehisa Yairi. "Anomaly Detection Using autoencoders with nonlinear dimensionality reduction." Proceedings of the MLSDA 2014 2nd Workshop on Machine Learning for Sensory Data Analysis. ACM, (2014).
[19] Beggel, Laura, Michael Pfeiffer, and Bernd Bischl. "Robust Anomaly Detection Images using Adversarial Autoencoders. arXiv preprint arXiv:1901.0060 (2019).
[20] Vu, Ha Son, al. " "Anomaly Detection with Adversarial Dual Autoencoders." arXiv preprint arXiv:1902.06924 (2019).
[21] Perera, Pramuditha, Ramesh Nallapati, and Bing Xiang. "OCGAN; One-class Novelty Detection Using GANs with Constrained Latent Representations. arXiv preprint arXiv:1903.08550 (2019).
[22] Gretton, Arthur, et al. "A Kernel Two-Sample Test." Journal of Machine Learning Research (2012)
[23] Breunig, Markus M., al. "LOF: Identifying Density-Based Local Outliers." " ACM sigmod record. Vol. 29. No. 2. ACM, (2000).
[24] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository http://archive.icc.uu....duu Irvine, CA: University of California, School of Information and Computer Science.
[25] Makhzani, Alireza, et al. "Adversarial autoencoders." " arXiv preprint 1rXiv:1511.0566 (2015).
[26] Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang Xiangnan He. "Generative Adversarial Active Learning for Unsupervised Outlier Detection." IEEE Transactions on Knowledge and Data Engineering (TKDE 2019)
[27] Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli, "Adversarially Learned One-Class Classifier for Novelty Detection". CVPR 2018.
[28] Yan Xia, Xudong Cao, Fang Wen, Gang Hua, Jian Sun. "Leaming Discriminative Reconstruc- tions for Unsupervised Outlier Removal". ICCV 2015
10

---

Under review as a conference paper at ICLR 2020
Appendices
A PROOFS
Theorem 1. Let Pz (u, 2). Then (x((X)((())) < W.(XX (2m"/2]/21172 Ad-
ditionally, if | S [4x=] then W (XX (2mm79)18117--
Proof. We will first show that W (x,(XX(Q(XX)) = (W,(X, (27//1)pp and then show that (X, G(Q(X))) < G(Q(X))) (3/"/"]|_)1
The maximum value of (z) will occur at μ. Then, sup pz (2 Thus
G(Q(X))
inf Q:Pa=Iz Pz
Epx Eps_E
pz(((()))
inf Epp Bex_egraxl((x.((((x)) Q:Pa=P2 (2Tm/2) Wc(X,
(
(2Tm\22
Similarly, to show that the rrightee Wasserstein divergence is a constant multiple of the true Wasserstein divergence:
W2(X,G(Q(X)) = inf Q:Pa=Px
Epx
inf Q:Po=P2
Epx
Pz))
(2T1/2))111/3
W G((XX)(((X))
(2mm/2) (2/1/)
Theorem 2. Let We be a Wasserstein divergence. Then We a Wasserstein divergence, with c' the prigr-weighted
Proof. Since is a Wasserstein divergence, we know that c(x1, > DV E supp( (P),
c(z, x) 0 V a E supp(P) (P), and [_,[c(2,, ax2) 2 4. V T[P, Since P2(2) 2 0Y z, d' will also fulfill the three aforementioned properties of c. Thus, W is Wasserstein divergence.
O
Theorem 3. Let Q be an encoder such that D(Pq, F2,F)= = where D(A, B,F) is the Maximum Mean Discrepancy between A an B, F is the of unbounded continuous functions and Pz NO E). Let X 52 + IR" be a centered random variable, X ~ Px. Let X(A), A C Ωt be outliers and let H (a, G(Q(a)) > ch, G(Q(()) V a ((A), h E X(H). Then, the optimal solution of OP-DMA is V 11-a 1-n/2-1eft di and 2fr())
be the inliers, where (X(A) px (x)dd a. Furthermore, let
to map such that (I(((())) 2 â and (H)( < 8, where
||. Il the Mahalanobis distance.
Proof. The Mahalanobis distance of Q(X) can itself be expressed random variable, 8 -Q( Let 4 be the CDF of 8. Then, 4b(l a) P(8 < 1 a) P(82 < (1 - (1-a)²) = (12 - a)
Let Y = =Q(X) where M M = Σ is the Choleski decomposition the covariance Σ. Since D(Pq, F), 00 and (AA, A, F) =0 iff A B, we thus know that Q(X) (O, E). Thus since
11

---

Under review as a conference paper at ICLR 2020
(X) is normally distributed and centered. Y thus is normally distributed with identity covariance. Since 8² = Q(X)= '(())== = YY", 2 is the CDF of the sum of squares of n normally distributed variables with mean 0 and 1. Thus, 25² is the Chi Squared distribution. The inverse Chi Squared CDF will therefore give us the distance such that 1 a percent of the points are within 5:
1-n/2-1ezt dt 2
ô =
Now, Iet us assume that for some parameter choice O' for Q that aP(Q(X(A)(0) 8)
8, B > O. Consequently, (1 P(Q( (1f))0)) (X(A) Px (x)dr= = a. Conversely, let us assume that there is a parameter configuration such that aP(Q(X S 8) 0 and so (1 ()Q(((
B, since (X)(X) > 6) and
5) 0.
Since Pz ~ W(..E), pz(di) < pz((2) for Idilmm > l|22lmm. Thus, since we assume c(a, ((Q(a)) c(h, G(Q(h)) Y X(A), e X(H), then Epx Eq(2)x) ((Q(),)(')) Bpx BQ(z)x)c(zp. G(Q(#, (')))p=(T;))
> Epy (o ((Q(z)1)))) (xp) Epx (o(2)(( ( (z) ())). Thus, mapping that places outliers away from the mean will have lower loss than one that places some percentage of outliers less than from the mean.
B DATASETS
Dataset
# Features
# Datapoints
% Outliers
Satellite
6435
32%
Pima
8
768
35%
WBC
278
5.6%
Arrythmia
274
452
15%
Breastw
638
35%
Letter
32
1600
6.25%
Cardio
21
1831
9.6%
Lympho
18
148
4.1%
Musk
166
3062
3.2%
Thyroid
3772
2.5%
Satimage-2
36
5803
1.2%
Table 3: Description of the dimensionality, number of datapoints, and contamination percentage of each dataset we tested our method on.
C
SENSITIVITY TO CONTAMINATION PARAMETER
While the contamination parameter a is not used during the training of OP-DM, is used to fit the Elliptic Envelope the encoded data after training. We test the sensitivity of this method to the value the contamination parameter by evaluating the Fl score of outlier detection on the Satellite dataset after encoding the data with OP-DMA. The results are shown in the figure below.
12

---

Under review as a conference paper at ICLR 2020
Sensitivty of OP-DMA to Contamination Parameter
0.7
0.6
F1 Score
0.5
0.4
0.3
0.1
0.2
0.3
0.4
0.5
Contamination Parameter
The Fl score after outlier detection using OP-DMA Elliptic Envelope the Satellite dataset, for various values of the contamination parameter a. The vertical dashed line shows the true percentage of outliers in the dataset.
Figure 5:
This analysis shows that long the contamination parameter is significantly less than the true contamination percentage the Fl-score is robust to different values of the contamination parameter.
13


Reviews:
[{"review_id": "ryehE_265r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an improved extension of the Wasserstein auto-encoder for anomaly detection. The novelty is in proposing a weighted reconstruction error that penalizes the mapping of data with high reconstruction errors (mostly anomalies) into high probability regions. The idea being that an outlier would have a higher reconstruction error, and hence should be mapped to low-probability region of the latent distribution. \n\nExperimental Results:  As a distribution mapping auto-encoder model, OP-DMA outperforms the deep learning based state-of-the-art models in the same domain. \n\nOverall Assessment: The authors have a nice idea of forcing the latent mappings of inputs to correlate with their reconstruction error. Overall, the method is promising, but I have the following concerns:\n\n* Using the reconstruction error as an anomaly score has been explored many years ago (check replicator neural networks), the novelty here is to enforce that on the latent space in the context of a variational auto-encoder. I am not sure if, from anomaly detection perspective, this is any better than simply using the reconstruction score. Why go the VAE route at all? \n* Is there a possibility that assuming a single multi-variate Gaussian, as a prior, too restrictive? Could it result in a high false alarm rate as well? I guess this could be answered by more experimental results on richer data sets (even synthetic is fine). \n \n* In most score based algorithms, the anomaly score is computed without assuming any prior knowledge about the contamination proportion. However, in the case of OP-DMA, the contamination parameter is used to train the auto-encoder that scores the data. This might result in an optimization that is very specific to the parameter setting. I strongly recommend a sensitivity analysis to study the robustness of the model against different values of contamination parameter.\n\n* Performance on synthetic data-set has not been presented. The set H in theorem 3 has not been defined. \n\n* Additional comparison with other non-distribution mapping state-of-the-art models such as LOF, oc-SVM, KNN would give a clearer idea of the performance. This is important, because in my past experience, non-deep learning methods give much better results on the benchmark data sets that the authors have evaluated their method on. In fact, a comparative analysis (See - https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173) gives a very nice comparison. However, since the authors provide results using Avg F1-score, instead of AUC curve, it was not possible to compare them myself.\n\n* In figure 3, in the training process, the authors have describe to add the divergence between the latent and prior distribution to the loss function, however, nothing like this is clearly shown in the figure.The references of figures in the text are either out of place or incorrect. Figure 1(a) and (b) in reference to the text are incorrect. Figure 2 is the misleading figure as it doesn't illustrate the anomaly detection process. Figure 3 has not been mentioned anywhere in the text. The authors have mentioned the comparison of their method with Wasserstien and variational auto-encoders in the text, while in table 2 and 4, AAE has also been shown as one of the method for comparison, which is never mentioned or described in the text.\n\n* Typo in caption of Figure 1 and the first line of section 3.3\n\nOverall, I am hesitant to recommend the paper before cross-checking the issue with contamination proportion and learning more about how a VAE framework is indeed important for anomaly detection.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyxflCHBtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel outlier detection approach, based on Wasserstein auto encoders. \n\nUnfortunately, I cannot comment on the overall scientific contribution of the paper, as I simply do not possess the expertise to judge it accurately. I will rely on the judgement of the other reviewers, whom I hope will have more experience and will better know the literature. I will report on a few issues with aspects related to the presentation below.\n\nIn fig. 1, the (a) and (b) should probaby appear below each diagram. \"on trained\" is repeated twice in the caption. \n\nIn fig. 2, the WAE acronym is defined only much later in the text. \n\nFig. 2 is also a bit confusing, since it seems like the order in which the diagrams appear should be swapped. Indeed, the text also refers to fig. 2 (b) before (a). The text just below fig. 2 also refers to Figure 1, but I think it should be 2? \n\nIn sec. 4.2, the text mentions table 2 when it should really be table 1. Also, table 1 should appear before table 2 in the body. \n\nIt looks as if the symbols () and [] are inverted? All references are referred to with () and text within parentheses (e.g. references to figures) have []. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HygKPD2hqr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an outlier detection method based on WAE framework. WAE is trained to ensure that 1) latent distribution follows a prior distribution 2) weighted reconstruction error is low where prior PDF is used to weight the reconstruction error.\n\nPositives\n------------\n1.I liked the intuition behind the proposed method and I felt its worth exploring. Paper points out that in previous works, there is no mechanism to prevent outliers from getting mapped to high probability areas in the model [21]. Authors claim that their method will over-come this issue. I believe this is the core contribution of the paper.\n\n2.I agree with authors point that WAE is a better choice than VAE for outlier detection because, former \"encourages the latent representations as a whole to match the prior \". \n\n3.I agree training with a distributional divergence loss along with a weighed reconstruction loss will be helpful for learning a robust representation (as outliers in the training dataset would be assigned a lower weight). \n\n4.Authors have compared the performance of their method on OOD dataset where they have  compared against three other baseline methods, where they obtain better performance in majority of cases.\n\n\nNegatives\n--------------\nA. Outlier detection and anomaly/novelty detection are two very different problems.  Outliers are 'bad eggs' coming from the same class as normal data. On the other hand, anomaly/novelty are unexpected data possibly coming from other classes. This is the taxonomy followed by majority of works. In my understanding this work is about 'outlier detection'. I hope authors will use the term 'outlier detection' consistency through the paper.\n\nB. Authors have not done a good survey on existing outlier detection methods.  Eg:\n    I. Chong You, Rene Vidal, Provable Self-Representation Based Outlier Detection in a Union of Subspaces, CVPR 17\n    II. Yan Xia, Xudong Cao, Fang Wen, Gang Hua, Jian Sun, Learning Discriminative Reconstructions for Unsupervised Outlier Removal, ICCV 15\n   III. Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli, Adversarially Learned One-Class Classifier for Novelty Detection, CVPR 18. (they have experiments on outlier detection)\n\nC. Although existing methods have not explicitly stated the problem identified in (1) above, their proposals are indirectly solving this problem. Therefore, authors should have compared with papers listed in (B) to demonstrate the effectiveness of their method for a meaningful comparison.\n\nD. This is the first time I'm seeing the OOD dataset used in the paper. Have other works published their results on this dataset? Can they be included in your paper?. If not, consider reporting results on standard datasets used in papers (B). I believe reporting results on at least two datasets is necessary to demonstrate the generalizability of the method.\n\nOther comments\n------------------------\na. What is the dimensionality used in the latent space? I believe a larger latent space may be required in modeling more complex data such as images. Is the weighting mechanism effective when a very large latent space is used due to the curse of dimensionality. \n\nb. I don't think the synthetic dataset experiment is giving any interesting insights. This space is better used if an additional dataset is used instead.\n\nIn conclusion, I like the idea presented in this paper; however, I believe experimental results needs to be improved significantly to demonstrate the effectiveness of the proposed method. I cannot recommend to accept the paper in its present condition. \n\n\nPost Rebuttal:\nAuthors have partially addressed my concerns. In light of new experiments provided, I'm changing my decision to weak accept.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an outlier detection method that maps outliers to low probability regions of the latent space. The novelty is in proposing a weighted reconstruction error penalizing the mapping of outliers into high probability regions. The reviewers find the idea promising.
They have also raised several questions. It seems the questions are at least partially addressed in the rebuttal, and as a result one of our expert reviewers (R5) has increased their score from WR to WA. But since we did not have a champion for this paper and its overall score is not high enough, I can only recommend a reject at this stage.
Decision: Reject