Suppressing Outlier Reconstruction in Autoencoders for Out-of-Distribution Detection
Authors: Sangwoong Yoon
Yung-Kyun Noh
Frank C. Park
Venue:  (2021)
Keywords: autoencoder
outlier detection
novelty detection
energy-based model

Abstract: While only trained to reconstruct training data, autoencoders may produce high-quality reconstructions of inputs that are well outside the training data distribution.  This phenomenon, which we refer to as outlier reconstruction, has a detrimental effect on the use of autoencoders for outlier detection, as an autoencoder will misclassify a clear outlier as being in-distribution.  In this paper, we introduce the Energy-Based Autoencoder (EBAE), an autoencoder that is considerably less susceptible to outlier reconstruction. 
The core idea of EBAE is to treat the reconstruction error as an energy function of a normalized density and to strictly enforce the normalization constraint. We show that the reconstruction of non-training inputs can be suppressed, and the reconstruction error made highly discriminative to outliers, by enforcing this constraint.  We empirically show that EBAE significantly outperforms both existing autoencoders and other generative models for several out-of-distribution detection tasks.

Under review as a conference paper at ICLR 2021
SUPPRESSING OUTLIER RECONSTRUCTION IN AUTOENCODERS FOR OUT-OF-DISTRIBUTIOO DETECTION
Anonymous authors Paper under double-bling review
ABSTRACT
While only trained to reconstruct training data, autoencoders may produce high- quality reconstructions of inputs that are well outside the training data distribution. This phenomenon, which we refer to outlier reconstruction, has detrimental effect on the use of autoencoders for outlier detection, as an autoencoder will misclassify clear outlier as being in-distribution. In this paper, we introduce the Energy-Bee -ased Autoencoder (EBAE), an autoencoder that is considerably less susceptible outlier reconstruction. The core idea of EBAE is to treat the recon- struction error as an energy function of a normalized density and to strictly enforce the normalization constraint. We show that the reconstruction of non-training in- puts can be suppressed, and the reconstruction error made highly discriminative outliers, by enforcing this constraint. We empirically show that EBAE signif- icantly outperforms both existing autoencoders and other generative models for several out-of distribution detection tasks.
INTRODUCTION
autoencoder (Rumelhart et al. 1986) is a neural network trained to reconstruct samples from training data distribution. As the quality of reconstruction is expected to degrade for inputs that are significantly different from training data, autoencoders are widely used in outlier detection (Japkow- icz et al. 1995) where an input with large reconstruction error is classified as out-of out-of-distribution (OOD). Such autoencoders for outlier detection have been applied in domains ranging from video surveillance (Zhao et al. 2017) to medical diagnosis (Lu & Xu 2018).
Contrary to widely-held belief, autoencoders are in fact capable of accurately reconstructing outliers, casting doubt on their reliability an outlier detector. Lyudchik (016) showed that autoencoder trained on MNIST with the digit seven excluded can reconstruct an image of the excluded digit, and Tong et al. (2019) reported that autoencoder trained on MNIST can reconstruct image with all zero pixels. The reconstruction of outliers is also observed for non-image data (Zong et al. 2018).
In this paper, we investigate this unexpected behavior of autoencoders more deeply, which refer to as outlier reconstruction. In the course of our investigation, we reproduce the findings of Lyudchik] (2016) and Tong et al. (2019), and additionally discover other interesting cases (Figure [T) Our experiments suggest that outlier reconstruction is not a fortuitous artifact of stochastic training but is, in fact, consequence of inductive biases inherent in an autoencoder.
Outlier reconstruction should be suppressed for an autoencoder-basel outlier detector, since a recon- structed outier undermines the detector's performance by being mistaken to be an inlier. Despite the long history of autoencoder research (Rumelhart et al.) 1986 Bank et al. 2020), the outlier re- construction phenomenon has only recently begun to receive attention (Lyudchik] (016) Tong al,) 2019 Zong et al.] (2018), with few works explicitly proposing solutions to the outlier reconstruc- tion problem (Gong et al. 2019). Previous works focused on regularization techniques that pre- vent autoencoder from being an identity mapping (and thus reconstructing all inputs). However, outlier reconstruction still occurs in popular regularized autoencoders, including denoising autoen- coders (DAE, Vincent et al. (2008)), variational autoencoders (VAE, Kingma & Welling (2014), and Wasserstein autoencoders (WAE, Tolstikhin et al, (2017)). ) as we shall show in our experiments (Table I.

---

Under review as a conference paper at ICLR 2021
MNIST
FNNIST
BIb X
Omniggic
Har 5
Upper
Lower
SVHN
Constentt
7
U
H
H
-10
8
7
U
H
EBAE
7
0
4.00
-3.75
-3.25
-3.00
-2.50
-2.25
Figure 1: (Left) Input images (first row), reconstructions from an autoencoder (AE, second row) and from EBAE (third row). AE and EBAE are trained MNIST. (Right) Distribution of reconstruction errors for inliers (CIFAR10) and outliers (the other curves).
In this paper, we propose the Energy-based Autoencoder (EBAE), an autoencoder in which the reconstruction of outliers is explicitly suppressed during training. In each training step of EBAE, "fake" samples with small reconstruction error are generated. These well-recoostrructed fake samples serve as probes for potential reconstructed outliers. Then, EBAF maximizes the reconstruction errors of the generated samples, while minimizing the reconstruction errors of "real" training samples. When the generated samples become indistinguishable training data, the gradients from the fake samples and real samples balance, and thus the training converges.
The training scheme naturally arises from defining a probability density for EBAE using its recon- struction error. The density of EBAE is given as (x( exp( -E((x))xX where E(x) is the reconstruction error of and Q is a normalization constant. This formulation of defining density using scalar function is often called an energy-bann -eergyybasedd model in the literature (Mnih & Hinton 2005 Hinton et al. 2006). and Fx(x) is called the energy of the density. Maximizing likelihood in this for- mulation results contrastive divergence learning (Hinton| 2002). which minimizes the energy of the training data while maximizing the energy of the samples from the model.
When generating samples with small reconstruction error during training, scheme specifically designed for EBAE. Our sampling scheme is based on Langevin Monte Carlo but leverages the latent space of an autoencoder to generate diverse samples which facilitates the training of EBAE.
a novel sampling
Setting the reconstruction error as the energy, EBAE incorporates two major outlier detection crite- ria, large reconstruction error (Japkowicz et al. 1995) and low likelihood (Bishop) 1994), since the two are equivalent in EBAE. Generally, the two criteria not necessarily overlap in other methods, e.g., VAE energy-based models (Zhai et al. 2016). Recent studies show that likelihood-basee outlier detector using a deep generative model, such as an auto-regressive model or fow-based model, fails to correctly classify certain obvious outliers (Nalisnick et al., 2019: [Hendrycks et al., 2019). However, EBAE is able to detect such outliers successfully while still using likelihood as the decision criterion.
The contributions of our paper can be summarized as follows:
We report and investigate various cases of outlier reconstruction in autoencoders:
We propose EBAE, an autoencoder significantly less prone to outlier reconstruction;
We present sampling method tailored for EBAE which efficiently generates diverse samples;
We empirically show that EBAE highly effective for outlier detection.
autoencoder-basee outlier detection. In Section 3, investigate outlier reconstruction in depth with illustrative examples. Section 4 describes EBAE. Related works are reviewed in Section 5. Section presents experimental results. Section 7 con- cludes the paper.
Section 2 provides brief introduction
2 BACKGROUND
2.1 PROBLEM SETTING
In this paper, we consider the outlier detection problem, which is also referred to as novelty de- tection, open set recognition, OOD detection in literature. The goal is classify outliers from
2

---

Under review as a conference paper at ICLR 2021
in-distribution samples, while no information regarding the outliers to be detected is provided during training. Formally, we are given a set of inliers x E X C R from the underlying data density function p(x) for training. Dx is the dimensionality of x and X' is the support of p(x). An outlier is typically defined as a sample from the p-sublevel set of data density {x (x(p(x) S p} (Seeiwwrrt et al.) 2005]. Note that sample located outside of the support X' belongs to the O-sublevel set and hence is an outlier for all p 2 0.
An outlier detection system typically produces scalar decision function c(x), and is predicted as an outlier if c(x)
for the threshold v/. Setting the threshold controls the trade-off between false positive rate and false negative rate. In our experiments, we shall use area under receiver operating characteristic curve (AUC) as threshold-ing preshold-independenn performance metric when evaluating outlier detection algorithms.
2.2 AUTOENCODER BASED OOD DETECTION
An autoencoder consists of an encoder fe(x) RD + R. and a decoder fa(z) RD - RR,, where D₂ is the dimensionality of the latent vector z. An input x sequentially processed through the encoder and the decoder, producing its reconstruction x fx(fe(x)). The reconstruction error is the discrepancy between x and X. L2 distance x x| is popular choice of discrepancy measure, but other error metries also applicable. A encoder and decoder deep neural networks and are jointly trained to minimize the mean reconstruction error of training data through stochastic gradient descent.
In an autoencoder-basee outlier detection system (Japkowicz et al. 1995). ) the reconstruction error of an autoencoder trained on in-distribution samples used the decision function c(x) = ||x x||², and an input with large reconstruction error is classified as OOD. However, in the next section, we shall show that an autoencoder could unexpectedly produce very small reconstruction error for inputs not drawn from the training distribution. Thus an uutoenccderr-basee outlier detection system may fail to detect such outliers.
3 OUTLIER RECONSTRUCTION
The outlier reconstruction is a phenomenon that an au- toencoder unexpectedly succeeds in reconstructing an in- put even though located outside of the training dis- tribution. In this section, provide illustrative exam- ples that show that outlier reconstruction is a consequence from the inductive biases of autoencoder.
EBAE
Smoothness of mappings When the training data distri- bution consists of multiple clusters, the outliers from the region between the clusters are likely to be reconstructed. Figure 2 depicts 2D synthetic data generated from mix- digureddepicts ture of two disconnected uniform distributions and their reconstruction from autoencoders with one-dimensional latent space. The outliers (red crosses) from the middle of two clusters show reconstruction errors (the length of thin black lines) smaller than some inliers (blue dots). Tong et al. (2019) noted this type of outlier reconstruction and mentioned that outliers "close to the mean" data or "in the convex hull" of data are likely to be reconstructed.
Figure AE and EBAE trained on bi- modal distribution.
This phenomenon arises from the inductive bias of an autoencoder that its encoder and decoder smooth mappings. The extreme of this inductive bias can be found in linear principal component analysis (PCA). PCA, a linear counterpart of an autoencoder (Bourlard Kamp 1988), would reconstruct any outliers which reside on the principal axis. Note that this phenomenon is consistent with the objective function of an autoencoder and PCA, as the objective does not penalize the reconstruction of outliers.
Compusitionality When there compositional structure in data, we can still observe a recon- structed outlier even if it lies outside of the convex hull of training data. The data compositional
3

---

Under review as a conference paper at ICLR 2021
if each datum can be broken down into smaller reusable components; For example, MNIST can be considered highly compositional, since digit image can be decomposed into smaller sub sub-patterns, such straight lines and curves. An outlier can be successfully reconstructed when composed of subset of components existing in the training data.
To demonstrate the effect of compositionality in outlier reconstruction, make two synthetic datasets both of which are clearly out- out-of- of-distributton with respect to MNIST. The first dataset is HalMMNIST (the seventh column in Figure D. consisting of MNIST images with randomly cho- sen upper or lower half replaced by zero pixels. ChimeraMNIST (the last column in Figure the second dataset, is a set of images generated by concatenating upper and lower halves of two ran- domly chosen digits. Although these images are not in the convex hull of MNIST digits, they share components found in MNIST. As shown in Figure T an autoencoder trained on MNIST have no problem reconstructing them. The classification AUCs from the reconstruction error are 0,482 for HalfMNIST and 0.69 for ChimeraMNIST, indicating poor classification.
It seems that an autoencoder learns to reconstruct each part of an image separately but is not able to judge whether the combination of the parts valid as a whole. This compositional way of processing facilitates generalization of model (Keysers et al. (019),, but the generalization of reconstruction in OOD inputs is not desirable for an autoencoder based outlier detector.
Distributed representation We suspect the outlier reconstruction due to compositional pro-
Recon eiter per pixal
0.06
0.8
MNIST9
cessing may be attributed to the distributed rep-
0.04
0.7
resentation (Mikolov et al.) 2013) used an autoencoder. To show the effect of the dis- tributed representation, we train autoencoders on MNIST with the digit 9 excluded (MNIST- not9) and measure the reconstruction error of the digit 9 (MNIST9) under multiple values of latent dimensionality Dz. Figure 3 shows the result. We observe the outlier reconstruction of MNIST9 possibly due to the compositional processing mentioned above. However, the outlier construction occurs only when D2 is large. The latent representation is more distribution for large D2, larger number of hidden neurons are used to represent imput. This observation suggests that the distributed representation used in an autoencoder enables the compositional processing and thus facilitates outlier reconstruction.
0.02
0.6
0.00
64
Latent dimersionality
Figure 3: Reconstruction errors of MNISTnot9 and MNIST9, The error bars denote 80-percentile around the means.
Meanwhile, the model we propose shortly is not vulnerable to outlier reconstruction in all cases we examined above, shown in Figure Figure 2 and Table even though it is based the same autoencoder architecture.
4 ENERGY-BASED AUTOENCODERS
4.1 DEFINITION
We propose Eneegy-based Autoencoder (EBAE), a generative model built from autoencoder described in Section 2.2 The density of EBAR is designed to be large for an input with small reconstruction error. Formally, given an encoder fx( and decoder fa(z), the density model of EBAE pa (x) is defined as follows:
||X x fa(fa(x))11, No f
(x) = Se
exp(-Eo(x)) Ex(x) =
<0. (1)
To ensure the integral exists, assume that the domain of is bounded, and exp( - Eg (x)) is continuous and bounded. Often, the quantity Ex( is referred to as an energy function, and the whole formulation is called energy-based model (Mnih & Hinton 2005 Hinton et al. 2006).
As in a conventional autoencoders (Japkowicz et al. 1995), outliers are detected based on the large reconstruction error in EBAE. This decision is equivalent to determining an outlier based on low likelihood (Bishop 1994), as the likelihood and the reconstruction error are in a linear relationship:
log (x) - log 2n where log Og is constant with respect
4

---

Under review as a conference paper at ICLR 2021
4.2 TRAINING
EBAE is trained to maximize the likelihood of data. The expectation of the likelihood gradient with respect the data density p(x) is written follows (the derivation in the supplementary material):
Ex=p(x) Vo po))] = px) [o)(
Vo log So
(2)
= (x)) V Ex)] + Txx (x [oF(x))
(3)
The expectation Ex- is with respect to training data, and the expectation Ex (x)) is with respect to samples generated from the model. Each gradient step would decrease the average re- construction error of the training data while increase the average reconstruction error of "fake" data x'. In practice, we approximate the expectations with a mini-batch of training data and generated samples in each iteration.
The second term (x)) ( sampling process tend to generate high- litlliikeiioodd samples which will produce small reconstruction error by definition. Therefore, the second term finds outliers which can be reconstructed by a current model and suppress them by applying the gradient. The training converges when px(x) become identical to p(x) as the two gradient terms are cancelled out. A converged EBAE will give small reconstruction errors only for in-distribution samples and large reconstruction errors for OOD inputs.
responsible for inhibiting the reconstruction of outliers. The
The suppression of outlier reconstruction in EBAE is originated from the enforcement of the nor- malization constraint, the second gradient term derived from the gradient of the normalization constant V log Sgg Note that by removing the second term in Eq. the expression reduces into the gradient of conventional autoencoder, only minimizing the reconstruction error inliers. Thus, from our energy- mesey--based formulation (I). conventional autoencoder training does not properly maximizes the likelihood neglects normalization, and the lack of normalization an explana- tion why outlier reconstruction occurs in autoencoders.
4.3 SAMPLING FROM EBAE
During training, samples are drawn from Pn by Markov Chain Monte Carlo (MCMC) method approximate the second expectation in Eq. For energy function based deep neural network, Langevin Monte Carlo (LMC) (Welling & Teh 2011)/Nee et (01] method is typically employed (Du Mordatch| 2019 Grathwohl al. 2020). However, we found that naive LMC mixes very slowly for the energy function of EBAE, possibly because the energy landscape is highly multi-modal.
Here, we propose two-stage sampling scheme which is specifi- cally designed for EBAE. The proposed method use the latent space structure in an autoencoder to traverse between energy basins. The scheme consists of two consecutive LMC chains, the latent chain and the visible chain, which are illustrated in Figure 5) The la- tent chain runs first in the latent space and aims to provide a good starting point for the visible chain, which runs in the visible space afterwards. After Тz steps of the latent chain, its sample 2y, is fed to the decoder, yielding the starting point for the visible chain Xo = fa(zr,) ). The visible chain runs for Tx steps and produces the sample. With the latent chain, the visible chain could start from diverse points with sufficiently high density, resulting in diverse samples with high quality.
Latent
Visible Chain
Figure 4: The illustration of two-stage sampling process.
The two chains can be formally written as follows:
Ag
Latent chain: Z Uniform, Ze+1 = Ze +
v. log (z))) + + N L2..).
(4)
2
Ax
Visible chain: Xo fa(zT.), X.+1 X +
Vx log po + ex, 6x N 1x)
(5)
2
where Oz, 0x. I,, and Ix are zero vectors and identity matrices defined in the latent space or the input space, accordingly. The step sizes A Ax and the variances Tg, 0x tuned separately to achieve faster training in practice as done in Du & Mordatch (2019); Grathhool et al. (2020). Po(z) the density of EBAE defined in (1) and qo (z)z) is auxiliary density function designed for the
5

---

Under review as a conference paper at ICLR 2021
latent chain to yield desirable starting point for the visible chain. We also define go(z) through the energy-based formulation with an energy function Hz)z):
go = a exp(-Ho(z)), Ho(z) = Eo(fa(z)), Vo No=1
< cc.
(6)
where Ve being the normalization constant. A sample z with high go (z(z) from the latent chain will have low (z), and therefore its projection the visible space fa(z) will have low B(fa(z)) by design.
Theoretically, running an infinitely long visible chain should suffice for generating samples from (x).. As the choice of initial point has negligible effect on the convergence of the chain in theory, the
4 R
4 5
8
9
8
2 1 -
3
2
5 6
introduction of the latent chain does not affect the asymptotic be- havior of sampling.
2
2 x ワシサカ
8
(
3
8
4
9
Figure shows the samples generated from conventional autoen- coder and EBAE trained on MNIST through the proposed two-stage sampling method. The samples from the autoencoder are signif- icantly different from MNIST digits. samples with small re- construction errors more likely be drawn, these non-MNIST images have smaller reconstruction errors than MNIST in-distribution images. Meanwhile, the sam- ples from EBAE visually similar to MNIST digits while also being diverse.
Figure 5: Samples from AE (Left) and from EBAE (Right).
4.4 TECHNICAL DETAILS IN TRAINING
Here, we describe a few technicalities for stable training and competitive performance with EBAE.
Pre-training as a conventional autoencoder Before training an EBAE using (q.(3) we initialize the network parameters by training the network as a conventional autoencoder via reconstruction er- ror minimization until convergence. Compared to computationally expensive LMC steps in EBAE training. the conventional autoencoder training is very stable and fast, also resulting in a good rep- resentation of data to start working with.
Spherical latent space (Davidson et al., 2018; Xu & Durrett 2018: Zhao al. 2019) The output of an encoder is projected to the surface of a unit ball through division by its norm. The spherical latent space removes the boundary effect for the latent chain EBAE, improving the stability of LMC. To reflect the spherical constraint during LMC, we employ Constrained LMC Brubaker et al. 2012), where 2, is projected to the sphere after every LMC step (Eq.44).
Regularization We regularize the energy of negative samples to prevent its blow up. During the training, we minimize the average squared energy of negative samples the negative likelihood of data. The regularization term is given as Lyeg = >en E(x))² / for the batch size B, and its gradient a VeLreg is added to the gradient of the negative log likelihood.
mini-batch as well as
5 RELATED WORK
Autoencoder as a probabilistic model While conventional autoencoders do not directly model probability distribution, but some autoencoder variants have probabilistic interpretation. In DAE Vincent et al.] 2008) and contractive autoencoders (Rifai et al. 2011). the reconstruction error related to the gradient of log density (Alain & Bengio 2014). Generative stochastic networks (Bengio et al. 2014) utilize this property to build model that can draw samples. Interestingly, Alain & Bengio (2014) reported that DAE might produce a spuriously small log density gradient estimate, I.c., small reconstruction error, for point in far from the support of data. VAE (Kingmal & Welling 20144) and its variants, including adversarial autoencoders Makhhani et al. (2015). WAE Tolstikhin et al. (2017), and Generative Probabilistic Novelty Detection Pidhorskyi al. (0118)
model a properly normalized probability density with the aid of a prior distribution.
Outlier detection Besides the reconstruction-bast based method mainly discussed in this paper (ap- kowicz et al.) 1995 An & Cho 2015) Zong et al.) 2018), autoencoders can be applied to outlier detection by learning representation of data. In Xu al. (2015). input data are transformed using

---

Under review as a conference paper at ICLR 2021
Table 1: MNIST hold-out class detection results. AUC scores are shown. The values in parentheses denote the standard error of mean after 10 training runs.
MNIST
2
3
5
6
9
ave
EBAE
.82)
.919.013)
.992.001)
.949.000)
(949).0))
.978, 003)
.938. 004)
.885,.12)
.929.00))
9344000)
.946
AE
.819
.131
843
.734
.661
.755
.84
.542
.902
.577
.677
DAE
.769
.124
872
935
.884
.793
.655
.533
910
.625
.731
VAB(R)
994
.391
.978
910
860
.999
916
.774
946
.721
.839
VAE(L)
9677
.326
976
.906
.798
.977
.988
.751
.955
.614
.813
WAE
817
.145
.955
.950
.751
.42
.853
.912
.907
.799
.805
GLOW
.003
.014
.624
.625
.364
.561
.583
.326
.721
.426
.505
PXCNN++
+
.757
.030
.663
.663
.483
.642
.596
.307
.810
.497
.545
IGEBM
.266
.401
.642
.644
.664
.752
.851
.572
.747
.522
.672
DAGMM
.386
.304
.407
.435
.444
.429
.446
.349
.609
.420
.423
the encoder in an autoencoder, then other outlier detection algorithms, such as one-class support machines (Schölkopf al., 200T], are applied. Deep autoencoding Gaussian mixture model (DAGMM. Zong et (201 trains an autoencoder jointly with mixture of Gaussian distributions defined in the latent space of the autoencoder. The key assumption of using the latent representation of an autoencoder is that outlier will reside far from the training samples in the latent space.
After Nalisnick al. (2019); Hendrycks et (2099) reported that deep generative models are not able to detect obvious outliers, a number of outlier detection methods have been proposed to remedy the problem. Some of those methods require training of multiple generative models (Ren et al., Choi et al.) 2018). Other methods are highly specialized to image data (Golan & El- Yaniv Tack et al. 2020 Serrà et al. 2020). In this paper, we focus on the generative approach which more generally applicable.
2019
2018
Energy-based models The energy-based formulation (Eq.(T) Mnih & Hinton (2005): Hinton et al. 2006 model density function has long history of research. Recently, number of works attempt to use a deep neural network to model an energy function, showing promising results on generative modeling and OOD detection. [uu & Mordatch 2019); [Grathwohl al. (2020)
a deep feed-forward network as energy function and train it via contrastive divergence learning which involves LMC sampling. /Zhao al. (2016) propose Deep Structured Energy-Basse Model (DSEBM), where the gradient of its energy function is set as the reconstruction error. Energy-based Generative Adversarial Networks (EBGAN) Zhao al. (2016) employs the reconstruction error of a deep autoencoder as a discriminator and call it an energy function. However, EBGAN does not explicitly model density function and trained via methods similar to other generative adversarial networks. Also, EBGAN requires separate generator to generate samples.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
We empirically compare EBAE to other generative models in outlier detection tasks. In experi- ments, each method is trained using inlier data and then asked discriminate outliers from inliers during test phase. As described in Section 2.1 we assume that a detection method produces a scalar decision function c(x) given input x, and its performance is measured AUC. Following the protocol of Ren et al. (2019) and Hendrycks et al. (2019), we use an additional OOD dataset differ- ent from the datasets used test phase tune model hyperpamraeters. Additional details on model implementation and datasets can be found in the supplementary material.
Our baselines includes autoencoders, deep generative models, and the state-of-the-an energy based model. We test conventional autoencoders (AE), DAE, VAE, and WAE as our autoencoder base- lines. Additionally, DAGMM (Zong et al., 2018), an autoencoder-ban reconstruction error and density in the latent space simultaneously in outlier detection is tested. For all autoencoder-ee based methods including EBAE, we use the identical network architectures and vary other hyperparameters including the latent dimensionality. For AE, DAE, WAE, and EBAE, reconstruction error as c(x). Since both reconstruction error and likelihood is available from VAE, test both options for c(x) and denote the results VAE(R) and VAB(L., respectively. Deep
approach which considers

---

Under review as a conference paper at ICLR 2021
generative model baselines. PixeICNN+ (PXCNN++, Salimans al. (017)) and Glow (Kingmal & Dhariwal 2018) are trained by maximizing likelihood, and their negative log likelihoods are used c(x). We implement deep energy- baaegy-based model (IGEBM, Du & Mordatch (2019) and use its energy as c(x).
In addition to widely used datasets, including MNIST, FashionMNIST (FMNIST), CIFAR-IO, SVHN, CelebA, and ImageNet resized to 32x32 (ImageNet32, |oodd et al. (2016), we addition- ally utilize two synthetic image datasets, Constant and Noise. Constant dataset, all pixels of an image has the same value uniform-randomly drawn from the set {0, ..., 255]. Images in Constant dataset are gray, i.e., their channels have the same values. An image in Noise dataset has its pixel values independently drawn from the uniform distribution on the set {O.... .5.).. Constant and Noise images are generated two sizes, x 28 and are clearly OOD for all mentioned datasets. Pixel values of images are scaled to the [0, 1] in all experiments.
x 32. Note that both image datasets
6.2 HOLD-OUT CLASS DETECTION
We set one class from MNIST as the outlier class and the rest as the inlier class, and repeat the pro- cedure for all classes in MNIST. This experiment simulates the situation where inlier distribution consists of multiple disconnected clusters. Constant dataset of size 28x28 used for select the best hyperparameters for each method. After hyperparameter selection, the training for EBAE under the selected hyperparameter is repeated for 10 different random seeds to demonstrate the variance of EBAE training. The results shown in Table
EBAE shows the highest AUC score for all classes except for 7, while still achieves the second best performance for detecting 7 and the best overall performance. Note that detecting 1, 7, 9 as outlier is more difficult than detecting other digit. From the discussion in Section 3 we suspect that 1, 7, and more readily represented by composition of sub-patterns in other digits.
6.3 OOD DETECTION
Table 2: OOD detection (in-distribution: CIFAR-10),
In this experiment, train generative models on CIFAR-10 or ImageNet 32x32 (ImageNet32). and test how well they classify images from other datasets as OOD. Here, we use MNIST images padded with zeros to make 32x32 for model selection. Other details are covered in the supplementary material. Results are shown Table 2 and Table 3 Out- lier reconstruction is severe for Constant and SVHN datasets, as noted from AUC lower than 0.5. The right panel of Figure compares the reconstruction етrоr distri- bution of Constant and SVHN from AE and EBAE trained on CIFAR-10, showing that the outlier reconstruction effectively suppressed in EBAE.
OOD
Constant FMNIST SVHN CelebA Noise
EBAE
.923
.819
.818
.789
1.0
AE
.006
.650
.175
.655
1.0
DAE
.001
.671
.175
.669
1.0
VAE(R)
.002
.700
.191
.662
1.0
VAE(L)
.002
.767
.185
.684
1.0
WAE
.000
.649
.168
.652
1.0
GLOW
384
.222
.260
.419
1.0
PXCNN+
.000
.013
.074
.639
1.0
IGEBM
.192
.216
.371
.477
1.0
Table 3: 0OD detection (in-distribution: ImageNet32).
OOD
Constant FMNIST SVIIN CelebA Noise
EBAE
.966
.994
.985
.949
1.0
AE
.005
995
.102
.325
1.0
DAE
.069
.991
.102
.426
1.0
7 CONCLUSION
VAE(R)
.936
.132
.501
1.0
VAE(L)
.028
.950
.132
.545
1.0
WAE
.069
.991
.081
.364
1.0
In this paper, we have investigated the un- expected reconstruction of outliers in au- toencoders. To fix the problem, we have proposed novel probabilistic view autoencoders, which leads experiments, an autoencoder trained by the proposed method less prone to reconstruct an outlier and is highly effective in outlier detection.
GLOW
.413
.856
.169
.479
1.0
PXCNN+
.000
.004
.027
.238
1.0
training scheme. In
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-359) 2014.
Jinwon An and Sungzoon Cho. Variational autoencoder hased anomaly detection using reconstruc- tion probability. Special Lecture IE, 2(1):1-18, 2015.
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. arXiv preprint urXiv:2003.0599, 2020.
Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski. Deep generative stochastic networks trainable by backprop. In International Conference on Machine Learning, pp. 226-234, 2014.
Christopher M Bishop. Novelty detection and neural network validation. IEE Proceedings Vision, Image and Signal processing, 141(4):117-222, 1994.
Hervé Bourlard and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):291-294, 1988.
Mareus Brubaker, Mathieu Salzmann, and Raquel Urtasun. family of meme methods implicitly defined manifolds. In Neil D. Lawrence and Mark Girolami (eds.). Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 161-172, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. html
URL
r. wptpss/////bbbbaaeeeeeeeeee
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint 188.vv/810001392 2018.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspher- ical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 856-865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based mod- els. In H. Wallach, H. Larochelle,
Beygelzimer, F. d Alche-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 3608- 3618. Curran Associates, Inc., 2019. URL http: http::/pppers.aetp ni www.:ii....cpcpppepeeeeeoopp gew.ratien---neeeeeeee and-modd mode ing wit -att---errr---aee---eeeeee
8619- 1611=mmoiiei11-eeerrttoo
pdf
Izhak Golan and Ran El- Yaniv. Deep anomaly detection using geometric transformations. In Ad- vances Neural Information Processing Systems, pp. 9758-9769 99699, 2018.
Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton den Hengel. Memorizing normality detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In IEEE International Conference on Computer Vision (ICCV), 2019.
Will Grathwohl,
Wang, Joern- Heen--Hnrik.J. Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly energy based model and you should treat like one. In International Conference on Learning Representations, 2020. URL https::/ openreview. nwt.//ee/eeee ftrrm//rrmmmmmm wot.orrmmmm...xxx0000
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2019. URL https://
openreview Seennreviewnnet/oone worttoouum/ hiammmiddd-ycci Hyxdd--yxxxxce)
Geoffrey Hinton, Simon Osindero, Max Welling, and Yee-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science. 30(4):725-731, 2006.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 17(8)1177--1800, 2002.

---

Under review as a conference paper at ICLR 2021
Nathalie Japkowicz, Catherine Myers, Mark Gluck, al. A novelty detection approach to classifi- cation. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 1, pp. 518-523, 1995.
Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashu- bin, Nikola Momchex, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, et al. Measur- ing compositional generalization: comprehensive method on realistic data. arXiv preprint arXiv:1912.09713, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Yuchen Lu and Peng Xu. Anomaly detection for skin disease images using variational autoencoder. 2018.
arXiv preprint
Olga Lyudchik. Outlier detection using autoencoders. Technical report, 2016.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: /51 I.05644, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-3119 19, 2013.
Andriy Mnih and Geoffrey Hinton. Learning nonlinear constraints with contrastive backpropagation. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 1302-1300 IEEE, 2005.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In International Conference on Learning Representations, 2019. URL httpttt:://pereeeeeeee
net htt////owmmm// H1 #ixxNhCCYY
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 ()12) 2011.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 1601. 00166/00067555 2016.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In Advances neural information processing systems, pp. 6822-6833, 2018.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for -iitt--disttibutionn detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 833-840, 2011.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning Internal Representations by Error Propagation, pp- 318-362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C. Williamson. Estimating the support a high-dimensional distribution. Neural computation, 2001.
10

---

Under review as a conference paper at ICLR 2021
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood- based generative models. In Interna- tional Conference Learning Representations, 2020. URL
forum? S=Sxx
Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Journal of Machine Learning Research, 6(Feb):211-22) 1-232, 2005.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive
learning distributionally shifted instances. arXiv preprint arXiv:2007.081766 2020.
Iyy Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-
encoders. arXiv preprint arXiv: 2017.
Alexander Tong, Roozbah Yousefzadeh. Guy Wolf, and Smita Krishnaswamy. Fixing bias in reconstruction-basee anomaly detection with lipschitz discriminators. arXiv preprint arXiv:1905.10710, 2019.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning ([CML-II), pp. 681-688, 2011.
Dan Xu, Elisa Ricci, Yan Yan, Jingkuan Song, and Nicu Sebe. Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553 2015.
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Pro- ceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 4503-4513, 2018.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. arXiv preprint arXiv:1605.07717, 2016.
Deli Zhao, Jiapeng Zhu, and Bo Zhang. Latent variables on spheres for autoencoders in high dimen- sions. arXiv, pp. arXiv-1912, 2019.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint 1rXhv:1609.03126. 2016.
Yiru Zhao, Bing Deng, Chen Shen, Yao Liu, Hongtao Lu, and Xian-Sheng Hua. Spatio-temporal au- toencoder for video anomaly detection. In Proceedings of the 25th ACM international conference Mullimedia, pp. 1933-1941, 2017.
Bo Zong, Qi Song. Martin Renqiang Min, Wei Cheng. Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In In- lernational Conference on Learning Representations, 2018. URL


Reviews:
[{"review_id": "vSEWhT6IXCS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author propose a variant of energy based model to define the energy via \\|x-f(x)\\| where f is a network.  The empirical studies are provided on MNIST, CIFAR and ImageNet32, which shows some advantage of the proposed algorithms. \n\n1.  It is not convincing to me why the proposed model should be better than existing energy based model. After defining the energy as \\|x-f(x)\\|, the remaining details of the algorithm are routines by following the known results. It is the nature of all EBMs to assign high density to training data and lower down the energy of the sampled data. Based on this intuition, several algorithms have been proposed further, such as CD and PCD back to decades ago. Then the question is why use exp(-\\|x-f(x)\\|) as energy should be better than exp(-g(x))?  What's the implicit prior we leverage here? Whether it is especially useful for OOD or generally true for all density model? If so, then why? otherwise, it looks not convincing that it can significantly outperform the standard energy based model if the model capacity is the same, which I don't find either in the paper. The authors only briefly mention the architecture for ImageNet32 experiments.  The authors should shorten the descriptions of known results on EBMs and focus more on why the defined energy is better than existing  on OOD. \n\n2.  The experiments of other baselines are potentially flawed.  The reported numbers are worse than they should be. For example, on CIFAR10 vs CelebA, GLOW can achieve 0.9 AUC, as reported here https://arxiv.org/pdf/2006.09273.pdf. \n\n3. The considered experiments are relative easier.  There are some more challenging experiments could be considered to make the results stronger. For example, CIFAR10 vs CIFAR100. Or doing the same setting as the experiments in Table 1 on CIFAR. \n\n4. Related works. There are other AE based works on OOD/Anomaly detection are not discussed, such as \n* Rate-Distortion Optimization Guided Autoencoder for Isometric Embedding in Euclidean Latent Space\n\n5. There are lots of recent works discussing about likelihood is not a good metric for OOD. Whether the proposed works contradicts those results? If yes, which design makes it better than all other LLD based model?", "extra_scores": {}, "extra_text": {}}, {"review_id": "lPoWl77-HN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address an important problem of autoencoders having low reconstruction error for OOD instances. They use the regular inlier reconstruction loss minimization with an additional term to maximize reconstruction loss for fake sampled OOD instances. A two stage Langevin Monte Carlo sampling technique is used for sampling in the proposed EBAE framework to generate diverse samples. Empirical analysis reveals that the proposed method outperforms existing autoencoder baselines. Some of my concerns with this paper are the following\n\na) This contribution \"We present a sampling method tailored for EBAE which efficiently generates diverse samples\" needs better clarification and presentation. Figure 5 conveys the message but the writeup and math need to be written more clearly.\n\nb) Table 1 is not rightly placed. It should not be in related work section. Also why are standard deviation values not reported for baselines ? Are those values taken from the corresponding papers directly ? For anomaly detection in particular it is not sufficient just to report AUC and other metrics like AUPR or F-1 score should also be reported. \n\nc) Table 2 and Table 3 the captions are not descriptive enough.\n\nd) Another relevant autoencoder baseline which should be considered for empirical analysis is Feature Bagging Autoencoder. (Outlier Detection with Autoencoder ensembles. Chen et al. 2017) \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SWoyaklTp2h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A generative autoencoder outlier detection called EBAE is proposed by introducing fake samples which produce small  reconstruction error, yet being outliers, during training to enhance the AE outlier detectability. The fake sample generation is done in 2 stages via MCMC method. Number of experiments are carried out on various datasets and the results are promising. The paper approaches an interesting topic that why AE  may fail to detect OOD samples. I have the following concerns/comments:\n\n-\tMy first question is about fake sample generation, it is not clear to me why the generated samples should be considered as outliers? According to section 4.4 AE is initialized by training it in a conventional way, so it is expected that the generated samples from this model to be close to the training samples, how do we guarantee that the fake samples which already have small reconstruction error are necessarily OOD?\n-\tThe convergence happens when P(x) and P_theta(x) are equal; in other words, the model would represent data distribution; thus the expectation is that the generated samples would look like the real ones more and more as the training goes forward? Is that being observed? What iteration results does Figure 5 depict?\n-\tHow many fake samples are augmented in each iteration? According to the abovementioned point, would you reduce it as training goes forward?\n-\tIn general, thinking of some simpler approaches to do the same thing:\no\tAs mentioned in the paper, the fact that AE may well reconstruct an outlier could be because of a good generalization, which results in inductive biases; then, overfitting could be also helpful to avoid that. How do you argue about that? and do you have any results showing the proposed method could outperform overfitting.\no\tThe proposed method utilizes data augmentation, and generating them via the sampling scheme presented in this paper is one way. Could we improve OOD detection by other augmentation like those artificial done in Figure 1?\n-\tThere are number of GAN based anomaly detection methods (e.g. GANomaly, AnoGAN,..) to estimate the distribution of data, how do you compare your method with them?\n-\tRegarding Figure 1:\no\tIt would be better to see the left and right results for the same dataset (either MNIST or CIFAR10).\no\tBy looking at the right plots, it is difficult to see any improvement  using EBAE?\n-\tIn Figure 3, due to scaling it is difficult to confirm “the outlier reconstruction occurs only when D_z is large”.\n-\tIn equation (1), p_theta(x) is defined according to E_theta(x), but in equation (2) (and also in supplementary document) it is changed to E(x), without theta subscript. Is it a typo? How do you discriminate them?\n-\tIn section 4.3 Figure 4 is referred as Figure 5 mistakenly.\n-\tIn table 2 and 3 please mention the metric (what the numbers in the table represent) in the caption.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzIWxLsPRn4", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper describes a new method for detecting outliers with deep autoencoders by suppressing the reconstruction of out-of-distribution data. The article first investigates the reasons why standard autoencoders (AEs) reconstruct outlier datapoints fairly well, and are therefore problematic when used to detect anomalies via the reconstruction loss. The main novel contribution is the Energy-based Autoencoder (EBAE), a variant of an autoencoder in which the reconstruction loss is directly used as an energy function, and therefore outliers should have high energy. This is done with a new gradient formulation that enforces normalization of the probability distribution by sampling from the learned model via a variant of Langevin Monte-Carlo sampling. This second term is supposed to punish the reconstruction of outliers. Results are shown on a MNIST holdout task (leave one class out), and for OOD detection in CIFAR-10 and a downsampled version of ImageNet, showing that EBAE learns to reconstruct samples from different datasets, as well as constant and noise inputs, with significantly higher reconstruction error than competing methods.\n\n### Evaluation\nThe field of anomaly detection with deep autoencoders is relevant for a large part of the ICLR community. The paper contributes a novel method that improves outlier detection on the tested benchmarks, so the paper has some significance for the field of anomaly detection. The paper is overall relatively well written, although there are some sections with quite a few grammar mistakes (e.g. Sec. 4.2) that should be corrected, and neither the figures nor their captions are very clear. The originality of the paper is quite limited, because it is to a large part based on known observations, and adds an incremental improvement over previous approaches tackling the same problem. The methods and the related work are described in sufficient detail. In the section on outlier reconstruction it is not clear which of the proposed reasons for outlier reconstruction by AEs are novel insights, and which are known facts from the literature. Only for the first argument (smoothness) a source is quoted. The experiments are quite extensive in comparing to different algorithms and with multiple outlier datasets, but the interpretation of these results is very short, almost non-existent.\n\nOverall I think this paper is about at the acceptance threshold. It is a nice but not a groundbreaking new method, and although the results on the tested benchmarks look good, these are relatively easy benchmarks for outlier detection, and so the relevance for real-world problems is not quite clear. In particular it is not clear if the sampling mechanism can deal with difficult outlier detection problems where inliers and outliers are similar (e.g. outliers produced via corruptions). Additionally I am concerned about the computational complexity of the method compared to other approaches. I think the paper would greatly benefit from a more exhaustive interpretation of the results.\n\n**Pros**:\n1. The paper shows a new method for outlier detection that improves existing AE-based techniques.\n2. There are a lot of experiments with different datasets and different competing algorithms for the three main tested datasets (MNIST, CIFAR-10, and ImageNet32).\n3. The experimental results show an advantage on the tested datasets both for class holdout and outliers from other datasets.\n\n**Cons**:\n1. Energy-based interpretations of autoencoders are well-known, so this is a rather incremental addition to the existing literature. \n2. The method adds computational complexity through the sampling process, and it is not clear how many samples are needed for the method to become effective. The supplementary material provides the number of steps to generate a sample, but I could not find the total number of samples generated (maybe I missed it). The authors say that the complexity is OK for the tested datasets, but I would assume for larger images and more complex datasets this could become a big issue, because there will not be an efficient sampling of outliers in such cases.\n3. There is almost no interpretation of the results. Why e.g. do almost all other methods have great difficulty with the constant inputs, but no problems with noise? Why are methods such as PXCNN++ performing so poorly on most datasets?\n4. The benchmarks are rather simple, because the tested outlier datasets are quite different from the training set. Tests e.g. on corrupted versions of the training data would be interesting to test the power of the method. Robust anomaly detection, i.e. detecting outliers under the assumption that a small fraction of training points are anomalies could also prove challenging, since such datapoints might be reproduced by the sampling process (see e.g. Beggel, L., Pfeiffer, M., & Bischl, B. (2019). Robust anomaly detection in images using adversarial autoencoders. ECML (pp. 206-222); Choi, H., Jang, E., & Alemi, A. A. (2018). Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392.) It would be good to talk more in the paper about the limitations of the method.\n5. In the tables on outlier detection I was missing the reconstruction error for inliers as a reference.\n6. The figures are not properly described in the captions, and some of them e.g. Fig. 2 cannot be understood without reading the full text.\n\n\n### Additional Comments\n1. In Fig. 1 (left) and Fig. 5 (right) it appears that all reconstructions of EBAE have much higher brightness than the original and reconstructions of AE. Why is that?\n2. In Fig. 1 (right) it is unclear why the reconstruction loss of EBAE is overall so much higher than for AE (the axes are shifted). The figure suggests a pretty high loss also for inliers, compared to AE, that's why I am also asking to provide the inlier values for Tables 2 and 3.\n3. Please improve the caption of Fig. 2, this is not understandable.\n4. In section 3 (towards the end) I don't understand the following sentence: \"The latent representation is more distribution for large D_z...\". Please correct.\n5. In Section 4.2. I am not sure about the reasoning why the sampling should produce outliers at all. It is only mentioned that sampling produces high-likelihood samples, but if only inliers are sampled, how does this help distinguishing from real outliers? It seems unlikely that proper outliers are sampled at all from this process.\n6. Section 6.1, Typo: \"hyperpamraeters\"\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use reconstruction error of autoencoder as the energy function and normalize the resulting density for detecting anomalous/OOD examples. Reviewers have raised several concerns with the paper, including, lack of insights into why the AE energy is better for OOD detection than other energy function parameterizations, and incremental nature of the proposed method. Authors have not responded to these concerns. The paper is not suitable for publication in its current form. 
Decision: Reject