Suppressing Outlier Reconstruction in Autoencoders for Out-of-Distribution Detection

1 INTRODUCTION . An autoencoder ( Rumelhart et al. , 1986 ) is a neural network trained to reconstruct samples from a training data distribution . As the quality of reconstruction is expected to degrade for inputs that are significantly different from training data , autoencoders are widely used in outlier detection ( Japkowicz et al. , 1995 ) where an input with a large reconstruction error is classified as out-of-distribution ( OOD ) . Such autoencoders for outlier detection have been applied in domains ranging from video surveillance ( Zhao et al. , 2017 ) to medical diagnosis ( Lu & Xu , 2018 ) . Contrary to widely-held belief , autoencoders are in fact capable of accurately reconstructing outliers , casting doubt on their reliability as an outlier detector . Lyudchik ( 2016 ) showed that an autoencoder trained on MNIST with the digit seven excluded can reconstruct an image of the excluded digit , and Tong et al . ( 2019 ) reported that an autoencoder trained on MNIST can reconstruct an image with all zero pixels . The reconstruction of outliers is also observed for non-image data ( Zong et al. , 2018 ) . In this paper , we investigate this unexpected behavior of autoencoders more deeply , which we refer to as outlier reconstruction . In the course of our investigation , we reproduce the findings of Lyudchik ( 2016 ) and Tong et al . ( 2019 ) , and additionally discover other interesting cases ( Figure 1 ) . Our experiments suggest that outlier reconstruction is not a fortuitous artifact of stochastic training but is , in fact , a consequence of inductive biases inherent in an autoencoder . Outlier reconstruction should be suppressed for an autoencoder-based outlier detector , since a reconstructed outlier undermines the detector ’ s performance by being mistaken to be an inlier . Despite the long history of autoencoder research ( Rumelhart et al. , 1986 ; Bank et al. , 2020 ) , the outlier reconstruction phenomenon has only recently begun to receive attention ( Lyudchik , 2016 ; Tong et al. , 2019 ; Zong et al. , 2018 ) , with few works explicitly proposing solutions to the outlier reconstruction problem ( Gong et al. , 2019 ) . Previous works focused on regularization techniques that prevent an autoencoder from being an identity mapping ( and thus reconstructing all inputs ) . However , outlier reconstruction still occurs in popular regularized autoencoders , including denoising autoencoders ( DAE , Vincent et al . ( 2008 ) ) , variational autoencoders ( VAE , Kingma & Welling ( 2014 ) ) , and Wasserstein autoencoders ( WAE , Tolstikhin et al . ( 2017 ) ) , as we shall show in our experiments ( Table 1 ) . In this paper , we propose the Energy-based Autoencoder ( EBAE ) , an autoencoder in which the reconstruction of outliers is explicitly suppressed during training . In each training step of EBAE , “ fake ” samples with small reconstruction error are generated . These well-reconstructed fake samples serve as probes for potential reconstructed outliers . Then , EBAE maximizes the reconstruction errors of the generated samples , while minimizing the reconstruction errors of “ real ” training samples . When the generated samples become indistinguishable to training data , the gradients from the fake samples and real samples balance , and thus the training converges . The training scheme naturally arises from defining a probability density for EBAE using its reconstruction error . The density of EBAE is given as pθ ( x ) = exp ( −E ( x ) ) /Ω , where E ( x ) is the reconstruction error of x and Ω is a normalization constant . This formulation of defining a density using a scalar function is often called an energy-based model in the literature ( Mnih & Hinton , 2005 ; Hinton et al. , 2006 ) , and E ( x ) is called the energy of the density . Maximizing likelihood in this formulation results in contrastive divergence learning ( Hinton , 2002 ) , which minimizes the energy of the training data while maximizing the energy of the samples from the model . When generating samples with small reconstruction error during training , we use a novel sampling scheme specifically designed for EBAE . Our sampling scheme is based on Langevin Monte Carlo but leverages the latent space of an autoencoder to generate diverse samples which facilitates the training of EBAE . Setting the reconstruction error as the energy , EBAE incorporates two major outlier detection criteria , large reconstruction error ( Japkowicz et al. , 1995 ) and low likelihood ( Bishop , 1994 ) , since the two are equivalent in EBAE . Generally , the two criteria do not necessarily overlap in other methods , e.g. , VAE or energy-based models ( Zhai et al. , 2016 ) . Recent studies show that a likelihood-based outlier detector using a deep generative model , such as an auto-regressive model or flow-based model , fails to correctly classify certain obvious outliers ( Nalisnick et al. , 2019 ; Hendrycks et al. , 2019 ) . However , EBAE is able to detect such outliers successfully while still using likelihood as the decision criterion . The contributions of our paper can be summarized as follows : • We report and investigate various cases of outlier reconstruction in autoencoders ; • We propose EBAE , an autoencoder significantly less prone to outlier reconstruction ; • We present a sampling method tailored for EBAE which efficiently generates diverse samples ; • We empirically show that EBAE is highly effective for outlier detection . Section 2 provides a brief introduction on autoencoder-based outlier detection . In Section 3 , we investigate outlier reconstruction in depth with illustrative examples . Section 4 describes EBAE . Related works are reviewed in Section 5 . Section 6 presents experimental results . Section 7 concludes the paper . 2 BACKGROUND . 2.1 PROBLEM SETTING . In this paper , we consider the outlier detection problem , which is also referred to as novelty detection , open set recognition , or OOD detection in literature . The goal is to classify outliers from in-distribution samples , while no information regarding the outliers to be detected is provided during training . Formally , we are given a set of inliers x ∈ X ⊂ RDx from the underlying data density function p ( x ) for training . Dx is the dimensionality of x and X is the support of p ( x ) . An outlier is typically defined as a sample from the ρ-sublevel set of data density { x|p ( x ) ≤ ρ } ( Steinwart et al. , 2005 ) . Note that a sample located outside of the support X belongs to the 0-sublevel set and hence is an outlier for all ρ ≥ 0 . An outlier detection system typically produces a scalar decision function c ( x ) , and x is predicted as an outlier if c ( x ) > η for the threshold η . Setting the threshold controls the trade-off between false positive rate and false negative rate . In our experiments , we shall use area under receiver operating characteristic curve ( AUC ) as a threshold-independent performance metric when evaluating outlier detection algorithms . 2.2 AUTOENCODER-BASED OOD DETECTION . An autoencoder consists of an encoder fe ( x ) : RDx → RDz and a decoder fd ( z ) : RDz → RDx , where Dz is the dimensionality of the latent vector z . An input x is sequentially processed through the encoder and the decoder , producing its reconstruction x̃ = fd ( fe ( x ) ) . The reconstruction error is the discrepancy between x and x̃ . L2 distance ||x− x̃||2 is a popular choice of discrepancy measure , but other error metrics are also applicable . A encoder and a decoder are deep neural networks and are jointly trained to minimize the mean reconstruction error of training data through stochastic gradient descent . In an autoencoder-based outlier detection system ( Japkowicz et al. , 1995 ) , the reconstruction error of an autoencoder trained on in-distribution samples is used as the decision function c ( x ) = ||x− x̃||2 , and an input with large reconstruction error is classified as OOD . However , in the next section , we shall show that an autoencoder could unexpectedly produce very small reconstruction error for inputs not drawn from the training distribution . Thus an autoencoder-based outlier detection system may fail to detect such outliers . 3 OUTLIER RECONSTRUCTION The outlier reconstruction is a phenomenon that an autoencoder unexpectedly succeeds in reconstructing an input even though it is located outside of the training distribution . In this section , we provide illustrative examples that show that outlier reconstruction is a consequence from the inductive biases of an autoencoder . Smoothness of mappings When the training data distribution consists of multiple clusters , the outliers from the region between the clusters are likely to be reconstructed . Figure 2 depicts 2D synthetic data generated from a mixture of two disconnected uniform distributions and their reconstruction from autoencoders with one-dimensional latent space . The outliers ( red crosses ) from the middle of two clusters show reconstruction errors ( the length of thin black lines ) smaller than some inliers ( blue dots ) . Tong et al . ( 2019 ) noted this type of outlier reconstruction and mentioned that outliers “ close to the mean ” of data or “ in the convex hull ” of data are likely to be reconstructed . This phenomenon arises from the inductive bias of an autoencoder that its encoder and decoder are smooth mappings . The extreme case of this inductive bias can be found in linear principal component analysis ( PCA ) . PCA , a linear counterpart of an autoencoder ( Bourlard & Kamp , 1988 ) , would reconstruct any outliers which reside on the principal axis . Note that this phenomenon is consistent with the objective function of an autoencoder and PCA , as the objective does not penalize the reconstruction of outliers . Compositionality When there is a compositional structure in data , we can still observe a reconstructed outlier even if it lies outside of the convex hull of training data . The data are compositional if each datum can be broken down into smaller reusable components ; For example , MNIST can be considered highly compositional , since a digit image can be decomposed into smaller sub-patterns , such as straight lines and curves . An outlier can be successfully reconstructed when composed of a subset of components existing in the training data . To demonstrate the effect of compositionality in outlier reconstruction , we make two synthetic datasets both of which are clearly out-of-distribution with respect to MNIST . The first dataset is HalfMNIST ( the seventh column in Figure 1 ) , consisting of MNIST images with a randomly chosen upper or lower half replaced by zero pixels . ChimeraMNIST ( the last column in Figure 1 ) , the second dataset , is a set of images generated by concatenating upper and lower halves of two randomly chosen digits . Although these images are not in the convex hull of MNIST digits , they share components found in MNIST . As shown in Figure 1 , an autoencoder trained on MNIST have no problem reconstructing them . The classification AUCs from the reconstruction error are 0.482 for HalfMNIST and 0.69 for ChimeraMNIST , indicating poor classification . It seems that an autoencoder learns to reconstruct each part of an image separately but is not able to judge whether the combination of the parts is valid as a whole . This compositional way of processing facilitates generalization of a model ( Keysers et al. , 2019 ) , but the generalization of reconstruction in OOD inputs is not desirable for an autoencoder-based outlier detector . Distributed representation We suspect the outlier reconstruction due to compositional processing may be attributed to the distributed representation ( Mikolov et al. , 2013 ) used in an autoencoder . To show the effect of the distributed representation , we train autoencoders on MNIST with the digit 9 excluded ( MNISTnot9 ) and measure the reconstruction error of the digit 9 ( MNIST9 ) under multiple values of latent dimensionality Dz . Figure 3 shows the result . We observe the outlier reconstruction of MNIST9 possibly due to the compositional processing mentioned above . However , the outlier reconstruction occurs only when Dz is large . The latent representation is more distribution for large Dz , as a larger number of hidden neurons are used to represent an input . This observation suggests that the distributed representation used in an autoencoder enables the compositional processing and thus facilitates outlier reconstruction . Meanwhile , the model we propose shortly is not vulnerable to outlier reconstruction in all cases we examined above , as shown in Figure 1 , Figure 2 , and Table 1 , even though it is based on the same autoencoder architecture .

Summary: The paper describes a new method for detecting outliers with deep autoencoders by suppressing the reconstruction of out-of-distribution data. The article first investigates the reasons why standard autoencoders (AEs) reconstruct outlier datapoints fairly well, and are therefore problematic when used to detect anomalies via the reconstruction loss. The main novel contribution is the Energy-based Autoencoder (EBAE), a variant of an autoencoder in which the reconstruction loss is directly used as an energy function, and therefore outliers should have high energy. This is done with a new gradient formulation that enforces normalization of the probability distribution by sampling from the learned model via a variant of Langevin Monte-Carlo sampling. This second term is supposed to punish the reconstruction of outliers. Results are shown on a MNIST holdout task (leave one class out), and for OOD detection in CIFAR-10 and a downsampled version of ImageNet, showing that EBAE learns to reconstruct samples from different datasets, as well as constant and noise inputs, with significantly higher reconstruction error than competing methods.