Deep Anomaly Detection with Outlier Exposure

1 INTRODUCTION . Machine Learning systems in deployment often encounter data that is unlike the model ’ s training data . This can occur in discovering novel astronomical phenomena , finding unknown diseases , or detecting sensor failure . In these situations , models that can detect anomalies ( Liu et al. , 2018 ; Emmott et al. , 2013 ) are capable of correctly flagging unusual examples for human intervention , or carefully proceeding with a more conservative fallback policy . Behind many machine learning systems are deep learning models ( Krizhevsky et al. , 2012 ) which can provide high performance in a variety of applications , so long as the data seen at test time is similar to the training data . However , when there is a distribution mismatch , deep neural network classifiers tend to give high confidence predictions on anomalous test examples ( Nguyen et al. , 2015 ) . This can invalidate the use of prediction probabilities as calibrated confidence estimates ( Guo et al. , 2017 ) , and makes detecting anomalous examples doubly important . Several previous works seek to address these problems by giving deep neural network classifiers a means of assigning anomaly scores to inputs . These scores can then be used for detecting outof-distribution ( OOD ) examples ( Hendrycks & Gimpel , 2017 ; Lee et al. , 2018 ; Liu et al. , 2018 ) . These approaches have been demonstrated to work surprisingly well for complex input spaces , such as images , text , and speech . Moreover , they do not require modeling the full data distribution , but instead can use heuristics for detecting unmodeled phenomena . Several of these methods detect unmodeled phenomena by using representations from only in-distribution data . In this paper , we investigate a complementary method where we train models to detect unmodeled data by learning cues for whether an input is unmodeled . While it is difficult to model the full data distribution , we can learn effective heuristics for detecting out-of-distribution inputs by exposing the model to OOD examples , thus learning a more conservative concept of the inliers and enabling the detection of novel forms of anomalies . We propose leveraging diverse , realistic datasets for this purpose , with a method we call Outlier Exposure ( OE ) . OE provides a simple and effective way to consistently improve existing methods for OOD detection . Through numerous experiments , we extensively evaluate the broad applicability of Outlier Exposure . For multiclass neural networks , we provide thorough results on Computer Vision and Natural Language Processing tasks which show that Outlier Exposure can help anomaly detectors generalize to and perform well on unseen distributions of outliers , even on large-scale images . We also demonstrate that Outlier Exposure provides gains over several existing approaches to out-of-distribution detection . Our results also show the flexibility of Outlier Exposure , as we can train various models with different sources of outlier distributions . Additionally , we establish that Outlier Exposure can make density estimates of OOD samples significantly more useful for OOD detection . Finally , we demonstrate that Outlier Exposure improves the calibration of neural network classifiers in the realistic setting where a fraction of the data is OOD . Our code is made publicly available at https : //github.com/hendrycks/outlier-exposure . 2 RELATED WORK . Out-of-Distribution Detection with Deep Networks . Hendrycks & Gimpel ( 2017 ) demonstrate that a deep , pre-trained classifier has a lower maximum softmax probability on anomalous examples than in-distribution examples , so a classifier can conveniently double as a consistently useful outof-distribution detector . Building on this work , DeVries & Taylor ( 2018 ) attach an auxiliary branch onto a pre-trained classifier and derive a new OOD score from this branch . Liang et al . ( 2018 ) present a method which can improve performance of OOD detectors that use a softmax distribution . In particular , they make the maximum softmax probability more discriminative between anomalies and in-distribution examples by pre-processing input data with adversarial perturbations ( Goodfellow et al. , 2015 ) . Unlike in our work , their parameters are tailored to each source of anomalies . Lee et al . ( 2018 ) train a classifier concurrently with a GAN ( Radford et al. , 2016 ; Goodfellow et al. , 2014 ) , and the classifier is trained to have lower confidence on GAN samples . For each testing distribution of anomalies , they tune the classifier and GAN using samples from that out-distribution , as discussed in Appendix B of their work . Unlike Liang et al . ( 2018 ) ; Lee et al . ( 2018 ) , in this work we train our method without tuning parameters to fit specific types of anomaly test distributions , so our results are not directly comparable with their results . Many other works ( de Vries et al. , 2016 ; Subramanya et al. , 2017 ; Malinin & Gales , 2018 ; Bevandic et al. , 2018 ) also encourage the model to have lower confidence on anomalous examples . Recently , Liu et al . ( 2018 ) provide theoretical guarantees for detecting out-of-distribution examples under the assumption that a suitably powerful anomaly detector is available . Utilizing Auxiliary Datasets . Outlier Exposure uses an auxiliary dataset entirely disjoint from test-time data in order to teach the network better representations for anomaly detection . Goodfellow et al . ( 2015 ) train on adversarial examples to increased robustness . Salakhutdinov et al . ( 2011 ) pre-train unsupervised deep models on a database of web images for stronger features . Radford et al . ( 2017 ) train an unsupervised network on a corpus of Amazon reviews for a month in order to obtain quality sentiment representations . Zeiler & Fergus ( 2014 ) find that pre-training a network on the large ImageNet database ( Russakovsky et al. , 2015 ) endows the network with general representations that are useful in many fine-tuning applications . Chen & Gupta ( 2015 ) ; Mahajan et al . ( 2018 ) show that representations learned from images scraped from the nigh unlimited source of search engines and photo-sharing websites improve object detection performance . 3 OUTLIER EXPOSURE . We consider the task of deciding whether or not a sample is from a learned distribution called Din . Samples from Din are called “ in-distribution , ” and otherwise are said to be “ out-of-distribution ” ( OOD ) or samples from Dout . In real applications , it may be difficult to know the distribution of outliers one will encounter in advance . Thus , we consider the realistic setting where Dout is unknown . Given a parametrized OOD detector and an Outlier Exposure ( OE ) dataset DOEout , disjoint from Dtestout , we train the model to discover signals and learn heuristics to detect whether a query is sampled from Din or DOEout . We find that these heuristics generalize to unseen distributions Dout . Deep parametrized anomaly detectors typically leverage learned representations from an auxiliary task , such as classification or density estimation . Given a model f and the original learning objective L , we can thus formalize Outlier Exposure as minimizing the objective E ( x , y ) ∼Din [ L ( f ( x ) , y ) + λEx′∼DOEout [ LOE ( f ( x ′ ) , f ( x ) , y ) ] ] over the parameters of f . In cases where labeled data is not available , then y can be ignored . Outlier Exposure can be applied with many types of data and original tasks . Hence , the specific formulation of LOE is a design choice , and depends on the task at hand and the OOD detector used . For example , when using the maximum softmax probability baseline detector ( Hendrycks & Gimpel , 2017 ) , we set LOE to the cross-entropy from f ( x′ ) to the uniform distribution ( Lee et al. , 2018 ) . When the original objective L is density estimation and labels are not available , we set LOE to a margin ranking loss on the log probabilities f ( x′ ) and f ( x ) . 4 EXPERIMENTS . We evaluate OOD detectors with and without OE on a wide range of datasets . Each evaluation consists of an in-distribution dataset Din used to train an initial model , a dataset of anomalous examples DOEout , and a baseline detector to which we apply OE . We describe the datasets in Section 4.2 . The OOD detectors and LOE losses are described on a case-by-case basis . In the first experiment , we show that OE can help detectors generalize to new text and image anomalies . This is all accomplished without assuming access to the test distribution during training or tuning , unlike much previous work . In the confidence branch experiment , we show that OE is flexible and complements a binary anomaly detector . Then we demonstrate that using synthetic outliers does not work as well as using real and diverse data ; previously it was assumed that we need synthetic data or carefully selected close-to-distribution data , but real and diverse data is enough . We conclude with experiments in density estimation . In these experiments we find that a cutting-edge density estimator unexpectedly assigns higher density to out-of-distribution samples than in-distribution samples , and we ameliorate this surprising behavior with Outlier Exposure . 4.1 EVALUATING OUT-OF-DISTRIBUTION DETECTION METHODS . We evaluate out-of-distribution detection methods on their ability to detect OOD points . For this purpose , we treat the OOD examples as the positive class , and we evaluate three metrics : area under the receiver operating characteristic curve ( AUROC ) , area under the precision-recall curve ( AUPR ) , and the false positive rate at N % true positive rate ( FPRN ) . The AUROC and AUPR are holistic metrics that summarize the performance of a detection method across multiple thresholds . The AUROC can be thought of as the probability that an anomalous example is given a higher OOD score than a in-distribution example ( Davis & Goadrich , 2006 ) . Thus , a higher AUROC is better , and an uninformative detector has an AUROC of 50 % . The AUPR is useful when anomalous examples are infrequent ( Manning & Schütze , 1999 ) , as it takes the base rate of anomalies into account . During evaluation with these metrics , the base rate of Dtestout to Dtestin test examples in all of our experiments is 1:5 . Whereas the previous two metrics represent the detection performance across various thresholds , the FPRN metric represents performance at one strict threshold . By observing performance at a strict threshold , we can make clear comparisons among strong detectors . The FPRN metric ( Liu et al. , 2018 ; Kumar et al. , 2016 ; Balntas et al. , 2016 ) is the probability that an in-distribution example ( negative ) raises a false alarm when N % of anomalous examples ( positive ) are detected , so a lower FPRN is better . Capturing nearly all anomalies with few false alarms can be of high practical value . 4.2 DATASETS . 4.2.1 IN-DISTRIBUTION DATASETS . SVHN . The SVHN dataset ( Netzer et al. , 2011 ) contains 32× 32 color images of house numbers . There are ten classes comprised of the digits 0-9 . The training set has 604 , 388 images , and the test set has 26 , 032 images . For preprocessing , we rescale the pixels to be in the interval [ 0 , 1 ] . CIFAR . The two CIFAR ( Krizhevsky & Hinton , 2009 ) datasets contain 32 × 32 natural color images . CIFAR-10 has ten classes while CIFAR-100 has 100 . CIFAR-10 and CIFAR-100 classes are disjoint but have similiarities . For example , CIFAR-10 has “ automobiles ” and “ trucks ” but not CIFAR-100 ’ s “ pickup truck ” class . Both have 50 , 000 training images and 10 , 000 test images . For this and the remaining image datasets , each image is standardized channel-wise . Tiny ImageNet . The Tiny ImageNet dataset ( Johnson et al . ) is a 200-class subset of the ImageNet ( Russakovsky et al. , 2015 ) dataset where images are resized and cropped to 64 × 64 resolution . The dataset ’ s images were cropped using bounding box information so that cropped images contain the target , unlike Downsampled ImageNet ( Chrabaszcz et al. , 2017 ) . The training set has 100 , 000 images and the test set has 10 , 000 images . Places365 . The Places365 training dataset ( Zhou et al. , 2017 ) consists in 1 , 803 , 460 large-scale photographs of scenes . Each photograph belongs to one of 365 classes . 20 Newsgroups . 20 Newsgroups is a text classification dataset of newsgroup documents with 20 classes and approximately 20 , 000 examples split evenly among the classes . We use the standard 60/40 train/test split . TREC . TREC is a question classification dataset with 50 fine-grained classes and 5 , 952 individual questions . We reserve 500 examples for the test set , and use the rest for training . SST . The Stanford Sentiment Treebank dataset ( Socher et al. , 2013 ) consists of movie reviews expressing positive or negative sentiment . SST has 8 , 544 reviews for training and 2 , 210 for testing .

Summary: This paper proposes fine-tuning an out-of-distribution detector using an Outlier Exposure (OE) dataset. The novelty is in proposing a model-specific rather than dataset-specific fine-tuning. Their modifications are referred to as Outlier Exposure. OE includes the choice of an OE dataset for fine-tuning and a regularization term evaluated on the OE dataset. It is a comprehensive study that explores multiple datasets and improves dataset-specific baselines.