SSD: A Unified Framework for Self-Supervised Outlier Detection

1 INTRODUCTION . Deep neural networks are at the cornerstone of multiple safety-critical applications , ranging from autonomous driving ( Ramanagopal et al. , 2018 ) to biometric authentication ( Masi et al. , 2018 ; Günther et al. , 2017 ) . When trained on a particular data distribution , referred to as in-distribution data , deep neural networks are known to fail against test inputs that lie far away from the training distribution , commonly referred to as outliers or out-of-distribution ( OOD ) samples ( Grubbs , 1969 ; Hendrycks & Gimpel , 2017 ) . This vulnerability motivates the use of an outlier detector before feeding the input samples to the downstream neural network modules . However , a key question is to understand what training information is crucial for effective outlier detection ? Will the detector require fine-grained annotation of training data labels or even access to a set of outliers in the training process ? Since neither data labels nor outliers are ubiquitous , the most compelling option is to design outlier detectors based on only unlabeled in-distribution data . However , we observe that most of the existing outlier detectors based on unlabeled data fail to scale up to complex data modalities , such as images . For example , autoencoder ( AE ) ( Hawkins et al. , 2002 ) based outlier detectors have achieved success in applications such as intrusion detection ( Mirsky et al. , 2018 ) , and fraud detection ( Schreyer et al. , 2017 ) . However , this approach achieves close to chance performance on image datasets . Similarly , density modeling based methods , such as PixelCNN++ ( Salimans et al. , 2017 ) and Glow ( Kingma & Dhariwal , 2018 ) are known to assign even a higher likelihood to outliers in comparison to indistribution data ( Nalisnick et al. , 2019 ) . In contrast , existing state-of-the-art OOD detectors achieve high success on image datasets but assume the availability of fine-grained labels for in-distribution samples ( Hendrycks & Gimpel , 2017 ; 1Our code is publicly available at https : //github.com/inspire-group/SSD Bendale & Boult , 2016 ; Liang et al. , 2018 ; Dhamija et al. , 2018 ; Winkens et al. , 2020 ) . This is a strong assumption since labels , in-particular fine-grained labels , can be very costly to collect in some applications ( Google AI Pricing , 2020 ) , which further motivates the use of unlabeled data . The inability of supervised detectors to use unlabeled data and poor performance of existing unsupervised approaches naturally give rise to the following question . Can we design an effective out-of-distribution ( OOD ) data detector with access to only unlabeled data from training distribution ? A framework for outlier detection with unlabeled data2 involves two key steps : 1 ) Learning a good feature representation with unsupervised training methods 2 ) Modeling features of in-distribution data without requiring class labels . For example , autoencoders attempt to learn the representation with a bottleneck layer , under the expectation that successful reconstruction requires learning a good set of representations . Though useful for tasks such as dimensionality reduction , we find that these representations are not good enough to sufficiently distinguish in-distribution data and outliers . We argue that if unsupervised training can develop a rich understanding of key semantics in in-distribution data then absence of such semantics in outliers can cause them to lie far away in the feature space , thus making it easy to detect them . Recently , self-supervised representation learning methods have made large progress , commonly measured by accuracy achieved on a downstream classification task ( Chen et al. , 2020 ; He et al. , 2020 ; Oord et al. , 2018 ; Misra & Maaten , 2020 ; Tian et al. , 2020 ) . We leverage these representations in our proposed cluster-conditioned framework based on the Mahalanobis distance ( Mahalanobis , 1936 ) . Our key result is that self-supervised representations are highly effective for the task of outlier detection in our self-supervised outlier detection ( SSD ) framework where they not only perform far better than most of the previous unsupervised representation learning methods but also perform on par , and sometimes even better , than supervised representations . What if access to a fraction of OOD data or training data labels is available ? How do we move past a detector based on unlabeled data and design a framework which can take advantage of such information ? Though access to outliers during training is a strong assumption , it may be feasible to obtain a few prior instances of such outliers ( Görnitz et al. , 2013 ) . We characterize this setting as few-shot OOD detection , where we assume access to very few , often one to five , samples from the targeted set of outliers . While earlier approaches ( Liang et al. , 2018 ; Lee et al. , 2018b ) mostly use such data to calibrate the detector , we find that access to just a few outliers can bring an additional boost in the performance of our detector . Crucial to this success is the reliable estimation of first and second order statistics of OOD data in the high dimensional feature space with just a few samples . Finally , if class labels are available in the training phase , how can we incorporate them in the SSD framework for outlier detection ? Recent works have proposed the addition of the supervised crossentropy and self-supervised learning loss with a tunable parameter , which may require tuning for optimal parameter setting for each dataset ( Hendrycks et al. , 2019b ; Winkens et al. , 2020 ) . We demonstrate that incorporating labels directly in the contrastive loss achieves 1 ) a tuning parameterfree detector , and 2 ) state-of-the-art performance . 1.1 KEY CONTRIBUTIONS . SSD for unlabeled data . We propose SSD , an unsupervised framework for outlier detection based on unlabeled in-distribution data . We demonstrate that SSD outperforms most existing unsupervised outlier detectors by a large margin while also performing on par , and sometimes even better than supervised training based detection methods . We validate our observation across four different datasets : CIFAR-10 , CIFAR-100 , STL-10 , and ImageNet . Extensions of SSD . We provide two extensions of SSD to further improve its performance . First , we formulate few-shot OOD detection and propose detection methods which can achieve a significantly large gain in performance with access to only a few targeted OOD samples . Next , we extend SSD , without using any tuning parameter , to also incorporate in-distribution data labels and achieve state-of-the-art performance . 2We refer to OOD detection without using class labels of in-distribution data as unsupervised OOD detection . 2 RELATED WORK . OOD detection with unsupervised detectors . Interest in unsupervised outlier detection goes back to Grubbs ( 1969 ) . We categorize these approaches in three groups 1 ) Reconstruction-error based detection using Auto-encoders ( Hawkins et al. , 2002 ; Mirsky et al. , 2018 ; Schreyer et al. , 2017 ) or Variational auto-encoders ( Abati et al. , 2019 ; An & Cho , 2015 ) 2 ) Classification based , such as DeepSVDD ( Ruff et al. , 2018 ; El-Yaniv & Wiener , 2010 ; Geifman & El-Yaniv , 2017 ) and 3 ) Probabilistic detectors , such as density models like Glow and PixelCNN++ ( Ren et al. , 2019 ; Nalisnick et al. , 2019 ; Salimans et al. , 2017 ; Kingma & Dhariwal , 2018 ) . We compare with detectors from each category and find that SSD outperforms them by a wide margin . OOD detection with supervised learning . Supervised detectors have been most successful with complex input modalities , such as images and language ( Chalapathy et al. , 2018a ; DeVries & Taylor , 2018 ; Dhamija et al. , 2018 ; Jiang et al. , 2018 ; Yoshihashi et al. , 2018 ; Lee et al. , 2018a ) . Most of these approaches model features of in-distribution data at output ( Liang et al. , 2018 ; Hendrycks & Gimpel , 2017 ; Dhamija et al. , 2018 ) or in the feature space ( Lee et al. , 2018b ; Winkens et al. , 2020 ) for detection . We show that SSD can achieve performance on par with these supervised detectors , without using data labels . A subset of these detectors also leverages generic OOD data to boost performance ( Hendrycks et al. , 2019a ; Mohseni et al. , 2020 ) . Access to OOD data at training time . Some recent detectors also require OOD samples for hyperparameter tuning ( Liang et al. , 2018 ; Lee et al. , 2018b ; Zisselman & Tamar , 2020 ) . We extend SSD to this setting but assume access to only a few OOD samples , referred to as few-shot OOD detection , which our framework can efficiently utilize to bring further gains in performance . In conjunction with supervised training . Vyas et al . ( 2018 ) uses ensemble of leave-one-out classifier , Winkens et al . ( 2020 ) uses contrastive self-supervised training , and Hendrycks et al . ( 2019b ) uses rotation based self-supervised loss , in conjunction with supervised cross-entropy loss to achieve state-of-the-art performance in OOD detection . Here we extend SSD , to incorporate data labels , when available , and achieve better performance than existing state-of-the-art . Anomaly detection . In parallel to OOD detection , this research direction focuses on the detection of semantically related anomalies in applications such as intrusion detection , spam detection , disease detection , image classification , and video surveillance . We refer the interested reader to Pang et al . ( 2020 ) for a detailed review . While a large number of works focuses on developing methods particularly for single-class modeling in anomaly detection ( Perera et al. , 2019 ; Schlegl et al. , 2017 ; Ruff et al. , 2018 ; Chalapathy et al. , 2018b ; Golan & El-Yaniv , 2018 ; Wang et al. , 2019 ) , some recent work achieve success in both OOD detection and anomaly detection Tack et al . ( 2020 ) ; Bergman & Hoshen ( 2020 ) . We provide a detailed comparison of our approach with previous work in both categories . 3 SSD : SELF-SUPERVISED OUTLIER/OUT-OF-DISTRIBUTION DETECTION . In this section , we first provide the necessary background on outlier/out-of-distribution ( OOD ) detection and then present the underlying formulation of our self-supervised detector ( SSD ) that relies on only unlabeled in-distribution data . Finally , we describe two extensions of SSD to ( optionally ) incorporate targeted OOD samples and in-distribution data labels ( if available ) . Notation . We represent the input space by X and corresponding label space as Y . We assume in-distribution data is sampled from PinX×Y . In the absence of data labels , it is sampled from marginal distribution PinX . We sample out-of-distribution data from PoodX . We denote the feature extractor by f : X → Z , where Z ⊂ Rd , a function which maps a sample from the input space to the d-dimensional feature space ( Z ) . The feature extractor is often parameterized by a deep neural network . In supervised learning , we obtain classification confidence for each class by g ◦f : X → Rc . In most cases , g is parameterized by a shallow neural network , generally a linear classifier . Problem Formulation : Outlier/Out-of-distribution ( OOD ) detection . Given a collection of samples from PinX × PoodX , the objective is to correctly identify the source distribution , i.e. , PinX or PoodX , for each sample . We use the term supervised OOD detectors for detectors which use in-distribution data labels , i.e. , train the neural network ( g ◦ f ) on PinX×Y using supervised training techniques . Unsupervised OOD detectors aim to solve the aforementioned OOD detection tasks , with access to only PinX . In this work , we focus on developing effective unsupervised OOD detectors . Background : Contrastive self-supervised representation learning . Given unlabeled training data , it aims to train a feature extractor , by discriminating between individual instances from data , to learn a good set of representations . Using image transformations , it first creates two views of each image , commonly referred to as positives . Next , it optimizes to pull each instance close to its positive instances while pushing away from other images , commonly referred to as negatives . Assuming that ( xi , xj ) are positive pairs for the ith image from a batch of N images and h ( . ) is a projection header , τ is the temperature , contrastive training minimizes the following loss , referred to as Normalized temperature-scaled cross-entropy ( NT-Xent ) , over each batch . Lbatch = 1 2N 2N∑ i=1 −log e uTi uj/τ∑2N k=1 1 ( k 6= i ) eu T i uk/τ ; ui = h ( f ( xi ) ) ‖h ( f ( xi ) ) ‖2 ( 1 )

Summary: The paper addresses the OOD problem without learning from class labels. It proposes to learn the feature representation with unsupervised learning, then applies Mahalanobis distance to measure how far a test data is away from the in-distribution data. The proposed framework also considers the cases of when class labels or OOD data are available. The experiments on image datasets (mainly cifar-10/100) demonstrate its advantage over other unsupervised OOD methods.