Learning and Evaluating Representations for Deep One-Class Classification

1 INTRODUCTION . One-class classification aims to identify if an example belongs to the same distribution as the training data . There are several applications of one-class classification , such as anomaly detection or outlier detection , where we learn a classifier that distinguishes the anomaly/outlier data without access to them from the normal/inlier data accessible at training . This problem is common in various domains , such as manufacturing defect detection , financial fraud detection , etc . Generative models , such as kernel density estimation ( KDE ) , is popular for one-class classification [ 1 , 2 ] as they model the distribution by assigning high density to the training data . At test time , low density examples are determined as outliers . Unfortunately , the curse of dimensionality hinders accurate density estimation in high dimensions [ 3 ] . Deep generative models ( e.g . [ 4 , 5 , 6 ] ) , have demonstrated success in modeling high-dimensional data ( e.g. , images ) and have been applied to anomaly detection [ 7 , 8 , 9 , 10 , 11 ] . However , learning deep generative models on raw inputs remains as challenging as they appear to assign high density to background pixels [ 10 ] or learn local pixel correlations [ 12 ] . A good representation might still be beneficial to those models . Alternately , discriminative models like one-class SVM ( OC-SVM ) [ 13 ] or support vector data description ( SVDD ) [ 14 ] learn classifiers describing the support of one-class distributions to distinguish them from outliers . These methods are powerful when being with non-linear kernels . However , its performance is still limited by the quality of input data representations . In either generative or discriminative approaches , the fundamental limitation of one-class classification centers on learning good high-level data representations . Following the success of deep learning [ 15 ] , deep one-class classifications [ 16 , 17 , 18 ] , which extend the discriminative one-class classification using trainable deep neural networks , have shown promising results compared to their kernel counterparts . However , a naive training of deep one-class classifiers leads to a degenerate solution that maps all data into a single representation , also known as “ hypersphere collapse ” [ 16 ] . Previous works circumvent such issues by constraining network architectures [ 16 ] , autoencoder ∗Equal contribution . pretraining [ 16 , 17 ] , surrogate multi-class classification on simulated outliers [ 19 , 20 , 21 , 22 ] or injecting noise [ 18 ] . In this work , we present a two-stage framework for building deep one-class classifiers . As shown in Figure 1 , in the first stage , we train a deep neural network to obtain a high-level data representation . In the second stage , we build a one-class classifier , such as OC-SVM or KDE , using representations from the first stage . Comparing to using surrogate losses [ 20 , 21 ] , our framework allows to build a classifier that is more faithful to one-class classification . Decoupling representation learning from classifier construction further opens up opportunities of using state-of-the-art representation learning methods , such as self-supervised contrastive learning [ 23 ] . While vanilla contrastive representations are less compatible with one-class classification as they are uniformly distributed on the hypersphere [ 24 ] , we show that , with proper fixes , it provides representations achieving competitive one-class classification performance to previous state-of-the-arts . Furthermore , we propose a distribution-augmented contrastive learning , a novel variant of contrastive learning with distribution augmentation [ 25 ] . This is particularly effective in learning representations for one-class classification , as it reduces the class collision between examples from the same class [ 26 ] and uniformity [ 24 ] . Lastly , although representations are not optimized for one-class classification as in end-to-end trainable deep one-class classifiers [ 16 ] , we demonstrate state-of-the-art performance on visual one-class classification benchmarks . We summarize our contributions as follows : • We present a two-stage framework for building deep one-class classifiers using unsupervised and self-supervised representations followed by shallow one-class classifiers . • We systematically study representation learning methods for one-class classification , including augmentation prediction , contrastive learning , and the proposed distribution-augmented contrastive learning method that extends training data distributions via data augmentation . • We show that , with a good representation , both discriminative ( OC-SVM ) and generative ( KDE ) classifiers , while being competitive with each other , are better than surrogate classifiers based on the simulated outliers [ 20 , 21 ] . • We achieve strong performance on visual one-class classification benchmarks , such as CIFAR10/100 [ 27 ] , Fashion MNIST [ 28 ] , Cat-vs-Dog [ 29 ] , CelebA [ 30 ] , and MVTec AD [ 31 ] . • We extensively study the one-class contrastive learning and the realistic evaluation of anomaly detection under unsupervised and semi-supervised settings . Finally , we present visual explanations of our deep one-class classifiers to better understand their decision making processes . 2 RELATED WORK . One-class classification [ 32 ] has broad applications , including fraud detection [ 33 ] , spam filtering [ 34 ] , medical diagnosis [ 35 ] , manufacturing defect detection [ 31 ] , to name a few . Due to the lack of granular semantic information for one-class data , learning from unlabeled data have been employed for one-class classification . Generative models , which model the density of training data distribution , are able to determine outlier when the sample shows low density [ 8 , 35 , 36 ] . These include simple methods such as kernel density estimation or mixture models [ 37 ] , as well as advanced ones [ 4 , 5 , 6 , 38 , 39 , 40 , 41 ] . However , the density from generative models for high-dimensional data could be misleading [ 9 , 12 , 42 , 43 ] . New detection mechanisms based on the typicality [ 44 ] or likelihood ratios [ 10 ] have been proposed to improve out-of-distribution detection . Self-supervised learning is commonly used for learning representations from unlabeled data by solving proxy tasks , such as jigsaw puzzle [ 45 ] , rotation prediction [ 46 ] , clustering [ 47 ] , instance discrimination [ 48 ] and contrastive learning [ 23 , 49 , 50 ] . The learned representations are then used for multi-class classification , or transfer learning , all of which require labeled data for downstream tasks . They have also been extended to one-class classification . For example , contrastive learning is adopted to improve the out-of-distribution detection under multi-class setting [ 51 ] , whereas our work focuses on learning from a single class of examples , leading to propose a novel distributionaugmented contrastive learning . Notably , learning to predict geometric transformations [ 20 , 21 , 22 ] extends the rotation prediction [ 46 ] to using more geometric transformations as prediction targets . Unlike typical applications of self-supervised learning where the classifier or projection head [ 23 ] are discarded after training , the geometric transformation classifier is used as a surrogate for oneclass classification . As in Section 4.1 , however , the surrogate classifier optimized for the selfsupervised proxy task is suboptimal for one-class classification . We show that replacing it with simple one-class classifiers consistently improve the performance . Furthermore , we propose strategies for better representation learning for both augmentation prediction and contrastive learning . Distribution-augmented contrastive learning is concurrently developed in [ 52 ] as a part of their multi-task ensemble model . While sharing a similar technical formulation , we motivate from fixing the uniformity of contrastive representations . We note that our study not only focuses on representation learning , but also on the importance of detection algorithms , which is under explored before . 3 A TWO-STAGE FRAMEWORK FOR DEEP ONE-CLASS CLASSIFICATION . In Section 3.1 , we review self-supervised representation learning algorithms , discuss how they connect to existing one-class classification methods , raise issues of state-of-the-art contrastive representation learning [ 23 ] for one-class classification , and propose ways to resolve these issues . Then , in Section 3.2 , we study how to leverage the learned representations for one-class classification . 3.1 LEARNING REPRESENTATIONS FOR ONE-CLASS CLASSIFICATION . LetA be the stochastic data augmentation process , which is composed of resize and crop , horizontal flip , color jittering , gray-scale and gaussian blur , following [ 23 ] , for image data . As in Figure 1 , selfsupervised learning methods consist of a feature extractor f parameterized by deep neural networks and the proxy loss L. Optionally , f is further processed with projection head g at training , which is then used to compute the proxy loss . Unless otherwise stated , normalize ( f ) , f/‖f‖2 is used as a representation at test time . Below , we discuss details of self-supervised learning methods . 3.1.1 EXTRACTING RICHER REPRESENTATION BY LEARNING WITH PROJECTION HEAD . While the efficacy of projection head has been confirmed for contrastive learning [ 23 ] or BYOL [ 53 ] , it is not widely adopted for other types of self-supervised learning , including rotation prediction [ 46 ] . On the other hand , Gidaris et al . [ 46 ] show that the lower-layer representation often perform better for downstream tasks as the last layer directly involved in optimizing the proxy loss becomes overly discriminative to the proxy task , while losing useful information of the data . Inspired by these observations , we adopt the projection head for augmentation prediction training as well . As in Figure 1a , we extend the network structure as g ◦ f , where g is the projection head used to compute proxy losses and f outputs representations used for the downstream task . Note that using an identity head g ( x ) =x recovers the network structure of previous works [ 20 , 21 , 46 ] . 3.1.2 AUGMENTATION PREDICTION . One way of representation learning is to learn by discriminating augmentations applied to the data . For example , the rotation prediction [ 46 ] learns deep representations by predicting the degree of rotation augmentations . The training objective of the rotation prediction task is given as follows : Lrot = Ex∼PX , A [ CrossEntropy ( y , pq◦g ( y|rot90 ( A ( x ) , y ) ) ) ] ( 1 ) where y ∈ { 0 , 1 , 2 , 3 } is a prediction target representing the rotation degree , and rot90 ( x , y ) rotates an input x by 90 degree y times . We denote the classifier pq◦g ( y|x ) as pq ( y|g ( x ) ) ∝ exp ( q◦g ( x ) ) [ y ] containing the representation g1 and a linear layer q with 4 output units for rotation degrees . Application to One-Class Classification . Although not trained to do so , the likelihood of learned rotation classifiers2 pq ( y= 0|g ( x ) ) is shown to well approximate the normality score and has been used for one-class classification [ 20 , 21 , 22 ] . A plausible explanation is via outlier exposure [ 19 ] , where the classifier learns a decision boundary distinguishing original images from simulated outliers by image rotation . However , it assumes inlier images are not rotated , and the classifier may not generalize to one-class classification task if it overfits to the proxy rotation prediction task . 3.1.3 CONTRASTIVE LEARNING . Unlike augmentation prediction that learns discriminative representations to data augmentation , contrastive learning [ 23 ] learns representation by distinguishing different views ( e.g. , augmentations ) of itself from other data instances . Let φ ( x ) = normalize ( g ( x ) ) , i.e. , ‖φ ( x ) ‖= 1 . Following [ 54 ] , the proxy task loss of contrastive learning is written as : Lclr = − Ex , xi∼PX , A , A′ [ log exp ( 1τ φ ( A ( x ) ) > φ ( A′ ( x ) ) ) exp ( 1τ φ ( A ( x ) ) > φ ( A′ ( x ) ) ) + ∑M−1 i=1 exp ( 1τ φ ( A ( x ) ) > φ ( A ( xi ) ) ) ] ( 2 ) where A and A′ are identical but independent stochastic augmentation processes for two different views of x. Lclr regularizes representations of the same instance with different views ( A ( x ) , A′ ( x ) ) to be similar , while those of different instances ( A ( x ) , A′ ( x′ ) ) to be unlike . Class Collision and Uniformity for One-Class Classification . While contrastive representations have achieved state-of-the-art performance on visual recognition tasks [ 23 , 24 , 49 , 55 ] and have been theoretically proven to be effective for multi-class classification [ 26 , 56 ] , we argue that it could be problematic for one-class classification . First , a class collision [ 26 ] . The contrastive loss in Eq . ( 2 ) is minimized by maximizing the distance between representations of negative pairs ( x , xi ) , x 6=xi , even though they are from the same class when applied to the one-class classification . This seems to contradict to the idea of deep one-class classification [ 16 ] , which learns representations by minimizing the distance between representations with respect to the center : ming , f Ex‖g ◦ f ( x ) − c‖2 . Second , a uniformity of representations [ 24 ] . It is proved that the optimal solution for the denominator of Eq . ( 2 ) is perfect uniformity asM→∞ [ 24 ] , meaning that φ ( x ) follows a uniform distribution on the hypersphere . This is problematic since one can always find an inlier x∈X in the proximity to any outlier x′ /∈X on the hypersphere , as shown in Figure 2a . In contrast , with reduced uniformity as in Figure 2b , it is easier to isolate outliers from inliers . 1We abuse notation of g not only to denote the projection head , but also the representation g ◦ f ( · ) . 2For presentation clarity , we use the rotation as an example for augmentations . Note that one may use more geometric transformations , such as rotation , translation , or flip of an image , as in [ 20 , 21 , 22 ] . pull pushgg○f One-Class Contrastive Learning . First , to reduce the uniformity of representations , we propose to use a moderateM ( batch size ) . This is in contrast with previous suggestions to train with largeM for contrastive representations to be most effective on multi-class classification tasks [ 23 , 55 , 57 ] . The impact of batch size M for one-class classification will be discussed in Section 5.1 . In addition , we propose distribution augmentation3 for one-class contrastive learning . The idea is that , instead of modeling the training data distribution PX , we model the union of augmented training distribution P⋃ a a ( X ) , where a ( X ) = { a ( x ) |x∈X } . Note that augmentation a for augmenting distribution is disjoint from those for data augmentation A that generates views . Inspired by Golan and El-Yaniv [ 20 ] , we employ geometric transformations , such as rotation or horizontal flip , for distribution augmentation . For example , as in Figure 3 , x and rot90 ( x ) ( German shepherds in the top row ) are considered as two separate instances and therefore are encouraged to be distant in the representation space . Not only increasing the number of data instances to train on ( e.g. , distribution augmentation by rotating 90◦ , 180◦ , 270◦ increases the dataset by 4 times ) , but it also eases the uniformity of representations on the resulted hypersphere . A pictorial example is in Figure 2c , where thanks to augmented distribution , the inlier distribution may become more compact .

Summary: This paper proposes a framework for deep one-class classification (an example application being anomaly detection).  The basic idea is to combine self-supervised representation learning (eg through a proxy task such as rotation prediction or contrastive learning), with a classical approach to one-class classification, such as one-class SVM or KDE.  This is in contrast to existing methods for deep one-class classification that use simulated outliers to form a surrogate classification loss and then train end-to-end.  The paper further improves on the first stage of representation learning, by introducing modifications to contrastive learning to make it more appropriate for one-class classification.  The main insight is to introduce distribution augmentation, where geometric transformations of images, such as rotation, are treated as separate instances, to be separated from the original view.  This is motivated from the perspective of reducing uniformity of the inliers across the unit hypersphere, to allow for better separation from outliers.