Year: 2021

Abstract:
We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. The framework not only allows to learn better representations, but also permits building one-class classifiers that are faithful to the target task. We argue that classifiers inspired by the statistical perspective in generative or discriminative models are more effective than existing approaches, such as a normality score from a surrogate classifier. We thoroughly evaluate different self-supervised representation learning algorithms under the proposed framework for one-class classification. Moreover, we present a novel distribution-augmented contrastive learning that extends training distributions via data augmentation to obstruct the uniformity of contrastive representations. In experiments, we demonstrate state-of-the-art performance on visual domain oneclass classification benchmarks, including novelty and anomaly detection. Finally, we present visual explanations, confirming that the decision-making process of deep one-class classifiers is intuitive to humans. The code is available at https: //github.com/google-research/deep_representation_one_class.

heading: 1 INTRODUCTION
text: One-class classification aims to identify if an example belongs to the same distribution as the training data. There are several applications of one-class classification, such as anomaly detection or outlier detection, where we learn a classifier that distinguishes the anomaly/outlier data without access to them from the normal/inlier data accessible at training. This problem is common in various domains, such as manufacturing defect detection, financial fraud detection, etc.
Generative models, such as kernel density estimation (KDE), is popular for one-class classification [1, 2] as they model the distribution by assigning high density to the training data. At test time, low density examples are determined as outliers. Unfortunately, the curse of dimensionality hinders accurate density estimation in high dimensions [3]. Deep generative models (e.g. [4, 5, 6]), have demonstrated success in modeling high-dimensional data (e.g., images) and have been applied to anomaly detection [7, 8, 9, 10, 11]. However, learning deep generative models on raw inputs remains as challenging as they appear to assign high density to background pixels [10] or learn local pixel correlations [12]. A good representation might still be beneficial to those models.
Alternately, discriminative models like one-class SVM (OC-SVM) [13] or support vector data description (SVDD) [14] learn classifiers describing the support of one-class distributions to distinguish them from outliers. These methods are powerful when being with non-linear kernels. However, its performance is still limited by the quality of input data representations.
In either generative or discriminative approaches, the fundamental limitation of one-class classification centers on learning good high-level data representations. Following the success of deep learning [15], deep one-class classifications [16, 17, 18], which extend the discriminative one-class classification using trainable deep neural networks, have shown promising results compared to their kernel counterparts. However, a naive training of deep one-class classifiers leads to a degenerate solution that maps all data into a single representation, also known as “hypersphere collapse” [16]. Previous works circumvent such issues by constraining network architectures [16], autoencoder
∗Equal contribution.
pretraining [16, 17], surrogate multi-class classification on simulated outliers [19, 20, 21, 22] or injecting noise [18].
In this work, we present a two-stage framework for building deep one-class classifiers. As shown in Figure 1, in the first stage, we train a deep neural network to obtain a high-level data representation. In the second stage, we build a one-class classifier, such as OC-SVM or KDE, using representations from the first stage. Comparing to using surrogate losses [20, 21], our framework allows to build a classifier that is more faithful to one-class classification. Decoupling representation learning from classifier construction further opens up opportunities of using state-of-the-art representation learning methods, such as self-supervised contrastive learning [23]. While vanilla contrastive representations are less compatible with one-class classification as they are uniformly distributed on the hypersphere [24], we show that, with proper fixes, it provides representations achieving competitive one-class classification performance to previous state-of-the-arts. Furthermore, we propose a distribution-augmented contrastive learning, a novel variant of contrastive learning with distribution augmentation [25]. This is particularly effective in learning representations for one-class classification, as it reduces the class collision between examples from the same class [26] and uniformity [24]. Lastly, although representations are not optimized for one-class classification as in end-to-end trainable deep one-class classifiers [16], we demonstrate state-of-the-art performance on visual one-class classification benchmarks. We summarize our contributions as follows: • We present a two-stage framework for building deep one-class classifiers using unsupervised
and self-supervised representations followed by shallow one-class classifiers. • We systematically study representation learning methods for one-class classification, including
augmentation prediction, contrastive learning, and the proposed distribution-augmented contrastive learning method that extends training data distributions via data augmentation.
• We show that, with a good representation, both discriminative (OC-SVM) and generative (KDE) classifiers, while being competitive with each other, are better than surrogate classifiers based on the simulated outliers [20, 21].
• We achieve strong performance on visual one-class classification benchmarks, such as CIFAR10/100 [27], Fashion MNIST [28], Cat-vs-Dog [29], CelebA [30], and MVTec AD [31].
• We extensively study the one-class contrastive learning and the realistic evaluation of anomaly detection under unsupervised and semi-supervised settings. Finally, we present visual explanations of our deep one-class classifiers to better understand their decision making processes.

heading: 2 RELATED WORK
text: One-class classification [32] has broad applications, including fraud detection [33], spam filtering [34], medical diagnosis [35], manufacturing defect detection [31], to name a few. Due to the lack of granular semantic information for one-class data, learning from unlabeled data have been employed for one-class classification. Generative models, which model the density of training data distribution, are able to determine outlier when the sample shows low density [8, 35, 36]. These include simple methods such as kernel density estimation or mixture models [37], as well as advanced ones [4, 5, 6, 38, 39, 40, 41]. However, the density from generative models for high-dimensional data could be misleading [9, 12, 42, 43]. New detection mechanisms based on the typicality [44] or likelihood ratios [10] have been proposed to improve out-of-distribution detection.
Self-supervised learning is commonly used for learning representations from unlabeled data by solving proxy tasks, such as jigsaw puzzle [45], rotation prediction [46], clustering [47], instance discrimination [48] and contrastive learning [23, 49, 50]. The learned representations are then used for multi-class classification, or transfer learning, all of which require labeled data for downstream tasks. They have also been extended to one-class classification. For example, contrastive learning is adopted to improve the out-of-distribution detection under multi-class setting [51], whereas our work focuses on learning from a single class of examples, leading to propose a novel distributionaugmented contrastive learning. Notably, learning to predict geometric transformations [20, 21, 22] extends the rotation prediction [46] to using more geometric transformations as prediction targets. Unlike typical applications of self-supervised learning where the classifier or projection head [23] are discarded after training, the geometric transformation classifier is used as a surrogate for oneclass classification. As in Section 4.1, however, the surrogate classifier optimized for the selfsupervised proxy task is suboptimal for one-class classification. We show that replacing it with
simple one-class classifiers consistently improve the performance. Furthermore, we propose strategies for better representation learning for both augmentation prediction and contrastive learning.
Distribution-augmented contrastive learning is concurrently developed in [52] as a part of their multi-task ensemble model. While sharing a similar technical formulation, we motivate from fixing the uniformity of contrastive representations. We note that our study not only focuses on representation learning, but also on the importance of detection algorithms, which is under explored before.

heading: 3 A TWO-STAGE FRAMEWORK FOR DEEP ONE-CLASS CLASSIFICATION
text: In Section 3.1, we review self-supervised representation learning algorithms, discuss how they connect to existing one-class classification methods, raise issues of state-of-the-art contrastive representation learning [23] for one-class classification, and propose ways to resolve these issues. Then, in Section 3.2, we study how to leverage the learned representations for one-class classification.

heading: 3.1 LEARNING REPRESENTATIONS FOR ONE-CLASS CLASSIFICATION
text: LetA be the stochastic data augmentation process, which is composed of resize and crop, horizontal flip, color jittering, gray-scale and gaussian blur, following [23], for image data. As in Figure 1, selfsupervised learning methods consist of a feature extractor f parameterized by deep neural networks and the proxy loss L. Optionally, f is further processed with projection head g at training, which is then used to compute the proxy loss. Unless otherwise stated, normalize(f), f/‖f‖2 is used as a representation at test time. Below, we discuss details of self-supervised learning methods.

heading: 3.1.1 EXTRACTING RICHER REPRESENTATION BY LEARNING WITH PROJECTION HEAD
text: While the efficacy of projection head has been confirmed for contrastive learning [23] or BYOL [53], it is not widely adopted for other types of self-supervised learning, including rotation prediction [46]. On the other hand, Gidaris et al. [46] show that the lower-layer representation often perform better for downstream tasks as the last layer directly involved in optimizing the proxy loss becomes overly discriminative to the proxy task, while losing useful information of the data.
Inspired by these observations, we adopt the projection head for augmentation prediction training as well. As in Figure 1a, we extend the network structure as g ◦ f , where g is the projection head used to compute proxy losses and f outputs representations used for the downstream task. Note that using an identity head g(x) =x recovers the network structure of previous works [20, 21, 46].

heading: 3.1.2 AUGMENTATION PREDICTION
text: One way of representation learning is to learn by discriminating augmentations applied to the data. For example, the rotation prediction [46] learns deep representations by predicting the degree of
rotation augmentations. The training objective of the rotation prediction task is given as follows: Lrot = Ex∼PX ,A [ CrossEntropy ( y, pq◦g ( y|rot90 (A(x), y)) )] (1)
where y ∈{0, 1, 2, 3} is a prediction target representing the rotation degree, and rot90(x, y) rotates an input x by 90 degree y times. We denote the classifier pq◦g(y|x) as pq(y|g(x)) ∝ exp(q◦g(x))[y] containing the representation g1 and a linear layer q with 4 output units for rotation degrees.
Application to One-Class Classification. Although not trained to do so, the likelihood of learned rotation classifiers2 pq(y= 0|g(x)) is shown to well approximate the normality score and has been used for one-class classification [20, 21, 22]. A plausible explanation is via outlier exposure [19], where the classifier learns a decision boundary distinguishing original images from simulated outliers by image rotation. However, it assumes inlier images are not rotated, and the classifier may not generalize to one-class classification task if it overfits to the proxy rotation prediction task.

heading: 3.1.3 CONTRASTIVE LEARNING
text: Unlike augmentation prediction that learns discriminative representations to data augmentation, contrastive learning [23] learns representation by distinguishing different views (e.g., augmentations) of itself from other data instances. Let φ(x) = normalize(g(x)), i.e., ‖φ(x)‖= 1. Following [54], the proxy task loss of contrastive learning is written as:
Lclr = − Ex,xi∼PX ,A,A′ [ log exp( 1τ φ(A(x))
>φ(A′(x))) exp( 1τ φ(A(x))>φ(A′(x)))+ ∑M−1 i=1 exp( 1τ φ(A(x))>φ(A(xi)))
] (2)
where A and A′ are identical but independent stochastic augmentation processes for two different views of x. Lclr regularizes representations of the same instance with different views (A(x),A′(x)) to be similar, while those of different instances (A(x),A′(x′)) to be unlike. Class Collision and Uniformity for One-Class Classification. While contrastive representations have achieved state-of-the-art performance on visual recognition tasks [23, 24, 49, 55] and have been theoretically proven to be effective for multi-class classification [26, 56], we argue that it could be problematic for one-class classification.
First, a class collision [26]. The contrastive loss in Eq. (2) is minimized by maximizing the distance between representations of negative pairs (x, xi), x 6=xi, even though they are from the same class when applied to the one-class classification. This seems to contradict to the idea of deep one-class classification [16], which learns representations by minimizing the distance between representations with respect to the center: ming,f Ex‖g ◦ f(x)− c‖2. Second, a uniformity of representations [24]. It is proved that the optimal solution for the denominator of Eq. (2) is perfect uniformity asM→∞ [24], meaning that φ(x) follows a uniform distribution on the hypersphere. This is problematic since one can always find an inlier x∈X in the proximity to any outlier x′ /∈X on the hypersphere, as shown in Figure 2a. In contrast, with reduced uniformity as in Figure 2b, it is easier to isolate outliers from inliers.
1We abuse notation of g not only to denote the projection head, but also the representation g ◦ f(·). 2For presentation clarity, we use the rotation as an example for augmentations. Note that one may use more
geometric transformations, such as rotation, translation, or flip of an image, as in [20, 21, 22].
pull pushgg○f
One-Class Contrastive Learning. First, to reduce the uniformity of representations, we propose to use a moderateM (batch size). This is in contrast with previous suggestions to train with largeM for contrastive representations to be most effective on multi-class classification tasks [23, 55, 57]. The impact of batch size M for one-class classification will be discussed in Section 5.1.
In addition, we propose distribution augmentation3 for one-class contrastive learning. The idea is that, instead of modeling the training data distribution PX , we model the union of augmented training distribution P⋃
a a(X ), where a(X ) = {a(x)|x∈X}. Note that augmentation a for augmenting distribution is disjoint from those for data augmentation A that generates views. Inspired by Golan and El-Yaniv [20], we employ geometric transformations, such as rotation or horizontal flip, for distribution augmentation. For example, as in Figure 3, x and rot90(x) (German shepherds in the top row) are considered as two separate instances and therefore are encouraged to be distant in the representation space. Not only increasing the number of data instances to train on (e.g., distribution augmentation by rotating 90◦, 180◦, 270◦ increases the dataset by 4 times), but it also eases the uniformity of representations on the resulted hypersphere. A pictorial example is in Figure 2c, where thanks to augmented distribution, the inlier distribution may become more compact.

heading: 3.2 BUILDING DEEP ONE-CLASS CLASSIFIERS WITH LEARNED REPRESENTATIONS
text: We present a two-stage framework for deep one-class classification that builds one-class classifiers on learned representations as in Figure 1. Compared to end-to-end training [16, 20, 21, 22], our framework provides flexibility in using various representations as the classifier is not bound to representation learning. It also allows the classifier consistent with one-class classification objective.
To construct a classifier, we revisit an old wisdom which considers the full spectrum of the distribution of the learned data representation. For generative approaches, we propose to use nonparametric kernel density estimation (KDE) to estimate densities from learned representations. For discriminative approaches, we train one-class SVMs [13]. Both methods work as a black-box and in experiment we use the default training setting except the kernel width where we reduce by 10 times than default. We provide detailed classifier formulations in Appendix A.1.

heading: 3.2.1 GRADIENT-BASED EXPLANATION OF DEEP ONE-CLASS CLASSIFIER
text: Explaining the decision making process helps users to trust deep learning models. There have been efforts to visually explain the reason for model decisions of multi-class classifiers [58, 59, 60, 61, 62, 63, 64] using the gradients computed from the classifier. In this work, we introduce a gradient-based visual explanation of one-class classification that works for any deep representations. To construct an end-to-end differentiable decision function, we employ a KDE detector, whose formulation is in Appendix A.1, built on top of any differentiable deep representations: ∂KDE(f(x))∂x = ∂KDE(f(x)) ∂f(x) ∂f(x) ∂x .

heading: 4 EXPERIMENTS
text: Following [20], we evaluate on one-class classification benchmarks, including CIFAR-10, CIFAR1004 [27], Fashion-MNIST [28], and Cat-vs-Dog [29]. Images from one class are given as inlier and those from remaining classes are given as outlier. We further propose a new protocol using CelebA eyeglasses dataset [30], where face images with eyeglasses are denoted as an outlier. It is
3We note that the distribution augmentation has been applied to generative modeling [25] as a way of improving regularization and generalization via multi-task learning.
420 super class labels are used for CIFAR-100 experiments [20].
more challenging since the difference between in and outlier samples is finer-grained. Last, in addition to semantic anomaly detection as aforementioned, we consider the defect detection benchmark MVTec [31] in Section 4.2.
We evaluate the performance of (1) representations trained with unsupervised and self-supervised learning methods, including denoising autoencoder [65], rotation prediction [46], contrastive learning [23, 48], and (2) using different one-class classifiers, such as OC-SVM or KDE. We use rotation augmentations for distribution-augmented contrastive learning, denoted as Contrastive (DA). We train a ResNet-18 [66] for feature extractor f and a stack of linear, batch normalization, and ReLU, for MLP projection head g. More experiment details can be found in Appendix A.4.

heading: 4.1 MAIN RESULTS
text: We report the mean and standard deviation of AUCs averaged over classes over 5 runs in Table 2. The mean of 5 datasets is weighted by the number of classes for each dataset. Besides those using self-supervised representations, we provide results using ImageNet pretrained ResNet-50 to highlight the importance of learning representations from in-domain distributions.
ImageNet pretrained ResNet-50 achieves the best performance of 84.0 mean AUC over 5 datasets. Compared to representations learned with denoising objective, it works particularly well on datasets such as CIFAR-100, cat-vs-dog, and CelebA, which we attribute it to the subset of ImaegNet classes is closely related to the classes of these datasets.
Similar to the findings from [20, 21], we observe significant performance gains with self-supervised learning. Moreover, while RotNet [20], an end-to-end trained classifier using rotation prediction, achieves 83.1 AUC, the RotNet representation evaluated with the KDE detector achieves 86.6, emphasizing the importance of a proper detector in the second stage. Finally, the quality of RotNet representation improves when trained with the MLP projection head, resulting in 88.2 AUC.
The representations learned with vanilla contrastive loss achieve 86.9 AUC with OC-SVM, which under-perform those trained with rotation prediction loss (88.2). The distribution-augmented contrastive loss achieves the highest mean AUC of 89.9 among all methods, demonstrating performance gains on all datasets by a large margin upon its vanilla counterparts.
Comparison to Previous Works. We make comparisons to previous works in Table 3. While some comparisons may not be fair as different works use different implementations, we note that our implementation is based on the common choices of network (e.g., ResNet-18) and optimizer (e.g., momentum SGD) for image classification. We advance the previous state-of-the-art on one-class
8 16 32 64 128 256 512
batch size
10 2
10 1
100
101
M M
D
Contrastive g f Contrastive (DA) g f
(a) MMDs with various batch sizes.
8 16 32 64 128 256 512
batch size
30
40
50
60
70
80
90
AU C
Contrastive f Contrastive g f Contrastive (DA) f Contrastive (DA) g f
(b) AUCs with various batch sizes.
0 1 2 3 4 5 6 7 8 Depth of MLP head
30
40
50
60
70
80
90
AU C
Contrastive f Contrastive g f Contrastive (DA) f Contrastive (DA) g f
(c) AUCs with various MLP depths.
Figure 4: Ablation study of contrastive representations trained with different batch sizes. We use (a) the MMD distance between the representations and the data sampled from uniform distributions to measure uniformity. Small MMD distance means being similar to uniform distribution. We also report (b) one-class classification performance in AUCs evaluated by kernel OC-SVMs and (c) the performance with MLP heads of different depths.
classification benchmarks by a large margin without test-time augmentation nor ensemble of models. We further improve the performance with model ensemble, which we report in Appendix A.2.2.

heading: 4.2 EXPERIMENTS ON MVTEC ANOMALY (DEFECT) DETECTION
text: Finally, we evaluate our proposed framework on MVTec [31] defect detection dataset, which is comprised of 15 different categories, including objects and textures. Instead of learning representations from the entire image, we learn representations of 32×32 patches. At test time, we compute the normality scores of 32×32 patches densely extracted from 256×256 images with stride of 4. For image-level detection, we apply spatial max-pooling to obtain a single image-level score. For localization, we upsample the spatial score map with Gaussian kernel [67]. Please see Appendix B for more implementation details and experimental results.
We report in Table 4 the detection and localization AUCs. We verify the similar trend to previous experiments. For example, while using the same representation, RotNet with KDE (RotNet†) significantly outperform the RotNet using built-in rotation classifier. Moreover, distribution-augmented contrastive learning (with rotation) improves the performance of vanilla contrastive learning. Due to a space constraint, we show localization results in Appendix B.4.

heading: 5 ANALYSIS AND ABLATION STUDY
text: In Section 5.1, we analyze behaviors of one-class contrastive representations and in Section 5.4, we report visual explanations of various deep one-class classifiers. Due to a space constraint, more studies, including an in-depth study on distribution-augmented contrastive representations and data efficiency of self-supervised learning for one-class classification, are in Appendix A.2.

heading: 5.1 UNIFORMITY, BATCH SIZE AND DISTRIBUTION AUGMENTATION
text: [23, 49, 55, 57] have shown substantial improvement on contrastive representations evaluated on multi-class classification using very large batch sizes, which results in uniformity. However, uniformity [24] and class collision [26] can be an issue for one-class classification as discussed in Section 3.1.3. Here we investigate the relations between performance and uniformity, and how we can resolve the issue via batch size, MLP head and distribution-augmented contrastive learning.
We measure the uniformity via MMD distance [68] between the learned representation and samples from uniform distributions on hyperspheres. Smaller MMD distance implies the distribution of representation is closer to uniform distributions. We train models with various batch sizes {23, . . . , 29}. We report in Figure 4 the average and standard deviation over 5 runs on CIFAR-10 validation set.
Distribution Augmentation. In Figure 4a, the representations from the projection head g◦f trained by standard contrastive learning (Contrastive g ◦ f ) are closer to be uniformly distributed when we increase the batch size as proven by [24]. Therefore, in one-class classification, the nearly uniformly distributed representation from contrastive learning is only slightly better than random guess (50 AUC) as in Figure 4b. In contrast, with distribution augmentations, the learned representations (Contrastive (DA) g ◦ f ) are less uniformly distributed and result in a significant performance gain. Batch Size. Following the discussion, large batch sizes result in nearly uniformly distributed representations (g ◦ f ), which is harmful to the one-class classification. On the other hand, small batch sizes (≤16), though less uniform, hinders us learning useful representations via information maximization [69]. As in Figure 4b, there is a trade-off of batch size for one-class classification, and we find batch size of 32 results in the best one-class classification performance. Further analysis showing the positive correlation between the uniformity measured by the MMD and the one-class classification performance is in Appendix A.2.2 and Figure 8.
MLP Projection Head. Similarly to [23], we find that f , an input to the projection head, performs better on the downstream task than g ◦ f . As in Figure 4c, the performance of f improves with deeper MLP head, while g ◦ f degrades, as it overfits to the proxy task. Lastly, we emphasize that all these fixes contribute to an improvement of contrastive representations for one-class classification. As in Figure 4 AUC drops when any of these components are missing.

heading: 5.2 ANALYSIS ON DIFFERENT DISTRIBUTION AUGMENTATIONS
text: The choice of distributions affects the performance. The ablation study using horizontal flip (hflip) and rotation augmentations is reported in Figure 7. Note that hflip is used only to augment distribution in this experiment. Interestingly, simply adding hflip improves the AUC to 90.7. This suggests a different insight from [52] who augments distribution as a means to outlier exposure [19]. Although we report the numbers with rotation augmentations in Table 2, with hflip, rot90, rot90+hflip as augmented distributions, we achieve the best mean AUC, 93.7, on CIFAR-10, without any test-time augmentation. Additional study on distribution augmentation follows in Appendix A.2.2.

heading: 5.3 APPLICATIONS TO UNSUPERVISED AND SEMI-SUPERVISED ANOMALY DETECTION
text: We conduct experiments for unsupervised anomaly detection, where the training set may contain a few outlier data.5 We study two settings: 1) Unsupervised settings for both learning representation and building detector, and 2) unsupervised setting for learning representation, but building detector with a small amount (as few as 50 data instances) of one-class data only. For both settings, we vary the outlier ratio in the training set from 0.5% to 10%. We show results in Figure 5. As in Figure 5a, we observe the decrease in performance when increasing the outlier ratio as expected. Rotation prediction is slightly more robust than contrastive learning for high outlier ratio. On the other hand, when classifier is built with clean one-class data, contrastive representations performs
5We follow the definition of [70] to distinguish unsupervised and semi-supervised settings of anomaly detection. Please see Appendix A.3 for additional description on their definitions.
better. Interestingly, contrastive learning benefits from outlier data, as it naturally learn to distinguish inlier and outlier. Due to space constraint, we provide more results and analysis in Appendix A.3.

heading: 5.4 VISUAL EXPLANATION OF DEEP ONE-CLASS CLASSIFIERS
text: We investigate the decision making process of our deep one-class classifiers using the tools described in Section 3.2.1. Specifically, we inspect by highlighting the most influential regions using two popular visual explanation algorithms, namely, integrated gradients (IG) [62] and GradCAM [61], for distribution-augmented contrastive representation, as well as RotNet and DAE on images from cat-vs-dog and CelebA eyeglasses datasets. For RotNet, we test using both a rotation classifier and KDE (RotNet∗ in Figure 6) to compute gradients.
As in Figure 6, the proposed visual explanation method based on the KDE one-class classifier permits highlighting human-intuitive, meaningful regions in the images, such as dog faces or eyeglasses instead of spurious background regions (Figure 6b). On the other hand, even though the classification AUC is not too worse (86.1 AUC on cat-vs-dog as opposed to 89.6 for ours), the visual explanation based on the rotation classifier (Figure 6c) suggests that the decision may be made sometimes based on the spurious features (e.g., human face in the background). We present more examples for visual explanation in Appendix A.6.

heading: 6 CONCLUSION
text: Inspired by an old wisdom of learning representations followed by building classifiers, we present a two-stage framework for deep one-class classification. We emphasize the importance of decoupling building classifiers from learning representations, which allows classifier to be consistent with the target task, one-class classification. Moreover, it permits applications of various self-supervised representation learning methods, including contrastive learning [23] with proper fixes, for one-class classification, achieving strong performance on visual one-class classification benchmarks. Finally, we exhibit visual explanation capability of our two-stage self-supervised deep one-class classifiers.

heading: A APPENDIX

heading: A.1 FORMULATIONS OF ONE-CLASS CLASSIFIERS
text: For completeness, we provide formulations of one-class classifiers, such as one-class support vector machine (OC-SVM) [13] and kernel density estimator, used in our work.
One-Class Support Vector Machine solves the following optimization problem to find support vectors describing the boundary of one-class distribution:
min w,ρ,ξi
‖w‖2
2 +
1
νn n∑ i=1 ξi − ρ (3)
subject to w>fi ≥ ρ− ξi, ξi ≥ 0,∀i = 1, · · · , n (4) where fi = f(xi) is a feature map. The decision score is given as follows:
s(x) = n∑ i=1 αik(xi, x)− ρ (5)
with coefficients αi> 0 for support vectors. Linear or RBF (kγ(x, y) = exp ( −γ‖x− y‖2 ) ) kernels are used for experiments.
Kernel Density Estimation is a nonparametric density estimation algorithm. The normality score of KDE with RBF kernel parameter γ is written as follows:
KDEγ(x) = 1 γ log [∑
y
exp ( −γ‖x− y‖2 ) ] (6)

heading: A.2 ADDITIONAL ABLATION STUDY

heading: A.2.1 WELL BEHAVED CONTRASTIVE REPRESENTATION
text: Linear Separability. A common belief is the good representations are linearly separable with respect to some underlying labels, which is supported by several empirical success [23]. The linear classifiers are proven to be nearly optimal [56] when we learn representations on data with all possible labels along with some additional assumptions. In one-class classification, although we violate the assumptions that we only train on one class of data, we are interested in the linear separability of the data to understand the characteristics of the learned embedding. To this end, in addition to training RBF kernel OC-SVM, we train OC-SVM with linear kernels on the learned embedding. We note that it is atypical to use linear OC-SVM, which is usually considered to be suboptimal for describing decision boundaries of one-class classification problems. However, with good representations, linear classifiers show good performance on one-class classification as shown in Table 5, and are better than existing works [20] in Table 2. Lastly, we note that OC-SVM with nonlinear RBF kernel is still better than linear models in Table 5. Similar observations are found in other problems such as regression [71] and generative models [72].
Parametric Models. In addition to studying from the perspective of discriminative model, we also dive into the generative model side. Instead of using the nonparametric KDE for density estimation, we use a parametric model, single multivariate Gaussian, whose density is defined as
p(x) ∝ det ( Σ )− 12 exp{− 1
2 (f(x)− µ)>Σ−1(f(x)− µ)
} . (7)
A shown in Table 5, the Gaussian density estimation (GDE) model shows competitive performance as KDE. It suggests the learned representations from distribution-augmented contrastive learning is compact as our expectation, which can be well approximated by a simple parametric model with single Gaussian. Compared with nonparametric methods, although parametric models have strong assumptions on the underlying data distributions, using parametric model is more data efficient if the assumption holds6. In addition, parametric models have huge advantage in computation during
6The error convergence rate is O(n−1/2), while that of nonparametric KDE is O(n−1/d) [3].
testing. Therefore, there is trade-off between assumptions of the data as well as model flexibility and computational efficiency. We note that the single Gaussian parametric model may not be a good alternative of KDE universally. A candidate with good trade-off is Gaussian Mixture Models, which can be treated as a middle ground of nonparametric KDE and parametric single Gaussian model. We leave the study for future works.

heading: A.2.2 ANALYSIS ON DISTRIBUTION AUGMENTED CONTRASTIVE REPRESENTATIONS
text: Relation to Outlier Exposure [19]. Distribution augmentation shares a similarity to outlier exposure [19] in that both methods introduce new data distributions for training. However, outlier exposure requires a stronger assumption on the data distribution that introduced outlier should not overlap with inlier distribution, while such an assumption is not required for distribution augmentation. For example, let’s consider rotation prediction as an instance of outlier exposure. When training the model on randomly-rotated CIFAR-10, where we randomly rotate images of CIFAR-10, the performance of rotation prediction representation drops to 60.7 AUC, which is slightly better than a random guess. On the other hand, rotation-augmented contrastive representation is not affected by random rotation and achieves 92.4 AUC.
Ensemble of Contrastive Representations. We note that most previous methods have demonstrated improved performance via extensive test-time data augmentation [20, 21, 22, 52]. While already achieving state-of-the-art one-class classification performance without test-time data augmentation, we observe marginal improvement using test time data augmentation. Instead, we find that an ensemble of classifiers built on different representations significantly improve the performance. In Table 6, we report the performance of distribution-augmented contrastive representations trained with different sets of augmented distributions on CIFAR-10. We observe that ensemble of 5 models trained with different seeds consistently improves the performance. Moreover, not only we improve one-class classification AUCs when aggregating scores across models trained with different distribution augmentations, we also observe lower standard deviation across different seeds. Finally, when ensemble over 5×5 = 25 models, we achieve 94.6 AUC.
Correlation between Uniformity and One-class Classification. We present scatter plots in Figure 8 showing the positive correlation between the uniformity (log(MMD)) and the one-class classification AUCs. Specifically, we compute the MMD using g ◦ f , an output of MLP projection head that are used to optimize the contrastive loss at training. We evaluate AUCs using both g ◦ f (left of Figure 8) and f (right of Figure 8). Data points in the plots are obtained from vanilla and DistAug contrastive models trained with various batch sizes, including 32, 64, 128, 256 and 512. For each configuration, we train 5 models with different random seeds.
From the left plot of Figure 8, we observe a strong positive correlation between MMD and AUC, suggesting that the more uniformly distributed (i.e., lower MMD), the worse the one-class classi-
fication performance is (i.e., low AUCs). We also compute the Pearson’s correlation coefficient, which results in 0.914. We also investigate the correlation between the MMD using g ◦ f and the AUC using f , as shown in the right plot of Figure 8. We observe highly positive correlation of 0.774 Pearsons correlation coefficient between the MMD using g ◦ f and the AUC using f . Overall, our results suggest that reducing uniformity of g◦f improves the one-class classification accuracy tested with both g ◦ f and f representations.

heading: A.2.3 DATA EFFICIENCY OF SELF-SUPERVISED REPRESENTATION LEARNING
text: While previous works on self-supervised learning have demonstrated the effectiveness on learning from large-scale unlabeled data, not much has shown for the data efficiency of these methods. Unlike multi-class classification tasks where the amount of data could scale multiplicative with the number of classes, data efficiency of representation learning becomes of particular interest for one-class classification as it is hard to collect large-scale data even without an annotation.
We present one-class classification AUCs of representations with various training data sizes, along with two baseline representations such as ResNet-18 using random weights or ImageNet-pretrained ResNet-50 in Figure 9. Note that we vary the training set sizes at representation learning phase only, but use a fixed amount (5000) to train classifiers for fair evaluation of the representation quality. We find that even with 50 examples, classifiers benefit from self-supervised learning when comparing against raw or deep features with random weights. The proposed distribution-augmented contrastive loss could match the performance of ImageNet-pretrained ResNet-50 with only 100 examples, while rotation prediction loss and vanilla contrastive loss require 250 and 1000 examples, respectively.

heading: A.3 UNSUPERVISED AND SEMI-SUPERVISED ANOMALY DETECTION
text: In this section, we conduct experiments for unsupervised anomaly detection. Note that one-class classification assumes access to training data drawn entirely from one-class, inlier distribution and is often referred to semi-supervised anomaly detection [70] as it requires human effort to filter out training data from outlier distribution. On the other hand, unsupervised anomaly detection assumes to include training data both from inlier and outlier distributions without knowing their respective labels. In other words, unsupervised anomaly detection may be viewed as an one-class classification with noisy data.
Here, we are interested in analyzing the impact of label noise (e.g., outlier examples are given as inlier) on one-class classification methods. To this end, we conduct experiments under unsupervised settings that includes different ratios, such as 0.5%, 1%, 2%, 5%, or 10%, of outlier examples in the train set without their labels. Note that the total amount of training examples for different settings remain unchanged. We show results in Figure 10 of 4 models: rotation prediction without
(“RotNet”) and with (“Rotation Prediction”) MLP projection head, contrastive (“Contrastive”) and distribution-augmented contrastive (“Contrastive (DA)”) learning. We also report thee performance evaluated with different classifiers, including OC-SVM with RBF (Figure 10a) and linear (Figure 10c) kernels, and Gaussian Density Estimator (GDE, Figure 10b). For all models, we observe performance degradation of deep one-class classifiers trained with outlier examples as expected. Rotation prediction has shown more robust when outlier ratio is high (5 or 10%). While showing stronger performance under one-class setting, kernel-based classifiers (OC-SVM with RBF kernel, KDE) have shown less robust than parameteric (GDE) or linear classifiers under high outlier ratios, as kernel-based methods focus more on the local neighborhood structures to determine a decision boundary.
In addition, we conduct a study under another semi-supervised setting, where we are given a small amount of labeled inlier data and a large amount of unlabeled training data composed of both inlier and outlier examples. Note that this is more realistic scenario for anomaly detection problems since it is easier to obtain some portions of labeled inlier examples than outlier examples. To demonstrate
its effectiveness, we apply our proposed framework without any modification by first training representations on unlabeled training data and then building classifiers on these representations using small amount of inlier data. Results are shown in Figure 11. Interestingly, we observe consistent improvement in classification performance for contrastive representations trained on data with higher proportions of outlier examples when classifier is trained on a pure one-class data. Plausible explanation is that the model learns better contrastive representations when trained with both inlier and outlier as it naturally learn to distinguish inlier and outlier. On the other hand, representations from rotation prediction still show performance degradation as it learns to classify inlier and outlier into the same category.

heading: A.4 DETAILS OF EXPERIMENTAL SETTING
text: We resize images into 64×64 for cat-vs-dog and CelebA datasets and 32×32 for the rest. Unless otherwise stated, models are trained for 2048 epochs with momentum (0.9) SGD and a single cycle of cosine learning rate decay [73]. L2 weight regularization with coefficient of 0.0003 is applied. We use scikit-learn [74] implementation of OC-SVMs with default value of ν. We use γ= 10/|f |·Var(f), which is 10 times larger than the default value for kernel OC-SVM. Same value of γ is used for KDE. We use scikit-learn implementation of Gaussian mixture model using a single component for Gaussian density estimator (GDE). No hyperparameters are tuned for GDE. Finally, all experiments are conducted using TensorFlow [75].
Other hyperparameters, such as learning rate or the MLP projection head depth, are cross-validated using small labeled data. While this may violate the assumption of one-class classification, supervised model selection is inevitable for deep learning models as their behaviors may be largely dependent on hyperparameters. To this end, we use 10% of inlier (500) and the same number of outlier examples of CIFAR-10 for hyperparameter selection, and use the same set of hyperparameters to test methods on other datasets, which could demonstrate the algorithm robustness. Learning rate
∈{0.1, 0.03, 0.01, 0.003, 0.001} and the depth ∈{0, · · · , 8} of an MLP projection head are tuned for all methods. In addition, the temperature τ∈{1, 0.5, 0.2, 0.1} and the batch size ∈{2n, n = 3, ..., 9} of contrastive loss are tuned. To this end, we train all models across all datasets using the same hyperparameter configurations, such as learning rate of 0.01, projection head of depth 8 ([512×8, 128]), temperature τ of 0.2, or batch size of 32.

heading: A.5 PER-CLASS AUCS

heading: A.6 MORE EXAMPLES FOR VISUAL EXPLANATION OF DEEP ONE-CLASS CLASSIFIERS

heading: B EXPERIMENTS ON MVTEC ANOMALY DETECTION DATASET

heading: B.1 DATASET DESCRIPTION
text: There are 15 different categories in MVTec anomaly detection dataset [31], where 10 are object (e.g., bottle, cable, transistor) and 5 are texture (e.g., grid, leather) categories. Each category comes with the training set containing∼241 images per category from normal data distribution and the test set containing ∼115 images per category from both normal and defective data distributions. The dataset also provides pixel-accurate annotation for defective regions, allowing to measure the defect localization performance. Images are of high-resolution, whose side length is as long as 1024.

heading: B.2 EXPERIMENTAL SETTING
text: First, we resize all images into 256×256 in our experiments. One challenge of learning representations from scratch on MVTec dataset is that the amount of training set is too small (∼241 images per category). As in Figure 9, self-supervised representation learning methods benefit from the large amount of training data and the performance degrades when the amount of training data is limited. Instead of learning the holistic image representation, we learn a patch representation of size 32×32 via the proposed self-supervised representation learning methods. Similarly to our previous experiments, we train ResNet-18 from scratch.
For evaluation, we compute both image-level detection and localization AUCs. We densely extract embeddings f from image patches with the stride of 4 and compute anomaly scores with KDE at each patch location, which results in a ( 256−2244 + 1)×( 256−224 4 + 1) = 57×57 anomaly score map. For image-level detection, we apply spatial max-pooling to obtain a single score. For localization, we upsample 57×57 into 256×256 with Gaussian kernel7 [67] and compare with the annotation.

heading: B.3 EXPERIMENTAL RESULTS
text: We report quantitative results in Table 12. Specifically, we report detection and localization AUCs averaged over object, texture, and all categories. Likewise, we run experiments 5 times with different random seeds and report the mean and standard deviation. We report the performance of RotNet (using rotation classifier, as in [20, 21]), the same representation but with KDE detectors as proposed, and RotNet trained with an MLP head and evaluated with KDE detector as proposed. In addition, we evaluate the performance of contrastive representations without (Vanilla) and with distribution augmentation (DistAug) as proposed. For comparison, we include the results of CAVGA Ru [76], which is also evaluated under the same setting. Below we highlight some points from our results:
1. The proposed two-stage framework, which trains a RotNet and constructs an one-class classifier, clearly outperforms an end-to-end framework, which trains the RotNet and uses a built-in rotation classifier [20, 21].
2. The proposed modification of RotNet with MLP projection head further improves the representation of RotNet for one-class classification.
7To be more specific, we apply transposed convolution to 57×57 score map using a single convolution kernel of size 32×32 whose weights are determined by Gaussian distribution and stride of 4.
3. The proposed distribution augmentation (rotation) improves the performance of contrastive representations.
4. We improve the localization AUC upon [76] both in detection and localization, both on object and texture categories.
5. Both RotNet and contrastive representations are effective for “object” categories, but not as much for “texture” categories. As reviewed by [77], semantic and texture anomaly detection problems are different and may require different tools to solve. The methods relying on geometric transformations are effective in learning representations of visual objects [46] and thus more effective for semantic anomaly detection, but less effective for texture anomaly detection.

heading: B.4 QUALITATIVE RESULTS ON LOCALIZATION

Review summaries:
This paper proposes an anomaly detection approach that has two stages: a first stage for learning a feature representation and a second stage to train either a one-class classifier based on OC-SVM or KDE.  The main contribution of the paper is the feature representation learning that relies on contrastive learning to optimise a self-supervised loss function which minimises the distance of the samples from the same image augmented with different data augmentation functions and maximises the distance of samples from different images augmented with the same augmentation functions.  The data augmentation functions used were horizontal flip and rotation (0,90,180,270).  Results on the public datasets CIFAR-10, CIFAR-100, Fashion MNIST, and Cat-vs-Dog show that the proposed method has better anomaly detection (measured with AUC) than the state of the art.  The paper also displays qualitative anomaly detection results and an ablation study that shows: a) how close to uniform distribution (on hypersphere) the feature representations are as a function of batch size, and b) how AUC is affected with batch size and depth of MLP project heads.

This paper proposes a framework for deep one-class classification (an example application being anomaly detection).  The basic idea is to combine self-supervised representation learning (eg through a proxy task such as rotation prediction or contrastive learning), with a classical approach to one-class classification, such as one-class SVM or KDE.  This is in contrast to existing methods for deep one-class classification that use simulated outliers to form a surrogate classification loss and then train end-to-end.  The paper further improves on the first stage of representation learning, by introducing modifications to contrastive learning to make it more appropriate for one-class classification.  The main insight is to introduce distribution augmentation, where geometric transformations of images, such as rotation, are treated as separate instances, to be separated from the original view.  This is motivated from the perspective of reducing uniformity of the inliers across the unit hypersphere, to allow for better separation from outliers.