Learning and Evaluating Representations for Deep One-Class Classification
Authors: Kihyuk Sohn
Chun-Liang Li
Jinsung Yoon
Minho Jin
Tomas Pfister
Venue: ICLR 2021 Poster (2021)
Keywords: deep one-class classification
self-supervised learning

Abstract: We present a two-stage framework for deep one-class classification. We first learn self-supervised  representations from one-class data, and then build one-class classifiers on learned representations. The framework not only allows to learn better representations, but also permits building one-class classifiers that are faithful to the target task. We argue that classifiers inspired by the statistical perspective in generative or discriminative models are more effective than existing approaches, such as a normality score from a surrogate classifier. We thoroughly evaluate different self-supervised representation learning algorithms under the proposed framework for one-class classification. Moreover, we present a novel distribution-augmented contrastive learning that extends training distributions via data augmentation to obstruct the uniformity of contrastive representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks, including novelty and anomaly detection. Finally, we present visual explanations, confirming that the decision-making process of deep one-class classifiers is intuitive to humans. The code is available at https://github.com/google-research/deep_representation_one_class.


Published as a conference paper at ICLR 2021
LEARNING AND EVALUATING REPRESENTATIONS FOR DEEP ONE-CLASS CLASSIFICATION
Kihyuk Sohn; Chun-Liang Li", Jinsung Yoon, Minho Jín & Tomas Pfister Google Cloud AI (kihyuks, chunliang, jinsungyoon, iin, tegpheoooei..t.sssseeeoo....e.. com
ABSTRACT
We present two-stage framework for deep one-class classification. We first learn self--sueervied representations from one-class data, and then build one-class clas- sifiers on learned representations. The framework not only allows to learn bet- ter representations, but also permits building one-class classifiers that are faithful to the target task. We argue that classifiers inspired by the statistical perspec- discriminative models are more effective than existing ap-
tive in generative proaches, such as a normality score from surrogate classifier. We thoroughly evaluate different self-supervised representation learning algorithms under the proposed framework for one-class classification. Moreover, we present a novel contrastive learning that extends training distributions via data augmentation to obstruct the uniformity of contrastive representations. experiments, we demonstrate state-of-the-as thrt--rrtt performance on visual domain one- class classification benchmarks, including novelty and anomaly detection. Finally, we present visual explanations, confirming that the decision-making process of deep one-class classifiers is intuitive to humans. The code is available at https: //github. com/ google e research deep www.rreeeeeeettteeeeeeeeee wwweaaattioooeeeeeeeeeeeeeeeeeee one_oon____e one_claas..
pessearch/deep_rpppeee
INTRODUCTION
One-class classification aims to identify if an example belongs to the same distribution as the training data. There are several applications of one-class classification, such as anomaly detection or outlier detection, where we learn classifier that distinguishes the anomaly/outlier data without access to them from the normal/inlier data accessible at training. This problem is common in various domains, such manufacturing defect detection, financial fraud detection, etc.
Generative models, such as kernel density estimation (KDE), is popular for one-class classifica- tion [1, 2] they model the distribution by assigning high density to the training data. At test time, low density examples are determined as outliers. Unfortunately. the curse of dimensionality hinders accurate density estimation in high dimensions [3]. Deep generative models (e.g. [4. 6)). have demonstrated success in modeling high-dimensional data (e.g., images) and have been applied to anomaly detection [7, 8, 10, 11]. However, learning deep generative models on raw inputs mains as challenging as they appear to assign high density to background pixels [10] or learn local pixel correlations [12]. A good representation might still be beneficial those models.
Alternately, discriminative models like one-class SVM (OC-SVM) [13] or support vector data de- scription (SVDD) [14] learn classifiers describing the support of one-class distributions to distin- guish them from outliers. These methods are powerful when being with non-linear kernels. How- ever, its performance still limited by the quality of input data representations.
In either generative or discriminative approaches, the fundamental limitation of one-class classi- fication centers on learning good high-level data representations. Following the success of deep learning [15]. deep one-class classifications [16, 17, 18]. which extend the discriminative one-class classification using trainable deep neural networks. have shown promising results compared to their kernel counterparts. However, naive training of deep one-class classifiers leads degenerate solution that maps all data into a single representation, also known as "hypersphere collapse" [16]. Previous works circumvent such issues by constraining network architectures [16], autoencoder
"Equal contribution.

---

Published as a conference paper at ICLR 2021
pretraining [16, 17], surrogate multi-class classification simulated outliers [19, 20, 21, 22] or injecting noise [18].
In this work, present two-stage framework for building deep one-class classifiers. As shown in Figure I, in the first stage, we train a deep neural network to obtain a high-level data representation. In the second stage, we build one-class classifier, such OC-SVM or KDE, using representa- tions from the first stage. Comparing to using surrogate losses [20, 21], our framework allows to build classifier that is more faithful one-class classification. Decoupling representation learning from classifier construction further opens up opportunities of using state-of-the-art representation learning methods, such as sepersupereieed contrastive learning [23]. While vanilla contrastive repre- sentations are less compatible with one-class classification as they are uniformly distributed on the hypersphere [24], we show that, with proper fixes, it provides representations achieving competi- tive one-class classification performance to previous state-of-the artss Furthermore, we propose a distribution-aaummentee contrastive learning. a novel variant of contrastive learning with distribution augmentation [25]. This is particularly effective in learning representations for one-class classifica- tion, as it reduces the class collision between examples from the same class [26] and uniformity [24]. Lastly, although representations are not optimized for one-class classification in end-to-end train- able deep one-class classifiers [16], we demonstrate state-of- satte-of-the-art classification benchmarks. We summarize our contributions follows:
the-ahrr art performance on visual one-class
We present a two-stage framework for building deep one-class classifiers using unsupervised and self-supervised representations followed by shallow one-class classifiers.
We systematically study representation learning methods for one-class classification, including augmentation prediction, contrastive learning, and the proposed distribution -utgmenedd con- trastive learning method that extends training data distributions via data augmentation.
We show that, with good representation, both discriminative (OC-SVM) and generative (KDE) classifiers, while being competitive with each other, are better than surrogate classifiers based on the simulated outliers [20, 21].
We achieve strong performance on visual one-class classification benchmarks, such as CIFAR- 10/100 [27], Fashion MNIST [28], Cat-vs-Dog [29). CelebA [30], and MVTec AD [31].
We extensively study the one-class contrastive learning and the realistic evaluation of anomaly detection under unsupervised and semi-supervised settings. Finally, we present visual explana- tions of our deep one-class classifiers to better understand their decision making processes.
2 RELATED WORK
One-class classification [32] has broad applications, including fraud detection [33], spam filter- ing [34], medical diagnosis [35], manufacturing defect detection [31], to name a few. Due to the lack of granular semantic information for one-class data, learning from unlabeled data have been employed for one-class classification. Generative models, which model the density of training data distribution, are able to determine outlier when the sample shows low density [8, 35, 36). These in- clude simple methods such as kernel density estimation or mixture models [37], as well as advanced ones [4, 5, 38, 39, 40, 41]. However, the density from generative models for high-dimensional data could be misleading [9, 12, 42, 43). New detection mechanisms based on the typicality [44] or likelihood ratios [10] have been proposed to improve -uttof-distribution detection.
Self-supervised learning is commonly used for learning representations from unlabeled data by solv- ing proxy tasks, such jigsaw puzzle [45], rotation prediction [46], clustering [47], instance dis- crimination [48] and contrastive learning [23, 49, 50). The learned representations are then used for multi-class classification, or transfer learning. all of which require labeled data for downstream tasks. They have also been extended to one-class classification. For example, contrastive learning adopted to improve the out-of-distribution detection under multi-class setting [51], whereas our work focuses on learning from a single class of examples, leading to propose a novel distribution- augmented contrastive learning. Notably, learning to predict geometric transformations [20, 21, 22] extends the rotation prediction [46] to using more geometric transformations as prediction targets. Unlike typical applications of self-supervised learning where the classifier or projection head [23] discarded after training, the geometric transformation classifier is used as surrogate for one- class classification. As in Section 4.1, however, the surrogate classifier optimized for the self- supervised proxy task is suboptimal for one-class classification. We show that replacing it with
2

---

Published as a conference paper at ICLR 2021
Fived
self-supee sepe-super proxy task
one-class classifier
Normality score
f00
g
gof(x
(eg, rotation prodiction, contrastive leamiig)
[0)
(egg. OC-SVM, KDE)
(a) Self-supervised representation learning
(b) One-class classifier
Figure I: Overview of our two-stage framework for building deep one-class classifier. (a) In the first stage, we learn representations from one-class training distribution using self-supervised learning methods, and (b) in the second stage, train one-class classifiers using learned representations.
Proxy Task
Normality Score Test Time (xx,)
Golan and El- Yaniv [20] Hendrycks et al. [21]
Lro
(0]f(z)) or Eye Pel (10 v)j)
Ours
Lrtt. Cer cir
Cdistaug
(f(x)(()) OC-SVM(((a.))
Table I: A comparison between one-class classifiers based self-supervised learning. Previous works [20, 21] train one-class classifiers using augmentation prediction with geometric transforma- tions (e.g. Lroa) and determine outliers using augmentation classifiers (f₂)) We learn representations using proxy tasks of different self-supervised learning methods, such as contrastive learning (Letr), and build simple one-class classifiers, such as KDE or OC-SVM, learned inlier representations.
simple one-class classifiers consistently improve the performance. Furthermore, we propose strate- gies for better representation learning for both augmentation prediction and contrastive learning.
Distribution-ane Distribution-anmmenteed contrastive learning is concurrently developed in [52] as a part of their multi-task ensemble model. While sharing a similar technical formulation, we motivate from fixing the uniformity of contrastive representations. We note that our study not only focuses on represen- tation learning, but also on the importance of detection algorithms, which is under explored before.
3 A TWO-STAGE FRAMEWORK FOR DEEP ONE- CLESS CLASSIFICATION
In Section 3.1, we review self-super suff-supereieed representation learning algorithms, discuss how they con- nect to existing one-class classification methods, raise issues of state-of-the-aae contrastive represen- tation learning [23] for one-class classification, and propose ways to resolve these issues. Then, in Section 3.2, we study how leverage the learned representations for one-class classification.
3.1 LEARNING REPRESENTATION FOR ONE- CLASS CLASSIHCATION
Let A be the stochastic data augmentation process, which composed resize and crop, horizontal flip, color jittering, gray- and gaussian blur, following [23], for image data. As in Figure 1, self- supervised learning methods consist of a feature extractor f parameterized by deep neural networks and the proxy loss L. Optionally, f is further processed with projection head g at training, which is then used to compute the proxy loss. Unless otherwise stated, normalize(f) ff f/liflla is used representation at test time. Below, we discuss details of self-supervised learning methods.
3.1.1 EXTRACTING RICHER REPRESENTATION BY LEARNING WITH PROJECTION HEAD
While the efficacy of projection head has been confirmed for contrastive learning [23] or BYOL [53], is not widely adopted for other types of self-supervised learning, including rotation prediction [46]. On the other hand, Gidaris et al. [46] show that the lower-layer representation often perform better for downstream tasks as the last layer directly involved in optimizing the proxy loss becomes overly discriminative to the proxy task, while losing useful information of the data.
Inspired by these observations, we adopt the projection head for augmentation prediction training well. As in Figure la, we extend the network structure g where is the projection head used to compute proxy losses and outputs representations used for the downstream task. Note that using an identity head g(x)) z recovers the network structure of previous works [20, 21, 46].
3.1.2 AUGMENTATION PREDICTION
One way of representation learning is to learn by discriminating augmentations applied to the data. For example, the rotation prediction [46] learns deep representations by predicting the degree of
3

---

Published as a conference paper at ICLR 2021
Inlier
Aug. Data
Outlier
(a) Perfect uniformity.
(b) Reduced uniformity. (c) Distribution augmentation.
Figure 2: One-class classification on different levels of uniformity of inlier distributions. (a) When representations uniform, isolating outliers is hard. (b) Reducing uniformity makes boundary be- tween inlier and outlier clear. (c) Distribution augmentation allows inlier distribution more compact.
rotation augmentations. The training objective of the rotation prediction task is given as follows:
Leot Px CrossEntropy (y, Uggg rot90 (A(x))
where y 1,, ( is prediction target representing the rotation degree, and (r, y) rotates an input by 90 degree y times. We denote the classifier Poog (y/2z) as (v)g((z)) x exp(gog(z))lv) containing the representation g¹ and linear layer q with 4 output units for rotation degrees.
Application to One-Class Classification. Although not trained to do so, the likelihood of learned rotation classifiers Py (y 0lg(x)) shown to well approximate the normality score and has been used for one-class classification [20, 21, 22]. A plausible explanation is via outlier exposure [19], where the classifier learns a decision boundary distinguishing original images from simulated out- liers by image rotation. However, it assumes inlier images are not rotated, and the classifier may not generalize to one-class classification task if it overfits to the proxy rotation prediction task.
3.1.3 CONTRASTIVE LEARNING
Unlike augmentation prediction that learns discriminative representations to data augmentation, con- trastive learning [23] learns representation by distinguishing different views (e.g., augmentations) of itself from other data instances. Let (xx) normalize i.c., [[(()] = Following [54], the proxy task loss of contrastive learning written as:
exp ( T (()())) Lelr = Ex,,, PAiAAA log exp(A)(((()) ( ((A(())))+ +I exp( (o(A(z)) ((A(z))))
(2)
where A and A' are identical but independent stochastic augmentation processes for two different views of x. Cet regularizes representations of the same instance with different views (A(z),A((z)) be similar, while those of different instances (A(z), A' (x()) to be unlike.
Class Collision and Uniformity for One-Class Classification. While contrastive representations
have achieved state-of-the-ee thetttr performance on visual recognition tasks [23, 24. 49, 55] and have been theoretically proven to be effective for multi-class classification [26, 56], we argue that it could be problematic for one-class classification.
First, a class collision [26]. The contrastive loss in Eq. (2) is minimized by maximizing the distance between representations of negative pairs (z) ) Iy, even though they are from the same class when applied to the one-class classification. This seems to contradict to the idea of deep one-class classification [16], which learns representations by minimizing the distance between representations with respect to the center: ming./ Ez g f(x) - cll2.
Second, a uniformity representations [24]. is proved that the optimal solution for the denomina- tor of Eq. (2) is perfect uniformity M - oo [24], meaning that ø(x;) follows a uniform distribution on the hypersphere. This is problematic since one can always find an inlier x E in the proximity to any outlier x' X' the hypersphere, shown in Figure 2a. In contrast, with reduced uniformity as in Figure 2b, it is casier to isolate outliers from inliers.
abuse notation of not only denote the projection head, but also the representation go f(-)- For presentation clarity, use the rotation example for augmentations. Note that may more geometric transformations, such rotation, translation, or flip of an image, in [20, 21, 22].
4

---

Published as a conference paper at ICLR 2021
Distribution-augmentte contrastive learning. Not only learning to discriminate differ- ent instances from an original distribution (e.g.. two images of different dogs on the left), it also learns to discriminate instances from different distributions created via augmentations, such as rotations (e.g., two images of the same dog with different rotation degrees on the top).
Figure 3:
pushh
One-Class Contrastive Learning. First, reduce the uniformity of representations, we propose to use moderate M (batch size). This is in contrast with previous suggestions to train with large M for contrastive representations to be most effective on multi-class classification tasks [23, 55, 57]. The impact of batch size M for one-class classification will be discussed in Section 5.1.
In addition, we propose distribution augmentation' for one-class contrastive learning- The idea that, instead of modeling the training data distribution Px. we model the union of augmented train- ing distribution PU, afx),, where a(x)= = (a(x) e Note that augmentation a for augmenting distribution is disjoint from those for data augmentation A that generates views. Inspired by Golan and Yaniv [20]. we employ geometric transformations, such as rotation or horizontal flip, for distribution augmentation. For example, in Figure 3, and rot90(x) (German shepherds in the top row) are considered as two separate instances and therefore are encouraged to be distant in the representation space. Not only increasing the number of data instances train (e.g., distribution augmentation by rotating 90", 180", 270 increases the dataset by times), but also eases the uni- formity of representations on the resulted hypersphere. A pictorial example is in Figure 2c, where thanks to augmented distribution, the inlier distribution may become more compact.
3.2 BUILDING DEEP ONE-CLASS CLASSIFIERS WITH LEARNED REPRESENTATIONS
We present two-stage framework for deep one-class classification that builds one-class classifiers on learned representations as in Figure 1. Compared to end-to-end training [16, 20, 21. 22], our framework provides flexibility in using various representations as the classifier is not bound to rep- resentation learning. It also allows the classifier consistent with one-class classification objective.
To construct a classifier, we revisit an old wisdom which considers the full spectrum of the distribu- tion of the learned data representation. For generative approaches, we propose to use nonparametric kernel density estimation (KDE) to estimate densities from learned representations. For discrimina- tive approaches, we train one-class SVMs [13]. Both methods work as black-box and in experiment we use the default training setting except the kernel width where we reduce by 10 times than default. We provide detailed classifier formulations in Appendix A.I.
3.2.1 GRADIENT-BASE EXPLANATION OF DEEP ONF-CCAAS CLASSIFIER
Explaining the decision making process helps users to trust deep learning models. There have been efforts visually explain the reason for model decisions of multi-class classifiers [58, 59, 60, 61, 62, 63, 64] using the gradients computed from the classifier. In this work, we introduce a gradient-based visual explanation of one-class classification that works for any deep representations. To construct an end-to-end differentiable decision function, we employ a KDE detector, whose formulation is in Ap- pendix A. built top of any differentiable deep representations: OKDE(/(=) aKDE(/(w) (w)) af(a) 8f(x)
de
EXPERIMENTS
Following [20], we evaluate on one-class classification benchmarks, including CIFAR-10, CIFAR- 100 [27], Fashion-MNIST [28], and Cat-vs-Dog [29]. Images from one class are given as inlier and those from remaining classes are given as outlier. We further propose a new protocol using CelebA eyeglasses dataset [30], where face images with eyeglasses are denoted
outlier. It is
3 Wee note that the distribution augmentation has heen applicd to generative modeling [25] as a way of improving regularization and generalization via multi-task learning. 220 super class labels are used for CIFAR-100 experiments [20].
5

---

Published as a conference paper at ICLR 2021
Representation
Classifier
CIARR-00
CIFAR- 100
E-MNIST
Cat-vs-Dog
CelebA
Mcan
ResNet--5 (ImageNet)
OC-SVM
80.0
83.7
91.8
74.5
81.4
84.0
KDE
83.7
90.5
74.6
82.4
83.7
Rotation Classifier
86.820.4
80.320.
87.4
86,1
$1.443.
83.1
RotNet (20)
KDE
89.3
81.9
94.6+0..
86,420.2
77.4
86.6
Denoising
OC-SVM KDE
83.4=1.0
75.2=1.0
93.9+0.4
57.3#1.
66.8+0.9
80.4
83.5
75.2
93.740.4
57.3
67.0a0.7
80.4
Rotation Prediction
OCSSVM
90.8
82.8
94.6120.
83.720..
65.8100
87.1
KDE
91.3
84. 14.1
95.8+0.3
86.4
69.5
88.2
OC-SVM
89.0g0.7
82,4208
93.920.3
87.7.0.5
83.512.
86.9
Contrastive
KDE
89.0.9.7
82.4.0.
93.6190.
87.7.00.4
84.612.5
86.8
OC-SVM
92.5=:0.6
86.5
94.8+n.a
89.6
84.5+111
89.9
Contrastive (DA)
KDE
92.4
86.5.0077
94.5+00.4
89.620.44
85.6
89.8
Table 2: We report the mean and standard deviation of one-class classification AUCs averaged over classes over 5 runs. The best methods are bold-faced for each setting. The per-class AUCs are reported in Appendix A.5. All methods are implemented and evaluated under the same condition.
CIFAR-10
CIFAR-100
FMNIST
cat-vs-lig
Table Comparison to previous one-class classification methods. denotes evaluation methods using test time data augmentation. Our methods are both more accurate and computationally efficient.
Ruff al.
64.8
Golan and EI YalYYnii |20]|
86.0
78.7
93.5
88.8
Bergman and Hoshca [22]*
88.2
94.1
Hendrycks [21]*
90.1
Huang et al. [36]]
86.6
78.8
93.9
Rotation prodiction
91.3-0.5
84.1
95.8=00a
86.4
Ours: Contrastive (DA)
92.5.006
86.5
94.8.10.3
89.620.5
more challenging since the difference between in and outlier samples is finer-grained. Last, in addi- tion to semantic anomaly detection as aforementioned, we consider the defect detection benchmark MV'Te [31] in Section 4.2.
We evaluate the performance of (1) representations trained with unsupervised and self-supervised learning methods, including denoising autoencoder [65]. rotation prediction [46]. contrastive learn- ing [23, 48], and (2) using different one-class classifiers, such as OC-SVM or KDE. We use rotation augmentations for distribution-augmenne
contrastive learning, denoted as Contrastive (DA). We train a ResNet-18 [66] for feature extractor f and stack of linear, batch normalization, and ReLU, for MLP projection head g. More experiment details can be found Appendix A.4.
4.1 MAIN RESULTS
We report the mean and standard deviation of AUCs averaged over classes over 5 runs in Table 2. The mean of 5 datasets is weighted by the number of classes for each dataset. Besides those using self-supervised representations, we provide results using ImageNet pretrained ResNet-50 to high- light the importance of learning representations from in-domain distributions.
ImageNet pretrained ResNer-50 achieves the best performance of 84.0 mean AUC over 5 datasets. Compared to representations learned with denoising objective, it works particularly well datasets such as CIFAR- 100, cat-vs-dog, and CelebA, which we attribute it to the subset of ImaegNet classes closely related to the classes of these datasets.
Similar to the findings from [20, 21]. we observe significant performance gains with self-supervised learning. Moreover, while RotNet [20], an end-to-end trained classifier using rotation prediction, achieves 83.1 AUC, the RotNet representation evaluated with the KDE detector achieves 86.6, 6,, em- phasizing the importance of a proper detector in the second stage. Finally, the quality of RotNet representation improves when trained with the MLP projection head, resulting in 88.2 AUC.
The representations learned with vanilla contrastive loss achieve 86.9 AUC with OC-SVM, which under-perform those trained with rotation prediction loss (88.2). The distribution-augman augmenteed con- trastive loss achieves the highest mean AUC of 89.9 9 among all methods, demonstrating performance gains all datasets by large margin upon its vanilla counterparts.
Comparison to Previous Works. We make comparisons to previous works Table 3. While some comparisons may not be fair as different works different implementations, note that our implementation is based on the common choices of network (c.g.. ResNet-18) and optimizer (c.g., momentum SGD) for image classification. We advance the previous state-of-the-art one-class

---

Published as a conference paper at ICLR 2021
Protocol
RotNet [20, 21]
RotNet!
RotNet (MLP head))
Vanilla CLRI
DistAug CLR
Detection
71.013.5
83.513.0
86.3
80.2
86.5
Localization 75.6
92.611.0
93.0
85.611.3
90.4.10.8
Table 4: Image-level detection and pixel-level localization AUCs on MVTec anomaly detection dataset [31]. We run experiments 5 times with different random seeds and report the mean and standard deviations. We bold-face the best entry of each row and those within the standard deviation. We use to denote the of KDE for one-class classifier.
Contrastive 9
(00)
MAD
13T
Contractue f
Contractive of
Contruative Contrastive (DAI g-1
Cottrastie (DA)
Cartrastive (D4) gxf
size
batch
Depth head
(a) MMDs with various batch sizes. (b) AUCs with various batch sizes. (c) AUCs with various MLP depths.
Figure 4: Ablation study contrastive representations trained with different batch sizes. We use (a) the MMD distance between the representations and the data sampled from uniform distributions to measure uniformity. Small MMD distance means being similar to uniform distribution. We also report (b) one-class classification performance in AUCs evaluated by kernel OC-SVMs and (c) the performance with ML. heads of different depths.
classification benchmarks by large margin without test-time augmentation ensemble of models. We further improve the performance with model ensemble, which we report in Appendix A.2.2.
4.2 EXPERIMENTS ON MVTEC ANOMALY (DEFECT) DETECTION
Finally, we evaluate proposed framework MVTee [31] defect detection dataset, which is com- prised of 15 different categories, including objects and textures. Instead of learning representations from the entire image, we learn representations of 32x32 patches. At time, compute the normality scores of x2x32 patches densely extracted from 256 x 256 images with stride of 4. For image-level detection, we apply spatial max-pooling to obtain single image-level score. For local- ization, we upsample the spatial score map with Gaussian kernel [67]. Please see Appendix B for more implementation details and experimental results.
We report in Table 4 the detection and localization AUCs. We verify the similar trend to previous experiments. For example, while using the same representation, RotNet with KDE (RoNNet) signif- icantly outperform the RotNet using built-in rotation classifier. Moreover, contrastive learning (with rotation) improves the performance of vanilla contrastive learning. Due to space constraint, show localization results in Appendix B.4.
Eetribution-augment
ANALYSIS AND ABLATION STUDY
In Section 5.1, we amalyze behaviors of one-class contrastive representations and in Section 5.4, we report visual explanations of various deep one-class classifiers. Due to a space constraint, more studies, including in-depth study on efficiency of self-supervised learning for one-class classification, are in Appendix A.2. 5.1 UNIFORMITY, BATCH SIZE AND DISTRIBUTION AUGMENTATION
contrastive representations and data
[23, 49, 55, 57] have shown substantial improvement contrastive representations evaluated on multi-class classification using very large batch sizes, which results uniformity. However, uni- formity [24] and class collision [26] be issue for one-class classification discussed in Section 1.3. Here we investigate the relations between performance and uniformity, and how we resolve the issue via batch size, MLP head and inttriuttionauugmeent
contrastive learning.
We measure the uniformity via MMD distance [68] between the learned representation and samples from uniform distributions on hyperspheres. Smaller MMD distance implies the distribution of rep- resentation is closer to uniform distributions. We train models with various batch sizes (2?...., 29). We report in Figure 4 the average and standard deviation over runs on CIFAR-10 validation

---

Published as a conference paper at ICLR 2021
Conteet for
Outlier
Outleer
5.30
000
Outlier Ratio
(a) Unsupervised settings for learn-
(c) Unsupervised setting for learn- ing representation and 1% (50) one- ing representation and 2% (100) tector with various outlier ratio. class data for building detector.
(b) Unsupervised setting for learn-
ing representation and building de-
one-class data for building detector.
Figure 5: Realistic evaluation of anomaly detection under (5a) unsupervised and (5b, 5c) one-class or semi-supervised settings. For unsupervised settings, we either learn representation or build detector from a training set containing both inlier and outlier data without their labels. For one-class or semi- supervised settings, we learn from a training set containing only a small amount of one-class (inlier) data. We report AUCs on CIFAR-IO and OC-SVM with RBF kernel is used for evaluation.
Distribution Augmentation. In Figure 4a, the representations from the projection head gof trained by standard contrastive learning (Contrastive g o f) are closer be uniformly distributed when we increase the batch size as proven by [24]. Therefore, in one-class classification, the nearly uniformly distributed representation from contrastive learning is only slightly better than random guess (50 AUC) as in Figure 4b. In contrast, with distribution augmentations, the learned representations (Contrastive (DA) g o f) are less uniformly distributed and result in a significant performance gain.
Batch Size. Following the discussion, large batch sizes result in nearly uniformly distributed rep- resentations (g o f), which harmful to the one-class classification. On the other hand, small batch sizes (=16), though less uniform, hinders us learning useful representations via information max- imization [69]. As in Figure 4b, there is a trade-off of batch size for one-class classification, and find batch size of 32 results in the best one-class classification performance. Further analysis showing the positive correlation between the uniformity measured by the MMD and the one-class classification performance is in Appendix A.2.2 and Figure 8.
MLP Projection Head. Similarly to [23], we find that f, an input to the projection head, performs better the downstream task than g As in Figure 4c, the performance of f improves with deeper MLP head, while g o f degrades, as it overfits to the proxy task.
Lastly, emphasize that all these fixes contribute to improvement of contrastive representations for one-class classification. As in Figure 4 AUC drops when any of these components are missing.
5.2 ANALYSIS ON DIFFERENT DISTRIBUTION AUGMENTATIONS
The choice of distributions affects the performance. The ablation study using horizontal flip (hflip) and rotation augmentations is reported in Figure 7. Note that hflip is used only to augment distribu- tion in this experiment. Interestingly, simply adding hilp improves the AUC to 90.7. This suggests different insight from [52] who augments distribution as a means to outlier exposure [19]. Although report the numbers with rotation augmentations in Table 2, with hflip, rot90, rot90+hhlip as augmented distributions, we achieve the best mean AUC, 93.7, on CIFAR-10. without any test-time augmentation. Additional study on distribution augmentation follows in Appendix A.2.2.
5.3 APPLICATIONS TO UNSUPERVISED AND SUMI-SUPERVISED ANOMALY DETECTION
We conduct experiments for unsupervised anomaly detection, where the training set may contain a few outlier 5 We study two settings: 1) Unsupervised settings for both learning representation and building detector, and 2) unsupervised setting for learning representation, but building detector with a small amount (as few as 50 data instances) of one-class data only. For both settings, we vary the outlier ratio in the training set from 0.5% to 10%. We show results in Figure 5. As in Figure 5a, we observe the decrease in performance when increasing the outlier ratio as expected. Rotation prediction is slightly more robust than contrastive learning for high outlier ratio. On the other hand, when classifier is built with clean one-class data, contrastive representations performs
'We follow the definition of [70] to distinguish unsupervised and semi-supervised settings of anomaly detection. Please see Appendix A.3 for additional description on their definitions.
8

---

Published as a conference paper at ICLR 2021
(a) Input
(b) Ours
(c) RotNet (d) RotNet"
(e) DAE
(f) Ours
RotNet (h) RotNet"
DAE
Figure 6: Visual explanations deep one-class classifiers on cat-vs-dog and CelebA eyeglasses
datasets. (a) input images, (b-e) images with heatmaps using integrated gradients [62], and (f-i) RotNet + KDE. More examples are in Appendix A.6.
those using GradCAM [61]. RotNet":
Figure 7: AUCs of contrastive representations trained from various augmented distributions on CIFAR-10. From left to right, we train by accumulating distributions. For example, an entry with "+rot90, hilip" is trained with distributions, i.e., original, hflip, rot90 and rot90, hflip.
21
Contrastive (DA)
21/0
sroooo Mio
Distributions
better. Interestingly, contrastive learning benefits from outlier data, as naturally learn to distinguish inlier and outlier. Due to space constraint, we provide more results and analysis in Appendix A.3,
5.4 VISUAL EXPLANATION OF DEEP ONE-CLASS CLASSIFIERS
We investigate the decision making process our deep one-class classifiers using tools described in Section 3.2. Specifically, we inspect by highlighting the most influential regions using two popular visual explanation algorithms, namely, integrated gradients (IG) [62] and GradCAM [61], for distribution-auggeenee cat-vs-dog and CelebA eyeglasses datasets. For RotNet, we test using both a rotation classifier and KDE (RotNet" in Figure 6) to compute gradients.
contrastive representation, well RotNet and DAE images from
As in Figure 6, the proposed visual explanation method based on the KDE one-class classifier per- mits highlighting human-intuitive, meaningful regions in the images, such dog faces or eyeglasses instead of spurious background regions (Figure 6b). On the other hand, even though the classifica- tion AUC is not too worse (86.1 AUC on cat-vs-dog as opposed 89.6 for ours), the visual expla- nation based on the rotation classifier (Figure 6c) suggests that the decision may be made sometimes based on the spurious features (e.g., human face in the background). We present more examples for visual explanation in Appendix A.6.
CONCLUSION
Inspired by an old wisdom of learning representations followed by building classifiers, we present two-stage framework for deep one-class classification. We emphasize the importance of decoupling building classifiers from learning representations, which allows classifier to consistent with the target task, one-class classification. Moreover, it permits applications of various self-supervised representation learning methods, including contrastive learning [23] with proper fixes, for one-class classification, achieving strong performance visual one-class classification benchmarks. Finally, we exhibit visual explanation capability of our two-stage sulf-supe deep one-class classifiers.

---

Published as a conference paper at ICLR 2021
REFERENCES
[1] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international confer- ence on Management of data, pages 93-104, 2000. 1
[2] Longin Jan Latecki, Aleksandar Lazarevic, and Dragoljub Pokrajac. Outlier detection with kernel density functions. In International Workshop on Machine Learning and Data Mining in Pattern Recognition, pages 61-75. Springer, 2007.
[3] Alexandre B Tsybakow. Introduction to nonparametric estimation. Springer Science & Busi- ness Media, 2008. 1, 15
[4] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volle Volume 48, pages 1747-1756, 2016. 1,2
[5] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolu- tions. In Advances in neural information processing systems, pages 10215-10224, 2018. 1, 2
[6] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013. 1,2
[7] Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. arXiv preprint 16X5::660..0777,, 2016.
[8] Bo Zong, Qi Song. Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detec- tion. In International Conference on Learning Representations, 2018. 1,2
[9] Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv: 0rXiv:1810.01392, 2018. 1,2
[10] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distriiutioo detection. In Advances in Neural Information Processing Systems, pages 14707-14718. 2019. 1,2
[11] Warren R Morningstar, Cusuh Ham, Andrew G Gallagher, Balaji Lakshminarayanan, Alexan- der A Alemi, and Joshua V Dillon. Density of states estimation for out- ciatof--iittrbution detec-
tion. arXiv preprint arXiv:2006.0922 2020. 1
[12] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect -fittt--isttibuution data. arXiv preprint arXiv:2006.08545, 2020. 1,2
[13] Bernhard Schölkopf, Robert C Williamson, Alex J Smola, John Shawe- Taylor, and John C Platt. Support vector method for novelty detection. In Advances in neural information pro- cessing systems, pages 582-588, 2000. 1,5, 15
[14] David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54 (1):45-66, 2004. 1
[15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-
444, 2015. 1
[16] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pages 4393-4402, 4402, 2018. 1, 2, 5,
[17] Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv preprint arXiv: 1102.068000066600 2018. 1,2
[18] Poojan Oza and Vishal M Patel. One-class convolutional neural network. IEEE Signal Pro- cessing Letters, 26(2):277-22 2018. 1,2
[19] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arxiv:1812 04660,, 2018. 2, 4, 8, 16
[20] Izhak Golan and Ran El-Y Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, pages 9758-97689 2018. 2, 3, 4, 5, 6, 7, 15, 16, 18, 20, 21, 25
10

---

Published as a conference paper at ICLR 2021
[21] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pages 15637-15648, 2019. 2, 4,5, 6, 7, 16, 25
[22] Liron Bergman and Yedid Hoshen. Classification baaiieeatio--baeeddd anomaly detection for general data. In Eighth International Conference on Learning Representations, 2020. 2, 4, 5, 6, 16
[23] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020. 2,
3,4, 5, 6, 7, 8, 9, 15
[24] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprini arXiv:2005. 10242, 2020. 2, 4, 7,
8
[25] Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In Proceedings of Machine Learning and Systems 2020, pages 10563-10576, 2020. 2,5
[26] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Interna- tional Conference on Machine Learning, pages 5628-5637, 2019. 2, 4,
[27] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. 2, 5
[28] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:/708.0m 077447 2017. 5,
[29] Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul. Asirra: a captcha that exploits interest-aligned manual image categorization. In ACM Conference Computer and Commu- nications Security, volume 7, pages 366-374, 2007. 2,5
[30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEFE international conference on computer vision, pages 3730- 3738, 2015. 2, 5
[31] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad a compre- hensive real-world dataset for unsupervised anomaly detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592-9600 2019. 2, 6, 7, 25, 26, 27, 28, 29, 30, 31, 32
[32] Mary M Moya, Mark W Koch, and Larry D Hostetler. One-class classifier networks for target recognition applications. STIN, 93:24043, 1993. 2
[33] Clifton Phua, Vincent Lee. Kate Smith, and Ross Gayler. A comprehensive survey of data mining-based fraud detection research. arXiv preprint 6109,1900,,619, 2010. 2
[34] Igor Santos, Carlos Laorden, Xabier Ugarte-Pedrero, Borja Sanz, and Pablo G Bringas. Spam filtering through anomaly detection. In International Conference on E-Business and Telecom- munications, pages 203 223-216.35 Springer, 2011.
[35] Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. International conference on information processing in medical imaging. pages 146-157. Springer, 2017. 2
[36] Chaoqing Huang, Jinkun Cao, Fei Yc, Maosen Li, Ya Zhang, and Cewu Lu. Inverse- autoencoder for anomaly detection. arXiv preprint arXiv: 191.1161,,,0676, 2019. 2, 6
[37]
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006. 2
[38] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise train- ing of deep networks. In Advances in neural information processing systems, pages 153 160,
2007. 2
[39] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014. 2

---

Published as a conference paper at ICLR 2021
[40] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, pages 4790-4798, 2016. 2
[41] Laurent Dinh, Jascha Sohl-Dicksteid and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: h 008000 2016. 2
[42] Eric Nalismick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshmi- narayanan. Do deep generative models know what they don't know? arXiv preprini 1rX.v:1810.091366 2018. 2
[43] Vít Škvára, Tomá3 Pevný, and Václav Šmídl. Are generative deep models for novelty detection truly better? arXiv preprint
1807.000220 2018. 2
[44] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint wwXi::196602294, 5, 2019. 2
[45] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69-84. Springer, 2016. 2
[46] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Sixth International Conference on Learning Representations, 2018. 2, 3, 6, 26
[47] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Com- puter Vision (ECCV), pages 132-149, 2018. 2
[48] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6210-6219, 2019. 2,
[49] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 0rXiv:1807.0377 003748, 2018. 4,
[50] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-97388 2020. 2
[51] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Contrastive training for improved out-of-distribution detection. arXiv preprint arXiv:2007.05566, 2020. 2
[52] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. arXiv preprint arXiv:2007.08176, 2020. 3. 8, 16
[53] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprins arXiv22006.07733, 2020. 3
[54] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Ad- vances in neural information processing systems, pages 1857-1865, 2016. 4
[55] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 1rXivv1905.09211 2019. 4, 5.
[56] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. arXiv preprint arXiv:2008. 10150, 2020. 4, 15
[57] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by max- imizing mutual information across views. In Advances in Neural Information Processing Sys- tems, pages 15535-15545, 2019. 5,7
[58] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
12

---

Published as a conference paper at ICLR 2021
[59] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klber-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015. 5
[60] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on com-
puter vision and pattern recognition, pages 2921-2929, 2016. 5
[61] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 218-626,220 2017. 5, 9, 21, 22, 23, 24
[62] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328, 2017. 5, 9, 21, 22, 23, 24
[63] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pages 9505-9515, 2018. 5
[64] Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attri- butions through regions. In Proceedings of the IEEE International Conference on Computer Vision, pages 4948-4957, 2019. 5
[65] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th inter- national conference on Machine learning, pages 1096-1103, 2008. 6
[66] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recogni-
tion, pages 770-778, 2016. 6
[67] Philipp Liznerski, Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller. Explainable deep one-class classification. arXiv preprint arXiv:2007.01760, 2020. 7, 25
[68] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
[69] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171- 5180, 2019. 8
[70] Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1-58, 2009. 8, 17
[71] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, 2016. 15
[72] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang. and Barnabás Póezos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pages 2203-2213, 2017. 15
[73] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Fifth International Conference on Learning Representations, 2017. 19
[74] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pret- tenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Per- rot, and E. Duchesnay. Seikit-learn: Machine learning in Python. Journal of Machine Learning
Research, 12:2825-28.3 2011. 19
[75] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. Tensorflow: A system for large-scale machine learning. In /2th {USENIX] Symposium on Operating Systems Design and Implementation ( (OSDI)) 16), pages 265-283 2016. 19
[76] Shashanka Venkataramanan. Hhan-Chuan Peng, Rajat Vikram Singh, and Abhijit Maha- lanobis. Attention guided anomaly localization in images. arXiv preprint 1rXiv:1911.0800 2019. 25, 26
13

---

Published as a conference paper at ICLR 2021
[77] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müllrr. A unifying review of deep and shallow anomaly detection. arXiv preprint arXiv:2009 2020. 26
14

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 FORMULATIONS OF ONE-CLASS CLASSIFIERS
For completeness, provide formulations of one-class classifiers, such one-class support vector machine (OC-SVM) [13] and kernel density estimator, used in our work.
One-Class Support Vector Machine solves the following optimization problem to find support vectors describing the boundary of one-class distribution:
min M.P.C:
3
(3)
2
subject to w fi T p- E. & 0, Vi 1,...
(4)
where (x,) is a feature map. The decision score is given as follows:
s(x) o;k(a;, -
(5)
with coefficients og > 0 for support vectors. Linear RBF (ki (x. exp (-7|kx y)(²)) kemnlss are used for experiments.
Kernel Density Estimation is nonparametric density estimation algorithm. The normality score of KDE with RBF kernel parameter y is written as follows:
(D),(r) = log Σ exp (->ll> - v112]
(6)
A.2 ADDITIONAL. ABLATION STUDY
A.2.1 WELL BEHAVED CONTRASTIVE REPRESENTATION
Linear Separability. A common belief is the good representations are linearly separable with re- spect to some underlying labels, which supported by several empirical success [23]. The linear classifiers are proven to be nearly optimal [56] when we learn representations data with all possi- ble labels along with some additional assumptions. In one-class classification, although we violate the assumptions that we only train of the data to understand the characteristics of the learned embedding. To this end, in addition to training RBF kernel OC-SVM, we train OC-SVM with linear kernels on the learned embedding. We note that it is atypical to use linear OC-SVM, which is usually considered to be suboptimal for describing decision boundaries of one-class classification problems. However, with good represen- tations, linear classifiers show good performance on one-class classification as shown in Table 5, and are better than existing works [20] in Table Lastly, we note that OC-SVM with nonlinear RBF kernel is still better than linear models in Table 5. Similar observations are found in other problems such as regression [71] and generative models [72].
class of data, we interested in the linear separability
Parametric Models. In addition studying from the perspective of discriminative model, we also dive into the generative model side. Instead of using the nonparametric KDE for density estimation, use parametric model, single multivariate Gaussian, whose density is defined
p(z) det (Σ) exp { 2 (f(x)) T (f(x) m)).
shown in Table 5, the Gaussian density estimation (GDE) model shows competitive performance as KDE. It suggests the learned representations from distribution-augmenti contrastive learning is compact as our expectation, which can be well approximated by a simple parametric model with single Gaussian. Compared with nonparametric methods, although parametric models have strong assumptions on the underlying data distributions, using parametric model is more data efficient the assumption h In addition, parametric models have huge advantage in computation during
'The error convergence O(n 1/2), while that of nonparametric KDE is Onn1///) [3].
15

---

Published as a conference paper at ICLR 2021
testing. Therefore, there is trade-off between assumptions of the data as well model flexibility and computational efficiency. We note that the single Gaussian parametric model may not good alternative of KDE universally. A candidate with good trade-off is Gaussian Mixture Models, which can be treated as a middle ground of nonparametrie KDE and parametric single Gaussian model. We leave the study for future works.
Representation
Classifier
CIFAR-10
CIFAR-
F-MNIST
Cat-vs-Dog
CelebA
Mean
OC-SVM (linear)
90.7=0.8
81.1
93.7=0.8
86.320.
88.4
86.7
OC-SVM (kernel)
92.5__066
86.510.7
94,820.3
89.610.8
84.5
89,9
Contrastive (DA)
KDE
9.74-007
86.510.7
94.5
89.610.4
85.6.0.5
89.8
GDF
92.0=0.5
86.020.8
95.5m003
88.920.3
90.62004
89.8
Table 5: One-class classification results using different one-class classifiers rotation-augmentte contrastive representations. In addition to OC-SVM and KDE, both of which with RBF kernels, we test with the linear OC-SVM and the Gaussian density estimator (GDE).
A.2.2 ANALYSIS ON DISTRIBUTION AUGMENTED CONTRASTIVE REPRESENTATIONS
Relation to Outlier Exposure [19]. Distribution augmentation shares a similarity to outlier ex- posure [19] in that both methods introduce new data distributions for training. However, outlier exposure requires a stronger assumption on the data distribution that introduced outlier should not overlap with inlier distribution, while such an assumption is not required for distribution augmen- tation. For example, let's consider rotation prediction as an instance of outlier exposure. When training the model on randomly-rotate CIFAR-IO, where we randomly rotate images of CIFAR-IO, the performance of rotation prediction representation drops to 60.7 AUC, which is slightly better than a random guess. On the other hand, cottion-augmenne contrastive representation not af- fected by random rotation and achieves 92.4 AUC.
Ensemble Contrastive Representations. We note that most previous methods have demon- strated improved performance via extensive test-time data augmentation [20, 21, 22, 52]. While already achieving state-of-the-art one-class classification performance without test-time data aug- mentation, we observe marginal improvement using test time data augmentation. Instead, that an ensemble of classifiers built on different representations significantly improve the perfor- mance. In Table 6, we report the performance of distribution- -aaamtoon- contrastive representations trained with different sets of augmented distributions on CIFAR- 10. We observe that ensemble of 5 models trained with different seeds consistently improves the performance. Moreover, not only we improve one-class classification AUCs when aggregating scores across models trained with different distribution augmentations, we also observe lower standard deviation across different seeds. Finally, when ensemble over 5x5 = 25 models, we achieve 94.6 AUC.
find
Representations
+hf0ip.root0
+rot90.hffip
+rot180
+rot
+rot270
Ensemble of
single model ensemble of 5 models
93.1+0.4
93.740.6
93.6.0.4
93.5
93.1
94.4
93.9
94.4
94.3
94.2
93.9
94.6
Table 6: Performance of single and ensemble models of distribution augmented contrastive represen- tations on CIFAR-10. For each augmented distribution, we report the mean and standard deviation of single model performance ("single model") and that of ensemble model whose ensemble score is aggregated from 5 models trained with different random seeds ("ensemble of 5 models"). "Ensemble of 5" aggregates score from 5 models with different augmentation distributions.
Correlation between Uniformity and One-class Classification. We present scatter plots in Fig- ure 8 showing the positive correlation between the uniformity (log(MMD) and the one-class clas- sification AUCs. Specifically, we compute the MMD using g f. an output of MLP projection head that are used to optimize the contrastive loss at training. We evaluate AUCs using both g f (left of Figure 8) and f (right of Figure 8). Data points in the plots obtained from vanilla and DistAug contrastive models trained with various batch sizes, including 32, 64, 128, 256 and 512. For each configuration, we train 5 models with different random seeds.
From the left plot of Figure 8, we observe a strong positive correlation between MMD and AUC, suggesting that the more uniformly distributed (i.e., lower MMD), the worse the one-class classi-
16

---

Published as a conference paper at ICLR 2021
Pearson corr: 0.914 (p-value: 0.000)
Pearson corr: 0.774 (p-value: 0.000)
Vanilla
Vanilla DistAug
DistAug
AUC (n
AUC (g
22
-45
-3.1
-30
-15
-40 -3.3
-30
-25
log MMD (g
log MMD (g
Figure 8: Scatter plots between uniformity metric (log (log(MMD) and one-class classification AUCs on CIFAR-10. MMD scores are obtained using go f. For AUCs, we show both evaluated using go f (left) and (right). Data points are obtained from vanilla and DistAug contrastive representations trained with batch sizes of 32, 64, 128, 256 and 512 using different random seeds.
fication performance is (i.e., low AUCs). We also compute the Pearson's correlation coefficient, which results in 0.914. We also investigate the correlation between the MMD using g f and the AUC using f. shown in the right plot of Figure 8. We observe highly positive correlation of 0.774 Pearsons correlation coefficient between the MMD using g o and the AUC using f. Overall, our results suggest that reducing uniformity of go improves the one-class classification accuracy tested with both and representations.
A.2.3 DATA EFFICIENCY OF SELF-SUPERVISED REPRESENTATION LEARNING
While previous works on self supervised learning have demonstrated the effectiveness on learning from large-scale unlabeled data, not much has shown for the data efficiency of these methods. Unlike multi-class classification tasks where the amount of data could scale multiplicative with the number of classes, data efficiency of representation learning becomes of particular interest for one-class classification as is hard to collect large-scale data even without an annotation.
We present one-class classification AUCs of representations with various training data sizes, along with two baseline representations such as ResNet-18 using random weights or ImageNet-pretraine ResNet-50 in Figure 9. Note that vary the training set sizes at representation learning phase only, but use a fixed amount (5000) to train classifiers for fair evaluation of the representation quality. We find that even with 50 examples, classifiers benefit from self-supervised learning when comparing against raw or deep features with random weights. The proposed distribution-augmentte contrastive loss could match the performance ImageNet- preetraindd ResNet-50 with only 100 examples, while rotation prediction loss and vanilla contrastive loss require 250 and 1000 examples, respectively.
A.3 UNSUPERVISED AND DEMI-SUPERVISED ANOMALY DETECTION
In this section, we conduct experiments for unsupervised anomaly detection. Note that one-class classification assumes access to training data drawn entirely from one-class, inlier distribution and often referred semi-supervised anomaly detection [70] as it requires human effort to filter out training data from outlier distribution. On the other hand, unsupervised anomaly detection assumes to include training data both from inlier and outlier distributions without knowing their respective labels. In other words, unsupervised anomaly detection may be viewed as one-class classification with noisy data.
Here, are interested in analyzing the impact of label noise (e.g., outlier examples are given as inlier) on one-class classification methods. To this end, we conduct experiments under unsupervised settings that includes different ratios, such 0.5%, 1%. 2%, 5%, 10%. of outlier examples in the train set without their labels. Note that the total amount of training examples for different set- tings remain unchanged. We show results in Figure 10 of 4 models: rotation prediction without
17

---

Published as a conference paper at ICLR 2021
50
85
AUC
80
75
70
Rotation
Contrastive
Dist.
68
ResNet-18 (Random)
ResNetdO (magelteti
60
50
100
250
1000
2500
5000
data
Figure 9: Self-supervised representations are trained from different data sizes, from 50 to 5000, on CIFAR-10. 10. Standard deviations are obtained by sampling subsets times. Classifiers are trained with the full (5000) train set for fair evaluation of representations. We provide baseline representa- tions of ResNet-18 using random weights and SaageNet-pretrainee ResNet-50. Standard deviations computed by running 5 times with different seeds.
Representation
Classifier
CIFAR-10
CIFAR- 100
F-NIST
cat-vs-dog
CelebA
Mean
OC-SVM (lineur)
50.1
50.1
50.3=0.9
$0.0
49.9
50.1
ResNet-18 18 (Random)
OC-SVM (kernel)
61.2a223
59.621
89.8209
50.7
57.1
66.5
KDE
60.5=24
58.7=2.2
89.4=1.
50.8
$7.3+0.8
65.9
OC-SVM (lincar)
67.9
71.0
77.0
61.0
58.0
70.9
ResNet-50 (ImageNet)
OC-SVM (kernel)
80.0
83.7
91.8
74.5
81.4
84.0
KDE
80.0
83.7
90.5
74.6
82.4
83.7
RotNet [20]
Rotation Classifier
86.8
80.3
87.4=1.7
86.1
$1.433
83.1
KDE
89.320.3
81.920.5
94.6:003
86.410.2
77.4+1.
86.6
OC-SVM (lineur)
72.6
62.42/2.1
55.0.3.
61.0a1.5
59.814..0
62.9
Denoising
OC-SVM (kernel)
83.4
75.2
93.9=004
$7.3+1.
66 810.
80.4
KDE
83.521.0
75.2g1
93.7.004
57.311.3
67.020.7
OC-SVM (lincar)
88.5
72.3=1.7
89.0=1.5
81.44.4
62.6
80.1
Rotation Prediction
OC-SVM (kemell)
90.8
82.8
94.6=0.3
83.720.6
65.8
87.1
KDE
91.320.3
84.1
95.8..03
86.420.
69.5
88.2
OC-SVM (linear)
85.621.2
74.921.4
92.3-111
82.811.1
91.7.t0.
82.2
Contrustive
OC-SVM (kemel)
89.0 02.0.
82.420
93.9203
87.7
83.5
86.9
KDE
89.020.7
82.420
93.6.005
87.720.4
84.612.5
86.8
OC-SVM (lincar)
90.7
81.1=13
93.7=0os
86.320.7
88.4+1
86.7
Contrastive (DA)
OC-SVM (kemell)
92.5
86.5
94.8=0.3
89.620.5
84.5
89.9
KDE
92.4
86.5
94.5=004
89.620.4
85.6+0..5
89.8
Table One-class classification results using different representations and one-class classifiers. We report the mean and standard deviation over runs of AUCs averaged over classes. The best methods are bold-faced for each setting.
("RotNet") and with ("Rotation Prediction") MLP projection head, contrastive ("Contrastive") and distribution-augmentte contrastive ("Contrastive (DA)") learning. We also report thee performance evaluated with different classifiers, including OC-SVM with RBF (Figure 10a) and linear (Fig- ure 10c) kernels, and Gaussian Density Estimator (GDE, Figure 10b). For all models, we observe performance degradation of deep one-class classifiers trained with outlier examples Rotation prediction has shown more robust when outlier ratio is high (5 or 10%). While showing stronger performance under one-class setting, kernel-based classifiers (OC-SVM with RBF kernel, KDE) have shown less robust than parameteric (GDE) or linear classifiers under high outlier ratios, as kernel-based methods focus more on the local neighborhood structures to determine a decision boundary.
expected.
In addition, we conduct a study under another sumi-supervised setting, where we are given a small amount of labeled inlier data and large amount of unlabeled training data composed of both inlier and outlier examples. Note that this is more realistic scenario for anomaly detection problems since it is easier obtain some portions of labeled inlier examples than outlier examples. To demonstrate
18

---

Published as a conference paper at ICLR 2021
Katel
Outler Ratio
E21
Ratio
500
5000
Outlir
(a) OC-SVM (RBF kernel)
(b) Gaussian density estimation
(c) OC-SVM (linear kernel)
Figure 10: Classification performance under unsupervised learning setting of representation and classifier. For unsupervised setting, training set contains both inlier and outlier examples without their labels, whereas for one-class setting, training set contains only inlier examples. For represen- tations trained with different outlier ratios, classification performances are evaluated with different classifiers, such as OC-SVMs with RBF and linear kernels, and Gaussian density estimation.
kuteber milicon
1use in miltion
miiicon
Outlier Ratio
Outtler Ratio
Outlerr Ratio
(a) 1 1% (50) onc-class data
(b) 2% (100) one-class data
(c) 10% (500) one-class data
Figure 11: Classification performance under unsupervised representation learning and one-class classifier learning settings. For unsupervised representation learning. training set contains both inlier and outlier examples without their labels. For one-class classifier learning, small portion of inlier data is used for training an OC-SVM with RBF kernel.
its effectiveness, we apply our proposed framework without any modification by first training rep- resentations on unlabeled training data and then building classifiers on these representations using small amount of inlier data. Results shown in Figure 11. Interestingly, we observe consistent im- provement in classification performance for contrastive representations trained on data with higher proportions of outlier examples when classifier is trained on pure one-class data. Plausible expla- nation is that the model learns better contrastive representations when trained with both inlier and outlier as it naturally learn distinguish inlier and outlier. On the other hand, representations from rotation prediction still show performance degradation as it learns to classify inlier and outlier into the same category.
A.4 DETAILS OF EXPERIMENTAL SETTING
We resize images into 64 x 64 for cat-vs-dog and CelebA datasets and 32x32 for the rest, Unless otherwise stated, models are trained for 2048 epochs with momentum (0.9) SGD and single cycle of cosine learning rate decay [73]. L2 weight regularization with coefficient of 0.0003 is applied. We use scikit-learn [74] implementation of OC-SVMs with default value of v. We use y= 19/1 which 10 times larger than the default value for kernel OC-SVM. Same value of y used for KDE. We use scikit-learn implementation of Gaussian mixture model using single component for Gaussian density estimator (GDE). No hyperparameters are tuned for GDE, Finally, all experiments conducted using TensorFlow [75].
Other hyperparameters, such as learning rate or the MLP projection head depth, are cross-validated using small labeled data. While this may violate the assumption of one-class classification, su- pervised model selection is inevitable for deep learning models as their behaviors may be largely dependent on hyperparameters. To this end, we use 10% of inlier (500) and the same number of out- lier examples of CIFAR-10 for hyperparameter selection, and use the same set of hyperparameters to test methods other datasets, which could demonstrate the algorithm robustness. Learning rate
19

---

Published as a conference paper at ICLR 2021
E,0.1, 0.03, 0.01, 0.003, 0.001} and the depth e(0.-.. .... 88 of an MLP projection head are tuned for 0.5, all methods. In addition, the temperature TE (..0, 0.2, } and the batch size E{2", , = 3, .... 9} of contrastive loss are tuned. To this end, we train all models across all datasets using the same hyper- parameter configurations, such as learning rate of 0.01, projection head of depth 8 ([12288 128), temperature of 0.2, or batch size of 32.
A.5 PER- PER-CLASS AUCS
Representation
RotNet [20]
Denoising
Rot. Prediction
Contrastive
Contrastive (DA)
Classifier
Rot. Cls
KDE
OC-SVM
KDB
OC-SVM
KDE
OC-SVM
KDE
OC-SVM
KDE
sirplane
80.3+0.
81.4=0.3
81.620.7
81.6+0.7
83.6
88.5+0.a
88,8
88.8 28.8
90.9
91. 0a.0
automobile
91 2
96.7=0.1
92.4
92.4
96.9+0
97.5
97.5=002
98.9 98.920.1
98. 9
bird
85.3
86.4=0.3
75.9=1.
75.9
87.9
88.2+0.a
87.7
87.7=0.4
88.1+0.1
88. 88.0a02
78.1
78.9.0.3
2..
72.311.9
79.010
78.3
82.0
82.1
83.1
k3.2
deer
85.9+0.5
88.0=0.0
82.3+..s
82.4+0.6
90.5+a02
90.2+002
82.4=1.8
82.3=1s
89.9a1.5
89.4+1.5
dog
86.71:a4
88.
83.1
83.11a8
89.5100
88.2...
89.2
89.2:003
90.31:1
90.22:13
frog
89.6 89.61995
90.7.004
86.720.5
86.8..06
94.1
94.61933
89.7
89.
93.5
93.5
horse
93.31:06
95.7,0.3
91.21004
91.22:044
96.7100
97.0go.1
95.6.0.2
95.62.022
98.22.0.1
98. 18.1
ship
91.8
94.0....
78.0
78.0
95.0...
95.7.o.1
86.0.6.7
85.8g..6
96.5.00
96.510a
truck
86.0s06
93.0=0.3
91.020.6
91.0s006
94.9+0
94,9=0.3
90.620.0
90.7=0.9
95.2 95.2s1.3
95.1
mean
86.82a4
89,320.
83.421.
83.5
90.8
91,3s0.
89.0
89.0
92.5
92.4
Table 8: Per-class one-class classification AUCs on CIFAR-10.
Representation
RotNet [20]
Demoising
Roi. Prediction
Contrastive
Contrastive (DA)
Classifier
Rot, Cls
KDE
OC-SVM
KDE
OC-SVM
KDE
OC-SVM
KDE
OC-SVM
KDE
0
77.7 7.00
g0.
74.620.7
74.4gos
77.
78.8
79.9
79.9
83.0g1.1
82.9
33.0009
75.4=0.
71.8=1.6
71.8
75.2+0
77.9=1.1
=0.7
81.5
84.640.3
84. 84.3+04
72.9 9
10.
81 31.5
81 4g
84.4
85.7
87.4
87.5 2..44
88.5 5
88.610e
81.3=0.3
77.6+1.6
77.6+1.6
15.
87.3
84.7=0.8
84.7=0x
86.3
86.4
4
78.4+004
80.8=0.5
76.1=1.
76. 16.15:1.
81.
84.8=0.a
89.8=0.0
89.9=n.0
92.6+0.5
92.6#0.
73.
73.7=0.5
70.2+0.5
70. 70.2+0.5
72.0+1
78.6=1.0
82.
82.7=1.2
84.4+1.3
84.5+1.
6
89.8 89.8
90.4=0.4
68.5
68.4+2.5
88. $8.6220.3
87.5+0.4
77.0=1.
77.0=1.
73.4+0.9
73.4+0.
1=0.7
66. 10.6
7..62..4
73.0+1.0
76.2+0.G
80.5
80,4
84.3+0.5
84. 84.2+0.6
8
83 3
05.3=0.4
81.7=0.9
81. 01.6
87.8+0.1
86.4=022
85 25.2 0000
85.2 =.
87.5+003
87.7
91.4
91.7
73.7
73.
88.01.7
88.8.9.
89, 10.0
89.2:..
94.1
10
86. 1
87.0=0.0
58.6=2.2
58.5+2.2
79.5+111
82.4
60.0=2.0
59.8 89.002.
85. 15111
85.2+1.1
83.7
85.0
76.5
76.5
87.2.00
84.920.
84.3
84.2:00.
87 1
87.8
12
82. 8104
84.3
7..0
79. 111.1
85.8
85.2
84.4 10.5
84 44.4
82.011.
82.0
64.1
70.2
70.2
70.810.7
73.6220.
29.3
79.2:1.4
S2.7
82.7
14
89.7193
90.3.0.2
1..0
13.0
91.3100
90.2..
89.1
89.1
93
93.4
15
400.
70.6:00.3
71.220.7
71.1
73.4+0.
74.9=0.5
21,5
41.4
76.1
75.8
16
78.0
79.4_.0.
74.7g1.0
74.8
9
80.5.0g
79.3
79.3gm.2
80.4107
80.3
17
92.5+004
93,7=002
88.8
8:0.5
93.9+0.3 3
94,6+0.2
91.2=0.3
91.2+0.8
07.5+0.1
97.5
18
90.8
92.5=0.3
80.3+0.7
50.2
93.4+0.3
93,1
87.4
87.2=n.
94.420.2
94,4+0.
19
4.0.
90.3
78.020.5
77. 77.8
89.3 1
89,8+0.2
84.0=1.6
83.921.
92.5+0.7
92.4
80.3
81,9
75.2
75.2
82.810 6
14.
82.420.
82.420.s
86.5207
86.5
Table 9: Per-class one-class classification AUCs on CIFAR-20.
Representation
RotNer [20]
Denoising
Rot. Prediction
Contrastive
Contrastive (DA)
Classifier
KDE
OC-SVM
KDB
OC-SVM
KDE
OC-SVM
KDB
OC-SVM
KDE
83.7=22
94,5
91.8en.7
92.3
94.5
95.2
93.420.2
93.3
93.0 33.0aa.9
92.3+1.
96.2+1.9
99,5-20.1
47.4
97.7+0.4
99.4+0
99.7+0.1
98.1 20.2
98. 2...
09.2+0.4
78.1
93.8=0.2
91.6 91.620.4
91. 6.0.4
93.1
94.2
93.3
93.5=n.2
93.1
93.4
83.7az.
95.0:204
91.1a1.1
91.0a1.1
93.3
95.4
92.2
91 91.4
93.220.5
92.2
83.5#2.1
91,9=0.1
93.350.2
92.7s0.2
92.1
93,9-002
93,4
92.5=n.2
94.2+0.2
93.440.3
87.611.
94.3
94.910.4
94.7
95.1
97.3103
93.8
93.02m.2
94.3
94.3
77.3
83.4
86.1
85.51.0.1
82.510-
85.9104
88.
87.4.0.1
87.61002
86.2
96.9
99.2.:0.1
96.910.5
96.91u.5
98.7
99.4
97.3
97.02m.2
98.510.1
98.4
92.2
96.2.10.5
90.910.s
90.01...
98.5100
98.3103
92.3g1.2
92.311.2
97.6 97.61.03
97.6103
95.0+13
12..a
93.3
92. 00.0
99.0.a 1
98.6=0.2
96.9=6.3
96.9.n.3
97.9
98.0=0.a
87.4
94.620.3
92.720.5
92.5a.0.
94.622
95.8
93.9
93.620.3
94.8
94.5
Table 10: Per-class one-class classification AUCs Fashion MNIST.
20

---

Published as a conference paper at ICLR 2021
Representation
RotNet [20)
Demoising
Rot. Prediction
Contrastive
Contrastive (DA)
Classifier
KDE
OC-SVM
KDB
OCC-VM
OC-SVM
KDE
OC-SVM
cat
86 4
86, =00
41.3a1.7
41.2s1.8
81.9
84.
89.8.0.4
89.7.0.4
91 31.600
91.7
dog
85.8
86.6=0.2
60.6+111
60.3s1.0
85.5
88.7+0.4
85.7
85.7=0.5
87.5
87,5+0.
86. 110
86.4..02
51.011.4
50.711.4
83.7.0s
86.410
87.7.0.5
87.710.4
89.610s
89.61o4
Table 11: Per-class one-class classification AUCs Cat-vs-Dog.
A.6 MORE EXAMPLES FOR VISUAL EXPLANATION OF DEEP ONE-CLASS CLASSIFIERS
(a) Input
(b) Ours
(c) RotNet (d) RotNet" (e) DAE
(f) Ours
(g) RotNet (h) RotNet"
(i) DAE
Figure 12: Visual explanations on CelebA eyeglasses dataset. (a) input images, (b-e) images with heatmaps using integrated gradients [62], and (f-i) those using GradCAM [61]. RotNet": RotNet
KDE.
21

---

Published as a conference paper at ICLR 2021
(a) Input
(b) Ours
(c) RotNet (d) RotNer" (c) DAR
(f) Ours
(g) RotNet (h) RotNet" (i) DAE
Figure 13: Visual explanations on CelebA eyeglasses dataset. (a) input images, (b-e) images with heatmaps using integrated gradients [62], and (f-i) those using GradCAM [61]. RotNet": RotNet + KDE.
22

---

Published as a conference paper at ICLR 2021
(a) Input
(b) Ours
(c) RotNet (d) RotNet" (c) DAR
(f) Ours
(g) RotNet (h) RotNet"
DAE
Figure 14: Visual explanations cat-vs-dog dataset. (a) input images, (b-c) images with heatmaps
using integrated gradients [62], and (f-i) those using GradCAM [61]. RotNet": RotNet + KDE.
23

---

Published as a conference paper at ICLR 2021
(a) Input
(b) Ours
(c) RotNet (d) RotNet" (e) DAE
(f) Ours
(g) RotNet (h) RotNet"
(i) DAE
Figure 15: Visual explanations on cat-vs-dog dataset. (a) input images, (b-e) images with heatmaps
using integrated gradients [62], and (f-i) those using GradCAM [61]. RotNet": RotNet + KDE.
24

---

Published as a conference paper at ICLR 2021
B EXPERIMENTS ON MVTEC ANOMALY DETECTION DATASET
B.I DATASET DESCRIPTION
There are 15 different categories in MV'Tec anomaly detection dataset [31], where 10 object (e.g., bottle, cable, transistor) and are texture (e.g., grid, leather) categories. Each category comes with the training set containing ~211 images per category from normal data distribution and the test set containing 115 images per category from both normal and defective data distributions, The dataset also provides pixel-accurate annotation for defective regions, allowing to measure the defect localization performance. Images are of high-resolution, whose side length is as long as 1024.
B.2 EXPERIMENTAL SETTING
First, we resize all images into 256 in our experiments. One challenge of learning representa- tions from scratch MVTec dataset is that the amount of training set is too small (~241 images per category). As Figure 9, self-supervised representation learning methods benefit from the large amount of training data and the performance degrades when the amount of training data is lim- ited. Instead learning the holistic image representation, we learn a patch representation of size 32x32 via the proposed supervupe representation learning methods. Similarly to previous experiments, we train ResNet-18 from scratch.
For evaluation, compute both image-level detection and localization AUCs. We densely extract embeddings f from image patches with the stride of 4 and compute anomaly scores with KDE at each patch location, which results in (256-224 + 1) x 256-224 + 1) 57 anomaly score map. For image-level detection, we apply spatial max-pooling to obtain a single score. For localization, we upsample x77 into 256 256 with Gaussian kemee'' [67] and compare with the annotation.
B.3 EXPERIMENTAL RESULTS
CAVGA-RI R. [76]
RotNet [20. 21J
RotNet KDE
RotNet (MIP head)
Vanilla Contrastive
DistAug Contrastive
AUC
+ KDE
Detection object
83.8
77.912.3
85.9
89.0_2.
83,8
88.611.4
Detection texture
78.2
73.2#3.5
75.5=311
81.0=344
73.0=2.6
82,5+2.2
Detection all
81.9
71.0=as
83.5=a.
86.3=24
80.2=1.
86.5
Localization object
74.111.6
95.7.0.7
96.4_...4
91.711.
94.4
Localization texture
78.8=3.1
86.4=1.8
86.3=2.0
73.4=1.8
82.5+1.5
Localization all
89
75.6=2.11
92.6=1.0
93,0=0.9
85.6=1.3
90.4
Table 12: Image-level detection and pixel-level localization AUC results MVTec anomaly de- tection dataset. We run experiments times with different random seeds and report the mean and standard deviations. We bold-face the best entry of each row and those within the standard deviation.
We report quantitative results in Table 12. Specifically, we report detection and localization AUCs averaged over object, texture, and all categories. Likewise, we run experiments 5 times with dif- ferent random seeds and report the mean and standard deviation. We report the performance of RotNet (using rotation classifier, as in [20, 21]), the same representation but with KDE detectors as proposed, and RotNet trained with an MLP head and evaluated with KDE detector as proposed. In addition, we evaluate the performance of contrastive representations without (Vanilla) and with dis- tribution augmentation (DistAug) as proposed. For comparison, we include the results of CAVGA R, [76], which is also evaluated under the same setting. Below we highlight some points from our results:
1. The proposed two-stage framework, which trains RotNet and constructs one-class classi- fier, clearly outperforms an end-to-end framework, which trains the RotNet and uses a built-in rotation classifier [20, 21].
2. The proposed modification of RotNet with MLP projection head further improves the represen- tation of RotNet for one-class classification.
"To be more specific, apply transposed convolution 57 x7x57 score map using a single convolution kernel of size 32 x whose weights determined by Gaussian distribution and stride of 4.
25

---

Published as a conference paper at ICLR 2021
3. The proposed distribution augmentation (rotation) improves the performance of contrastive rep- resentations.
4. We improve the localization AUC upon [76] both in detection and localization, both on object and texture categories.
5. Both RotNet and contrastive representations are effective for "object" categories, but not as much for "texture" categories. As reviewed by [77], semantic and texture anomaly detection problems are different and may require different tools to solve. The methods relying on geo- metric transformations effective in learning representations of visual objects [46] and thus more effective for semantic anomaly detection, but less effective for texture anomaly detection.
B.4 QUALITATIVE RESULTS ON LOCALIZATION
input
groundtruth
localization
input
groundtruth
localization
Figure 16: Visualization of defect localization on bottle category MVTec dataset [31] using rep- resentations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
26

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
Figure 17: Visualization of defect localization cable category of MVTec dataset [31] using rep- resentations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
27

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
500
500
500
500
50
500
500
800
500
500
500
Partets
5.00 0
Cantvir
500
500
Facteos
Geten
500
Gatee
500
500
500
500
500
Figure 18: Visualization of defect localization capsule category of MVTec dataset [31] using representations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
28

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
Figure 19: Visualization of defect localization category of MVTec dataset [31] using rep- resentations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
29

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
Figure 20: Visualization of defect localization transistor category of MVTec dataset [31] using representations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
30

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
Figure 21: Visualization of defect localization grid category of MVTec dataset [31] using repre- sentations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
31

---

Published as a conference paper at ICLR 2021
input
groundtruth
localization
input
groundtruth
localization
Figure 22: Visualization of defect localization on leather category of MVTec dataset [31] using representations trained with rotation prediction on patches. From left to right, defective input data in test set, ground-truth mask, and localization visualization via heatmap.
32


Reviews:
[{"review_id": "p42RjN6OEcm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a two-stage representation learning approach to deep one-class classification.\n\nIn the first stage, a mapping f to a versatile high-level latent representation is learned using self-supervised learning for a contrastive learning proxy task. In the second stage, the same mapping f is used to map the data to the latent space, whereafter a traditional one-class classifier such as OC-SVM or KDE, is applied. \nIt is shown that the one-class task puts somewhat different requirements on the representation than with a multi-class classification task, both 1) in terms of uniformity of the data points in the representation, which is desired for multi-class tasks but not fully beneficial for one-class tasks, and 2) in terms of minimizing or maximizing the distance between different instances of the negative class - for multi-class tasks you want the distances maximized, while for one-class tasks you want the negative (inlier) examples close together. 1) is addressed by using smaller batch sizes in training while 2) is addressed by distribution augmentation that will render a compact inlier distribution in the representation.\n\nThis paper is overall a good paper that will be interesting to a certain audience at ICLR. \n+ It is well written, well motivated, with a clear argument and as far as I can see, technically correct. \n+ The experiments are well designed, valid and exhaustive, with comparison to a range of baselines as well as an ablation study. \n+ Moreover, the visual explanation of what the different representations have focused on is highly interesting.\n+ I appreciate the comprehensive grounding of the contribution in both new and old related work. The reference list contains all the relevant state of the art, as well as references to more classical work such as [13,14,29,47,53].\n\nThe paper is not highly seminal, but more incremental in nature, putting together and modifying existing methodology. However, since it is very well done, the work is absolutely worth acceptance. \n\nA criticism is that there are some repetition in the line of argument, for example between 2.1.2 second paragraph and 2.1.3 first paragraph. A more compact, e.g., section 2.1 would render more space for results which now have been pushed to the appendix to a large degree. Another suggestion for improvement could be to indicate more clearly in figure 1(b) that f is kept fixed in this step. This could be done e.g. with a different color of the f box in figure 1(b).\n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "gDmcRn5g1eB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for deep one-class classification (an example application being anomaly detection).  The basic idea is to combine self-supervised representation learning (eg through a proxy task such as rotation prediction or contrastive learning), with a classical approach to one-class classification, such as one-class SVM or KDE.  This is in contrast to existing methods for deep one-class classification that use simulated outliers to form a surrogate classification loss and then train end-to-end.  The paper further improves on the first stage of representation learning, by introducing modifications to contrastive learning to make it more appropriate for one-class classification.  The main insight is to introduce distribution augmentation, where geometric transformations of images, such as rotation, are treated as separate instances, to be separated from the original view.  This is motivated from the perspective of reducing uniformity of the inliers across the unit hypersphere, to allow for better separation from outliers.\n\nPositives:\n+ strong empirical results, with improved performance over existing methods for one-class classification\n+ validation of two stage framework, by showing improved performance with RotNet representation with KDE detector versus RotNet end-to-end [20]\n+ validation of improvements to contrastive learning for one-class classification, such as distribution augmentation, batch size selection, use of MLP project head\n\nMinor negatives:\n- I think the paper would flow a little better if the related work section was moved earlier in the paper, rather than coming only after the detailed description of the method.\n- In describing distribution augmentation and contrasting it with standard data augmentation for contrastive learning, it is clarified that the two sets of augmentations are disjoint.  I would it have found it helpful if the paper was explicit about which data augmentations were used for the contrastive learning, as this did not seem to be stated in the paper.\n\nOverall I found this to be a nice paper with strong empirical results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dBoT0EJ9J7m", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an anomaly detection approach that has two stages: a first stage for learning a feature representation and a second stage to train either a one-class classifier based on OC-SVM or KDE.  The main contribution of the paper is the feature representation learning that relies on contrastive learning to optimise a self-supervised loss function which minimises the distance of the samples from the same image augmented with different data augmentation functions and maximises the distance of samples from different images augmented with the same augmentation functions.  The data augmentation functions used were horizontal flip and rotation (0,90,180,270).  Results on the public datasets CIFAR-10, CIFAR-100, Fashion MNIST, and Cat-vs-Dog show that the proposed method has better anomaly detection (measured with AUC) than the state of the art.  The paper also displays qualitative anomaly detection results and an ablation study that shows: a) how close to uniform distribution (on hypersphere) the feature representations are as a function of batch size, and b) how AUC is affected with batch size and depth of MLP project heads.\n\nThis paper has outstanding results on the datasets CIFAR-10, CIFAR-100, Fashion MNIST, and Cat-vs-Dog, but it is missing result on a challenging dataset, such as Mvtec [51].  It is also missing results on anomaly localisation (e.g., Venkataramanan, Shashanka, et al. \"Attention Guided Anomaly Detection and Localization in Images.\" arXiv preprint arXiv:1911.08616 (2019)), so it scores slightly below acceptance for results given that it is hard to assess how the method would perform in a more realistic anomaly detection problem.  In terms of the proposed method, it is quite similar to [20,21], with the difference that it uses more data augmentation functions and rely on contrastive loss.  Therefore, it scores slightly below acceptance on novelty as well.  \n\nOne argument that seems contradictory is the one for class collision and uniformity.  In particular, if pre-training forces all inlier samples to be on a hyper-sphere, wouldn't it be advantageous to have a uniform distribution given that outliers could be easily detected as not lying on the hyper-sphere?  Of course, this would probably require a change in the OC-SVM classifier.  Can the authors comment on that?\n\nAlso, the argument on Sec. 2.1.3, on the effect of projection heads, says that \"I(g(f(x));x) <= I(f(x);x), so f can retain more information than g, thus more suitable for downstream tasks that are not necessarily correlated with the proxy tasks\".  If we push this argument, then I(f(x);x) <= I(x;x), so we should use x for downstream tasks.  Can the authors comment on that?", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Oa-hBrsupn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThey investigated the effectiveness of self-supervised learning (SSL) for one class classification (OCC).  \nHere is what I think are contributions relative to existing literature - \nEmpirically improved AUC for multiple OCC datasets, here are the techniques that were useful - \nUsed “distribution augmentation” [DistAug] for learning representation for OCC, and in ablation studies show DistAug leads to  improvement over standard augmentation\nUsed KDE and OCSVM on top of learned representation, and showed improvement over using the classification head training during SSL\nUsed a smaller batch size (=32) \nUsed a MLP head during SSL\nThe authors also included a section of visualizing explanation using existing techniques to illustrate how their method leads to more reasonable decisions.  \n\nStrength:\nThe paper is well written. I appreciate the clarity, and good coverage of the current literature.\nThe ablation studies are thorough, which make the empirical improvement solid. \n\nConcerns:\nThe uniformity argument is weak. The authors state the empirical improvement on OCC using their method hinges on the DistAug technique, which is motivated to reduce the uniformity of the learned representation. When achieved the inliers will live in the dense regions on the hypersphere, and outliers will live on the non-occupied region.  This assumes all the test inputs are projected onto the hypersphere, including the outlier.  From my understanding, the authors used f() for OCC, not \\phi() which is the normalized (i.e. hypersphere) output. In this case, there are many ways that OCC can be achieved even if \\phi() of the training inputs are uniform on the hypersphere.  Suppose  both the inliers and outliers after f() live on hyperspheres, just with a different radius, then  after normalization they can both be uniformly distributed  on  the  same hypersphere.  \nOne question is if there is a difference in using f()  or \\phi() for OCC. \nFurthermore, the authors try to back this claim up using Figure 4, but I cannot  seem to connect the dots here. \nThey authors used MMD to a uniform distribution to measure how uniform the representations are.  The less uniform (i.e. higher MMD), the better it should be for OCC.  The  correlation between MMD and AUC does not  seem  to be very strong.  E.g., for the (DA) gf variant, the 2 metrics actually  seem negatively correlated.  \nThis again, makes me wonder if “less uniformity” really is why their technique led to an improvement in OCC. \nIf this is not why, then we should find another explanation for why there was  an improvement. \nThere is always the concern that the improvement comes from extra hyperparameter tuning. Did the author also tune for good hyperparameters for the non DistAug version as described in A.3?  \n\n\nOverall, a fairly thorough empirical investigation into better techniques for using SSL for OCC.  It can be a decent contribution along the lines  of one of the  “improve techniques …”  papers if the above concerns can be addressed.  In fact, I think not focusing on selling DistAug, but really identifying what contributes to  the gain empirically makes this  paper stronger. \n\n\nReferences:\n[DistAug] Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In Proceedings ofMachine Learning and Systems 2020, pages 10563–10576, 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the one-class classification problem, proposing to learn a self-supervised representation and a distribution-augmented contrastive learning method; thorough results and analysis show that the method is effective and backs up their claims in terms of the underlying mechanism for why it works. In general, reviewers thought the paper was well-written, well-motivated/argued, and presents a thorough related work comparison and experimentation, though the novelty was found to be somewhat low. Several reviewers brought up some possible weaknesses in terms of demonstrating uniformity of the representations as well as suggesting additional datasets. Through an interesting discussion, the authors provided additional visualizations and results on the Mvtec dataset. This further bolstered the arguments in the paper. 

Overall, this is a strong paper with a clear argument and contribution, and so I recommend acceptance. 
Decision: Accept (Poster)