Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images
Authors: Philipp Liznerski
Lukas Ruff
Robert A. Vandermeulen
Billy Joe Franks
Klaus Robert Muller
Marius Kloft
Venue: Accepted by TMLR ()

Abstract: Due to the intractability of characterizing everything that looks unlike the normal data, anomaly detection (AD) is traditionally treated as an unsupervised problem utilizing only normal samples. However, it has recently been found that unsupervised image AD can be
drastically improved through the utilization of huge corpora of random images to represent anomalousness; a technique which is known as Outlier Exposure. In this paper we show that specialized AD learning methods seem unnecessary for state-of-the-art performance, and furthermore one can achieve strong performance with just a small collection of Outlier Exposure data, contradicting common assumptions in the field of AD. We find that standard classifiers and semi-supervised one-class methods trained to discern between normal samples and relatively few random natural images are able to outperform the current state of the art on an established AD benchmark with ImageNet. Further experiments reveal that even one well-chosen outlier sample is sufficient to achieve decent performance on this benchmark (79.3% AUC). We investigate this phenomenon and find that one-class methods are more robust to the choice of training outliers, indicating that there are scenarios where these are still more useful than standard classifiers. Additionally, we include experiments that delineate the scenarios where our results hold. Lastly, no training samples are necessary when one uses the representations learned by CLIP, a recent foundation model, which achieves state-of-the-art AD results on CIFAR-10 and ImageNet in a zero-shot setting.

Published in Transactions on Machine Learning Research (11/2022)
Exposing Outlier Exposure: What Can Be Learned From Few, One, and Zero Outlier Images
Philipp Liznerski' TŪ Koiserslautern
limnrskii@cs.unie www.eess...ii.....mm
Lukas Ruff Aignostics, Berlin
hukas.raffSaiinosticcc
Robert A. Vandermeulen"
ML Group TU Berlin, Berlin Institute for the Foundations of Learning and Data
Billy J. Franks
franks@cs. uni-ss.nikkkkee
TU Kaiserslauterv
Rebus- Müller
de
Google Research, Brain Team, ML Group TU Berlin, MPII, and Korea University
Marius Kloft
kloft@cs. uni-kl. de
TU Kaisersloutern
Reviewed on OpenReview: https: // opw.r/veew.eeee..e net/forun's 3y7
Abstract
Due to the intractability of characterizing everything that looks unlike the normal data, anomaly detection (AD) is traditionally treated as an unsupervised problem utilizing only normal samples. However, it has recently been found that unsupervised image AD can be drastically improved through the utilization of huge corpora of random images to represent anomalousness; technique which is known Oullier Exposure. In this paper show that specialized AD learning methods seem unnecessary for state-of-the-art performance, and furthermore one achieve strong performance with just small collection of Outlier Exposure data, contradicting common assumptions in the field of AD. We find that standard classifiers and semi-supervised one-class methods trained to discern between normal samples and relatively few random natural images are able to outperform the current state the art on an established AD benchmark with ImageNet. Further experiments reveal that even one well-chosen outlier sample sufficient to achieve decent performance on this benchmark (79.3% AUC). We investigate this phenomenon and find that one-class methods more robust to the choice of training outliers, indicating that there are scenarios where these are still more useful than standard classifiers. Additionally, we include experiments that delineate the scenarios where our results hold. Lastly, no training samples are necessary when one uses the reprosentations Icarned by CLIP, a recent foundation model, which achieves state-of-the-arl AD results on CIFAR-10 and ImageNet in a zero-shot selting.
1 Introduction
A Anomaly detection (AD) (Chandola et al. 2009 is the task of determining whether sample is anomalous compared to a corpus of data. Recently there has been a great interest in developing novel deep methods for
equal contribution.
Our code availahle https://github http:///gthab..cm///zerr/////// com/1isnerakiieed
Part of this work has been presented ICMI. 2021 UDL Workshop Ruff al. 2021b

---

Published in Transactions on Machine Learning Research (11/2022)
AD Ruff et al. 2021a Pang et al. 2021 Most prior work performs AD in unsupervised way utilizing only un anbleeled corpus of mostly normal data (Golan & Fl-Yaniv 2018 Hendrycks et al. 2019b Bergman et al. Tack et al. 2020). While AD can be interpreted as a classification problem of "normal vs. anomalous, it is classically treated as an unsupervised problem due to the rather tricky issue finding or constructing a dataset that captures everylhing different from the normal dataset.
2020;
One often has, in addition to normal data, access some data which is known be anomalous, Hendryeks et al. (0199a) noted that, for an image AD problem, one has access to virtually limitless amount of random natural images from the internet that are presumably not normal. They term the utilization of such auxiliary data Outlier Exposure (OE). Many top-performing AD methods on standard image AD benchmarks utilize tens of thousands of OE samples combined with sell-supervised learning (Hendrycks et al. 2019bb or transfer learning (Reiss et al. 2021 Deecke et al., 2021 to achieve state-of-thes detection performance.
For clarity, we here delincate throe basic approaches to anomaly detection: Unsupervised: These are methods trained on unlabeled data that assumed to be mostly normal. This is the classic and most common approach to AD.
Unsupervised OE: These are adaptations of unsupervised methods that incorporate auxiliary OE data that is not normal. Elsewhere this is also called "semi- pemiisuppervied" AD (Görnitz et al., 2013 Ruff et al. 2020 Supernised OF: This indicates standard classification methods trained to discern between normal data and an auxiliary OE dataset that is not normal.
Using unsupervised OE rather than supervised OE to discern between the normal data and OE samples seems intuitive since the presented anomalies likely do not completely characterize "anomalousness! Figure illustrates this classic intuition and highlights the differences between these approaches 2D toy dataset. The benefits of unsupervised OE when incorporating (a few specific) known anomalies has also been observed
in previous works TTx 2001
Görnitz et al. 2013
Ruff et al. 2020
normal
nomeel
Unfor Cinfortunately not true
(a) Train data: ideal
(b) Supervised OE on ideal
(c) Unsupervised OE on ideal
for deep A on on
mages
22
(d) Train data: skewed
(c) Supervised OE on skewed
(f) Unsupervised OE on skewed
Figure 1: This figure visualizes a deceptively reasonable intuition that one might have for deep AD. It shows the decision boundaries of a supervised OE method and an unsupervised OE method on two toy datasets: ideal (a-c) and skewed (d-f). The skewed scenario occurs naturally when there are not enough OE samples cover the ambient space, the data dimensionality is high, the OF data is clustered. Unsupervised OF (c f) learns compact decision region for the normal class that generalizes well in both scenarios. A supervisod OE approach (b + e), the other band, learns decision region that generalizes well in the ideal case where the outlier training data is fully representative, but not in the skewed case. While this intuition is true for shallow AD settings, like in this 2D toy example (a-f), our results suggest that a deep approach does not follow these phenomena for which supervised OE performs remarkably well.
2

---

Published in Transactions on Machine Learning Research (11/2022)
In this paper we present surprising experimental results that challenge the assumption that deep AD on images needs an unsupervised approach (with or without QE). Using the same OF setup as Hendrycks et al. (2019b), which is common in the literature, we find that a standard classifier outperforms current state-of-the-are art AD
methods on the one vs. rest AD benchmark with CIFAR-10 and ImageNet. The one vs. rest benchmark has
been recommended as a standard evaluation protocol to validate AD methods (Emmot1 el al.)
2013) and
is used as a litmus test in virtually all deep AD papers published at top-tier venues; s0e e.g-
Ruff al.
2018
Deecke et al.,
2018
Golan & El- Ylnivv
2018 Akcay et al. 2018 [enddyyks et al. 2019b Abati et al.
2019 Perera et al.
2019
Wang et al.) 2019a Ruff et al.) 2020 Bergman & Hoshen 2020 Kim et al.) 2020
Deecke et al.] 2021 Further challenging common assumptions, we find that OF does not seem to require huge amounts of data to represent "anomalousness." " A classifier requires only 256 random OE samples to outperform the state of the art on ImageNet and only one well-chosen OE sample to score reasonably compared to unsupervised methods and classical AD approaches.
Liznerski et al. 2021
OE, however, does not solve all types of AD problems, in particular when the normal datasct is highly diverse when anomalies are very subtle. For instance, we demonstrate that the methods need more OE samples to achieve top performance on the less studied, more challenging leave-one-clss-outt AD benchmark [Brgmann & Hoshen (0200); Ahmed & Courville (2020); Deecke et al. (2021) where many classos are combined to form a multimodal normal class. On the DTD (describable textures) one vs. rest benchmark and on MVTec-AD, a recent manufacturing dataset, we show that random natural images are not very informative as OF.
We also investigate transfer learning approaches to AD that have recently improved AD on images Reiss et al., 2021 Deecke et al.) 2021). Using CLIP (Radford et al.) 2021), recent foundation model, we find that it is possible to set state of the art on CIFAR-10 and ImageNet without any additional training data. While transfer learning and standard classification work well, we still find advantages of unsupervised OE over supervised OE When there are very few (< 32) OE samples or the OE samples are not very informative, unsupervised OE approaches outperform classifiers, indicating a certain robustness with respect to the training outliers. On low-dimensional datasets like Mashion-MNIST and MNIST, where shallow approaches without OE perform well, the intuition described in Figure [ is still valid: Standard classifier require a lot of OE data to perform well, whereas one-class methods perform competitively even with just one OE sample.
In conclusion, the primary message of this paper is neither that we propose yet another hato-of-tho-art method nor that one of our investigaled met hodhoologee is of general superiorily, but that there is surprisingly strong performance of off the-she-el classifiers, transfer learning, and just a few OE samples contradicting widespread common assumptions for deep AD on well-established AD benchmarks. Through this work, we want to encourage rethinking how previous AD results extend to deep learning.
2 Related work
We here briefly review recent developments deep AD, including self-supervision, transfer learning, and outlier exposure. We further clarify the differences between out-of-distribution detection and AD and discuss non-natural image AD benchmarks.
Deep anomaly detection While there exist many shallow methods for AD, it has been observed that these methods perform poorly on bigh-dimensional data (Huang & LeCun 2006 Kriegel et al., 2008 Erfani
et al.
2015 2016). To address this, deep approaches to AD that scale well with higher dimensions have been proposed (Ruff et al.] 2021a Pang et al., 2021). The most common approaches to deep AD employ autoencoders trained on normal data, where samples not reconstructed well at test time are deemod anomalous
(Hawkins et al.] 2002; Sakurada & Yairi 2014 Chen et al.). 2017 Zhou & Paffenroth| 2017 Nguyen al.,
2019
Kim et al.) 2020
Deep generative models detect anomalies via variety of methods (Schlegl et al.
2017
Deecke et al. 2018
Zenati et al. 2018 Schlogl et al. 2019), yet their effectiveness has been called into
question (Nalisnick et al. 2019).
A recent avenue of research uses sele-superiiiion for deep AD on images Gidaris et al.] 2018 Golan &
Yanivy 2018 Wang et al. 2019b Hendrycks et al.] (2099b Tack et al. 2020: Sohn et al. 2021 p. One of
the best-performing AD methods is the self-supervised approach from Tack et al. (2020), which combines
Hendrycks et. al. 2019bh'ss AD method with contrastive representation learning (Chen et al. 2020
Tack
3

---

Published in Transactions on Machine Learning Research (11/2022)
et al. (2020) train their network transformed normal data that it maps similar transformations of sample close together, while sufficiently distorted transformations and other samples are mapped away. The network then has to classify each sample's type of transformation as in (Hendrycks et al. 2019b). For a test sample, both the trained network's certainty on predicting correct transformations and the similarity of the sample with its nearest neighbor in feature space determine its anomaly score: the larger the certainly and similarity, the smaller the anomaly score.
More recently, transfer learning- based approaches to AD (Bergman et al., 2020 Reiss et al. 2021 Deecke
et al. 2021p that fine-tune supervised classifiers trained on ImageNet have shown to outperform self-supervised
methods such as Tack et al. (2020) on common benchmarks. To the best of our knowledge, |Reis et al. (2021
is the best performing AD method on CIFAR-10 that does not use OE (Hendrycks et al. 2019a]. Their
method fine-tunes a ResNet pre-trained on ImageNet on a deep one-class loss (Ruff et al., 2018) and applies
continual learning to avoid a feature collapse. Since they use ImageNet pre-trained models, they do not. validate their method on the ImageNot one vs. rest benchmark.
The state of the art on the CIFAR-10 10 benchmark, however, takes advantage of OE (Hendrycks et al. 2019a), which follows the idea of using large unstructured corpus of images as "auxiliary anomalies" during training. For example, Hendrycks et al. (0019b) use OE to improve their self-supervised method by training the network to predict the uniform distribution for all transforms on OF samples, while leaving training on normal samples unchanged. Reiss et al. (2021) and Deecke et al. 2021 combined transfer learning and OE, which yields the currently best performing methods.
Out-of-distribution detection and anomaly detection A field of research related to AD is out-of- distribution (OOD) detection, where the aim is to detect anomalous samples that do not belong to any of the given classes of a multi-class classification task (Lee et al. 2018). One can always apply an AD method to OOD by using it separately from the classifier, treating all training samples of all given classes as normal, ignoring the available class labels. However, AD methods typically (and expectedly) perform worse than specialized OOD methods that take advantage of the in-distribution labels and confidence scores of a trained classifier (Liang el al. 2018 Tack el al.) 2020; Heuddyykss el al.] 2022). Such methods define the anomaly score to be large when the maximum of the softmax outputs (Liang et al., 2018) or logits (Hendrycks et al., 2022 is small, f.c. when the classifier is uncertain about the classification of sample. Conversely to AD methods, which are applicable OOD problems, OOD methods cannot be applied to AD setups due to the absence of in-distribution class labels. Note that the type of auxiliary supervision via OE we utilize in this paper hence differs from the kind of supervision applied in OOD, as we do not discriminate between different classes of normality but only between normal samples and auxiliary outliers.
Non-natural image AD Recently there has been increasing attention on image AD on "non-natural" images (e.g. medical images or technical images from manufacturing), where anomalies tend to be more subtle. For example, the MVTec-A datasct consists of photos from manufacturing with, for instance, screws being normal and defective screws being anomalous (Bergmann et al. 2019). In this paper we instead focus on the common and well-established one vs. rest benchmarks with natural images, aiming to detect images of natural classes that are semantically different from the normal class. For other types of image data, random natural images from the web are likely not informative as OE We show this in Section 5.2 and Appendix for the example of MVTec-AD and, less prominently, for DTD. However, one can see that both transfer learning and OE can work well in other settings as many state-of-the-arr methods on MVTec-AD rely on one of these. For instance, Liznerski et al. (2021): |: Schlüter et al. [2021) : Li et al. (2021) employ OE in the form of synthetically generated anomalies and Defard et al. 2021] Gudovskiy et al. 2002 Roth et al. 2022 use
transfer learning-base based methods.
3 Methods
In this section, we introduce the methods that we will use for our experimental evaluation. We first motivate why AD typically follows an unsupervised approach and not viewed as binary classification problem. Afterwards, we introduce deep one-class classification as well as CLIP (Radford et al. 2021 for zero-shot AD.
4

---

Published in Transactions on Machine Learning Research (11/2022)
3.1 AD classification problem
Traditionally AD is understood as the problem of estimating the support (or level sets of the support) of the normal data-generating distribution. This is known as density level set estimation (Polonik) 1995 Tsybakov 1997p. This follows the assumption that normal data is concentrated whercas anomalies are not concentrated Schölkopf & Smola 2002). Steinwart et al.] (2005) remark that density level set est can also be interpreted as a binary classification problem between the normal and an anomalous distribution. Many classic AD mothods (e.g., kernel density cstimation or one-class SVMs) implicitly assume the anomalics to follow a uniform distribution, i.. they make an uninformative prior assumption on the anomalous distribution (Steinwart. et. al., 2005)- These methods, as well binary classifier trained to discriminate between normal samples and uniform noise, are in fact asymptotically consistent density level set estimators (Steinwart et al., 2005 Vert & Vert 2006). Practically, however, it is preferable to estimate the level set directly rather than classifying siing against uniform noise. Such a classification approach is particularly ineffective and inefficient in high dimensions since it would require massive amounts of noise samples to properly fill the sample space. As we show in our experiments, however, we find that this intuition does not seem to extend to deep anomaly detection on images.
3.2 Deep one-class classification
Deep onc-class classification (Ruff et al. 2018 was introduced as doep learning extension of the onc-class classification approach to anomaly detection [Schölkopf et al. 2001 Tax 2001 Deep SVDD (Ruff et al., 2018) is trained to map normal samples close to a center e in feature space, thereby following the concentration assumption (Schölkopf & Smola 2002) mentioned above. For a neural network Φa with parameters 0, the Deep SVDD objective is given by ming - (0,) - cll" Ruff et al. (2020) proposed an extension of Anomaly Detection (Deep SAD). Dcep SAD trains a network to concentrate normal data near the center c, while mapping anomalous samples away from that center. Hence, this follows an unsupervised OE approach to AD. Here, we present a principled modification of Deep SAD based on the cross-entropy loss, which we call hypersphere classification (HSC). We find that this modification improves performance over Deep SAD and use it in our experiments as prototypical representative of the unsupervised OE approach to AD. Potentially, the unsupervised OF approach by developing OF variant of CST TTack et al. 2020), which we leave to future work and is out of scope of this paper.
Deep SVDD that incorporates known anomalies, called Deep Semt--uperrsse
could further improve
Let D = {(=11,1)........................ (In, yn)} be a dataset with x; E Rd and y E f0, 1) where y = 1 denotes normal and y = anomalous instances. Let Φo Rdd -> Ii" be a neural network and 1: TR" -> [0, 1] a function that maps the output to probabilistic score. Then, the cross-entropy loss is given by
y log (dbo(a))) + (1-w) wi) log (1 (1=1()((x;)))-
(1)
For a standard binary classifier, is a linear layer followed by a sigmoid, and the decision region of the mapped samples (01)). .o....oo( (x,) half-space S. In this case, the preimage of S, (5'(S)) not guaranteed
be compact. To encourage the preimage of our normal decision region to be compact, we choose to be radial basis function: 1(z)= exp (- )|= In this case, becomes
Vi (1)0(2)))1 (1-v,) log (1- exp (- [1
If there are no anomalies, the HSC loss simplifies to 1 Ei (3))) For c 0, we thus recover Deep n
SVDD as a special case. Similar to Deep SVDD SVDD/SAD, we define our anomaly score as s(x) = [ Motivated by robust statistics (Hampel et al. 2005 Huber & Ronchetti 2009), we also considered replacing with radial functions that replace the squared-norm with more robust alternatives. Here, we found the pseudo Huber loss (Charbonnier et al. 1997) to consistently yield the best results. We refer to Appendix B for detailed analysis.
5

---

Published in Transactions on Machine Learning Research (11/2022)
3.3 Contrastive language-iman image pre-training
To challenge the assumption that transfer learning approaches require OE for stale-of-the-arl detection performance, we consider zero-shot approach to AD using the features of the contrastive language-maae pre-training (CLIP) model (Radford et al. 2021). CLIP is trained on a massive datasct of 400 million (image, text) pairs with an objective to align corresponding pairs in feature space while keeping other pairs apart. Let (Iy R.)) denote an (image, text) pair, = (ar.) and obtained by networks fu and fu, and consequently u, and v; the representations of the i-th data pair. CLIP uses the following losses: the text-to-image image loss ((0"") and the image-to-text loss 1." ((u-)))
(a())) the corresponding representations
(w)))) log exp((vi, ui) eT)
exp( ( Uz) e*)
4 ((-
log
(2)
Ex-repp( exp( ux) e*)
Lk=1 exp((u; uk) eT)
where (.--) denotes the cosine similarity and T is a temperature parameter. CLIP's final objective is
N
((0+))) + ((a-tu) 12.
min fu.fusT
(3)
N
2=
Radlord et al. [2021 report that, without any fine-tuning on the downstream lask, CLIP able to pur a fully supervised linear classifier with ResNet-50 features on several classification benchmarks, including ImageNet. For this, they use the names of the dataset classes as potential text candidates and predict the class whose text has the largest alignment with given image. For diitr---dittriuttoon detection, Fort et al. (2021) explored using CLIP by taking the in-distribution and out-of-distribution text labels as candidates. We use CLIP in a similar way to perform zero-shot AD, where we use the text pair (V1, 02) = ("a photo of a [NORMAL CLASS)" "a photo of something"). For a lest image æ,, we compute its anomaly score as
exp((fu(()) 100)
s(x) Ei exp((fu (vx)) 100)
(4)
Fine-tuning CLIP for AD with OE also is straightforward. We simply minimize the score 4 for normal samples and maximize it for OE samples. Since this corresponds to binary cross-entropy loss, this is an instance of supervised OF, which we term "BCE with CLIP" (or just BCF-CI.).
On the legitimacy of transfer learning for AD The use of transfer learning improved the performance of deep AD approaches significantly. Yet, it seems at least questionable whether the of pre-trained models is experimentally sound since there may be semantic overlap between the pre-training data and the anomalies seen at test time. While it is technically true that the AD model is still unsupervised and does not exploit. knowledge of test. samples, the reported performance on typical image AD benchmarks might be spurious as the model may not gencralize well to other data. Radford et al. 2021), however, have investigated the overlap data used for pre-training CLIP, where they remove all the data that overlaps with the downstream tasks and observe only an insignificant drop in performance on average. This suggests that experiments and results based on CLIP reasonably explore generalization performance. We still want to raise awareness for this somewhat problematic trend in deep AD, however, for which we propose future research below.
4 Experimental setups
Before we present our results, we explain the experimental setup. In particular, we introduce the common one vs. rest benchmark, the CIFAR-10, ImageNet, CUB, DTD, Pashion-MNIST, and MNIST datasets, and the state-of-the-art AD methods we consider in our experiments.
One vs. rest benchmark The one vs. rest evaluation protocol is a ubiquitous benchmark in the deep AD literature (Ruff et al. 2018 Golan & Ylniv 2018 Hendrycks et al. 2019a Ruff al. [2020] Sohn et al.
2021 Deecke et. al. 2021
Liznerski et al. 2021 Reiss et. al.) 2021 This benchmark constructs AD settings

---

GT (Golan & Published in Transactions on Machine Learning Research (11/2022)
from classification datasets (e.g., CIFAR-10) by considering the "one" class (e.g., "airplane") as being normal and the "rest" classes (e.g., "automobile", "bird", ...) as being anomalous at test time. In each experiment, we train a model using only the training set of the normal class and samples from an OE set that are not contained in the anomaly classes of the benchmark. We use the same OE auxiliary datasets as suggested in previous works (Hendrycks et al. 2019a b Liznerski et al.) 202T). To evaluate detection performance, we consider the commonly used Area Under the ROC curve (AUC) on the one vs. rest test sets. This is repeated over classes and multiple random seeds.
Datasets For our experiments we focus on the well-established one vs, rest versions of CIFAR-10 and ImageNet. We also consider less common datasets, for which there are yet no AD results in the literature. If not mentioned otherwise, we use all available classes as our one vs. rest classes.
CIFAR-10: For CIFAR-10 (Krizhevsky et al. 2009), we use 80 Million Tiny Images (80MTT)((orra))) (Torrllb et al. 2008) as OE, with CIFAR-10 and CIEAR-100 images removed. This follows the experimental spt in Hendrycks et al. 2019b].
ImageNet-30: For ImageNet (Deng et al. 2009); we use subset of 30 classes as the one vs. rest classes, For OE, we use ImageNet-22KK with (019b)).
which are the samn classes used in Hendrycks et al. (2019b).
ImageNet-1K KK removed, again following Hendrycks et al.
2011 is more challenging dataset where each of the 200 classes consists of images of a specific bird type (e.g.. "black footed albatross", "blue jay", ...).
CUB: CUB (Caltech- UCSD Mards-200-2011) (Wah et al.
We again use ImageNet-222) as OE.
DTD: DTD (Describable Textures Dataset) (Cimpoi et al. 2014 is a dataset containing 47 different classes of images of textures like "cracked" and "braided". We use ImageNet-227 as OE MNSSI (Xiao et al. 2017), we consider a grayscale version of CIFAR-100 as OF,, as suggested in Liznerski et al. 2021 }.
Fashion-MNISTA For Fashion- MNIST
MNIST: For MNIST (Deng) 2012
we use EMNIST Cohen et al.
2017 as OE.
End-to-end methods We present results from end-to-end methods (without transfer learning) including all methods that achieve statc-of-tho-art performance on the CIFAR-10 and ImageNNet-30 one vs. rest benchmarks.
Unsupervised: Shorthands for unsupervised methods are DSVDD (Rulf et al. 2018
El- Yaniv 2018), GT+ (Hendrycks et al., 2019b), and CST Tack et al. (2220)
2020 as unsupervised OE Unsupervised OE: We implement HSC from Section 3.2 and DSAD (Ruff et al.
methods and also report the results from the unsupervised OE variant of GT+ (Hendrycks et al. 2019b). Supervised OF: BCE denotes standard binary cross-entropy classifier. We also implement the Focal loss classifier with 2 (Lin et al. 2017/, BCE variant for imbalanced classes that was also presented in Hendrycks et al. 2019b). Results from Hendrycks et al. (019bb) are marked with asterisk as Focal*,
Transfer lcarning- based methods We consider the following transfer learning-based methods
Unsupervised: We implement a zero-shot anomaly detector using CLIP's 's feature space as described in Section 3.3 and use DN2 (Bergman et al. 2020) and PANDA (Reiss et al. 2021 as shorthands respectively for these unsupervised methods from the literature.
Supervised OE: We consider a finc-tuned version of CLIP with binary cross-cntropy classifier, denoted as BCE-CL. When available, we also report the results of ADIP (Deecke et al. 2021 and the supervised OE variant of PANDA (Reiss et al. 2021
We provide all network architecture and optimization details in Appendix G and investigate the impact of y for the Focal loss in Appendix [F] We report the mean AUC performance over all classes and sceds in the main paper. Individual results per class and method are given in Appendix || Each experiment is averaged over 10 seeds if not mentioned otherwise.
5 On the usefulness of samples in deep AD
Traditionally, AD methods utilize an unsupervised approach due to the assumption that it is impossible to characterize everything that is not normal. As mentioned above, later works introduced the idea of including large collection of random images (OE) during training that serve as auxiliary examples of anomalousness to
7

---

Published in Transactions on Machine Learning Research (11/2022)
improve unsupervised AD methods (Hendrycks et al. 2019a). Models are trained with these OE samples using loss that is essentially inverting a given unsupervised AD loss. We here look into whether an unsupervised OE approach, instead of a straightforward supervised OE approach using standard binary cross-entropy, is really necessary.
5.1 Supervised OE achieves soate-of-thheart results
The first work on OE (Hendrycks et al. 2019a applied an unsupervised OE method to two experimental setups: CIFAR-10 with 80MTI as OE and ImageNet-33 with ImageNet-22K as OF. Here we consider the same experimental setups with the basic HSC and BCE classifiers. We also include experiments on the mostly unexplored one vs. rest versions of CUB, DTD, Fashion-MNIST, and MNIST. The results are shown in Table In this section we only consider end-to-end methods (i.e., no transfer learning).
Table 1: Mean AUC detection performance in % for end-to-end methods on the CIFAR-10, ImageNet-30, CUB, DTD, Fashion-MNISS and MNIST one vs. rest benchmarks.
Unsupervised GT+*
Unsupervised OE
Supervised OE
DSVDD
CSI*
GT+
DSAD
HSC
Focal*
Focal
BCE
CIFAR-10
64.8*
90.1
94.3
95.6
94.5
95.9
87.3
95.8
96.1
ImageNet-30
61.1
84.8
91.6
85.7
96.7
97.3
56.1
97.5
97.7
CUB
59.0
X
X
81.1
83.2
X
83.5
84.1
DTD
56.8
X
X
72.4
72.7
X
73.2
73.3
Fashion-MNIST
86.3
X
X
86.4
87.3
x
87.7
86.4
MNIST 97.6
X
97.6
98.2
X
98.4
98.4
Discussion Surprisingly, we find that the choice of AD method has little impact on performance. On CIFAR-10, all unsupervised OF methods yield comparable detection performance, while the supervised methods (Focal and BCE) show state-of-the-art performance, with BCE attaining the overall best mean AUC. On ImageNet, Deep SAD, HSC, Focal, and BCE all outperform the current state of the art (CSI*) by significant margin. We are unsure as to why the Focal* performs so poorly in Hendrycks al. (2019b) since their experimental code is not publicly available. On CUB and DTD, observe similar behavior, but the methods perform slightly worse, most likely due to the more fine-grained class categorization in CUB and the more subtle anomalies in DTD. These benchmarks, as they show room for improvement, thus present themselves useful for future AD evaluation. On the rather low-dimensional Fashion-MNIST and MNIST datasets, OE is not able to improve significantly over unsupervised approaches, and the different OE approaches again perform similarly. Overall, our results show that a vanilla. classifier using OE outperforms all previous deep AD approaches.
Our experiments seem to suggest that the inclusion of OE does not just improve AD performance, it also changes the problem into typical supervised classification problem that does not require compact decision
boundary (cf. Figure 1. This stands in contrast to previous observations in shallow AD (Tax 2001
Görnitz
ett al. 2013). While there is an abundance of OE data for the sorts of AD problems we investigate here, we want to highlight that this data likely not very helpful for AD problems where anomalies are more subtle, For example, in the realm of manufacturing, pictures of cats and trucks are not as useful for OE as industrial images, which are not so widely available. Nonetheless, one may still have few anomalous examples to incorporate during training. As one removes OE data, would expect the behavior of unsupervised OE approach to transition to the behavior of an unsupervised method that outperforms supervised OE, Our next experiment investigates this transition.
5.2 Few OE samples are sufficient
To investigate the effect of the size of the OE corpus, we perform experiments varying the OE training set size from just one sample (2 1) to using the maximal amount. of OE data. Our results for the
8

---

Published in Transactions on Machine Learning Research (11/2022)
high-dimensional datasets CIFAR-10, ImageNet-30, and CUB are shown in Figure With sufficiently few samples, HSC outperforms BCE, which seems to indicate regime where the unsupervised OE approach is advantageous. Interestingly, this regime seems to be quite small, with supervised OE needing only samples to outperform unsupervised OE on CIFAR-10 and only ~ 32 samples for such a transition on ImageNet and CUB. Remarkably, BCE classification out purperorms previous state-of-the-art methods on ImageNet with only using 256 OE samples. A training set with so few outliers and high-dimensional data represents an instance of skewed data (see Figure and caption), where the common intuition suggests that supervised OE should not generalize well. As before, as sufficient OE data is available.
that the choice of the specific method seems negligible soon
1.0
2.0
0.8
3.6
0.
20 21
13
14
(a) CIFAR-10 (over classes 10 seeds)
(b) CUB (over 200 classes 2 seeds)
1.0
0.9
0.8
0.7
0.6
0.5
HSC
10
13
14
17
2
(c) ImageNet-30 (over 30 classes seeds)
Figure Mean AUC detection performance in % on the CIFAR-10, CUB, and ImageNet-30 one vs. rest benchmarks when varying the number of 80MTI and ImageNet-22KK OE samples, respectively.
Results with varied OE set size for Pashion-MNIST and MNIST are presented in Figure 3 contrast the experiments on high-dimensional datasets, the amount OE has little impact on performance for the well-performing HSC, whereas BCE requires lot of OE to perform competitively. It seems that the intuition of Figure does still hold for low-dimensional datasets where shallow methods perform well.
We find that, in
3.
0.8
25
IGC
0.
(a) Rishion-MNIST (over 10 classes x seeds)
(b) MNIST (over 10 classes 5 seeds)
Figure 3: Mean AUC detection performance in % on the FNSHI and MNIST rest benchmarks when varying the number of grayscale CIFAR-100 and EMNIST OE samples, respectively.
9

---

Published in Transactions on Machine Learning Research (11/2022)
There are settings that require more OE data The one rest image-AD benchmark represents certain type of typical AD problem where the normal data distribution is roughly unimodal since the normal samples are drawn from just one class. However, one can also create more challenging, and perhaps less typical, AD benchmark by considering the "rest" classes (c.g., "automobile", "bird", ...) as being normal and the "one" class (e.g.: "airplane") as being anomalous at lest time. The distribution of normal samples this leave-one-class-out approach becomes multimodal. Figure f shows the AD performance of HSC and BCE on the CIFAR-10 and ImageNet-30 leave-one-classout image-AD benchmark, when one varies the size of the OE training set.
10
15
10
11
(a) CIFAR-10 (over 10 classes seeds)
(b) ImsgeNet-30 (over 30 classes 2 seeds)
Figure Mean AUC detection performance in % on the CIFAR-10 and ImageNet-30 leave-one-clas--out benchmarks when varying the number of 80MTI and ImageNet-22N1 22K OE samples, respectively.
Compared to previous experiments it seems that more OE data is required to achieve strong performance. This indicates that more OE samples are necessary when the normal class is not concentrated. We also report results when one uses the full OE dataset in Appendix L3 where we see that the various methods perform similarly with respect to one another as in the one vs. rest tasks, albeit overall slightly worse due to the more difficult problem setting. Notably BCE is the best performing method, further supporting its effectiveness at AD with OE.
There are settings where random images are not informative as OE As mentioned in Section 2 this paper focuses on natural images because random natural images are likely not informative when used as OE in other settings. To demonstrate this, we apply our methods the one vs. rest DTD benchmark and the AV dataset. DTD contains different classes of textures. MVTeecAD contains image-AD scenarios for detecting manufacturing defects for varicty of object types (c.g., screws, bottles, wires, sections of carpet). For example, one scenario consists of training set containing images of normal screws and test set containing images of normal screws along with images of screws with defects, which serve anomalies. Figure 5 shows results when varying the number of ImageNet-225 samples used as OE for training. We find that the OE training set size has little impact on the AD performance, especially for MVTec-AD.
(a) DTD (over classes x seeds)
(b) MVTec-AD (over classes 2 seeds)
Figure 5: Mean AUC detection performance in % on the DTD one rest and on the MVTec-AD benchmarks when varying the number of ImageNet-22K OE samples.
10

---

Published in Transactions on Machine Learning Research (11/2022)
We also show results using the full ImageNet-222 22k dataset as OE for various methods for MVTec-AD in Table 18 in Appendix I.4 As expected, OF performs more poorly on MVTec-AD than state-of-the-art methods: HSC scores around 70% while Roth et al. 2002)) score around 99% AUC. Contrary to the experiments with CIFAR-10, CUB, and ImageNet-30, we obscrve that HSC is a bit stronger than BCE (66%). There seems to be some evidence that HSC performs better when the OE data is not very informative, either when there a
dearth of OE data like in Section 5.2. or when the OE data simply isn't relevant to the AD scenario. We explore this a bit further in Section 5.4
Diversity of OE data is important To measure the impact of data diversity we also experiment with varying the number of OE classes, We defer these results to Appendix A In summary, we find that performance overall increases with OF data diversity, but already using just one OF class still performs relatively well.
While our results show that surprisingly few samples are needed to achieve competitive performance for end-to-end models on the standard one vs. rest AD benchmarks, more recent methods in deep AD tend to use transfer learning. Interestingly, we find that one can achievo statc-of-tho-art AD performance using pre-trained models with no Iraining sumples (i.e., in a zero-shot setting).
5.3 Transfer learning enables zero-shot AD with state-of-the-ard performance
More recent progress in deep AD has been achieved through transfer learning, which has further improved the state of the art standard deep AD benchmarks. These methods use network pre-trained on large datasets to provide rich representations as starting basis for deep AD. Again, we look into how useful auxiliary data is for these algorithms. Here we investigate the effect of using OE data (vs. no OE data at all) and the implicit use of an extrancous dataset via pre-training. We also consider the situation where no normal data used and find that CLIP outperforms all previous end-to-end methods on virtually all datasets. Table 2 shows the results for transfer learning-based methods on the one vs. rest benchmarks. Note that, since DN2, PANDA, and ADIP employ ImageNet pre-trained networks, they cannot be compared on the ImageNet AD benchmark in fair way.
Discussion These results highlight the remarkable efficacy of pre-training. Disre- garding the CLIP results for now, we see that pre-training improves over previous unsupervised deep results on CIFAR-10. The results of PANDA indicates that OE is still useful for deep AD: OE provides addit ional information that not learned in the pre-training task. Undoubtedly, the most interesting result here is the observa- tion that can outperform all previous state-of-the-art methods using no addi-
Table 2: Mean AUC detection performance in % for methods with transfer learning on the CIFAR-10, ImageNet-30, CUB, DTD, Fashion-MNIST,, and MNIST one vs. rest benchmark.
Unsupervised PANDA*
Supervised OE ADIP*
DN2*
CLIP
PANDA
BCE-CL
CIFAR-10
92.5
96.2
98.5
98.9
99.1
99.6
ImageNet-30
X
X
99.88
X
X
99.90
CUB
X
X
97.1
x
X
97.5
DTD Fashion-MNIST
X
90.2
X
X
94.6
X
89.0
X
X
94.7
MNIST
X
59.0
X
X
96.0
tional training from the benchmarks. In a zero-shot setting, CLIP (unsupervised) outperforms all previous end-to-end (see Table I and unsupervised methods on all datasets apart from MNIST. Fine-tuning CLIP with OF along with normal samples (BCF-CT.), further improves CLIP's results setting a new state of the art on the CIFAR-10 and ImageNet benchmarks. Remarkably, CLIP performs similarly on DTD that consists of textures instead of natural images and, on CUB, outperforms end-to-end methods by a large margin (13.4% AUC), essentially solving this quite challenging benchmark.
Similar to results in other areas of deep learning (Bommasani et al. 2021p; the use of large pre-trained networks offers an effective and convenient way to improve performance. Though transfer learning seems to be a natural endpoint for a certain class of deep AD problems, it still leaves a more general question about the difference between supervisod and unsupervised approaches to deep AD in sottings where transfer learning is not appropriate. This may happen when there are very subtle semantic novelties (Vaze et al. 2022). : when one simply cannot use pre-trained network; for example, when one must train network from scratch due
11

---

Published in Transactions on Machine Learning Research (11/2022)
to architectural considerations (e.g., when one requires smaller architecture due to hardware constraints). when there are security concerns regarding the white-box nature of pre-trained networks (Samek et al. 2021 p. Additionally, some AD techniques offer no obvious way to utilize a pre-trained network, like the recently introduced explainable one-class variant (Liznerski et al. 2021 or methods based on probabilistic modols.
5.4 On the robustness of HSC vs. BCE
Our previous experiments (Section 5.2 have shown that, though end-lo-end BCE overall outperforms HSC, an unsupervised OE approach is more effective when only very few (< 32) OE samples are available or when the OE data is not very informative. This indicates a cortain degree of robustness to the anomalous training samples for HSC. This robustness with regards to OE likely inherited from the learning objectives of unsupervised and semi-supervised AD, which encourage the normal representations to be concentrated thereby avoiding the issue with skewed OE data (see Figure D So, even high-dimensional datasets, there seems to be a regime, albeit a very small one, where the intuition with skewed data in Figure does hold.
To demonstrate the robustness of HSC when one has few OE samples we investigate the extreme case where an OE dataset consists of only one sample in more detail. How much can a single sample help or hinder the different ap- proaches to deep AD? This is analogous to the experiments in Section 5.2 with the number of OF samples fixed to one. To investigate robustness, we scarch for the OE sample that gives the worst test AUC for each class and report the average AUC over all classes. While it is desirable for a method to be unaffected by detrimental OE examples, one would still want an AD method to exploit beneficial OF examples. To measure this trade-off, we perform an analogous experiment where we search for the best. performing OE sample and again report the average test AUC over all classes.
Table 3: Mean AUC detection performance in % for the best and worst single OE samples on the CIFAR-10 AD benchmark with 80MTI OE and on the ImageNet- 10 AD benchmark with ImageNet-22KK (without the 1K classes) as OF.
CIFAR-10 HSC
ImageNet-10 HSC BCE
BCE
Best OE Worst OE
77.7
69.9
79.3
75.5
43.3
31.6
39.2
26.3
As it is computationally prohibitive test every possible OE sample, we utilize an evolutionary algorithm attempt to minimize or maximize the class AUC. A detailed description of the algorithm can be found in Appendix To ensure that this optimization scheme does not find poor local minima, we also evaluate the AUC of many randomly chosen OE samples (see Appendix DD. We find that the evolutionary algorithm almost always finds better optima. We report our results for CIFAR-10 and for the first 10 classes of ImageNet-30 Table 3
Figure f shows optimal OE examples on CIFAR-10 for the normal classes "ship" and "cat," and on ImageNet-Il 10 for "airplane" and "dragonfly."
NOTTH
OOTT
wozst
WO1S
(a) "ship" is normal
(b) "cat." is normal
(c) "airplane" is normal
(d) "dragonfly" is normal
Figure 6: OE samples for CIFAR-10 with 8OMTI as OE (a-b) and for ImageNet-10 with ImageNet-222 as OE (c-d). The first row shows normal samples, the next two rows show the best samples for HSC (top) and BCE (bottom), and the last two rows show the worst samples for HSC (top) and BCE (bottom).
12

---

Published in Transactions on Machine Learning Research (11/2022)
Discussion On both datasets, we observe that HSC performs better than BCE when using both the best and the worst QF samples. Looking at the samples chosen for the optima, there also appears to be more consistency within a setting (class, dataset, best or worst) for HSC. For example on "best" with "ship," " HSC's images are all brownish outdoor photos, whereas the BCE samples vary from greenery to stylized text to an image containing mostly sky. This likely due to the fact that HSC has, in some sense, an initial notion of anomalousness due to its unsupervised term. For instance, the most useful OE samples are those not already contained in this notion of anomalousness, resulting in HSC having stable region for selecting OE samples that yield the greatest improvement. BCE lacks this notion so it can benefit from large variety of OF samples. With HSC, choosing one optimal sample achieves roughly the same performance as using 32 random ones. Interestingly, it seems that near-distribution outliers are less useful as OE samples since samples with similar color patterns as the normal ones occur more often as the "worst." samples. Previous works have found that near-dittrbuttin OE samples are useful for OE (Lee et al. 2018 Goyal et al. 2020), however, our results suggest that this may not hold when little OE data is available.
HSC focuses on low frequency features To gain further insight into the difference between BCE and HSC, we extend the previous experiments to include frequency-domain corruptions. This sort of analysis has been insightful in other works on deep learning (Yin et al.) 2019). We find that HSC is again more robust than BCE since it's generally less affected by the frequency corruptions and tends to focus on low frequency signals in the input. We defer the results and discussion to Appendix B and also show some more examples for the best and worst single OE samples in Appendix !
6 Broader impact
Anomaly detection methods on images may be deployed on tasks which have societal implications such as screening images or automated surveillance, and it is thus imperative that these tasks are done in a fair and transparent way. The use of OE is potentially harmful as there may be OE images biasing the model towards detecting certain entities as anomalous. Our paper aids in this since it demonstrates that no huge corpora are necessary, which enables controlled selection of OE samples. Further, we have shown that HSC is more robust, and the fact that it chooses optimal OE samples that coincide with human intuition suggests that it is more interpretable than BCE, where the rationale for optimal OE samples is opaque. This makes HSC more suitable for critical applications, We trained some of our models with the 80MT1 dataset, which is known to contain problematic data such as offensive labels, but were required to do so to be comparable with the previous line of research.
7 Conclusion
We presented surprising results that challenge common assumptions in AD. Neither does deep AD on natural images seemn to require specialized AD methodologies nor huge amounts of Outlier Exposure (OE). A standard classifier outperforms all end-to-end methods on the common one vs. rest benchmark, for which only needs 256 random OE samples on ImageNet and only one well-chosen OE sample for competitive performance. We showed some limitations of the few-OE approaches when applied to settings where the normal dataset is multimodal or the anomalies are more subtle. Using transfer learning, standard classifiers set a new state of the art on CIFAR-10 10 and, in a zero-shot setting without using any additional training data, on ImageNet. Despite the overall strong performance of standard classifiers, we find that semi-supervised one-class methods are more robust to the choice of OE when only few OE examples are available. Our results provide insights about deep AD that are useful for future research.
8 Acknowledgements
PL, BJF, and MK acknowledge support by the Carl-Zeiss Foundation, the German Research Foundation (DFG) awards KL 2698/22 /2-1 and KL 2698/5-1, and the German Foderal Ministry of Education and Rescarch (BMBF) awards 01$$18051,, B300077 and (1/12000 RV acknowledges support by the Federal Ministry of Education and Research (BMBF) for the Berlin Institute for the Foundations of Learning and Data
13

---

Published in Transactions on Machine Learning Research (11/2022)
(BIFOLD) (011S18037A). This work was supported in part by the German Ministry for Education and Research under Grant Nos. 01IS14013A-F, 01GQ1115, 01GQ0850, 01IS18025A 031L0207D, and 01TS18037A. KRM was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea Dovernmen((MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea Universily and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Pla, Explanation and Verification of Explanation).
References
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In CREE/CVE Conference on Computer Vision and Pattern Recognition, pp. 481-490, 2019.
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In AAAT Conference on Artificial Intelligence, pp. 3154 3162, 2020.
Samet Akcay, Amir Abarghou,, and Toby P Breckon. GANomaly: Semi-supervised anomaly detection via adversarial training. In Asian Conference on Compuler Vision, pp. 622 637, 2018.
Liron Bergman and Yedid Hoshen. Classification-aased anomaly detection for general data. In International Conference on Learning Representations, 2020.
Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. orXiv preprinl arXiv:2002.104 10445, 2020.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mytec ad a comprehensive real-world dataset for unsupervised anomaly delection. In Proceedings of the IEEE/CVE conference on compuler vision and pattern recognition, pp. 9592 9600, 2019.
Tobias Blickle and Lothar Thiele. A mathematical analysis of tournament selection. In ICGA, volume 95, pp. 9-15. Citescer, 1995.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohgg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Xi preprint arXic:2108.0777 07258, 2021.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM Computing Surveys, 41(3):1-58, 2009.
Pierre Charbonnier, Laure Blanc-Féraud, Gilles Aubert, and Michel Barland. Detorministic edge-presorving regularization in computed imaging. IEEE Transactions on Image Processing, 6(2):298 311, 1997.
Jinghui Chen, Saket Sathe, Charu C Aggarwal, and Deepak S Turaga. Outlier detection with autoencoder ensembles. In SIAM International Conference on Data Mining, pp. 90-98, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive Icarning of visual ropresentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEFE Conf. on Computer Vision and Pallern Recognition (CVPR), 2014.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks, pp. 2921 2926, 2017.
Lucas Doecke, Robert A Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in. Databases, pp. 3 17, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Lucas Deecke, Lukas Ruff, Robert A Vandermeulen, and Hakan Bilen. Transfer-bass based semantic anomaly detection. In International Conference on Machine Learning, pp.
PMLR, 2021.
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: a patch distribution modcling framework for anomaly detection and localization. In International Conference on Pattern Recognition, pp. 475 489. Springer, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In IEEE/CVE Conference on Compuler Vision and Pollern Recogrition, pp. 248 255, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141 142, 2012.
Andrew F Emmott, Shubhomoy Das, Thomas Dicttorich, Alan Fern, and Weng-Keen Wong. Systematic construction of anomaly detection benchmarks from real data, KDD 2013 Workshop on Outlier Detection and Description, pp. 16 21, 2013.
Sarah Erfani, Mahsa Baktashmotlagh, Sutharshan Rajasegarar, Shanika Karunasekera, and Chris Leckie. RISVM: A randomised nonlinear approach to large-acale anomaly detection. In AAAI Conference on Artificial Intelligence, pp. 432 438, 2015.
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High-dimensional and large-scale anomaly detection using linear one-class SVM with deep learning. Pollern Recognalion, 58:121-134 2016.
Stanislav Fort, Jie Ren, and Balaji Takshminarayanan. Exploring the limits of out-of-distribution detection. Advances in Newral Information Processing Systems, 34, 2021.
Félis-Antoine Fortin, Frangois-Michel De Rainville, Marc-André Gardner Gardner, Marc Parizcau, and Christian Gagné. Deap: Evolutionary algorithms made easy. The Journal of Machine Learning Research, 13(1):2171-2175 2012.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation ion learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Aduances in Neural Information Processing Systems, pp. 9758 9769, 2018.
Nico Görnitz, Marius Kloft, Konrad Ricck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235 262, 2013.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In International Conference on Machine Learning, pp. 3711-3721. PMLR, 2020.
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 98 98-107, 2022.
Frank Hampel, Elvezio M. Ronchetti, Peter J Rousseeuw, and Werner Stahel. Robust Statislics: The Approuch Based on Influence Functions. John Wiley & Sons, 2005.
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier detection using replicator neural networks. In International Conference on Data Warehousing and Knowledge Discovery, volume 2454, pp. 170 180, 2002.
Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning improve model robustness and uncertainty. In Advances in Newral Information. Processing Systems, pp- 11637-15644 2019b.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Stcinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world scttings. In International Conference on Machine Learning, pp. 8759 8773. PMLR, 2022.
Fu Jie Huang and Yann LeCun. Largo-scale learning with SVM and convolutional nets for generic object categorization. In IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp- 284-291, 2006.
Peter J Huber and Elvezio M Ronchetti. Robust Slatislics. John Wiley & Sons, 2nd edition, 2009.
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, volume 37, pp. 448-456, 2015.
Ki Hyun Kim, Sangwoo Shim, Yongsub Lim, Jongseob Jeon, Jeongwoo Choi, Byungchan Kim, and Andre S. Yoon. RaPP: Novelty detection with reconstruction along projection pathway. In International Conference Learning Representations, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Hans-Peter Kriegel, Matthias Schubert, and Arthur Zimek. Angle-based outlier detection in high-dimensional data. In International Conference on Knowledge Discovery & Data Mining, pp. 444-452, 2008.
Alex Krizhevsky, Geolfrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems, pp. 396 404, 1990.
Kimin Lee, Honglak Lee, Kibok Loe, and Jinwoo Shin. Training confidenco-calibrrtte classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9664-99999 9974, 2021.
Shiyu Liang, Yixuan Li, and R Srikant. Fahancing the reliability of out-of-distributiio image detection in neural networks. In International Conference Learning Representations, 2018.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In International Conference on Computer Vision, pp. 2980-2985 2988, 2017.
Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Müller. Explainable deep ono-class classification. In International Conference on Learning Representations, 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019.
Duc Tam Nguyen, Zhongyu Lou, Michael Klar, and Thomas Brox. Anomaly detection with multiple- hypotheses predictions. In International Conference on Machine Learning, volume 97, pp. 4800 4809, 2019.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Compuling Surveys, 54(2), 2021.
16

---

Published in Transactions on Machine Learning Research (11/2022)
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. OCGAN: One-class novelty detection using GANs with constrained latent representations. In JEEE/CVE Conference on Compuler Vision and Paltern Recognition, pp. 2898 2906, 2019.
Wolfgang Polonik. Measuring mass concentrations and estimating density contour clusters-an excess mass approach. The Annals of Statistics, 23(3):855-8811 881, 1995.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, volume 139, pp. 8748 8763, 2021.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 2806 2814, 2021.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. Tn Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14318 14328, 2022.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A. Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloî. Deep one-class classification. In International Conference on Machine Learning, volume 80, pp. 4390-4399 2018.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In International Conference on Learning Representations, 2020.
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-78)) 795, 2021a.
Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Claus-Robert Müller, and Marius Kloft. Rethinking assumptions in deep anomaly detection. In TCML 2021 Workshop on Uncertainty & Robustness in Deep Learning, 2021b.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In 2nd Workshop on Machine Learning for Sensory Data Analysis (MLSDA 2014), pp- 4 11, 2014.
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher Anders, and Explaining deep neural networks and beyond: review of methods and applications. Proceedings of the IEFE, 2347777777 2021.
Müller.
Thomas Schlegl, Philipp Secböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging, pp. 146-157 2017.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Georg Langs, and Ursula Schmidt-Erfurth. f- AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Analysis, 54:30-44, 2019.
Hannah M Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Sclf-supervised oft-f--disti detection and localization with natural synthetic anomalies (nsa). arXio preprinl arXiv:2109. STf::2109.112222 15222, 2021.
Bernhard Schölkopf and Alex J Smola. Learning with Kernels. MIT press, 2002.
Bernhard Schölkopf, John C Platt, John Shawe- Shawe-Taylor, Alex Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural Compulation, 13(7):144331177) 2001.
17

---

Published in Transactions on Machine Learning Research (11/2022)
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification. In International Conference on Learning Represenlations, 2021.
Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Journal of Machine Learning Research, 6(Fcb):211-232, 2005.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, volume 33, pp. 11339-1185,, 2020.
David Martinus Johannes Tax. One-Class Classification. PhD thesis, Delft University of Technology, 2001.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. TEEE Transactions on Pallern Analysis and Machine Intelligence, 30(11):1958 1970, 2008.
Alexandre Tsybakov. On nonparametric estimation of density level sets. The Annals of Statistics, 25(3): 948-969, 1997.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew ZZiserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
Régis Vert and Jean-Philippe Vert. Consistency and convergence rates of onc-class SVMs and related algorithmss Journal of Machine Learning Research, 7(May):817 854, 2006.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. The caltech-uesd birds-200-2011 dataset. California Institute of Technology, 2011.
Technical Report
Jingjing Wang, Sun Sun, and Yaoliang Yu. Multivariate triangular quantile maps for novelty detection. Advances in Neural Information Processing Systems, pp. 5061-5017 5072, 2019a.
Siqi Wang, Yijie Zeng, Xinwang Liu, En Zhu, Jianping Yin, Chuanfu Xu, and Marius Kloft. Effective end-to-end unsupervised outlier detection via inlier priority of discriminative network. In Advances in Neural Information Processing Systems, pp. 5960 5973, 2019b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. 2017.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective model robustness in computer vision. In Advances in Neural Information Processing Systems, pp. 13255-13265, 2019.
Xinjie Yu and Mitsuo Gen. Introduction to evolutionary algorithms. Springer Science & Business Media, 2010.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference, 2016.
Houssam Zenati, Manon Romain, Chuan-Sheng Foo, Bruno Lecouat, and Vijay Chandrasekhar. Adversarially learned anomaly detection. In IEEE International Conference on Data Mining, pp. 727-736, 2018
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In International Conference on Knowledge Discovery 6 Dala Mining, pp. 665 674, 2017.
18

---

Published in Transactions on Machine Learning Research (11/2022)
A Diversity of the Outlier Exposure data
Here evaluate how data diversity influencos detection performance for unsupervised and superviscd OE, again comparing HSC to BCE. For this purpose, instead of 80MTI, now use CIFAR-100 as OE varying the number of anomaly classes available for the CIFAR-10 benchmark. The OE data is varied by choosing k classes at random for each random seed and using the union of these classes as the OE dataset.
1.0
0.9
AUC
0.8
BCE
0.7
HSC
1
2
4
16 k
32
64
100
Figure 7: Mean AUC detection performance in % (over 10 classes with 10 seeds per class) on the CIFAR-10 with CIFAR-100 OE one vs. rest benchmark when varying the number of l classes that comprise the OE dataset.
The results are presented in Figure 7 As expected, the performance increases with the diversity of the OE dataset. Interestingly, drawing OE samples from just k = class, i.e. binary classification between the normal class and a single OE class (which is not present as an anomaly class at test time!) already yields good detection performance the CIFAR-10 benchmark. For example, training standard classification network to discern between automobiles and beavers performs competitively an automobile anomaly detector, even when no beavers are present as anomalies during lest lime.
Compared to Section 5.2 in the main paper, where we see a transition to BCE outperforming HSC at 2 8 OE samples, we here see that with even one OE class we have passed this transition: there are already enough samples a single class that BCE outperforms HSC. The lakeaway is that OF sample diversity is not as important as one may expect, simply having many OE smaples suffices enter the regime where BCE outperforms HSC. Again, with many samples, BCE and HSC's performances are comparable.
B HSC focuses on low frequency features
To gain further insight into the difference between BCE and HSC and why the best (and worst) found OE samples are quite different for these methods, we repeat the experiment from Section 5.4 in the main paper with frequency-domain corruptions. That is, we low-pass-filee (LPF) or high-pass filter (HPF) the entire dataset (training, testing, and OF) and then proceed exactly as we did Section 5.4 An LPF removes all higher frequencics and preserves only global information such as scenery's color. This roughly corresponds to blurring images. An HPF removes all lower frequencies and roughly corresponds to edge detection. The rest benchmarks are in Table 4 Figure g contains
AUC scores for the CIFAR-10 and ImageNet-10 examples of the best and worst OE samples and also shows filtered examples. We provide details on the filter implementation in Appendix C where we evaluate the general performance of BCE and HSC for varying filter magnitudes.
HSC seems to be more robust than BCE since it's generally less affocted by the frequency corruptions. Focusing on CIFAR-10, we see that for HSC frequency corruption makes little change performance except for the Best OF HPF experiment, where HSC's performance is significantly lower than in the T.P. variant,
19

---

Published in Transactions on Machine Learning Research (11/2022)
causing HSC to behave similarly to BCE. This may have some implication that when useful signal is contained in the OF sample, corresponding to the "Best OF" experiments, it is concentrated in the low frequency spectrum. It seems as though BCE is not capable of exploiting this data. Interestingly it was also found in Yin et al. (2019) that low frequency features tend do be more robust ("low frequency bias results improved robust to corruptions"). On ImageNet we observe that HSC again performs better on the LPF experiments.
Table 4: Mean AUC detection performance in % for the best and worst single OE samples CIFAR-10 using 80MTI as OE and on ImageNet-10 using ImageNet-22K (with the 1K classes removed) as OE. All images have been either high-pass-fileen filtered (HPF) or low-pass-filteed (LPF) both during training and testing. Arrows indicate the change compared to Table
CIFAR-10
ImageNet
HSC
BCE
HSC
BCE
Best OE LPF
77.5+
68.5
77.2
73.8
Worst OE LPF
44.1-
31.1
44.6
26.1-
Best OE HPF
68.8
66.4
75.0 y
77.3
Worst OE HPF
43.6-
38.0
44.1 f
27.9
best
bese
Donate
workt
worst
(a) "ship" normal
(b) "cat" normal
(c) "airplane" normal
(d) "dragonfly" normal
Dorm
best
Worst
(c) "ship" is normal
(I) "cat" is normal
(g) "airplane" is normal
(h) "dragonfly" is normal
Figure 8: OE samples for low-pass-ilterre (a-d) and high-pass-filteeed (e-h) versions of CIFAR-10 with 80MTI as OE (a,b.e,f) and ImageNet-10 10 with ImageNet- 22k as OE (c,d,g.h). In each figure, the first row shows normal samples, the next two rows the best samples for HSC (top) and BCE (bottom), and the last two rows the worst samples for HSC (top) and BCE (bottom). The last column shows the filterod version of the images, which what the network sees during training and testing.
C Frequency sensitivity analysis
To understand why so few OE samples are that effective, we investigate the general detection performance of HSC and BCE for images with limited frequency spectra. Analog to Appendix B either low-pass (LPP)
20

---

Published in Transactions on Machine Learning Research (11/2022)
high- pass-filter (HPF) all images, both during training and testing, both normal and anomalous samples. We train and evaluate either HSC or BCE for varying OF dataset sizes and different magnitudes of filters. Note that, due to computational constraints, we decrease the number of epochs and restrict the augmentations for the frequency experiments. Figures g
and 12 show the results on LPF CIFAR-10, HPF CIFAR-10, LPF TmageNet-330, and HPF ImageNeet-30, respectively. Each point the plots corresponds to the mean AUC detection performance over all classes and seeds per class. Different colors/marker correspond different amounts of random OE samples used. The magnitudes shown the horizontal axes correspond to the number of rows and columns removed in the frequency domain. For example, LPF with magnitude of m sets the first and last rows and the first and last columns of the Fourier-transformed image to zero, before applying the inverse Fourier transformation. A magnitude of 0 corresponds to unfiltered images, a magnitude of 15 on CIFAR-10 images (which have a resolution of 32 x 32) corresponds to filtered images where just "pixels" " in the center remain the frequency domain. Similarly, an HPF with a magnitude of m sets the center of size m X zero. The extended OE robustness experiments in Appendix and Appendix D use magnitude of 14 for CIFAR-10 and magnilude of 110 for ImageNet.
ABC
15
magritade
magntade
(a) HSC
BCE
Figure LPF CIFAR-10 with 8OMTI OE AD benchmark.
1.0
1.0
AVC
(a) HSC
(b) BCE
Figure 10: HPP CIFAR-10 with 80MTT OE AD benchmark.
21

---

Published in Transactions on Machine Learning Research (11/2022)
10
10
50
20
0.0
2.0
17
1.0
166
4.5
15
moggitide
(a) HSC
(b) BCE
Figure 11: LPF ImageNet-30 with ImageNet-224, (with ImageNet-30 removed) OE AD benchmark.
10
10
20
2.0
2.
0.8
6.2
0.2
0.6
1.6
(a) HSC
(b) BCE
Figure 12: HPF ImageNet-30 with ImageNet-221 (with ImageNet- 1k removed) OE AD benchmark.
D OE robustness results with random search
In Section 5.4 we use an evolutionary algorithm to find the best and worst single OE samples for AD. The main difference between an evolutionary algorithm and a pure random search that the former "fine-tunes" samples by picking the next ones depending on the best-performing previous ones. However, the algorithm also explores completely new samples. To show that this is the case and the algorithm docsn't 'tt quickly converge to local optimum, we here present results using a pure random search for the best single OE samples and compare those to the ones in the main paper.
Table shows the mean AUC over ten classes (analogue to Table 3 in the main paper and Table in Appendix B.. We see that pure random search consistently yields slightly worse samples on CIFAR-10 and mostly very similar performing samples on ImageNet-10, There is a minor exception for unfiltered images with BCE, where random scarch found on average 2% better performing single OE samples, resulting in the same conclusion, though. Figure I3 shows examples for the best samples found via random search for unfiltered and filtered images. We see that the results quite similar to those found via the evolutionary algorithm.
22

---

Published in Transactions on Machine Learning Research (11/2022)
Table 5: Mean AUC detection performance in % for the best single OF samples on the CIFAR-10 AD benchmark using 80MTI as OE and on the ImageNet-10 AD benchmark using ImageNet-22K (with the 1K classes removed) as OE All images have been either unfiltered, high-pass pash--as--fitteee filtered (HPF), or low-pass-filerred (LPF). Here use pure random search instead of evolutionary algorithm.
CIFAR-10
ImageNet
HSC
BCE
HSC
RCE
Best OF
76.5
68.7
79.5
77.8
Best OE LPF
77.0
66.6
76.7
73.3
Best OE HPF
68.1
65.9
75.0
77.2
Home
(a) "ship" normal
(b) "cat" is normal
(c) "airplane" is normal
(d) "dragonfly" normal
(e) "ship" is normal
"cat," normal
(g) "airplane" is normal
(h) "dragonfly" normal
"ship" is normal
(i) "cat" is normal
(k) "airplane" is normal
(l) "dragonfly" is normal
Figure 13: Best OE samples for CIFAR-10 with SOMTI as OE (a-b, c-f, i-j) and ImageNet-10 with ImageNet- 22k as OE (c-d, g-h, k-l). each figure, the first row shows normal samples, and the next two rows the best. samples found via HSC (top) and BCE (bottom). normal) are low-pass-filtreed in Figures (i-I) they Figures (e-h), all samples (train, test, anomalous, and catb--pass-ilteeed.. These figures also contain separate last column showing the filtered version of the images, which is what the network sees during training testing. Here we use pure random search instead of an evolutionary algorithm.
E Hypersphere Classifier sensitivity analysis
In this section, we show results for the Hypersphere Classifier (HSC) (Section 3.2 when varying the radial function 1(z) = exp (-h(2)). For this, we run the CIFAR-10 one vs. rest benchmark with 80MTI as OE, as presented in Table D in the main paper, for different functions R" + (0, + h(z). We also alter training to be with without data augmentation in these experiments. The results presented in Table i We see that data augmentation leads to an improvement in performance even in the case where have the full 80MTI dataset as OE. HSC shows the overall best performance with data augmentation and using the
robust Pseudo-Huber loss h(z) -1
23

---

Published in Transactions on Machine Learning Research (11/2022)
Table 6: Mean AUC detection performance in % (over 10 seeds) on the CIFAR-10 one vs. rest benchmark using 80MTI as OE for different choices of h(z) the radial function of HSC.
Dala augment.
||
||=|||
||=|[
||
+1-1
w/o
90.6
92.3
89.1
91.8
w/
92.5
94.1
94.5
96.1
F Focal Loss with varying 7
Here include results showing how mean AUC detection performance changes with the Focal loss. Since we balance every batch to contain 128 normal and 128 OE samples during training, we set the weighting factor to be = 0.5 (Lin et al. 2017p. Again, note that = 0 corresponds to standard binary cross entropy. Figure 14 shows that mean AUC performance is insensitive to the choice of y on the CIFAR-10 and ImageNet-30 one vs. rest benchmarks.
1.00
1.00
0.98
0.98
0.96
AUC
AUC
0.96
0.94
0.92
0.94
Focal
Focal
0.0
0.5
2.0
4.0
0.0
0.5
2.0
4.0
Y
Y
(a) CIFAR-10
(b) ImageNet-30
Figure 14: Focal loss detection performance in mean AUC in when varying y on the CIFAR-10 with 80MTI OE (a) and ImageNet-30 with ImageNet-22K OE (b) one vs. rest benchmarks.
G Network architectures and optimization
We provide details on the network architectures and optimization below, where we distinguish between end-to-end methods and the ones that use transfer learning (CI.I--based).
G.1 End-to-end methods
We always use the same underlying network do each experimental setting for our HSC, Deep SAD, Focal, and BCE implementations control architectural effects. For Focal and BCE, the output of the network Φy is followed by linear layer with sigmoid activation. For the experiments JMNISST we standard LeNet-style networks (LeCun al.) 1990) having three convolutional layers followed by two fully connected layers. We use batch normalization [Toff & Szegedy 2015 and (leaky) ReLU activations in these networks. For our experiments on ImageNet, CUB, DTD, and MVTec-AD we use the same WideResNet (Zagoruyko Komodakis 2016) as Hendrycks et al. (0019b). which has ResNet- as architectural backbone. We use Adam (Kingma & Bal 2015) for optimization and balance every batch contain 128 normal and 128 OE samples during training. For data augmentation, we use standard color jitter, random cropping, horizontal flipping, and Gaussian pixel noise. Due to computational constraints, there is
CIFAR-10 and
24

---

Published in Transactions on Machine Learning Research (11/2022)
exception for the OE robustness experiments (Section 5.4 the frequency analysis (Appendices B and C
and the leave-one-clas-oout experiment with varying OF sizes (Figure 4 where use no augmentat ions for CIFAR-10 and only random cropping for ImageNet. We provide further dataset-specifie details below.
CIFAR-10 On CIFAR-10, we use LeNet-style networks having three convolutional layers and two fully connected layers. Each convolutional layer is followed by batch normalization, a leaky ReLU activation, and max-pooling, The first fully connected layer is followed by batch normalization, and Icaky ReLU aclivation, while the last. layer is just a linear transformation. The number of kernels in the convolutional layers are, from first to last: 32-64-128. The fully connected layers have 512-256 units respectively. We use Adam (Kingma & Ba 2015) for optimization and balance every batch to contain 128 normal and 128 OE samples during training. We train for 200 epochs starting with learning rale of r 0.001 and have learning rate milestones at 100 and 150 epochs. The learning rate is reduced by factor of 10 at every milestone. For the OE robustness experiments (Section 5.4) the frequency analysis (Appendices B and C and the leavo-one-class-on experiment with varying OE sizes (Figure 4 we trained for 30 epochs with milestone at 25 instead.
ImageNet On ImageNet, we exactly the same WideResNet (Zagouykoo & Komodakis, 2016 used in Hendryoks el al.] (2019b), which has a ResNet-18 as architectural backbone. We use Adain (Kingma & Ba 2015) for optimization and balance every batch to contain 128 normal and 128 OE samples during training. We train for 150 epochs starting with a learning rate of 0.001 and milestones at epochs 100 and 125. The learning rale is reduced by a factor of 10 al every milestone. For the OE robust ness experiments (Section 5.4), the frequency analysis (Appendices B and C and the leave-one-class-ot experiment with varying OE sizes (Figure we trained for 30 epochs with milestone at 25 instead.
CUB On CUB, we use the same setup as we used on ImageNet. However, we balance every batch to contain 30 normal and 30 OE samples during training since there are only 30 training samples per class. Also, we used subset of ImageNet-22l where we sampled images per ImageNet-22k class as OE. This speeds up data loading but should not have an impact the performance because with 150 epochs and 30 training samples the trainer only sees up to 4500 OE samples.
DTD On DTD, similar to CUB, we use the same setup as we used on ImageNet but balance every batch to contain 40 normal and 40 OF samples during training since there are only 40 training samples per class. Again, we use the subset of ImageNet-22k OE.
MV AD On MV, we use almost the same setup as we used on ImageNet. However, we train for 300 epochs starting with learning rate of n = 0.001 and milestones epochs 200 and 250.
Fashion-MNIST
On MASSI, we use a similar setup as we used on CIFAR-10. Here the network consists of only two convolutional layers with 16 and 32 kernels, respectively.
MNIST On MNIST, we use the same setup as on Fashion-MNIST but don't use any data augmentation.
G.2 CLIP
Apart from the following changes, use the same setup for our CLIP-based implementations for the end-to-end methods. One of the changes is that we use the pre-trained CLIP network architecture (Radford ee al. 2021). For fine-tuning CLIP with a BEC classifier, we use SGD with Nesterov momentum instead of ADAM, train for 80 epochs starting with a learning rate of 7 = 0.0001, and have learning rate milestones at 50, 60, 70, and 75 epochs. The learning rate is reduced by factor of 10 at every milestone. This applies completely to ImageNet and MVToc-AD. For CIFAR-10, CUB, DTD, Rashion-MNIST and MNIST, we do the same but start with learning rafe of v 0.00002 instead.
25

---

Published in Transactions on Machine Learning Research (11/2022)
H The evolutionary algorithm for finding OE samples
Section 5.4 we search for sample the OE dataset that either particularly harmful or useful for the AD model. common approach to perform a discrete search large set to use evolutionary algorithms Yu & Gen 2010; Fortin et al. 2012). We here stick to simple version that uses tournament selection Blicklle & Thiele 1995 with three competitors. In our scenario, the individuals are images of the OE dataset. and randomly initialized.
Algorithm Evolve OE samples
Input: AD model @, OE dataset D, normal train set X, full test set Xtest-
Output: A collection of optimal single training outliers (Individuals) D'.
Define:
COLN(p): returns TRUE with chance p, else FALSE.
RND(D, k); selects samples in randomly.
TRAIN d): randomly initializes é's weights, then trains with normal dataset and the single training outlier [V)L(()): computes and returns AUC using one vs. rest approach on Xtest-
Algorithm:
# Initialize:
D' RND(D,6) 64) all d' & D' do
TRAIN(6. d')
(V)
Set fitness of d' to a end for
for 50 iterations do
# Select:
Dtemp f-
for all diemp Dremp do
Replace diomp in Dteny with the best in RND(DD',)) [best according to fitness]
end for
D' - Dremp
# Mate:
for all 20, [ do
if COIN(0.05) then
d' < D'
4g + D[
for all j [0 do
P <- RNNDDD, 10000)
sort(P) according to (lpp 11 llp
d,Il²) for every C P.
P < P|: 50), get the 50 samples with least average distance to d' and dz Replace D' + with 1))(
end for
end if end for
# Mutate:
for d' E D' do
if COIN(0.55) then
P RND(D, 10000)
sort(P) according to Up d'|l² for every p P.
P < Pi: 50), get the 50 samples with least distance to d'. Replace in D' with 1ND(P.1).
end if
end for
# Evaluate:
for all d' D' do TRAIN(o, d') EVAL(e) Set fitness d' end
end for
26

---

Published in Transactions on Machine Learning Research (11/2022)
Mutating individual works by first sampling random images from the OE set and then randomly picking one among those 50 in that subset that have the least 7,2 distance to the individual. Mating works similarly, it picks one with the least L distance to both parents. The fitness of an individual is the test AUC of an AD model trained with the full normal dataset and just the individual as the only training outlier. The algorithm's 's objective to either maximize (*best" " OE samples) or minimize ("worst" OE samples) the average fitness of its final generation.
This experimental setup is exceptionally computationally expensive, with the evaluation of a single individual's fitness requiring full training of neural network. Because of this decrease the number of training epochs, restrict the augmentations, and average over just two different random seeds to evaluate an individual's fitness (see Appendix Gp. On ImageNet, we only consider the first 10 classes (see Appendix | out of the 30 that are used in Hendrycks et al. (0019)), but evaluate these with the full 30-classes one vs. rest benchmark.
Algorithm 1 provides a detailed explanation of the evolutionary algorithm. For the sake of readability, we fix the evolutionary hyperparameters to the values used in our experiments. We set the generation size to 64, the number of generations to 50, the tournament size to 3, the mating chance to 0.05, the mutation chance to 0.55, the initial candidate pool size to 10000, and the final candidate pool size to 50. We apply this algorithm to each class separately, thereby changing the normal set X and test set Xtest- For finding the worst single OE samples instead of the best ones, we change the selection procedure to simply replace dremp with the worst in RND(D', 3) instead of the best.
27

---

Published in Transactions on Machine Learning Research (11/2022)
Results on individual classes
We report results averaged over all classes in the main paper. Here we provide full class-wisc results for all and Appendix B We also include ligures for each class for the varying amount of OE experiment (Section 5.2 for CIFAR-10 and ImageNet-30.
experiments of Sections 5.1
5.3 5.4
1.1 Image AD with end-to-end end methods on the one vs. rest benchmark
Here, we report results for the one vs. rest benchmark experiments from Section 5.1 for all individual classes and end-to-end methods. For CIFAR-10, we report results in Table 7 including results from the literature. For CIFAR-10, ImageNet-30,
Mushion-MNISTI and MNIST, we report results with standard deviations for our implementations Tables
and respectively. For CUB and DTD, we report class- wise results instead in form of plot in Figures and 16 as there are many classes (47 for DTD and 200 for CUB) to report concisely in Table.
Table 7: Mean AUC detection performance in % (over 10 seeds) for all individual classes and end-lo-end methods the CIFAR-10 vs. rest benchmark with SOMTI OE from Section 5.1
Unsupervised
Unsupervised OE
Supervised OE
Class
SVDD*
DSVDD*
GT
GT+*
CSI*
GT+*
DSAD
HSC
Focal*
Focal
BCE
Airplane
65.6
61.7
74.7
77.5
94.2
96.3
87.6
95.9
96.4
Automobile
40.9
65.9
95.7
96.9
99.1
99.3
98.1
98.7
93.9
98.7
98.8
Bird
65.3
50.8
78.1
87.3
98.1
93.7
89.8
92.7
78.6
92.3
93.0
Cat
50.1
59.1
72.4
80.9
86.4
88.1
87.4
89.8
79.9
88.8
90.0
Deer
75.2
60.9
87.8
92.7
93.9
97.4
95.0
96.6
81.7
96.6
97.1
Dog
51.2
65.7
87.8
90.2
93.2
94.3
93.0
94.2
85.6
94.1
94.2
Frog
71.8
67.7
83.4
90.9
95.1
97.1
96.9
97.9
93.3
97.8
98.0
Horse
51.2
67.3
95.5
96.5
98.7
98.8
96.8
97.6
87.9
97.6
97.6
Ship
67.9
75.9
93.3
95.2
97.9
98.7
97.1
98.2
92.6
98.0
98.1
Truck
48.5
73.1
91.3
93.3
95.5
96.2
97.4
92.1
97.5
97.7
Mean AUC
58.8
64.8
86.0
90.1
94.3
95.6
94.5
95.9
87.3
95.8
96.1
Table 8: Mean AUC detection performance in % (over 10 seeds) with slandard deviations for all individual
classes for our end-to-end implementations on the CIFAR-10 one vs. rest benchmark with SOMTI OE from Section 5.1
Unsupervised OE
Supervised OE
Class
DSAD
HSC
Focal
BCE
Airplane
94.2 + 0.34
96.3 + 0.13
95.9 = 0.11
96.4 0.17
Automobile
98.1 0.19
98.7 + 0.07
98.7 = 0.09
98.8 0.06
Bird
89.8 = 0.54
92.7 = 0.27
92.3 = 0.32
93.0 + 0.14
Cat
87.4 0.38
89.8 = 0.27
88.8 =
90.0 0.27
Deer
95.0 + 0.22
96.6 + 0.17
96.6 = 0.10
97.1 + 0.10
Dog
93.0 0.30
94.2 + 0.13
94.1 = 0.21
94. 0.12
Frog
96.9 = 0.22
97.9 = 0.08
97.8 0.07
98.0 0.09
Horse
96.8 = 0.15
97.6 + 0.10
97.6 = 0.16
97.6 = 0.09
Ship
97.1 = 0.21
98.2 = 0.09
98.0 0.11
98.1 0.08
Truck
96.2 + 0.22
97.4 + 0.13
97.5 = 0.12
97.7 + 0.16
Mean AUC
94.5 + 3.30
95.9 + 2.68
95.8 = 2.97
96.1 2.71
28

---

Published in Transactions on Machine Learning Research (11/2022)
Table Mean AUC detection performance in % (over 10 seeds) for all individual classes for our end-to-end implementations on the ImageNet-30 one vs. rest benchmark with ImageNet-22K OE from Section 5.1 Note
that for GT4*, Focal*, and CSI reported in Table in the main paper, Hendrycks et al. b019b and Tack et al (2020 do not provide results per class basis.
Unsupervised DSVDD
Unsupervised OE
Supervised Focal
Class
DSAD
HSC
BCE
Acorn
62.7 + 2.94
98.5 + 0.28
98.8 + 0.42
99.0 + 0.15
99.0 + 0.19
Airliner
55.8 + 1.76
99.6 + 0.16
99.8 + 0.10
99.9 + 0.02
99.8 + 0.04
Ambulance
17.3 + 3.58
99.0 + 0.13
99.8 + 0.13
99.2 0.14
99.9 0.07
American alligator
73.0 + 0.65
92.9 = 1.06
98.0 = 0.32
94.7 0.67
98.2 + 0.27
Banjo
56.8 + 2.22
97.0 = 0.51
98.2 0.41
97.0 0.33
98.7 0.22
Barn
67.6 1 1.32
98.5 1 0.29
99.8 1 0.05
98.7 1 0.24
99.8 1: 0.08
Bikini
59.7
96.5 1 0.84
98.6
97.2
0.30
Digital clock
61.2
99.4 + 0.33
96.8 + 0.79
99.8 + 0.03
97.2 + 0.29
Dragonfly
61.9 3.02
98.8 + 0.28
98.4 + 0.16
99.1 + 0.21
98.3 0.04
Dumbbell
51.3 1.09
93.0 + 0.53
91.6 + 0.88
94.0 + 0.04
92.6 + 0.97
Forklift
48.0 5.00
90.6 = 1.43
99.1 = 0.33
94.2 + 0.90
99.5 0.09
Goblet
63.3 = 0.25
92.4 = L.05
93.8 = 0.38
93.8 + 0.27
94.7 + 1.43
Grand piano
58.4 0.16
99.7 + 0.06
97.4 + 0.37
99.9 1 0.04
97.6 1 0.34
Hotdog
62.0 2.46
95.9 1 2.01
98.5 1 0.34
97.2 0.05
98.8 0.34
Hourglass
60.2 2.98
96.3 + 0.37
96.9 = 0.26
97.5 + 0.17
97.6 + 0.48
Manhole cover
63.9 0.56
98.5 + 0.29
99.6 + 0.34
99.2 0.09
99.8 0.01
Mosque
73.8 + 0.35
98.6 = 0.29
99.1 + 0.26
98.9 + 0.30
99.3 + 0.15
Nail
49.4 1.05
92.8 = 0.80
94.0
93.5
1.37
Parking meter
46.3 + 2.64
98.5 = 0.29
93.3 + 1.64
99.3 1 0.04
94.7 + 0.76
Pillow
10.8 1.22
99.3 + 0.14
94.0 + 0.47
99.2 0.14
94.2 0.42
Revolver
50.6 + 1.59
98.2 + 0.30
97.6 + 0.25
98.6 + 0.11
97.7 + 0.68
Rotary dial telephone
59.1 1.68
90.4 = 1.99
97.7 = 0.50
92.2 + 0.33
98.3 0.75
Schooner
78.0 1.94
99.1 = 0.18
99.2 = 0.20
99.6 + 0.02
99.1 + 0.26
Snowmobile
70.9 1.99100 0.79
97.7 + 0.86
99.0 + 0.22
98.1 1 0.15
99.1 0.25
Soccer ball
56.0 1.68
97.3 + L.70
92.9 + 1.18
98.6 + 0.13
93.6 + 0.61
Stingray
79.5 0.98
99.3 + 0.20
99.1 + 0.33
99.7 + 0.04
0.04
99.2 + 0.10
Strawberry
65.0 + 3.32
97.7 + 0.64
99.1 + 0.20
99.1 0.03
99.2 0.22
Tank
69.5 + 1.83
97.3 + 0.51
98.6 + 0.18
97.3 + 0.47
0.47
98.9 + 0.13
Toaster
33.1 0.42
97.7 = 0.56
92.2 0.78
98.3
92.2 0.65
Volcano
88.0 + 3.27
89.6 = 0.44
99.5 = 0.09
91.6 + 0.90
99.4 + 0.19
Mean
61.1 + 10.61
96.7 = 2.98
97.3 = 2.53
97.5 + 2.43
97.7 + 2.34
Table 10: Mean AUC detection performance in % (over 10 seeds) for all individual classes for our end-to-end implementations the MaSHion-MNIST one rest benchmark with grayscale CIFAR-100 OE from Section
5.1
Unsupervised DSVDD
Unsupervised OE HSC
Supervised OE
Class
DSAD
Focal
BCE
Top
82.8 1: 1.10
85.2 1: 2.00
83.5 + 4.13
78.6 t 2.41
75.3 +: 2.12
Trouser
94.8 + 4.92
96.1 1.90
98.1 = 0.18
95.9 + 1.30
93.5 + 0.81
Pullover
79.3 + 2.42
82.4 + 0.95
83.6 = 2.59
88.9 + 1.55
89.4 + 1.37
Dress
88.8 + 2.22
88.1 + 1.96
88.0 + 2.26
84.5 + 2.04
82.6 2.88
Coat
82.9 + 3.71
84.1 + 2.04
85.7 + 1.71
86.2 + 0.62
83.1 2.20
Sandal
75.1 9.60
1.18
90.5 0.76
92.7 + 0.97
91.7 2.09
Shirt
75.5 1.02
73.0 2.40
74.5 3.16
78.4 0.74
74.7 1.58
Sneaker
96.6 0.53
0.65
97.1 0.49
0.69
95.0 0.31
Bag
2.67
75.9 5.94
78.3
T
89.4 2.01
Ankle boot
1 0.23
91.6 1.91
94.0 1.36
1 1.78
89.8 1.79
Mean
86.3 + 8.06
86.4 + 7.44
87.3 + 7.38
87.7 + 5.76
86.4 + 6.84
29

---

Published in Transactions on Machine Learning Research (11/2022)
Table Mean AUC detection performance (over seeds) for all individual classes for our end-to-end implementations on the MNIST one vs. rest benchmark with EMNIST OE from Soction 5.1
Unsupervised DSVDD
Unsupervised OF
Supervised
Class
DSAD
HSC
Focal
BCE
Zero
99.0 + 0.27
99.3 0.39
99.6 = 0.03
99.4 + 0.34
99.4 = 0.16
One
99.8 1 0.02
99.7 0.12
99.9 1 0.02
99.7 1 0.18
99.3 - 0.34
Two
92.3 1.17
98.0 0.12
98.0 + 0.49
97.6 0.68
97.5 0.23
Three
94.9 0.36
98.6 0.35
98.5 =
99.5 0.10
99.6 0.13
Four
94.4 0.50
96.5 1.02
96.7 =
1.01
98.2 0.52
Five
94.2 1.35
97.6 0.24
1
97.9 0.25
97.3
99.0 0.09
99.6 0.08
99.6 1 0.08
0.07
99.7 0.0T
Seven
96.6 0.57
97.1 0.39
97.4 1 0.63
97.6 0.65
97.7 0.45
Fight
90.9 + 1.47
93.5 2.49
96.6 + 0.81
97.3 0.32
97.3 = 0.57
Nine
96.9 +: 0.36
96.3 1.06
97.6 + 0.27
97.8 + 0.31
97.7 = 0.57
Mean
95.8 2.83
97.6 1.81
98.2 1.14
98.4 0.97
98.4 0.95
0.9
0.8
AUC
0.6
0.5
26.0 EO.0TH 8.E2%
800 7.99%
class
Figure 15: Mean AUC detection performance in % (over 10 seeds) for all individual classes for our end-to-end implementations on the CUB one vs. rest benchmark with ImageNet-22k OE from Section
0.9
0.8
AUC
190
0.6
D5VT0 860%
0.5
16.8 12.15 6-00
19.3%%
beidy
surey
extee
class
Figure 16: Mean AUC detection performance in % (over 10 seeds) for all individual classes for our end-to-end implementations on the DTD one vs. rest benchmark with ImageNet-222 22k OE from Section 5.1
30

---

Published in Transactions on Machine Learning Research (11/2022)
1.2 Image AD with transfer learning- based methods the rest benchmark
Here report results for the one vs. rest benchmark experiments from Section 5.3 for all individual classes and transfer learning-busse based methods. For CIFAR-10, we report results in Table including results from the litcrature (left) and results with standard deviations for our implementation (right). For ImageNet-300, Fashion-MViST, and MNIST, we report results with standard deviations for our implementations Tables 13 and respectively. For CUB and DTD, we report class- wise results instead form of plot 18 as there are too many classes (47 for DTD and 200 for CUB) to report concisely in a
Figures 17
Table.
Table 12: Left: Mean AUC detection performance in % (over 10 scods) for all individual classes and transfer learning- based methods on the CIFAR-10 one vs. rest benchmark with SOMTI OE from Section 5.3 Our transfer learning- bearning-based based implementations for the same setup.
Right:
Unsupervised PANDA*
Supervised OE
Unsupervised CLIP
Supervised OE BCE-CL
Class
DN2*
CLIP
PANDA
ADIP*
BCE-CL
Class
Airplane
93.9
99.4
99.2
99.7
Airplane
99.40 + 0.00
99.74 + 0.02
Automobile
97.7
99.4
99.8
99.8
Automobile
99.37 +
99.82 =
Bird
85.5
X
97.4
X
98.6
99.2
Bird
97.37 = 0.00
99.29 = 0.04
Cat
85.5
X
97.0
97.0
98.8
Cat
96.99 = 0.00
98.86 = 0.04
Deer
93.6
98.1
99.3
99.6
Deer
98.05 = 0.00
99.62 0.02
Dog
91.3
97.4
99.2
Dog
97.36 =
99.26 =
Frog
94.3
X 98.1
99.6
99.9
Frog
98.05 = 0.00
99.89 = 0.01
Horse
93.6
99.0
99.8
99.8
Horse
98.99 = 0.00
99.84 = 0.02
Ship
95.1
99.7
99.6
99.8
Ship
99.71 = 0.00
99.86 0.01
Truck
95.3
X
99.3
99.5
99.9
Truck
99.30 = 0.00
99.90 = 0.01
Mean AUC
92.5
96.2
98.5
98.9
99.1
99.6
Mean AUC
98.46 + 0.96
99.61 + 0.34
Table 13: Mean AUC detection performance in % (over 10 seeds) for all individual classes for transfer learning-basee implementations the Pashion-MNIST one rest benchmark with grayscale CIFAR-100 OE from Section 5.3
Unsupervised CLIP
Supervised OE BCE-CL
Class
Top
82.9 + 0.00
84.7 1 0.34
Trouser
87.7 + 0.00
98.5 0.03
Pullover
91.6 + 0.00
93.8 0.06
Dress
87.0 + 0.00
97.0 + 0.07
Coat
91.8 + 0.00
91.8 0.34
Sandal
78.7 + 0.00
99.3 + 0,02
Shirt
81.3 + 0.00
81.9 0.22
Sneaker
97.3 + 0.00
97.8 + 0.09
Bag
92.4 + 0.00
99.6 0.02
Ankle boot
99.1 + 0.00
99.4 + 0.04
Mean
89.0 1 6.36
94.7 1 5.49
31

---

Published in Transactions on Machine Learning Research (11/2022)
Table 14: Mean AUC detection performance (over 10 seeds) for all individual classes for our transfer
learning-baad sraning-basodd based implementations on the ImageNet-30 one vs. rest benchmark with ImageNet-22K OE from Section 5.3
Unsupervised CLIP
Supervised OE BCE-CL
Class
Acorn
99.78 0.00
99.96 + 0.01
Airliner
100.00 = 0.00
100.00 + 0.00
Ambulance
100.00 = 0.00
100.00 + 0.00
American alligator
99.98 0.00
100.00 1 0.00
Banjo
100.00 +: 0.00
100.00 1 0.00
Barn
100.00 0.00
100.00 + 0.00
Bikini
99.87 + 0.00
100.00 + 0.00
Digital clock
99.69 0.00
99.93 + 0.02
Dragonfly
100.00 = 0.00
100.00 + 0.00
Dumbbell
99.91 0.00
99.97 0.02
Forklift
100.00 +: 0.00
100.00 + 0.00
Goblet
99.29 0.00
99.81 1: 0.04
Grand piano
100.00 = 0.00
98.36 + 4.87
Hotdog
99.99 0.00
100.00 + 0.00
Hourglass
99.69 0.00
99.97 + 0.02
Manhole cover
100.00 + 0.00
100.00 + 0.00
Mosque
100.00 = 0.00
100.00 + 0.00
Nail
99.61 = 0.00
99.97 + 0.01
Parking meter
99.52 0.00
99.97 + 0.01
Pillow
99.95 = 0.00
100.00 + 0.00
Revolver
100.00 0.00
100.00 + 0.00
Rotary dial telephone
99.84 + 0.00
99.21 + 1.75
Schooner
100.00 0.00
100.00 1 0.00
Snowmobile
99.99 + 0.00
100.00 + 0.00
Soccer ball
99.97 0.00
99.98 + 0.05
Stingray
100.00 + 0.00
100.00 + 0.00
Strawberry
99.81 = 0.00
99.97 - 0.07
Tank
100.00 = 0.00
100.00 - 0.00
Toaster
99.44 + 0.00
99.88 1 0.01
Volcano
100.00 1: 0.00
99.99 1 0.04
Mean AUC
99.88 0.19
99.90 0.32
Table 15: Mean AUC detection performance in (over 10 seeds) for all individual classes for our transfer learning-base basod implementations on the MNIST one vs. benchmark with EMNIST OE from Scction 5.1
Unsupervised CLIP
Supervised OE
Class
BCE-CI CL
Zero
73.8 0.00
98.2 + 2.22
One
38.3 + 0.00
99.6 + 0.08
Two
69.8 0.00
11.61
Three
85.2 + 0.00
99.4 1 0.06
Four
64.8 0.00
98.4 + 0.15
Five
49.4 + 0.00
96.6 + 0.33
Six
24.7 0.00
88.5 + 21.08
Seven
77.9 0.00
97.8 + 0.06
Eight
40.0 0.00
95.6 + 0.37
Nine
66.1 0.00
96.9 1 0.43
Mean
59.0 18.76
96.0 + 3.70
32

---

Published in Transactions on Machine Learning Research (11/2022)
1.0
0.9
0.8
AUC
0.6
0.5
DCC 2.7%%
class
Figure 17: Mean AUC detection performance % (over seeds) for all individual classes for our transfer learning-basee based implementations on the CUB one vs. rest benchmark with ImageNet-22k OE from Section 5.3
1.0
0.9
0.8
AUC
0.7
0.6
0.5
CUP 50 10% + 10.4% 4.7%%
colbed
ame
my
class
Figure 18: Mean AUC detection performance in % (over 10 seeds) for all individual classes for our transfer
learning-best based implementations on the DTD one vs. rest benchmark with ImageNet-22k OE from Section 5.3
33

---

Published in Transactions on Machine Learning Research (11/2022)
1.3 Image AD on the leave-one oolve-ose-cass---e out benchmark
rest image-AD benchmark (see Section 4 In Section 5.2 we also present results on the more uncommon ceave-one-class-00 benchmark when varying the OE training set sizc. Hero, we present results when using the full OE training set size, analogue to the
In the main paper, evaluate our methods the ubiquitous
experiments Sections 5.1 and 5.3 Tables 17 and 16 show results for leave-one-class-ontt CIFAR-10 and
leave-one-class-out ImageNet-30, respectively.
Note that, to apply CLIP the leave-one-class-out setting with K normal classes, use the text tuple "a photo of (NORMAL CLASS KY", "a photo of something"). For test image we compute its anomaly score as
(U1,..., uk+1) (*a photo of (NORMAL CLASS_1J " "_",
(a 100)
s(x) EKt exp((fu(=z) 100)
(5)
Table 16: Mean AUC detection performance in % (over 2 sceds) with standard deviations for all individual classes for our implementations on the ImageNet-30 leave-one-clas-oout AD benchmark with ImageNet-222k (with ImageNet- Ik removed) OE.
Unsupervised
Unsupervised OE
Supervised OE BCE
Anomaly class
DSVDD
CLIP
DSAD
HSC
Focal
BCE-CL
Acorn
49.7 +: 1.02
98.8 + 0.00
92.0 + 0.21
92.6 +:2 0.54
92.2 + 0.51
90.4 +: 0.02
99.7 + 0.05
Airliner
47.6 + 3.67
99.9 + 0.00
93.8 + 0.89
92.9 + 1.71
94.7 + 0.22
93.9 + 0.09
99.9 + 0.01
Ambulance
50.5 + 1.57
99.7 + 0.00
0.00
93.4 + 0.97
92.3 + 0.92
93.1 + 1.46
93.2 + 1.66
99.9 H 0.02
American alligator
53.4 4.66
98.6 1 0.00
91.7 1 0.89
91.3 1 0.12
91.5 1 1.01
92.1 0.01
99.7 1 0.07
Banjo
50.3 1.30
0.00
85.3 1 0.98
85.6 1 0.97
85.3 1 0.54
86.0 0.04
99.9 1 0.02
Barn
51.5 0.96
# 0.00
0.73
90.6 0.95
0.17
89.3 2.04
+ 0.05
Bikini
50.6 1.03
97.9 + 0.00
0.04
90.0 +
87.4 0.72
90.6 0.44
+ 0.03
Digital clock
49.1 + 1.33
+
0.42
84.8 +
84.7 + 0.79
85.1 0.83
E 0.01
Dragonfly
51.2 + 3.23
99.8 + 0.00
94.2 + 0.01
93.9 + 0.24
93.8 + 0.58
93.5 = 0.72
99.9 = 0.01
Dumbbell
46.1 2.87
97.7 = 0.00
80.7 1.12
82.7 = 0.78
81.7 1.39
83.3 + 0.71
99.4 = 0.07
Forklift
50.6 1.20
96.8 = 0.00
82.1 0.37
82.0 + 0.42
82.9 - 0.34
81.9 - 0.71
99.6 = 0.12
Goblet
46.8 1 0.13
96.3 1 0.00
84.2 1 2.64
83.9 1 0.15
84.4 1 1.30
83.5 0.99
98.8 1 0.14
Grand piano
46.2 + 2.79
99.5 # 0.00
82.2 1 1.00
82.4 + 0.01
83.4 + 0.71
82.6 + 1.08
99.9 1 0.02
Hotdog
50.4 + 1.74
99.7 # 0.00
93.2 + 0.62
93.9 + 0.49
94.3 + 0.27
94.2 + 0,23
99.7 + 0.03
Hourglass
50.9 + 2.30
87.3 = 0.00
87.1 + 0.89
85.9 + 0.62
86.9 + 0.71
85.5 - 1.14
96.0 = 0.29
Manhole cover
51.8 1: 5.59
93.0 1 0.00
84.7 1 1.74
84.9 1 1.47
87.8 1 0.48
88.3 1 0.09
99.6 1 0.04
Mosque
49.4 1 6.93
99.7 1 0.00
91.7 1 0.04
89.4 1 1.22
88.1 1 0.84
89.1 1 0.29
99.9 1 0.06
Nail
49.3 1: 0.87
97.1 1 0.00
86.2 1 0.44
87.7 1 0.74
86,6 1 0.08
85,4 1.49
97.0 1 0.05
Parking meter
51.1 + 2.91
92.4 0.00
85.1 + 0.96
82.1 + 0.26
81.9 0.87
81.3 + 0.17
96.3 + 0.55
Pillow
47.9 1 0.37
99.5 + 0.00
91.6 + 0.65
90.6 + 0.84
91.0 0.65
90.6 + 0.36
99.9 + 0.02
Revolver
54.7 1: 0.47
99.6 1 0.00
91.3 1 0.66
88.9 1 0.91
89.4 1 0.15
90.5 1 0.65
99.8 1 0.04
Rotary dial telephooe
50.0 0.68
97.3 1 0.00
85.2 1.38
84.9 T 1.09
82.9 T 0.38
82.7 0.41
99.3 1 0.10
Schooner
42.0 0.25
# 0.00
0.12
94.2 + 0.65
95.6 0.18
94.4 + 0.61
+ 0.01
Snowmobile
48.2 2.40
#
0.01
88.0
87.5 1.40
86.5 0.16
+ 0.01
Saccer ball
51.1 0.20
99.4
0.46
89.8
0.51
90.2 0.92
+ 0.00
Stingray
48.6 6.35
97.1 0.00
91.8 0.17
90.9 + 1.21
90.8 + 0.53
90.4 0.39
99.8 + 0.01
Strawberry
52.0 2.28
99.5 0.00
95.5 4 0.95
96.4 0.34
96.5 0.23
95.0 0.48
99.8 - 0.01
Tank
51.8 2.40
98.4 0.00
86.8 + 1.76
82.7 0.72
84.9 0.19
84.6
99.8 + 0.01
Toaster
48.3 1.40
97.2 = 0.00
84.0 + 0.08
83.3 - 1.03
86.1 = 0.72
84.3 = 0.32
98.0 + 0.19
Volcano
51.3 2.07
99.6 1 0.00
89.2 1 1.10
89.2 1 0.77
88.5 1 0.65
88.4 1 0.94
99.9 1 0.01
Mean
49.7 2.41
97.8 + 2.73
88.8 4.20
88.3 + 4.16
88.5 4.14
88.2 4.12
+ 1.05
34

---

Published in Transactions on Machine Learning Research (11/2022)
Table 17: Mean AUC detection performance in % (over 2 seeds) with standard deviations for all individual classes for our implementations on the CIFAR-10 leave-one-clas-ouut AD benchmark with 80MTI OE.
Unsupervised
Unsupervised OF
Supervised OE BCE
Class
DSVDD
CLIP
DSAD
HSC
Focal
BCF-CI.
Airplane
62.3 + 8.90
96.2 + 0.00
85.5 + 0.52
87.0 + 0.52
88.4 + 0.28
87.9 + 0.17
99.2 + 0.03
Automobile
52.7 + 0.69
96.0 + 0.00
85.6 0.81
87.2 + 0.56
89.9 0.71
90.3 = 0.01
99.3 + 0.02
Bird
54.2 + 4.00
93.5 + 0.00
83.6 = 0.04
83.8 + 0.03
86.2 + 0.49
86.0 = 0.45
97.9 + 0.15
Cat
39.6 + 2.98
+ 0.00
82.4 0.10
82.4 + 0.80
83.4 0.52
81.1 = 0.36
97.9 + 0.01
Deer
56.6 + 0.49
79.5 + 0.00
75.6 = 0.11
74.9 + 0.06
76.9 + 0.42
77.0 = 0.13
96.6 + 0.03
Dog
49.9 = 4.06
90.6 = 0.00
82.7 0.15
82.8 +
84.1 0.75
84.6
97.9 +
Frog
53.2 = 11.75
94.0 0.00
85.1 = 0.10
85.5 + 0.35
86.1 + 0.28
86.4 = 0.31
98.3 + 0.12
Horse
47.7 =
93.4 0.00
85.2 0.57
85.9 + 0.08
88.4 0.04
88.2
+ 0.09 99.4 + 0.03
Ship
54.7 = 5.87
97.8 0.00
89.7 = 0.12
90.7 + 0.06
91.3 + 0.17
92.0 = 0.35
Truck
51.1 + 1.86
90.0 + 0.00
86.7 0.23
88.0 + 0.20
88.9 0.41
89.2 0.37
99.0 + 0.01
Mean
52.2 + 5.66
+ 4.89
84.2 3.51
84.8 + 4.08
86.4 3.94
86.6 3.96
98.4 + 0.83
1.4 Image AD on the MVTec-AD benchmark
As mentioned in Section our paper focuses on natural images because random images from the web are likely not informative as OF for other data. In Section 5.2 we demonstrate this on the manufacturing datasot. MVTec-AD, where anomalics are rather subtle, for varying OE sot sizos. Here we show results using the full ImageNet-222k dataset
OE in Table 18
Table 18: Mean AUC detection performance in % (over 2 seeds) with standard deviations for all individual classes for implementations the MVTec-AD benchmark with ImageNet22k OF.
Unsupervised
Unsupervised OE
Supervised OE
Class
DSVDD
CLIP
DSAD
HSC
Focal
BCE
BCE-CL
Bottle
83.0 2.38
34.8 + 0.00
78.0 + 6.07
73.6 + 0.95
66.9 6.47
71.9 + 5.28
76.0 = 4.92
Cable
60.7 3.02
48.7 = 0.00
71.3 + 2.74
69.1 1 0.42
61.5 3.47
54.8 4.76
72.1 =
Capsule
61.4 + 4.09
43.3 + 0.00
53.2 + 7.64
59.7 5.88
56.0 = 11.81
57.5 + 3.97
59.2 + 1.07
Carpet
54.0 1.89
72.5 1 0.00
55.6 + 2.96
63.5 5.12
55.3 0.22
57.1 + 0.52
88.1 + 0.63
Grid
47.0 2.09
48.9
59.4
4.93
49.9
1
Hazelnut
80.5 1 0.11
48.1 + 0.00
70.2 1 6.56
79.2 3.38
73.2 5.14
78.8 1 2.82
62.8 1 1.12
Leather
69.3 + 3.21
99.9 + 0.00
80.1 + 3,87
84.6 0.48
87.3 4.52
87.3 +: 1.61
87.4 # 12.50
Metal nut
55.7 + 0.20
45.0 + 0.00
47.9 + 5.33
62.7 5.76
61.1
54.5 + 4.42
75.3 1 7.61
61.3 2
69.4 1
58.8 10.27
55.8 4.95
70.3 2.96
53.8 11.25
1 76.6 + 1.33
Screw
49.5 + 1.52
64.4 + 0.00 0.00
39.0 + 1.30
41.9 + 1.84
59.5 2 4.08
62.0 + 7.20
Tile
63.7 + 0.05
78.0 + 0.00
95.4 + 1.06
95.3 + 1.15
95.5 + 1.82
98.2 + 0.20
Toothbrush
52.9 6.53
58.3 +
+ 70.4 + 7.19
2.99
59.6 1.81
66.9 + 4.72
68.0 + 6.76
Transistor
61.9 + 6.73
51.0 + 0.00
64.9 + 2.83
64.6 3.87
66.9 + 0.08
72.5 + 6.65
Wood
87.7 + 0.70
35.5 + 0.00
85.1 + 11.27
93.8 + 1.67
73.6 + 9.12
75.7 + 20.57
94.3 + 0.73
Zipper
70.8 + 1.52
36.5 + 0.00
67.1 + 6.22
60.3 + 10.50
31.8 + 25.64
58.6 + 0.54
74.5 + 0.67
Mean
64.0 + 11.81
56.4 + 17.57
66.1 + 14.89
70.1 + 14.78
65.4 + 12.44
66.1 + 12.93
76.2 + 10.96
35

---

Published in Transactions on Machine Learning Research (11/2022)
1.5 Varying the OE size
include plots for all individual classes in Figure 21 for CIFAR-10 and in Figures 19 and 20 for ImageNet-30, respectively. Additionally, for the experiments on varying the diversity of OE data on CIFAR-10 with CIFAR-100 OE, we added the plots for all individual classes in Figure 22
For the experiments on varying the number of OE samples (Section 5.2
67
(a) Class: acorn
(b) Class: airliner
(c) Class: ambulance
(d) Class: american alligator
Class: banjo
Class: barn
(g) Class: bikini
(h) Class: digital clock
(i) Class: dragonfly
Class: dumbbell
(k) Class: forklift
(1) Class: goblet
(m) Class: grand piano
(n) Class: hotdog
(o) Class: hourglass
Figure Mean AUC detection performance % (over seeds) for all classes of the ImageNet-30 one vs. rest benchmark from Section 5.2 when varying the number of ImageNet-22KK OE samples. These plots correspond to Figure 2(c). but here report the results for all individual classes (from class (acorn) to class 15 (hourglass)).
36

---

Published in Transactions on Machine Learning Research (11/2022)
Class: manhole
(b) Class: mosque
(c) Class: nail
(d) Class: parking meter
(e) Class: pillow
(1) Class:: revolver
(g) Class: rotary dial telephone
(h) Class: schooner
Class: snowmobile
(i) Class: socoer ball
(k) Class: stingray
(I) Class: strawberry
67
(m) Class: tank
(n) Class: toaster
(o) Class: volcano
Figure 20: Mean AUC detection performance in % (over seeds) for all classes of the ImageNet-30 one rest benchmark from Section 5.2 when varying the number of ImageNet-22XK OE samples. These plots correspond to Figure [cc) but here report the results for all individual classes (from class 16 (manhole cover) to class 30 (volcano)).
37

---

Published in Transactions on Machine Learning Research (11/2022)
1.0
0.9
0.0
0.8
0.3
AUC
01
10
10
20
21
(a) Class: airplane
(b) Class: automobile
1.0
1.0
0.9
0.9
0.6
0.3
ABC
ADC
10
10
19
20
21
(c) Class: bird
(d) Class: cat
1.0
1.0
0.9
0.9
AUC
(c) Class: deer
(f) Class: dog
SCE BBC
0.3
BCE
(g) Class: frog
(h) Class: horse
0.7
0.7
0.6
0.6
0.5
BCB
0.5
BCZ:
RBC
ISC
Class: ship
Class: truck
Figure 21: Mean AUC detection performance in (over seeds) for all classes of the CIFAR-10 one vs. benchmark from Section 5.2 when varying the number of 80MTI OE samples. These plots correspond to Figure 2(a) but here report the results for all individual classes.
38

---

Published in Transactions on Machine Learning Research (11/2022)
1.00
0.9
0.9
0.95
AUC
AUC
ALC
0.8
0.8
0.85
0.7
0.80
HSC
HSC
HSC
64
100
100
16
32
64
100
(a) Class:: airplanc
(b) Class: automobile
(c) Class: bird
0.90
0.9
0.85
0.90
0.80
AUC
0.8
0.75
0.80
0.70
BCE
0.75
BCE HSC
HSC
0.65
0.70
(d) Class: cat.
(c) Class: deerr
(f) Class: dog
0.95
AUC
0.900
0.90
0.85
0.85
BCB
ICE
HSC
FSCC
HSC
(g) Class: frog
(h) Class: horse
(i) Class: ship
0.95
0.90
AUC
0.85
0.50
BSC
2
16
32
64
100
(i) Class: truck
Figure 22: Mean AUC detection performance % (over seeds) for all CIFAR-10 classes from the experiment in Appendix A on varying the number of classes & of the CIFAR-100 OE data. These plots correspond Figure 7 but here report the results for all individual classes.
39

---

Published in Transactions on Machine Learning Research (11/2022)
1.6 Robustness towards choice of OE samples
Here provide class-wise results for the best and worst single OE samples found via evolutionary algorithm (Section 5.4). On ImageNet, only consider the first 10 classes of the ones used by (Hendrycks ct al. 2019b): "Acorn". " "Airliner", "Ambulance", "American alligator", "Banjo", "Barn", "Bikini", "Digital clock", "Dragonfly", "Dumbbell". However, the full 30-class one vs. rest benchmark was used to evaluate for each of these ten classes. Tables 19 and 20 show the performance of the best or worst single OF sample (from 80MTI) for each class in CIFAR-10, while the former shows results for unfiltered images and the latter for low-pass-filteres filtered high-pass- filtered images. Tables 21 and 22 show the same for ImageNet- 10 with ImageNet-221K as OF. Figures 23
26 27 and 28 show the 4-5 best and worst samples for HSC and BCE, on CIFAR-10 or ImageNet-Ie 10, for unfiltered or filtered images, respectively.
24
Table 19: Class-wise AUC detection performance in % for the best and worst single OE samples found via evolutionary algorithm (Section 5.4 on the CIFAR-10 one vs. rest AD benchmark using 80MTI as OE.
Best OE
Worst OE
Class
HSC
BCE
HSC
BCE
Airplane
85.2
76.1
37.4
29.9
Automobile
78.1
71.5
36.4
32.9
Bird
74.0
67.9
47.2
35.8
Cat
72.5
64.9
38.7
35.1
Deer
79.9
68.2
55.1
29.8
Dog
71.6
70.6
41.0
35.0
Frog
84.2
71.7
47.7
24.8
Horse
66.2
66.2
41.9
38.7
Ship
86.5
72.8
46.2
27.1
Truck
78.8
69.1
41.2
26.9
Mean AUC
77.7
69.9
43.3
31.6
Table 20: Class-wise AUC detection performance in % for the best and worst single OE samples found via an rest AD benchmark using 80MTI OE. All images are either low-pass-filtered (LPF) or high-pass-fittredd (HPF), both during training and testing.
evolutionary algorithm (Section 5.4 the CIFAR-10
Best OF
Worst OE
Class
HSC LPF
BCE LPF
HSC HPF
BCE HPF
HSC LPF
BCE LPF
HSC HPF
BCE HPF
Airplane
83.0
68.5
68.5
64.6
36.0
22.1
40.7
32.5
Automobile
75.0
69.5
60.1
69.0
47.5
32.6
40.2
40.3
Bird
63.5
67.9
40.3
39.1
46.4
39.2
Cat
73.6
63.2
65.5
63.3
44.0
34.2
48.6
39.4
Deer
76.9
69.2
68.5
63.8
38.0
31.7
47.2
38.9
Dog
74.2
64.33
69.4
67.1
43.0
34.1
53.3
37.3
Frog
83.5
73.1
78.2
70.3
40.1
28.2
38.0
37.0
Horse
72.5
66.5
66.7
50.9
35.5
40.3
Ship
85.7
75.8
74.8
72.2
47.8
23.0
35.5
35.8
Truck
79.9
71.0
67.9
69.2
53.0
30.1
37.3
39.7
Mean AUC
77.5
68.5
68.8
66.4
41.1
31.1
43.6
38.0
40

---

Published in Transactions on Machine Learning Research (11/2022)
Table 21: Class-wise AUC detection performance in for the best and worst single OF samples found an ovolutionary algorithm (Section 5.40 on the ImageNet-30 one vs. rest AD benchmark using ImageNet-221t (with the IK classes removed) OE.
Best OE
Worst OE
Class
HSC
BCE
HSC
BCE
Acorn
84.0
76.6
43.6
22.3
Airliner
91.1
80.8
41.3
15.7
Ambulance
84.0
86.4
24.0
22.9
American alligator
81.2
75.7
54.6
30.1
Banjo
81.3
75.8
37.2
26.8
Barn
78.4
68.0
40.0
29.0
Bikini
66.7
67.0
41.5
38.9
Digital clock
67.7
73.2
32.5
28.1
Dragonfly
86.6
84.8
36.1
13.6
Dumbbell
71.8
66.5
40.8
35.9
Mean AUC
79.3
75.5
39.2
26.3
Table 22: Class-wise AUC detection performance in % for the best and worst single OE samples found via evolutionary algorithm (Section 5.4 the ImageNet-30 one rest AD benchmark using ImageNet-222 (with the 1K classes removed) as OE All images are either low-pass-fileent filtered (LPF) or (HPF), both during training and testing.
Best OE
Worst OE
Class
HSC LPF
BCE LPF
HSC HPF
BCE HPF
HSC LPT
BCE LPF
HSC HPF
BCE HPF
Acorn
82.8
83.8
80.4
75.9
41.2
20.0
45.8
31.0
Airliner
78.2
77.3
83.6
86.3
44.6
21.7
52.5
18.0
Ambulance
86.7
76.3
79.1
83.3
53.6
26.1
44.4
29.2
American alligator
67.4
78.3
43.4
42.2
29.0
Banjo
83.0
74.8
71.3
75.9
48.0
21.5
39.1
31.9
Barn
79.6
71.5
73.0
77.6
45.2
25.6
44.4
26.2
Bikini
70.4
68.0
68.6
65.9
47.4
35.6
45.9
34.9
Digital clock
63.3
65.7
68.4
76.2
41.2
35.9
34.9
26.0
Dragonfly
51.4
Dumbbell
73.2
68.1
63.8
67.2
48.1
29.1
40.0
32.8
Mean AUC
77.2
73.8
75.0
77.3
44.6
26.1
44.1
27.9
41

---

Published in Transactions on Machine Learning Research (11/2022)
"airplane" is normal
(b) "automabile" is normal
(c) "bird" is normal
(d) "cat" is normal
"deer" normal
"dog" normal
SC
(g) "frog" is normal
(h) "horse" normal
(i) "ship" is normal
(i) "truck" is normal
Figure 23: Optimal OE samples for CIFAR-10 with 80MTI as OE The first row shows normal samples, the next two rows the best samples found via HSC (top) and BCE (bottom), and the last two rows the worst samples found via HSC (top) and BCE (bottom).
42

---

Published in Transactions on Machine Learning Research (11/2022)
(a) "airplane" is normal
(b) "automobile" is normal
(c) "bird" is normal
(d) "cat" is normal
(e) "deer" is normal
(f) "dog" normal
Donate
(g) "frog" normal
(h) "horse" is normal
(i) "ship" is normal
"truck" is normal
Figure 24: Optimal OF samples for www-pass-ine filtered CIFAR-10 with low-pees-iittered 80MTT as OF. The first row shows normal samples, the next two rows the best samples found via HSC (top) and BCE (bottom), and the last two rows the worst samples found via HSC (top) and BCE (bottom). The last column shows the www-pass-filteedd filtered version of the images, which what the network sees during training and testing.
43

---

Published in Transactions on Machine Learning Research (11/2022)
nama
"airplane" is normal
(b) "automobile" is normal
(c) "bird" is normal
(d) "cat" is normal
(e) "deer" is normal
(f) "dog" normal
ADCHOM SWIT
(g) "frog" normal
(h) "horse" is normal
(i) "ship" is normal
"truck" is normal
Figure 25: Optimal OE samples for nigh-pass-filtered CIFAR-10 with high-pass-filtreed 80MTI OF. The first row shows normal samples, the next two rows the best samples found via HSC (top) and BCE (bottom), and the last two rows the worst samples found via HSC (top) and BCE (boltom). The last column shows the high-pass filtered version of the images, which is what the network sees during training and testing.
44

---

Published in Transactions on Machine Learning Research (11/2022)
"acorn" is normal
(b) "sirliner" is normal
"ambulance" is normal
JU
(d) "american alligator" nor- mal
(e) "banjo" normal
ff "barn" is normal
"bikini" normal
"digital clock" normal
(i) "dragonfly" normal
3
(i) "dumbbell" is normal
Figure 26: Optimal OE samples for ImageNetIk lk with ImageNet22k (with the 1K classes removed) OE. The first row shows normal samples, the next two rows the best samples found via HSC (top) and BCE (bottom), and the last two rows the worst samples found via HSC (top) and BCE (bottom).
45

---

Published in Transactions on Machine Learning Research (11/2022)
"acorn" normal
(b) "airliner" normal
(c) "ambulance" is normal
Musio
(d) "american alligator" mal
(e) "banjo" normal
"barn" is normal
(g) "bikini" normal
(h) "digital clock" normal
(i) "dragonfly" normal
(1) "dumbbell" is normal
Figure 27: Optimal OE samples for low-pass-filtere filtered ImageNet with low-pass-ilttred ImageNet22k (with the 1K classes removed) as OE. The first row shows normal samples, the next two rows the best samples found via HSC (lop) and BCE (bottom), and the last two the worst samples found via HSC (lop) and BCE (bottom). The last column shows the low-pass oow-pas--ilteeeed filtered version of the images, which is what the network soes during training and testing.
46

---

Published in Transactions on Machine Learning Research (11/2022)
"acorn" normal
(b) "airliner" normal
(c) "ambulance" is normal
(d) "american alligator" mal
(e) "banjo" normal
"barn" is normal
(g) "bikini" normal
(h) "digital clock" normal
(i) "dragonfly" normal
(1) "dumbbell" is normal
Figure 28: Optimal OF samples for high-pass-filtered ImageNet1k 1k with high-pass-filtreed ImageNet22k (with the 1K classes removed) as OE. The first row shows normal samples, the next two rows the best samples found via HSC (lop) and BCE (bottom), and the last two the worst samples found via HSC (lop) and BCE (bottom). The last column shows the high mass-filtered version of the images, which is what the network soes during training and testing.
47


Reviews:
[{"review_id": "Tt83jXmzG9", "reviewer": "Reviewer_Lp9d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. The paper poses a very interesting angle about AD problems and several common assumptions are contradicted. \n2. They provide surprising results that outperform the SOTA benchmarks by using only few OE examples, which sounds exciting. \n3. The paper offers a detailed discussion about the motivation, challenges, and current SOTA works, which is very helpful to follow their ideas. \n\nWeakness\n1. The empirical studies are strong enough but may lack some theoretical insights, and it will be great if the authors can provide additional explanations why the results look so surprising either from the theoretical side or intuitive side.\n2. The motivation for why you use CLIP is not clear. From the current version, just inspired by the previous work [Fort 2021]? \n3. The AD assumption in real-world applications.  In fact, OE examples are not easy to get in practice since we probably need additional efforts to define normal and anomaly. Unlike imagenet, we are easy to define which samples are normal or not but if we do not know, does the method still works well?   It involves the basic assumption of AD setting but we may have many challenges in defining OE in real-world applications even though only few samples are needed as the author mentioned. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows very interesting results that challenge common assumptions in deep anomaly detection. They found only a small number of Outlier exposure data is enough to outperform all end-to-end methods on common AD benchmarks. By leveraging transfer learning techniques, they propose zero-shot learning with CLIP methods and show superior performance on imagenet.  In addition, they found that semi-supervised one-class methods are more robust to the choice of OE when only few OE data are available. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, the paper is well-written and easy to follow. I am glad to read these interesting findings and recommend to accept. \n\nIn the current stage, I have no big concerns to change but it would be helpful if the authors can well address the weakness I listed above.  ", "broader_impact_concerns": "I did not find broader impact concerns ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JvgNg70Fjh", "reviewer": "Reviewer_x73K", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper is clearly written.\n- Several observations made in the paper could be of interest to the community, e.g., the zero-shot AD performance of CLIP, and OE with fewer samples.\n\nWeaknesses\n\n- The paper proposes a new form of OE, but its novelty over Deep-SVDD is not actually delineated, as I feel it is quite a direct extension of Deep-SVDD. Also, I do not get why the proposed method is more “principled modification” over the previous approaches, that as mentioned in Section 3.2.\n- The experiments are conducted with comparably narrow one-class benchmarks of CIFAR-10 and ImageNet-30, with a single major performance metrics of AUC. I generally feel that the claims made in the paper requires a more extensive evaluation: e.g., to support whether OE indeed needs fewer samples to improve semantic anomaly detection.\n- The technical novelty of the method seems limited: for the supervised OE results (Section 5.1), the improvements of the proposed losses is quite marginal over the standard OE. In case of few-shot results (Section 5.2), I feel the results are not as dramatic as claimed - it still requires ~32K OE samples to reliably recover the full-shot results, and the few-shot results (I consider it as from 2^1 ~ 2^4) do not actually improve much from the zero-shot results. Also, the experiments do not verify whether this trends will persists on other OOD benchmarks.\n- The results in Section 5.4 is somewhat inconclusive: nether HSC and BCE shows much robustness on varying OE samples - for such a claim, I personally think the evaluation benchmark should need more diversity and thoroughness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on implementing Outlier Exposure (OE) in anomaly detection (AD) as a binary classification problem, and considers to cross-entropy based losses, namely BCE and HSC, where the latter adopts the radial basis kernel and turns out to be an extension of Deep-SVDD with negative (outlier) samples. With these losses, the paper observes that (a) this “supervised” version of OE can outperform OE, and (b) the full effectiveness of OE can be actually made with less (not much less in my opinion, though) outlier samples. Next, the paper considers a simple zero-shot AD method via CLIP, showing that this can achieve near-perfect performances on the one-class benchmarks of CIFAR-10 and ImageNet-30. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Major requests\n\n- I feel the experiment in Section 5.2 could be strengthen if it can provide results from the standard OE, i.e., results OE with fewer samples. If the proposed method can show better few-shot efficiency, it will be a good addition.\n- Results on more datasets and diverse evaluation metrics would directly strengthen the paper.\n- It is quite confusing that the paper conducts one-class experiments on ImageNet-30 or (the full) ImageNet-1k - although Section 4 says as the former (30), but the captions on the remaining tables says as the latter (1k). The paper should clarify that.\n- Missing error bars in all the tables, while the paper states that all the experiments are averaged over 10 seeds.\n\nMinor requests\n\n- I think Section 3 (”Methods”) could be better organized - namely, as Preliminaries and Methods. For me, it is quite hard to extract technical novelty from the existing works given the current text.\n- For the few-shot experiments, I think it would be interesting if the paper could explore the effect of data augmentation techniques to the detection performances.", "broader_impact_concerns": "None that I am aware of. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6jGEO17rM7", "reviewer": "Reviewer_tVkG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This submission is very well written and gives a nice overview if the SOTA in the field. I found it a pleasure to read, and I believe that its background can serve newcomers to the field very well.\n\nThe scope of the contributions of this submission is somewhat modest - an empirical investigation of several aspects of OE. Still, it is able to disambiguate several aspects of current methods e.g. if unsupervised losses are also used, how well OE works without transferring external weights. Also the precise experiments for OE accuracy as a function of OE dataset diversity and size is informative. \n\nOther advantages are the new SOTA number of OE with CLIP weights. I found the zero-shot CLIP results interesting - however it s limited scope should be acknowledged as it only applies when a single class name for the normal data exists and is known - which may not always be the case.\n\nThe submission has some weaknesses. I think the primary one is that results are only presented for Cifar10 and ImageNet when OE works well (as OE similar to the anomalies can be found). OE is less successful when the OE dataset is not similar to the anomalies - this is the main reason why it cannot be used in many real cases. This is only partially acknowledged and not investigated. In this reviewer's view, this is a critical limitation.\n\nThe unsupervised method used here (SVDD) is no longer the leading self-supervised AD method, it is conceivable that a combination of CSI/DROC with OE might give more favorable tradeoffs for the unsupervised setting. So the investigation is incomplete in this aspect.\n\nFinally, the limited scope of the publication is a limitation but in my understanding of the TMLR guidelines, not a critical one.\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates outlier exposure (OE) for anomaly detection (AD) on two commonly used object classification datasets (Cifar10 and ImageNet). Three main settings are investigated: unsupervised, supervised and pretrained. Different OE hyparameters are tested: number and diversity of images. Additionally a zero-shot setting is evaluated which merely relies on the supervision of the normal class label. Results show that even small OE dataset can achieve strong performance and that standard AD losses eg. SVDD nearly do not help at all when OE are available.   \n\nNew knowledge:\n* While the knowledge that not many OE images are needed is not really new given Deecke et al. and Reiss et al., I am not aware of a precise investigation of the accuracy vs. number of images in OCC case.\n* The exact interplay between SVDD + OE losses has not been sufficiently investigated before, although the results are expected.\n* SOTA numbers for transfer OE using CLIP weights.\n* I think the zero-shot CLIP experiment is new for the OCC setting (although some results were presented in Fort et al., NeurIPS'21)", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "While the contributions of this work are modest - it adds more knowledge. None the less, a precise contributions statement with the claimed contributions should be added. The discussion above can serve as aid for their specification.\n\nOE should be examined on other datasets beyond Cifar10 and ImageNet. In my experience it works far less effectively on other datasets and current papers give a misleading sense of success. There are many datasets that can be evaluated such as CUB, Dynamic textures, MVTec (already mentioned in the text as failuure case, but a proper experiment should be performed), F/MNIST etc. The leave-one-class-out setting should also be investigated as both SVDD and OE will be less effective there. This reviewer does not think an OE paper is acceptable without such an investigation.  \n\nThis reviewer does not find the small number of images needed for OE surprizing - this is highly expected given the SOTA in few-shot learning and can probably even be improved further. I suggest removing the \"surprise\" from the text. ", "broader_impact_concerns": "Broader impact is sufficiently addressed", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received one Accept (Reviewer Lp9d), one Leaning Accept (Reviewer tVkG), and one Leaning Reject (Reviewer x73K). Reviewer x73K's main concerns are related to the technical novelty of the paper. The authors addressed these concerns in their answer, but did not manage to convince the reviewer.

Nevertheless, all reviewers (including Reviewer x73K) acknowledge that some of the observations made by the authors in their study were unknown in the community and are of interest. The reviewers also acknowledge that the paper is very well written and that the experimental evaluation is mostly solid. 

As mentioned before, there is however and exception to this, with Reviewer tVkG requesting additional results on datasets mentioned in their review for the readers to get (quoting the reviewer) "a better sense of the (limited) success of OE". We believe this to be a valid suggestion and therefore recommend the authors to follow it for the final version of the paper.
Decision: Accept (poster)