---
abstract: 'We critically appraise the recent interest in out-of-distribution (OOD) detection, questioning the practical relevance of existing benchmarks.  While the currently prevalent trend is to consider different datasets as OOD, we posit that out-distributions of practical interest are ones where the distinction is semantic in nature, and evaluative tasks should reflect this more closely.  Assuming a context of computer vision object recognition problems, we then recommend a set of benchmarks which we motivate by referencing practical applications of anomaly detection.  Finally, we explore a multi-task learning based approach which suggests that auxiliary objectives for improved semantic awareness can result in improved semantic anomaly detection, with accompanying generalization benefits.' 
author:
- |
    Faruk Ahmed[^1]\
    Aaron Courville[^2]\
bibliography:
- 'references.bib'
title: Detecting semantic anomalies
---

Introduction
============

In recent years, concerns have been raised about modern neural network based classification systems providing incorrect predictions with equally high confidence [@guocalibration].  A possibly related finding is that classification-trained CNNs find it much easier to overfit to low-level properties such as texture [@shapeiclr2019], canonical pose [@rotatedtruck], or contextual cues [@terraincognita] rather than learning globally coherent characteristics of objects.  A subsequent worry is that such a classifier trained on data sampled from a particular distribution is likely to be misleading when encountering novel situations in deployment.  For example, silent failure might occur due to equally confident categorization of an unknown object into a known category, due to having overfit to local biases.  This last concern is one of the primary motivating reasons for wanting to be able to detect when test data comes from a different distribution than that of the training data.  This problem has been recently dubbed as *out-of-distribution* (OOD) detection in [@danbaseline], and is also synonymously referred to as *anomaly detection* or *novelty detection* in the contemporary deep learning context.  Evaluation is typically carried out by benchmarks of the style proposed in [@danbaseline], where different datasets are treated as OOD after training on a particular in-distribution dataset.  This area of research has since steadily developed [@odin; @oe; @taylor; @multiplelabels; @kimin], with similarly themed additions to the tasks, and improved results in OOD detection on them. 

#### Current benchmarks are ill-motivated

Despite such tasks rapidly becoming the standard benchmark for OOD detection in the community, we suggest that, taken as a whole, they are not very well-motivated.  For example, the object recognition dataset [<span style="font-variant:small-caps;">CIFAR-10</span>]{}(consisting of images of objects placed in the foreground), is typically trained and tested against noise, or different datasets such as downsampled [<span style="font-variant:small-caps;">LSUN</span>]{}(a dataset of scenes), or <span style="font-variant:small-caps;">SVHN</span> (a dataset of house numbers), or [<span style="font-variant:small-caps;">tiny-Imagenet</span>]{}(a different dataset of objects).  For the simpler cases of noise, or datasets with scenes or numbers, low-level image statistics are sufficient to tell them apart.  While the latter choice of [<span style="font-variant:small-caps;">tiny-Imagenet</span>]{}might seem reasonable, it has been noted that particular datasets have particular biases related to the specific data collection and curation quirks [@datasetbias; @Tommasi2017], which renders the problem of treating different datasets for OOD detection questionable.  It is possible that we are only getting better at distinguishing such idiosyncrasies [@haters] (also see Appendix C).  Perhaps due to these tasks being intrinsically trivial, most approaches typically report very flattering performance on them. 

#### Semanticity is relevant

Therefore, we call into question the practical relevance of these benchmarks, which are currently treated as standard by the community.  While they might have some value as a testbed for diagnosing peculiar pathologies [@glowfail], we argue that in practical contexts of being able to detect novelty, the nature of novelty that is of primary concern is *semantic* in nature.  The implicit goal for the current style of benchmarks is that of detecting a wide variety of covariate shifts, which mostly consist of extraneous factors.  We suggest that this is misguided: non-semantic covariate shift is something we should be robust to, while semantic covariate shift should be flagged down.  For example, a shift in low-level properties, such as lighting for instance, or even a contextual one, such as a shift in background detail, should not deter an object recognizer from performing a correct categorization.  Quite the contrary, we would like our classifiers to systematically generalize [@fodor; @lakeandbaroni; @dimaandshikhar], arriving at decisions by assimilating a globally coherent understanding of the relevant semantics for the task.  For example, object recognizers should successfully recognize a cow even when it is sitting on a seashore, without getting confused by the atypical context [@terraincognita].  Similarly, we would like action classifiers to correctly identify an activity as *running* even when the running person is wearing boxing gloves, and not misclassify the action due to having associated *boxing* only with the presence of boxing gloves; a scene should be correctly categorized as a *bedroom* by a scene classifier even if there is a large picture of a church on the wall. 

#### Context matters

The connotations carried by the terms *anomaly detection* and *novelty detection* are unclear without context, and convey meaning only after acknowledging the semantics of interest for a given task.  For example, in scene classification, the concerned semantic of interest is the composition of components from across the entire frame.  In the context of such a task, a kitchen with a bed in the middle is an anomalous observation (if such a composition has not been previously observed).  However, when performing object recognition, the concerned semantic is not the composition of components anymore, but the identity of the foreground object, and now the unusual context should not prevent correct object recognition.  Similarly, in activity recognition from videos, the concerned semantic is the temporal evolution of the action across frames.  The task of *out-of-distribution detection* as presented in [@danbaseline], presumably intended to be broader and more flexible, suffers from imprecision due to this context-free breadth.  Being unclear about the nature of the out-distribution leads to a corresponding lack of clarity about the implications of underlying methodologies in proposed approaches.  The current benchmarks and approaches therefore carry a risk of potential misalignment of evaluative performance with field performance in practical anomaly detection problems. 

In this paper, we offer the following contributions. 

*1.  Semantic anomalies are what matter:* We provided a motivating discussion about the relevance of semanticity in the context of a task when we think about practical anomaly detection problems. 

*2.  More meaningful benchmarks for anomaly detection:* Based on this discussion, and real life examples of anomaly detection (Section 2), we recommend a style of benchmarks that are more pragmatic than the currently adopted set of tasks.  In our application-specific discussions, we shall presume the context of an object recognition task, but the theme applies generally. 

*3.  Improved semantic awareness improves semantic anomaly detection:* Following our discussion about the importance of semanticity, we investigate the effectiveness of multi-task learning with auxiliary self-supervised objectives (Section 4,5).  We evaluate representative anomaly detection methods on our recommended benchmarks, and demonstrate that such augmented objectives can result in improved anomaly detection, as well as accompanying generalization benefits. 

Motivation and proposed tasks
=============================

In order to develop meaningful benchmarks, we begin by considering some practical applications where being able to detect anomalies, in the context of classification tasks, would find use. 

*Nature studies and monitoring:* Biodiversity scientists want to keep track of variety and statistics of species across the world.  Online tools such as *iNaturalist* [@inaturalistapp] enable photo-based classification and subsequent cataloguing in data repositories from pictures uploaded by naturalists.  In such automated detection tools, a potentially novel species should result in a request for expert help rather than misclassification into a known species, and detection of undiscovered species is in fact a task of interest.  A similar practical application is camera-trap monitoring of members in an ecosystem, notifying caretakers upon detection of invasive species [@pests; @willi].  Taxonomy of collected specimens is often backlogged due to the human labour involved.  Automating digitization and identification can help catch up, and often new species are brought to light through the process [@historical], which obviously depends on effective detection of novel specimens. 

*Medical diagnosis and clinical microbiology:* Online medical diagnosis tools such as *Chester* [@chester] can be impactful at improving healthcare levels worldwide.  Such tools should be especially adept at being able to know when faced with a novel pathology rather than categorizing into a known subtype.  Similar desiderata applies to being able to quickly detect new strains of pathogens when using machine learning systems to automate clinical identification in the microbiology lab [@plosone]. 

*AI safety:* Amodei et al.  [@amodei] discuss the problem of distributional shift in the context of autonomous agents operating in our midst, with examples of actions that do not translate well across domains.  A similar example in the vein of [@amodei], grounded in a computer vision classification task, is the contrived scenario of encountering a novel vehicle (that follows different dynamics of motion), which might lead to a dangerous decision by a self-driving car which fails to recognize unfamiliarity.  Having compiled the examples above, we can now try to come up with a set of tasks more aligned with realistic applications.  The basic assumptions we make about possible evaluative tasks are: (i) that anomalies of practical interest are semantic in nature; (ii) that they are relatively rare events whose correct detection is of more primary relevance than minimizing false positives; and (iii) that we do not have access to examples of anomalies (as some existing works assume [@odin; @kimin]).  These assumptions guide our choice of benchmarks and evaluation. 

  Subset                      Number of members   Total training images   Total test images
  -------------------------- ------------------- ----------------------- -------------------
  Dog (*hound dog*)                  12                   14864                  600
  Car                                10                   13000                  500
  Snake (*colubrid snake*)            9                   11700                  450
  Spider                              6                   7800                   300
  Fungus                              6                   7800                   300

  : Sizes of proposed benchmark subsets from ILSVRC2012.  Sample images are in the Appendix.  The training set consists of roughly 1300 images per member, and 50 images per member in the test set (which come from the validation set images in the ILSVRC2012 dataset). 

\[tab:benchmark\]

#### Proposed benchmarks

A very small number of recent works [@ganomaly; @zenati] have considered a case that is more aligned with the goals stated above.  Namely, for a choice of dataset, for example [<span style="font-variant:small-caps;">MNIST</span>]{}, train as many versions of classifiers as there are classes, holding out one class every time.  At evaluation time, score the ability of being able to detect the held out class as anomalous.  This is a setup more clearly related to the task of being able to detect semantic anomalies, holding dataset-bias factors invariant to a significantly greater extent, and incorporating sufficient variability (every class is held out once) to be maximally informative and diagnostic.  In this paper, we shall explore this setting with [<span style="font-variant:small-caps;">CIFAR-10</span>]{}and [<span style="font-variant:small-caps;">STL-10</span>]{}, and recommend this as the default benchmark for evaluating anomaly detection in the context of object recognition.  Similar setups apply to different contexts. 

While the hold-out-class setting for [<span style="font-variant:small-caps;">CIFAR-10</span>]{}and [<span style="font-variant:small-caps;">STL-10</span>]{}is a good setup for testing anomaly detection of disparate objects, a lot of applications, including some of the ones we described earlier, require detection of more fine-grained anomalies.  For such situations, we propose a suite of tasks comprised of subsets of [<span style="font-variant:small-caps;">Imagenet</span>]{}(ILSVRC2012 [@imagenet2012]), with fine-grained subcategories.  For example, the <span style="font-variant:small-caps;">spider</span> subset consists of members *tarantula*, *Argiope aurantia*, *barn spider*, *black widow*, *garden spider*, *wolf spider*.  We also propose <span style="font-variant:small-caps;">fungus</span>, <span style="font-variant:small-caps;">dog</span>, <span style="font-variant:small-caps;">snake</span>, and <span style="font-variant:small-caps;">car</span> subsets.  These subsets have varied sizes, with some of them being fairly small (see Table \[tab:benchmark\]).  Although this is a significantly harder task, we believe this setting aligned with the practical situations we described above, where sometimes large quantities of labelled data are not always available, and a particular fine-grained selection of categories is of interest.  We emphasize that the point of such benchmarks, as with most benchmarks, is to provide a reasonably well-aligned and general setup for researchers to develop and investigate approaches to anomaly detection, while not being overwhelmingly expensive to run.  For very particular tasks, such as our motivating examples described above, we advise practitioners to curate particular benchmarks to evaluate fine-tuned models upon. 

#### Evaluation

Current works tend to mainly use both Area under the Receiver-Operator-Characteristics (AUROC) and Area under Precision-Recall curve (AUPRC) to evaluate performance on anomaly detection.  The AUROC score is more easily interpretable, being a measure of how likely a randomly chosen positive example (an anomaly in our case) is to be scored higher than a randomly chosen negative one.  However, in situations where positive examples are not only much rarer, but also of primary interest for detection, AUROC scores are a poor reflection of detection performance; *precision*, which is the proportion of true positives among positive detections, is more relevant [@rocfawcett; @prroc; @Avati2018].  We shall not inspect AUROC scores because in all of our settings, normal examples significantly outnumber anomalous examples, and AUROC scores are known to be often overly optimistic and hence potentially misleading in imbalanced datasets [@prroc].  Additionally, it has been shown in [@prroc] that optimizing for performance under AUROC does not imply optimizing for AUPRC.  Precision and recall are calculated as $$\begin{aligned}
    \text{precision} &=& \frac{\text{true positives}}{\text{true positives + false positives}}, \\
    \text{recall} &=& \frac{\text{true positives}}{\text{true positives + false negatives}},\end{aligned}$$ and a precision-recall curve is then defined as a set of precision-recall points $$\begin{aligned}
    \text{PR curve} \triangleq \{\text{recall}(t), \text{precision}(t), -\infty < t  < \infty\},\end{aligned}$$ where $t$ is a threshold parameter. 

The area under the precision-recall curve is calculated by varying the threshold $t$ over a range spanning the data, and creating a finite set of points for the PR curve.  One alternative is to interpolate these points, producing a continuous curve as an approximation to the true curve, and computing the area under the interpolation by, for example, the trapezoid rule.  Interpolation in a precision-recall curve can sometimes be misleading, as studied in [@averageprecision], who recommend a number of more robust estimators.  Here we use the standard approximation to average precision as the weighted mean of precisions at thresholds, weighted by the increase in recall from the previous threshold $$\begin{aligned}
    \text{average precision} = \sum_k \text{precision}_k ( \text{recall}_k - \text{recall}_{k-1}),\end{aligned}$$ which is the implementation in Scikit-Learn [@sklearn]. 

Related work
============

#### Evaluation

As we have discussed earlier, the style of benchmarks widely adopted today follows the recommendation in [@danbaseline].  Among follow-ups, the most significant successor has been [@odin] which augmented the suite of tests with slightly more reasonable choices: for example, they consider [<span style="font-variant:small-caps;">tiny-Imagenet</span>]{}as out-of-distribution for in-distrbution datasets such as [<span style="font-variant:small-caps;">CIFAR-10</span>]{}, which makes more sense than only comparing to noise and scene datasets, as in [@danbaseline].  However, on closer inspection, we find that [<span style="font-variant:small-caps;">tiny-Imagenet</span>]{}shares semantic categories with [<span style="font-variant:small-caps;">CIFAR-10</span>]{}, such as species of {dogs, cats, frogs, birds}, so it is unclear how such choices of evaluative tasks correspond to realistic anomaly detection problems.  Some work in the area of *open-set recognition* has come close to a realistic setup: in [@openset2], detection of novel categories is tested with a set of images corresponding to different classes that were discontinued in later versions of Imagenet, but later work [@openset4] relapsed into treating very different datasets as novel.  Also, using one particular split of a collection of unseen classes as OOD does not provide as varied and as finely diagnostic a view as holding out each class at a time, which is what we recommend.  As mentioned earlier, a small number of works in OOD detection [@ganomaly; @zenati] have already used the hold-out-class style of tasks for evaluation.  Unfortunately, due to a lack of a motivated discussion about the choice, the community has ignored this, and continue to adopt the set of benchmarks in [@danbaseline; @odin]. 

#### Approaches to OOD detection

In  [@danbaseline], the most natural baseline for a trained classifier is presented, where the detection score is simply given by the predictive confidence of the classifier (MSP).  Follow-up work in [@odin] proposed adding a small amount of adversarial perturbation, followed by temperature scaling of the softmax.  There is no ablation study, but we have found temperature scaling to be responsible for most of the gains over [@danbaseline].  Methodologically, the approach suffers from having to pick a temperature and perturbation weight per anomaly-dataset.  Confidence calibration has also been explored in [@taylor], and was shown to improve complementary approaches like MSP and ODIN. 

Using auxiliary datasets as surrogate anomalies has been shown to improve performance on existing benchmarks in [@oe].  This approach is limited, due to reliance on other datasets, but a more convincing variant has been explored in [@kimin] where a GAN is used to generate negative samples.  However, [@kimin] suffers from the methological issue discussed earlier, with hyperparameters being optimized per anomaly-dataset.  We believe that such contentious practices arise from a lack of a clear discussion of the nature of the tasks we should be concerned with, and a lack of grounding in practical applications which would dictate proper methodology.  The primary goal of our paper is to help fill this gap. 

Among existing works, the closest in terms of approaching the essence of our discussion about semantic properties being the discriminative factor of interest is [@multiplelabels], where the training set is augmented with semantically similar labels.  While we approve of the spirit, it is not always practical to assume access to a corpora providing such labels.  This is especially true for the motivating practical applications we described earlier.  In the next part of the paper, we explore a way to induce greater semantic awareness in a classifier, with the hope that such improved representation would lead to corresponding improvements in semantic anomaly detection and generalization. 

Encouraging semantic representation with multi-task learning
============================================================

We hypothesize that classifiers that learn representations which are more oriented toward capturing semantic properties rather than overfitting to low-level properties would naturally lead to better performance at detecting semantic anomalies.  Overfitting to low-level features such as colour or texture without consideration of global coherence might result in potential confusions in situations where the training data is not representative and unbiased.  For a lot of existing datasets, it is quite possible to achieve good generalization performance without learning semantic distinctions, a possibility that spurs the search for removing algorithmic bias [@fair], and which is often exposed in embarrassing ways [@racist; @ads].  As an extreme example, if the training and testing data consists of only one kind of animal which is furry, the classifier only needs to learn to detect fur-texture, and can ignore other meaningful characteristics such as the number of legs.  Such a system would fail to recognize another furry, but nine-legged animal as anomalous, while achieving good test performance.  Motivated by this line of thinking, we ask the question of how we might encourage classifiers to learn more semantically meaningful representations. 

Caruana [@montreal] describes how sharing parameters for learning multiple tasks, which are related in the sense of requiring similar features, can be a powerful tool for inducing domain-specific inductive biases in a learner.  Hand-design of inductive biases requires complicated engineering, while using the training signal from a related task can be a much easier way to achieve similar goals.  Even when related tasks are not explicitly available, it is often possible to construct one.  A recent example of an application of this idea is [@rei-2017-semi], where the task of sequence labelling is augmented with the auxiliary self-supervised task of predicting neighbouring words, with shared representation learning parameters.  This was found to improve performance on the original task of sequence labelling. 

                             [<span style="font-variant:small-caps;">CIFAR-10</span>]{}   [<span style="font-variant:small-caps;">STL-10</span>]{}
  ------------------------- ------------------------------------------------------------ ----------------------------------------------------------
  Classification only                             $95.87 \pm 0.05$                                            $85.51 \pm 0.17$
  Classification+rotation                         $96.54 \pm 0.08$                                           $88.98 \pm 0.30 $

  : Multi-task augmentation with the self-supervised objective of predicting rotation improves generalization.  Subsequently, we investigate if this improvement also correlates to improved semantic understanding, as measured indirectly through the task of detecting semantic anomalies. ( Results are averages over 3 trials.) 

\[table:rotationhelps\]

We explore such a framework for augmenting object recognition classifiers with auxiliary tasks that require semantic understanding.  Recently, there has been significant interest in self-supervision applied to vision [@doersch2015; @pathak2016; @noroozi2016; @zhang2017; @cpc; @gidaris2018; @cluster], exploring tasks that induce semantic representations without requiring expensive annotations.  These tasks naturally lend themselves as auxiliary tasks for encouraging inductive biases towards semantic representations.  Here, we use the recently introduced rotation task of [@gidaris2018], which asks the learner to predict the orientation of a rotated input.  This blends in most readily in the standard multi-task learning setup, where typically there is hard sharing of representation learning parameters, with a shared input space and addition of an extra task at the end.  A shortcut solution is possible in such a setting where the networks learns to use a subset of the parameters in the shared representation to predict rotation, and the other part for predicting object category, leading to no improvements.  Empirically, we find that this does not appear to happen in our settings: in Table \[table:rotationhelps\], we show significantly improved generalization performance of classifiers on [<span style="font-variant:small-caps;">CIFAR-10</span>]{}and [<span style="font-variant:small-caps;">STL-10</span>]{}when augmented with the auxiliary task of predicting rotation.  Details of experimental settings, and the correspondence to anomaly detection, are in the next section. 

We note that the application of strategies for encouraging more semantic representations through multi-task learning is complementary to the choice of scoring anomalies, which has been the focus of most existing work.  Additionally, as in standard multi-task learning setups, it enables further augmentation with more auxiliary tasks [@mltselfsup], which we leave for future empirical exploration. 

Evaluation
==========

We study the two existing representative baselines of maximum softmax probability (MSP) [@danbaseline], and ODIN [@odin] on the proposed benchmarks.  For ODIN, it is unclear how to choose the hyperparameters for temperature scaling and the weight for adversarial perturbation without assuming access to anomalous examples, an assumption we consider unrealistic in most practical settings.  We fix $T = 1000$, which is the most common setting in [@odin], and $\epsilon = 5\text{e-}5$, which is the setting there for <span style="font-variant:small-caps;">CIFAR 80/20</span>.  We initially used $T = 1$, as suggested in [@odin] for <span style="font-variant:small-caps;">CIFAR-80/20</span>, but found that this setting underperforms for our hold-out-class experiments on [<span style="font-variant:small-caps;">CIFAR-10</span>]{}. 

![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/cifar10_anomaly_0.png "fig:") ![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/cifar10_anomaly_2.png "fig:") ![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/cifar10_anomaly_3.png "fig:")\
![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/stl10_anomaly_1.png "fig:") ![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/stl10_anomaly_2.png "fig:") ![ Plots of training and testing costs, accuracies, and average precision corresponding to hold-out-class experiments with three categories each from CIFAR-10 (top) and STL-10 (bottom), using the MSP method [@danbaseline].  While classification performance is not correlated with performance at anomaly detection (compare absolute test accuracy numbers with average precision scores across columns), the “pattern” of improvement in anomaly detection appears roughly related to generalization (compare the coarse shape of test accuracy curves with that of average precision curves across iterations), indicating that there is some connection between generalization and the ability to detect semantic anomalies.  In general, we should expect greater semantic understanding to lead to improvements in both generalization as well as anomaly detection.[]{data-label="fig:corr"}](figures/stl10_anomaly_3.png "fig:")

\[tab:cifar\]

Experimental settings
---------------------

#### Architectural/training choices for [<span style="font-variant:small-caps;">CIFAR-10</span>]{}and [<span style="font-variant:small-caps;">STL-10</span>]{}

Our base network for all [<span style="font-variant:small-caps;">CIFAR-10</span>]{}experiments is a Wide ResNet [@wrn] with 28 convolutional layers and a widening factor of 10 (WRN-28-10) with the recommended dropout rate of 0.3.  Following [@wrn], we train for 200 epochs, with an initial learning rate of 0.1 which is scaled down by 5 at the 60th, 120th, and 160th epochs, using stochastic gradient descent with Nesterov’s momentum at 0.9.  We train in parallel on 4 Pascal V100 GPUs with batches of size 128 on each.  For [<span style="font-variant:small-caps;">STL-10</span>]{}, we use the same architecture but append an extra group of 4 residual blocks with the same layer widths as in the previous group.  We also use a widening factor of 4 instead of 10, and batches of size 64 on each of the 4 GPUs.  We use the same optimizer settings as with [<span style="font-variant:small-caps;">CIFAR-10</span>]{}.  In both cases, we apply standard data augmentation of random crops (after padding) and random horizontal reflections. 

#### Architectural/training choices for [<span style="font-variant:small-caps;">Imagenet</span>]{}

For experiments with the proposed subsets of [<span style="font-variant:small-caps;">Imagenet</span>]{}, we replicate the architecture we use for [<span style="font-variant:small-caps;">STL-10</span>]{}, but add a downsampling average pooling layer after the first convolution on the images.  We do not use dropout, and use a batch size of 64; otherwise all other details follow the setup from the experiments on [<span style="font-variant:small-caps;">STL-10</span>]{}.  The standard data augmentation steps of random crops to a size of $224 \times 224$ and random horizontal reflections are applied. 

#### Auxiliary rotation prediction

For adding rotation prediction as an auxiliary task, we follow the standard recipe from typical multi-task learning setups where the representation space is shared through hard parameter sharing.  This means that all we do is append an extra linear layer alongside the one that is responsible for object recognition.  We weight the rotation loss with a co-efficient (0.5 for [<span style="font-variant:small-caps;">CIFAR-10</span>]{}, 1.0 for [<span style="font-variant:small-caps;">STL-10</span>]{}, and a mix of 0.5 and 1.0 for [<span style="font-variant:small-caps;">Imagenet</span>]{}), which we chose by performing a coarse hyperparameter search for the best combined performance at predicting object category and rotation on the validation set.  The optimizer and regularizer settings are kept the same, with the learning rate decayed along with the learning rate for the classifier at the same scales. 

\[tab:stl\]

We emphasize that this procedure is not equivalent to data augmentation, since we do not optimize the linear classification layer for rotated images.  Only the rotation prediction linear layer gets updated for inputs corresponding to the rotation task, and only the linear classification layer gets updated for non-rotated, object-labelled images.  Asking the classifier to be rotation-invariant would require the auxiliary task to develop a disjoint subset in the shared representation that is not rotation-invariant, so that it can succeed at predicting rotations.  This encourages an internally split representation, thus diminishing the potential advantage we hope to achieve from a shared, mutually beneficial space. 

Discussion
----------

#### Self-supervised multi-task learning is promising

In Tables \[tab:cifar\] and \[tab:stl\] we report average precision scores on [<span style="font-variant:small-caps;">CIFAR-10</span>]{}and [<span style="font-variant:small-caps;">STL-10</span>]{}for the baseline scoring methods MSP [@danbaseline] and ODIN [@odin].  We note that ODIN, with the hyperparameter settings we have fixed for all experiments, continues to outperform MSP most of the time.  Then, we augment our classifiers with the auxiliary rotation-prediction task and find that anomaly detection as well as test set accuracy are markedly improved for both scoring methods.  As we have remarked earlier, a representation space with greater semanticity should be expected to bring improvements on both fronts.  All results report mean $\pm$ standard deviation over 3 trials.  In Table \[tab:imagenet\], we repeat the same process for all of our proposed Imagenet subsets.  Full results, corresponding to individual members of the subsets, are in the Appendix, while here we only show the average performance across all members of the subset.  The results indicate that multi-task learning with self-supervised auxiliary tasks is a promising complement for anomaly detection, while also improving generalization. 

#### High test set accuracy does not imply semantic representation

By our hypothesis, training methods developed solely to improve generalization, without consideration of the affect on semantic understanding, might perform worse at detecting semantic anomalies.  This is because it is often possible to overfit to low-level or contextual discriminatory patterns, which are almost surely biased in small datasets for complex domains such as natural images, and perform reasonably well on the test set.  To illustrate this, we run an experiment where we randomly mask out a $16 \times 16$ region in [<span style="font-variant:small-caps;">CIFAR-10</span>]{}images from within the central $21 \times 21$ region.  We find that while this leads to improved test accuracies, anomaly detection suffers (numbers are averages across hold-out-class trials):

  Method                      Accuracy       Av.  Prec.  with MSP
  ---------------------- ------------------ --------------------
  Base model              96.03 $\pm$ 0.00    34.92 $\pm$ 0.41
  Random-center-masked    96.27 $\pm$ 0.05    34.41 $\pm$ 0.74
  Rotation-augmented      96.83 $\pm$ 0.02    40.22 $\pm$ 0.16

This hints that while the masking strategy is effective as a regularizer, it might come at the cost of less semantic representation.  Such training choices can therefore result in models with seemingly improved generalization but which have a poorer understanding of object coherence, due to potentially overfitting on biases in local statistics or contextual cues in the dataset.  For comparison, the rotation-augmented network achieves both a higher test set accuracy as well as an improved average precision.  This example might be yet another cautionary tale about developing techniques that might inadvertently lead to neural networks achieving reassuring test set performance while following an internal *modus operandi* very misaligned with the pattern of reasoning we hope they discover.  This can have unexpected consequences when such models are deployed in real life applications. 

\[tab:imagenet\]

Conclusion
==========

We provided a motivated discussion of what the goals of anomaly detection should be, grounded in the context of machine learning classification systems.  While there is significant recent interest in the area, current research suffers from questionable benchmarks and methodology.  In this paper, we propose a set of benchmarks which are better aligned with realistic applications. 

We also explored the effectiveness of a multi-task learning framework with an auxiliary self-supervised objective that has been shown to induce semantic representations.  Our results demonstrate improved anomaly detection along with improved generalization under the augmented objective.  This suggests that such semantic-learning inductive biases could have an important role to play in developing neural networks with improved semantic representations, in the context of the problem they are tasked with. 

Additionally, the ability to detect semantic anomalies also provides us with an indirect notion of semanticity in the representations learned by our mostly opaque deep models. 

Acknowledgements {#acknowledgements .unnumbered}
================

We thank Rachel Rolland for referencing and discussing the motivating examples of anomaly detection in nature studies.  Ishaan Gulrajani and anonymous reviewer R1 provided useful feedback.\
\
This work was enabled by the computational resources provided by Compute Canada. 

=0mu plus 2mu

Imagenet benchmarks
===================

In this section, we present details of the Imagenet-based benchmark we proposed.  For constructing these datasets, we first sorted all subsets by the number of members, as structured in the Imagenet hierarchy.  We then picked from among the list of top twenty subsets, with a preference for subsets that are more closely aligned with the theme of motivating practical applications we provided.  We also manually inspected the data, to check for inconsistencies, and performed some pruning.  For example, the *beetle* subset, while seeming ideal, has some issues with labelling: *leaf beetle* and *ladybug* appear to overlap in some cases.  Finally, we settled on our choice of 5 subsets.  The following table lists the members under every proposed subset. 

       Dog (hound)           Car       Snake (colubrid)        Spider             Fungus
  --------------------- ------------- ------------------ ------------------ ------------------
      Ibizan hound         Model T      ringneck snake       tarantula          stinkhorn
        bluetick          race car        vine snake      Argiope aurantia        bolete
         beagle          sports car     hognose snake       barn spider      hen-of-the-woods
      Afghan hound         minivan      thunder snake       black widow         earthstar
       Weimaraner         ambulance      garter snake      garden spider        gyromitra
         Saluki              cab          king snake        wolf spider        coral fungus
         redbone         beach wagon     night snake                        
       otterhound           jeep         green snake                        
   Norweigian elkhound   convertible     water snake                        
      basset hound          limo                                            
   Scottish deerhound                                                       
       bloodhound                                                           

\[tab:breakdown\]

In Figures \[hounddogs\],\[fig:cars\],\[fig:snakes\],\[fig:spiders\],\[fig:fungi\] we show samples of images.  The sets are collected by first resizing such that the shorter side is of length 256 pixels, followed by a center crop.  It is obivous that due to intrinsic dataset bias, some categories may be viewed as anomalous without careful inspection of the object of interest.  For example, owing to their smaller size, ringneck snakes are most often photographed when held in human hands, and race cars are usually pictured on race tracks.  Such dataset biases have historically been hard to account for, and we recommend more thoughtful curation of specific datasets for specific tasks for our proposed style of benchmarks to be more reflective of field performance. 

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/Ibizan.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/bluetick.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/beagle.jpg "fig:")

\

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/Afghan.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/Weimaraner.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/Saluki.jpg "fig:")

\

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/redbone.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/otterhound.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/elkhound.jpg "fig:")

\

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/basset.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/deerhound.jpg "fig:")

[0.25]{} ![ Dog (hound dog)[]{data-label="hounddogs"}](figures/dog/bloodhound.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/modelt.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/racer.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/sports.jpg "fig:")

\

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/minivan.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/ambulance.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/cab.jpg "fig:")

\

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/waggon.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/jeep.jpg "fig:")

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/convertible.jpg "fig:")

\

[0.25]{} ![ Car[]{data-label="fig:cars"}](figures/car/limo.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/ringneck.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/vine.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/hognose.jpg "fig:")

\

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/thunder.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/garter.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/king.jpg "fig:")

\

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/night.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/green.jpg "fig:")

[0.25]{} ![ Snake (colubrid)[]{data-label="fig:snakes"}](figures/snake/water.jpg "fig:")

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/tarantula.jpg "fig:")

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/aurantia.jpg "fig:")

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/barn.jpg "fig:")

\

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/blackwidow.jpg "fig:")

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/garden.jpg "fig:")

[0.25]{} ![ Spider[]{data-label="fig:spiders"}](figures/spider/wolf.jpg "fig:")

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/stinker.jpg "fig:")

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/bolete.jpg "fig:")

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/hen.jpg "fig:")

\

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/earthstar.jpg "fig:")

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/gyromitra.jpg "fig:")

[0.25]{} ![ Fungus[]{data-label="fig:fungi"}](figures/fungus/coral.jpg "fig:")

Expanded results for Imagenet-subset experiments
================================================

In this section, we show expanded results for the Imagenet experiments, corresponding to every hold-out-class experiment. 

  ---------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --
  *Dog*                                                                                                                                                         
  (lr)[2-4]{} (lr)[5-7]{} Anomalous dogs          MSP                ODIN             Accuracy             MSP                ODIN             Accuracy         
  Ibizan hound                              25.56 $\pm$ 2.43   26.34 $\pm$ 2.65   85.19 $\pm$ 0.65   22.85 $\pm$ 2.54   24.27 $\pm$ 2.60   85.99 $\pm$ 0.73     
  bluetick                                  34.37 $\pm$ 4.32   39.19 $\pm$ 1.43   85.50 $\pm$ 0.82   29.60 $\pm$ 4.39   31.70 $\pm$ 1.99   85.36 $\pm$ 0.59     
  beagle!                                    18.29 $\pm$ 1.54   17.05 $\pm$ 1.21   86.33 $\pm$ 1.18   19.79 $\pm$ 2.02   18.52 $\pm$ 1.69   86.37 $\pm$ 0.71     
  Afghan hound                              20.05 $\pm$ 3.07   18.69 $\pm$ 1.22   84.16 $\pm$ 0.59   20.62 $\pm$ 1.26   20.56 $\pm$ 1.37   83.44 $\pm$ 0.98     
  Weimaraner                                31.04 $\pm$ 2.22   36.87 $\pm$ 2.90   83.68 $\pm$ 0.36   27.65 $\pm$ 2.52   30.59 $\pm$ 1.03   83.62 $\pm$ 1.40     
  Saluki                                    26.64 $\pm$ 2.50   31.75 $\pm$ 2.01   83.76 $\pm$ 1.08   28.20 $\pm$ 1.46   29.27 $\pm$ 1.30   84.74 $\pm$ 0.19     
  redbone                                   17.93 $\pm$ 0.59   18.66 $\pm$ 0.59   86.61 $\pm$ 1.48   19.14 $\pm$ 0.39   19.54 $\pm$ 1.15   86.01 $\pm$ 1.10     
  otterhound                                22.71 $\pm$ 0.77   23.31 $\pm$ 0.83   84.50 $\pm$ 0.54   21.32 $\pm$ 3.71   22.90 $\pm$ 4.28   84.24 $\pm$ 0.56     
  Norweigian elkhound                       28.82 $\pm$ 2.16   36.55 $\pm$ 0.61   83.91 $\pm$ 1.35   34.64 $\pm$ 6.16   41.61 $\pm$ 6.55   84.33 $\pm$ 0.84     
  basset hound                              18.39 $\pm$ 0.91   16.23 $\pm$ 0.58   86.34 $\pm$ 0.76   21.33 $\pm$ 3.10   19.46 $\pm$ 1.72   86.45 $\pm$ 0.65     
  Scottish deerhound                        26.83 $\pm$ 2.97   26.52 $\pm$ 2.23   83.95 $\pm$ 0.61   26.64 $\pm$ 1.17   24.87 $\pm$ 0.52   83.80 $\pm$ 0.02     
  bloodhound                                16.43 $\pm$ 1.17   19.04 $\pm$ 0.91   87.17 $\pm$ 0.24   24.19 $\pm$ 6.23   25.42 $\pm$ 6.60   88.69 $\pm$ 0.84     
  Average                                   23.92 $\pm$ 0.49   25.85 $\pm$ 0.09   85.09 $\pm$ 0.14   24.66 $\pm$ 0.58   25.73 $\pm$ 0.87   85.25 $\pm$ 0.17     
  ---------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --

  ---------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --
  *Car*                                                                                                                                                         
  (lr)[2-4]{} (lr)[5-7]{} Anomalous cars          MSP                ODIN             Accuracy             MSP                ODIN             Accuracy         
  Model T                                   26.77 $\pm$ 1.21   31.20 $\pm$ 1.22   72.92 $\pm$ 0.49   32.09 $\pm$ 0.86   36.10 $\pm$ 1.84   72.52 $\pm$ 0.62     
  race car                                  22.48 $\pm$ 2.53   27.12 $\pm$ 3.90   79.65 $\pm$ 1.85   20.32 $\pm$ 5.47   22.41 $\pm$ 6.41   74.67 $\pm$ 3.39     
  sports car                                16.20 $\pm$ 1.77   13.86 $\pm$ 0.44   80.97 $\pm$ 1.97   16.80 $\pm$ 0.93   15.58 $\pm$ 0.96   81.00 $\pm$ 0.48     
  minivan                                   17.19 $\pm$ 2.57   17.78 $\pm$ 1.68   79.25 $\pm$ 1.89   17.32 $\pm$ 3.08   18.45 $\pm$ 2.67   80.01 $\pm$ 0.98     
  ambulance                                 11.13 $\pm$ 1.78   9.51 $\pm$ 0.97    75.71 $\pm$ 2.44   11.24 $\pm$ 0.84   10.61 $\pm$ 1.25   75.78 $\pm$ 0.31     
  cab                                       26.17 $\pm$ 2.42   27.93 $\pm$ 2.30   75.92 $\pm$ 3.77   28.57 $\pm$ 1.91   29.39 $\pm$ 2.52   76.74 $\pm$ 3.09     
  beach wagon                               24.82 $\pm$ 0.85   26.30 $\pm$ 2.00   78.75 $\pm$ 1.09   24.50 $\pm$ 1.64   25.22 $\pm$ 1.89   79.81 $\pm$ 1.27     
  jeep                                      25.47 $\pm$ 0.38   26.99 $\pm$ 2.70   74.67 $\pm$ 1.37   27.92 $\pm$ 5.01   27.74 $\pm$ 3.28   72.84 $\pm$ 0.47     
  convertible                               20.00 $\pm$ 2.63   18.35 $\pm$ 1.17   76.86 $\pm$ 0.81   15.32 $\pm$ 2.01   14.79 $\pm$ 2.24   76.26 $\pm$ 1.74     
  limo                                      25.16 $\pm$ 1.43   25.87 $\pm$ 0.83   77.04 $\pm$ 1.52   22.53 $\pm$ 2.08   23.49 $\pm$ 1.17   77.54 $\pm$ 1.19     
  Average                                   21.54 $\pm$ 0.62   22.49 $\pm$ 0.54   77.17 $\pm$ 0.10   21.66 $\pm$ 0.19   22.38 $\pm$ 0.46   76.72 $\pm$ 0.19     
  ---------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --

  ------------------------------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --
  *Snake*                                                                                                                                                         
  (lr)[2-4]{} (lr)[5-7]{} Anomalous snakes          MSP                ODIN             Accuracy             MSP                ODIN             Accuracy         
  ringneck snake                              20.18 $\pm$ 2.98   20.56 $\pm$ 2.78   71.08 $\pm$ 3.13   20.84 $\pm$ 0.77   23.22 $\pm$ 1.07   69.03 $\pm$ 0.46     
  vine snake                                  16.07 $\pm$ 4.51   17.19 $\pm$ 3.91   67.94 $\pm$ 3.96   15.94 $\pm$ 1.05   16.15 $\pm$ 0.96   72.65 $\pm$ 6.44     
  hognose snake                               16.82 $\pm$ 0.38   16.65 $\pm$ 0.46   67.95 $\pm$ 2.81   19.70 $\pm$ 1.22   19.32 $\pm$ 0.77   69.85 $\pm$ 2.50     
  thunder snake                               17.06 $\pm$ 2.94   19.18 $\pm$ 3.31   71.86 $\pm$ 7.58   21.26 $\pm$ 0.35   23.08 $\pm$ 0.70   69.34 $\pm$ 7.08     
  garter snake                                21.45 $\pm$ 4.35   22.16 $\pm$ 3.81   67.26 $\pm$ 2.19   22.67 $\pm$ 1.13   23.12 $\pm$ 1.83   68.29 $\pm$ 5.90     
  king snake                                  17.37 $\pm$ 0.39   16.55 $\pm$ 1.19   66.45 $\pm$ 5.38   19.47 $\pm$ 3.72   17.96 $\pm$ 2.74   68.13 $\pm$ 2.74     
  night snake                                 21.70 $\pm$ 4.01   20.50 $\pm$ 3.36   76.56 $\pm$ 0.78   23.28 $\pm$ 0.79   24.12 $\pm$ 1.26   78.71 $\pm$ 3.91     
  green snake                                 12.42 $\pm$ 3.31   13.49 $\pm$ 3.57   71.07 $\pm$ 6.41   13.15 $\pm$ 1.11   13.94 $\pm$ 0.23   71.74 $\pm$ 2.75     
  water snake                                 24.50 $\pm$ 3.10   26.36 $\pm$ 3.24   67.46 $\pm$ 6.52   25.77 $\pm$ 3.59   29.62 $\pm$ 5.04   66.85 $\pm$ 0.70     
  Average                                     18.62 $\pm$ 0.93   19.18 $\pm$ 0.79   69.74 $\pm$ 1.63   20.23 $\pm$ 0.18   21.17 $\pm$ 0.12   70.51 $\pm$ 0.48     
  ------------------------------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --

  ------------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --
  *Spider*                                                                                                                                                         
  (lr)[2-4]{} (lr)[5-7]{} Anomalous spiders          MSP                ODIN             Accuracy             MSP                ODIN             Accuracy         
  tarantula                                    19.45 $\pm$ 0.73   22.91 $\pm$ 2.37   60.67 $\pm$ 0.66   24.27 $\pm$ 3.25   26.07 $\pm$ 2.73   60.07 $\pm$ 0.86     
  Argiope aurantia                             12.97 $\pm$ 0.39   12.49 $\pm$ 0.48   69.70 $\pm$ 2.17   12.82 $\pm$ 0.51   12.01 $\pm$ 0.21   69.17 $\pm$ 1.57     
  barn spider                                  23.03 $\pm$ 3.03   23.83 $\pm$ 2.95   75.69 $\pm$ 0.55   21.41 $\pm$ 1.41   23.54 $\pm$ 1.21   76.56 $\pm$ 1.85     
  black widow                                  29.24 $\pm$ 4.39   37.96 $\pm$ 5.68   61.79 $\pm$ 0.87   37.08 $\pm$ 7.50   42.64 $\pm$ 8.87   62.63 $\pm$ 0.09     
  garden spider                                17.36 $\pm$ 1.33   15.51 $\pm$ 0.88   77.81 $\pm$ 2.29   16.57 $\pm$ 1.58   15.88 $\pm$ 1.42   76.38 $\pm$ 1.94     
  wolf spider                                  25.15 $\pm$ 2.98   32.23 $\pm$ 1.91   64.73 $\pm$ 0.45   25.48 $\pm$ 1.75   30.69 $\pm$ 1.28   67.19 $\pm$ 0.37     
  Average                                      21.20 $\pm$ 0.56   24.15 $\pm$ 0.72   68.40 $\pm$ 0.21   22.90 $\pm$ 1.29   25.10 $\pm$ 1.78   68.68 $\pm$ 0.77     
  ------------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --

  ----------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --
  *Fungus*                                                                                                                                                       
  (lr)[2-4]{} (lr)[5-7]{} Anomalous fungi          MSP                ODIN             Accuracy             MSP                ODIN             Accuracy         
  stinkhorn                                  52.43 $\pm$ 1.15   56.37 $\pm$ 1.98   90.91 $\pm$ 0.54   54.37 $\pm$ 4.65   59.10 $\pm$ 5.71   92.27 $\pm$ 0.97     
  bolete                                     51.04 $\pm$ 0.42   52.82 $\pm$ 3.10   89.19 $\pm$ 0.94   49.43 $\pm$ 2.05   53.07 $\pm$ 3.48   89.22 $\pm$ 1.09     
  hen-of-the-woods                           44.83 $\pm$ 1.52   48.04 $\pm$ 0.84   89.41 $\pm$ 1.64   48.87 $\pm$ 2.00   51.37 $\pm$ 2.44   90.13 $\pm$ 0.33     
  earthstar                                  34.90 $\pm$ 3.26   36.79 $\pm$ 2.16   86.70 $\pm$ 1.91   41.96 $\pm$ 7.66   43.24 $\pm$ 4.92   86.46 $\pm$ 0.62     
  gyromitra                                  46.75 $\pm$ 0.42   49.06 $\pm$ 2.64   86.79 $\pm$ 1.66   44.90 $\pm$ 1.94   49.20 $\pm$ 1.51   86.39 $\pm$ 0.18     
  coral fungus                               25.42 $\pm$ 2.60   24.44 $\pm$ 3.04   86.36 $\pm$ 1.25   25.58 $\pm$ 1.15   25.22 $\pm$ 2.81   86.35 $\pm$ 0.80     
  Average                                    42.56 $\pm$ 0.49   44.59 $\pm$ 1.46   88.23 $\pm$ 0.45   44.19 $\pm$ 1.86   46.86 $\pm$ 1.13   88.47 $\pm$ 0.43     
  ----------------------------------------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ -- --

Trivial baseline for OOD detection on existing benchmarks
=========================================================

To demonstrate that the current baselines are trivial with low-level details, we tested OOD detection with CIFAR-10 as the in-distribution by simply looking at likelihoods under a mixture of 3 Gaussians, trained at a pixel-level, channel-wise.  We find that this simple baseline performs quite well at the benchmarks in [@odin], as we show below:

  OOD dataset              Average precision
  ----------------------- -------------------
  TinyImagenet (crop)            96.84
  TinyImagenet (resize)          99.03
  LSUN                           58.06
  LSUN (resize)                  99.77
  iSUN                           99.21

We see that this method does not do well on LSUN.  When we inspect LSUN, we find that the images are cropped patches from scene-images, and are mostly of uniform colour, with little variation and structure in them.  We believe that this results in the phenomenon reported in [@glowfail], where one distribution that “sits inside” the other because of a similar mean but lower variance ends up being more likely under the wider distribution. 

We found that this simple baseline underperforms severely on the hold-out-class experiments on CIFAR-10, achieving an average precision of a mere 11.17% across the 10 tests, indicating that trivial statistics might not as useful at telling apart semantic anomalies, which are of greater practical interest. 

[^1]: Mila-Université de Montréal.  E-mail: faruk.ahmed@umontreal.ca

[^2]: Mila-Université de Montréal; CIFAR Fellow.  E-mail: aaron.courville@umontreal.ca
