UNSUPERVISED ANOMALY DETECTION FROM SEMANTIC SIMILARITY SCORES
Authors: Nima Rafiee
Rahil Gholamipoor
Markus Kollmann
Venue:  (2021)
Keywords: Anomaly Detection
Out-of-Distribution Detection
Novelty Detection

Abstract: In this paper we present SemSAD, a simple and generic framework for detecting examples that lie out-of-distribution (OOD) for a given training set. The approach is based on learning a semantic similarity measure to find for a given test example the semantically closest example in the training set and then using a discriminator to classify whether the two  examples show sufficient semantic dissimilarity such that the test example can be rejected as OOD.  We are able to outperform previous approaches for anomaly, novelty, or out-of-distribution detection in the visual domain by a large margin. In particular we obtain AUROC values close to one for the challenging task of detecting examples from CIFAR-10 as out-of-distribution given CIFAR-100 as in-distribution, without making use of label information. 

Under review as a conference paper at ICLR 2021
UNSUPERVISED ANOMALY DETECTION FROM SEMAN- TIC SIMILARITY SCORES
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we present SemSAD, a simple and generic framework for detecting examples that lie is based on learning a semantic similarity measure find for given test example the semantically closest example in the training set and then using discriminator to classify whether the two examples show sufficient semantic dissimilarity such that the test example can be rejected as OOD. We are able to outperform previous approaches for anomaly, novelty, or out-of of-distributton detection in the visual do- main by large margin. In particular the challenging task of detecting examples from CIFAR-10 as out-of-distribution given CIFAR- 100 as in-distribution, without making of label information.
(OOD) for a given training set. The approach
obtain AUROC values close to one for
INTRODUCTION
Anomaly detection or novelty detection aims at identifying patterns in data that are significantly dif- ferent to what is expected. This problem is inherently a binary classification problem that classifies examples either as in-distribution in-distribution (training set). A natural approach to OOD detection is to learn a density model from the training data and compute the likelihood ratio of OOD examples. However, in practice this ap- proach frequently fails for sigh-dimensional data (Nalisnick et al. (2019)) where has been shown that deep generative models can assign higher likelihood to OOD examples than to in-distribution examples. This surprising result is likely the consequence of how existing deep generative models generalise. For example, Variational Autoencoders (Kingma & Welling (2014)) generalise by su- perposition of examples, which is consequence of the stochastic nature of the posterior that can map different examples to the same point in latent space. As superposition an averaging process that reduces the information content it can be expected that examples of lower complexity than the training examples can map to high likelihood regions in latent space. Note that it is possible for datapoint to have high likelihood under distribution yet be nearly impossible to be sampled, property known as asymptotic equipartition property information theory Cover & Thomas 200T). For autoregressive generative models, such as PixeICNN (van den Oord al. (2016)). it has been shown that the axxelby-pixxt generation process is strongly determined by the local surrounding of pixels (Chen et al. (2018)) where the fact that nearby pixels of training examples frequently share the same color can explain why mono-chromatic images are assigned a high likelihood (Nalisnick) al. (2019). Local pixel correlations also seem be responsible for the failure of generative mod- els based on Normalising Flows to assign correct likelihood values to OOD examples Schirrmeister al. (2020).
out-of-distribution, given sufficiently large sample from the
As consequence, most of the current OOD detection approaches make use of a score function s(x) to classify test examples as in-distribution or OOD. In case that the examples of the training set are labelled, a simple score can be given by s(x) = max, p(y]z), with p the softmax probability for predicting class labels, y E {1,.., K} (Hendrycks & Gimpel| (2017)) If s(x) below threshold the test example is classified as OOD. Labelled data allows to learn representations that are associated with the semantic information shared by the examples in the training set, which can be used for OOD detection. However, the approach suffers from the problem that the scores for in-distribution examples can be widely distributed across the interval of possible score values, s(x) E [1/K, 1], especially if the number of labels low and the classification task is hard, which strongly increases the false-positive rate. Consequently, better performance was found for approaches that use labeled data for leamning higher dimensional representation that encodes for

---

Under review as a conference paper at ICLR 2021
semantic information (Lee et al. (2018b)) In this representation space the in-distribution occupies just a small volume and a random feature vector would be most likely classified as OOD. Another simplification arises if the OOD detection problem is supervised, with some OOD examples labelled as such and contribute to the training set. In this case the OOD detection problem boils down to an unbalanced classification problem (Chalapathy & Chawla (2019)). In general OOI) detection benefits from separating the factors of variation for the in-distribution in either relevant (e.g. object identity) or irrelevant (e.g. compression artefacts) using prior knowledge, where the relevant factors are typically those that carry salient semantic information. In line with the arguments put forward by Ahmed & Courville (2020), this separation helps an OOD model systematically generalise, e.g. whether we are allowed to re-colour or add noise to images for data augmentation. Generalisation over the training set necessary, as learning under insufficient inductive bias would result in misclassification of examples from an in-distribution test set as OOD. Labeled data provide this additional information, as relevant factors can be defined as those that help the classification task, with the limitation that there might be more factors involved in characterising the in-distribution than those needed to predict the labels.
In this work, we introduce general framework for OOD detection problems that does not require label information. Our framework can be widely applied to OOD detection tasks, including visual, audio, and textual data with the only limitation that transformations must be a priori known that conserve the semantics of training examples, such as geometric transformations for images, proxim- ity of time intervals for audio recordings (van den Oord et al. (2018)) or randomly masking a small fraction of words in sentence or paragraph (Devlin al. (2019). For visual data we show new tfat-oof-the-ar OOD classification accuracies for standard benchmark data sets, surpassing even the accuracies that include labels as additional information. The key contributions of this work are
We propose a new OOD detection framework that is applicable in absence of labeled in- distribution data or OOD examples that are labeled as such.
We show that our approach strongly improves OOD detection for challenging tasks in the visual domain
We find that identifying semantically close examples in the training set is central for reliable OOD detection
2 RELATED WORK
Unsupervised Methods using in-distribution labels. Many OOD detection methods make use of labels to generate scores that are either based on class prediction probabilities or on intermediate rep- resentations for an in-distribution classification task. For example, Hendrycks & Gimpel (Hendrycks) & Gimpel (2017)) used the maximum of the softmax probabilities (MSP) discriminate between OOD and in in-distribution. More recent approaches Lee et al. (2018a); Winkens et al. (2020); Zhang which density distribution (e.g.
et al. 2020) use labels to learn an intermediate representation multivariate normal distribution or deep generative network) can be fitted, which then can be used to compute the likelihood of OOD examples. As labels implicitly provide information about the semantic relation of examples in the training set, approaches using label information typically show higher accuracy than unsupervised methods. These approaches can be improved by introducing additional parameters or training strategies. For example, MSP was improved by introducing a tem- perature parameter (Liang al. (2018)) alternative losses (Lee et al. (2018a); V yas et al. (0188).
auxiliary objectives (Devries & loylor/ (2018): Hendrycks et al. (2019b); Mohseni et al. 2020
or outlier exposure (Hendrycks et al. (2019a)). Intermediate representations were improved using
multi-head network architecture (Shalev et al. (2018), contrastive learning Winkens et al. 2020
metric learning Masana et al. (2018)).
General Unsupervised Methods. If label information is absent, other means must be found to impose an inductive bias on the OOD detection model to generalise over the training set. Existing approaches can be separated in methods that learn generalisable features based on (i) self-supervised learning tasks Golan & El- Yaniy (0018), transformations that destroy semantics Choi & Chung 22000), match of encoder-decoder architectures Xiao et al. (2020), make use of semantically related auxiliary outlier distribution Schiirmeister et al. (2020). The work that is most related to ours is Geometric- Tranttrrc-TTansfommmatooo Classification (GEOM), proposed by Golan & El-Yaniy (2018)
2

---

Under review as a conference paper at ICLR 2021
and improved by Bergman & Hoshen (2020), which belongs to the class of self-supervisee learn- ing approaches (Hendrycks et al. (2019b)). The central idea of GEOM is to construct an auxiliary in-distribution classification task by transforming each image of the training set by one of 72 differ- ent combinations of geometric transformations with fixed strength, such as rotation, reflection, and translation. The task is predict which of the 72 transformations has been applied, given trans- formed image. GEOM gives examples that show high prediction uncertainty high OOD score. The relevant features learned by this task are salient geometrical features, such as the typical orientation of an object. Our approach differs from GEOM by the fact that we define the relevant features as those that are invariant under geometric and other transformations, such as cropping and color jitter, which are chosen of moderate strength to not change the semantics of the images in the training set.
METHOD
An intuitive approach for OOD detection is to Iearn representation that densely maps the in- distribution to a small region within a lower dimensional space (latent space), with the consequence that OOD) examples will be found outside this region with high probability. The representation should include the salient semantic information of the training set, to ensure that test examples from the in-distribution are not misclassified as OOD, but disregard irrelevant factors of variation that would prevent dense mapping. As learning this mapping by Autoencoders is difficult, we split the OOD detection task into finding semantically dense mapping of in-distribution onto a d-dimensional wnt--hypershhere by contrastive learning, followed by classifying neighbouring ex- amples on the unit-hypersphere as semantically close or distant.
3.1 LEARNING SEMANTIC SIMILARITY
A contrastive objective can be used align feature vectors h(z) e Rd that are semantically similar and at the same time distributes examples of the training set almost uniformly over the unit-hypersphere (Wang & Isola (2020); Chen al. (2020). This representation allows iden- tify for any test example the semantically close example from the training set. The mapping h(z)= f (x)/II](()) can be learned from training a deep neural network f(x) to minimise the
contrastive loss
Ma)TM(a')/r
[[h] -E ((aa") log E
Exrne TT, (a)) (chho))hhemeeal)
where 7 denotes a temperature parameter. Here, each positive pair x') is the result of sampling from distribution of transformations TT x²) that conserve semantics between ø and z', with (x"))) the marginal of Th xx). For datasets used to benchmark object recognition tasks, samples single example from the training and independently apply random transformations, such as geometric transformations, colour distortions, or cropping (Appendix D). The negative pairs can be generated by applying random transformations to different training examples. We emphasise that the types of transformations and their strengths essentially define the semantics we want to encode and thus determine for example, the image of a black swan is classified as OOD for an in dn-distribution that contains only white swans. The design of transformations that capture the underlying semanties of the training dataset requires either higher level understanding of the data or extensive sampling of different combinations of transformations with evaluation on an in-distribution validation set.
(x, Th (Ir, x²) can be generated by picking
3.2 LEARNING SEMANTIC DIFFERENCES
As the encoder h(x) maps any example on the unit-hypersphere, including OOD examples, we have the situation that OOD examples can be close to training examples without sharing semantic information. The obvious reason is that the contrastive objective homogeneously distributes training examples on unit-hypersphere, giving no preferred direction for OOD examples to cluster. As a consequence, the representation h(x) cannot be directly used for OOD detection. The idea is now to make use of the fact that nearby examples on uni--hypersphere share semantic information if both come from the in-distribution but don't share semantic information if one of the two examples is
3

---

Under review as a conference paper at ICLR 2021
In-distribution
Semantic Space
2
Samples
Training O OTest(indiittriuttion) @ Test (OOD)
Figure I: Illustration of mapping examples from the in-distribution onto a unit-hypersphere. In this representation, feature vectors from the in-istributtio are semantically similar if they approximately align and semantically diverse if they are separated by a large angle. If OOD examples are mapped onto the miithyyprsppher, they can align with training examples without being semantically similar. A discriminator trained to classify pairs of feature vectors from the training set semantically close or distant, with sharing or not sharing the same semantic neighhourhood as target, can then be used for detecting OOD examples.
OOD. We therefore train a score function s(x, x²) to detect semantic differences between nearby examples on the unit-hypersphere, which are determined by the readily trained encoder h(z)). A sta- tistically meaningful score can be given by the likelihood ratio between the probability Ppos of test example Frest and its next nearest neighbour the training set unezt = arg max, h²" (a))( to be semantically close in relation to the likelihood P Pneg that the same two examples semantically distant
(Tinett Ptest) ) = log Prox(Fnest; Jitest)
(2)
Preelenezt.: (Enezt: Ftest)
For the distribution of positive examples we use Ppos (z, 2¹) = (1-2) x²) 21 =lwesss)/((()) with 2 Bernoulli distributed random variable of mean μ. We introduced with S(x) the semantic neighbourhood of x, which is defined by the k-nearest neighbours of x, using cosine similarity h" (æ))(x') as semantic similarity measure (Fig. 2). The type of transformations are similar to Tr but with reduced strength to ensure that the relevant factors of variation are conserved (Fig. 3 and Appendix D). For negative examples we take Pnie x') T, (a') TT, with (z) the marginals, which implies that x and x' are almost always derived from two different images the training set. The negative transformations, T., are allowed to include stronger and more diverse transformations than Tp (Fig. and Appendix D). In principle negative examples can be constructed that are harder to classify, such augmenting P Pneg by pairs that are independent transformations of the same example (Choi & Chung 2020)). However, we found that this reduces the performance (Table 2). As shown in Appendix A, the score s" s"(x,, x') maximises the training objective J[s;; 7], given by
J[s:T] B(
log (a)) + -BB log a(a))]
Here, we defined (a) = 1/(1 + e=a) and a = s(x, x') - log(7) and realise the score function s(x,,x") by deep neural network. It can be shown that the optimal solution s* (x, x²) is invariant
4

---

Under review as a conference paper at ICLR 2021
Original
Weak
Strong
Extreme
CIFAR-10 semantic nelghbourhocd
orig nal
CIFAR-
original
se mamatiic neighbourhood
Figure 2: Semantic Neighbourhoods for examples from CIFAR-10/100.
Figure 3:
Transformation strengths used in training Ppos (weak to strong) and Pneg (weak to extreme)
to any variation in y > (Appendix A). We introduced y as is notoriously hard to learn ratios of probability densities in high dimensional spaces, which is central problem of generative ad- versarial networks (Azadi et al. (019)). In general, s(x, x²) learned by the objective Eq. 3 can deviate significantly from the optimal generalising likelihood ratio s" (x., This deviation is most apparent if Ppos (I, æ²) is close to zero where (2, x²) ) is non-zero and vice versa, as shown in Fig. 4. In this case the objective can be maximised by any decision boundary that lies in the region between the distributions Ppos (x,, x²) and Pneg (x, x'). To smoothen the score function s(x, x²) we sample y each iteration of the learning process and thereby effectively sample over an ensemble of gradients (Appendix B). Inspired by the lottery ticket hypothesis that training a deep neural net- work under constant objective mainly affects the weights of small subnetwork (Frankle Carbin (019)). ) we can reason that sampling over y affects the weights for an ensemble of overlapping sub- networks, As a consequence, s(x, x²) is the prediction from an ensemble models, which typically results in higher prediction accuracies, less variance with respect weight initialisation, and higher robustness to overfitting. The effect of uniform sampling of 7 on stabilising the decision bound- ary and thus observing the train/test sets from the in-distribution within the positive score range is shown in Appendix C. Although only the difference score values between test example and the in-distribution test set is relevant for OOD detection, examples from the in-distribution should be sufficiently distant from Preg(I, for optimal performance. can be further shown that for the extreme case > Do the score function learns the optimal weights to realise importance sampling for P Poos by sampling from Pneg (Appendix B).
4 TRAINING AND EVALUATION PROTOCOLS
4.1 TRAINING
Experiments were carried out using either ResNet18 or ResNet34 neural network architectures, for both the encoder f(x) and the discriminator s(x, x²). We substituted ReLU activations for the dis- criminator by ELU units to reduce the overconfidence induced by ReLU (Meinke Hein 2020 which resulted in strong reduction of unusually large spikes in the training loss curve for the dis- criminator. For contrastive learning a MLP head was attached to the last hidden layer that projects d 128 dimensional feature vector, h(r)), whereas for the discriminator the MLP head projects
5

---

Under review as a conference paper at ICLR 2021
cifar100-train
cf
cfer110-train-pes
cffar100-tren tra
cifar10-test
svhn-test
500
Frequency
400
300
200
100
-200
300
0OD score
Distributions over the OOD detection score, s(x, x²), trained on CIFAR-100 pos/neg pairs (Ppos in blue; Prey in red) as described in Section 4.1 and applied to semantic nearest- neighbour pairs from the test sets of SVHN and CIFAR-10 (out-dittributions) in comparison to semantic neirest-n pairs of the CIFAR-100 test/train sets (in-distributions).
Figure 4:
to scalar output, ((). Note that the ResNets for encoder and discriminator don't share param- eters. We train the contrastive loss at batch size 2048 and the discriminator batch size 128, using ADAM(AMSGrad) optimiser. We applied random transformations to each example in the training set before presenting them to encoder and discriminator. The transformations consist of combi- nations of random cropping followed by resizing to the original size, random horizontal flipping, random color jitter, grey scaling, and gaussian blurring (Appendix D). For training the encoder, h(x), we used the same transformations with the same strength reported in Chen et al. (2020)) We set the temperature parameter to T = which is the value reported in Winkens et al. (2020)) for the same datasets used in this work. Unless otherwise specified, positive pairs for training the discriminator are two independent transformations of a single image from the training set, where the transformation strength is bounded by strong transformations (Fig. and Appendix D), make sure that we don't transform out of the in-distribution. As pairs generated from independent trans- formations of the same image are typically semantically closer than any two semantically close images of the in-distribution test set (Fig. 2), the latter would be erroncously classified as OOD. To avoid mis-claasifiiation we augment the transformed positive pairs with fraction of semantic sim- ilar pairs, with pairing partners randomly selected from the semantic neighbourhood. The strength of augmentation is determined such that train/test sets from the in-distribution reside on the posi- tive OOD-score side yet remain inside the sensitive range of the logistic sigmoid function (Fig. 4). Unless otherwise specified, we take a semantic neighbourhood size of 4 and substitute a fraction 1/32 of transformed pairs in minibatch with semantically similar pairs from the training set. For regularisation, we use weight decay of 10-6 and uniformly sample f ~ U(1, 10) at each itera- tion of the learning process. Negative pairs constructed by transforming two different examples from the training set, including also 'extreme' transformations and gaussian blur.
4.2 EVALUATION
We evaluate the results using Area Under the Receiver Operating Characteristic curve (AUROC), which has the advantage to be scale-invariant measures how well predictions are ranked, rather than their absolute values and iassiiicattt--tteeeoodd---aaaata ples are separated from the training set. However, for any practical setting of OOD detection clas- sification threshold is needed and can be chosen such that the false positive rate of an in-distribution test set is close to some threshold, e.g. Precision-Ree Recall curve (AUPR) as in this work we have no class imbalance between the OOD test set and the in-distribution test-set. As observed significant shifts of OOD-sccoes for in-distribution
measures how well OOD sam-
0.05. We do not report values for Area Under the

---

Under review as a conference paper at ICLR 2021
Table I: Out-of-distributioo detection performance (% AUROC). Reported values for SemSAD lower bounds.
Din CIFARIO
CIFARIO CIFARI00
CIFAR 100
Method
D Dout SVHN
CIFARIO
Unsupervised methods using in-distribution labels
Softmax probs. (Hendrycks & Gimpel, 2017)
89.9
86.4
77.1
Liang et al. 2018)
96.7
85.8
77.2
Mahalanobis (Lee et al. 20188)
99.1
88.2
77.5
Residual flows (Zisselman & Tamar
2020
99.1
89.4
77.1
Outlier exposure [Hendrycks et al. 2019a
98.4
93.3
75.7
Rotation pred. (Hendrycks al. (019b)
98.9
90.9
Gram matrix (Sastry & Oore 2020
99.5
79.0
67.9
Contrastive Aug. (Winkens al.
2020)
99.5
92.9
78.3
OpenHybrid (Zhang et al. 2020)
99.8
95.1
85.6
General unsupervised methods
Likelihood Regret (Xiao et al. 2020)
86.6
SVD-RND (Choi & Chung 2020)
96.9
Hierachical-AD (Schirrmeister et al. 2020
99.0
86.8
62.5
SemSAD (Ours)
100
99.9
99.9
train/test sets between training runs (Appendix C). we suggest for any practical applications to carry out majority vote over 5 independent training runs, where after each example is classified as OOD if the OOD-score is significantly lower than the OOD-scores of the in-distribution test set.
EXPERIMENTAL RESULTS
In our experiments, we focus on difficult OOD detection problems in the visual domain (Nalisnick al. (2019)), in particular The main results are summarised in Table 1, where we used an identical setup (e.g. same hyperpa- rameters, transformation strengths, and network size) for all datasets and averaged AUROC values over 5 subsequent rums using ResNet1 8 and 5 subsequent runs using ResNet34. We compare our unsupervised method (SemSAD) with supervised methods that use label information and/or OOD examples that are labeled as such. Surprisingly, we find that SemSAD outperforms not only all unsupervised methods on these problems but also all supervised methods we are aware of. This result is especially striking as supervised methods typically outperform unsupervised methods, as semantic representations learned on label information are typically advantage. Our interpretation is that learning representation for the semantic information shared between pairs of examples allows to identify a larger set relevant features that characterise the in-distribution than from the large number of examples that share the same label. The more of the relevant features be identified the tighter the in-distribution can be characterised, which helps OOD detection. The performance gain our method strong for all OOD detection problems considered in this work but most apparent for CIFAR-100 100 as in-distribution and OOD examples from CIFAR-IO, with increase state-of-the-art AUROC for unsupervised methods using label information from 0.625 to 0.999. Note that the classes of CIFAR-10 and CIFAR- 100 are mutually exclusive, and thus CIFAR- can be used as OOD test set.
CIFAR-IOICIFAR-I00, and CIFAR-IOO/CIFAR
We carried out further experiments to see the effects of hyperparameter values and the transforma- tions used for training the discriminator (Table 2). As expected, if we destroy semantic information by using extreme transformations for generating positive (in-distribution) pairs the performance is significantly reduced, whereas Gaussian blur on negative examples has positive effect. The experi- ments further show that the semantic neighbourhood size should be taken small enough to make sure that the pairs generated from semantic neighbourhoods and used for training s(x,x') are semanti- cally close. In general, we observed better performance we broaden the distribution of negative pairs, e.g. by augmenting examples with gaussian blur and extreme transformations. The per-

---

Under review as a conference paper at ICLR 2021
Table 2: Average over AUROC values from 5 independent training for CIFAR-IO0CIIFAN (in/out distribution) for different setups. The lowest AUROC values among the 5 runs are shown in brackets. Reported AUROC values are lower bounds. We applied gaussian blurring on negative samples (blur), extreme transformations negative pairs (2, x') derived from extreme transformations of the same image (correlated neg), and changed the fraction of semantically similar pairs (μ)) per minibatch, the sampling range for % and the semantic neighbourhood size (N), AUROC is computed for CIFAR-100/10 test sets with 10k examples.
positive samples (extreme transf.), and using correlated
AUROC
blur
extreme transf.
A
7
N
correlated neg
0.980 (0.915)
1/32
L1 10
0.999 (0.999)
1/32
~ ~11 10
4
0.999 (0.999)
1/32
0.991 (0.963)
1/32
~ ~u11,
4
0.998 996
1/32
~1,11
32
0.729 (0.564)
1/32
(1,1
4
0.999 (0.999)
[0,
formance of our method is tightly connected to the ability of the encoder h(x) and discriminator s(x) to extract or sensitive to features that allow to generalise over the training-set and are thus specific to the in-distribution. These generalisable features are orthogonal to the features that change under the transformations we use training. The type transformations we use in this work, e.g. cropping, horizontal flip, and colour jitter, are generic in the sense that they are designed to conserve the semantics for images that are related to physical objects. In general, the type of transformations used must match the generalisable features of the in-distribution. For example, if all examples of in-distribution have a specific horizontal orientation, then horizontal flip must be excluded from the transformations.
CONCLUSION
In this work proposed SemSAD a new OOD detection approach that is based on scoring seman- tically similar pairs of examples from the in-distribution. We showed that our method outperforms supervised and unsupervised methods on challenging OOD detection tasks in the visual domain. The definition of semantic similarity within our approach requires to identify transformations that are applicable to individual examples and are orthogonal the salient semantic factors of the distribution. Although semantic similarity can be broadly defined "everything that is not noise", high predictive power can be expected if the semantic similarity score catches the higher order fea- tures that specific to the in-distribution. In practice, there are problems where the definition of semantic similarity is challenging. For example, if genome sequence data the input and the ef- fect on the phenotype of the organism is the true underlying semanties. In this case, it is unclear how transformations of the genome sequence should look like that lead to the same phenotype. In contrast, for the important problem of protein folding, such transformations multiple sequence alignments of sequences that likely conserved the function of a protein.
be inferred from
AUTHOR CONTRIBUTIONS
ACKNOWLEDGMENTS
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Faruk Ahmed and Aaron C. Courville. Detecting semantic anomalies. In AAAI, pp. 3154-3162, 2020.
Samanch Azadi, Catherine Olsson, Trevor Darrell, lan J. Goodfellow, and Augustus Odena. Dis- criminator rejection sampling. In International Conference on Learning Representations, 2019.
Liron Bergman and Yedid Hoshen. Classification-base anomaly detection for general data. In International Conference Learning Representations. ICLR, 2020.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey.
CoRR, abs/1901.03407, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/20M2..05709, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad. and Picter Abbeel. Pixelsnail: An improved au- toregressive generative model. In International Conference Machine Learning, pp- 863-871, 2018.
Sung-lk Choi and Sae- Young Chung. Novelty detection via blurring. In International Conference on Learning Representations, 2020.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, 2001. URL https: ///oi..or///1 1022207720/001
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In NAACL-HLT, 2019.
Terrance Devries and Graham W. Taylor. Learning confidence for cut-of-distributiio detection in neural networks. ArXiv, abs/ 04065. 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Izhak Golan and Ran EI- Yaniv. Deep anomaly detection using geometric transformations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems, pp. 978 9781-9791, 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pp. 15637-15648, 2019b.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibaate classifiers for detecting out-of-diitribution samples. In International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distributtion samples and adversarial attacks. In Adyances in Neural Information Process- ing Systems, 71 7167-7177, 2018b.
Shiyu Liang. Yixuan Li, and R. Srikant. Enhancing the reliability of out-of--istribution image detection in neural networks. In International Conference Learning Representations, 2018.

---

Under review as a conference paper at ICLR 2021
Mare Masana, Idoia Ruiz, Joan Serrat, Joost de Weijer, and Antonio M. López. Metric learning for novelty and anomaly detection. In British Machine Vision Conference, 2018.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't t know. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Sina Mohseni, Mandar Pitale, Jbs Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable dittri--ditiitttionn detection. In AAAI, 2020.
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Tch, Dilan Görür, and Balaji Lakshminarayanan.
Do deep generative models know what they don't 't know? In International Conference Learning Representations, 2019.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-o--distributiin examples with gram matrices. In International Conference on Machine Learning. 2020.
Robin Tibor Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. CoRR, abs/2006. 10848, 2020.
10848
Gabi Shalev, Yossi Adi, and Joseph Keshet. Out-of distributit detection using multiple semantic label representations. In Advances in Neural Information Processing Systems, pp. 7386-7396, 2018.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, 2016.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, CoRR, abs/1 807 307.03778, 2018.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-of-distribution detection using an ensemble self supervised leave-out classifiers. In Proceedings of the European Conference Computer Vision, pp. 560,574,77 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. CoRR, abs/2005. 100/20 2020.
J. Winkens, R. Bunel, Abhijit Guha Roy, Robert Stanforth, V. Natarajan, Joseph R. Ledsam, Patri- cia MieWiliisms, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, T. cemgil, S. Eslami, and O. Ronneberger. Contrastive training for improved out-of-distributtin detection. ArXiv, 05..20007.05666, 2020.
Zhisheng Xiao, Q. Yan, and Y. Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. ArXiv, abs/2003.00977 2020.
Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. CoRR, abs/2003. 12506, 2020.
Ev Zisselman and Aviv Tamar. Deep residual flow for novelty detection. CoRR, abs/2001.00 1.05119. 2020.
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
The objective
J[s; = -Ypos
log (a)) + Yney ( log (1 - a(a))
(4)
with (aa) = 1/(1+ + t = s(2, x²) + logl Ypos
/neg)). /nos///gg))
and = (Ypass kneg), has the upper
bound -[ 2 J[s;7] for Npos: Yang > O, where 8* is given by
x') = log Pro x')
(5)
P Preg(0r,0")
under the condition that pos and Peg have the same support that where Ppos is non-zero also Pneg is non-zero and vice versa.
To prove that assertion we make use of variational calculus (see e.g. C. Bishop, Patter Recognition and ML, Appendix D) to compute the functional derivative 6 which defined by the integral over arbitrary test function, 712
J(s + enj
/
8J [s ös(y, y²) n(y, ddydd
(6)
de
l==0
/ [hoo Ppo x²) o(a)) Pneg (nne(a, (a)( m(z, of dardd' (7)
where have used that da(a)/ a{a) o(a)). The optimum can be computed from 5J/ōs, b=)=- O, which results in
Ypos P (po (1, (1-oaa))
Yneg ThegPnegh Pneg (z (a) 0
(8)
Ypon P Pppal (2)
(9)
Tpos P paa (1r, x²) + Yneg Prog (x,
+ e- ((r,tt) -log (oog(("oou// (moy)
Pros (x,
s" (2, log
Viipos Yneg
(10)
Pneg (I,
Note that J[s; 7] is not bounded from below, so the optimum maximum.

---

Under review as a conference paper at ICLR 2021
B APPENDIX
We show that optimising the objective ][s; Ypos; "neg] by gradient ascent, with ?nos: randomly sampled in each optimisation step, leads averaging over ensemble of gradients, A gradient based optimisation method in its simplest form updates the parameters 0, which determine the function s(r, by the rule
- 0 +=0+aVoJ(sn)
(11)
with a the learning rate and
VoJ[s:n]
=
Ypoo (araa))
(1 -1- o(a)) V (a)) Vos(1,
"nng (n [o(a)
= )a
+
"poo (=(2z²))
Ypos
(os(E,22)
Pney
- (m,a') ~PPna
Inss s(a,aa)) V Vos(z, 2')
Yneg
(12)
1
Ypus
This result shows that for given s(x, x²), random values of Yposs Tneg > 0 weight the expected gradients for the positive examples and for the negative examples differently. As consequence, VoJ[s;;] takes different directions for each parameter update, given fixed matnc and fixed initial conditions.
we consider the following limiting cases for 8*
lim [oJ[s:]]
[(ra,"))PPp. [Vos(x, - (e (ostaa")oss Vos(x,,2))
(13)
Ypus -1 Tneg 700
P pos (20, V Vos(z,
((xp)))~PPp [VS((,,))]] ((t,ee))PP
Pang (I, x²)
(14)
and
lim [aJ[s;]]
(es(ana") Vas(an() B(e,rr) (Vo((x, (15)
Yoos 7000
They
(2ng( Vos(z,
E(
E(z,a)) (es((I
P pos (I,
(16)
see that the objective learns the optimal importance weights of importance sampling. As in this work we have the situation that P (2) x') = O for some negative examples, the case Tpos + 00, Ynng + 1 should be applied, which is why we set Ypos and sample uniformly from Tneg E N1, with N > 1.
12

---

Under review as a conference paper at ICLR 2021
C APPENDIX
2009
209
-20
-000
1000
280
750
E.00
090
000
10
200
000 scme
000
Figure 5: Random shifts of in nn-distribution test/train sets (green) for different training runs as indi- cator for instability the decision boundary. Shown are results for independent training runs for (111, 10) (left column) and for = 1 (right column), using the same setup as used to compute Table 1 but with ResNet18.
13

---

Under review as a conference paper at ICLR 2021
D APPENDIX
D.1 TRAINING CONTRASTIVE ENCODER
In order train the contrastive encoder, we use a modified version of resnet18 with 128-dimensional projection head to make it suitable for cifar10 and cifar100 datasets with relatively small image size. In particular we remove the Maxpooling layer and subsitute the first 7 x 7 convolutional layer of stride 2 with a convolutional layer of kernel size 3 3 with padding and stride of 1. For the optimisation task, we use the Adam optimiser with learning rate 10-4 and weight decay of 10-6. The network is trained for 1500 epochs batch size 2048.
D.2 TRAINING DISCRIMINATOR
To train the discriminator, we use ResNet18/34 and apply the same modifications as for the con- trastive encoder. In addition, all the ReLU activation functions are replaced with ELU and the projecting head maps to scalar value. Note that encoder and discriminator don't share parameters. The discriminator is trained with initial learning rate of 5 10 5 using AMSGrad optimiser with weight decay of 10-6 on batch size of 128 samples in each iteration. The learning rate is multiplied by 0.2 and 0. after 200 and 500 epochs, respectively. To generate the positive pairs for training, we first find the 4 examples with the highest cosine similarity score among 10k random examples for each example in training set, from which one is randomly selected with equal chance. During the training procedure μ 1/32 3. 3.125% of each batch includes semantically similar pairs. For the y, in each iteration value is uniformly chosen from the range (11, 10). The hyperparameters and their default values shown in Table
Table 3: Hyperparameters for discriminator Hyperparameter
Default Value
Batch size
128
Initial learning rate
5 10-5
Portion of semantic samples
3.125%
Size of neighbourhood Number of samples for best neighbour selection
4
10k
?
~ ((1
Weight decay
10-6
D.3 GEOMETRIC TRANSFORMATTONS
As data augmentation to train the contrastive encoder, we use the same transformations as in Chen let al. (2020). including, randomly chosen geometric transformation from the set (Cropping. Hori- zontal Flip, Color Jitter, GrayScale, Gaussian Blurring]. Pytorch snippets for encoder transforma- tions be found in Table 4 To train the discriminator make two sets of transformations, one for positive samples and one for negative ones. The main intuition to shape a set of transforma- tion for positive samples is keep them in-distribution according the original training samples. For the special case of cropping we make three different categories as weak, strong and extreme cases. Table 5S shows their cropping scale according to pytorch standard. To make positive pairs we randomly apply both weak and strong ranges for cropping, random horizontal flipping and color jittering on the same image from the training set and for negative pairs we apply all ranges of weak, strong and extreme cropping, horizontal flipping, color jittering, and Gaussian blurring on two ran- domly selected images. The details and pytorch snippet for positive and negative transformations can be found in Table 7 and O respectively. Note that for augmentation with semantically similar pairs from the training set there is no transformation applied on positive pairs.
14

---

Under review as a conference paper at ICLR 2021
Table 4: Transformations for contrastive encoder
Transformation
Pytorch snippet
Cropping Horizontal Flip Color Jitter
transforms.i transforms.RRnndooReesssn RandommResizedCroppp22 scale=(0.00s)) 1))
transforms. transforms.Randomnii Randoom/HoorzzontllPPiip = 0.5)
transforms.. Canirommssnanddmpppyyyyyll C, 0.4, 0.4, transforms. SiftoomssRRndooGGayysclllllll = 0.2)
)} p=0.8)
GrayScale
Table 5: Weak, strong and extreme cropping
Strenght
Pytorch snippet
Weak Strong
transforms.
hat.ooonssRaaaodmmeeiieeeCorroo scale= 1))
transforms. transforms.Randomneee hww.annssssnoommeeeeeeeeeereooo scale=(0.3,, 0.7))
Extreme
transforms.
(cale=(0.08)) 0.3))
Table 6: Transformations for negative samples
Transformation
Pytorch snippet
Cropping Horizontal Flip Color Jitter
transforms.
transformssRaaddoRReszeeede RandommResizedCrppp22 scale= 1))
transforms.. transforms. Random/HorizootallPippp = 0.5)
transforms. Raar transforms.Colod ColorJJiter(O..4. 0.4, 0.4, 0.1 )), p=0.8) GaussianBlur(o 2(0.1,2.0), 0.2) link to github
Gaussian Blurring
Table 7: Transformations for positive samples
Transformation
Pytorch snippet
Cropping Horizontal Flip Color Jitter
transfoms..RnddomRReino RandommReszzedCropp3322 scale=(0.4, 1))
transforms.
transforms. transforns.RanddmmHoriine RandommHoorzzomtllPPppyy = 0.5)
transforms. Random CwassfornssColooritten.... 0.4, 0.1, )}, p=0.8
15


Reviews:
[{"review_id": "xCuu7VxXG0I", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe authors present a new Algorithm for performing unsupervised anomaly detection in diverse applications such as visual, audio and text data. They propose a two-step method in which first they utilise contrastive learning in order to find a semantically dense map of the data onto the unit-hypersphere. Then, they classify neighbouring pairs of test examples as in- or out-of- distribution based on the amount of the shared semantic information. Finally, they show that in several anomaly detection problems in the field of visual data their proposed method outperforms several existing methods.\n\n##########################################################################\n\nReasons for score: \n\nI recommend to accept the paper since the authors deal with an important problem and they propose a clear and well-written method that outperforms in their empirical applications, at least, several existing approaches. Please find below cons that I suggest the authors to address in the rebuttal period.\n\n##########################################################################\n\nCons: \n\n1) Although the authors refer to several existing anomaly detection methods I would suggest to add a separate and relatively small literature review section in the paper. In that section the authors should list the most relevant, existing, anomaly detection methods and briefly explain them. This will improve the readability of the paper.\n\n2) The authors identify that the main limitation of the proposed approach is the definition of a semantic similarity which in some applications can be very difficult. Therefore, I suggest the authors to perform a sensitivity analysis of their results with respect to the transformations that they use. I propose to add one or two tables similar to Table 1 in which they will compare versions of their method resulting from using different/misspecified transformations with the competing methods. They could for example add some 'noise' in the transformation that they use and re-perform the comparisons.\n\n3) The authors should make, within their main text, reference to the Figures and the Algorithms that they present. By giving briefly the utility of each of their Figures and Algorithms they will improve substantially the readability of the paper.\n\n##########################################################################\n\nMinor comments: \n\n1) Define d in 'd-dimensional' in page 2. \n\n2) Conduct an extensive search for typos, correct for example the punctuation in 'everything that is not noise' at the bottom of page 7.\n\n\n\n \n\n\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "krfrHCB7zCB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an OOD detector algorithm that first learns a function to reduce the data dimensionality followed by learning a classifier discrimination model to separate in-distribution data from OOD.\n\nPro:\n1. The paper compares many baseline algorithms\n2. The paper tries to address an important problem (OOD detector)\n\nCon:\n1. The paper title is 'A General Framework...', however, the few datasets selected for experiments represent a very narrow domain. The paper title should be narrowed down or more domains should be included in experiments.\n2. There are gaps in the intuitions such as why would two instances in the same neighborhood in the reduced dimension not be expected to have similar labels.\n\nMain Comments:\n\n1. The overall approach is that of reducing the dimensionality of the data by projecting it onto a lower dimensional manifold (surface of hyper sphere) and then using a discriminator. This approach is not novel in general.\n\n2. While the paper claims that this is a general technique, it depends on the concept of 'semantic neighborhood' for which it only provides CIFAR variants as evidence. We do not know (contrary to claims) whether it might work on other types of data (audio, text, etc.)\n\n3. Section 4: \"Our interpretation ... includes all semantic information ... helps OOD detection. In contrast, learning from label information ... mainly the semantics that help predicting labels.\" -- The paper does admit that the 'semantic neighborhood' is ill defined (Section 6, Conclusion). Yet the paper assumes, in Section 4, that the proposed technique (using pairwise distance metric) learns it well for the image data it was tested on. It is hard to see how this interpretation is justified. My assumption is that the algorithm has only learned what is necessary for the task of OOD just as a classification algorithm will learn what is necessary for labeling. There are many critical decisions that have gone in to design the proposed OOD detector (such as the distance metric to use, which features to use for the discriminator, etc.). It is more conceivable that in the end the algorithm has learned just enough representation that makes the combined design choices work well on the specific dataset. It is hard to generalize given that the experiments cover so few datasets. I suggest the paper remove 'semantic neighborhood' terminology.\n\n4. Section 2.2: \"...belief in the lottery hypothesis...\" -- Many of the subnetworks might be sharing weights and are therefore not independent. This point becomes more important because as discussed in Section 3.1, a small network was used which increases the likelihood of weight-sharing. So, the true ensemble effect might be absent in reality.\n\n5. Section 2.2: \"The idea is now to make use of the fact that nearby examples on unit-hypersphere share semantic information if both come from the in-distribution but don’t share semantic information if one of the two examples is OOD.\" -- It is not clear to me why any two close examples would not share semantic similarities assuming that the mapping function is smooth. In case the contrastive objective results in such as case, then we might have very noisy labeled data.\n\n6. Section 3.1: \"We train at batch sizes of either 1024 or 2048 using ADAM optimizer.\" -- These batch sizes are quite large than conventional (e.g. 32, 64). Is there a reason for that?", "extra_scores": {}, "extra_text": {}}, {"review_id": "H3tb-TnfPcA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n- Presents GenAD as a general framework for anomaly detection\n- Method builds on top of contrastive training and proposes to learn a discriminator to distinguish between semantically similar and dissimilar pair of examples\n- Results are SOTA but need verification through code and methods clarification\n\nClarity/Quality:\n\nPaper is overall written OK but several typos/grammatical errors as highlighted below:\n\n- “For visual data we show new state-of-the OOD classification accuracies for standard benchmark data sets” -> new state of the \n  “art” OOD classification\n- “The contrastive objective aligns feature vectors h = h(x)” -> consider using different symbols for the vector output and the \n   encoder function\n- “A statically meaningful score” -> statistically?\n- The notation “Pneg(x, x’) = Ppos(x)Ppos(x’)” is unclear. Is marginalization implied? (end of page 3)\n- “mainly affects the weights of a small subnetwork Frankle & Carbin (2019)” - missing parentheses around reference\n- “If we belief in the lottery hypothesis” -> belief to believe\n- “We expect to see a significant increase in OOD detection performance upon increasing network size, which left to future work.” \n -> which is left to future work\n\nNovelty:\n\nCentral claim - Contrastive training maps example to unit hypersphere but it is possible OOD examples can be in same neighborhood. Hence need a semantic discriminator and introduces it along with algorithms for sampling positives/negatives. \n\nSignificance:\n\nThe central idea is simple and well motivated. If (and it is a big if) the results are verified, this could be a very important paper in the field of OOD detection. \n\nQuestions/Comments/Clarification\n\n- Assumption is all OOD is semantic which may not always hold true especially if there are stylistic varaiations introduced using different imaging equipment\n- Unclear why gamma (Yneg and Ypos) was introduced\n- Unclear how encoder and discriminator are trained? Is it jointly or separately? Are these the same networks? Architecture diagram for network setup is needed to clarify details\n- Why does the discriminator enable learning of semantic dissimilarity?\n- While algorithm for sampling of positives is specified how are negatives sampled?\n- In Table 2, ablation corresponding to T(x) should be similar to results from (Winkens et al. 2020) right? However the \n  corresponding values are much higher (78.3 vs 89.3). The only difference seems network sizes. Not sure how these results came \n  about?\n- In Appendix C2 - “To train the discriminator s(x, x0 ), we use almost the same network structure as our contrastive encoder but \n  with smaller width and the MLP layer projects to a scalar output.” -> so do you not train the discriminator on top of contrastive \n  representations? If yes, then how is the network pruned to smaller width?\n- Why was ADAM used instead of LARS as in Chen et al?\n- Claim of general framework for OOD detection is strong as no results shown on non visual domains.\n\nOverall, this is an interesting idea but the method needs a lot more clarification and results need verification. Would encourage authors to share code to help verify the methods/results.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ITNaKYgFAz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n**UPDATE**\n\nI acknowledge that I have read the author responses as well as the other reviews. I appreciate the clarifications and improvements made to the paper and have increased my score 5.\n\nMy concerns about the generality of the framework (as also pointed out by Rev1) still hold, however, as an evaluation on non-image data is still missing. I encourage the authors to extend their work further into this direction, but as is, I would keep my recommendation to reject.\n\n#####\n\n**Summary**\n\nThis work presents a generic approach for out-of-distribution (OOD) detection or anomaly detection (AD) called GenAD. GenAD consists of two steps: First, (i) learning a spherical representation via contrastive learning to capture semantic similarities, followed by (ii) training a classifier to discern between semantically similar and dissimilar pairs of samples, given the representation from (i). An experimental evaluation on in-distribution vs. out-of-distribution dataset pairs (CIFAR-10 vs. SVHN, CIFAR-10 vs. CIFAR-100, CIFAR-100 vs. CIFAR-10) is presented which shows that GenAD outperforms previous OOD methods on these settings.\n\n\n**Pros**\n+ OOD detection is an important open problem that is relevant and of interest to the community.\n+ GenAD seems to improve over previous methods in the visual domain.\n+ GenAD, in principle, is applicable to general types of data (e.g., images, audio, text, etc.).\n\n\n**Cons**\n- There are some critical details missing about the specific choices made for sampling negative pairs, which makes it hard to assess the technical correctness and merit of the presented approach. In general, I find it hard to follow and exactly understand all the relevant details from reading the description of the method in Section 2.\n- Though the applicability of the approach to general types of data is emphasized, the experimental evaluation only includes image data.\n- Some recent related work from the out-of-distribution [6, 9, 10, 5] and deep anomaly detection [7, 3, 1, 2, 4, 8] lines of research are missing which also study representations that are effective for detecting semantic out-of-distribution samples and propose various solutions.\n\n\n**Recommendation**\n\nAs is, I recommend to reject this paper primarily due to a lack of clarity and missing details in the description of the approach, which makes it hard to assess the technical correctness and merit of GenAD.\n\nIn particular, how are $P_{pos}$ (via transformation or neighborhood or both?) and $P_{neg}$ exactly modeled in the experiments?\nIn Section 2.2, $P_{neg}$ is defined as the product of positive marginals, but how is this implemented?\nHow are the negative minibatch $\\{x_k^r\\}_{k=1}^N$ and the negative set of transformations in $T^{negative}$ in Algorithm 1 defined and chosen?\n\nThese details should be clarified and explained.\n\n\n**Additional feedback and ideas for improvement**\n- Include the missing details and try to explain the approach more clearly (there is one page of space currently left).\n- Include other types of data in the experimental evaluation, which would strengthen the generality claim of the proposed approach.\n\n\n**Minor Comments**\n\n1. The title of the paper is very generic.\n2. The figures in the paper are disproportionately large and waste quite some whitespace.\n3. The batch sizes reported in the experiments are uncommonly large (1024, 2048). What is the reason for this choice?\n4. I think Algorithm 2 can be removed, as it just describes $k$-NN using cosine similarity, right?\n5. Section 1: ‘Note that it is possible for a datapoint to have high likelihood under a distribution yet be nearly impossible to be sampled, a property known as asymptotic equipartition property in information theory.’ Citation?\n6. Section 1: ‘Intuitively, the OOD detection problem should be independent of the hardness of an in- distribution classification task.’ Why? I could imagine the hardness of an in-distribution classification task can be due to a complex in-distribution, for which the OOD detection problem is also more difficult.\n7. Make use of page 8 in the main paper, e.g. move interesting claims and derivations to the main paper.\n8. Section 3.1: ‘[...] - for both the encoder $f(x)$ and the *classifier* $s(x,x′)$.’ I would avoid to use the discriminator term.\n9. Table 1: Add space between method names and citations.\n10. Section 4: ‘[...], with increase in state-of-the-art AUROC from 0.783 to > 0.999.’ What about the 0.856 of OpenHybrid in Table 1?\n11. Section 4: ‘Note that $h(x)$ *encodes features* of semantic similarity but not necessarily *features that allow* to score semantic dissimilarity.’\n12. Section 4: ‘In fact, we observe for CIFAR-100 that examples from the same semantic neighbourhood do not always share the same label.’ Could you include some example images?\n\n\n#####\n\n**References**\n\n[1] F. Ahmed and A. Courville. Detecting semantic anomalies. In AAAI, pages 3154–3162, 2020.\n\n[2] L. Bergman and Y. Hoshen. Classification-based anomaly detection for general data. In ICLR, 2020.\n\n[3] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In NeurIPS, pages 9758–9769, 2018.\n\n[4] S. Goyal, A. Raghunathan, M. Jain, H. V. Simhadri, and P. Jain. DROCC: Deep robust one-class classification. In ICML, pages 11335–11345, 2020.\n\n[5] P. Kirichenko, P. Izmailov, and A. G. Wilson. Why normalizing flows fail to detect out-of-distribution data. arXiv preprint arXiv:2006.08545, 2020.\n\n[6] A. Meinke and M. Hein. Towards neural networks that provably know when they don’t know. In ICLR, 2020.\n\n[7] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft. Deep one-class classification. In ICML, pages 4393–4402, 2018.\n\n[8] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A unifying review of deep and shallow anomaly detection. arXiv preprint arXiv:2009.11732, 2020.\n\n[9] R. T. Schirrmeister, Y. Zhou, T. Ball, and D. Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. arXiv preprint arXiv:2006.10848, 2020.\n\n[10] Z. Wang, B. Dai, D. Wipf, and J. Zhu. Further analysis of outlier detection with deep generative models. arXiv preprint arXiv:2010.13064, 2020.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a two stage approach for anomaly detection - first train a low dimensional embedding potentially using self-supervised learning methods, and then train a discriminator on top of the embedding that takes in pairs of examples and outputs a score which can be used for anomaly detection. A test example is paired with the next nearest neighbor. A common concern of the reviewers was on the claim of the paper to be a general approach for anomaly detection whereas experiments are reported only on vision datatsets. The authors have addressed this by making changes to the title and to the claims made in the paper. However R1 and R2 still have concerns about insufficient empirical evaluations, in particular lack of non-vision datasets. 

As the paper aims to tackle the problem where OOD examples are spread through the sphere, appearing mixed with normal examples, I think fitting a nonparametric density model (eg, using KDE) or parametric density model (eg, a mixture model) on the embeddings is a natural baseline to compare with. 

I encourage the authors to strengthen the empirical section of the paper based on reviewers' comments and resubmit to a future venue. 
Decision: Reject