UNSUPERVISED ANOMALY DETECTION FROM SEMANTIC SIMILARITY SCORES

1 INTRODUCTION . Anomaly detection or novelty detection aims at identifying patterns in data that are significantly different to what is expected . This problem is inherently a binary classification problem that classifies examples either as in-distribution or out-of-distribution , given a sufficiently large sample from the in-distribution ( training set ) . A natural approach to OOD detection is to learn a density model from the training data and compute the likelihood ratio of OOD examples . However , in practice this approach frequently fails for high-dimensional data ( Nalisnick et al . ( 2019 ) ) , where it has been shown that deep generative models can assign higher likelihood to OOD examples than to in-distribution examples . This surprising result is likely the consequence of how existing deep generative models generalise . For example , Variational Autoencoders ( Kingma & Welling ( 2014 ) ) generalise by superposition of examples , which is a consequence of the stochastic nature of the posterior that can map different examples to the same point in latent space . As superposition is an averaging process that reduces the information content it can be expected that examples of lower complexity than the training examples can map to high likelihood regions in latent space . Note that it is possible for a datapoint to have high likelihood under a distribution yet be nearly impossible to be sampled , a property known as asymptotic equipartition property in information theory Cover & Thomas ( 2001 ) . For autoregressive generative models , such as PixelCNN ( van den Oord et al . ( 2016 ) ) , it has been shown that the pixel-by-pixel generation process is strongly determined by the local surrounding of pixels ( Chen et al . ( 2018 ) ) , where the fact that nearby pixels of training examples frequently share the same color can explain why mono-chromatic images are assigned a high likelihood ( Nalisnick et al . ( 2019 ) ) . Local pixel correlations also seem to be responsible for the failure of generative models based on Normalising Flows to assign correct likelihood values to OOD examples Schirrmeister et al . ( 2020 ) . As a consequence , most of the current OOD detection approaches make use of a score function s ( x ) to classify test examples as in-distribution or OOD . In case that the examples of the training set are labelled , a simple score can be given by s ( x ) = maxy p ( y|x ) , with p ( y|x ) the softmax probability for predicting class labels , y ∈ { 1 , .. , K } ( Hendrycks & Gimpel ( 2017 ) ) . If s ( x ) is below a threshold the test example is classified as OOD . Labelled data allows to learn representations that are associated with the semantic information shared by the examples in the training set , which can be used for OOD detection . However , the approach suffers from the problem that the scores for in-distribution examples can be widely distributed across the interval of possible score values , s ( x ) ∈ [ 1/K , 1 ] , especially if the number of labels are low and the classification task is hard , which strongly increases the false-positive rate . Consequently , better performance was found for approaches that use labeled data for learning a higher dimensional representation that encodes for semantic information ( Lee et al . ( 2018b ) ) . In this representation space the in-distribution occupies just a small volume and a random feature vector would be most likely classified as OOD . Another simplification arises if the OOD detection problem is supervised , with some OOD examples labelled as such and contribute to the training set . In this case the OOD detection problem boils down to an unbalanced classification problem ( Chalapathy & Chawla ( 2019 ) ) . In general OOD detection benefits from separating the factors of variation for the in-distribution in either relevant ( e.g . object identity ) or irrelevant ( e.g . compression artefacts ) using prior knowledge , where the relevant factors are typically those that carry salient semantic information . In line with the arguments put forward by Ahmed & Courville ( 2020 ) , this separation helps an OOD model to systematically generalise , e.g . whether we are allowed to re-colour or add noise to images for data augmentation . Generalisation over the training set is necessary , as learning under insufficient inductive bias would result in misclassification of examples from an in-distribution test set as OOD . Labeled data provide this additional information , as relevant factors can be defined as those that help the classification task , with the limitation that there might be more factors involved in characterising the in-distribution than those needed to predict the labels . In this work , we introduce a general framework for OOD detection problems that does not require label information . Our framework can be widely applied to OOD detection tasks , including visual , audio , and textual data with the only limitation that transformations must be a priori known that conserve the semantics of training examples , such as geometric transformations for images , proximity of time intervals for audio recordings ( van den Oord et al . ( 2018 ) ) , or randomly masking a small fraction of words in a sentence or paragraph ( Devlin et al . ( 2019 ) ) . For visual data we show new state-of-the-art OOD classification accuracies for standard benchmark data sets , surpassing even the accuracies that include labels as additional information . The key contributions of this work are • We propose a new OOD detection framework that is applicable in absence of labeled indistribution data or OOD examples that are labeled as such . • We show that our approach strongly improves OOD detection for challenging tasks in the visual domain • We find that identifying semantically close examples in the training set is central for reliable OOD detection 2 RELATED WORK . Unsupervised Methods using in-distribution labels . Many OOD detection methods make use of labels to generate scores that are either based on class prediction probabilities or on intermediate representations for an in-distribution classification task . For example , Hendrycks & Gimpel ( Hendrycks & Gimpel ( 2017 ) ) used the maximum of the softmax probabilities ( MSP ) to discriminate between OOD and in-distribution . More recent approaches Lee et al . ( 2018a ) ; Winkens et al . ( 2020 ) ; Zhang et al . ( 2020 ) use labels to learn an intermediate representation on which a density distribution ( e.g . multivariate normal distribution or deep generative network ) can be fitted , which then can be used to compute the likelihood of OOD examples . As labels implicitly provide information about the semantic relation of examples in the training set , approaches using label information typically show higher accuracy than unsupervised methods . These approaches can be improved by introducing additional parameters or training strategies . For example , MSP was improved by introducing a temperature parameter ( Liang et al . ( 2018 ) ) , alternative losses ( Lee et al . ( 2018a ) ; Vyas et al . ( 2018 ) ) , auxiliary objectives ( Devries & Taylor ( 2018 ) ; Hendrycks et al . ( 2019b ) ; Mohseni et al . ( 2020 ) ) , or outlier exposure ( Hendrycks et al . ( 2019a ) ) . Intermediate representations were improved using a multi-head network architecture ( Shalev et al . ( 2018 ) , contrastive learning Winkens et al . ( 2020 ) , metric learning Masana et al . ( 2018 ) ) . General Unsupervised Methods . If label information is absent , other means must be found to impose an inductive bias on the OOD detection model to generalise over the training set . Existing approaches can be separated in methods that learn generalisable features based on ( i ) self-supervised learning tasks Golan & El-Yaniv ( 2018 ) , transformations that destroy semantics Choi & Chung ( 2020 ) , match of encoder-decoder architectures Xiao et al . ( 2020 ) , or make use of a semantically related auxiliary outlier distribution Schirrmeister et al . ( 2020 ) . The work that is most related to ours is Geometric-Transformation Classification ( GEOM ) , proposed by Golan & El-Yaniv ( 2018 ) and improved by Bergman & Hoshen ( 2020 ) , which belongs to the class of self-supervised learning approaches ( Hendrycks et al . ( 2019b ) ) . The central idea of GEOM is to construct an auxiliary in-distribution classification task by transforming each image of the training set by one of 72 different combinations of geometric transformations with fixed strength , such as rotation , reflection , and translation . The task is to predict which of the 72 transformations has been applied , given a transformed image . GEOM gives examples that show high prediction uncertainty a high OOD score . The relevant features learned by this task are salient geometrical features , such as the typical orientation of an object . Our approach differs from GEOM by the fact that we define the relevant features as those that are invariant under geometric and other transformations , such as cropping and color jitter , which are chosen of moderate strength to not change the semantics of the images in the training set . 3 METHOD . An intuitive approach for OOD detection is to learn a representation that densely maps the indistribution to a small region within a lower dimensional space ( latent space ) , with the consequence that OOD examples will be found outside this region with high probability . The representation should include the salient semantic information of the training set , to ensure that test examples from the in-distribution are not misclassified as OOD , but disregard irrelevant factors of variation that would prevent dense mapping . As learning this mapping by Autoencoders is difficult , we split the OOD detection task into finding a semantically dense mapping of in-distribution onto a d-dimensional unit-hypersphere by contrastive learning , followed by classifying neighbouring examples on the unit-hypersphere as semantically close or distant . 3.1 LEARNING SEMANTIC SIMILARITY . A contrastive objective can be used to align feature vectors h ( x ) ∈ Rd that are semantically similar and at the same time distributes examples of the training set almost uniformly over the unit-hypersphere ( Wang & Isola ( 2020 ) ; Chen et al . ( 2020 ) ) . This representation allows to identify for any test example the semantically close example from the training set . The mapping h ( x ) = f ( x ) /||f ( x ) || can be learned from training a deep neural network f ( x ) to minimise the contrastive loss L [ h ] = −E ( x , x′ ) ∼Th ( x , x′ ) [ log eh ( x ) Th ( x′ ) /τ Exneg∼Th ( x ) [ eh ( x ) Th ( xneg ) /τ ] ] , ( 1 ) where τ denotes a temperature parameter . Here , each positive pair ( x , x′ ) is the result of sampling from a distribution of transformations Th ( x , x′ ) that conserve semantics between x and x′ , with Th ( x′ ) the marginal of Th ( x , x′ ) . For datasets used to benchmark object recognition tasks , samples ( x , x′ ) ∼ Th ( x , x′ ) can be generated by picking a single example from the training set and independently apply random transformations , such as geometric transformations , colour distortions , or cropping ( Appendix D ) . The negative pairs can be generated by applying random transformations to different training examples . We emphasise that the types of transformations and their strengths essentially define the semantics we want to encode and thus determine if , for example , the image of a black swan is classified as OOD for an in-distribution that contains only white swans . The design of transformations that capture the underlying semantics of the training dataset requires either higher level understanding of the data or extensive sampling of different combinations of transformations with evaluation on an in-distribution validation set .

Summary: The authors present a new Algorithm for performing unsupervised anomaly detection in diverse applications such as visual, audio and text data. They propose a two-step method in which first they utilise contrastive learning in order to find a semantically dense map of the data onto the unit-hypersphere. Then, they classify neighbouring pairs of test examples as in- or out-of- distribution based on the amount of the shared semantic information. Finally, they show that in several anomaly detection problems in the field of visual data their proposed method outperforms several existing methods.