No Shifted Augmentations (NSA): strong baselines for self-supervised Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) or out-of-distribution ( OOD ) detection requires using only available indistribution ( ID ) samples for training a classifier to decide upon the relative normalcy of samples at test time , without knowledge of the nature of the OOD data . OOD detection is an important problem with practical applications in , for example , industrial defect detection , fraud detection , autonomous driving , biometrics , spoofing detection and many other domains ( Tack et al. , 2020 ) . 1.1 BACKGROUND . For natural images ( which according to the manifold hypothesis lie in a compact set in a suitable space ) , OOD detection translates to finding as tight a decision boundary as possible around the normal set , while excluding unseen samples from other classes or distributions . This detection was traditionally done with either generative ( Zhai et al. , 2016 ) or discriminative models ( Schölkopf et al. , 2000 ) on top of shallow features . Deep representations subsequently provided a large boost in performance . However , the density of deep generative image models have often proved to be ineffective ( Kingma & Dhariwal , 2018 ) , with poorly calibrated likelihoods away from observed data . There have instead been two recent directions to learn suitable deep features used for OOD evaluation : a ) supervised pre-training on an external dataset . b ) Self-supervised learning ( SSL ) pre-training on either the normal set only or also on an external dataset . A variety of learned metrics , scoring functions , or one-class classifiers have then been employed on top of these learned features and this general paradigm has shown to be highly effective in many cases ( Sohn et al. , 2020 ) . The best recent results with SSL-based training for OOD has been with contrastive learning ( Tack et al. , 2020 ; Sohn et al. , 2020 ; Sehwag et al. , 2021 ; Winkens et al. , 2020 ) . Contrastive learning has been shown to distribute ID data uniformly on the hypersphere ( Wang & Isola , 2020 ) . While this helps general multi-class SSL training , it hurts OOD detection , as it makes isolating outliers from the single class harder ( Sohn et al. , 2020 ) . This uniformity also makes OOD detection much more sensitive to pollution in the inlier training data ( Han et al. , 2021 ; Sohn et al. , 2020 ) . See Appendix H for more background on related methods . If some labeled OOD data are available as negatives , semi-supervised learning maybe be used ( Ruff et al. , 2018 ; Han et al. , 2021 ) . If no such labeled negatives are available , one way to soften the effect of a contrastively learned uniform representation is to introduce artificial negative samples as proxies for these outliers . Using hard augmentations ( e.g . 90 degree rotations ) has been termed Distributional Shifting ( Tack et al. , 2020 ; Sohn et al. , 2020 ; Li et al. , 2021 ; Tian et al. , 2021 ) . Such augmentations are intended to make the in-distribution data less uniform , and thus easier to isolate from OOD data . However , they also make the significant assumption that the data is not ( fully or partially ) invariant to those augmentation ( s ) , and that the augmentations are a good proxy for the true negative distribution . The other direction is using features from a model pre-trained on a large external dataset , with the hope of producing universal features that can work in any OOD detection scenario . This can be done in either supervised ( Reiss et al. , 2020 ) or self-supervised manner ( Xiao et al. , 2021 ) . However , the assumption that such representative labeled samples will be available in the latter case , or that learned image features from general datasets such as ImageNet will transfer well may be restrictive at best . 1.2 CONTRIBUTION . In this work , we first investigate the training dynamics of contrastive SSL methods , and show that their performance decays significantly over long term training . We find that uniformity or noncompactness of the learned ID representation is the main reason for this decay . We study this effect on positive-pairs-only SSL using SimSiam ( Chen & He , 2020 ) , and show that in such cases , the decay does not happen . We propose an architectural modification that can be applied generally across such networks , and show extensive analysis that this modification improves performance and always encourages learning a more compact ID representations . In doing so we are able to learn high quality One-class classifiers without resorting to distributionally shifted augmented samples as negatives , hence we term the resulting methodology No Shifted Augmentations ( NSA ) . We summarize our contributions as follows : • We investigate and empirically verify and quantify that the the non-uniformity and compactness of learned ID is a main factor of the final OOD detection performance , independent from the quality of the learned features . • We propose an assumption-free , simple and novel architectural modification for inducing a non-uniform learned ID representation , and show that this works very well with both SimSiam and SimCLR and produces solid performance improvements . • We identify , investigate , and solve a gradient problem in SimSiam ( and also BYOL ) that greatly affects the proper propagation of the norms inside the network ; we solve it and notice much higher stability , especially in the low batch size training regime . • We consider improved feature scoring methods for OOD detection , including in our proposed solution a Mahalanobis Cosine score on nearest neighbors , related to methods in open-set metric learning . We then present a computationally efficient method of feature ensembling that also boosts performance . • We show unexpected case ( s ) ( e.g . SVHN ) where the usual ImageNet-Pretrained ResNet methods fail catastrophically on One-Class Classification Anomaly Detection tasks , even with feature adaptation . We show that training from scratch without using shifted augmentations avoids this . • We extensively evaluate and ablate the proposed models with a wide variety of different datasets and scenarios , separating the contributions of representation learning , scoring , data augmentation , and additional variations like ensembling . We show our solutions have comparable performance against more complex methods . More importantly they show state-of-the-art performance , by a wide margin , achieving robustness for small batch sizes and in the presence of polluted data . 2 METHODOLOGY . 2.1 SELF-SUPERVISED LEARNING ( SSL ) SETUP . A general SSL setup resembling SimSiam ( Chen & He , 2020 ) is depicted in Figure 1.2 . SimCLR is a very similar architecture that omits a prediction network , so we use SimSiam for this illustration . The figure shows both the forward pass and backward pass of two random augmentations x1 and x2 of an input image x . Both are encoded by the shared encoder ( the convolutional backbone network ) f into y1 and y2 . Both augmentations are projected into z1 and z2 by a projection network g. A prediction network h transforms zi into pi = h ( zi ) and the whole network learns to match the output of the prediction network fed with the projection of one view pi = h ( zi ) to the projection of the other view zj = g ( yj ) and vice versa , by minimizing their negative cosine similarity : D ( pi , zj ) = − pi ‖pi‖2 · zj ‖zj‖2 . ( 1 ) Most current SSL algorithms use a projection head g on top of a convolutional feature extractor , as this was empirically found to help learn a much better final representation ( Chen et al. , 2020a ; b ; Grill et al. , 2020b ; Chen & He , 2020 ) . However , it was also found that the learned projection is much worse in downstream tasks ( Chen et al. , 2020a ) including for OOD ( Sohn et al. , 2020 ) . Therefore , the output of the encoder backbone f used during feature evaluation in most OOD methods is based on SSL , as it learns a much better representation . We call the outputs of f the learned embedding . 2.2 IS REPRESENTATION QUALITY THE ONLY IMPORTANT THING FOR OOD ? . To illustrate that the quality of learned representation is only one factor for OOD performance , we train SimCLR on one class of CIFAR10 and evaluate the learned representation for OOD detection against all other classes ; a similar setup is used in ( Sehwag et al. , 2021 ; Tack et al. , 2020 ) , however , we train for much longer . We repeat this experiment for 4 different classes , and evaluate the learned representations for OOD detection for each throughout training . Figure 2a ( blue curve ) shows the mean performance evaluation of the 4 classes ; it is evident that performance reaches a peak very fast ( within few hundred epochs ) then deteriorates significantly as training progresses . Appendix A shows the same phenomena using 2 other evaluation metrics , with detailed graphs for each class . One easy way to explain this behavior is the deteriorating quality of the learned representation . We carry out two more evaluations to examine this hypothesis . In Figure 2c we use linear evaluation ( Zhang et al. , 2016 ) and weighed k-NN ( Wu et al. , 2018 ) to evaluate the quality of learned representation . Figure 3d shows that the quality of the representation barely changes during training , i.e . there is only the expected small decrease after the peak and then just fluctuations , and that linear separability holds . Weighted k-NN results supporting the same fact are in Appendix A . These results tell two stories . ( a ) Methods based on density estimation show that performance drops down significantly as training progresses . ( b ) Doing supervised classification on the frozen features shows they are still of high quality and maintain nearly the same linear separable performance of the learned representation . This points to a change of the underlying distribution of the embedding space . To study what is happening , we use the von Mises-Fisher ( vMF ) distribution which is a fundamental probability distribution on the ( n− 1 ) -dimensional hyper-sphere Sd−1 ⊂ Rd . Its probability density function is fn ( z , µ , κ ) = Cn ( κ ) eκµ T z , where µ is the mean direction , and κ is the concentration parameter . The vMF shape depends on κ : for high values , the distribution has a mode at the mean direction µ ; for κ = 0 it is uniform on the hyper-sphere Sd−1 . vMF has been successfully used in both analyzing ( Wang & Isola , 2020 ; Lee et al. , 2018a ; 2021 ) and learning ( Hasnat et al. , 2017 ; Davidson et al. , 2018 ; Kumar & Tsvetkov , 2018 ) deep neural networks . We fit a vMF ( Banerjee et al. , 2005 ; Sra , 2012 ) to the learned normalized embeddings of SimCLR , and study how κ changes during training . Figure 2b shows how SimCLR starts with a relatively large κ ( high concentration ) then reduces monotonically ( low concentration , more uniform ) as training progresses – this is true for both ID and OOD . In Figure 2f we notice the same behaviour using another tool : the Maximum Mean Discrepancy ( MMD ) ( Gretton et al. , 2012 ) between the learned representation and samples from a uniform distribution on a unit hypersphere , as suggested in ( Sohn et al. , 2020 ) . MMD measures the distance between two probability distributions , so a high MMD means a less uniform and more concentrated distribution , and a low MMD the opposite . Observe that this gives an explanation of the decaying performance seen in Figure 2a ( blue curve ) . It is not the quality of the representation that is essential , but rather its distribution . As the ID data distribution gets more uniform , the probability p to find an inlier sample x ∈ ID arbitrarily close to an outlier query sample x′ ∈ OOD increases , i.e . ID and OOD get more and more indistinguishable . This is exactly the intuition behind many methods for AD , for example ( Ruff et al. , 2018 ; Goyal et al. , 2020 ; Ruff et al. , 2021 ) Given this analysis , and the fact that contrastive methods are proven to converge to a uniform distribution ( Wang & Isola , 2020 ) , we take the natural step and perform the same analysis on a non-contrastive SSL method , SimSiam ( selected over BYOL as it is simpler and retains most of the performance ) . Results are shown in Figure 2c , 2d , where indeed there is big difference from SimCLR : OOD performance stays nearly constant after reaching the peak . Also in Figure 2e , 2f we see that although the representation is also becoming more uniform , it is at a much slower rate , and is able to maintain high values for κ and MMD even after many epochs of training . It is also important to mention that this analysis shows that the number of training epochs ( or use of early stopping criteria ) is a very important hyper-parameter to choose for contrastive methods used for OOD . This needs to be carefully tuned , and we see that non-contrastive methods have a big advantage in being less sensitive to the number of epochs . Other works that noticed importance of early stopping in this context include ( Reiss et al. , 2020 ; Han et al. , 2021 ; Reiss & Hoshen , 2021 )

Summary: In this paper, authors study anomaly detection under fully unsupervised setting. This work explores novel architectural modifications on SimSiam (Chen & He, 2020) to the self-supervised feature learning step, that facilitates compact in distribution (ID) distributions to be learned on out-of-distribution (OOD) detection. It performs robustness to polluted ID. In addition, the paper investigates how the geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier.