No Shifted Augmentations (NSA): strong baselines for self-supervised Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) or out-of-distribution ( OOD ) detection requires using only available indistribution ( ID ) samples for training a classifier to decide upon the relative normalcy of samples at test time , without knowledge of the nature of the OOD data . OOD detection is an important problem with practical applications in , for example , industrial defect detection , fraud detection , autonomous driving , biometrics , spoofing detection and many other domains ( Tack et al. , 2020 ) . 1.1 BACKGROUND . For natural images ( which according to the manifold hypothesis lie in a compact set in a suitable space ) , OOD detection translates to finding as tight a decision boundary as possible around the normal set , while excluding unseen samples from other classes or distributions . This detection was traditionally done with either generative ( Zhai et al. , 2016 ) or discriminative models ( Schölkopf et al. , 2000 ) on top of shallow features . Deep representations subsequently provided a large boost in performance . However , the density of deep generative image models have often proved to be ineffective ( Kingma & Dhariwal , 2018 ) , with poorly calibrated likelihoods away from observed data . There have instead been two recent directions to learn suitable deep features used for OOD evaluation : a ) supervised pre-training on an external dataset . b ) Self-supervised learning ( SSL ) pre-training on either the normal set only or also on an external dataset . A variety of learned metrics , scoring functions , or one-class classifiers have then been employed on top of these learned features and this general paradigm has shown to be highly effective in many cases ( Sohn et al. , 2020 ) . The best recent results with SSL-based training for OOD has been with contrastive learning ( Tack et al. , 2020 ; Sohn et al. , 2020 ; Sehwag et al. , 2021 ; Winkens et al. , 2020 ) . Contrastive learning has been shown to distribute ID data uniformly on the hypersphere ( Wang & Isola , 2020 ) . While this helps general multi-class SSL training , it hurts OOD detection , as it makes isolating outliers from the single class harder ( Sohn et al. , 2020 ) . This uniformity also makes OOD detection much more sensitive to pollution in the inlier training data ( Han et al. , 2021 ; Sohn et al. , 2020 ) . See Appendix H for more background on related methods . If some labeled OOD data are available as negatives , semi-supervised learning maybe be used ( Ruff et al. , 2018 ; Han et al. , 2021 ) . If no such labeled negatives are available , one way to soften the effect of a contrastively learned uniform representation is to introduce artificial negative samples as proxies for these outliers . Using hard augmentations ( e.g . 90 degree rotations ) has been termed Distributional Shifting ( Tack et al. , 2020 ; Sohn et al. , 2020 ; Li et al. , 2021 ; Tian et al. , 2021 ) . Such augmentations are intended to make the in-distribution data less uniform , and thus easier to isolate from OOD data . However , they also make the significant assumption that the data is not ( fully or partially ) invariant to those augmentation ( s ) , and that the augmentations are a good proxy for the true negative distribution . The other direction is using features from a model pre-trained on a large external dataset , with the hope of producing universal features that can work in any OOD detection scenario . This can be done in either supervised ( Reiss et al. , 2020 ) or self-supervised manner ( Xiao et al. , 2021 ) . However , the assumption that such representative labeled samples will be available in the latter case , or that learned image features from general datasets such as ImageNet will transfer well may be restrictive at best . 1.2 CONTRIBUTION . In this work , we first investigate the training dynamics of contrastive SSL methods , and show that their performance decays significantly over long term training . We find that uniformity or noncompactness of the learned ID representation is the main reason for this decay . We study this effect on positive-pairs-only SSL using SimSiam ( Chen & He , 2020 ) , and show that in such cases , the decay does not happen . We propose an architectural modification that can be applied generally across such networks , and show extensive analysis that this modification improves performance and always encourages learning a more compact ID representations . In doing so we are able to learn high quality One-class classifiers without resorting to distributionally shifted augmented samples as negatives , hence we term the resulting methodology No Shifted Augmentations ( NSA ) . We summarize our contributions as follows : • We investigate and empirically verify and quantify that the the non-uniformity and compactness of learned ID is a main factor of the final OOD detection performance , independent from the quality of the learned features . • We propose an assumption-free , simple and novel architectural modification for inducing a non-uniform learned ID representation , and show that this works very well with both SimSiam and SimCLR and produces solid performance improvements . • We identify , investigate , and solve a gradient problem in SimSiam ( and also BYOL ) that greatly affects the proper propagation of the norms inside the network ; we solve it and notice much higher stability , especially in the low batch size training regime . • We consider improved feature scoring methods for OOD detection , including in our proposed solution a Mahalanobis Cosine score on nearest neighbors , related to methods in open-set metric learning . We then present a computationally efficient method of feature ensembling that also boosts performance . • We show unexpected case ( s ) ( e.g . SVHN ) where the usual ImageNet-Pretrained ResNet methods fail catastrophically on One-Class Classification Anomaly Detection tasks , even with feature adaptation . We show that training from scratch without using shifted augmentations avoids this . • We extensively evaluate and ablate the proposed models with a wide variety of different datasets and scenarios , separating the contributions of representation learning , scoring , data augmentation , and additional variations like ensembling . We show our solutions have comparable performance against more complex methods . More importantly they show state-of-the-art performance , by a wide margin , achieving robustness for small batch sizes and in the presence of polluted data . 2 METHODOLOGY . 2.1 SELF-SUPERVISED LEARNING ( SSL ) SETUP . A general SSL setup resembling SimSiam ( Chen & He , 2020 ) is depicted in Figure 1.2 . SimCLR is a very similar architecture that omits a prediction network , so we use SimSiam for this illustration . The figure shows both the forward pass and backward pass of two random augmentations x1 and x2 of an input image x . Both are encoded by the shared encoder ( the convolutional backbone network ) f into y1 and y2 . Both augmentations are projected into z1 and z2 by a projection network g. A prediction network h transforms zi into pi = h ( zi ) and the whole network learns to match the output of the prediction network fed with the projection of one view pi = h ( zi ) to the projection of the other view zj = g ( yj ) and vice versa , by minimizing their negative cosine similarity : D ( pi , zj ) = − pi ‖pi‖2 · zj ‖zj‖2 . ( 1 ) Most current SSL algorithms use a projection head g on top of a convolutional feature extractor , as this was empirically found to help learn a much better final representation ( Chen et al. , 2020a ; b ; Grill et al. , 2020b ; Chen & He , 2020 ) . However , it was also found that the learned projection is much worse in downstream tasks ( Chen et al. , 2020a ) including for OOD ( Sohn et al. , 2020 ) . Therefore , the output of the encoder backbone f used during feature evaluation in most OOD methods is based on SSL , as it learns a much better representation . We call the outputs of f the learned embedding . 2.2 IS REPRESENTATION QUALITY THE ONLY IMPORTANT THING FOR OOD ? . To illustrate that the quality of learned representation is only one factor for OOD performance , we train SimCLR on one class of CIFAR10 and evaluate the learned representation for OOD detection against all other classes ; a similar setup is used in ( Sehwag et al. , 2021 ; Tack et al. , 2020 ) , however , we train for much longer . We repeat this experiment for 4 different classes , and evaluate the learned representations for OOD detection for each throughout training . Figure 2a ( blue curve ) shows the mean performance evaluation of the 4 classes ; it is evident that performance reaches a peak very fast ( within few hundred epochs ) then deteriorates significantly as training progresses . Appendix A shows the same phenomena using 2 other evaluation metrics , with detailed graphs for each class . One easy way to explain this behavior is the deteriorating quality of the learned representation . We carry out two more evaluations to examine this hypothesis . In Figure 2c we use linear evaluation ( Zhang et al. , 2016 ) and weighed k-NN ( Wu et al. , 2018 ) to evaluate the quality of learned representation . Figure 3d shows that the quality of the representation barely changes during training , i.e . there is only the expected small decrease after the peak and then just fluctuations , and that linear separability holds . Weighted k-NN results supporting the same fact are in Appendix A . These results tell two stories . ( a ) Methods based on density estimation show that performance drops down significantly as training progresses . ( b ) Doing supervised classification on the frozen features shows they are still of high quality and maintain nearly the same linear separable performance of the learned representation . This points to a change of the underlying distribution of the embedding space . To study what is happening , we use the von Mises-Fisher ( vMF ) distribution which is a fundamental probability distribution on the ( n− 1 ) -dimensional hyper-sphere Sd−1 ⊂ Rd . Its probability density function is fn ( z , µ , κ ) = Cn ( κ ) eκµ T z , where µ is the mean direction , and κ is the concentration parameter . The vMF shape depends on κ : for high values , the distribution has a mode at the mean direction µ ; for κ = 0 it is uniform on the hyper-sphere Sd−1 . vMF has been successfully used in both analyzing ( Wang & Isola , 2020 ; Lee et al. , 2018a ; 2021 ) and learning ( Hasnat et al. , 2017 ; Davidson et al. , 2018 ; Kumar & Tsvetkov , 2018 ) deep neural networks . We fit a vMF ( Banerjee et al. , 2005 ; Sra , 2012 ) to the learned normalized embeddings of SimCLR , and study how κ changes during training . Figure 2b shows how SimCLR starts with a relatively large κ ( high concentration ) then reduces monotonically ( low concentration , more uniform ) as training progresses – this is true for both ID and OOD . In Figure 2f we notice the same behaviour using another tool : the Maximum Mean Discrepancy ( MMD ) ( Gretton et al. , 2012 ) between the learned representation and samples from a uniform distribution on a unit hypersphere , as suggested in ( Sohn et al. , 2020 ) . MMD measures the distance between two probability distributions , so a high MMD means a less uniform and more concentrated distribution , and a low MMD the opposite . Observe that this gives an explanation of the decaying performance seen in Figure 2a ( blue curve ) . It is not the quality of the representation that is essential , but rather its distribution . As the ID data distribution gets more uniform , the probability p to find an inlier sample x ∈ ID arbitrarily close to an outlier query sample x′ ∈ OOD increases , i.e . ID and OOD get more and more indistinguishable . This is exactly the intuition behind many methods for AD , for example ( Ruff et al. , 2018 ; Goyal et al. , 2020 ; Ruff et al. , 2021 ) Given this analysis , and the fact that contrastive methods are proven to converge to a uniform distribution ( Wang & Isola , 2020 ) , we take the natural step and perform the same analysis on a non-contrastive SSL method , SimSiam ( selected over BYOL as it is simpler and retains most of the performance ) . Results are shown in Figure 2c , 2d , where indeed there is big difference from SimCLR : OOD performance stays nearly constant after reaching the peak . Also in Figure 2e , 2f we see that although the representation is also becoming more uniform , it is at a much slower rate , and is able to maintain high values for κ and MMD even after many epochs of training . It is also important to mention that this analysis shows that the number of training epochs ( or use of early stopping criteria ) is a very important hyper-parameter to choose for contrastive methods used for OOD . This needs to be carefully tuned , and we see that non-contrastive methods have a big advantage in being less sensitive to the number of epochs . Other works that noticed importance of early stopping in this context include ( Reiss et al. , 2020 ; Han et al. , 2021 ; Reiss & Hoshen , 2021 )

Summary: The paper studied the issues of recent self-supervised contrastive learning based anomaly detection and presented few modification to resolve those issues. Specifically, the paper pointed out the issue of contrastive learning for OOD detection when trained longer as their embedding distributions become more uniform, and proposed an architectural modification by adding a L2 normalization layer after encoder. While SimSiam, a non-contrastive self-supervised learning algorithm, suffers less from uniformity issue, authors found adding L2 normalization layer to SimSiam effective. Authors also pointed out some gradient flow issue of SimSiam and proposed adding another L2 normalization layer at the end of projection layer. Evaluation on anomaly detection verify the importance of the proposed L2 normalization layers, but the per