No Shifted Augmentations (NSA): strong baselines for self-supervised Anomaly Detection
Authors: Mohamed Yousef
Tom Bishop
Unmesh Kurup
Venue: ICLR 2022 Submitted (2022)
Keywords: self-supervised learning
anomaly detection

Abstract: Unsupervised Anomaly detection (AD) requires building a notion of normalcy, distinguishing in-distribution (ID) and out-of-distribution (OOD) data, using only available ID samples. Recently, large gains were made on this task for the domain of natural images using self-supervised contrastive feature learning as a first step followed by kNN or traditional one-class classifiers for feature scoring.
Learned representations that are non-uniformly distributed on the unit hypersphere have been shown to be beneficial for this task. We go a step further and investigate how the \emph {geometrical compactness} of the ID feature distribution makes isolating and detecting outliers easier, especially in the realistic situation when ID training data is polluted (i.e. ID data contains some OOD data that is used for learning the feature extractor parameters).

We propose novel architectural modifications to the self-supervised feature learning step, that enable such compact ID distributions to be learned. We show that the proposed modifications can be effectively applied to most existing self-supervised learning objectives with large gains in performance. Furthermore, this improved OOD performance is obtained without resorting to tricks such as using strongly augmented ID images (e.g. by 90 degree rotations) as proxies for the unseen OOD data, which imposes overly prescriptive assumptions about ID data and its invariances.

We perform extensive studies on benchmark datasets for one-class OOD detection and show state-of-the-art performance in the presence of pollution in the ID data, and comparable performance otherwise. We also propose and extensively evaluate a novel feature scoring technique based on the angular Mahalanobis distance, and propose a simple and novel technique for feature ensembling during evaluation that enables a big boost in performance at nearly zero run-time cost compared to the standard use of model ensembling or test time augmentations. Code for all models and experiments will be made open-source.

Under review as a conference paper at ICLR 2022
No SHIFTED AUGMENTATIONS (NSA): COMPACT DISTRIBUTIONS FOR ROBUST SELF-SUPERVISED ANOMALY DETECTION
Anonymous authors Paper under double-bling brind review
ABSTRACT
Unsupervised Anomaly detection (AD) requires building notion of normaley, distinguishing in- on-distribttion (ID) and out-of of-ttiiibbuttiinnn (OOD) data, using only available ID samples. Recently, large gains were made on this task for the domain of natural images using self- supervised contrastive feature learning as first step followed by kNN or traditional one-class classifiers for feature scoring. Learned representations that are non-uniformly distributed on the unit hypersphere have been shown to be beneficial for this task. We go a step further and investigate how the geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier, especially in the realistic situation when ID training data is polluted (i.e. ID data contains some OOD data that used for learning the feature extractor parameters). We propose novel architectural modifications to the self-supervised feature learning step, that enable such compact distributions for ID data to be learned. We show that the proposed modifications can be effectively applied to most existing self-supervised objectives, with large gains in performance. Furthermore, this improved OOD performance is obtained without resorting to tricks such as using strongly augmented ID images (e.g. by 90 degree rotations) as proxies for the unseen OOD data, as these impose overly prescriptive assumptions about ID data and its invariances. We perform extensive studies on benchmark datasets for one-class OOD detection and show oftt-e--tttt-ttttt performance in the presence pollution in the ID data, and comparable performance otherwise. We also propose and extensively evaluate novel feature scoring technique based on the angular Mahalanobis distance, and propose a simple and novel technique for feature ensembling during evaluation that enables big boost in performance nearly zero run-time cost compared to the standard use of model ensembling or test time augmentations. Code for all models and experiments will made open-source.
INTRODUCTION
Anomaly detection (AD) out-of-distribution (OOD) detection requires using only available in- distribution (ID) samples for training a classifier to decide upon the relative normalcy of samples at test time, without knowledge the nature of the OOD data. OOD detection is an important problem with practical applications in, for example, industrial defect detection, fraud detection, autonomous driving, biometries, spoofing detection and many other domains (Tack al. (020)).
1.1 BACKGROUND
For natural images (which according to the manifold hypothesis lie in a compact set in suitable space), OOD detection translates to finding as tight decision boundary as possible around the normal set, while excluding unseen samples from other classes or distributions. This detection was traditionally done with either generative (Zhai et al. 2016) discriminative models (Schölkopf) et al. 2000) on top of shallow features. Deep representations subsequently provided large boost in performance. However, the density of deep generative image models have often proved to be ineffective (Kingma & Dhariwal 2018), with poorly calibrated likelihoods away from observed data. There have instead been two recent directions learn suitable deep features used for OOD evaluation:

---

Under review as a conference paper at ICLR 2022
a) supervised pre-training on either the normal set only or also on an external dataset. A variety of learned metrics, scoring functions, one-class classifiers have then been employed on top of these learned features and this general paradigm has shown to be highly effective in many cases (Sohn et al. 2020).
external dataset. b) Self-supervised learning (SSL) pre-training
The best recent results with SSL-based training for OOD has been with contrastive learning (Tack Winkens et al.] (020). Contrastive learning has been shown to distribute ID data uniformly on the hypersphere (Wang & Isola, 2020). While this
et al. 2020; |Shhn et al. 20200 /Sewwag et a 2021 I
helps general multi-class SSL training, it hurts OOD detection, as it makes isolating outliers from
the single class harder (Sohn et al. 2020p. This uniformity also makes OOD detection much more sensitive to pollution in the inlier training data (Han et al. 2021 Sohn et al. 2020). See Appendix H for more background on related methods.
If some labeled OOD data are available as negatives, semi-supervised learning maybe be used (Ruff eet al.] [2018 |Han et al., (02T). no such labeled negatives are available, one way to soften the effect of contrastively learned uniform representation is to introduce artificial negative samples as proxies for these outliers. Using hard augmentations (e.g. 90 degree rotations) has been termed Distributional
Shifting (Tack et al., 2020; Sohn et al., 2020; Li et al. 2021 Tian et al. (021). Such augmentations are intended to make the in-distribution data less uniform, and thus easier to isolate from OOD data. However, they also make the significant assumption that the data is not (fully partially) invariant to those augmentation(s), and that the augmentations are good proxy for the true negative distribution.
The other direction is using features from model pre- pre-trained a large external dataset, with the hope of producing universal features that can work in any OOD detection scenario. This can be done in either supervised (Reiss al. 2020) or self-supervised manner (Xiao al., [021]. However, the assumption that such representative labeled samples will be available in the latter case, or that learned image features from general datasets such as ImageNet will transfer well may be restrictive at best.
1.2 CONTRIBUTION
In this work, we first investigate the training dynamics of contrastive SSL methods, and show that their performance decays significantly over long term training. We find that uniformity non- compactness of the learned ID representation is the main reason for this decay. We study this effect on -oliive--pairs-nlly SSL using SimSiam (Chen & He, 2020), and show that in such cases, the decay does not happen.
We propose architectural modification that can be applied generally across such networks, and show extensive analysis that this modification improves performance and always encourages learning more compact ID representations. In doing so we are able to learn high quality One-class classifiers without resorting to distributionally shifted augmented samples as negatives, hence we term the resulting methodology No Shifted Augmentations (NSA). We summarize our contributions as follows:
We investigate and empirically verify and quantify that the the non-uniformity and compact- ness of learned ID is main factor of the final OOD detection performance, independent from the quality of the learned features.
We propose an assumption free, simple and novel architectural modification for inducing a non-uniform learned ID representation, and show that this works very well with both SimSiam and SimCLR and produces solid performance improvements.
We identify, investigate, and solve a gradient problem in SimSiam (and also BYOL) that greatly affects the proper propagation of the norms inside the network; we solve it and notice much higher stability, especially in the low batch size training regime.
We consider improved feature scoring methods for OOD detection, including in our proposed solution a Mahalanobis Cosine score on nearest neighbors, related to methods in open-set metric learning. We then present computationally efficient method of feature ensembling that also boosts performance.
We show unexpected case(s) (e.g. SVHN) where the usual ImageNet- Pretrained ResNet methods fail catastrophically on One-Class Classification Anomaly Detection tasks, even with feature adaptation. We show that training from scratch without using shifted augmenta- tions avoids this.
2

---

Under review as a conference paper at ICLR 2022
stop
grad
X-
21
21
V1
21
W/
(1)
9:
21
9
| ll
Pt
P1 h(21)
P1
(f R2)
(82 &1
22
M2
12
o(0²)
32
|ealla
2g
stop
grod
82
||
Pa
lg
h(2g)
P2
p
Figure 1: An illustration of modified SimSiam with the yellow boxes showing our proposed changes described in Sec. 2.3 2.4 In the original version the gradients of E; = z</1|=i[|a are prevented from flowing backwards in all possible paths by the stop-grad operation. In modified version, thanks to the added operations, we can mimic the flow of the gradients of & = 2 = that were blocked.
We extensively evaluate and ablate the proposed models with wide variety of different datasets and scenarios, separating the contributions of representation learning, scoring, data augmentation, and additional variations like ensembling, We show our solutions have comparable performance against more complex methods. More importantly they show state-of-the-ane performance, by wide margin, achieving robustmess for small batch sizes and in the presence of polluted data.
METHODOLOGY
2.1 SELF-SUPERVISED LEARNING (SSL) SETUP
A general SSL setup resembling SimSiam (Chen & He 2020 is depicted in Figure 1.2 SimCLR is very similar architecture that omits a prediction network, so we use SimSiam for this illustration. The figure shows both the forward pass and backward pass of two random augmentations æ and 22 of an input image æ. Both are encoded by the shared encoder (the convolutional backbone network) f into y1 and y2. Both augmentations are projected into 21 and 22 by projection network g. A prediction network h transforms 2; into p. = h(2)) and the whole network learns to match the output of the prediction network fed with the projection of one view p; h(2,) to the projection of the other view 2j (W;) and vice versa, by minimizing their negative cosine similarity:
D(pr. 23)
Zj
(1)
1
projection head g top of a convolutional feature extractor, as this was empirically found to help learn a much better final representation (Chen et al. 2020a Grill et al. 2020b; Chen & He, 2020). However, it was also found that the learned projection is much worse in downstream tasks (Chen et al.) 2020a) including for OOD (Sohn et al. 2020). Therefore, the output of the encoder backbone used during feature evaluation in most OOD methods is based on SSL, as it learns a much better representation. We call the outputs of f the learned embedding.
Most current SSL algorithms
2.2 IS REPRESENTATION QUALITY THE ONLY IMPORTANT THING FOR OOD
To illustrate that the quality of learned representation only one factor for OOD performance, we train SimCLR on one class of CIFAR 10 and evaluate the learned representation for OOD detection against all other classes; similar setup is used in (Sehwag al. 2021 Tack al. 2020), however, we train for much longer. We repeat this experiment for 4 different classes, and evaluate the learned representations for OOD detection for each throughout training. Figure 2a (blue curve) shows the mean performance evaluation of the 4 classes; it is evident that performance reaches peak very fast (within few hundred epochs) then deteriorates significantly as training progresses. Appendix A] the same phenomena using 2 other evaluation metrics, with detailed graphs for each class.
One easy way to explain this behavior the deteriorating quality of the learned representation. We linear evaluation
earry out two more evaluations examine this hypothesis. In Figure 2c (Zhang et al. 2016) and weighed k-NN (Wu et al.) 2018) to evaluate the quality of learned repre- sentation. Figure shows that the quality of the representation barely changes during training, i.e.
3

---

Under review as a conference paper at ICLR 2022
there is only the expected small decrease after the peak and then just fluctuations, and that linear separability holds. Weighted k-NN results supporting the same fact are in Appendix A
These results tell two stories. (a) Methods based on density estimation show that performance drops down significantly as training progresses. (b) Doing supervised classification on the frozen features shows they are still of high quality and maintain nearly the same linear separable performance of the learned representation. This points to a change of the underlying distribution of the embedding space.
To study what is happening, we use the Mises-Fisher (vMF) distribution which is a fundamental probability distribution on the (n - -iddmmnniio hyper-sphere 5 C Rd. Its probability density
function is (z, μ,, k) = C. (,))ee where is the mean direction, and is the concentration parameter. The vMF shape depends on k: for high values, the distribution has a mode at the mean direction μı; for
it is uniform
the hyper-sphere 5th vMF has been successfully used
in both analyzing (Wang& & Isola 2020; Lee et al. 201 2021) and learning (Hasnat et al. 2017
Davidson al. 2018 Kumar & Tsvetkov 2018) deep neural networks.
We fit a vMF (Banerjee et al. 2005 Sra 2012 to the learned normalized embeddings of SimCLR, and study how changes during training. Figure shows how SimCLR starts with a relatively large & (high concentration) then reduces monotonically (low concentration, more uniform) as training progresses this is true for both ID and OOD. In Figure 2f we notice the same behaviour using another tool: the Maximum Mean Discrepancy (MMD) (Gretton et al. (012)) between the learned representation and samples from uniform distribution on a unit hypersphere, as suggested in (Sohn) et al. 2020). MMD measures the distance between two probability distributions, so a high MMD) means a less uniform and more concentrated distribution, and a low MMD the opposite.
Observe that this gives an explanation of the decaying performance seen in Figure 22 (blue curve). is not the quality of the representation that is essential, but rather its distribution. As the ID data distribution gets more uniform, the probability p to find an inlier sample E ID arbitrarily close to an outlier query sample z' E OOD increases, i.e. ID and OOD get more and more indistinguishable. This is exactly the intuition behind many methods for AD, for example (Ruff et al. 2018 Goyal et al., 2020 Ruff et al. 202T)
Given this analysis, and the fact that contrastive methods proven to converge a uniform distribution (Wang & Isola, 2020), we take the natural step and perform the same analysis on a non-contrastive SSL method, SimSiam (selected over BYOL as it is simpler and retains most of the performance). Results are shown in Figure 2c 2d where indeed there is big difference from SimCLR: OOD performance stays nearly constant after reaching the peak. Also in Figure 2e 2 we see that although the representation is also becoming more uniform, it is at much slower rate, and is able to maintain high values for and MMD even after many epochs of training.
It is also important to mention that this analysis shows that the number of training epochs (or use of early stopping criteria) is a very important hyper-parane pperrpparmettr to choose for contrastive methods used for OOD. This needs to be carefully tuned, and we see that non-contrastive methods have a big advantage in being less sensitive to the number of epochs, Other works that noticed importance of
early stopping in this context include (Reiss et al. 2020 Han et al. 2021 Reiss & Hoshen 2021)
2.3 LEARNING A DENSE REPRESENTATION
(Schmidt et al. 2018) show that the sample complexity of robust learning can be significantly larger than that of standard learning. While some works tried to address this difference with extra positive or negative data, (Pang et al.) 2019) propose the interesting idea of manipulating the local sample distribution of the training data via appropriate training objectives such that by inducing high-density feature regions, there would be locally sufficient samples to train robust classifiers and return reliable predictions. We propose pursuing the same direction with the OOD detection problem, and propose an architectural modification that can help induce high density feature space.
We propose adding differentiable lz-normalization operation after the encoder f and before the projection head g. As such, the output ys = f(x) is transformed into Ji = = yy//lyy la (as in Figure 1.2). The intuition is that adding the normalization step would deprive the model of any gradients when the norm of the embedding is changed, thus enforcing the model to learn directional transformations as those would be the only way actually decrease the loss function. Regularizing the model this
4

---

Under review as a conference paper at ICLR 2022
SimCLR C-Cos
Sim kappa
Lineer probe
1600
1400
0.05
1200
0.20
75
600
0.4
600
60
2.5
(a)
(b)
(c)
SImSlam, C-Cos
SImSiam, kappa
SimSiam, MMD
norm
W narm wil norm
10070 -W1 norm
3.6
85
2500
3.5
2000
0.45
1700
1000
1000
2000
3000
4003
503C
1000
2000
3000
4030
5000
1000
2000
3002
4030
5000
(d)
(e)
(I)
Figure 2: Analysis of training and evaluation of SimCLR and SimSiam CIFARIO for OOD with and without the proposed normalization. Every point on these plots represents the average from 4 independent classes. First row, SimCLR evaluated with (a) c-Cos (b) к from fitting a vMF (c) linear probe accuracy. Second row, SimSiam evaluated with (d) c-Cos (e) x from fitting a vMF (f) Maximum Mean Discrepancy (MMD) with uniform distribution.
way should give a finer directional control on the learned embedding (the cosine distance makes more sense), and yield more efficient use of volume and thus denser representation.
We perform a series of experiments to empirically verify our intuition. Figure 2a (red curve) shows evaluation logs for SimCLR after adding normalization. It is evident the training is much more regularized now, the decrease in OOD performance after the peak is much smaller and performance maintained for thousands of epochs. Figure 2bb (red curve) shows a denser learned representation when compared to original SimCLR. We can see the same behaviour for SimSiam in Figure 2d 2e
One thing to note here that greatly strengthens the analysis made in the previous section, the linear probe accuracy with and w/o norm (Figure 2c), they are essentially the same, even after 5000 epochs, on the other hand there big deference between their OOD performance that point (see Figure (2)). Further showing that it is problem of feature distribution not feature quality.
Lastly, we would like to emphasize that while having a dense ID representation can be important for OOD detection with clean ID data, it is much more important in the presence of pollution in the ID data. In this polluted setup, some OOD data are mixed in during training and considered ID by the loss. In this case, compact ID data distribution decreases chance of other OOD data to be considered as ID as much as possible. This is empirically verified in the experimental section.
2.4 SIMSIAM AND BYOL GRADIENT FLOW PROBLEM
To avoid the degenerate solution of a collapsed representation, the authors of SimSiam (Chen & He 2020) found it crucial to have gradient blocking operation (stop-rra)) that blocks gradients starting from & from flowing back to other parts of the network. This lack of gradients acts as a regularizer and makes it hard for the optimizer to reach the trivial solution of collapsed representation.
5

---

Under review as a conference paper at ICLR 2022
Note that the same analysis presented here equally applies to BYOL (Grill et al. 2020b): the existence of the momentum encoder enforces an implicit stopp-grad.
However both (Chen & He 2020, Grill et al. 2020b) do not study the effect stop-ge ad may have on proper gradient flow in the network. Studying Figure L.2, we can see that the 12-norm of the output of the prediction network p.: p:/l|p:]e | gives proper gradients that flows back to other parts of the network. The exact opposite happens for the Iz-norm of the encoder network's 's output: all gradients of that operation gets blocked by the -topp-grad though it an integral part of the loss function.
Proposed solution Simply removing the stop-grad operation converges quickly to a collapsed representation (Chen & He 2020). Another apparent straightforward solution to minimize the norm of the output representation of the encoder f. but this limits the representation ability of the encoder and also rapidly collapses. One last trial would be removing normalization altogether in D, however this converges to a sub-optimal representation with an unbounded norm (Grill et al. 2020b).
Our proposed solution is instead based on a simple observation. The missing gradient from & carries two pieces of information: (a) moving the output of the encoder 2, to be closer to the output of the predictor ps (which is the information we want to hide from the optimizer). (b) encouraging the projector g learn 22-norm invariant representation; this facilitates training the predictor network and thus also the encoder, and is what we want to maintain.
In order maintain (b) while discarding (a), we propose small modification (see Figure 1.2p: we apply a differentiable 12- Iz-normalization to the projection 2,, such that the new projection is á 2 ///z////2 This gives the network proper gradients for learning Iz-normalized representation satisfying the loss function DD while still having the stop-gr grad operation and avoiding related collapse.
2.5 FEATURE EVALUATION FOR OUT-OF--DSTTRBUTION DETECTION
Many recent works on OOD detection are based on scoring given test sample using its distance to the nearest training sample (Tack et al. 2020 Sehwag et al. 2021 Lee et al., 2018b). This provides simple evaluation bascline with strong results. Many metrics have been used for this, the most commonly used are the Mahalanobis distance (Lee et al. 2018b Schwag et al., (021) and the Cosine distance (Tack et al. 2020) evaluated at the output of the network after the last convolutional block.
To take the advantages of both of these distances, we go a step further and score features with the arccosine of the Mahalanobis Cosine similarity (i.e. angular Mahalanobis distance) (Beveridge (et al.] (005). which have been proposed and studied well previously in the field of face recognition (Beveridge et al., 20055 Vinay et al., 2015). Mahalanobis Cosine is the Cosine similarity between vectors after projection into the Mahalanobis space, where the famous Mahalanobis distance the Euclidean distance computed between vectors after also projection into the Mahalanobis space.
If Em y are a training and test sample respectively, and the projection of their features f(m),, f(v into the Mahalanobis space is u, e using the sample covariance matrix Σm and mean pm of the training data {rm J.M then the Mahalanobis Cosine distance Dasc and our scoring Sk-cos are
u = Em 1/2 - pm), DMC(Em. y) = |ull2 ||01|2'
Sxco((y) = arccos max (M y)) m
Se-Cor considers only the distance to the nearest training sample and is used for most of our experi- ments. In Appendices C and G we study a variant Sc-oor, that especially robust to pollution; it com- putes distance to Pm: the mean vector of the training data { So-Cory)) = arccos (D Parci (Pm> u)).
Feature ensembling We also propose another evaluation scheme where all the intermediate feature maps of the network are scored independently, then their scores are all summed together. The idea is to get both high level (from final layers) and low level (from initial layers) OOD scores using different feature maps. Note that this is different from (Lee et al. 2018b) that learn a weighted sum of all the feature scores, where the weights are learned on a validation set; our proposed method is a simple sum and doesn't 't need validation set. It also attains a huge runtime saving when compared to test-time augmentation (TTA) used in (Tack et al. (200)) or model ensembling used in (Sohn et al. 2020), as it requires single model forward pass single instance. It consists of three steps:

---

Under review as a conference paper at ICLR 2022
1. Computing a score for each feature map, using either Sk-cos Se-Cos both.
2. Normalizing the range of the scores to be between 0 and I, using the range of training data scores.
3. Summing the normalized scores.
Due to space limitations we show detailed feature evaluation results in Appendix [ along with comparing different evaluation metrics. We then show extensive ablations in Appendix [ In Section 3.2 we firstly consider experimental results without ensembling, before comparing to methods that use ensembling in Section 3...44 Any result that includes ensembling strictly has the suffix, which indicates using the Ens. combination whose components are precisely described in Appendix
EMPIRICAL EVALUATION
3.1 EXPERIMENTAL SETUP
We perform a thorough evaluation of our proposed normalization modifications on SimSiam, BYOL, and SimCLR (with and without negative shifted augmentations). The problem discussed in Section 2.4 doesn't apply to SimCLR (there is no stop-grad), so while both modifications are applied to SimSiam, our normalized variant of SimCLR only includes the change proposed in Section 2.3 for ÿ.
We evaluate in the one-vs-all OOD detection setting for CIFAR-10, CIFAR- 100 super-classes (Krizhevsky al., 2009), Fashion-MNIST (Xiao al. 2017). and SVHN (Netzer al. 2011). In
this setting one class is treated as the normal class and the rest are treated as outliers.
For all results presented we use a ResNet-18, trained with Adam (Kingma & Ba 2014) with learning rate of 0.0001 and cosine learning rate decay (Loshchilov & Hutter (2016). For SimCLR we used 2-layer MLP projection head with an architecure similar (Sohn al. 2020). For SimSiam we used 3-layer MLP for both the projector and the predictor. SimCLR is trained for 500 epochs, and SimSiam and BOL for 4000 epochs. All models are trained on Nvidia V100 16GB GPUs and written in Pytorch (Paszke et al. 2019). All our reported results are without test-time augmentation.
For brevity, we often refer to SimSiam as SS and SimCLR SC. Also SS(n), SC(n) and BYOL(n) indicate inclusion of the proposed normalization, and SC(-) is with negative augmentations. Unless otherwise stated, our ensemble-free results use k-Cos Scoring. More details of the datasets, and additional comparisonssddecrriiions of competing methods can be found in Appendices B and An extensive ablation study including training 640 different models provided in Appendix
3.2
RESULTS
3.2.1
ONE-CLASS CLASSIFICATION
Table compares ansemble-free versions of our basclines BYOL, SimSiam, and SimCLR with and without normalization against current state of the art (without ensembling) DROC (Sohn et al. (222). RotNet (Golan & El- Yaniy 2018), and GOAD (Bergman & Hoshen 2020). Extended comparison of our results (without ensembling) with many other published methods can be found in Appendix [
We see that adding normalization is consistently effective across BYOL, SS, and SC in all scenarios. Also, SS(n) and B (O)((n) always achieve very competitive results compared to methods that use negative augmentations (SC(-) and DROC) while making much less assumptions about the underlying training data.
3.2.2 PERFORMANCE UNDER POLLUTION
Table also shows a comparison of the performance under the very realistic scenario that some 5% of the training data polluted with OOD data. It can be that adding the proposed normalization drastically reduces the effect of polluted data, and achieves state-of-the-ar results (without ensembling).
One important take-away from the table is that, as predicted by our analysis in Section 2.3 positive only SSL (e.g. SS or B BYOL) is much more suited to real world OOD detection which possibly
7

---

Under review as a conference paper at ICLR 2022
Data
BYOL
BYOL(n)
S8
SS(n)
SC
SC(n)
SCG-)
SC(n-)
RotNet
DROC
GOAD
90.5
91.7
86.3
88.9
90.7
92,9
92.5
88.2
C10
0.1
82.9
85.3
83.2
86.3
65.6
79.6
83.9
80,3
83.0
79.6
80.2
81.4
84.3
80.5
84.0
84.7
87.0
81.9
86.5
74.5
C100
0.1
76.2
77.8
78.9
80.3
75.9
80.0
80.5
82.8
95.3
95.1
95.9
95.0
94.6
94.9
94.7
95.7
94.6
94.5
94.1
IMNIST
0.1
61.3
73.2
63.0
75.3
46.5
53.1
78.7
80,9
76.6
Table Results of baselines and proposed variants compared to state of the art, without ensembling. is the ratio of outlier pollution data inside the training set. SS is SimSiam, SC is SimCLR.
BYOL(1)EE
SS(n)
SC(n)ε
SC(n-)
CSI
STOC
# Inference steps / example # Trained models/ class
160
I
1
60
Requires good approximation of
X
X
X
X
X
/
Assumes rotation-variant data
X
X
X
/
/
/
Data
0,0
91.9
92.5
90.3
93.0
94.3
92.1
CIFARIO
0.1
88.3
88.5
86.7
87.8
84.5
89.9
0.0
96.2
96.1
96.3
95.9
95.5
IMNIST
0.1
87.9
87.8
87.5
90.9
85.7
0.0
83.4
86.6
89.4
89.6
CIFAR100
0.1
80.7
82.5
83.0
85.7
Table 2: Performance compared to state of the art under different ensembling setups. p is the ratio of outliers inside the training data.
contain small subset of polluted data compared to contrastive SSL (without negative augmentations) which suffers big performance drops in this scenario. In the case where negative augmentations are suitable assumption, proposed SC(n-) gets better results.
3.2.3 EFFECT OF NORMALIZATION
Table examines the effect of the two proposed normalizations on SS on different batch size to asses stability training. As we can see for both batch-sizes both proposed normalizations do help performance, and their combinations (proposed) is the best. We can also see for low batch sizes SS without norm suffers a big performance loss, which the proposed normalization fixes; this a very important property large batch training is not always option.
3.2.4 EFFECT OF FEATURE ENSEMBLING
Table 2 shows results of the feature ensembling proposed in Section 2.5 techniques that also utilize ensembling: CSI (Tack et al. 2020) and STOC Yoon et al. 202 For
compared ttatc-of-the-art
CIFARIO
CIFARION
batch size
32
512
32
512
without norm
77.97
89.56
74.46
81.38
only normalize
85.76
91.63
78.7
82.27
only normalize
81.73
89.22
73.1
81.15
normalzie both
92.9
91.7
81.73
84.31
Table 3: An ablation of SimSiam showing the importance of both the normalization schemes proposed in Section 2.3 and 2.4 especially for in low batch-size training.
8

---

Under review as a conference paper at ICLR 2022
Pre-trained (IM images)
Trained from scratch
Pre-trained (IB images)
Pre-trained and adapted
Supervised
Self-supervised
SS(n)
SCin)
PTRI8
PT R50
R152
PT (SC)
PT R50
PT R50
94,9
93.8
61
66
70
Table 4: OOD detection performance of various pre-trained backbones on SVHN
fair comparison, we also state the relative computational budget (training or inference) for each. The proposed feature ensembles offer big computational savings compared to CSI and STOC at roughly the same scores. Lastly, improvements by ensembling are more significant when there are outliers in the training data.
3.2.5 PRETRAINED BACKBONES ON SVHN
Some concurrent works (Reiss& & Hoshen 2021 Fort et al. (021) claim that ImageNet pre- pre-trained models can act as universal OOD detector and can work well on nearly any in-domain distribution. (Anonymous) 2022) showed that for classes not present in the pre-training data, pre-trained models perform poorly at OOD detection.
We confirm this here, and show that even for very simple datasets (e.g. SVHN) that require different kinds of discriminative features than those required for natural images, pre-trained models perform very poorly. Table ssoos that regardless of the model size (ResNet 18 152) size of pre-training dataset (from IM images to IB images (Yalniz et al.) (219)). or using fully-supervised or SSL pre-training, or even with state-of-the-art feature adaptation after pre-training (Reiss & Hoshen 2021), all catastrophically fail at SVHN compared to training from scratch, which can get a nearly perfect score (we report other related pre-training results from our implementation in Appendix FF where as expected performance is good in situations the OOD task is similar enough to the pretraining dataset; we see this as not truly practical OOD detection benchmark however).
CONCLUSION
We have considered a general framework to detection of Anomalies in images: using various SSL methods deep feature extractors; followed by metric learning for outlier scoring. For each stage we have studied what does and doesn't It work in a variety of scenarios, and proposed remedies and improvements that are also robust in the of polluted training data and small batch sizes.
We have investigated and studied compactness of representation distributions as an important and very sensitive factor to the final OOD detection performance. Our experiments demonstrated that regardless of the quality of learned features, the ID representation compactness is critical. As its distribution gets closer uniform, the OOD detection performance deteriorates significantly.
We have motivated, proposed, and studied an assumption-free, novel architectural modification for inducing this non-uniformity, and use it to solidly improve performance across contrastive and non-contrastive SSL-based OOD detection. We also studied several variants of feature scoring that work well across these different methods. More importantly, under the real world setting of totally unsupervised AD, where the ID training data can be polluted by some OOD outliers, our proposed modifications provide ofte-of-the-ar performance among competing methods.
Previous state-of ofate-of-the-ar literature for OOD detection was based on contrastive- bbased SSL with negative "distributionally-shiin
augmentations (e.g. 90 degree rotations). big hurdle with the applicability of these methods is that the assumptions they make about the training data can be partially or fully invalid in real-world scenarios. Using our proposed "No Shifted Augmentations" (NSA) modifications, both contrastive and non-contrastive methods get boost in their baseline performance, making them comparable to negative augmentation based techniques, but much more applicable to open world scenarios where little is controlled about ID data.
While model ensembling or Test-Time Time Augmentation is known in literature to be very effective for OOD, increased training/inference computational requirements can be often prohibitive. We went further and studied using hight-weight multi-level feature ensembling for OOD. This enabled us to show state-of-the-art performance in terms of AUCROC, with huge savings in computational budget.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Anonymous. Improving and assessing anomaly detectors for large-seale settings. In Submitted to The Tenth International Conference on Learning Representations, 2022. URL https://
openreview.
wxuupllppp under review.
Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9). 2005.
Liron Bergman and Yedid Hoshen. Classification-basel anomaly detection for general data. In International Conference on Learning Representations. 2020.
J Ross Beveridge, David Bolme, Bruce A Draper, and Marcio Teixeira. The csu face identification evaluation system. Machine vision and applications, 16(2):128-13) 138, 2005.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning. 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566, 2020.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. WAIC, but Why? Generative Ensembles for Robust Anomaly Detection. October 2018.
Tim R Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomezak. Hyperspherical variational auto-encoders. arXiv preprint arXiv: 1804:0084400 2018.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image Anomaly Detection with Generative Adversarial Networks. ECMI/PKDD, 11051(3):3-7,, 201 88
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. CoRR, 2019. URL
0868996
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanaan. Exploring the limits of out-of-distribution 2021.
detection. arXiv preprint
Izhak Golan and Ran Yaniv. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, 2018.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In International Conference on Machine Learning, pp. 3711-3721. PMLR, 2020.
Will Grathwohl, Kuan-Chieh Wang. Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. CoRR, 2019. URL httttt/p////xiii wwwp:iiivirrrrrre.o 3t2.:b/91/1....2.3322 03263v2
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal Machine Learning Research, 13(1):723-773, 2012.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent: A New Approach to Self-SSupr Learning. June 2020a.
Jean- nan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006. 07733, 2020b.
10

---

Under review as a conference paper at ICLR 2022
Sungwon Han, Hyeonho Song, Seungeon Lee, Sungwon Park, and Meeyoung Cha. Elsa: Energy- based learning for semi-supervised anomaly detection. arXiv preprint arXiv:2103.15290 2021.
Md Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, Liming Chen, et al. von mises- fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv: 044404, 2017.
K He, Kaiming He, Haoqi Fan, H Fan, Yuxin Wu, Y Wu, S Xie, Saining Xie, R Girshick arXiv preprint arXiv 1911.05722. Ross Girshick, and 2019. Momentum Contrast for Unsupervised Visual Representation Learning. arXiv.org, November 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pp. 15663-15674 2019.
Chaoqin Huang, Fei Ye, Jinkun Cao, Maosen Li, Ya Zhang, and Cewu Lu. Attribute Restoration Framework for Anomaly Detection. November 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 14442.66800 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, 2018.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
Sachin Kumar and Yulia Tsvetkov. Von mises-fisher loss for training sequence to sequence models with continuous outputs. arXiv preprint arXiv:1812.04611 2018.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Directional analysis of stochastic gradient descent via von mises-fisher distributions in deep learning. arXiv preprint arXiv:1810.00150, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, 2018b.
Kuang-Hue' Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. arXiv preprint 0rXiv:2109.12900 2021.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. arXiv preprint arXiv:2104.04015, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint /60iv:/608. (00.:/0800....883.. 2016.
Ishan Misra and Laurens van der Maaten. Self-Supervised Learning Pretext-Invariant Representa- tions. December 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do Deep Generative Models Know What They Don't 't Know? October 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems Workshop Deep Learning and Unsupervised Feature Learning, 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. March 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CORR, 2018.
Tianyu Pang, Kun Xu, Yinpeng Dong, Chao Du, Ning Chen, and Jun Zhu. Rethinking softmax cross-entropy loss for adversarial robustness. arXiv preprint arXiv: /9055 10.../05500000 10626, 2019.
11

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. OCGAN: One-class Novelty Detection Using
GANs with Constrained Latent Representations. March 2019.
Stanislav Pidhorskyi, Ranya Almohsen, Donald A Adjeroh, and Gianfranco Doretto. Generative Probabilistic Novelty Detection with Adversarial Autoencoders. July 2018.
Tal Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. arXiv preprint Stti:22006.08844 2021.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation. October 2020.
Douglas A Reynolds. Gaussian mixture models. Encyclopedia of biometrics, 741:659-663, 2009,
Oliver Rippel, Patrick Mertens, and Dorit Merhof. Modeling the Distribution of Normal Data in Pre-Tree Trained Deep Features for Anomaly Detection. May 2020.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018.
Lukas Ruff, Robert A Vandermeulen, Billy Joe Franks, Klaus-Robee Müller, and Marius Kloft. Rethinking assumptions in deep anomaly detection. In ICML 2021 Workshop on Uncertainty & Robustness in Deep Learning, 2021.
Thomas Schlegl. Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery. March 2017.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data, arXiv preprint arXiv:1804.11285, 2018.
Bernhard Schölkopf, Robert C Williamson, Alex J Smola, John Shawe-Taylor, and John C Platt. Support vector method for novelty detection. In Advances in Neural Information Processing Systems, 2000.
Vikash Sehwag, Mung Chiang, and Prateek Mittal, (SSD): unified framework for self supf-supereieed outlier detection. In International Conference on Learning Representations, 2021. URL https
///openreview,n/ net/ fotp://o (xpm888
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for deep one-class classification. arXiv preprint arX1v:2011.02578, 2020.
Suvrit Sra. A short note on parameter approximation for von mises-fisher distributions: and a fast implementation of (x). Computational Statistics, 27(1):17-1990, 2012.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. Advances in Neural Information Processing Systems, 33:1 13:11899-18822 2020.
Yu Tian, Guansong Pang, Fengbei Liu, Seon Ho Shin, Johan W Verjans, Rajvinder Singh, Gustavo Carneiro, et al. Constrained contrastive distribution learning for unsupervised anomaly detection and localisation in medical images. arXiv preprint arXiv:2103.03423, 2021.
Vinay, Vinay S Shekhar, KN Balasubramanya Murthy, and S Natarajan. Performance study of Ida and kfa for gabor based face recognition system. Procedia Computer Science, 57:960-969, 2015.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9939-9939. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2022
Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karhikesalingam, and Simon Kohl. Contrastive training for improved cuttof--distrbbution detection. arXiv preprint arXiv:2007.05566, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 2017.
Zhisheng Xiao, Qing Yan, and Yali Amit. Do We Really Need to Learn Representations from In-domain Data for Outlier Detection? May 2021.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint PrXiv:/905.00544 2019.
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan O Arik, Chen-Yo Yu Lee, and Tomas Pfis- ter. Self-trained one-class classification for unsupervised anomaly detection. arXiv preprint arXiv:2/06.061 15, 2021.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International Conference on Machine Learning, 2016.
Richard Zhang. Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision, 2016.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. Febru- ary 2018.
A PLOTS FOR EVALUATIONS DURING TRAINING
Here we show the detailed version of the summary (mean) plots presented in Figure in the main text. They show the same curves for 4 distinct classes of CIFAR10, and also under 3 different metrics. They appear in Figure (SimCLR (SimCLR AUC and Linear probes during training), Figure 4 (SimCLR Kappa and MMD during training), Figure 5 (SimSiam AUC during training), Figure (Sima Kappa and MMD during training), and Figure 7 (SimCLR with norm AUC). The same findings as in the main text are seen hold across these detailed variants.
B DATASET DETAILS
The main datasets the splits and protocols specified TTack et al. 2020) and (Sohn et al. 2020): we add SVHN using the same basic protocol one class as inlier v.s. the remaining as outliers. We resized images to 32 32 for all datasets apart from ImageNet30 which uses the standard ImageNet ResNet-18 18 architecture's 's transformation of 224 224-pixel center-cropped region from the 256 256 input image.
CIFAR 10, CIFAR 100 and ImageNet30, fashion-MNIST following
C COMPARISON OF SCORING WITH DIFFERENT FEATURE EVALUATIONS, METRICS & ENSEMBLING
As discussed in Section 2.5 we compare the default scoring used in our main experiments (the encoder's 's last layer features using Sy-Cor) with a feature ensembling evaluation scheme consisting of these scores summed across feature maps from the encoder backbone network, and projection heads from either SimSiam or SimCLR. In the following tables we use following feature maps:
conv block n: The output feature map from blockn of the convolutional backbone. Which is first directly flattened from 2D 1D before evaluation.
13

---

Under review as a conference paper at ICLR 2022
GDB
Oos
0.005
14
R2
13
[3]
90
90
80
80
70
70
Epoch
Epoch
Epoch
50
50
50
4k
(a)
(b)
(c)
M
60
60
50
50
40
30
20
Epoch
20
Epoch
2k 8k
(d)
Figure 3: Training SimCLR classes 0,1,2,3 from CIFAR10 and evaluating the representation
during training. (a) Using Gaussian density estimation (GDE) (Reynolds) 2009). (b) k-Cos cosine distance to the closest (k=l) point in ID data. (c) c-Cos cosine distance the mean direction in ID
data. Both these are defined in more detail in Section 2.5 (d) Linear evaluation (Zhang et al. 2016) of learned embedding.
(d) Weighted nearest neighbor classifier (k-NN) (Wu et al. 2018)) Solid lines
represent & = and dashed lines represent k 20
3010
13| D
B1ID
1
000
MND
0.4
1500
0.3
1000
0.2
0.1
Epoch
Epoch
2k
4k
6k
(a)
(b)
Figure 4: Training SimCLR on classes 0,1,2,3 from CIFARIO and monitoring the learned vMF ID and OOD representations during training. (a) progress during training; solid lines ID data and dashed are OOD data. (b) The MMD (Gretton et al. 2012) between the learned representation and samples from uniform distribution on unit hypersphere (lower is more uniform).
conv (1x1): Same conv _block_n but after pooling to size of 1x1.
head layer.n: The output feature map from layer n of the projection head.
All Conv blocks: The sum of the scores of all convolutional blocks, using the distance specified in column, then summed across table columns merged in table.
14

---

Under review as a conference paper at ICLR 2022
GDE
Oo6
90
90
90
80
70
70
70
60
Epoch
Epoch
Epach
50
50
50
2k
3k
4k
Figure 5: Same experiment as Figure
but with SimSiam.
MMD
DI
2500
0.5
2000
1500
0.3
1000
0.2
500
Epoch
Epoch
1k
2k
3k
4k
4k
Figure 6: Same experiment as Figure but with SimSiam.
All blocks: The sum of the scores of all network internal feature maps, using the distance specified column, then summed across table columns merged in table.
Ens.: The sum of k-Cos and k-Cos (Mah) for 2D feature maps (e.g. convolutional feature maps) and c-Cos and c-Cos (Mah) for ID feature maps (e.g. projection heads).
We also investigate different metric functions for computing the OOD score, namely:
k-Cos: Cosine distance to closest (k=1) training vector, introduced Section 2.5
k-Cos (Mah): Same but evaluated in Mahalanobis space.
C-Cos: Cosine distance to the mean of all the training set vectors, introduced in Section 2.3
C-Cos (Mah): Same but evaluated in Mahalanobis space.
the same configuration as in Sohn et al. GDE: A Gaussian Kernel density estimator, 2020).
Tables 6 show an extensive evaluation of presented models at most internal layers and a variety of scoring metrics. We should highlight the fact that each number presented in those tables is the average across all classes in the dataset, with one experiment per class i.e. 10 experiments for CIFAR10 10, and 20 experiments for CIFAR100 100. We notice:
The Cosine Mahalanobis distance mostly outperforms most other evaluation metries, some- times by large gap.
There is a consistent improvement associated with ensembling of feature scores. This is true across models and across datasets.
We note this type feature ensembling can be considered free, computation-wise, compared with the ensembling used in e.g. CSI, that slows down the method significantly at inference.
15

---

Under review as a conference paper at ICLR 2022
CIFARIO
CIFARI00
IN30
IMNIST
SVHN
SS(n) SimSiam with norm, aka "NSA"
91.54
84.09
80.88
95.15
94.91
Ours
2. SC(n) SimCLR(w norm, no neg aug)
89.27
83.95
75.17
94.61
93.84
3. SC) SimCLR(w//o norm, neg aug)
86.49
80.51
75.17
94.61
93.84
4. SS(-) SimC norm, neg aug)
91.12
86.68
76.97
95.47
92.17
5. Pretrained ResNer 18 [Ours]
93.63
92.64
99.78
94.35
61.06
Pretraining FT
6. Pretrained Res [Ours]
94.45
94.68
99.91
95.
66. 28
1. Pretrained R5s [Ours
95.82
95.21
99.96
94.99
63.41
8. PT ResNet50 (DROC (Sohn ct al. 2020
80.0
83.7
91.8
10. 5esNet152 (DN2 (Keiss et 2020 1
92.5
94.1
94.5
52 (PANDA (Reiss al.
2020
96.2
94.1
95.6
12. Supr-Sup. PT R50. Xiao et al, et 2021
93.8
92.6
94.4
Dist. Shifting
13. CST (SimCLR loss only)
87.9
13. CSI (SimCLR neg aug only)**
90.1
86.5
83.1
14. CSI (Full)**
94.3
89.61
91.6
15. DROC, Contrastive
89.0
82.4
93.9
16. DROC, Contrastive DA
92.5
86.5
94.8
19. DeepSVDD (Ruf et al,
2018
64.8
67.0
84.8
Other Methods
20. DROCC (Goyal ct al 2020
74.2
21. Geom (Golan et al.) GGlln & Ei Yanıy 2018
86.0
78.7
93.5
22. GOAD (Bergman) (Bergman & Hoshen 2020
88.2
74.5
94.1
23. ARNet (Huang) (Huang cl al 2019
86.6
78.8
93.33
24. Hendryks et al. (Hendrycks et al. 2019
90.1
79.8
85.7
93.2
25.SSD (Schwag et al. 2021
90.0
Table 5: One-Class Classification Summary results reported in the literature on various datasets, plus
some of our results; all figures are AUC. indicates methods trained external additional data, which may overlap in scope with the "unseen" OOD data. indicates method using test-time data augmentation/ ensembling during evaluation, which can involve drastically slower inference.
k-Cos
k-Cos (Mah)
c-Cos
C-Cos (Mah)
GDE
conv.block.
80.29
83.58
65.35
83,52
conv_block,
85.87
83.59
75.81
83.53
conv block
92.21
91.20
88.35
91.12
conv_block (Ix1)
71.29
70.85
68.39
67.37
67.39
block 2 (1x1)
73.19
71.93
70.94
69.66
69.39
conv block 3 (Ix1)
81.46
83.47
73.60
79.66
79.32
conv_block.44 (Ix1)
91.85
91.54
84.88
90.52
91.04
head_layer
82.64
87.84
71.18
87.56
head_layer_2
80.62
83.08
70.50
82.76
head_layer
79.67
77.41
51.93
54.64
All Conv blocks
92.37
90.79
All Conv blocks
92.01
All blocks
91.62
92.23
80.50
90.86
89.96
All blocks
92.86
88,74
All blocks
92.86
Ens.
92.5
Table 6: Different feature ensembling methods: SimSiam w norm on Cifar10. Note that items spanning multiple columns imply summation of the corresponding features.
16

---

Under review as a conference paper at ICLR 2022
k-Cos
k-Cos (Mah)
c-Cos
c-Cos (Mah)
GDE
conv block
73.36
76.42
58.54
76.39
conv conv_block.3
79.31
77.83
68.39
77.80
conv block.l.k...
85.47
84.46
78.19
84.40
conv_block_1 (Ix1)
66.35
67.51
60.75
64.77
64.76
conv_block (Ix1)
68.05
67.99
61.93
65.46
65.37
conv_block (1x1)
74.26
76.49
64.76
72.78
72.75
conv blockblock.o 4 (Ix1)
84.76
84.09
72.08
82.74
83.15
head_layer_1
76.95
81.46
64.50
81.24
head layer 2
74.78
77.46
62.95
77.29
head layer 3
75.23
73.86
52.65
68.15
All Conv blocks
86.45
84.75
All Conv blocks
85.89
All blocks
85.19
87.11
72.50
86.17
All blocks
87.54
85.79
All blocks
87.25
Ens.
86.6
Table 7: Different feature ensembling methods; SimSiam norm on Cifar100
k-Cos
k-Cos (Mah)
c-Cos
c-Cos (Mah)
GDE
conv block
79.62
82.40
64.71
82.32
cony_block_3
81.08
80.22
73.38
80. 14
conv hunv_block_4 4
88.92
89.47
82.83
89.42
conv. wnw.kb (1x1)
71.63
71.53
68.56
68.12
68. 17
conv_block (Ix1)
73.85
72.67
70.60
70.12
69.79
conv_block. 3 (Ix1)
74.97
77.00
68.97
74.47
73.50
conv block 4 (1x1)
86.68
89.27
75.68
88.43
88.39
head layer
73.88
81.05
48.36
80.63
head_layer:
65.19
72.33
41.26
46.18
All Conv blocks
90.65
89.61
All Conv blocks
90.57
All blocks
87.37
90.16
75.65
89.12
All blocks
90.20
89.10
All blocks
90.35
Ens.
90.3
Table 8: Different feature ensembling methods: SimCLR norm Cifar
17

---

Under review as a conference paper at ICLR 2022
GDE
Oos
0.005
13
[3]
90
80
80
80
70
60
Epoch
Epoch
Epach
50
50
50
4k
Figure 7: Same experiment as Figure but after adding the normalization proposed in Section 2.3
MMD
000
110000
DOD
2000
0.5
0.4
1500
0.3
1000
0.2
500
0.1
Epoch
Epoch
1k
3k
Figure 8: Same experiment
Figure but after adding the normalization proposed in Section 2.3
D CIFAR 10 PER-CLASS RESULTS
Here in Table we show the per class results for each of the CIFAR 10 classes trained for one-class classification against the others, and compare our NSA method (SimSIAM with Norm, no ensembling augmentations), with (i) last layer features only and (ii) summing all features, with other recent works that also report these results.
E ADDITIONAL POLLUTION RESULTS
Table 10 shows additional CIFARIO results in the presence of pollution, also more comparisons.
F ADDITIONAL COMPARISON WITH PREVIOUS RESULTS FROM LITERATURE
Table 5 gives more detailed overview of related OOD / One Class Classification results from the literature, and comparison with variants of our baseline (ensemble free) variants of SimSiam, SimCLR,
Dog
Frog
Ship
Blck et 2020 Lnc. eusembling angementttions
90.0
99.1
93.2
86.1
93.8
93.4
95.2
95.6
97.9
95.5
S4.3
DXCC, OC V (Selin et al 2020
88.8
97.5
87.7
82.0
82.4
89.2
89.7
95.6
86.0
90.6
89.0
DROC Crustian KDE (DA) Sela A020
91.0
98.9
83.2
89.4
90.2
93.5
98.1
96.5
95.1
92.4
Rot Pred, SV- from
Spln aran
83.6
96.9
87.9
79.0
90.5
89.5
94.1
96.7
95.0
94.9
90.8
Denosing OC-VVM
KotNet Kat, Cis, Golan ranyy 2013
M0.3
91.2
85.3
85.9
36.7
89.6
95.3
91.8
36.0
36.8
NSA (SimSEX mirm
90.4
9966
85.2
85.7
841
92.9
92.9
94.5
96.3
90.99
91.52
nomi)
[Ours]
Table 9: CIFARIO Per class results
18

---

Under review as a conference paper at ICLR 2022
p=0
p=0.05
p=0.10
A (p=0.1 p==0
SimSiam(w w norm)
91.54
88.08
86.21
5.33
SimSiam (/oo norm)
89.56
85.21
81.79
7.77
SinCLR(no neg. w norm)
89.27
83.1
77.49
11.78
SinCLR neg. w/o norm)
86.49
71.1
63.5
25.99
SimCLR (neg aug, w norm)
92.91
86.1
83.38
9.53
SimCLR (neg aug, w/o norm)
91.12
81.5
77.94
13.18
Pretrained IN18
92.63
90.9
89.3
3.33
Pretrained IN50
94.45
92.18
89.49
4.96
Pretrained IN152
95.82
94.48
91.39
4.43
DROC (Sohn et al.) 2020)
89
76.5
73.0
16.0
DROC (DA) (Sohn et al.) 20200)
92.5
85.0
80.5
12.0
CSI (results from (Han et al. 2021)
94.3
88.2
84.5
9.8
Semi- Superisuee (5% labeled) ELSA (Han al.) Semi-SSpervised (5% labeled) ELSA+ (Han et al.)
85.7
83.5
81.6
4.1
95.2
93.0
91.1
4.1
Table 10: CIFARIO pollution experiments. p is the ratio of outlier data inside the training set. the last column is the loss in performance between training with clean data and 10% polluted data. Clearly, our proposed modifications reduces the drop in all cases. With SimSiam we beat standard ELSA which uses 5% labeled data to maintain robustness to pollution, and come close to ELSA+, which uses TTA and other tricks from CSI on top of the baseline SSL approach.
along with our implementation of pretrained ResNets (as an additional basline for comparison to other pretrained methods.
The pretrained results also serve to indicate where these types of approaches can fall down, and the issues in a with fairly comparing these pretrained methods with from-scratch trained approaches in the Anomaly/OOD detection context (It unsurprising that pretraining a representation on ImageNet does well distinguishing "unseen" ImageNet30 classes in One-Class classifier scenario, and similarly for e.g. CIFAR, where the same types of class are present; however SVHN is less similar and therefore the pretrained represenation is less useful).
More details and notes on these competing methods follow. In particular, we clarify that "(n)" indicates a method using our proposed modifications, and "w/o norm" is the standard version of the architecture (SimCLR or SimSiam) without these modifications. "(-)" means including strong distributionally negative shifted augmentations, using randomly the four 90, 180, and 270 degree rotations, following the approach of CSI (Tack et al. 2020).
We note in particular that CSI's method combines many parts (contrasiiv++classiicttto losses and scoring functions, plus ensembling) which each contribute something and add up to give good results. Our goal of instead showing baseline SimCLR results with / without the norm, and with/without shifted augmentations is to create more straightforward bascline compare with. We note improved results are possible with our method adding these additional features such as ensembling, but also at additional computational cost, is the case with CSI.
On the other hand, our pretrained baselines obtain very good results equal or exceeding PANDA (Reiss al. 2020''s 's fine-tuned results datasets that similar in nature ImageNet that they are pretrained on; this exemplifies the benefits our chosen scoring metric SNSA = Sy-Cor on good representations general. However also show datasets where this approach falls down, compared with self-supervised methods.
For each of the methods, here is detailed description of the features, networks, scoring functions etc. used for comparison:
1. NSA [ours]; features = SimSiam (with norm), ResNet18; scoring = KNN + Mahalanobis Cosine, last layer features.
2. features = SimCLR(w norm) + NO negative shifting augmentations, ResNet18, scoring KNN Mahalanobis Cosine, last layer features. [our baseline]
19

---

Under review as a conference paper at ICLR 2022
3. features SimCLR(w/o norm) NO negative shifting augmentations, ResNet18; scoring KNN + Mahalanobis Cosine, last layer features. [our bascline]
4. features = SimCLR(w/o norm) + With strong rotation negative shifting augmentations, ResNer18; scoring KNN + Mahalanobis Cosine, last layer features [our baseline, closest to CSI (without ensembling) DROC+DA]
5. Pretrained 8 (on ImageNet), no fine-tuning; scoring KNN Mahalanobis Cosine,
last layer features [our baseline]
6. Pretrained ResNet50 (on ImageNet), fine-tuning; scoring KNN Mahalanobis Cosine, last layer features [our baseline]
7. Pretrained ResNet152 (on ImageNet), no fine-tuning: scoring = KNN Mahalanobis Cosine, last layer features [our baseline]
8. Pretrained ResNet-50 on ImageNet, results from DROC (Sohn al. 2020
9. Rippel al. (Rippel al., 2020) Pretrained EfficientNet-BA on Imagenet + Mahalanobis (no results on our benchmark datasets).
10. Reiss et al (Reiss al. 2020) Simple baseline "DN2" Pretrained ResNet- 152 on ImageNet, no fine-tuning + kNN scoring from last layer, presumably euclidean distance
11. Reiss et al. Reiss et al., 2020) PANDA-EWC Pretrained ResNet-152 on ImageNet, Fine- tuned last 2 layers each dataset, with compactness loss kNN=2 scoring from last layer, presumably euclidean distance.
12. Xiao et al. (Xioo et al. 2021) Self-supervised Pretrained ResNet-50. SimCLRv2, Gaussian Mixture Model Mahalanobis distance
13. (a) CSI, SimCLR loss only; features SimCLR (w/o norm), ResNer18 scoring Sim- only contrastive(cosine norm). [their results Table 15]: (b) same but with full CSI loss, contrastive With strong rotation negative shifting augmentations; scoring contrastive Sim-only (cosine).
14. Full CSI; features SimCLR(w/o norm) + With strong rotation negative shifting augmen- tations, ResNet18 scoring contrastive(cosine norm) rotation prediction, on shifted transforms, with ensembles. [their main results, essentially combining many parts]
15. DROC (Sohn et al. features SimC norm) + NO negative shifting augmentations: scoring = (OC-SVM). [their results]
2020) Contrastive (=Deep Representation One-class Classification);
16. DROC (Sohn et al.
2020) Contrastive DA (=Deep Representation One-class Classification); features = SSmCL.R(W)) norm) + With strong rotation negative shifting augmentations. scoring (OC-SVM). [their results]
17. ELSA (Han al.
2021) NO negative shifting augmentations with 1 1% labeled outliers
Han et al. 2021
18. ELSA+ (Han et al. 2021 With strong rotation negative shifting augmentations (like CSI), and with ensembles also with 1% labeled outliers (Han al.
2021
19. DeepSVDD, Ruff al. (Rulf et al. 2018). with LeNet architecture Autoencoder adaptated
features. [All results apart from CIFARTA 10 from Reiss(Reiss et al. 2020
20. DROCC, Goyal et al. (Goyal et al.
2020
LeNet architecture
21. Geom Golan et al. GGolan & El- Yaniv
2018
WRN- 16-8 Arch.
22. GOAD, Bergman al. (Bergman & Hoshen 2020
WRN-16-4 architecture (CIFAR 100 2020)
results from CSI with ResNet18] (Bergman & Hoshen
23. ARNet (formerly called Inv. Trans AE) Huang et all Huang et al. 2019
24. Rot Trans, Hendryks et al. (Hendrycks et al. 2019)- WRN- 1644 architecture (CIFAR 100 results from CSI with ResNet18; IN30 are ResNet18 Rot-Trans+AAtn+ fMNIST results from Reiss (Reiss et al. 2020)
25. SSD (Shwwag et al. 202I): features SimCLR(w/o norm) NO negative shifting augmen- tations scoring Mahalanobis [their results]
20

---

Under review as a conference paper at ICLR 2022
G DETAILED ABLATION STUDY
In Table we show an extensive ablation study demonstrating results on CIFARIO, CIFARI00 and IMNIST of each variant of the methods study, with and without our normalization enhancements, for different pollution settings, and under 5 different feature evaluations metrics (same as those in Appendix C and the feature ensemble (Ens.) proposed in Section 2.3] We would like stress that this study includes training 640 different models and evaluating each model using 6 different metrics.
We take few important notes:
For all examined situations, the proposed normalization always brings a noticeable im- provement. The highest improvement for SimCLR in the presence of pollution; this is consistent with our analysis in the main text.
For different datasets, different algorithms, and different pollution ratios, the proposed ensembling has the best performance most of the time, and the second best otherwise, which shows how generic our proposed ensembling scheme is.
k-Cos (Mah) is the metric most often getting second best results among all evaluated, and as such was chosen for our simple baseline (ensemble-fiee comparison).
Although in many times is not the best or second best metric, c-Cos is the metric that gets the largest boost in performance after applying normalization, which is expected because as the ID representation gets more compact, the center much more representative of the distribution.
H BACKGROUND AND RELATED WORK
H.1 ANOMALY AND OOD DETECTION
Outlier detection is important in a variety of practical tasks, such as detecting problems in a production process, detecting security events, and acting upon novelties. The most general case is unsupervised novelty detection (or poisoned data): there are outliers in the training set, and we have no information about them. Then there degrees of supervision, semi-supervised (a few outliers are labeled) and fully supervised (all outliers are labeled). Furthermore, in the sub-field of anomaly detection it is assumed that there are no outliers in the training data. A challenge OOD detection that the notion of in- and out-of -fittriiiitiii is not well defined, and task dependent. A good OOD method would generalize to different notions of nut-of-distributin and datasets, e-g. with respect to color, style, perspective and content. Recent approaches in Anomaly/OOD detection can be categorized in four groups:
Densi methods are based on the assumption that models trained to fit the in- distribution data will be less confident on out-of-distribution data in terms of likelihood of the outputs. Using the likelihood as detection score has been shown to be weak metric (Nalisnick et al. 2018 2018| Choi et al. 2018). and modifications such as entropy, energy (Du & Mordatch] 2019 Grathwohl al. 2019) and WAIC (Choi et al. 2018) have been proposed.
One-class classifiers are classic approach for outlier detection and have been adopted to deep learning settings. They find a decision boundary that separates ID and OOD samples.
A margin is introduced to allow generalization (Schölkopf et al.) 2000 Ruff al. 2018).
Reconstrucctio--asse methods model the ID training data by training an encoder and decoder network to reconstruct the in-distribution data. The reconstruction will generalize less for OOD data such that the reconstruction loss can be used a the detection metric. Auto-encoders (Zong al. 2018 Pidhorskyi al. 2018) and GANs (Schlegi al. 2017
Deecke et al. 2018 Perera et al. 2019).
Self-supervised methods leverage the representations learned from self-supervision, com- bined with different detection scores. The current totte-of-the-rrt in OOD detection is CSI (Tack et al. 2020), using representations learned by SimCLR (Chen et al. 2020a) and the distance to the closest training point in latent space as detection score. Other approaches train networks with predefined tasks such as permutations of image patches or rotations
(Golan & El-Yaniv 2018 Hendryeks et al. 2019 Bergman & Hoshen 2020)
21

---

Under review as a conference paper at ICLR 2022
Table Detailed ablation study. Here p the ratio of outlier data inside the training set. Norm is whether normalization is applied or not. Ens. is our proposed feature ensembling. Best results are in bold. Second best results are underlined.
Data
Algo
p
Norm
k-Cos
k-Cos (MAH)
c-Cos
e-Cos (MAH)
GDE
Ens.
0
x
84.9
88.5
51.3
83.8
86.2
88.9
0
V
89.9
90.5
79.5
89.0
89.5
91.9
BYOL
62.7
82.9
63.4
82.0
80.5
85.3
0.1
77.5
85.3
78.5
83.6
83.0
88.3
0
X
86.5
89.5
59.3
85.8
87.9
90.1
0
91.6
91.7
84.9
90.5
91.0
92.5
SimSiam
0.1
65.8
83.2
67.9
81.5
80.7
84.3
0.1
80.6
86.3
82.5
84.9
84.3
88.4
79.3
86.3
45.6
84.9
84.7
87.8
0
86.0
88.9
75.2
87.9
88.0
90.3
SimCLR
C10
0.1
53.1
65.6
62.6
65.9
64.7
80.3
68.8
79.8
80.0
78.7
78.2
86.7
88.6
90.7
79.1
90.0
90.1
91.1
0
91.2
92.9
87.3
92.5
92.6
93.0
SimCLR(-)
0.1
x
71.4
79.6
83.4
80.8
80.2
86.3
77.9
83.9
87.2
83.6
83.1
87.8
78.6
79.6
49.4
74.8
76.8
81.3
0
81.0
80.2
59.5
77.5
78.2
83.4
BYOL
0.1
x
74.1
76.2
50.8
71.4
73.4
77.7
0.1
78.0
77.8
59.5
75.0
75.7
80.7
79.7
81.4
55.6
77.1
78.8
83.3
0
84.5
84.3
72.1
82.7
83.1
86.6
SimSiam
0.1
X
73.1
78.9
54.7
75.4
75.4
79.7
0.1
79.7
80.3
67.2
78.4
78.4
82.5
76.3
77.0
35.5
76.3
78.2
84.2
C100
0
80.1
82.0
68.4
82.6
82.7
86.9
SimCLR
0.1
x
69.2
75.9
46.5
75.0
73.8
79.9
0.1
75.8
80.0
65.6
79.2
78.6
83.0
83.4
84.7
68.5
84.9
85.9
87.9
SimCLR(-)
85.8
87.0
80.3
87.4
87.8
89.4
0.1
X
78.1
80.5
67.3
81.1
81.0
84.3
0.1
81.2
82.8
79.5
83.9
83.6
85.8
90.5
95.3
84.7
95.0
95.4
95.9
0
93.2
95.1
91.2
94.8
95.0
96.2
BYOL
0.1
38.1
61.3
84.9
75.5
75.2
86.7
0.1
48.0
73.2
86.9
80.9
80.9
87.9
92.7
95.9
84.7
95.7
95.8
95.8
93.9
95.0
90.7
94.8
94.9
96.1
SimSiam
0.1
40.3
63.0
88.4
73.3
72.6
86.1
0.1
52.7
75.3
90.3
80.0
79.8
87.8
87.6
94.6
70.4
95.0
95.1
96.1
FMNIST
91.3
94.9
86.8
94.9
95.0
96.3
SimCLR
0.1
30.9
46.5
88.4
55.5
55.0
86.3
0.1
34.9
53.1
90.6
61.1
60.6
87.5
0
X
92.7
94.7
86.3
94.5
94.5
95.6
SimCLR(-)
94.2
95.7
90.4
95.6
95.6
95.9
0.1
60.9
78.7
90.5
83.7
83.5
90.4
0.1
65.3
80.9
92.1
84.4
84.3
90.9
22

---

Under review as a conference paper at ICLR 2022
H.2 SEEE-SUPERV LEARNING (SSL)
SSL is a form of unsupervised learning, tackling it through means of supervised learning from pseudo-labels that easily be generated. One line of computer vision research uses augmentations as pseudo-labels. These augmentations can be generated at no additional human cost, for example 90 degree rotations results in four labels. In jigsaw tasks (Noroozi & Favaro 2016) the image split in grids, for example 2x2 or 3x3, and shuffled, the resulting position is the prediction target.
Another more recent direction is constrastive learning (Oord al. 2018 He al. 2019 Misra van der Maaten 2019 Grill et al. (020aa. In SimCLR (Chen et al. 2020a b) every image in a batch augmented twice, and the objective
minimize the distance of the latent representations of the same origin image, while maximizing the distance to other images in the batch.
Another recent SSI. direction is non-contrastive or positive samples only SSL. Bootstrap Your Own Latent (BYOL) (Grill et al.) 2020b) was the first example of this class of algorithms to achieve very competitive results, that even surpasses SimCLR. BYL gets away from the problem representation collapse (first enemy of SSL, usually handled by negative samples) by introducing assymetry in the network architecute through the idea of prediction network after the project head, it also uses an exponential moving average of the weights of the network target representation. SimSiam (Chen & He 2020) made a significant analysis on B BYL and found that using a moving average of the weights wasnot necessary and just a simple stop- taoo--rrdddd operation enough.
23


Reviews:
[{"review_id": "qbqT7fn7B9o", "reviewer": "Reviewer_9Mbn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) This work looks solid, and the experiments in this paper are sufficient, which consider almost all key parameters or components of the proposed method and demonstrate the effectiveness of the method. Besides, the method achieves state-of-the-art performance on OOD task.\n\n(2) The paper spots the reliability issue on long term training of the contrastive Self-Supervised Learning (SSL) methods as deep feature extractors on OOD. The authors claim that no-uniformity and compactness of the learned ID distribution is the main reason to improve the performance on OOD.\n\n\n\n\n\nWeaknesses:\n\n(1) The proposed method seems simple, the proposed NSA is built on a modified (l2-norm) SimSiam architecture.\n\n(2) The title states that authors targets on anomaly detection, but this paper works on out-of-distribution detection. They are not the same problem: the ID for anomaly detection is one-class; and that for out-of-distribution detection is multi-class.\n\n(3) This manuscript is not very well organized. The organization of tables, figures and text is in a mess. Tables and Figures shall not be placed between text, and the distance from the table to its reference is somehow long. For example, Table 6 should not be place behind the conclusion, which should be the end of the paper. And the paper exceeds the 9-page limit. \n\n\n\n\n\nImprovements:\n\n(1) It would be better to combine Figure3 and Figure 7, Figure 4 and Figure8. The organization of figures makes readers hard to follow. \n\n(2) I wonder the slight difference between with and without normalization showed in Figure 2, especially on SimCLR (the same as Figure 4 and Figure8). In addition, Figure 2(b) miss to show the result of SimCLR w/o norm.\n\n(3) The equation in Figure 1 is l1-norm, but the paper states to employ l2-norm. Figure 1 is the most important figure in this paper. This confuses me a lot.\n", "summary_of_the_paper": "In this paper, authors study anomaly detection under fully unsupervised setting. This work explores novel architectural modifications on SimSiam (Chen & He, 2020) to the self-supervised feature learning step, that facilitates compact in distribution (ID) distributions to be learned on out-of-distribution (OOD) detection. It performs robustness to polluted ID. In addition, the paper investigates how the geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier.", "summary_of_the_review": "Overall, the paper presents some good empirical results of OOD detection on benchmarks. This paper is well motivated and solid, but the writing needs to be improved. However, it is somehow weak in terms of technical novelty. Therefore, I vote this paper marginally below the acceptance threshold. \nGiven that I am not very familiar with OOD, I would like to check other reviewers' opinions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hXyjgVzK46", "reviewer": "Reviewer_UoZi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__-=-Pros-=-__\n\n * __Problem Significance:__ Using unsupervised learning as a step towards totally unsupervised AD is an approach that is sensible and worth investigating. The problem identified by the authors makes sense and solving it by finding useful unconcentrated representations is reasonable and could be of some general interest outside of deep AD as well.\n* __Approach:__ The authors provide a sensible line of reasoning for their approach and it doesn't seem ad-hoc. This point is meant to contrast a paper whose approach seems to be developed through shotgun testing and doesn't provide much insight to why an approach works.\n* __Experimental Setup:__ Several different approaches using standard datasets and setups. Reasonable competitors. Small novel contribution with the vBM distribution approach\n* __Experimental Results:__ Results strongly indicate that the proposed method does improve the representation for AD.\n* __Clarity:__ Generally good.\n\n__-=-Cons-=-__\n * __Novelty:__ The proposed change is quite small. The value of spherical representations has been observed quite a few times, especially in contrastive learning, so inserting it into a few more layers isn't really making a contribution that will likely be broadly useful.\n *__Lack of Theoretical Analysis:__ If this paper would be very strong had a bit of relevant theoretical analysis. For example analyzing the distribution of an optimal representation for some very simple problem setting, perhaps even showing theoretically that its not uniform. Analysis of the form of \"Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere\" (Wang & Isola 2020) would be great.\n *__Experimental Results/ Competitor Approaches/ Missing Citations:__ The proposed method is still outshone by other deep AD methods. In particular [outlier exposure](https://openreview.net/forum?id=HyxCxhRcY7) (OE) methods work very well as well as transfer-learning based methods, and [combining the two](https://proceedings.mlr.press/v139/deecke21a.html) achieves 99%+ AUC on your CIFAR10 experiments. This is somewhat counter to the assertion that \"the assumption...that learned image features from general datasets such as ImageNet will transfer well in all cases may be restrictive at best.\" I think the non-OE non-transfer learning based methods are interesting in their own right, but I think these other approaches/results should at least be mentioned.\n *__Some parts could use more detail:__ The paper should really include a precise description of SimSiam and SimCLR since they are so integral to the contribution you are making. BYOL should also be introduced (also the full name of BYOL should be mentioned somewhere instead of going directly to the acronym). This should be in the appendix, at the very least. \"Mahalanobis space\" isn't particularly standard or common nomenclature and should be described precisely somewhere (in the appendix if necessary).\n* __Quite a few style and grammar errors:__ At end.\n\n__-=-Verdict-=-:__ As is the paper is just a bit worse than what I would expect out of ICLR, however there is a lot of easy room for improvement.\n\n\n__-=-Other Misc. Errors-=-__\n* __Bibliography:__ There are numerous errors in here and the author should go through all the cited works carefully. Below are some examples\n  * There are several citations that reference arXiv version despite the existence of peer-reviewed and/or archival versions. I only checked two arXiv works but they bot had peer reviewed archival versions (Ruff 2020 has a [workshop version](https://github.com/lukasruff/Classification-AD), Tack 2020 has a [NeurIPS paper](https://proceedings.neurips.cc/paper/2020/hash/8965f76632d7672e7d3cf29c87ecaa0c-Abstract.html) ).\n  * Sehwag 2021 is missing a venue entirely.\n  * Tack 2020 \"Csi\" should all be capitalized, you can do this with {CSI}\n  * Schölkopf 1999, I'm not sure what is standard procedure now, but you may want to consider using \"NeurIPS\" since there [is some controversy surrounding its old name)[https://www.nature.com/articles/d41586-018-07476-w].\n* __Capitalization:__ There are quite a few random and incorrect capitalized words. Examples\n  * p.1 \"Deep Neural Networks\" should be \"Deep neural networks\"\n  * p.1 \"Industrial\" should be \"industrial\"\n  * p.1 \"Self-Supervised Learning\" should be \"Self-supervised learning\"\n*__Figures in weird locations__: Several of the figures referred to in the main text are in the appendix. The main text should really be self-contained, so you should at least give a summary of the figure in the main text and treat the figures in the appendix as additional information.\n* __Some figure keys are too small:__ Figure 2 right is the worse offender of this\n* __Figure 2 caption:__ y axis label overlaps the graph, should contain a brief description of kappa and MMD \n* __Small errosr:__\n  * p. 7: bottom missing space in \"Section2.3\"\n  * p. 6: Not sure what is meant by \"high\" and \"low\" level scores\n  * \" are two random training and test samples..\" should be \"are a training and test sample respectively\"\n  * MMD should be introduced with full term (Maximum Mean Discrepancy) and briefly described\n  * \"use the tool of the von Mises-Fisher\" shouldn't use the word \"tool\" in my opinion, its just a distribution.\n  * Figure 1: should use \\left< foo \\right> to get correct angle brackets\n  * (1): replace comma with period", "summary_of_the_paper": "In this paper the authors propose a way to improve the representation learned from contrastive learning when the downstream task is anomaly detection/out of distribution detection (AD), specifically an unsupervised AD problem where one _only_ uses the unlabelled training data which is assumed to contain only, or mostly, nominal data. Several popular contrastive learning methods map samples to points on a unit hypersphere and find a representation that encouraged to be uniform over the sphere. This is problematic for AD since one needs the nominal data to be concentrated so that it is clear when a test sample lies away, thereby being indicative of an anomaly. To remedy this issue the authors enforce the representation at multiple levels to be on the unit sphere (they look at SimSiam and SimCLR in particular) thereby making it impossible to enforce the contrastive similarity loss to use scale as a way to measure similarity, it must always use direction. This is demonstrated to be effective and makes representations which are more concentrated, useful for AD, and are more robust to pollution and varied batch sizes.", "summary_of_the_review": "Paper is borderline, could be improved significantly by improving the exposition. I've included several suggestions for this.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aW0ypY9aLJT", "reviewer": "Reviewer_p72E", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ (+) The paper introduces several insightful contributions: e.g. AD correlation with distribution uniformity measured by fitting a vMF distribution or by MMD; or solution to the gradient-flow issue in SimSiam.\n+ (+) The proposed improvements to SSL anomaly detection do not need to make assumptions on the data invariances\n+ (+) The proposed multi-scale feature ensamble is an effective way to exploit cues at different semantics/resolutions without increasing inference time\n+ (+) The proposed improvements are claimed to hold even under mild data pollution\n\nMAJOR\n\n+ (-) The paper is not well organized: starting from Sec. 2.2 going forward the reader is continuously referred to the appendix to see plots, tables and details that the reviewer finds essential to understand, appreciate and validate the authors' claims.\n\nMINOR\n\n+ typos: ood evaluation -> OOD evaluation (Sec 1, page 1); to actually to decrease -> to actually decrease (Sec. 2.3 page 5); \n+ Table B is references but it does not exists in the paper nor in the appendix.", "summary_of_the_paper": "The paper contributes to the field of self-supervised anomaly detection by i) verifying the correlation between AD performance and in-distirbution (ID) representation uniformity and how to improve it with a simple normalization; ii) introducing a simple solution to a gradient-flow issue in previous work; and iii) proposing a multi-level feature ensembling as a cheaper but quantitatively effective alternative to test-time data augmentation and model ensambling. They build on SimSiam and SimCLR to support all of their claims.", "summary_of_the_review": "On one hand the paper has good contributions, provides insights into the problem that are supported by empirical evidence; on the other hand is poorly organized and written: most of the key results are supported in the appendix, which should be optional - not critical - to the reading of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-J48vx8f0SZ", "reviewer": "Reviewer_ft5T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength\n  - The paper made a good presentation with motivating empirical results to justify their proposed method.\n  - The proposed method is rather simple, but shown to be effective. \n\n- Weakness\n  - Ablation study on the proposed Mahalanobis cosine similarity is missing. I wonder they are really the necessary component that improves upon existing one class classifiers proposed by [Lee et al., 2018b](https://proceedings.neurips.cc/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf) or those studied in [Sohn et al., 2021](https://openreview.net/forum?id=HCSgyPUfeDj).\n  - [Sohn et al., 2021](https://openreview.net/forum?id=HCSgyPUfeDj) suggested to use L2 normalized encoder output for one-class classification. For models without normalization, I wonder whether authors use L2 normalized encoder output or unnormalized ones.\n  - The motivation for adding L2 normalization layer at the end of the projection layer of SimSiam in Section 2.4 needs clarification. It is unclear what it means by \"all the gradients of that operation gets blocked by the stop-grad, though it is an integral ...\". Also, there seems no specific ablation study provided (by default, I assume \"w/ norm\" model is with L2 normalization both at the end of encoder and projection layer and \"w/o norm\" with no L2 normalization layer at all).\n  - While mentioned multiple times, there is no empirical result using BYOL, so it is unclear their behavior will be also similar to SimSiam. \n  - I would be a bit careful when saying \"Feature ensembling does not require a validation set\". It is rather heuristic and has been shown (e.g., [Defard et al., 2021](https://arxiv.org/pdf/2011.08785.pdf) or [Rippel et al., 2021](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9412109)) specifically for anomaly detection that one should be careful to select correct layers to aggregate.\n  - Feature ensembling could be effective in practice, but it makes comparison to other methods somewhat convoluted, especially when readers want to understand the source of improvement. It would be beneficial to have results using only encoder output in Table 1 or Table B.\n\n- Misc\n  - For Figure 3 ~ 8, legends are not properly cited.\n  - Mahalinobis -> Mahalanobis.\n", "summary_of_the_paper": "The paper studied the issues of recent self-supervised contrastive learning based anomaly detection and presented few modification to resolve those issues. Specifically, the paper pointed out the issue of contrastive learning for OOD detection when trained longer as their embedding distributions become more uniform, and proposed an architectural modification by adding a L2 normalization layer after encoder. While SimSiam, a non-contrastive self-supervised learning algorithm, suffers less from uniformity issue, authors found adding L2 normalization layer to SimSiam effective. Authors also pointed out some gradient flow issue of SimSiam and proposed adding another L2 normalization layer at the end of projection layer. Evaluation on anomaly detection verify the importance of the proposed L2 normalization layers, but the per", "summary_of_the_review": "- The technical contribution of the paper is very simple (adding few L2 normalization layers). While showing some good empirical evidences, the overall contribution of the paper is marginal and does not extend state-of-the-art. Also, while proposing many small techniques, they are not fully verified in experiments (please see my comment above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates how the geometrical compactness of in-distribution examples affects OOD detection performance and proposes architectural modifications to enable compact in-distribution embeddings. All the reviewers agreed that the paper has several interesting contributions. I agree with the authors that simplicity is a strength, not a weakness. 

My main concern is that the paper's contributions feel a bit scattered. For instance, the paper does a detailed evaluation of normalization and compactness, but makes a few other minor contributions (as detailed by 
the authors at https://openreview.net/forum?id=7VH_ZMpwZXa&noteId=m-1y5byLbwS​). However, the latter contributions feel a bit narrow to specific methods and are not as comprehensively tested as the claims around normalization.

Overall, the reviewers and I think that the current version falls below the acceptance threshold. I encourage the authors to revise the draft and resubmit to a different venue.
Decision: Reject