Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors propose a way to improve the representation learned from contrastive learning when the downstream task is anomaly detection/out of distribution detection (AD), specifically an unsupervised AD problem where one _only_ uses the unlabelled training data which is assumed to contain only, or mostly, nominal data. Several popular contrastive learning methods map samples to points on a unit hypersphere and find a representation that encouraged to be uniform over the sphere. This is problematic for AD since one needs the nominal data to be concentrated so that it is clear when a test sample lies away, thereby being indicative of an anomaly. To remedy this issue the authors enforce the representation at multiple levels to be on the unit sphere (they look at SimSiam and SimCLR in particular) thereby making it impossible to enforce the contrastive similarity loss to use scale as a way to measure similarity, it must always use direction. This is demonstrated to be effective and makes representations which are more concentrated, useful for AD, and are more robust to pollution and varied batch sizes.
main_review: __-=-Pros-=-__

 * __Problem Significance:__ Using unsupervised learning as a step towards totally unsupervised AD is an approach that is sensible and worth investigating. The problem identified by the authors makes sense and solving it by finding useful unconcentrated representations is reasonable and could be of some general interest outside of deep AD as well.
* __Approach:__ The authors provide a sensible line of reasoning for their approach and it doesn't seem ad-hoc. This point is meant to contrast a paper whose approach seems to be developed through shotgun testing and doesn't provide much insight to why an approach works.
* __Experimental Setup:__ Several different approaches using standard datasets and setups. Reasonable competitors. Small novel contribution with the vBM distribution approach
* __Experimental Results:__ Results strongly indicate that the proposed method does improve the representation for AD.
* __Clarity:__ Generally good.

__-=-Cons-=-__
 * __Novelty:__ The proposed change is quite small. The value of spherical representations has been observed quite a few times, especially in contrastive learning, so inserting it into a few more layers isn't really making a contribution that will likely be broadly useful.
 *__Lack of Theoretical Analysis:__ If this paper would be very strong had a bit of relevant theoretical analysis. For example analyzing the distribution of an optimal representation for some very simple problem setting, perhaps even showing theoretically that its not uniform. Analysis of the form of "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere" (Wang & Isola 2020) would be great.
 *__Experimental Results/ Competitor Approaches/ Missing Citations:__ The proposed method is still outshone by other deep AD methods. In particular [outlier exposure](https://openreview.net/forum?id=HyxCxhRcY7) (OE) methods work very well as well as transfer-learning based methods, and [combining the two](https://proceedings.mlr.press/v139/deecke21a.html) achieves 99%+ AUC on your CIFAR10 experiments. This is somewhat counter to the assertion that "the assumption...that learned image features from general datasets such as ImageNet will transfer well in all cases may be restrictive at best." I think the non-OE non-transfer learning based methods are interesting in their own right, but I think these other approaches/results should at least be mentioned.
 *__Some parts could use more detail:__ The paper should really include a precise description of SimSiam and SimCLR since they are so integral to the contribution you are making. BYOL should also be introduced (also the full name of BYOL should be mentioned somewhere instead of going directly to the acronym). This should be in the appendix, at the very least. "Mahalanobis space" isn't particularly standard or common nomenclature and should be described precisely somewhere (in the appendix if necessary).
* __Quite a few style and grammar errors:__ At end.

__-=-Verdict-=-:__ As is the paper is just a bit worse than what I would expect out of ICLR, however there is a lot of easy room for improvement.


__-=-Other Misc. Errors-=-__
* __Bibliography:__ There are numerous errors in here and the author should go through all the cited works carefully. Below are some examples
  * There are several citations that reference arXiv version despite the existence of peer-reviewed and/or archival versions. I only checked two arXiv works but they bot had peer reviewed archival versions (Ruff 2020 has a [workshop version](https://github.com/lukasruff/Classification-AD), Tack 2020 has a [NeurIPS paper](https://proceedings.neurips.cc/paper/2020/hash/8965f76632d7672e7d3cf29c87ecaa0c-Abstract.html) ).
  * Sehwag 2021 is missing a venue entirely.
  * Tack 2020 "Csi" should all be capitalized, you can do this with {CSI}
  * Schölkopf 1999, I'm not sure what is standard procedure now, but you may want to consider using "NeurIPS" since there [is some controversy surrounding its old name)[https://www.nature.com/articles/d41586-018-07476-w].
* __Capitalization:__ There are quite a few random and incorrect capitalized words. Examples
  * p.1 "Deep Neural Networks" should be "Deep neural networks"
  * p.1 "Industrial" should be "industrial"
  * p.1 "Self-Supervised Learning" should be "Self-supervised learning"
*__Figures in weird locations__: Several of the figures referred to in the main text are in the appendix. The main text should really be self-contained, so you should at least give a summary of the figure in the main text and treat the figures in the appendix as additional information.
* __Some figure keys are too small:__ Figure 2 right is the worse offender of this
* __Figure 2 caption:__ y axis label overlaps the graph, should contain a brief description of kappa and MMD 
* __Small errosr:__
  * p. 7: bottom missing space in "Section2.3"
  * p. 6: Not sure what is meant by "high" and "low" level scores
  * " are two random training and test samples.." should be "are a training and test sample respectively"
  * MMD should be introduced with full term (Maximum Mean Discrepancy) and briefly described
  * "use the tool of the von Mises-Fisher" shouldn't use the word "tool" in my opinion, its just a distribution.
  * Figure 1: should use \left< foo \right> to get correct angle brackets
  * (1): replace comma with period
summary_of_the_review: Paper is borderline, could be improved significantly by improving the exposition. I've included several suggestions for this.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studied the issues of recent self-supervised contrastive learning based anomaly detection and presented few modification to resolve those issues. Specifically, the paper pointed out the issue of contrastive learning for OOD detection when trained longer as their embedding distributions become more uniform, and proposed an architectural modification by adding a L2 normalization layer after encoder. While SimSiam, a non-contrastive self-supervised learning algorithm, suffers less from uniformity issue, authors found adding L2 normalization layer to SimSiam effective. Authors also pointed out some gradient flow issue of SimSiam and proposed adding another L2 normalization layer at the end of projection layer. Evaluation on anomaly detection verify the importance of the proposed L2 normalization layers, but the per
main_review: - Strength
  - The paper made a good presentation with motivating empirical results to justify their proposed method.
  - The proposed method is rather simple, but shown to be effective. 

- Weakness
  - Ablation study on the proposed Mahalanobis cosine similarity is missing. I wonder they are really the necessary component that improves upon existing one class classifiers proposed by [Lee et al., 2018b](https://proceedings.neurips.cc/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf) or those studied in [Sohn et al., 2021](https://openreview.net/forum?id=HCSgyPUfeDj).
  - [Sohn et al., 2021](https://openreview.net/forum?id=HCSgyPUfeDj) suggested to use L2 normalized encoder output for one-class classification. For models without normalization, I wonder whether authors use L2 normalized encoder output or unnormalized ones.
  - The motivation for adding L2 normalization layer at the end of the projection layer of SimSiam in Section 2.4 needs clarification. It is unclear what it means by "all the gradients of that operation gets blocked by the stop-grad, though it is an integral ...". Also, there seems no specific ablation study provided (by default, I assume "w/ norm" model is with L2 normalization both at the end of encoder and projection layer and "w/o norm" with no L2 normalization layer at all).
  - While mentioned multiple times, there is no empirical result using BYOL, so it is unclear their behavior will be also similar to SimSiam. 
  - I would be a bit careful when saying "Feature ensembling does not require a validation set". It is rather heuristic and has been shown (e.g., [Defard et al., 2021](https://arxiv.org/pdf/2011.08785.pdf) or [Rippel et al., 2021](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9412109)) specifically for anomaly detection that one should be careful to select correct layers to aggregate.
  - Feature ensembling could be effective in practice, but it makes comparison to other methods somewhat convoluted, especially when readers want to understand the source of improvement. It would be beneficial to have results using only encoder output in Table 1 or Table B.

- Misc
  - For Figure 3 ~ 8, legends are not properly cited.
  - Mahalinobis -> Mahalanobis.

summary_of_the_review: - The technical contribution of the paper is very simple (adding few L2 normalization layers). While showing some good empirical evidences, the overall contribution of the paper is marginal and does not extend state-of-the-art. Also, while proposing many small techniques, they are not fully verified in experiments (please see my comment above).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors study anomaly detection under fully unsupervised setting. This work explores novel architectural modifications on SimSiam (Chen & He, 2020) to the self-supervised feature learning step, that facilitates compact in distribution (ID) distributions to be learned on out-of-distribution (OOD) detection. It performs robustness to polluted ID. In addition, the paper investigates how the geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier.
main_review: Strengths:

(1) This work looks solid, and the experiments in this paper are sufficient, which consider almost all key parameters or components of the proposed method and demonstrate the effectiveness of the method. Besides, the method achieves state-of-the-art performance on OOD task.

(2) The paper spots the reliability issue on long term training of the contrastive Self-Supervised Learning (SSL) methods as deep feature extractors on OOD. The authors claim that no-uniformity and compactness of the learned ID distribution is the main reason to improve the performance on OOD.





Weaknesses:

(1) The proposed method seems simple, the proposed NSA is built on a modified (l2-norm) SimSiam architecture.

(2) The title states that authors targets on anomaly detection, but this paper works on out-of-distribution detection. They are not the same problem: the ID for anomaly detection is one-class; and that for out-of-distribution detection is multi-class.

(3) This manuscript is not very well organized. The organization of tables, figures and text is in a mess. Tables and Figures shall not be placed between text, and the distance from the table to its reference is somehow long. For example, Table 6 should not be place behind the conclusion, which should be the end of the paper. And the paper exceeds the 9-page limit. 





Improvements:

(1) It would be better to combine Figure3 and Figure 7, Figure 4 and Figure8. The organization of figures makes readers hard to follow. 

(2) I wonder the slight difference between with and without normalization showed in Figure 2, especially on SimCLR (the same as Figure 4 and Figure8). In addition, Figure 2(b) miss to show the result of SimCLR w/o norm.

(3) The equation in Figure 1 is l1-norm, but the paper states to employ l2-norm. Figure 1 is the most important figure in this paper. This confuses me a lot.

summary_of_the_review: Overall, the paper presents some good empirical results of OOD detection on benchmarks. This paper is well motivated and solid, but the writing needs to be improved. However, it is somehow weak in terms of technical novelty. Therefore, I vote this paper marginally below the acceptance threshold. 
Given that I am not very familiar with OOD, I would like to check other reviewers' opinions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper contributes to the field of self-supervised anomaly detection by i) verifying the correlation between AD performance and in-distirbution (ID) representation uniformity and how to improve it with a simple normalization; ii) introducing a simple solution to a gradient-flow issue in previous work; and iii) proposing a multi-level feature ensembling as a cheaper but quantitatively effective alternative to test-time data augmentation and model ensambling. They build on SimSiam and SimCLR to support all of their claims.
main_review: + (+) The paper introduces several insightful contributions: e.g. AD correlation with distribution uniformity measured by fitting a vMF distribution or by MMD; or solution to the gradient-flow issue in SimSiam.
+ (+) The proposed improvements to SSL anomaly detection do not need to make assumptions on the data invariances
+ (+) The proposed multi-scale feature ensamble is an effective way to exploit cues at different semantics/resolutions without increasing inference time
+ (+) The proposed improvements are claimed to hold even under mild data pollution

MAJOR

+ (-) The paper is not well organized: starting from Sec. 2.2 going forward the reader is continuously referred to the appendix to see plots, tables and details that the reviewer finds essential to understand, appreciate and validate the authors' claims.

MINOR

+ typos: ood evaluation -> OOD evaluation (Sec 1, page 1); to actually to decrease -> to actually decrease (Sec. 2.3 page 5); 
+ Table B is references but it does not exists in the paper nor in the appendix.
summary_of_the_review: On one hand the paper has good contributions, provides insights into the problem that are supported by empirical evidence; on the other hand is poorly organized and written: most of the key results are supported in the appendix, which should be optional - not critical - to the reading of the paper.
review_title: null

Meta-review: The paper investigates how the geometrical compactness of in-distribution examples affects OOD detection performance and proposes architectural modifications to enable compact in-distribution embeddings. All the reviewers agreed that the paper has several interesting contributions. I agree with the authors that simplicity is a strength, not a weakness. 

My main concern is that the paper's contributions feel a bit scattered. For instance, the paper does a detailed evaluation of normalization and compactness, but makes a few other minor contributions (as detailed by 
the authors at https://openreview.net/forum?id=7VH_ZMpwZXa&noteId=m-1y5byLbwS​). However, the latter contributions feel a bit narrow to specific methods and are not as comprehensively tested as the claims around normalization.

Overall, the reviewers and I think that the current version falls below the acceptance threshold. I encourage the authors to revise the draft and resubmit to a different venue.
Decision: Reject