Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised idea for unsupervised anomaly detection. Specifically, this framework enables high-performance AD without any labels via SRR, which is an ensemble approach to propose candidate anomaly samples that are refined from training. This way allows more robust fitting of the anomaly decision boundaries and also better learning of data representations.  Multiple examples are used to demonstrate the proposed methods on effectiveness and robustness. 
main_review: **Strength**
1. the authors fully explore the power of unsupervised AD and shows outperformed results by using the proposed SRR scheme based on the GOAD framework. 
2. The performance is improved by leveraging the ensemble of the OCC, which works well but may result in additional computational cost. 
3. Enough details, such as the sensitivity of hyperparameters, are provided to reproduce the experiments on multiple tasks, including tabular and image datasets. 

**Weakness**
1. The idea sounds promising but may not be the first work. The performance is enhanced by an ensemble of multiple tricks. It would be helpful to see a detailed ablation study, which may show more insights.
2. The baseline is mainly consisting of GOAD, OC-SVM, etc. Some recent SOTA baselines are missed, for example, NeuTral [1]
[1] Qiu, Chen, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. "Neural Transformation Learning for Deep Anomaly Detection Beyond Images." ICML 2021. 
3. The baseline varies case by case, for example, Cutpaste is used for MVTec datasets, have you considered the SOTA performance, for example in this link - https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad  
Cupaste only ranks the 9th, have you compared with the other SOTA baselines?  I, therefore, have a concern, how do you choose the baseline method in different datasets? I suggest to choose more rather than a specific one. 


summary_of_the_review: Overall, the paper is well-written and the results and performance look solid. My major concern is the novelty, specifically compared with the GOAD method. In addition, the code is not uploaded, so I am not sure how it works for reproducibility and time cost.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an ensemble approach, called SRR (Self-supervise, Refine, Repeat), for robust unsupervised anomaly detection. The proposed approach trains an ensemble of K detectors together with a joint self-supervised feature extractor g on K disjoint subsets of the data. The ensemble is then used to filter the training data, keeping only the data points that are collectively deemed normal by the K detectors. This training-data filtering process is repeated until the self-supervised feature extractor has converged. A final detector is then trained using the refined data and converged self-supervised feature extractor. Experiments on tabular and image datasets are presented which show that the proposed ensemble approach is more robust at high anomaly contamination ratios than respective state-of-the-art single detectors.
main_review: *Pros*
+ The experimental results demonstrate significant anomaly detection performance improvements for the proposed SRR approach, especially at high anomaly ratios.
+ The paper is overall presented and written well, as well as technically sound.
+ The paper is well placed well into the existing literature, up to including recent works.

*Cons*
- The methodological novelty of the proposed SRR approach is rather low (ensemble learning is standard to improve robustness and the individual detection method from Sohn et al. (2020) is not new)
- The experiments do not contain a comparison to any specifically robust AD approach (e.g. robust PCA for the tabular and robust autoencoders for the image dataset). 
- While the paper includes ablation studies on key components and hyperparameters (ensemble size, data rejection confidence, updating the self-supervised feature extractor), I think there should also be a comparison to just using the final ensemble on the converged self-supervised extractor vs. training an additional final model. Is there much of a difference left?


Some additional minor points:

- I don't agree with the framing that most prior AD works "all depend on some labeled data" as expressed in the abstract and introduction. The bulk of AD research is on unsupervised methods (see Chandola et al. (2009) and the recent reviews by Ruff et al. (2021) and Pang et al. (2021)). However, I agree that most methods assume fairly clean training data. This view should be updated in my mind.
- The GDE abbreviation is used before it is explained.
- p.6: "For the N setting [...]" Typo?
summary_of_the_review: Though I think the methodological novelty of the proposed approach is rather low, and that the experimental comparison should be somewhat extended (where I expect the improvements of SRR to hold up), I am overall positive towards accepting this work since robust anomaly detection is a relevant problem of high practical significance, for which the proposed SRR approach demonstrates significant improvements over current state-of-the-art methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper tackles an unsupervised anomaly detection problem where the training set contains an unknown portion of anomalies.
When anomalies are contained in the training set, it is known that classical AD approaches' performance degrades.
The idea is to filter out potential anomaly samples (data refinement) by ensemble model.
Each model in the ensemble is trained on a disjoint set of training data and then used as a classifier to determine potential anomalies.
Then the data refinement process uses a hard assignment excluding anomalies from the training set.
Refinement and ensemble training is repeated iteratively until convergence.
The proposed framework is validated on the four tabular datasets and four image datasets.
main_review: [Stregnth]
The effectiveness of the proposed framework is validated on top of contrastive learning-based models, which are state-of-the-arts.
Extensive experiments on both tabular and image datasets support the effectiveness of the framework.
Ablation studies decouple the effects of each hyperparameter. Also, the representation update study shows that re-training representation with the refined dataset is important.

[Weakness]
- Although the proposed framework is tested on contrastive models, the idea of data refinement itself is independent of these models.
The framework can be applied to other types of anomaly detectors but it is not shown.

- The iterative refinement has been studied [1,2,3] previously, however, no comparison or discussion is addressed with those approaches.
Although they use AE-based models, the idea of iterative refinement can be deployed on top of contrastive models as well. What makes SRR more effective than these methods? What is the main factor that makes SRR more competitive than these methods?

[1] Xia, Yan, et al. "Learning discriminative reconstructions for unsupervised outlier removal." Proceedings of the IEEE International Conference on Computer Vision. 2015.

[2] Beggel, Laura, Michael Pfeiffer, and Bernd Bischl. "Robust anomaly detection in images using adversarial autoencoders." arXiv preprint arXiv:1901.06355 (2019).

[3] Pang, Guansong, et al. "Self-trained deep ordinal regression for end-to-end video anomaly detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

- Hyperparameter selection requires tuning. Although it is shown in Figure 7, that any value of gamma improves over baseline, it is still important how to choose this value. The paper suggested Otsu's method as a solution to predict the approximate anomaly ratio of the dataset but it is not shown in the main experiments. (MVTec experiment is provided in A.5) In the main experiments, hyperparameter gamma is tuned with two times of anomaly ratio which this setting requires prior knowledge on anomaly ratio. I wonder how effective Otsu's method is in other scenarios.

- The training requires heavy computation as the framework requires ensemble learning on top of contrastive learning. 

[Questions]
- What is the convergence condition? Is it necessary to train with the framework until the data refinement gives marginal change to the data? How long does it take?

- What is the important difference of SRR from the previous refinement methods? Is SRR more effective than the previous refinement approaches?

- How does SRR perform when trained with Otsu's method rather than using true anomaly ratio?


[Post rebuttal]
The authors addressed all my concerns.
I raise the score, assuming the rebuttal materials will be included in the revised version.
Rebuttal materials here means,
- OTsu's method experiment
- detailed discussion on the difference between SRR and the previous iterative works
- Convergence analysis (maybe this one in the appendix)
summary_of_the_review: The paper proposes a framework to refine data and train contrastive models for unsupervised anomaly detection problems.
The extensive experiments show the effectiveness of the method.
However, the main experiments require prior knowledge of the true anomaly ratio which is unavailable in real-world problems.
Discussions about the important difference between the proposed method and the previous iterative methods would make the paper more convincing.

[Post rebuttal]
The authors addressed all my concerns.
I raise the score, assuming the rebuttal materials will be included in the revised version.
Rebuttal materials here means,
- OTsu's method experiment
- detailed discussion on the difference between SRR and the previous iterative works
- Convergence analysis (maybe this one in the appendix)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a data refinement approach combined with self-supervised representation to robust one-class classification, which is commonly used in the anomaly detection scenario. The proposed data refinement approach is designed based on an ensemble of one-class classifiers. 
main_review: The authors propose a novel AD framework to enable inspect defects with one-class, which is called SRR and applicable on unlabeled datasets.SRR employs an ensemble of multiple OCCs to give the potential anomaly refined samples from training. SRR brings the advantages of making the anomaly decision boundaries more robust and giving better data representations. The proof and experiment results are well organized. 
summary_of_the_review: The paper is ready for acceptance.
review_title: null

Meta-review: The paper worked on fully unsupervised anomaly detection and proposed to use self-supervised representation learning to improve the performance of one-class classification. This is a borderline case close to acceptance but cannot make it. Specifically, it is useful, but its novelty is the main issue, since it is not surprising that self-supervised representation learning can improve one-class classification without representation learning (this part is still much of the taste of ICLR) and an ensemble of multiple models can improve upon a single model (which is just "bootstrap aggregating" or "bagging" used everyday in practice and known to machine learning and statistics societies a very long time ago). After seeing the rebuttal, the concerns were not really addressed well and the issues were only partially solved. Thus, the paper is not enough to guarantee an acceptance to ICLR unfortunately.
Decision: Reject