Self-Supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection
Authors: Jinsung Yoon
Kihyuk Sohn
Chun-Liang Li
Sercan O Arik
Chen-Yu Lee
Tomas Pfister
Venue: ICLR 2022 Submitted (2022)
Keywords: Anomaly detection
Data refinement
Iterative training

Abstract: Anomaly detection (AD) - separating anomalies from normal data - has many applications across domains, from manufacturing to healthcare. While most previous works have been shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-supervised representations using a data refinement process. Our proposed data refinement approach is based on an ensemble of one-class classifiers (OCCs), each of which is trained on a disjoint subset of training data. Representations learned by self-supervised learning on the refined data are iteratively updated as the refinement improves. We demonstrate our method on various unsupervised AD tasks with image and tabular data. With a 10% anomaly ratio on CIFAR-10 image data / 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the state-of-the-art one-class classification method by 6.3 AUC and 12.5 average precision / 22.9 F1-score.

Under review as a conference paper at ICLR 2022
SELF-SUPERVISE, REFINE, REPEAT: IMPROVING UN- SUPERVISED ANOMALY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Anomaly detection (AD) separating anomalies from normal data - has many ap- plications across domains, from manufacturing to healthcare. While most previous works have been shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-supervised representations using a data refinement process. Our proposed data refmement approach is based on an ensemble of one- class classifiers (OCCs), each of which is trained on a disjoint subset training data. Representations learned by self-supervise learning on the refined data are iteratively updated as the refinement improves. We demonstrate our method on various unsupervised AD tasks with image and tabular data. With 10% anomaly ratio on CIFAR- 10 image data / 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the state-of-the-ar one-class classification method by 6.3 AUC and 12.5 average precision / 22.9 F1-score.
INTRODUCTION
Anomaly detection (AD), the task of distinguishing anomalies from normal data, plays a crucial role in many real-wer salld applications such as detecting faulty products using visual sensors manufacturing. fraudulent behaviors at credit card transactions, or adversarial outcomes at intensive care units.
AD has been considered under various settings based on the availability of negative (normal) and positive (anomalous) data and their labels at training, as overviewed in Sec. 2. Each application scenario is dominated by different challenges. When entire positive and negative data are available along with their labels (Fig. la), the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in label distributions [4, 16, 20, 27, 32, 35J. When only negative labeled data are available (Fig. lb), the problem is "one-class classification' [23, 26, 33, 45, 47, 51, 53]. Various works have also extended approaches designed for these to settings with additional unlabeled data (Fig. Ic,d,e) [10, 18, 24, 46, 58) in semi-supervised setting. While there exist many prior works in these settings, they all depend on some labeled data, which is not desirable all application scenarios.
Unsupervised AD, on the other hand, poses unique challenges in the absence of any labeled data information, and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal. For example, some recent studies [7, 60] have applied one-class classifiers (OCCs) that are known to yield impressive performance when trained on negative samples [7, 23, 26, 33, 51] to unsupervised AD, but their performance for unsupervised AD has been quite sub-optimal. cate-of-the--a Deep OCCs [51] with different anomaly ratios in unlabeled training data the average precision significantly drops even when a small portion (2%) of training data is contaminated with anomalies.
Fig. 2 illustrates this, showing the unsupervised AD performance
Our framework SRR (Self-supervise, Refine, Repeat), overviewed in Fig. 3, brings novel approach to unsupervised AD with the principles of sulf-super learning without labels and iterative data refinement based on the agreement of OCC outputs. We propose to improve the of-te-of-the-rrt performance of OCCs, e.g. [33, 51], by refining the unlabeled training data so as to address the fundamental challenges elaborated above. SRR iteratively trains deep representations using refined

---

Under review as a conference paper at ICLR 2022
Negative (N) sample,
Positive
sample,
Unlabeled (U) sample
(a) P+N
(b) N
(c) P+N+U
(d) P+U
(e) N
(1) U
Figure 1: AD problem settings. Blue and red dots are for labeled negative (normal) and positíve (anomalous) samples, respectively. Grey dots denote unlabeled samples. While previous works mostly focus supervised (a. b) or semi-supervised (c, d, e) settings, we tackle AD problem using only unlabeled data (f) that may contain both negative and positive samples.
data while improving the refinement of unlabeled data by excluding samples. For the data refinement process, we employ an ensemble of OCCs, each of which is trained on disjoint subset of unlabeled training data. The samples are declared as normal if there is consensus between all the OCCs. The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting. Most prior unsupervised AD works [23, 26, 33, 51] assume that the data contains entirely negative samples, which makes them not truly unsupervised as they require having humans to do the data filtering. Similar to ours, some prior unsupervised AD works [7, 45, 60] considered evaluating on an unsupervised setting where there exist small percentage of anomalous samples in the training data, i.e. operating in 'truly' unsupervised setting without having the need for humans do any filtering in the training data. However, these methods often suffered from significant performance degradation as the ratio of anomalous sample ratio has increased (see Sec. 4.2). We would like highlight that our method distinguishes from the literature by bringing a data-centric approach (refining the unlabeled data) to unsupervised anomaly detection beyond the model-centric approaches (improving the model itself). Our framework SRR aims to provide robustness in performance as the anomalous sample ratio increases, as shown in Fig. 2.
ceeenniilll-possttiie (anomalous)
We conduct extensive experiments across various datasets from different domains, including semantic AD (CIFAR-IK [29]. Dog-vs-Cat [19]), real-world manufacturing visual AD use (MVTec [8]), and tabular AD benchmarks. We consider methods with both shallow [36, 47] and deep [7, 33, 51] models.
Average Precision
We evaluate models at different anomaly ratios of unlabeled training data and show that SRR signifi-
cantly boosts performance. For example, in Fig. 2,
Baseline (Contrastive)
SRR improves more than 15.0 average precision (AP)
Proposed model (SRR)
with 10% anomaly ratio compared to a state-of-the art one-class contrastive representation model [51] on CIFAR-10. Similarly, on MV'Tec SRR retains a strong performance, dropping less than 1.0 AUC with 10% anomaly ratio, while the best existing OCC [33] drops more than 6.0 AUC. We further investigate the efficacy of design choices, such the number of ensemble classifiers, thresholds, and refinement of deep representations via ablation studies.
Anomaly ratio (%) in training data
Figure 2: Performance of our proposed model and a baseline OCC using contrastive repre- sentation [51] on CIFAR-10 with different anomaly ratios in the training data.
2 RELATED WORK
There are various existing works under the different settings described in Fig. 1:
The positive + negative setting is often considered supervised binary classification problem. The challenge arises due to the imbalance in label distributions as positive (anomalous) samples are rare. As summarized in [12], to address this, wtr/nnndr--spplpin [16, 20], weighted optimization [4, 27], synthesizing data of minority classes [32, 35], and hybrid methods [2. 22] have been studied.
The negative setting is often converted to one-class classification problem, with the goal of finding a decision boundary that includes as many one-class samples as possible. Shallow models for this setting include one-class support vector machines [47] (OC-SVM), support vector data
2

---

Under review as a conference paper at ICLR 2022
description [53] (SVDD), kernel density estimation (KDE) [31], and Gaussian density estimation (GDE) [44]. There are also auto-encoder based models [59] that treat the reconstruction error as the anomaly score. Deep learning based OCCs have been developed, such as Deep OCC [45], geometric transformation [23], or outlier exposure [26]. Noting the degeneracy or inconsistency of learning objectives of existing end-to-end trainable Deep OCCs, [51] proposed deep representation OCC, two-stage framework that learns self-supervised representations [17, 28] followed by shallow OCCs. That work was extended for texture anomaly localization with CutPaste [33]. The robustness to very low anomaly ratios of these methods under the unsupervised setting was explored in [7, 60].
The superviie setting defined as utilizing a small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data. Depending on which labeled samples are given, this setting can split into three sub-categories. When only some positive/negative labeled samples are provided, we denote that as PU/NU setting. Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given [1,41, 52]. The PNU setting is a more general semi-supervised setting where subsets of both positive and negative labeled samples are given. Deep SAD [46] and SU-IDS [39] are included in this category. We show the significant outperformance of our proposed SRR framework compared to Deep SAD, for multiple benchmark datasets when not using any labeled data (see Sec. 4.2).
The unlabeled setting has received relatively less attention despite its significance in automating machine learning. The popular methods for this setting include isolation forest [36] and local outlier factor [13]. However, they are difficult scale, and less compatible with recent advances in representation learning. While OCCs, such as OC-SVM, SVDD, or their deep counterparts, apply to unlabeled settings by assuming the data all negative, and the robustness of those methods has also been demonstrated in part [7, 60]. in practice we observe a significant performance drop with a high anomaly ratio, shown in Fig. 2. In contrast, our proposed framework is able to maintain high performance across anomaly ratios.
Data refinement has been applied to AD in some prior works. [5, 42] generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent AD performances in unsupervised settings. [6, 30, 40, 55, 59] used the reconstruction errors of the auto-encoder as an indicator for removing possible anomalies. [21] and [38] used data refinement for AD in supervised and semi-supervised settings. Additional discussions can be found in Appendix A.9.
Self- Self-training [37, 48] is an iterative training mechanism using predicted pseudo labels as targets for model training. It has regained popularity recently with its successful results in semi-supervised image classification [9, 50, 57]. To improve the quality of pseudo labels, employment of an ensemble of classifiers has also been studied. [14] trains an ensemble of classifiers with different classification methods to make consensus for noisy label verification. Co-training [11] trains multiple classifiers, each of which is trained on the distinct views, to supervise other classifiers. Co-teaching [25] and DivideMix [34] share a similar idea in that they both train multiple deep networks on separate data batches to learn different decision boundaries, thus becoming useful for noisy label verification. While sharing a similarity, the proposed framework has clear differences from the previous works. SRR performs iterative training with data refinement (with robust ensemble methods) and self supervuee learning for unsupervised learning of an anomaly detector.
3 PROPOSED FRAMEWORK
Self-supervise, Refine, and Repeat (SRR) is an iterative training framework, where we refine the data (Sec. 3.1) and update the representation with the refined data (Sec. 3.2), followed by OCCs on refined representations. Fig. overviews the framework and Algorithm 1 provides the pseudo code.
Notation. We denote the training data as D = (xx)) where x; E X and N is the number of training samples. Bi E {0, 1} is the corresponding label to x,, where denotes normal (negative) and 1 denotes anomaly (positive). Note that labels are not provided in the unsupervised setting.
Let us denote a feature extractor as g X + Z. y may include any data preprocessing functions, an identity function (if raw data is directly used for one-class classification). and learned or learnable representation extractors such as deep neural networks. Let us define an OCC as f Z + [-oo, oo] that outputs anomaly scores given the input features (x). The higher the score f(((x)), the more anomalous the sample is. The binary anomaly prediction is made after thresholding: 1 (f(((x)) 2 wy).
3

---

Under review as a conference paper at ICLR 2022
Data refinement (Ensemble one-class classifier)
Train
Inference
Subset
Ona-class classiffer 1
Intersect
Train
Representation learner
Uniabeled data (D)
One-class dassifier
One-class classifter 2
Subeet 2
One-class classifier K
Subset K
Figure 3: Block diagram of SRR composed of representation learner (Sec. 3.2), data refinement (Sec. 3.1), and final OCC blocks. The representation learner updates the deep models using refined data from the data refinement block. Data refinement is done by an ensemble OCCs, each of which is trained on K disjoint subsets of unlabeled training data. Samples predicted as normal by all classifiers are retained in the refined data, and are used to update the representation learner and final OCC. The process is repeated iteratively until convergence. Convergence graphs can be found in the Appendix A.8.
3.1 DATA REFINEMENT
naive way to generate pseudo labels of unlabeled data to construct an OCC on raw data or learned representations in [51] and threshold the anomaly score to obtain binary label for normal vs. anomalous. As we update the model with refined data that excludes samples that are predicted to be anomalous, it important to generate pseudo labels of training data as accurately possible.
To this end, instead of training single classifier (unlike most previous works on data refinement for AD [5. 42, 59]). we train an ensemble K OCCs and aggregate their predictions to generate pseudo labels. We illustrate the data refinement block in Fig. 3 and as REFINEDATA in Algorithm 1. Specifically, we randomly divide the unlabeled training data DD into disjoint subsets Dı,..., DK. and train K different OCCs (fi,..., fK) on corresponding subsets (D1, Dx). Then, we estimate .i.....,a
a binary pseudo-label of the data x; E 'D as follows:
K
0. =
II
(F)((()))) mk)
(1)
k=1
1
Σ (fx(o(x))) N
The max s.t.
where 1(-) is the indicator function that outputs 1/0 if the input is True/False. (g(x))) represents an anomaly score of x; for an OCC ft. in equation 2 is threshold determined as percentile of the anomaly score distribution (x))(
To interpret equation 1, x; is predicted as normal, i.c. 9. =0 if all K OCCs predict it normal. While this may too strict and potentially reject many true normal samples the training set, find that empirically, it is critical to be able to exclude true anomalous samples from the training set. The effectiveness of the employment of ensemble of classifiers is empirically shown in Sec. 4.3.
3.2 REPRESENTATION UPDATE
SRR follows the idea of deep representation OCCs [51], where in the first stage a deep neural network trained with self-supervised learning (such as rotation prediction [23], contrastive [51], or CutPaste [33]) to obtain mcaningful representations of the data, and in the second stage OCCs are trained on these learned representations. Such a two-stage framework shown to beneficial as prevents the 'hypersphere collapse' of the deep OCCs by the favorable inductive bias it brings with the architectural constraints [45].
Here, propose to conduct self-supervised representation learning jointly with data refinement.
More precisely, train feature extractor g using D (x: = 00. subset of unlabeled data
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 SRR: Self-supervise, Refine, Repeat.
Input: Training data D {x: 1 Ensemble count (K), threshold (γ)
Output: Refined data (D), trained OCC (f), feature extractor (g)
1: function REFINEDATA(D, 7)
2: Train OCC models {fx [= on {D*} K disjoint subsets the training data D.
3: Compute thresholds 7ı is for percentile of anomaly distributions (equation 2).
4: Predict binary labels y5 (equation 1).
5: Return D = (x: : 91 0, x; ED)
6: end function
7:
function SRR(D, 7)
8: Initialize the feature extractor g.
9: while not converged do
10:
D = REFINEDATA(D,, g. K, 7).
11:
Update using D with self-supervised learning objectives. end while
12:
13:
D = REFINEDATA(D, K)
14:
Train an OCC model (f) on refined data (D).
end function
D that only includes samples whose predicted labels with an ensemble OCC from Sec. 3.1 are negative. We also update DD as we proceed with representation learning. The proposed method illustrated in Algorithm as SRR. In contrast to previous works [33, 51] that use the entire training data for learning self-supervised representation, we find it necessary to refine the training data even for learning deep representations. Without representation refinement, the performance improvements of SRR are limited, as shown Sec. 4.3.2. Last, for test-time prediction, we train an OCC on refined data D using updated representations by g as in line 13-14 in Algorithm 1.
3.3 UNSUPERVISED MODEL SELECTION
As SRR is designed for unsupervised AD, labeled validation data for hyperparameter tuning is typically not available and the framework should enable robust model selection without any reliance on labeled data. Here, we provide insights how select important hyperparameters, and later in Sec. 4 perform sensitivity analyses for these hyperparameters.
Data refinement of SRR introduces hyperparameterss: the number of OCCs (K) and the percentile threshold (7). There is trade-off between the number of classifiers for the ensemble and the size of disjoint subsets for training each classifier. With large K, we aggregate prediction from many classifiers, each of which may contain randomness from training. This comes at a cost of reduced performance per classifier as we use smaller subsets train them. In practice, we find K works well across different datasets and anomaly ratios. y controls the purity and coverage of refined data. If y is large, and thus classifiers reject too many samples, the refined data could be more pure and contain mostly the normal samples; however, the coverage of the normal samples would be limited. On the other hand, with a small 7, the refined data may still contain many anomalies and the performance improvement with SRR would be limited. We empirically observe that SRR is robust to the selection of y when is chosen from a reasonable range. In our empirical experiments, we find 2x of the true anomaly ratio be reasonable choice. In other words, it is safer to higher than the expected true anomaly ratio. some cases, the true anomaly ratio may not be available at all; for such scenarios, we propose Otsu's 's method [49] estimate the anomaly ratio of the training data for determining the threshold (experimental results are in the Appendix A.5).
EXPERIMENTS
We evaluate the efficacy of proposed framework for unsupervised AD tasks tabular (Sec. 4.1) and image (Sec. 4.2) data types. We experiment varying ratios of anomaly samples in unlabeled training data and with different combinations of representation learning and OCCs. In Sec. 4.3,
5

---

Under review as a conference paper at ICLR 2022
provide performance analyses to better explain major constituents of the performance, as well as sensitivity to hyperparameter values.
Implementation details: To reduce the computational complexity of the data refinement block, utilize a simple OCC such as GDE in the data refinement block. In a two-stage model, we only update the data refinement block 1st, 2nd, 5th, 10th, 20th, 50th, 100th, 500th epochs instead of every epoch. After 500 epochs, we update the data refinement block per each 500th epoch. Each run of experiments requires a single V100 GPU. Additional discussions can be found in Appendix A.10.
4.1 EXPERIMENTS ON TABULAR DATA
Datasets. Following [7, 60), we test the efficacy of SRR on variety of tabular datasets, including KDDCup, Thyroid, or Arrhythmia from the UCI repository [3]. We also use KDDCup-Rew where the labels of KDDCup splits, we use 50% of normal samples for training. In addition, we hold out some anomaly samples (amounting 10% of the normal samples) from the data. This allows to simulate unsupervised settings with an anomaly ratio of up to 10% of entire training set. The rest of the data is used for testing. We conduct experiments using 5 random splits and 5 random seeds, and report the average and standard deviation of 25 Fl-scores (with scale 0-100) for the performance metric.
reversed so that an attack represents anomaly [60]. To construct data
Models. We mainly compare with GOAD [7] (the state-of-the-art AD model in the tabular domain) and implement SRR on top of it. GOAD utilizes random transformation classification as the pretext task of self- superviee learning, and the normality score is determined by whether transformations are accurately included in the transformed space of the normal samples. We re-implement GOAD [7] with few modifications. First, instead of using embeddings to compute the loss, we use parametric classifier, similarly to augmentation prediction [51]. Second, we follow the two-stage framework [51] to construct deep OCCs. For the clean training data setting, our implementation achieves 98.0 for KDD, 95.0 for KDD-Rev. 75.1 for Thyroid, and 54.8 for Arrhythmia Fi-scores, which are comparable to those reported in [7]. Please see Appendix A.3 for formulation and implementation details.
1D0 Rov
Thyrold
Arrtychima
S1R-5CAD
SCA
SAN--GOAD
138
004D
G24D
GOXD
122
OC-OAN (1B cente]]
SC (b/kkmd)
DC-SV icerne))
Anomaly noo (5) trasiing dats
Ansmay tit training
62
Anomaly
0a
so
Anomsly radoo (k) traiiing
Figure 4: Unsupervised AD performance (FI-score) using OC-SVM (with rbf kemne),, GOAD [7], and GOAD with the proposed method SRR on various tabular datasets. Shaded areas represent the standard deviation.
Results. We show results of GOAD (the baseline) and GOAD with SRR in Fig. 4. The ranges of the noise ratio are set to 0% for the anomaly ratios in the original dataset (if anomaly ratios are larger than 10%. we set the maximum anomaly ratio as 10%). For KDD-Rev, we set the maximum anomaly ratio to 2.5% because the performance of GOAD (without SRR) drops significantly even with small ratio of anomalies in the training data. Fig. 4 shows that integrating SRR significantly improves GOAD (the offte--o-theaann methods for tabular OCC). The improvements are more significant especially at higher anomaly ratios. This underlines how SRR can achieve significant improvements for both small (Thyroid & Arrhythmia) and large-scale datasets (KDD& & KDD-Rev).
4.2 EXPERIMENTS ON IMAGE DATA
Datasets. We evaluate SRR on visual AD benchmarks, including CIFAR-10 [29], f-MNIST [56]. Dog-vs-Cas Datt [19], and MVTec [8]. For CIFAR- f-MNIST, and Dog-vs-Cat datasets, samples from
Note that the experimental settings with contaminated training data in GOAD [7] and DAGMM [60] are slightly different from ours. Our contamination ratio is defined as the anomaly ratio over the entire training data, while their contamination ratio is the anomaly ratio over all the anomalies in the dataset.
6

---

Under review as a conference paper at ICLR 2022
one class set to be normal and the rest from other classes set to be anomaly. Similar to the experiments on tabular data in Section. 4.1, we swap a certain amount of the normal training data with anomalies given the target anomaly ratio. For MVTec, since there available for training, we borrow 10% of the anomalies from the test set and swap them with normal samples in the training set. Note that 10% of samples borrowed from the test set are excluded from evaluation. For all datasets, we experiment with varying anomaly ratios from 0% to 10%.
anomalous data
We use area under ROC curve (AUC) and average precision (AP) metrics quantify the performance for visual AD (with scale 0- 0-100). When computing AP, we set the minority class of the test set as label and majority as label (e.g.. normal samples are set as label for CIFAR- 10 experiments as there are more anomaly samples that are from 9 classes in the test set). We run all experiments with 5 random seeds and report the average performance for each dataset across all classes. Per-class AUC and AP are reported in Appendix A.4.
Models. For semantic AD benchmarks, CIFAR-10, 10, f-MNIST, and Dog-vs-Cat, we compare the SRR with two-stage OCCs [51] using various representation learning methods, such as distribution- augmented contrastive learning [51], rotation prediction [23, 28] and its improved version [51], and denoising autoencoder. For MVTec benchmarks, we use CutPaste [33] as baseline and compare to its version with SRR integration. For both experiments, we use the ResNet- 18 architecture, trained from random initialization. using the hyperparameters from [51] and [33]. The same model and hyperparameter configurations are used for SRR with K = 5 classifiers in the ensemble. We set y as twice the anomaly ratio of training data. For 0% anomaly ratio, set ry as 0.5. Finally, a Gaussian Density Estimator (GDE) on learned representations is used as OCC.
90
Average Precision
60
85-
AUC
SRR+Contrastive
SRR+ Conttaasseee
Contrastive
Contrastive
Rotation
40
Rotation
75-
RotNet
RotNet
DAE
30
DAE
2
Anomaly ratio (%) in training data
Anomaly ratio (%) in training data
Figure 5: Unsupervised AD performance with various OCCs on CIFAR-10. For SRR we adapt distrbbution-augmeneed contrastive representation learning [51]. (Left) AUC, (Right) Average Preci- sion (AP).
Results. Fig. shows significant performance drop with increased anomaly ratio, regardless of representation learning. For example, the AUC of distributtin-aagmennee contrastive representa- tion [51] drops from 92.1 83.6 when anomaly ratio becomes 10%. Similarly, the improved rotation prediction representation [28] drops from 90.0 to 84.1. On the other hand, SRR effectively handles the contamination in the training data and achieves 89.9 AUC with 10% anomaly ratio, reducing the performance drop by 74.1% An 'oracle' upper bound would be the removal of all anomalies from the training data (which is the same as the performance at 0% anomaly ratio for the same size of the data). As Fig. 5 shows, the performance of SRR is similar to this oracle upper bound performance (less than 2.5 AUC difference) even with high anomaly ratios (10%). The results are also similar in other metrics, such as AP in Fig. 5 or Recall at Precision of 70 and 90 in Fig. 9 in the Appendix.
We repeat experiments on 3 additional visual AD datasets and report results Fig. 6. We observe consistent and significant improvements over the baseline across different datasets and different one-class classification methods. Note that the improvement is more significant at higher anomaly ratios. For instance, MVTec dataset, SRR improves AUC by 4.9 and AP by 7.1 compared to the state-of-the-aa CutPaste OCC with an anomaly ratio 10%. In the Appendix (Sec. A.4), we also illustrate per-class performance of SRR compared with the state-of-the-art.
7

---

Under review as a conference paper at ICLR 2022
SARsDuePasle CutPaste
SRR+Contrastive Contractive
34-
AUC
82-
SRRtContrsstvve
Contrastive
715
Anomaly rato iN) training data
Anomaly training
Anomaly ratio (8) in training data
01.
SRR+CutPaste
Siif #ilttCCntrastive
CutPaste
Centrastoo Sentrastive
Contrastive
115
Anomaly ratio (%) training data
Anomaly ratio (%) training data
Anomaly ratle (%) in tntiiinnn data
(a) MVTec
(b) F-MNIST
(c) Dog-vs-Cal
Figure 6: Unsupervised AD performance on (a) MVTec (b) f-MNIST, and (c) Dog-vs-Cat datasets with varying anomaly ratios. We use state-of-the- one-class classification models for baselines, such as distribution -abbbuttonnngmeneeed contrastive representations [51] for CutPaste [33] for MVTec, and build SRR on top of them.
f- f-MNIST and Dog-vs Dog-vs-Cat, or
4.3 PERFORMANCE ANALYSES
In this section, we conduct sensitivity analyses on two hyperparameters of SRR, namely, the number K for ensemble OCCs and the percentile threshold y that determines normal and anomalies in the data refinement module. In addition, we show the importance of updating the representation with refined data. Ablation studies are conducted on two visual AD benchmarks, CIFAR-10 and MVTec. More experimental results and discussions can be found in the Appendix.
4.3.1 SENSITIVITY TO HYPERPARAMETERS
SRR is designed for unsupervised AD and it is important to ensure robust performance against changes in the hyperparameters as model selection without labeled data would be very challenging. Fig. presents the sensitivity analyses SRR with respect to various hyperparameters. In Fig. 7a, we observe the performance improvement we increase the number of classifiers for ensemble. This is particularly effective on CIFAR (top in Fig. 7a), where the number of samples in the training data is large enough that even with large K. the number of samples to train each OCC (N/ K) would be sufficient. In Fig. 7b, we observe that SRR performs robustly when 7 is set to be larger than the actual anomaly ratio (10%). When 7 is less than 10%, however, all the way to the baseline (7 = 0). Our results show that SRR improves upon baseline regardless of the threshold. In addition, suggests that y could be to be anywhere from the true anomaly ratio and and 2x the anomaly ratio to maximize its effectiveness. When the true anomaly ratio is unknown, as discussed in Appendix A.5, Otsu's method could be used.
see significant drop in performance,
4.3.2 ITERATIVELY UPDATING REPRESENTATION WITH REFINED DATA
It is possible to decouple representation learning and data refinement of SRR, which would result in a three-stage framework, where we learn representations until convergence without data refinement, followed by the data refinement and learning OCC. Fig. 7c shows that SRR using a pre-trained then fixed representation (i.e. when data refinement is used for OCC only) already improves the performance upon the baseline with no data refinement at any stage of learning representation or classifier. Improvements on CIFAR- 10 are 4.9 in AUC and 8.0 in AP; and on MVTec are 1.3 in AD and in 1.3 in AP. This underlines the effectiveness of our data refinement module. More results on applying SRR on raw tabular features or learned (and fixed) visual representations are provided in Appendix A.2. Moreover, resulting in another improvement in AUC by 1.4, AP by 4.5 on CIFAR- 10, and AUC by 3.2, AP by 4.8 on MVTec, upon SRR using fixed representation.
find that learning representation with refined data plays crucial role,
8

---

Under review as a conference paper at ICLR 2022
160
Proceion
AUC
623
AUC
333
No data retrement
Data retrement only oCC
Data refiremen for epreserittiin OCC
Ensomble count [K]
Threshold (y)
AUC
Precison
dots rifommert
Preciain
Dats rnfimmert for rpprssetation 6occ
Eats rimomartts ony SCc
AUC
909
Ensemble count [K]
Thewshold (y)
AUC
Average Precison
(a) Ensemble count K
(b) Percentile threshold
(c) Representation update
Figure Ablation studies on (top) CIFAR-10 and (bottom) MVTec under 10% anomaly ratio setting with respect to (a) ensemble count K, (b) percentile threshold y, and (c) data refinement with or without representation update.
4.4 QUANTIFYING THE REFINEMENT EFFICACY
We evaluate how many normal and anomalies are excluded by the proposed data refinement block. As in Fig. 8(a, b), with data refinement, we can exclude more than 80% anomalies the training set without removing too many normal samples. For instance, among 4% anomalies in CIFAR-10 -- data, SRR is able to exclude 80% anomalies while removing less than 20% normal samples. Such high recall of anomalies of SRR is not only useful for unsupervised AD, but also could be useful for improving the anmotation efficiency when a budget for active learning is available. Fig. 8(c, d) demonstrates the removed normal and abnormal samples the data refinement module training epochs. It shows that better representation learning (as training epochs increase) consistently improves the efficacy of the data refinement.
Anorialia ma sumples
Anorilbu Eamola samphe
Aeeddyyrte (%1 baaiing data
Atorady thii baning del
(a) CIFAR-10
MVTec
(c) CIFAR-10
(d) MVTec
Figure 8: Percentage of excluded anomalous and normal samples by data refinement (a, b) with different anomaly ratios in training data and (c, d) over training epochs for 10% anomaly ratio.
CONCLUSION
AD has wide range of practical use cases. challenging and costly aspect of building an AD system that anomalies
and not easy detect by humans, making them difficult label. To this end, we propose a novel AD framework to enable high performance AD without any labels called SRR. SRR can be flexibly integrated with any OCC, and applied raw data or trainable representations. SRR employs an ensemble of multiple OCCs to propose candidate anomaly samples that are refined from training, which allows more robust fitting of the anomaly decision boundaries as well as better learning of data representations. We demonstrate the state-of-the-arr AD performance of SRR on multiple tabular and image data.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
SRR has the potential to make significant positive impact in real-world AD applications where detect- ing anomalies is crucial, such for financial crime elimination, cybersecurity advances, or improving manufacturing quality. We note potential risk associated with using SRR: when representations are not sufficiently good, there will a negative cycle of refinement and representation updates/OCC. While rely the existence of good representations, in some applications they may be difficult to obtain (c.g., cybersecurity). This paper focuses on the unsupervised setting and demonstrates strong AD performance, opening new horizons for human-in in-the-loop AD systems that are low cost and robust. We leave these explorations to future work.
REPRODUCIBILITY STATEMENT
The source code of SRR will be published upon acceptance. For reproducibility, we only public tabular and image datasets to evaluate the performances of SRR. Complete descriptions of those datasets be found in Datasets subsections in Sec. 4.1 and 4.2. Detailed experimental settings and hyperparameters can be found Models subsections in Sec. 4.1 and 4.2. The implementation details (including hyperparameters that we used) on GOAD [7] are described in Sec. A.3.
REFERENCES
[1] S. Akcay, A. Atapour-Abarghouei, and T. Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In ACCV, 2018.
[2] R. Alejo, R. M. Valdovinos, V. Gareía, and J. Decheco-Sanchez, hybrid method face class overlap and class imbalance on neural networks and multi-class scenarios. PRL, 34(4))380-388, 2013.
[3] A. Asuncion and D. Newman. UCI machine learning repository, 2007.
[4] S. Barua, M. M. Islam, X. Yao, and K. Murase. Mwmote-majority weighted minority over- sampling technique for imbalanced data set learning. IEEE Trans knowledge and data
engineering, 26(2):405-425, 2012. 1,2
[5] L. Beggel, M. Pfeiffer, and B. Bischl. Robust anomaly detection in images using adversarial autoencoders. Machine Learning and Knowledge Discovery in Databases, 2019. 3, 4, 18, 21
[6] A. Berg, J. Ahlberg, and M. Felsberg. Unsupervised learning of anomaly detection from contaminated image data using simultaneous encoder training. arXiv preprint 190.::1905.10000 2019.
[7] L. Bergman and Y. Hoshen. Classification-bunt based anomaly detection for general data. In International Conference on Learning Representations, 2019. 1, 6, 10, 14, 15, 19
[8] Bergmann, M. Fauser, D. Sattlegger, and Steger. MVTec AD-a a comprehensive real-world dataset for unsupervised anomaly detection. In CVPR, 2019. 2, 6
[9] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: holistic approach to semi-supervised learning. Advances in Neural Information Processing
Systems, 32, 2019. 3
[10] G. Blanchard, G. Lee, and C. Scott. Semi-supervised novelty detection. JMLR, 11:2973-3009, 2010.
[11] A. Blum and T. Mitchell, Combining labeled and unlabeled data with co-training. In CLT, 1998.
3
[12] Branco, L. Torgo, and R. Ribeiro. A survey of predictive modelling under imbalanced distributions. arXiv preprint arXiv:1505.0Ie 2015. 2
[13] M. M. Breunig, H.-P. Kriegel, R. T. Ng. and J. Sander. Lof: identifying density-based local outliers. In International conference on management of data, 2000. 3, 20
[14] C. E. Brodley, M. A. Friedl, et al. Identifying and eliminating mislabeled training instances. In Proc National Conference Artificial Intelligence, 1996. 3
10

---

Under review as a conference paper at ICLR 2022
[15] E. J. Candès, X. Li, Y. Ma, and J. Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011. 20
[16] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002. 1,2
[17] Chen, S. Korblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In ICML., 2020. 3
[18] F. Denis. PAC learning from positive statistical queries. In ICALT, 1998.
[19] J. Elson, J. R. Douceur, J. Howell, and J. Saul. Asirra: captcha that exploits interest-aligned manual image categorization. In Proc Computer and Communications Security, 2007. 2,
[20] A. Estabrooks, T. Jo, and N. Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-36 2004. 1,
[21] J.-C. Feng, F.T. Hong, and W.-S. Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In CVPR, 2021. 3
[22] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera. A review ensembles for the class imbalance problem: bagging. boosting-. and hybrid- approaches. IEEE Trans on Systems, Man, and Cybernetics, 42(4):463 2011.
[23] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 9781-9791, 2018. 1, 3, 4, 7, 13, 14
[24] N. Görnitz, M. Kloft, K. Rieck, and U. Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235-262, 2013.
[25] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. W. Tsang, and M. Sugiyama. Co-teaching: robust training of deep neural networks with extremely noisy labels. In NIPS, 2018. 3
[26] D. Hendrycks. M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure.
In International Conference on Learning Representations, 2018. 1, 2, 3, 14
[27] Hwang, S. Park, and Kim. new weighted approach to imbalanced data classifica- tion problem via support vector machine with quadratic cost function. Expert Systems with Applications, 38(7):8580885)
2011. 1,2
[28] N. Komodakis and S. Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018. 3, 7, 13, 14
[29] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009, 2,
[30] C.-H. Lai, D. Zou, and G. Lerman. Robust subspace recovery layer for unsupervised anomaly detection. In International Conference Learning Representations, 2019. 3
[31] L. J. Latecki, A. Lazarevic, and D. Pokrajac. Outlier detection with kernel density functions. In International Workshop on Machine Learning and Data Mining in Pattern Recognition, 2007.
[32] S. S. Lee. Noisy replication in skewed binary classification. Computational statistics & data analysix, 34(2):165-191, 2000. 1.2
[33] C.-L. Li, K. Sohn, J. Yoon, and T. Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In CVPR, 2021. 1, 2, 3,4, 4,5, 7, 8, 17, 19, 20
[34] J. Li, R. Socher, and S. C. Hoi. Dividemix: Learning with noisy labels semi-supervised learning. In ICLR, 2019. 3
[35] A. Liu, J. Ghosh, and C. E. Martin. Generative oversampling for mining imbalanced datasets.
In DMIN, pages 66-72, 2007. 1,2
[36] T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation forest. In ICDM, 2008. 2,
[37] G. J. McLachlan, Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365369,, 1975.
[38] X. Meng. S. Wang, Z. Liang, D. Yao, J. Zhou, and Y. Zhang. Semi-supervised anomaly detection in dynamic communication networks. Information Sciences, 571:527-542, 2021.
11

---

Under review as a conference paper at ICLR 2022
[39] E. Min, J. Long, Q. Liu, J. Cui, Z. Cai, and J. Ma. Su-ids: A semi- sumervupe and unsupervised framework for network intrusion detection. In International Conference on Cloud Computing and Security, 2018. 3
[40] M. Mohseni, J. Yap. W. Yolland, A. Koochek, and S. Atkins Can self-training identify suspicious ugly duckling lesions? In CVPR, 2021. 3
[41] Münoz-Marí, F. Bovolo, L. Gómez-Chova, Bruzzone, and G. Camp-Vall Valls. Semisupervised one-class support vector machines for classification of remote sensing data. IEEE transactions on geoscience and remote sensing, (188))318-3119)) 2010. 3
[42] G. Pang, C. Yan, C. Shen, A. v. d. Hengel, and X. Bai. Self-trained deep ordinal regression for end-to-end video anomaly detection. In CVPR, 2020. 3. 3,4, 18, 21
[43] J. Ren, J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lakshmi- narayanan. Likelihood ratios for out-of-distiibution detection. arXiv preprint arXiv: 02845, 2019. 21
[44] D. A. Reynolds. Gaussian mixture models. Encyclopedia of biometrics, 741:659 663, 2009.
[45] L. Ruff. R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and
M. Kloft. Deep one-class classification. In ICML,, 201 8. 2, 3,4
[46] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. In ICLR, 2020. 1,3
[47] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, J. C. Platt, al. Support vector method for novelty detection. In NIPS, 1999. 1,2
[48] H. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transac- tions on Information Theory, 11(3):363-371, 1965. 3
[49] M. Sezgin and B. Sankur. Survey over image thresholding techniques and quantitative perfor- mance evaluation. Journal of Electronic imaging, 13(1):146 2004. 5, 18
[50] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020. 3
[51] K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pfister. Learning and evaluating representations for 5, 6, 7, 8, 13, 14, 16, 18, 19, 20, 21
deep one-class classification. In ICLR, 2021.
[52] H. Song, Z. Jiang, A. Men, and B. Yang. A hybrid semi-supervised anomaly detection model for high-dimensional data. Computational intelligence and neuroscience, 2017.
[53] D. M. Tax and R. P. Duin. Support vector data description. Machine learning, 54(1):45-66. 2004. 1,3
[54] S. Wold, K. Esbensen, and P. Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987. 20
[55] Y. Xia, X. Cao, F. Wen, G. Hua, and J. Sun. Learning discriminative reconstructions for unsupervised outlier removal. In ICCV, 2015. 3, 18, 21
[56] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 178.07770..07747, 2017. 6
[57] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020. 3
[58] B. Zhang and W. Zuo. Learning from positive and unlabeled examples: A survey, In 2008 International Symposiums Information Processing, 2008. 1
[59] C. Zhou and R. C. Paffenroth. Anomaly detection with robust deep autoencoders. In KDD, 2017. 3, 4, 20
[60] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen. Deep autoen- coding gaussian mixture model for unsupervised anomaly detection, In ICLR, 2018. 1, 2, 3,
12

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 ADDITIONAL RESULTS WITH DIFFERENT METRICS
There are various performance metrics of OCCs. In the main manuscript, we mainly use AUC, Average Precision (AP) and Fl-score as the evaluation metries. In this subsection, we report the performances of the proposed model (SRR) and baselines in terms on Recall at Precision 70 and 90 as the additional metries on CIFAR-10 dataset.
Recall at Precision 90
Recall at Precision
30
Contrastive
SRR+ CoRtrasttet
Rotation
30
20
Contrastive
RotNet
20
Rotation
DAE
RotNet
DAE
Anomaly ratio (%) in training data
Anomaly ratio (%) in training data
Figure 9: Performance various OCCs on CIFAR-10 dataset. SRR is applied on top of Contrastive
[51]. (Left) Recall at Precision 70, (Right) Recall at Precision 90.
As in Fig. 9, we observe similar trends with these two additional metrics as well. For example, the performances of SRR are robust across various anomaly ratios. On the other hand, all the other OCCs show consistent and significant performance degradation as the anomaly ratio increases. SRR performs 15.8 and 10.9 better than the state-of-the-an OCC [51] terms of recall at precision and 90, respectively.
A.2
SRR ON RAW TABULAR FEATURES LEARNED IMAGE REPRESENTATIONS
Thyroid
Anhythmia
Auemon Prrectsin toton FI-ccare
Aveengee Precision iniioinn// Fl-core
Baseline
Baseline
SAR on raw features
raw features
Bascline
11.
raw
123
Average
Average
Average
(a) KDD-Rev
(b) Thyroid
(c) Arrhythmia
OFAR-10 Roth et representation
CIFAR-10 / Rotation representation
AICC Ruwrngg Precition
AUC: Average Preeeiee woon
Baseline
SRR on learned represenations
SRR on leamndd ropresenations
SAA on learned ropresenations
Average ge
Average
Average
(d) RotNet [23]
(e) Rotation [28]
(f) Contrastive [51]
Figure 10: Performance of SRR on (top) raw tabular features and (lower) learned image representa-
tions. SRR consistently outperforms baseline and in some cases (e.g., Thyroid, and Contrastive [51]),
the performance improvements significant.
13

---

Under review as a conference paper at ICLR 2022
SRR is also applicable raw tabular features learned image representation without representation update using data refinement. In this section, we demonstrate the performance improvements by SRR without representation update to verify the effectiveness of data refinement block of SRR for shallow OCCs.
Fig. 10 (upper) demonstrates consistent and significant performance improvements when apply SRR on top raw tabular features. Specifically, the Average Precision (AP) improvements are 10.2, 29.0, and 4.1 with KDD-Rev, Thyroid, and Arrhythmia tabular datasets, respectively.
We also apply SRR on top of various learned image representations. As can be seen in Fig. 10 (lower), the performance improvements of SRR consistent across various different learned image representations (without representation update). For instance, the AP improvements are 9.2, 1.0, and 12.1 with learned image representations using RotNet [23], Rotation [28], and Contrastive [51]. respectively.
A.3 IMPLEMENTATION DETAILS ON GOAD [7] FOR TABULAR DATA EXPERIMENTS
classification-buse AD method, GOAD [7], has demonstrated strong AD performance on tabular datasets. Unlike previous works [23, 26] that formulate parametric classifier for multiple transfor- mation classification, GOAD employs distance-based classification multiple transformations, For the set of transformations Tm X -x>77 D, m 1, M, the loss function of GOAD is written in equation 3 with the probability defined in equation 4.
-mm log Pmmm[Tm (x))],
(3)
cm 12
((([T(()) = Σn. exp((II/(Tmmmma)
(4)
c. 12)
where the centers Cm 's are updated by the average feature over the training set. While it is shown to perform well [7], find that the distance-based formulation is necessary, and achieve the similar performance, not worse, to [7] using a parametric classifier when computing the probability:
P(m(TT(x)) exp ( (x)) + ba)
expep (w f (Tm (x)) + b,))
exp
The formulation in equation 5 is easier to optimize than its original form in equation 4 as it can be fully optimized with backpropagation without alternating updates of feature extractor f and centers Car- Once we learn representation by optimizing the loss in equation 3 using equation 5, follow two-stage one-class classification framework of [51] to construct a set of Gaussian density estimation OCCs for each transformation. Finally, aggregate maximum normality scores from set of classifiers as the normality score.
In Table summarize the implementation details, such network architecture or hyperparameters, and AD performance under clean training data setting that reproduces the results in [7].
14

---

Under review as a conference paper at ICLR 2022
Datasets
KDD
KDD-Rev
Thyroid
Arrhythmia
F-score [7]
9..4
98.9
74.5
52.0+23
F-score (ours)
98.0402
95.010.2
75.1
54.8
f (feature)
Linear(8),
Lea0)))LU(002) x5
Optimizer
Momentum SGD (momentum= 0.9)
Learning rate
0.001
L2 weight regularization
0,00003
Batch size
64
Random projection dimension
32
M (number of transformations)
32
256
Train steps
2
2
Table 1: The AD performance under clean only data setting of GOAD in [7] and our implementation. Our implementation demonstrates comparable, if not worse, performance to those reported in [7]. Our implementation also shares most hyperparameters across datasets except the M, the number of transformations, and the train steps, which are closely related to the size of training data.
A.4 PER-CLASS AUC AND AP
In the main manuscript, we report the mean and standard deviation of AUC and AP across all classes in each dataset. this section, we report the mean and standard deviation of AUC and AP for each class in each dataset (including CIFAR-10 (Table 2), MVTec (Table 3), fMNIST (Table 4), and Dog-vs-Car (Table 5) datasets).
A.4.1 PER-CLASS AUC AND AP ON CIFAR- 10 DATASET
15

---

Under review as a conference paper at ICLR 2022
anomaly ratios
0%
10%
Classes / Methods
Baseline [51]
SRR + [51]
Baseline [51]
SRR + [51]
AUC
Airplane
90.5
90.4104
85.3108
87.6107
Automobile
98.8+0.1
98.7
95.2+0.5
97.9+0.
Bird
87.5+0.6
87.1
1..
84.7#112
Cat
81.5
80.3+0.7
68. 1..
78.5
Deer
90.3
90.2
79.9
87.4110
Dog
90.8
90.8
79.2:1:.5
89.610.
Frog
92.0+0.6
91.4+09
78.5
87.5
Horse
97.8+0.1
97.940.1
93.3+00
96.0106
Ship
96.5+0.2
96.4+0.1
91.5+009
95,2+05
Truck
95. 10.3
95.8+0.1
88.9=0.4
94.7+00
Average Precision
Airplane
68.3
67.9
11.
61.8
Automobile
93.91002
94.110.1
85.913.
91.51007
Bird
62.2+2.1
61.7+24
42.4+3.0
57.5+23
Cat
44.0+2.4
42.9+1.
30.3#38
44.940.
Deer
59.5+08
58.8+12
42.9+3.5
53.6+15
Dog
59. 16.5
59.8+447
39.7+9.1
64.5+20
Frog
72.7+20
70.4+69
38.5 38.5+110
73.6+3.
Horse
89.010.7
89.2
80.6119
86.2
Ship
83.9
83.7
74.9124
79.8
Truck
79.0
81.3+04
67.0
75.8
Table 2: The AD performance per class on CIFAR-10 dataset in terms of AUC and AP. Class represents the label of normal samples.
A.4.2 PER-CLASS AUC AND AP ON MVTEC DATASET
16

---

Under review as a conference paper at ICLR 2022
anomaly ratios
0%
10%
Classes Methods
Baseline [33]
SRR + [33]
Baseline [33]
SRR + [33]
AUC
Bottle
9.
98.8107
94.31008
98.7100
Cable
77.7=0.7
77.9+26
72.9
80,4
Capsule
96.1
97.2+13
92.4=1177
96.8+0.7
Hazelnut
97.2+007
97.5+04
83.5+3.6
96.7=00
Metal Nut
98.5+110
98.6104
91.2:1:8
98.7110.
Pill
92.1 13.1
91.1425
85.1
92.1
Screw
1.64
86.8+25
19.9
86.4+14
Toothbrush
1.55
98.34111
98.6+1.3
99.3+112
Transistor
96.4+20
96.7+1.
93.9+28
95.7+0.7
Zipper
99.4+00
98,0+06
91.1+299
99.2+0.7
Carpet
90.0
90.8
80.416.1
91.2127
Grid
99.210.5
99.71003
92.917.6
99.7102
Leather
99.810.3
99.7100
98.011.1
100.0100
Tile
93.1 155
93.3+27
86.4+34
92.4
Wood
88.8
98.5+14
92.1
99.0+0.
Average Precision
Bottle
98.8
99.3++0
9..3+11.
98.0+0.6
Cable
81.8+0.9
82.3+17
70.8+4.3
74.1+30
Capsule
99.010.3
99.31103
79.5
89.8127
Hazelnut
97. 10.6
97.0104
88.813.6
93.61119
Metal Nut
99.6+0.3
99,6+0.1
76. 1844
92.1+3.1
Pill
98.2+0.7
97.9+06
56.642.6
67.3422
Screw
93.8
93,6111
60.1
63.7
Toothbrush
99.2+t09
99.2
97.4
98.6+20
Transistor
93.0+3.
93.7+24
97.8+1.6
97.1405
Zipper
99.8+0.2
99.2+t.3
86. 1128
98.0+15
Carpet
94.4
95.7
64.218.2
71.61442
Grid
99.5+0.3
99,8+02
90.9+6.1
99,4+00
Leather
99.9+0.2
99.9+02
97.2+11
100.0+0.1
Tile
96.1 10.9
96.4114
82.4
86.4128
Wood
99.4
99.4
88. 1:312
97.2
Table 3: The AD performance per class on MVTec dataset in terms of AUC and AP. Class represents the label of normal samples.
A.4.3 PER-CLASS AUC AND AP ON FMNIST DATASET
A.4.4 PER-CLASS AUC AND AP ON DOG-VS-CAT DATASET
17

---

Under review as a conference paper at ICLR 2022
anomaly ratios
0%
10%
Classes Methods
Baseline [51]
SRR + [51]
Baseline [51]
SRR + [51]
AUC
T-shrr//oop
9.
93.2110
1.
84.7.110
Trouser
99.2+0.3
99.2+0.
92.9=08
94.5+119
Pullover
93.8+0.2
93,8+0.2
78.4
90.1
Dress
94.420.5
94,0+0.
78,4
85.8
Coat
95.110.1
94.9
80.4
88.7
Sandal
95.4
95.5
67.614.
81.7#18
Shirt
87.9+0.2
87.9+02
73.9
81.0+25
Sneaker
99. +0.0
99.2+t.
80.2+49
94.7136
Bag
97. 97.8+002
97.6+0.3
58.2+333
68.6+23
Ankle boot
98.6 63.6
98.5+00
76.6+2.6
87.1
Average Precision
T-shir//top
68.7
67.9.8.9
43.0
56.8137
Trouser
96.5129
98.2.10.3
78.21:52
87.1137
Pullover
62.7+1.2
63.6+1.
27. 27.2+69
57.1
Dress
72.3+3.7
71.6+27
36. 36.1
57.3426
Coat
66.2+1.1
65.6+10
24.3+339
50.7499
Sandal
70.9+28
70.8+24
17.8
40.1
Shirt
49.6+0.9
49.7+00
27.2+558
37.6+447
Sneaker
91.910.4
92.9100
21.01:3.
81.611114
Bag
92.210.4
91.8
13.212.
20.11337
Ankle boot
87.9
86.5
19.6+3.1
48.0+16.5
Table 4: The AD performance per class on IMNIST dataset in terms of AUC and AP. Class represents the label of normal samples.
anomaly ratios
0%
10%
Classes / Methods
Baseline [51]
SRR [51]
Baseline [51]
SRR [51]
AUC
Dog
89.7
89.2+004
74.7+2.1
79. 19.3
Cat
88.2
88.0
70.1
70.7
Average Precision
Dog
91.611.7
91.0
80.0122 71.643.
84.0.1.2 71.5+448
Cat
90.2+0.3
19.8
Table 5: The AD performance per class on Dog-vs-Cat dataset in terms of AUC and AP. Class represents the label of normal samples.
A.5 SRR WITH OTSU'S METHOD
In the main manuscript we show that the performances of the proposed unsupervised anomaly detection framework (SRR) are significantly better than the tfate-of-the-art OCCs if the given data fully unlabeled. Although SRR does not need any labeled data, needs a true anomaly ratio for hyper-parameter optimization. In this subsection, we further extend SRR to the setting where there no access to the true anomaly ratio as well. We utilize the Otsu's 'ss method [49] to identify the threshold between normal and anomalous samples and use that the hhper-paramene parameter (r) instead of twice the true anomaly ratio. Note that other previous works [5, 42, 55] also utilized intra-variance minimization, in similar vein to Otsu's method, for determining the threshold for data refinement.
18

---

Under review as a conference paper at ICLR 2022
The key idea of the Otsu's method is to find the threshold that minimizes the intra-class variance. This defined as the weighted sum of variances the two classes. Let us denote the normality scores as {st):== and threshold as 77. Then, try to find the threshold (7) that minimizes the weighted sum of the variance (wo(n) x ao(n) + wi (n) x a (1)) where wo( Σ> I(s; < n)/NN and wh (1) Ei T[(s; 2 n)/NN (1) and d (1) are the variances of each class. The optimal threshold (1)") determined as
min (n) x (n) + wi (1) (1))).
We use the twice rf as the hyperparameter (y) SRR.
We evaluate the performances of Otsu's 's method (on top of SRR) in comparison to the state-of-the-art OCC [33] and original SRR with the knowledge of the true anomaly ratio using MVTec dataset.
96
Average Precision
95.0
94
92
00.0
87.5
88
SuttCut (with Otsu methad)
SRRtCutPastc (with anornaly ratio)
82.5
SRR+ CutPaste (with Otsu method)
CutPate (with anomaly
CuPaatt
CutPaste
10
Anomaly ratio (%) in training data
Anomaly ratio (%) training data
Figure 11: Unsupervised anomaly detection performances with Otsu's method on top of SRR on MVTTc dataset. (Left) AUC, (Right) Average Precision (AP).
Fig. 11 demonstrates that even without true anomaly ratio, the performance of SRR can be signifi- cantly better than the state-of-the-as OCC [33] with Otsu's 's method. Also, Fig. 11 shows that the knowledge of true anomaly ratio is crucial information for maximizing the performance of SRR in fully unsupervised settings.
We further extend the experimental results using Otsu's method with other datasets such as CIFAR-10 and Thyroid datasets. Table 6 are the overall results Otsu's method yields only slight degradation compared SRR with true anomaly ratio; however, it still significantly outperforms SOTA OCC baselines.
Datasets / Method SOTA OCC
SRR with Otsu's method
SRR
CIFAR-10
0.855 0.585
0.906 / 0.703
0.910
0.709 0.709
Thyroid
0.506
0.623
0.639
Table 6: Additional ablation studies. SOTA OCC methods are [51] for CIFAR-10 dataset and GOAD [7] for Thyroid dataset. We introduce 6% noise on MVTec dataset and 1.5% noise on the Thyroid dataset. Metrics are (AUC/AP) for CIFAR-10 dataset and F1 score for Thyroid dataset.
A.6 ADDITIONAL ABLATION STUDIES
To better understand the source of gains, we include comparisons between the final ensemble model on the converged self-supervised extractor (SRR without final OCC) with the proposed SRR (using an additional final OCC). Also, to further support the novelty of SRR, report additional experimental results only with ensemble model without data refinement (Ensemble Only).
As can be seen in Table 7, the proposed version of SRR (with additional final OCC) outperforms significantly. which is attributed to the fact that while fitting the individual OCC models in the ensemble, do not exclude the possible anomaly samples for diversity of the trained submodels, so the anomaly decision boundaries can be fitted robustly. Therefore, the ensemble model can be somewhat less accurate for classifying difficult samples (near the decision boundary between normal
19

---

Under review as a conference paper at ICLR 2022
and abnormal) compared to the final OCC used (that is trained with only normal samples). Also, employment of ensemble learning without data refinement (Ensemble only), yields much worse performance than the proposed method (SRR), underlining the importance of the core data refinement idea of SRR.
Datasets Method
SOTA OCC
Ensemble Only
SRR without final OCC
SRR
MVTec
0.905 0.845
0.911 0.849
0.922 0.870
0.937 / 0.887
CIFAR-10
0.855 / 0.585
0.862 0.599
0.890 / 0.677
0.910 / 0.709
Table Additional ablation studies. SOTA OCC methods are CutPaste [33] for MVTee dataset and [51] for CIFAR-10 dataset. We introduce 6% noise both datasets. Metrics are (AUC/AP).
A.7 ADDITIONAL BASELINES
We add extra baselines from robust anomaly detection literature: Standard PCA [54], Robust PCA [15] and Local Outlier Factor (L.OF) [13] for tabular data and Robust autoencoder (Robust AE) [59] for image data. As can be seen in Table 8, the performance of PCA and LOF are highly degraded even with a small amount anomalies in the training data. For Robust PCA and Robust AE, the performance degradation is less but still significant in comparison to SRR. Overall, SRR outperforms other benchmarks in fully unsupervised settings, underlining the importance of data refinement in improving the robustness anomaly ratio in training data, the core constituent of SRR framework.
Datasets / Method
PCA
Robust PCA
LOF
Robust AE
SRR
CIFAR-10
0.636 / 0.114
0.910 / 0.709
Thyroid
0.299
0.377
0.338
0.506
KDD
0.836
0.893
0.873
0.942
Table 8: Additional experiments with extra baselines from robust anomaly detection literature. We introduce 6% noise on CIFAR- 10 and KDD datasets, For Thyroid dataset, we introduce 1.5% noise. Metrics are (AUC/AP) for image data and Fl score for tabular data.
A.8 CONVERGENCE GRAPIS
The proposed SRR framework is converged when the iterative training of the self-supervised learning converged. Depending on the data refinement model training, corresponding self-supervised models also trained with differently refined training data. Usually, the data refinement model and self-supervised models converge after a similar number of epochs. Fig. 12 illustrate the convergence graphs of SRR with MVTTc and CIFAR-10 datasets.
Represene Representation learrr learriing train loss
Represente rntorssetattion tin araning train loss
$3.0
Selected samples (%)
82.5
88
delected samples camples
$2.0
78
74
80.5
72
80.0
3.4
72.5
68
0.2
Epochs
Epochs
Figure 12: Convergence graphs of SRR with (left) MV'Tec dataset, (right) CIFAR-10 dataset.
20

---

Under review as a conference paper at ICLR 2022
A.9 ADDITIONAL DISCUSSIONS: RELATED WORKS FOR DATA REFINEMENT
In this subsection, we further discuss the related works [5, 42, 55] for data refinement that we briefly mentioned in Section 2. We highlight the differences between SRR and [5, 42, 55] below.
First, [5, 55] are based on auto-encoders and they directly utilize the reconstruction errors as informa- tive signals for anomaly detection and iterative data refinement. However, prior work [43] discovered that the reconstruction is not a good informative signal for outlier detection. Also as shown in Fig. 5, the performance of DAE (reconstruction errors based anomaly detection model) works much worse than alternatives.
Second, SRR utilizes ensemble learning for data refinement to improve the robustness which is critical in unsupervised anomaly detection. On the other hand, [5, 42, 55] utilize single model for data refinement, As can seen in Fig. 7a, the impact of the ensemble model for data refinement significant.
Third, [5, 42, 55] directly utilize the (pseudo) abnormal samples for model training in addition to (pseudo) normal samples. For instance, the model in [42] is trained via two-class ordinal regression where two classes come from (pseudo) normal and (pseudo) abnormal samples. Directly relying on the (imperfectly) labeled abnormal samples can be harmful for the anomaly detection due the overfitting problems and high False Positive Rates (FPR) in (pseudo) abnormal samples. For instance, with 80% of the recall (for the abnormal sample discovery), the precisions (for the abnormal sample discovery) are 28.2% for CIFAR-10 10 and 31.7% with MVTèc datasets (with 6% abnormal ratio) using SRR framework. It means that the majority of the (pseudo) abnormal samples actually the normal samples (71.8% for CIFAR-IO and 68.3% for MVTec datasets). This would be strong evidence that directly utilizing the (pseudo) abnormal samples for training can be harmful.
Lastly, [42, 55] are based on the end-to-end anomaly detection models which are empirically less accurate than two-stage models [51].
A.10 ADDITIONAL DISCUSSIONS: COMPUTATIONAL COMPLEXITY
Note that when applying SRR top of representation learning, the computational complexity of the representation learning part is not changed. With SRR, the additional computations come from training the ensemble models top of learned representations. Note that use shallow one-class classifiers (such as GDE or one-class SVM) for submodels; thus, the additional computational complexity is marginal. We would like mention that all the experiments done single V100 GPU and each experiment needs at most 12 hours for training (the additional training time caused by the SRR framework is an average 13. 13.1% of the total training time). The computations of the ensemble parts can be further improved by the model parallelization.
21


Reviews:
[{"review_id": "s8i6aOVOLC7", "reviewer": "Reviewer_7NNr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros*\n+ The experimental results demonstrate significant anomaly detection performance improvements for the proposed SRR approach, especially at high anomaly ratios.\n+ The paper is overall presented and written well, as well as technically sound.\n+ The paper is well placed well into the existing literature, up to including recent works.\n\n*Cons*\n- The methodological novelty of the proposed SRR approach is rather low (ensemble learning is standard to improve robustness and the individual detection method from Sohn et al. (2020) is not new)\n- The experiments do not contain a comparison to any specifically robust AD approach (e.g. robust PCA for the tabular and robust autoencoders for the image dataset). \n- While the paper includes ablation studies on key components and hyperparameters (ensemble size, data rejection confidence, updating the self-supervised feature extractor), I think there should also be a comparison to just using the final ensemble on the converged self-supervised extractor vs. training an additional final model. Is there much of a difference left?\n\n\nSome additional minor points:\n\n- I don't agree with the framing that most prior AD works \"all depend on some labeled data\" as expressed in the abstract and introduction. The bulk of AD research is on unsupervised methods (see Chandola et al. (2009) and the recent reviews by Ruff et al. (2021) and Pang et al. (2021)). However, I agree that most methods assume fairly clean training data. This view should be updated in my mind.\n- The GDE abbreviation is used before it is explained.\n- p.6: \"For the N setting [...]\" Typo?", "summary_of_the_paper": "This paper proposes an ensemble approach, called SRR (Self-supervise, Refine, Repeat), for robust unsupervised anomaly detection. The proposed approach trains an ensemble of K detectors together with a joint self-supervised feature extractor g on K disjoint subsets of the data. The ensemble is then used to filter the training data, keeping only the data points that are collectively deemed normal by the K detectors. This training-data filtering process is repeated until the self-supervised feature extractor has converged. A final detector is then trained using the refined data and converged self-supervised feature extractor. Experiments on tabular and image datasets are presented which show that the proposed ensemble approach is more robust at high anomaly contamination ratios than respective state-of-the-art single detectors.", "summary_of_the_review": "Though I think the methodological novelty of the proposed approach is rather low, and that the experimental comparison should be somewhat extended (where I expect the improvements of SRR to hold up), I am overall positive towards accepting this work since robust anomaly detection is a relevant problem of high practical significance, for which the proposed SRR approach demonstrates significant improvements over current state-of-the-art methods.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "qqGr8g0RQnt", "reviewer": "Reviewer_gVix", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Stregnth]\nThe effectiveness of the proposed framework is validated on top of contrastive learning-based models, which are state-of-the-arts.\nExtensive experiments on both tabular and image datasets support the effectiveness of the framework.\nAblation studies decouple the effects of each hyperparameter. Also, the representation update study shows that re-training representation with the refined dataset is important.\n\n[Weakness]\n- Although the proposed framework is tested on contrastive models, the idea of data refinement itself is independent of these models.\nThe framework can be applied to other types of anomaly detectors but it is not shown.\n\n- The iterative refinement has been studied [1,2,3] previously, however, no comparison or discussion is addressed with those approaches.\nAlthough they use AE-based models, the idea of iterative refinement can be deployed on top of contrastive models as well. What makes SRR more effective than these methods? What is the main factor that makes SRR more competitive than these methods?\n\n[1] Xia, Yan, et al. \"Learning discriminative reconstructions for unsupervised outlier removal.\" Proceedings of the IEEE International Conference on Computer Vision. 2015.\n\n[2] Beggel, Laura, Michael Pfeiffer, and Bernd Bischl. \"Robust anomaly detection in images using adversarial autoencoders.\" arXiv preprint arXiv:1901.06355 (2019).\n\n[3] Pang, Guansong, et al. \"Self-trained deep ordinal regression for end-to-end video anomaly detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n- Hyperparameter selection requires tuning. Although it is shown in Figure 7, that any value of gamma improves over baseline, it is still important how to choose this value. The paper suggested Otsu's method as a solution to predict the approximate anomaly ratio of the dataset but it is not shown in the main experiments. (MVTec experiment is provided in A.5) In the main experiments, hyperparameter gamma is tuned with two times of anomaly ratio which this setting requires prior knowledge on anomaly ratio. I wonder how effective Otsu's method is in other scenarios.\n\n- The training requires heavy computation as the framework requires ensemble learning on top of contrastive learning. \n\n[Questions]\n- What is the convergence condition? Is it necessary to train with the framework until the data refinement gives marginal change to the data? How long does it take?\n\n- What is the important difference of SRR from the previous refinement methods? Is SRR more effective than the previous refinement approaches?\n\n- How does SRR perform when trained with Otsu's method rather than using true anomaly ratio?\n\n\n[Post rebuttal]\nThe authors addressed all my concerns.\nI raise the score, assuming the rebuttal materials will be included in the revised version.\nRebuttal materials here means,\n- OTsu's method experiment\n- detailed discussion on the difference between SRR and the previous iterative works\n- Convergence analysis (maybe this one in the appendix)", "summary_of_the_paper": "The paper tackles an unsupervised anomaly detection problem where the training set contains an unknown portion of anomalies.\nWhen anomalies are contained in the training set, it is known that classical AD approaches' performance degrades.\nThe idea is to filter out potential anomaly samples (data refinement) by ensemble model.\nEach model in the ensemble is trained on a disjoint set of training data and then used as a classifier to determine potential anomalies.\nThen the data refinement process uses a hard assignment excluding anomalies from the training set.\nRefinement and ensemble training is repeated iteratively until convergence.\nThe proposed framework is validated on the four tabular datasets and four image datasets.", "summary_of_the_review": "The paper proposes a framework to refine data and train contrastive models for unsupervised anomaly detection problems.\nThe extensive experiments show the effectiveness of the method.\nHowever, the main experiments require prior knowledge of the true anomaly ratio which is unavailable in real-world problems.\nDiscussions about the important difference between the proposed method and the previous iterative methods would make the paper more convincing.\n\n[Post rebuttal]\nThe authors addressed all my concerns.\nI raise the score, assuming the rebuttal materials will be included in the revised version.\nRebuttal materials here means,\n- OTsu's method experiment\n- detailed discussion on the difference between SRR and the previous iterative works\n- Convergence analysis (maybe this one in the appendix)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KBQavNXQihj", "reviewer": "Reviewer_MspQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n1. the authors fully explore the power of unsupervised AD and shows outperformed results by using the proposed SRR scheme based on the GOAD framework. \n2. The performance is improved by leveraging the ensemble of the OCC, which works well but may result in additional computational cost. \n3. Enough details, such as the sensitivity of hyperparameters, are provided to reproduce the experiments on multiple tasks, including tabular and image datasets. \n\n**Weakness**\n1. The idea sounds promising but may not be the first work. The performance is enhanced by an ensemble of multiple tricks. It would be helpful to see a detailed ablation study, which may show more insights.\n2. The baseline is mainly consisting of GOAD, OC-SVM, etc. Some recent SOTA baselines are missed, for example, NeuTral [1]\n[1] Qiu, Chen, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. \"Neural Transformation Learning for Deep Anomaly Detection Beyond Images.\" ICML 2021. \n3. The baseline varies case by case, for example, Cutpaste is used for MVTec datasets, have you considered the SOTA performance, for example in this link - https://paperswithcode.com/sota/anomaly-detection-on-mvtec-ad  \nCupaste only ranks the 9th, have you compared with the other SOTA baselines?  I, therefore, have a concern, how do you choose the baseline method in different datasets? I suggest to choose more rather than a specific one. \n\n", "summary_of_the_paper": "This paper proposes a self-supervised idea for unsupervised anomaly detection. Specifically, this framework enables high-performance AD without any labels via SRR, which is an ensemble approach to propose candidate anomaly samples that are refined from training. This way allows more robust fitting of the anomaly decision boundaries and also better learning of data representations.  Multiple examples are used to demonstrate the proposed methods on effectiveness and robustness. ", "summary_of_the_review": "Overall, the paper is well-written and the results and performance look solid. My major concern is the novelty, specifically compared with the GOAD method. In addition, the code is not uploaded, so I am not sure how it works for reproducibility and time cost.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2auIET28sH", "reviewer": "Reviewer_s1MH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a novel AD framework to enable inspect defects with one-class, which is called SRR and applicable on unlabeled datasets.SRR employs an ensemble of multiple OCCs to give the potential anomaly refined samples from training. SRR brings the advantages of making the anomaly decision boundaries more robust and giving better data representations. The proof and experiment results are well organized. ", "summary_of_the_paper": "The authors propose a data refinement approach combined with self-supervised representation to robust one-class classification, which is commonly used in the anomaly detection scenario. The proposed data refinement approach is designed based on an ensemble of one-class classifiers. ", "summary_of_the_review": "The paper is ready for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper worked on fully unsupervised anomaly detection and proposed to use self-supervised representation learning to improve the performance of one-class classification. This is a borderline case close to acceptance but cannot make it. Specifically, it is useful, but its novelty is the main issue, since it is not surprising that self-supervised representation learning can improve one-class classification without representation learning (this part is still much of the taste of ICLR) and an ensemble of multiple models can improve upon a single model (which is just "bootstrap aggregating" or "bagging" used everyday in practice and known to machine learning and statistics societies a very long time ago). After seeing the rebuttal, the concerns were not really addressed well and the issues were only partially solved. Thus, the paper is not enough to guarantee an acceptance to ICLR unfortunately.
Decision: Reject