Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection
Authors: Jinsung Yoon
Kihyuk Sohn
Chun-Liang Li
Sercan O Arik
Chen-Yu Lee
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: Anomaly detection (AD), separating anomalies from normal data, has many applications across domains, from security to healthcare. While most previous works were shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-supervised representations using a data refinement process. Our proposed data refinement approach is based on an ensemble of one-class classifiers (OCCs), each of which is trained on a disjoint subset of training data. Representations learned by self-supervised learning on the refined data are iteratively updated as the data refinement improves. We demonstrate our method on various unsupervised AD tasks with image and tabular data. With a 10% anomaly ratio on CIFAR-10 image data / 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the state-of-the-art one-class classifier by 6.3 AUC and 12.5 average precision / 22.9 F1-score.

Published in Transactions on Machine Learning Research (08/2022)
Self-supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection
Jinsung Yoon, Kihyuk Sohn, Chun-Liang Li, Sercan ö. Arik, Chen- Yu Lee, Tomas Pfister finsunggoon, kahyuuk, chanliang, soarik, chengulee, tpfister) &google.com
Google Cloud AI
Reviewed on OpenReview: https: openreview. net/ net/ forum? id- b3v1UrtF6G
Abstract
Anomaly detection (AD), separating anomalies from normal data, has many applications across domains, from security to healthcare. While most previous works were shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, unlabeled. To lackle this problem effectively, we propose to improve the robustness of onc-class classification trained on self-supervieed representations using data refinement process. Our proposed data refinement approach is based one-class classifiers (OCCs), each of which is trained on a disjoint subset of training data. Representations learned by self-supervised learning on the refined data are iteratively updated as the data refinement improves. We demonstrate our method on various unsupervised AD lasks with image and labular data. With a 10% anomaly ratio on CIFAR-10 image data 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the statc-of-the-art onc-class classifier by 6.3 AUC and 12.5 average precision 22.9 F1-score.
ensemble of
1 Introduction
Anomaly detection (AD), the task of distinguishing anomalies from normal data, plays crucial role in many real-world applications such as detecting faulty products from vision sensors in manufacturing, fraudulent behaviors at credit card transactions, or adversarial outcomes at intensive care units such as death, heart atlack, or blood poisoning.
AD has been considered under various settings based on the availability of negative (normal) and positive (anomalous) data and their labels at training, as overviewed in Sec. 2 Each application scenario is dominated by different challenges. When all positive and negative samples
available along with their labels (Fig. b))
the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in
label distributions (Chawla et al. 2002 Estabrooks et al. 2004 Hwang et al. 2011 Barua et al.
2012 Lee
2000
Liu et al. 2007). When only negative labeled data are available (Fig. 1b), the problem is loneclass
classification
Schölkopf et al.
1999 Tax Duin 2004 Ruff al. 2018 Hendrycks et. al. 2018 Golan
El-Yan Yaniy 2018
Sohn al. [201 Li ct al. 2021). Various works have also extended approaches designed for
these to settings with additional unlabeled data (Fig. [)d,)) (Zhang & Zuo 2008 Blanchard et al.) 2010;
Görnitz et al.) 2013 Ruff et al. 2020 in semi-supervised setting. While there exist many prior works in these settings, most of them roly on some labeled data, which is not desirable in many application scenarios.
Unsupervised AD, on the other hand, poses unique challenges in the absence of any labeled data information, and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal.
For example, some recent studics [Zong al., 2018 Bergman & Hoshen 2019) have applied one-class classifiers
(OCCs) that are known yield the paato-of-tho-art performance when trained on negative samples (Golan
& Elliyniv 2018: Hendryeks et al. 2018 Bergman & Hoshen 2019. Sohn et al. 2021 Li et al. 2021 to

---

Published in Transactions on Machine Learning Research (08/2022)
Negative (N) sample,
Positive
(P) sample,
Unlabeled (U) sample
(a) P+N
(b) N
(c) P+N+U
(d) P+U
(e) N+U
(f) U
Figure 1: Anomaly Detection (AD) problem settings. Blue and red dots are for labeled negative (normal)
and positive (anomalous) samples, respectively. Grey dots denote unlabeled samples. While previous works mostly focus on supervised (a, b) or semi-supervised (c, d, e) settings, we tackle AD problem using only unlabeled data (I) that may contain both negative and positive samples.
unsupervised AD, but their performance for unsupervised AD (when the unlabeled data contain both positive and negative samples) has been significantly degraded. Fig. 2 illustrates this, showing the unsupervised AD performance of state-of-the-art Deep OCCs (Sohn et al. 2021] with different anomaly ratios in unlabeled training data the average precision significantly drops even with small contamination ratio (2%) of the training data.
Our framework SRR (Self-supervise, Refine, Repeat), overviewed in Fig. 3 brings a aatta-centri@@approaadd to unsuper- vised AD with the principles self-supervised learning without labels and iterative data refinement based on the agreement of OCC outputs. We propose to improve the stato-of-the-art
Average Precision
performance of OCCs, e.g. (Sohn el al. 2021 Li et al. 2021 by refining the unlabeled training data so as to address the challenges elaborated above. SRR iteratively trains deep rep- resentations using refined data while improving the refinement of unlabeled data by excluding potentially-positive (anomalous) samples. For the data refinement process, we employ an ensem- ble of OCCs, each of which is trained on disjoint subset of unlabeled training data. The samples are declared normal if there is consensus between all the OCCs. The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting.
05
Baseline (Contrastive) Proposed model (SRR)
20
Anomaly ratio (%) in training data
Performance of the proposed model (SRR) and a baseline OCC using con- trastive representation (Sohn et al.) 2021 on CIFAR-10 with different anomaly ratios in the training data.
Figure 2:
Most prior AD works (Golan & El-Yaniv 2018 Hendrycks al.
2018 Sohn et al. 2021 Li et al.] 20221] assume that the data contain entirely negative samples, which makes them not truly unsupervised as they require having humans to do the data
filterin. Most prior works are not actually designed for fully-ussupervieed anomaly detection. Some prior unsupervised AD works (Ruff et al. 2018 Zong et al.) 2018 Bergman & Hoshen 2019) consider evaluating on an unsupervised setting where there exist a small percentage anomalous samples in the training data, i.e. operating in 'truly' unsupervised setting without having the need for humans to do any filtering in the training data. However, these methods often suffer from significant performance degradation as the ratio of anomalous sample ratio increases (see Sec. 4.2 Our method distinguishes from the literature by bringing data-centric approach to unsupervised AD beyond the model-centric approaches. We show the value of SRR in improving robustness of the performance as the anomaly ratio increases, as shown in Fig. 2
We conduct extensive experiments across various datasets from different domains, including semantic AD (CIFAR-10 (Krizhevsky & Hinton 2009), Dog-vs-Cat [Elson et al. 2007)), ), real-workd manufacturing visual AD use case (MVTec (Bergmann et al. 2019) and real-world tabular AD benchmarks (e.g., detecting
Definitions of the data-centric approaches be found here hattps://specrrum feee. www.:iiwwwe.deeeeeeeeeeeee data- centric-ai datttaaaa-ttt-i--aa
2

---

Published in Transactions on Machine Learning Research (08/2022)
medical or network anomalies). We consider methods with both shallow (Shhlkkof et al. 1999 Liu et al. 2008) and deep (Sohn et al.) 2021: Li et. al.
2021: Bergman & Hoshen 2019) models. We evaluate models at different anomaly ratios of unlabeled training data and show that SRR siguificantly boosts performance. For example, in Fig. 2 SRR improves more than 15.0 average precision (AP) with a 10% anomaly ratio compared to a stale-of-Uhe-art one-class deep model (Sohn et al., 2021) on CIFAR-10. Similarly, on MVTec, SRR retains a strong performance, dropping less than 1.0 AUC with 10% anomaly ratio, while the best existing OCC (Li et al. 2021) drops more than 6.0 AUC. The contributions of this paper are summarized below.
We propose novel data-centric framework, SRR, for unsupervised anomaly detection using the ensemble of one-class classifiers as a data refinement module.
The proposed framework is a model-agnostic approach that can be applicable on top of any anomaly detection framework, considering different ways of applying self-supervised representation learning or one-class classification.
SRR achieves significant robustness improvements with various anomaly ratios- in other words, the users do not need to worry about manually filtering the possible anomalies from the training data to minimize contamination. We demonstrate the superior performances on multiple tabular and image datasets.
2 Related Work
There are various existing works under different scttings described in Fig.
Learning from both positives & negatives setting is often considered as a supervised binary classi- fication problem. The challenge arises due to the imbalance in label distributions as positive samples are (Chawla et al.) [202]]
rarc. As summarized in (Branco et al.] 2015), to address this, over-
Estabrooks et al.
2004), weighted optimization (Hwang et al., 2011 Barua el al. 2012) synthesizing data of
minority classes
Lee 2000
Liu et al..
2007,, and hybrid methods (Galar et al.
2011 have been studied.
Learning only from the negatives sctting is often converted to a onc-class classification (OCC) problem, with the goal of finding a decision boundary that. includes as many one-class samples as possible. Shallow
models for this setting include one-class support vector machines (Schölkopf et al. 1999) (OC-SVM), support
vector data description (Tax & Duin 2004) (SVDD), kernel density estimation (KDE) Lattecki et al. 2007).
and Gaussian density estimation (GDE) (Reynolds) 2009). There are also auto-encoder based models (Zhou
& Paffenroth 2017) that treat the reconstruction error the anomaly score. Deep learning based OCCs have
been developed, such as Deep OCC (Ruff et al. 2018), geometric transformation (Golan & El-Yaniv) 2018
or outlier exposure (Hendrycks et al. 2018). Noting the degeneracy or inconsistency of learning objectives of existing end-to-end ble Deep OCCs, (Sohn et al.] [2021] propose a deep representation OCC, a two-stage
framework that Icarns self-supervised representations [ooodaaii & Gidaris 2018 Chen et al.) 2020) followod by shallow OCCs. That work is extended for texture anomaly localization with CutPaste (Li el al. 2021
Robustness against very low anomaly ratios of these in unsupervised setting explored in (Zong et al.
2018
Bergman & Hoshen 2019).
The semi-supervised learning setting utilizes small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data. Depending on which labeled samples are given, this setting can be split into three sub-categories. When only some positive/negative /negative labeled samples are provided, we denote that as a PU (positive + unlabeled) / NU (negative + unlabeled) setting. Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given (Münoz-Marí et al., 2010; Song et al.] 2017 Akcay et al., 2018). The PNU (positive + negative + unlabeled) setting is the more general semi-supervvsed setting where subsets of both positive and negative labeled samples are given. Deep SAD (Ruff et al./ (2200) and SU-IDS (Min et al. 2018) belong to this category. We show significant outperformance of SRR compared to Deep SAD, on multiple benchmark datasets without using any labeled data (see Sec. 4.2
The unlabeled setting has received relatively less attention despite its significance in minimizing labeling costs. The popular methods for this sctting include isolation forest (Liu et al.) 2008) and local outlier factor (Breunig et al., (2000). These are difficult to scale, and less compatible with recent advances in representation learning. While OCCs, such as OC-SVM, SVDD, or their deep counterparts, can also be
3

---

Published in Transactions on Machine Learning Research (08/2022)
applied to unlabeled settings by assuming the data are all negative, and the robustness of those methods has been demonstrated in part (Zong et al. 2018 Bergman & Hoshen 2019). In practice, we observe significant. performance drop with a high anomaly ratio, shown in Fig. 2 In contrast, our proposed framework is able to
maintain high performance across different anomaly ratios.
Data refinement has been applied to AD in some prior work. (Pang et al. 2020 Beggel et al., 2019)
generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent
AD performance in unsupervised settings. (Mohseni et al. 2021 Xia et al. 2015 Zhou & Paffenroth 2017 7 Lai et al.
2019 Berg et al., 2019) use the reconstruction errors of the auto-encoder
an indicator for use data refinement for AD in
removing possible anomalies. (Feng et al. 2021) and (Meng et al. 2021
supervised and semi-supervised settings. SRR differentiates from (Xia et al. 2015 Beggel et al. 2019 Pang
et al.
2020 in multiple key ways:
Xia et al.
2015 Beggel et al. 2019) are based on auto-encoders and they directly utilize the reconstruction
errors as informative signals for AD and iterative data refinement. However, prior work (Ren et al., 2019
discovered that the reconstruction is not good informative signal for outlier detection. Also as shown in Fig. [ the performance of DAE (reconstruction errors basod AD) is much worse than alternatives.
SRR utilizes ensemble learning for data refinement to improve the robustness which is critical in unsupervised AD. On the other hand, (Xia et al.,
2015 Beggel et al.] 2019; Pang et al., 2020) utilize a single model for data refinement. As can be seen in Fig. Sa the impact of the ensemble model for data refinement is significant.
(iia et al., 2015 Beggel et al. 2019 Pang et al. 2020 directly utilize the (pseudo) abnormal samples for model training in addition to (pseudo) normal samples. For instance, the model in (Pang et al. 2020) is trained via two-class ordinal regression where two classes come from (pscudo) normal and (pseudo) abnormal samples. Directly relying on the (imperfectly) labeled abnormal samples can be harmful for AD due to the overfitting problems and high False Positive Rates (FPR) in (pseudo) abnormal samples
(Xia et al., 2015 Pang et al. 2020) are based on the end-to-end AD models which are empirically less
accurate than two-stage models (Sohn et al. 2021
1975 is an iterative training mechanism using predicted pseudo labels as targets. It has regained popularity recently with its successful results in semi-supervised image classification (Berthelot et al. 2019 Sohn et al., 2020; Xie et al.] 2020). To improve the quality of pseudo labels, employment of an ensemble of classifiers has also been studied. (Brodley et al.) 1996) trains ensemble of classifiers with different classification methods to make a consensus for noisy label verification, while co- training (Blum& & Mitchell 1998) trains multiple classifiers, each of which is trained on the distinct views, to supervise other classifiers. Co-teaching (Han et al., 2018) and DivideMix (Li et al. 2019) share similar ideas they both train multiple deep neural networks on separate data batches to learn different decision boundaries, thus, becoming useful for noisy label verification. While sharing a similarity, SRR has clear differences from the previous works SRR performs ilerative training with data refinement (with robust. ensemble methods) and self-supervised learning for unsupervised AD.
Self-training (Scudder 1965 McLachlan
3 Proposed Framework
Refine, and Repeat (SRR) is an ilerative training framework, where we refine the data (Sec. 3.1 and update the representation with the refined data (Sec. 3.2 followed by OCC training on refined representations. Fig. overviews the framework and Algorithm I provides the pseudo-code.
Sell-supervise,
Notation. We denote the training data as D= [ where x, E X' and N is the number of training samples. y/ E (0, 1} is the corresponding label to x,, where 0 denotes normal (negative) and denotes anomaly (positive). Note that labels are not provided in the unsupervised setting.
FFor instance, with 80% of the recall (for the abnormal sample discovery), the precision (for the abnormal sample discovery) is 28.2% for CIFAR-10 and 31.7% with MV (with 6% abuormal ratio) using SRR framework the majority of the (pscudo) abnormal samples are actually the normal samples (71.8% for CIFAR-10 and 68.3% for MVTee datasets). This would be strong evidence that directly utilining the (pseudo) abnormal samples for training can be harmful.
4

---

Published in Transactions on Machine Learning Research (08/2022)
Data refinement (Ensemble one-class classifier)
Train
Inference
One-class classifier
Subset
Intersect
Train
Representation learner
Unlabeled data (D)
One-class classifier
One-class classifier
Subset 2
One-class
Subset K
classifler
Figure 3: Block diagram of SRR composed of representation learner (Sec. 3.2 data refinement. (Sec. 3.1 and final OCC blocks. The representation learner updates the corresponding deep neural network models using refined data from the data refinement block. Data refinement is implemented by ensemble of OCCs, each of which is trained on K disjoint subsets of unlabeled training data. Samples predicted as normal by all classifiers are retained in the refined data, and are used to update the representation learner and final OCC. The training process SRR is repeated until the convergence of the representation learner's loss (if no improvement observed for epochs). Convergence graphs are provided in the Appendix A.3
Let us denote feature extractor as g Z. g may include any data preprocessing functions, an identity function (if raw data is directly used for onc-class classification), and learned or learnable representation extractors such as deep neural networks. Let us define OCC as ʃ 2 - [-00, oo] that outputs anomaly scores given the input features g(x), such that the higher the score f(g(x)) is, the more anomalous the sample x would be predicted as. The binary AD is made after thresholding: 1 (((((x)) n))
3.1 Data refinement
A naive way to generate pscudo labels for unlabeled data would be to construct an OCC on raw data learned representations as in (Sohn el al. 2021), ) pick a threshold for the anomaly score and obtain a binary label for samples as normal vs. anomalous. As we update the model with refined data that exclude samples that are predicted to be anomalous,
important to generate pseudo labels for training data accurately, To this end, instead of training single classifier (unlike most previous works on data refinement for AD (Pang et al. 2020 Beggel et al.) 2019: Zhou & Paffenroth 2017)), train ensemble of K OCCs and aggregate their predictions to generate pseudo labels.
We illustrate the data refinement block in Fig. and as REFINEDATA in Algorithm [ Specifically, we randomly divide the unlabeled training data D into K disjoint subsets DI,..., Dr. and train different OCCs (f1..., on corresponding subsets (D1...., Dk). Then, we estimate a binary pscudo-labol of the data x; E as follows:
Ú:
II (fa(g(xx)) 2 m
(1)
N
100
max 1y s.t.
Σ 2()(
(2)
where 1(-) is the indicator function that outputs 1/0 if the input is True/False. fa [x(x)))) represents an anomaly score of x; for an OCC fa- Ti in Eq. is threshold determined as percentile of the anomaly score distribution (fx(g(x)))
To interpret Eq. D x; is predicted as normal, i.c. d1 = 0, if all K OCCs predict as normal. While this may be too strict and potentially reject many true normal samples in the training set, find that empirically, it is
5

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm 1 SRR: Self-supervise, Refine, Repeat.
Input: Train data {xx //= Ensemble count (K), threshold (n)
Output: Refined data (D), trained OCC (f), feature extractor (g)
function REFINEDATA(D, 7)
2: Train OCC models {fa 1 on {Dx) K disjoint subsets of the training data D.
3: Compute thresholds yn is for percentile of anomaly distributions using Eq. 2
4: Predict binary labels úJ using Eq. !I
5: Return D = (x: : Ds = 0, x, ED}.
end function
function SRR(D, K,
8: Initialize the feature extractor g.
9: while not converged do
10: D = REFINEDATA 7)
11:
Update g using D with self-supervised objectives.
12:
end while
13: D REFINEDATA(D,, g, r),
14: Train OCC model (f) on refined data (D).
15: end function
critical to be able to exclude true anomalous samples from the training set. The effectiveness of employment of an ensemble of OCCs to improve the robustness against overfitting OCCs is empirically shown in Sec. 5 More specifically, Fig shows that the performance is much better for higher ensemble counts, compared single classifier training along with the feature extractor.
3.2 Representation update
SRR follows the idea of deep representation learning to perform OCCs (Sohn et al. 2021 where in the first stage, deep neural network is trained with self-supervised learning (such rotation prediction (Golan & El- Yaniv 2018). contrastive (Sohn et al., 2021). or CutPaste [Li et al. 2021 to obtain meaningful representations of the data, and in the second stage OCCs are trained on these learned representations. Such a two-stage framework is shown to beneficial as prevents the "hypersphere collapse' of the deep OCCs by the favorable inductive bias it brings with the architectural constraints [Ruff et al. 2018
We propose to conduct self-supervised representation learning jointly with data refinement. More precisely, train feature extractor g using D = {x, 0 = 0), subset of unlabeled data D that only includes samples whose predicted labels with ensemble OCC from Sec. 3.1 are negative. We also update D as proceed with representation learning. Overall method is illustrated in Algorithm [ Unlike previous works (Sohn et al. 2021 Li et al. 2021 that use the entire training data for representation learning, we empirically show that it is beneficial to refine the training data even for learning of the deep representations. Without representation refinement, the performance improvements of SRR are limited, as shown Sec. [ Lastly, to get predict test-time, we train an OCC on refined data D on the updated representations by g as in line 13-14 in Algorithm
3.3 Unsupervised model selection
As SRR is designed for unsupervised AD, labeled validation data for hyperparameter tuning typically not available and the fremework should enable robust model selection without any reliance on labeled data. Here, we provide insights on how to select important hyperparameters, and later in Sec. 5.3 we perform sensitivity analyses for these hyperparameters.
Data refinement of SRR introduces two hyperparameters: the number of OCCs (K) and the percentile threshold (7). There is a trade-off between the number of classifiers for the ensemble and the size of disjoint subsets for training each classifier. With large K, aggregate prediction from many classifiers, each of which

---

Published in Transactions on Machine Learning Research (08/2022)
may contain randomness from training. This comes at the cost of reduced performance per classifier we use smaller subsets to train them. In practice, we find K=5 works well across different datasets and anomaly ratios, Y controls the purity and coverage of refined data if y is large, and thus classifiers reject too many samples, the refined data could be more pure and contain mostly the normal samples; however, the coverage of the normal samples would be limited. On the other hand, with a small 7, the refined data may still contain many anomalics and the performance improvement with SRR would be limited. We empirically observe that SRR is robust to the selection of y when it is chosen from a reasonable range. In our experiments, we find 1-2 times of the true anomaly ratio to be often reasonable choice. In other words, it is safer to use y higher than the expected true anomaly ratio. In some cases, the true anomaly ratio may not be available. For such scenarios, we propose Otsu's method (Sezgin & Sankur 2004) to estimate the anomaly ratio of the training data for determining the threshold (experiment resulls are in Sec.
4 Anomaly Detection Performance
We evaluate the efficacy of our proposed framework for unsupervised AD tasks for tabular (Sec. 4.1 and image (Sec. 4.2 data types. We experiment varying ratios of anomaly samples in unlabeled training data and with different combinations of representation learning and OCCs. In Sec. 5 we provide performance analyses to better explain major constituents of the performance, as well as the sensitivity analysos.
Implementation details: To reduce the computational complexily of the data refinement block, we utilize a simple OCC such as GDE in the data refinement block. In a two-stage model, we only update the data refinement block at 1st, 2nd, 5th, 10th, 20th, 50th, 100th, 500th epochs. After 500 epochs, we update the data refinement block per each 500th epoch. Each experimental run is performed on a single V100 GPU. Additional discussions can be found in Appendix [
4.1 Experiments on tabular data
Datasets. Following (Zong et al. 2018 Bergman & Hoshen 2019), we test the performance of SRR on a varicty of real- world tabular AD datasets, including network (KDDCup) and medical (Thyroid, Arrhythmia) AD from the UCT repository (Asuncion & Newman 2007p. We also use KDDCup-Rev, where the labels of KDDCup are reversed so that an attack represents the anomaly (Zong et al. 2018). To construct the data splits, we utilize 50% of normal samples for training. In addition, we hold out some anomaly samples (amounting to 10% of the normal samples) from the data. This allows to simulate unsupervised settings with anomaly ratio of up to 10% of entire training set. Rest of the data is used for testing 3 We conduct. experiments using 5 random splits and 5 random seeds, and report the average and standard deviation of 25 F1-scores (with scale 0-100) for the performance metric.
Models. We focus on comparisons with GOAD (Bergman & Hoshen 2019) (the state-of-the-art AD model in the tabular domain), with and without SRR. GOAD utilizes random transformation classification as the pretext. task of self-supervised representation learning, and the normality score is determined by whether transformations are accurately included in the transformed space of the normal samples. We re-implement GOAD [Bergman & Hoshen, 2019) with a few modifications. First, instead of using embeddings to compute the loss, we use a parametric classifier, similarly to augmentation prediction (Sohn et al. 2021). Second, we follow the two-stage framework (Sohn et al. 2021) to construct deep OCCs. For the clean training data setting, our implementation achieves 98.0 for KDD, 95.0 for KDD-Rev, 75.1 for Thyroid, and 54.8 for Arrhythmia Fi-scores, which are comparable to those reported in (Bergman & Hoshen 2019). Please see Appendix C for formulation and implementation details.
We also include the comparisons with OC-SVM, with radial basis function (rbf) kernel. For comparisons with other conventional baselines including Standard PCA (Wold et al. 1987). Robust PCA (Candès et al. 2011h, and Local Outlier Factor [Brunngg et al. 2000), please see the Appendix (we do not present them here as they are far worse than the stato-of-tho-art on the experimented datasots).
Note that the experimental settings with contaminated training data in GOAD (ergmann & Hoshen et 2018) are slightly different from ours. The contamination ratio in this paper defined as anomaly ratio over the entire training data, while their contamination ratio is defined the anomaly ratio over all the anomalies in the dataset.
2019 and DAGMM Zong
7

---

Published in Transactions on Machine Learning Research (08/2022)
Results.
KDD
KDD-Rev
Thyroid
Arrhythmia
GOAD
55
152
OC-SVM (rbf kemel)
42
33
Flacore
03.
07.5
85.0
SRA+COAD
SAK+ SHA+GOAD
SAR+GOAD
43
125
GOAD
GOAD
GOAD
5V- kemel)
keme!)
OC SVM (rbf kemee)
35
25
25
65
15
15
23
24
26 Anomaly ratio (%) training data
Anomaly ratie (%5 in training data
Anomaly ratio training data
Anomaly ratio training data
Figure 4: Unsupervised AD performance (FI-score) using OC-SVM (with rbf kernel), GOAD (ergman & Hoshen 2019), and GOAD with the proposed method (SRR) on four tabular datasets. Shaded areas represent the standard deviations.
Fig. 4 presents the results for SRR compared to the GOAD bascline, as well as OC-SVM. Overall, we observe superior AD performance with SRR consistently across datasets. For KDD-Rev, the performance of GOAD (without SRR) drops significantly even with a small anomaly ratio in training data, and beyond 1%, the drop becomes very significant. On the other hand, SRR can preserve the high anomaly detection performance despite the contamination in the training data, and at 2.5% anomaly ratio, we can observe almost the double F-score. The improvements on other datasets are smaller, but similarly we can observe significant F-score differences when the training data is contaminated with many anomaly samples. On two small-scale datasets, Thyroid & Arrhythmia, we observe more variance in the results due to data randomness when fitting the models, expected. The improvements of SRR also highlights its data efficiency, the proposed refinement mechanism can be effective even with small subset of samples.
4.2 Experiments on image data
Datascts. We evaluate SRR on various visual AD benchmarks, including real-world mamufacturnng AD dalaset (MVTec (Bergmann el al. 2019)) and semantic AD datasets (CIFAR-10 (Krizhevsky & Hinton 2009). f-NIST (iiao et al. 2017). Dog-vs-Car Cat (Elson et al.) 2007)). ). For CIFAR-10, f-MNIST, and Dog-vs ves-vs-Caa Cat datasets, samples from one class are set to be normal and the rest from other classes are set to be abnormal. Similar to the experiments on tabular data Sec. 4.1 we swap a certain amount of the normal training data with anomalies given the target anomaly ratio. For MVTec, since there are no anomalous data available for training, we borrow 10% of the anomalies from the test and swap them with normal samples in the training set. Note that 10% of samples borrowed from the test set excluded from evaluation. For all datasets, we experiment with varying anomaly ratios from 0% to 10%.
We use AUC and average precision (AP) metrics to quantify the AD performance (with scale 0-100). When computing AP, we set the minority class of the test set as label 1 and majority as label 0. We run all experiments with 5 random seeds and report the average performance for each dataset across all classes.
Models. For semantic AD benchmarks, CIFAR-10, F-NNST, and Dog-vs-Cat, we compare the SRR with two- stage OCCs (Sohn et al. 2021) using various representation learning methods, such distribation-augnn contrastive learning (Sohn et al. )021), rotation prediction [Komodakis & Gidaris 2018 Golan & El- Yaniv 2018) and its improved version, and denoising autocncoder. For MVTec benchmarks, we use CutPaste (Li el al. 2021b as the baseline and compare to its version with SRR integration. For both experiments, we use the ResNet-18 architecture, trained from random initialization, using the hyperparameters from (Sohn et al,, 2021) and (Li et al. 2021). The same model and hyperparameter configurations are usod for SRR with k== classifiers in the ensemble. We set y as twice the anomaly ratio of training data. For 0% anomaly ratio, we set 7 as 0.5. Finally, Gaussian Density Estimator (GDE) learned representations is used as the OCC.
8

---

Published in Transactions on Machine Learning Research (08/2022)
For comparisons with other conventional baselines inchuding Standard PCA Wold et al. 1987 Robust PCA (Candès et al. 2011p, and Robust autoencoder (Zhou & Paffenroth 2017). please see the Appendix (we do not present them here as they are far worse than the state-of-the-art on the experimented datasets).
Results.
70
Aver age Precision
Recail at Precision
00
AUC
SRR++Canrasstie+tOO
20
ToR--Cotrastiie-OC
SRR+ Contrastiv+OOC
Contrastive+OcC
Contrastive+OCC
Contrastive+OCC
Rotation+DCC
40
Retation+OCC
RoCation+OCC
RotNet
DAE
DAE
DAE
Anomaly ratio training
Anomaly (%) training
Anomaly ratio training data
Figure 5: Unsupervised AD performance with various OCCs CIFAR-10 dataset. For SRR we adapt distributionn-augmentee contrastive representation learning (Sohn et al. 2021 (Left) AUC, (Middle) Average Precision (AP), (Right) Recall at Precision 70.
On CIFAR-10 dataset, Fig- 5 compares SRR to other alternatives. In general, most methods suffer from significant performance drop with increased anomaly ratio, despite using representation learning. For example, the AUC for AD based drops from 92.1 to 83.6 when anomaly ratio becomes 10%. Similarly, with the improved rotation prediction representation (Komodakis & Gidaris 2018), the AD performance drops from 90.0 to 84.1 in AUC. On the other hand, SRR effectively handles the contamination in training data and the reduction in performance is much lower with increased anomaly ratio it achieves 89.9 AUC with 10% anomaly ratio, reducing the performance drop by 74.1% compared to the best alternative. One 'oracle' upper bound would be removal of all anomalics from training data, which is the same as the performance at 0% anomaly ratio for the same size of the data. As Fig, 5 shows, the performance of SRR is similar to this oracle upper bound, with less than 2.5 AUC difference, until high anomaly ratios (10%). The results are also similar in other metrics, such as AP and Recall at Precision of 70, described in Fig. 5
contrastive representation (Sohn et al. 2021
Next, we show additional results on 3 visual AD datasets Fig. comparing the best alternatives with representation learning and OCC. The improvements of SRR, particularly at higher anomaly ratios, are consistent across different cases. For example, on MVTec dataset, SRR improves AUC by 4.9 and AP by 7.1 compared to the state-of-the-art CutPaste OCC with anomaly ratio of 10%. On Dog-vs-Cat task, we observe the improvement to be smaller, which we attribute to similarity of the learned representation for these two classes, that makes anomaly definition more ambiguous.
5 Performance Analyses
In this section, we focus on providing more insights on the important constituents of our framework with empirical results. We first present ablation studics for better understanding of the source of gains with different components of the framework. Next, we quantify the refinement accuracy and show the efficacy of the proposed refinement block, as one of the key components of the proposed framework. Finally, present scnsitivity analyses on two key hyperparameters of SRR, it is crucial for an unsupervised AD method not to be sensitive to its hyperparameters.
5.1 Ablation studies
To shed more light on the source of gain coming from different components, we present ablation studies. Specifically, we include comparisons with the final ensemble model on the converged self-supervised extractor (SRR without the final OCC), without using data refinement but only employing an ensemble OCCs, applying data refinement only for OCCs (but not for the representation updates), and data refinement with majority voting (SRR with majority voting).
9

---

Published in Transactions on Machine Learning Research (08/2022)
SRR+CutPasti
25
060
3RR+Contrasiiv+OOC
CutPaste
973
Contrastive+OCC
150
85
00
68
SAR+Contrastiv+OOC
Contrastive+OCC
72
760
Anomaly ratio (%) in training data
Anemaly ratio (%) training
Anomaly ratio (%) in training data
80
CRR-CutPaste
40.0
Average Procision
15.3-
CutPaste
Average Prectsion
Average Precision
22.5
60.3
82.5
01.3
65.3
860
5RR+ Centrasttrttteeeeee
SBR+
Contrastive+OCC
250
Contrastiv++OCC
725
12
Anomaly ratio (%) in training data
Anomaly ratio (%) in training data
Anomaly aly ratio (%) in training data
(a) MVTee
(b) M-NIIT
(c) Dog-vs- CatCvs-Cat
Figure 6: Unsupervised AD performance (a) MVTec (b) F-NNIST, and (c) Dog-vs- Cat. datasets with varying anomaly ratios. We use state--f-theeart one-class classification models for baselines, such as distribution- augmented contrastive representations (Sohn et al. 2021) for f-MNIST and Dog-vs-Cat, or CutPaste LLi
ct al. 2021 for MVTec, and build SRR on top of them.
Table 1: Ablation studies. SOTA OCC methods are CutPaste (Li al. 2021
for MVTec dataset and (Sohn)
al. 2021 for CIFAR-10 dataset with 6% noise. Metrics are (AUC/AP)
Methods / Datasets
MVTec
CIFAR-10
SOTA OCC
0.905=00.02 0.845:10.035
0.855+100017 / 0.58510.003
SRR without final OCC
0.922=00002 0.870=#0.022
0.890=10.007
0.677+0.019
SRR without data refinement (Ensemble of OCCs)
0.911=0.023
0.849=000037
0.862+0.0000 / 0.599+0.003
Data refinement for only OCC SRR with majority voting
0.918=0.008 0.858=0.039
0.885+000006
0.644+0.016
0.925=0.017
0.873=00002
0.89310.005
0.675+.0.014
SRR
0.937:0.018 / 0.887=0.032
0.910=0.005 / 0.709-10.013
In Table. 1. the proposed version of SRR (with an additional final OCC) outperforms the version without the final OCC. Because while fitting the individual OCC models in the ensemble, we do not exclude the possible anomaly samples for diversity of the trained submodels, so the anomaly decision boundaries can be fitted robustly. Ensemble of one-class classifiers is employed to identify the possible anomalous samples in the training set rather than yielding final anomaly score predictions. Therefore, we do not exclude the possible anomalous samples train the weak one-class classifier. In that regard, if we directly utilized the outputs of the ensemble for the final anomaly scores, the performance would be worse (as shown in Table []. Also, employment of ensemble of OCCs without data refinement, yields worse performance than the proposed (SRR), underlining the importance of the core data refinement idca of SRR, which was further eluded in Sec. 5.2 Lastly, as opposed lo applying data refinement. only for OCCs, we find that learning representation with refined data plays a crucial role, resulting in another improvement compared to SRR using fixed representations trained on the entire datasct. The proposed version of SRR achieved statistically significant improvements over alternatives with CIFAR-10 datasets terms of both AUC and AP but not with MVTec datasets due to their small sample size per category (100s-500s).
10

---

Published in Transactions on Machine Learning Research (08/2022)
Anomolous compicc
sample
sample
Norma samples
Sammure snomaio mmp
anerrreu Remuedt Fomnul corpies
reccettage
6% Anomaly ralio dsi training data
26
22h
Anomaly race 196) intraiiing data
Eps
Epocts
(a) CIFAR-10
(b) MVTec
(c) CIFAR-10
(d) MVTec
Figure 7: Percentage of excluded anomalous and normal samples by data refinement (a, b) with different.
anomaly ratios in training data and (c, d) over training epochs with 10% anomaly ratio.
5.2 Refinement efficacy
It is crucial for the proposed data refinement block being accurate in filtering the correct anomaly samples, to minimize the impact of contamination. We evaluate the accuracy of this filtering of the proposed data refinement block. As Fig. 7 a, b) shows, with data refinement, we can exclude more than 80% of anomalies in the training set without removing too many normal samples. For example, among 4% anomalics in CIFAR-10 data, SRR able exclude 80% anomalies while removing less than 20% normal samples. Such a high recall of anomalics of SRR not only useful for unsupervisod AD, but can also help improving the annotation efficiency when & budget for active learning is available. Fig. 7e, d) demonstrates the removed normal and abnormal samples by the data refinement module over training epochs. It shows that better representation learning (as the number of training epochs increases) consistently improves the efficacy of the data refinement. This positive reinforcement between better representation learning and higher filtering accuracy, constitute one key aspect of the proposed framework.
5.3 Sensitivity to hyperparameters
31
730
Everzge Prectsion
575
05.
ADC
025
AUC
Average
550
58-
525
Ensemble count
Threshold (y)
Average Precision
AUC
furerage
22
Ensemble count
Threshold
(a) Ensemble count K
(b) Percentile threshold
Figure 8: On (top) CIFAR-10 and (bollom) MVTTe under 10% anomaly ratio setting, sensitivity analyses for the (a) ensemble count K, (b) percentile threshold 7.
SRR designed for fully-unsupervieedd AD, thus, is important to ensure robust performance against changes in the hyperparameters, as model selection without labeled data would be very challenging. To demonstrate this, conduct sensitivity analyses for the key hyperparameters, shown in Fig.
11

---

Published in Transactions on Machine Learning Research (08/2022)
96
AUC
92
90
SRR CutPaste (with true anomaly ratio) CRR+Cutaaste CutPaste (with Otsu method)
86 CutPaste
10
Anomaly ratio (%) in training data
Figure 9: Unsupervised AD performance for SRR with Otsu's method on MVTec dataset AUC.
Fig. 8a shows the impact of the number of OCCs in the ensemble, K. In general, the performance saturates even after a small number OCCs, and a typical value of K around ~-10 seems applicable across most datasets. On CIFAR-10 (top in Fig. Saj, we observe slight improvements with higher K as the mumber of samples to train cach OCC (N/K) would be sufficient.
Fig. [b. shows the impact of the threshold parameter, 7. Overall, an intermediate value seems optimal and the performance does not change much around it. We observe that SRR performs robustly when is set to be larger than the actual anomaly ratio (10%). When is less than 10,, however, we observe some drop in performance as 7 decreases. Yel, we nole that SRR still improves over alternatives regardless of the threshold. The analyses suggest that y could be set to be anywhere from the true anomaly ratio and and 2x the anomaly ratio to maximize its effectiveness. For the scenario of unknown true anomaly ratio, we propose Otsu's method (sce Sec. 6
6 SRR with unknown anomaly ratio: Otsu's method
Proposed AD framework (SRR) is significantly better than alternatives in fully-unsupervised setting, shown in wide range of scenarios. Despite being in fully--usuperviieed setting, one information reliance of SRR is the true anomaly ratio. In practice, many applications have good estimate for it from domain knowledge - c.g. banks know typical credit card fraud ratio. Yet, there could be some applications that the true anomaly ratio may also be completely unknown. this section, we propose an extension of SRR to address this scenario performing unsupervised AD without any information on anomaly ratio.
A key component would be automatic threshold selection. Some previous works (Xia et al., 2015 Beggel et al. 2019 Pang et al. 2020) studied metries like intra-class variance minimization for determining the threshold for data refinement. In a similar vein, propose integrating Otsu's method (Sezgin & Sankur 2004) into SRR to identify the threshold between normal and anomalous samples. We use this threshold for selecting the mothod is aiming to find the threshold that minimizes the intra-class variance. This is defined as the weighted sum of variances of the two classes [ Let us denote the normality scores as [ and threshold as r). Then, we pick the threshold (17) that minimizes the weighted sum of the variance (wo(1) oo(n) + w1 wi(n) o (7))
(7) instead of twice of the true anomaly ratio. The key idea of the Otsu's
where wp(n) = EN I[s;; < n)/N and w1(1) = Σ I[s 2 n)/N. ou(1) and 01 (1) are the variances cach class. The optimal threshold (17*) is determined as
17" min wn(n) x o)(1) wi (1)) (1)))
(3)
We use the twice of '7* as the hyperparameter (7) in SRR based on the sensitivity analyses described in Sec. 5.3
We evaluate the performance of SRR with Otsu's method and compare to the state-of-the-art OCC 2021 and the original SRR with the knowledge of the true anomaly ratio on MVTec datasct. Fig.
ct al.
More details be found here https://en http:/wwwwiieeia..oooii/i//////a methho
12

---

Published in Transactions on Machine Learning Research (08/2022)
Table 2: Performances with Otsu's method. SOTA OCC methods are (Sohn et al, 2021 for CIFAR-10
dataset and GOAD (Bergman & Hoshen 2019) for Thyroid dataset. We introduce 6% noise on MVTec dataset and 1.5% noise on the Thyroid dataset. Metries are (AUC/AP) for CIFAR-10 dataset and Fl score for Thyroid dataset.
Methods / Datasots
CIFAR-10
Thyroid
SOTA OCC
0.855 / 0.585
0.506
SRR with Otsu's method
0.906 / 0.703
0.623
SRR
0.910 / 0.709
0.639
demonstrates that even without true anomaly ratio, the performance of SRR can be significantly better than the statc-of-the-art OCC (Li et al. 2021) with Otsu's method. Per results, the knowledge of true anomaly ratio is definitely useful information for maximizing the performance of SRR in fully unsupervised settings. We further extend the experimental results using Otsu's method with other datasets such as CIFAR-10 and Thyroid. Similarly, Table 2 show that Otsu's method yields only slight degradation compared to SRR with true anomaly ratio; however, still significantly outperforms SOTA OCC baselines.
7 Conclusion
AD has wide range of use cases with significant importance in real world applications, from detecting security threats to the financial system to identifying faulty behaviors of manufacturing machines. A challenging and costly aspect of building an AD system is that anomalies are rare and not easily detectable by humans, yielding high complexity and cost. To this end, we propose, SRR, a canonical AD framework to enable high performance AD without any labels. SRR employs an ensemble of multiple OCCs to propose candidate anomaly samples that are refined from training, which allows more robust fitting of the anomaly decision boundaries well as better learning of data representations. SRR can be flexibly integrated with any OCC, and applied raw data of SRR on multiple tabular and image datasets from varions applications. We provide detailed analyses on the key contributing factors of SRR, which we hope to provide further guidance in AD research. Lastly, we extend SRR to the scenario of not possessing any information on the anomaly ratio. We leave some important aspects, explainability and reliability of AD, to future work.
trainable representations. We demonstrate the toate--ffthheart AD performance
References
Samet Akcay, Amir Atapour-Abarhhouii, and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In ACCV, 2018.
Arthur Asuncion and David Newman. UCT machine learning repository, 2007.
Sukarna Barua, Md Monirul Islam, Xin Yao, and Kazuyuki Murase. Mwmote- majority weighted minority oversampling technique for imbalanced data set learning, IEEE Trons on knowledge and data engineering, 26(2):405-425, 2012.
Laura Beggel, Michael Pfeiffer, and Bernd Bischl. Robust anomaly detection in images using adversarial autoencoders. Machine Learning and Knowledge Discovery in Databases, 2019.
Amanda. Berg, Jörgen Ahlberg, and Michael Felsberg- Unsupervised learning of anomaly detection from contaminated image data using simultaneous encoder training. arXiv preprint ar
2019.
Liron Bergman and Yedid Hoshen. Classifccation-assed anomaly detection for general data. In International Conference on Learning Representations, 2019.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. MVTTc AD-a comprehensive real-world dataset for unsupervised anomaly detection. In CVPR, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2022)
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning, Aduances in Neural Information Processing Systems, 32, 2019.
Gilles Blanchard, Gyemin Loe, and Clayton Scott. Semi-supervisod novelty detection. JMLR, 11:2973-300 3009, 2010.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In CLT, 1998.
Paula Branco, Luis Torgo, and Rita Ribciro. survey of prodictive modelling under imbalanced distributions. arXiv preprinl ar Xiv::5505 01 01658, 2015.
Markus M Breunig, Hans-Peter Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In International conference on management of data, 2000.
Carla E Brodley, Mark A Friedl, et al. Identifying and eliminating mislabeled training instances. In Proc National Conference Artificial Intelligence, 1996.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1 37, 2011.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321 357, 2002.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
Jeremy Elson, John R Douceur, Jon Howell, and Jared Saul. Asirra: captcha that exploits interest-aligned manual image categorization. In Proc Computer and Communications Security, 2007.
Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-36, 2004.
Jia- Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In CVPR, 2021.
Mikel Galar, Alberto Fernandez, Edurne Barrenechea, Humberto Bustince, and Francisco Herrera. A review on ensembles for the class imbalance problem: bagging., boosting, and hybrid-based approaches. IEFF Trons Systems, Mon, and Cybernetics, 42(4):463 484, 2011.
Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 9781 9791, 2018.
Nico Görnitz, Marius Kloft, Konrad Ricck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235 262, 2013.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W Tsang, and Masashi Sugiyama. Co-teaching: robust training of deep neural networks with extremely noisy labels. In N7PS, 2018.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2018.
Jae Pil Hwang, Seongkeun Park, and Funtai Kim. A new weighted approach to imbalanced data classification problem via support vector machine with quadratic cost function. Expert Systems with Applications, 38(7): 8580 8585, 2011.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rolations.
In International Conference on Learning Representations (ICLR), 2018
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Chich-Hsii Lai, Dongmian Zou, and Gilad Lerman. Robust subspace recovery layer for unsupervised anomaly detection. In International Conference on Learning Represenlations, 2019.
Longin Jan Latecki, Aleksandar Lazarevic, and Dragoljub Pokrajac. Outlier detection with kernel density functions. In International Workshop on Machine Learning and Data Mining in Pattern Recognition, 2007.
Sauchi Stephen Leo. Noisy replication in skewed binary classification. Compulational statistics & data analysis, 34(2):165 191, 2000.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Sclf-uupervised learning for anomaly detection and localization. In CVPR, 2021.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2019.
Alexander Liu, Joydeep Ghosh, and Cheryl E Martin. Generative oversampling for mining imbalanced datasets. In DMIN, pp. 66 66-72, 2007.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In JCDM, 2008.
Geoffrey McLachlan. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365-366) 1975.
Xuying Meng, Suhang Wang, Zhimin Liang, Di Yao, Jihua Zhou, and Yujun Zhang. Semi-superrised anomaly detection in dynamic communication networks. Information Sciences, 571:527-542, 2021.
Erxue Min, Jun Long, Qiang Liu, Jianjing Cui, Zhiping Cai, and Junbo Ma. Su-ids: A semi-supervised and unsupervised framework for network intrusion detection. In International Conference on Cloud Computing and Security, 2018.
Mohammadreza Mohseni, Jordan Yap, William Yolland, Arash Koochek, and Stella Atkins. Can self-training identify suspicious ugly duckling lesions? In CVPR, 2021.
Jordi Münoz-Mari, Francesca Bovolo, Luis Gómez-Chova Lorenzo Bruzzone, and Gustavo Camp-Vall Valls. Semisupervised one-class support vector machines for classification of remote sensing data. IEEE transactions on geoscience and remote sensing, 3888))3188-311 2010.
Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai. Self-trained deep ordinal regression for end-to-end video anomaly detection. In CVPR, 2020.
Jie Ren, Peter J Lin, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distributtin detection. Advances in neural information processing systems, 32, 2019.
Douglas A Reynolds. Gaussian mixture models. Encyclopedia of biometrics, 741:659 663, 2009.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, 2018.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Roberl Müller, and Marius Kloft. Deep semi-supervised anomaly detection. In ICLR, 2020.
Bernhard Schölkopf, Robert C Williamson, Alexander Smola, John Shawe- Shawe-Taylor, John C Platt, al. Support vector method for novelty detection. In NIPS, 1999.
Henry Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):3 371, 1965.
Mchmet Sezgin and Bülent Sankur. Survey over image thresholding techniques and quantitative performance evaluation. Journal of Flectronic imaging, 13(1):146-166 166116,,665 2004.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Tiang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. NeurIPS, 2020.
Kihyuk Sohn, Chun-Liang Lı, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluating representations for doep onc-class classification. In ICLR, 2021.
Hongchao Song, Zhuqing Jiang, Aidong Men, and Bo Yang. hybrid semi-supervised anomaly detection model for high-dimensional data. Computational intelligence and neuroscience, 2017.
David MJ Tax and Robert PW Duin. Support vector data description. Machine learning, 54(1):45 66, 2004.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 211-3):37-5. 1987.
Yan Xia, Xudong Cao, Fang Wen, Gang Hua, and Jian Sun. Learning discriminative reconstructions for unsupervised outlier removal. In ICCV, 2015.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: Nll:1708.07777 2017.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Bangzuo Zhang and Wanli Zuo. Learning from positive and unlabeled examples: survey. In 2008 International Symposiums on Information Processing, 2008.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders, In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 665-674, 2017.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In ICLR, 2018
16

---

Published in Transactions on Machine Learning Research (08/2022)
A Additional results
A.1 SRR on raw tabular features learned image representations
Thyroid
Arrhythmia
Average Precosion Forro
Baseline
SRR raw features
Prezition
17.
Baseline
Acourege Practsion/
153
Average Procksn
F1-score
(a) KDD-Rev
(b) Thyroid
(c) Arrhythmia
CIFAR-10 RoiNet represintation
CIFAR-10 Rotation represectation
astive representation
AUC Avvrrgge Precsoon
ANC: Auverage teeettioon
Base ine
5RR on learned represenations
SRA en leamned represerations
SAR en leamed rppresenations
ANC
AUC
Average Frecision
AUC
Average Precision
AUC
Average Precision
(d) RotNet (Golan & El- Yaniv 2018
(e) Rotation
Komodakis & Gi-(f)
Contrastive
Sohn et al.
daris
2018
2021
Figure 10: Performance of SRR (top) tabular features and (lower) learned image representations.
SRR consistently outperforms bascline and in cases (o.g., Thyroid, and Contrastive (Sohn et al. 2021 the performance improvements significant.
SRR also applicable on raw tabular features or learned image representation without representation update using data refinement. In this section, we demonstrate the performance improvements by SRR without representation update to verify the effectiveness of data refinement block of SRR for shallow OCCs.
Fig. 10 (upper) demonstrates consistent and significant performance improvements when we apply SRR on top of raw tabular features. Specifically, the Average Precision (AP) improvements are 10.2, 29.0, and 4.1 with KDD-Rev, Thyroid, and Arrhythmia tabular datasets, respectively. We also apply SRR on top of various learned image representations. As can be seen in Fig. 10 (lower), the performance improvements of SRR are consistent across various different learned image representations (without representation update). For instance, the AP improvements are 9.2, 1.0, and 12.1 with learned image representations using RotNet. (Golan)
& El- Yaniy 2018). Rotation (Komodakis & Gidaris 2018). and Contrastive (Sohn et al. 2021 respectively.
A.2 Additional baselines
Table 3: Additional experiments with extra baselines from robust AD literature. We introduce 6% noise on CIFAR-10 and KDD datasets. For Thyroid dataset, introduce 1.5% noise. Metrics (AUC/AP) for image data and Fl score for tabular data.
Methods Datasets
CIFAR-10
Thyroid
KDD
PCA Robust PCA LOF Robust AE SRR
0.299
0.836
0.377
0.893
0.338
0,873
0.636 / 0.174
0.910
0.709
0.506
0.942
17

---

Published in Transactions on Machine Learning Research (08/2022)
We add extra basclines from robust AD literature: Standard PCA
Wolld al. 1987 Robust PCA (Candès et al. 2011) and Local Outlier Factor (I.OF) (Breunig et al. 2000) for tabular data and Robust autoencoder (Robust AE) (Zhou & Paffenroth, 2017) for image data. In Table 8 the performance of PCA and LOF are highly degraded even with a small amount of anomalics in the training data. For Robust PCA and Robust AE, the performance degradation is less but still significant in comparison SRR. Overall, SRR outperforms other benchmarks in fully unsupervised settings, underlining the importance of data refinement improving the robustness with contaminated data, as the core constituent of SRR.
A.3
Convergence graphs
Representation leamnng train loss
Representation leaming train loss
Selected samples (9)
82
1.2
Selected samples (%)
82.5
80
1.0
81.5
76
74
80.5
72
0.4
79.5
68
0.2
50
100
150
200
250
250
500
750
1000
1250
1500
1750
2000
Epochs
Epochs
Figure 11: Convergence graphs of SRR with (left) MVTec dataset, (right) CIFAR-10 dataset.
Convergence is a well-known challenge in alternating training. It is more problematic in unsupervised lcarning sel tings due to the lack of validation set to evaluate the performance convergence. In the proposed framework, feature extractor loss used for the convergence criteria (if no improvement is observed in the loss for 5 epochs) and data refinement block intermittently updated with much lower frequency. The data refinement block consists of shallow one-class classifiers (not neural networks) trained on top of learned representations. Empirically, when the feature extractor's loss converges, the data refinement block is also converged. Fig. illustrate the convergence graphs of SRR with MVTec and CIFAR-10 datasets.
B Computational complexity
Note that when applying SRR on top of representation learning, the computational complexity of the representation learning part is not changed. With SRR, the additional computations come from training the ensemble models on top of learned representations. Note that we use shallow one-class classifiers (such as GDE or one-class SVM) for submodels; thus, the additional computational complexity marginal. We would like to mention that all the experiments are done on a single V100 GPU and each experiment needs most 12 hours for training (the additional training time caused by the SRR framework is average 13.1% of the total training time). The compulations of the ensemble parts can be further improved by the model parallelization.
C Implementation details for GOAD (Bergman & Hoshen, 2019
A classification-bar AD method, GOAD [Brggman & Hoshen 2019 has demonstrated strong AD perfor-
mance on tabular datasots. Unlike previous works (Golan & El-Yaniv 2018 Hendrycks et al. 2018 that formulate parametric classifier for multiple transformation classification, GOAD employs distance-based classification of multiple transformations. For the set of transformations Ti, :x D. 1,... M, the loss
18

---

Published in Transactions on Machine Learning Research (08/2022)
function of GOAD is written as in Eq. 4 with the probability defined in Eq.
5
L Bm, log ((m(Tm((2) (x))
(4)
exp( (1 (x)) cm 13)
pm Σn exp(-1lf((TT)) (x)) G. 12)
(5)
where the centers Cm 's are updated by the average feature over the training set. While shown to perform well (Bergman & Hoshen 2019), find that the distance-based formulation is necessary, and achieve
the similar performance, if not worse, to (Bergman & Hoshen
2019 using parametric classifier when
computing the probability:
exp (w fI (x)) + bm)
P(m.(Tm())) En exp (w (T)) + b,.)
(6)
The formulation Eq. G is easier to optimize than its original form in Eq. as can be fully optimized with backpropagation without alternating updates of feature extractor f and centers Cm- Once we learn representation by optimizing the loss in Eq. using Eq. G we follow two-stage one-class classification framework of (Sohn al. 2021) to construct set of Gaussian density estimation OCCs for each transformation. Finally, aggregate maximum normality scores from set of classifiers as the normality score.
In Table we summarize the implementation dctails, such network architecture or hyperparameters, and AD performance under clean training data setting that reproduces the results in (Bergman & Hoshen 2019
Table 4: The AD performance under clean only data setting of GOAD in (Bergman & Hoshen 2019 and our implementation. Our implementation demonstrates comparable, if not worse, performance to those reported in (Bergman & Hoshen 2019). Our implementation also shares most hyperparameters across datasets except the M, the number transformations, and the train steps, which are closely related to the size of training data.
Datasets
KDD
KDD-Rev
Thyroid
Arrhythmia
F-score
|Bergman & Hoshen
2019
98. 08.4
98.9+0.3
74.5+1.1
52.0+2.3
F-score (ours)
98.0+..2
95.0+u.2
75.1+2.4
54.843.2
f (feature) Optimizer
Linear(8)) (8), (ea0)
x5
Momentum SGD (momentum- 0.9)
Learning rate Batch size
0.001
64x M
L2 weight regularization
0.00003
Random projection dimension
32
M
32
256
Train steps
2
2
19


Reviews:
[{"review_id": "IQR2xO2GTo", "reviewer": "Reviewer_Gw8t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n-The paper is clear and easy to follow.\n\n-The paper includes thorough experimental results with ablation studies and sensitivity analysis on hyperparameters. It is interesting to see the performance improvement brought by retraining the representations using refined training data.\n\nCons:\n\n-The proposed method iteratively trains a distiller that excludes possible anomalies based on ensemble models and a feature extractor based on self-contrastive learning using the distilled data. Such an ensemble training nature may bring heavy computation costs.\n\n-Another concern is that the convergence of the distiller and the feature extractor in the iterative training procedures may not be guaranteed. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the unsupervised anomaly detection problem where the unlabeled training data contains an unknown portion of anomalies. As the anomalies in training data degrade the performance of one-class classification based on unsupervised anomaly detection methods, they propose an ensemble method that trains multiple one-class classifiers on disjoint subsets of training data, uses them for detecting potential anomalies, and then excludes the detected anomalies from the training data. They iteratively conduct the ensemble training and the training of a feature extractor on the filtered training data using self-supervised objectives until its convergence. Lastly, a one-class classifier is trained on the finally refined data. Experiments on various datasets and settings demonstrate the effectiveness of the proposed method.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "-It would interesting to see discussions in what situations they cannot converge. In section A.3 some convergence graphs are provided. It would be helpful if more insights and discussions on why they always converge in experiments can be given. \n\n-As the proposed method is based on ensemble models and iterative training, comparing the computational time and cost with baseline methods may also be needed.\n\n-In table 1, it would be helpful if the standard deviation results can also be reported.\n\n\n", "broader_impact_concerns": "-The proposed approach seems more general than the given algorithm. It does not depend on the ensemble training for data refinement or the self-supervised training for learning representations. Other frameworks can also be applied, and it would help make the work more impactful if such discussions can be provided.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nL7tMsbDW8", "reviewer": "Reviewer_GJKz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1). Overall, this paper is well-written, and very easy to follow, even though I am not very familiar with the anomaly detection task.\n\n2).The idea is simple and natural. Using (multiple) one-class classiers for anomaly detection seems technically sound to me. There are K detectors together with a joint self-supervised feature extractor g on K disjoint subsets of the data. The final detector is an integration of the K detectors, so that the final performance is improved.\n\n3). The effectiveness of the proposed framework is validated on top of contrastive learning-based models, which are state-of-the-art. Extensive experiments on both tabular and image datasets validate the effectiveness and superiority of the framework. Ablation studies well isolate the effects of each hyperparameter.\n\n\n**Weaknesses**:\n\n1). Although the idea is simple and natural, the methodological novelty is somewhat limited. Actually, it is well known that emsemble methods can improve the model robustness in many recognition and detection tasks.\n\n2). The real contributions of this paper should be further clarified. This is not the first work that focuses on the problem setting of unsupervised anomaly detection, but in the abstract and Fig. 1, I think the authors discuss too much about the concept of this problem setting.\n\n3). As we know, the iterations of K detectors and feature extractors and alternative. How can we ensure that this process will certainly converge? It would be nice if the authors could provide some theoretical discussions or insights.\n\n\n**In Summary**:\n\nOverall, although the technical novelty is not too high, I still think this is a good paper and I would like to give an accept for it. If it is finally accepted, please consider my questions listed in the weakness points as well as the suggestions in the requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a self-supervised ensemble method for unsupervised anomaly detection. In contrast to the previous unsupervised model-centric method, the proposed method is data-centricc, and this allows more robust fitting of the anomaly decision boundaries and also better learning of data representations. Experiments on tabular and image datasets validate that the proposed  approach is more robust at high anomaly contamination ratios than respective state-of-the-art single detectors.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "As I listed in the weaknesses, it would be nice if the authors could further list their contributions in the introduction. Moreover, I think the authors should slightly reduce the discussion of the unsupervised problem setting.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ASmBHUQgI5", "reviewer": "Reviewer_CDcs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Unsupervised anomaly detection is a challenging task.\n2. The author provided empirical evidence that this training scheme based on ensemble and self-supervised learning may be helpful.\n3. The author studied the problem of unknown anomaly ratios and introduced a method for such a case.\n\n## Weaknesses\n\nThe biggest problem is that, for almost all parts, the author only stated what they did but not why they did so. Since all components are not novel (self-supervised learning, one-class classifier, ensemble, etc.), it is crucial that all design choices are supported by a reasonable explanation, theoretical justification, or empirical evidence. For example:\n\n- The author proposed to use an ensemble of one-class classifiers. It is well known that bagging could improve the expressiveness of models and enhance weak learners. But what if we only use a single classifier and train it along with the feature extractor? What will the problem be? In other words, why is an ensemble preferable or necessary? If we choose ensemble, why do we choose unanimity voting over other methods such as majority voting? These issues were not well explained, referred to a reliable source, or empirically supported.\n- The author chose to train the feature extractor and one-class classifiers simultaneously. However, the benefit is unclear. Since the number of positive instances is small (by definition of anomaly detection), it seems no problem to train a feature extractor with self-supervised learning first with all training instances. We can even use pre-trained ones if available. The author did show some empirical evidence from an ablation study, which is, however, based on a single setting and a single trial. The statistical significance may not be high enough.\n- Another design choice is the last one-class classifier. The explanation \"the proposed version of SRR (with an additional final OCC) outperforms the version with the final OCC, which is attributed to the fact that while fitting the individual OCC models in the ensemble, we do not exclude the possible anomaly samples for diversity of the trained submodels, so the anomaly decision boundaries can be fitted robustly\" is not understandable.\n\nAnother issue is the experimental setting based on semi-synthetic datasets. It is questionable if treating one class as normal and others as abnormal from the CIFAR-10, Fashion MNIST, and Asirra datasets reflects actual anomaly detection applications in the real world.\n\nThere are other expression issues, strawman arguments, and typos, which are listed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studied the unsupervised anomaly detection problem and proposed a method based on self-supervised learning, an ensemble of one-class classifiers, and threshold-based sample selection.\nThe proposed method was validated empirically on the semi-synthetic image (CIFAR-10, Fashion MNIST, Asirra, MVTec AD) and tabular datasets.\n\nConcretely, the proposed method, referred to as \"self-supervise, refine, repeat\" (SRR), first partitions the training set of normal and abnormal instances into subsets and trains multiple one-class classifiers;\nThen, the abnormal training instances are detected based on a predefined and fixed thresholding hyperparameter; Meanwhile, the feature extractor is trained only on selected normal training instances via self-supervised learning; Finally, a single one-class classifier is trained on selected normal training instances.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- What are \"adversarial outcomes at intensive care units\"?\n- \"They all depend on some labeled data\", and \"Most prior unsupervised AD works assume that the data contain entirely negative samples\"... are strawman arguments and not justifiable.\n- Vacuous intensifiers like \"impressive performance\", \"quite sub-optimal\", \"holistic approach,\" and \"fundamental challenges\" are not scientific writing and are debatable.\n- In my opinion, this work is not data-centric: there is only one application -- anomaly detection, and the main contribution is still model and training method design.\n- Either all \"percentile\" need to be replaced by \"quantile\", or $\\gamma$ in Eq. (2) needs to be multiplied by $100$.\n- \"We proposed that it is beneficial to...\" sounds unnatural.\n- The criterion of convergence is not given.\n- Figure 4: F-score?\n- Otsu's method: why weighted?\n", "broader_impact_concerns": "I do not foresee such concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All 3 reviewers recommended weak accept after the rebuttal because the authors have successfully addressed most concerns. I think the paper can be accepted for publication at TMLR after a minor revision.

There are 2 concerns not yet well addressed:

> Another issue is the experimental setting based on semi-synthetic datasets. It is questionable if treating one class as normal and others as abnormal from the CIFAR-10, Fashion MNIST, and Asirra datasets reflects actual anomaly detection applications in the real world.

> The criterion of convergence is not given.

Regarding the first concern, the argument that other papers also did so is rather not a good answer (at least to me). Please be scientific and consider whether other papers are correct or not on this point. What is missing from other papers and this paper is what underlying assumption you made in unsupervised learning without any fitting target. Specifically, do you need any kind of the cluster assumption, the manifold assumption, or in general the low-density separation assumption for unsupervised modeling of the given data? For your proposed method to work, do you need the normal/anomalous data to form clusters or low-density regions? Real-world anomalous data may not possess any cluster structure like "other classes" in multi-class benchmark datasets. This is what our reviewer was worried about and what you should carefully discuss about in the camera-ready paper.

Regarding the second concern shared by all 3 reviewers, I think it's fine to mention the difficulty of such analysis given the currently available tools in math/theory. However, please be aware that when the loss converges, the model being trained may not converge, whenever your optimization problem is non-convex. This may be the case if the model jumps from a solution to another equally good solution at every update. Thus, a converged loss only means your optimization achieves an equilibrium of training but not a converged model itself. Please remove the misleading claims related to your Answer 12 in the camera-ready paper. Thanks!
Decision: Accept (poster)