Self-Supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) , the task of distinguishing anomalies from normal data , plays a crucial role in many real-world applications such as detecting faulty products using visual sensors in manufacturing , fraudulent behaviors at credit card transactions , or adversarial outcomes at intensive care units . AD has been considered under various settings based on the availability of negative ( normal ) and positive ( anomalous ) data and their labels at training , as overviewed in Sec . 2 . Each application scenario is dominated by different challenges . When entire positive and negative data are available along with their labels ( Fig . 1a ) , the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in label distributions [ 4 , 16 , 20 , 27 , 32 , 35 ] . When only negative labeled data are available ( Fig . 1b ) , the problem is ‘ one-class classification ’ [ 23 , 26 , 33 , 45 , 47 , 51 , 53 ] . Various works have also extended approaches designed for these to settings with additional unlabeled data ( Fig . 1c , d , e ) [ 10 , 18 , 24 , 46 , 58 ] in a semi-supervised setting . While there exist many prior works in these settings , they all depend on some labeled data , which is not desirable in all application scenarios . Unsupervised AD , on the other hand , poses unique challenges in the absence of any labeled data information , and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal . For example , some recent studies [ 7 , 60 ] have applied one-class classifiers ( OCCs ) that are known to yield impressive performance when trained on negative samples [ 7 , 23 , 26 , 33 , 51 ] to unsupervised AD , but their performance for unsupervised AD has been quite sub-optimal . Fig . 2 illustrates this , showing the unsupervised AD performance of state-of-the-art Deep OCCs [ 51 ] with different anomaly ratios in unlabeled training data – the average precision significantly drops even when a small portion ( 2 % ) of training data is contaminated with anomalies . Our framework SRR ( Self-supervise , Refine , Repeat ) , overviewed in Fig . 3 , brings a novel approach to unsupervised AD with the principles of self-supervised learning without labels and iterative data refinement based on the agreement of OCC outputs . We propose to improve the state-of-the-art performance of OCCs , e.g . [ 33 , 51 ] , by refining the unlabeled training data so as to address the fundamental challenges elaborated above . SRR iteratively trains deep representations using refined data while improving the refinement of unlabeled data by excluding potentially-positive ( anomalous ) samples . For the data refinement process , we employ an ensemble of OCCs , each of which is trained on a disjoint subset of unlabeled training data . The samples are declared as normal if there is a consensus between all the OCCs . The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting . Most prior unsupervised AD works [ 23 , 26 , 33 , 51 ] assume that the data contains entirely negative samples , which makes them not truly unsupervised as they require having humans to do the data filtering . Similar to ours , some prior unsupervised AD works [ 7 , 45 , 60 ] considered evaluating on an unsupervised setting where there exist a small percentage of anomalous samples in the training data , i.e . operating in ‘ truly ’ unsupervised setting without having the need for humans to do any filtering in the training data . However , these methods often suffered from significant performance degradation as the ratio of anomalous sample ratio has increased ( see Sec . 4.2 ) . We would like to highlight that our method distinguishes from the literature by bringing a data-centric approach ( refining the unlabeled data ) to unsupervised anomaly detection beyond the model-centric approaches ( improving the model itself ) . Our framework SRR aims to provide robustness in performance as the anomalous sample ratio increases , as shown in Fig . 2 . We conduct extensive experiments across various datasets from different domains , including semantic AD ( CIFAR-10 [ 29 ] , Dog-vs-Cat [ 19 ] ) , real-world manufacturing visual AD use case ( MVTec [ 8 ] ) , and tabular AD benchmarks . We consider methods with both shallow [ 36 , 47 ] and deep [ 7 , 33 , 51 ] models . We evaluate models at different anomaly ratios of unlabeled training data and show that SRR significantly boosts performance . For example , in Fig . 2 , SRR improves more than 15.0 average precision ( AP ) with a 10 % anomaly ratio compared to a state-of-theart one-class contrastive representation model [ 51 ] on CIFAR-10 . Similarly , on MVTec SRR retains a strong performance , dropping less than 1.0 AUC with 10 % anomaly ratio , while the best existing OCC [ 33 ] drops more than 6.0 AUC . We further investigate the efficacy of our design choices , such as the number of ensemble classifiers , thresholds , and refinement of deep representations via ablation studies . 2 RELATED WORK . There are various existing works under the different settings described in Fig . 1 : The positive + negative setting is often considered as a supervised binary classification problem . The challenge arises due to the imbalance in label distributions as positive ( anomalous ) samples are rare . As summarized in [ 12 ] , to address this , over-/under-sampling [ 16 , 20 ] , weighted optimization [ 4 , 27 ] , synthesizing data of minority classes [ 32 , 35 ] , and hybrid methods [ 2 , 22 ] have been studied . The negative setting is often converted to a one-class classification problem , with the goal of finding a decision boundary that includes as many one-class samples as possible . Shallow models for this setting include one-class support vector machines [ 47 ] ( OC-SVM ) , support vector data description [ 53 ] ( SVDD ) , kernel density estimation ( KDE ) [ 31 ] , and Gaussian density estimation ( GDE ) [ 44 ] . There are also auto-encoder based models [ 59 ] that treat the reconstruction error as the anomaly score . Deep learning based OCCs have been developed , such as Deep OCC [ 45 ] , geometric transformation [ 23 ] , or outlier exposure [ 26 ] . Noting the degeneracy or inconsistency of learning objectives of existing end-to-end trainable Deep OCCs , [ 51 ] proposed a deep representation OCC , a two-stage framework that learns self-supervised representations [ 17 , 28 ] followed by shallow OCCs . That work was extended for texture anomaly localization with CutPaste [ 33 ] . The robustness to very low anomaly ratios of these methods under the unsupervised setting was explored in [ 7 , 60 ] . The semi-supervised setting is defined as utilizing a small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data . Depending on which labeled samples are given , this setting can be split into three sub-categories . When only some positive/negative labeled samples are provided , we denote that as a PU/NU setting . Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given [ 1 , 41 , 52 ] . The PNU setting is a more general semi-supervised setting where subsets of both positive and negative labeled samples are given . Deep SAD [ 46 ] and SU-IDS [ 39 ] are included in this category . We show the significant outperformance of our proposed SRR framework compared to Deep SAD , for multiple benchmark datasets when not using any labeled data ( see Sec . 4.2 ) . The unlabeled setting has received relatively less attention despite its significance in automating machine learning . The popular methods for this setting include isolation forest [ 36 ] and local outlier factor [ 13 ] . However , they are difficult to scale , and less compatible with recent advances in representation learning . While OCCs , such as OC-SVM , SVDD , or their deep counterparts , apply to unlabeled settings by assuming the data is all negative , and the robustness of those methods has also been demonstrated in part [ 7 , 60 ] , in practice we observe a significant performance drop with a high anomaly ratio , shown in Fig . 2 . In contrast , our proposed framework is able to maintain high performance across anomaly ratios . Data refinement has been applied to AD in some prior works . [ 5 , 42 ] generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent AD performances in unsupervised settings . [ 6 , 30 , 40 , 55 , 59 ] used the reconstruction errors of the auto-encoder as an indicator for removing possible anomalies . [ 21 ] and [ 38 ] used data refinement for AD in supervised and semi-supervised settings . Additional discussions can be found in Appendix A.9 . Self-training [ 37 , 48 ] is an iterative training mechanism using predicted pseudo labels as targets for model training . It has regained popularity recently with its successful results in semi-supervised image classification [ 9 , 50 , 57 ] . To improve the quality of pseudo labels , employment of an ensemble of classifiers has also been studied . [ 14 ] trains an ensemble of classifiers with different classification methods to make a consensus for noisy label verification . Co-training [ 11 ] trains multiple classifiers , each of which is trained on the distinct views , to supervise other classifiers . Co-teaching [ 25 ] and DivideMix [ 34 ] share a similar idea in that they both train multiple deep networks on separate data batches to learn different decision boundaries , thus becoming useful for noisy label verification . While sharing a similarity , the proposed framework has clear differences from the previous works . SRR performs iterative training with data refinement ( with robust ensemble methods ) and self-supervised learning for unsupervised learning of an anomaly detector . 3 PROPOSED FRAMEWORK . Self-supervise , Refine , and Repeat ( SRR ) is an iterative training framework , where we refine the data ( Sec . 3.1 ) and update the representation with the refined data ( Sec . 3.2 ) , followed by OCCs on refined representations . Fig . 3 overviews the framework and Algorithm 1 provides the pseudo code . Notation . We denote the training data as D= { xi } Ni=1 where xi ∈X and N is the number of training samples . yi ∈ { 0 , 1 } is the corresponding label to xi , where 0 denotes normal ( negative ) and 1 denotes anomaly ( positive ) . Note that labels are not provided in the unsupervised setting . Let us denote a feature extractor as g : X → Z . g may include any data preprocessing functions , an identity function ( if raw data is directly used for one-class classification ) , and learned or learnable representation extractors such as deep neural networks . Let us define an OCC as f : Z → [ −∞ , ∞ ] that outputs anomaly scores given the input features g ( x ) . The higher the score f ( g ( x ) ) , the more anomalous the sample x is . The binary anomaly prediction is made after thresholding : 1 ( f ( g ( x ) ) ≥ η ) .

Summary: The paper tackles an unsupervised anomaly detection problem where the training set contains an unknown portion of anomalies. When anomalies are contained in the training set, it is known that classical AD approaches' performance degrades. The idea is to filter out potential anomaly samples (data refinement) by ensemble model. Each model in the ensemble is trained on a disjoint set of training data and then used as a classifier to determine potential anomalies. Then the data refinement process uses a hard assignment excluding anomalies from the training set. Refinement and ensemble training is repeated iteratively until convergence. The proposed framework is validated on the four tabular datasets and four image datasets.