Self-Supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) , the task of distinguishing anomalies from normal data , plays a crucial role in many real-world applications such as detecting faulty products using visual sensors in manufacturing , fraudulent behaviors at credit card transactions , or adversarial outcomes at intensive care units . AD has been considered under various settings based on the availability of negative ( normal ) and positive ( anomalous ) data and their labels at training , as overviewed in Sec . 2 . Each application scenario is dominated by different challenges . When entire positive and negative data are available along with their labels ( Fig . 1a ) , the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in label distributions [ 4 , 16 , 20 , 27 , 32 , 35 ] . When only negative labeled data are available ( Fig . 1b ) , the problem is ‘ one-class classification ’ [ 23 , 26 , 33 , 45 , 47 , 51 , 53 ] . Various works have also extended approaches designed for these to settings with additional unlabeled data ( Fig . 1c , d , e ) [ 10 , 18 , 24 , 46 , 58 ] in a semi-supervised setting . While there exist many prior works in these settings , they all depend on some labeled data , which is not desirable in all application scenarios . Unsupervised AD , on the other hand , poses unique challenges in the absence of any labeled data information , and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal . For example , some recent studies [ 7 , 60 ] have applied one-class classifiers ( OCCs ) that are known to yield impressive performance when trained on negative samples [ 7 , 23 , 26 , 33 , 51 ] to unsupervised AD , but their performance for unsupervised AD has been quite sub-optimal . Fig . 2 illustrates this , showing the unsupervised AD performance of state-of-the-art Deep OCCs [ 51 ] with different anomaly ratios in unlabeled training data – the average precision significantly drops even when a small portion ( 2 % ) of training data is contaminated with anomalies . Our framework SRR ( Self-supervise , Refine , Repeat ) , overviewed in Fig . 3 , brings a novel approach to unsupervised AD with the principles of self-supervised learning without labels and iterative data refinement based on the agreement of OCC outputs . We propose to improve the state-of-the-art performance of OCCs , e.g . [ 33 , 51 ] , by refining the unlabeled training data so as to address the fundamental challenges elaborated above . SRR iteratively trains deep representations using refined data while improving the refinement of unlabeled data by excluding potentially-positive ( anomalous ) samples . For the data refinement process , we employ an ensemble of OCCs , each of which is trained on a disjoint subset of unlabeled training data . The samples are declared as normal if there is a consensus between all the OCCs . The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting . Most prior unsupervised AD works [ 23 , 26 , 33 , 51 ] assume that the data contains entirely negative samples , which makes them not truly unsupervised as they require having humans to do the data filtering . Similar to ours , some prior unsupervised AD works [ 7 , 45 , 60 ] considered evaluating on an unsupervised setting where there exist a small percentage of anomalous samples in the training data , i.e . operating in ‘ truly ’ unsupervised setting without having the need for humans to do any filtering in the training data . However , these methods often suffered from significant performance degradation as the ratio of anomalous sample ratio has increased ( see Sec . 4.2 ) . We would like to highlight that our method distinguishes from the literature by bringing a data-centric approach ( refining the unlabeled data ) to unsupervised anomaly detection beyond the model-centric approaches ( improving the model itself ) . Our framework SRR aims to provide robustness in performance as the anomalous sample ratio increases , as shown in Fig . 2 . We conduct extensive experiments across various datasets from different domains , including semantic AD ( CIFAR-10 [ 29 ] , Dog-vs-Cat [ 19 ] ) , real-world manufacturing visual AD use case ( MVTec [ 8 ] ) , and tabular AD benchmarks . We consider methods with both shallow [ 36 , 47 ] and deep [ 7 , 33 , 51 ] models . We evaluate models at different anomaly ratios of unlabeled training data and show that SRR significantly boosts performance . For example , in Fig . 2 , SRR improves more than 15.0 average precision ( AP ) with a 10 % anomaly ratio compared to a state-of-theart one-class contrastive representation model [ 51 ] on CIFAR-10 . Similarly , on MVTec SRR retains a strong performance , dropping less than 1.0 AUC with 10 % anomaly ratio , while the best existing OCC [ 33 ] drops more than 6.0 AUC . We further investigate the efficacy of our design choices , such as the number of ensemble classifiers , thresholds , and refinement of deep representations via ablation studies . 2 RELATED WORK . There are various existing works under the different settings described in Fig . 1 : The positive + negative setting is often considered as a supervised binary classification problem . The challenge arises due to the imbalance in label distributions as positive ( anomalous ) samples are rare . As summarized in [ 12 ] , to address this , over-/under-sampling [ 16 , 20 ] , weighted optimization [ 4 , 27 ] , synthesizing data of minority classes [ 32 , 35 ] , and hybrid methods [ 2 , 22 ] have been studied . The negative setting is often converted to a one-class classification problem , with the goal of finding a decision boundary that includes as many one-class samples as possible . Shallow models for this setting include one-class support vector machines [ 47 ] ( OC-SVM ) , support vector data description [ 53 ] ( SVDD ) , kernel density estimation ( KDE ) [ 31 ] , and Gaussian density estimation ( GDE ) [ 44 ] . There are also auto-encoder based models [ 59 ] that treat the reconstruction error as the anomaly score . Deep learning based OCCs have been developed , such as Deep OCC [ 45 ] , geometric transformation [ 23 ] , or outlier exposure [ 26 ] . Noting the degeneracy or inconsistency of learning objectives of existing end-to-end trainable Deep OCCs , [ 51 ] proposed a deep representation OCC , a two-stage framework that learns self-supervised representations [ 17 , 28 ] followed by shallow OCCs . That work was extended for texture anomaly localization with CutPaste [ 33 ] . The robustness to very low anomaly ratios of these methods under the unsupervised setting was explored in [ 7 , 60 ] . The semi-supervised setting is defined as utilizing a small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data . Depending on which labeled samples are given , this setting can be split into three sub-categories . When only some positive/negative labeled samples are provided , we denote that as a PU/NU setting . Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given [ 1 , 41 , 52 ] . The PNU setting is a more general semi-supervised setting where subsets of both positive and negative labeled samples are given . Deep SAD [ 46 ] and SU-IDS [ 39 ] are included in this category . We show the significant outperformance of our proposed SRR framework compared to Deep SAD , for multiple benchmark datasets when not using any labeled data ( see Sec . 4.2 ) . The unlabeled setting has received relatively less attention despite its significance in automating machine learning . The popular methods for this setting include isolation forest [ 36 ] and local outlier factor [ 13 ] . However , they are difficult to scale , and less compatible with recent advances in representation learning . While OCCs , such as OC-SVM , SVDD , or their deep counterparts , apply to unlabeled settings by assuming the data is all negative , and the robustness of those methods has also been demonstrated in part [ 7 , 60 ] , in practice we observe a significant performance drop with a high anomaly ratio , shown in Fig . 2 . In contrast , our proposed framework is able to maintain high performance across anomaly ratios . Data refinement has been applied to AD in some prior works . [ 5 , 42 ] generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent AD performances in unsupervised settings . [ 6 , 30 , 40 , 55 , 59 ] used the reconstruction errors of the auto-encoder as an indicator for removing possible anomalies . [ 21 ] and [ 38 ] used data refinement for AD in supervised and semi-supervised settings . Additional discussions can be found in Appendix A.9 . Self-training [ 37 , 48 ] is an iterative training mechanism using predicted pseudo labels as targets for model training . It has regained popularity recently with its successful results in semi-supervised image classification [ 9 , 50 , 57 ] . To improve the quality of pseudo labels , employment of an ensemble of classifiers has also been studied . [ 14 ] trains an ensemble of classifiers with different classification methods to make a consensus for noisy label verification . Co-training [ 11 ] trains multiple classifiers , each of which is trained on the distinct views , to supervise other classifiers . Co-teaching [ 25 ] and DivideMix [ 34 ] share a similar idea in that they both train multiple deep networks on separate data batches to learn different decision boundaries , thus becoming useful for noisy label verification . While sharing a similarity , the proposed framework has clear differences from the previous works . SRR performs iterative training with data refinement ( with robust ensemble methods ) and self-supervised learning for unsupervised learning of an anomaly detector . 3 PROPOSED FRAMEWORK . Self-supervise , Refine , and Repeat ( SRR ) is an iterative training framework , where we refine the data ( Sec . 3.1 ) and update the representation with the refined data ( Sec . 3.2 ) , followed by OCCs on refined representations . Fig . 3 overviews the framework and Algorithm 1 provides the pseudo code . Notation . We denote the training data as D= { xi } Ni=1 where xi ∈X and N is the number of training samples . yi ∈ { 0 , 1 } is the corresponding label to xi , where 0 denotes normal ( negative ) and 1 denotes anomaly ( positive ) . Note that labels are not provided in the unsupervised setting . Let us denote a feature extractor as g : X → Z . g may include any data preprocessing functions , an identity function ( if raw data is directly used for one-class classification ) , and learned or learnable representation extractors such as deep neural networks . Let us define an OCC as f : Z → [ −∞ , ∞ ] that outputs anomaly scores given the input features g ( x ) . The higher the score f ( g ( x ) ) , the more anomalous the sample x is . The binary anomaly prediction is made after thresholding : 1 ( f ( g ( x ) ) ≥ η ) .

Summary: This paper proposes an ensemble approach, called SRR (Self-supervise, Refine, Repeat), for robust unsupervised anomaly detection. The proposed approach trains an ensemble of K detectors together with a joint self-supervised feature extractor g on K disjoint subsets of the data. The ensemble is then used to filter the training data, keeping only the data points that are collectively deemed normal by the K detectors. This training-data filtering process is repeated until the self-supervised feature extractor has converged. A final detector is then trained using the refined data and converged self-supervised feature extractor. Experiments on tabular and image datasets are presented which show that the proposed ensemble approach is more robust at high anomaly contamination ratios than respective state-of-the-art single detectors.