Automatic Unsupervised Outlier Model Selection

1 Introduction . The lack of a universal learning model that performs well on all problem instances is well recognized [ 53 ] . Therefore , effort has been directed toward building a toolbox of various models and algorithms , which has given rise to the problem of algorithm selection and hyperparameter tuning ( i.e. , model selection ) . The same problem applies to outlier detection ( OD ) ; a long list of detectors has been developed in the last decades [ 2 ] , with no universal “ winners ” [ 8 ] . In supervised learning , model selection can be done via performance evaluation of each trained model on labeled hold-out data . In contrast , unsupervised OD does not have access to any labels , nor is there a universal objective function that could guide model selection ( cf . clustering where a loss function enables model comparison ) . Unsupervised model selection for OD is challenging exactly because both model evaluation and comparison are not feasible—which renders any trial-and-error techniques like grid search or iterative strategies like Bayesian hyperparameter optimization [ 57 ] inapplicable . Consequently , there has been no principled work on unsupervised outlier model selection—rather , the choice of a model for a new task ( or dataset ) remains “ a black art ” . A typical approach is to use popular OD algorithms , like LOF [ 6 ] and iForest [ 31 ] ( often with default hyperparameters ) which are shown to be competitive on average on many benchmark datasets . However , as noted earlier , none of 1Code available at URL : https : //github.com/yzhao062/UOMS 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . these methods can universally outperform others on all tasks [ 8 ] . We argue that model selection is exactly how one can “ break the performance ceiling ” for OD . In this work , we tackle the unsupervised outlier model selection ( UOMS ) problem systematically . To that end , we introduce ( to the best of our knowledge ) the first UOMS approach that selects an effective model to be employed on a new detection task without any model evaluation ( using labels ) or model comparison ( via loss criteria ) . Our proposed method , called METAOD , is based on meta-learning , and stands on the prior performances of a large collection of existing detection models on an extensive corpora of historical outlier detection benchmark datasets . In a nutshell , the idea is to estimate a candidate model ’ s performance on the new task ( with no labels ) based on its prior performance on similar historical tasks . We remark that METAOD is strictly a model selection technique – that picks one model ( a detector and its associated hyperparameter ( s ) ) from a pool of ( existing ) candidate models – and not yet-another outlier detection algorithm itself . In leveraging meta-learning , we establish a connection between the UOMS problem and the cold-start problem in collaborative filtering ( CF ) , where the new task in UOMS is akin to a new user in CF ( with no available evaluations , hence cold-start ) and the model space is analogous to the item-set . Differently , OD necessitates the identification of a single best model ( i.e. , top-1 rank ) , whereas CF typically operates in a top-k setting . In CF , future recommendations can be improved based on user feedback which is not applicable to OD . Moreover , METAOD requires the effective learning of task similarities based on characteristic dataset features ( namely , meta-features ) that capture the outlying properties within a dataset , whereas user features ( location , age , etc . ) in CF may be readily available . In summary , the key contributions of this work include the following : • First Approach to Unsupervised Outlier Model Selection : We propose METAOD , ( to our knowledge ) the first effort on unsupervised model selection for OD tasks . Notably , given a new dataset ( i.e. , at test time ) , it does not rely on any ground-truth labels for model evaluation or any loss or heuristic criterion for model comparison . METAOD stands on meta-learning in principle , and historical collections of outlier models and benchmark datasets in practice . • Problem Formulation : We establish a correspondence between UOMS and CF under cold-start , where the new task “ better likes ” a model that performs better on similar historical tasks . • Specialized Meta-features : We design novel meta-features to capture the outlying characteristics in a dataset toward effectively quantifying task similarity specifically among OD tasks . • Effectiveness and Efficiency : Through extensive experiments on two benchmark testbeds that we have constructed , we show that selecting a model by METAOD for each given task significantly outperforms always using a popular model like iForest , as well as other possible meta-learning approaches that we tailored for UOMS . Moreover , METAOD incurs negligible run-time overhead ( < 1 second ) at test time . • Open-source Platform : We open-source1 METAOD and our meta-learning database for the community to use it for UOMS in practice , and to extend it with new datasets and models . We expect the growth of the database would make meta-learning based approaches , like METAOD , more powerful and also help foster further research on this new direction to an important problem . 2 Related Work . 2.1 Model Selection for Outlier Detection ( OD ) . Most outlier mining work have focused on developing new , better methods for detection on different types of data [ 2 ] . In comparison , there are only a few work on the outlier model selection problem –which detector and hyperparameter ( s ) to use on a new task– all of which require some labeled data . Recent work include AutoOD [ 29 ] that focuses on automatic neural architecture search , however , it is limited to deep autoencoder based detection , and more importantly it relies on hold-out labeled data for evaluation . Similarly , PyODDS [ 30 ] and TODS [ 25 ] both require ground truth labels . To our knowledge , there is no existing work on the general unsupervised outlier model selection ( UOMS ) problem , which is considerably more challenging , as ( 1 ) model evaluation is infeasible due to the lack of any ground truth labels , and ( 2 ) model comparison of different heterogeneous detectors is infeasible due to the lack of a universal OD loss function . We note that some OD methods do have a loss function ; e.g . auto-encoders [ 9 , 63 ] aim to minimize reconstruction error for modeling the inliers , and one-class classification ( OCSVM [ 38 ] , SVDD [ 46 ] , etc . ) aims to maximize margin to the origin or minimize the radius of a data-enclosing hyperball . There is also work on model selection for one-class models [ 7 , 54 ] , however , those are limited to this specific family of methods and do not apply in the general case . Our proposed METAOD is not limited to one specific model family , but can select among any ( heterogeneous ) set of detectors . 2.2 Model Selection in ML , AutoML , Meta-Learning . Model selection refers to the process of algorithm selection and/or hyperparameter optimization ( HO ) . With the advent of complex ( e.g . deep ) models , HO in high dimensions has become impractical to be human-powered [ 59 ] . As such , automating ML pipelines has seen a surge of attention [ 18 ] . Meta-learning has been a key contributor to the AutoML effort [ 41 , 49 , 58 ] . Supervised Model Selection : Most existing work focus on the supervised setting , and use hold-out data with labels . Randomized [ 3 ] , bandit-based [ 27 ] , and Bayesian optimization ( BO ) techniques [ 40 ] are various state-of-the-art ( SOTA ) approaches to HO . Specifically sequential model-based BO [ 19 , 22 ] evaluates hold-out performance at various initial hyperparameter configurations ( HC ) , where a ( smooth ) surrogate function is fit to the resulting 〈HC , performance〉 pairs , which is then used to strategically query other HCs , e.g. , via hyper-gradient based search [ 15 ] . Meta-learning has also been employed [ 13 , 52 ] , e.g. , to find promising initialization for ( i.e . warm-starting ) BO [ 14 , 51 ] . Note that all of these approaches rely on multiple model evaluations ( i.e. , performance queries ) for various HCs , and hence can not be applied to the unsupervised outlier model selection problem . Unsupervised Model Selection : Unsupervised ML tasks ( e.g. , clustering ) poses additional challenges for model selection [ 12 , 47 ] . Nonetheless , those exhibit established objective criteria that enable model comparison , unlike OD . For example , BO methods still apply where the surrogate can be trained on 〈HC , objective value〉 pairs , for which meta-learning can provide favorable priors . Task-independent meta-learning [ 1 ] , that simply identifies the globally best model on historical tasks , applies to the unsupervised setting and hence OD . This can be refined by identifying the best model on not all , but similar tasks , where task similarity is measured in the meta-feature space via clustering [ 23 ] or nearest neighbors [ 33 ] . This type of similarity-based recommendations points to a connection between algorithm selection and collaborative filtering ( CF ) , first recognized by Stern et al . [ 44 ] . The most related to UOMS is CF under cold start , where evaluations are not-available ( in our case , infeasible ) for a new user ( in our case , task ) . There have been a number of work using meta-learning for the cold-start recommendation problem [ 4 , 26 , 50 ] , and vice versa , using CF solutions for ML algorithm selection [ 32 , 56 ] . We tailor these to UOMS and compare to METAOD in the experiments . 3 Unsupervised Outlier Model Selection via Meta-learning . 3.1 Problem Statement . We consider the model selection problem for unsupervised outlier detection , which we refer to as UOMS ( unsupervised outlier model selection ) hereafter . Given a new dataset , without any labels , the problem is to select both ( i ) a detector/algorithm and ( ii ) its associated hyperparameter ( s ) ( HP ) . The former is a discrete choice , given the finite set of existing detection algorithms . The latter is continuous , and hence induces infinitely many candidate models . Under certain assumptions , such as performance changing smoothly in the HP space , a HP configuration can be selected iteratively based on evaluations on several other carefully-chosen configurations . Importantly however , OD is not amenable for such iterative search over models—evaluations are not possible due to the lack of labels and absence of a universal objective criterion . The selection of a model , therefore , is to be done without building or evaluating any model on the new dataset . Given this constraint , we discretize the HP space for each candidate detector to make the search space tractable , which induces a finite pool of models denoted M = { M1 , . . . , Mm } . Each model M ∈M can be seen as a { detector , configuration } pair , where the configuration depicts a specific set of values for the detector ’ s HP ( s ) . ( See Appendix A for details . ) Then , the UOMS problem is stated as follows : Problem 1 ( Unsupervised Outlier Model Selection ( UOMS ) ) Given a new input dataset ( i.e. , detection task ) Dtest = ( Xtest ) without any labels , Select a model M ∈M to employ on Xtest . 3.2 Proposed METAOD In this work we consider the UOMS problem and propose a meta-learning based solution , leveraging past experience on historical detection tasks . As such , our METAOD relies on • a collection of historical outlier detection datasets Dtrain = { D1 , . . . , Dn } , namely , a meta-train database with ground truth labels , i.e. , { Di = ( Xi , yi ) } ni=1 , and • the historical performances of the pool of candidate models , M , on the meta-train datasets . We denote by P ∈ Rn×m the performance matrix , where Pij corresponds to the j-th model Mj ’ s performance2 on the i-th meta-train dataset Di . Note that model performance can be evaluated on the historical meta-train datasets as they contain ground truth labels , which however is not the case for any newcoming task at test time . Our METAOD consists of two-phases : offline ( meta- ) training of the meta-learner on Dtrain , and online prediction that enables unsupervised model selection at test time for Dtest . Arguably , the running time of the offline phase is not critical . In contrast , model selection for a newcoming task should incur small run-time overhead , as it precedes the actual building of the selected OD model . Fig . 1 summarizes the process and the major components of METAOD , where we highlight the components transferred from offline ( meta-learning ) to online stage ( model selection ) in blue . We also provide the detailed steps of METAOD in pseudo-code , for both meta-training ( offline ) and model selection ( online ) , in Appendix D Algo . 1 .

Summary: This paper presents a data-driven approach to model selection for unsupervised outlier detection. The proposed method, METAOD takes a large number of training datasets with outlier labels and a pool of outlier detection models (methods plus their parameters) and uses a matrix factorization method to learn a model performance predictor, which, for given a test dataset, can predict/select a high performing model from the model pool without requiring test time model evaluation.  The performance of METAOD is evaluated with two implementations with the same pool of 300+ models (8 distinct outlier detection methods with different parameter configurations) but two different groups of training datasets.