Automatic Unsupervised Outlier Model Selection
Authors: Yue Zhao
Ryan Rossi
Leman Akoglu
Venue: NeurIPS 2021 Poster (2021)
Keywords: outlier detection
anomaly detection
unsupervised learning
model selection
automated machine learning

Abstract: Given an unsupervised outlier detection task on a new dataset, how can we automatically select a good outlier detection algorithm and its hyperparameter(s) (collectively called a model)? In this work, we tackle the unsupervised outlier model selection (UOMS) problem, and propose MetaOD, a principled, data-driven approach to UOMS based on meta-learning. The UOMS problem is notoriously challenging, as compared to model selection for classification and clustering, since (i) model evaluation is infeasible due to the lack of hold-out data with labels, and (ii) model comparison is infeasible due to the lack of a universal objective function. MetaOD capitalizes on the performances of a large body of detection models on historical outlier detection benchmark datasets, and carries over this prior experience to automatically select an effective model to be employed on a new dataset without any labels, model evaluations or model comparisons. To capture task similarity within our meta-learning framework, we introduce specialized meta-features that quantify outlying characteristics of a dataset. Extensive experiments show that selecting a model by MetaOD significantly outperforms no model selection (e.g. always using the same popular model or the ensemble of many) as well as other meta-learning techniques that we tailored for UOMS. Moreover upon (meta-)training, MetaOD is extremely efficient at test time; selecting from a large pool of 300+ models takes less than 1 second for a new task. We open-source MetaOD and our meta-learning database for practical use and to foster further research on the UOMS problem. 

Automatic Unsupervised Outlier Model Selection
Yue Zhao
Ryan A. Rossi Adobe Research ryrossi @adobe. com
Leman Akoglu Carnegie Mellon University lakoglu@andrev. cmu edu
Carnegie Mellon University zhaoy@cmu. edu
Abstract
Given unsupervised outlier detection task on new dataset, how can au- tomatically select a good outlier detection algorithm and its hyperparameter(s) (collectively called model)? In this work, we tackle the unsupervised outlier model selection (UOMS) problem, and propose METAOD, a principled, data-driven approach to UOMS based meta-learning. The UOMS problem is notoriously challenging, as compared to model selection for classification and clustering, since model evaluation infeasible due to the lack of hold-out data with labels, and (ii) model comparison infeasible due to the lack of universal objective function. METAOD capitalizes on the performances of a large body of detection models on historical outlier detection benchmark datasets, and carries over this new
prior experience automatically select effective model to be employed dataset without any labels, model evaluations or model comparisons. To capture task similarity within our meta-learning framework, we introduce specialized meta- features that quantify outlying characteristics of dataset. Extensive experiments show that selecting model by METAOD significantly outperforms no model selection (e.g. always using the same popular model or the ensemble of many) well other meta-learning techniques that we tailored for UOMS. Moreover upon (meta- beeta-)training, METAOD is extremely efficient at test time; selecting from a large pool of 300+ models takes less than second for a new task. We open-soured METAOD and our meta-learning database for practical use and to foster further research on the UOMS problem.
Introduction
The lack of universal learning model that performs well all problem instances well recognized [53]. Therefore, effort has been directed toward building toolbox of various models and algorithms, which has given rise to the problem of algorithm selection and hyperparameter tuning (i.e., model selection). The same problem applies to outlier detection (OD); a long list of detectors has been developed the last decades with no universal "winners" " S
In supervised learning, model selection can be done via performance evaluation each trained model on labeled hold-out data. In contrast, unsupervised OD does not have access to any labels, nor is there a universal objective function that could guide model selection (cf. clustering where a loss function enables model comparison). Unsupervised model selection for OD is challenging exactly because both model evaluation and comparison are nol srasible-which renders any trial-and-error techniques like grid search iterative strategies like Bayesian hyperparameter optimization (57) inapplicable. Consequently, there has been no principled work on unsupervised outlier model Diocction-rathn the choice of model for new task (or dataset) remains "a black art". typical approach is to use popular OD algorithms, like LOF [6] and iForest (31) (often with default hyperparameters) which are shown be competitive on average on many benchmark datasets. However, as noted earlier, none of
CCode available at URL: httphpps//ggiiaab...o/yaaaaaaaaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

these methods can universally outperform others on all tasks [8. We argue that model selection is exactly how one can "break the performance ceiling" for OD.
In this work, we tackle the unsupervised outlier model selection (UOMS) problem systematically. To that end, we introduce (to the best of our knowledge) the first UOMS approach that selects effective model to be employed on new detection task without any model evaluation (using labels) or model comparison (via loss criteria). Our proposed method, called METAOD, is based on meta-learning, and stands on the prior performances of large collection of existing detection models on an extensive corpora of historical outlier detection benchmark datasets. In nutshell, the idea to estimate a candidate model's 's performance on the new task (with no labels) based on its prior performance on similar historical tasks. We remark that METAOD is strictly a model selection technique that picks one model (a detector and its associated typerparameter(s) from a pool of (existing) candidate models and not yet-another outlier detection algorithm itself.
In leveraging meta-learning, we establish a connection between the UOMS problem and the cold-start problem collaborative filtering (CF), where the new task in UOMS is akin to a new user in CF (with no available evaluations, hence cold-start) and the model space is analogous to the item-set. Differently, OD necessitates the identification of a single best model (i.e., top-1 rank), whereas CF typically operates in a top-k setting. In CF, future recommendations can be improved based on user feedback which not applicable OD. Moreover, METAOD requires the effective learning of task similarities based on characteristic dataset features (namely, meta- meta-features) that capture the outlying properties within dataset, whereas user features (location, age, etc.) in CF may be readily available.
In summary, the key contributions of this work include the following:
First Approach to Unsupervised Outlier Model Selection: We propose METAOD, (to our knowledge) the first effort on unsupervised model selection for OD tasks. Notably, given a new dataset (i.e., at test time), it does not rely any ground-truth labels for model evaluation any loss or heuristic criterion for model comparison. METAOD stands on meta-learning in principle, and historical collections of outlier models and benchmark datasets in practice.
Problem Formulation: We establish a correspondence between UOMS and CF under cold-start, where the new task "better likes" a model that performs better on similar historical tasks.
Specialized Meta-features: We design novel meta- featuress to capture the outlying characteristics in a dataset toward effectively quantifying task similarity specifically among OD tasks.
Effectiveness and Efficiency: Through extensive experiments on two benchmark testbeds that we have constructed, we show that selecting a model by METAOD for each given task significantly outperforms always using a popular model like iForest, as well as other possible meta-learning approaches that we tailored for UOMS. Moreover, METAOD incurs negligible run-time overhead (<1 second) at test time.
We open-source METAOD and our meta-learning database for the community use it for UOMS in practice, and to extend it with new datasets and models. We expect the growth of the database would make meta meta-learning based approaches, like METAOD, more powerful and also help foster further research on this new direction an important problem.
Open-source Platform:
2
Related Work
2.1 Model Selection for Outlier Detection (OD)
Most outlier mining work have focused on developing new, better methods for detection on different types of data (2). In comparison, there are only few work the outlier model selection problem -which detector and hyperparameter(s) to use on a new task- all of which require some labeled data. Recent work include AutoOD 29 that focuses
automatic neural architecture search, however, it is limited to deep autoencoder based detection, and more importantly it relies on hold-out labeled data for evaluation. Similarly, PyODDS (30 and TODS 25 both require ground truth labels.
To our knowledge, there is no existing work on the general unsupervised outlier model selection (UOMS) problem, which is considerably more challenging. as (1) model evaluation is infeasible due to the lack of any ground truth labels, and (2) model comparison of different heterogeneous detectors is infeasible due to the lack of universal OD loss function.

---

We note that some OD methods do have loss function; e.g. auto auto-encoders 19 63 aim to minimize reconstruction error for modeling the inliers, and one-class classification (OCSVM 38, SVDD [46] etc.) aims to maximize margin to the origin or minimize the radius of a data-enclosing hyperball. There is also work on model selection for one-class models 7 [44 however, those are limited to this specific family of methods and do not apply in the general case. Our proposed METAOD is not limited to one specific model family, but can select among any (heterogeneous) set of detectors.
2.2 Model Selection in ML, AutoML, Meta-Learning
Model selection refers to the process of algorithm selection and/or hy hyperparameter optimization (HO), With the advent of complex (e.g. deep) models, HО in high dimensions has become impractical to be
[59]. As such, automating ML pipelines has seen a surge of attention [8]
Meta-learning has been a key contributor to the AutoML effort
58
Supervised Model Selection: Most existing work focus on the supervised setting, and use hold-out data with labels. Randomized (3), bandit-based [27). and Bayesian optimization (BO) techniques are various state-of-the-art (SOTA) approaches to HO. Specifically sequential model-based BO evaluates hold-out performance at various initial hyperparameter configurations (HC), where a (smooth) surrogate function is fit to the resulting (HC, performance) pairs, which is then used to strategically query other HCs, e.g., via apprrggrddeet based search [15). Meta-learning has also been employed 13 [2], c.g., to find promising initialization for (i.e. warm-starting) BO [1,
40
19
22
Note that all of these approaches rely multiple model evaluations (i.e., performance queries) for various HCs, and hence cannot be applied to the unsupervised outlier model selection problem.
Unsupervised Model Selection: Unsupervised ML tasks (e.g.. clustering) poses additional chal- lenges for model selection [12 47). Nonetheless, those exhibit established objective criteria that enable model comparison, unlike OD. For example, BO methods still apply where the surrogate can be trained on (HC, objective value) pairs, for which meta-learning can provide favorable priors.
Task task-innependdett meta-lcarning [1]. that simply identifies the globally best model on historical tasks, applies the unsupervised setting and hence OD. This can be refined by identifying the best model on not all, but similar tasks, where task similarity is measured in the meta-feature space via clustering 23 or nearest neighbors [33] This type of similarity -based recommendations points to a connection between algorithm selection and collaborative filtering (CF), first recognized by Stern er al. A4 The most related to UOMS is CF under cold start, where evaluations are not-available (in our case, infeasible) for new user (in our case, task). There have been number of work using meta-learning for the cold-start recommendation problem 4 26 50), and vice versa, using CF solutions for MI. We tailor these to UOMS and compare to METAOD in the experiments.
algorithm selection 332 [6].
Unsupervised Outlier Model Selection via Meta-learning
3.1 Problem Statement
We consider the model selection problem for unsupervised outlier detection, which we refer to as UOMS (unsupervised outlier model selection) hereafter. Given a new dataset, without any labels, the problem is to select both (i) detector/algorithh and (fi) its associated hyperparameter(s) (HP). The former a discrete choice, given the finite set of existing detection algorithms. The latter continuous, and hence induces infinitely many candidate models.
Under certain assumptions, such as performance changing smoothly in the HP space, HP configura- tion can be selected iteratively based on evaluations on several other carefully-chosen configurations. Importantly however, OD is not amenable for such iterative search over models- not possible due to the lack of labels and absence of universal objective criterion. The selection of a model, therefore, is be done without building or evaluating any model on the new dataset. Given this constraint, we discretize the HP space for each candidate detector to make the search space tractable, which induces finite pool of models denoted M (M,,..., M. Each model M E - can be seen (detector, conf i iguration) pair, where the conf i guration depicts a specific set of values for the detector's 's HP(s). (See Appendix for details.) Then, the UOMS problem is stated as follows:
are
3

---

Problem 1 (Unsupervised Outlier Model Selection (UOMS)) Given a new input dataset (i.e., de-
tection task) D rest (Xeen) without any labels. Select model M M to employ on X
3.2 Proposed METAOD
In this work we consider the UOMS problem and propose meta-learning based solution, leveraging past experience historical detection tasks. As such, our METAOD relies on
database with ground truth labels, i.e., (D; yi) [D...... D.... namely, meta- train and the historical performances of the pool of candidate models, M, on the meta-train datasets. We denote by P E R2xm the performance matrix, where Pij corresponds to the j-th model Mj's performance on the i-th meta-train dataset D,
a collection of historical outlier detection datasets vain (Di..
Note that model performance can be evaluated the historical meta-train datasets as they contain ground truth labels, which however is not the case for any newcoming task at test time.
Our METAOD consists of two-phases: offline (meta-)training of the meta-learner on D and online prediction that enables unsupervised model selection at test time for Dist- Arguably, the running time of the offline phase is not critical. In contrast, model selection for newcoming task should incur small run-time overhead, as precedes the actual building of the selected OD model. Fig. D summarizes the process and the major components of METAOD, where highlight the components transferred from offline (meta-learming) to online stage (model selection) blue. We also provide the detailed steps of METAOD in pseudo-code, for both meta-training (offline) and model selection (online), in Appendix D Algo.
3.2.1 (Meta-)Training (Offline)
In principle, meta-learning carries over prior experience set of historical tasks to "do better" on a new task. Such improvement can be unlocked only if the new task resembles and thus can build on at least some of the historical tasks (such learning ice-skating given prior experience with roller-blading), rather than representing completely unrelated phenomena. This entails defining an effective way to capture task similarity between an input task and the historical tasks at hand.
In machine learning, similarity between meta-train and test datasets are quantified through character- istic features of a dataset, also known as meta-featured Those typically capture statistical properties of the data distributions. (See survey
40 for various types of meta-features.)
To capture prior experience, METAOD first constructs the performance matrix P by running/building and evaluating all the models in our defined model space M all the meta-train datasets D Draain 3l To capture task similarity, it then extracts a set of d meta-features from each meta-train dataset, denoted by M ( )) Rnd where (-) depicts the feature extraction module. We defer the details on the meta-feature specifics to $3.3
At this stage, it is easy to recognize the connection between the UOMS and the collaborative filtering (CF) under cold start problems. Simply put, meta-train datasets are akin to existing users in CF that have prior evaluations a newcoming user with no prior evaluations (and in our case, no possible future evaluation either), which however exhibits some pre-defined features.
set of models that are akin to the item-set in CF. The test task is akin to
Capitalizing on this connection, we take a matrix factorization based approach where P is approxi- mated by the dot product of what-we-call dataset matrix U R=*k and model matrix V m The intent is to capture the inherent dataset-to-modee affinity via the dot product similarity in the k-dimensional latent space, such that P. & U,V,² where matrix subscript denotes the row.
What loss criterion is suitable for the factorization? In CF the typical goal is top-k item recommenda- tion. In METAOD, we aim to select the model with the best performance on task which demands top-1 optimization. Therefore, we discard least squares and instead optimize the rank-based (row- or
Area under the precision-recall curve (Average Precision AP): can be substituted with any other measure. 'Note that this step takes considerable compute-ting time, which however amortizes to "do better" for future tasks. To this effect, we open-source trained meta-learner be readily deployed.

---

dataset-wise) discounted cumulative gain (DCG) 21
max U.V
Σ (..
(1)
The factorization is solved via alternating optimization, where initialization plays important role for such non-convex problems. We find that initializing U, denoted U based on meta-features facilitates stable training. potentially by hinting at inherent similarities among datasets as compared to random initialization. Specifically, an embedding function <(-) is used to set U == ((M) for
IRd + RR, k d. Details objective criteria and optimization are deferred $3.4
By construction, matrix factorization is transductive. On the other hand, we would need Utest to be able to estimate performances of the model set M new dataset X To this end, one can learn an (inductive) multi-output regression model that maps the meta-features onto the latent features. We simplify by learning regression function f R + I that maps the (lower dimensional) embedding features o (M)) (which are also used to initialize U) onto the final optimized U. Note that this requires inductive embedding function Φ--) to be applicable newcoming datasets. In implementation, we use PCA for ((-) and a random forest regressor for f()) although METAOD is flexible to accommodate any others provided they are inductive.
Remark: METAOD improves over the existing methods that use CF in machine learning model selection (see 2.2 in two aspects. First, METAOD builds specialized landmarker features tai- lored for capturing outlying characteristics of dataset, while the existing ML model selection mainly uses generic statistical features (see (33) Second, METAOD uses customized (back- propagatableesmoooth rank-based loss in CF for more effective top-1 optimization (see (3.4), while existing approaches mainly leverage mean squared loss (MSE).
3.2.2 Prediction for Unsupervised Model Selection (Online)
Meta-training stage yields the estimated functions ()), <(.). and f(-) as well as the model matrix
V E Rm* which we save for test time (See Fig. Given a new dataset X for OD, METAOD first computes the corresponding meta- features as Mist == (Xees) E Rd.. Those are then embedded via d(Mess E RR which are regressed to obtain the latent features, i.e.. Uwe == ((o(Mes) R
Model set performances are predicted P Pest == Uws V IR"" Finally, the model with the largest predicted performance is outputted as the selected model, that is,
arg max ( (f(o(1((xss))) )
(2)
Notice that model selection by Eq. (2) for newcoming dataset is solely based on its meta-features and other pre-trained components from meta-learning. does not rely on ground-truth labels or any OD model evaluations, therefore, METAOD provides unsupervised outlier model selection. Further, it does not require choosing or tuning any values at test time, and hence is fully automatic. In terms of computation, test-time embedding by o (PCA) and regression by f (regression trees) take near-constant time given the small number of meta-fcatures, embedding dimensions, and trees of fixed depth. Moreover, we use meta-features with computational complexity linear in the dataset size as we describe next.
Remark:
3.3 Meta-Features for Outlier Detection
A key part of METAOD is the extraction of meta-features that capture the important characteristics of an arbitrary datasct. Existing outlier detection models have different methodological designs (c.g., density, distance, angle, etc. based) and different assumptions around the topology of outliers (e.g., global, local, clustered). As a result, we expect different models to perform differently depending on the input dataset and the nature of outliers it schibits-henee no "winner". In our meta-learning approach, the goal is to identify the datasets in the meta-train database that exhibit similar characteristies to given test dataset, and focus on models that do well on those similar datasets. This is akin to recommending to a new user those items liked by similar users.
To this end, we extract meta-features that can be organized into two categories: (1) statistical features, and (2) landmarker features. Broadly speaking, the former captures statistical properties of the underlying data distributions: e.g., min, max, variance, skewness, covariance, etc. of the features and
5

---

Figure 1:
METAOD
Optimize DCG UV")
overview; components that transfer from offline (meta-learning) to online (model selection) phase shown in blue; namely, meta-feature extractors (v), embedding model (Ø), regressor f. dataset matrix U, and model matrix V. For the online phase, the input dataset Xestt and the predicted model performance Piest are denoted in yellow.
P m)
(0(0.)
X R)
models
train
Ap iraeeeeatur edtretto
+
U
entedcing
(n
regnession
(Offline) Meta- Leatner Training
(Online) Model Selection
V reb sabacikar
M test d)
+ Hintuctool
Utest (1 (xx
Prest Uss
X
Geductioly
(1 m)
feature combinations. (See Appendix B Table for for the complete list.) These kinds of meta-features have been commonly used in the AutoML literature [5].
The optimal set of meta-features has been shown to application-dependent [49] Therefore, perhaps more important are the landmarker features, which are problem-specific and aim to capture
the outlying characteristies a dataset. The idea is to apply a few the fast, easy-to- dasy-to-construct OD models on dataset and extract features from (i) the structure of the estimated OD model, and (ii) its output outlier scores. For the OD-specific landmarkers, we use four OD algorithms: iForest [3], HBOS [17), LODA (34), and PCA [20] (reconstruction error outlier score). We choose the four OD algorithms due to their efficiency and diversity (as group). First, they are all fast algorithms and able to handle large, iighddimensiona datasets (2] This makes the meta-feature generation efficient and practical in the real world. Second, these four OD algorithms as group show decent diversity (i.e., internal detection mechanism) to capture rich outlying characteristics. Consider iForest as example. It creates a set of what-is-called extremely randomized trees that define the model structure, from which we extract structural features such as average horizontal and vertical tree imbalance. As another example, LODA builds on candom-projection histograms from which we extract features such as entropy. In addition, based on the list of outlier scores from these models, we compute features such as dispersion, max consecutive gap in the sorted order, etc. We elaborate on the details of the landmarker features in Appendix B.2
3.4 Meta-Learning Objective and Training
3.4.1 Rank-based Criterion
typical loss criterion for matrix factorization is the mean squared error (MSE), a.k.a. the Frobenius norm of the error matrix P UV". While having nice properties from optimization perspective, MSE does not (at least directly) concern with the ranking quality. In contrast, our goal is rank the models for each dataset row-wise, model selection concern with picking the best possible model to employ. Therefore, we use vank-based criterion called DCG from the information retrieval literature 21.. For a given ranking, DCG is given
DCG Σ belr
(3)
log,(r (T +
where red_r depicts the true relevance of the item ranked the r-th position and b is scalar (typically set to 2). In our setting, we use the performance of a model to reflect its true relevance to dataset. As such, DCG for dataset is re-written
BP.,
DCG; =
Σ 3=1 log, Σm I
(4)
S PI))
where p.y = (U, Vz) is the predicted performance that dictates the ranking order. Intuitively, ranking high- gegh-performing models the top leads to higher DCG, and a larger increases the emphasis on the quality of models at the higher rank positions.

---

A challenge with DCG is that it is not differentiable. unlike MSE, as it involves ranking/sorting, Specifically, the sum term in the denominator Eq. 4 uses the (nonsmooth) indicator function to obtain the position of model j as ranked by the estimated performances. We circumvent this challenge by replacing the indicator function by the (smooth) sigmoid approximation [16] as follows.
P,
DCG, ~ sDCG, =
(5)
1ogg(1 (1 Ek=1 (P Pt))
3.4.2 Initialization & Alternating Optimization
Overall we optimize the smoothed criterion, sDCG, over all meta-train datasets D main (Di)==1 as
min
L Σ (P, U.V"), U.V
by alternatingly solving for U as we fix V (and vice versa) by gradient descent. We initialize by leveraging the meta-fcatures, which are embedded to a space with the same size as U. By capturing the latent similarities among the datasets, such initialization not only accelerates convergence [62 but also facilitates convergence to better local optimum. V is initialized from unit Normal.
As we aim to maximize the total dataset- wise DCG, we make pass over meta-train datasets one by one at each epoch. For brevity, we give the gradients for U, and V, Eq.s and (8, respectively.
aL aU,
m
6P
In (2)
Σ
Σ
- V,)
33 In
(3))
j=l
0L = - (2)
Σ
Pis
Σ
(1 -
(8)
0V,
B3 In (3))
/=1
where wij = (U;, (V& Vj)) and 83 = 3/2 + Enzj a (wfxx): see derivations in Appendix
Experiments
4.1 Experiment Setting
Model Set and Evaluation. We pair 8 SOTA OD algorithms and their corresponding hyperpa- rameters to compose model set M with 302 unique models. (See Appendix [a] 2 for the complete list.) We evaluate METAOD and the baselines on 2 testbeds introduced below, resp. with 100 and 62 datasets, via cross-validation where datasets are split into metat-rain/test in each fold. For each testbed, we first generate the performance matrix P, by evaluating the models from M against the benchmark datasets in the testbed. For randomized detectors (random-splly trees/random projections/etc., we run five independent trials and record the average performance. For consistency, all models are built using the PyOD library [6] on an Intel i7-9700 @3.00 GHz, 64GB RAM, 8-core workstation. We compare two methods statistically, using the pairwise Wilcoxon signed rank test on performances across datasets (significance level 0.05).
Testbed Setup. Meta-learning works well if new task can leverage prior knowledge; e.g., mastering motorcycle can benefit from bike riding experience. As such, METAOD relies on the assumption that a newcoming test dataset shares similarity with some meta-train datasets. We create two testbeds with different train/test dataset similarity, systematically study the effect of task similarity.
1. Proof-of-Concept (POC) testbed contains 100 datasets that form clusters of similar datasets, where 5 different detection tasks ("siblings") are created from each one of 20 "mothersets". Stress Testing (ST) testbed consists of 62 independent datasets from 3 different public-domant OD dataset repositories, which exhibit relatively lower similarity
another.
We refer to Appendix F for the complete list of datasets and details on testbed generation. Fig.
illustrates the differences between POC and ST testbeds, where the meta-fcatures of their constituting
7

---

Figure 2: 2-D embedding of datasets in (left) POC and (right) ST. POC exhibits higher task similarity, wherein "siblings" (marked by same color) form clusters. ST contains indepen- dent datasets with no apparent clusters.
datasets are embedded to 2-D by 1-SNE 48 By construction, POC consists of clusters and hence exhibits higher task/dataset similarity as compared to ST.
Baselines. Being the first work for UOMS, METAOD does not have immediate competing baselines. Therefore we employ simple ideas and tailor some existing methods for comparison. We also create 2 variations of METAOD (marked with f) for ablation analysis.
In Appendix F we give detailed descriptions of all 10 baselines. Briefly, they organized as follows: (i) no model selection always employs the same popular model, namely (1) LOF (6) or (2)
iForest [3 or the ensemble of all the models called (3) Mega Ensemble (ME); (#i) simple meta- learners include (4) Global Best (GB) that selects the model with the largest avg. performance across
meta-train datasets, (5) ISAC (23) and (6) ARGOSMART (AS) [33]; and (i0z) optimization-bused
meta-learners include (7) Supervised Surrogates (SS) 55 and (8) ALORS 32
Variants of METAOD are (9) |METAOD_S CC where performance and meta-feature matrices are con- catenated as C = P. M] E R=x(m+d), before factorization, C as UV- Given a test dataset, zero- concatenated meta-features are projected and reconstructed as [P testi Mies]] := [0 ... O; Mnew] VV and (10) METAOD __ where U is fixed at ((M) after the embedding step and only V is optimized.
Additionally, report Empirical Upper Bound (EUB) (only) for POC, the performance of the best model on a dataset's ss 4 "siblings"; this (valuable) information is not available in practice- "upper bound"". For ST with lower task similarity, we include Random Selection (RS) baseline.
4.2 POC Testbed Results
Testbed Setting. POC testbed is built simulate the scenario where there are similar meta-train train tasks to given test task. We use the benchmark dataset:" by Emmott et al. [1I], who created "childsets" from 20 independent "mothersets" by sampling. Consequently, the childsets generated from the same motherset using the same generation properties (e.g., the frequency of anomalies) can be deemed as "siblings" with large similarity. We build the POC testbed by using siblings from each motherset, resulting in 100 datasets. We split them into 5 folds for cross-validation, each test fold containing 20 independent childsets without siblings.
Figure Comparison of avg. rank (lower is better) of methods w.r.t per- formance across datasets in POC. Mean AP across datasets (higher is better) shown on lines. METAOD is the top- performing meta-learner, and compara- ble to EUB.
12
11
10
8
7
2
ME LOF GB
0.2051 EUB
0.2035 MstaOD
0.1970 SS
ISAC 0.1858
1980
MetaOD
0.1728
1.1853
IForest 0.1536
0.1966 MetaOD_F
Results. In Fig. B observe that METAOD is superior to all baseline methods w.r.t. the average rank and mean average precision (MAP), and performs comparably to the Empirical Upper Bound (EUB). Table (left) shows that METAOD is the only meta-learner that not significantly different from both EUB (MAP=0.205)) and the 4-/h best model (0.2185). Moreover, METAOD is significantly better than the baselines that do not employ any model selection (LOF (282) iForest (0.1536), and ME (0. (382),, as well as all the other meta-learners including GB (0.1524), ISAC (0.1658) and ALORS (0. (7.1728). For the full POC evaluation, see Appendix G.T
Averaging all models (ME) does not lead to good performance as one may expect. As shown in Fig. ME is the worst baseline by average rank in the POC testbed. Using single detector, e.g.,
https://Lie library.r....... oregonstate httpe///ooooeeeeaaaaaa//////////

---

Ours
Baseline
p-value
Ours
Baseline
p-value
Table 1: Pairwise statistical test results between METAOD and baselines by
MetaOD
EUB
0.0522
MctaOD
Wilcoxon signed rank test. Statisti- 58-th Best
0.0517
MetaOD
4-th Best
0.0929
MetaOD
RS
0.0001
MetaOD
LOF
0.0013
MetaOD
LOF
0.0001
cally better method shown in bold (both marked bold if signíficance). In (left) POC, METAOD is the only meta-learner with no diff. from both EUB and the th best model. In (right) ST, METAOD is the only meta-learner with no statisti- cal diff. from the 58-th best model. It is statistically better than all except iForest.
MetaOD
Forest
0.0090
MetaOD iForest
0.1129
MetaOD
ME
0.0004
MetaOD
ME
0.0001
MetaOD
GB
0.0051
MetaOD GR
0.0030
MetaOD
ISAC
0.0019
MctaOD ISAC
0.0006
MetaOD
0.2959
MetaOD
0.0009
MetaOD S8
0.7938
MctaOD SS
0.0190
MetaOD
ALORS
0.0025
MetaOD
ALORS
0.0001
MetaOD
MetaOD C
0.6874
MetaOD
MetaOD C
0.0001
MelaOD
MetaOD_F
0.1165
MetaOD
MetaOD
0.0001
iForest, is significantly better. This is mainly because some models perform poorly on any given dataset, and ensembling all the models indiscriminately draws overall performance down. Using selective ensembles [36] could be beneficial, however, ensembles of many models are expensive to build in practice. contrast, METAOD is fast at test time and selects without building any models.
Meta-learners perform significantly better than methods without model selection. In particular, four meta-learnen (METAOD, SS, METAOD C, METAOD F) significantly outperform single outlier detection methods (LOF and iForest) as well as the Mega Ensemble (ME) that averages all the models. METAOD respectively has 58.74%, 32.48% and 47.25% higher MAP over LOF, iForest, and ME. These results signify the benefits of model selection.
Optimization-basee meta learners generally perform better than simple meta learners. Top-3 meta Iearners by average rank (METAOD, SS, and METAOD C) are all optimization-but and significantly outperform simple meta-learners like ISAC as shown in Fig. B Simple meta-learners weigh meta-features equally for task similarity, whereas others learn which meta-features matter (e.g.. regression on meta-features), leading to better results. We find that METAOD respectively achieves 33.53% 22.74%, and 4.73% % higher MAP than simple meta-learners including GB, ISAC, and AS.
4.3
ST Testbed Results
Testbed Setting. When meta-train datasets lack similarity to the test dataset, is hard capitalize on prior experience. In the extreme case, meta-learning may not perform better than no-model-selection baselines, e.g., single detector. To investigate the impact of the train/test similarity on meta-learning performance, we build the ST testbed that consists of 62 public-domain datasets from 3 different repositories (See Appendix(E]Tablee
with relatively low similarity as shown in Fig. For evaluation on ST, we use leave-one-out cross validation; each time using 61 datasets as meta-train.
12
10
7
Figure 4: Comparison of avg. rank (lower is better) of methods w.r.t. per- formance across datasets in ST. Mean AP (higher is better) shown on lines. METAOD outperforms all baselines.
MetaOD C 0.1946
0.3382 MetaOD
MetaOD 0,2594
0.3197 |Forest
RS 0.2582
0.3137 GB
LOF 0.2155
0.28114 SS
ALORS 0.2981
0.2882 ISAC
ME 0.2689
0.2704 AS
Results. For the ST testbed, METAOD still outperforms all baseline methods w.rt. average rank and MAP as shown in Fig. 4 Table 1 (right) shows that METAOD (0.3382) could select, from a pool of 302, the model that is as good as the 58-th best model (top 20%) per dataset (0.3513) in this challenging testbed. The comparable model changes from the 4-th best per dataset in POC to 58-th best in ST. which is expected due to the lower task similarity to leverage in ST. Notably, all other baselines are worse than the 80-th best model with statistical significance. Moreover, METAOD is significantly better than all baselines except iForest. Note that METAOD also significantly outperforms RS, showing that it is able to exploit the meta-train database despite limited task similarity and not simply resorting to random picking. These results suggest that METAOD is a good choice under various extent of similarity among train/test datasets. We refer Appendix G.2 for detailed ST results on individual ST datasets.
Training stability affects performance for optimization-base methods. Notably, several optimization- meta-learners, such as ALORS and METAOD CC do not perform well for ST. We find that the training process of matrix factorization is not stable when latent similarities are

---

weak. In METAOD, we employ two strategies that help stabilize the training. First, we leverage meta-feature based (rather than random) initialization. Second, we use cyclical learning rates that help escape saddle points for better local optima 43 Consequently, METAOD (0.3382) significantly outperforms ALORS (0.2981) and METAOD C (0.1946) with 13.45% and 73.799 33.9% higher MAP.
Global methods outperform local methods under limited task similarity. In ST, datasets are less similar and simple meta-learners that leverage task similarity locally often perform poorly. For example, AS selects the model based on the I-NN, and is likely to fail if the most similar meta-train task still quite dissimilar to the current task. Notably, the global meta-learner GB outperforms AS and ISAC. Note the opposite ordering among these methods in POC as shown in Fig. 3 In short, effectiveness of simple meta-learners tends to be sensitive the train/test dataset similarity, which makes them hard to use in general. In contrast, METAOD performs well in both settings.
4.4 Runtime Analysis
Empowered by meta-training, METAOD (meta-feature generation and model selection) takes less than second on most test datasets, shown in Fig. where it incurs negligible overhead relative to building/rainnin the selected outlier model (=10% on avg.). Fig. corroborates the statement by showing the comparison on the 10 largest datasets in POC.
60%
100%
120%
140%
Figure 5: METAOD running time at test time in sec.s (left), and percentage of time relative to building the selected model (right). Notice that it is fast, and incurs negligible computational overhead.
25
(MA]aCC]] mata-
meta-leature feature generation (M4tsO()) model sellctonnn mels-feature
Figure 6: Time for METAOD vs. train- ing of the selected model (on 10 largest datasets in POC). METAOD incurs only negligible overhead (diff. shown w/ black arrows).
baiing
Saconds
10
abalone
gab
ingsag
letter
magic
per tole
sknn
wove
wine
yearp
Notably, meta-feature extraction may be trivially parallelized whereas the model selection is even faster, e.g., using SUOD (60, effectively taking constant time (See $3.2.2
5 Conclusion
We addressed the unsupervised outlier model selection (UOMS) problem without relying on any labels, model evaluations or comparisons for the first time. Our proposed METAOD is a meta-leamerr, and builds on extensive pool of historical outlier detection datasets and models. Given new task, selects a model based on the past performances of models on similar historical tasks. To effectively capture task similarity, we designed novel problem-specific meta-features. Importantly, METAOD is (i) fully automatic, requiring no supervision at test time, and (ii) lightweight, incurring relatively small selection time overhead prior to outlier model building. Extensive experiments two large testbeds showed that METAOD significantly improves detection performance over always using some of the most popular outlier models as well as several other meta-learners tailored for UOMS.
We open-source METAOD and our meta- meta-nearning database for use in practice. We expect meta- learning to become more powerful as the meta-train database grows. Therefore, we also share all our code and testbeds with the community to stimulate further advances in automating UOMS. Future work can address UOMS in the continuous hyperparameter space, leverage self-aware learning (28 and conformal prediction [39 to estimate the confidence selection, and explore the potential bias and fairness issues in OD model selection [10 42).
10

---

References
[I] M. Abdulrahman, P. Brazdil, N. van Rijn, and J. Vanschoren. Speeding up algorithm selection using average ranking and active testing by introducing runtime. Mach. Learn., 107(1):79-108, 2018.
[2] C. C. Aggarwal. Outlier Analysis. Springer, 2013.
[3] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012.
[4] H. Bharadhwaj. Meta-learning for user cold-start recommendation. In IJCNN, pages 1-8. IEEE, 2019.
[5] B. Bilalli, A. Abelló Gamazo, and T. Aluja Banet. On the predictive power meta-features in OpenML. International Journal of Applied Mathematics and Computer Science, 27(4):697-712, 2017.
[6] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density- local outliers. In SIGMOD Conference, pages 93-104, ACM, 2000. SIGMOD Record 29(2), June 2000.
[7] E. Burnaev, P. Erofeev, and D. Smolyakov. Model selection for anomaly detection. In ICMV, volume 9875 of SPIE Proceedings, page 987525. SPIE, 2015.
[8] G. O. Campos, A. Zimek, J. Sander, R. J. G. B. Campello, B. Micenková, E. Schubert, Assent, and M. E. Houle. On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min. Knowl. Discov,, 30(4):891 2016.
[9] J. Chen, S. Sathe, C. C. Aggarwal, and D. S. Turaga. Outlier detection with autoencoder ensembles. In SDM, pages 90-98. SIAM, 2017.
[10] I. Davidson and S. S. Ravi. A framework for determining the fairness of outlier detection. In G. D. Giacomo, A. Catalá, B. Dilkina, M. Milano, S. Barro, A. Bugarín, and J. Lang. editors, ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pages 2465-2472. IOS Press, 2020.
[11] A. Emmott, S. Das, G. Dietterich, Fern, and W.-K Wong. Anomaly detection meta-analysis benchmarks. 2016.
[12] X. Fan, Y. Yue, P. Sarkar, and Y. X. R. Wang. A unified framework for tuning hyperparameters in clustering problems. CoRR, abs/1910.08018, 2019.
[13] M. Feurer, B. Letham, and E. Bakshy. Scalable meta-learning for bayesian optimization. CoRR, abs/1802.02219. 2018.
[14] M. Feurer, T. Springenberg, and F. Hutter. Initializing bayesian hyperparameter optimization via meta-learming In AAAI, pages 1128 1778-1155 1135. AAAI Press, 2015.
[15] L. Franceschi, M. Donini, Frasconi, and M. Pontil. Forward and reverse gradient-based hyperparameter optmizzation. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 1165 1173. PMLR, 2017.
[16] J. Fréry, A. Habrard, M. Sebban, O. Caclen, and He-Guelton. Efficient top rank optimization with gradient boosting for supervised anomaly detection. In ECML/PKDD, volume 10534, pages 20-35, 2017.
[17] M. Goldstein and A. Dengel. Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm. K7-2012: Poster and Demo Track, pages 59-63, 2012.
[18] X. He, K. Zhao, and X. Chu. Automl: survey of the state-of-the-art. Knowl. Based Syst., 10662, 2021.
11

---

[19] F. Hutter, H. H. Hoos, and K. Leyton-Brown Sequential model-basee optimization for general algorithm configuration. In C. A. C. Coello, editor, LION, volume 6683 of Lecture Notes in Computer Science, pages 507-523. Springer, 2011.
[20] T. Idé and H. Kashima. Eigenspace-basee anomaly detection in computer systems. In W. Kim, ACM, 2004.
R. Kohavi, J. Gehrke, and W. DuMouchel, editors, KDD, pages
[21] K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.
[22] D. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. J. Global Optimization, 13(4):455-492, 1998.
[23] S. Kadioglu, Y. Malitsky, M. Sellmann, and K. Tiemney Isac - -o--snnncesspeciiie algorithm configuration. ECAL, volume 215 of Frontiers Artificial Intelligence and Applications, pages 751-756. IOS Press, 2010.
[24] H.-P. Kriegel, M. Schubert, and A. Zimek. Angle-based outlier detection in sigh-dimensional data. In Y. Li, B. Liu, and S. Sarawagi, editors, KDD, pages 144-452 ACM, 2008
[25] K. Lai, D. Zha, G. Wang, J. Xu, Y. Zhao, D. Kumar, Y. Chen, P. Zumkhawaka, M. Wan, D. Martinez, and X. Hu. TODS: an automated time series outlier detection system. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 16060-16062. AAAI Press, 2021.
[26] H. Lee, J. Im, S. Jang, Cho, and Chung. MeLU: Meta-learned user preference estimator for cold-start recommendation. In KDD, pages 1073-1082 ACM, 2019.
[27] L. Li, K. G. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Mach. Learn. Res., 18: 18:185:1-185:52 2017.
[28] L. Li, M. L. Littman, T. J. Walsh, and A. L. Strehl. Knows what it knows: framework for self-aware learning. Mach. Learn., 82(3):399-443, 2011.
[29] Y. Li, Z. Chen, D. Zha, K. Zhou, H. Jin, H. Chen, and X. Hu. Autood: Neural architecture search for outlier detection. In 37th IEEE International Conference on Data Engineering. ICDE 2021, Chania, Greece, April 19-22, 2021, pages 2117-2122. IEEE, 2021.
[30] Y. Li. D. Zha, P. K. Venugopal. N. Zou, and X. Hu. Pyodds: An end-to-end outlier detection system with automated machine learning. In A. E. F. Seghrouchni, G. Sukthankar, Liu, and M. Steen, editors, Companion of The 2020 Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 153-157. ACM / IW3C2, 2020.
[31] F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation forest. In ICDM, pages 413-422. IEEE Computer Society, 2008
[32] M. Misir and M. Sebag. Alors: An algorithm recommender system. Artif. Intell., 244:291-314, 2017.
[33] M. Nikolic, F. Maric, and P. Janicic. Simple algorithm portfolio for sat. Artif. Intell. Rev., 40(4):457-465, 2013.
[34] T. Pevný. Loda: Lightweight on-line detector of anomalies. Mach. Learn., 102(2))275-304, 2016.
[35] S. Ramaswamy, Rastogi, and Shim. Efficient algorithms for mining outliers from large data sets. In W. Chen, J. F. Naughton, and P. A. Bernstein, editors, SIGMOD Conference, pages 427 438. ACM, 2000. SIGMOD Record 29(2), June 2000.
[36] S. Rayana and Akoglu, Less more: Building selective anomaly ensembles. ACM Trans. Knowl. Discov. Data, 10(4):42:1-4) 1-4:333, 2016.
12

---

[37] B. Schölkopf. J. C. Platt. J. Stawe-Taylow J. Smola, and R. C. Williamssn. Estimating the support of a high high-dimensiioal distribution. Neural Computation, 13(7):143--1471 July 2001.
[38] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe- Shawe-Taylor, and J. C. Platt. Support vector method for novelty detection. In NIPS, pages 582-588. The MIT Press, 1999.
[39] G. Shafer and Vovk. A tutorial on conformal prediction. Mach, Learn. Res., 9:371 -211, 2008.
[40] B. Shahriari, K. Swersky, Z. Wang. R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proc. IEEE, 104(1):148-175, 2016.
[41] R. E. Shawi, M. Maher, and S. Sakr. Automated machine learning: State-of-the-an art and open challenges. CoRR, abs/1906.02287, 2019.
[42] S. Shekhar, N. Shah, and L. Akoglu, Fairod: Fairnes--wware outlier detection. In M. Fourcade, B. Kuipers, S. Lazar, and D. Mulligan, editors, AIES '21: AAAI/ACM Conference on Al, Ethics, and Society, Virtual Event, USA, May 19-21, 2021, pages 210-220. ACM, 2021.
[43] L. N. Smith. Cyclical learning rates for training neural networks. In WACV, pages 464-472
IEEE Computer Society, 2017.
[44] D. H. Stern, H. Samulowitz, R. Herbrich, T. Graepel. L. Pulina, and A. Tacchella. Collaborative expert portfolio management. In AAAI. AAAI Press, 2010.
[45] J. Tang, Z. Chen, A. W. Fu, and D. W. Cheung. Enhancing effectiveness of outlier detections for low density patterns. In PAKDD, volume 2336 of Lecture Notes in Computer Science, pages 535-548, 2002.
[46] D. M. J. Tax and R. P. W. Duin. Support vector data description. Mach. Learn., 54(1):45-66 -6, 2004.
[47] S. Vaithyanathan and B. Dom. Generalized model selection for unsupervised learning in high dimensions. In NIPS, pages 970-976. The MIT Press, 1999.
[48] L. van der Maaten and G. Hinton. Visualizing high-dimensional data using t-sne. J. of Mach. Lear. Res., 9:2579-2605, 2008
[49]
Vanschoren. Meta-learning. In Automated Machine Learning, pages 35-61. Springer, 2019.
[50] M. Vartak, Thiagarajan, C. Miranda, Bratman, and H. Larochelle. A meta-learning perspective cold-start recommendations for items. In NeurIPS, pages 6904-6914, 2017.
[51] M. Wistuba, N. Schilling, and L. Schmidt-Thieme, Learning hyperparameter optimization initializations. In DSAA, pages 1-10. IEEE, 2015.
[52] M. Wistuba, N. Schilling, and L. Schmidt-Thieme. Scalable gaussian process-based transfer surrogates for hyperparameter optimization. Mach. Learn., 107(1):43-78, 2018.
[53] D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. IEEE Trans. Evolutionaar Computation, 1(1):67-8) 1997.
[54] Y. Xiao, H. Wang, L. Zhang, and W. Xu. Two methods of selecting gaussian kernel parameters for one-class SVM and their application to fault detection. Knowl. Based Syst., 59:75-84, 84, 2014.
[55] L. Xu, Hutter, J. Shen, H. H. Hoos, and K. Leyton-Brown Brown. Satzilla2012: Improved algorithm selection based on cost-sensitive classification models. Proceedings of SAT Challenge, pages 57-58, 2012.
[56] C. Yang, Y. Akimoto, D. W. Kim, and M. Udell. OBOE: collaborative filtering for AutoMI. model selection. In KDD, pages 1173-1183, ACM, 2019.
[57] I.. Yang and A. Shami. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415:295-316, 2020.
13

---

[58] Q. Yao, M. Wang, H. J. Escalante, I. Guyon, Y. Hu, Y. Li, W. Tu, Q. Yang. and Y. Yu. Taking human out of learning applications: A survey on automated machine learning. CoRR. abs/181 110.110.1111 13306, 2018.
[59] T. Yu and H. Zhu. Hyper-parameter optimization: A review of algorithms and applications. CoRR, 0as/2003.05689) 2020.
[60] Y. Zhao, X. Hu, C. Cheng, C. Wang, C. Wan, W. Wang. J. Yang. H. Bai, Z. Li, C. Xiao, Y. Wang.
Z. Qiao, J. Sun, and L. Akoglu. SUOD: Accelerating large-scale unsupervised heterogeneous outlier detection. Proceedings of Machine Learning and Systems, 2021.
[61] Y. Zhao, Z. Nasrullah, and Z. Li. PyOD: A python toolbox for scalable outlier detection. J. Mach. Learn, Res., 20:96: 1-96:7, 2019.
[62] Zheng, J. Yang, and Zhu. Initialization enhancer for non-negative matrix factorization. Eng. Appl. Artif. Intell., 20(1):101-110; 2007.
[63] C. Zhou and R. C. Paffenroth. Anomaly detection with robust deep autoencoders. In KDD, pages 665-674, ACM, 2017.
14


Reviews:
[{"review_id": "k2CBuB9TCwA", "reviewer": "Reviewer_EPJc", "summary": "The paper proposes a method, named MetaOD, which for each input data set selects the most suited outlier detection model together with its relevant hyper-parameter values from a set of given models. The selection is done in a supervised manner. MetaOD works as follows. During the meta-training stage, it considers a set of $m$ models $M = \\{M_1, ..., M_m\\}$ and a set of $n$ data sets $D = \\{D_1, ..., D_n\\}$. Each model $M_i$ includes model architecture and model configuration (hyper-parameter values). MetaOD computes performance matrix $P$ where cell $P_{ij}$ is the performance of method $M_j$ on data set $D_i$. Then it learns a data matrix $U \\in R^{n \\times k}$ and a model matrix $V \\in R^{m \\times k}$ where $U_i \\times V_j^T$ approximates $P_{ij}$. The functions learnt from $U$ and the matrix $V$ are then used during test time to choose the best model (together with its configuration) for each input data set.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unsupervised outlier detection is a challenging task as it lacks a universal objective function and reliable evaluation data sets. Methods proposed so far have their own notion of outlier scores and do not consistently have best performance. This makes it hard in practice to choose a suitable OD method for the data set at hand. As a plus point, the paper tackles the issue by mapping unsupervised OD model selection to the collaborative filtering cold-start problem, enabling the transfer of well-established solutions in CF to OD.\n\nI find the solution of finding $U$ and $V$ through matrix factorisation on $P$ (solved by alternating optimisation) straightforward and standard. I have two concerns though. First, while $V$ is used directly during test time, $U_{\\mathit{test}}$ is just a mapping of the test set through a series of three functions, one of which is regression function $f(\\cdot)$. Function $f(\\cdot)$ on the other hand is not optimised together with $U$ and $V$ during meta-training stage, and hence might be sub-optimal to reconstructing matrix $P$. Is there a way to learn $f(\\cdot)$ during alternating optimisation? Second, MetaOD focuses on selecting the best model. It would be more interesting if the authors solve the more general problem of selecting top-k models (to build an ensemble) and make the top-1 a special case.\n\nThe experiments look extensive with 2 settings: POC and stress tests. I find the baselines and data sets sufficient. In addition, MetaOD achieves good empirical performance and takes about 1 second to choose the best model for each input data set.\n\nThe writing has room for improvement. For instance, a table summarising the notations used would make the paper more readable. The part about data-specific meta-feature extraction is important and should be explained in the main part of the paper (e.g. through an example data set).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do address the technical limitations of the paper that they are able to identify.\n\nThe authors do not identify any societal impact of the paper. However, one thing they could explore, perhaps in future work, is how biases embedded in base OD models and historical data sets could influence the selection of best model for incoming test sets.", "review_text": "", "extra_scores": {"ethical_concerns": "As mentioned above.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ft0ccHqn_t", "reviewer": "Reviewer_ZaAH", "summary": "This paper proposes a data-driven method, METAOD to unsupervised outlier model selection problem based on meta-learning. METAOD makes use of many detection models on historical outlier detection benchmark datasets and selects an effective model to be employed on a new dataset without. Authors introduce meta-features that quantify the outlying characteristics of a dataset. Experiments have shown the promise of METAOD. The authors have implemented an open-source tool METAOD and our meta-learning database for practical use.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors tackle a very challenging problem, unsupervised outlier model selection. Evaluating outlier detection methods is very challenging because of no ground truths and a lack of objective function makes unsupervised outlier model selection. The authors approach the problem as a collaborative filtering recommendation problem, and this is quite interesting.\n\nOverall, I feel this method has some novelty. The selection of an outlier detection method is challenging, and the method proposed can be a practical solution. This paper utilises prior knowledge about outlier detection models (their performance on historical labelled datasets) and the similarity of an (unlabelled) test dataset to historical datasets to determine the best model for the test dataset. A key step is to find the similarity, i.e., meta-features. Two types of meta-features, statistical and landmark features, are designed to capture the similarities of two datasets in the general case and outlying aspects, respectively. The design of the method is reasonable. \n\nHowever, I have some specific questions about the implementation. \n\nThe success of the proposed method depends on several factors, and the two major ones are as the following: 1) the outlying characteristics can be precisely quantified. 2) The performance of a method can be accurately measured. \n\nThe features and measures proposed are heuristics, and they need justification. The statistic features are very general and they may not reveal any outlying characteristics since otherwise, outlier detection can be based on these general statistics features. Furthermore, data sets have different dimensionalities. Can statistical properties of data sets with different dimensionalities be comparable?\n\nLandmark features:  what are justifications for using the four OD methods? Do the features capture outlying chrematistics of data sets? \n\nThe meta-features can be many. Will this cause curse of dimensionality?\n\nEquation (2) seems arbitrary for me. Why are the three functions used?\n\nAs the authors discussed, it is very difficult to quantify the performance of an outlier detection method. How do the authors resolve this problem?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QQkrKuNKsc", "reviewer": "Reviewer_4gCH", "summary": "The paper proposes a model selection approach called METAOD for outlier detection. METAOD maintains historical performances of various outlier detection models on different datasets, where the performance of a detection model on a new dataset is estimated based on the historical information. The setting is thus similar to meta-learning. METAOD is motivated by Collaborative Filtering, where the compatibility of a user (i.e., a dataset) and an item (i.e., a model) is estimated. A set of meta-features are crafted for charactering a dataset. Experiments show that METAOD benefits outlier detection model selection, compared with no model-selection baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper targets a novel task, i.e., model selection for outlier detection (unsupervised). The task poses a new challenge that traditional model selection methods could not tackle, that it is hard to evaluate models on datasets without ground truths.\n\nQuality: The proposed technique is, in general, intuitive and reasonable. A CF-based matrix factorization is used to learn the latent space for measuring model-data similarity. A forward mapping function f is learned to transform data features to the latent space. A PCA step is included as pre-processing. In the CF step, a back-propagatable DCG-based loss is developed to replace traditional MSE loss. However, it would be better to report this advantage through experiments.\n\nClarity: The paper is well-structured and well-written.\n\nSignificance: Model selection is a crucial task for outlier detection. The paper (partially) tackles the important problem, as long as practitioners have already accumulated a significant number of historical datasets and evaluated models. The testbed setup consists of two scenarios (POC and ST), which is an interesting setting. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Pft9cRjpSE", "reviewer": "Reviewer_21Jt", "summary": "This paper presents a data-driven approach to model selection for unsupervised outlier detection. The proposed method, METAOD takes a large number of training datasets with outlier labels and a pool of outlier detection models (methods plus their parameters) and uses a matrix factorization method to learn a model performance predictor, which, for given a test dataset, can predict/select a high performing model from the model pool without requiring test time model evaluation.\n\nThe performance of METAOD is evaluated with two implementations with the same pool of 300+ models (8 distinct outlier detection methods with different parameter configurations) but two different groups of training datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please find below some detailed comments on the originality, significance, quality and clarity of the paper.\n\n1. Originality and significance\n\nStrong points:\n\n(a) The idea of using meta-learning for selecting outlier detection models is interesting and sounds novel to me.\n\n(b) There is not a model that can suit all outlier detection problems, and it is important and a good idea to try to select a high performing model for each specific dataset.\n\n(c)  It is infeasible to conduct test time model selection or evaluation for outlier detection problems, hence using historical data and the similarity between test data and historical data is a good direction to explore.\n\nWeakness:\n\n(a) For the proposed method/framework to be applicable, it would be good if the authors could discuss more about the limitations of the proposed method, in particular, it would be ideal if some guidelines/analysis or evaluation can be provided regarding the extend of similarity of test and train data for the proposed method to be effective.\n\n(b) In Section 2.2, more discussions should be given on the difference between the key ideas/methods used by METAOD and the existing methods which use CF solutions for ML model selection. \n\n2. Quality\n\nStrong points:\n\n(a) The overall idea of using meta learning to capture models' experience on outlier detection in different historical data.\n\n(b) The linking between unsupervised outlier detection model selection and collaborative filtering.\n\n(c) The creation and use of meta-features\n\n(d) A comprehensive open-source implementation and evaluation\n\nWeakness:\n\n(a) The success of METAOD relies on the similarity between test data and training data. Given the \"unknown\" nature of outliers, it is quitely likely that in practice such similarity may not exist or may be very weak. As mentioned above, discussions and ideally evaluations on this applicability problem of METAOD should be provided, and the authors may also want to consider extend METAOD with the ability to output \"null\" when no good models can be picked up from the model pool.\n\n(b) How the different factors, e.g. ways of discretizing hyperparameters, number of models to include in the pool, number of training datasets, performance evaluation metrics used for creating P, etc. will affect the overall performance of METAOD?\n\n3. Clarity\n\nOverall the paper is not difficult to follow, but the presentation can be improved by providing more clarity on some of the details and correcting the English problems, as detailed below.\n\n(a) The description in Section 3.2 sometimes is not easy to follow. It would be better if an outline of the process and key parts of METAOD could be given at the start of the section. I understand that Fig. 5 in Appendix D may use lots of space, but it helps me a lot to understand the working of METAOD. The authors may consider to have a simplified version of Figure 5. The current writing of Section 3.2 is a bit loose and it could be tighten up too  to leave room to move Figure 5 to the main text and organise the writing in this section around the Figure.\n\n(b) There are some typos and other English problems. Examples are:\n\n*Line 11: selecting -> select\n*Line 27: winner -> winners\n*Line 179: largest performance -> best performance?\n*Line 282: affect -> effect\n*Line 286: footnotes 6,7 and 8 are not included in the main paper\n*Line 309: whom -> who", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should provided more discussions on the limitations of the proposed method, especially the limitations related to the applicability of the method.\n\nCurrently there are no discussions provided on the societal impact of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers have praised the importance of selecting outlier detectors in an automatic way and recognized the proposed meta-learning solution as a novel contribution. 

Important points were raised during the reviewing and discussion period. They concern: 
  - the influence of the distribution similarity between train and test to make the meta-learning succeed, 
  - the effect of some hyperparameters involved and 
  - some rewriting of the presentation to ease reading (notation + English grammar typos) and to better motivate the use of a collaborative filtering approach.

The paper is accepted subject to the above points are addressed in the camera-ready.
Decision: Accept (Poster)