Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a data-driven approach to model selection for unsupervised outlier detection. The proposed method, METAOD takes a large number of training datasets with outlier labels and a pool of outlier detection models (methods plus their parameters) and uses a matrix factorization method to learn a model performance predictor, which, for given a test dataset, can predict/select a high performing model from the model pool without requiring test time model evaluation.

The performance of METAOD is evaluated with two implementations with the same pool of 300+ models (8 distinct outlier detection methods with different parameter configurations) but two different groups of training datasets. 
main_review: Please find below some detailed comments on the originality, significance, quality and clarity of the paper.

1. Originality and significance

Strong points:

(a) The idea of using meta-learning for selecting outlier detection models is interesting and sounds novel to me.

(b) There is not a model that can suit all outlier detection problems, and it is important and a good idea to try to select a high performing model for each specific dataset.

(c)  It is infeasible to conduct test time model selection or evaluation for outlier detection problems, hence using historical data and the similarity between test data and historical data is a good direction to explore.

Weakness:

(a) For the proposed method/framework to be applicable, it would be good if the authors could discuss more about the limitations of the proposed method, in particular, it would be ideal if some guidelines/analysis or evaluation can be provided regarding the extend of similarity of test and train data for the proposed method to be effective.

(b) In Section 2.2, more discussions should be given on the difference between the key ideas/methods used by METAOD and the existing methods which use CF solutions for ML model selection. 

2. Quality

Strong points:

(a) The overall idea of using meta learning to capture models' experience on outlier detection in different historical data.

(b) The linking between unsupervised outlier detection model selection and collaborative filtering.

(c) The creation and use of meta-features

(d) A comprehensive open-source implementation and evaluation

Weakness:

(a) The success of METAOD relies on the similarity between test data and training data. Given the "unknown" nature of outliers, it is quitely likely that in practice such similarity may not exist or may be very weak. As mentioned above, discussions and ideally evaluations on this applicability problem of METAOD should be provided, and the authors may also want to consider extend METAOD with the ability to output "null" when no good models can be picked up from the model pool.

(b) How the different factors, e.g. ways of discretizing hyperparameters, number of models to include in the pool, number of training datasets, performance evaluation metrics used for creating P, etc. will affect the overall performance of METAOD?

3. Clarity

Overall the paper is not difficult to follow, but the presentation can be improved by providing more clarity on some of the details and correcting the English problems, as detailed below.

(a) The description in Section 3.2 sometimes is not easy to follow. It would be better if an outline of the process and key parts of METAOD could be given at the start of the section. I understand that Fig. 5 in Appendix D may use lots of space, but it helps me a lot to understand the working of METAOD. The authors may consider to have a simplified version of Figure 5. The current writing of Section 3.2 is a bit loose and it could be tighten up too  to leave room to move Figure 5 to the main text and organise the writing in this section around the Figure.

(b) There are some typos and other English problems. Examples are:

*Line 11: selecting -> select
*Line 27: winner -> winners
*Line 179: largest performance -> best performance?
*Line 282: affect -> effect
*Line 286: footnotes 6,7 and 8 are not included in the main paper
*Line 309: whom -> who
limitations_and_societal_impact: The authors should provided more discussions on the limitations of the proposed method, especially the limitations related to the applicability of the method.

Currently there are no discussions provided on the societal impact of the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a model selection approach called METAOD for outlier detection. METAOD maintains historical performances of various outlier detection models on different datasets, where the performance of a detection model on a new dataset is estimated based on the historical information. The setting is thus similar to meta-learning. METAOD is motivated by Collaborative Filtering, where the compatibility of a user (i.e., a dataset) and an item (i.e., a model) is estimated. A set of meta-features are crafted for charactering a dataset. Experiments show that METAOD benefits outlier detection model selection, compared with no model-selection baselines.
main_review: Originality: The paper targets a novel task, i.e., model selection for outlier detection (unsupervised). The task poses a new challenge that traditional model selection methods could not tackle, that it is hard to evaluate models on datasets without ground truths.

Quality: The proposed technique is, in general, intuitive and reasonable. A CF-based matrix factorization is used to learn the latent space for measuring model-data similarity. A forward mapping function f is learned to transform data features to the latent space. A PCA step is included as pre-processing. In the CF step, a back-propagatable DCG-based loss is developed to replace traditional MSE loss. However, it would be better to report this advantage through experiments.

Clarity: The paper is well-structured and well-written.

Significance: Model selection is a crucial task for outlier detection. The paper (partially) tackles the important problem, as long as practitioners have already accumulated a significant number of historical datasets and evaluated models. The testbed setup consists of two scenarios (POC and ST), which is an interesting setting. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method, named MetaOD, which for each input data set selects the most suited outlier detection model together with its relevant hyper-parameter values from a set of given models. The selection is done in a supervised manner. MetaOD works as follows. During the meta-training stage, it considers a set of $m$ models $M = \{M_1, ..., M_m\}$ and a set of $n$ data sets $D = \{D_1, ..., D_n\}$. Each model $M_i$ includes model architecture and model configuration (hyper-parameter values). MetaOD computes performance matrix $P$ where cell $P_{ij}$ is the performance of method $M_j$ on data set $D_i$. Then it learns a data matrix $U \in R^{n \times k}$ and a model matrix $V \in R^{m \times k}$ where $U_i \times V_j^T$ approximates $P_{ij}$. The functions learnt from $U$ and the matrix $V$ are then used during test time to choose the best model (together with its configuration) for each input data set.
main_review: Unsupervised outlier detection is a challenging task as it lacks a universal objective function and reliable evaluation data sets. Methods proposed so far have their own notion of outlier scores and do not consistently have best performance. This makes it hard in practice to choose a suitable OD method for the data set at hand. As a plus point, the paper tackles the issue by mapping unsupervised OD model selection to the collaborative filtering cold-start problem, enabling the transfer of well-established solutions in CF to OD.

I find the solution of finding $U$ and $V$ through matrix factorisation on $P$ (solved by alternating optimisation) straightforward and standard. I have two concerns though. First, while $V$ is used directly during test time, $U_{\mathit{test}}$ is just a mapping of the test set through a series of three functions, one of which is regression function $f(\cdot)$. Function $f(\cdot)$ on the other hand is not optimised together with $U$ and $V$ during meta-training stage, and hence might be sub-optimal to reconstructing matrix $P$. Is there a way to learn $f(\cdot)$ during alternating optimisation? Second, MetaOD focuses on selecting the best model. It would be more interesting if the authors solve the more general problem of selecting top-k models (to build an ensemble) and make the top-1 a special case.

The experiments look extensive with 2 settings: POC and stress tests. I find the baselines and data sets sufficient. In addition, MetaOD achieves good empirical performance and takes about 1 second to choose the best model for each input data set.

The writing has room for improvement. For instance, a table summarising the notations used would make the paper more readable. The part about data-specific meta-feature extraction is important and should be explained in the main part of the paper (e.g. through an example data set).
limitations_and_societal_impact: The authors do address the technical limitations of the paper that they are able to identify.

The authors do not identify any societal impact of the paper. However, one thing they could explore, perhaps in future work, is how biases embedded in base OD models and historical data sets could influence the selection of best model for incoming test sets.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a data-driven method, METAOD to unsupervised outlier model selection problem based on meta-learning. METAOD makes use of many detection models on historical outlier detection benchmark datasets and selects an effective model to be employed on a new dataset without. Authors introduce meta-features that quantify the outlying characteristics of a dataset. Experiments have shown the promise of METAOD. The authors have implemented an open-source tool METAOD and our meta-learning database for practical use.

main_review: The authors tackle a very challenging problem, unsupervised outlier model selection. Evaluating outlier detection methods is very challenging because of no ground truths and a lack of objective function makes unsupervised outlier model selection. The authors approach the problem as a collaborative filtering recommendation problem, and this is quite interesting.

Overall, I feel this method has some novelty. The selection of an outlier detection method is challenging, and the method proposed can be a practical solution. This paper utilises prior knowledge about outlier detection models (their performance on historical labelled datasets) and the similarity of an (unlabelled) test dataset to historical datasets to determine the best model for the test dataset. A key step is to find the similarity, i.e., meta-features. Two types of meta-features, statistical and landmark features, are designed to capture the similarities of two datasets in the general case and outlying aspects, respectively. The design of the method is reasonable. 

However, I have some specific questions about the implementation. 

The success of the proposed method depends on several factors, and the two major ones are as the following: 1) the outlying characteristics can be precisely quantified. 2) The performance of a method can be accurately measured. 

The features and measures proposed are heuristics, and they need justification. The statistic features are very general and they may not reveal any outlying characteristics since otherwise, outlier detection can be based on these general statistics features. Furthermore, data sets have different dimensionalities. Can statistical properties of data sets with different dimensionalities be comparable?

Landmark features:  what are justifications for using the four OD methods? Do the features capture outlying chrematistics of data sets? 

The meta-features can be many. Will this cause curse of dimensionality?

Equation (2) seems arbitrary for me. Why are the three functions used?

As the authors discussed, it is very difficult to quantify the performance of an outlier detection method. How do the authors resolve this problem?

limitations_and_societal_impact: See above.
review_title: null

Meta-review: All reviewers have praised the importance of selecting outlier detectors in an automatic way and recognized the proposed meta-learning solution as a novel contribution. 

Important points were raised during the reviewing and discussion period. They concern: 
  - the influence of the distribution similarity between train and test to make the meta-learning succeed, 
  - the effect of some hyperparameters involved and 
  - some rewriting of the presentation to ease reading (notation + English grammar typos) and to better motivate the use of a collaborative filtering approach.

The paper is accepted subject to the above points are addressed in the camera-ready.
Decision: Accept (Poster)