AutoMS: Automatic Model Selection for Novelty Detection with Error Rate Control
Authors: Yifan Zhang
Haiyan Jiang
Haojie Ren
Changliang Zou
Dejing Dou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model Selection
FDR
Novelty Detection

Abstract: Given an unsupervised novelty detection task on a new dataset, how can we automatically select a ''best'' detection model while simultaneously controlling the error rate of the best model? For novelty detection analysis, numerous detectors have been proposed to detect outliers on a new unseen dataset based on a score function trained on available clean data. However, due to the absence of labeled data for model evaluation and comparison, there is a lack of systematic approaches that are able to select a ''best'' model/detector (i.e., the algorithm as well as its hyperparameters) and achieve certain error rate control simultaneously. In this paper, we introduce a unified data-driven procedure to address this issue. The key idea is to maximize the number of detected outliers while controlling the false discovery rate (FDR) with the help of Jackknife prediction. We establish non-asymptotic bounds for the false discovery proportions and show that the proposed procedure yields valid FDR control under some mild conditions. Numerical experiments on both synthetic and real data validate the theoretical results and demonstrate the effectiveness of our proposed AutoMS method. The code is available at https://github.com/ZhangYifan1996/AutoMS.

AutoMS: Automatic Model Selection for Novelty
Detection with Error Rate Control
Yifan Zhang'* Haiyan Jiang Haojie Ren Changliang Zou' Dejing Dou
1 School of Statisties and Data Sciences, LPMC, KLMDASR and LEBPS, Nankai University, Tianjin, China
2 Machine Learning Department, MBZUAL, Abu Dhabi, UAE
Baidu Research, Baidu Inc., Beijing, China
School of Mathematical Science, Shanghai Jiao Tong University, Shanghai, China yfzhang_starr sttrgs_stttmmmm Cmail. nankai edu. cn, haiyana jjiangembzuai. ac. haojierenesjit juu edu, cn, zoucl @nankai edu. en, doudej ing@baidu com
Abstract
Given an unsupervised novelty detection task on new dataset, how can we au- tomatically select "best" detection model while simultancously controlling the етror rate of the best model? For novelty detection analysis, numerous detectors have been proposed function trained on available clean data. However, due to the absence of labeled anomalous data for model evaluation and comparison, there is lack systematic approaches that are able to select the "best" model/detector (i.e., the algorithm well as its hyperparameters) and achieve certain error rate control simultaneously. In this paper, we introduce unified data-driven procedure address this issue. The key idea is to maximize the number of detected outliers while controlling the false discovery rate (FDR) with the help of Jackknife prediction. We estab- lish non-asymptotie bounds for the false discovery proportions and show that the proposed procedure yields valid FDR control under some mild conditions. Nu- merical experiments on both synthetic and real data validate the theoretical results and demonstrate the effectiveness of our proposed AutoMS method. The code is available at: https:/ //gi thub. wwtuuu.ccmm//// com comm/hangYiaae Mianngiffann ann an 1996/ 1996/AuttM... AutoMS.
detect outliers on a new unseen dataset based on score
Introduction
With the development of big data, novelty detection has become critical for many machine learning applications, such as network intrusion detection [1], fraud detection in finance [2, 3], event detection in the earth sciences [4], and so on. For these applications, there are a very large number of "normal" instances (i.e., inlier) but lack of "abnormal" data (i.e., novelty, outlier). because collecting abnormal data is much more expensive or difficult than collecting normal data [5]. This leads us to situation where the available historical dataset contains only normal observations, and the basic goal is to detect outliers on new unseen test dataset. Novelty detection has achieved remarkable success in many application domains with numerous methodssalgorihhe learning literature [6-8], however, two problems need to be addressed in the current novelty detection models, (1) lack of error rate control in detection algorithms, and (2) ad-hoc model selection of Aggoritns/hyperpprmmeee.
being developed in machine
The most common way to solve the novelty detection problem is a plausible score function that assigns a score to each point in the test dataset. Generally, data with large scores are more likely to be detected outliers. A decision threshold is given to determine
detection algorithm to learn
Y. Zhang did this work during her internship at Baidu Research.
i Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

whether a data point is an outlier or not. There is an obvious off between precision and recall: a small threshold would detect more outliers but make more false discoveries as well. Therefore, more viable way is to reformulate the novelty detection problem as multiple hypothesis testing problem, and adopt some error rate control procedure as suitable criteria to choose the threshold [9].
Model selection (including algorithm selection or hyperparameter tuning) is a well-known feature of these novelty detection algorithms and arises naturally since none of these methods can universally outperform others on all tasks as noted in Wolpert and Macready [10], Zhao et al. [11]. Due to the special setting of novelty detection, model evaluation and model comparison are not feasible when only "clean" data are provided as training dataset. Those strategies are usually ad-hoc [11, 12], which may greatly hamper their applicability to novelty detection in general cases. Furthermore, these methods only target the recall through model selection without considering the false discoveries. Therefore, it is desirable to develop a unified framework to select the "best" detector with certain error rate control.
We see that designing a data-driven procedure select the "best" model/detector while simultancously achieving some eπror rate control remains an open challenge. In this paper, we propose a data- driven procedure, AutoMS, implement automatic model selection for novelty detection with error rate control. We apply the Jackknife technique [13] to fully explore the clean data and avoid the randomness caused by data-splitting. With the help of Jackknife prediction, we select from a pool of diversified models/detectors the "best" detector that maximizes the number of detected outliers while controlling the false discovery rate. The theoretical guarantees for the FDR control of the selected "best" detector requires further investigation because the error distribution of the "best" detector is different from that of a given detector. We address this issue by carefully studying the bound of false discovery proportion (FDP) and FDR which is calculated by taking expectation of FDP.
To our best knowledge, this is the first work systematically select the "best" model while achieving certain error rate control. The main contributions of this work are summarized as follows. (I) The proposed AutoMS can select the best model and simultaneously control the error rate of the best model. Notably, AutoMS is a unified data-drives framework, and it does not rely on any labeled anomalous data for model selection.
(2) To our best knowledge, it is the first effort to select the "best" model/detector with theoretical guarantees in the view of FDR control. We establish non-asymptotic bounds for the FDP and show that the proposed AutoMS yields valid FDR control.
(3) The AutoMS can be easily coupled with commonly used novelty detection algorithms. Extensive numerical experiments indicate that AutoMS outperforms other methods significantly, with respect to both error rate control and detection power.
2 Related Work
Reformulating the outlier detection problem multiple testing problem, commonly-used error rate in this setting is the false discovery rate (FDR): the expected propor- tion of true inliers among the test points reported as outliers. The FDR control [14-16] has been fully studied in the statistical literature and is a particularly useful tool to maintain the ability to reliably detect true alternatives without excessive false discovery results when large-scale hypotheses are simultaneously tested.
FDR Control.
Along this line, Bates et al. [17] pioneered a method to split the clean dataset into two independent parts, one for training the score function and one for validating the empirical cumulative distribution function of the score function which will be used for constructing p-values for the new observations. They showed that the FDR control can be achieved with thcoretical finite sample guarantees. It involves, however, one-time random split of the clean dataset, which leads to "p-value lottery" problem. That is, those p-values based on sample-spliting are sensitive and difficult to be reproduced in practice. Moreover, the procedure in Bates et al. [17] can only control the FDR for a given novelty detection model, and no model efficiency is considered in their paper.
Model Selection in Novelty Detection. As described in Sect. 1, various detection algorithms
have been developed for different problems or different types of data, such as LOF [18], one-class SVM [19], iForest [20] and so on. More discussions can be found in Pimentel et al. [6], Aggarwal [21]. However, no universal learning model performs well on all problem instances Wolpert and Macready [10]. Therefore, it raises the problem of model selection, including algorithm selection

---

and hyperparameter tuning. For example, AutoOD [12] proposed an automated outlier detection framework focusing on automatic neural architecture search. Recent work METAOD [11] built measure for task similarity and use meta-learning to select the most similar task compared to the historical ones. Most existing work is usually based on sophisticated black-box machine learning algorithms to learn the space of the normal data. While generally effective in practice, they rarely offer a clear guarantee about the output.
We note that some statistical criteria can be used for model selection in novelty detection; e.g. using voluue and excess-mas [22] or some clustering validation metrics [23] to measure the quality of score function, and using IREOS [24] to assess the separability. However, Ma et al. [25] indicated that those criteria are not useful in practice and cannot lead universal detector for general cases. Our proposed AutoMS is not limited to one specific algorithm or model but can select the "best" one among any set of models/detectors.
Methodology: Automatic Model Selection
In novelty detection problems, we are given a clean training dataset of inliers D = 1
containing independently and identically distributed (i.id.) observations Z, which is drawn Po- The goal of the novelty detection problem is to identify the outlier set O = (Z, E U : Z, ~ Po] among new unseen test dataset 2/ = (2
One of the most common approaches in addressing novelty detection to employ detection algorithm M to learn a plausible score function Sm(-)) with the clean dataset 'D and use the score SM (Z,) assess whether Z, E U is an outlier. Generally, it is assumed that the larger the score SM (.). the more likely Z; is outlier. Hence, the decision set (detected outlier set) is selected as: Om-- {Z, E E (74 > L.M), where LM is threshold associated with detector M. Denote the subset of containing all inliers as OF = (2 E u : Z; ~ Po} with size no, and OU = U. Let |A| denote the number of the elements in set A. Denote a b max(a, b}.
Taking a multiple testing perspective, the null hypothesis is that a new observation an inlier, and the alternative asserts it is an outlier. That is, the novelty detection problem is translated into multiple testing problem as follows,
Hoo: Zi Po V.s. H1s Ze ~ Po i= 1, n.
3.1 Problem Definition
Problem (Model Selection of Unsupervised Novelty Detection) The model selection problem for unsupervised novelty detection problem can be stated as follows.
Given a training dataset of inliers D = 1Z j= a new unlabeled dataset 44 = (2,) which may contain outliers, and all models G = UM;
Select the best model M E such that the FDR of M is controlled under the target level a.
The false discovery proportion (FDP) and true discovery proportion (TDP) are defined as follows
16m no
1 nol
FDP-
TDP
(2)
1
10
The false discovery rate (FDR) is the expectation of FDP, i... FDR = E(FDP). The true detection rate (TDR) is defined as the expectation of TDP, i..., TDR = E(TDP). A detector is superior to another if the former has a larger TDR for the same FDR level.
3.2 Construction of AutoMS
We begin with the FDR control for given detector M and then give the AutoMS of how to select the "best" detector from G. For detector M, we learn a score function (-(() based on the historical
clean dataset D and evaluate the score 5M Zz)) at the new data point Z, E U. The canonical approach
to FDR control multiple testing problem to apply the Benjamini-Hochherg (BH) procedure [14- 16] to the p-values of scores (M()))) for any Z; E U. That is, the data-dependent threshold L is
3

---

selected as follows,
nlM(()
)
I = inf 1>0: 0: 112, E u: SM (z.) 2 tt V1 S saj.
where GM(()- = PH, (S,(Zz) 2 t/D) is the p-value of (M, for Zi E U when Ho holds.
However, is quite a challenge estimate reliable p-values (() since Po is unknown and the score function (.) may be derived from a sophisticated black-box machine learning algorithm. Bates et al. [17] suggested randomly splitting 'D into disjoint parts with equal sample size, one for training the score function (.)() and the other for estimating (m((t with the help of conformal inference. This magle-random-sslitiin based method suffers from, (1) the estimation of the p-values depends on the random split; (2) different split ratios also lead to different results.
To yield more stable estimates of p-values in practice, we propose to learn the score function (.(()) on the whole clean dataset D, where our key idea stems from the Jackknife estimate. Denote Di = D DVL2, } as the subset of D with the j-th observation removed. Define (- as the score function trained on D(- Then we can evaluate m scores (S 1 (2z) on the whole clean dataset D. The p-value function can be estimated by
Gm.M(t) = K2, D: : S 1 (Z,) 2 t}
m
The Jackknife technique provides an approximation of the score, s/ (2,), for any Z, E D. Under the null Ho that Z; E U is from Po. (2,) can be used for estimating the empirical distribution of SM (Z.). And the corresponding threshold LM is determined via
n.m. LM = inf {0 < fm,M H2z u SM(Z) 2 2 S Saj. }
(4)
where Mm.M is the upper bound for t and its choice will be discussed in Sect. 4 in detail, and S the same score function trained on D. Then, the detected set is identified as
θm {Z: ut: SM, 2 LM}.
(5)
Further, we select the "best" detector M via
M arg max MED
(6)
Algorithm 1 AutoMS: Automatic Model Selection
1: Input: New dataset L4, clean dataset D, target FDR level a and a pool of detectors g.
2: for M do
3: for = 1, m do
4: Train score function sl (-) based on Dl-1l (i.e. DI = D({
5: Compute the score at Z, E D. i.e. s/ (2,):
6:
end for
7: Learn score function (.).) based on D;
8: Compute the scores SM 7R) for any Z; U, and find threshold T.M by Eq. (4);
9: The detected set is θ = {2: E u (Z.) 2 LM}.
10: end for
11: Select the "best" detector M from via Eq. (6).
12: Output: θ = (Z, E 4: S (Za) 2 L.
We refer to procedure as AutoMS short for Automatic Model Selection and it is summarized in Algorithm 1. In fact, our AutoMS consists of two phases: offline training in lines 2-7 and online prediction in lines 8-9 of pseudo-code in Algorithm 1. Note that the running time of offline training not critical. Meanwhile, the additional running time is the detector selection in line 1 11, which should incur a small run-time overhead due only maximizing finite set.
Note that the pool of detectors G = UM can be a set of different algorithms or set of hyperpa- rameters for one typical algorithm, or both, which fits the definition of model selection including

---

algorithm selection and hyperparameter tuning. Note that the size of G, [g], can be infinite, and [ is allowed increase as the size of D. More details will be discussed in Sect. 4.
Rationale. Here is the intuition why the selected detector obtained by AutoMS is the "best" one. The general idea is as follows. A good detector needs to achieve a large TDR while controlling the false discoveries under the target FDR level a. With FDR control, the selected detector which has the largest number of discoveries is roughly the one with the largest TDR. Otherwise, simply selecting the largest number of discoveries without FDR control, is not even a correct criterion, because in this case, most detected discoveries may be inliers and true outliers may not be detected. For every detector M E G, with FDR control, its FDP can be approximately controlled around the target FDR level a, as the proposed AutoMS resets the corresponding novelty threshold for every M. Therefore, there are roughly M (1 a) true outliers in the detected set OM. with the TDR being
approximated by (0x1(1_a) Since the number of true outliers |O] and the target FDR level are [0]
fixed in given novelty detection task, selecting the largest [( namely the largest [0M-) |. leads to the "best" detector M
[0]
Why Jackknife not Jackknife+. Barber et al. [26] introduced the Jackknife+ as a new method for constructing predictive confidence intervals. They provide non-asymptotic coverage guarantees under no assumptions beyond the training and test dataset being exchangeable and also study the theoretical guarantees for the original Jackknife with stability assumptions (see Section 5 of their paper). Jackknife+ needs to calculate m leave-one-out predictions for one test point while the original Jackknife only calculates one predicted value [26]. The model evaluation cost (counts the number of times evaluating a fitted model on a new point) of Jackknife+ is mn whereas the original Jackknife of cost n, indicating that Jackknife+ has a much higher computation complexity compared to the original. Thus, since most novelty detection algorithms satisfy the stability assumption, we are more inclined to apply the original Jackknife technique rather than Jackknife+ to detectors that meet stability assumption, and we also give theoretical guarantees to achieve the non-asymptotic bounds of FDP and maintain FDR control.
4 Statistical Performance Guarantees
In this section, we will theoretically establish NON-asymptotic bounds for FDP of the selected detector M by AutoMS and further show that it can control the FDR under the following mild assumptions. Define the number of detectors wm = | and let A,, = o(n) be pre-specified sequence.
Assumption 1 (Density). For any detector M E G and sufficiently large m, the conditional density of SM given training dataset D uniformly bounded above by a universal constant cf >> where Z E l is a new point.
Assumption 2 (Learning stability). For any fixed M, the score SM satisfies: for large enough m,
P([SM(Z) - -um (2) < Km) 2 1 Sm.
with some sequences satisfying Ki.. = o(1) and Sm 0(1) as co, and some function jiM- Assumption 3 (Rates). Denote Bn = 2m + 36m- Suppose that Bnn o(1), nw By = (1) and mww / 2/3 o(1).
Our main theoretical result the validity of the AutoMS method for both FDP and FDR control is given by the next theorem.
Theorem (FDR control). Suppose Assumptions 1-3 hold. Let 0 < 8 < and 0 < a There exist universal constants Ci > 0 and C2 >> O so that the FDP of the proposed method satisfies
FDP S a 1 4nBm
+ +C.
wm
+ C2p
8 W mn
An
5A213
6
with probability at least 1 28, and
limsup FDR(LM*) S a (m,n)-+00
(8)
where W mn
Am77 + n
(1 15Bm + 2m Bm)
4/3
5

---

It is worth mentioning that even if all detectors control the FDR, we cannot assert without theoretical proof that the optimal detector M can still guarantee the FDR control. And Bates al. [17] only gives the expectation version that FDR can be controlled for a given detector. Briefly, we first prove that the FDP of each detector can be bounded uniformly through the uniform convergence of the Jackknife estimator. Hence, the selected "best" detector M with the largest number of discoveries will have the same non-asymptotic bounds for FDP. Then we give the asymptotic FDR control for the optimal detector. Detailed proofs for theoretical results are provided in Appendix E.
In Eq. (4), we specify a data-driven upper bound Em when determining L. We define E... m (An/n) nn) where Gm.. (t) is defined in Eq. (3) and G (m. inf { 2 0: Gm. S y} with 0 S y S 1, The upper bound fm.M for threshold LM can facilitate the derivation of non- asymptotic bounds for FDP but is actually not required in practice. In other words, this makes an implicit assumption that a non-trivial (not oo) threshold can be attained within the interval (0, [..M]. By Theorem 1, we can see that large A, would yield faster convergence of FDP. However, too large A., may lead to a trivial threshold. In fact, how large An could be depends on the number of identifiable outliers in 2 [27], which is certainly unknown to Alternatively, by examining the requirements in Assumption 3, we can set A, min (mwm) )², n n!/23 for simplicity.
Remark 1 (Stability). Since we use the Jackknife to estimate FDP, certain stability conditions seem to be necessary. In Assumption 2, j needs not to be close to an optimal population detector. We only need the SM concentrate around jr. This is just stability assumption rather than consistency assumption. When the sampling stability fails to hold, the s distribution than (t). In that case, we may resort to the sample-splitting procedure instead.
J.M may have a substantially different
Remark 2 (Rates). In general, Assumption 3 is used to ensure the uniform consistency of the estimated p-values, which performs an important role in the convergence of the FDR control. Specifically, A, = (n) is a pre-specified sequence which gives a non-trivial lower bound of the true p-value function (t) with (t( 2 A- o(1). The Bm, is a quantity used to measure the difference of two p-values estimated in different ways, as in Lemma C.3 and Lemma C.2. And B,, = o(1) indicates the two p-values should be close enough which is very mild condition, and it can be expected that the p-values will be close if the training sample sizes of inliers m and test sample sizes of inliers no large enough in practice. In the proposed AutoMS, the number of detectors wm = [g] can go to infinity with some certain rate related to the training sample size m and test sample size n. The assumptions nww B,, = o(1) and mwm/AA 2/3 o(1) indicate the required rate of wm to ensure the uniform consistency among detectors.
Remark 3 (FDP bound). Although the FDP bound might be slightly larger than the target FDR level finite samples, the FDP bound converges asymptotically to the target FDR level a. Actually, in novelty detection tasks, it is not necessary to obtain procedures of finite FDR control with additional assumptions, instead, a powerful and stable procedure which controls FDR asymptotically is good enough for most cases.
5 Experiments and Evaluation
5.1 Experiment Setting
In this section, we evaluate the performance of the proposed AutoMS in terms of FDR control and detection power (TDP) through extensive numerical studies both synthetic and real data. We report the results based on 100 replication experiments throughout this paper. We construct a detector set G with 83 candidate models by combining 6 novelty detection algorithms and their corresponding hyperparameters (See Table S1 in Appendix A for the complete list). These models/detectors are available in the Python "PyOD" [28] package. Note that the size of detector set G can be infinite and our AutoMS can still be used.
Performance Measures. Since we focus on both error rate control and detection power, we use FDR = E(FPP) and TDR E(TDP) as performance measures for each model in this paper. According to the definitions in Eq. (2) of Sect. 3, we have FDP = 1 - Precision, and TDP = Recall. Correspondingly, the empirical FDR and empirical TDR are calculated by averaging of FDPs and TDPs over 100 replications, respectively. We say a detection procedure is superior to its competitor if it has a larger TDR for the same target FDR level a, that is, a superior detector has a larger empirical TDR whose empirical FDR is below or around the target FDR level a. The preset target FDR level

---

should be an acceptable level, which means all detectors that control the FDR at this level a should be accepted. As long as the FDR of the selected detector below preset level a, we have achieved our goal on FDR control and we should focus on improving TDR.
Baseline methods. To evaluate the performance of the proposed AutoMS method, we compare the proposed AutoMS with two different types of baselines: (i) model selection meth- ods (METAOD [11]) and (ii) error- rate-controlled methods (SRS-based methods [17]). And we also consider two versions of the proposed AutoMS:
Table 1: Model-selection and eroo-rate-contro methods.
Method
Model Selection
Error Rate Control
SRS [17]
METAOD [11]
X
AutoMS- SRS
AutoMS-JK
AutoMSS-JK and AutoMS--RS, where AutoMS-JK uses Jackknife technique and AutoMS-SRS replaces the Jackknife technique with single-random-splitnng (SRS) technique for estimating score. We explore the performances of the four methods with respect to FDR and TDR. It is worth mention- ing that SRS is guaranteed with FDR control for any given detector, but there is no model selection considered in SRS-based methods. METAOD is model selection method using meta-learning without considering error rate control, while the AutoMS can do both model selection and error rate control, shown in Table 1. For more details on the baseline methods, Refer to Appendix B.
5.2 Evaluation of the Jackknife
Firstly, We would like to show the ad- vantage of Jackknife prediction with toy example using synthetic data. We
0120
1.00
TDP (Recall)
0.75
generate a training dataset DD and
test dataset U from a Gaussian mix- ture model, where their sample sizes are m = n = 2000 respectively and u contains 10% outliers. More details of the setting can be found in Sect. 5.3, In Fig. I, we compare the Jackknife pre- diction to the (SRS) method in Bates et al. [17] with different split ratios when the detector M the OCSVM with default hy- perparameter in the PyOD package [28]. Focusing on one training set DD we ran- domly split D 100 times for each split ratio of SRS method while Jackknife technique only has one prediction result. We can see from Fig. I that the Jackknife provides a quite tight FDP control and achieves a high TDP while finding more true outliers. As discussed above, the SRS-based methods depend on the random split and the split ratio and are usually unstable due to the randomness of the segmentation, making it hard to reproduce.
0.50
0.05-
1100
0.00
1:5 1:4
1:3
1:2 1.1 2:1 31 Split Ratio
4: 3TT
1:5 1:4 1:3 1:2
2:1 3141115
Split Ratio
Figure 1: Performance of SRS-OCSVM [17] under differ- ent split ratios. Each box-plot shows the distribution of FDP and TDP under 100 simulations. The red line shows the corresponding results from Jackknife prediction. The black dashed line is the target FDR level a 0.1.
The Jackknife estimate helps improve the stability and the accuracy of the scores due to the usage of more samples to train the score function. Meanwhile, the Jackknife procedure will require bit more computation compared to one-time data splitting, if m not too small. As the Jackknife procedure for SM
-1 (-) is trained offline, one can always replace the Jackknife with cross- vols--valiation to reduce the computing burden to a certain extent. Instead of omitting one observation, we can randomly partition D into & equal-sized folds and remove one fold from D as the subsample to train the score function. In the following of this section, we use 100 folds in the AutoMS -JK procedure.
5.3 Synthetic Data
Data Description. We consider the same synthetic setting as in Bates et al. [17]. We will compare the performance of different methods (the proposed AutoMS-JK, AutoMS-SRS, SRS-based methods and METAOD) when the target FDR level is set to a 0.1. Then we will show the results under different target FDR levels. The data are generated by sampling each data point Z, E R250 from a multivariate Gaussian mixture model, Z, W.+ VaV, where W, is one of 50 cluster centers sampled
7

---

independently from the uniform distribution on |-3, 3 and V E R8 are i.idd from ( 1). The difference between inliers and outliers is the constant with Hp a 1 v.s. H1 clean dataset DD with m 2000 observations and test dataset // with n = 2000 observations in which 90% are sampled as inliers. We aim to identify the remaining 10% of outliers U.
We generate
AutoMS-JK
AutoMS-SRS
SRS-XNN
SAS-OCSVM
SRS-10F
SRS-11BOS
SRS-1loores
SRS-10DA
METAOD
1.00
0.75
0.30
0.25
1,00
TEP Giirall
0.50
0.25
0.00
G (aaan
6 Aaaer Cimen
(00 baspe ON mane
011 0r uon
120
000 900 00
On
(11
Hyperparameters
Figure 2: Comparisons of FDP (more stable around the dashed line) and TDP (the larger the better) among AutoMS-JK, AutoMS-SRe SRS, SRS-based methods (with different hyperparameters) and METAOD. The dashed line the target FDR level a which set to a 0.1.
Results. With the given target FDR level 0.1, the AutoMS (either AutoMS-JK JK or AutoMS-SiD SRS) outperforms all baseline methods as shown in Fig. 2. (a) All the SRS-based methods can perfectly control the FDR with FDPs scattering around the given target FDR level But in terms of detection power, the TDP performance SRS-based methods vary greatly with different hyperparameters and different algorithms. Some SRS-based methods, such SRS-LODA. SRS-HOBOL even lose detection power with almost zero TDP/recall. This necessitates model selection among numerous novelty detection methods. (b) On the other hand, METAOD gives large FDPs, which means that most of the reported points are not true outliers. This phenomenon can be explained as METAOD does not employ any FDR control procedure. And the TDPs of METAOD do not perform well because the model selection of METAOD is based on historical data which may not be suitable for the data we use. (c) AutoMS-JK and AutoMS-SRS are superior all SRS-based methods and METAOD with respect to the average TDR, with the average FDR slightly above the target FDR level O, as is more difficult control the FDR the selected detector than given detector. Additionally, AutoMS-JK gives higher TDR than AutoM--SRS which indicates that using the Jackknife method instead of SRS enhances detection power. Although both AutoMS and SRS- methods can control the FDR, AutoMS has better detection power with a larger TDR compared to SRS-base methods. (d) The reason why the FDR control AutoMS more difficult than SRS-based methods is that the FDP distribution of our selected detector is different from given detector. Take simple example: there are 10 standard Gaussian variables, each with a mcan 0. The largest variable among those 10 does not have mean O. Therefore, the selected detector is more difficult to control FDR.
Fig. shows the performance of AutoMS different target FDR levels. In particular, SRS-LOF bestt is the best SRS-LOF detector with the largest TDR among all its candidate hyperparameterss. Although all SuSS--ased-bes methods can control the FDR under any target FDR level, the average TDPs of methods at every target FDR level. The average FDP of AutoMS-JK/Auton SRS is slightly larger than the given target FDR level a, but their difference FDP does not increase as a becomes larger. The FDR of AutoMS-JK has smaller deviation above the target a than AutoMS-SRA Actually, in novelty detection tasks, it is not necessary to obtain procedures with
AutoMs-SRS and SRS-based-bel methods under
SRS are higher than all
8

---

AutoMs-JK
AutoMS-SRS
SRS-RNN
SRS-LOF-
SRS-OCSVM-
1.00
0.1
0.75
FDP -a
0.0
0.50
-0.1
0.25
-0.2
0.00
0.05
0.1
0.15
0.2
0.05
0.1
0.15
0.2
Target FDR level a
Target FDR level
Figure 3: Performance of AutoMS-JK, AutoMS-SRS and SRS-based-bos methods with different target FDR level a. The difference between the FDP and the target FDR level shows the deviation of FDP from a.
finite FDR control, instead, a powerful and stable procedure that controls FDR asymptotically around the target FDR level is good enough for most cases. That is what AutoMS does. AutoMS can uniformly offer much higher TDPs than SRS-based-best methods under different target FDR levels.
5.4 Real Data
We compare the proposed AutoMS with two different types of baselines: (i) model selection methods (METAOD [11]) and (ii) aaror-rat-coontrolle methods (SRS- based methods [17]), as in Table
Datasets. Four real-world datasets for outlier detection are considered, and the details of the datasets are summarized in Table 2. We use empirical TDR (Recall) and empirical FDR (I-Precision) as performance measures. For real data implementation, we construct the training dataset 'D by randomly sampling m min (00000, N inlier samples from the total No inliers. Then we focus on test dataset generating mechanism under which 90% of the n min 22000, (N, N/1/6} observations each test dataset UI are sampled from No inliers, and we seek identify the remaining 10% of outliers (i.e., 10% outlier ratio). For results on 11 more real-world datasets [29] used in Zhao et al. [11], refer to Appendix D.
Table 2: Summary real-world datasets for novelty detection.
Credit Card [30]
Covertype [31]
Satellite [32]
Shutrle [33]
Features
30
10
36
9
# Inliers No
284,807
283,301
5,732
45,586
Outliers N₁
492
2747
703
12,414
Results. Fig. 4 shows the performance of different methods on real-world datasets, and Table gives the empirical FDR and empirical TDR of different methods, where SRS-best is the best SRS detector with the largest TDR among all detectors. From Fig. 4 and Table 3, we can observe that AutoMS can maintain the FDPs well around the target FDR level 0.1, while METAOD has inflated the empirical FDP to values around or above 0.5 on all real-world datasets, indicating that in the detected set O obtained by METAOD, around 50% of the detected outliers are not true outliers. It worth mentioning that, all SRS- methods can maintain the FDR level around 0.1 as well as expected from their theoretical guarantees. As for true detection rate (TDR), AutoMS has reasonable empirical TDR on all datasets, while SRS- baseb methods sometimes perform too conservatively, have no detected outliers and their performance varies greatly from dataset to dataset.
In addition, the performances of SRS-LODA-best and SRS-LOF-best are rather different, particularly on the "Covertype" dataset and "Credit Card" dataset, which means SRS-based based-best methods can not give consistent good result on different tasks and its performance can be severely affected by base detector the SRS used. On the other hand, AutoMS keeps accurate and stable FDR levels on all datasets and has reasonable TDR for different tasks. Overall, AutoMS can uniformly offer much higher TDRs than RRS-based-bee best methods across all settings and AutoMS can maintain the FDR

---

K
AutoMS-SRS
SRS-LODA
SRS-LOF-beS.
METAOD
0.8-
1.00
FDP (1-Precision)
0.6
0.75
TDP (Recall)
0.4
0.50
0.25
0.0
0.00
Covertype Credit Card
Satellite
Shuttle
Covertype Credit Card Satellite
Shuttle
Figure 4: Performance on four different real-world datasets. Each box-plot shows the distribution of FDP and TDP. The dashed line is the target FDR level = 0.1.
level with comparable TDR compared with METAOD. Table 4 shows results under different target FDR levels (refer to Appendix C for more results and details).
Table 3: Empirical FDR and TDR of different methods when the target FDR level is set a 0.1.
Dataset
AutoMS-JK
AutoMS-SRS
SRS-best
METAOD FDR
FDR
TDR
FDR
TDR
FDR
TDR
TDR
Covertype
0.106
0.933
0.105
0.905
0.090
0.890
0.487
0.960
Credit Card
0.099
0.758
0.094
0.757
0.086
0.674
0.508
0.856
Satellite
0.114
0.968
0.134
0.974
0.069
0.963
0.495
0.948
Shuttle
0.100
0.942
0.102
0.926
0.088
0.914
0.573
0.689
Table 4: Empirical FDR and TDR under different target FDR levels a on "Shuttle" dataset.
Shuttle
0.05
0.10
a FDR
0.15
0.20
Method
FDR
TDR
FDR
TDR
TDR
FDR
TDR
AutoMS-JK
0.042
0.883
0.100
0.942
0.151
0.951
0.209
0.953
AutoMS-SRS
0.051
0.659
0.102
0.926
0.159
0.952
0.216
0.956
SRS-HBOS-bess
0.024
0,400
0.083
0.554
0.131
0.733
0.177
0.748
SRS-OCSVM-best
0.010
0.139
0.088
0.914
0.133
0.942
0.177
0.946
METAOD 0.573
0.689
0.573
0.689
0.573
0.689
0.573
0.689
Conclusion
The detector selection problem in novelty detection analysis challenging problem and it has not been thoroughly investigated in the literature, especially in terms of theoretical guarantees. In this paper. we propose an effective data-driven procedure, named AutoMS, that can select the "best" detector which offers high TDR and controls FDR simultaneously. The AutoMS is fully automatic without any supervision and is easily implemented. We give theoretical results showing that the proposed AutoMS can achieve the non-asymptotic bounds of FDP and maintain FDR control under mild conditions. Extensive experiments on synthetic and real-world datasets show AutoMS significantly improves the detection power and yields accurate FDR control.
Acknowledgments and Funding Disclosure
We thank anonymous area chair and reviewers for their helpful comments. Changliang Zou's s research was supported by the National Natural Science Foundation of China Grants (Nos. 1192510 106. 12231011, 11931001 and 11971247). Haojie Ren's research was supported by the National Natural Science Foundation of China Grants (No.12101 and the Shanghai Sailing Program. This work was partially supported by the Technology Innovation Institute and MBZUAI joint project (NO. TWWARRC/2073/20111) Energy-based Probing for Spiking Neural Networks.
10

---

References
[1] Hung-Jen Liao, Chun-Hung Richard Lin, Ying-Ching Lin, and Kuang- Yuan Tung. Intrusion detection system: A comprehensive review. Journal of Network and Computer Applications, 36 (1):16-24, 2013.
[2] Siddhartha Bhattacharyya, Sanjeev Jha, Kurian Tharakunnel, and Christopher Westland. Data mining for credit card fraud: A comparative study. Decision support systems, 50(3):602-613, 2011.
[3] Tru-Hsuan Lin and Jehn-Ruey Jiang. Credit card fraud detection with autoencoder and proba- bilistic random forest. Mathematics, 9(21):2683, 2021.
[4] Jan Verbesselt, Achim Zeileis, and Martin Herold. Near real-time disturbance detection using satellite image time series. Remote Sensing of Environment, 123:98- 108, 2012.
[5] Rémi Domingues. Probabilistic modeling for novelty detection with applications to fraud identification. arXiv preprint
2019.
[6] Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215-249, 2014.
[7] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint 10 14051, 2021.
[8] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojeiech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifyyng review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
[9] Quentin Rebjock, Baris Kurt, Tim Januschowski, and Laurent Callot. Online false discovery rate control for anomaly detection in time series. Advances in Neural Information Processing Systems, 34, 2021.
[10] David Wolpert and William G Macready. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997.
[1I] Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. Advances in Neural Information Processing Systems, 34, 2021.
[12] Yuening Li, Zhengzhang Chen, Daochen Zha, Kaixiong Zhou, Haifeng Jin, Haifeng Chen, and Xia Hu. Autood: Neural architecture search for outlier detection. In 202/ IEEE 37th International Conference on Data Engineering (ICDE), pages 2117-2122. IEEE, 2021.
[13] Bradley Efron. The jackknife, the bootstrap and other resampling plans. SIAM, 1982.
[14] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: series B (Methodological), 57(1):289-300, 1995.
[15] Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate multiple testing under dependency. Annals of Statistics, pages 1 1165-118,, 2001.
[16] John D Storey, Jonathan E Taylor, and David Siegmund. Strong control, conservative point esti- mation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2004.
[17] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. arXiv preprint arXiv:2/04.088) 2021.
[18] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying dessity-based local outliers. Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.

---

[19] Bernhard Schölkopf, John C Platt, John Shawe- Taylor. Alex J Smola, and Robert C Williamson. Estimating the support of a high -igensiieniional distribution. Neural Computation, 13(7):1443- 1471, 2001.
[20] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth IEEE International Conference on Data Mining, pages 413-422, IEEE, 2008.
[21] Charu C Aggarwal. Outlier Analysis. Springer, 2017.
[22] Nicolas Goix, How to evaluate the quality of unsupervised anomaly detection algorithms?
arXiv preprint 1607.00600.0// 10007001555 2016
[23] Thanh Trung Nguyen, Uy Quang Nguyen, al. An evaluation method for unsupervised anomaly detection algorithms. Journal of Computer Science and Cybernetics, 32(3):259-272, 2016.
[24] Henrique O Marques, Ricardo JGB Campello, Jörg Sander, and Arthur Zimek. Internal evaluation of unsupervised outlier detection. ACM Trunsactions on Knowledge Discovery from Data (TKDD), 14(4):1-42, 2020.
[25] Martin Q Ma, Yue Zhao, Xiaorong Zhang, and Leman Akoglu. A large-scale study on unsuper- vised outlier model selection: Do internal strategies suffice? arXiv preprint arXiv:2104.01422 2021.
[26] Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1))486-555 507, 2021.
[27] Lilun Du, Xu Guo, Wenguang Sun, and Changliang Zou. False discovery rate control under general dependence by symmetrized data aggregation. Journal of the American Statistical Association, justaccepe 1-34, 2021.
[28] Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96): 2019, http: jmlrr org/ papers/ v20/19-0000 011 html.
[29] Andrew Emmott, Shubhomoy Das, Thomas Dietterich, Alan Fern, and Weng Keen Wong. Anomaly detection meta-analysis benchmarks. Oregon State University, 2016. URL https: / //ir. library. oregonstate halp/ooncer//datii /ddt///att//4429/155555
[30] Credit Card. Credit card fraud detection, 2016. URL creditcan teardfraud...
ulbb
[31] Covertype. Forest covertype, 1998.
URL http://odds.cs cs. www.ybboooosooooooooooooo edu/
forestcoverrcovettyye---ee -aterrtttteddaaat......m
[32] Satellite. Satellite image, 1993,
URL http:///oddB..s.sttnymnee edu/
satimage- 2-amang---aateee d-ttase...
[33] Shuttle. Statlog (shuttle), 1993. URL
Statlog+ (Shuttle).
[34] Markus Goldstein and Andreas Dengel. Histogram-based outlier score (hbos): A fast un- supervised anomaly detection algorithm. K1-2012: Poster and Demo Track, pages 59-63, 2012.
[35] Tomáš Pevný. Loda: Lightweight on-line detector of anomalies. Machine Learning, 102(2): 275-304 2016.
[36] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427-438, 2000.
[37] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. The Journal of Machine Learning Research, 112293-3000, 2010.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See at the end of Sect. 1, we summarize our main contributions. And the experimental results in Sect. 5 validate the theoretical results in
Sect. 4.
(b) Did you describe the limitations of your work? [Yes] See Remark in Sect. 4, we discuss the stability limitation (Assumption of detector algorithm.
(c) Did you discuss any potential negative societal impacts your work? [No] Our work is foundational research and not tied to particular applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We read the ethics review guidelines and ensured that our paper conforms them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Assump- tion I-Assumption 3 in Sect. 4.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix E in the Supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material in Sect. 3, we give the pseudo-code our AutoMS procedure, and release the code on github https: // //g github...... com/ www.coo///aa/eeeeee /t/////naaaanngggg//ee 199t/9999//.......
URL)? [Yes] See Algorithm 1
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] We describe the generation of our synthetic data in Sect. 5.3 and give the candidate detectors we use in Appendix A of the Supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Our experimental results are all shown with boxplots.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use the Python "PyOD" package and have cited the creator. The real data we use is open source and the URLs are cited.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "w4bHLYK4ZA9", "reviewer": "Reviewer_VjGX", "summary": "The authors propose a model selection method for novelty detection with false discovery rate (FDR) control. Given a detection model $M$, a detection threshold $L_M$ is selected based on the Benjamini-Hochberg (BH) procedure so that the FDR of $M$ is less than $\\alpha$. To estimate the p-values in the BH procedure precisely, the authors propose to apply the Jackknife estimation, which extends the existing work by Bates et al. After estimating $L_M$ for each model $M \\in G$, the model that most detects the novelties with $L_M$ is selected as the best model $M^*$. The authors also give theoretical results to show that the FDP of $M^*$ is non-asymptotically bounded and the FDR of $M^*$ is asymptotically bounded by $\\alpha$. Experiments using synthetic or real datasets demonstrate the advantage of the proposed method against the work by Bates et al. or METAOD.\n", "questions": "It seems possible to apply the model selection using Equation (6) even when SRS is used to estimate $L_M$. If this is true, I would like to see the results of \"model selection + SRS\" in Fig.2-4.\n\nIn Fig.4:\n- Why SRS-kNN and SRS-OCSVM are not compared?\n- Does AutoMS include kNN/OCSVM models as candidates?\n\nIn Algorithm 1:\n- Procedure of learning $S_M^{[-j]}$ should be added between line3-4.\n- line 11: Input -> Output\n\nIn line 189, what is \"the ADNER method\"?\n", "limitations": "The computational overhead of applying the k-fold Jackknife against the original SRS should be assessed in the experiment.\n", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Hyperparameter tuning or model selection is especially hard in unsupervised settings like novelty dectecion. This paper proposes a simple yet effective approach for this problem from the viewpoint of \"maximize_{M \\in G} #detection(M), subject to FDR(M) \\lt \\alpha$.\"\n- The control of FDR of $M$ is mainly achieved by the existing framework of Bates et al., but its Jackknife extention is proposed.\n\nWeaknesses\n- The computational overhead of applying the Jackknife procedure is not negligible, especially when the training set is large\n- Experimental results, e.g. Fig.3, suggest that the FDR control of $M$ gets slightly worse by applying the Jackknife compared to the original SRS (by Bates et al.).\n\n\n=====POST-REBUTTAL COMMENTS========\n\nThanks for the authors' response. The newly added experimental results and authors' response addressed a part of my concerns. I have raised my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "v8HMNqIH8-e", "reviewer": "Reviewer_47FE", "summary": "This work automatically selects a best detection model while simultaneously controlling the false discovery rate.\nThe experimental results shows that the proposed method can control the false discovery rate (FDR) and the true discovery rate (TDR) simultaneously.", "questions": "I have a couple of concerns and questions regarding the method, and empirical evaluations.\n\n1. In page5 rationale,  you explain why we can select the model which detects the most outliers as the \"best\" one.  This idea seems heuristic somehow, but I am still confused about this part.  Consider there are two models  A and B,  and TDP_A =1 and TDP_B = 1. But B is more aggressive and have detect some false novelty.  Based on your strategy, we will choose B as the best,  but actually A is better than B because the FDP of A is smaller.  The issue may be  that you assume that all the models will keep the FDR level $\\alpha$ constant   .\n\n2. The main theoretical result is based on assumption 4.1-4.3. It is very common to require the assumption of learning stability and density. However, the assumption of rates seems  a little complicated here. Could you give more explanation about this assumption?\n\n3. In section 5.2, the results shows that the Jackknife method is better than simple split  for estimating the score function.  However, in later experiments,  it seems that SRS has lower FDP than AutoMS. \nBased on this, could you please adding another version AutoMS-based method in section 5.3 and 5.4, which is AutoMS-simple,  which just replace the Jackknife method with simple split? \n\n\nOther minor comments:\npage5 line 189:  ADNER --> AutoMS??\n\n###########\n\nupdate: Thanks for the author's response. I will keep my score as 6.\n", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is very clearly written and easy to understand. I really enjoy reading this paper and it makes interesting contribution.  The key idea is  estimating more \"stable\" p-value for better FDR and then adding extra step (i.e., model selection) for additional TDR control.  It's not surprising to see in experiments this method have better TDR than those methods without controlling TDR.\n\nThis paper can be seen a good extension work of Bates et al. [17]. The authors replace the simple split conformal prediction with Jackknife technique for more \"accurate\" estimated p-value,  by fully exploring the clean data and avoid the randomness caused by data-splitting. This idea is very straightforward.  Another contribution is selecting the best model from a pool of detectors. “Best\" here means that the model detected the most outliers in the new dataset, which is not novel technique as well. Overall,  the novelty of this work is limited.  The manuscript will benefit from adding explanation about the novelty of such combination of two existing techniques, theoretically or practically.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LVyq5tbqOk3", "reviewer": "Reviewer_MLPQ", "summary": "This paper proposes a general AutoML framework for novelty detection and controlling the error rate of the model. The framework consists of an automated model selection procedure with FDR control. The theoretical bound is provided for AutoMS. Extensive experiments are conducted to demonstrate its effectiveness.", "questions": "None", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1: The paper proposed a unified framework that can be combined with different base detectors\n2: The paper provides a theoretical bound of FDR\n3: Experiments are conducted to evaluate the effectiveness of AutoMS on both synthetic and real-world data.\n\nWeaknesses\n1. Only several real-world datasets are selected in the experiments. As a comparison, the previous work MetaOD has performed experiments on hundreds of datasets. The authors are encouraged to conduct a more thorough comparison with MetaOD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept