Derandomized novelty detection with FDR control via conformal e-values
Authors: Meshi Bashari
Amir Epstein
Yaniv Romano
Matteo Sesia
Venue: NeurIPS 2023 poster (2023)
Keywords: Conformal inference
Derandomization
E-values
False discovery rate
Out-of-distribution testing
Testing for outliers
Uncertainty
TL;DR: Powerful derandomization framework for conformalized novelty detection under rigorous false discovery rate control.

Abstract: Conformal inference provides a general distribution-free method to rigorously calibrate the output of any machine learning algorithm for novelty detection. While this approach has many strengths, it has the limitation of being randomized, in the sense that it may lead to different results when analyzing twice the same data and this can hinder the interpretation of any findings. We propose to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance. This solution allows the evidence gathered from multiple analyses of the same data to be aggregated effectively while provably controlling the false discovery rate. Further, we show that the proposed method can reduce randomness without much loss of power compared to standard conformal inference, partly thanks to an innovative way of weighting conformal e-values based on additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.

Derandomized novelty detection with FDR control via conformal e-values
Meshi Bashari
Amir Epstein Citi Innovation Lab Tel Aviv. Israel amir. seirreepseenn@itii.com
Department of Electrical and Computer Engineering
Technion IIT
Haifa, Israel
meshi.. b@campus. technion. ac.
Yaniv Romano
Department of Electrical and Computer Engineering Department of Computer Science Technion IIT Haifa, Israel yromano@technion icl
Matteo Sesia
Department of Data Sciences and Operations
University Southern California
Los Angeles, California, USA sesia@marshall usc.
Abstract
Conformal inference provides a general distribution-free method to rigorously calibrate the output of any machine learning algorithm for novelty detection. While this approach has many strengths, it has the limitation of being randomized, in the sense that it may lead to different results when analyzing twice the same data, and this can hinder the interpretation of any findings. We propose make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance. This solution allows the evidence gathered from multiple analyses of the same data to be aggregated effectively while provably controlling the false discovery rate. Further, we show that the proposed method can reduce randomness without much loss of power compared to standard conformal inference, partly thanks to an innovative way of weighting conformal e- values based additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.
Introduction
1.1 Background and motivation
common problem statistics and machine learning is to determine which samples, among a collection of new observations, were drawn from the same distribution 1963) Riani et al., 2009, Chandola et al., 2009). This task is known as novelty detection, out-of- distribution testing, or testing for outliers, and it arises in numerous applications within science,
reference data set (Wilks
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

engineering, and business, including for example in the context of medical diagnostics (Tarassenko) it al. [995). security monitoring (Zhang et al., 2013), and fraud detection (Ahmed et al., (016). This paper looks at the problem from model-free perspective, in the sense that it does not rely on parametric assumptions about the eafa-rennr distributions, which are generally unknown and complex. Instead, we apply powerful machine learning models for one-class (Moya et al. 1993) or binary classification to score the new samples based on how they conform to patterns observed the reference data, and then we translate such scores into rigorous tests using conformal inference.
Conformal inference (Vladimir al., 2005 Lei et al. 2013) provides flexible tools for extracting provably valid novelty detection tests from any black- box model. The simplest implementation is based on random sample splitting. This consists of training a classifier on subset of the reference data, and then ranking the output score for each test point against the corresponding scores evaluated out-of-sample for the hold-out reference data. As the latter do not contain outliers, the aforementioned rank is uniformly distributed under the null hypothesis that the test point is not an outlier (Laxhhamarr and Falkman 2015] Smith et al., 2015] Guan and Tibshirani] 2022), as long as some relatively mild exchangeability assumptions hold. In other words, this calibration procedure yields conformal p-value that can be utilized to test for outliers while rigorously controlling the probability of making a false discovery-incorrecte labeling an inlier data point as an "outlier". Further, split-conformal inference produces only weakly dependent p-values for different test points (Bates et al. (023). allowing exact control of the expected proportion of false discoveries the false discovery rate (FDR)-wall the powerful Benjamini- Hooiiirrr (BH) filter (Benjamini and Hochberg 1995).
As visualized in Figure Ta a limitation of split-conformal inference is that it is randomized its results for a given data set are unpredictable because they depend on how the reference samples are divided between the training and calibration subsets. However, higher stability is desirable in practice, as randomized methods generally tend to be less reliable and more difficult to interpret (Murdoch et al.) 2019; Yu and Kumbier 2020). This paper addresses the problem of making conformal inferences more stable by developing a principled method to powerfully aggregate tests for outliers obtained with repeated splits of the same data set, while retaining provable control of the FDR. This problem challenging because dependent p-values for the same hypothesis are difficult to aggregate without incurring into significant loss of power (Vovk and Wang 2020 Vovk et al. 2022).
1.0
10
0.8
5
o Coo
Inlier Outlier
0.4
-5
-5
0.2
-10
-10
10
10
10
10
(a) Standard conformal.
(b) derandomized conformal.
Figure 1: Demonstration on two-dimensional synthetic data of standard conformal (a) and derandom- ized conformal (b) inferences for novelty detection. Circles denote true inliers and squares denote outliers. The colors indicate how often each test point is reported possible outlier over 100 independent analyses of the same data. By carefully aggregating evidence from 10 distinct analyses based independent splits of the same data, the proposed derandomized approach discovers the same outliers consistently and is less likely to make random false discoveries.
1.2
Main contributions
This paper utilizes carefully constructed conformal e-values (Vovk and Wang 2021) instead of p- values to quantify statistical significance when testing for outliers under FDR control. The advantage of e-values is that they make possible to aggregate the results of mutually dependent tests relatively simple way, enabling an effective approach to derandomize conformal inferences. Our method inspired by Ren and Barber (2023) that leverages e-value ideas efficiently, as different types of e- e-values can be constructed but not all would be powerful
contribution develop martingale-

---

in our context due to the discrete nature of the statistical evidence in conformal inference. We further refine this method and boost power by adaptively weighting our conformal e-values based on an estimate of the out-of-sample accuracy of each underlying machine learning model. A preview of the performance of our solution is given by Figure [b] which shows that our method can achieve power comparable to that of standard conformal inferences while mitigating the algorithmic randomness.
1.3 Related work
This paper builds upon e-values (Vovk and Wang 2021 J: quantitative measures of statistical evidence, alternative to p-values, that lend themselves well to the derandomization of data-splitting procedures and to FDR control under dependence (Wang and Ramdas 2022). There exist several generic methods for converting any p-value into an e-value (Vovk and Wang (021). While those p-to-e calibrators could be applied for our novelty detection problem, their power turns out to be often quite low due to the fact that conformal p-values are discrete and cannot take very small values unless the sample size extremely large; see the Supplementary Section S5 for more details.
Therefore, we propose a novel construction of (slightly generalized) e-values inspired by the work of Ren and Barber (023) on the derandomization of the knockoff filter (Barber and Candès 2013), which focused on a completely different high-dimensional variable selection problem. A different approach for producing e-values in the context of conformal inference can also be found in Iggattiadis itt al. (202)), although the latter did not focus on derandomization. Our approach differs from that of Ignatiadis et al. (2023) because construct e-values simultaneously for the whole test set, aiming to control the FDR, instead of operating one test point at a time, Simulations show that our approach tends yield higher power, especially if the test data contain many outliers.
Our second novelty consists of developing a principled method for assigning data- fata-driven weights to conformal e-values obtained from different machine learning models, in such way to further boost power. This solution re- pe-puuoosss transductive (Vovk] 2013) conformal inference ideas to leverage information contained in the test data themselves while calibrating the conformal inferences, increasing the power to detect outliers similarly to Marandon et al. (002) and Liang et al. (2022). )
While this paper focuses on derandomizing split-conformal inferences, there exist other distribution- free methods that can provide finite-sample tests for novelty detection, such as full-conformal inference (Vladimir et al. 2005) and cross- (Barber et al. 2021). Those techniques are
more computationally expensive but have the advantage of yielding relatively more stable conformal p-values because they do not rely on single random data split. However, full-conformal inference and cross- cross-validee validation+ also produce conformal p- p-values with more complicated dependencies. which make it difficult to control the FDR without large losses in power (Benjamini and Yekutieli 2001) or very expensive computations (Fithian and Lei 2022) Liang et al. 2022).
Finally, prior works studied how to stabilize conformal predictors by calibrating the output of an ensemble of simpler models (Löfström et al. 2013] Beganovic and Smirnov [0088] [
[Linusoon et al.) 2020: Kim et al. 22200] Gupta et al.) 2022). However, we consider distinct problem as we focus on derandomizing conformal novelty detection methods while controlling the FDR.
2 Relevant technical background
2.1 Notation and problem setup
Consider observations, X; E Rd, sampled exchangeably (or, for simplicity, independent and identically distributed) from some unknown distribution Po, for all E D= [n] [I....,n] n). Then, imagine observing a test set of Thea "unlabeled" samples X, E IR.. The problem to test, for each j E D In + Mhea] [n]. the null hypothesis that X, is also an inlier, in the sense that it was randomly sampled from Po exchangeably with the data in D. We refer to a rejection of this null hypothesis the discovery that X; is outlier, and we indicate the set of true inlier test points as Duull, with niuil |Diulll- For each j E Drest, define R2 as the binary indicator of whether X, is labeled by our method as an outlier. Then, the goal is to discover as many true outliers as possible while controlling the FDR, defined as FDR = [ R2)/ max (1, ΣSeD,.. R2D].
3

---

2.2 Review of FDR control with conformal p-values
After randomly partitioning D into two disjoint subsets Derain and Deal. of cardinality Aerain and Neal = n Terain respectively, the standard approach for computing split-conformal p-values begins by training one-class classification model on the data indexed by D This model is applied out-of-sample to compute conformity scores S; and S, for all calibration and test points i E Dent and j E Drest. with the convention that larger scores suggest evidence of an outlier. Assuming without loss of generality that all scores take distinct values (otherwise, ties can be broken at random by adding a little noise), a conformal p-value ú(X,) for each j E Durst then calculated by taking the relative rank of Šu among the 5; for all i E Deat: ū(X;) = (1+2
11;; S 5. + Reaa)). If the null hypothesis for X3 is true, S is exchangeable with S, for all i E Deal. and i(X;) ;) is uniformly distributed on {1/(1+ Real), 2/ (1+mM) Mlaa)),..... ................................ 1}. Since this distribution is stochastically larger than the continuous uniform distribution on [0, 1I, one can say that @(X,) is a valid conformal p-value. Note however that the p-values ú(X; and úi ((x,,) for two different test points j.j E Drest are not independent of one another, even conditional on Dirains because they share the same calibration data.
Despite their mutual dependence, conformal p-values can be utilized within the BH filter to simulta- neously probe the Thest hypotheses for all test points while controlling the FDR. A convenient way to explain the BH filter is as follows (Storey 2002). Imagine rejecting the null hypothesis for all test points j with ú(X,) S 8, for some threshold E [0, 1]. By monotonicity of a(X,), this amounts to rejecting the null hypothesis for all test points j with S 2 I, for some appropriate threshold E IR. An intuitive estimate of the proportion of false discoveries incurred by this rule is:
1 EiED DEDDNN
I($1 2 2
FDP(1) 1 neal
Thesst
+
EjEDem.. I($ 2 t
This can be understood by noting that EjeDim /1 2 8J is the total number of discoveries, while the numerator should behave similarly to the (latent) number of false discoveries in Drest due the exchangeability of S, and S under the null hypothesis. With this notation, it can be shown that the BH filter applied at level a E (0, 1) computes an adaptive threshold
mmi B = min {t E (S:) FDP(()= S }
(2)
and rejects all null hypotheses j with 8, > iBH, see Rava et al. 2021 1 for a derivation of this connection. This procedure was proved by Bates et al. (202)) control the FDR below a.
2.3 Review of FDR control with AdaDetect
Recently, Marandon al. (2022) proposed AdaDetect, a more sophisticated version of the method reviewed in Section 2.2 The main innovation of AdaDetect is that it leverages a binary classification model instead of one-class classifier. In particular, AdaDetect trains binary classifier to distinguish the inlier data in Derain from the mixture of inliers and outliers contained in the union of Deal and Drest. The key idea achieve FDR control is that the training process should remain invariant to permutations of the calibration and test samples. While the true inlier or outlier nature of the observations Drest is obviously unknown at training time, AdaDetect can still extract some useful information from the test data which would otherwise be ignored by the more traditional split-conformal approach reviewed in Section 2.2 In particular, AdaDetect can leverage the test data to automatically tune any desired model number of discoveries, A similar idea also motivates the alternative method of integrative conformal p-values proposed by Liang et al. (2022) although the latter requires the additional assumption that some labeled outlier data are available, and is therefore not discussed in equal detail within this paper.
in order to approximately maximize the
Despite more sophisticated use of the available data compared to the coffo-colflrmal method reviewed in Section [.2] AdaDetect still suffers from the same limitation that must calibrate its inferences based on a single random data subset D.att and thus its results remain aleatory. For simplicity, Section 3.J begins by explaining how to derandomize standard split-conformal inferences; then, the proposed method will be easily extended in Section 3.3 to derandomize AdaDetect.

---

3 Method
3.1 Derandomizing split-conformal inferences
Consider K 2 1 repetitions of the split-conformal analysis reviewed in Section 2.2 each starting with an independent split of the same reference data into D train and cal For each repetition is E [K]
(kk after training the machine learning model on train and computing conformity scores on D(al and Drest, one can estimate the false discovery proportion corresponding the rejection of all test points
with scores above a fixed rejection threshold t E I, similarly to ①I with:
EieD 1155 2 #} 115) 2 t
(k)
Riest
FDP
(1)
on
Meal
Ejer
Note that the estimate in differs slightly from that in [ as it lacks the "+1" constant term in the numerator and denominator. While it is possible to include the "+1" terms in (3. this is not needed by our theory and we have observed that it often makes our method unnecessarily conservative. For any fixed Onh E (0, 1). let ((k) be the corresponding BH threshold (2) at the nominal FDR level Onhi
((8) = min(t E D(k)
FDP (k) (1) S (bbh): cal-test
where ()()) cal-test 1
iE Drna (t). For each test point E Drest, define the following rescaled indicator of whether (((k) exceeds {(N):
T( 2 (00)
c = +1+ neal):
(5)
+ EiED!) T(S 2 {(1))
Intuitively, this quantifies not only whether the j-th null hypothesis would be rejected by the BH filter at the nominal FDR level Olsh, but also how extreme / is relative to the calibration scores. In other words, a large ej
(k) suggests that the test point may be an outlier, where this variable can take any of the following values: 0, 1, (1 + (ecal) /eal. (1 + nccat/((eaa - -I), (1 + neal). This approach, inspired by Ren and Barber (2023), is not the only possible way of constructing e-values to derandomize conformal inferences, as discussed in Supplementary Section S5 However, we will show that it works well in practice and typically achieves higher power compared standard p-to-e calibrators (Vovk and Wang 202T) applied to conformal p-values. This advantage partly derives from the fact that (5) can gather strength from many different test points, and partly from the fact that it is not proper e-value according to the original definition of Vovk and Wang (202T), in the sense that its expected value may larger than one even if X, an inlier. Instead, we will show that our e-values satisfy relaxed average validity property (Ren and Barber 2023) that is sufficient to guarantee FDR control while allowing more numerous discoveries.
After evaluating (5) for all j E Drest and all k E [K], we aggregate the evidence against the j-th null hypothesis into single statistic ej by taking weighted average:
K W
K
(k)
Σ
(k) =1,
based on some appropriate normalized weights (*). Intuitively, the role of (R(R) is to allow for the possibility that the machine learning models based on different realizations of the training subset may not all be equally powerful at separating inliers from outliers. In the remainder of this section, we will take these weights to be known a-priori for all & E K), thus representing relevant side information; e.g., in the sense of Genovese et al. (2006) and Ren and Candès (2033) For simplicity, one may think for the time being of trivial uninformative weights (h) allow these weights to be data-driven, but such an extension is deferred to Section 3.2 for conciseness.
K. Of course, it would be preferable to
Having calculated aggregate e-values ēú with the procedure described above, which is outlined by Algorithm|S ST in the Supplementary Material, our method rejects the null hypothesis for all j E D Deest whose ē, greater than adaptive threshold caleulated by applying the eBH filter of Wang and

---

Ramdas 2022). which is outlined for completeness by Algorithm S2 in the Supplementary Material. We refer to Wang and Ramdas (2022) for more detailed discussion of the eBH filter. Here, it suffices to recall that the eBH filter computes an adaptive rejection threshold based on the nesst input e-values and on the desired FDR level a E (0, 1). Then, our following result states that the overall procedure is guaranteed to control the FDR below under relatively mild exchangeability assumption.
Assumption 3.1. The inliers in and the null test points are exchangeable conditional on the non-null test points.
Theorem 3.2. Suppose Assumption 3.1 holds. Then, the e-values computed by Algorithm S7 satisfy:
B S Ritest
(6)
jED
The proof of Theorem 3.2 is in the Supplementary Section S2 Combined with Theorem 2 from Ren and Barber (2033) this result implies our method controls the FDR below the desired target level a.
Corollary 3.3 (Ren and Barber (.. The eBH filter of Wang and Ramdas (2022) applied at level
(0, 1) to e-values (e satisfying 46) guarantees FDR S a.
Remark 3.4. Assumption 3.1 does not require that the inliers are independent of the outliers. Remark 3.5. Theorem(322 holds regardless of the value of the hyper-parameter obb of Algorithm SI which appears in See Section 3.4 for further details about the choice of aihh
3.2 Leveraging data-driven weights
Our method can be extended leverage adaptive weights based on the data in DD and Drest as long as each weight (k( is invariant to permutations of the test point with the corresponding calibration samples in (()) In other words, we only require that these weights be written in the form of cal-
(k( = (D(th)
cal cal-trsi
The function w may depend on (())
train but not on cal or Drest- An example of a useful weighting scheme satisfying this property is at the end of this section. The general method summarized by Algorithm S3 in the Supplementary Material, which extends Algorithm ST] This produces e-values that control the FDR in conjunction with the eBH filter of Wang and Ramdas (2022).
Theorem 3.6. Suppose Assumption 3./ holds. Then, the e-values computed by Algorithm S3 satisfy (
as long as the adaptive weights obey
An example of a valid weighting function applied in this paper is the following. Imagine having some prior side information suggesting that the proportion of outliers in Drest is approximately Y E (0, 1)- Then, natural choice to measure the quality of the k-thh model is to let (i) = |,<()), where ((k) is the standard t-statistic for testing the difference in means between the top [iees 771 largest values in D(A)
cal-test and the remaining ones. See Algorithm $5 in the Supplementary Material for further details. Intuitively, Algorithm tends to assign larger weights to models achieving stronger oft-ol-sample separation between inliers and outliers. Of course, this approach may not always be optimal but different weighting schemes could be easily accommodated within our framework.
3.3 Derandomizing AdaDetect with E- E-AdaDetect
The requirement discussed in Section 3.2 that the data-adappive weights should be invariant to permu- tations of the calibration and test samples is analogous to the idea utilized by AdaDetect (Marandon) et al. 2022) to train more powerful machine learning models leveraging also the information con- tained in the test set; see Section 2.3 This implies that Theorem remainss valid even if our method is implemented based on K machine learning models each trained by looking also at the unordered
union of all data points ((A) Duest for each l E [K] See Algorithm SS in the Supplementary Material for a detailed implementation of this extension of our method, which we call E- E-AdaDetect.
3.4 Tuning the FDR hyper-parameter
As explained in Section 3.1 our method involves hyyee-parameter abb controlling the BH thresholds g(k) in (4) Intuitively, higher values of abb tend to increase the number of both test and calibration

---

scores exceeding the rejection threshold at each of the K iterations. Such competing effects make generally unclear whether increasing ohh leads to larger e-values in (5) and hence higher power. This trade-off was studied by RRn and Barber (2023) while derandomizing the knockoff filter, and they suggested setting Olb < a. In this paper, we adopt abh = a/10, which we have observed to work generally well in our context, although even higher power can sometimes be obtained with different values of abhh especially if the number of outliers in the test set is large. While we leave to future research to determine whether further improvements are possible, it is worth noting that straightforward extension of our method, not explicitly implemented in this paper, can be obtained by further averaging e-values obtained with different choices of Crbb- Such extension does not affect the validity of (6) due to the linearity of expected values.
4
Numerical experiments
4.1 Setup and performance metrics
This section compares empirically the performance of AdaDetect and proposed derandomized method described in Section 3.33 namely E- AdaDetect. Both procedures are deployed using a binary logistic regression classifier (Marandon et al. 2022) the base predictive model. The reason why we focus on derandomizing AdaDetect instead of traditional split-conformal inferences based on a one-class classifier (Bates et al., 2023) is that we have observed that AdaDetect often achieves higher power on the data considered in this paper. which makes it more competitive benchmark. However, additional experiments reporting on the performance of our derandomization method applied in combination with one-class classifiers can be found in the Supplementary Sections (S.2 and S6.2
As the objective of this paper is powerfully detect outliers while mitigating algorithmic randomness, we assess the performance of each method over M = 100 independent analyses based on the same fixed data and the same test set. For each repetition m of the novelty detection analysis based on the fixed data, we identify subset (mm Drest of likely outliers (the rejected null hypotheses) and evaluate the average power and false discovery proportion, namely
Power
1
Σ
M
R(m))
Dawn-mnn null
FDR
1 M mm
R n D
(8)
M
DDean null
max (m)), 1,13
mim1
where Desn Drest proportion defined in 8b is not the FDR, which is the quantity we can theoretically guarantee to control. In fact, FDR R[FDR], with expectation taken with respect all randomness in the data. Nonetheless, we will see that this average false discovery proportion is also controlled in practice within all data sets considered in this paper. The advantage of this setup is that it makes it natural to estimate algorithmic variability by observing the consistency of each rejection across independent analyses. In partieular, after defining Rj.m as the indicator of whether the j-th null hypothesis was rejected in the m-th analysis, we can evaluate the average variance in the rejection events:
Desll indicates the true outliers the test set. The average false discovery
M
Variance
Σ m==1
RRi. R₂)²
(9)
Thestt
M
where R, = (1/M) = Ry.m- Intuitively, it would be desirable to maximize power while simultaneously minimizing both the average false discovery proportion and the variability. In practice, however, these metrics often compete with one another; hence why we focus on comparing power and variability for methods designed control the FDR below the target level a 0.1.
4.2 Experiments with synthetic data
Synthetic reference and test data consisting of 100-dimensional vectors X are generated as follows. The reference set contains only inliers, drawn i.i.d. from the standard normal distribution with independent components, A A/, J100). Unless specified otherwise, the test set contains 90% inliers and 10% outliers, independently sampled from M(,, 1100). The first 5 entries of yu are equal to a constant parameter, to which we refer as the signal amplitudee while the remaining 95 entries are zeros. The size of the reference set is 2000, with 1000 samples in the training subset and 1000 in the calibration subset. The size of the test set Thast = 1000. Both E- AdaDDtett and AdaDetect applied based on the same logistic regression classifier with default
7

---

4.2.1 The effect of the signal strength
Figure[Joompann the performance of E- E-AdaDetect (applied with K' = 10) to that of AdaDetect, as a function the signal amplitude. The results confirm both methods control the FDR but ours is less variable, as expected. The comparison becomes more interesting when looking at power: AdaDetect tends to detect more outliers on average the signal strength is low, but E-AdaDetect can also outperform by that metric if the signals are strong. This may be explained as follows. If the signal strength high, most true discoveries produced by AdaDetect are relatively stable across different analyses, while false discoveries may be more aleatory, consistently with the illustration of Figure I Such situation ideal for derandomization, which explains why E-AdaDetect is able to simultaneously achieve high power and low false discovery proportion. By contrast, if the signals weak, the true outlier discoveries produced by AdaDetect are relatively scarce and unpredictable, thus behaving not so differently from the false findings. In this case, one could argue that stability becomes even more important to facilitate the interpretation of any findings, and that may justify some loss in average power.
AdaDetect
0.75
0.8
0.6
E E-AdaDetect
Verience
0.02
0.50
0.4
5800 Signal amplitude
9.6.2
1/ Signal amplitude
Signal Cmppitude
Figure 2: Performance on synthetic data of the proposed derandomized outlier detection method, E- E-AdDDetect, applied with K = 10, compared to that of its randomized benchmark, AdaDetect, as a function of the signal strength. Both methods leverage a logistic regression binary classifier. Left: average proportion of true outliers that are discovered (higher better). Center: average proportion of false discoveries (lower is better). Right: variability of the findings (lower is better).
4.2.2 The effect of the number of analyses K
Figure investigates the effect of varying the number of analyses K aggregated by E-AdaDetect. Here, signal amplitude is fixed 3.4 (strong signals), while K is varied between and 30. As expected, the results show that the variability of the findings obtained with E- AddDeteett decreases as K increases. The average proportion of false discoveries obtained with E- E-AdaDetect also tends to decrease when K is large, which can be understood by noting that spurious findings are less likely to be reproduced consistently across multiple independent analyses of the same data. Regarding power, the average number of true outliers detected by E- E-AdaDetect appears monotonically increase with K, although this is not always true in other situations, as shown in the Supplementary Section S4.1 In fact, if the signals are weak, E-AdaDetect may lose some power with larger values of K (although some K > may be optimal), consistently with the results shown in Figure 2 Thus, we recommend practitioners to utilize larger values of K in applications where higher power is expected. Finally, note that the power of E- AdAdDteect is generally lower compared to that of AdaDetect. in the special case of K although this is not practically relevant value of K because it does not allow any derandomization. The reason why the power of E-AdaDetect is lower when K is FDR-controlling
that this method relies the eBH filter. The latter is relatively conservative strategy because it requires no assumptions about the dependencies of the input statistics.
4.2.3 The effect of the weighting strategy
This section highlights the practical advantage of being able to use data-adaptive model weights within E- E-Adetttec.. For this purpose, we carry out experiments similar to those of Figure [ but leveraging a logistic regression model trained with different choices of of the K analyses. Specifically, we fit sparse logistic regression model using K = 10 different values of the regularization parameter. To induce higher variability in the predictive rules, one model was trained with regularization parameter equal to 0.0001, while the others were trained with regularization parameters equal to 1, 10, 50, and 100, respectively. Then, we apply E- E-AdaDetect using different weighting schemes: constant equal weights ('uniform'), data-driven weights calculated
in each

---

1.0
0.025
AdaDetect
Power
Varisnce
0.020
0.015
0.4
0.0
0.005
20
30
3
3
5
10 15
Number of Rerations
Number itorations
Number of iterations
Figure Performance on synthetic data of E-AdaDetect, as function of the number K of derandomized analyses, compared to AdaDetect. Note that the latter can only be applied with single data split (or iteration). The signal amplitude 3.4. Other details are as in Figure
with the I-statistic approach ("t-test") summarized by Algorithm S5 and simple alternative trimmed average data-driven approach ("avg. score") outlined by Algorithm Sn the Supplementary Material. The results in Figure 4 show that the data-driven aggregation scheme based on t-statistics the most effective one, often leading to much higher power. We have chosen not compare E AdaDetect to the automatic AdaDetect prper-prraeeter tuning strategy proposed in Section 4.5 of Marandon al. 2022) because found that it does not perform very well in experiments, possibly due to the relatively low sample size.
1.00
1.0
0.8
AdaDetect (t-test)
0.020
0.6
DettDtttct (avg. score)
0.015
0.50
0.4
022
0.00
0-0-0-0
0.0
0.000
Signal amplitude
Signal amplitude
Signal amplitude
Figure 4: Performance on synthetic data E- AdaDeteec applied with different model weighting schemes, as function of the signal strength. The model weights are designed to prioritize the results of randomized analyses based on models that are more effective at separating inliers from outliers. The t-test approach tends to lead to higher power. Other details are in Figure 2
To further demonstrate the effectiveness of data-driven weighting, turn analyze the performance of E-AdaDetect on four real-world outlier detection data sets: musk, shuttle, KDDCup99, and credit card. We refer Supplementary Section [S6 for more information regarding these data. Similar to Figure our E-AdaDetect method is applied K = 10 times to each data set, each time leveraging different predictive model follows. Half of the models are random forests implemented with varying max-depth hyper-parameters (10, 12, 20, 30, and 7). while the other half are support vector machines with RBF kernel with varying width hyper-parameters (0.1, 0.001, 0.5, 0.2, and 0.03). This setup is interesting because different models often tend to perform differently in practice, and usually unclear a-priori which combination of model and hyper-parameten is optimal for given data set. Figure 5 summarizes the results, demonstrating that both data-driven weighting schemes ("t-test" and "avg. score") lead to more numerous discoveries compared to the "uniform" weighting baseline, and that the "t-test" approach is the most powerful weighting scheme here. These results are in line with the synthetic experiment presented in Figure 4 Lastly, the variance metries reported in Figure 5 also suggest that data-driven weighting further enhances the algorithmic stability.
4.3 Additional results from experiments with synthetic and real data
Sections 244S6lnn the Supplementary Material present the results of several additional experiments. In on experiments with synthetic data. Section (ection[S5]dese describes comparisons
particular, Section|S4| with alternative derandomization approaches based on different types of p-to-e calibrators (Vovk) and Wang T02T) operating one test point at a time, which turn out to yield lower power compared to our martingale-based method. The results also show that our method compares favorably to an alternative derandomization method based on an e-value construction that first appeared in Ignatiadis) al. 2023), especially if the test set contains numerous outliers or if the nominal FDR level is not

---

10
1.00
E-AdaDttect (uniform)
Power
E-AdaDttect (avg, score)
AdaDetec (tstest)
Vartance
0.5
0.50
0.01
0.25
0.0
musk
shuttle
KODCup39
creditcard
0.00
musk
shuttle
KODCup99
creditcard
0.00
musk
shuttle
KDDCup99
creditcard
Dataset
Dataset
Dataset
Figure 5: Performance of E- E-AddDetect, applied with different model weighting schemes, four real data sets. All methods utilize a combination of random forest and support vector machine models. The "t-test" weighting approach leads to the highest power and lowest algorithmie variability, all while controlling the FDR below the nominal 10% level.
too low. Finally, Section S6 describes additional numerical experiments based on several real data sets also studied in Bates et al. (2023) and Marandon et al. (2022). These results confirm that our martingale based e-value method can mitigate the algorithmic randomness of standard conformal inferences and AdaDetect while retaining relatively high power.
5
Discussion
Our experience suggests that e-values are often less powerful than p-values in measuring the statistical evidence against single hypothesis. Yet, e-values can be useful to aggregate multiple dependent tests of the same hypothesis (Vovk and Wang 2021 a task that would otherwise require very conservative adjustments within the p-value framework (Vovk et al. 2022). Further, have shown that e-values lend themselves well to multiple testing because they allow efficient FDR control under arbitrary dependence (Wang and Ramdas] 2022), and even their relatively weak individual evidence can accumulate rapidly when large number of hypotheses is probed. The opportunity arising from the combination of these two key properties was recently leveraged to derandomize knockoffs (Ren) and Barber 2023), but until now it had not been fully exploited in the context of conformal inference.
While this paper has focused derandomizing split-conformal and AdaDetect inferences for novelty detection, the key ideas could be easily extended. For example, one may utilize e-values to derandomize conformal prediction intervals in regression (Lei and Wasserman 2014 Romano et al.) 2019) or prediction sets for classification (Lei et al.] Romano et al.) 2020)) while controlling the false coverage rate over large test set (Weinstein and Ramdas] 2020). different direction for future research may explore the derandomization of cross-validation- (Barber et al. (021).
We conclude by discussing two limitations of this work. First, the proposed method is more computa- tionally expensive compared to standard conformal inference or AdaDetect, and therefore one may be limited to applying it with relatively small numbers K of analysis repetitions when working with very large data sets. That being said, it is increasingly recognized that stability is an important goal in data science (Murdoch et al. 2019). and thus mitigating algorithmic randomness may often justify the deployment of additional computing resources. Second, we have shown that our method sometimes leads to reduction in the average number of findings compared to randomized alternatives, especially in applications where few discoveries are expected. Therefore, in situations with few anticipated discoveries, practitioners considering applying method should carefully weigh the anticipated gains in stability versus a possible reduction in power.
Mathematical proof's and additional supporting results are in the Supplementary Material. Software implementing the algorithms described this paper and enabling the reproduction the associated numerical experiments is available at http://hhw...ooe//s////////////a
10

---

Acknowledgements
R. and M. were supported by the Israel Science Foundation (grant No. 729/21). Y. R. thanks the Career Advancement Fellowship. Technion, for providing research support. Y. R. also thanks Citi Bank for the generous financial support. M. was partially supported by NSF grant DMS 2210637 and by an Amazon Research Award.
References
M. Ahmed, A. N. Mahmood, and M. R. Islam. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278-288, 2016.
R. F. Barber and E. Candès. Controlling the false discovery rate via knockoffss Ann. Stat., 43(5): 2055-20855 2015.
R. F. Barber, E. Candès, A. Ramdas, Tibshirani, al. Predictive inference with the jackknife+. Ann. Stat., 49(1))486-500 2021.
S. Bates, E. Candès, L. Lei, Y. Romano, and M. Sesia. Testing for outliers with conformal p-values. Ann. Stat., 51()):149 178, 2023.
D. Beganovic and E. Smirnov. Ensemble cross-conformal prediction. In 2018 IEEE International Conference on Data Mining Workshops, pages 870-877. IEEE, 2018.
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B, 57(1):289-300, 1995.
Y. Benjamini and D. Yekutieli. The control of the false discovery rate in multiple testing under dependency. Ann. Stat., pages 1165-1188, 2001.
V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), (1):)11-55 2009.
W. Fithian and L. Lei. Conditional calibration for false discovery rate control under dependence. Ann. Stat., (091)333 2022.
C. R. Genovese, K. Roeder, and L. Wasserman. False discovery control with p-value weighting. Biometrika, 9333)550--524, 2006.
L. Guan and R. Tibshirani. Prediction and outlier detection in classification problems. J. R. Stat. Soc. Series B, 84(2):524, 2022.
C. Gupta, A. K. Kuchibhotla, and A. Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022.
N. Ignatiadis, R. Wang, and A. Ramdas. E-values as unnormalized weights multiple testing. arXiv preprint arXiv:2204.1244 12447, 2023. version 4.
B. Kim, C. Xu, and R. Barber. Predictive inference is free with the juckknifcet-fffe--bottsaag Adv. Neural Inf. Process. Syst., 33:4138-4144 2020.
R. Laxhammar and G. Falkman. Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories. Annals of Mathematics and Artificial Intelligence, 74(1-2):67 2015.
J. Lei and L. Wasserman. Distribution-free prediction bands for non-parametric regression. J. R. Stat. Soc. Series B, 76(1):71-96, 2014.
J. Lei, J. Robins, and L. Wasserman. Distribution--fee prediction sets. J. Am. Stat. Assoc., 108(501): 278-287, 2013.
Z. Liang, M. Sesia, and W. Sun. Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers. arXiv preprint
2022.
H. Linusson, U. Johansson, and H. Boström. Efficient conformal predictor ensembles. Neurocomput- ing, 397:266-278. 2020.
T. Löfström, U. Johansson, and H. Boström. Effective utilization of data in inductive conformal prediction. In Proc. Int. Jt. Conf. Neural Netw: IEEF, 2013.
A. Marandon, Lei, D. Mary, and E. Roquain. Machine learning meets false discovery rate. arXiv preprint arXiv:2208.066 0060088..668..88 2022.

---

M. M. Moya, M. W. Koch, and L. D. Hostetler. One-class classifier networks for target recognition applications. NASA STI/Recon Technical Report N, 93:24043, 1993.
W. J. Murdoch, C. Singh, K. Kumbier, Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. Proc. Natl. Acad. Sei. U.S.A., 116(44):2201- 22080, 2019.
B. Rava, W. Sun, G. M. James, and X. Tong. A burden shared is burden halved: A fainess--ajjsteed approach to classification. arXiv preprint arXiv:2110.05720, 2021.
Ren and R. Barber. Derandomised knockoffs: leveraging e-values for false discovery rate control. J. R. Stat. Soc. Series B, 09 2023.
Z. Ren and E. Candès. Knockoffs with side information. Ann. Appl. Stat,, 17(2):1152-111 2023.
M. Riani, A. C. Atkinson, and A. Cerioli. Finding an unknown number multivariate outliers. J. R. Stat. Soc. Series B, 71(2):4477445 2009.
Y. Romano, E. Patterson, and E. Candes. Conformalized quantile regression. Adv. Neural Inf. Process.
Syst. 32, 2019.
Y. Romano, M. Sesia, and E. Candès. Classification with valid and adaptive coverage. Adv. Neural Inf. Process. Syst., 33, 2020.
Smith, Nouretdinov, R. Craddock, C. Offer, and A. Gammerman. Conformal anomaly detection of trajectories with multi-class hierarchy. In International symposium statistical learning and data sciences, pages 281-290. Springer, 2015.
D. Storey. direct approach to false discovery rates. R. Stat. Soc. Series B, 64(3):479-498, 2002.
L. Tarassenko, P. Hayton, N. Cerneaz, and M. Brady. Novelty detection for the identification of masses mammograms. In 1995 Fourth International Conference on Artificial Neural Networks, pages 442-447. IET, 1995.
V. Vladimir, Alex, and Glenn. Algorithmic learning in random world. Springer, 2005.
Vovk. Transductive conformal predictors. In IFIP International Conference on Artificial Intelli- gence Applicutions and Innovations, pages 348-360. Springer, 2013.
V. Vovk and R. Wang. Combining p-values via averaging. Biometrika, 107(4):791-808, 2020.
V. Vovk and R. Wang. E-values: Calibration, combination and applications. Ann. Stat., 49(3), jun 2021.
V. Vovk, B. Wang, and R. Wang. Admissible ways of merging p-values under arbitrary dependence. Ann. Stat., 50(1):351-375, 2022.
R. Wang and A. Ramdas. False discovery rate control with e-values. Stat. Soc. Series B, 84(3): 822-852, 2022.
A. Weinstein and A. Ramdas. Online control of the false coverage rate and false sign rate. In International Conference Machine Learning, pages 10193-10202. PMLR, 2020.
S. S. Wilks. Multivariate statistical outliers. Sankhya: The Indian Journal of Statistics, Series A, pages 407-426, 1963.
B. Yu and K. Kumbier. Veridical data science. Proc. Natl, Acad. Sci. U.S.A., 1928)39920-3329, 2020.
M. Zhang, A. Raghunathan, and N. K. Jha. MedMon: Securing medical devices through wireless monitoring and anomaly detection. IEEE Transactions Biomedical circuits and Systems, 7(6): 871 871-881, 2013.
12


Reviews:
[{"review_id": "LJh4y239AD", "reviewer": "Reviewer_gnkM", "summary": "This papers proposes a way to reduce the randomness in novelty detection methods (detecting out-of-distribution points) that are based on the split-conformal inference paradigm. This is done by ensembling over several ($K$) train-validation splits of the dataset. The main technical point is to aggregate the evidence from the $K$ individual predictions by averaging their E-values, as a replacement to considering the p-value in the traditional approach with only one train-validation split. It is shown that the FDR (false discovery rate) can still be controlled with these quantities. The method is further enhanced by using certain weighted averages of the E-values, taking into account the estimated powers of each one. The methods are evaluated on synthetic and real data, showing that the proposed derandomized method works better if the fraction of outliers is larger.", "questions": "* Please mention what K is in Fig. 1.\n* line 228: Should this be D^(k)_cal \\cup D_test?\n* How is AdaDetect different from E-AdaDetect with K=1 (Fig. 3)?\n* Please carefully re-write the Proof of Theorem 3.6 in Appendix S2, as I believe there are many typos and several things that could be explained better. In particular:\n   - Please explain the first equality after line 35.\n   - It seems that the martingale runs from l'=l down to l'=0, correct?\n   - It would be good to explain in detail, over which random variables the various expectations E after line 26 run and which random variables are held fixed (conditioned on).\n   - Right-hand-sides of Eq. (S1) and (S2): Should there be t instead of \\hat{t}^{(k)}?\n   - line 31: The word \"outliers\" should be \"inliers\"\n   - Should the inf be changed to sup, or maybe the inequality sign reversed?\n   - line 34: inside the curly brackets, the l should be l'?", "limitations": "Yes, this is discussed in a fair way after line 351.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed novelty detection approach is original in that it combines evidences from different algorithm runs into a single evidence score. This required a formulation of the algorithms' evidences in terms of E-values instead of p-values that are commonplace for conformal methods, to avoid a loss in detection power. While prior novelty detection works have touched upon E-values, the current paper succeeds in controlling the FDR based on these E-value evidences. The methodological details (Sec. 3.1) and mathematical proof (Theorems 3.2 and 3.6) of this FDR control are strongly inspired by existing works, but this technique is new in the area of novelty detection.\n\nAs to significance, the method does reduce the variance as promised and keeps its FDR guarantee. It has higher power than competitor methods in some regimes, but quite consistently weaker power in other regimes, particularly for low fraction of outliers. Secondly, while the problem and the solution is built around quite specific requirements, it is plausible that the presented techniques can be applied to the derandomization of other inference methods with statistical guarantees, as the authors say in their Conclusion.\n\nThe clarity of the paper is exceptional (but see my comments on Appendix S2 below), the explanations and discussions are to the point.", "weaknesses": "As the biggest weakness of the paper I see that, despite the novelty of the approach, the considered problem requirements are relatively specific and the solution therefore narrow, as it concerns the de-randomization of certain novelty detection algorithms that are based on conformal inference and that aim at a mathematical control of the FDR.\n\nAlso, the method's performance (power) falls behind other methods in certain application regimes. It would be good to have some rules beforehand to know in which regime to apply which method.\n\nIf space permits, I would like to see some real data experiments (see Sec. 4.3) in the main text rather than only in the Supplement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. \n\n### Broader relevance of our work\n\nIt is true that this work focuses explicitly on de-randomizing conformal inferences for novelty detection tasks, which may seem like a relatively narrow scope compared to the broader range of possible conformal inference applications. However, the novelty detection idea of testing the exchangeability between a calibration set and a new test point is essentially the foundation of all conformal inference. \n\nThis is why it makes sense to start tackling de-randomization in conformal inference from the perspective of novelty detection, but it is not hard to see how our ideas could be extended to other tasks, including regression and classification (after replacing the FDR with analogous concepts such as the FCR). Such extensions are suggested in Section 5. Space limitations prevent us from developing these extensions fully within the present paper, but this is certainly an interesting direction for follow-up work.\n\n### The effect of de-randomization on power\n\nThe goal of our method is to decrease the algorithmic randomness of existing conformal inference techniques, not to increase their power. \n\nOur results consistently demonstrate the effectiveness of our approach in achieving this goal (e.g., refer to Section 4). \nReducing algorithmic randomness is important because it enhances the reproducibility, stability, and interpretability of any findings. There is no clear reason why de-randomization should also lead to higher power in general; it is a nice surprise that it sometimes does. However, it must be remembered that power may not always be the most meaningful concept in applications with high algorithmic randomness, as it refers to an expected behavior averaged over many hypothetical repeated experiments. In reality, algorithmic randomness can make the results of any analysis highly unpredictable, regardless of what the average power might be, and this is precisely the issue that motivates our work.\n\nRegarding your suggestion of providing some guidelines for practitioners on when to expect our method to also boost power, we would like to point out that we have already touched upon this issue in the paper. Specifically, we have explained how our method tends to reduce power in applications with few expected discoveries (e.g., refer to Section 5). Unfortunately, it might be challenging to predict power improvements more precisely without much stronger assumptions than those made in this paper.\n\n### Other questions\n\nIn Figure 3, the difference between AdaDetect and E-AdaDetect is that these two methods utilize different decision rules. The former computes p-values, while the latter relies on E-values. E-values are generally less powerful than p-values, but they have the advantage of facilitating de-randomization. This is why E-AdaDetect is at a disadvantage when applied with K=1, but this special case is not practically interesting (as it does not allow de-randomization) and it is only shown for completeness.\n\nWe also thank you for bringing some typos and imprecisions to our attention; we will certainly fix them in the revised version of the paper."}, {"review_id": "FLW7GvHsvD", "reviewer": "Reviewer_Ed41", "summary": "The main limitation of conformal prediction lies in its inherent randomness. However, this paper presents an innovative solution by introducing a derandomized version of conformal prediction, specifically applied to the field of novelty detection. Through the incorporation of conformal e-values, the proposed method successfully reduces the element of randomness while providing provable and effective control over the False Discovery Rate (FDR).\n\nThe key contribution of this research lies in its pioneering use of e-values, instead of traditional p-values, within the framework of conformal prediction. This innovative approach significantly simplifies the aggregation process, reducing randomness without compromising the overall detection power.", "questions": "*", "limitations": "*", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is a highly innovative and inspiring work that highlights the potential of e-values as a superior alternative to p-values for derandomizing conformal prediction through the aggregation of multiple dependent tests of the same hypothesis.\n\nThe paper introduces a novel approach for constructing e-values and provides a rigorous guarantee of false discovery rate (FDR) control, with Theorem 3.2 being the main contribution of the research.\n\nFurthermore, the practical aspect of the paper lies in the deployment of conformal e-values in AdaDetect, which presents a derandomized version of AdaDetect. It is important to note that E-AdaDetect is not merely a simple combination but rather a specific practical application demonstrating how to aggregate e-values using data-adaptive weights.\n\nThe experimental results showcased in the paper reveal that the derandomized AdaDetect exhibits comparable power to its randomized counterpart, while effectively controlling the FDR. This finding is particularly surprising (even higher power? )and highlights the potential benefits of adopting the \"new\"(not same as the classical definition) e-values .", "weaknesses": "The main paper primarily focuses on experimental results obtained from synthetic data, which is sampled from simple Gaussians. It may be considered relatively easier compared to real-world scenarios. Therefore, I would suggest including at least one experiment using more complex synthetic data or real data from the supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. \n\nWe also appreciate your suggestion of including an additional experiment utilizing more complex synthetic data or real data; if space permits we will insert these results in the revised manuscript."}, {"review_id": "vTSxESwmNO", "reviewer": "Reviewer_TGb1", "summary": "The paper employs conformal e-values, as opposed to p-values, to quantify statistical significance during outlier testing under FDR control. This approach enables the principled aggregation of results from mutually dependent tests, thereby providing a solution to de-randomize (split) conformal inferences.", "questions": "The paper focuses on derandomizing split-conformal inferences. I wonder if jackknife+ would inherently solve the problem.\n\nThe authors mention that their method can be extended to leverage adaptive weights based on the data. Could they discuss potential strategies for choosing these weights in practice?", "limitations": "See weakness.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper addresses a significant issue of the problem of the randomness in conformal inferences. \n\nThey propose a method to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance, also merging the idea of de-randomizing conformal novelty detection and FDR control.\n\nThe proposed method has the potential to significantly improve the stability and interpretability of conformal inferences.", "weaknesses": "The paper could significantly benefit from revisions aimed at improving clarity. The current presentation of ideas and concepts is convoluted, making it difficult for readers to follow and understand the arguments and methodologies proposed. \n\nWhile the authors' approach is novel, it builds upon existing studies and techniques. The authors could strengthen the originality of their work by further highlighting the unique aspects of their approach and how it differs from previous methods.\n\nThe authors provide simulations to demonstrate the effectiveness of their method, but it would be beneficial to see more empirical evaluations, including comparisons with other state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review; we appreciate the effort and honest feedback. We are sorry to hear you found the paper a bit hard to understand, but we hope we can answer your questions here.\n\n- Clarity. Other reviewers found the paper to be clear, but we can try to make it even more accessible. It would help us if you could be a little more specific about which sections or concepts you found confusing.\n\n- Jackknife+. Section 1.3 explains the relation of our work with Cross-Validation+, of which the Jackknife+ is a special case. In short, the Jackknife+ is not a satisfactory solution for the problem considered in this paper due to the difficulty of controlling the FDR. However, in the future it may be possible to combine our work with the Jackknife+, as we suggested in Section 5.\n\n- Data-driven weights. The extension of our method based on data-driven weights is explained in Section 3.2. This also includes a detailed description of the specific approach adopted in the experiments of this paper, presented in Section 4.2.3. See also Algorithm S5 in the Supplementary Material for further details. Is it possible that you read this part of the paper quickly or missed the Supplementary Material?\n\n- Novelty. Our paper makes two new contributions. First, it is the first one to carefully study the problem of de-randomizing conformal inferences using e-values, while controlling the FDR. We think this will have direct practical impact and is also likely to open new directions of research. Second, our paper combines and innovates upon ideas from different fields, introducing a key technical novelty that allows power boosting through data-driven e-value weighting. We think this adaptive weighting strategy will be useful beyond the scope of this paper. In summary, it is true that our results are achieved by building upon the works of others, but the importance of their prior contributions is clearly acknowledged.\n\n- Simulations. The paper presents extensive simulations and empirical comparisons with state-of-the-art methods. Is it possible that you missed some of the details described in Section 4 and in the Supplementary Material? Or is there anything specific that you would like to see added?"}, {"review_id": "p1huUtfCH9", "reviewer": "Reviewer_ccpQ", "summary": "This paper applied the derandomized e-value to the conformal novelty detection, which reduces the randomness of original approach using conformal p-value. The authors also refined the method by adaptively weighting the conformal p-values based on an estimate of the out-of-sample accuracy of each underlying machine learning model. Simulations with synthetic and real data are conducted to compare the performance with the original approach.", "questions": "1. What is the superiority of data-driven weights over the fixed width? Is the weighted approach in Ren and Baber [2022] applicable here?\n\n2. In Figure 2, why is the power of E-AdaDetect higher than Ada-Detect while the FDR value is still lower? It seems contradictory to common sense.", "limitations": "The main limitation is the novelty. Despite that this is the first work to deploy derandomized e-value to split conformal inference, the framework and the theory is well studied in previous works. In addition, the technical contribution is minor.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is the first work to apply the derandomized e-value to the conformal inference, which makes it more stable compared with the original approach.", "weaknesses": "### 1. The novelty of this paper is limited.\n\nConsidering the previous works by applying derandomization and E-value to Knockoff filter [Ren et al., 2020, Ren and Baber 2023], it is straightforward to extend it to the conformal setting.\n\n### 2. The challenges are not stated clearly.\n\n (1) From the connection between the conformal BH filter and Eq. (2) (given by Rava et al. [2021]), it would be easy to extend the construction in [Ren and Baber 2022] to conformal e-value in Eq. (5). In addition, the weighted e-value is also proposed by Ren and Baber [2022].\n\n (2) Given the Theorem 2 from Ren and Baber [2022], all the technical difficulty falls in proving Theorem 3.2. However, the proof strategy is from  Rava et al. [2021].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and constructive feedback. We are addressing your comments below.\n\n### Novelty and data-driven weights\nOur data-adaptive weighting method involves technical innovations and is different from the use of “side-information” weights discussed in Ren and Baber [2022]. The key distinction is that our method weights the conformal e-values in a data-driven way, leveraging additional information extracted from the same data used to compute the conformal e-values themselves. This is challenging because standard methods for weighted hypothesis testing typically assume the weights to be independent, obtained from separate data, and it is not clear where such independent information could come from in our case. For example, Ren and Baber [2022] discuss how to leverage prior information obtained from a completely independent source in the context of the knockoff filter. Their approach is thus not applicable here.\n\nFinally, our experiments show that our data-driven weighting method leads to higher power (e.g., refer to Figure 4), and we believe this method could also be useful beyond the scope of this paper.\n\n### Novelty and de-randomization in conformal inference\nThe novelty of this work is enhanced by the fact that it is the first paper to specifically address the important problem of de-randomizing conformal inferences. We think this research can have broad practical impacts due to the rapidly spreading adoption of conformal inference methods and the well-known importance of mitigating algorithmic randomness in sensitive applications.\n\n### Novelty and relation to Ren and Baber [2022] and Rava et al. [2021]\nAlthough our methods and proofs involve some technical similarities (which we clearly acknowledged), the problem studied in this paper is completely different from both that of Ren and Baber [2022] and that of Rava et al. [2021]. Ren and Baber [2022] focus on de-randomizing conditional independence tests based on knockoffs; they do not study conformal inference. Rava et al. [2021] study a problem closely related to conformal classification but they do not tackle de-randomization at all.\n\n### Figure 2\nFinally, in Figure 2, it should not be surprising that the power of E-AdaDetect can sometimes be higher than that of Ada-Detect while the FDR is lower. The two methods may often produce completely different orderings of the test points. Therefore, it is perfectly plausible that one method sometimes provides better separation between the inliers and the outliers. There would only be an intuitive contradiction here if the two methods constructed their inferences by applying a different significance threshold to the same test statistics; but that is clearly not the case."}]

Author rebuttal:

Meta-review: This paper addresses a derandomized conformal e-value for outlier detection. It also introduces a data-adaptive weighting method where conformal e-values are weighted in a data-driven way, leveraging additional information extracted from the same data used to compute the conformal e-values themselves. The outlier detection method proposed in this paper seems to be original, although there are some concerns on its limited novelty. The authors did a good job in their rebuttal, resulting in an increase in the overall score. All reviewers feel that the paper has interesting contributions and is deserved for publication.
Decision: Accept (poster)