Title: Derandomized novelty detection with FDR control via conformal e-values
Venue: NeurIPS 2023
Accepted: yes
Keywords: Conformal inference

Derandomization

E-values

False discovery rate

Out-of-distribution testing

Testing for outliers

Uncertainty
Abstract: Conformal inference provides a general distribution-free method to rigorously calibrate the output of any machine learning algorithm for novelty detection. While this approach has many strengths, it has the limitation of being randomized, in the sense that it may lead to different results when analyzing twice the same data and this can hinder the interpretation of any findings. We propose to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance. This solution allows the evidence gathered from multiple analyses of the same data to be aggregated effectively while provably controlling the false discovery rate. Further, we show that the proposed method can reduce randomness without much loss of power compared to standard conformal inference, partly thanks to an innovative way of weighting conformal e-values based on additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.

Derandomized novelty detection with FDR control via conformal e-values

Abstract

Conformal inference provides a general distribution-free method to rigorously calibrate the output of any machine learning algorithm for novelty detection. While this approach has many strengths, it has the limitation of being randomized, in the sense that it may lead to different results when analyzing twice the same data, and this can hinder the interpretation of any findings. We propose to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance. This solution allows the evidence gathered from multiple analyses of the same data to be aggregated effectively while provably controlling the false discovery rate. Further, we show that the proposed method can reduce randomness without much loss of power compared to standard conformal inference, partly thanks to an innovative way of weighting conformal evalues based on additional side information carefully extracted from the same data. Simulations with synthetic and real data confirm this solution can be effective at eliminating random noise in the inferences obtained with state-of-the-art alternative techniques, sometimes also leading to higher power.

1 Introduction

1.1 Background and motivation

A common problem in statistics and machine learning is to determine which samples, among a collection of new observations, were drawn from the same distribution as a reference data set (Wilks, 1963; Riani et al., 2009; Chandola et al., 2009). This task is known as novelty detection, out-ofdistribution testing, or testing for outliers , and it arises in numerous applications within science,

Amir Epstein

engineering, and business, including for example in the context of medical diagnostics (Tarassenko et al., 1995), security monitoring (Zhang et al., 2013), and fraud detection (Ahmed et al., 2016). This paper looks at the problem from a model-free perspective, in the sense that it does not rely on parametric assumptions about the data-generating distributions, which are generally unknown and complex. Instead, we apply powerful machine learning models for one-class (Moya et al., 1993) or binary classification to score the new samples based on how they conform to patterns observed in the reference data, and then we translate such scores into rigorous tests using conformal inference.

Conformal inference (Vladimir et al., 2005; Lei et al., 2013) provides flexible tools for extracting provably valid novelty detection tests from any black-box model. The simplest implementation is based on random sample splitting. This consists of training a classifier on a subset of the reference data, and then ranking the output score for each test point against the corresponding scores evaluated out-of-sample for the hold-out reference data. As the latter do not contain outliers, the aforementioned rank is uniformly distributed under the null hypothesis that the test point is not an outlier (Laxhammar and Falkman, 2015; Smith et al., 2015; Guan and Tibshirani, 2022), as long as some relatively mild exchangeability assumptions hold. In other words, this calibration procedure yields a conformal p-value that can be utilized to test for outliers while rigorously controlling the probability of making a false discovery -incorrectly labeling an inlier data point as an 'outlier". Further, split-conformal inference produces only weakly dependent p-values for different test points (Bates et al., 2023), allowing exact control of the expected proportion of false discoveries-the false discovery rate (FDR)-with the powerful Benjamini-Hochberg (BH) filter (Benjamini and Hochberg, 1995).

As visualized in Figure 1a, a limitation of split-conformal inference is that it is randomized-its results for a given data set are unpredictable because they depend on how the reference samples are divided between the training and calibration subsets. However, higher stability is desirable in practice, as randomized methods generally tend to be less reliable and more difficult to interpret (Murdoch et al., 2019; Yu and Kumbier, 2020). This paper addresses the problem of making conformal inferences more stable by developing a principled method to powerfully aggregate tests for outliers obtained with repeated splits of the same data set, while retaining provable control of the FDR. This problem is challenging because dependent p-values for the same hypothesis are difficult to aggregate without incurring into a significant loss of power (V ovk and Wang, 2020; Vovk et al., 2022).

10

5

0

5

10

10

5

0

5

10

(a)

Standard

conformal.

10

5

0

5

10

10

5

0

5

10

(b)

conformal.

1.2 Main contributions

This paper utilizes carefully constructed conformal e-values (Vovk and Wang, 2021) instead of pvalues to quantify statistical significance when testing for outliers under FDR control. The advantage of e-values is that they make it possible to aggregate the results of mutually dependent tests in a relatively simple way, enabling an effective approach to derandomize conformal inferences. Our contribution is to develop a martingale-based method inspired by Ren and Barber (2023) that leverages e-value ideas efficiently, as different types of e-values can be constructed but not all would be powerful

in our context due to the discrete nature of the statistical evidence in conformal inference. We further refine this method and boost power by adaptively weighting our conformal e-values based on an estimate of the out-of-sample accuracy of each underlying machine learning model. A preview of the performance of our solution is given by Figure 1b, which shows that our method can achieve power comparable to that of standard conformal inferences while mitigating the algorithmic randomness.

1.3 Related work

This paper builds upon e-values (Vovk and Wang, 2021): quantitative measures of statistical evidence, alternative to p-values, that lend themselves well to the derandomization of data-splitting procedures and to FDR control under dependence (Wang and Ramdas, 2022). There exist several generic methods for converting any p-value into an e-value (Vovk and Wang, 2021). While those p-to-e calibrators could be applied for our novelty detection problem, their power turns out to be often quite low due to the fact that conformal p-values are discrete and cannot take very small values unless the sample size is extremely large; see the Supplementary Section S5 for more details.

Therefore, we propose a novel construction of (slightly generalized) e-values inspired by the work of Ren and Barber (2023) on the derandomization of the knockoff filter (Barber and Candès, 2015), which focused on a completely different high-dimensional variable selection problem. A different approach for producing e-values in the context of conformal inference can also be found in Ignatiadis et al. (2023), although the latter did not focus on derandomization. Our approach differs from that of Ignatiadis et al. (2023) because we construct e-values simultaneously for the whole test set, aiming to control the FDR, instead of operating one test point at a time. Simulations show that our approach tends to yield higher power, especially if the test data contain many outliers.

Our second novelty consists of developing a principled method for assigning data-driven weights to conformal e-values obtained from different machine learning models, in such a way as to further boost power. This solution re-purposes transductive (Vovk, 2013) conformal inference ideas to leverage information contained in the test data themselves while calibrating the conformal inferences, increasing the power to detect outliers similarly to Marandon et al. (2022) and Liang et al. (2022).

While this paper focuses on derandomizing split-conformal inferences, there exist other distributionfree methods that can provide finite-sample tests for novelty detection, such as full-conformal inference (Vladimir et al., 2005) and cross-validation+ (Barber et al., 2021). Those techniques are more computationally expensive but have the advantage of yielding relatively more stable conformal p-values because they do not rely on a single random data split. However, full-conformal inference and cross-validation+ also produce conformal p-values with more complicated dependencies, which make it difficult to control the FDR without large losses in power (Benjamini and Yekutieli, 2001) or very expensive computations (Fithian and Lei, 2022; Liang et al., 2022).

Finally, prior works studied how to stabilize conformal predictors by calibrating the output of an ensemble of simpler models (Löfström et al., 2013; Beganovic and Smirnov, 2018; Linusson et al., 2020; Kim et al., 2020; Gupta et al., 2022). However, we consider a distinct problem as we focus on derandomizing conformal novelty detection methods while controlling the FDR.

2 Relevant technical background

2.1 Notation and problem setup

Consider n observations, X i ∈ R d , sampled exchangeably (or, for simplicity, independent and identically distributed) from some unknown distribution P 0 , for all i ∈ D = [ n ] = { 1 , . . . , n } . Then, imagine observing a test set of n test 'unlabeled' samples X j ∈ R d . The problem is to test, for each j ∈ D test = [ n + n test ] \ [ n ] , the null hypothesis that X j is also an inlier , in the sense that it was randomly sampled from P 0 exchangeably with the data in D . We refer to a rejection of this null hypothesis as the discovery that X j is an outlier , and we indicate the set of true inlier test points as D null test , with n null test = |D null test | . For each j ∈ D test , define R j as the binary indicator of whether X j is labeled by our method as an outlier. Then, the goal is to discover as many true outliers as possible while controlling the FDR, defined as FDR = E [( ∑ j ∈D null test R j ) / max { 1 , ∑ j ∈D test R j } ] .

2.2 Review of FDR control with conformal p-values

After randomly partitioning D into two disjoint subsets D train and D cal , of cardinality n train and n cal = n -n train respectively, the standard approach for computing split-conformal p-values begins by training a one-class classification model on the data indexed by D train . This model is applied out-of-sample to compute conformity scores ˆ S i and ˆ S j for all calibration and test points i ∈ D cal and j ∈ D test , with the convention that larger scores suggest evidence of an outlier. Assuming without loss of generality that all scores take distinct values (otherwise, ties can be broken at random by adding a little noise), a conformal p-value ˆ u ( X j ) for each j ∈ D test is then calculated by taking the relative rank of ˆ S j among the ˆ S i for all i ∈ D cal : ˆ u ( X j ) = (1 + ∑ i ∈D cal I { ˆ S j ≤ ˆ S i } ) / (1 + n cal ) . If the null hypothesis for X j is true, ˆ S j is exchangeable with ˆ S i for all i ∈ D cal , and ˆ u ( X j ) is uniformly distributed on { 1 / (1 + n cal ) , 2 / (1 + n cal ) , . . . , 1 } . Since this distribution is stochastically larger than the continuous uniform distribution on [0 , 1] , one can say that ˆ u ( X j ) is a valid conformal p-value. Note however that the p-values ˆ u ( X j ) and ˆ u ( X j ′ ) for two different test points j, j ′ ∈ D test are not independent of one another, even conditional on D train , because they share the same calibration data.

Despite their mutual dependence, conformal p-values can be utilized within the BH filter to simultaneously probe the n test hypotheses for all test points while controlling the FDR. A convenient way to explain the BH filter is as follows (Storey, 2002). Imagine rejecting the null hypothesis for all test points j with ˆ u ( X j ) ≤ s , for some threshold s ∈ [0 , 1] . By monotonicity of ˆ u ( X j ) , this amounts to rejecting the null hypothesis for all test points j with ˆ S j ≥ t , for some appropriate threshold t ∈ R . An intuitive estimate of the proportion of false discoveries incurred by this rule is:

This can be understood by noting that ∑ j ∈D test I { S ( k ) j ≥ t } is the total number of discoveries, while the numerator should behave similarly to the (latent) number of false discoveries in D test due to the exchangeability of ˆ S i and ˆ S j under the null hypothesis. With this notation, it can be shown that the BH filter applied at level α ∈ (0 , 1) computes an adaptive threshold

and rejects all null hypotheses j with ˆ S j ≥ ˆ t BH ; see Rava et al. (2021) for a derivation of this connection. This procedure was proved by Bates et al. (2023) to control the FDR below α .

2.3 Review of FDR control with AdaDetect

Recently, Marandon et al. (2022) proposed AdaDetect , a more sophisticated version of the method reviewed in Section 2.2. The main innovation of AdaDetect is that it leverages a binary classification model instead of a one-class classifier. In particular, AdaDetect trains a binary classifier to distinguish the inlier data in D train from the mixture of inliers and outliers contained in the union of D cal and D test . The key idea to achieve FDR control is that the training process should remain invariant to permutations of the calibration and test samples. While the true inlier or outlier nature of the observations in D test is obviously unknown at training time, AdaDetect can still extract some useful information from the test data which would otherwise be ignored by the more traditional split-conformal approach reviewed in Section 2.2. In particular, AdaDetect can leverage the test data to automatically tune any desired model hyper-parameters in order to approximately maximize the number of discoveries. A similar idea also motivates the alternative method of integrative conformal p-values proposed by Liang et al. (2022), although the latter requires the additional assumption that some labeled outlier data are available, and is therefore not discussed in equal detail within this paper.

Despite a more sophisticated use of the available data compared to the split-conformal method reviewed in Section 2.2, AdaDetect still suffers from the same limitation that it must calibrate its inferences based on a single random data subset D cal , and thus its results remain aleatory. For simplicity, Section 3.1 begins by explaining how to derandomize standard split-conformal inferences; then, the proposed method will be easily extended in Section 3.3 to derandomize AdaDetect .

3 Method

3.1 Derandomizing split-conformal inferences

Consider K ≥ 1 repetitions of the split-conformal analysis reviewed in Section 2.2, each starting with an independent split of the same reference data into D ( k ) train and D ( k ) cal . For each repetition k ∈ [ K ] , after training the machine learning model on D ( k ) train and computing conformity scores on D ( k ) cal and D test , one can estimate the false discovery proportion corresponding to the rejection of all test points with scores above a fixed rejection threshold t ∈ R , similarly to (1), with:

Note that the estimate in (3) differs slightly from that in (1) as it lacks the '+1' constant term in the numerator and denominator. While it is possible to include the '+1' terms in (3), this is not needed by our theory and we have observed that it often makes our method unnecessarily conservative. For any fixed α bh ∈ (0 , 1) , let ˆ t ( k ) be the corresponding BH threshold (2) at the nominal FDR level α bh :

where ˜ D ( k ) cal -test = { ˆ S ( k ) i } i ∈D test ∪D ( k ) cal . For each test point j ∈ D test , define the following rescaled indicator of whether ˆ S ( k ) j exceeds ˆ t ( k ) :

Intuitively, this quantifies not only whether the j -th null hypothesis would be rejected by the BH filter at the nominal FDR level α bh , but also how extreme ˆ S ( k ) j is relative to the calibration scores. In other words, a large e ( k ) j suggests that the test point may be an outlier, where this variable can take any of the following values: 0, 1, (1 + n cal ) /n cal , (1 + n cal ) / ( n cal -1) , . . . , (1 + n cal ) . This approach, inspired by Ren and Barber (2023), is not the only possible way of constructing e-values to derandomize conformal inferences, as discussed in Supplementary Section S5. However, we will show that it works well in practice and it typically achieves higher power compared to standard p-to-e calibrators (Vovk and Wang, 2021) applied to conformal p-values. This advantage partly derives from the fact that (5) can gather strength from many different test points, and partly from the fact that it is not a proper e-value according to the original definition of V ovk and Wang (2021), in the sense that its expected value may be larger than one even if X j is an inlier. Instead, we will show that our e-values satisfy a relaxed average validity property (Ren and Barber, 2023) that is sufficient to guarantee FDR control while allowing more numerous discoveries.

After evaluating (5) for all j ∈ D test and all k ∈ [ K ] , we aggregate the evidence against the j -th null hypothesis into a single statistic ¯ e j by taking a weighted average:

based on some appropriate normalized weights w ( k ) . Intuitively, the role of w ( k ) is to allow for the possibility that the machine learning models based on different realizations of the training subset may not all be equally powerful at separating inliers from outliers. In the remainder of this section, we will take these weights to be known a-priori for all k ∈ [ K ] , thus representing relevant side information ; e.g., in the sense of Genovese et al. (2006) and Ren and Candès (2023). For simplicity, one may think for the time being of trivial uninformative weights w ( k ) = 1 /K . Of course, it would be preferable to allow these weights to be data-driven, but such an extension is deferred to Section 3.2 for conciseness.

Having calculated aggregate e-values ¯ e j with the procedure described above, which is outlined by Algorithm S1 in the Supplementary Material, our method rejects the null hypothesis for all j ∈ D test whose ¯ e j is greater than an adaptive threshold calculated by applying the eBH filter of Wang and

Ramdas (2022), which is outlined for completeness by Algorithm S2 in the Supplementary Material. We refer to Wang and Ramdas (2022) for a more detailed discussion of the eBH filter. Here, it suffices to recall that the eBH filter computes an adaptive rejection threshold based on the n test input e-values and on the desired FDR level α ∈ (0 , 1) . Then, our following result states that the overall procedure is guaranteed to control the FDR below α , under a relatively mild exchangeability assumption.

Assumption 3.1. The inliers in D and the null test points are exchangeable conditional on the non-null test points.

Theorem 3.2. Suppose Assumption 3.1 holds. Then, the e-values computed by Algorithm S1 satisfy:

The proof of Theorem 3.2 is in the Supplementary Section S2. Combined with Theorem 2 from Ren and Barber (2023), this result implies our method controls the FDR below the desired target level α .

Corollary 3.3 (Ren and Barber (2023)) . The eBH filter of Wang and Ramdas (2022) applied at level α ∈ (0 , 1) to e-values { ¯ e j } j ∈D test , satisfying (6) guarantees FDR ≤ α .

Remark 3.4 . Assumption 3.1 does not require that the inliers are independent of the outliers.

Remark 3.5 . Theorem 3.2 holds regardless of the value of the hyper-parameter α bh of Algorithm S1, which appears in (4). See Section 3.4 for further details about the choice of α bh .

3.2 Leveraging data-driven weights

Our method can be extended to leverage adaptive weights based on the data in D and D test , as long as each weight w ( k ) is invariant to permutations of the test point with the corresponding calibration samples in D ( k ) cal . In other words, we only require that these weights be written in the form of

The function ω may depend on D ( k ) train but not on D ( k ) cal or D test . An example of a useful weighting scheme satisfying this property is at the end of this section. The general method is summarized by Algorithm S3 in the Supplementary Material, which extends Algorithm S1. This produces e-values that control the FDR in conjunction with the eBH filter of Wang and Ramdas (2022).

Theorem 3.6. Suppose Assumption 3.1 holds. Then, the e-values computed by Algorithm S3 satisfy (6) , as long as the adaptive weights obey (7) .

An example of a valid weighting function applied in this paper is the following. Imagine having some prior side information suggesting that the proportion of outliers in D test is approximately γ ∈ (0 , 1) . Then, a natural choice to measure the quality of the k -th model is to let ˜ w ( k ) = | ˜ v ( k ) | , where ˜ v ( k ) is the standard t-statistic for testing the difference in means between the top ⌈ n test · γ ⌉ largest values in ˜ D ( k ) cal -test and the remaining ones. See Algorithm S5 in the Supplementary Material for further details. Intuitively, Algorithm S5 tends to assign larger weights to models achieving stronger out-of-sample separation between inliers and outliers. Of course, this approach may not always be optimal but different weighting schemes could be easily accommodated within our framework.

3.3 Derandomizing AdaDetect with E-AdaDetect

The requirement discussed in Section 3.2 that the data-adaptive weights should be invariant to permutations of the calibration and test samples is analogous to the idea utilized by AdaDetect (Marandon et al., 2022) to train more powerful machine learning models leveraging also the information contained in the test set; see Section 2.3. This implies that Theorem 3.6 remains valid even if our method is implemented based on K machine learning models each trained by looking also at the unordered union of all data points in D ( k ) cal ∪ D test , for each k ∈ [ K ] . See Algorithm S4 in the Supplementary Material for a detailed implementation of this extension of our method, which we call E-AdaDetect .

3.4 Tuning the FDR hyper-parameter

As explained in Section 3.1, our method involves a hyper-parameter α bh controlling the BH thresholds ˆ t ( k ) in (4). Intuitively, higher values of α bh tend to increase the number of both test and calibration

scores exceeding the rejection threshold at each of the K iterations. Such competing effects make it generally unclear whether increasing α bh leads to larger e-values in (5) and hence higher power. This trade-off was studied by Ren and Barber (2023) while derandomizing the knockoff filter, and they suggested setting α bh < α . In this paper, we adopt α bh = α/ 10 , which we have observed to work generally well in our context, although even higher power can sometimes be obtained with different values of α bh , especially if the number of outliers in the test set is large. While we leave it to future research to determine whether further improvements are possible, it is worth noting that a straightforward extension of our method, not explicitly implemented in this paper, can be obtained by further averaging e-values obtained with different choices of α bh . Such extension does not affect the validity of (6) due to the linearity of expected values.

4 Numerical experiments

4.1 Setup and performance metrics

This section compares empirically the performance of AdaDetect and our proposed derandomized method described in Section 3.3, namely E-AdaDetect . Both procedures are deployed using a binary logistic regression classifier (Marandon et al., 2022) as the base predictive model. The reason why we focus on derandomizing AdaDetect instead of traditional split-conformal inferences based on a one-class classifier (Bates et al., 2023) is that we have observed that AdaDetect often achieves higher power on the data considered in this paper, which makes it a more competitive benchmark. However, additional experiments reporting on the performance of our derandomization method applied in combination with one-class classifiers can be found in the Supplementary Sections S4.2 and S6.2.

As the objective of this paper is to powerfully detect outliers while mitigating algorithmic randomness, we assess the performance of each method over M = 100 independent analyses based on the same fixed data and the same test set. For each repetition m of the novelty detection analysis based on the fixed data, we identify a subset R ( m ) ⊆ D test of likely outliers (the rejected null hypotheses) and evaluate the average power and false discovery proportion, namely

where D non-null test = D test \ D null test indicates the true outliers in the test set. The average false discovery proportion defined in (8) is not the FDR, which is the quantity we can theoretically guarantee to control. In fact, FDR = E [ ̂ FDR ] , with expectation taken with respect all randomness in the data. Nonetheless, we will see that this average false discovery proportion is also controlled in practice within all data sets considered in this paper. The advantage of this setup is that it makes it natural to estimate algorithmic variability by observing the consistency of each rejection across independent analyses. In particular, after defining R j,m as the indicator of whether the j -th null hypothesis was rejected in the m -th analysis, we can evaluate the average variance in the rejection events:

where ¯ R j = (1 /M ) ∑ M m =1 R j,m . Intuitively, it would be desirable to maximize power while simultaneously minimizing both the average false discovery proportion and the variability. In practice, however, these metrics often compete with one another; hence why we focus on comparing power and variability for methods designed to control the FDR below the target level α = 0 . 1 .

4.2 Experiments with synthetic data

Synthetic reference and test data consisting of 100-dimensional vectors X are generated as follows. The reference set contains only inliers, drawn i.i.d. from the standard normal distribution with independent components, N (0 , I 100 ) . Unless specified otherwise, the test set contains 90% inliers and 10% outliers, independently sampled from N ( µ, I 100 ) . The first 5 entries of µ are equal to a constant parameter, to which we refer as the signal amplitude , while the remaining 95 entries are zeros. The size of the reference set is n = 2000 , with 1000 samples in the training subset and 1000 in the calibration subset. The size of the test set is n test = 1000 . Both E-AdaDetect and AdaDetect are applied based on the same logistic regression classifier with default hyper-parameters.

4.2.1 The effect of the signal strength

Figure 2 compares the performance of E-AdaDetect (applied with K = 10 ) to that of AdaDetect , as a function of the signal amplitude. The results confirm both methods control the FDR but ours is less variable, as expected. The comparison becomes more interesting when looking at power: AdaDetect tends to detect more outliers on average if the signal strength is low, but E-AdaDetect can also outperform by that metric if the signals are strong. This may be explained as follows. If the signal strength is high, most true discoveries produced by AdaDetect are relatively stable across different analyses, while false discoveries may be more aleatory, consistently with the illustration of Figure 1. Such situation is ideal for derandomization, which explains why E-AdaDetect is able to simultaneously achieve high power and low false discovery proportion. By contrast, if the signals are weak, the true outlier discoveries produced by AdaDetect are relatively scarce and unpredictable, thus behaving not so differently from the false findings. In this case, one could argue that stability becomes even more important to facilitate the interpretation of any findings, and that may justify some loss in average power.

1.0

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

4.2

4.4

4.6

Signal amplitude

0.00

0.25

0.50

0.75

1.00

Power

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

4.2

4.4

4.6

Signal amplitude

0.0

0.2

0.4

0.6

0.8

AdaDetect

E-AdaDetect

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

4.2

4.4

4.6

Signal amplitude

0.00

0.01

0.02

4.2.2 The effect of the number of analyses K

Figure 3 investigates the effect of varying the number of analyses K aggregated by E-AdaDetect . Here, the signal amplitude is fixed to 3.4 (strong signals), while K is varied between 1 and 30. As expected, the results show that the variability of the findings obtained with E-AdaDetect decreases as K increases. The average proportion of false discoveries obtained with E-AdaDetect also tends to decrease when K is large, which can be understood by noting that spurious findings are less likely to be reproduced consistently across multiple independent analyses of the same data. Regarding power, the average number of true outliers detected by E-AdaDetect appears to monotonically increase with K , although this is not always true in other situations, as shown in the Supplementary Section S4.1. In fact, if the signals are weak, E-AdaDetect may lose some power with larger values of K (although some K > 1 may be optimal), consistently with the results shown in Figure 2. Thus, we recommend practitioners to utilize larger values of K in applications where higher power is expected. Finally, note that the power of E-AdaDetect is generally lower compared to that of AdaDetect in the special case of K = 1 , although this is not a practically relevant value of K because it does not allow any derandomization. The reason why the power of E-AdaDetect is lower when K = 1 is that this method relies on the eBH filter. The latter is relatively conservative as an FDR-controlling strategy because it requires no assumptions about the dependencies of the input statistics.

4.2.3 The effect of the weighting strategy

This section highlights the practical advantage of being able to use data-adaptive model weights within E-AdaDetect . For this purpose, we carry out experiments similar to those of Figure 2, but leveraging a logistic regression model trained with different choices of hyper-parameters in each of the K analyses. Specifically, we fit a sparse logistic regression model using K = 10 different values of the regularization parameter. To induce higher variability in the predictive rules, one model was trained with a regularization parameter equal to 0 . 0001 , while the others were trained with regularization parameters equal to 1 , 10 , 50 , and 100 , respectively. Then, we apply E-AdaDetect using different weighting schemes: constant equal weights ('uniform'), data-driven weights calculated

1

3

5

7

10

15

20

30

Number of iterations

0.4

0.6

0.8

Power

1

3

5

7

10

15

20

30

Number of iterations

0.0

0.2

0.4

0.6

0.8

1.0

E-AdaDetect

AdaDetect

1

3

5

7

10

15

20

30

Number of iterations

0.005

0.010

0.015

0.020

0.025

with the t-statistic approach ('t-test') summarized by Algorithm S5, and a simple alternative trimmed average data-driven approach ('avg. score') outlined by Algorithm S6 in the Supplementary Material. The results in Figure 4 show that the data-driven aggregation scheme based on t-statistics is the most effective one, often leading to much higher power. We have chosen not to compare E-AdaDetect to the automatic AdaDetect hyper-parameter tuning strategy proposed in Section 4.5 of Marandon et al. (2022) because we found that it does not perform very well in our experiments, possibly due to the relatively low sample size.

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

Signal amplitude

0.00

0.25

0.50

0.75

1.00

Power

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

Signal amplitude

0.0

0.2

0.4

0.6

0.8

1.0

E-AdaDetect (uniform)

E-AdaDetect (t-test)

E-AdaDetect (avg. score)

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

Signal amplitude

0.000

0.005

0.010

0.015

0.020

To further demonstrate the effectiveness of data-driven weighting, we turn to analyze the performance of E-AdaDetect on four real-world outlier detection data sets: musk , shuttle , KDDCup99 , and credit card . We refer to Supplementary Section S6 for more information regarding these data. Similar to Figure 4, our E-AdaDetect method is applied K = 10 times to each data set, each time leveraging a different predictive model as follows. Half of the models are random forests implemented with varying max-depth hyper-parameters (10, 12, 20, 30, and 7), while the other half are support vector machines with an RBF kernel with varying width hyper-parameters (0.1, 0.001, 0.5, 0.2, and 0.03). This setup is interesting because different models often tend to perform differently in practice, and it is usually unclear a-priori which combination of model and hyper-parameters is optimal for a given data set. Figure 5 summarizes the results, demonstrating that both data-driven weighting schemes ('t-test' and 'avg. score') lead to more numerous discoveries compared to the 'uniform' weighting baseline, and that the 't-test' approach is the most powerful weighting scheme here. These results are in line with the synthetic experiment presented in Figure 4. Lastly, the variance metrics reported in Figure 5 also suggest that data-driven weighting further enhances the algorithmic stability.

4.3 Additional results from experiments with synthetic and real data

Sections S4-S6 in the Supplementary Material present the results of several additional experiments. In particular, Section S4 focuses on experiments with synthetic data. Section S5 describes comparisons with alternative derandomization approaches based on different types of p-to-e calibrators (Vovk and Wang, 2021) operating one test point at a time, which turn out to yield lower power compared to our martingale-based method. The results also show that our method compares favorably to an alternative derandomization method based on an e-value construction that first appeared in Ignatiadis et al. (2023), especially if the test set contains numerous outliers or if the nominal FDR level is not

musk

shuttle

KDDCup99

creditcard

Dataset

0.0

0.5

1.0

Power

musk

shuttle

KDDCup99

creditcard

Dataset

0.00

0.25

0.50

0.75

1.00

E-AdaDetect (uniform)

E-AdaDetect (t-test)

E-AdaDetect (avg. score)

musk

shuttle

KDDCup99

creditcard

Dataset

0.00

0.01

0.02

too low. Finally, Section S6 describes additional numerical experiments based on several real data sets also studied in Bates et al. (2023) and Marandon et al. (2022). These results confirm that our martingale-based e-value method can mitigate the algorithmic randomness of standard conformal inferences and AdaDetect while retaining relatively high power.

5 Discussion

Our experience suggests that e-values are often less powerful than p-values in measuring the statistical evidence against a single hypothesis . Yet, e-values can be useful to aggregate multiple dependent tests of the same hypothesis (Vovk and Wang, 2021)-a task that would otherwise require very conservative adjustments within the p-value framework (Vovk et al., 2022). Further, we have shown that e-values lend themselves well to multiple testing because they allow efficient FDR control under arbitrary dependence (Wang and Ramdas, 2022), and even their relatively weak individual evidence can accumulate rapidly when a large number of hypotheses is probed. The opportunity arising from the combination of these two key properties was recently leveraged to derandomize knockoffs (Ren and Barber, 2023), but until now it had not been fully exploited in the context of conformal inference.

While this paper has focused on derandomizing split-conformal and AdaDetect inferences for novelty detection, the key ideas could be easily extended. For example, one may utilize e-values to derandomize conformal prediction intervals in regression (Lei and Wasserman, 2014; Romano et al., 2019) or prediction sets for classification (Lei et al., 2013; Romano et al., 2020) while controlling the false coverage rate over a large test set (Weinstein and Ramdas, 2020). A different direction for future research may explore the derandomization of cross-validation+ (Barber et al., 2021).

We conclude by discussing two limitations of this work. First, the proposed method is more computationally expensive compared to standard conformal inference or AdaDetect , and therefore one may be limited to applying it with relatively small numbers K of analysis repetitions when working with very large data sets. That being said, it is increasingly recognized that stability is an important goal in data science (Murdoch et al., 2019), and thus mitigating algorithmic randomness may often justify the deployment of additional computing resources. Second, we have shown that our method sometimes leads to a reduction in the average number of findings compared to randomized alternatives, especially in applications where few discoveries are expected. Therefore, in situations with few anticipated discoveries, practitioners considering applying our method should carefully weigh the anticipated gains in stability versus a possible reduction in power.

Mathematical proofs and additional supporting results are in the Supplementary Material. Software implementing the algorithms described in this paper and enabling the reproduction of the associated numerical experiments is available at https://github.com/Meshiba/derandomized-novelty-detection.

References

Ahmed, A. N. Mahmood, and M. R. Islam. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems , 55:278-288, 2016.

F. Barber and E. Candès. Controlling the false discovery rate via knockoffs. Ann. Stat. , 43(5): 2055-2085, 2015.

F. Barber, E. Candès, A. Ramdas, R. J. Tibshirani, et al. Predictive inference with the jackknife+. Ann. Stat. , 49(1):486-507, 2021.

Bates, E. Candès, L. Lei, Y. Romano, and M. Sesia. Testing for outliers with conformal p-values. Ann. Stat. , 51(1):149 - 178, 2023.

Beganovic and E. Smirnov. Ensemble cross-conformal prediction. In 2018 IEEE International Conference on Data Mining Workshops , pages 870-877. IEEE, 2018.

Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B , 57(1):289-300, 1995.

Benjamini and D. Yekutieli. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. , pages 1165-1188, 2001.

Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR) , 41(3):1-58, 2009.

Fithian and L. Lei. Conditional calibration for false discovery rate control under dependence. Ann. Stat. , 50(6):3091-3118, 2022.

R. Genovese, K. Roeder, and L. Wasserman. False discovery control with p-value weighting. Biometrika , 93(3):509-524, 2006.

Guan and R. Tibshirani. Prediction and outlier detection in classification problems. J. R. Stat. Soc. Series B , 84(2):524, 2022.

Gupta, A. K. Kuchibhotla, and A. Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition , 127:108496, 2022.

Ignatiadis, R. Wang, and A. Ramdas. E-values as unnormalized weights in multiple testing. arXiv preprint arXiv:2204.12447 , 2023. version 4.

Kim, C. Xu, and R. Barber. Predictive inference is free with the jackknife+-after-bootstrap. Adv. Neural Inf. Process. Syst. , 33:4138-4149, 2020.

Laxhammar and G. Falkman. Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories. Annals of Mathematics and Artificial Intelligence , 74(1-2):67-94, 2015.

Lei and L. Wasserman. Distribution-free prediction bands for non-parametric regression. J. R. Stat. Soc. Series B , 76(1):71-96, 2014.

Lei, J. Robins, and L. Wasserman. Distribution-free prediction sets. J. Am. Stat. Assoc. , 108(501): 278-287, 2013.

Liang, M. Sesia, and W. Sun. Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers. arXiv preprint arXiv:2208.11111 , 2022.

Linusson, U. Johansson, and H. Boström. Efficient conformal predictor ensembles. Neurocomputing , 397:266-278, 2020.

Löfström, U. Johansson, and H. Boström. Effective utilization of data in inductive conformal prediction. In Proc. Int. Jt. Conf. Neural Netw. IEEE, 2013.

Marandon, L. Lei, D. Mary, and E. Roquain. Machine learning meets false discovery rate. arXiv preprint arXiv:2208.06685 , 2022.

M. Moya, M. W. Koch, and L. D. Hostetler. One-class classifier networks for target recognition applications. NASA STI/Recon Technical Report N , 93:24043, 1993.

J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. Proc. Natl. Acad. Sci. U.S.A. , 116(44):2207122080, 2019.

Rava, W. Sun, G. M. James, and X. Tong. A burden shared is a burden halved: A fairness-adjusted approach to classification. arXiv preprint arXiv:2110.05720 , 2021.

Ren and R. F. Barber. Derandomised knockoffs: leveraging e-values for false discovery rate control. J. R. Stat. Soc. Series B , 09 2023.

Ren and E. Candès. Knockoffs with side information. Ann. Appl. Stat. , 17(2):1152-1174, 2023.

Riani, A. C. Atkinson, and A. Cerioli. Finding an unknown number of multivariate outliers. J. R. Stat. Soc. Series B , 71(2):447-466, 2009.

Romano, E. Patterson, and E. Candes. Conformalized quantile regression. Adv. Neural Inf. Process. Syst. , 32, 2019.

Romano, M. Sesia, and E. Candès. Classification with valid and adaptive coverage. Adv. Neural Inf. Process. Syst. , 33, 2020.

Smith, I. Nouretdinov, R. Craddock, C. Offer, and A. Gammerman. Conformal anomaly detection of trajectories with a multi-class hierarchy. In International symposium on statistical learning and data sciences , pages 281-290. Springer, 2015.

D. Storey. A direct approach to false discovery rates. J. R. Stat. Soc. Series B , 64(3):479-498, 2002.

Tarassenko, P. Hayton, N. Cerneaz, and M. Brady. Novelty detection for the identification of masses in mammograms. In 1995 Fourth International Conference on Artificial Neural Networks , pages 442-447. IET, 1995.

Vladimir, G. Alex, and S. Glenn. Algorithmic learning in a random world . Springer, 2005.

Vovk. Transductive conformal predictors. In IFIP International Conference on Artificial Intelligence Applications and Innovations , pages 348-360. Springer, 2013.

Vovk and R. Wang. Combining p-values via averaging. Biometrika , 107(4):791-808, 2020.

Vovk and R. Wang. E-values: Calibration, combination and applications. Ann. Stat. , 49(3), jun 2021.

Vovk, B. Wang, and R. Wang. Admissible ways of merging p-values under arbitrary dependence. Ann. Stat. , 50(1):351-375, 2022.

Wang and A. Ramdas. False discovery rate control with e-values. J. R. Stat. Soc. Series B , 84(3): 822-852, 2022.

Weinstein and A. Ramdas. Online control of the false coverage rate and false sign rate. In International Conference on Machine Learning , pages 10193-10202. PMLR, 2020.

S. Wilks. Multivariate statistical outliers. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 407-426, 1963.

Yu and K. Kumbier. Veridical data science. Proc. Natl. Acad. Sci. U.S.A. , 117(8):3920-3929, 2020.

Zhang, A. Raghunathan, and N. K. Jha. MedMon: Securing medical devices through wireless monitoring and anomaly detection. IEEE Transactions on Biomedical circuits and Systems , 7(6): 871-881, 2013.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applied the derandomized e-value to the conformal novelty detection, which reduces the randomness of original approach using conformal p-value. The authors also refined the method by adaptively weighting the conformal p-values based on an estimate of the out-of-sample accuracy of each underlying machine learning model. Simulations with synthetic and real data are conducted to compare the performance with the original approach.\n\nSTRENGTHS:\nThis paper is the first work to apply the derandomized e-value to the conformal inference, which makes it more stable compared with the original approach.\n\nWEAKNESSES:\n### 1. The novelty of this paper is limited.\n\nConsidering the previous works by applying derandomization and E-value to Knockoff filter [Ren et al., 2020, Ren and Baber 2023], it is straightforward to extend it to the conformal setting.\n\n### 2. The challenges are not stated clearly.\n\n (1) From the connection between the conformal BH filter and Eq. (2) (given by Rava et al. [2021]), it would be easy to extend the construction in [Ren and Baber 2022] to conformal e-value in Eq. (5). In addition, the weighted e-value is also proposed by Ren and Baber [2022].\n\n (2) Given the Theorem 2 from Ren and Baber [2022], all the technical difficulty falls in proving Theorem 3.2. However, the proof strategy is from  Rava et al. [2021].\n\nQUESTIONS:\n1. What is the superiority of data-driven weights over the fixed width? Is the weighted approach in Ren and Baber [2022] applicable here?\n\n2. In Figure 2, why is the power of E-AdaDetect higher than Ada-Detect while the FDR value is still lower? It seems contradictory to common sense.\n\nLIMITATIONS:\nThe main limitation is the novelty. Despite that this is the first work to deploy derandomized e-value to split conformal inference, the framework and the theory is well studied in previous works. In addition, the technical contribution is minor.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper employs conformal e-values, as opposed to p-values, to quantify statistical significance during outlier testing under FDR control. This approach enables the principled aggregation of results from mutually dependent tests, thereby providing a solution to de-randomize (split) conformal inferences.\n\nSTRENGTHS:\nThe paper addresses a significant issue of the problem of the randomness in conformal inferences. \n\nThey propose a method to make conformal inferences more stable by leveraging suitable conformal e-values instead of p-values to quantify statistical significance, also merging the idea of de-randomizing conformal novelty detection and FDR control.\n\nThe proposed method has the potential to significantly improve the stability and interpretability of conformal inferences.\n\nWEAKNESSES:\nThe paper could significantly benefit from revisions aimed at improving clarity. The current presentation of ideas and concepts is convoluted, making it difficult for readers to follow and understand the arguments and methodologies proposed. \n\nWhile the authors' approach is novel, it builds upon existing studies and techniques. The authors could strengthen the originality of their work by further highlighting the unique aspects of their approach and how it differs from previous methods.\n\nThe authors provide simulations to demonstrate the effectiveness of their method, but it would be beneficial to see more empirical evaluations, including comparisons with other state-of-the-art methods.\n\nQUESTIONS:\nThe paper focuses on derandomizing split-conformal inferences. I wonder if jackknife+ would inherently solve the problem.\n\nThe authors mention that their method can be extended to leverage adaptive weights based on the data. Could they discuss potential strategies for choosing these weights in practice?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main limitation of conformal prediction lies in its inherent randomness. However, this paper presents an innovative solution by introducing a derandomized version of conformal prediction, specifically applied to the field of novelty detection. Through the incorporation of conformal e-values, the proposed method successfully reduces the element of randomness while providing provable and effective control over the False Discovery Rate (FDR).\n\nThe key contribution of this research lies in its pioneering use of e-values, instead of traditional p-values, within the framework of conformal prediction. This innovative approach significantly simplifies the aggregation process, reducing randomness without compromising the overall detection power.\n\nSTRENGTHS:\nThis paper is a highly innovative and inspiring work that highlights the potential of e-values as a superior alternative to p-values for derandomizing conformal prediction through the aggregation of multiple dependent tests of the same hypothesis.\n\nThe paper introduces a novel approach for constructing e-values and provides a rigorous guarantee of false discovery rate (FDR) control, with Theorem 3.2 being the main contribution of the research.\n\nFurthermore, the practical aspect of the paper lies in the deployment of conformal e-values in AdaDetect, which presents a derandomized version of AdaDetect. It is important to note that E-AdaDetect is not merely a simple combination but rather a specific practical application demonstrating how to aggregate e-values using data-adaptive weights.\n\nThe experimental results showcased in the paper reveal that the derandomized AdaDetect exhibits comparable power to its randomized counterpart, while effectively controlling the FDR. This finding is particularly surprising (even higher power? )and highlights the potential benefits of adopting the \"new\"(not same as the classical definition) e-values .\n\nWEAKNESSES:\nThe main paper primarily focuses on experimental results obtained from synthetic data, which is sampled from simple Gaussians. It may be considered relatively easier compared to real-world scenarios. Therefore, I would suggest including at least one experiment using more complex synthetic data or real data from the supplementary material.\n\nQUESTIONS:\n*\n\nLIMITATIONS:\n*", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers proposes a way to reduce the randomness in novelty detection methods (detecting out-of-distribution points) that are based on the split-conformal inference paradigm. This is done by ensembling over several ($K$) train-validation splits of the dataset. The main technical point is to aggregate the evidence from the $K$ individual predictions by averaging their E-values, as a replacement to considering the p-value in the traditional approach with only one train-validation split. It is shown that the FDR (false discovery rate) can still be controlled with these quantities. The method is further enhanced by using certain weighted averages of the E-values, taking into account the estimated powers of each one. The methods are evaluated on synthetic and real data, showing that the proposed derandomized method works better if the fraction of outliers is larger.\n\nSTRENGTHS:\nThe proposed novelty detection approach is original in that it combines evidences from different algorithm runs into a single evidence score. This required a formulation of the algorithms' evidences in terms of E-values instead of p-values that are commonplace for conformal methods, to avoid a loss in detection power. While prior novelty detection works have touched upon E-values, the current paper succeeds in controlling the FDR based on these E-value evidences. The methodological details (Sec. 3.1) and mathematical proof (Theorems 3.2 and 3.6) of this FDR control are strongly inspired by existing works, but this technique is new in the area of novelty detection.\n\nAs to significance, the method does reduce the variance as promised and keeps its FDR guarantee. It has higher power than competitor methods in some regimes, but quite consistently weaker power in other regimes, particularly for low fraction of outliers. Secondly, while the problem and the solution is built around quite specific requirements, it is plausible that the presented techniques can be applied to the derandomization of other inference methods with statistical guarantees, as the authors say in their Conclusion.\n\nThe clarity of the paper is exceptional (but see my comments on Appendix S2 below), the explanations and discussions are to the point.\n\nWEAKNESSES:\nAs the biggest weakness of the paper I see that, despite the novelty of the approach, the considered problem requirements are relatively specific and the solution therefore narrow, as it concerns the de-randomization of certain novelty detection algorithms that are based on conformal inference and that aim at a mathematical control of the FDR.\n\nAlso, the method's performance (power) falls behind other methods in certain application regimes. It would be good to have some rules beforehand to know in which regime to apply which method.\n\nIf space permits, I would like to see some real data experiments (see Sec. 4.3) in the main text rather than only in the Supplement.\n\nQUESTIONS:\n* Please mention what K is in Fig. 1.\n* line 228: Should this be D^(k)_cal \\cup D_test?\n* How is AdaDetect different from E-AdaDetect with K=1 (Fig. 3)?\n* Please carefully re-write the Proof of Theorem 3.6 in Appendix S2, as I believe there are many typos and several things that could be explained better. In particular:\n   - Please explain the first equality after line 35.\n   - It seems that the martingale runs from l'=l down to l'=0, correct?\n   - It would be good to explain in detail, over which random variables the various expectations E after line 26 run and which random variables are held fixed (conditioned on).\n   - Right-hand-sides of Eq. (S1) and (S2): Should there be t instead of \\hat{t}^{(k)}?\n   - line 31: The word \"outliers\" should be \"inliers\"\n   - Should the inf be changed to sup, or maybe the inequality sign reversed?\n   - line 34: inside the curly brackets, the l should be l'?\n\nLIMITATIONS:\nYes, this is discussed in a fair way after line 351.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}