SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The function  $\omega$  may depend on  $\mathcal{D}_{\mathrm{train}}^{(k)}$  but not on  $\mathcal{D}_{\mathrm{cal}}^{(k)}$  or  $\mathcal{D}_{\mathrm{test}}$ . An example of a useful weighting scheme satisfying this property is at the end of this section. The general method is summarized by Algorithm S3 in the Supplementary Material, which extends Algorithm S1. This produces e-values that control the FDR in conjunction with the eBH filter of Wang and Ramdas (2022).
Theorem 3.6. Suppose Assumption 3.1 holds. Then, the  $e$ -values computed by Algorithm S3 satisfy (6), as long as the adaptive weights obey (7).
An example of a valid weighting function applied in this paper is the following. Imagine having some prior side information suggesting that the proportion of outliers in  $\mathcal{D}_{\mathrm{test}}$  is approximately  $\gamma \in (0,1)$ . Then, a natural choice to measure the quality of the  $k$ -th model is to let  $\tilde{w}^{(k)} = |\tilde{v}^{(k)}|$ , where  $\tilde{v}^{(k)}$  is the standard t-statistic for testing the difference in means between the top  $\lceil n_{\mathrm{test}}\cdot \gamma \rceil$  largest values in  $\tilde{\mathcal{D}}_{\mathrm{cal - test}}^{(k)}$  and the remaining ones. See Algorithm S5 in the Supplementary Material for further details. Intuitively, Algorithm S5 tends to assign larger weights to models achieving stronger out-of-sample separation between inliers and outliers. Of course, this approach may not always be optimal but different weighting schemes could be easily accommodated within our framework.
3.3 Derandomizing AdaDetect with E-AdaDetect
The requirement discussed in Section 3.2 that the data-adaptive weights should be invariant to permutations of the calibration and test samples is analogous to the idea utilized by AdaDetect (Marandon et al., 2022) to train more powerful machine learning models leveraging also the information contained in the test set; see Section 2.3. This implies that Theorem 3.6 remains valid even if our method is implemented based on  $K$  machine learning models each trained by looking also at the unordered union of all data points in  $\mathcal{D}_{\mathrm{cal}}^{(k)} \cup \mathcal{D}_{\mathrm{test}}$ , for each  $k \in [K]$ . See Algorithm S4 in the Supplementary Material for a detailed implementation of this extension of our method, which we call E-AdaDetect.
3.4 Tuning the FDR hyper-parameter
As explained in Section 3.1, our method involves a hyper-parameter  $\alpha_{\mathrm{bh}}$  controlling the BH thresholds  $\hat{t}^{(k)}$  in (4). Intuitively, higher values of  $\alpha_{\mathrm{bh}}$  tend to increase the number of both test and calibration
scores exceeding the rejection threshold at each of the  $K$  iterations. Such competing effects make it generally unclear whether increasing  $\alpha_{\mathrm{bh}}$  leads to larger e-values in (5) and hence higher power. This trade-off was studied by Ren and Barber (2023) while derandomizing the knockoff filter, and they suggested setting  $\alpha_{\mathrm{bh}} < \alpha$ . In this paper, we adopt  $\alpha_{\mathrm{bh}} = \alpha / 10$ , which we have observed to work generally well in our context, although even higher power can sometimes be obtained with different values of  $\alpha_{\mathrm{bh}}$ , especially if the number of outliers in the test set is large. While we leave it to future research to determine whether further improvements are possible, it is worth noting that a straightforward extension of our method, not explicitly implemented in this paper, can be obtained by further averaging e-values obtained with different choices of  $\alpha_{\mathrm{bh}}$ . Such extension does not affect the validity of (6) due to the linearity of expected values.
4.1 Setup and performance metrics
This section compares empirically the performance of AdaDetect and our proposed derandomized method described in Section 3.3, namely E-AdaDetect. Both procedures are deployed using a binary logistic regression classifier (Marandon et al., 2022) as the base predictive model. The reason why we focus on derandomizing AdaDetect instead of traditional split-conformal inferences based on a one-class classifier (Bates et al., 2023) is that we have observed that AdaDetect often achieves higher power on the data considered in this paper, which makes it a more competitive benchmark. However, additional experiments reporting on the performance of our derandomization method applied in combination with one-class classifiers can be found in the Supplementary Sections S4.2 and S6.2.
As the objective of this paper is to powerfully detect outliers while mitigating algorithmic randomness, we assess the performance of each method over  $M = 100$  independent analyses based on the same fixed data and the same test set. For each repetition  $m$  of the novelty detection analysis based on the fixed data, we identify a subset  $\mathcal{R}^{(m)} \subseteq \mathcal{D}_{\mathrm{test}}$  of likely outliers (the rejected null hypotheses) and evaluate the average power and false discovery proportion, namely
$$
\widehat {\text {P o w e r}} = \frac {1}{M} \sum_ {m = 1} ^ {M} \frac {\left| \mathcal {R} ^ {(m)} \cap \mathcal {D} _ {\text {t e s t}} ^ {\text {n o n - n u l l}} \right|}{\left| \mathcal {D} _ {\text {t e s t}} ^ {\text {n o n - n u l l}} \right|}, \quad \widehat {\mathrm {F D R}} = \frac {1}{M} \sum_ {m = 1} ^ {M} \frac {\left| \mathcal {R} ^ {(m)} \cap \mathcal {D} _ {\text {t e s t}} ^ {\text {n u l l}} \right|}{\max  \{| \mathcal {R} ^ {(m)} | , 1 \}}, \tag {8}
$$
where  $\mathcal{D}_{\mathrm{test}}^{\mathrm{non - null}} = \mathcal{D}_{\mathrm{test}}\setminus \mathcal{D}_{\mathrm{test}}^{\mathrm{null}}$  indicates the true outliers in the test set. The average false discovery proportion defined in (8) is not the FDR, which is the quantity we can theoretically guarantee to control. In fact,  $\mathrm{FDR} = \mathbb{E}[\widehat{\mathrm{FDR}} ]$ , with expectation taken with respect all randomness in the data. Nonetheless, we will see that this average false discovery proportion is also controlled in practice within all data sets considered in this paper. The advantage of this setup is that it makes it natural to estimate algorithmic variability by observing the consistency of each rejection across independent analyses. In particular, after defining  $R_{j,m}$  as the indicator of whether the  $j$ -th null hypothesis was rejected in the  $m$ -th analysis, we can evaluate the average variance in the rejection events:
$$
\widehat {\text {V a r i a n c e}} = \frac {1}{n _ {\text {t e s t}}} \sum_ {j = 1} ^ {n _ {\text {t e s t}}} \frac {1}{M - 1} \sum_ {m = 1} ^ {M} \left(R _ {j, m} - \bar {R} _ {j}\right) ^ {2}, \tag {9}
$$
where  $\bar{R}_j = (1 / M)\sum_{m = 1}^{M}R_{j,m}$ . Intuitively, it would be desirable to maximize power while simultaneously minimizing both the average false discovery proportion and the variability. In practice, however, these metrics often compete with one another; hence why we focus on comparing power and variability for methods designed to control the FDR below the target level  $\alpha = 0.1$ .
4.2 Experiments with synthetic data
Synthetic reference and test data consisting of 100-dimensional vectors  $X$  are generated as follows. The reference set contains only inliers, drawn i.i.d. from the standard normal distribution with independent components,  $\mathcal{N}(0, I_{100})$ . Unless specified otherwise, the test set contains  $90\%$  inliers and  $10\%$  outliers, independently sampled from  $\mathcal{N}(\mu, I_{100})$ . The first 5 entries of  $\mu$  are equal to a constant parameter, to which we refer as the signal amplitude, while the remaining 95 entries are zeros. The size of the reference set is  $n = 2000$ , with 1000 samples in the training subset and 1000 in the calibration subset. The size of the test set is  $n_{\mathrm{test}} = 1000$ . Both E-AdaDetect and AdaDetect are applied based on the same logistic regression classifier with default hyper-parameters.
4.2.1 The effect of the signal strength
Figure 2 compares the performance of E-AdaDetect (applied with  $K = 10$ ) to that of AdaDetect, as a function of the signal amplitude. The results confirm both methods control the FDR but ours is less variable, as expected. The comparison becomes more interesting when looking at power: AdaDetect tends to detect more outliers on average if the signal strength is low, but E-AdaDetect can also outperform by that metric if the signals are strong. This may be explained as follows. If the signal strength is high, most true discoveries produced by AdaDetect are relatively stable across different analyses, while false discoveries may be more aleatory, consistently with the illustration of Figure 1. Such situation is ideal for derandomization, which explains why E-AdaDetect is able to simultaneously achieve high power and low false discovery proportion. By contrast, if the signals are weak, the true outlier discoveries produced by AdaDetect are relatively scarce and unpredictable, thus behaving not so differently from the false findings. In this case, one could argue that stability becomes even more important to facilitate the interpretation of any findings, and that may justify some loss in average power.
Figure 2: Performance on synthetic data of the proposed derandomized outlier detection method, E-AdaDetect, applied with  $K = 10$ , compared to that of its randomized benchmark, AdaDetect, as a function of the signal strength. Both methods leverage a logistic regression binary classifier. Left: average proportion of true outliers that are discovered (higher is better). Center: average proportion of false discoveries (lower is better). Right: variability of the findings (lower is better).
4.2.2 The effect of the number of analyses  $K$
Figure 3 investigates the effect of varying the number of analyses  $K$  aggregated by E-AdaDetect. Here, the signal amplitude is fixed to 3.4 (strong signals), while  $K$  is varied between 1 and 30. As expected, the results show that the variability of the findings obtained with E-AdaDetect decreases as  $K$  increases. The average proportion of false discoveries obtained with E-AdaDetect also tends to decrease when  $K$  is large, which can be understood by noting that spurious findings are less likely to be reproduced consistently across multiple independent analyses of the same data. Regarding power, the average number of true outliers detected by E-AdaDetect appears to monotonically increase with  $K$ , although this is not always true in other situations, as shown in the Supplementary Section S4.1. In fact, if the signals are weak, E-AdaDetect may lose some power with larger values of  $K$  (although some  $K > 1$  may be optimal), consistently with the results shown in Figure 2. Thus, we recommend practitioners to utilize larger values of  $K$  in applications where higher power is expected. Finally, note that the power of E-AdaDetect is generally lower compared to that of AdaDetect in the special case of  $K = 1$ , although this is not a practically relevant value of  $K$  because it does not allow any derandomization. The reason why the power of E-AdaDetect is lower when  $K = 1$  is that this method relies on the eBH filter. The latter is relatively conservative as an FDR-controlling strategy because it requires no assumptions about the dependencies of the input statistics.
4.2.3 The effect of the weighting strategy
This section highlights the practical advantage of being able to use data-adaptive model weights within E-AdaDetect. For this purpose, we carry out experiments similar to those of Figure 2, but leveraging a logistic regression model trained with different choices of hyper-parameters in each of the  $K$  analyses. Specifically, we fit a sparse logistic regression model using  $K = 10$  different values of the regularization parameter. To induce higher variability in the predictive rules, one model was trained with a regularization parameter equal to 0.0001, while the others were trained with regularization parameters equal to 1, 10, 50, and 100, respectively. Then, we apply E-AdaDetect using different weighting schemes: constant equal weights ('uniform'), data-driven weights calculated
Figure 3: Performance on synthetic data of E-AdaDetect, as a function of the number  $K$  of derandomized analyses, compared to AdaDetect. Note that the latter can only be applied with a single data split (or iteration). The signal amplitude is 3.4. Other details are as in Figure 2.
with the t-statistic approach ("t-test") summarized by Algorithm S5, and a simple alternative trimmed average data-driven approach ("avg. score") outlined by Algorithm S6 in the Supplementary Material. The results in Figure 4 show that the data-driven aggregation scheme based on t-statistics is the most effective one, often leading to much higher power. We have chosen not to compare E-AdaDetect to the automatic AdaDetect hyper-parameter tuning strategy proposed in Section 4.5 of Marandon et al. (2022) because we found that it does not perform very well in our experiments, possibly due to the relatively low sample size.
Figure 4: Performance on synthetic data of E-AdaDetect applied with different model weighting schemes, as a function of the signal strength. The model weights are designed to prioritize the results of randomized analyses based on models that are more effective at separating inliers from outliers. The t-test approach tends to lead to higher power. Other details are as in Figure 2.
To further demonstrate the effectiveness of data-driven weighting, we turn to analyze the performance of E-AdaDetect on four real-world outlier detection data sets: musk, shuttle, KDDCup99, and credit card. We refer to Supplementary Section S6 for more information regarding these data. Similar to Figure 4, our E-AdaDetect method is applied  $K = 10$  times to each data set, each time leveraging a different predictive model as follows. Half of the models are random forests implemented with varying max-depth hyper-parameters (10, 12, 20, 30, and 7), while the other half are support vector machines with an RBF kernel with varying width hyper-parameters (0.1, 0.001, 0.5, 0.2, and 0.03). This setup is interesting because different models often tend to perform differently in practice, and it is usually unclear a-priori which combination of model and hyper-parameters is optimal for a given data set. Figure 5 summarizes the results, demonstrating that both data-driven weighting schemes ("t-test" and "avg. score") lead to more numerous discoveries compared to the "uniform" weighting baseline, and that the "t-test" approach is the most powerful weighting scheme here. These results are in line with the synthetic experiment presented in Figure 4. Lastly, the variance metrics reported in Figure 5 also suggest that data-driven weighting further enhances the algorithmic stability.
4.3 Additional results from experiments with synthetic and real data
Sections S4-S6 in the Supplementary Material present the results of several additional experiments. In particular, Section S4 focuses on experiments with synthetic data. Section S5 describes comparisons with alternative derandomization approaches based on different types of p-to-e calibrators (Vovk and Wang, 2021) operating one test point at a time, which turn out to yield lower power compared to our martingale-based method. The results also show that our method compares favorably to an alternative derandomization method based on an e-value construction that first appeared in Ignatiadis et al. (2023), especially if the test set contains numerous outliers or if the nominal FDR level is not
Figure 5: Performance of E-AdaDetect, applied with different model weighting schemes, on four real data sets. All methods utilize a combination of random forest and support vector machine models. The "t-test" weighting approach leads to the highest power and lowest algorithmic variability, all while controlling the FDR below the nominal  $10\%$  level.
too low. Finally, Section S6 describes additional numerical experiments based on several real data sets also studied in Bates et al. (2023) and Marandon et al. (2022). These results confirm that our martingale-based e-value method can mitigate the algorithmic randomness of standard conformal inferences and AdaDetect while retaining relatively high power.
Our experience suggests that e-values are often less powerful than p-values in measuring the statistical evidence against a single hypothesis. Yet, e-values can be useful to aggregate multiple dependent tests of the same hypothesis (Vovk and Wang, 2021)—a task that would otherwise require very conservative adjustments within the p-value framework (Vovk et al., 2022). Further, we have shown that e-values lend themselves well to multiple testing because they allow efficient FDR control under arbitrary dependence (Wang and Ramdas, 2022), and even their relatively weak individual evidence can accumulate rapidly when a large number of hypotheses is probed. The opportunity arising from the combination of these two key properties was recently leveraged to derandomize knockoffs (Ren and Barber, 2023), but until now it had not been fully exploited in the context of conformal inference.
While this paper has focused on derandomizing split-conformal and AdaDetect inferences for novelty detection, the key ideas could be easily extended. For example, one may utilize e-values to derandomize conformal prediction intervals in regression (Lei and Wasserman, 2014; Romano et al., 2019) or prediction sets for classification (Lei et al., 2013; Romano et al., 2020) while controlling the false coverage rate over a large test set (Weinstein and Ramdas, 2020). A different direction for future research may explore the derandomization of cross-validation+ (Barber et al., 2021).
We conclude by discussing two limitations of this work. First, the proposed method is more computationally expensive compared to standard conformal inference or AdaDetect, and therefore one may be limited to applying it with relatively small numbers  $K$  of analysis repetitions when working with very large data sets. That being said, it is increasingly recognized that stability is an important goal in data science (Murdoch et al., 2019), and thus mitigating algorithmic randomness may often justify the deployment of additional computing resources. Second, we have shown that our method sometimes leads to a reduction in the average number of findings compared to randomized alternatives, especially in applications where few discoveries are expected. Therefore, in situations with few anticipated discoveries, practitioners considering applying our method should carefully weigh the anticipated gains in stability versus a possible reduction in power.
Mathematical proofs and additional supporting results are in the Supplementary Material. Software implementing the algorithms described in this paper and enabling the reproduction of the associated numerical experiments is available at https://github.com/Meshiba/derandomized-novelty-detection.
Y. R. and M. B. were supported by the Israel Science Foundation (grant No. 729/21). Y. R. thanks the Career Advancement Fellowship, Technion, for providing research support. Y. R. also thanks Citi Bank for the generous financial support. M. S. was partially supported by NSF grant DMS 2210637 and by an Amazon Research Award.
M. Ahmed, A. N. Mahmood, and M. R. Islam. A survey of anomaly detection techniques in financial domain. Future Generation Computer Systems, 55:278-288, 2016.
R. F. Barber and E. Candès. Controlling the false discovery rate via knockoffs. Ann. Stat., 43(5): 2055-2085, 2015.
R. F. Barber, E. Candès, A. Ramdas, R. J. Tibshirani, et al. Predictive inference with the jackknife+. Ann. Stat., 49(1):486-507, 2021.
S. Bates, E. Candès, L. Lei, Y. Romano, and M. Sesia. Testing for outliers with conformal p-values. Ann. Stat., 51(1):149 - 178, 2023.
D. Beganovic and E. Smirnov. Ensemble cross-conformal prediction. In 2018 IEEE International Conference on Data Mining Workshops, pages 870-877. IEEE, 2018.
Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B, 57(1):289-300, 1995.
Y. Benjamini and D. Yekutieli. The control of the false discovery rate in multiple testing under dependency. Ann. Stat., pages 1165-1188, 2001.
V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1-58, 2009.
W. Fithian and L. Lei. Conditional calibration for false discovery rate control under dependence. Ann. Stat., 50(6):3091-3118, 2022.
C. R. Genovese, K. Roeder, and L. Wasserman. False discovery control with p-value weighting. Biometrika, 93(3):509-524, 2006.
L. Guan and R. Tibshirani. Prediction and outlier detection in classification problems. J. R. Stat. Soc. Series B, 84(2):524, 2022.
C. Gupta, A. K. Kuchibhotla, and A. Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022.
N. Ignatiadis, R. Wang, and A. Ramdas. E-values as unnormalized weights in multiple testing. arXiv preprint arXiv:2204.12447, 2023. version 4.
B. Kim, C. Xu, and R. Barber. Predictive inference is free with the jackknife+-after-bootstrap. Adv. Neural Inf. Process. Syst., 33:4138-4149, 2020.
R. Laxhammar and G. Falkman. Inductive conformal anomaly detection for sequential detection of anomalous sub-trajectories. Annals of Mathematics and Artificial Intelligence, 74(1-2):67-94, 2015.
J. Lei and L. Wasserman. Distribution-free prediction bands for non-parametric regression. J. R. Stat. Soc. Series B, 76(1):71-96, 2014.
J. Lei, J. Robins, and L. Wasserman. Distribution-free prediction sets. J. Am. Stat. Assoc., 108(501): 278-287, 2013.
Z. Liang, M. Sesia, and W. Sun. Integrative conformal p-values for powerful out-of-distribution testing with labeled outliers. arXiv preprint arXiv:2208.11111, 2022.
H. Linusson, U. Johansson, and H. Boström. Efficient conformal predictor ensembles. Neurocomputing, 397:266-278, 2020.
T. Lofstrom, U. Johansson, and H. Boström. Effective utilization of data in inductive conformal prediction. In Proc. Int. Jt. Conf. Neural Netw. IEEE, 2013.
A. Marandon, L. Lei, D. Mary, and E. Roquain. Machine learning meets false discovery rate. arXiv preprint arXiv:2208.06685, 2022.