Robust Conformal Outlier Detection under Contaminated Reference Data
Authors: Meshi Bashari
Matteo Sesia
Yaniv Romano
Venue: ICML 2025 poster (2025)
Keywords: Conformal Prediction
Hypothesis Testing
Out-of-Distribution Detection
Contaminated Data
Primary area: probabilistic_methods
TL;DR: Powerful conformal outlier detection framework under contaminated data, utilizing a limited annotation budget.

Abstract: Conformal prediction is a flexible framework for calibrating machine learning predictions, providing distribution-free statistical guarantees. In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate. However, obtaining a perfectly labeled inlier reference set is often unrealistic, and a more practical scenario involves access to a contaminated reference set containing a small fraction of outliers. This paper analyzes the impact of such contamination on the validity of conformal methods. We prove that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error control, shedding light on the inherent robustness of conformal methods. This conservativeness, however, typically results in a loss of power. To alleviate this limitation, we propose a novel, active data-cleaning framework that leverages a limited labeling budget and an outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers. By removing only the annotated outliers in this ``suspicious'' subset, we can effectively enhance power while mitigating the risk of inflating the type-I error rate, as supported by our theoretical analysis. Experiments on real datasets validate the conservative behavior of conformal methods under contamination and show that the proposed data-cleaning strategy improves power without sacrificing validity.

Robust Conformal Outlier Detection under Contaminated Reference Data
Meshi Bashari Matteo Sesia 23 Yaniv Romano
Abstract
1. Introduction
1.1. Background and Motivation
Conformal prediction is a flexible framework for calibrating machine learning predictions, provid- ing distribution-free statistical guarantees. In out- lier detection, this calibration relies on reference set of labeled inlier data to control the type-I er- ror rate. However, obtaining a perfectly labeled inlier reference set is often unrealistic, and more practical scenario involves access to a contami- nated reference set containing a small fraction of outliers. This paper analyzes the impact of such contamination on the validity of conformal meth- ods. We prove that under realistic, non-adversarial settings, calibration contaminated data yields conservative type-I error control, shedding light on the inherent robustness of conformal methods. This conservativeness, however, typically results in a loss of power. To alleviate this limitation, we propose a novel, active data-cleaning frame- work that leverages a limited labeling budget and outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers. By removing only the annotated outliers in this "suspicious" subset, we can effectively enhance power while mitigating the risk of inflating the type-I error rate, as sup- ported by our theoretical analysis. Experiments real datasets validate the conservative behav- ior of conformal methods under contamination and show that the proposed data-cleaning strategy improves power without sacrificing validity.
This paper studies the problem of outlier detection: given reference dataset (e.g., a collection of legitimate financial transactions) and an unlabeled test point (a new transaction), our goal is to determine whether the test point is an outlier (a fraudulent transaction) by assessing its deviation from the reference data distribution. Naturally, we aim to maximize the detection of outliers by harnessing the capabilities of complex machine learning (ML) models. However, these models typically lack type-I error rate control, potentially re- sulting in unreliable detections. In our running example, the type-I error is the probability of falsely flagging legitimate transaction as fraudulent. As such, uncontrolled error rates can lead to costly unnecessary investigations of legitimate transactions and negatively impact customer experience.
The broad need for reliable ML systems has sparked surge of interest in conformal prediction-a versatile framework that can provide statistical guarantees for any "black-box" predictive model (Vovk et al., 2005). This framework for- mulates the outlier detection task as a statistical test, where the null hypothesis that the new data point is not an out- lier (Laxhammar & Falkman, 2015; Bates et al., 2023). To derive a decision rule guaranteeing type-I error control, con- formal inference relies on a reference (calibration) set of inlier data points. These points are assumed to be sampled i.i.d. from an unknown distribution, independent of the data used to train the outlier detection model.
In practice, however, it often difficult to obtaín a perfectly clean reference dataset that contains outliers (Park et al., 2021; Zhao et al., 2019; Chalapathy & Chawla, 2019; Jiang et al., 2022). In our example, more realistic scenario would assume instead access to a slightly contaminated reference set, mostly legitimate transactions with a few unnoticed outliers (Zhao et al., 2019). But this setting poses new challenges for conformal prediction methods, potentially invalidating the error control guarantees or, shall see, often reducing the power to detect true outliers at test time.
'Department of Electrical and Computer Engineering, Technion IT, Haifa, Israel 'Department of Data Sciences and Operations, University of Southern California, Los Angeles, California, USA 'Department of Computer Science, University of Southern Cali- fornia, Los Angeles. California, USA 'Department of Computer Science, Technion IIT, Haifa, Israel. Correspondence to: Meshi Bashari <meshi. b @ campus. technion.ac.ll>.
1.2. Outline and Contributions
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
While type-I emror control in conformal inference theoret- ically requires perfectly clean reference data, in practice

---

Robust Conformal Outlier Detection under Contaminated Reference Data
contaminated data often makes these methods overly conser- vative, reducing the power to detect true outliers rather than inflating the type-I error rate. This empirical observation motivates the first question explored in this paper:
at http://ggiiii..coommmmiiooooooos
1.3. Related Work
Recently, there has been growing interest in studying the statistical properties of conformal inference methods under more realistic scenarios, moving beyond the idealized as- sumption of perfectly clean and exchangeable observations to account for various types of distribution shift (Tibshirani et al., 2019; Einbinder et al., 2024; Sesia et al., 2024; Barber ct al., 2023; Gibbs & Candès, 2021; Zaffran et al., 2022; Feldman et al., 2023; Gibbs & Candès, 2024; Podkopaev & Ramdas, 2021; Si al., 2023; Prinster et al.). This paper draws inspiration from several prior works in this area.
Q1: When does conformal outlier detection with contami- nated reference data yield valid type-l error control?
In Section 2.3, we present the first contribution of this pa- per: novel theoretical analysis that identifies common conditions under which this conservative behavior arises. Unfortunately, this conservativeness often comes at the cost of reduced detection power, particularly when targeting low type-I error rates. To address this issue, we investi- gate data-driven cleaning strategies aimed at mitigating the contamination in the reference dataset.
Tibshirani et al. (2019) introduced a weighted conformal pre- diction approach to address covariate shift between calibra- tion and test data, later extended by Podkopaev & Ramdas (2021) to accommodate label shift. Both settings, however, involve a different form of distribution shift from the one we study here. Barber et al. (2023) extend this line of work by analyzing the effects of general distribution shifts on the validity of conformal methods, focusing however on worst-case scenarios; see also Farinhas et al. (2024).
A straightforward approach to cleaning the contaminated set is to remove all data points flagged as likely outliers by the detection model. However, this method is unsatisfactory, as it risks inadvertently removing inliers along with outliers, resulting in an "overly clean" reference set. This, in turn, distorts the inlier distribution and inflates the type-I error rate above the desired nominal level.
This challenge motivates our second and main contribu- tion. In Section 3.3, we introduce an approach to clean the contaminated reference set by leveraging limited labeling budget (e.g.. 50 annotations). The outlier detection model is first used to identify suspected outliers within the contami- nated reference set. The limited budget then strategically allocated to annotate these points, thereby avoiding the un- intended removal of inliers. While this is practical and intuitive approach, it naturally prompts a critical question:
In contrast, our work moves away from this worst-case perspective. We aim to explain why conformal outlier de- tection with contaminated reference data often results in conservative type-I error rate, rather than investigating type-I error inflation, which, while theoretically possible in adversarial settings, appears less common in practice. Fur- thermore, we focus on developing methods to address this over-conservativeness. boosting detection power.
A more closely related line of work investigates the robust- ness of conformal prediction to label noise (Einbinder et al., 2024; Sesia et al., 2024; Clarkson al., 2024; Penso et al., 2025) or other forms of data contamination (Zaffran et al., 2023; 2024; Feldman & Romano, 2024). Specifically, Ein- binder et al. (2024) and Sesia et al. (2024) show that, under certain assumptions, conformal prediction for classification with noisy labels often results in conservative type-I error rates. Furthermore, Sesia et al. (2024) proposes a method to address this conservativeness by leveraging an explicit "label noise model" that captures the relationship between the true and contaminated labels in the calibration dataset.
Q2: How does the selective annotation and partial removal of outliers from a contaminated reference set affect the va- lidity of conformal inferences?
We analyze the validity of this active labeling approach for trimming outliers in the contaminated set. Our theoretical results identify the conditions required to achieve approxi- mate type-I error control, even when the data are selectively annotated and not all outliers are removed. This analysis also highlights key factors that can inflate the error rate, of- fering practitioners guiding principles to enhance the power of conformal methods the presence of contaminated data.
Finally, in Section 4, we empirically validate our theory and proposed data-cleaning approach through comprehensive experiments on real-world datasets. The experiments con- firm that conformal inference with contaminated data tends to be conservative. Furthermore, they demonstrate that our method significantly boosts power, particularly when the target I error rate is low and the number of outliers in the contaminated set is small.
In contrast, this paper avoids relying on an explicit model for the contaminated data, as such models can be difficult to estimate in practice within our context. Instead, we utilize a pre-trained black-box outlier detection model and a limited annotation budget to selectively and reliably trim outliers from the contaminated set. Furthermore, it is important to emphasize that the method proposed by Sesia et al. (2024) is primarily designed for classification tasks with relatively balanced data, whereas outlier detection naturally involves
A software that implements the proposed method is available
2

---

Robust Conformal Outlier Detection under Contaminated Reference Data
extreme class imbalance. This distinction underscores the need for solutions specifically tailored to outlier detection.
Proposition 2.1 intuitively states that the conformal p-value defined in (2) provides well-calibrated rule for flagging new data point as a likely outlier. Rejecting Ho when Pn+1 I a ensures type-I eπror control at level a while avoiding excessive conservatism. Specifically, the type-I error rate closely matches a when the sample size n is large and the nonconformity scores have a continuous distribu- tion with no ties a mild condition that can be achieved in practice by adding small random noise to the scores.
2. Setup and Preliminary Results
2.1. Inference with Clean Calibration Data
Conformal inference for outlier detection requires a refer- ence (or calibration) set, Deal= = [n] := {l..... n), con- taining n data points. The reference set typically assumed to be clean, consisting solely of inliers, which are i.i.d. sam- ples from an unknown distribution 1.o (exchangeability may sometimes suffice, but this work assumes i.i.d. inliers). Un- der this assumption, Deal may be referred to as Dialier.
What remains unclear, and serves as the starting point of this paper, is how conformal p-values behave when the cali- bration dataset is contaminated, containing not only inliers but also a fraction of misplaced outliers.
The goal is to determine whether a new observation, X an indee--independeetll sampled from Po-or an oullier, sampled from a different distribution P1 * Fo. This can be formulated as hypothesis testing problem, where the null hypothesis Ho claims that Xn+1 is an inlier:
2.2. Inference with Contaminated Calibration Data
In this paper, we consider a more general setting where the calibration dataset, indexed by Deal = [ra] may contain both inliers (Dinlier), sampled i.i.d. from a distribution Po- and outliers (Doutlier) sampled i.i.d. from a different dis- tribution P1 * Po- Thus, Deat Dintier U U Dottlerr- The numbers of inliers and outliers, respectively no = |Dinlier and n. = (outtlier |. are treated as fixed, with n = nn. The goal remains to test the null hypothesis Ho that a new data point Xn+1 is an inlier, independently sampled from Po- Formally, this setup can be written as:
X; lid. Po. Vi E Dintiers ~ ind.
Dinlier = Deal = [m],
Ho: X,+1 ~ Po-
The split-conformal method, a simple and computation- ally efficient approach, uses pre-trained outlier detection any machine learning model-to to com- pute nonconformity scores that quantify how different data point is from the reference distribution. The model, repre- sented by a score function 8, trained on a separate dataset Dursius which similar to but independent of Deal- Typi- cally, larger dataset of inliers, assumed to be i.i.d. samples from Fo. is randomly split into Derain and Deal-
X; lid Po, Vi E Dinlier:
X, 1 P1, Vi E Doutliers
Dinlier UDoutlier = Deal
[n],
Ho Xx+1 ~ Po.
In the following, we first analyze the behavior of standard conformal p-values, computed as in (2), when applied to contaminated data scenarios described by (3). Subsequently, we will propose a novel method for computing more re- fined conformal p-values by approximately cleaning the calibration set to remove undesired outliers.
The model tries to learn a score function such that larger values of s 11)+1) indicate stronger evidence that the test point may be an outlier. Conformal inference rigorously quantifies this evidence, providing a principled decision rule for rejecting Hi when the evidence is strong enough, while controlling the type-I error rate-the probability of incorrectly rejecting Ho when X.+1 i is actually an inlier.
2.3. Explaining the Conservativeness
This statistical evidence is quantified by computing a con- formal p-value, defined as:
Empirical results suggest that contamination by outliers in the calibration data often makes standard conformal p- values overly conservative, resulting in a type-I error rate significantly lower than the desired nominal level a.
PA+1 1+ Ei [[x(X,) 2 s(Xn+1)
1
We begin by examining Figure 1, which provides some in- sight into this behavior based on the analysis of the "shuttle" dataset (shu), as detailed in Section In this example, the nonconformity scores of outlier data points in the contam- inated calibration set are typically larger than those of the inliers. This pattern reflects the goal of a well-designed out- lier detection model: to differentiate outliers from inliers by assigning higher scores to the former. Consequently, confor- mal p-values computed using (2) will be inflated relative to
Thus, larger values of 1(),-1) correspond to smaller val- ues of Pa+1, and the test point Xn+1 can be confidently classified as an outlier (rejecting Ho) when fo+t is smaller than given significance level a E (0, 1).
Proposition 2.1 (from Vovk et al. (2005)). Under (1), if the null hypothesis Ho is true, then for any a E (0, 1): P (Pu+1 S a) < a. Further, if s(X) has a continuous distribution under Po, then P (fat1 S a) 2 a011(nn+) 1))
3

---

Robust Conformal Outlier Detection under Contaminated Reference Data
the ideal scenario in which all calibration points are inliers, reducing our power detect true outliers test time.
for this paper, we also include an additional result (Corol- lary A.3) in Appendix A, which reaches qualitatively similar conclusions by adopting approach more closely aligned with Theorem from Sesia et al. (2024).
Inliers
600
A direct corollary of Lemma 2.2 is that standard conformal p-values are conservative when outlier scores are typically larger than inlier scores. Formally, this condition is:
Outliers
400
Assumption 2.3. E (Q1
200
a.
If Assumption 2.3 fails for instance, when outlier scores are smaller than inlier scores-date contamination may in- validate standard conformal p-values, inflating the type-I error rate. However, it is more common in practice that Assumption 2.3 holds, in which case contamination tends to reduce calibration power, and more so if ni is large. In particular, Assumption 2.3 holds if: (i) the outlier detection model is relatively accurate, and (ii) the outlier distribution P1 is not adversarial. Our experiments will show this power loss can be substantial, motivating the need for new methods that can approximately "clean up" the calibration data.
-0.1
0.0
0.1
0.2
Scores
Figure I: Histogram of nonconformity scores for inliers and outliers in a contaminated calibration subset of the "shuttle" data, with a contamination rate of 5%. The vertical lines indicate the (1 - at) empirical quantile of all calibration scores (black), as well as separately for inliers (blue) and outliers (red), with 0.02.
This phenomenon is further corroborated by extensive numerical experiments presented in Section 4 and Ap- pendix B.2, which consistently demonstrate this conser- vative behavior across nine different datasets.
3. Methods
3.1. Key Idea: Boosting Power by Cleaning the Data
While the conservativeness of conformal prediction meth- ods in the presence of contaminated calibration data has already been observed and studied theoretically in different contexts (Einbinder et al., 2024; Sesia et al., 2024; Clarkson et al., 2024), prior works did not focus on outlier detection. Therefore, is helpful to introduce a new theoretical result that precisely quantifies the inflation of standard conformal p-values that we often observe in practice. This will serve as foundation for the novel method of computing adaptive conformal p-values presented in the next section.
Ideally, we would like to remove all nı outliers from Deal. restoring the ideal behavior of conformal p-values calibrated on a clean dataset, as described in Proposition 2.1, and likely boosting power. However, manually labeling the entire contaminated calibration set Deat is often impractical, especially when n = |Deat| is large. At the same time, utilizing only small calibration set is not always desirable.
A large calibration set often needed because the small- est conformal p-value obtainable through (2) scales as 1/n. Thus, a large is critical for achieving high confidence in identifying outliers, especially in sreedle-in-a-haystaee scenarios (Bates et al., 2023), where a few outliers must be detected in a large test set dominated by inliers. In such cases, the ability to obtain very small p-values is essen- tial to achieve non-trivial power while controlling the false discovery rate (Benjamini & Hochberg, 1995).
Let F denote the empirical cumulative distribution function (CDF) of the scores in Doutlier and Quail represent the [(1-a)(1 + n)]]-th smallest score in the calibration set. Lemma 2.2. Under the setup defined in (3), Ha is true, then for any E 10,
P (Pn+1 a) < a a- n2
140 +1 (1 (1-a- a1-a-B(Ae [ (Q
3.2. A Simple but Unsatisfactory Approach: Naive-Trine
The above challenge underscores the need for a method to mitigate the impact of outliers in the calibration dataset with- out requiring exhaustive annotation. An intuitive approach is to forgo annotations and simply remove all "suspicious" data points with large nonconformity scores. For instance, one could remove the top scores from Deat, where m is a fixed guess of the true number of outliers ny in the calibration set. We refer to this approach as Naive-Trim.
This result related to Theorem 1 in Sesia et al. (2024), which studies the behavior of conformal prediction sets for multi-class classification (Lei al., 2013; Romano et al., 2020) calibrated with contaminated data. The key distinc- tion is in our treatment of the calibration set: we assume that no and ni are fixed, whereas Sesia et al. (2024) con- sider a mixture model where the observed proportions of data points from different classes in the calibration set are random. While treating no and ny as fixed is convenient
While teive-Tree can reduce conservativeness by re-

---

Robust Conformal Outlier Detection under Contaminated Reference Data
moving some outliers, it is not a satisfactory solution as it risks "over-compensattng". By potentially removing also true inliers with large nonconformity scores, it can signif- icantly skew the inlier score distribution to the left. This side effect is problematic, as it tends to invalidate conformal p-values and inflate the type-I error rate, over-correcting the conservativeness of standard conformal p-values.
labeling budget m
Sorted scores
Outlier detection model
Dail
Figure 2: A schematic illustration of the proposed active data-cleaning of the contaminated reference set (Algo- rithm 1). The approach begins by computing nonconfor- mity scores of the contaminated reference data Deal using a pretrained outlier detection model, where blue circles denote inliers and red circles denote outliers. The scores are then sorted in increasing order, and the top m samples-those most likely to be outliers-arr are selected for annotation. Af- ter removing the annotated outliers, the resulting (partially) cleaned set DLu is used for calibration.
This issue is particularly pronounced when m > ni or in noisy settings where the outlier detection model cannot perfectly distinguish between inliers and outliers. For ex- ample, as shown in Section 4, applying Naive Serive-Traae to the dataset illustrated in Figure I results in uncontrolled inflation of the type-I error error rate.
To address this challenge, we will now present more so- phisticated method, which we refer to as Label 1eel--riin im. This approach utilizes limited labeling budget to remove outliers from Deal in more reliable manner, mitigating the risk of over-correcting the conformal p-value.
Algorithm Label-trim calibration (construction phase)
1: Input: labeling budget m, contaminate calibration
Deal = {X:}._. score function s(-), obtained by a pre-trained outlier detection model;
3.3. The Label -Tr im Method
Consider having limited budget to label m < n calibration samples, where m is much smaller than n. We aim to utilize this budget to remove as many outliers as possible from the calibration set without altering the inlier score distribution.
2: Compute the calibration scores S; = sX,). Vi Deat-
3: Sort the calibration scores, such that S1 S
Sa(n) where и : [n] + [n] is the corresponding permu- tation of the indices.
4: Annotate the m largest scores Diabeled :
A practical approach to annotate the m largest scores in Deat, these are most likely outliers based on the model. Denote these annotated samples as Diabeled S Deal. and let Diutlier denote the subset of annotated data points that are true outliers. Removing these outliers from Deat yields a smaller, cleaner calibration set, which we call D S Deaal
> n - m). with Y=()) = if X=()) an inlier and Y=()) = 1 otherwise.
{(Sa()) Y=(o)
5: Construct the trimmed calibration set cal
{T(() : S n - m] U {5: j E D Diabeled and Y; = 0}.
6: Output: trimmed calibration set DLT
The Label-Trin method then calculates a refined con- formal p-value, now denoted as phtt1. following the same procedure as in (2) with Deal replaced by the (partially) cleaned calibration set DIT:
Algorithm 2 Label-trim calibration (testing phase)
1: Input: test point X,, X, score function s; trimmed calibration set DIT type-I error level a;
1 1+ EiEniy [s(X,)) 2 s(xx+1)
Compute the conformal p-value /.T according (4).
Px+1
1+ D
(4)
3: Output: reject the null hypothesis Ho if pht S a, classifying Xn+1 as an outlier.
Algorithms 1 and 2 summarize this procedure, which in- tuitively offers advantages over both the standard method for computing p in (2), by potentially increasing power, and the Naive-Trim approach. by mitigating the risk of over-correcting pn+1- A schematic illustration of the con- struction of D by Algorithm is shown in Figure 2.
revealing the true labels of some calibration points but not others, which could disrupt the exchangcability typically assumed among inlier data points in conformal inference. Therefore, this justification requires novel proof techniques and does not follow directly from existing results.
The following theorem provides justification for Label-Trim, demonstrating that PLT is an ap- proximately valid p-value under relatively mild conditions. While our method is intuitive, this result is nontrivial for two reasons. First, Label-Trimg cannot guarantee the removal of all outliers from the calibration set, as it may be that m < ni or some outliers are not among the m largest scores. Second, involves annotating the m largest scores,
Following notation similar to that of Lemma 2.2, let ATT denote the empirical CDF of the scores in DL.T Putlier Define also QIT as the itr smallest element in {S,) (S.)igpun
{oc), with å == [(1 (n + 1)] and n |Ptal
Theorem 3.1. Consider the setup in (3), with Ho being true.
5

---

Robust Conformal Outlier Detection under Contaminated Reference Data
For any fixed a E (0, 1), assume that m a(nn + 1). Then,
calibration Peat- and test sets of inliers Disler and outliers Doutier To simulate a realistic setting, we construct the training and contaminated calibration sets with the same contamination rate of r%. The inlier Dialier and outlier test test sets are used to compute the type-I error and power of the outlier detection model, respectively. To ensure fair comparisons, all conformal methods use the same outlier detection model, trained on Drain. Performance metrics are evaluated across 100 random splits of the data. The size of each dataset, along with the details of how Dimin. Deal, and Diest are constructed are provided in Appendix B.1.
P (pot
I a) S a +
70 +1
- E
no + (11
LT
a1
FTT (PHa))
The upper bound on the type-I error rate provided by Theo- rem 3.1 resembles that of Lemma 2.2 and can be interpreted as follows: Label-Tr's -rrm produces approximately valid conformal p-values if: (i) the labeling budget is small rela- tive to the calibration set size, i.e., S a + 1); and (ii) the calibration set contains large number of inliers, no.
4.1. Tabular Data
However, the upper bound in Theorem 3.1 also suggests that Label-Trinn may remain overly conservative. sim- ilar to standard conformal p-values, if (i) not all outliers are removed from the calibration set (n 0 with high probability), and (ii) the remaining outlier scores are gener- ally larger than the remaining inlier scores, consistent with E[FIT (QIT,)] < 1 - a, akin to Assumption 2.3.
We now compare the performance of the different methods on three benchmark tabular datasets for outlier detection, previously used in the conformal literature (Bates et al., 2023). Since conclusions are similar across datasets, we focus here on results for the shuttle dataset (shu). Results for the credit card (cre) and KDDCup99 (KDD) datasets are presented in Appendix B.2.1. For all conformal methods, Isolation Forest (Liu et al., 2008) as the base outlier detection model, implemented using s sci k kitt-ti-- teltt-llaare with default hyperparameters (Buitinck et al., 2013).
This potential conservative behavior arises naturally from the use of a limited labeling budget, especially when the model guiding the construction of the set Diabeled fails to effectively detect true outliers. Nevertheless, we will see in the next section, Label -Trim often enhances power.
Figure 3 presents the performance metrics of each method as a function of the contamination rate r. Following the left panel in that figure, we can see that the Standard con- formal method results in conservative type-I error control, with decrease in the enror rate the outlier proportion increases a behavior that is aligned with Lemma 2.2. No- tably, the type-I error of the Oracle method is tightly centered around a, as guaranteed by Proposition 2.1. The Naive-Trim method does not control the type-I error rate, emphasizing the need for reliable data-cleaning procedures. In striking contrast, our Labe 1-rim method achieves a valid type-I error rate. At lower outlier proportions, the empirical type-I error is close to but the method becomes more conservative as the outlier proportion increases. This observation aligns with the upper bound on the error rate derived in Theorem 1. Notably, as the contamination rate in the training data increases, the outlier detection model's ability to distinguish between inliers and outliers weakens. This, in turn, adversely affects the effectiveness of forming subset of data points for annotation, as demonstrated in the right panel of Figure 3. The Small-Cleat method also controls the type-I error but is more conservative than Label - Trim due to its much smaller reference set, which becomes even smaller as the contamination rate increases. Observe how the power of the Small-CIe method is lower than that of the andard approach, despite the latter using a contaminated reference set. By contrast, our proposed Label- - --r--iim trim method significantly improves the power of the St Standard method and even achieves near- oracle performance when the outlier proportion is low.
4. Experiments
We turn to evaluate the performance of conformal outlier de- tection methods under contaminated data. The experiments presented in this section are conducted on nine benchmark datasets: three tabular datasets, listed in Section 4.1, and six visual datasets, listed in Section 4.2.
Methods
We compare the following methods:
The basic conformal method that uses the contaminated reference set Deal = Pinlier U Doutteer- Oracle: An infeasible benchmark method where the reference contains only inliers, i.e., Deat = Dintier- Naive-Trim: The baseline method from Section 3.2, which removes the top 5% non-conformity scores from Deat where r = 72/(no + n1).
Standard:
1abel-Trin Our proposed reliable data-cleaning method from Section 3.3, applied with a labeling bud- get of m 50 annotations to label the m data points with the largest non-conformity scores from Deal- Small-clean: A baseline method that uses the la- beling budget to construct a small, clean reference set by (i) randomly selecting m data points from Deal and (ii) extracting the true inliers from this subset.
Setup and performance metrics In all experiments, we randomly split a given dataset into disjoint training D Dirann
6

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.04
0.6
125
Standard
Type- L-Error
100
Oracle (infeasible)
0.03
Power
0s
44
Trimmed
75
Naive-Trim (invalid)
0.02
0.3
50
Small -lean
Label-Trim
0.01
0.2
Target type-I level
Labeling budget
0.90
0.1
0.0
0.01
0.02
0.03
0.04
0.05
0.0
0.01
0,02
0.03
0.04
0,55
0.0
0.01
0.02
0.03
0.04
0.05
outliers
Contamination rate
Contamination rate
Contamination rate
Figure 3: Comparison of conformal outlier detection methods on a tabular dataset ("shuttle") as function of the contamina-
tion rate r. The target type-I eπror rate is a = 0.02. Left: Empirical type-I error. Middle: Average detection rate (power), where higher values indicate better performance. Right: Number of outliers trimmed by the Label -rrmm method. Results are averaged across 100 random splits of the data.
0.025
100
Oracle (infeasible)
Type- Error
0,020
Small-Clean
Trimmed
80
0.015
Power
Label-Trim
60
Target type-I level
0.010
0.2
Labeling budget
0.005
# outliers
70
0.00n
100
18
3
80
g0
100
10
20
30
40
50
69
80 D0)
Ing
Labeling budget
Labeling budget
Labeling budget
Figure 4: Comparison of conformal outlier detection methods on a tabular dataset ("shuttle") as a function of the labeling
budget m. The contamination rate fixed to 0.03, Other details are as in Figure
Next, we study the effect of the labeling budget on the perfor- mance of our Label-Trim method. As shown in Figure 4, increasing the labeling budget brings the Label-T iim method closer to the Oracle e in terms of both type-I error and power. Notably, even with modest budget of 40-50 annotations, the power Label-Trim is nearly indis- tinguishable from that of the Cracle. This attributed to the method's effective trimming of outliers, as shown in the right panel. Notably, for labeling budgets > 50, the condition in Theorem 3.1 no longer holds, and yet the label-Tring method still achieves valid type-I error con- trol at level a in practice. This highlights the robustness of the proposed method to the choice of m beyond the restrictions specified in Theorem 3.1, where we attribute this robustness to the adversrraai nature of the outlier distribution and the underlying detection model.
sition 2.1. For instance, with m 80 and a 0.02, the lower bound is approximately which aligns closely with the empirical error rate shown in the left panel of Figure 4. Overall, these results highlight the benefits of selectively cleaning a relatively large contam- inated set compared to relying on a small clean reference set, offering both improved stability and higher power.
1/(m 1) 0.0076,
Next, we examine how the target error level a affects the performance of different methods. Figure 5 shows that our Label-Trim method performs particularly well at low type-I error rates, especially when a is smaller than the contamination rate (r 3%). This behavior can be explained as follows. For a relatively accurate model, the outliers primarily distort the tail of the empirical distribution of nonconformity scores-see Figure 1. Consequently, the influence of these outliers the rejection rule p S from (2), p.t from (4), diminishes as increases.
Figure 4 also illustrates that the Small 1alll-Clean method lags behind Label-Trin im both in terms of power and con- servativeness. For small labeling budgets of m < 45, the coarse granularity of conformal p-values (2) renders the method powerless; the smallest achievable p-value in this case is 1/(m + 1) > 0.02 a. Even for slightly larger labeling budgets, the conservative nature of the conformal the 'plus/ I' term in (2) continuues to have a significant impact. This effect is rigorously quanti- fied by the lower bound on type-I error provided in Propo-
Additional experiments In Appendix B.2. we extend the experiments presented above. These include evalua- tions under higher contamination levels and additional out- lier detection models, such as One-Class SVM (Schölkopf et al., 2001) and Local Outlier Factor (Breunig et al., 2000). We also examine the effect of different outlier injection strategies and the robustness the proposed method under test-time distribution shifts the outlier population.

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.00
Standard
0.6
Type Type-1-Ermi 1-Error
Oracle (infeasible)
0.4
Power
at
Naive-Trim (ivalidd)
Small-Clean
0.02
0.2
Label-Trim
Target type-I level
0.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Figure 5: Comparison of conformal outlier detection methods tabular dataset ("shuttle") as function of the target type-I emor rate a. The contamination rate r is fixed to 3%. Other details are as in Figure 3.
4.2. Visual Data
niques. Numerical experiments with real data confirmed that standard conformal outlier detection methods are con- servative under contaminated data and demonstrated that our SabellTriine method can significantly enhance power.
In what follows, we compare all methods using benchmark visual datasets for outlier detection, Similar to Zhang et al. (2024), we construct six datasets, where the inlier samples are always images from CIFARIO (Krizhevsky & Hinton, 2009; Krizhevsky et al., 2009) and the outlier samples vary across datasets. Specifically, the outliers are drawn from (1) MNIST (Deng, 2012), (2) SVHN (Netzer et al., 201 (3) Texture (Cimpoi et al., 2014), (4) Places365 (Cimpoi et al., 2014), (5) TinylmageNet (Torralba et al., 2008), and (6) CIFAR100 (Krizhevsky et al., 2009). For all datasets, we use the outlier detection model proposed by Sun al. (2021), ReAct, which operates on feature representations extracted by a pre-trained ResNet-18 model. More details are in Appendix B.1.
However, the experiments also reveal limitation of our Label-Tris 1-Trim method: while it improves power compared to standard conformal inference, it often remains too conser- vative, particularly when the labeling budget is very limited, leaving room for further improvement. A promising direc- tion for future research is enhance Label-Trih with active learning strategies (Makili al., 2012; Fannjiang et al., 2022; Prinster et al.), enabling the removal of more outliers without increasing the labeling budget.
While we provide robustness results for calibration under contaminated reference data, our analysis assumes that the inlier calibration and test points are drawn i.i.d. from the same distribution. An important direction for future work is extend this setting beyond the i.i.d. assumption, pos-
Table I summarizes the results for all six datasets. Overall, we can see a trend similar to that of the tabular data: the Standard and Small Amall-CC 1-Clean methods are valid but con- servative, the Naive Trim fails to control the type-I error, and our Label-Tri achieves significant boost in power while practically controlling the type-I error. Notably, our Label-Tring imm method attains near-oracle performance for low contamination rates. Detailed results for each dataset, along with experiments with additional outlier detection models- speciillyey the ReAct method (Sun et al., 2021) with pre-trained VGG-19 model and the SCALE method (Xu al., 2024) with ResNet-18 model-are provided in Appendix B.2.2.
sibly by building ideas from Tibshirani al. (2019); Podkopaev & Ramdas (2021); Sesia et al. (2024); Barber et al. (2023).
Another limitation of the Label-Tris approach is its reliance on actively collecting new annotations. In scenarios where flexible labeling budget is unavailable but access to small, clean reference set is feasible, this dependency becomes restrictive. As our experiments demonstrate, the limited sample size imposes fundamental constraint on the power of conformal outlier detection methods. This raises an intriguing question for future research: given a small clean reference set and larger, contaminated reference set, how can we effectively and safely clean the contaminated data enhance detection power at test time?
5. Discussion
In this work, we studied the robustness of conformal predic- tion under contaminated reference data. Motivated by empir- ical evidence, we characterized the conditions under which conformal outlier detection methods become too conserva- tive. To improve power, we proposed the Label-Trim method, which leverages an outlier detection model and limited labeling budget to remove outliers from the con- taminated reference set. We also provided a theoretical justification for this approach, employing novel proof tech-
One potential solution could involve using the small clean reference set to calibrate a base outlier detection model. This calibrated model could then be employed to clean the larger contaminated set by removing detected outliers, while carefully accounting for inliers mistakenly classified as outliers. Exploring such semi-supervised data- cleaning approach represents promising direction for future work,
8

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table I: Comparison of conformal outlier detection methods on six visual datasets for varying contamination rate r and target type-I error level a. The empirical type-I error values are averaged across all datasets. The empirical power is presented relative to the Standars rd method (higher is better), and averaged across all datasets. Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination
1%
3%
5%
Method
Power
Type-I Eror
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (+ 0.0317)
0.008 (+ 0.0003)
1.0 (# 0.0354)
0.005 (+ 0.0003)
1+ 0.0408)
0.004 (+ 0.0002)
Oracle (infeasible)
1.166 (1: 0,0336)
0.01 (# 0.0003)
1.549 (#: 0.0425)
0.01 (#: 0.0003)
1.961 (# 0,0531)
0.009 (#: 0.0004)
Naive-Trim (invalid)
1.659 (+ 0.0342)
0.017 (+ 0.0004)
2.79 (+ 0.045)
0.027 (+ 0.0006)
4.16 (+ 0.0596)
0.036 (+ 0.0007)
-naal-Cleean
0:t 0.0)
0.0 (+ 0.0)
0.0 0.0)
0.0 0.0)
0.0 (t: 0.0)
0.0 0.0)
Label-Trim
1.166 (#: 0.0336)
0.01 0.0003)
1,517 (== 0.042)
0.01 (# 0.0003)
1.786 (# 0.0498)
0.008 (= 0.0003)
(a) Target type-I error rate a = 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (L 0.0174)
0.027 (1 0.0006)
1.0 (L 0.0189)
0.019 (1 0.0005)
(L 0.0212)
0.015 (L 0.000S)
Oracle (infcasible)
1.062 (# 0.0176)
0.03 (+ 0.0006)
1.235 (+ 0.0192)
0.029 (+ 0.0006)
1.448 (# 0.023)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.146 (1 0.0175)
0.035 (L 0,0006)
1,487 (1 0,0186)
0.043 (1 0.0007)
1.882 (1 0.0224)
0.052 (1 0.0008)
Small-Clean
0.714 (1: 0.0448)
0.02 (1: 0.0021)
0.869 (#: 0.0501)
0.02 (1: 0.002)
1.033 (1 0.0613)
0.021 (1: 0.0023)
I Label-Trim
1.041 (+ 0.0177)
0.029 (+ 0.0006)
1.139 (+ 0.019)
0.025 (+ 0.0006)
1.215 (+ 0.0226)
0.021 (+ 0.0006)
(b) Target type-I error rate a 0.03
though we anticipate that establishing the theoretical validity of such method may not be straightforward.
were funded by the European Union (ERC, SafetyBounds, 101 163414... Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. This research was also partially supported by the Israel Science Foundation (ISF grant 729/21). Y. R. acknowledges additional support from the Career Advancement Fellowship at the Technion.
Another direction related to the above discussion is how to account for uncertainty in the labeling process. In our cur- rent formulation, we assume that all calibration points are labeled as inliers, but some may in fact be outliers. Impor- tantly, we have no indication of which points are mislabeled, nor any signal of uncertainty in the labels. Exploring how to incorporate such uncertainty into our framework could enhance its practical utility. The line of work presented in Stutz et al. (2023); Javanmardi et al. (2024); Caprio et al. (2025) may offer a valuable starting point for such an ex- tension: it introduces techniques for handling ambiguous labels, though in the context of multi-class classification rather than outlier detection.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Notably, our paper also provides practical guidance on how to annotate data in situations where there is uncertainty about whether a point is an inlier or an outlier. As indicated by Lemma 2.2 and supported by our empirical results, when there is uncertainty about a point's label, treating it an inlier is a conservative strategy that preserves type-I error control. This observation suggests compelling connection between label ambiguity (Stutz et al., 2023; Javanmardi et al., 2024; Caprio et al., 2025) and contamination in the reference set, which merits further investigation.
References
KDD Cup 1999 Data Set. https://www. kaggle. com/ml g-u b/ credi teadiitcardraus Not normal- ized, without duplicates, categorial attributes removed. Accessed: January, 2021.
Credit Card Fraud Detection Data Set. https: //ww: hww.:/u///redttaaddafeeeeee cessed: January, 2021.
kaggle. . www.co////...... wwtpe//ooocccc//eeeee
Ac-
Acknowledgments
M. S. was partly supported by NSF grant DMS 2210637 and by a Capital One CREDIF Research Award. Y. R. and M. B.
Statlog (Shuttle) Data Set. http: cs. s stonybrook. http://hutteedataaae aset. Accessed: January, 2021.

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, J. J. Conformal prediction beyond exchangeability. Ann. Stat., 51(2):816-844 2023.
Fannjjang. C., Bates, S., Angelopoulos, A. N., Listgarten, J., and Jordan, M. I. Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences, 119(4)):22044699119 2022.
Bates, S., Candès, E., Lei, L.,, Romano, Y., and Sesia, M. Testing for outliers with conformal p-values. Ann. Stat., 51(1):149- - 178, 2023.
Farinhas, A., Zerva, C., Ulmer, D. T., and Martins, A. Non- exchangeable conformal risk control. In International Conference on Learning Representations, 2024.
Benjamini, Y. and Hochberg, Y. Controlling the false dis- covery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B, 57(1):289-300 1995.
Feldman, S. and Romano, Y. Robust conformal prediction using privileged information. In Conference on Neural Information Processing Systems, 2024.
Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J. Lof: identifying density-basee local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93 93-104, 2000.
Feldman, S., Ringel, L., Bates, S., and Romano, Y. Achiev- ing risk control in online learning settings. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Buitinck, L.., Louppe, G., Blondel, M., Pedregosa, F., Mueller, A., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., Layton, R., VanderPlas, J., Joly, A., Holt, B., and Varoquaux, G. API design for ma- chine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pp. (08-122,20) 2013.
Gibbs, I. and Candès, E. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 1660-1672, 2021.
Gibbs, I. and Candès, E. J. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162):1 2024.
Caprio, M., Stutz, D., Li, S., and Doucet, A. Confor- malized credal regions for classification with ambigu- ous ground truth. Transactions on Machine Learn- ing Research, 2025. ISSN 2835-8856. URL https: //openrevien net. forum? d==.77
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7770-778.20 2016.
Chalapathy, R. and Chawla, S. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03400 2019.
Javanmardi, A., Stutz, D., and Hüllermeier, E. Conformal- ized credal set predictors. Advances in Neural Informa- tion Processing Systems, 37:1 179877177111111 2024.
Chen, Y. Pytorch cifar models. https: com/ chenyaofo, c http://ttoofaaarrmeooeeeeee Accessed: 2025-5-17.
Jiang, X., Liu, J., Wang, J., Nie. Q.. Wu, K., Liu, Y., Wang, C., and Zheng, F. Softpatch: Unsupervised anomaly de- tection with noisy data. Advances in Neural Information Processing Systems, 35:15433-15 2022.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, (0), 2009.
Clarkson, J., Xu, W., Cucuringu, M., and Reinert, G. Split conformal prediction under data contamination. Pro- ceedings of the Thirteenth Symposium on Conformal and Probabilistic Prediction with Applications, volume 230 of Proceedings of Machine Learning Research, pp. 5-27. PMLR, 2024.
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 and CIFAR-100 datasets. URI: https://www. cs. toronto. edu/kriz/cifar. himl, 2009.
Deng. I.. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
Laxhammar, R. and Falkman, G. Inductive conformal anomaly detection for sequential detection of anomalous sub-trajece sub-trajectories. Annals of Mathematics and Artificial Intelligence, 74(1-2):67-93 2015.
Einbinder, B.-S., Feldman, S., Bates, S., Angelopoulos, A. N., Gendler, A., and Romano, Y. Label noise ro- bustness of conformal prediction. Journal of Machine Learning Research, 25(328):1-66, 2024.
Lei, J., Robins, J., and Wasserman, L. Distribution-free prediction sets. Am. Stat. Assoc., (088501):278-225 2013.
10

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Liu, F. T., Ting, K. M., and Zhou, Z.-H. Isolation forest. In 2008 eighth IEEE international conference data mining, pp. 413-422. IEEE, 2008.
Sun, Y., Guo, C., and Li, Y. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
Makili, L. E., Sánchez, J. A. V., and Dormido-Canin S. Active learning using conformal predictors: application to image classification. Fusion Science and Technology, 62(2))347-355, 2012.
Tibshirani, J.J., Foygel Barber, R., Candès, E., and Ramdas, A. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
Torralba, A., Fergus, R., and Freeman, W. T. 80 million tiny images: A large data set for nonparametric object and scene recognition. TPAMI, 2008.
Netzer, Y.. Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised learning, volume 2011, pp. Granada, 2011.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world. Springer, 2005.
Xu, K., Chen, R., Franchi, G., and Yao, A. Scaling for training time and post-hoc out-of-distribuut detection enhancement. In The Twelfth International Conference on Learning Representations, 2024.
Park, J., Moon, J.-H., Ahn, N., and Sohn, K.-A. What is wrong with one-class anomaly detection? In Inter- national Conference Learning Representations Work- shops, 2021.
Yang. J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al. Openood: Benchmarking generalized out-of-distributtin detection. Advances in Neural Information Processing Systems, 35: 22598-32611, 2022.
Penso, C., Goldberger, J., and Fetaya, E. Estimating the conformal prediction threshold from noisy labels. arXiv preprint arXiv:2501. 12749, 2025.
Podkopaev, A. and Ramdas, A. Distribution- free uncertainty quantification for classification under label shift. In Un- certainty in artificial intelligence, pp. 844-853, PMLR, 2021.
Zaffran, M., Féron, O., Goude, Y., Josse, J., and Dieuleveut, A. Adaptive conformal predictions for time series. In In- ternational Conference on Machine Learning, pp. 25834- 25866. PMLR, 2022.
Prinster, D., Stanton, S. D., Liu, A., and Saria, S. Conformal validity guarantees exist for any data distribution (and how to find them). In Forty-first International Conference on Machine Learning.
Zaffran, M., Dieuleveut, A., Josse, J., and Romano, Y. Conformal prediction with missing values. In Interna- tional Conference on Machine Learning, volume 202, pp. PMLR, 2023.
Romano, Y., Sesia, M., and Candès, E. Classification with valid and adaptive coverage. Ady. Neural Inf. Process. Syst. 33, 2020.
Zaffran, M., Josse, J., Romano, Y., and Dieuleveut, A. Pre- dictive uncertainty quantification with missing covariates. arXiv preprint arXiv:2405. 15641, 2024.
Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C. Estimating the support of high- dimensional distribution. Neural computation, 13(7):
Zhang, J., Yang, J., Wang, P., Wang, H., Lin, Y., Zhang, H., Sun, Y., Du, X., Li, Y., Liu, Z., Chen, Y., and Li, H. Openood v1.5: Enhanced benchmark for out-of- distribution detection. Journal of Data-centric Machine Learning Research, 2024.
1443-1471, 2001.
Sesia, M., Wang, Y. R., and Tong, X. Adaptive conformal classification with noisy labels. J. R. Stat. Soc. Series B, pp. qkae| 14, 2024.
Zhao, Z., Cerf, S., Birke, R., Robu, B., Bouchenak, S., Mokhtar, S. B., and Chen, L. Y. Robust anomaly de- tection on unreliable data. In IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 630-637. IEEE, 2019.
Si, W., Park, S., Lee, I., Dobriban, E., and Bastani, O. PAC prediction sets under label shift. arXiv preprint arXiv:2310. 12964, 2023.
Stutz, D., Roy, A. G., Matejovicova, T., Strachan, P., Cemgil, A. T., and Doucet, A. Conformal prediction under am- biguous ground truth. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreview.oo net/ forum? http://ooommmmmcce22222222 1d=CAd6V2gxxc.

---

Robust Conformal Outlier Detection under Contaminated Reference Data
A. Mathematical Proofs
A.1. Auxiliary Technical Results
In this section, we begin by introducing two useful propositions, Proposition A.I and Proposition A.2, which will be used later in the proofs presented here.
Proposition A.1. Let be a dataset containing n scores, and define the threshold Q Qi-a as
Q1-a:== : i-th smallest element in DUfoo),
where
: [(1 an an 1)].
For any test point X the following holds:
s(Xn-1) > Qi-a if and only if pnt1 Sa,
where Pa+1 is the conformal p-value (2).
Proof of Proposition A.I. The proof follows the definition of conformal p-value from (2), and its relation to the empirical
quantile function:
1 Σ i== [(x(X;) 2 s (Xn+1))]
Pat1
aa
n n+1
+ >in
[[s(X,) 2 s(Xn+1)] La(n+11 1)]
Pat1
n n+1
1 +
[|s(X,)) 2 s(N+11) Ia(n + 1)]
1=1
1 n
[[s(X:) < s(Xx+1)] S La(n 1)]
[(((.)) < s(Xn+1)) 2n+1 La01---ann---- + 1)] i
I[s(X.) < s(X [(1 (1 1)7
(5)
The labeled steps above can be explained as follows.
(i) The values of pn+1 are discrete, taking values from { (x+T) 2
1). Therefore, po+1 & for some
1 E [n fm+1] 1]. We explicitly prove that p S a iff Pu+1 < [o(++l) follows:
+ Assume pn+1 5 <a(n+1)]
n+1 Therefore, Pm+1 [o(n+()]
u+1 < a(a+))
n+1 a.
& Assume Pn+1 S a. then n+1 I a. This implies that & S ann 1). Since k is an integer, it follows that k < (n(n + 1)]. Therefore, P.+1 n+I
&
< [a(n+1)]
n+1
(ii) This step follows directly from the equality n [(1 a)- 1)] + [o(n 1)j- We explicitly prove this equality follows:
The tern [(1 (n + 1)] represents the smallest integer greater than equal to (1 a) + 1). Hence, we can write:
[(1 - a) + +1] = (1 - a) + 1) + +
where 0.5 51 < 1.
12

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Similarly, the term [o(nn + 1)] represents the largest integer less than or equal to a(n 1). Thus:
La(n 1)] = a(n + 1) -2,
where 0 N 62 < 1.
Adding these two terms gives:
La(n 1)] + [[1 a)(n + 1)] = a(n + 1) 62 + (1 a) + 1) + =n+ (81 -2)) -
Since [a(n + 1)] + [[1 (n + 1)] must be an integer and 61, 62 E [0, 1), it follows that 81 - 82 = 0.
Therefore, [a(n + 1)] + (1-a) + 1)1 = n +
To complete the proof, we now show that (5) holds if and only s(Xn+1) > Qi-a-
Assume s(Xn+1) > Qi-a. By definition, E [[x) S Q1-al = [[1 - an + 1)7. Then, (5) holds since
a
Σ I[s(X;)) s(xn+1)] 2 Σ [ss(X)) s &Qi-a [(1 (11-a0 a) 1)].
We prove this direction by contradiction, assuming that (5) holds. Now, suppose that s(Xn+1) S Qi-a also holds, implying that
< @1-a] 2 [[1 a(1-a)(n+ 1)7, Σ [((x))) < s(X E [ss(x))<
which contradicts the assumption (5). Therefore, we conclude that s(Xn+1) > Q1-a. The last step above can be explained follows.
(i) Recall that by definition, Qı-a is specific value in (s(X,))² and Σ Isx,x,) Q1-al [(1 (n
1)1. This implies that
n
Σ [(((,)) <1-a)= [(1 an + 1)] Σ [(((,,) Q1-al < [(1 - a) 1)].
In sum, Put1 I a holds if and only if (5) holds, and the latter holds if and only if s(X,11) > Qi-a- This completes the proof.
Proposition A.2. Let 'D be dataset containing scores, and let Sn+1 be test score. Define the following thresholds:
Q1-a == i-th smallest element in D D(ooo,,
where
: [[1 a) + 1)].
Similarly, let
2 = i-th smallest element in D DUfSu4ll
It follows that
Q 2 Qu+1 1-a almost surely.
Proof of Proposition A.2. Since the largest possible score is oc, the set Dfooo} almost surely contains scores that are greater or equal to those in D DU(+++)) Consequently, Q1-a := ?-th smallest element in D Dfoo} is almost surely greater than or equal to pp+1 = i-th smallest element in DS
13

---

Robust Conformal Outlier Detection under Contaminated Reference Data
A.2. Explaining the Conservativeness of Standard Conformal p- a-Values
A.2.1. PROOF OF LEMMA 2.2
Proof of Lemma 2.2. To simplify the notation define the random score S := s(X,) for all i Deal Lin 1}. Throughout the proof, refer to the calibration as the set of nonconformity scores corresponding the calibration points. Without loss of generality, assume that the inliers in Deal are located at the first nu indices. Let Dinlier [Ro] denote the set of indices corresponding to the inlier scores. Consequently, define Dottlier {no + 1, roo ......
as the set of indices corresponding to the outlier scores in Deal- We assume the scores have no ties (which can always be achieved by adding a negligible random noise to the scores output by any model).
n
Given fixed realization of the score vector (81,..., 8n, 8n+1) E R define the following two events:
Ein: the unordered set of inlier scores, including the test score, (.,,..., Sng Sn+1] = {81,.... Sngs Sn-1):
Foutt the unordered set of outlier scores {Smo+1... Snott....,,A.
,,S) =
Under the setup defined in (3), when Ho is true, the test score Sn+1 and the inlier scores in the calibration set are i.id. from Po. Therefore, by exchangeability, the following holds for each inlier index E Dintier Ufn + 1]:
P (S.+1 8, Ein+ Eowt)
(6)
Since the calibration almost-surely distinct, the probability of a null point obtaining any outlier score is zero Therefore, for each outlier index Duttier:
PP PS 5j Eins Eout) = 0.
To obtain an upper bound on the type-I error rate, P (fn+1 S a), we use the equivalence established in Proposition A.1. According to this result, the following holds:
P (pn+1 =a)= =P(Sttt > Q1al))
where @pal_ is the fear-th smallest element in (8:)%_ U foo} and ical == [(1 a) 1)]-
Moreover, define Q as the êear-th smallest score in 1n+1 By Proposition A.2. Qu 2 + almost surely.
Now, we obtain an upper bound for TP (Sa+1 Qgaal Ein, Fout). where the probability is taken over random permutations
14

---

Robust Conformal Outlier Detection under Contaminated Reference Data
of the scores conditional Einy Bout.
IP Sn+1 > Qi Lins Eout) P SSpt > Qita | Bins Eout)
= E I Sp+1 > Q Einy Bout
Σ
IR T [Sn+1 si] II
8; Q
Fin Font
ie ialior (x+1]
Σ
8/ Q
TP P(S++1 8x Piny Fout)
inDinter Ufa+1}
Σ
I 8; > Q
no +
iEDinlter Uf+t1]
(8) 1
[a(+1))
Σ
85 > 2
no +1
iEDoutler
a
+
1
an
Σ
1
[ > 2
To +
iE DEDoutler
a
711
1
Σ
8t S Q7-1
120 +
T1
Douttier
(Mi)
S
111
1 - a
Σ
T
|so S 2
+1
iED iEDontiie
T1
(1-a a
F (Qi=aa) ))
no
+
where F1 is the empirical CDF of the outlier scores. The labeled steps above can be explained follows.
(i) i > Q is measurable with respect to the o-algebra generated by Ein Eour- This follows because Q is the êeal-th smallest element of {s1,..., Sn+1), }. which fully determined by these variables. Thus, we can pull out of the expectation.
(ii) + is the tea smallest score {s1,... 8n, 8at1) and [n + 1] = Doutlier U Dinlier Ufn + 1}. By definition,
n+1
Σ
I [[., > 1 +
Σ
1 Q = Σ
s > 2 = La(n + 1)] & a(n + 1)
DEDler Un++1}
iEDoutiler
1=1
and therefore.
Σ
I 1/0, > Q & a(n++1)-
Σ
iED Uf++1}
iEDouctier
(iii) Since Q 2 + almost surely, increasing the threshold (i.e., using Qual)) results in an equal or larger value of the sum.
15

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Now, we derive upper bound for P (fn+1 a)) follows:
P Pfn+1 5a)= =P
Sn+1 Q9
Fin: Fout)) )]
E
Snt1
1
(1 a M (Qi-l
E
f1g
+
n
a
-
no
+1
with this expectation being taken over different realizations of the inlier and outlier nonconformity scores.
A.2.2. AN ALTERNATIVE VIEW BASED ON MIXTURE DISTRIBUTIONS
Next, we provide an additional theoretical result concerning the conservativeness of conformal outlier detection methods, to supplement the result presented in Section 2.3 from a point of view closer that Sesia al. (2024).
Specifically, we consider a contaminated calibration set, Peat. which may include both inliers (samples i.ddd from IP)) and outliers (samples i.i.d. from P1 * Po). The goal remains to test the null hypothesis Ho that data point X,41 is an inlier, independently sampled from Fo-
This setup differs from (3) in that the calibration set is drawn from a mixed distribution, where the proportion of outliers in the population is denoted by E [0,1), Hence, the numbers of inliers and outliers in the calibration set are random, rather than fixed. Formally, this setup expressed as:
X, iid Pmixed = (1-8) 8) Po +5 Pl.
Vi Deals
(8)
Ho: : X ind. Po-
Let Fo and Fi denote the CDFs P. and Pl, respectively, and Qual, represent the [(1 (1--ad an + 1)]-th smallest score in the calibration set.
Corollary A.3 (Conservativeness)) Under the setup defined in (8), Ho is true, then, for any a (0, 1).
PP (fm+1 a S a -8 Fo( - Fi (((()=a))
Corollary A.3 reformulates Theorem in Sesia al. (2024) under the setup in (8). This result quantifies the behavior of conformal outlier detection methods in the presence of contaminated data and establishes guarantees on the type-I error rate. This result complements our analysis of the conservativeness of these methods.
Proof of Corollary A.3. The proof adapts the argument of Theorem in Sesia et al. (2024) to the outlier detection setting considered here. Specifically, we follow the structure the original proof, making adjustments to account for the presence of inliers and outliers in the calibration set.
By Proposition A. we have
Pn4l Sa + Sh+1 Qcal
16

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Under the null, upper bound P (fn+1 Sa) follows:
Po (fn+1 Sa) Po (pn+1 S (x) + Piixed (fn-1 S a) - P Piixed (fa+1 S a)
= I' mixed (pn+1 S a)) mixed (u+t1 S ()) - Po (Po+1 S sa]
S a [P mixxdd (pu+1 1 a) - Po (Pa+1 [a]
[[(1 F) (pn+1 S a.) + 6PI (Pn+1 S sai) - Po (pn+1 S a)]
ò [L'] (pn+1 [a) Po (pn-1 sa]]
5
Pi Sut1 > ( - Po [Sn+1 (
Po
Sh+1 S Q1 - IP|
SSt+1 S Q(
a
a 5 E
Po (Su+1 E Qua)] Deal - P1 SSa+1 I & Deal)
a SE Fo (2 Fi (Q )]
A.3. Validity of the Label-Trim Method
1. PROOF OF THEOREM 3.1
MAIN STEPS
Proof of Theorem 3.1. As in the proof of Lemma 2.2, define the random score S; == s(X,) for all Deal Ufn + 1}. By Proposition A.1, for any fixed a (0, 1), the probability of type-I error, IP (p!T) a). can be expressed
P (PIT a) = P (Sn+1 Q1-
Consider the augmented set (StheDL U (S which includes the test score Sn+1. Define QT follows:
@l itr smallest element in hee U (S
By Proposition A.2, it holds that QT 2 Q
LL.+1 almost surely.
Now, consider an imaginary "mirror" version of this method that applies the label-trim algorithm with two key differences:
it uses a larger labeling budget, m = m +
it treats { as the calibration set instead of { that is, it includes the test point in the annotation process, preserving the exchangeability with the calibration inliers.
LT Let U {n + 1} denote the indices of the trimmed augmented calibration set produced by the mirror procedure, let iinlier notlier Diubeled denote the indices of the corresponding labeled data points, and define Diabeled Plabeled as the corresponding subsets of inliers and outliers, respectively. Under this mirror procedure, the empirical quantile Q I.T.+ corresponds to
d = -th smallest element in (S.) 1 U (S,+1),
I.T where ürr = [(1 - aT + 1)] and îu == IDeall
By construction of this mirror procedure, inr < â.T almost surely, because TLT & LT almost surely and thus
iLT = [1 (n + 1)7 s [[1 an + 1) âr..
Using the fact that äo 5 âur almost surely, we prove in Appendix A.3.2 that, almost surely,
@T.
ÓLT,++1
17

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Since already knew that Q QI-a+! this implies:
O 2 01-a+! 2 -
(10)
Therefore, the type-I error rate of the Label-Trim approach can be bounded from above by the type-I error rate of the mirror procedure, which can be studied with an approach similar to that of the proof of Lemma 2.2.
Dur Let outtier denote the outlier indices remaining in D LT with i
D outliarl As in the proof of Lemma 2.2, define Em n+1 respectively. Appendix A.3.3 proves that
and Poot as two unordered realizations of the inlier and outlier scores in [S:]-=
Dinlter P (S > Q Fia: Fouu: D Dlabeled labeled) S a + no no +1 (1 - P
(1
(1))
inlier
from which it follows immediately, by marginalizing over Ein; Eout, Diubeled, and Dabblled that
P (S > 2 S E + 1 no + ( (1-a) -PTT (श )
A.3.2. PROOF OF THEOREM 3.1 PROOF OF EQUATION (9)
Proof. We prove (9) by analyzing two distinct cases, depending on whether S,+1 among the m + 1 largest scores or not. Recall that S a(n 1), by assumption, and T. < n. It is easy see that this implies that, almost surely,
it [[1 - (n 1)1 S n + (m 1).
(12)
The mirror procedure labels and potentially removes the largest m + 1 scores out of the 1 total scores. Thus, QTT
smaller than all the outliers removed during trimming, i.e., @ < Sy for Sy Diabelod
outlier
Suppose S is among the m + 1 largest scores in {
1=+1
This case is illustrated below:
m
In this scenario, the mirror trimming approach ad- D (n 1} ditionally labels the test score Sa+1, which, under the null hypothesis, is an inlier and is thus not removed. As result, both trimming procedures LT yield the same set; i.e., Deat = DLT Conse-
772 1
DUT fn + 1}
quently, @LT, QLTa+t
Dur U fn + 1}
Suppose Sn+1 is not among the m + largest scores in {S.)+t] Within this case, there are two sub-cases to consider.
The (m + 1)-th largest score in an inlier:
D fn+1) 1}
In this case, the mirror trimming approach addi- tionally labels an inlier, which is not removed. As result, both trimming procedures yield the
+
same set; i.c., D = DLT Consequently,
DUT u (m+ 1}
I.
11-a = 1-a
DLT U (n +1
18

---

Robust Conformal Outlier Detection under Contaminated Reference Data
The (m + 1)-th largest score in an outlier:
This is the interesting case where i n and the set LT
771
contains one fewer outlier score than DLT follows from (12) that the threshold + is smaller than the r + 1 largest scores.
Dau U 1]
778 +
Then, in this region, Deal and Deal contain the same scores. Therefore, the åLr-th smallest score corresponds to the same score in both D U(n+ + 1} and Deat (n++1) Since iLT S êLT, it follows D U (
Dul u (n+
that Q QLI I QL
Score
A.3.3. PROOF OF THEOREM 3.1 PROOF OF EQUATION (11)
Proof.
P Snt1 Q Ein Eout Diabeled, D
inlier Labeled
P Sat1
@1 Eins Eout, Diubeled; Piabeled]
Σ
I 84 @IT,n+1]
I[[0+1 8i] Bin, Eout, Diabelod; D Plabeled
Piller ((++1)
Σ
Snnt = Emg Eout Diabeledd D Disbeled) 8 Q
JED mileer Uf++1}
(6))
Σ
> uL+
PP PSS+1 = = Emn Eout)
IED Un++1}
Σ
8 Q1-a
no +1
iE inniee U(++1}
a(ñ +1)
Σ
1-a
n0 +1
iED LT outlier
1
a(n LT
+1)
Σ
I 1
2TT.o+11 +1
1o +1
A DED outller
a<n LT
+1)
Σ
I
[
Q1-a
Q
+1
14g +
LLT cursher
1
aT +1) -1)- - RLT
Σ
I S +1 +1)
no +1
iEDLT
(T-
-
a +
N1o
+
1
Top
+1
The labeled steps above can be explained follows.
[ st > 2
[+ is measurable with respect to the o-algebra generated by Eins Eout, Diabeled, and D
labeled since
inlier
gLT++1 is the - smallest element of (81,..., Sn+1) } \ Diibetted Dj
Dinlier inlier
1-a
Plabeled
19

---

Robust Conformal Outlier Detection under Contaminated Reference Data
(ii) The mirror procedure is applied (5 preserving the exchangeability of the test score S2+1 with the calibration inliers. Hence, the resulting labeled sets Diabeled and D Paaeleed contain additional information about Su+1 beyond Emn Eout.
(iii) By definition, Q1-x+1 is the ihr-th smallest element of {St [S)) U (Sa+1), where irr = [(1 - (n + 1)].
Consequently, La(ñ + 1)] scores in {S; { se U 1 are larger than @LT
(iv) The set D ootlier is either equal to D outlier or contains one fewer outlier, and n E (n!T, n 1} almost surely.
(v) Recall from (10) that Q 2 @L.,++1 almost surely.
20

---

Robust Conformal Outlier Detection under Contaminated Reference Data
B. Supplementary Experiments and Implementation Details
B.1. Datasets
Table 2 summarizes details of the three tabular benchmark datasets. For all tabular datasets, we perform random subsampling to construct contaminated train, calibration, and test sets. Specifically, for the shuttle and KDDCup99 datasets, the train set contains 5,000 samples, and the calibration set contains 2,500 samples, both with contamination rate of 3%, unless stated otherwise. The inlier and outlier test sets consist of 950 and 50 samples, respectively. For the credit-card dataset, the train set contains 2,000 samples, while the calibration and test sets follow the same setup as the Shuttle and KDDCup99 datasets.
Table 2: Summary of tabular datasets
Dataset
Shuttle (shu)
Credit-card (cre)
KDDCup99 (KDD)
Total Samples
58,000
284,807
494,020
Number of Outliers
12,414
492
396,743
Number of Features 9
29
41
For visual datasets, we use the OpenOOD benchmark (Zhang et al., 2024; Yang et al., 2022). Specifically, for each dataset and contamination rate, we perform a one-time training of an outlier detection model, which operates on feature representations extracted from a pre-trained backbone. We include results with the following model configurations:
ReAct with ResNet-18: ReAct (Sun et al., 2021) operates on feature representations extracted from a pre-trained ResNet-18 model (Zhang et al., 2024; He et al., 2016). The model applies a percentile-basee threshold (set 90%) to truncate activations, where the threshold is computed on the contaminated train set. These truncated activations then pass through the fully connected layer of the pre-trained ResNet-18. The outlier score is computed using an energy-based log-sum-exp function applied to these truncated activations.
ReAct with VGG-19: Same as above, but with a pre-trained VGG-19 backbone (Chen) instead of ResNet-18.
SCALE with ResNet-18: SCALE (Xu et al., 2024) operates on feature representations extracted from pre-trained ResNet-18 18 model (Zhang et al., 2024; He et al., 2016). The model rescales the activations using sample-specific factor, defined as the sum of all activations divided by the sum of activations below a certain percentile (set to 65%). Similar to ReAct, the outlier score computed using an energy-based log-sum-exp function applied to the rescaled activations.
After training, we save the outlier scores for the remaining outlier samples and the CIFAR-10 test set. We randomly subsample this pool of scores to construct the calibration and test sets, ensuring that all sets are disjoint.
The sizes of the train and calibration sets are 2,000 and 3,000, respectively, with the same contamination rate. The inlier and outlier test sets consist of 950 and 50 samples, respectively.
B.2. Additional Experiments
B.2.1. TABULAR DATASETS
In this section, we provide additional real-data experiments conducted on the credit card (cre) and KDDCup99 datasets, complementing the analysis provided for the shuttle dataset in the main manuscript. Figures 12 corresponds and extends the figures presented in the main text.
We further report supplementary results across all three tabular datasets considered in this work, including: additional outlier detection models (Figures 13 and 14); higher contamination levels (Figure 15); strategic outlier injection (Figures 16 to 19); and test-time distributional shifts in the outlier population (Figure 20).
Results as a function of the contamination rate In Figure 3 of the main manuscript, we analyze the performance of conformal outlier methods as a function of the contamination rate r. Here, we repeat the same experiment on the credit-card
21

---

Robust Conformal Outlier Detection under Contaminated Reference Data
and KDDCupp9 datasets (Figures 6 and 7). The performance trends similar to the one presented in the main manuscript: both the Standard rd and Small 1 -Cllean methods achieve valid type-I error rate but exhibit conservative behavior. In contrast, the Naive ve Trimm method fails to control the type-I error rate. The Label- -rrm method attains improved power while practically controlling the type-I enror at level a.
0.8
125
Standard
Typo 1 E-rrrrr
0.03
as
Trimmed
100
Oracle (Infeasible)
Power
75
Naive-Trim (Invalid)
0.02
Small-Clean
0.4
60
Label-Trim
0.01
25
Target type-I level
Labeling budget
0.0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
00
0.01
0.02
0.03
0.04
0.05
outliers
Contamination rale
Contamination rate
Contamination rale
Figure 6: Comparison of conformal outlier detection methods real dataset "credit-card" as function of the contamination
rate Other details in Figure 3.
Standard
Typo 1 Error
o
on
Power
08
as
Trimmed
700
75
Oracle (infeasible)
Naive-Trim (invalid)
0.03
Small-Cless
0.02
0.
80
Label-Trim
0.01
n2
2
Target type-I level
Labeling budget
0.0
0.00
0.02
0.03
0.04
0.05
0.0
0.01
0.02
0.04
0.05
0.0
0..11
0.02
0.03
0.04
0.05
outliers
Contamination rate
Contamination rate
Contamination rate
Figure 7: Comparison of conformal outlier detection methods on real dataset "KDDCup99" as a function of the contamination
rate r. Other details are as in Figure 3.
Results as a function of the labeling budget In Figure of the main manuscript, we evaluate the performance of the Label-Trim, Small-Clean and, Oracle methods as a function of the labeling budget m. Figures 8 and 9 extend this analysis to the credit-card and KDDCup99 datasets, respectively. Consistent with the trends observed in Figure increasing the labeling budget improves the performance of the Label-Tras method in terms of both type-I error and power. Notably, although the condition in Theorem 3.1 is longer satisfied for 50, the Label-Tring method still maintains valid type-I error control at the desired level a.
For the KDDCup99 dataset, however, we observe that the Small malan method shows higher power than the Oracle across several labeling budgets (m 2 55). This is due to the high variability in the dataset and the small sample size used by this method, resulting in significant variance in type-I error. illustrate this, Figure 10 presents box plot showing the variability and instability of the Small-Cles method this regime. While this leads higher average power, this variability is undesirable in practice.
0.025
100
Oracle (infeasible)
Type-1 Error
0.00
0.6
Small-Clean
Trimmed
80
0.015
Power
Label-Trim
0.4
60
Target type-I level
0.010
0.2
Labeling budget
0.005
# outliers
70
0.000
30
100
3
80
9n
100
10
20
30
40
50
69
8n
9n
100
Labeling budget
Labeling budget
Labeling budget
Figure 8: Performance on real dataset "credit-card" as function of the labeling budget m. The contamination rate is
r 0.03. Other details are as Figure 3.
22

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0,023
100
Oracle (infeasible)
Type Error
0.000
Smail-Clean
Trimmed
66
0.13
Power
Label-Trim
0.2
60
Target type-I level
0.010
0.1
40
Labeling budget
0.005.
# outliers
0.00n
0.9
1n0
6
900
100
g0
90
100
Labeling budget
Labeling budget
Labeling budget
Figure 9: Performance on real dataset "KDDCup99" function of the labeling budget The contamination rate is
0.03. Results are averages across 400 random splits of the data. Other details are as in Figure 3.
Oracle (infeasible)
0.10
Small-Clean
Type- 1 --Eror
0.om.
0.8
Label Trim
Power
Target type-1 level
Labeling budget
# outliers
con
0.02
0.00
108
105
Labeling budget
Labeling budget
Figure 10: Performance real dataset "KDDCup99" function of the labeling budget m. The contamination rate is 0.03. Results are averages across 400 random splits of the data. Other details are as in Figure 3.
Results as a function of the target type-I error level In Figure 5 we examine the performance of conformal outlier detection methods as a function of the target type-I error level a. Here, we replicate the experiments on the credit-card and KDDCup99 datasets (Figures and 12). line with the trends observed Figure 5, the Label-Trs im method performs well low type-I error rates. Notably, for a 0.01, the Label-Tae im. method outperforms the baselines, while practically controlling the type-I error level a, even though the condition of Theorem. 3.1 is not satisfied in this case. This highlights the robustness of our approach.
0.08
0.8
Standard
Type-1 EEror
Oracle (infeasible)
0.04
Power
0s
Naive-Trim (ivalidd)
ad
Small-Clean
0.02
Label-Trim
02
Targel type-I level
0.00
n0
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.09
0.04
0.05
Figure 11: Comparison of conformal outlier detection methods on real dataset "credit-card" as a function of the target type-I enror rate a. The contamination rate is fixed 3%; other details are as Figure 3.
23

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.06
Standard
Type- 1-Errorrrr
0.6
Oracle (infeasible)
0.04
Power
Naive-Trim (inalidd)
04
Small-Clean
0.02
Label-Trim
0.2
Target type-I level
0.00
0.01
0.02
0.03
0.04
0.05
0.01
0.02
0.03
0.04
0.05
Figure 12: Comparison of conformal outlier detection methods on real dataset "KDDCup99" as function of the target type-I emor rate a. The contamination rate is fixed to 3%; other details are as in Figure 3.
Results with additional outlier detection models Here, we consider two additional outlier detection models: Local Outlier Factor (LOF) with 100 estimators and One-Class Support Vector Machine (OC-SVM) with an RBF kemel,, both implemented via sci l tlaarn. Figures 13 and 14 present the performance of all methods when applied with LOF and OC-SVM, respectively, As can be seen, the proposed Label 1 Trim controls the type-I error across both models. When the outlier detection model less discriminative (e.g.. LOF or OC-SVM on the KDDCup99 dataset), the candidate set becomes less effective and power decreases. Nevertheless, our method improves over the baselines, including the Small 1 CCeen method.
Type-1 Error 0.
123
09 96
100
0.3
Trimmed
Power
75
a.2
44
50
0.01
0.2
25
0.01
0.02
0.03
0.04
0.05
0.
0.02
0.02
0.03
0.
0.05
0.0%
0.02
0.03
0.04
0.05
Contamination rate
Contamination
Contamination
(a) shuttle
0.04
125
0.0
Type- E-Errr
100
a
Trimmed
Power
78
0.02
30
6.1
0.2
25
0.0
0.01
0.03
0.04
0.05
0.0
0.01
02
0.03
0.05
6.0
0.01
02
0.15
Contamination rate
Contamination rate
Contamination rale
(b) credit-card
0.08
Power 0:26
125
Standard
Type- --rrrr
0.g
ais
o.20
Trimmed
100
75
Naive-Trim (invalid)
Small-Clean
Oracle (infeasible)
0.10
30
Label-Trim
0.02
0.05
Target type-1 level
0.00
Labeling budget
0.91
0.02
0.03
0.05
0.0
0.01
0.02
0.03
0.04
0.05
0.0
0.07
0.02
0.03
0.04
0,05
outilers
Contamination rate
Contemination rate
Contamination rate
(c) KDDCup99
Figure 13: Comparison of conformal outlier detection methods on real datasets as a function of the contamination rate r. All methods utilize a Local Outlier Factor model. Other details are as in Figure 3 in the main manuscript.
24

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.6
125
Type- --E-rrrrr
0.0
00
100
Trinamed
ans
Power
0
6.l
0.2
2
0.1
1.00
0.02
0.03
0.04
0.9
0.01
0.02
0.04
0.05
100
0.02
0.03
0.04
0.05
Contamination
Contamination rate
Contamination rate
(a) shuttle
128
0.00
Power 0.125
Type: 1-Brrne
100
0,000
Trimmed
004
0.75
78
0.00
50
0.02
0,025
0,01
0.03
0.04
0.05
0o
0.01
0.02
0.03
0.04
0.05
0.0
0.05
Contamination rate
Contamination rate
Contamination rate
(b) credit-card
Power 0,100
15
Standard
Type- I-Brror
u.me
100
Oracle (infeasible)
Trimmed
Naive-Trim (invald)
aog
0.0
Small-Clean
0,044
Label-Trim
0.02
2082
Target type-I level
0.00
090
Labeling budget
0.01
0.02
0.6
00
0.01
0.02
3.04
0.05
0.0
0.02
0.03
0.04
0.5
outliers
Contamination rate
Contamination rate
Contamination rate
(c) KDDCup99
Figure 14: Comparison of conformal outlier detection methods on real datasets as a function of the contamination rate r. All
methods utilize a One-Class SVM model. Other details are as in Figure 3 in the main manuscript.
25

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Results with higher contamination rate Figure 15 shows the performance of conformal outlier detection methods function of the contamination rate evaluated higher levels than those considered in the main manuscript. The performance trend aligns with that observed at lower contamination rates, across all datasets. Both the Standard and Small-Clean methods control the type-I enror at the nominal level but remain overly conservative. In contrast, Na Naive-Trim fails to control the type-I error. 'he proposed Label -Trim method achieves improved power while controlling the type-I error at level a.
0.10
90
Type: E-rrrr
0.08
Trinmed
300
Power
04
0.04
200
0.2
0.02
100
0.00
0.0
0.00
0.07
0.00
0.11
0.13
0.1
0.00
0.07
9.15
0.08
0.07
0.15
Contamination
Contamination
Contamination
(a) shuttle
0.00
0.8
150
Type: I-Eroor
04
Power
0.0
Trimmed
135
10g
0.00
75
0.2
50
0.00
0.0
0.05
0.07
0.00
0.12
0.13
0.15
0.05
0,07
0.15
0.05
0.07
0.15
Contamination rate
Contamination rate
Contamination rate
(b) credit-card
125
Standard
Type E-Errr
ton
300
Oracle (infeasible)
0075
Power
Trimmed
Naive-Trim (invalid)
200
Small-Clean
050
Label-Trim
0.025
9.1
100
Target type-1 leve]
0.000
0.0
Labeling budget
0.55
0.07
0.09
0.15
0.05
0.07
0.15
0.05
0.07
0g
outliers
Contamination rate
Contamination rate
Contamination rate
(c) KDDCup99
Figure 15: Comparison of conformal outlier detection methods on real tabular datasets as function of the contamination
rate r. Other details are as in Figure 3 in the main manuscript.
Results with strategic outlier injection We extend the experimental setting from the main manuscript by evaluating different outlier injection strategies. Instead of injecting outliers at random, we selected outliers that more closely resemble inliers--peeiffcclll. outliers whose nonconformity scores fall below a given score percentile. Figure 16 illustrates the nonconformity scores for both outliers and inliers under different outlier injection strategies for the "shuttle" dataset, highlighting that lower-percentile outliers increasingly resemble inliers. This sets the stage to evaluate the performance of our proposed method under these challenging settings. Figure 17 presents the performance of all conformal methods on the same dataset and outlier injection strategies, function of the target type-I error level. Following that figure, we that Label-Tr - -rrmm-rrmm im. controls the type-I eиror while improving power. Similar trends hold the credit-card and KDDCup99 datasets (Figures 18 and 19).
26

---

Robust Conformal Outlier Detection under Contaminated Reference Data
1000
1000
1000
Inliers
750
750
750
Outliers
500
250
250
250
-0.1
0.00
-0.1
0.0
0.3
0.2
-0.1
0.3
0.1
0.2
Scores
Scores
(a) Random outlier injection
(b) 70th percentile
(c) 50th percentile
Figure 16: Histogram of nonconformity scores inliers and outliers in a contaminated calibration subset of the "shuttle" data, with contamination rate 5% for different outlier injection strategies.
0.00
0.6
Type-1 -Emor
6.4
Power
0.4
602
02
0.00
0.0
0.01
0.02
043
0.04
0.0
0.1
0.02
0.33
0.44
0.s
(a) Random outlier injection
0.00
Type: Sppe---Eror
0.
0.04
Power
0.
0.02
0.00
0.02
0.94
0.1
0.02
0.3
0.44
0.88
(b) 70th percentile
04
Standard
Type: 1-Error
0.00
Oracle (infeusible)
0.3
044
Power
Naive-Trim (invalid)
02
Small-Clean
Label Trim
0.2
Target type-T level
6.0
0.0
0.91
0.02
0.00
0.94
0.05
0.91
0.02
0.03
0.94
0.65
(c) 50th percentile
Figure 17: Performance on a real dataset "shuttle" as a function of the target type-I error rate for different outlier injection strategies. Other details are as in Figure in the main manuscript.
27

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.8
Type-1 -Error
604
06
Power
04
602
0.2
0.0
0.11
043
0.94
040
0.11
0.02
0.03
0.44
0.05
(a) Random outlier injection
0.06
0.8
Type- Type-1-Emi L-Error
06
604
Power
04
022
02
0.39
0.0
0.01
0.02
0.94
0.15
0.01
0.02
0.03
0.04
0.05
(b) 70th percentile
6.06
Standard
Type- I-Error
0.6
Oracle (infensible)
0.04
Power
0.
Small-Cie Clean
Naive-Trim (invalid)
0.02
0.2
Label Trim
Target type-1 level
0:00
0.0
0.01
0.02
0.00
0.04
0.05
0.01
0.02
0.3
0.44
0.65
(c) 50th percentile
Figure 18: Performance on a real dataset "credit-card" as a function of the target type-I error rate a for different outlier injection strategies. Other details are as in Figure in the main manuscript.
28

---

Robust Conformal Outlier Detection under Contaminated Reference Data
0.06
Type- 1-Emo-rr
0.6
Power
0.02
02
0.00
0.0
0.01
0.02
0.03
0.94
0.05
0.01
0.02
0.03
0.04
0.05
(a) Random outlier injection
0.00
0.6
Type-1 --Error
0,0
Power
04
0.02
0.
0.00
0.0
0.01
0.02
0.03
0.94
0.55
0.11
6.02
0.03
0.34
6.65
(b) 90th percentile
0.5
600
Type-1 1-Eror
Power 0.4
Rgg
22
0.02
0,
0.00
0.0
0.01
0.02
0.03
0.94
0.05
0.11
0.02
0.03
0.44
0.65
P
(c) 80th percentile
Standard
Type-1-55 --Eorr
0.00
Oracle (infessible)
02
Power
Naive-Trim (invalid)
0.04
Small Clean
0.1
Label-Trim
0.02
Target type-I level
0.00
0.0
0.01
0,02
0.03
0.94
0.05
0.01
0.02
0.05
0.44
0.65
(d) 70th percentile
Figure 19: Performance on real dataset "KDDCup99" as function of the target type-I error rate for different outlier injection strategies. Other details in Figure in the main manuscript.
29

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Results with test-time drifting outliers We further investigate the robustness of the proposed method under test-time distribution shift in the outlier population. Using the injection strategies described earlier, we simulate shift the "shuttle" dataset where the outlier distribution gradually changes over time. Specifically, while the training and calibration sets contain Sighh-ercentile outliers, the test set progressively includes more challenging, low-percentile outliers. As shown in Figure 20, Label -rrim m maintains type-l error control throughout the distribution shift, outperforming all baselines and approaching oracle performance.
n025
Type Errror
0.20
0.15
0.10
0,008
(30-100)
(20-90)
(10-80)
(0-70)
(0-60)
(0-50)
(0-40)
(0-30)
(0-80)
(0-90)
(0-100)
(10-100)
(20-100)
(40-100)
(50-100)
(60-100)
(70-100)
Outlier test distribution (percentile range)
1.0
Standard
Oracle (infeasible)
0.8
Naive-Trim (invalid)
Power
0.
Small- Clean
Label-Trim
0.4
Target type-1 level
0.2
0.0
(30-100)
(20-90)
(10.80)
(0:70)
(0-60)
(0-50)
(0-40)
(0.30)
(0-80)
(0-90)
(8-100)
(0-100))
(20-100)
(4-1100)
(50.100)
(60-100)
(70- (on)
Outlier test distribution (percentile range)
Figure 20: Performance on real dataset "shuttle" function of drift in the outlier test distribution. Outliers in the training and calibration sets are drawn from the 30th to 100th percentile of the outlier score distribution, while the test set varies across different percentile ranges. Other details are as in Figure 3 in the main manuscript.
30

---

Robust Conformal Outlier Detection under Contaminated Reference Data
B.2.2. VISUAL DATASETS
Table 3 summarizes the results across all six datasets for target type-I error rate of 0.02, showing trends consistent with those observed in Table 1 of the main manuscript.
Table 3:
Comparison of conformal outlier detection methods on six visual datasets for varying contamination rate r and target type-I error level 0.02. The empirical type-I error values are averaged across all datasets. The empirical power presented relative to the Standard method (higher is better), and averaged across all datasets. Results averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 0.0204)
0.017 (#: 0.0004)
(= 0.0238)
0.012 0.0004)
1.0 (#: 0.0272)
0.009 (#: 0.0003)
Oracle (infeasible)
1.096 (1 0.0214)
0.021 (1 0.000S)
1.33 (1 0.025)
0.02 (11. 0.0005)
1.674 (1 0.0311)
0.02 (1: 0.0006)
Haive-Trims (invalid)
1.249 (L 0.0219)
0.026 (1 0.0005)
1.777 (1 0,0254)
0.035 (1 0.0006)
2,452 (1 0.0324)
0.044 (1 0.0007)
Small-Clean
0.819 (+ 0.0592)
0.018 (#: 0.002)
0.613 (+ 0.0753)
0.011 (# 0.0018)
0.406 (# 0.0769)
0.006 (# 0.0013)
Label-Trim
1.079 (:L: 0.0212)
0.02 (1: 0.0005)
1.23 (1 0.0247)
0.017 (#: 0.0004)
1.381 (1 0.0298)
0.014 (1 0.0005)
(a) Target type-I error rate a 0.02
We also report performance averaged across the six datasets for additional models: ReAct (Sun et al., 2021) with VGG-19 backbone and SCALE (Xu al., 2024) with ResNet-18 backbone, shown in Tables 4 and 5, respectively. These results align with our main findings and further demonstrate the effectiveness of the proposed - method.
Following this, we provide detailed results for each dataset, reporting type-I error rates and the power of each method, with the power reported relative to the Standard method (normalized to 1). The actual power of the Standard method included in the last row of each table. Specifically, Tables to 11 present the detailed per-dataset results corresponding Tables I and 3. Additionally, detailed per-dataset results for the additional models are provided in Tables 12 17 and Tables 18 to 23, corresponding to Table 4 and Table 5, respectively.
31

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 4: Comparison of conformal outlier detection methods on six visual datasets for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained VGG-19. Other details are as in Table the main manuscript.
Contamination rale
3%
5%
Method
Power
Type-1 Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (= 0,0317)
0.009 (# 0.0004)
1.0 (= 0.0377)
0.006 (# 0.0003)
1.0 (#: 0.0422)
0.004 (# 0.0002)
Oracle (infeasible)
1,095 (+ 0.033)
0.01 (+ 0.0004)
1.368 (+: 0.0418)
0.01 (+ 0.0004)
1.633 (# 0.0488)
0.009 (+ 0.0003)
Naive-Trim (invalid)
1.464 (#: 0.0373)
0,018 (#: 0.0005)
2.5 (1 0.0531)
0,031 0.0006)
3,447 0.0592)
0.04 (= 0.0008)
Small-Clean
0.0 (+
(+ 0.0)
(+ 0.0)
0.0 (+ 0.0)
0.0 0.0)
0.0 0.0)
Label-Trim
1.095 (:: 0.033)
0.01 0.0004)
1.364 (#: 0.0418)
0.01 (#: 0.0004)
1.606 (# 0.0487)
0.009 (#: 0.0003)
(a) Target type-I error rate
0.01
Contamination
1%
3%
5%
Method
Power
Type-T Firor
Power
Type-T Frror
Power
Type-I Error
Standard
1.0 (+ 0.0251)
0.018 (+ 0.0005)
1.0 (+ 0.027)
0.014 (+ 0.0004)
(+ (+ 0.0291)
0.01 (+ 0.0004)
Oracle (infeasible)
1.068 (1 0.0247)
0.021 (1: 0.0005)
1.221 (1 0.0297)
0.02 (1: 0.0005)
1.37 (1 0.0323)
0.019 (:1 0.0006)
Naive-Trim (invalid)
1.252 (# 0.0256)
0.028 (= 0.0006)
1.759 (# 0.0326)
0.039 (# 0.0007)
2.212 (# 0.0345)
0.048 (# 0.0008)
Small-Clean
0.813 (1 0.0593)
0.017 (1 0.0019)
0.593 (.L 0.0737)
0,011 0.0018)
0.352 (1 0.0663)
0.005 (.L 0.0013)
Label-Trim
1.056 (+: 0.0244)
0.02 0.0005)
1.177 0.0292)
0.019 0.0005)
1.253 0.0312)
0.016 (+ 0.0005)
(b) Target type-I error rate a
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-1 Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (1L 0.0202)
0.028 (1 0.0006)
1.0 (1 0.0238)
0.022 (1 0.0005)
1.0 (L 0.0247)
0.017 (LL 0.0005)
Oracle (infeasible)
1.074 (+1 0.02)
0.031 (# 0.0006)
1.194 (1L 0.0256)
0.03 (::: 0.0007)
1,318 (1 0.0271)
0.029 (1: 0.0007)
Naive-Trin Trim (invalid)
1.166 (+ 0.0202)
0.037 (+ 0.0007)
1.51 (+ 0.0248)
0.048 (+ 0.0008)
1.848 (+ 0.0272)
0.056 (+ 0.0009)
Small-Clean
0.709 (#: 0.045)
0.019 (= 0.0018)
0.85 (# 0.0541)
0.021 (#: 0.002)
0.932 (# 0.0568)
0.019 (#: 0.002)
Label-Trim
1.048 (1 0.0202)
0.03 (t: 0.0006)
1.118 (E 0.0248)
0.027 0.0006)
1.167 (1 0.026)
0.023 (t: 0.0006)
(c) Target type-I crror rate a 0.03
32

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 5: Comparison of conformal outlier detection methods on six visual datasets for varying contamination rate r and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with a pretrained ResNet-18, Other details are as in Table in the main manuscript.
Contamination rale
3%
5%
Method
Power
Type-1 Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (= 0,0358)
0.009 (# 0.0003)
1.0 (= 0.0375)
0.006 (# 0.0003)
1.0 (# 0.0407)
0.004 (# 0.0002)
Oracle (infeasible)
1.144 (+ 0.037)
0.01 (+ 0.0003)
1.472 (+ 0.0474)
0.01 (+ 0.0003)
1.708 (# 0.0522)
0.01 (+ 0.0004)
Naive-Trim (invalid)
1.736 (#: 0.0414)
0,017 (#: 0.0004)
2.93 0.0553)
0,028 0.0006)
4.144 0.0614)
0.038 (1 0.0007)
Small-Clean
0.0
(+ 0.0)
(+ 0.0)
0.0 (+ 0.0)
0.0 0.0)
0.0 0.0)
Label-Trim
1.144 (#: 0.0371)
0.01 0.0003)
1.456 (:: 0.0469)
0.01 (#: 0.0003)
1.627 0.0501)
0.009 (#: 0.0003)
(a) Target type-I error rate
0.01
Contamination
1%
3%
5%
Method
Power
Type-T Firor
Power
Type-T Frror
Power
Type-I Error
Standard
1.0 (+ 0.0236)
0.018 (+ 0.0004)
1.0 (+ 0.0267)
0.013 (+ 0.0004)
1.0 (+ 0.0288)
0.01 (+ 0.0003)
Oracle (infeasible)
1.076 (1 0.0244)
0.021 (1: 0.0005)
1.306 (1 0.0279)
0.02 (1: 0.0005)
1.583 (1 0.0316)
0.02 (1: 0.0005)
Naive-Trim (invalid)
1.247 (# 0.0253)
0.027 (= 0.0006)
1.86 (= 0.0294)
0.036 (# 0.0006)
2.532 (# 0.0335)
0.047 (# 0.0008)
Small-Clean
0.831 (L 0.0598)
0.018 (1 0.0019)
0.609 (.L 0.0765)
0,011 0.0017)
0.348 0.0744)
0.005 (.L 0.0014)
Label-Trim
1.062 (+ 0.0243)
0.02 0.0005)
1,226 0.0284)
0.018 0.0005)
1.368 0.0315)
0.015 (+ 0.0004)
(b) Target type-I error rate a 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (1 0.0201)
0.027 (1 0.0006)
1.0 (1 0.0212)
0.021 (1L 0.0005)
1.0 (. 0.022)
0.016 (LL 0.0004)
Oracle (infeasible)
1.082 (+1 0.0204)
0.031 (# 0.0006)
1.233 (1: 0.0227)
0.029 (1: 0.0006)
1.398 (1 0.0237)
0.03 (#: 0.0006)
Naive-Trin Trim (invalid)
1.19 (+ 0.0197)
0.036 (+ 0.0006)
1.547 (+ 0.0229)
0.044 (+ 0.0007)
1.9 (+ 0.0239)
0.055 (+ 0.0009)
Small-Clean
0.712 (+: 0,045)
0.019 (= 0.0019)
0.862 (#: 0.055)
0.02 (#: 0.002)
0.971 (# 0.0606)
0.021 (#: 0.0021)
Label-Trim
1.05 (:: 0.0204)
0.029 (t: 0.0006)
1.116 (t 0.022)
0.025 (t: 0.0006)
1.178 (1 0,0224)
0.022 (:: 0.0005)
(c) Target type-I crror rate a 0.03
33

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 6: Comparison of conformal outlier detection methods Texture dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained ResNet-18, 18. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-T Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0295)
0.008 (t. 0.0003)
1.0 (:: 0.0369)
0.006 (1: 0.0003)
1.0 CE 0.0405)
0.004 (1: 0.0002)
Oracle (infcasible)
1.173 (+ 0.0295)
0.01 0.0003)
1.455 0.0432)
0.01 (# 0.0003)
1.824 0.051)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.722 (+ 0.035)
0.017 (= 0.0004)
2.668 0.0428)
0.028 0.0006)
4.008 0.0599)
0.037 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
0.0 (1L 0.0)
0.0 0.0)
Label- Trim
1,173 (+ 0.0295)
0.01 (+ 0.0003)
1.448 (+ 0.0428)
0.01 (+ 0.0003)
1.73 (+ 0.0477)
0.009 (+ 0.0003)
Standard Power
0.194 (+ 0.0057)
0.159 (+ 0.0059)
0.123 (+ 0.005)
Target type-I rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0202)
0.017 (+ 0.0005)
1.0 (+ 0.0239)
0.012 (+ 0.0004)
1.0 0.0264)
0.009 (+ 0.0003)
Oracle (infcasible)
1.109 (+: 0.0199)
0.021 (= 0.0005)
1.315 0.0259)
0.02 (#: 0.0005)
1.623 0.0346)
0.02 (=: 0.0006)
Naive-Trim (invalid)
(t: 0.0193)
0.026 (t: 0.0006)
1.781 0.026)
0.035 0.0007)
2.431 0.0334)
0.045 (t: 0.0007)
Small-Clean
0.819 (+ 0.0599)
0.018 (#: 0.002)
0.702 (#: 0.0824)
0.014 0.0025)
0.478 (# 0.0842)
0.007 (#: 0.0016)
Label- Trim
1.089 (LL 0.0199)
0.02 (L 0.0005)
1.219 (1 0.0253)
0.017 (L 0.0004)
1,353 (L 0.0303)
0.014 (. 0.0005)
Standard Power
0.337 (#: 0.0068)
0.272 (+: 0.0065)
0.224 (# 0.0059)
Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (L 0.0155)
0.027 (1 0.0006)
1.0 (L 0.0197)
0.02 (.L. 0.0005)
1.0 (L 0.0215)
0.015 (. 0.0005)
Oracle (infeasible)
1.068 (+ 0.0152)
0.03 0.0006)
1,224 0.0191)
0.03 (+ 0.0006)
1.427 0.0249)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.15 (1: 0.0152)
0.036 (# 0.0006)
1.514 (1: 0.0178)
0.043 0.0007)
1.919 (1: 0.0229)
0.052 (1: 0.0008)
Small-Clean
0.743 (#: 0.0441)
0.02 0.002)
0.883 (#: 0.0549)
0.021 0.0025)
1.062 (1: 0.0665)
0.024 (#: 0.0027)
Label Trim
1,046 (t: 0.0153)
0.029 (+: 0.0006)
1.13 ( 0.019)
0.025 (+: 0.0005)
1.215 (: 0.0245)
0.021 (t: 0.0005)
Standard Power
0.421 (t: 0.0065)
0.357 (t: 0.007)
0.309 (t: 0.0067)
Target type-I error rate 0.03
34

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 7: Comparison of conformal outlier detection methods on SVHN dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained ResNet-18, 18. The empirical power is presented relative to the St anddardd method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-T Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0255)
0.008 (t: 0.0003)
1.0 (:: 0.0316)
0.004 0.0002)
1.0 (t: 0.0388)
0.002 (1: 0.0002)
Oracle (infcasible)
1.192 (# 0.0262)
0.01 0.0003)
L664 0.0368)
0.01 (# 0.0003)
2.208 0.052)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.573 (+ 0.0271)
0.016 (+ 0.0004)
2.689 (+ 0.0353)
0.025 0.0006)
4.073 0.0533)
0.033 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
0.0 O.L 0.0)
(.0 0.0)
Label- Trim
1.192 (+ 0.0262)
0.01 (+ 0.0003)
1.601 (+ 0.0367)
0.009 (+ 0.0003)
1,895 (+ 0.0448)
0.007 (+ 0.0003)
Standard Power
0.271 (+ 0.0069)
0.191 (+ 0.006)
0.137 (+ 0.0053)
Target type-I
0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0168)
0.016 (+ 0.0004)
1.0 (+ 0.0218)
0.01 (+ 0.0003)
1.0 0.0245)
0.007 (+ 0.0003)
Oracle (infcasible)
1.096 (+: 0.0173)
0.021 (= 0.0005)
1.404 0.0211)
0,02 (#: 0,0005)
1.828 0.0304)
0.02 (= 0.0006)
Naive-Trim (invalid)
1.2 0.0175)
0.026 (t: 0.0005)
1.721 0.0216)
0.032 0.0006)
2.42 (:: 0.0281)
0.041 (t: 0.0007)
Small-Clean
0,833 (+ 0.053)
0.02 (#: 0.0026)
0.598 (#: 0.0754)
0.01 (#: 0.0016)
0.509 (# 0.0929)
0.008 (#: 0.0019)
Label- Trim
1.08 (1 0.0176)
0.02 (L 0.0005)
1.277 (1L 0.0228)
0.016 (L 0.0004)
1.469 (L 0.0288)
0.013 (. 0.0004)
Standard Power
0.429 (+: 0.0072)
0.332 (#: 0.0072)
0.25 (= 0.0061)
Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (LL 0.0145)
0.026 (1 0.0005)
1.0 (L 0.0169)
0.017 0.0004)
1.0 (L 0.0204)
0.012 (. 0.0004)
Oracle (infeasible)
1.063 (+: 0.014)
0.03 0.0006)
1,246 0.0158)
0.029 0.0006)
1,513 0.0212)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.115 (#: 0.0137)
0.035 (1: 0.0006)
1.395 (1: 0.0148)
0.041 0.0007)
1.825 (1: 0.0199)
0.049 (1: 0.0008)
Small-Clean
0.73 (:: 0.0419)
0.021 (#: 0.0027)
0.948 (#: 0.0447)
0.022 0.0019)
1.092 (1: 0.0625)
0.022 (1: 0.0023)
Label Trim
1.042 (t: 0.0143)
0.029 (+: 0.0006)
1.144 (+ 0.0151)
0.024 (+: 0.0005)
1.266 (+ 0.0209)
0.019 (t: 0.0006)
Standard Power
0.517 (:: 0.0075)
0.44(1 0.0074)
0.355 (t: 0.0072)
Target type-I error rate 0.03
35

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 8: Comparison of conformal outlier detection methods on Places365 dataset (outliers) and CIFAR-10 10 dataset (inliers) for varying contamination rate r and target type-I error level o. All methods utilize the ReAct (Sun et al., 2021) method with pretrained ResNet-18. The empirical power is presented relative to the Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,033)
0.009 (t: 0.0003)
1.0 C:: 0.0364)
0.006 (1: 0.0003)
1.0 CE: 0.0431)
0.004 (1: 0.0002)
Oracle (infcasible)
1.139 (+ 0.0339)
0.01 0.0003)
1.47 0.0423)
0.01 (# 0.0003)
1.839 0.0558)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.624 (+ 0.0339)
0.017 (+= 0.0004)
2.73 0.0487)
0.028 0.0006)
4.12 (# 0.069)
0.039 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
O.L 0.0)
0.0 0.0)
Label-Trine Trim
1,139 (+ 0.0339)
0.01 (+ 0.0003)
1.466 (+ 0.0421)
0.01 (+ 0.0003)
1,707 (+ 0.0531)
0.009 (+ 0.0003)
Standard Power
0.193 (+ 0.0064)
0.148 (+ 0.0054)
0.115 (+ 0.005)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0209)
0.017 (+ 0.0005)
1.0 (+ 0.0241)
0.013 (+ 0.0004)
1.0 0.0298)
0.009 (+ 0.0003)
Oracle (infcasible)
1.091 (+: 0.0224)
0.021 (= 0.0005)
1,271 (#: 0.0258)
0.02 (#: 0,0005)
1.599 0,0308)
0.02 (= 0.0006)
Naive-Trim (invalid)
1.246 (+: 0.0225)
0.027 (t: 0.0006)
1,753 (t: 0.0277)
0.036 0.0007)
2.462 0.0376)
0.046 (1: 0.0007)
Small-Clean
0.799 (+ 0.0632)
0.018 (#: 0.0021)
0.549 (#: 0.0716)
0.01 (#: 0.0016)
0.255 (# 0.0595)
0.003 (#: 0.0009)
Label- Trim
1.071 (LL 0.0219)
0.02 (L 0.0005)
1.187 (LL 0.0247)
0.017 (L 0.0004)
1,361 (L 0.0306)
0.015 (.1 0.0005)
Standard Power
0.316 (+: 0.0066)
0.261 (+: 0.0063)
0.213 (# 0.0064)
Target type-I error rate 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (LL 0.018)
0.027 (1 0.0006)
1.0 (L 0.0204)
0.02 (.L. 0.0005)
1.0 (L 0.021)
0.015 (. 0.0005)
Oracle (infeasible)
1.055 (+ 0.0189)
0.03 0.0006)
1,231 0.0213)
0.029 0.0006)
1.403 0.025)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.145 (#: 0.0199)
0.036 (1: 0.0006)
1.496 (1: 0.0217)
0.044 0.0007)
1.883 (1: 0.0261)
0.054 (1: 0.0008)
Small-Clean
0.746 (#: 0.0469)
0.021 (#: 0.002)
0.833 (#: 0.0504)
0.018 0.0018)
1.029 (1: 0.0593)
0.022 (1: 0.0022)
Label Trim
1,038 (t: 0.0188)
0.029 (+: 0.0006)
1.149 (+ 0.0211)
0.025 (+: 0.0006)
1.186 (+ 0.0228)
0.022 (t: 0.0006)
Standard Power
0.395 (:: 0,007
0.335 (t: 0.0068)
0.302 (t: 0.0063)
Target type-I error rate 0.03
36

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 9: Comparison of conformal outlier detection methods MNIST dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained ResNet-18, 18. The empirical power is presented relative to the St ataddard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0248)
0.008 (t: 0.0003)
1.0 (:: 0.0283)
0.004 (1: 0.0002)
1.0 CE: 0.0363)
0.003 (1: 0.0002)
Oracle (infcasible)
1.241 (+ 0.0291)
0.01 0.0003)
1.758 0.0352)
0.01 (# 0.0003)
2.554 0.0601)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.629 (+ 0.0292)
0.016 (+ 0.0004)
2.722 (+: 0.0363)
0.023 0.0005)
4.718 0.0552)
0.03 (+ 0.0006)
Small-Clean
0.0 0.0)
(1 0.0)
(d:: 0.0)
0.0 0.0)
O.L 0.0)
0.0 (= 0.0)
Label- Trim
1,241 (+ 0.0291)
0.01 (+ 0.0003)
1.636 (+ 0.0331)
0.009 (+ 0.0003)
2,113 (+ 0.0555)
0.007 (+ 0.0003)
Standard Power
0.282 (+ 0.007)
0,208 (+ 0.0059)
0.131 (+ 0.0048)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0173)
0.016 (+ 0.0004)
1.0 (+ 0.0205)
0.01 (+ 0.0003)
1.0 0.027)
0.007 (+ 0.0003)
Oracle (infcasible)
1.117 (+: 0.0179)
0.021 (= 0.0005)
1.452 (#: 0.0209)
0.02 (#: 0,0005)
1.971 0.0297)
0.02 (= 0.0006)
Naive-Trim (invalid)
1.231 (t: 0.0179)
0.025 (t: 0.0005)
1.759 0.0207)
0.031 0.0006)
2.531 0.0273)
0.038 (t: 0.0007)
Small-Clean
0,844 (+ 0.0514)
0.019 (#: 0.0019)
0.617 (#: 0.0738)
0.01 (#: 0.0017)
0.438 (# 0.0817)
0.005 (#: 0.0011)
Label- Trim
1.096 (LL 0.0181)
0.02 (L 0.0005)
1.308 (.L 0.0206)
0.015 (L 0.0004)
1.493 (LI 0.0295)
0.012 (.1 0.0004)
Standard Power
0.465 (#: 0.008)
0.36 (# 0.0074)
0.264 (# 0.0071)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (L 0.0147)
0.025 (1 0.0005)
1.0 (L 0.0151)
0.016 0.0004)
1.0 (L 0.0201)
0.011 (.L 0.0004)
Oracle (infeasible)
1.072 (+ 0.0149)
0.03 0.0006)
1,295 0.0155)
0.029 0.0006)
1.617 0.0186)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.116 (1: 0.0144)
0.034 (1 0.0006)
1.424 (1: 0.0142)
0.039 0.0007)
1.851 (1: 0.0178)
0.046 (1: 0.0007)
Small-Clean
0.699 (#: 0.0394)
0.019 (#: 0.0019)
0.884 (#: 0.0464)
0.019 0.0018)
1.164 (1: 0.0576)
0.02 (# 0.002)
Label Trim
1.049 (t: 0.0148)
0.029 (+: 0.0006)
1.158 (+ 0.016)
0.023 (t: 0.0005)
1.273 (+ 0.0218)
0.017 (t: 0.0005)
Standard Power
0.576 (::: 0.0085)
0.483 (t: 0.0073)
0.382 (t: 0.0077)
Target type-I error rate 0.03
37

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 10: Comparison of conformal outlier detection methods CIFAR-100 dataset (outliers) and CIFAR-10 dataset
(inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained ResNet-18. The empirical power is presented relative to the St andard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0393)
0.009 (t: 0.0003)
1.0 (:: 0.0391)
0.007 (1: 0.0003)
1.0 (E: 0.042)
0.005 (1: 0.0003)
Oracle (infcasible)
1.116 (+ 0.0417)
0.01 0.0003)
1.463 0.05)
0.01 (# 0,0003)
1,588 0.0477)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.733 (+ 0.0397)
0.018 (+ 0.0005)
3.06 0.0585)
0.03 (+ 0.0006)
4.054 0.0631)
0.041 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.1 0.0)
0.0 0.0)
(.0 0.0)
Label- Trim
1.116 (+ 0.0417)
0.01 (+ 0.0003)
1.463 (+ 0.05)
0.01 (+ 0.0003)
1,587 (+ 0.0468)
0.009 (+ 0.0004)
Standard Power
0.145 (+ 0.0057)
0.118 (+ 0.0046)
0.104 (+ 0.0044)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,023)
0.018 (+ 0.0005)
1.0 (+ 0.0265)
0.014 (+ 0.0004)
1.0 0.0264)
0.011 (+ 0.0004)
Oracle (infcasible)
1.082 (+: 0.0256)
0.021 (= 0.0005)
1,265 0.0299)
0,02 (#: 0,0005)
1.476 0.0317)
0.02 (# 0.0006)
Naive-Trim (invalid)
1.294 (t: 0.0266)
0.027 (t: 0.0006)
1.831 0.0288)
0.038 0.0006)
2.464 0.0372)
0.049 (t: 0.0008)
Small-Clean
0,844 (+ 0.0709)
0.019 (#: 0.0022)
0.686 (#: 0.0797)
0.012 (# 0.0017)
0.392 (# 0.078)
0.006 (#: 0.0014)
Label- Trim
1.064 (L 0.0247)
0.02 (L 0.0005)
1.193 (.L 0.0293)
0.018 (L 0.0004)
1,288 (LL 0.0296)
0.016 (.L 0.0005)
Standard Power
0.255 (#: 0.0059)
0.225 (+: 0.006)
0.19 (= 0.005)
Target type-I error rate 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (1L 0.0204)
0.027 (1 0.0006)
1.0 (L 0.0214)
0.022 0.0005)
1.0 (L 0.0228)
0.018 (. 0.0005)
Oracle (infeasible)
1.054 (+: 0.0212)
0.031 (+ 0.0006)
1.184 0.0228)
0.03 (+ 0.0006)
1.33 (= 0.0246)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.184 (#: 0.0213)
0.036 (1: 0.0006)
1.523 (1: 0.0222)
0.046 0.0007)
1.94 (1: 0.026)
0.056 (1: 0.0009)
Small-Clean
0.709 (#: 0.0527)
0.021 (#: 0.0023)
0.842 (#: 0.0545)
0.021 0.0019)
0.947 (#: 0.0628)
0.02 (# 0.002)
Label Trim
1.029 (+: 0.0212)
0.029 (+: 0.0006)
1.114 (+ 0.0222)
0.026 (+: 0.0006)
1.16 (::: 0.0229)
0.023 (t: 0.0006)
Standard Power
0.332 (::: 0.0068)
0.302 (t: 0.0064)
0.263 (t: 0.006)
Target type-I error rate 0.03
38

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 11: Comparison of conformal outlier detection methods TinyImageNet dataset (outliers) and CIFAR-I0 dataset
(inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained ResNet-18. The empirical power is presented relative to the St anddrd method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0381)
0.009 (t: 0.0003)
1.0 (:: 0.0398)
0.006 (1: 0.0003)
1.0 CE 0.0439)
0.004 (1: 0.0002)
Oracle (infcasible)
1.137 (+ 0.0409)
0.01 0.0003)
1.492 0.0474)
0.01 (# 0.0003)
1.754 0.0518)
0.009 (# 0.0004)
Naive-Trim (invalid)
1.675 (+ 0.0401)
0.018 (+ 0.0004)
2.872 (+: 0.0485)
0.028 0.0006)
3.986 0.0568)
0.039 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
O.0 0.0)
(.0 0.0)
Label- Trim
1,137 (+ 0.0409)
0.01 (+ 0.0003)
1.488 (+ 0.0472)
0.01 (+ 0.0003)
1.681 (+ 0.0511)
0.009 (+ 0.0003)
Standard Power
0.168 (+ 0.0064)
0.133 (+ 0.0053)
0.115 (+ 0.005)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0239)
0.018 (+ 0.0004)
1.0 (+ 0.0262)
0.013 0.0004)
1.0 (+ 0.0293)
0.01 (+ 0.0004)
Oracle (infcasible)
1.084 (+: 0.0253)
0.021 (= 0.0005)
1,275 (#: 0.0263)
0.02 (#: 0.0005)
1.544 0.0293)
0.02 (# 0.0006)
Naive-Trim (invalid)
1.284 (t: 0.0274)
0.027 (t: 0.0006)
1.819 (t: 0.0277)
0.036 0.0006)
2.406 0.0308)
0.046 (1: 0.0008)
Small-Clean
0.777 (+ 0.0566)
0.015 (# 0.0015)
0.525 (#: 0.0687)
0.009 (#: 0.0015)
0.366 (# 0.0653)
0.005 (#: 0.001
Label- Trim
1.074 (L 0.0249)
0.02 (L 0.0005)
1.193 (1L 0.0254)
0.017 (L 0.0004)
1.322 (1 0.03)
0.015 (.1 0.0005)
Standard Power
0.285 (#: 0.0068)
0.243 (+: 0.0064)
0.209 (# 0.0061)
Target type-I error rate 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (LL 0.0213)
0.027 (1 0.0006)
1.0 (L 0.0202)
0.02 (.L. 0.0005)
1.0 (L 0.0216)
0.016 (1 0.0005)
Oracle (infeasible)
1.062 (+: 0.0216)
0.03 0.0006)
1,23 0.0205)
0.03 (+: 0.0006)
1.397 0.0236)
0.03 (+ 0.0007)
Naive-Trim (invalid)
1.168 (#: 0.0203)
0.036 (1: 0.0006)
1.569 (1: 0.0207)
0.045 0.0007)
1.877 (1: 0.0217)
0.054 (1: 0.0009)
Small-Clean
0.656 (#: 0,0438)
0.017 (#: 0.0017)
0.825 (#: 0.0496)
0.019 0.0019)
0.904 (+: 0.0589)
0.02 (#: 0.0024)
Label Trim
1.04 (+: 0.0219)
0.029 (+: 0.0006)
1.139 (+ 0.0208)
0.026 (+: 0.0006)
1.19 (:: 0.0225)
0.022 (t: 0.0006)
Standard Power
0.368 (::: 0,0078)
0.318 (t: 0.0064)
0.29 (:: 0.0063)
Target type-I error rate 0.03
39

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 12: Comparison of conformal outlier detection methods Texture dataset (outliers) and CIFAR-10 10 dataset (inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with pretrained VGG-19. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-T Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.026)
0.008 (t: 0.0003)
1.0 (:: 0.0312)
0.005 0.0002)
1.0 CE: 0.0342)
0.003 (1: 0.0002)
Oracle (infcasible)
1.119 (# 0.0263)
0.01 0.0004)
1.421 0.032)
0.01 (# 0.0004)
1.833 0.0437)
0.009 (# 0.0003)
Naive-Trim (invalid)
1.368 (+ 0.0278)
0.017 (= 0.0005)
2.168 0.0379)
0.029 0.0006)
3.383 0.0481)
0.037 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
O.L 0.0)
(.0 0.0)
Label- Trim
1.119 (+ 0.0263)
0.01 (+ 0.0004)
1.401 (+ 0.0321)
0.01 (+: 0.0003)
1.712 (+ 0.0429)
0.008 (+ 0.0003)
Standard Power
0.239 (+ 0.0062)
0.008 (+ 0.0003)
0.188 (+ 0.0059)
0.005 (+ 0.0002)
0.139 (+ 0.0048)
0.003 (+ 0.0002)
Target type-I error rate 0.01
Contamination rate
1% Type-1 Liror
3%
5%
Method
Power
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0202)
0.017 (+ 0.0005)
1.0 (+ 0.0223)
0.012 (+ 0.0004)
1.0 0.0249)
0.008 (+ 0.0003)
Oracle (infcasible)
1.083 (+: 0.0186)
0.021 (= 0.0005)
1,253 (#: 0.0246)
0,02 (#: 0.0005)
1.502 0.0292)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.223 (t: 0.0201)
0.027 (t: 0.0006)
1.605 0.0241)
0.037 0.0007)
2.15 (:: 0.0295)
0.045 (1: 0.0008)
Small-Clean
0,841 (+ 0.052)
0.018 (#: 0.0018)
0.752 (#: 0.0805)
0.016 0.0025)
0.437 (# 0.071)
0.006 (#: 0.0012)
Label- Trim
1.072 (L 0.0185)
0.02 (LL 0.0005)
1.2 (1 0.0234)
0.018 (L 0.0005)
1.323 (LI 0.0276)
0.015 (.L 0.0005)
Standard Power
0.331 (+: 0.0067)
0.017 (== 0.0005)
0.283 (#: 0.0063)
0.012 (+ 0.0004)
0.237 (# 0.0059)
0.008 (#: 0.0003)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (L 0.0164)
0.027 (1 0.0006)
1.0 (L 0.02)
0.02 (.L. 0.0005)
1.0 0.0202)
0.015 (.L. 0.0005)
Oracle (infeasible)
1.075 (+: 0.0153)
0.031 (# 0.0006)
1.179 0.0201)
0.03 (+: 0.0007)
1.356 0.022)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.146 (1: 0.0155)
0.036 (#: 0.0007)
1.408 (1: 0.0187)
0.045 0.0008)
1.75 (#: 0.0228)
0.053 (1: 0.0009)
Small-Clean
0,801 (#: 0,0382)
0.022 (#: 0.0018)
0.937 (#: 0.0569)
0.025 0.0028)
1.016 (1: 0.0548)
0.02 (# 0.0021)
Label Trim
1,053 (t: 0.016)
0.03 (+: 0.0006)
1.117 (+ 0.02)
0.027 (+: 0.0006)
1.208 (+ 0.0221)
0.022 (+: 0.0006)
Standard Power
0.408 (t: 0.0067)
0.027 (t: 0.0006)
0.353 (t: 0.0071)
0.02 (::: 0.0005)
0.312 (t: 0.0063)
0.015 (t: 0.0005)
Target type-I error rate 0.03
40

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 13: Comparison of conformal outlier detection methods on SVHN dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level o. All methods utilize the ReAct (Sun et al., 2021) method with pretrained VGG-19. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0297)
0.009 (t: 0.0003)
1.0 (:: 0.0355)
0.006 (1: 0.0003)
(E 0.0463)
0.004 (1: 0.0002)
Oracle (infcasible)
1,085 (+ 0.0305)
0.01 0.0004)
1,287 0.0384)
0.01 (# 0.0004)
1.607 0.0452)
0.009 (# 0.0003)
Naive-Trim (invalid)
1.355 (+ 0.0356)
0.018 (+ 0.0005)
2.261 0.05
0.031 0.0006)
3.202 0.053)
0.04 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 (1: 0.0)
0.0 0.0)
0.0 0.0)
0.0 0.0)
Label- Trim
1,085 (+ 0.0305)
0.01 (+ 0.0004)
1,287 (+ 0.0384)
0.01 (+: 0.0004)
1,573 (+ 0.0467)
0.009 (+ 0.0003)
Standard Power
0.182 (+ 0.0054)
0.009 (+ 0.0003)
0.145 (+ 0.0052)
0.006 (+ 0.0003)
0.12 (+ 0.0056)
0.004 (+ 0.0002)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.026)
0.018 (+ 0.0005)
1.0 (+ 0.0274)
0.014 (+ 0.0004)
1.0 (+ 0.0285)
0.01 (+ 0.0003)
Oracle (infcasible)
1,038 (+: 0.026)
0.021 (= 0.0005)
1.167 0.0298)
0.02 (#: 0.0005)
1.289 0.0294)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.222 (t: 0.0269)
0.028 (t: 0.0006)
1.77 0.0354)
0.039 0.0007)
2.198 0.0315)
0.048 (1: 0.0008)
Small-Clean
0,894 (+ 0.0609)
0.019 (#: 0.0019)
0.504 (#: 0.07)
0.009 (#: 0.0015)
0.389 (# 0.0722)
0.006 (#: 0.0015)
Label- Trim
1.035 (LL 0.0256)
0.02 (LL 0.0005)
1.146 (.L 0.0293)
0.019 (1 0.0005)
1,217 (L 0.0301)
0.017 (.L 0.0005)
Standard Power
0,249 (+: 0.0065)
0.018 (== 0.0005)
0,216 (#: 0.0059)
0.014 (# 0.0004)
0.196 (# 0.0056)
0.01 (# 0.0003)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0219)
0.028 (1 0.0006)
1.0 (L 0.0251)
0.023 0.0005)
1.0 (L 0.0248)
0.018 (.L 0.0005)
Oracle (infeasible)
1.074 (+: 0.0225)
0.031 (# 0.0006)
1,206 0.0281)
0.03 (+ 0.0007)
1.302 0.0261)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.224 (1: 0.0216)
0.037 (# 0.0007)
1.626 (1: 0.0277)
0.048 0.0008)
1.962 (1: 0.0257)
0.056 (1: 0.0009)
Small-Clean
0.788 (#: 0,0485)
0.02 0.0019)
0.878 (#: 0.0536)
0.019 0.0018)
0.986 (+: 0.0625)
0.02 (# 0.0021)
Label Trim
1,05 (+t 0.0224)
0.03 (+: 0.0006)
1.134 (+ 0.0265)
0.028 (+: 0.0006)
1.16 (+: 0.0247)
0.024 (+: 0.0006)
Standard Power
0.307 (::: 0.0067)
0.028 (t: 0.0006)
0.266 (t: 0.0067)
0.023 (t: 0.0005)
0.241 (t: 0.006)
0.018 (:: 0.0005)
Target type-I error rate 0.03

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 14: Comparison of conformal outlier detection methods on Places365 dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level o. All methods utilize the ReAct (Sun et al., 2021) method with pretrained VGG-19. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0289)
0.009 (t: 0.0004)
1.0 (:: 0.0383)
0.006 (1: 0.0003)
1.0 (E: 0.0382)
0.005 (1: 0.0003)
Oracle (infcasible)
1.084 (+: 0.0295)
0.01 0.0004)
1.444 0.0407)
0.01 (# 0.0004)
1.602 0.0488)
0.009 (# 0.0003)
Naive-Trim (invalid)
1.454 (+ 0.0346)
0.018 (+ 0.0005)
2.623 (+ 0.0521)
0.03 (# 0.0006)
3.405 0.0592)
0.04 (+ 0.0008)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
O.0 0.0)
0.0 0.0)
Label-Trine Trim
1,084 (+ 0.0295)
0.01 (+ 0.0004)
1.441 (+ 0.0404)
0.01 (+: 0.0003)
1,578 (+ 0.0476)
0.009 (+ 0.0003)
Standard Power
0.176 (+ 0.0051)
0.009 (+ 0.0004)
0.134 (+ 0.0051)
0.006 (+ 0.0003)
0.12 (+ 0.0046)
0.005 (+ 0.0003)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0236)
0.018 (+ 0.0005)
1.0 (+ 0.0251)
0.014 (+ 0.0004)
1.0 (+ 0.0296)
0.01 (+ 0.0004)
Oracle (infcasible)
1.074 (+: 0.0238)
0.021 (= 0.0005)
1,247 0.0291)
0.02 (#: 0.0005)
1.35 (= 0.0332)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.236 (+: 0.0236)
0.028 (t: 0.0006)
1.781 0.0308)
0.039 0.0007)
2.666 0.0346)
0.048 (E: 0.0009)
Small-Clean
0.71 (#: 0.0577)
0.015 (#= 0.0017)
0.569 (#: 0.0713)
0.011 (# 0.0016)
0.268 (# 0.0606)
0.004 (#: 0.0013)
Label- Trim
1.058 (LL 0.0241)
0.02 (LL 0.0005)
1.198 (L 0.0281)
0.019 (1 0.0005)
1,231 (L 0.0315)
0.016 (.L 0.0005)
Standard Power
0.259 (+: 0.0061)
0.018 (== 0.0005)
0.226 (#: 0.0057)
0.014 (# 0.0004)
0.205 (# 0.0061)
0.01 (# 0.0004)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0188)
0.028 (1 0.0006)
1.0 (L 0.0221)
0.022 0.0005)
1.0 (L 0.0261)
0.017 (.L. 0.0005)
Oracle (infeasible)
1,088 (+ 0.0187)
0.031 (# 0.0006)
1.197 0.0233)
0.03 (+ 0.0007)
1.32 (= 0.0288)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.186 (1: 0.0185)
0.037 (#: 0.0007)
1.499 (1: 0.0249)
0.048 0.0008)
1.83 (#: 0.026)
0.056 (1: 0.0009)
Small-Clean
0.652 (#: 0.045)
0.017 (#: 0.0017)
0.85 (# 0.0541)
0.021 0.0018)
0.921 (#: 0.054)
0.02 (# 0.002)
Label Trim
1,056 (+: 0.019)
0.03 (+: 0.0006)
1.12 ( 0.0232)
0.027 (+: 0.0006)
1.168 (+ 0.0276)
0.023 (+: 0.0006)
Standard Power
0.321 (t: 0.006)
0.028 (t: 0.0006)
0.294 (t: 0.0065)
0.022 (t: 0.0005)
0.263 (t: 0.0069)
0.017 (t: 0.0005)
(c) Target type-I error rate 0.03
42

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 15: Comparison of conformal outlier detection methods MNIST dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate r and target type-I error level o. All methods utilize the ReAct (Sun et al., 2021) method with pretrained VGG-19. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0315)
0.009 (t: 0.0004)
1.0 (:: 0.0376)
0.007 (1: 0.0003)
1.0 CE: 0.0428)
0.005 (1: 0.0003)
Oracle (infcasible)
1,089 (+ 0.0325)
0.01 0.0004)
1.349 0.0414)
0.01 (# 0.0004)
1.547 0.0488)
0.009 (# 0.0003)
Naive-Trim (invalid)
1.493 (+ 0.0352)
0.018 (+ 0.0005)
2.499 (+ 0.0571)
0.031 0.0006)
3.315 0.0664)
0.041 (+ 0.0008)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
0.0 0.0)
(.0 0.0)
Label- Trim
1,089 (+ 0.0325)
0.01 (+ 0.0004)
1.347 (+ 0.0414)
0.01 (+: 0.0004)
1.549 (+ 0.0492)
0.009 (+ 0.0003)
Standard Power
0.162 (+ 0,005
0.009 (+ 0.0004)
0.134 (+ 0.005)
0.007 (+ 0.0003)
0.116 (+ 0.005)
0.005 (+ 0.0003)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0234)
0.018 (+ 0.0005)
1.0 (+ 0.0283)
0.014 (+ 0.0004)
1.0 (+ 0.0298)
0.01 (+ 0.0004)
Oracle (infcasible)
1.068 (+: 0.0225)
0.021 (= 0.0005)
1.219 0.0313)
0.02 (#: 0.0005)
1.36 (= 0.0343)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.246 (+: 0.0245)
0.028 (t: 0.0006)
1.76 0.0341)
0.039 0.0007)
2.167 0.0395)
0.049 (t: 0.0009)
Small-Clean
0.781 (+ 0.0592)
0.017 (#: 0.0019)
0.617 (#: 0.0743)
0.012 (#: 0.0018)
0.302 (# 0.0582)
0.005 (#: 0.0011)
Label- Trim
1.053 (LL 0.0224)
0.02 (LL 0.0005)
1.176 (1L 0.0318)
0.019 (1 0.0005)
1.242 (1 0.0326)
0.016 (. 0.0005)
Standard Power
0,244 (#: 0.0057)
0.018 (== 0.0005)
0.214 (#: 0.0061)
0.014 (# 0.0004)
0.194 (# 0.0058)
0.01 (# 0.0004)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0195)
0.028 (1 0.0006)
1.0 (L 0.0247)
0.023 0.0006)
1.0 (L 0.0262)
0.018 (.L 0.0005)
Oracle (infeasible)
1.062 (+ 0.0196)
0.031 (# 0.0006)
1.189 0.0279)
0.03 (+: 0.0007)
1.296 0.0305)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.178 (1: 0.0208)
0.037 (# 0.0007)
1.501 (1: 0.0257)
0.049 0.0008)
1.799 (1: 0.0319)
0.057 (1: 0.0009)
Small-Clean
0.648 (#: 0.0456)
0.017 (#: 0.0018)
0.888 (#: 0.0515)
0.023 0.002)
0.964 (# 0.0563)
0.021 (#: 0.0018)
Label Trim
1.039 (+: 0.0196)
0.03 (+: 0.0006)
1.109 (+ 0.0258)
0.027 (+: 0.0006)
1.145 (+ 0.0268)
0.024 (+: 0.0006)
Standard Power
0.305 (:: 0.0059)
0.028 (:: 0.0006)
0.276 (t: 0.0068)
0.023 (t: 0.0006)
0.25 (:: 0.0066)
0.018 (t: 0.0005)
(c) Target type-I error rate 0.03
43

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 16: Comparison of conformal outlier detection methods CIFAR-100 dataset (outliers) and CIFAR-10 dataset
(inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained VGG-19, The empirical power is presented relative to the Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0388)
0.009 (t: 0.0004)
1.0 (:: 0.0466)
0.007 (1: 0.0003)
1.0 CE: 0.0472)
0.005 (1: 0.0003)
Oracle (infcasible)
1,089 (+: 0.0409)
0.01 0.0004)
1.34 0.0508)
0.01 (# 0.0004)
1.554 0.0539)
0.009 (# 0.0003)
Naive-Trim (invalid)
1.558 (+ 0.0455)
0.018 (= 0.0005)
2.816 (+: 0.0647)
0.032 0.0007)
3.58 (# 0.0691)
0.042 (+ 0.0008)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
O.L 0.0)
0.0 0.0)
Label-Trine Trim
1,089 (+ 0.0409)
0.01 (+ 0.0004)
1.34 (+ 0.0508)
0.01 (+: 0.0004)
1.572 (+ 0.053)
0.009 (+ 0.0003)
Standard Power
0.137 (+ 0.0053)
0.009 (+ 0.0004)
0.111 (+ 0.0052)
0.007 (+ 0.0003)
0.1 (+ 0.0047)
0.005 (+ 0.0003)
Target type-I error
0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0288)
0.019 (+ 0.0005)
1.0 (+ 0.0308)
0.015 (+ 0.0005)
1.0 0.0307)
0.012 (+ 0.0004)
Oracle (infcasible)
1.07 0.0289)
0.021 (= 0.0005)
1,219 0.0332)
0,02 (#: 0,0005)
1.316 0.0362)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.293 (t: 0.0301)
0.028 (t: 0.0006)
1.811 0.0395)
0.04 (:: 0.0007)
2.24 (::: 0.0396)
0.051 (t: 0.0009)
Small-Clean
0,831 (+ 0.0636)
0.018 (#: 0.002)
0.646 (#: 0.0826)
0.013 (#: 0.0021)
0.317 (# 0.0665)
0.005 (#: 0.0016)
Label- Trim
1.057 (LL 0.0281)
0.02 (LL 0.0005)
1.17 (L 0.0327)
0.019 (L 0.0005)
1,233 (LI 0.0343)
0.017 (.L 0.0006)
Standard Power
0.217 (+: 0.0063)
0.019 (== 0.0005)
0.192 (#: 0.0059)
0.015 (+: 0.0005)
0.175 (# 0.0054)
0.012 (# 0.0004)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.023)
0.028 (1 0.0006)
1.0 (L 0.0272)
0.023 0.0006)
1.0 (L 0.0269)
0.019 (.L 0.0005)
Oracle (infeasible)
1.061 (+: 0.0233)
0.031 (# 0.0006)
1.183 0.0278)
0.03 (+ 0.0007)
1.292 0.0306)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.184 (1: 0.0249)
0.038 (1: 0.0007)
1.512 (1: 0.0294)
0.05 (1: 0.0008)
1.868 (1: 0.0324)
0.059 (1: 0.0009)
Small-Clean
0.706 (#: 0,0461)
0.02 0.002)
0.816 (#: 0.0583)
0.021 0.0022)
0.856 (+: 0.0579)
0.018 (#: 0.0021)
Label Trim
1,039 (t: 0.0226)
0.03 (+: 0.0006)
1.111 (+ 0.0275)
0.028 (+: 0.0006)
1.154 (+ 0.0291)
0.024 (+: 0.0006)
Standard Power
0.282 (+: 0.0065)
0.028 (t: 0.0006)
0.254 (t: 0.0069)
0.023 (t: 0.0006)
0.228 (t: 0.0062)
0.019 (:: 0.0005)
Target type-I error rate 0.03
44

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 17: Comparison of conformal outlier detection methods TinyImageNet dataset (outliers) and CIFAR-I0 dataset
(inliers) for varying contamination rate r and target type-I error level a. All methods utilize the ReAct (Sun et al., 2021) method with a pretrained VGG-19, 19. The empirical power is presented relative to the Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (L 0,0353)
0.009 (t: 0.0004)
1.0 C:: 0.037)
0.007 (1: 0.0003)
1.0 (E: 0.0446)
0.005 (1: 0.0002)
Oracle (infcasible)
1.104 (#: 0.0383)
0.01 0.0004)
1.37 0.0474)
0.01 (# 0.0004)
1.653 0.0523)
0.009 (# 0.0003)
Naive-Trim (invalid)
1,556 (+ 0.0452)
0.018 (= 0.0005)
2.635 (+= 0.0555)
0.031 0.0006)
3.796 0.0594)
0.04 (+ 0.0008)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
O.0 0.0)
(-0 0.0)
Label- Trim
1.104 (+ 0.0383)
0.01 (+ 0.0004)
1.37 (+ 0.0474)
0.01 (+: 0.0004)
1.651 (+ 0.0526)
0.009 (+ 0.0003)
Standard Power
0.151 (+ 0.0053)
0.009 (+ 0.0004)
0.128 (+ 0.0047)
0.007 (+ 0.0003)
0.105 (+ 0.0047)
0.005 (+ 0.0002)
Target type-I error rate 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Liror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0284)
0.019 (+ 0.0005)
1.0 (+ 0.0279)
0.014 (+ 0.0004)
1.0 0.0313)
0.011 (+ 0.0004)
Oracle (infcasible)
1.074 (+: 0.0281)
0.021 (= 0.0005)
1,221 0.0303)
0.02 (#: 0.0005)
1.4 (#: 0.0317)
0.019 (#: 0.0006)
Naive-Trim (invalid)
1.295 (+: 0.0281)
0.028 (t: 0.0006)
1.826 0.0314)
0.039 0.0007)
2.351 0.0323)
0.049 (t: 0.0008)
Small-Clean
0,823 (+ 0.0624)
0.017 (#: 0.0018)
0.471 (#: 0.0636)
0.008 (#: 0.0012)
0.401 (# 0.0691)
0.005 (#: 0.001)
Label- Trim
1.06 (1 0.0275)
0.02 (1L 0.0005)
1.173 (1L 0.0302)
0.019 (1 0.0005)
1,272 (1 0.0311)
0.016 (.L 0.000S)
Standard Power
0.239 (+: 0.0068)
0.019 (== 0.0005)
0,213 (#: 0.0059)
0.014 (# 0.0004)
0.187 (# 0.0059)
0.011 (# 0.0004)
(b) Target type-I error
0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0217)
0.028 (1 0.0006)
1.0 (L 0.0239)
0.023 0.0005)
1.0 (L 0.0239)
0.018 (.L 0.0005)
Oracle (infeasible)
1,086 (+: 0.0207)
0.031 (# 0.0006)
1,209 0.0264)
0.03 (+ 0.0007)
1.344 0.0247)
0.029 (+ 0.0007)
Naive-Trim (invalid)
1.196 (#: 0.0201)
0.037 (# 0.0007)
1.512 (1: 0.0223)
0.048 0.0008)
1.879 (1: 0.0245)
0.057 (1: 0.0009)
Small-Clean
0.662 (#: 0,0464)
0.018 (#: 0.0018)
0.733 (#: 0.05)
0.017 0.0016)
0.848 (#: 0.0556)
0.017 (1: 0.0018)
Label Trim
1.054 (t: 0.0218)
0.03 (+: 0.0006)
1.12 (: 0.0256)
0.027 (+: 0.0006)
1.167 (+ 0.0254)
0.023 (+: 0.0006)
Standard Power
0,313 (t: 0.0068)
0.028 (:: 0.0006)
0.278 (t: 0.0066)
0.023 (t: 0.0005)
0.252 (t: 0.006)
0.018 (t: 0.0005)
(c) Target type-I error rate 0.03
45

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 18: Comparison of conformal outlier detection methods Texture dataset (outliers) and CIFAR-10 10 dataset (inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with pretrained ResNet-18. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0416)
0.009 (t: 0.0003)
1.0 (:: 0.0373)
0.006 0.0003)
1.0 (E 0.036)
0.005 (1: 0.0002)
Oracle (infcasible)
1.129 (+ 0.0436)
0.01 0.0003)
1.399 0.0502)
0.01 (# 0.0003)
1.545 0.0506)
0.01 (= 0.0004)
Naive-Trim (invalid)
1.757 (+ 0.0508)
0.018 (+ 0.0004)
2.978 (+ 0.061)
0.029 0.0006)
4.043 0.0598)
0.04 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
0.0 0.0)
0.0 0.0)
Label-Trine Trim
1.129 (+ 0.0436)
0.01 (+ 0.0003)
1.399 (+ 0.0502)
0.01 (+: 0.0003)
1,526 (+ 0.0482)
0.009 (+ 0.0003)
Standard Power
0.146 (++ 0.0061)
0.009 (+ 0.0003)
0.125 (+ 0.0047)
0.006 (+ 0.0003)
0.107 (+ 0.0038)
0.005 (+ 0.0002)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Eiror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.029)
0.018 (+ 0.0004)
1.0 (+ 0.0269)
0.014 (+ 0.0004)
1.0 (+ 0.0286)
0.011 (+ 0.0003)
Oracle (infcasible)
1.089 (+: 0.0286)
0.021 (= 0.0005)
1.292 0.0312)
0.02 (#: 0,0005)
1,518 0.0278)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.295 (+: 0.0312)
0.027 (t: 0.0006)
1.904 0.0338)
0.037 0.0006)
2.51 (:: 0.0339)
0.048 (E: 0.0008)
Small-Clean
0.792 (+ 0.0585)
0.017 (#: 0.0018)
0.636 (#: 0.0789)
0.012 (# 0.0019)
0.288 (# 0.0601)
0.005 (#: 0.0013)
Label- Trim
1.069 (LL 0.0285)
0.02 (LL 0.0005)
1.228 (1L 0.0309)
0.018 (L 0.0005)
1,333 (LI 0.0291)
0.015 (.L 0.0004)
Standard Power
0.259 (+: 0.0075)
0.018 (=: 0.0004)
0.226 (+: 0.0061)
0.014 (# 0.0004)
0.188 (# 0.0054)
0.011 (# 0.0003)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (LL 0.024)
0.028 (1 0.0006)
1.0 (L 0.0222)
0.022 0.0005)
1.0 (L 0.0205)
0.017 (. 0.0005)
Oracle (infeasible)
1.089 (+: 0.0227)
0.031 (# 0.0006)
1,228 0.0248)
0.029 0.0006)
1.394 0.0218)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.204 (#: 0.023)
0.036 (1: 0.0006)
1.557 (1: 0.0251)
0.045 0.0007)
1.919 (1: 0.0242)
0.056 (1: 0.0009)
Small-Clean
0.659 (#: 0,0441)
0.019 (#: 0.0018)
0.809 (#: 0.0541)
0.019 0.002)
0.837 (#: 0.0579)
0.017 (#: 0.002)
Label Trim
1.057 (t: 0.0231)
0.03 (+: 0.0006)
1.115 (+ 0.0238)
0.026 (+: 0.0006)
1.173 (+ 0.021)
0.023 (+: 0.0005)
Standard Power
0.338 (:: 0,0081)
0.028 (:: 0.0006)
0.307 (t: 0.0068)
0.022 (t: 0.0005)
0.263 (t: 0.0054)
0.017 (t: 0.0005)
(c) a 0.03
46

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 19: Comparison of conformal outlier detection methods on SVHN dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with pretrained ResNet-18. The empirical power is presented relative to the Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0308)
0.009 (t: 0.0003)
1.0 (:: 0.0386)
0.006 (1: 0.0003)
1.0 (E 0.0401)
0.004 (1: 0.0002)
Oracle (infcasible)
1.144 (+ 0.0321)
0.01 0.0003)
1,526 0.0466)
0.01 (# 0.0003)
1.868 0.058)
0.01 (= 0.0004)
Naive-Trim (invalid)
1.743 (+ 0.033)
0.017 (+ 0.0004)
2.838 0.0519)
0.027 0.0005)
4.211 0.0669)
0.038 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
O.L 0.0)
(-0 0.0)
Label- Trim
1.144 (+ 0.0321)
0.01 (+ 0.0003)
1.513 (+ 0.0462)
0.01 (+ 0.0003)
1.749 (+ 0.0529)
0.009 (+ 0.0003)
Standard Power
0.191 (+ 0.0059)
0.009 (+ 0.0003)
0.143 (+ 0.0055)
0.006 (+ 0.0003)
0.115 (+ 0.0046)
0.004 (+ 0.0002)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Eiror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0189)
0.017 (+ 0.0004)
1.0 (+ 0.027)
0.012 (+ 0.0003)
1.0 (+ 0.0299)
0.01 (+ 0.0003)
Oracle (infcasible)
1.082 (+: 0.0204)
0.021 (= 0.0005)
1.342 (#: 0.0262)
0,02 (#: 0,0005)
1.642 0.0331)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.214 (t: 0.021)
0.027 (t: 0.0006)
1.858 0.0271)
0.036 0.0006)
2.462 0.0359)
0.046 (1: 0.0008)
Small-Clean
0.788 (+ 0.0566)
0.018 (#: 0.002)
0.662 (#: 0.0764)
0.011 (# 0.0015)
0.46 (# 0.0819)
0.007 (#: 0.0016)
Label- Trim
1.068 (LL 0.0207)
0.02 (LL 0.0005)
1.253 (.1 0.0275)
0.017 (LL 0.0005)
1.403 (LI 0.0332)
0.014 (. 0.0004)
Standard Power
0.336 (#: 0.0064)
0.017 (=: 0.0004)
0,256 (+: 0.0069)
0.012 (#: 0.0003)
0.216 (# 0.0064)
0.01 (# 0.0003)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0173)
0.027 (1 0.0006)
1.0 (L 0.0196)
0.02 (.L. 0.0005)
1.0 (L 0.0229)
0.015 (. 0.0004)
Oracle (infeasible)
1.075 (+: 0.017)
0.031 (+ 0.0006)
1.247 0.0215)
0.029 0.0006)
1.39 (= 0.0246)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.162 (1: 0.0163)
0.036 (1: 0.0006)
1.519 (1: 0.0195)
0.044 0.0007)
1.805 (1: 0.0245)
0.054 (1: 0.0008)
Small-Clean
0.686 (#: 0.0437)
0.019 (#: 0.002)
0.857 (#: 0.0528)
0.019 0.0017)
0.913 (#: 0.0586)
0.018 (1: 0.0018)
Label Trim
1.046 (t: 0.0176)
0.029 (+: 0.0006)
1.122 (+ 0.0212)
0.025 (+: 0.0005)
1.179 (+ 0.0225)
0.022 (t: 0.0005)
Standard Power
0.411 (t: 0,007
0.027 (t: 0.0006)
0.342 (t: 0.0067)
0.02 (:: 0.0005)
0.311 (t: 0.0071)
0.015 (t: 0.0004)
(c) a 0.03
47

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 20: Comparison of conformal outlier detection methods on Places365 dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with pretrained ResNet-18. The empirical power is presented relative to the St Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0354)
0.009 (t: 0.0003)
1.0 (:: 0.0381)
0.006 (1: 0.0003)
1.0 (E 0.0462)
0.004 (1: 0.0002)
Oracle (infcasible)
1.164 (# 0.0361)
0.01 0.0003)
1.489 0.0456)
0.01 (# 0.0003)
1.773 0.0555)
0.01 (= 0.0004)
Naive-Trim (invalid)
1.729 (+ 0.0387)
0.017 (+ 0.0004)
2,944 0.0532)
0.028 0.0006)
4.136 0.0545)
0.038 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
0.0 (:L: 0.0)
(-0 0.0)
Label- Trim
1.164 (+ 0.0361)
0.01 (+ 0.0003)
1.483 (+ 0.0452)
0.01 (+ 0.0003)
1,704 (+ 0.0533)
0.009 (+ 0.0003)
Standard Power
0.174 (+ 0.0061)
0.009 (+ 0.0003)
0.133 (+ 0.005
0.006 (+ 0.0003)
0.112 (+ 0.0052)
0.004 (+ 0.0002)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Eiror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0221)
0.018 (+ 0.0004)
1.0 (+ 0.0258)
0.013 (+ 0.0004)
1.0 (+ 0.0296)
0.01 (+ 0.0003)
Oracle (infcasible)
1.068 (+: 0.0237)
0.021 (= 0.0005)
1.291 0.0272)
0.02 (#: 0,0005)
1.564 0.0325)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.233 (t: 0.0249)
0.027 (t: 0.0006)
1.782 0.0278)
0.036 0.0006)
2.483 0.0308)
0.046 (1: 0.0008)
Small-Clean
0,823 (+ 0.0588)
0.018 (#: 0.0019)
0.595 (#: 0.074)
0.011 (# 0.0016)
0.331 (# 0.0746)
0.005 (#: 0.0014)
Label- Trim
1.058 (LL 0.0236)
0.02 (LL 0.0005)
1.212 (1L 0.0272)
0.018 (L 0.0005)
1,357 (L 0.0312)
0.015 (. 0.0004)
Standard Power
0.302 (#: 0.0067)
0.018 (=: 0.0004)
0.25 (# 0.0065)
0.013 (+ 0.0004)
0.207 (# 0.0061)
0.01 (# 0.0003)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0201)
0.028 (1 0.0006)
1.0 (L 0.0203)
0.021 0.0005)
1.0 (L 0.0225)
0.016 (1 0.0004)
Oracle (infeasible)
1.084 (+: 0.0218)
0.031 (# 0.0006)
1,229 0.0211)
0.029 0.0006)
1.383 0.0228)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.186 (#: 0.0217)
0.036 (1: 0.0006)
1.534 (1: 0.0216)
0.044 0.0007)
1.866 (1: 0.0213)
0.055 (1: 0.0009)
Small-Clean
0.712 (#: 0.0449)
0.019 (#: 0.0018)
0.832 (#: 0.0525)
0.019 0.0018)
1.025 (1: 0.0644)
0.023 (#: 0.0022)
Label Trim
1.044 (t: 0.0206)
0.029 (+: 0.0006)
1.121 (+ 0.0208)
0.026 (+: 0.0005)
1.167 (+ 0.0219)
0.022 (+: 0.0005)
Standard Power
0.374 (:: 0.0075)
0.028 (t: 0.0006)
0.327 (t: 0.0066)
0.021 (t: 0.0005)
0.291 (t: 0.0065)
0.016 (t: 0.0004)
(c) a 0.03
48

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 21: Comparison of conformal outlier detection methods MNIST dataset (outliers) and CIFAR-10 dataset (inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with pretrained ResNet-18. The empirical power is presented relative to the Standard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0286)
0.008 (t: 0.0003)
1.0 (:: 0.0284)
0.004 (1: 0.0002)
1.0 CE 0.0366)
0.003 (1: 0.0002)
Oracle (infcasible)
1.174 (+ 0.0308)
0.01 0.0003)
1.255 0.0389)
0.01 (# 0.0003)
1.961 0.045)
0.01 (= 0.0004)
Naive-Trim (invalid)
1.641 (+ 0.0326)
0.016 (+ 0.0004)
2.781 0.0382)
0.025 0.0006)
3.78 (# 0.0441)
0.032 (+ 0.0006)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
O.0 0.0)
(.0 0.0)
Label- Trim
1.174 (+ 0.0308)
0.01 (+ 0.0003)
1.555 (+ 0.0366)
0.01 (+ 0.0003)
1,701 (+ 0.041)
0.008 (+ 0.0003)
Standard Power
0.251 (+ 0.0072)
0.008 (+ 0.0003)
0.181 (+ 0.0051)
0.004 (+ 0.0002)
0.152 (+ 0.0056)
0.003 (+ 0.0002)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Eiror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.0195)
0.017 (+ 0.0004)
1.0 (+ 0.0216)
0.01 I (+ 0.0003)
1.0 0.0244)
0.008 (+ 0.0003)
Oracle (infcasible)
1.095 (+: 0.0198)
0.021 (= 0.0005)
1.416 0.0218)
0.02 (#: 0,0005)
1.776 0.0279)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.212 (t: 0.0195)
0.026 (t: 0.0006)
1.84 0.0223)
0.032 0.0006)
2.456 0.0243)
(:: 0.0007)
Small-Clean
0.849 (+ 0.0589)
0.02 (#: 0.0021)
0.605 (#: 0.0715)
0.009 (# 0.0015)
0.366 (# 0.0772)
0.005 (#: 0.0014)
Label- Trim
1.076 (L 0.0197)
0.02 (LL 0.0005)
1.3 (1 0.0227)
0.016 (L 0.0005)
1.424 (1 0.0272)
0.013 (.L 0.0004)
Standard Power
0.415 (#: 0,0081)
0.017 (== 0.0004)
0.317 (#: 0.0069)
0.011 (#: 0.0003)
0.257 (# 0.0063)
0.008 (#: 0.0003)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (LL 0.0161)
0.026 (1 0.0006)
1.0 (L 0.0164)
0.018 0.0005)
1.0 (L 0.0191)
0.012 (. 0.0004)
Oracle (infeasible)
1,086 (+: 0.0154)
0.031 (# 0.0006)
1,272 0.0167)
0.029 0.0006)
1,518 0.02)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.161 (1: 0.0144)
0.035 (1: 0.0006)
1.483 (1: 0.0168)
0.041 0.0007)
1.852 (1: 0.0172)
0.048 (1: 0.0008)
Small-Clean
0.778 (#: 0.0434)
0.022 (#: 0.002)
0.848 (#: 0.0471)
0.018 0.0018)
1.075 (1: 0.0562)
0.02 (# 0.0021)
Label Trim
1,056 (t: 0.0161)
0.029 (+: 0.0006)
1.138 (+ 0.0162)
0.024 (+: 0.0005)
1.237 (+ 0.0195)
0.019 (t: 0.000S)
Standard Power
0.505 (:: 0,0081)
0.026 (t:: 0.0006)
0.43 (:: 0.007)
0.018 (t: 0.0005)
0.362 (t: 0.0069)
0.012 (t: 0.0004)
(c) a 0.03
49

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 22: Comparison of conformal outlier detection methods CIFAR-100 dataset (outliers) and CIFAR-10 dataset
(inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with a pretrained ResNet-18. The empirical power is presented relative to the St andard method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0.0399)
0.009 (t: 0.0003)
1.0 (:: 0.0432)
0.007 (1: 0.0003)
1.0 (E: 0.047)
0.006 (1: 0.0003)
Oracle (infcasible)
1.113 (+ 0.0396)
0.01 0.0003)
1.355 0.049))
0.01 (# 0.0003)
1.478 0.0566)
0.01 (# 0.0004)
Naive-Trim (invalid)
1.848 (+ 0.0507)
0.018 (+ 0.0005)
3.161 0.0652)
0.03 (# 0.0006)
4.447 0.0722)
0.042 (+ 0.0007)
Small-Clean
0.0 0.0)
(1 0.0)
(1: 0.0)
0.0 0.0)
0.0 0.0)
(= 0.0)
Label- -rim
1.116 (+ 0.04)
0.01 (+ 0.0003)
1.355 (+ 0.0496)
0.01 (+ 0.0003)
1.491 (+ 0.0582)
0.01 (+ 0.0004)
Standard Power
0.124 (+ 0.0049)
0.009 (+ 0.0003)
0.103 (+ 0.0045)
0.007 (+ 0.0003)
0.091 (+ 0.0043)
0.006 (+ 0.0003)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0.027)
0.019 (+ 0.0005)
1.0 (+ 0.027)
0.014 (+ 0.0004)
1.0 0.0329)
0.012 (+ 0.0004)
Oracle (infcasible)
1.057 (+: 0.0281)
0.021 (= 0.0005)
1.24 (1: 0.0302)
0.02 (#: 0,0005)
1.507 0,0364)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.305 (t: 0.0289)
0.028 (+: 0.0006)
1.957 (t: 0.0336)
0.038 0.0006)
2.747 0.0395)
(:: 0.0008)
Small-Clean
0.932 (+ 0.0653)
0.02 (#: 0.0018)
0.533 (#: 0.0766)
0.01 (#: 0.0017)
0.38 (# 0.0889)
0.007 (#: 0.0018)
Label- Trim
1.046 (L 0.0276)
0.021 (_L 0.0005)
1.178 (1L 0.0299)
0.018 (L 0.0005)
1,344 (L 0.0349)
0.016 (.L 0.0005)
Standard Power
0.232 (# 0.0063)
0.019 (== 0.0005)
0.197 (=: 0.0053)
0.014 (# 0.0004)
0.163 (# 0.0053)
0.012 (# 0.0004)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (1 0.0218)
0.028 (1 0.0006)
1.0 (L 0.0244)
0.023 0.0005)
1.0 (L 0.0251)
0.018 (. 0.0005)
Oracle (infeasible)
1,088 (+: 0.0232)
0.031 (# 0.0006)
1,231 0.0255)
0.029 0.0006)
1.376 0.0272)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.224 (#: 0.0223)
0.037 (#: 0.0007)
1.658 (1: 0.0276)
0.047 0.0007)
2.043 (1: 0.0274)
0.058 (1: 0.0009)
Small-Clean
0.758 (#: 0,0461)
0.021 (#: 0.0017)
0.897 (#: 0.0659)
0.023 0.0024)
0.97 (#: 0.0695)
0.021 (#: 0.0022)
Label Trim
1,051 (t: 0.0231)
0.03 (+: 0.0006)
1.11 (: 0.0244)
0.026 (+: 0.0006)
1.173 ( 0.0261)
0.024 (+: 0.0005)
Standard Power
0.304 (:: 0.0066)
0.028 (:: 0.0006)
0.261 (ct 0.0064)
0.023 (t: 0.0005)
0.237 (t: 0.0059)
0.018 (t: 0.0005)
(c) a 0.03
50

---

Robust Conformal Outlier Detection under Contaminated Reference Data
Table 23: Comparison of conformal outlier detection methods TinyImageNet dataset (outliers) and CIFAR-10 dataset
(inliers) for varying contamination rate and target type-I error level a. All methods utilize the SCALE (Xu et al., 2024) method with a pretrained ResNet-18. The empirical power is presented relative to the St and method (higher is better). Results are averaged across 100 random splits of the data, with standard errors presented in parentheses.
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-I Error
Power
Type-I Error
Standard
1.0 (1 0,0383)
0.009 (t: 0.0003)
1.0 (:: 0.0394)
0.006 0.0003)
1.0 (E: 0.0383)
0.005 (1 0.0002)
Oracle (infcasible)
1.138 (# 0.0399)
0.01 0.0003)
1.436 0.0534)
0.01 (# 0.0003)
1.625 0.0475)
0.01 (= 0.0004)
Naive-Trim (invalid)
1.696 (+ 0.0424)
0.018 (+ 0.0004)
2.88 0.0621)
0.029 0.0006)
4.248 0.0707)
0.041 (+ 0.0007)
Small-Clean
0.1 0.0)
(1 0.0)
0.0 0.0)
0.0 0.0)
0.0 0.0)
(=0 0.0)
Label-Trine Trim
1,138 (+ 0.0399)
0.01 (+ 0.0003)
1.433 (+ 0.0535)
0.01 (+ 0.0003)
1,591 (+ 0.0467)
0.009 (+ 0.0004)
Standard Power
0.159 (+ 0.0061)
0.009 (+ 0.0003)
0.122 (+ 0.0048)
0.006 (+ 0.0003)
0.102 (+ 0.0039)
0.005 (+ 0.0002)
(a) 0.01
Contamination rate
1%
3%
5%
Method
Power
Type-1 Eiror
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (+ 0,0248)
0.018 (+ 0.0004)
1.0 (+ 0.0317)
0.014 (+ 0.0004)
1.0 0.027I)
0.011 (+ 0.0004)
Oracle (infcasible)
1.063 (+: 0.0255)
0.021 (= 0.0005)
1.254 0.0309)
0.02 (#: 0,0005)
1.49 (= 0.0318)
0.02 (# 0.0005)
Naive-Trim (invalid)
1.223 (t: 0.0263)
0.028 (t: 0.0006)
1.816 0.0321)
0.037 0.0006)
2.536 0.0365)
0.049 (t: 0.0008)
Small-Clean
0,803 (+ 0.0609)
0.017 (# 0.0018)
0.625 (#: 0.0816)
0.013 (# 0.0022)
0.263 (# 0.064)
0.003 (#: 0.001)
Label- Trim
1.053 (L 0.0255)
0.02 (LL 0.0005)
1.183 (.L 0.0325)
0.018 (L 0.0005)
1,344 (1 0.0334)
0.016 (.1 0.0004)
Standard Power
0.273 (#: 0.0068)
0.018 (=: 0.0004)
0.227 (=: 0.0072)
0.014 (# 0.0004)
0.187 (# 0.0051)
0.011 (# 0.0004)
(b) 0.02
Contamination rate
1%
3%
5%
Method
Power
Type-I Error
Power
Type-1 Error
Power
Type-I Error
Standard
1.0 (LL 0.0215)
0.028 (1 0.0006)
1.0 (1 0.024)
0.022 0.0005)
1.0 (L 0.0221)
0.017 (. 0.0005)
Oracle (infeasible)
1.07 0.0222)
0.031 (# 0.0006)
1.188 0.0264)
0.029 0.0006)
1.328 0.0256)
0.03 (+ 0.0006)
Naive-Trim (invalid)
1.202 (#: 0.0203)
0.037 (# 0.0006)
1.533 (1: 0.0266)
0.046 0.0007)
1.916 (1: 0.0288)
0.057 (1: 0.0009)
Small-Clean
0.678 (#: 0.0476)
0.017 (#: 0.0018)
0.932 (#: 0.0576)
0.025 0.0025)
1.005 0.0573)
0.024 (#: 0.0022)
Label Trim
1.046 (t: 0.0222)
0.03 (+: 0.0006)
1.09 (+ 0.0256)
0.026 (+: 0.0005)
1.14 (+: 0.0234)
0.023 (+: 0.0005)
Standard Power
0.336 (+: 0.0072)
0.028 (t: 0.0006)
0.295 (t: 0.0071)
0.022 (t: 0.0005)
0.266 (t: 0.0059)
0.017 (t: 0.0005)
(c) a 0.03
51


Reviews:
[{"review_id": "A56SoKObED", "reviewer": "Reviewer_2tsw", "summary": "This paper analyzes the impact of such contamination on the validity of conformal methods. The paper proves that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error control.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We’ve conducted additional experiments to answer your questions and further support and clarify our contributions. These results are available at https://tinyurl.com/rcod-exps, and we refer to them as *Supp-Figure X* and *Supp-Table Y* in our responses below.\n\n> **R1:** In most experiments, m is fixed to 50. Generally, how should it be set to achieve good model performance?\n\nWe view the annotation budget m as primarily determined by resource constraints (how much it costs to label the data points) rather than as a tuning parameter that can be optimized by a general mathematical argument. This is because the power of our method depends not just on m, but also on the performance of the outlier detection model and the underlying contamination level, both of which are hard to characterize theoretically without much stronger assumptions than we make in this work.\n\nThat said, Figure 3 in the main manuscript provides interesting practical insight. In that experiment, the calibration set contains approximately 80 outliers. As shown in the figure, even a small annotation budget (e.g., m ≈ 20) is enough for Label-Trim to achieve power nearly matching that of the oracle conformal method that uses clean reference data. This suggests that even a modest budget (on the order of a quarter of the expected number of outliers) can already yield strong performance. Even more encouragingly, the power of Label-Trim appears to vary quite smoothly with the labeling budget in these experiments, indicating graceful performance degradation under tighter annotation constraints.\n\nThank you for giving us the opportunity to further highlight this important aspect of Figure 3. We will clarify this message and the associated practical guidance in the revised version.\n\n\n> **R2:** How does the proposed method perform when applied to real data that is inherently contaminated?\n\nTo answer this question, which also relates to Reviewer Zuz4’s comments, we conducted additional experiments simulating various realistic contamination scenarios. We summarize these results at the end of this response.\n\nWe also wish to clarify that all our existing experiments are already based on real-world datasets, meaning both the inliers and outliers originate from actual applications. Since our method relies on selective annotation under a limited budget, we simulate contamination in the reference set by injecting outliers. This setup reflects a realistic scenario where labeling resources are constrained and full manual annotation is impractical due to the need for domain expertise.\n\nMoreover, a controlled contamination process is essential for evaluating our method because computing the performance metrics requires knowing the ground-truth labels. Specifically, to estimate type-I error, we need to know which test points are inliers, and to measure detection power, we need to know which test points are outliers.\nSummary of additional experiments:\n\n- **Varying contamination rate.** We extend Figure 2 by increasing the contamination rate up to 15%. As shown in Supp-Figure 1, Label-Trim maintains type-I error control while continuing to outperform standard conformal and the “small clean” baseline in terms of power.\n- **Strategic outlier injection.** Instead of injecting outliers at random, we selected outliers that resemble inliers—those falling below a given score percentile. Supp-Figure 5 (shuttle dataset) shows how these lower-percentile outliers increasingly resemble inliers, while Supp-Figure 4 demonstrates that Label-Trim still controls the error and improves power. Similar trends hold on the credit-card and KDDCup99 datasets (Supp-Figures 6–7).\n- **Test-time distribution drift.** On the shuttle dataset, we simulate drift in the outlier distribution by contaminating the calibration set with high-percentile outliers and gradually shifting to harder, low-percentile outliers at test time. As shown in Supp-Figure 8, Label-Trim remains robust, maintaining error control and power throughout the distribution shift.\n\nWe will revise the paper to clarify how our experimental design reflects real-world constraints and to incorporate discussion of these new results demonstrating robustness to a range of contamination scenarios.", "claims_and_evidence": "This paper focuses on detecting outliers with conformal prediction in the context of contaminated reference data, which is both an interesting and important problem.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The authors provide a solid and new theoretical analysis of the proposed method in terms of the type-I error rate.", "experimental_designs_or_analyses": "Extensive experiments on real data validate that standard conformal outlier detection methods are conservative under contamination and show that the proposed method improves power without sacrificing validity in practice.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weaknesses:\n1. In most experiments, m is fixed to 50. Generally, how should $m$ be set to achieve good model performance? \n2. This paper primarily considers injected outliers within the contaminated calibration set. How does the proposed method perform when applied to real data that is inherently contaminated?", "other_comments_or_suggestions": "No.", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kaYnrh3qDB", "reviewer": "Reviewer_Zuz4", "summary": "This paper studies conformal outlier detection with contaminated reference sets. It theoretically shows that non-adversarial contamination induces conservative type-I error control, explaining empirical performance gaps. To address power loss, the authors propose Label-Trim: an active data-cleaning framework leveraging limited labeling budgets to annotate and remove suspected outliers from high-scoring regions. Theoretical analysis proves Label-Trim maintains approximate error control under practical conditions. Experiments on tabular and vision datasets validate that standard conformal methods become conservative under contamination, while Label-Trim recovers detection power without inflating errors, achieving near-oracle performance when contamination rates are low.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We appreciate the opportunity to provide some clarifications. To address your questions and strengthen the empirical foundation of our claims, we have also conducted a range of new experiments available at https://tinyurl.com/rcod-exps, which we reference below as Supp-Figure X and Supp-Table Y.\n\nOur goal in this paper is to address a critical gap in conformal outlier detection: robustness to contamination in the reference set. To the best of our knowledge, no prior work offers a fully satisfactory solution to this problem. While our proposed method is deliberately simple, its theoretical guarantees are non-trivial, and our new experiments demonstrate its practical robustness.\n\n> **R1:** The method’s simple pipeline lacks deeper algorithmic novelty\n\nWe understand this concern, though we believe the intuitive nature of our approach is a practical strength. Moreover, while the method is intentionally simple, establishing its validity is technically non-trivial. Our analysis required developing a novel proof technique, which we believe offers theoretical and methodological value beyond the specific application studied in this paper.\n\n> **R2:** Assumes i.i.d. inliers\n\nYou're right that conformal prediction traditionally assumes i.i.d. inliers and a clean reference set. Our work relaxes the second of these assumptions: we allow the reference set to be contaminated with non-i.i.d. outliers. The assumption we retain is that the inliers themselves are i.i.d. We agree that relaxing the i.i.d. assumption on the inliers is an exciting direction for future work. However, doing so would likely require additional assumptions on the dependency structure, going outside the scope of this paper. We will clarify this point in the revision.\n\n> **R3:** Focus on low contamination ≤5%\n\nThank you for pointing this out. As described in our response to Reviewer 2tsw, we’ve extended our experiments to include higher contamination levels, and continue to observe consistent trends.\n\n> **R4:** Systematically test Label-Trim with alternative models\n\nWe appreciate the suggestion. In addition to Isolation Forest (tabular) and ReAct with a ResNet backbone (visual) used in the main manuscript, we’ve now added results for:\nTabular data: Local Outlier Factor (LOF) and One-Class SVM (OCSVM)\nVisual data: ReAct with a VGG19 backbone and SCALE with a ResNet backbone\nAs shown in Supp-Figures 2–3, Label-Trim consistently controls the type-I error across all models. As expected, when the outlier detection model is less effective (e.g., LOF, OCSVM), the candidate set becomes noisier and power decreases. Still, our method improves over the baseline. For the visual data, Supp-Tables 1–2 show results that closely mirror the trends in the main manuscript. We’ll include these in the revision.\n\n> **R5:** Random outlier injection oversimplifies real-world scenarios where outliers may strategically mimic inliers\n\nWe completely agree and have addressed this in our response to Reviewer 2tsw, where we design and test more challenging outlier scenarios.\n\n> **R6:** Real-world drift are untested\n\nWe share your interest in this important issue. In our response to Reviewer 2tsw, we present new experiments simulating drift in the outlier distribution over time, and find that Label-Trim remains robust.\n\n> **R7:** Explore adaptive labeling budgets instead of fixed m=50.\n\nWe refer you to our response to Reviewer 2tsw, where we discuss the role of the annotation budget m, and show that Label-Trim performs well across a range of values. We also highlight that its performance degrades gracefully as the budget decreases.\n\n> **R8:** How does Label-Trim perform on naturally contaminated datasets?\n\nPlease see our response to Reviewer 2tsw, where we explain why controlled contamination is necessary for rigorous evaluation and provide additional experiments modeling more realistic contamination.\n\n> **R9:** Would the method fail catastrophically when the number of outliers exceeds the labeling budget m?\n\nNo. Both our theoretical results and experiments show that type-I error control is preserved regardless of how small m is relative to the number of outliers. For instance, in Figure 3 of the main manuscript and Supp-Figure 1 (right panel), the number of outliers significantly exceeds the labeling budget, yet Label-Trim still controls the error and achieves meaningful power gains.\n\n> **R10:** The focus on low-error regimes (α=0.01-0.03) matches safety-critical applications but ignores moderate α\n\nWe’ve added experiments evaluating performance for higher type-I error thresholds. As shown in Supp-Figure 9, Label-Trim continues to perform well across all tested α levels. We'll incorporate these results into the revised version.", "claims_and_evidence": "The paper’s key claims—conservative error control under contamination and Label-Trim’s power recovery—are supported by theoretical proofs (Lemma 2.2, Theorem 3.1) and experiments on tabular/vision datasets. Empirical validation includes score distribution analysis (Fig.1), type-I error trajectories (Fig.2,4), and detection rate comparisons (Fig.2-3, Table1). Near-oracle performance at low contamination is numerically confirmed.  Experiments exclude high contamination rates (>5%), limiting claims about robustness to realistic but challenging data shifts.", "methods_and_evaluation_criteria": "The proposed Label-Trim method aligns with practical constraints: it uses a pre-trained outlier detector to prioritize high-scoring calibration samples for limited manual labeling, then trims confirmed outliers. This leverages model confidence to focus labeling efforts, avoiding random or exhaustive cleaning. Evaluation employs standard conformal metrics (type-I error rate, detection power) on 3 tabular  and 6 vision datasets, with controlled contamination rates (1%-5%). Baseline comparisons include Oracle (clean reference), Standard (no cleaning), Naive-Trim (remove top scores without labels), and Small-Clean (random labeling).\n\nWhile the datasets are established in outlier detection, the contamination simulation (random outlier injection) oversimplifies real-world scenarios where outliers may strategically mimic inliers. The 5% contamination cap excludes high-pollution cases common in practice (e.g., 10%-20%). The focus on low-error regimes (α=0.01-0.03) matches safety-critical applications but ignores moderate α settings. Isolation Forest (tabular) and ReAct (vision) are reasonable model choices, but using only one detector per data type (without testing alternatives) limits confidence in generalizability. The dependence on detector quality is not systematically tested, weakening claims about robustness across detector architectures.", "theoretical_claims": "The proofs for Lemma 2.2 and Theorem 3.1 are mathematically correct under their assumptions (e.g., i.i.d. inliers, fixed contamination).", "experimental_designs_or_analyses": "Experiments are sound for core claims: controlled contamination (1%-5%) tests type-I error/power trade-offs. Tabular (Isolation Forest) and vision (ResNet) benchmarks are standard. However:\n\n1. Contamination is simulated via random outlier injection, ignoring realistic contamination scenarios.\n2. High contamination (>5%) and real-world drift (e.g., temporal shifts) are untested.\n3. Only one detector per data type is used; architecture variations are unexplored.", "relation_to_broader_scientific_literature": "This paper is the first to bridge conformal prediction with outlier detection under reference set contamination, introducing a novel intersection of these fields. Prior conformal works focused on covariate/label shifts (Tibshirani et al., 2019) or label noise (Sesia et al., 2024), but none addressed **contaminated calibration sets** in outlier detection. Unlike semi-supervised anomaly detection (Jiang et al., 2022), which assumes partial labels, Label-Trim uses limited labels to clean reference data while preserving conformal guarantees—a unique hybrid approach. Theoretically, it extends conservative error bounds (Sesia et al., 2024) to **unknown outlier distributions**, avoiding explicit noise modeling. Compared to worst-case robustness analyses (Barber et al., 2023), this work identifies practical conservatism under non-adversarial contamination, aligning with empirical patterns. Experiments validate the framework on both tabular and vision data, broadening conformal methods beyond traditional single-domain applications.\n\n- Tibshirani, R. J., Foygel Barber, R., Cande`s, E., and Ramdas, A. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.\n- Sesia, M., Wang, Y. R., and Tong, X. Adaptive conformal classification with noisy labels. J. R. Stat. Soc. Series B, pp. qkae114, 2024.\n- Barber, R. F., Cand`es, E. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. Ann. Stat., 51(2):816–845, 2023.\n- Jiang, X., Liu, J., Wang, J., Nie, Q., Wu, K., Liu, Y., Wang, C., and Zheng, F. Softpatch: Unsupervised anomaly detection with noisy data. Advances in Neural Information Processing Systems, 35:15433–15445, 2022.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. **Originality:** This paper is the first to bridge conformal prediction with outlier detection under reference set contamination, bridging anomaly detection and robust statistics.\n\n2. **Practicality:** Label-Trim’s simplicity suits real-world deployment.\n\n3. **Clarity:** Figures (e.g., score distributions) intuitively explain conservative behavior.\n\n**Weaknesses:**\n\n1. **Moderate innovation:** The method’s simple pipeline (score-sort-label) lacks deeper algorithmic novelty compared to recent advances.\n\n2. **Assumption-heavy theory:** Relies on independent and identically distributed inliers, limiting real-world applicability.\n\n3. **Narrow scope:** Focus on low contamination (≤5%) excludes high-noise scenarios common in practice.", "other_comments_or_suggestions": "1. **Dynamic Budget Allocation:** Explore adaptive labeling budgets (e.g., increasing *m* when contamination is suspected) instead of fixed *m=50*.\n2. **Detector-Agnostic Analysis:** Systematically test Label-Trim with alternative architectures (e.g., autoencoders, One-Class SVM) to assess generalizability.", "questions_for_authors": "1. How does Label-Trim perform on naturally contaminated datasets? Synthetic noise may not reflect real-world anomalies.\n2. How does Label-Trim perform on high contamination rate (>10%) data? \n3. Would the method fail catastrophically when the number of outliers exceeds the labeling budget *m*?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E2Io5RAyZj", "reviewer": "Reviewer_HWrV", "summary": "This manuscript, titled \"Robust Conformal Outlier Detection under Contaminated Reference Data,\" focuses on the problem of conformal outlier detection in the presence of contaminated reference data. It discovers that in non-adversarial scenarios, data contamination makes conformal prediction methods conservative and reduces their detection power. To address this, the manuscript proposes the Label-Trim method, which utilizes a limited labeling budget and an outlier detection model to remove outliers from the contaminated data. Theoretical analysis demonstrates that this method can approximately and effectively control the error rate under certain conditions. Experiments comparing multiple methods on several datasets show that the Label-Trim method can significantly enhance the detection power while controlling the type-I error rate, and it performs particularly well in scenarios with low error rates and low contamination rates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and encouraging feedback. To fully address your comments, we have conducted additional experiments, available at https://tinyurl.com/rcod-exps. In our responses below, we refer to these as Supp-Figure X and Supp-Table Y.\n\n> **R1:** The effectiveness of the Label - Trim method depends on the accuracy of the pre-trained outlier detection model. The paper does not thoroughly explore how to select or improve the outlier detection model for better performance of the overall system.\n\nYou're absolutely right that the performance of Label-Trim depends on the quality of the outlier detection model used to rank points for annotation. To explore this further, and in response to your concern (as well as R4 from Reviewer Zuz4), we conducted new experiments using outlier detection models with lower performance than Isolation Forest, specifically Local Outlier Factor and one-class SVM on the tabular datasets. As shown in Supp-Figures 2–3, when the underlying model struggles to distinguish inliers from outliers, Label-Trim does not show meaningful power gains over the standard conformal method. This is expected, as the model’s scores no longer reliably identify outliers.\n\nThat said, as with any outlier detection task, we recommend using the best model available or fine-tuning a given model using a small set of labeled outliers, something that naturally improves performance. Importantly, Label-Trim is model-agnostic: it imposes no constraints on model complexity or architecture. This ensures that our approach remains compatible with future developments in outlier detection.\n\n> **R2:** Although the paper uses visualizations to present experimental results, there is a lack of visual aids to help readers understand the working mechanism of the Label - Trim method.\n\nThank you for this excellent suggestion. We agree that a visual explanation would help communicate the logic and steps of the method more clearly. In the revised version, we will include a new schematic illustration that walks through the Label-Trim pipeline.\n\n> **R3:** While the synthetic data experiments are useful, the authors could explore a wider range of outlier injection strategies.\n\nWe appreciate this point and want to clarify that all our experiments are conducted on real-world datasets (both inliers and outliers originate from real applications). That said, we agree that modeling more nuanced corruption is important. In our response to Reviewer 2tsw (R4), we include additional experiments with more strategic outlier injection strategies, which make the detection task more challenging.\n\n> **R4:** In real - world datasets, the preprocessing steps might be too simplistic.\n\nWe’re not entirely sure what specific aspect of preprocessing you are referring to. In our work, we follow the preprocessing protocols established in prior studies (Bates et al., 2023; Zhang et al., 2024; Yang et al., 2022). Our main departure is the introduction of contamination into the training and calibration sets to reflect realistic scenarios more closely. If there are particular concerns about the preprocessing that we overlooked, we would be happy to address them.\n\n> **R5:** There could be more diverse baselines considered.\n\nWe agree that comparisons with a broader set of baselines are useful. In addition to the four methods evaluated in the main manuscript, we have now added results for two additional outlier detection models on tabular data (Local Outlier Factor and one-class SVM), as well as two new models for visual data (ReAct with a VGG19 backbone and SCALE with a ResNet backbone). We’ve also included new experiments under higher contamination rates (up to 15%) and across a range of type-I error levels. As shown in the updated results, type-I error is consistently controlled, and power improves when the detection model separates inliers from outliers with reasonable accuracy. \n\n> **R6:** While type - I error rate and power are important metrics, additional metrics could be considered.\n\nWe focused on type-I error and power because they are the standard evaluation metrics in the conformal prediction and outlier detection literature. These metrics directly capture the validity (false positive control) and utility (detection rate) of the method. To provide additional insight, we also report the number of trimmed outliers in the contaminated reference set as a measure of cleaning effectiveness.", "claims_and_evidence": "The claims in the manuscript are supported by clear and convincing evidence. Theoretically, the conservativeness of the conformal outlier detection method and the effectiveness of the Label-Trim method are proven through derivations. Experimentally, the comparison results on multiple real-world datasets support the claims of the article.", "methods_and_evaluation_criteria": "In this manuscript, the proposed Label-Trim method and evaluation criteria are well-suited to the problem of outlier detection with contaminated reference data. The Label-Trim method effectively deals with contaminated data by using a limited labeling budget and an outlier detection model. The use of multiple benchmark datasets and comparison with multiple baseline methods comprehensively assess the method's performance.", "theoretical_claims": "I have checked the correctness of the proofs for the theoretical claims in the paper. The proofs of Lemma 2.2 and Theorem 3.1 are particularly crucial. For Lemma 2.2, which analyzes the conservativeness of standard conformal p-values under contaminated data, the proof process is clear and the logic is rigorous. The proof of Theorem 3.1, which validates the Label-Trim method, is also well-structured. It constructs an imaginary \"mirror\" version of the method to analyze the relationship between different quantiles. It provide solid theoretical support for the claims in the paper.", "experimental_designs_or_analyses": "I've examined the experimental designs and analyses in the paper, and they are generally sound and valid. They use a diverse set of benchmark datasets, including three tabular datasets (shuttle, credit card, KDDCup99) and six visual datasets. This variety helps capture different data characteristics and real-world scenarios, enhancing the generalizability of the results.", "relation_to_broader_scientific_literature": "The paper's key contributions are closely related to the broader scientific literature in multiple ways. The Label-Trim method in this paper builds on the existing understanding of conformal prediction. By addressing the over-conservativeness problem in contaminated data scenarios, it fills a gap in the literature. It provides a new approach to enhance the power of conformal methods while maintaining type-I error control, which is an important addition to the body of knowledge on outlier detection and conformal prediction.", "essential_references_not_discussed": "After a thorough review, there don't seem to be any essential related works that are not cited or discussed in the paper. The paper comprehensively references prior research on conformal inference under distribution shifts, robustness to data contamination, and outlier detection.", "other_strengths_and_weaknesses": "S1: The paper's focus on outlier - robust conformal outlier detection with contaminated reference data fills a significant gap in the existing literature. While many studies assume clean reference data in conformal prediction, this work directly addresses the practical issue of contamination, offering new insights into the behavior of conformal methods under such conditions.\nS2: The Label - Trim method is a creative solution. By combining a pre - trained outlier detection model with a limited labeling budget to selectively clean the contaminated reference set, it presents a unique approach to enhancing the power of conformal outlier detection while maintaining type - I error control. This method offers a practical alternative to existing data - cleaning strategies in the context of conformal prediction.\nS3: The theoretical analysis of the conservativeness of conformal methods in the presence of contaminated data and the validation of the Label - Trim method contribute to the theoretical understanding of conformal prediction. The results can serve as a foundation for further research in robust conformal inference and outlier detection.\nS4: The paper is well - structured, with a clear introduction that motivates the research problem, followed by detailed sections on setup, methods, experiments, and discussion. Each section is logically connected, making it easy for readers to follow the flow of the research.\n\nW1: The effectiveness of the Label - Trim method depends on the accuracy of the pre-trained outlier detection model. If the outlier detection model performs poorly, the performance of the Label - Trim method may be severely affected. The paper does not thoroughly explore how to select or improve the outlier detection model for better performance of the overall system.\nW2: Although the paper uses visualizations to present experimental results, there is a lack of visual aids to help readers understand the working mechanism of the Label - Trim method.\nW3: While the synthetic data experiments are useful, the authors could explore a wider range of outlier injection strategies.\nW4: In real - world datasets, the preprocessing steps might be too simplistic. \nW5: There could be more diverse baselines considered.\nW6: While type - I error rate and power are important metrics, additional metrics could be considered.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aOPoOM4uAe", "reviewer": "Reviewer_6UJm", "summary": "The authors analyze the impact of contamination on the validity of conformal methods. They show that under realistic, non-adversarial\nsettings, calibration on contaminated data yields conservative type-I error control. This conservativeness, however, typically results\nin a loss of power. To alleviate this limitation, they propose a novel, active data-cleaning framework that leverages a limited labeling budget and an outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the careful review and encouraging feedback. We answer your question below.\n\n> **R1:** How does the concept of contaminated reference set relate to the ambiguous ground truth case?\n\nTo the best of our understanding, the papers you referenced study ambiguity in the labeling process within a multi-class classification setting, where some examples may plausibly belong to multiple classes and this ambiguity is explicitly reflected in the labels. In contrast, our work focuses on outlier detection rather than classification, and our setting does not involve any explicit label ambiguity: all calibration points are labeled as inliers, but some may in fact be outliers. The contamination is entirely latent: we have no indication of which points are mislabeled, nor any signal of uncertainty in the labels.\n\nThat said, our results do offer some connection to the point of view of the papers you mentioned. As shown in Lemma 2.2 and supported by our experiments, when there is uncertainty about a point’s status, treating it as an inlier is a conservative strategy that preserves type-I error control. This highlights a potentially interesting connection between label ambiguity and reference set contamination, though the two settings at this point still appear quite fundamentally different in how they represent and handle possible labeling errors.\n\nIn any case, we agree this connection is worth exploring further in the future and will cite the papers you mentioned while incorporating this discussion into Section 5 of the revised manuscript.", "claims_and_evidence": "The claims made by the authors are supported by clear evidence, both theoretical and experimental in nature.", "methods_and_evaluation_criteria": "The proposed methods seem correct to show the validity of the proposed method.", "theoretical_claims": "There are two novel theoretical claims in the paper, whose proofs seem both largely correct.", "experimental_designs_or_analyses": "The experiments appear to be sound and valid, and support the theoretical claims.", "relation_to_broader_scientific_literature": "The related literature is discussed in sufficient depth. A suggestion for further related works is given in the Questions section.", "essential_references_not_discussed": "None; see also Question **Q1**.", "other_strengths_and_weaknesses": "The approach seem novel, and intuitively appealing. The results proved are very interesting, and the impact of the assumptions (and more in general of the limitations) is discussed at length.", "other_comments_or_suggestions": "See the Questions section.", "questions_for_authors": "**Q1** How does the concept of contaminated reference set relate to the ambiguous ground truth case? Refer e.g. to \n\nhttps://openreview.net/forum?id=CAd6V2qXxc\n\nhttps://openreview.net/forum?id=L7sQ8CW2FY\n\nhttps://proceedings.neurips.cc/paper_files/paper/2024/hash/d42a8bf2f40555d4a5120300f98c88f6-Abstract-Conference.html)\n\nI'd like the authors to comment on this (possible) relationship.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper addresses conformal outlier detection in the presence of contaminated reference data. It is shown that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error, which leads to a loss of power. To alleviate this limitation, a method referred to as Label-Trim is proposed to remove as many outliers as possible from the calibration set without altering the inlier score distribution. The upper bound on the type-I error rate is provided. During the discussion period after the rebuttal, all of reviewers agree that the paper  has valuable contributions.
Decision: Accept (poster)