Robust Conformal Prediction Using Privileged Information
Authors: Shai Feldman
Yaniv Romano
Venue: NeurIPS 2024 poster (2024)
Keywords: Conformal Prediction
Uncertainty Quantification
Distribution Shift
Corrupted Data
Privileged Information
Primary area: safety_in_machine_learning

Abstract: We develop a method to generate prediction sets with a guaranteed coverage rate that is robust to corruptions in the training data, such as missing or noisy variables. 
Our approach builds on conformal prediction, a powerful framework to construct prediction sets that are valid under the i.i.d assumption. Importantly, naively applying conformal prediction does not provide reliable predictions in this setting, due to the distribution shift induced by the corruptions. 
To account for the distribution shift, we assume access to privileged information (PI). The PI is formulated as additional features that explain the distribution shift, however, they are only available during training and absent at test time.
We approach this problem by introducing a novel generalization of weighted conformal prediction and support our method with theoretical coverage guarantees. 
Empirical experiments on both real and synthetic datasets indicate that our approach achieves a valid coverage rate and constructs more informative predictions compared to existing methods, which are not supported by theoretical guarantees.

Robust Conformal Prediction Using Privileged Information
Shai Feldman
Department of Computer Science
Technion, Israel wwwhheeh..n..... htw.dmaannnsttecccooo.o.coo ac..l
shai fel.dallmmnnee
Yaniv Romano
Departments of Electrical
Computer Engineering and of Computer Science
Technion, Israel
premano@cs..eehhioon ac.
Abstract
We develop a method to generate prediction sets with guaranteed coverage rate that is robust to corruptions in the training data, such missing noisy variables. Our approach builds on conformal prediction, a powerful framework construct prediction sets that valid under the i.i.d assumption. Importantly, naively applying conformal prediction does not provide reliable predictions in this setting, due the distribution shift induced by the corruptions. To account for the distribution shift, we assume access to privileged information (PI). The PI is formulated additional features that explain the distribution shift, however, they are only available during training and absent at test time. We approach this problem introducing novel generalization of weighted conformal prediction and support our method with theoretical coverage guarantees. Empirical experiments on both real and synthetic datasets indicate that our approach achieves valid coverage rate and constructs more informative predictions compared to existing methods, which are not supported by theoretical guarantees.
Introduction
1.1 Motivation
Uncertainty quantification plays a pivotal role in increasing the reliability machine learning models. In this paper, focus situations where the training data corrupted, e.g., due missing variables or noisy labels. 'hese corruptions are ubiquitous high-stakes applications- such as diagnosing diseases, predicting financial outcomes, personalizing treatment plans for patients-inn in which the data-collection process is complex, resource-intensive, or time-consuming [1-5].
One way to enhance the trustworthiness of data-driven predictions is to provide uncertainty set containing the correct outcome at a user-specified coverage rate, e.g., 90%. Conformal prediction (CP) [6] is general framework for constructing such reliable prediction sets, however, it assumes that the training and test data samples are drawn i.i.d from the same distribution. This assumption does not hold in the problem setting consider in this work. in which the training data is corrupted or biased version of the ground truth. For instance, consider a medical application in which training data have missing incorrect labels for some patients in non-random pattern. Another example is a situation where we have missing feature values in the training data but, at test-time, we observe the full set of features. These examples illustrate common sources for distribution shift between
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the training and test data, which breaks the coverage guarantee of traditional CP techniques. In this work, we address this gap and propose a novel calibration technique, called privileged conformal prediction (PCP), which constructs provably valid uncertainty sets despite being employed with corrupted samples. Technically, we achieve this by utilizing privileged information information-addiin additional data available only during training time to account for the distribution shift induced by the corruptions.
1.2 Problem setup
Suppose we are given training samples (M,), (.()), Z,, M,)) where X² = (MM)
X' is the observed covariates, Yoobs = (M.) E y' is the observed response, Z4 E 2 is the privileged information (PI), and M, E (0. 1} is the corruption indicator. Specifically, if M; X or ye are corrupted, and if M, O, then X and y, correctly reflect the ground truth. In our setup, we require that the privileged information Z, explains the corruption occurences M,. Formally, we assume that the clean data variables are independent of the corruption indicator given the privileged information, i.e., (X), (0)) IL M| | Z.
then either
At inference time, we aim to provide reliable predictions for the clean test response Y = + (0) given the clean version of the features: X Xn+1 (0). That is, even though the observed Xobs yobs might be corrupted, the test X yust are always uncorrupted. Crucially, at test-time, we do not have access to the test privileged information Ziest 7 nor the clean test label yust Moreover, we assume that the PI Z is always clean and correctly reflects the ground truth. We now emphasize the importance of this problem setup by providing several examples.
Example 1 (Noisy response). Here, we refer to Y1(1) as noisy version of the ground truth response (.()). For instance, Y:uhs (M.) could be a label obtained either by non-expert annotator or an expert annotator, and Z: could be information about the annotator, such as their level of expertise. In this case, M,- = 0 (M, = 1) indicates that the annotator chose the correct (wrong) label y.uhs = (0) (Y, = (1)). In contrast, the features are always uncorrupted: X²()) = (1)).. Notice that the test Ylesi= = Yn+1 (0) always refers to the clean response. In this setup, since the PI, Z, is the information about the annotator, it is likely to explain the corruption appearances M,. That is, it is sensible to believe that the assumption (X (0), (0)) TL M Z is approximately satisfied. Example (Missing features). Here, (0) is the full clean feature vector, and (1) is a partial version of it, i.e., (1) 'NA' for some entries j. For example, consider an application where participants are requested to fill a user experience (UX) questionnaire, in which the goal is to predict user engagement. This trial consists of expert participants, who tend to fully answer the questionnaire, resulting in full X,(O), and non-experts, who tend to partially answer it, resulting in the incomplete (1). The PI Z; could be the level of expertise of the participant. Also, the response is always uncorrupted: Y:))) = Y1)).. We remark that at test time, the full feature vector X = Xx+1 (0) is completely available. Since the PI Zu is the information about the participant, it is likely to explain the missing indices My. Therefore, for this choice of Pl, we have good reason to believe that the conditional independence requirement, X (0), Y (0) IL M Z, is approximately satisfied.
Example (Missing response). Consider a medical setup where patients are being selected for a costly diagnosis, such MRI scan. Here, (0)
(1) is the more standard medical measure- ments of the i-th patient, such as age, gender, medical history. and disease- -specffc measurements. The PI Z; is the information manually collected by the doctor to choose whether the patient should be examined by an MRI scan. This information is obtained through, e.g., a discussion of the doctor with the patient, or a physical examination, and could include, for instance, shortness of breath, swelling, blurred vision, etc. The response Y,(0) is the disease diagnosis obtained by the MRI scan, and Y.(1) 'NA'. The missingness indicator M, equals O if the doctor decides to conduct an MRI scan, and 1 otherwise. At test time, our goal is to assist the doctors in future decisions before examining the patients, and hence the test pi Zuest is unavailable. This task is relevant in situations where the number of available doctors is insufficient to examine all patients. Here, Z; explains the missingness M,, and Mr does not depend on X; or Y, given Z,
With the above use cases in mind, our goal is to construct an uncertainty set C(XleM) C y for the unknown clean test variable yrest = (0) given the clean features test = Xn+1 (0). Importantly, this uncertainty set should be statistically valid and satisfy the following coverage requirement:
P(yttess E C(Xtes)) 2 1 - at
(1)
where 1 - a E (0, 1) is a pre-specified coverage rate, e.g., 90%. This property is called marginal coverage, as the probability is taken over all samples (I),, Yi(),, Z, M.)a+.
2

---

which are assumed to be drawn exchangeably (e.g., i..d.) from Px(o), Y0, The challenge in achieving (1) lies in the fact that there distribution shift between the training data ((), Y. {L, and the test data (X,, (0), Y (0)). Indeed, naively calibrating the model with the corrupted data may produce invalid uncertainty estimates [7]. Also, calibrating using only the clean data would result in biased predictions as the clean training samples are drawn from (x) Y (0)]M=0- while the test samples are drawn from (x(
To bypass this bias, we assume that the privileged information explains away the corruption appear- ances. Formally, we require that the corruption indicator is independent of the clean data conditional on the value of the privileged information. (X (0). Y (0)) TL M Z. This assumption implies that our setting is special case of covariate shift, with the covariates being the privileged information Z. Since the test PI Ztest = Za+1 is unknown at test time, conformal methods that account for covariate shift, such as weighted conformal prediction (MCP) [8] cannot be applied directly in the setup. In this paper, we re- re-formulate weighted conformal prediction and show how to construct uncertainty sets that satisfy the coverage requirement in (1) although Z is unavailable.
1.3 Our contribution
We introduce privileged conformal prediction (PCP)-a novel calibration scheme that effectively handles corrupted data, and constructs provably valid uncertainty sets in the sense of (1). Our key assumption is that the corruption indicator does not depend on the observed clean data given the privileged information, namely, (Y)) (0)) TL M Z. This assumption implies that the privileged information explains away the corruption appearances. Building on WCP, we offer a specialized calibration scheme that carefully utilizes only the observed training privileged data {Z, n to attain a valid predictive inference at test time. To enhance statistical efficiency, we further adapt PCP for scarce data, building on leave-one-out arguments [9, 10]. Importantly, all methods we offer are supported by theoretical valid coverage rate guarantee. Numerical experiments on both synthetic and real data show that naive conformal prediction techniques do not provide reliable uncertainty estimates, in contrast with the proposed PCP. To the best of our knowledge, this work is the first to propose a calibration scheme that generates statistically valid prediction sets, assuming that the privileged information explains away the corruption appearances. Software implementing the proposed method and reproducing our experiments is available at https: / /gitt hub. com/ Shai 128 pcp.
2 Background and related work
2.1 Conformal prediction
Conformal Prediction (CP) [6] is a powerful framework for constructing prediction sets that hold a marginal coverage rate guarantee, in the sense of (1). The general recipe construct such prediction sets is as follows. First, split the data into a proper training set, indexed by Ij, and a calibration set, indexed by Zz. Then, fit given learning model f. e.g., a random forest or neural network, on the training data. Next, evaluate the holdout prediction error of f by applying non-confornity score function S(-) IR to the calibration samples: S: S(X,, Y:; fi, Vi E Iz. Popular score functions include the absolute residual S(x, v: f) |f(x) y| in regression cases, where f is mean estimator, or - f(z), in classification settings, where f(x), the estimated probability of the y label given X = The latter is known the homogeneous prediction sets (HPS) score [6]. Other score functions include the CQR score [11] for regression tasks and the APS score [12] for classification tasks. Armed with the non-conformity scores, the conformal procedure proceeds by
computing the (1 + (1 - o-) empirical quantile of the calibration scores:
Q (1 1/ (1 -thth empirical quantile of the scores {8: (Sehe
(2)
where 1 - a is a user-specified coverage level. Lastly, the prediction set for the test point is given by CCP {u S(xtes y: f) < Q
The above procedure is guaranteed to generate predictive sets with valid marginal coverage (1) under the assumption that the calibration and test samples are exchangeable. We now turn to describe weighted conformal prediction (MCP) which designed to handle exchangeability violations that arise from covariate shifts.
3

---

2.2 Weighted conformal prediction
Weighted Conformal Prediction (WCP) [8] extends the conformal prediction framework handle covariate shifts. The key idea behind WCP is to weight the distribution of the calibration scores when taking their quantile in (2), so that the weighted scores 'look exchangeable with the test non-confornity score. For the interest of space, we will not present the general form of WCP, and instead focus on the setup presented in this work, in which (0), (())) 11 M Z. Under this assumption, the corruption indicator induces covariate shift between the observed clean calibration samples and the test sample, which explained by Z. That is, the clean calibration samples are drawn from (0), M-D- while the test sample is drawn from P Nevertheless, their distributions are equal conditionally on Z: ( [00) place, we follow the recipe of WCP and construct prediction set as follows. First, we compute the ratio likelihoods between the test and train data:
Px -1)) With this in
dP2
(2)
(2)
P(M = 0)
w(2
(3)
d. d.Patann
flest (0)
( (2)
P(M 1= 2)
0(M-0)
We define the set of uncorrupted calibration indexes as: INd weights are formulated as:
(i IE I2, M₁ =}. The normalized
w (1))
w (iest)
Pe( (
EKETY w (Za) + w (Zuajj Prest ((0S))
(4)
kezy w/ Zk) + w (Zust)
Then, the calibration threshold for the test point is defined as the 1 - a empirical quantile of the weighted distribution of the scores:
SCP (Zies)=
Quantile
1 O;
Σ p: (zissts)) + Prest (Ziest ) VEZY
(5)
and, the prediction set for the test sample is defined similarly to CP:
C (Xtest Z(ee ty S(X ,y: S Q (Z)
Remarkably, WCP produces uncertainty sets that achieve the desired marginal coverage rate (I) despite the induced covariate shift. Nonetheless, to implement this method, we must have access to Zist, which is required to obtain wZ In our problem setup, however, we assume that Ziest is unavailable, and thereby WCP cannot be directly applied. This highlights the key challenge we aim to tackle in this paper, but before describing our method we first outline additional related work.
2.3 Additional related work
The concept of learning from privileged information was introduced by [13], which proposes tech- niques to leverage additional knowledge available during training improve the prediction accuracy and accelerate algorithm convergence rate. This idea has been further explored to train models that are more robust to distribution shifts in the context of domain adaptation [14-16]. The method proposed in [17] utilizes PI to handle datasets containing weak labels and to obtain more accurate predictions. Furthermore, [18] combined model distillation with privileged information as way to enhance learning from multiple models and data representations. The integration of PI with traditional conformal prediction to generate more informative uncertainty estimates was explored in [19, 20]. This line of work stands in striking contrast with our proposal, as we present a novel robust conformal calibration procedure based on Pl. More broadly, there have been developed conformal methods that advance beyond the exchangeability assumption, such as WCP, among other contributions [7, 21-26). However, none of these works utilize to ensure the validity of the constructed prediction sets.
Proposed method
In this section, we present our main contribution, the privileged conformal prediction (PCP) method. Since the setup we study in this paper has not been explored in the literature of conformal prediction, we start by suggesting naive approach to achieve (1). Beyond serving baseline method for our PCP, this naive approach also reveals the challenges involved in constructing valid prediction sets when the calibration data is corrupted.

---

3.1 A naive approach: Two-Staged Conformal
Recall that WCP cannot be directly applied in our setup, since Z is unknown. To overeome this, the naive approach presented below consists of two stages: (i) estimate the unknown Z from the feature vector X and (ii) employ WCP with the estimated privileged information.
While this approach is intuitive, the estimation of Zust must be done in care: if the prediction of Z is incorrect, then WCP would not provide the desired coverage guarantee. As way out, instead of providing point estimate, we will construct an interval C² (Xtes) for Zrest given X using conformal prediction. This interval is guaranteed to contain the true PI Ztex with probability 1- 3. where 3 is miscoverage rate of our choice, e.g. 8 = 0.01. Since we do not know which E C2 (uss)) is the correct Z we sweep over all possible elements C² (X compute their weights, w(2), and take the largest weight:
W conservative (xtest)
w(2). =EC (XIm))
max
(6)
The intuition behind taking the largest weight lies in Lemma 1, which states that the larger the test weight is, the larger the threshold QPeP produced by WCP, which, in turn, increases the size of the prediction set. Armed with ( we can run WCP with a nominal coverage level 1 - a + 3 using the clean calibration samples and their weights {(X;" Y:ths, w (e(Zi)) and the conservative test weight, w (X (Xtes). We denote the weighted score quantile provided by WCP in (5) with this conservative test weight by QNCP Comservative" The prediction set is therefore defined as:
Ciwo-Staged (X = { S(Xles, y. S
QuCP conservative
An outline of this procedure is given in Algorithm 2 in Appendix B.1. The proposition below states that the uncertainty set generated by this naive approach is guaranteed to contain the test label y despite the presence of corrupted labels in the calibration set.
Proposition 1. Suppose that {(x,(0) X²(1), Y;()), Y((1), ; Z4, M.) are exchangeable, the ob- served covariates are clean, i.e., Vi: X (0) (1)), the covariate shift assumption holds, ie., (0), (0)) IL M Z, and Pz is absolutely continuous with respect to PZ|M=0- Then, the prediction set CTwo-Stayed( (aaas)(( from (7) achieves a valid coverage rate:
P(y CTo CTToo-Saaedd (aaeed((tss)
21-a.
The proof is given in Appendix A.1. While this two-staged approach constructs valid prediction sets, has several limitations. First, it requires predicting not only ytest but also Zest, and the prediction of the latter is anticipated to increase the uncertainty encapsulated in the resulting prediction set for ylest, This algorithm also requires iterating over all 2 E C2 (Xtest), which can be computationally expensive, especially when 2 is continuous for Ziest might contain unlikely, or off-support values of Z.
multi-dimmnsional. In addition, and perhaps more
importantly, the prediction set C²2 (X(tess)) (eesi This can lead to extreme  cvmservvatvee (Xles), which, in turn, results in unnecessarily large prediction sets for Y rest Moreover, this naive method assumes that the calibration features X reflect the ground truth, i.e., X = (0), and thus the coverage guarantee does not hold in situations where the features are missing or noisy. This discussion emphasizes the challenges in designing calibration scheme that not only provides robust coverage guarantees but is also computationally and statistically efficient. In the next section, present our main proposal which fully resolves all limitations of this naive approach.
3.2
Our main proposal: Privileged Conformal Prediction
In this section, we introduce our procedure to construct prediction sets with a valid coverage rate under the setting of corrupted samples. We begin similarly to CP, as described in Section 2.1, and split the data into a training set, Zi, and a calibration set, Iz. Next, we a predictive model f on the training data, and compute non-conformity score for each calibration sample:
S S(X."h y, f), Vi E Iz.
Similarly to MCP and Two-Staged methods, we rely on the likelihood ratio of the training and test distributions, and compute the weight of the i-th sample: w; == P(M-0) The problem in
P0-=0
5

---

WCP is that the scores threshold Q (2 from (5) depends on Zee, Here, we follow the intuition behind the two-staged baseline and propose an algorithm that provides a fixed threshold Q that is not function of 7est This threshold can be thought of as a conservative estimate, or upper bound of Quee/ (z which based on the calibration data, and does not require Z To achieve this, we consider every calibration point i E I2 the i-th score threshold Q. The final PCP test score threshold, Q is defined as the (1 3) empirical quantile of the calibration thresholds {Qi)ieTa where BE (0, a) is a level of our choice, e.g.. 3 = 0.01.
test point, and run WCP subroutine to obtain
Formally, we consider the i-th sample as a test point and compute the normalized weight of the j-th sample:
p
Wy
Vi,j Iz.
kEzy wy W/
Notice that p) extends the WCP weights, Py. from (4), since p. = Pj n+1 Now, we compute the i-th threshold Q; by applying WCP using the uncorrupted calibration data:
1 (1-
Qi = Quantile
3;
Σ v/,os; + p1000 JEIN
(8)
Next, we extract from (Qi)ee,, a conservative estimate of Qaee (Zust) = Qu+1, denoted by QPCP:
(1-3) 3: Σ
QeeP := Quantile
+
Tz + 800
Iz +
iEZs
Finally, for new input data Xlesi, we construct the prediction set for Yisst as follows:
CPCP -vu S(X y, S QPCF)
For convenience, Algorithm I summarizes the above procedure and Algorithm 3 details more efficient version of this procedure. We now show that the prediction sets constructed by PCP achieve valid marginal coverage rate.
Theorem Suppose that ((X.(0), Y(O), Y,(1), Ze, M.Y+ are exchangeable, (X()), II M Z, and Pz is absolutely continuous with respect to P2|M=0 Then, the prediction set C C constructed according to Algorithm I achieves the desired coverage rate:
P(yles E CCe 21
The proof is given in Appendix A.2. We remark that while Theorem requires that the PI satisfies the conditional independence assumption, i.e., (X))), (()) TL M Z, in Appendix A.5 we relax this assumption and provide a lower bound for the coverage rate for settings where the conditional independence assumption is not exactly satisfied. We pause here emphasize the significance of Theorem 1. The key challenge in proving this result lies in the fact that the (Q, (eINeeunnn++) are not exchangeable. This is attributed to the fact that for every i ITe the threshold Q: is defined using its own score S,, while the test Q does not rely on its corresponding score Sa+1 S(Xtess yest f). As side comment, if the thresholds were exchangeable, the proof was much simpler, as QPCE would be greater than Q with probability 1 - 3. In this case, CCCP includes CRCP at a high probability, meaning that it achieves the desired coverage rate. Due to the lack of exchangeability, the argument above is incorrect. Indeed, the validity of PCP does not follow directly from the guarantee of WCP, and it requires additional technical steps.
We now turn to discuss the role of First, we emphasize that Theorem holds for any choice of 3 E (0, a). Therefore, / only affects the sizes of the uncertainty sets. Intuitively, as 3 -> a, a higher quantile of the weighted distribution
the scores is taken, and lower quantile of the Q is taken. Similarly, as 3 - 0 a lower quantile of the weighted distribution of the scores is taken, and a higher quantile of the Q,'s is taken. An optimal /3 can be considered as the |3 that leads to the narrowest intervals. Such optimal 3 can be practically computed with a grid of values for 3 in (0, a), using validation set. Nonetheless, in our experiments, we directly chose that is close to O. Appendix E.5 conduct ablation study analyzing the effect of 3 on synthetic dataset.

---

Lastly, we note that while the real ratios of likelihoods, are required to provide the validity guarantee in Theorem 1, PCP can be employed with estimates of w,. These weights can be estimated in the following way. The first step is estimating the conditional corruption probability given Z, ie.. PPM = 0 = 2), using the training and validation sets with any off-the-shelf classifier. We remark that this classifier can be fit on unlabeled data, as this classifier only requires the PI Z and the corruption indicator M. We denote the model outputs by P(M
Z = z). Next, we estimate the marginal corruption probability directly from the data: p(M 0) 1 - i- M,. Finally, the p -(M=0) estimated weights are computed according to (3): ú; = ((2)) = p(M=0)
Even though PCP is not guaranteed to attain the nominal coverage level when employed with the estimates ti; the experiments from Section 4.3 indicate that does achieve a conservative coverage rate in this case. The effect of inaccurate estimates of the coverage rate attained by PCP could be exciting future direction to explore, perhaps by drawing on ideas from [26].
Algorithm 1: Privileged Conformal Prediction (PCP)
Input:
Data (Xxthh, y, Ze, M.) e X x x Z {0, 13. S is n, weights {w,)i=1, miscoverage level a E (0, 1), level BE (0, a), an algorithm f, a score function S, and test point X x.
Process:
Randomly split f1...., n into two disjoint sets I1. I2. Fit the base algorithm f on the training data {(X,"h ),bb)),
Compute the scores S. = S(x:hh, y: f) for the calibration samples, i ZT.
Compute a threshold Q; for each calibration sample according to (8).
Compute QPCP, the (1 (3 quantile of (Q: Nez, according to (9).
Output:
Prediction CFCP (x) {y: : S(x, yi s Q.
3.3 Privileged Conformal Prediction for scarce data
In this section, we present an adaptation of PCP to handle situations where the sample size is small. While PCP is computationally light, it requires splitting the data into training and calibration sets. This restriction is significant for small datasets in which the reduction in computations from the data splitting comes the expense of statistical efficiency. To avoid data splitting, we build on the leave-one-out jackknife+ method [9, 27], and, in particular, its weighted version JAW [21]. The method we propose, which we refer to as LCO-PCP, better utilizes the training data compared PCP. For the interest of space, we refer to Appendix B.3 for the description of LOO-PCP. The following Theorem states that prediction C106-PCP constructed by LOO-PCP is guaranteed to achieve valid coverage rate under our setup. In Appendix A.3 we provide the proof, which relies results from [21].
Theorem Suppose that ((X, Y,(),, Y,(1), Z₁, M.)+ are exchangeable, (X), (0)) IL M Z. and Pz is absolutely continuous with respect to PZ|M=0. Then, the prediction set CLoS-PCP (Xles) constructed according to Algorithm 4 satisfies:
P((Yes E C.000-PPP (xtes)) 2 2a.
We note that in contrast to split CP, here, the coverage guarantee appears with a factor 2 in a. We refer the reader to [9] for detailed explanation of why the factor 2 is necessary and cannot be removed. Nonetheless, it is well-known that this jackknife approach greatly improves statistical efficiency compared to split conformal methods.
4 Applications
In this section, we exemplify our proposal in three real-life applications. In all experiments, we randomly split the data into training, validation, calibration, and test sets. We fit a base learning model on the training data and use the validation set to avoid overfitting. We calibrate the model using the
7

---

calibration data with the proposed PCP or with a baseline technique, and evaluate the performance on the test set. In all experiments, the calibration schemes are applied to achieve a 1 a 90% marginal coverage rate. We use the CQR [1 I] non-conformity score in regression tasks, and the homogeneous prediction sets (HPS) non-conformity score [6, 28] classification tasks. Appendix D describes the full details about the network architecture, training strategy, datasets, corruption technique, and this experimental protocol. The specific formulation of the PI is described in each experiment. In this section, we focus on three use cases: causal inference, missing response, and noisy response. We demonstrate the applicability of PCP on more datasets, and under different corruptions, including additional causal inference tasks in Appendix E.I, more response corruptions in Appendix E.3 and in Appendix E.4.1, and missing features settings in Appendix E.4.2.
4.1 Causal inference: semi-synthetic example
We begin with a causal inference example. in which the goal is to obtain inference for individual treatment effects [29]. In this setting, (0) = Xx(() E X' denotes the features, Z; denotes the privileged information, M, E {0, 1 denotes the binary treatment indicator, and (0)),, Y, (1) E R denote the counterfactual outcomes under control and treatment conditions, respectively. Recall that we only observe y: = (M,) and that the PI explains the treatment pattern
1 I .M] Z. In this experimental setup, our goal is to construct a prediction set that covers the true test potential outcome under control conditions, i.e., Y (0), at a user-specified level 1 - a = 90%. Alternatively, we could also aim to predict the outcome under treatment Y (1). However, in this experiment, we focus on 1 (0) since the dataset we use is highly imbalanced and there are few samples from the treatment group. This task is compelling since it can be used to generate valid uncertainty interval for the individual treatment effect (ITE), (1) - (,)),, which is a great interest for many problems [30-33]. For instance, the work in [34] shows how to construct valid interval for the ITE by combining intervals for Yn+1 (0) and (=+1(1) (1). We remark that providing statistically valid prediction intervals for (0) is challenging due to the distribution shift between the observed control responses, which are drawn from Pr(o) M-0 -0, whereas the test control response is drawn from (o)- Moreover, in this example, we intentionally design M, to induce such distribution shift; see Appendix D.1 for more details on the definition of M,.
We test the applicability of our method on the semi-synthetic Infant Health and Development Program (IHDP) dataset [35], in which the objective is to find the effect of specialist home visits on a child's future cognitive test scores. That is, the feature vector X, contains covariates describing the child's characteristics, the treatment M, is the specialist home visits indicator, and the potential outcomes (,))), (;()) are the future cognitive test scores. Since this dataset does not originally contain privileged information variable, we artificially define it as the entry in X; that correlates the most with (0). This feature is then removed from X,, so it is unavailable inference time.
Since the IHDP dataset contains only 747 samples, we apply 1O0-PCP in this example and compare it to the following calibration techniques. The first method is naive jackk i fet, which uses only the control samples and does not account for distribution shifts. The second and third techniques are two versions of JAW [21], which is weighted conformal version of the jackknife+. The first version (Naive WCP) naively uses an estimate of Pasix as the likelihood ratio weights instead of PM12 as zrest is unknown. The second (Infeasible WCP) is an infeasible method which requires access to the unknown test privileged information Zrest for the computation of the likelihood ratio weights which can be considered as an oracle calibration process. Importantly, the infeasible JAW and the proposed method use the true corruption probabilities when computing the weights w(()) from (3).
Figure I reports the coverage rates and interval lengths of each calibration scheme. This figure shows that naive jackkifer and the naive JAW achieve a lower coverage rate than desired. This anticipated, as both schemes do not accurately account for the distribution shift. In contrast, the infeasible JAW and PCP achieve the desired coverage rate. This is not a surprise, as JAW is guaranteed to attain the nominal coverage level when applied with the correct weights [21, Theorem 1]. However, this method is infeasible to implement in contrast with our proposal, which, according to Theorem 2, is guaranteed to cover the response at the desired rate without using the test privileged information. Furthermore, Figure reveals that PCP constructs intervals with approximately the same width as the ones generated by the infeasible JAW. This indicates that we do not lose much in terms of statistical efficiency by not having access to the test privileged information Z
8

---

Coverage rate
Interval length
6-
80
Naive CP
Naive WCP
Infeasible WCP
Privileged CP
Naive CP
Naive WCP
Infeasible WCP
Privileged CP
Method
Method
Figure 1: Causal inference experiment: IHDP dataset. The coverage rate and average interval length achieved by naive jackknife+ (Naive CP), naive JAW which considers only X to cope with the distribution shift (Na vve WCP), infeasible JAW which uses Z (In feas bll WCP), and the proposed method vegeed CP). The metrics are evaluated over 50 random data splits.
4.2 Missing response variable: semi-synthetic example
In this section, we study the performance of PCP and compare it to baselines in a missing response setting using six real datasets: Facebook1,2 [36], Bio [37], House [38], Meps [39] and Blog [40]. Since these datasets do not originally contain privileged information, we artificially define Z; as the feature from X, that correlates the most with Y; and then remove from X₃. Furthermore, since all response variables are present in these datasets, we artificially remove the responses in 20% of the samples. We intentionally set the missing probability in a way that induces a distribution shift between the missing and observed variables. In Appendix D.1 we provide the full details about the corruption process and how we impute the missing data.
We compare the proposed method (PCP) to the following calibration schemes: a naive confor- mal prediction (Naive CP); naive WCP, which considers only X cope with the distribution shift; the two-staged baseline Sta--d: and an infeasible weighted conformal prediction (Infeasible WCP) which has access the test privileged information Zrest, Importantly, the baseline Two-Staged, the infeasible WCP, and PCP use the real corruption probabilities when computing the weights w(2) in (3). In contrast, Naive WCP estimates the corruption probability conditioned on X from the data. Figure 2 presents the performance of each calibration scheme, showing that the naive approach (Naive CP) consistently produces invalid prediction intervals. Naive CP does not provide guarantees under distribution shifts. Figure
This is anticipated, also shows that Swa-St generates too wide intervals, resulting conservative coverage rate of approximately 95%. By contrast, the infeasible WCP and the proposed PCP consistently achieve desired 90% level. Crucially, PCP is comparable the interval length to the infeasible WCP. In conclusion, this experiment demonstrates that PCP constructs intervals that are both reliable and informative.
Coverage rate
95
Method
Naive CP Naive WCP
85
Two Staged
$8
Infeasible WCP Privileged
facebook_1
facebook_2
house
meps19
blog
Dataset
Interval length
1.0
0.5
0.0
facebook
facebook 2
bio
house
meps19
blog
Dataset
Figure 2: Missing response experiment. The coverage rate and average interval length obtained by
various methods; see text for details. Performance metrics are evaluated over random data splits.

---

4.3 Noisy response variable: real example
In what follows, we examine the performance of the proposed technique on the CIFAR- ION [41] image recognition dataset that contains noisy labels. Here, X is an image of one out of ten possible objects, and Y its corresponding label. The noisy response, (1), is the label annotated by one human annotator, while (0) denotes the clean label obtained from CIFAR-10 [42]. That is, M = 0 indicates that the annotator correctly labeled the image. Similarly to [17], we define the privileged data as information about the annotators. Specifically, the variable Z: contains two features: (i) the number of unique labels suggested by three annotators for the i-th sample, and (ii) the time took to annotate the corresponding sample batch, which contains ten images. In this experiment, we compare our method (PCP) to the following calibration schemes: a naive conformal prediction, applied either with the noisy labels Naive CP (clean + noisy) or ignoring them Naive CP (only c clean): ; the two-staged baseline (Two Staged CP); an infeasible WCP (Infeasible WCF) which assumes access to the unknown test privileged information Z Additionally, since the corruption probabilities are not given in this dataset, we estimate them from the data and use these estimates compute the weights w in (3).
Figure 3 presents each calibration scheme's coverage rate and uncertainty set size. This figure shows that Naive CP applied with noisy labels tends to overcover the clean label. This behavior is consistent with the work in [25, 43], which suggests that naive CP constructs conservative uncertainty sets when employed on data with dispersive label-noise. Figure 3 also indicates that calibrating the model only on the clean samples leads to invalid prediction sets that tend to undercover the clean test label. Observe also that the two-stage baseline is overly conservative, it encapsulates the error in predicting both Zre and the label. In contrast, the coverage rate of infeasible WCP and our proposed PCP is much closer to the desired level, yet slightly conservative. We suggest two possible explanations for this behavior: (i) the weights used are only estimates of the true likelihood ratios; (ii) in this real data we consider here, the PI may not fully explain the corruption mechanism. This highlights the robustness of our method to violations of our assumptions in the specific use-case studied here. Lastly, remark that the prediction sets of PCP have similar set size the sets constructed by the infeasible WCP, which is in line with the results from Section 4.1 and Section 4.2.
Coverage rate
95
Set size
90
Naive CP (clean (only clean) Naive WCP
Naive CP
Two Staged CP
Infeasible WCP
Privileged CP
Naive (clean + noisy) CP (only clean) Naive WCP
Two Staged CP
Infeasible WCP
Privileged CP
Method
Method
Figure 3: Noisy response experiment: CIFAR-ION dataset. Average coverage and set size obtained by various methods; see text for details. The metrics are evaluated over 20 random data splits.
5 Discussion and impact statement
In this paper, we introduced PCP, a novel calibration scheme to reliability quantify prediction uncertainty in situations where the training data is corrupted. The validity of our proposal is supported by theoretical guarantees and demonstrated in numerical experiments. The key assumption behind our method is that the features and responses are independent of the corruption indicator given the privileged information. This conditional independence resembles the strong ignorability assumption in causal inference [44-46]. While acquiring PI that satisties this requirement can be challenging, our work relaxes the strong ignorability assumption, as the confounders are allowed to be absent during inference time. An additional restriction we make is that the true conditional corruption probability must be known to provide theoretical coverage validity. However, our numerical experiments indicate that estimating these probabilities leads to reliable uncertainty estimates. A promising future direction would be to theoretically analyze the effect of inaccurate weights on the coverage guarantee, e.g., by borrowing ideas from [26]. Finally, we should note that there are potential social implications of our method, akin to many other works that aim to advance the ML field.
10

---

Acknowledgments and Disclosure of Funding
Y.R. and S.F. were supported by the ISRAEL SCIENCE FOUNDATION (grant No. 729/21). Y.R. thanks the Career Advancement Fellowship, Technion. Y.R. and S.F. thank Stephen Bates for insightful discussions and for providing feedback on this manuscript.
References
[1] H.A. Kahn and C.T. Sempos. Statistical Methods in Epidemiology. Monographs in epidemiolggy and biostatistics. Oxford University Press, 1989.
[2] David E Lilienfeld and Paul Stolley. Foundations of epidemiology. Oxford University Press, USA, 1994.
[3] Steven Piantadosi. Clinical trials: a methodologic perspective. John Wiley & Sons, 1997.
[4] Steve Selvin. Statistical analysis of epidemiologic data, volume 35. Oxford University Press, 2004.
[5] Floyd Fowler Jr. Survey research methods. Sage publications, 2013.
[6] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, NY, USA, 2005.
[7] Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani, Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816 845, 2023.
[8] Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
[9] Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, )9(1): 507, 2021.
[10] Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74:9-28, 2015.
[11] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
[12] Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, pages 3581-3591, 2020.
[13] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6):544-5557 2009.
[14] Saeid Motiann Domain Adaptation and Privileged Information for Visual Recognition. West Virginia University, 2019.
[15] Adam Breitholtz. Towards practical and provable domain adaptation. PhD thesis, Chalmers Tekniska Hogskola (Sweden), 2023.
[16] Judith Hoffman. Adaptive learning algorithms for transferable visual recognition. University of California, Berkeley, 2016
[17] Yanshan Xiao, Zexin Ye, Liang Zhao, Xiangjun Kong, Bo Liu, Kemal Polat, and Adi Alhudhaif. Privileged information learning with weak labels. Applied Soft Computing, 142:1 142:110298, 2023.
[18) David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. International Conference on Learning Representations, 2016.
[19] Meng Yang, Ilia Nouretdinov, and Zhiyuan Luo. Learning by conformal predictors with additional information. In Artificial Intelligence Applications and Innovations: 9th IFIP WG 12.5 International Conference, AIAI 2013, Paphos, Cyprus, September 30-October 2, 2013, Proceedings 9, pages 394-400. Springer, 2013.

---

[20] Niharika Gauraha, Lars Carlsson, and Ola Spjuth. Conformal prediction in learning under privi- leged information paradigm with applications drug discovery. In Conformal and Probabilistic Prediction and Applications, pages 147-156. PMLR, 2018.
[21] Drew Prinster, Anqi Liu, and Suchi Saria. Jaws: Auditing predictive uncertainty under covariate shift. Advances in Neural Information Processing Systems, 35:35907-359200 2022.
[22] Maxime Cauchois, Suyash Gupta, Alnur Ali, and John Duchi. Predictive inference with weak supervision. arXiv preprint arXiv:2201.082: 2022.
[23] Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association, pages 1 66, 2024.
[24] Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. In International Conference on Machine Learning, pages 40578-406044 PMLR, 2023.
[25] Matteo Sesia, YX Wang, and Xin Tong. Adaptive conformal classification with noisy labels. arXiv preprint arXiv:2309.05092, 2023.
[26] Yonghoon Lee, Edgar Dobriban, and Eric Tchetgen Tchetgen. Simultaneous conformal prediction of missing outcomes with propensity score e-discretization. arXiv preprint arXiv:2403.04613, 2024.
[27] Chirag Gupta, Arun K. Kuchibhotla, and Aaditya Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 108:6,,,,, 2022.
[28] Jing Lei, James Robins, and Larry Wasserman. Distribution-fre prediction sets. Journal of the American Statistical Association, 108(501):278-22) 287, 2013.
[29] Miguel A Hernán and James M Robins. Causal inference, 2010.
[30] Jennie E Brand and Yu Xie. Who benefits most from college? evidence for negative selection in heterogeneous economic returns to higher education. American sociological review, 75(2):273- 302, 2010.
[31] Stephen Morgan. Counterfactuals, causal effect heterogeneity, and the catholic school effect on learning. Sociology of education, pages 341-374, 2001.
[32] Yu Xic, Jennie Brand, and Ben Jann. Estimating heterogeneous treatment effects with observational data. Sociological methodology, 42(1)3314-347 2012.
[33] Jean-Pierre Florens, James Heckman, Costas Meghir, and Edward Vytlacil. Identification of treatment effects using control functions in models with continuous, endogenous treatment and heterogeneous effects. Econometrica, 76(5):1 191-1206, 2008
[34] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, (3(5):9119938 2021.
[35] Jennifer Hill. Bayesian nonparametric modeling for causal inference. Journal of Computa- tional and Graphical Statistics, 20(1):217-240, 2011.
[36] Facebook comment volume data set.
https://archive ics uci edu mm/ wooeeeeeetttlllleetaattteeeeee Accessed: January, 2019.
[37] bio.
/ Physicochemical properties of protein tertiary structure data set. https: of S.-Prootiin-TerrtaayyyySStuuuu
Accessed: January, 2019.
[38] house. House sales in king county, USA. https://wwe kaggle e. com /har/ http///arrll////oo////oo foxem/ cousesalepprediitiion/netaaaaa Accessed: July, 2021.
12

---

[39] meps 19. Medical expenditure panel survey, panel 19. https://meps.
ahrq. golog/gg///eeeeb////ee attaa_sstttt stats/download data files detail.jsp? cboPufluumeer=#C-1188 81. Accessed: January, 2019.
[40] blog_ data. Blogfeedback data set. https:: http://aaaciveeeeeeeeeeeeeeeeeee ics uci wtw../......eeemmm//tttsttttttt. BlogFeedback. Accessed: January, 2019.
[41] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[43] Bat-Sheva Einbinder, Shai Feldman, Stephen Bates, Anastasios N Angelopoulos, Asaf Gendler, and Yaniv Romano. Conformal prediction is robust to dispersive label noise. In Conformal and Probabilistic Prediction with Applications, pages 624 626. PMLR, 2023.
[44] Donald B Rubin. Bayesian inference for causal effects: The role of randomization. The Annals of statistics, pages 34-58, 1978.
[45] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[46] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015.
[47] Douglas Almond, Kenneth Y. Chay, and David S. Lee. The costs of low birth weight. The Quarterly Journal of Economics, 120(3):0031-1083 2005.
[48] Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
[49] David S Yeager, Paul Hanselman, Gregory M Walton, Jared Murray, Robert Crosnoe, Chandra Muller, Elizabeth Tipton, Barbara Schneider, Chris S Hulleman, Cintia P Hinojosa, et al. A national experiment reveals where a growth mindset improves achievement. Nature, 573 369, 2019.
[50] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[51] Carlos Carvalho, Avi Feller, Jared Murray, Spencer Woody, and David Yeager. Assessing treatment effect variation in observational studies: studiee Results from a data challenge. Observational Studies, 5(2):21-35, 2019.
[52] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.
[53] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 785-794, New York, NY, USA, 2016. ACM.
[54] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-28 22330, 2011.
[55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, hegf-peeformance deep learning library. In Advances Neural Information Processing Systems 32, pages 8024-8035 Curran Associates, Inc., 2019.
13

---

[56] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
14

---

A Theoretical results
A.1 Proof of Proposition
Proof. For the sake of this proof, we re-define the scores quantile T function of test weight wt
conservative from Section 3.1 as
(w)) == Quantile
1-a++ a + 8; Σ
Py(w)os, + Pn- P (w))0w))
(10)
5EZ9
where py(w), Pn+ (w) are defined as:
W
Py(w) EvETy WK + w
Pn+i
EKETY Wk +
For ease of notation, we denote the test weight by Wn+1 = w( and its conservative coun- conservativve which is defined in (6). Note that by the definition of Q, get terpart by 1n+1 #++1
Quee conservative ((in+1) Since the observed calibration points ((.,,M.). Z:) Jizz, and the test point (Xn+1 (0), Zn+1) are exchangeable (we assume that X2()) = (1)). then CP [6] guarantees that
the prediction set Cº² (Xtee) satisfies the coverage requirement:
PM E C² (xie)) 2 - 3
and therefore:
P(w Zn++1) (w(z) 2 C (x 21-
Following this, we get: (P 71+1) 1++1)==0 < in+1) 2 1 3. Assuming (X))), Y()) IL M Z, there a covariate shift between the calibration and test samples, where the covariates are the privileged information Z. Specifically, the calibration covariates (Zihezy are drawn from PZIM=0 while the test covariates Zrest are drawn from Pz. Importantly, both the calibration and response response variables have the same distribution conditional on Z: Priz- Thus, [8, Theorem 1] states that:
IP (Y {y S(X y) S ((wm+ ((w++)))))))))))))))))))) 2 a B. We note that Lemma I states that Q(w) is non-decreasing, i.e, 2 w2 + Q(wi) 2 Q(w2). Finally, we combine everything together to get:
(Yrest E @Two-Staged
(xee))
= P (Yles E {y : S(x'", ,y) S Q ((wn+t)))
= P (S(xtat yls)) < ((ū++1))
> IP (S(X) ylau) @(w++1))+)))). Wn+1 2 Wn+1)
2 IP (S(X)e)) ylu) Q(w++1)) Wn+1 2 Wn+1)
1 (S(Xte ytex)= (wn=1) or Wnt1 < Wn+1)
2 1 (S(x tee ylu)) > Q (wn=1)) - PP(n++1 Wn+1)
2 (a B) (8)
1 - a.
A.2 Proof of Theorem
Proof. For the sake of this proof, we re-define the scores quantile Q from (8) as function of test weight w, similarly to (10):
(ww) == Quantile
1- a +3; Σ
Py(w)ōs, + P (w)
(11)
JEIN
where pj (w), Pn+1 (w) are defined as:
Wi
w
()(w) Every wh + w'
Pati(w) = Σ EkEzy We + w
15

---

Note that by the definition of Q, we get Qi Q(w;). Furthermore:
QPCP
Quantile
1 - 3;
8Q,
[zl + 600
+
IEIg
Quantile
1 9
(Q(w.))
[Tz + 800
I2
+
VEI2
Since (ww) is non-decreasing function wf as proved in Lemma 1, we get:
QPCP =
=o(ouantie (1
800 ))
3;
Sw: +
Iz
IZ
iEZ
Thus, QFEP can be considered as it was computed from the following weight:
1n+1 Quantile
1- B:
Σ
8m.
+
800
Iz
+
I2
TET
in the sense that QT = @(#x++)). Therefore:
C (FCe((X)tt) {u S(X QPc3) {v: S(X .9) S ((@n++)))
The true weight of the n + 1 sample is: Wn+1 F(M-0)2-2 iz Now, since {Z, bEETgu(t14) are exchangeable, then [wi (ei are exchangeable and thus [11, Lemma 2] states that:
P(M=0)
P(wn+1 S xn+1) 1-8.
Assuming (0), (0)) TT M Z, there is covariate shift between the calibration and test samples, where the covariates are the privileged information Z. Specifically, the uncorrupted calibration covariates {Z, kezy from Pz. Importantly, both the calibration and response response variables have the same distribution conditional on Z: Priz- Thus, [8, Theorem 1] states that:
drawn from P while the test covariates Z are drawn
P (yrest E (y S(X v) < ((wn+1))) 2 - a
Note that Lemma 1 states that Q(w) is non-decreasing, i..., w1 2 W2 Q(wi) 2 Q(w2). Finally, we combine all together and get:
IP (Y E CPCP (X
= IP (y E {yy S(xtos ,y) S Q(#n++1))
PP (S(X yreu) < Q(wn+1))
2 P (S(Xiu y ((@@@@@))) unt1 2 Wn+1)
2 IP (S(X yoa) < Q(wm+)))) 1), Wn+1 2 Wn+1)
1 P (S(Xxe yles) > (wn+1) ) or Wn+1 Wp+1)
2 1 P (S(x y > ((wn+1)) P(un+1 < wn+1)
2 (a - B) (B)
1 - a.
A.3 Proof of Theorem 2
Proof. For the sake of this proof, we re-define the prediction set as function of the test weight [0, 1]:
y EY:: En(w))(S < S(x, < -al.)ssssst....)) -77
CCO (4)
IEI
16

---

where y = =a- - 3. and (s(w), Pu+1 (w) are defined as:
Wi
w
(w(w) ENETH Wk + (w)
= E ZNETu we + w
The prediction set generated by LOO-PCP is: Choo-PEP where:
(1- 8
Wn+11 Quantile
Ows
1
Therefore, our goal is to show that this prediction set covers the response variable at the desired coverage rate:
P(y E 1,111
C (x(eet) 2 1 - 2a.
(12)
The proof consists of two steps. In the first step, we show that C PCP (Xist)) achieves valid marginal coverage, namely:
P(ytest E C:00-P0P (X 2 11-27
"m+1
In the second step, we show that C PCP (Xtea) is a super set of Wa41 @n+1
PCP (Xtts)) with a high
probability. From these two steps, we will conclude (12).
We define the matrix of residuals, similarly to [9, 21], denoted by p(n+1)x(++1), with entries:
+oc
= J, i i
R
((X,(0),
where f is the model f fitted all samples except with the points and removed, namely, {1, n + j. For simplicity, we denote (w) == wy for E (1,..,), n and wx+1(w) = w. We follow the definition [21] of "strange" points ((w)) C T U U + 1}:
((w) i I unn+11 U
(.(w) 0,
Σ jET=U(n+1)
p)(w)] {Ri > Rg R3 R1
begin by showing that Yiat E Cum ( (Xtes) + n + & ((wn+1). Suppose that Yies PCP (Xtest). Then, by definition of C sC2 (Xtss))): "wat1
#n+1
Σ (, < S(Xte yieu, yieu; JEI
1-イ>
Σ
P5(wn+1)1[D) < Butia)
JEI
Σ jEZ
P5(un+1) > Rs
Σ
P)(Wn+1)1(Bmtt) > Rgntt1
=
JEI+U(n+1}
Therefore: + 1 $ ((wn41), by the definition of G. We deduce that: P(ytest E C #1 G(wn+1)) In [21] it is shown that:
Ctoc-POP (X1ea)
P(n+ +1++1e ((wn+1)) 2 22=111112 1-27
Thus:
P(ytes E wat1
(Xtest) 2 P(n + 1 & ((wn-1)) 2 1 27.
We now turn to the second step of the proof. Similarly to Lemma 1, we show that C C1000-FCP (z) a monotonic function of w, i.c., if W1 2 W2 then C
CL03-PCP (x) CLa0-PCE (z). Suppose that
y E CLOU-PCP (xr). Then:
W2
Σ < <1 ME
17

---

Since wi 2 w2, we get Pi(wi) & (((22) for all i E fl....,)) n). Therefore:
Pi < U {}} & Σ (i(w2) < S(z,v: < 1-
VE
RE
Meaning that y C PCP (x) well. Lastly, recall that according [11, Lemma 2] the weight Wn+1 satisfies:
P(wn+1 S (n+1) 2 1- B.
Finally, we combine everything together get:
YYest E
C0100-PPP (Xtest) 0n+1
P yist E C100-PCP (Xtea), un+1 2 Wn+t
Wh+1
2 P yust E We+1
PCP (X Wn+1 2 Wh+1
(yleu & C (Xles) or Wn+1 < Wn+1)
yest & PCP (O (x IP (wn+1 < wa+1)
2 P
2 (2x)
(5)
1
2 2
a - 2 - (3)
A.4 Lemma about weighted quantiles
Lemma 1. Suppose that wi, St E I for S sic S n. Further, suppose that 7 E [0, 1), and
I ul....... Denote the normalized weights Pi(w), Pn+1(w)))ww)))) as:
Di(w)=
KEE We + Pn+1(w)) (w) ΣxEI Wk + w
W/;
w
Then, the weighted quantile:
Q(w) = Quantile
> JEI
pi(w)os, + +
a non-decreasing function of w. i.e., W1 2 W2 = Q(wi) 2 Q(w2).
Proof. Without loss of generality, suppose that {S: 1 sorted in an increasing order, i.e., Si+1 2 S. For the sake of this proof, we define Sn+1 xc. For ease notation, denote I' I U (n+ + 13 The formal definition of Q(w) is:
and I, I' (1....;) i}.
Q(w)
S min (SET) p(())27)'
Suppose that wi 2 w2. Then, by the definition of ps. we get that for all i E I:
pi(wi) S pw()).
Therefore for all i I:
Σ Pa (w) < Σ P)(ww).
KET,
KET,
For i = n + 1 the above equation is trivially satisfied as EKEInei (w(w1) = EkEInE, Pa(u2) = 1.
Thus,
min
iE fI.... n +1 Σ px(w)) (px(u))27)
IN
kET:
min i (1, (1,.... 1) Σ pk(w2) 27
keI.
18

---

Therefore,
min (ii..ie P pa(w))27
2 S (in Pa (wa)27)
and finally,
Q ((w)).
A.5 Relaxing the conditional independence assumption
In this section, present relaxations for conditional independence assumption, (X))), Y ( T I M Z, in Theorem The first relaxation allows X (0) to depend on M given the PI at the expense of using the following weights P(M X x, x). The second result relaxes the conditional independence to hold approximately. up to some error, denoted by E. We begin with formalizing the first result and then turn the second one.
Theorem 3 (Robustness of PCP to dependence of the features and corruption indicator). Suppose that {(X,(0), (1). Y2) ((),, Z, M.) are exchangeable. (Y) TT M) (X, Z), the features are uncorrupted, (() (1), and (x) is absolutely continuous with respect to Px ( ZIM=0 Denote by C (X the prediction set constructed according to Algorithm with the weights:
PPM = 0)
We P(M=0 X(0) (,) Z Z.)
Then, this prediction achieves the desired coverage rate:
P(y E CPCP (x 2 1
Proof. This result follows from the proof of Theorem I, except for the following changes. Here, get that then there a covariate shift between the calibration and test samples, where the covariates X (0), Z. Therefore, the following weight wa+1 of this setup:
P(M
W Wn+1 = P(M = 0 X(0) = x (0), = Zn+1)
satisfies:
TP (Ylest E {v : -(X1es, y) S ((+++1)))))))))))))) 1)} 2 - a 8
where Q is defined as in Appendix A.2. The rest of the proof is as in Appendix A.2.
We now turn to present an initial extension of Theorem to setting where the conditional in- dependence assumption is not fully satisfied. For the simplicity of this extension, assume (0) IL Z 2.
We note that the independence assumption Y()) IL M Z = 2 is equivalent to assuming that the density of (()) M = m, = same for m E {0, 1}. formally:
fr(o) M=0, (v:
In this extension, we relax this assumption and instead require that Vr E X, there exists En E such that the difference between the two densities is bounded by Ep:
Vy E V 3 M0
0, - 1, S Ez.
Theorem 4 (Robustness of PCP to conditional independence violation). Suppose that ((,, (1()) Y,(O). Y((1,, Z, M,) 1++1 corruption indicator given the PI, X(0 IL M Z, the probability Pz is absolutely continuous with respect to P2|==0. and Va E X there exists & E RR such that:
1=1 are exchangeable, the features are independent of the
Vy V z Z Z) M=
0, 2) fY M=IN-=R22= 1,,, E.
Then, the coverage rate of the prediction set CCP C (Xtest) constructed according Algorithm is lower bounded by:
Pey
PCP (X 2 1- Ex.2| (X)
19

---

Proof. We define by V variable that is drawn from:
(7, V) Pz Pr
By the definition of V. and according to Theorem PCP covers with 1 a probability:
P(V E C (X)) 2 1 a.
Notice that V X = Z = 2 equals in distribution to (0) M = 0, = I, 2 = 2 by definition.
We denote the coverage rate of PCP over (0) M = 0, = x, Z = bb by 80,
f0,x.= (0) E C (X) X x, 7 = 2, M 0) P(V C (X) X Ir = =).
Similarly, we denote the coverage rate of PCP over Y (0) M 1,X 2, 2 2 by 31,0,2
21 = P(Y(O) E (PCP(X) M 1, X = 2, = 2) 2)=
1100) 1, x, x22)) 2)
This probability can be lower bounded by:
SEcomiO (ECKST())
X=x z)dy /yEcoa(z)) (fy(o) M=N., Ex)dY
30, - |CPCF (sr)6ex
We now compute the conditional coverage rate of PCP:
(0Y CPCP (X) Z = == x) (P) CPCP M =
=xx,===>>> =
+ P(Y(0) E (X) M 2,,
= x, =
2 So. P(M = 0
z,Z=2)
+ (B0. - 1C (x))e,)))) = = = ==
: P(M = 0 | = =z,Z=
+ Bo, P(M = 1| XX = =I, 7 = 2) - - (x) = = (=)
B0,0. (P(M = 0, x, 7 = =2) P(M = x, ==
|OPCP (x))es, = | x, =)
30,0. - -C P(M |1| xxx, xx =
PPVV (X) x, 7 - CCPCP (x))e, = x| x =xx,=>))
By marginalizing this result we get:
((Y E CPCP (C)P(()))
P(Y(O) E (X) 2 = =, = xx (x, 2)dadz
SEX
E (X) x= x, 2 = 2)) ( 00,23. 2 2)dad=
ZEX
(S) = || X = x, = =x)] (2, 2)dadz
1
1|x,Z)]
This result provides lower bound for the coverage rate of PCP in the setting where the conditional independence assumption is not exactly satisfied. Intuitively, as ε, decreases, i.e., as the two distributions (()) M m, Z = for E {0, 1} are closer to each other, the lower bound is tighter, and closer to the target level. Similarly, as the two distributions diverge, the lower bound becomes looser.
20

---

B Algorithms
B.1 Two-Staged Conformal
In this section, we outline the two-staged conformal prediction algorithm (Two-Ster Staged).
Algorithm 2: Two-Staged Conformal Prediction (Two-Staged)
Input:
Data (X,tts Y.obs Zi, M,) E Ex x x x sis Weights (=:);-1-
Miscoverage level a E 1).
Level (0.
An algorithm f for Y.
An algorithm f2 for Z.
A score function S.
A test point X X I.
Process:
Randomly split {1, n into two disjoint sets I1, Iz. Fit the base algorithm f on {(X,thx Y, Jiez, and the algorithm jz on {(Xçhh Zi)
Compute the scores S?² = S(X,t Z, fz) St = S(X Y: ff for the calibration samples, e I2.
Compute a threshold Q² as the (1- 1/-1/ - th empirical quantile of the scores {Si)eez
Construct prediction set for Z: (z)= S(x, f2) S Q²).
Compute the conservative test weight consevvative (I) max (w) w(2).
Compute the test threshold Comsevvative according (5) with the clean calibration points
{(X,, Y,, wi( [e)), and the conservative test weight, eoooerrvatie) with nominal coverage
level of 1 - a + B.
Output:
Prediction set Iwo -Iwo-Staaeee Staged(x) = {y S(x, y5 M <
QuCe conservative
B.2
Efficient Privileged Conformal Prediction
Since the complexity of Algorithm is squared the number of calibration samples, here, we provide more efficient algorithm with complexity that is linear in the number of calibration samples. The efficient version is detailed in Algorithm The proof Theorem Section A.2 shows that these two algorithms are identical, in the sense that they produce exactly the same outputs.
B.3 Privileged Conformal Prediction for scarce data
In this section, we describe the leave-one-out privileged conformal prediction algorithm (LOO-PCP) which is designed handle scarce data more efficiently. This procedure is summarized in Algorithm 4.
C Datasets details
C.1 General real dataset details
Table displays the size of each data set, the feature dimension, and the feature that is used as privileged information in the tabular data experiments.
C.2 CIFAR-1ON dataset details
CIFAR-1ON [41] is variation of the CIFAR-10 [42] in which the labels are given by human annotators. In this task, X; an image from the CIFAR-10 dataset. The response Y, E (1,... 10} the noisy image label chosen by the first human annotator. The ratio of noisy samples is 17.23%. The
21

---

Algorithm 3: Elficient Privileged Conformal Prediction (PCP)
Input:
Data (Xrots y, Zi, M,) x 10,1), sisn. n
Weights {wi)
Miscoverage level a E (0,
Level a).
An algorithm f.
A score function S.
A test point Xtest = I.
Process:
Randomly split (1,..., n into two disjoint sets I,, Iz.
Fit the base algorithm f on {Xpths
Compute the scores S; = S ((pbb y,ous f) for the calibration samples, i Z3..
Compute an estimated test weight based on the calibration samples:
Wn+11 == Quantile (1 3; Eiez,
1 Sw. + m 8x0
m2+1
Compute QPEP = @(mm+1), where Q is defined (11).
Output:
Prediction set CPCP (z) {y : S(x 9: QPCP).
Algorithm 4: Privileged Conformal Prediction for scarce data (LOO-PCP)
Input:
Data (Xoth, yooks Zi, M,) E X x y x 2 x (0. 1).1 < i < weights (w:) miscoverage
level a (0, 1). level a), an algorithm f. score function S, and test point X
Process:
for i 4-1 to n do
Define the current training set as I {1...., 1, 1,., n}.
Fit the base algorithm on (X,, X))jEz, to obtain f
Compute the score S; ((x,t y,obs f-).
Guess the weight of the + 1 sample:
Wn+1 = Quantile (I
3: EiE (l....) n+1 Sw, + n+I dx) Σjeme my +0 for f n..
Compute: Pt
Define the threshold 7 == a 3. Output:
Prediction 1000- PCP (x) y yyV: Eiero pil(S ((,,,,<<)
privileged information Z: has two features: the working time of the first annotator, and the number of different labels chosen by the three human annotators.
C,3 CIFAR-10C dataset details
CIFAR-1OC [50] is a variation of the CIFAR-IO [42] dataset in which the images are contaminated by artificial corruptions. Here, we randomly apply one of the following corruptions for 15% of the images: snow, defocus blur, pixelate, or fog. The corruption severity level is either 4 or 5, chosen with equal probability. A label of corrupted image is flipped according to the severity and corruption type. The severities 4.5 are assigned the values 0.92, and 0.95, respectively, and the snow, defocus blur, pixelate, and fog corruptions are assigned with 0.95, 0.93, 0.9, 0.93, 0.94, respectively. The flip probability is the nultiplication of the value assigned with the corresponding severity and corruption type. In total, 13.09% of the labels are flipped. Importantly, in training time, both the image and the label are corrppted, while at inference time, only the image corrupted and the performance computed with respect to the clean label. In the dispersive noise setting, the label is uniformly flipped
22

---

Table 1: Information about the real data sets.
Dataset Name
# Samples
X/Z/Y Dimensions
Z description
facebookl [36]
40948
52/1/1
Number of posts comments
facebook2 [36]
81311
52/1/1
Number posts comments
Blo [37]
45730
8/1/1
Fractional area of exposed non polar residue
House [38]
21613
17/1/1
Square foutage of the apartments interior living space
Meps19 [39]
15785
138 /1/1
Overall rating of feelings
Blog [40]
52397
279/1/1
The time between blog post publication and base-time
THDP [35]
747
24/1/1
Birth weight
Twins [47, 48]
12042
50/1/1
The birth weight of the lighter twin
NSLM [49]
10391
10/1/1
Synthetic normally distributed random variable
CIFAR-ION [41]
50000
32x32x3/1/2
Annotation time Label variability Corruption severity type
CIFAR-IOC [50]
40000
32x32x3/1/2
into a wrong one. In the contractive noise setting, if the image is affected by either snow or defocus blur corruption, then the noisy label is deterministically set to 2, or to 1 if the original label was already 2. If the image is corrupted by either pixelate, or fog, then the noisy label is deterministically set to 7, or to 6 if the original label was already 7. The privileged information contains two features: the severity level and the corruption type. Since CIFAR 10C contains only 10000 samples, we add 30000 clean samples from CIFAR- to our dataset. This way, 60% of the 10000 CIFAR- 10C images are corrupted, while the other 30000 CIFAR- 10 samples are clean, total, 15% of the images are corrupted.
C.4 IHDP dataset details
The Infant Health and Development Program (IHDP) dataset [35], is semi-synthetic data containing in which the response variable is the future cognitive test scores. The feature vector includes information about the child such as child-bin weight, head circumference, weeks born preterm, birth order, first born indicator, neonatal health index, twin status, and more. The treatment M the specialist home visits indicator. The privileged information is defined as the covariate in X; with the highest correlation to Y, We then remove this feature from X,, so it cannot be used at inference time. Since this dataset is semi-synthetic, every sample point includes both potential outcomes (O), (1(1) for every sample i. We therefore need to choose which samples are treated with M = 0 and which are treated with M = 1. In section D.1 we explain how M is chosen to intentionally induce distribution shift between the observed and test distributions.
C.5 Twins dataset details
The Twins dataset contains information about twin births in the USA between 1989 and 1991. The raw data was provided by [47], and [48] introduced it as new benchmark. In this dataset, the treatment = 1 indicates for being born the heavier twin, the covariates X include features about the twins and their parents, and the outcome corresponds to the mortality of each of the twins in their first year of life. Since the records of the two twins are available, their mortality rates are considered as two potential outcomes, where the treatment is the indicator of being born heavier. We follow the protocol of [48] and focus only on twins with birth weight less than 2kg. The privileged information variable is set to the birth weight of the lighter twin. Since both potential outcomes are observed in the data, we must selectively hide one of them to simulate an observational study. We choose the treatment probability explained in Section D.
C.6 NSLM dataset details
The National Study of Learning Mindsets (NSLM) dataset [49] is semi-synthetic data that was analyzed in the 2018 Atlantic Causal Inference Conference workshop on heterogeneous treatment effects [51]. See [51, Section 2] for more information about this dataset. In our experiments,
23

---

follow the protocol introduced in [34, to generate two synthetic potential outcomes and synthetic Pl variable. Specifically, begin by scaling the data to have 0 mean and standard deviation of We split the dataset into a training set and validation set, containing 80% and 20% samples from the entire data, respectively. Using these training and validation sets, we fit a neural network with one hidden layer of size 32 to predict Y from X. The training parameters are detailed in Section D.1. This network function is denoted by (-) Similarly, we an XGBoost classifier to predict the original treatment variable M given in the data from the feature vector X. We set the max __ephh then calibrate the estimated propensity score
and n estimators parameters to 2, 10, respectively, to have the same mean the marginal treatment probability. This calibrated estimated propensity score is denoted by é(X;). We generate a new treatment variable M,, synthetic PI variable Z,, and semi-synthette target variable (0) follows.
Z: ~ (0, 0.2?)
E, = 1 1 2 Quantile Z) or Z; S Quantile(O.L. Z)}
M, (er(min(0.8.. +1+ E.)((x))
= 0.228 + 0.0,s < 0.07) - 0.051 < -0.69) - 0.081(X,, E (1, (0) Ao + 7. + (1 + E.)
C.7 Synthetic dataset details In this section, we present the synthetic dataset used in the ablation study of the parameter , in Appendix E.5.
The feature vectors are uniformly sampled as follows:
X; Un)(1,5)10
where Uni(a, b) is unifrom distribution in the range (a, b). The PI is sampled as:
E N),
F Uni(-1,1)
E 1),
P. Pois(cos( E3 + 0.1)) E?,
Z, P. + 2
Above, Pois(A) poisson distribution with parameter X, and N(u, o²) normal distribution with mean and variance 0². Finally, the label is defined as:
8 Uni(0, 1)a
3/11.111
U, Iz,<-3 + -3322111++18 +8 L2
E, ~ 1)),
Y. 0,3X,3 + 0.82, + 0.2 U,E,
D Experimental setup
D.1 General setup
In all experiments, except for scarce data experiments, we split the data into training set (50%). calibration (20%), validation (10%) used for early stopping, and a test set (20%) evaluate performance. See Section D.2 for the specific details in the scrace data experiments. Then, we normalize the feature vectors and response variables to have zero mean and unit variance. In experiments involving missing variables, we impute them with a linear model fitted on variables that are always observed from X, Y, Z. The linear model trained on samples from the training and validation sets. For datasets that are not originally corrupted, the corruption probability is determined as follows. First, for IHDP and Twins datasets, we fit linear model on the entire data to predict Y given X, Z, and use its predictions as initial value. For other datasets, we take Z as the initial value.
24

---

We take the initial values, set the maximal value as the 85% quantile, and divide by the 90% quantile of the initial values. Then, we zero the lowest 75% values. Then, we raise all values to the exponent that achieves an average of 0.20. The result is the corruption probability. Therefore, by definition, the average corruption probability is 20%. all experiments, we fit base learning model and wrap its output with a calibration scheme. In regression tasks, the model is trained to learn the 5% and 95% conditional quantiles of Y X. In Table 2 we summarize the model we used for each dataset for both tasks. For neural network models, we used an Adam optimizer [52] with le-4 learning rate, and batch size of 128. The network is composed hidden layers sizes: 32, 64, 64, 32, 0. dropout, and leaky relu as activation function. For xgboost and random forest models, 100 estimators. We train the networks for 1000 epochs, but stop the training earlier if the validation loss does not improve for 200 epochs, and in this casc, the model with the lowest validation loss is chosen. In our experiments, we use the xgboost package [53] and the scikit-learn package [54] from random forest. The neural networks were implemented with the pytorch package [55]. Regarding the hyper-paramett of the calibration schemes, in all experiments, we set the parameter /8 of PCP to 0.05.
used
B = 0.005 and the parameter 3 of Two-
Table 2: The learning models used for each dataset.
Dataset Name
Base learning model
Corruption probability estimator
Facebook1 [36]
Neural network
Neural network
Facebook2 [36]
Neural network
Neural network
Bio [37]
Neural network
Neural network
House [38]
Neural network
Ncural network
19ps19 [39]
Neural network
Neural network
Blog [40]
Neural network
Neural network
IHDP [35]
XGBoost
XGBoost
Twins (47, 48]
XGBoost
XGBoost
NSLM [49]
XGBoost
XGBoost
CIFAR-1ON [41]
Resnet-18
Resnet- 88 when using or Random forest when using only
CIFAR-1OC (50]
Resnet-18
Resnet-18 when using x or Random forest when using only s
D.2 Experimental setup for scarce data experiment
In this section, we detail the experimental setup employed in the experiment in Section 4.1. In this experiment, we split the data into training set (30%), a validation set (10%), and test set (60%). Furthermore, the nominal coverage level was set to - 2a = 90%. For each training sample we fit an XGBoost quantile regression model using the entire training set, except for the i-th sample, to obtain a leave-one-out model. Then, we calibrated the model outputs with each calibration scheme, using the leave-one-out models.
D.3 Experimental setup for CIFAR-1ON and CIFAR-10C experiments
We use a ResNet- 18 network [56) as a base model. we used both X and Z as an input to the model, we forward X through the CNN, and then through a linear layer with output dimension of 16. Then, we concatenate this result with 7 and forward it through network with hidden layers with sizes 32, 64, 64, 32. When fitting the CNN, train the model with batches of size 32 for 50 epochs, and choose the model with the lowest validation loss. Additionally, we apply random augment transform improve performance. For the Two- Staged and Naive WCP calibration schemes, we use only X to estimate the corruption probability, and for the infeasible WCP and PCP we use only Z estimate it.
D.4 Machine's spec
The resources used for the experiments are:
CPU: Intel(R) Xeon(R) E5-2650 v4.
25

---

GPU: Nvidia titanx, 1080ti, 2080ti.
OS: Ubuntu 18.04.
D.5
Computational resources
The computation efficiency of the proposed algorithm is dominated by the efficiency of the base learning model. The reason that our calibration scheme only requires one pass over all calibration samples, explained in Section B.2.
E Additional experiments
In this section, we provide additional experiments and supply additional results. In all experiments, whenever possible, we display the performance of uncalibrated model, Naive CP which uses clean and noisy samples, Naive CP which uses only the clean samples, Wei ghted CP with the following weights: (i) estimated using only from X, (ii) estimated using Z, and (iii) oracle weights as a function of Z. We further employ the Two-St Staged algorithm and PCP with these three options for the weight function. is important to note that it is not applicable in practice to apply Weight ed CP or Two-Sten Staddt CP with weights estimated from Z, since they require the test privileged information Zlest We conduct these experiments for demonstration purposes. Nevertheless, PCP applicable with any of these weights, as it does not use Zess.
E.1 Causal inference tasks experiments
We follow the protocol in Section 4.1, when our goal is to estimate the uncertainty of unknown response under no treatment + (0). As explained in Section 4.1, valid prediction intervals for (0), 1 (1) can be combined to construct reliable interval for the individual treatment effect (TTE), which is valuable many applications [30-33].
We begin with the semi-synthetic IHDP [35] dataset, in which the objective is to analyze the effect of specialist home visits on future cognitive test scores. See Section C.4 for more information about this dataset. Figure 4 displays the coverage rate and interval lengths of the prediction intervals constructed by the calibration schemes. This figure shows that the naive techniques: Uncal ibbrttdd, naive jackknife+ (Naive CP), and naive JAW, which uses weights estimated only from X, do not achieve the desired coverage level. This is not surprise, as the naive approaches do not hold statistical guarantees. In contrast, JAW which uses Zrest and the proposed PCP construct uncertainty intervals that achieve the desired coverage level. This also anticipated since these methods are supported by theoretical guarantees. Nevertheless, the versions of JAW that achieve valid coverage rate are infeasible in practice, as they require Ziest, which unknown in our setting. In conclusion, Figure suggests that PCP is the only applicable calibration scheme that achieves valid coverage level in this experiment.
Next, we turn to the Twins dataset [47] which contains records about newborn twin babies, and the response variable is the mortality indicator. In Section C.5 we provide additional information about this dataset. Figure 5 presents the performance of each calibration scheme on the Twins dataset. This figure indicates that Naive CP tends undercover the response variable while the two-staged baseline tends to overcover it. In contrast, observe that WCP and PCP achieve the desired 90% coverage rate when employed with oracle corruption probabilities, or with probabilities estimated from Z. However, we remark that these versions of WCP cannot be applied in practice, since they require Zrest which is unavailable in our setup. The only practical version of WCP is with corruption probabilities estimated from X, which does not achieve the nominal coverage level. Notice, however, that the proposed PCP can be applied with all versions of corruption probabilities, as it does not require access to Zies To conclude, PCP and Two- Staged are the only calibration schemes that are both applicable in practice and guaranteed to generate uncertainty sets with a valid coverage rate. In addition, Figure reveals that PCP achieves comparable set size to the infeasible WCP, indicating that PCP does not lose much statistical efficiency by not using the test PI tistt
Lastly, we consider the semi-synthetic National Study of Learning Mindsets (NSLM) dataset [49], which examines behavioral interventions. See [51, Section 2] for information on the dataset, and Appendix C.6 for our adaptation for this dataset. The performance of cach calibration scheme is
26

---

provided in Figure 6. This figure shows the same trend: the naive methods undercover the response variable while the proposed PCP constructs valid uncertainty sets.
Coverage rate
Corruption probability
90
estimated from x estimated from Z oracle
Uncalibrated
Naive CP
Weighted CP
Privileged CP
Method
Interval length
7.5
5.0
2.5
Uncalibrated
Naive CP
Weighted CP
Privileged CP
Method
Figure 4: IHDP dataset experiment. The coverage and average interval length achieved by an uncalibrated quantile regression (Uncal libaatebr,ddd a naive jackknife+ (Naive CP), JAW (Wei ght CP) which estimates the corruption probability from either X (orange), Z (green), or uses the oracle probabilities (red), and the proposed method (Privi vi leged CP) with the three options for the corruption probabilities. All methods are applied attain coverage rate at level 2a 90%. The metrics are evaluated over 50 random data splits.
Coverage rate
Corruption probability
estimated from estimated from Z oracle
Naive
Weighted
Two Staged CP Method
Privileged CP
Set Size
1.2
Naive CP
Weighted CP
Two Staged
Privileged CP
Method
Figure 5: Twins dataset experiment. The coverage rate and average set size achieved by naive conformal prediction (Naive CP), Weighted CP which estimates the corruption probability from either X (orange), Z (green), or uses the oracle probabilities (red), the baseline Two Staged CP, and the proposed method prrivileged CP) with the three options for the corruption probabilities. All methods are applied to attain a coverage rate at level 1 - 90%. The metrics are evaluated over 20 random data splits.
E.2 Noisy response variable: CIFAR-ION dataset
In this section, we provide additional results from the CIFAR- 1ON experiment conducted in Sec- tion 4.3. In Figure 7 display the performance naive CP, and WCP, Two-St Staged, PCP, applied either with corruption probabilities estimated from X or from Z. This figure shows that WCP and PCP do not achieve the desired coverage rate when the corruption probabilities are estimated from X. In contrast, WCP and PCP attain valid coverage rate when the corruption probabilities are estimated from Z. This not a surprise, as guaranteed by [8, Theorem 1] and by Theorem 1. Lastly, this figure shows that Two- Staged constructs uncertainty sets that are too conservative. This is also anticipated since the prediction sets of Two-St ged encapsulate the uncertainty in both and Y. Importantly, we note that WCP cannot be used with corruption probabilities estimated from Z
27

---

Coverage rate
Corruption probability
90
estimated from X estimated from Z oracle
Uncalibrated
Naive CP
Weighted CP Method
Two Staged CP
Privileged CP
Interval length
Uncalibrated
Naive CP
Weighted CP Two Staged CP Privileged CP Method
Figure 6: NSLM dataset experiment. The coverage rate and average interval length achieved by uncalibrated quantile regression (Unca (Uncalibrate (ilibbrreed)) naive conformal prediction (Naive CP), We Weighted CP which estimates the corruption probability from either X (orange), Z (green), uses the oracle probabilities (red), the baseline Two Staged CP and the proposed method (Privileged CP) with the three options for the corruption probabilities. All methods are applied attain coverage rate level 1 90%. The metrics evaluated over 20 random data splits.
(We i siighted CP in color green in Figure 7) since it requires access to Zest which is unknown in setup.
Coverage rate
Corruption probability
95
estimated from X estimated from Z
90
Naive CP (ceisy+clean)
Naive CP (only clean)
Weighted CP Method
Two Staged CP Privileged CP
Set size
Naive CP (leasy+
Naive CP (only clean)
Weighted CP
Two Staged CP Privileged CP
Method
Figure 7: Noisy response experiment: CIFAR-10N dataset. The coverage rate and average set size length achieved by. naive conformal prediction (Naive CF), using either all calibration samples (noisy clean) or only the uncorrupted ones (only clean), Wei Weighted CP which estimates the corruption probability from either X (orange), (green), the baseline Two Staged CP and the proposed method (P Privill ville CP) with the two options for the corruption probabilities. All methods are applied attain coverage rate level 20 random data splits.
90%. The metrics are evaluated over
E.3 Noisy features and responses: CIFAR-10C dataset
We demonstrate our proposed method on the CIFAR-10C [50] dataset, which contains pairs of noisy image with a noisy label. In short, some of the images are corrupted, e.g., with defocus blur, and the responses corrupted images are artificially corrupted. Nevertheless, we note that the test images are corrupted well, in the sense that X() X(1) for all samples. In this experiment, we examine two options: dispersive noise, which randomly flips the label into different one with uniform probability, and an adversarial noise, which deterministically flips the label into different
28

---

one. In Section C.3 we provide the full details about this dataset. Lastly, in the following experiments, we set the desired coverage rate 80% since the model outputs extremely uncertain predictions for 10% of the samples.
Figure 8 displays the coverage rates and set sizes achieved by each calibration scheme, the CIFAR-I00 10C dataset corrupted with dispersive noise. This figure indicates that by considering the noisy labels, Naive CP achieves conservative coverage rate, which is consistent with the work of [25, 43]. Nevertheless, when applied without the noisy labels, Naive CP tends to undercover the correct response, similarly the two-staged baseline Two-Staçed. Observe also that the feasible version of WCP, which uses weights estimated only from X achieves under coverage. In striking contrast, our proposed PCP covers the response the desired coverage rate. This trend also applies for the CIFAR-106 1OC dataset corrupted with adversarial noise, as presented in Figure 9. This figure illustrates that all feasible baselines suffer from undercoverage. except for the two-staged baseline which achieves an over-conservative coverage rate. In contrast, our proposal achieves the nominal coverage level.
Coverage rate
90
Corruption probability
estimated from X estimated from Z oracle
80
Naive CP (ceaa))
Naive CP (only clean)
Weighted CP Method
Two Staged CP
Privileged CP
Set size
1.5
1.0
Naive CP (cles)+)lean)
Naive CP (only clean)
Weighted CP
Two Staged CP Privileged CP
Method
Figure 8: Dispersive label-noise experiment: CIFAR-10C dataset. The coverage rate and average set size length achieved by, naive conformal prediction (Naive CP), using either all calibration samples (noisy + clean) or only the uncorrupted ones (only clean), Weighted CP which estimates the corruption probability from either X (orange), Z (green), or uses the oracle probabilities (red), the baseline Two St Staged CP and the proposed method sPrivileged CP) with the three options for the corruption probabilities. All methods are applied to attain a coverage rate at level 1 l = 90%. The metries are evaluated over 20 random data splits.
E,4 Tabular data experiments
In this section, we conduct a series of experiments with the datasets: facebookl, facebook2, bio, house, meps19, and blog. In each experiment, apply different corruption to either the covariates or the response variable. Additionally, for the Two-Staged, Infeas ble WCP and the proposed PCP, we use the oracle corruption probabilities when computing the weights we(2). However, the Naive WCP method uses corruption probabilities estimated only from x using a neural network classifier. Section C.1 provides information about these datasets and Section D.I describes the experimental setup.
E.4.1 Noisy response
We examine two artificial noise functions cornupting the response. The first noise is contractive, which reduces the variability by averaging the response with its mean: Y(1) = (Y + (0))- The second is dispersive, which adds to the response variable normally distributed random noise with mean and standard deviation 5 times the standard deviation of (
We begin with the dispersive noise experiment. Figure 10 shows the performance of each calibration scheme this setup. This figure indicates that the uncalibrated model and naive CP construct too conservative intervals. This result is consistent with the findings of [25, 43] which suggest that
29

---

Coverage rate
Corruption probability
80
estimated from X estimated from Z oracle
Naive CP (neasy+clean)
Naive CP (only clean)
Weighted CP
Two Staged CP
Privileged CP
Method
Set size
Naive CP cleas))
Naive CP (only clean)
Weighted CP
Two Staged CP Privileged CP
Method
Figure 9: Contractive label-noise experiment: CIFAR-10C dataset. The coverage rate and average size length achieved by, naive conformal prediction (Naive CP), using either all calibration samples (noisy + clean) or only the uncorrupted ones (only clean), Wei ghted CP which estimates the corruption probability from either X (orange), (green), or uses the oracle probabilities (red), the baseline Two Staged CP and the proposed method (Privil sivveedd CP) with the three options for the corruption probabilities. All methods are applied to attain a coverage rate at level 1 I-a = 90%. The metrics are evaluated over 20 random data splits.
naively applying CP method on data with dispersive label-noise leads to conservative uncertainty sets. Nevertheless, Naive WCP achieves coverage rate that is too low, possibly because the corruption probability estimates are not sufficiently accurate. Also, the two-staged baseline tends to output conservative intervals, anticipated. Lastly, Infe ible ble WCP and the proposed PCP consistently achieve the desired coverage rate for all datasets. This is not a surprise, as thcoretical guarantee supports this result.
We now turn to the contractive noise experiment, and report Figure 11 the coverage rate and interval lengths of the prediction intervals constructed by each calibration scheme. This figure shows that the uncalibrated model and naive CP generate intervals that undercover the correct outcome. This is anticipated, as the contractive noise confuses these techniques to 'think' that the underlying uncertainty small, leading them to produce too small prediction intervals. In addition, the baselines Na i ve WCP and Two Staged CP construct too wide intervals, that tend to overcover the response. In contrast, Inae ible WCP and the proposed PCP achieve desired coverage rate.
E.4.2 Missing features
In this section, we study the setting where the entries in the corrupted feature vector X(1) (1) are missing. Specifically, we artificially delete 20% of the features with the highest correlation to Y, from (0) to obtain (1).). The corruption indicator M, is defined similarly to other experiments, explained in Section D.I. Figure 12 shows the performance of each calibration scheme. This figure indicates that the uncalibrated model and naive CP tend to undercover the response variable. Also, the Naive WCP produces intervals with large variability. This behavior probably results from inaccurate estimates of the corruption probability PM x, its oracle counterpart, Infeas ible WCP, which uses the true corruption probabilities, precisely achieves the nominal coverage level. Furthermore, the baseline Two Staged CP generates too wide intervals that tend to overcover the response. In contrast, the proposed PCP consistently achieves the desired coverage rate a = 90%.
E.5 Ablation study of the effect /
This section studies the effect of the parameter 3 on the prediction sets constructed by PCP. We apply PCP on a synthetic data, introduced Appendix C.7 with different values of / E (0, ar). We follow the experimental protocol described in Appendix D. and report the coverage rate and average length achieved by PCP in Figure 13. This figure indicates that the smallest intervals are achieved for 9 that close to 0, and the interval sizes are increasing function of 3. Yet, it is important to understand
30

---

100
Method
Coverage rate
Uncalibrated
90
Naive CP
Naive WCP
Two Staged CP
Infeasible WCP
50
Privileged CP
facebook_1
facebook_2 2
bio Dataset
house
meps19
blog
Interval length
0.75
0.50
0.25
0.00
facebook_1 1
facebook_ 2
bio
house
meps19
blog
Dataset
Figure 10: Dispersive label-noise experiment: tabular datasets. The coverage and average interval length achieved by an uncalibrated quantile regression (Uncal mollib ibrated), naive conformal prediction which uses both clean and noisy samples (Naive CP), WCP which estimates the corrup- tion probability from X (Naive WCP), the baseline Two Staged CP, WCP which uses the oracle corruption probabilities (ean uses to the oracle corruption probabilities. All methods are applied to attain coverage rate at level & 90%. The metrics are evaluated over 20 random data splits.
WCP), and the proposed method (Privi leged CP) that
that different results could be obtained for different datasets. Therefore, recommend choosing using& a validation set, with grid of values for / in (0, a).
31

---

Method
Coverage rate
Uncalibrated
90
Naive CP
Naive WCP
Two Staged CP
Infeasible WCP
Privileged CP
facebook_1 1 facebook_2
bio
house
meps19
blog
Dataset
Interval length
0.75
0.50
0.25
0.00
facebook_1 1
facebook_2 2
bio
house
meps19
blog
Dataset
Figure 11: Contractive label-noise experiment: tabular datasets. The coverage and average interval length achieved by an uncalibrated quantile regression (Uncal riccalibrrs ibrated), naive conformal prediction which uses both clean and noisy samples (Naive CP), WCP which estimates the corrup- tion probability from X (Naive WCP), the baseline Two Staged CP, WCP which uses the oracle corruption probabilities (Infeasible WCP), and the proposed method (Privileged CP) that uses the oracle corruption probabilities, as well. All methods are applied to attain coverage rate at level a 90%. The metrics are evaluated over 20 random data splits.
Method
Coverage rate
Uncalibrated
90
Naive CP
Naive WCP
Two Staged CP
Infeasible WCP
Privileged CP
facebook_1
facebook
house
meps19
blog
Dataset
Interval length
0.75
0.50
0.25
0.00
facebook_ 1 facebook 2
bio
house
meps19
blog
Dataset
Figure 12: Missing features experiment: tabular datasets. The coverage and average interval length achieved by an uncalibrated quantile regression (Uncal sinaaibbrtedd) naive conformal predic- tion which uses both clean and noisy samples (Naive CP), WCP which estimates the corruption probability from X (Naive WCP), the baseline Two Staged CP, WCP which uses the oracle corruption probabilities st.feasible WCP), and the proposed method (Priv vileged CP) that uses to the oracle corruption probabilities. All methods are applied to attain coverage rate at level
1 & 90%. The metrics are evaluated over 20 random data splits.
32

---

100.0
Y coverage
97.5
92.5
90.0
0.00
0.02
0.04
0.06
0.08
0.10
beta
Interval length
12.5
10.0
7.5
0.00
0.02
0.04
0.06
0.08
0.10
beta
Figure 13: Ablation study of B. The coverage rate and average interval length achieved by PCP applied attain a coverage rate level 1 data splits.
90%. The metries are evaluated over 20 random
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The empirical experiments are provided in Section 4 and match the theoretical results in Appendix A. The problem setup is discussed in Section D.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: In Section 5 we discuss the limitations of our method, and in Sections 3.3 and 1.3 we review its computational efficiency.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification:
The theoretical claims are proved in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Appendix D we detail the full experimental setup and in Appendix C we describe all information about the datasets we used.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
35

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The software package implementing method and reproducing the periments is attached to the supplementary material. The datasets we used are publicly
Justification:
available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /ttp://nips...e cc/
public /wwp///gu/e/s//eeeeeee Cet//CoooSSSummm ssiiisssscppppp for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa: / /nips. cc/ public/ teliii/guiie gu/guee/e/////// CCodeoddSuuaal Submi as wwwoooooioooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer:
[Yes]
Justification:
In Appendix D we detail the full experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experiments are demonstrated with a boxplot which describes the variability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix D.4 provide the spec of the machines we used and in Ap- pendix D.5 we discuss the computation efficiency.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /teess///euuiipp. http:p//lll/ttttcccs..e.......ae
Answer: [Yes]
Justification: This paper does not involve human subjects participants, and all datasets used are publicly available.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section we discuss the potential positive and negative impacts of our method.
Guidelines:
37

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: In this paper, we introduce a calibration scheme that wraps predictive models. The datasets and models we use are standard, and thus we do not feel that our paper poses potential risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All sources we used are publicly available and we cited every asset we used in this work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
38

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw swi thoode com// dat aset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "js3Idojsis", "reviewer": "Reviewer_hB3A", "summary": "This paper is studying the problem of conformal prediction in the presence of data (covariate or label) corruption leveraging privileged data. They build upon the framework of weighted conformal prediction by introducing a novel leave-one-out weighting technique which produces a conservative (upper-bound) estimate of the original threshold of the weighted conformal prediction. They further theoretically and experimentally evaluate the coverage validity of their method.", "questions": "- How to compute w_i?\n- How to tune \\beta?\nI might ask more questions after i hear the response of the authors.", "limitations": "There is a limitation section that covers most of the limitations. However, I believe some of the limitations discussed in the very last section of the paper (specifically all the theoretical assumptions) must be presented and discussed much earlier in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presented framework is very general and can potentially be applied in a range of problems in practice.\n- Their method is very intuitive and insightful in the sense that their leave-one-out technique can potentially be applied to adjacent problems in conformal prediction.\n- The presentation of the paper is very nice. I particularly like the authors decision to present the \"two-stage naive\" method first to prepare the readers to fully absorb the underlying challenge of the problem.", "weaknesses": "I am willing to show a thumb up for this paper. However, before that I would appreciate if the authors can respond to the following concerns:\n\n1) I dont understand how one can compute the quantiles (Q_i) using the eq.(8). It looks like one need to know the values of w_i in order to compute Q_i. However, I can not find an explanation in the paper on how to compute w_i using data. A formulation of w_i is given in the paper using the probability distribution of M and Z. Are the authors assuming that the distribution of (Z, M) are known? if not how one can compute w_i? If the w_i is meant to be estimated from data, first how to do that estimation and second how does that estimation affects the coverage validity theorems? This is an important concern, and either way, this should be clearly stated before presenting the algorithm. In the current format, the algorithm is not complete!\n\n2) It is interesting, yet concerning, that there is no trace of the choice of \\beta in the presented theory. Can the authors comment on the choice of \\beta from point of views of theory and practice? It is odd to me how the algorithm might produce meaningful prediction sets for both \\beta = 0.00001 and \\beta = 0.99999! It is then a natural question/expectation that how one should tune \\beta in practice and how that choice affect the prediction set size and coverage validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive and valuable feedback and suggestions. In what follows, we address your concerns in detail.\n\n\n\n## The weights $w_i$\n \nWe thank the reviewer for raising this point. As the reviewer suggested, the real ratios of likelihoods, $w_i$, are required to provide the validity guarantee in Theorem 1. While PCP can be applied with estimates of $w_i$, which can be computed with estimates of $\\mathbb{P}(M=0 \\mid Z=z)$ according to equation (3), the validity guarantee does not hold in this case. This restriction is similar to WCP which also requires the true weights $w_i$ to provide a validity guarantee. The effect of inaccurate estimates of $w_i$ on the coverage rate attained by PCP could be an exciting future direction to explore, e.g., by borrowing ideas from [1].\n \n \nFollowing the reviewer's comment we updated Algorithm 1 (PCP) in the text, and added $w_i$ as an input. Furthermore, in Section 3.2 we explained if the real weights $w_i$ are unavailable, then they can extracted from estimates of $\\mathbb{P}(M=0 \\mid Z=z)$ according to equation (3), and this conditional probability can be estimated from the training data, using any off-the-shelf classifier. We also clarified in the updated manuscript that Theorem 1 does not hold if PCP is not used with the oracle weights.\n \n \n[1] Yonghoon Lee, Edgar Dobriban, and Eric Tchetgen Tchetgen. Simultaneous conformal prediction of missing outcomes with propensity score $\\epsilon$-discretization. arXiv preprint arXiv:2403.04613, 2024.\n\n\n \n## The choice of $\\beta$\n \nThank you for bringing up this important point, which was also raised by reviewer R7TW. First, we emphasize that Theorem 1 holds for any choice of $\\beta\\in (0,\\alpha)$. Therefore, $\\beta$ only affects the sizes of the uncertainty sets. Intuitively, as $\\beta \\rightarrow \\alpha$, a higher quantile of the weighted distribution of the scores is taken, and a lower quantile of the $Q_i$’s is taken. Similarly, as $\\beta \\rightarrow  0$ a lower quantile of the weighted distribution of the scores is taken, and a higher quantile of the $Q_i$’s is taken. An optimal $\\beta$ can be considered as the $\\beta$ that leads to the narrowest intervals. Such optimal $\\beta$ can be practically computed with a grid of values for $\\beta$ in $(0,\\alpha)$, using a validation set.\nNonetheless, to maintain Algorithm 1 (PCP) simple and intuitive, we did not optimize for $\\beta$ in the paper. In our experiments, we chose $\\beta$ that is close to 0, so the quantile of the weighted distribution of the scores chosen by PCP is close to the quantile chosen by (the infeasible) WCP. In response to this comment, we added this discussion to the text, and we conducted an ablation study for the choice of $\\beta$ on a synthetic dataset. The results of the ablation study are provided in the PDF file in the global response. This experiment indicates that the smallest intervals are achieved for $\\beta$ that is close to 0, and the interval sizes are an increasing function of $\\beta$. Yet, it is important to understand that different results could be obtained for different datasets. Therefore, we recommend choosing $\\beta$ using a validation set, as explained above. Thank you for the opportunity to discuss this issue.\n\n\n## Limitations\nWe thank the reviewer for raising this topic. We added a discussion about the limitations of PCP in Section 3.2. Specifically, we clarified that the true conditional corruption probability $\\mathbb{P}(M=1 \\mid Z=z)$ must be known to provide a theoretical coverage validity guarantee.\n \nWe thank the reviewer for these comments, which greatly improve our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHojheBsRZ", "reviewer": "Reviewer_UnMc", "summary": "The authors introduce a calibration method called Privileged Conformal Prediction (PCP) to generate prediction sets that guarantee coverage on uncorrupted test data, even when the target label (Y) and/or input features (X) in the calibration data samples are corrupted (e.g., missing or noisy variables). The key innovation is leveraging privileged information (PI)—additional features (Z) available during training but not at test time—to handle distribution shifts induced by corruptions. They assume that the input and target features of clean data (X(0),Y(0)) are independent of the corruption indicator variable (M) given the privileged information (Z). This allows the authors to treat this setting as a specific case of covariate shift (and weighted conformal prediction), with the added challenge that the PI variable is not available at test time (Ztest). To address this, they propose a reformulation of the weighted conformal prediction framework, where an estimate of the non-conformity score threshold without Ztest is obtained by considering a conservative estimate based on a quantile of the calibration thresholds. Experiments on real and synthetic datasets show that PCP achieves valid coverage rates and constructs more informative predictions than existing methods that lack theoretical guarantees", "questions": ".", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-presented, motivated, clear, and easy to follow. The problem seems relevant and novel within the context of conformal prediction, as far as I know. The authors effectively present the hypothesis and challenges of their problem. Discussing the two-stage approach before introducing the proposed solution is beneficial, as one might initially consider estimating Ztest from the input features and building the weight based on that estimate. I think the proposed method, where the authors estimate the non-conformity score threshold without Ztest by considering a conservative estimate based on a quantile of the calibration thresholds, is well-motivated and supported by theoretical guarantees, as presented in Theorem 1. Moreover, experimental results show the benefits of the proposed approach.", "weaknesses": "I do not see major weaknesses with this work, authors address the limitations of their approach in Section 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for the time and effort you put into this review. We are also very appreciative of your encouragement and positive assessment of our work. Your feedback is very important to us! Thank you again for your support.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qBpyVZISrZ", "reviewer": "Reviewer_3YMV", "summary": "This paper introduces a method to create prediction sets with guaranteed coverage in the presence of training data that is corrupted by missing or noisy variables. The approach is an extension of conformal prediction that works by assuming access to privileged information available during training. This information is used to account for distribution shifts caused by the corruptions. The proposed method is supported by theoretical coverage guarantees and empirical examples are used to demonstrate that the approach produces more reliable and informative predictions compared to existing methods on real and synthetic datasets.", "questions": "Could some discussion around what constitutes \"ideal\" privileged information for this particular method be possible?", "limitations": "Yes, sufficiently addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper stands out for its originality in addressing issues caused by corrupted training data through a novel extension of conformal prediction. Through a rather common assumption of access to privileged information the authors are able to develop an effective solution for obtaining conformal prediction sets despite the distribution shift cased by the corrupted data. The main results are rigorously proven and the presentation of the results and proofs are clear. The difficulty in proving Theorem 1 is well-explained and thus the the quality and significance in obtaining the result is evident. The practically of this result is apparent in that it potentially opens new possibilities for applying conformal prediction in high-stakes applications where there may be corrupted data.", "weaknesses": "The empirical evaluation, though comprehensive, could benefit from a broader range of real-world datasets. Also some discussion around access to and potential surrogates for the privileged information could better highlight the scope of the work. For example, some discussion on what could constitute privileged information and ideally even what characteristics of the information would make it most useful to the method would be interesting and really enhance the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We very much appreciate your positive feedback and interest in our work. We thank the reviewer for classifying our contribution as a novel one. We also thank the reviewer for their helpful comments and suggestions. In what follows, we address your comments in detail.\n$ \\newcommand{\\indep}{\\perp \\\\\\!\\\\!\\\\! \\perp}$\n\n## The Privileged Information\nWe thank the reviewer for raising this point which is related to a comment raised by Reviewer R7TW. In the updated manuscript, in Example 1 and Example 2 we expanded about the intuition behind the privileged information. Specifically, in the noisy response setup outlined in Example 1, we explained that since the PI, $Z_i$, is the information about the annotator, it is likely to explain the corruption appearances $M_i$. That is, in this example, the features $X_i(0)$ and the ground truth label $Y_i(0)$ do not provide additional knowledge about the corruption indicator given $Z_i$, i.e., $ X_i(0), Y_i(0) \\indep M_i \\mid Z_i=z$.\n \nAdditionally, we added the following example inspired by a real-world medical task. Consider a medical setup where patients are being selected for a costly diagnosis, such as an MRI scan. Here, $X_i(0)=X_i(1)$ is the more standard medical measurements of the i-th patient, such as age, gender, medical history, and disease-specific measurements. The PI $Z_i$ is the information manually collected by the doctor to choose whether the patient should be examined by an MRI scan. This information is obtained through, e.g., a discussion of the doctor with the patient, or a physical examination, and could include, for instance, shortness of breath, swelling, blurred vision, etc. The response $Y_i(0)$ is the disease diagnosis obtained by the MRI scan, and $Y_i(1)='\\texttt{NA}'$ is the missing value. The missingness indicator $M_i$ equals 0 if the doctor decides to conduct an MRI scan, and 1 otherwise. At test time, our goal is to assist the doctors in future decisions before examining the patients, and hence the test PI $Z_\\text{test}$ is unavailable. This task is relevant in situations where the number of available doctors is insufficient to examine all patients. Here, $Z_i$ explains the missingness $M_i$, and $M_i$ does not depend on $X_i$ or $Y_i$ given $Z_i$.\n\n\n\nFinally, we note that PCP can produce valid uncertainty sets even if the independence assumption $(X \\indep M) \\mid Z$ is not satisfied. In this case, we instead assume that both $X,Z$ explain the corruption indicator, namely, $(Y \\indep M) \\mid (Z,X) $, and that the features are uncorrupted $X(0)=X(1)$. Also, the weights $w_i$ are instead defined as: $w_i = \\frac{\\mathbb{P}(M=0)}{ \\mathbb{P}(M=0 \\mid X=x_i, Z=z_i)}$. We will clarify this point in the revised manuscript as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWF0CTO5p8", "reviewer": "Reviewer_R7TW", "summary": "This paper focuses on conformal prediction with \"privileged information\" in constructing prediction intervals with missingness/noise. The proposed quantile-of-quantile approach handles the difficulty that the \"privileged information\" is not available for test data and achieves finite-sample coverage.", "questions": "1. [Conditional independence assumption] Consider the scenario with split conformal prediction and Z is a function of X that is probably determined with the first half of training data, e.g. Z is the feature from X that correlates the most with Y. In this case, the conditional independence assumption may not hold precisely and a robustness result would be better to make theory complete.\n\n2. [Choice of beta] Ablation study on the choice of the hyperparameter beta in the simulation would be helpful in understanding the role of beta and a tradeoff in terms of accuracy could be expected.\n\n\n3. [Definition of Z] In some real applications, the \"privileged information\" is not pre-defined, and instead, it is usually learned from data. It would be very interesting to investigate the \"optimal\" partition of information in X into features and \"privileged information\". Sufficient dimension reduction, self-supervised training, and relevant techniques could be promising.", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and organized, which presents both solid theoretical guarantees and comprehensive simulation studies. The presentation is very good.", "weaknesses": "The intuition behind the notion of \"privileged information\" is not well explained. It would be better if examples of Z in real applications could be introduced and justified in the beginning. There are also hyperparameters, e.g. \\beta, in the proposed approach, for which ablation studies are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

Thank you for your time and effort in reviewing our submission and providing valuable feedback and suggestions.

In response to the reviewers' comment, we attached to this reply a PDF file containing the results of an ablation study analyzing the effect of $\beta$ on Algorithm 1 (PCP).

Once again, we thank the reviewers for helping us improve our paper!

Meta-review: This paper considers the problem of conformal prediction in the presence of data corruption assuming the availability of some privileged information (i.e. additional features that explain the distribution shift). They build upon the framework of weighted conformal prediction by introducing a novel leave-one-out weighting technique which produces a conservative (upper-bound) estimate of the original threshold of the weighted conformal prediction. The authors evaluate the coverage validity of their method both theoretically and experimentally.

All the reviewers agree that the paper provides a novel set of ideas and methodologies for the important problem of robust CP in the presence of corruption in the training data. Hence, I would like to recommend the paper to be accepted.
Decision: Accept (poster)