JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Authors: Andrew Prinster
Anqi Liu
Suchi Saria
Venue: NeurIPS 2022 Accept (2022)
Keywords: conformal prediction
uncertainty quantification
covariate shift
jackknife+
influence functions
error assessment
auditing

Abstract: We propose \textbf{JAWS}, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered on the core method \textbf{JAW}, the \textbf{JA}ckknife+ \textbf{W}eighted with data-dependent likelihood-ratio weights. JAWS also includes computationally efficient \textbf{A}pproximations of JAW using higher-order influence functions: \textbf{JAWA}. Theoretically, we show that JAW relaxes the jackknife+'s assumption of data exchangeability to achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of the sample size or the influence function order under common regularity assumptions. Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees to the reverse task: estimating the probability that a prediction is erroneous, based on user-specified error criteria such as a safe or acceptable tolerance threshold around the true label. We then propose \textbf{JAW-E} and \textbf{JAWA-E} as the repurposed proposed methods for this \textbf{E}rror assessment task. Practically, JAWS outperform state-of-the-art predictive inference baselines in a variety of biased real world data sets for interval-generation and error-assessment predictive uncertainty auditing tasks.

JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Drew Prinster
Anqi Liu
Department of Computer Science Johns Hopkins University
Department of Computer Science Johns Hopkins University
Baltimore, MD 21211 drew@cs. jhu. edu
Baltimore, MD 21211 aliu@cs juu edu
Suchi Saria
Department of Computer Science Johns Hopkins University Baltimore, MD 21211 ssariaăcs jhu edu
Abstract
We propose JAWS, a series of wrapper methods for distribution-free uncertainty quantification tasks under covariate shift, centered the core method JAW. the JAckknife+ Weighted with data-dependent likelihood-rathe weights. JAWS also includes computationally efficient Approximations of JAW using higher- order influence functions: JAWA. Theoretically, we show that JAW relaxes the jackknife+'s 's assumption of data exchangeability achieve the same finite-sample coverage guarantee even under covariate shift. JAWA further approaches the JAW guarantee in the limit of the sample size the influence function order under common regularity assumptions. Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees the reverse task: estimating the probability that prediction is erroneous, based on user-speeified error criteria such as a safe acceptable tolerance threshold around the true label. We then propose JAW-E and JAWA-E as the repurposed proposed methods for this Error assessment task. Practically, JAWS outperform state-of- the-art predictive inference baselines in a variety of biased real world data sets for interval-generation and arror-assessment predictive uncertainty auditing tasks.
Introduction
Auditing the uncertainty under data shift Principled quantification of predictive uncertainty is crucial for enabling users calibrate how much they should should trust given prediction (Thicbes et al. 202 Ghosh et al. 202 Tomsett et al. 2020 Bhatt et al., 202T]. Uncertainty-baat -based predictor auditing can be considered type of uncertainty quantification performed post-hoc, for example by regulator without detailed knowledge of predictor's 's architecture and with limited resources (Schulam and Saria 2019]. Data shift poses major challenge to uncertainty quantification due to violation of the common assumption that the training and test data are exchangeable, or more specifically independent and identically distributed (i.i.d.) [Ovadia et al. 2019 Ulmer all, 2020 Zhou and Levine 2021 Chan et al. 2020). Therefore, it is essential to develop convenient tools for users or regulators to audit the uncertainty of given prediction even when training data biased.
Predictor auditing: Interval generation In this work we distinguish between two types of predictive uncertainty auditing. We describe the first type interval generation, which refers common
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

goal in the distribution-fre free uncertainty quantification literature: to generate predictive confidence interval (or set) that covers the true label with at least user-specified probability. For instance, an auditor might ask for predictive intervals that contain the true label with at least, say 90% frequency.
Predictor auditing: Error assessment While predictive interval generation has been central focus of the distribution-free uncertainty quantification literature [Angeloppulos and Bates 202T). in some applications the reverse computation may be more actionable: estimating the probability that a prediction is erroneous or not, based on user-specified error critieria such as a safe or acceptable tolerance region around the true label. We thus refer to this task as error assessment. For instance, take the setting of chemical or radiation therapy dose prediction for cancer treatment, where administering a dose within approximately +10% of the optimal dose is considered safety-critical (see Appendix A. for details). Whereas predictive interval generation could fail to provide safety assurance (e.g., if the predictive confidence interval is larger than the safe tolerance region), error assessment would give worst-case probability of the prediction being safe. Similar examples could be formulated in other applications, such as incision planning in surgical robotics and autonomous vehicle navigation.
Coverage We assume a standard regression setup with a multiset of training data {(X1, Yi). (Xn, Yn)) and test point (Xn+1; Yn+1) with unknown label Yn+1, where (Xi, Y;) R x R for all i E {1....,, + 1}. Also, we denote a predictor ğ A Y2), .... (Xn, Y,))),
where A is a model-fitting algorithm. For a predictive interval (or set) Caudit I {subsets of IR}, a coverage guarantee gives lower bound to the probability that the interval covers the true test label:
P(Yn+I E Candit (Xn+1)) 2 21-a.
(1)
The coverage guarantee provides the basis for both
and error-assessment auditing, though it is important to note that in this work we focus on marginal rather than conditional coverage (see (Foygel Barber al. 202T for more details on this distinction). Standard conformal prediction methods (Vokk et al., 2005] Shater and Vovk 2008 Vovk 2013] along with the jackknife+ and related methods [Barber al. 2022]] which we refer to together "predictive inference" methods, provide a framework for generating predictive intervals with finite-sample guaranteed coverage.
Exchangeability Standard conformal prediction and the jackknife+ rely two crucial notions of exchangeability: data exchangeability, that is that the training and test data are all exchangeable (e.g., i..dd);; and secondly that the model-fitting algorithm AA treats the data symmetrically (Barber eet al. 2022]. In common situations of dataset shift, however, the data exchangeability assumption is violated. Empirically, the coverage performance of standard conformal prediction methods can suffer under data shift (Tibshirani et al. 2019, Podkopaev and Ramdas 2021
Neural network predictor
Random forest predictor
Frequency
Jacokmifee, covariate shift
Juckkkiffes, IID data (no shill)
475
165
6%0
355
X0
4:5
165
6%0
305
Coverage
Coverage
Figure 1: Jackknife+ loses coverage on the airfoil dataset under covariate shift (details in Section
In this work, we build on the jackknife+ method due its beneficial compromise between the statistical and computational limitations of other conformal prediction methods [Barber et al., 2021 However, jackknife+ coverage performance can still degrade under data shift, such as shown in Figure I and in some applications its computational requirements can still be limiting. To address these concerns and make extensions to error assessment, we develop JAWS, a series of wrapper methods for distribution- uncertainty quantification under covariate shift (see Table I for key properties).
Table I: Summary of key properties for JAWS methods (details in Section
Guarantee (under covariate shift) Finite sample
Method
Task
Asymptotic
Avoids retraining
JAW
Interval generation
X
JAWA
Interval generation
JAW-E
Error assessment
X
JAWA-E
Error assessment
X
2

---

Our contributions can be summarized as follows:
1. We develop JAW: a jackknife+ method with data-dependent likelihood-ratio weights for predic- tive interval generation under covariate shift. We show that JAW achieves the same rigorous, finite-sample coverage guarantee as jackknife+ [[Barber et al. 202T while relaxing the data exchangeability assumption to allow for covariate shift.
2. We develop JAWA: a sequence of computationally efficient approximations to JAW that uses higher-order influence functions to avoid retraining. Under assumptions outlined in Giordano et al. [2019a] regarding the regularity of the data, Hessian of the objective (local strong convexity). and the existence and boundedness of higher order derivatives, we provide an asymptotic guarantee for the JAWA coverage in the limit of the sample size influence function order.
3. We propose general approach to repurposing any distribution-fre predictive inference method to the error assessment task, with rigorous guarantees for the coverage probability estimation. Our approach applies methods that assume exchangeable data and to methods like JAW and JAWA that allow for covariate shift-JAW-E and JAWA-F refer to the error assessment versions.
4. We demonstrate superior empirical performance of JAWS over other distribution-fre predictive inference baselines on a variety of benchmark datasets under covariate shift.
2 Background and related work
2.1 Standard conformal prediction
Conformal prediction has grown into a broad research field since arising in the 1990s Vovk et al.
Shater and Vovk 2008 Balasubramanian et al. 2014 Angelopoulos and Bates 202 Standard conformal prediction methods generate a prediction interval (or set) with a samie- coverage guarantee in (. which is distributionnffee in the sense that the guarantee applies to any ex- changeable data distribution [Lei and Wasserman 2014 Lei et al. 2018). With the exchangeability assumptions in Section D standard conformal prediction methods rely a pre-fit fit score function
2005
S: IRd x R + I (in regression, the absolute-ve value residual score S(x, v) = ly [(x))] is commonly used). A conformal prediction interval at confidence level 1 1-aa a is then determined by a corresponding quantile on multiset of (exchangeable) score values.
Split conformal and full conformal are two main types of standard conformal prediction, and each bears its own limitation (Vok et al. 2005 [Shafer and Vovk 2008). Split conformal generates scores on labeled holdout data and is computationally efficient due to not requiring retraining. but sample splitting to obtain the holdout set can reduce model accuracy (Papadopouloss, 2008 Lei et al. 2018 Vovk 2012). On the other hand, full conformal prediction avoids the holdout set requirement, but at the heavy computational cost of retraining the model on every possible target value (or, in practice, on a fine grid of target values) [Ndiaye and Takeuchi 2019
Zeni el al. 2020
2.2 Covariate shift
Under the covariate shift assumption, the Y|X distribution is assumed to be the same between training and test data but the marginal X distributions may change (Sugiyama et al. 2007 Shimodaira 2000]
(X,, X2) lid Px X i L,.,, n
(Xu+1, Xn+1) Px x Py|x, independently.
Rich literature exist in this domain see Appendix A.2 for more details. Uncertainty quantification is relatively less explored under covariate shift, though recent work (Ovadia et al. [019 Zhou and
Levine 2021
Chan et al. 2020) emphasizes its importance, especially in deep learning.
2.3 Conformal prediction under covariate shift and beyond exchangeability
Tibshirani et al. 2019 develop the idea of weighted exchangeability for adapting conformal prediction to the covariate shift setting. Random variables Vi,..., Vn are weighted exchangeable with weight functions W1, -..; W., if their joint density O can be factorized as [(11, v,) II =
(v,)
g(vi, vn), where g is independent of ordering on its inputs. For covariate shift as in 2 if Px is
absolutely continuous with respect to Px, then the data {(X,, Y.)) are weighted exchangeable with
weight functions W1 wn = 1 and Wn+1 = dPx xdPx/xx (ibshiraai et al. 2019
3

---

If {or} represents a set of scores for standard conformal prediction, then we can represent the empirical distribution of {u;} as a+I
+ Ei= on, + n+I n where 6,, denotes a point mass v; [Barber al. 2022]. By, extension, weighted conformal prediction uses the weighted empirical distribution defined as 2i=1 (r)6vv + (2)1000600. with weights given by
(X,))
w(r) w w(x;) w(x)
(z) Σj=1 ((x)) + w(x) 12= 1, ... R
and P Pi+i(()
(3)
where w = /dPx//XXxxx so D (Xx+1) can thought of as a normalized likelihood ratio weight for each i E {1,..., ni + }]. Corollary 1 in (Tibshirani et al. 2019 provides the coverage guarantee of weighted conformal prediction that takes the form of | but relaxes the exchangeable data assumption to allow for covariate shift. However, weighted split and weighted full conformal inherit the same statistical and computational limitations, respectively, from their standard (exchangeable) variants.
The recent work of [Barber et al. 2022 provides a novel extension of conformal prediction and the jackknife+ to unknown violations of the exchangeability assumption, including "nonexchangeable
jackknife+" defined with fixed weights. The key difference between the nonexchangeable jackknife+
in Barber al. [2022] and our proposed JAW method is that Barber et al. [2022] use fixed weights to
compensate for unknown exchangeability violations (not limited to covariate shift) but at the expense of a bounded but generally nonzero "coverage gap" (drop in guaranteed coverage relative if the
data were exchangeable), whereas our JAW method with data-dependent weights assumes covariate shift but does not suffer from any similar coverage gap. See Appendix A.3 for more details.
2.4 Jackknife+
The jackknife+ [Barber et al. 2021]. which is closely related to cross conformal prediction (ookk et al. 2018). offers compromise between the statistical limitation of split conformal and the computational limitation of full conformal, at the cost of slightly weaker coverage guarantee. The jackknife+ predictive interval can most easily be understood as a modification to a predictive interval from the classic jackknife resampling method [Miller 1974 Steinberger and Leeb 2018 2016]. For
a set of point masses {dor) at values U1. Un, let Q n++I Ou denote the level / quantile on the
{ (+ 5.,} denote the level B quantile on I Sn, + 0 Then, denoting the model trained without the ith the empirical distribution E n+I
empirical distribution Ee= a+I 5og + n+I 5-x and let QJ
point as fi: (( (i), (X,-1. Yi-1), (X1+1, Yi+1).... (Xn: Yn))) and the leave-one-out residual 12:00 = =YY - (X,)), the jackknife prediction interval can be written as
Clackkniff (Xn+1) 10%+ ()( Qt
}}-
R+ [ (((..+))-R.
(4)
In contrast, obtain the jackknife+ predictive interval in Barber et al. model prediction ((Xn+1) in 4 with (Xn+1))
2021] by replacing the full
Čjuckknilet (Xn+1)= = [ 11111111110. (x_))))))) }. Qi { n+1 pp Rço0 }}. (5) P,ax
n+1
Barber et al. 2021 prove that, with the same exchangeability assumptions as in standard conformal prediction, the jackknife+ prediction interval satisfies
P(Y441
Öjackkmilet (X,+1)} 2 2a.
(6)
2.5 Approximating leave-one-out models with higher-order influence functions
Influence functions (IFs) [Cook] 1977] have a long history in robust statistics for estimating the dependence of parameters on sample data. Recently, IFs have become more widespread in machine learning for uses including model interpretability Koh and Liang [217 and approximating classic resampling- based uncertainty quantification methods including bootstrap [Schulam and Sarial 2019). jackknife, and leave-k-out cross validation [Giordano et al. 2019b al. In each of these cases. IFs enable approximation of the parameters that would be obtained if the model were retrained on resampled data by instead estimating the effect of corresponding reweighting. In prior work, Alaa and Van Der Schaar (2020) proposed approximating the leave-one-out models required by the jackknife+ with higher-order IFs, but their work assumes exchangeable or i.i.d. train and test data.

---

Let & denote the fitted parameters for predictor şi trained on the full training data. Given Assumptions 1-4 in Giordano et al. 20hic]-which which require that 0 is a local minimum of the objective function, that the objective is k - times continuously differentiable with bounded norms, and that the objective is strongly convex the neighborhood of 9 the the k-th order leave-one-out IF refers to the k-th order directional derivative of the model parameters & with respect to the data weights, in the direction of the leave-one-out change in weights (See AA.4 for more details). With each of these kth order leave-one-out IFs for l {1.... }), denoted with condensed notation 8t, 8 we can construct K-th order Taylor series approximation to estimated the leave-one-out model parameters A_;:
K
6 K =
In this work we implement the algorithm proposed by Giordano et al. 2019a compute higher-order IFs, a recursive procedure based on foreward- -mode automatic differentiation Maclaurin et al. 2015) for memory efficiency in computing higher-order directional derivatives. Our introduction of IFs is highly simplified-we we refer to Appendix A.4 and to Giordano et al. (2019a] for more details.
2.6 Error assessment
Whereas conformal prediction and related methods generate prediction intervals that control the error probability (miscoverage level (r) at a user-specified level, we refer to the reverse task as error assessment: estimating the probability that prediction is erroneous not, based on user-specified error criteria. For instance, a user might define an error as any deviation between the prediction ū(Xx+11) and the true label Yn+1 greater than some acceptable tolerance threshold that is, when [Yn+1- - ((),-1)l > T. In Section 3.3 we present a general approach to repurposing predictive inference methods with validity under covariate shift to error assessment.
We note that for score functions that are monotonic in y. such as S(x, y) (i)), guarantees for this eтror assessment task can be obtained using conformal predictive distributions as described by Vovk et al. 2017] (also see Vovk et al. 2020]. Vovk and Bendtsen (2018), Xie and Zheng (2022). )). In regression tasks assuming exchangeable data, CPDs generate a probability distribution for the label over IR. However, CPDs require that score functions be monotonic in y, whereas we allow for certain non-monotone conformity scores such as the commonly used absolute-value residual ly -;rr]; Moreover, CPDs assume exchangeable data, whereas our approach extends to covariate shift.
3 Proposed approach and theoretical results
3.1 JAW: Jackknife+ weighted with data-dependent weights
We present JAW, the JAckknife+ Weighted with data-dependent likelihood-ratio weights, defined by the following predictive interval:
GHAW (X++1)
Q7 {pr (Xn+1) - (x40)) J.
(Xn+1) .6 (X_ RR.00
(8)
where R2.00 = (x:((x) Y.J. with p." (x) for i {l, n + 1} as in Bp where QT (pT0 (X (Xn+1)
0p ((Xm11)-R((o0 } denotes the level a quantile of the empirical distribution Σ [p] (Xn+1) 1)
(x..(xx))) + p (X - - and where ( (Xn+1) (x.,(xx } is the
level 1 - a quantile for En= (Xn+1) ( (x + p xx+1)= -
We choose to define JAW using likelihood-ratio weights w(X,) dPx (x,)/dXx in the p," (x) to address covariate shift, but a similar result holds for other instances of weighted exchangeability and corresponding data-dependent weight functions (see Appendix B. We show that CJAW (Xn+1) satisfies the same coverage guarantee the jackknife+ except relaxing the data exchangeability assumption to allow for covariate shift, which we state formally in the following theorem.
Theorem Assume data under covariate shift from to Px, then the JAW interval in K satisfies
If Px is absolutely continuous with respect
IP PpnYn11 GAAW (Xn+1)} 2 1 22.
(9)
5

---

Remark 1. The results from Tibshirami et al. 2019 do not directly imply Theorem The approach in Tibshirani al. 2019 relies on leveraging the weighted exchangeability of the data to reweight the nonconformity scores [Vi,.. ,,.,, V so they can be treated as exchangeable, and for the jackknife+ this approach would entail treating [X=((xa x1) R.00 as implicit nonconformity scores. But, datapoints, whereas (1((++)) f is trained on n datapoints. Thus, no reweighting can make p=: equivalent in distribution to J and thereby allow us to treat the reweighted (=x((n+1) + R2:00 and fi(X.-1) + R2,41 as exchangeable.
observe that for E {1, n), fi= is trained on
Proof sketch: Our proof technique for Theorem extends the jackknife+ coverage guarantee proof in Barber et al. 2021 to the covariate shift setting for JAW using likelihood ratio weights as in Tibshirani al. 2019 The full proof is given Appendix C.T but the outline is as follows:
Setup: Following Barber et al. 2021 we define a set of leave-two-out models {ū-().))) We then generalize the notion of "strange" points described in Barber et al. 2021 to covariate shift.
1. Bounding the total normalized weight of strunge points: We establish deterministically that the total normalized weight of strange points cannot exceed 2a:
2. Weighted exchangeability using the leave-nwo-ond models: Using the leave-two-out model con- struction, we leverage weighted exchangeability to show that the probability that a test point n + 1 is strange is thus bounded by 2a.
3. Connection to JAW: Lastly, we show that the JAW interval can only fail to cover the test label value Y if
is strange point.
While JAW assumes access to oracle likelihood ratio weights, in practice this information often has to be estimated. See Appendix D.5 for discussion and experiments of JAW with estimated weights.
3.2 JAWA: Using higher-order influence functions to approximate JAW without retraining
For computationally efficient JAW Approximations that avoid retraining n leave-one-out models, we propose the JAWA sequence, which approximates the ceave-one-out models required by JAW using higher-order influence functions. For each training point E {l, n). define the K-th order influence function approximation to the leave-one-out refit parameters p_;, obtained from Algorithm 4 in Giordano et al. (211 as given by equation 7 and let = be the model with with these approximated parameters glf-K for each E {1,...,n] n). Then, the prediction interval for the K-th order JAWA (i.e., for JAWA-K) is given by
CJAWA-K (X.+1) n,
Qn (p"(("X"X (Xm+1) pp R.T.K K,000
Qi (X 6 (X + 1 }).
(10)
with R. RII-K.LOO
Y. pp" (x) in 33. and quantiles defined analogously to JAW.
We now provide an asymptotic coverage guarantee for ČIAWA-K (Xn+1) that holds either in the limit n,a of the sample size in the limit of the influence function order, under regularity conditions formally described in Giordano et al. 2019a These assumptions concern the regularity and continuity of the training data, local convexity of the objective (or that the Hessian is strongly positive definite), and the existence and boundedness of the objective's 's 1st through K + 1th order directional derivatives.
Theorem 2. Assume data under covariate shift from 2 and that Px is absolutely continuous with respect to Px. Let Assumptions 1- 4 and either Condition or Condition 4 from Giordano al. [2019a] hold uniformly for all n. Then, in the limit of the training sample size n + DC or in the limit of the influence function order K- 0o, the JAWA-K interval in (TOp satisfies
PP(Y+1 CUAWA-K ((x+1)) 2-
(11)
We leave the proof to Appendix but we note that the result follows by combining Propositions 1
and Giordano et al. 2019a with the JAW coverage guarantee that we present in Theorem |
3.3 Error assessment under covariate shift
We now propose a general approach repurposing predictive inference methods with validity under covariate shift from predictive interval generation to the reverse task: estimating the probability that

---

prediction is erroneous or not, based on user-specified error criteria. For example, consider user that defines a prediction p((,+1)) as erroneous, relative to the true label Y 1. if it is farther than
some acceptable tolerance threshold r from Y=+1: i.e., if Ya+1 jú( Xx, *++ > т. For this common regression error criterion, our approach to adapting method such as JAW P or weighted split
conformal prediction (Tibshirani et al. 2019] to error assessment reduces to first defining the set of
labels that would nor be considered erroneous, E [ T, (((x-1) + T], and then finding
the method's largest predictive interval contained within E, call it n, (Xn+1). The coverage guarantee for Cnraushi (Xn-1) then yields lower bound on P(Yn+1 E E}, the probability of no error (or an upper bound on the error probability). See Figure 2 for an illustration of this example.
((xx+1)
p(Xness)
p(X
Interval E
V
Vgl
Vf
V
VL.
Vil
V2
V/
+(X QU (Xns() 6yp)
Q5 -uye)
Figure 2: Illustration of approach to repurposing predictive inference method "w-audit" to error assessment. The interval E = [pt(x++ -T, ((Xx+1)++ +T) is shown in violet, the lower score values (V:*) in blue, the upper score values {V;") in red, and the interval (w--adit(X,.44 in green. Each vertical line at a location VÚ on the real line represents a point mass öy, with height corresponding to the normalized likelihood ratio weight p," (Xn-1).
Generally, a user must specify error criteria by a test point score function S Rd x RR + R (for conformal prediction, a nonconformity score), well as minimum and maximum acceptable score values 7 and t,, where -- r+; i.e., (x(X++1) is considered erroneous if S(X,+1, Yn+1) 1) T or if 7+ < ((X+11, Ya+)). (For nonnegative S, we might let -- = 0.) Then, the values of y for which observing Yn+1 y would not imply 10( is erroncous are:
E {v E RR : I S(Xn+1,9) }-
(12)
Now, assume a predictive inference method with predictive sets that can be written in the form Om-andin (Xn+1) {v E IR: QT ( 1) s S(Xn41, S Qt-a-vi (X (Xn+1)0yp }
(13)
with valid coverage guaranteed under covariate shift. (Note, 13 gives the JAW interval by setting 5(x.y) = y - fx))), V.L = =Tx:(X..+1) - Ti(X - R.00 and V = -
7 (X,++)) +
R.000 see Appendix B.3 Similarly, (13 gives the prediction interval for weighted split conformal
prediction (Tibshirani et al. 2019] for absolute value residual scores when S(z y) = by - (xx)). and
for all calibration data we let V. = =B ((X;)) and V.L = 0.) Then, defining
w-audit min (fa' : - - (p. 1) } (x (X > S r)) (14)
0
we can estimate the probability of (T(Xn+1) not resulting in an error as in 12) as:
w-audit
if ax w-audit exists
P(Yn+1 E}
a'
(15)
otherwise.
While the target coverage for Cna (Xn+1) is used in 15) the following theorem gives the worst- case error assessment guarantee for covariate shift (proof in Appendix C.3) Corollary i Appendix
B.3 Corollary 2 in Appendix B.4give the error assessment guarantees for JAW-E and JAWA-E
respectively. Appendix B.2 gives the analogous guarantee for exchangeable data.
Theorem 3. Assume a predictive inference method of the form 13 has coverage guarantee
P(YY+1 E Č.-auuii (xoquu((X++))) (X++1)} 2 1 - C1 C2 with C1, C1,C C2 E R. under covariate shift 2 where
Px is absolutely continuous with respect to Px. Define E as in 112 and a a as in 14 Then, 1 - c10 andii C2 if af exists and apgandi 1-c2
P(Yn+1 E FF)
(16)
0
otherwise.
7

---

Experiments
4.1 Datasets and creation of covariate shift
We conduct experiments on five UCT datasets Dua and Graff] [2017] with various dimensionality (Table : airfoil self-noise, red wine quality prediction [Cortez al.) 2009), energy converters, superconduetivity (Hamidieh] 2018), and communities and crime [Redmond and Baveja, 2002].
Table 2: Statistics for the UCI datasets. Only the first 2000 samples were used for the wave and superconductivity datasets (for wave, the first 2000 samples Adelaide data).
Dataset
samples
of features
Label range
Airfoil self-noise (airfoil)
1503
5
[103.38, 140.987]
Red wine quality (wine)
1599
11
[3. 8]
Wave energy converters (wave)
2000
48
[1226969, 1449349]
Superconductivity (superconduct)
2000
81
[0.2, 136.0] [0. 1]
Communities and crime (communities)
1994
99
We use exponential tilting induce covariate shift on the test data, based on the approach used in Tibshirani et al. 2019]. We first randomly sample 200 points for the training data, and then sample the biased test data from the remaining datapoints that are not used for training with probabilities proportional to exponential tilting weights. See Appendix D.I for additional details.
4.2 Baselines
Baselines for comparison to JAW We compared JAW to the following baselines:
1. Naive estimates are based on training data residuals |Y; -i((X)),, which suffers from overfitting.
2. Jackknife uses the classic Jackknife resampling as in 4
which replaces the prediction (((x++1) in jackknife with (X=(Xn+1)) 4. Jackknife-mm Barber et al. [021] is a more conservative alternative to jackknife+ that guarantees
3. Jackknife+ follows
coverage at the I - at level with exchangeable data, but usually with overly-wide intervals.
Cn.a
Gjjckkenfo-mm (X.+1)=
min (X.(X.,1))
Qt (R.000) max (X:(Xn+1) + QT- QT-
5. Cross validation+ (CV+) (Barber al. 202 similar to jackknife+ but splits data into K folds and replaces the ( with ju (X++1 the model trained with the kth subset removed.
6. Split method follows split conformal prediction, which uses half the data for training and the other half for generating the nonconformity scores.
Weighted split is a version of split conformal with likelihood ratio weights to maintain coverage under covariate shift, as in Tibshirani et al. 2019
Baselines for comparison to JAWA For influence function orders E (1,2,3), we compared the proposed JAWA-K method with -th order influence function approximations of the jackknife-based baselines that we used as comparisons to JAW-we thus refer to these approximations as IF-K jackknife, IF-K jackknife+, and IF-K jackknife-mm. Each baseline compared to JAWA-K is thus also approximated with the same K -thh order leave-one-ous influence function models.
4.3 Experimental results
We report experimental results on the predictive interval-generation task for both JAW and JAWA and on the eгror assessment task for JAW, compared baselines. Additional experimental details and supplementary experiments can be found in Appendix D including for estimated likelihood ratio weights D.5 ablation study with shift magnitudes in D.6 and coverage histograms in D.9
4.3.1 Interval generation results for JAW: Coverage and interval width
Figure compares JAW and its baselines, firstly regarding mean coverage and secondarily regarding median interval width, on all five UCI datasets for both neural network and random forest predictors,
Additional analysis in Appendix D and code at httt::///gsttub...oo//rrrrppssps

---

averaged over 1000 experimental replicttss. See Appendix D.2 for predictor function details. Meeting the target coverage level of 1 a the primary goal of the interval- -eeeerttgnnertioon audit task, but for methods that meet or nearly meet the target coverage level, smaller interval widths are more informative. Additionally, smaller variance in coverage indicates a more reliable or consistent method.
As seen in Figure B the JAW predictive interval coverage is above the target level of 0.9 across all datasets, for both random forest and neural network functions, along with the jackknife-mm and weighted split methods. However, JAW's interval widths are generally smaller and thus more informative than those of jackknife-min mm (which are often overly large, as noted in Barber al. (02TT). Weighted split and JAW perform similarly on mean coverage and median interval width (both methods have coverage guarantees under covariate shift), but JAW avoids sample splitting and as result has lower coverage variance than weighted split for all dataset and predictor conditions (see Appendix D.3). which suggests that JAW's s predictive intervals are more reliable.
JAW
naive
jackknile
jackenife+
jackknife-mm
CV+
spiR
weighted soli
1.0
1.0
coverage
0..
08-
0.8
08-
08-
0.6-
0.44
Neural not.
Randem
Mosiol
Sandom
Mound
Nandom
unddm
Intervall w dtth
200000
Median
20-
10
100
08
Nearal
Londom foc
Neurainer
Hardbm '01
Neural 101.
Hancom for
Hounal not.
Nandom
nD
Mourafret
Random
(a) Airfoil
(b) Wine
(c) Wave
(d) Superconduct
(e) Communities
Figure 3: Mean coverage (first row) and median interval width (second row) for neural network and random forest predictors UCI datasets. Dashed line is the target coverage level (1
0.9).
Error bars show the standard error of 1000 repeated experiments. JAW maintains target coverage under covariate shift for all predictor and dataset conditions along with jackknife-mm and weighted split-however, JAW's intervals are generally smaller and thus more informative than jackknife- ss and JAW's coverage variance smaller and thus more reliable than weighted split's (Appendix D.3)
4.3.2 Interval generation results for JAWA: Coverage and interval width
Figure 4 evaluates JAWA coverage and interval width compared to baselines for IF orders K E [1, with neural network predictor (see Appendix D.2 for predictor details). As with the JAW experiments, coverage at the target level of 1 - a = 0.9 is the primary goal, while secondarily, smaller intervals are more informative for methods that meet or nearly meet target coverage. For three of the five datasets (airfoil, wine, and communities), JAWA is the only method that consistently reaches or nearly reaches the target coverage level. JAWA and all the baselines perform well on the wave datasets, and in the superconduct dataset JAWA still outperforms approximations of jackknife and jackknife+ for all IF orders. Appendix D.7 provides an example empirical comparison of JAWA and JAW runtimes, which demonstrates that JAWA can be orders of magnitude faster to compute.
4.3.3 Error assessment results for JAW-E: AUC
We now turn to an error-assessment audit task where the goal is to evaluate a method's ability to estimate the probability that a given prediction erroneous or not, based on the error criterion |Y++1- ((Xn+11) > r. Let E- = [T(XX++) - T. (i(X++1) + 7]. Then, the goal is estimate the probability that ((X,+1) is correct, i.e., Yn+1 E; or an error, i.e., Yn+1 & E. For five predictive interval-generiae
methods repurposed to the error assessment task (JAW-E, jackknfeeeE. cross validation+E, split conformal-E -E, and weighted split conformal-E), Figure 5 reports the area under the receiver operating characteristic curve (AUROC) for 50 repeated experiments with neural network predictor, with dataset-specifee values of (see Appendix D.4 for details and additional experiments with random forest predictor). Better performing methods have higher AUROC values for all values r. For most tolerance levels and datasets, JAW achieves AUROC values comparable to jackknife+ and CV+ well as higher AUROC values than split and weighted split conformal prediction. The

---

Fipcklnife
Fipdktnie+
F-pdketleem
300
Weei coverage
0.0
0.6
102
30d
211
2nd
3md
155
3rd
102
2nd
375
Inteoval with
Med
2011
3r
2rd
3rd
20d
315
102
2nd
102
2nd
(a) Airfoil
(b) Wine
(c) Wave
(d) Superconduct
(e) Communities
Figure 4: Mean coverage (first row) and median interval width (second row) for JAWA and baselines for influence function orders E {1, 2, 3). Dashed line is the target coverage level (1 & = 0.9). Error bar shows the standard error of 200 repeated experiments. JAWA more consistent than baselines in reaching or nearly reaching the target coverage level across datasets and influence function orders, and it is more computationally efficient than JAW (Appendix D.7
JAW
jecknnlee+
CV+
Spitt conformal
Weighted split conformal
3-c
(a) Airfoil (b) Wine
(c) Wave
(d) Superconduct
(e) Communities
Figure 5: AUROC values for tolerance levels 7 across the three datasets for the neural net predictor, averaged across 50 experiment replicates. Results for random forest predictor in Appendix D.4
comparable performance of JAW and jackknife+ is likely due tradeoff between the benefit of JAW's validity under covariate shift and reduced effective sample size inherent to likelihood-ratio weighting, as jackknife+'s 's and CV+'s AUROC degrades with reduced sample size (Appendix (.6).
5 Conclusion
In this paper, we develop JAWS, a series of wrapper methods for distribution- free predictive uncer- tainty auditing tasks when the data exchangeability assumption is violated due to covariate shift. We also propose general approach to repurposing any distribution-fee predictive inference method to the error assessment task. We provide rigorous saipee guarantees for JAW and JAW-E on the interval generation and eтror assessment tasks respectively, and analogous asymptotic guarantees for the computationally efficient JAWA and JAWA-E. We moreover demonstrate superior performance of the JAWS series variety of datasets. In supplementary experiments JAWS' limitations: weight estimation can address the assumed access to oracle weights with similar empirical performance (Appendix D.5] and JAW's increased coverage variance with covariate shift can be explained by reduced effective sample size due to importance weighting (Appendix [.6]. Additionally, note that JAW and JAWA share a limitation with weighted conformal prediction (Tbsshirani et al. 2019 of potentially producing overly large intervals in extreme covariate shift cases where a test point's normalized likelihood ratio approaches or exceeds a. In the future, we aim to address the problems of reducing coverage variance and improving predictive interval sharpness.
investigate number of
Acknowledgments
This work was supported by the National Science Foundation grant IIS- 18S-188 We thank Yoav Wald for helpful discussions and advice, as well as Peter Schulam for sharing code that facilitated our influence function approximation implementation and AUC experiments.
10

---

References
Scott Thiebes, Sebastian Lins, and Ali Sunyaev. Trustworthy artificial intelligence. Electronic Markets, 31(2):447-466 2021.
Soumya Ghosh, Q Vera Liao, Karthikeyan Natesan Ramamurthy, Jiri Navratil, Prasanna Sattigeri, Kush R Varshney, and Yunfeng Zhang. Uncertainty quantification 360: A holistic toolkit for quantifying and communicating the uncertainty of ai. arXiv preprint arXiv:2106.01410. 2021.
Richard Tomsett, Alun Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani Srivastava, Gavin Pearson, and Lance Kaplan. Rapid trust calibration through interpretable and uncertainty- aware ai. Patterns, 1(4):100049, 2020.
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as form of transparency: Measuring. communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 401-413, 2021.
Peter Schulam and Suchi Saria. Can you trust this prediction? auditing pointwise reliability after learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1022-1031. PMLR, 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Dennis Ulmer, Lotta Meijerink, and Giovanni Cinà. Trust issues: Uncertainty estimation does not enable reliable ood detection medical tabular data. In Machine Learning for Health, pages 341-354. PMLR, 2020.
Aurick Zhou and Sergey Levine. Amortized conditional normalized maximum likelihood: Reliable out of distribution uncertainty estimation. In International Conference on Machine Learning, pages 12803-12812. PMLR, 2021.
Alex Chan, Ahmed Alaa, Zhaozhi Qian, and Mihaela Van Der Schaar. Unlabelled data improves bayesian uncertainty calibration under covariate shift. In International Conference Machine Learning, pages 1392-1402. PMLR, 2020.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-fiee uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-42,, 2021.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer Science & Business Media, 2005.
Glenn Shafer and Vladimir Vovk, A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Vladimir Vovk. Transductive conformal predictors. In IFIP International Conference on Artificial Intelligence Applications and Innovations, pages 348 360. Springer, 2013.
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife+. The Annals of Statistics, 49(1):486-507, 2021.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint 2rXiv:2202.134111 2022.
Ryan Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
11

---

Aleksandr Podkopaev and Aaditya Ramdas. Distribution-Fri free uncertainty quantification for clas- sification under label shift. In Uncertainty in Artificial Intelligence, pages 844-853. PMLR. 2021.
Ryan Giordano, Michael I Jordan, and Tamara Broderick. A higher-order swiss army infinitesimal jackknife. arXiv preprint arXiv:1907.12116, 2019a.
Vineeth Balasubramanian, Shen-Shyang Ho, and Vladimir Vovk. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
Jing Lei and Larry Wasserman. Distribution-fres prediction bands for non pana-eerattticc regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(1):71-998 2014.
Jing Lei, Max G'Slll, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-fre predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094-1111, 2018.
Harris Papadopoulos. Inductive conformal prediction: Theory and application to neural networks. INTECH Open Access Publisher Rijeka, 2008
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pages 475-490. PMLR, 2012.
Eugene Ndiaye and Ichiro Takeuchi. Computing full conformal prediction with approximate homotopy. Advances in Neural Information Processing Systems, 32, 2019.
Gianluca Zeni, Matteo Fontana, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. arXiv preprint arXiv:2005.07972, 2020.
Masashi Sugiyama. Matthias Krauledat, and Klaus-Robers Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Vladimir Vovk, Ili Nouretdinov, Valery Manokhin, and Alexander Gammerman. Cross-conformal predictive distributions. In Conformal and Probabilistic Prediction and Applications, pages 37-51. PMLR, 2018.
Rupert G Miller. The jackknife-a review. Biometrika, 61(1):1-15, 1974.
Lukas Steinberger and Hannes Leeb. Conditional predictive inference for high-dimensional stable algorithms. arXiv preprint arXiv: 180..01422 2018
Lukas Steinberger and Hannes Leeb. Lcave-one-out prediction intervals in linear regression models with many variables. arXiv preprint 16Xi::602..0800) 2016
Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):15-18, 1977.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pages 1885-1894 PMLR, 2017.
Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1147. PMIR, 2019b.
Ahmed Alaa and Mihacla Van Der Schaar. Discriminative jackknife: Quantifying uncertainty in deep learning via higher-order influence functions. International Conference on Machine Learning. pages 165-174. PMLR, 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Autograd: Effortless gradients in numpy. In ICML 2015 AutoML workshop, volume 238, 2015.
12

---

Vladimir Vovk, Jieli Shen, Valery Manokhin, and Min-ge Xie. Nonparametrie predictive distributions based on conformal prediction. In Conformal and Probabilistic Prediction and Applications, pages 82-102. PMLR, 2017.
Vladimir Vovk, Ivan Petej. Ili Nouretdinov, Valery Manokhin, and Alexander Gammerman. Compu- tationally efficient versions conformal predictive distributions. Neurocompptiing 397:292-300 2020.
Vladimir Vovk and Claus Bendtsen. Conformal predictive decision making. In Conformal and Probabilistic Prediction and Applications. pages 52-62. PMLR, 2018.
Min-ge Xie and Zheshi Zheng. Homeostasis phenomenon in conformal prediction and predictive distribution functions. International Journal of Approximate Reasoning, 141:131-145,
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL Bttppttt://archivee... ics uci edu/mll
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a supercon- ductor. Computational Materials Science, 154:346-354, 2018.
Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research, 141(3):660-66) 2002.
Mary Feng, Gilmer Valdes, Nayha Dixit, and Timothy Solberg. Machine learning radiation oncology: opportunities, requirements, and needs. Frontiers in oncology, 8:1 2018.
Elizabeth Huynh, Ahmed Hosny, Christian Guthier, Danielle S Bitterman, Steven F Petit, Daphne A Haas- Kogan, Benjamin Kann, Hugo JWL Aerts, and Raymond Mak. Artificial intelligence in radiation oncology. Nature Reviews Clinical Oncology, 17(12):771-781 2020.
Saul Weingart, Lulu Zhang, Megan Sweeney, and Michael Hassett. Chemotherapy medication errors. The Lancet Oncology, 19(4)ee991e119) 2018.
Marcel Van Herk. Errors and margins in radiotherapy. In Seminars in radiation oncology, volume 14, pages 52-64. Elsevier, 2004.
Howard Gurney. How to calculate the dose of chemotherapy. British journal of cancer, 86(8): 1297-1302, 2002.
Michael R Cohen, Roger W Anderson, Richard M Attilio, Laurence Green, Raymond J Muller, and Jane M Pruemer. Preventing medication enrors in cancer chemotherapy. American Journal of Health- Shuulth-System Pharmacy, 53(7):737-746, 1996.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift.
Journal of Machine Learning Research, 10(9), 2009.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
Yaoliang Yu and Csaba Szepesvári. Analysis of kernel mean matching under covariate shift, arXiv preprint arXiv: 1650.4600.6 2012.
Kai Zhang, Vincent Zheng, Qiaojun Wang, James Kwok, Qiang Yang, and Ivan Marsic. Covariate shift hilbert space: solution via sorrogate kernels. In International Conference on Machine Learning, pages 385-395. PMLR, 2013.
13

---

Yin Zhao, Longjun Cai, et al. Reducing the covariate shift by mirror samples in cross domain alignment. Advances in Neural Information Processing Systems, 34, 2021.
Anqi Liu and Brian Ziebart. Robust classification under sample selection bias. Advances in neural information processing systems, 27, 2014.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart. Robust covariate shift regression. In Artificial Intelligence and Statistics, pages 1270-1279. PMLR, 2016.
John C Duchi, Tatsunori Hashimoto, and Hongseok Namkoong. Distributionally robust losses against mixture covariate shifts. Under review, 2, 2019.
Ashkan Rezaei, Anqi Liu, Omid Memarrast, and Brian D Ziebart. Robust fairness under covariate shift. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9419- 9427, 2021.
Luisa Turrin Fernholz. Von Mises calculus for statistical functiooals, volume 19. Springer Science Business Media, 2012.
HG Landau. On dominance relations and the structure of animal societies: Ii the condition for score structure. The bulletin of mathematical biophysics, 15(2):143-144, 1953.
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Samyadeep Basu, Philip Pope. and Soheil Feizi. Influence functions deep learning are fragile. arXiv preprint arXiv:2006. 14651, 2020.
Fengpei Li, Henry Lam, and Siddharth Prusty. Robust importance weighting for covariate shift. In International Conference on Artificial Intelligence and Statistics, pages 352-362. PMLR, 2020.
Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael Jordan. Uncertainty sets for image classifiers using conformal prediction, arXiv preprint ar 200.:2009.4493, 2020.
14


Reviews:
[{"review_id": "wXwVHGXuS5w", "reviewer": "Reviewer_qDpq", "summary": "This work introduces a new method, JAW, that incorporates importance weighting into Jackknife+. They demonstrate that JAW extends Jackknife+’s coverage guarantee to situations with covariate shift. Finally, they demonstrate the effectiveness of this method on experiments.", "questions": "* In line 144-145: $\\hat{\\mu}\\_{-i}$ includes $(X\\_{n+1}, Y\\_{n+1})$, but that doesn’t seem to be the case for the definition of $\\hat{\\mu}_{-i}$ in [1]. Are these supposed to be (slightly) different things?\n* I may have missed this, but what are the values of $\\beta$ for Figs. 3 and 4?\n* Also regarding Fig 3., it seems that JAW and Jackknife-mm have comparable performance when using random forests, while Jackknife-mm can have much larger interval widths with neural networks. Is there an intuitive reason for this?\n* Why are Naive, Jackknife-mm, and Jackknife missing from Fig. 4? Particularly Jackknife-mm as it seemed like the most competitive baseline in Fig. 3.\n\n[1] Barber, Rina Foygel, et al. \"Predictive inference with the jackknife+.\" The Annals of Statistics 49.1 (2021): 486-507.", "limitations": "* The main limitations of this method are adequately discussed in the work: in particular, the computational difficulty and the variance of JAW.\n* This method, like other IW-based methods, requires knowing the covariate shift from train to test, which may not be applicable in many settings. Is there a way to incorporate weight estimates?", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* An valuable work in developing the Jackknife+ framework and extending its utility to an important use case.\n* Good technical contribution\n\nWeaknesses:\n* The method so far seems infeasible for larger-scale problems", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H5YsLQ8iZgq", "reviewer": "Reviewer_MNhq", "summary": "This paper studies conformal prediction under covariate shift and presents a weighted version of jackknife+ for solving it.", "questions": "- The notations in the paper seem not consistent. For example, the quantile functions in Eq. (4) and Eq. (6) use different notations.\n- Is the equation of likelihood ratio correct in Line 105, Page 3? Does X_i follow the training distribution or the test distribution?", "limitations": "None.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem this paper studies is significant.\n- Overall, the paper is well written.\n- The theoretical analysis looks solid.\n\nWeaknesses:\n- The major problem of this paper is its originality. The authors claim that the weighted version of jackknife+ proposed in [Barber et al., 2022] uses fixed and not data-dependent weights (and thus differs from the authors' proposed weighted jackknife+). However, in my understanding, Barber et al. (2022) consider conformal prediction in a more general setting (non-exchangeable), and therefore, the weights in their method are not specified. More importantly, I think the weights should reduce to those appearing in the proposed method of this paper if we consider the covariate shift setting (a special case of non-exchangeability), which implies the proposed method of this paper can be regarded as a special case of [Barber et al., 2022]. And obtaining these weights under covariate shift looks not difficult as long as we follow [Tibshirani et al., 2019] and [Barber et al., 2022].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8mPVzk6z-a4", "reviewer": "Reviewer_3P4m", "summary": "Jackknife+ is an approach to construct a predictive interval with coverage guarantee under the exchangeable assumption. This paper extends Jackknife+ for covariate shift. In particular, the paper proves that the proposed Jackknife+ Weighted with data-dependent weight (JAW) predictive interval satisfies the same coverage guarantee as the original Jackknife+ given the likelihood ratio. The paper empirically demonstrated that the proposed JAW predictive interval satisfies a target coverage level, while generating a small interval width, compared to six baselines. \n", "questions": "Thanks for introducing an extension of Jackknife+ for covariate shift.\n\n* As mentioned before my main concern lies in the practicality of Jackknife+ (though I believe that Jackknife+ could be useful) and the benefit of JAW compared to conformal prediction for covariate shift. In particular, Jackknife+ requires keeping n models to construct a predictive interval; in the case of a large dataset like ImageNet, which consists of 1.2M images, this means I need to keep 1.2M resnet152 to run Jackknife+, which does not sound feasible. In this sense, split conformal prediction is a significantly appealing approach, thus its extension to covariate shift (i.e., weighted split conformal prediction (WSCP) by Tibshirani et al. [2019]) would be enough for conformalized prediction intervals under covariate shift. In other words, the motivation to use JAW seems to be weak, though Jackknife+/JAW themselves have unique algorithmic features. Why or when can JAW be more useful than WSCP? If WSCP is less sample-efficient (as it needs a hold-out set), what is the range of the number of samples where JAW is better than WSCP? In practice, we anyway need a holdout set for model selection or likelihood ratio estimation, then why not properly exploit this holdout set along with WSCP for conformalized prediction sets?\n\n* I believe that WSCP is also a good comparing approach, but it is missing in the main experiments, though SCP is included. I think a comparison to WSCP is needed to quantitatively understand the benefit of JAW as a conformalized prediction set under covariate shift. \n\n* Barber et al. 2022 is clearly related work, but the proposed approach is not well contrasted to Babar et al. 2022. Why does Babar et al. 2022 make a stronger assumption (i.e., a fixed weights)? What are the main challenges on removing the stronger assumption? What's the key ideas on addressing the challenges?", "limitations": "The authors mentioned the high variance of coverage rate and computational overhead. In addition to this, the assumption that the likelihood ratio is known should be part of the limitations since otherwise JAW cannot get the desired guarantee. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### originality\nCompared to Barber et al 2022 (which is the most related work), this paper removes the strong assumption on the likelihood ratio (i.e., the likelihood ratio is fixed) to satisfy the coverage guarantee; this is novel, though it would be more instructive to state key differences on the generalization from Barber et al 2022. The proof technique seem to be a combination of Tibshirani et al. [2019] and Barber et al. [2021], but anyway this is a novel combination.\n\n### quality\nThis paper may theoretically sound and the empirical results support the claim.\n\n### clarity\nOverall clearly written, but the following may enhance the readability.\n* Barber et al 2022 also proposed the Jackknife+ extension for covariate shift, so it’s better to introduce it in Section 2 for clear comparison.\n* line 145: I guess (X_{n+1}, Y_{n+1}) is a typo; the algorithm considers the point up to n.\n* line 165: it contains an incomplete math notation. \n* references to Appendix are broken.\n* line 234: Jacknife-mm is not well defined; i is not a free-variable.\n\n### significance\nExtending uncertainty quantification approaches for covariate shift is required. But, the main concern lies in the usefulness of Jacknife+ in practical setups (i.e., it needs to keep n models to construct a predictive interval). So, extending Jackknife+ for covariate shift may not be a significant issue\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7X2rYIgoSE", "reviewer": "Reviewer_AAyc", "summary": "The authors focus on introducing a new method named JAW to estimate predictive inference under covariate shift, and it achieves superior performance over the baselines on different data sets and architectures of black-box model. The proposed method is a weighted version of the jackknife+, and the weights are not a new technique. What's more, the idea of a weighted version of conformal prediction under covariate shift is not new, so the innovation of this paper is limited.\n", "questions": " 1. The style of the citations is different for different sections.\n\n  2. There are writing issues in line 107.\n\n  3. There seems to be reference error in line 109 and line 110, line 168", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength：\n  The authors analyze the problem scenario and the motivation clearly. \n  \nWeaknesses：\n  1. There seems to be an error in line 145, the model is trained with the ith datapoint removed, but the definition A(·) seems to be trained with the ith datapoint. And compared with \\hat{\\mu} defined in line 77, \\hat{\\mu_{-i}} seems have one more element, do authors mean \\hat{\\mu_{-i}(X_{n+1})}? Imprecise use of symbols makes definitions ambiguous.\n\n  2. Many symbols are not formally defined when used. Although this is understandable due to space constraints, the authors had better add the note \"as defined in [ ]\" at least, or it will cause reading difficulties, and the reader needs to guess the meaning of each symbol according to the previous papers.\n\n  3. The paper lacks innovation, the idea is based on [1] and the technique is from [1], then this technique is added to [2]. Hope the authors can put forward more analysis and improvement.\n\n\n[1] Ryan J Tibshirani, et al. Conformal prediction under covariate shift. NeurIPS, 2019.\n\n[2] Rina Foygel Barber, et al. Predictive inference with the jackknife+. The Annals of Statistics, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept