Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift
Authors: Jiawei Ge
Debarghya Mukherjee
Jianqing Fan
Venue: NeurIPS 2024 poster (2024)
Keywords: prediction intervals aggregation
unsupervised domain shift
minimal width
adequate coverage
distribution shift
Primary area: learning_theory

Abstract: As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts.
A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. 
The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. 
In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain.
Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation.
Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.

Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift
Jiawei Ge
Debarghya Mukherjee"
Operations Research & Financial Engineering Princeton University
Department of Mathematics and Statistics Boston University
ig5300@princeton.cor
mdeb@bu.edo edu
Jianqing Fan
Operations Research Financial Engineering Princeton University
j jgtanpprinceton.ed edu
Abstract
As machine learning models increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with dis- tribution shifts. A distribution shift occurs when the underlying Cata-generating process changes, Icading to deviation in the model's performance. The predic- tion interval, which captures the range of likely outcomes for given prediction, serves as crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) bounded density ratio, and ii) mreasure-preserving transformation. Our proposed methodologics are computationally efficient and casy to implement. Beyond illustrating the performance of method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous the- oretical guarantees, coupled with finite sample bounds, regarding the coverage and width our prediction intervals. Our approach excels in practical applica- tions and is underpinned by solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.
Introduction
In the modern era of big data and complex machine learning models, extensive data collected from diverse sources are often used to build a predictive model. However, the assumption of independent and identically distributed (i.i.d.) data frequently violated in practical scenarios. Take algorith- mie fairness as example: historical data often exhibit sampling biases towards certain groups, like females being underrepresented in credit card data. Over time, the differences group propor- tions have diminished, leading to distribution shifts. Consequently, models trained on historical data may face shifted distributions during testing, and proper adjustments are needed. Distribution shift has garnered significant attention from statistical and machine learning communities under various names, i.c., transfer learning [PY09, WKW16) domain adaptation [FVRA21], I]. domain generaliza- tion + 22, WLL 22], continual learning [DLAM+2] 221, MLJ+ 22], multitask learning [ZY21]
"quul contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

etc. While numerous methods are available in the literature for training predictive models under dis- tribution shift, uncertainty quantification under distribution shift has received relatively scant atten- tion despite its crucial importance, One notable exception is conformal prediction under distribution shift; [TFBCRT 19] proposed variant of standard conformal inference methods to accommodate test data from a distinct distribution from the training data under the covariate shift. Recently, [GC21] introduced an adaptive conformal inference approach suitable for continuously changing distribu- tions over time. Additionally, quantile regression under distribution shift offers another avenue for addressing uncertainty quantification under distribution shift [ERS+2] 22]].
Although few methods exist for constructing prediction intervals under distribution shift, most focus primarily on ensuring coverage guarantee rather than minimizing interval width. This prompts the immediate question:
Can we generate prediction intervals in the target domain that provide both i) coverage guarantee and ü) minimal width?
This paper seeks to address this question by leveraging model aggregation techniques [NW15, MNW16, CENI4, Vov15, HKNCI4]. Suppose we have K' different methods for constructing prediction intervals in the source domain. Our proposed approach efficiently combines these methods to produce prediction intervals in the largel domain with adequate coverage and minimal width. When individual methods are the elementary basis functions, such as the kernel basis, the resulting aggregation is indeed construction of the prediction interval based the basis functions. Our methodology draws inspiration primarily from recent work [FGM23] on prediction interval aggregation under the i.i.d. setting. However, key distinction lies in our focus on unsupervised domain adaptation, where we can access labeled samples from the source and unlabeled samples from the target domain. Certain assumptions regarding the similarities between these domains are necessary to facilitate knowledge transfer from the source to the target domain. We explore two types of similarities in this paper: i) covariate shift, where we assume that the distribution of the response variable Y given X is consistent across both domains, albeit the distribution of X may differ, and ii) domain shift, where we assume that the condi- tional distribution of Y given X remains unchanged up to a measure- mcasure-preserving preserving transformation. Covariate shift is a well-explored concept in transfer learning and has also garnered attention in uncertainty quantification. It allows different distributions of X while maintaining identical conditional distributions Y|X across domains. For constructing conformal prediction intervals within this framework, see [TFBCRR9, HL23, YKT22, LC21] and references therein. On the other hand, distribution shift is more general, allowing both the distribution of X and the conditional distribution of Y|X to differ across domains. Our methods in this context draw upon domain matching principles via transport map, as proposed in [CFT14] and further elaborated in subsequent works like [CFTR16, CFHRI7, RHS17). among others. The key assumption is the existence of a measure-prserrvnndddmman---l-
map T from the target to the source domain, such that the conditional distribution of Y|X on the target domain matches ( conditional distributions matches upon domain alignment. The case where the domain- map is the optimal transport map has received considerable attention in the literature, c.g., [CFT14, CFTR16, CFHR17, 2L]. Empirical evidence supports the efficacy of domain alignment through optimal transport maps across various datasets. For instance, in [2], a variant of this method is applied for domain adaptation in image recognition tasks, such as recognizing similarities between USPS [Hu194], MNIST (LBBH98), and SVHN digit images [NWC+11], as well as between different types of images in the Office-home dataset [VECP17), including artistic and product images. Additionally, in [CFT14], the authors explore the impact of domain alignment via optimal transport maps on the face recognition problem, where different poses give rise to distinct domains. However, most of these works concentrate on training predictors that perform well on the target domain without any guarantee regarding uncertainty quantification. To our knowledge, this is the first work to propose a method with rigorous theoretical guarantees for constructing prediction intervals on the target domain under the domain- aligning assumption within an unsupervised domain adaptation framework. We now summarize our contributions.
the source domain, i.c.,
This paper introduces a novel methodology for aggregating various prediction methods available on the source domain to construct unified prediction interval on the target domain under both covariate shift and domain shift assumptions. Our approach simple and easy to implement and requires solving a convex optimization problem, which can even be
Our Contributions:

---

simplified to linear program problem in certain scenarios. We also establish rigorous theoretical guarantees, presenting finite sample concentration bounds demonstrate that our method achieves adequate coverage with a small width. Furthermore, our methodology extends beyond model aggregation; can be used to construct efficient prediction intervals from any convex collection of candidate functions. In the paper, we adopt this broader perspective, discussing how the aggregation of prediction intervals emerges as a particular case. Lastly, we validate the effectiveness of our approach by analyzing real-world datasets.
We also want to highlight the differences between our method and related method proposed in [FGM23]. We deal with unsupervised domain adaptation, i.e., we do not observe any label from the target domain, in contrast to [FGM23], which only deals with i.i.d. data, Hence, significant changes in methodology are required to address the domain shift. Furthermore, as pointed out in Section 3, the shift may cause the optimization problem non-convex, for which we need to introduce a convex surrogate (e.g., the hinge function), leading to additional theoretical challenges.
2 Notations and preliminaries
Notation The covariates of the source and the target domains are denoted by X' and Xy,, respec- tively, and X' == Xs U Nr. The space of the label is denoted by V. We use the notation Es (resp. Er) to denote the expectation with respect to the source (resp. target) distribution. The expectation with respect to sample distribution is denoted by E,,s and E,,r. We use ps (resp. pr) to denote the probability density function of X on the source and the target domain, respectively. Throughout the paper, we use to denote universal constants, which may vary from line to line.
2.1 Problem formulation
Our setup aligns with the unsupervised domain adaption; we assume to have ns i.i.d. labeled samples (Xs,i, & ~ (s( Y) from the source domain, and i.i.d. unlabeled samples {XT 1T ~ (x(X) from the target domain. Given any a > O, ideally, we want to construct a valid prediction interval with minimal width on the target domain:
(2.1) In many practical contexts, the preferred prediction interval takes the form of m(X) = (X). where m(X) is predictor for Y given X (an estimator of Er[Y X)). and (XX) gauges the uncertainty of the predictor m(X). The optimizer of (2.1) takes this simplified form when the distribution of Y E X] is symmetric around O. Moreover, it offers straightforward interpretation as the pair (m, g) is predictor and a function quantifying its uncertainty. Within the framework of this simplified prediction interval, need to estimate m and g. Estimating the conditional mean function m is relatively easy and has been extensively studied; one may use any suitable catpenerrri/ooooparmmmettt
min un
Ex(u(X) I(X)), S.t. Pr (I(X) S Y x(X)) 2 a
method. Upon estimating m, we need to estimate g so that the prediction interval [m () + g(X] has both adequate coverage and minimal width. This translates into solving the following optimization problem:
min JEF
Ex[s(X)], s.t. F - m(())² > f(X)) Sa.
(2.2)
Let fo be the solution of the above optimization problem. Then the optimal prediction interval is [m)(x) + /=((=)] However, the key challenge here is that we do not observe the response variable Y from the target, and consequently, solving (2.2) becomes infeasible. Hence, we must rely on transferring our knowledge acquired from labeled observations in the source domain, which neces- sitates making certain assumptions regarding the similarity between the two domains. Depending on the nature of these assumptions regarding domain similarity, our findings are presented in two sections: Section 3 addresses covariate shift under the bounded density ratio assumption, while Sec- tion 4 considers more general distribution assumption under measure- preesure-rresrviigg transformations. Furthermore, as will be shown later, this problem, though well-defined, not easily implementable. Therefore, we propose surrogate convex optimization problem in this paper and provide its theo- retical guarantees.
2.2 Complexity measure
The complexity of the function class F is usually quantified through the Rademacher complexity, defined as follows.
3

---

Definition 2.1 (Rademacher complexity). Let F be a fufunttion class and {X+]²- be a set of samples drawn i.i.d. from a distribution DD. The Rademacher complexity of F is defined as
sup Σ Eaf(x)) [EF
(F) Ee
(2.3)
i=1
where (6i) are ii.dd Rademacher random variables that equals to +1 with probability 1/2 each.
3 Covariate shift with bounded density ratio
Setup and methodology In this section, we focus on the covariate shift problems, where the marginal densities (X( and (x( of the covariates may vary between the source and target do- mains, albeit the conditional distribution Y|X remains the same. Denote by mo (z) = Er [ = a] = Es Y z/., the conditional mean function. For the ease of the presentation, we assume mo is known. If unknown, one may use the labeled source data to estimate it using suitable parametric/non-paae paramorrre estimate (e.g.. splines, local polynomial, or deep neural networks), sub- sequently substituting mo with nn in our approach. The density ratio of the source and the target distribution of is denoted by wo(x) == pr(x)/ps(1)) We henceforth assume that the density ratio is uniformly bounded:
Assumption 3.1. There exists W such that suPzeXs wo(x) S W.
If Wig is known, (2.2) has the following sample level counterpart:
sen (x))) s.t. En.s which is NP-hard owing to the presence of the indicator function. However, in many practical scenarios, it is observed that the shape of the prediction band does not change much if we change the level of coverage (i.e., o); only the bands shrink/expand. Indeed, the true shape determines the average width; if the shape is wrong, then the width of the prediction band is quite likely to be unnecessarily large. Therefore, to obtain prediction interval with adequate coverage and minimal width, one should first identify the shape of the prediction band and then shrink/expand it appropriately get the desired coverage. This motivates the following two steps procedure:
S a,
(3.1)
Step 1:
(Shape estimation) Obtain an initial estimate fuit by solving (3.1) for a = (to capture the shape):
minjer Ex)], s.t. s(X:) 2 (Y, -Y: (x.)) V1 < ns wo wo (Xa) 0. (3.2)
Step 2: (Shrinkage) Refine Jint by scaling down using A(a). defined as:
X(a) inf { { 2 0: En, (X (x) (x)) S oz
(3.3)
The final prediction interval is:
Pl (I) [[00 mo(x) V
+ v Xa)
(3.4)
In Step 1, we relax (3.1) by effectively setting a = 0. This relaxation aids in determining the optimal shape while also converting (3.1) into a convex optimization problem (equation (3.2)) as long as F is a convex collection of functions. Furthermore, in (3.2), we only consider those source observations for which wn(x) O, as otherwise, the samples are not informative for the target domain. In practice, Wo is typically unknown; one may use the source and target domain covariates to estimate Wg- Various techniques are available for estimating the density ratio (e.g.. [USS* + 16, CMSE22, Qin98, GSH [08 and references therein). However, any such estimator (0x) can be non-zero for where wo (2) = 0 due to estimation error. Consequently, v may not be efficient in selecting informative source samples. To mitigate this issue, we propose below modification of (3.2), utilizing hinge funetion hs(t) == max (0, (t/5) + 1}:
min JEF
[x,r
(3.5)
subject to En,s(((X)h, ((Y ((Y-mol 10mmmmX))+ - f(x)) Se.
with 5 and e should be chosen based on sample size ni and the estimation accuracy of ú. When v = we (i.e., the density ratio is known), then by choosing = 0 and 5 - 0. (3.5) recovers (3.2). As hi convex, the optimization problem (3.5) is still a convex optimization problem. We summarize our algorithm in Algorithm 1.

---

Algorithm Prediction intervals with bounded density ratio
1: Input: mo (or rit if unknown), density ratio estimator v, function class F, sample Ds {(Xg,,, Ys f== and Dr {Xr,ali " parameters 6, coverage level a.
2: Obtain funt by solving (3.5).
3: Obtain the shrink level X(a) by solving (3.3) with wp replaced by ú.
4: Output: (1) defined in (3.4).
Theoretical results We next present theoretical guarantees of the prediction interval obtained via Algorithm 1. For technical convenience, we resort to data-splitting: we divide the source data into two equal parts (Ds,1 and 2s,2), use Ds,1 and Dr to solve (3.5), and Ds,2 to obtain the shrink level {(a). Without loss of generality, we assume mo = (otherwise, we set Y < Y mo(X)). A careful inspection of Step 1 reveals that fiit aims to approximate a function f defined as follows:
arg B subject to y f() almost surely on target domain.
(3.6)
In other words, fini estimates f* that has minimal width among all functions covering the response variable. This is motivated by the philosophy that the right shape leads to a smaller width. The following theorem provides finite sample concentration bound on the approximation error of fint: Theorem 3.2. Suppose Y - ff*(X) < B on the source domain and has density bounded by L. Also assume || S By for all f E F. Then for
c 2 L2 V
B+5
(Es [xx(X) wo (W + W'W (1
(3.7)
5
we have with probability at least 1 - 3e=
Er Er[fmi(X)] Er[f(()] + 2R (F f+) +2BB 2nt
where W ||00||o0.
The bound in the above theorem depends on the Rademacher complexity of F (the smaller, the better), the estimation error of Wo, and an interplay between the choice of (c. 8). The lower bound on e in (3.7) depends on both 8 and 1/5. Although it is not immediate from the above theorem why we need to choose e to as small as possible, will be apparent our subsequent analysis; indeed e is large in (3.5), then fint = 0 will be a solution of (3.5). Consequently, the shape will not be captured. Therefore, one should first choose ō (say 8*), that minimizes the lower bound (3.7), and then set e = e* equal to the value of the right-hand side (3.7) with = 8*, which ensures that c* is optimally defined to capture the shape accurately. Once the shape is identified, properly in Step 2 to attain the desired coverage and reduce the width. Although ideally }(a) < 1, it is not immediately guaranteed as we use separate data (D5,2) for shrinking. The following lemma for any fixed
shrink it
long the sample size is large enough. Recall that the data were split into exactly half with size n = [Dsl-
shows that \(a)=
Lemma 3.3. Under the aforementioned choice of (c",8)), we have with high probability:
1
Σ
w(X,) (x)= Sa,
n2/2 VEDs.2
for all large ns- provided that u is a consistent estimator of wy Hence, A(a) 1
Our final theorem for this section provides coverage guarantee for the prediction interval given by Algorithm 1.
Theorem 3.4. For the prediction interval obtained in (3.4), with probability greater than 2e-
Pr y > (a)fm((X) 1 Ds UDr) a < Es (X) +(X)]++ + W') 2ng
+
C
ny
for some constant C > 0 and W' ||00|/oo
5

---

Theorem 3.4 validates the coverage of the prediction interval derived through Algorithm 1, achieving the desired coverage level as the estimate of wh improves and sample size expands. Theorems 3.2 and 3.4 collectively demonstrate the efficacy of our method in maintaining validity and accurately capturing the optimal shape of the prediction band, which in turn leads to small interval widths.
Remark 3.5. our optimization problem, we' we've substituted the indicator loss with the hinge loss function to ensure convexity. However, if's worth noting that if we know the subset of Xs where wo(ar) > beforehand, we could directly optimize (3.2). This approach would be easy to implement and wouldn't 't involve tuning parameters (ò, e). A special case is when wo wo(z) for all E Xs (as true in our experiment), which simplifies the condition in (3.2) to f(Xi) 2 (Y:=)((X))) for all 1 < i S ng. However, if this information is unavailable, one can still employ (3.2) by enforcing the constraint on all source observations. While this approach might result in wider prediction intervals, it is easy to implement and doesn't require tuning parameters.
Domain shift and transport map
Setup and methodology In the previous section, we assume uniform bound on the density ratio. However, this may not be the case in reality; it is possible that there exists E (rsp(pp))ssppp(3)) which immediately implies that wo (x) oc. In image recognition problems, the source data are images taken during the day at some place, and the target data are images taken at night, then this directly results in an unbounded density ratio (due to the change in the background color). Yet a transport map could effectively model this shift by adapting features from the source to correspond with those of the target, maintaining the underlying patterns or object recognition capabilities across both domains. To perform transfer learning in this setup, we model the domain shift via a measure transport map To that preserves the conditional distribution, as elaborated the following assump- tion:
Assumption 4.1. There exists a measure transport map To : Nr - Xss Le., To(XT) - Xs, such that: PY X = A Ps(Y X To(x)), Vz E Xr..
This assumption allows the extrapolation of source domain information to the target domain via To. enabling the construction of prediction intervals at æ E Xr by leveraging the analogous intervals at Tz( X'.. Inspired by this observation, we present sentially consists of two key steps: i) constructing a prediction interval in the source domain and ii) transporting this interval to the target domain using the estimated transport map To. If To (or its estimate) is not given, it must be estimated from the source and the target covariates. Various meth- ods are available in the literature (e.g., [DNWP22, SDF+ 17, MTOL220, DGS21J). ]), and practitioners can pick a method at their convenience. Notably, the processes described in equations (4.1) and (4.2) follow the methodology (i.e., (3.2) and (3.3)) from Section 3 for scenarios without shift (i.e., wp = 1), adding a slight Ò to ensure coverage even when F is complex. In Algorithm 2, we assume
methodology in Algorithm 2 that
Algorithm 2 Transport map
1: Input: conditional mean function mg on the source domain, transport map estimator To. func- tion class F. sample Ds {(Xs,,, Ys and Dy. (Xr,) Je. parameter 6, coverage level 1 a.
2: Obtain funi by solving:
minser 1 nn EE s.t. f(Xs,) 2 (Ys,: mo mo((xs))) [ns].
(4.1)
3: Obtain the shrink level
1 i(a) == inf {X 0: iss
Ens 1l (x..-mm S sa)
(4.2)
4: Output: (1) = mo To(x)= + A(aa) (finte o fo(x)) + (6)
the conditional mean function mo the source domain is known. In cases where the conditional

---

mean function mo on the source domain is unknown, it can be estimated using standard regression methods from labeled source data, after which mi is replaced by this estimate, m.
Remark 4.2 (Model aggregation). Suppose we have K different methods (fi,..., JK) for con- structing prediction intervals in the source domain. In the context of model aggregation, (4.1) then reduces to:
K
minay OO
11
j=1
subject
Oj I (Xs,)) 2 (Ys; - mo(X [ms]
j=1 2 0, visjsk.
In other words, the function class F is a linear combination of the candidate methods. The problem then simplified to a linear program problem, which can implemented efficiently using standard solvers.
Theoretical results We now present theoretical guarantees of our methodology to ensure that our method delivers what it promises: prediction interval with adequate coverage and small width. For technical simplicity, we split data here: divide the labeled source observation with two equal parts (with n2 observations each), namely Ds.1 and Ds,2. We use Ds,1 to solve (4.1) and obtain the initial estimator fini.. and Ds.2 to solve (4.2), i.c. obtaining the shrinkage factor X(a). Henceforth, without loss of generality, we assume my = and present the theoretical guarantees of our estimator. We start with an analog of Theorem 3.2, which ensures that with high probability finit To approximates the function that has minimal width among all the functions in F. composed with To that covers the labels on the target almost surely:
Theorem 4.3. Assume the function class is By-bounded and Lipsschh,, Define
A = min To(X)) F,Y² < fo To(X) a.s. on target domain)
Then we have with probability 21 1-
Erlsii o fo(X)) S + 4Rng I,
Ey ELTEE [(o(() - Tx(X) + 4Br 2ns
The upper bound on the population width of finit o (I( consists of four terms: the first term is the minimal possible width that can be achieved using the functions from F, the second term involves the Rademacher complexity of F, the third term encodes the estimation error of To. and the last term is the deviation term that influences the probability. Hence, the margin between the width of the predicted interval and the minimum achievable width is small, with the convergence rate relying on the precision of estimating To and the complexity of F, as expected.
We next establish the coverage guarantec of our estimator of Algorithm 2, obtained upon suitable separate dataset Ds.2-
truncation of fiit. As mentioned, the shrinkage operation performed Therefore, it is not immediate whether the shrinkage factor X(a) is smaller than 1, i.c. whether we are indeed shrinking the confidence interval (Å(a) > 1 is undesirable, as it will widen finit. increasing the width of the prediction band). The following lemma shows that with high probability, i(a)= <
Lemma 4.4. With probability greater than or equal e, we have:
(a- Pag opng
P(((a) 1 Ds,1, DT) S
1
where
Png = Ps (x² (X) + Ds.1,
(8)+6 Dr 5 ( BelyY ns + Ros Rns(F))
+
7

---

Here Png is the conditional probability of a test observation Y falling outside I- (ait(() + 6, / (X) + 6). which small as evident from the above lemma. In par- ticular, for model aggregation, if J is the linear combination of K functions, then Pnş is of the order v VK/ng. Hence, the final prediction interval is guaranteed to be a compressed form of finit with overwhelmingly high probability. We present our last theorem of this section, confirming that the prediction interval derived from Algorithm achieves the intended coverage level with a high probability:
Theorem 4.5. Under the same setup of Theorem 4.3, along with the assumption that fs(y x) is
uniformly bounded by G, we have with probability greater than 1 cng -1-0 that
Pr y 2 =(a) (fmit o To(X) + +6) Ds (Ds -
s CV log ns
GLFF EE [[66()] -To6x))To(x)]]
ng
As for Theorem 4.3, the bound obtained in Theorem 4.5 also depends two crucial terms: Rademacher complexity of F and estimation error of To. Therefore, the key takeaway of our the- oretical analysis is that the prediction interval obtained from Algorithm 2 asymptotically achieves nominal coverage guarantee and minimal width. Furthermore, the approximation error intrinsically depends the Rademacher complexity of the underlying function class and the precision in esti- mating To-
Remark 4.6 (Measure preserving transformation). In our approach, To is employed to maintain measure transformation, although it may not necessarily be an optimal transport map. Yet, estimat- ing To can be challenging in many practical scenarios. In such cases, simpler transformations like linear or quadratic adjustments are often utilized to align the first few moments of the distributions. Various methods provide such simple solutions, including, but not limited to, CORAL [SFS17] and ADDA [THSDI7].
5
Application
In this section, we illustrate the effectiveness of our method by applying it to five different datasets: i) airfoil dataset [DG19], ii) real estate data [Yeh] 8). iii) energy efficiency data [TX12b], iv) appliance energy prediction data [Can17], and v) ET Dataset (ETT-small) [ZZP+21]. The first four datasets freely available in the UCI repository, and the last dataset can be found in this GitHub link. Here, we illustrate the procedure using the airfoil dataset, and the details of our experiments using the other datasets can be found in Appendix C. The airfoil dataset includes 1503 observations, fea- turing a response variable Y (scaled sound pressure level) and a five-dimensional covariate X (log of frequency, angle of attack, chord length, free-stream velocity, log of suction side displacement thickness). We assess and compare the performance of our prediction intervals in terms of cover- age and width with those generated by the weighted split conformal prediction method described in (TFBCR19) 19). We use the same data-generating process described in (TFBCR19) to facilitate a direct comparison. We experiments 200 times; each time, randomly partition the data into two parts D Purain and Drest, where Derain contains 75% of the data, and Diest contains 25% of the data. Following [TFBCRI9], we shift the distribution of the covariates of Drest by weighted sampling with replacement, where the weights are proportional to
w(x) exp(x 3), where B-
These reweighted observations in Drest, which we call Dahift act as observations from the target domain. Clearly, by our data generation mechanism wp(r) = fr((x) fs(() = e exp(x: (3), where e is the normalizing constant. The source and target domains share the same support under this configuration. As our methodology is developed for unsupervised domain adaptation, we do not use the label information of Dahift to develop the target domain's 's prediction interval.
Density ratio estimation We use the probabilistic classification technique to estimate the density based on the source and the target covariates. Let XI,... Xn, be the covariates in dataset D and tng be the covariates in dataset Dainn The density ratio estimation proceeds two steps: (1) logistic regression is applied to the feature-class pairs {(,,, C where C 0 for
X ..... X
8

---

i i...... π1 and C = for i n1 + + R2 + n2, yielding an estimate of P(C x), denoted as p(z); (2) the density ratio estimator then defined as (xxx m2 1-)(2) explanations are provided in Appendix B.
1.....,
p(x)
Further
Implementation of our method and results As the mean function ma = Y X (which is the same the source and the target domain) is unknown, we first estimate it via linear regression, which henceforth will be denoted by v(xx). To construct a prediction interval, we con- sider the model aggregation approach, i.e., the function class F is defined the linear combination of the following six estimates:
(1) Estimator 1(fi:: A neural network based estimator with depth= width=10 that estimates the 0.85 quantile function of (Y r X x.
(2) Estimator 2(f2): A fully connected feed forward neural network with depth=2 and width=50 that estimates the 0.95 quantile function of (Y m X=
(3) Estimator 3(s): A quantile regression forest estimating the 0.9 quantile function of (Y
(x))) X æ.
(4) Estimator 4(fx): A gradient boosting model estimating the 0.9 quantile function of (Y
(X))² X = œ.
(5) Estimator 5(fs): An estimate of B[(Y (x)) X = xx] using random forest.
(6) Estimator 6(f6): The constant function 1.
Here, the quantile estimators are obtained by minimizing the corresponding check loss. The im- plementation of our method is summarized as follows: (1) We divide the training data Dusio into two halves D1 U DD. We utilize dataset D1 to derive mean estimator and six aforementioned esti- mates. We also employ the covariates from D1 and Dahift to compute a density ratio estimator. (2) We further split D2 into two equal parts D2,1 and D2.2. D2,1, along with covariates from Dahaft, used to find the optimal aggregation of the six estimates to capture the shape, i.e., for obtaining finit- The second part 22,2 is used to shrink the interval to achieve a 0.95 coverage, i.e. to estimate )(a). (3) We evaluate the effectiveness of our approach in terms of the coverage and average bandwidth on the Daun dataset.
In Figure 1, we present the histograms of the coverage and the average bandwidth of our method, and more general version of weighted conformal prediction in [TFBCRR9] over 200 experiments (see Appendix B for details), which show that our method consistently yields a shorter prediction interval than the weighted conformal prediction while maintaining coverage. Over 200 experiments, the average coverage achieved by our method was 0.964029 (SD= = 0.04), while the weighted conformal prediction method achieved average coverage of 0.9535 (SD 0.036). Additionally, the average width of the prediction intervals for our method was 13.654 (SD = 2.22), compared to 20.53 (SD 4.13) for the weighted conformal prediction. Regarding the performance of intervals over 95% coverage, our method achieved this in 72.5% of cases with an average width of 14.35 (SD = 2.22). In contrast, the weighted conformal prediction method did so in 57% of cases with an average width of 21.4 (SD= = 4.39). Boxplots are presented in Appendix B for further comparison.
5.1 Robustness of our method
One of the strengths of our approach lies in its resilience to the misspecification of certain com- ponents. The core idea behind our method is to combine multiple predictors to create a prediction interval that ensures sufficient coverage while maintaining narrow average width. These individ- ual prediction intervals may be based on estimators of conditional quantiles, means, variances, and other metries. If some of the component prediction intervals exhibit poor performance, whether due to inadequate coverage or excessive width, our method typically assigns them lower weights, making it robust to their deficiencies. In contrast, other conformal methods that heavily depend single component tend to underperform, particularly with respect to average width. To illustrate this phenomenon, evaluate method under model misspecification using simple one-dimensional simulation setup.
X ~ unif ([-1,)), E~ unif([11, X 111
== V1 +22x.

---

(a) Coverage of our method
(b) Bandwidth of our method
CoARge--GE nabod
(c) Coverage of weighted conformal
(d) Bandwidth of weighted conformal
Le mndiina sonerni
Figure Experiments on Airfoil data using Algorithm
Max depth
Avg. width Our Method
Avg. width WVAC
3
2.07(0.975)
3.08(0.9712)
5
2.07(0.95)
3.28(0.9664)
7
2.068(0.94)
3.33(0.97)
15
2.08(0.97)
5.00(0.97)
Table 1: Robustness of our method and WVAC. The number inside the parenthesis is the median of coverage over these Monte Carlo iterations.
As mentioned in the previous subsection, our method aggregates six predictor intervals, including an estimator for the conditional variance function (Estimator 5). The weighted variance-adjusted conformal prediction interval (WVAC) relies on accurately estimating this conditional variance. We estimate the conditional variance using random forest with varying depths ((3, 5, 7, 15}). For 2500 samples, keeping 75% as source data and resampling
simulation purpose, we generate remaining 25% with weighted samples proportional w(xx) x (1 + exp(-2x)) increases, overfitting leads to poor out-of-sample variance predictions. Table summarizes our findings over 100 Monte Carlo iterations, showing that WVAC's average width increases with depth, while method's average width remains stable.
depth
6 Conclusion
This paper focuses on unsupervised domain shift problems, where we have labeled samples from the source domain and unlabeled samples from the target domain. We introduce methodologies for constructing prediction intervals on the target domain that are designed to ensure adequate coverage while minimizing width. Our analysis includes scenarios in which the source and target domains are related either through a bounded density ratio or a measure- preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. We further establish rigorous finite sample theoretical guarantees regarding the coverage and width of our prediction intervals. Finally, we demonstrate the practical effectiveness of our methodology through its application the airfoil dataset.
measure-preserving
10

---

References
[Can17] Luis Candanedo. Appliances Energy Prediction. UCI Machine Learning Repository,
2017. DOI: htpps///8o..rr/10024333/////
[CEN14] Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4B1), Rhodes, Greece, September 19-21, 2014. Proceedings 10, pages 231 Springer, 2014.
[CFHRI7] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. Advances in neural infor- mation processing systems, 30, 2017.
[CFT14] Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part 14, pages 274-289. Springer, 2014.
[CFTRI6] Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 39())1183333333 1865, 2016.
[CMSE22] Kristy Choi, Chenlin Meng. Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics, pages 2552-2573. PMLR, 2022.
[DG19] Dheeru Dua and Casey Graff. Uci machine learning repository. https: ics. uci, edu, 2019.
[DGS21] Nabarun Deh, Promit Ghosal, and Bodhisattva Sen. Rates of estimation of optimal transport maps using plug-in estimators via barycentric projections. Advances in Neu- ral Information Processing Systems, 34:29736-29753, 2021.
[DLA*21]] Matthias De Lange. Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-338 3385, 2021.
[DNWP22] Vincent Divol, Jonathan Niles-Weed and Aram-Alexandre Pooladian. Optimal trans- port map estimation in general function spaces. arXiv preprint arXiv:2212.03722, 2022.
ERS +22 Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von Kügelgen, Hamed Hassani, George J Pappas, and Bernhard Schölkopf. Probable domain generalization via quantile risk minimization. Advances Neural Information Processing Systems, 35: 35:17340-17358, 2022.
[FGM23] Jianqing Fan, Jiawei Ge, and Debarghya Mukherjee. Utopia: Universally trainable optimal prediction intervals uggregation. arXiv preprint arXiv:2306. 16549, 2023.
[FVRA21] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pages 877-894, 2021.
[GC21] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660-1672, 2021.
[GSH+08] 08] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. 2008
[KKCC14] Mohammad Anwar Hosen, Abbas Khosravi, Saeid Nahavandi, and Douglas Creighton. Improving the quality of prediction intervals through optimal aggrega- tion. IEEE Transactions on Industrial Electronics, 62(7)) 4429, 2014.
11

---

[HL23] Xiaoyu Hu and Jing Lei. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association, pages 1-19, 2023.
[Hu194] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Trans- actions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
[LBBH98] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
[LC21] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and indi- vidual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(5):911-933 2021.
[LGR* + 18] Jing Lei, Max Gelll, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution free predictive inference for regression. Journal of the American Statis- tical Association, 113(523):1094-1111 2018.
[Mau16] Andreas Maurer. A vector-contraction inequality for rademacher complexities, In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pages 3-17, Springer, 2016.
[MLJ+22] 22] Zheda Mai, Ruiwen Li. Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott San- Online continual learning in image classification: An empirical survey. Neuro- computing, 469:28-51, 2022.
[MNW16] Katarzyna Naciejowska, Jakub Nowotarski, and Rafal Weron. Probabilistic forecast- ing of electricity spot prices using factor quantile regression averaging. International Journal of Forecasting, 32(3):957-965 2016.
[MTOL20] Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal trans- port mapping via input convex neural networks. In International Conference Ma- chine Learning, pages 6672-6681. PMLR, 2020.
[NW15] Jakub Nowotarski and Rafat Weron. Computing electricity spot price prediction in- tervals using quantile regression and forecast averaging. Computational Statistics, 30:791-803, 2015.
[NWC+11) Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011.
[PY09] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions knowledge and data engineering, 22(10:1345--135) 2009.
[Qin98] Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika, 85(3):619-630, 1998.
[RHS17] levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, Septem- ber 18-22 22-22, 2017, Proceedings, Part II 10, pages 737-753. Springer, 2017.
[RPC19] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile re- gression. Advances in neural information processing systems, 32, 2019.
[SDF+ 17] Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint 17X1v:/711.022888 2017.
[SFS17] Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation computer vision applications, pages 153- 171, 2017.
12

---

[TFBCRI9] Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances neural information process- ing systems, 32, 2019.
[THSDI7] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discrimi- native domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
[TX12a] Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and buildings, 49:560-567, 2012.
[TX12b] Athanasios Tsanas and Angeliki Xifara. Energy Efficiency. UCI Machine Learning Repository, 2012. DOI:
[USS 16]
Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Mat- suo. Generative adversarial nets from density ratio estimation perspective. arXiv preprint arXiv:1610.029000 02920, 2016.
[VECP17] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep hashing network for unsupervised domain adaptation. Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5018- 5027, 2017.
[Vov15] Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74:9-28, 2015.
[WKW16] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. survey of transfer learn- ing. Journal of Big data, 3:1-40, 2016.
[WLL+22] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing to unseen domains: A survey domain generalization. IEEE transactions knowledge and data engineering, 35(8):8052-800 8072, 2022.
[XLW+20] Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4394- 4403, 2020.
[Yeh18] I-Cheng Yeh. Real Estate Valuation. UCI Machine Learning Repository, 2018. DOI: http:///oo.orr//1.244421C
[YH18] I-Cheng Yeh and Tzu-Kuan Hsu, Building real estate valuation models with compar- ative approach through case-based reasoning. Applied Soft Computing, 65:260-271, 2018.
[YKT22] Yachong Yang, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Dou- bly robust calibration prediction sets under covariate shift. arXiv preprint arXiv:2033.01761, 2022.
[ZLQ++22] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain gener- alization: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4396-4415, 2022.
[ZY21] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):586-5609, 2021.
[ZZP*21] H Zhou, S Zhang, J Peng, S Zhang, Li, H Xiong, and W Zhang Informer. Be- yond efficient transformer for long sequence time-series forecasting. 2021, 35. DOI: https://doi. 1609/aaa. v35i12, 17325:1 1012::111066111 101-11115,, 2021.
13

---

A Proofs
A.1 Proof of Theorem 3.2
First, we show that for our choice of (e, 6), as depicted in Theorem 3.2, f is a feasible solution of equation (3.5). Consider wo instead of wh By definition of f*,
Pr S (X)) =1 Es [xx(X)]x=ss((x)
[o(x]1ly>)=(x)> = 0 a.s. on the source domain.
This implies:
wo (X; (Y²- (x.))
/2
1EDs,
1
Wo (ox)) (X2 - (Xi)) (x)))(x))
n2//
iEDs,1
1
Σ
wo(X))hs -V2- (x,)) 1
ns/2
iEDs,
/2 iEDsa
where the first equality follows from the fact that wo (x) 0 a.s. the source do- main, the second equality follows from the fact that hs(()) 0 for all I, and the last in- equality follows from the fact that hs(Y," - (X,)) S 1 when Y? fX, I 0. Since W. by Hoeffding's 's inequality, we have with probability at least
wo (oo(X)1 (x) (5Y 1ee
wo(X.)hs (Y? - (x,))) s Es (1x( + W
n2// DED
ns
t
= =Pr (f - -8 < fx)) + W
ns
S L8 W
where upper bound on the density of y2 f* (X). Call this event 22 that the above bound holds. At this event we have:
Σ
(X,)ha (Y? (xx)))
ns/2
kEDs.
1
wo Wo(X.)la (Y22 - (Xxi) +
Σ
(w(X,) mo(X:)) (Y? - (xx)))
ns/2
ns/2 iEDs.1
iEDs,
ns/2
1
B 8+8
2
10 (X,) - wo (X.)l,
nn
11
where the last inequality follows from the fact that (1) < (B+6)/5 S B. Finally, to bound the last summand, we again apply Hoeffding's inequality. As | S W we have with probability greater than or equal to 1 - e:
/2
(x,))) - wo woo (x,)) < Es [Ix(X) mxo( + (W- - W
L ns
n2// 2
If we denote the event f2 where the above inequality holds, then on the event f2 L2, we have:
Σ
hs (x? (X,))
I
B 8
< L5 + W
Es [|((() wx)x + (W + W'),
cc
ng
8
ns
14

---

Furthermore,
P(Q, 092) 2 P(L) + ((02) 12 2t
Therefore, we conclude that with probability 2 1 - 2 is feasible solution.
We now proof Theorem 2.2 the event ft n 92,, when f is feasible solution. Then have,
P,, P P,, (f*(((()))))) this event, by the optimality of finit in equation (3.5). Then have:
Er(siiit (X)) = Pny (fii + (Pr Fny) (f)it(X)
S Pnr + (Pr- - Pn,) (X))
Ex]s + (PnT - Prr) (f* - (X)) ET(s (X)] + sup (IP PP) (X)
feF
Finally f f is upper bounded by F' = By + |s ccc (as is uniformly upper bounded by F). Therefore, by Mcdiarmid's inequality, we have with probability e=
sup (Pny - (f - - - Er sup SEF
(T
(r)) + F 2nT
feF
Call this event U2.. Furthermore, by standard symmetrization:
Er
sup LEF
(P (F s 2R (F-
where Rnr (F - f*) is the Rademacher complexity of F f" Therefore, on 2 we have:
t
Er Er(fmi(X)] < Er(f(()) 2m (F f*) FF F"
2nt
and P(n (4) 2 - 31 This completes the proof.
A.2 Proof of Lemma 3.3
We prove the lemma into two steps; first we show that fini satisfies IP fmit(X) S with high probability for some small Next we argue that, on Ds.2, we have 7 with high probability for some small 7. Then as long as S a, we conclude the proof of the lemma.
(2/ns) LiEDsa (X)) > (x.)(()))
Step 1: Note that, by feasibility, fiii satisfies:
Σ
w(X))o((? - -imie(x)) S E.
n2//
VEDs.
This implies:
Er ha y (Y²- (r²-funtt fims((x)))
= =Es wol (xo(X) (xn-mii()))
Σ
wo wo(X))ho(Y3 {mie((ii) + (Ps--PP PPns/2) wo wo(X)ha(Y2 (x))
ns/2
iEDs.1
Σ
10 ((X))h&(Y? - Pmnn(())) + ns/2 iEDs.1
Σ
(wo(X,) - w(x,)))s(Y?² -iine (Xi))
ns/2
iEDs.1
+ (P's P P2s/2) wo(X)hs(2 (X))
B
<
|10 wo |l (P).s) + sup 5
(Ps Pnss2) wo (o(X))((2222)) -(x))
fEF
15

---

Now, as h2 - f(X)) - (B + 8)/6 and wo - W, we have by Mediarmid's 's inequality, with probability
sup feF
(Ps P P2g/2) wo -fx))
B +
Es
sup fEF
Pn P2s/2) wo wo(X))s F(x)) (P'
+ W
71.
2nss/ 1t0s:2/x((ull)) (wyhs o f) + W B
ng
Meanwhile, as in the proof of Theorem 3.2, with probability 2 - --
j (oller(P.).s)) I Es [|x(X) w(Xil (W + 1n) ns
Choosing t = 10 log ns we obtain that with probability 2 - 2ng"
Er h1s 17 - fin((X))))
B + 8
10 log ns Es [x)((() - xx)( + (W w")
I +
ns
B + 6
10 10 ns
3Rng/22FFupll W
5
ns
B+ S cc+ 5
10 log ns
Es [1((((X) - wol (x) [x] (2W + W')
22 f).
ns
We next bound the Rademacher complexity of /2g (wohs (wohs with Sns/2 Sn,/.. i.i.d.
f). By symmetrization, we have
Rademacher(1122)
Rns Fms/2.F((nll o 0) = 2E.
sup
Σ Gonn(X))a(Y?
fEF
ns//
2Es
sup feF
Σ
Soo (woo Y? -(x())
[0(x, v) whs(u))
ns/2
We first show that R22 wo(ar) which lies within | W. W). The second argument of á is y2 - -((X) (on the source domain), which is bounded by B. Therefore, hss hy2 -(()) is bounded above by (B + 6)/8. The derivative
Lipschitz function on its domain. The first argument of
of he is 0 for < -6 and 8 for -8. Hence, we have the following:
B 5)2
W2
B+W++
Vede, M (1) ab((0)
82
82
We next apply vector valued Ledoux Talagrand contraction inequality on the function & (equation (1) of 6]), obtain the following bound on the Rademacher complexity:
2E.
sup fEF
Σ GO (wo(Xx)) Y? - f(x.))
ng/2
/2
W +
< 2/2
Es
sup SEF
n
+ W +
22/
Es
ns/22 Σ Ci
+ Es
Σ Si
Rns/e(F)
ns/2 tEDs,
B + W + 8
[|woll
Es YY4
5 2
+ R2)/2(FF
/ng//2 /
ns//
16

---

Using this, we obtain the following:
Ex h1s (Y² - Sunit fx))
B +
5 log (n
Set
Es (x() _o(x)]l + (2W + Wn)
ns/
B + W + 5
))
|woll [100ll + (E4
+ 4v2
/ (F))
8
Vns
W + 8
5 blg (ns/2) n /s/
S 4V2
E [1o((ss) - m(xs)ll + (2W + VVy
| | + B
+ R2)
Vns/2
B + + 8)
5 blog (ns/22 ns/2
W + V [Y4)
< <c+
w(Xs) E - (2W + W
+ Rng (2
8
vns/2
Choosing
5lg (ns/2)
B 8
5 log (ns/2) ng. n2//2 /2
E= L8 W
V
Es [lx(X) (x)il + (W + +W')
ns//
we obtain
ET h6 (y²- fintfit()))
B + W +
5 ns
$ L8 +
(Es [|(((X) wo()) (W + w
+ Rng/2(FF)
8
ns
llg ng
$
L(B + W)
L(B+W)
(Es [xx wx((X]l + (W W
+ Rng/2(F)
ns
5 log ns
+ (Es [|x(X) wo wx)))]l (W + w
Rng/2
ns
(by choosing d to balance the terms)
A
Call the above event 22 This completes the proof of Step 1.
Step 2: Coming back to Ds,2, we have:
Σ ((XXs,)ly?>smmel I Σ [xx(
Σ
wo
/ 2
ns/2 iEDs,2
ns/2 iEDs,
tEDs2
Furthermore, by Hoeffding's inequality, have with probability 1-
ng/ /2 VED2
Σ
wo (X, (mn(())) S Es (X) y2 Sm)
+ +W
V
7g
S Eg wo wo(XXho (r2 Prait (x))
W
ng
Er (hs (Y² (x2- (--mmm(())) W
V
ns
Mcanwhile, with probability 2 -
Σ
(x(X)) - wo((x)) S Es [X) wo(xx]] + (W + W')
ng/2 (EDsa
ns
17

---

Therefore, with t 10 10 ris. we have with probability 2 1 - 2mg"" 10:
1 Σ
10 logns
y?
S Es [|xo(X) wo(X)]] + (W + W
n2//2 iEDa.2
ns
+ Er his fiic(((())) + W
10 nts
Call this event flg. Therefore, on 121 n l52 have:
10 10g
Σ ii I y2>mmux Es [Xx mx( + (2W + W
n tEDs2
ng
This completes the proof of Step 2. For any fixed a > O, we have I a as long as ng is large enough and Eg (X) - (X)))] []]]= is small enough, and as consequence X(a) S This completes the proof.
A.3 Proof of Theorem 3.4
Recall that we construct the prediction intervals using data splitting, from the first part of the data (namely D1), we estimate finit and use the second part of the data (namely D2) to estimate X(x).
Conditional on D,, define function class G = G(f)) as:
g = [9x(2, = wo(x)1 ga (x)20 A2
As G only depends on scalar parameter À (as wi and fini are fixed conditionally on Ds,1, Dr), is VC class of function with VC-dim S 2.
Pr (r² 2 ((a))
= Es un (X) y= Y=
Σ ns/2 iEDs.2
wo (X:) -M + (Ps -P (x
Σ
d ((X)) xx Maa ns/2 sEDg,
Σ
- ((x)) ((a))
ns/2 iEDs,
(Pss - Pnn/222 (na)2))o()(1, (() (x)20
(A.1)
Now, by the definition of \((x) (see Step 2), we have:
Σ
a.
/2
ns/2 DEDs2
We use a similar technique to control the second summand in the proof of Theorem 3.2. By using the fact that the indicator function is less than one, we have:
Σ (wo(X)) - ú ((X)) Y2 M2
V (x(X)) wo(X,))
/2 SED
ns/2 tEDs,
Applying Hoeffding's inequality (with the fact that ||00| W' and |wal o0 S W), we have with probability greater than or equal to 1 et:
1
(x,))) wo(x)) Xx)) < Es [x(X) w (x)I (W + W')
t
n
ns
VEDS2
To control the third summand (A.1), note that, conditional on Ds.1 and Dr (i.e., assuming finit fixed), and using the fact that |/o]|oo < ||wolloo < W for all g G. we have by Mediarmid's
18

---

inequality with probability greater than or equal to 1 - e
t
sup gEG
(P' - Pn /2g Es LgEg
sup
(Ps -P (= Ds,1, Dj + W
ng
t
S nR /2 (9 Ds.. Dr) + W
ng
Now conditional on Ds,1, Dr. a VC class function with VC dimension S Therefore,
C
Rms/2 (g Ds,,,, s V
ns
for some constant > 0. Thus, we have
C W
sup (F's Pnn/22 S ns
gEg
ns
Combining the bounds, we have, with probability 2 1-2
Pr y Ma)) a
V ?
+ V/
C
Es (X) wo + (2W + W')
n2/
ngg
This completes the proof.
A.4 Proof of Theorem 4.3
We start with the following decomposition:
Lnit [x(X)] Er(iint To(X)) Rrisi To(X) - fini To(X)]
= Finti (X) + Erifmin o Ta(X) finit [x())]
I Es (X)] + Ly Tx(X)]
where the second equation follows from the fact that when X ~ Pr. then (X) ~ Ps. and the last line follows from the fact F LF Lipschitz. A similar argument in the proof of Theorem 3.5 [FGM23) yields:
L
Ess fimt (X)) < A + 4Rng (F) 4 2ns
with probability 2 -t We then finish the proofs.
A.5 Proof of Lemma 4.4
By the definition of A(a), we have
{11/2 Σ 1 2 +8) > za}.
((a) 2 ns/2 iEDs.2
Now by an application of Chernoff bound for binomial distribution, we have:
(o (o-pog 2ng Gyng
Σ 2 +6) > a Ds.1, Dr <e
P
ns/2
iEDs
Hence, we have the following:
(o (o-pan) ) Pag Gyng
P(((a) Ds.1, Dr) S
19

---

We next establish the high probability bound on Pag. We define function (s(z) which is when
S -, 0 when and -x/8 when - I x S 0.
Pas Es
1 [x22
(s(fini -22] - Es
Σ Es(fini (X:) -Y?) + (Pus/2 PPs) ls ((finit()) y²)
n1s/ /2 ieDs,
$ sup fef
(Pus/2 Ps) (s(((X) - -Y²)
4
Es[Y
t
+ R 12
n1
ngg
where the first inequality used (a(x) 2 [x - -), second inequality uses the fact that sample aver- age of ls over D is by the definition of finit. third inequality uses Ledoux- Ledoux-Talagrand contraction inequality observing that is 1/8-Lipschitz. This completes the proof.
A.6 Proof of Theorem 4.5
Pr y 2 faai o To(X) +0)
Pr (Y² ((att To(X) + +5))
+ +6) - PP (r² ((nit T +6)) + +|P ( Fini o To(X)+
T + T2.
(A.2)
We start with analyzing the first term:
T = Pr y 2 (fait o To(X) + +6)
S xr J 1 fr(y/ XT = x)) pr(x) dyda
Jxm J 1 fsly Xs To(x)) (r(x) dydr
=Je
Jxs I ( fsly Xs 2) PT (z)) VT. (2) dydr
-Jee / (22 fs(y Xs 2) ps(2) dydr
= =Ps(Y²2 2
(X)+
Therefore, we need high probability upper bound on PPs ((a)((mi(x)= Ds UT)). Towards that end, we start with the following expansion:
Ps ( 2 (x) + Ds DsUDr)
1 Σ
(P'ns/2 (x22)to)(hm(()4t) + ns/2 iEDs22
Ps) 1 Ly=2)t))(hm(()+ (A.3)
Now, note that, by the definition of )(x), we have:
Q 1 1
< Σ 1122M01(ha((XH9)) aa.
ns/2 ns/2 iEDs.
To bound the second term in (A.3), we use:
(Png m2///2
Ps) (122)4))))(mmm))))))
sup X20
(TP (2ny/2 Ps)) 1 1y22M((n((X)4)) Z,
20

---

To bound the supremum we use standard techniques from the empirical process theory. Define collection of functions g =
( y
= 0 Note that, here we condition on Ds,1, so
treat fiii constant function. For notational simplicity, suppose
vr, = Es sup A20
(/Png/2-PPP - 1 (x22Mmm((X)+0) Ds.1 = Es
(Fns/2 Pss) ((X,Y)) Ds.1 Loeg
sup
As the functions in G are uniformly bounded by 1 (and consequently, (X Y) Σ 1), have by Talagrand's concentration inequality of the suprema of the empirical process:
1 4
4t
Zn 2 1 + /2t
Ds,1
See
(A.4)
ng
3ng
Therefore, we need an upper bound on V to obtain a high probability upper bound on Z, Towards that end, observe that 9 VC class with VC-dim less than or equal (as an indicator function of collection of functions with one parameter). Hence, we have, by symmetrization and Dudley's metric entropy bound:
C VTs
V S 2Eg
sup gEG
Σ
c Yi)
D2.1
12
iED52
Therefore, going back to (A.4), we have with probability
- ---
Ci
C2
41
Zn S
+
Vl
V
nS
ng
nzg
3/2
3ns
Hence, we have:
Ps (2 (ma)((mme()))) + 06
Ds DDUD - 2
t
ngs
with probability 2 et This completes the proof of Ti. To obtain bound on T2, note that:
T2
PT (r² Maa) ( o fo(X)+ Pr (Y² M(a)( To(X) 6))
Ier (Pr(r22 Ma) +6) XT x)
-Pr(YY2 S (Maa)( + 6) XT = xi) pr( dz
- -1/ (Ma))
Fy (Ma) + +)) pr(x) dr
I SG $Gf i(a) (unn (T - fint pr(x) dæ
Xx
< &GL
fx(
Here, the penultimate inequality the fact that the conditional distribution of Y? given Xy is Lipschitz (as the density of Y given X, bounded), and the last inequality uses the fact that finit is Lipschitz we have assumed all functions in F are Lipschitz.
B Details of the experiment
B.1 Density ratio estimation via probabilistic classification
Suppose we observe {XI,..., X,,,., } from a distribution P (with density p) and (Xnyt1,.... X from another distribution Q (with density ÿ) We are interested in es- timating wo(x) q(x)/p(z), where assume is absolutely continuous with respect to P
21

---

(otherwise, the density ratio can be unbounded with positive probability). Define, R1 + mane for < I n and C, for 11 + S i < и + n2. Consider the augmented dataset 'D = {(X,, Signn We can think that this dataset is generated from a mixture distribution pp(.X) + + p)g(2) where p P(C 1).
binary random variables {CI..... +n2) such that C;
For this mixture distribution, the posterior distribution of C given X is:
P(X = C - PC 1)
P(C = 1|x x²) = P(X C PXX -
C 0) 0)
pq(x)
mq(x) + (1 p)p(a)
(p/(1 p))
(p/(1 p)) (x: +1
This implies:
wo(x) p P(C=1|X=2)
1
Now, from the data, we can estimate A n2/(m1 +22 and P X x) by any classification technique (e.g., using logistic regression, boosting, random forest, deep neural networks etc). Let ((x) be one such classifier. Then we can estimate (x() by (m1/n))((x))(( ((x))).
B.2 General weighted conformal prediction
The weighted conformal prediction method, presented in [TFBCR19) 19), consists of two main steps:
1. Split the source data into parts; estimate the conditional mean function E[Y X = x]. say ((r) using the first part of the source data.
2. Use the second part of the source data and the target data construct weight wxX.) and the score function S(x, y) = yg j(x)] to construct the confidence interval.
In Section we have implemented generalized version of it, where we modify the score function as follows:
1. We estimate the conditional standard deviation function v vvr((Y X = x) along with the conditional mean function from the first part of the data. Call it ā
2. We use the modified score function s(z, v)
(/x))//(x).
The rest of the method is the same as [TFBCR19). This additional estimated conditional variance function allows more expressivity and flexibility to the conformal prediction band, as observed in Section 5.2 of [LGR+ 18], as this captures the local heterogeneity of the conditional distribution of Y given X.
B.3 Boxplots compare coverage and bandwidth
In this subsection, we present two boxplots to compare the variation in coverage and average width of the prediction bands between our method and the generalized weighted conformal prediction described in the previous subsection).
(a) Average Bandwidth
(b) Coverage
660
0.0
Megiel Cobrre
Met gloed Corfene
22

---

Real Estate Data
Outcome
Our Method
WVAC
WQC
Coverage (Median)
0.98
0.962
0.971
Coverage (IQR)
0.058
0.048
0.048
Bandwidth (Median)
36. 889
46.392
46.858
Bandwidth (IQR)
15.001
19.027
14.189
Bandwidth (Median for Coverage > 95%)
40.774
50.703
51.031
Table 2: Experimental results for the real estate data
The boxplots immediately show that our methods yield similar coverage (even with lesser vari- ability) with significantly lower average width than the generalized weighted conformal prediction method.
C Additional Experiments
In this section, we present the details of the additional experiments based four other datasets, as mentioned in Section 5. We compare our methodology against two other widely used confor- mal methods, namely Weighted Variance-adjusted Conformal (WVAC) [TFBCRR9), and Weighted Quantile-adjusted Conformal WWQC)-which a variant of WVAC, where the score function is changed to the conditional quantile [RPC19].
C.1 Experiment 1: Real estate data
The Real Estate Valuation dataset available at the UCI Machine Learning Repository contains data used to estimate the value of real estate properties. Collected from Taipei. Taiwan, this dataset is useful for predictive modeling in the real estate sector. It consists of 414 instances, with each entry representing different real estate transaction. This dataset was originally introduced in [TX12a), In this dataset, the goal to predict the house price per unit area based on the 6 other features, namely transaction date, ii) house age, iii) distance to the nearest MRT station, iv) number of convenience stores, v) latitude and vi) longitude. The construction of shifted data (with 8 = (-1, 0, -1, 0, 1, 1)) and implementation procedure results over 200 Monte Carlo iterations. It is evident from the table that our method produced small average width in comparison the other methods while maintaining the coverage guarantee.
the same
Section 5, Table 2 and Figure 3 presented the
C.2 Experiment 2: Energy efficiency data
The Energy Efficiency dataset available at the UCI Machine Learning Repository is designed to help predict the heating load and cooling load requirements of buildings. The dataset was originally introduced in [YH18). The dataset includes various building parameters such as: wall area, ii) surface area, iii) roof area, iv) orientation, etc. The goal is to predict the heating load based on 8 other covariates. The construction of shifted data (with 3 (-1, 1, -,,, 0, -)) and imple- mentation procedure are the same in Section 5. Our results over 200 Monte Carlo iterations are presented in Table 3 and Figure The table shows that our method produced a smaller bandwidth than WVAC. While WQC has smaller median bandwidth, it sacrifices coverage. The last row indicates that for experiments with coverage > 95%, WQC's median average width significantly larger than ours. Thus, whenever WQC provides adequate coverage, its bandwidth is much larger than ours.
C33 Experiment 3: Appliances Energy Prediction Dataset
Appliances Energy Prediction Dataset is freely available from the UCI repository. This dataset is a time series data with 28 covariates and one response variable. We used data from 2016-01-11 to 2016-02-15 (5000 samples) as our training set and data from 2016-05-13 to 2016-05-27 (2000
23

---

Energy efficiency data
Outcome
Our Method
WVAC
WQC
Coverage (Median)
0.995
0.969
0.973
Coverage (IQR)
0.047
0.036
0.05
Bandwidth (Median)
4.332
5.045
2.842
Bandwidth (IQR)
1.358
3.269
2.551
Bandwidth (Median for Coverage > 95%)
4.373
5.681
4.94
Table Experimental results for energy efficiency data
samples) as our testing set. Since the source and target data are from different time periods, this experiment involves non-synthetic real-world time shift. The results based our Algorithm 2 are presented below:
Appliances Energy Prediction Dataset
Outcome
Our Method
WVAC
WQC
Coverage
0.95
1.00
1.00
Bandwidth
461.69
6809.87
2032.12
Table 4: Experimental results for the Appliances Energy Prediction Dataset
C.4 Experiment 4: ETDataset
We applied our method to the ETDataset (ETT-small) from [ZZP+21], which contains hourly-level data from electricity transformers at different stations, including load and oil temperature measurements. Each data point consists of 8 features, including the date of the point, the predictive value "oil temperature", and 6 different types of external power load features. For our experiment, we used the data from one transformer during the period from July 1, 2016, to November 2, 2016, our source data, and data from the As our source data and the target data are from different locations, we have a geographical covariate shift; see [ZZP+21) 21] for more details, Our results are as follows:
time period from the other transformer as our target data.
ETDataset
Outcome
Our Method
WVAC
WQC
Coverage Bandwidth
0.976
0.982
0.842
41.525
57.9
54.981
Table 5: Experimental results for the ETttattset
C.5 Experiment 5: Airfoil data
In Section 5, we have implemented our Algorithm on the Airfoil data [TFBCR19] to showcase the efficacy of our method. Here, additionally, implement Algorithm on the same dataset to evaluate the performance of our second method based on optimal transport-based domain alignment. Here, the data shifting procedure (to create data from the target domain with shifted distribution) is different from the rest of the experiments; we use the linear transformation + Az + b with = diag(1.5 (1.5, 1.2, 1.6, 2, 1.8) and b = (1, O, O, 1, 0) to generate covariates on the target domain. As before, we split the data into two parts; we keep 75% of the data as it is, and shift the rest 25% of the data. The results are given Table and Figure 5. The second column of Table 6, namely Our Method (Without OT) is the aggregation method proposed in this paper, without domain alignment

---

Airfoil data
Outcome
Our Method
Our Method
WWC
WQC
(without OT)
Coverage (Median)
0.928
0.749
0.984
0.952
Coverage (IQR)
0.035
0.22
0.024
0.077
Bandwidth (Median)
15.075
18,512
36.298
32.143
Bandwidth (IQR)
1.638
3.089
10.619
8,364
Bandwidth (Median for
16.429
25.268
37.783
36,433
Coverage 95%)
Table 6: Experimental results for the airfoil data using optimal transport
through optimal transport, i.e., we use source data to construct the prediction interval and use the same prediction interval on the target domain. As evident from Table 6, our method outperforms all other methods in terms of the average width of the prediction interval while maintaining good coverage guarantee.
25

---

(a) Our-Width Width
Our Coverage
(a) Our- Width Our-Width
(b) Our Cov-Creea
(c) WQC-Width
(d) WQC-Coverage
(c) WQC-Width
(d) WQC-Coverage
(e) WVAC-Widi
Cove-Coverage
(c) WVAC- Width
(f) WVAC-Coverage
Figure 3: Experiments real estate data
Figure 4; Experiments on Energy efficiency data
(a) Our-Width
(b) Our-Coverage
(c) Our Oidh-idth
(e) WQC- Width
(1) WQC-Coverage
(g) WVAC-Width
(h) WVAC-Coverage
Figure Experiments on Airfoil data using optimal transport
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 3, 4 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: We have proposed method for aggregating prediction intervals along with theoretical guarantees and experiments in this paper. Our theory is valid under certain assumptions clearly mentioned in the main paper. One limitation of the theory is that do not know the performance of our method if the assumptions are violated, as is true for any theory of methods. Furthermore, one needs verify the methodology on various real datasets to under the applicability of our method, which is outside the scope of this paper. As these are quite standard limitations, we refrain from including them in the main draft. However, if the referees feel that this should be included in the main draft, we would be happy to oblige.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
27

---

judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
be accomplished in various ways.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
28

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See Section 5.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cm
onsiiiccyyyyl for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nniss. wttcc//pbbbc wccpublicc///cccccccccccccccclbb /ellii//uiieeee des/ htt////ooubuissseeeeeeeeeeeeeeee asionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures sy symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Both methods in the experiment took approximately 10 minutes run on MacBook Pro laptop (with M2 Max CPU, 10 Cores, 32 GB RAM, and no GPU).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tp:r//////.../eeee cc/public/Echiccuuieee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
30

---

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
See Section 5.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
31

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi wwwoeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurlPS Code Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
32

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "kAWQ1XwDh7", "reviewer": "Reviewer_6ToN", "summary": "This work proposes a method to construct prediction intervals in distribution shift problems, where unlabeled data from the target domain is available. The method is inspired by Fan et al (2023) in the i.i.d. setting and assumes we can estimate a transform (defined by reweighting or a transport map) that maps the joint distribution from the source domain to the target.", "questions": "Given the above questions on the theoretical motivation, novelty or significance, I believe there should be extensive empirical evaluation for the proposed method.\n\nIn light of question 3 above, I also wonder if the observed performance difference between the proposed method and conformal prediction could be attributable to the choice of different conformity scores; how does the method compare to an implementation of split conformal prediction using the same $(y - \\hat m(x)) / f(x)$ as the conformity socre?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem studied is relevant.\n- The method appears reasonable and sound.", "weaknesses": "- My main concern is about the novelty of this work: given the assumed condition on domain shift (namely a good approximation to the reweighting or transport map is readily available) it is a natural application of the method in Fan et al (2023).  The proof will require some additional effort but given the assumptions I do not expect them to be technically challenging.\n\n- While the paper employed assumptions on domain shift are common in literature, I am uncertain about their relevance, especially in high-dimensional problems.  It is well-known that the bounded density ratio assumption in Section 3 is unrealistic in high dimensions. And while I have come across past works on the transport map assumption, I do not know of realistic examples where the assumption actually holds so that the coverage guarantees established in this work could be relevant: it is much easier to think of scenarios where the assumption does not necessarily hold, but heuristic methods based on this assumption achieve better predictive performance; the latter is more often the goal in previous works.\n\n- Finally, I have doubts about the novelty/utility of Fan et al (2023), which seems quite similar to conformal prediction.  It could be alternatively summarized as applying quantile estimation to the \"conformity score\" of $(y - \\hat m(x)) / f(x)$, and compared with conformal prediction the main difference is the lack of sample splitting (in split conformal) or leave-one-out operations (as in full conformal).  I wonder if the removal of sample splitting is really a wise choice: the coverage guarantees in Fan et al (2023) and this work relies on the Rademacher complexity of the function class for $f$.  If the function class is defined by general ML models the guarantees would be rather unreliable in high-dimensional problems, in sharp contrast to the distribution-free guarantees in (split/full) conformal prediction.  Otherwise, we would need further sample splitting to construct a finite set of candidates for $f$, in which case we could just do split conformal prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading our paper and for your insightful comments. We answer your questions and concerns in the following.\n\n1. **Novelty compared to Fan et al. (2023)**:  Please see our global response. \n\n2. **Assumptions on the covariate shift**:  Covariate shift is a common assumption, even for high-dimensional data, e.g., see [1] or [2]. The impact of domain shift in our analysis is primarily through the estimation error of the density ratio and the optimal transport map. We acknowledge that high-dimensional settings can pose challenges, such as the curse of dimensionality. However, these issues can often be mitigated by incorporating more structure; without any structural assumption, we do not expect to gain any theoretical guarantee under covariate shift in high-dimension. An example of a structured distribution shift is the *exponential tilt* assumption (the one we used in our experiments),  or that only a subset of variables undergoes the shift (sparsity type assumption). \nSimilarly, we can assume some structure in the optimal transport map to avoid the curse of dimensionality. While our analysis is quite general, we believe that with specific structural assumptions, our method and analysis can be appropriately adjusted to incorporate these structures. Therefore, we agree that we need some structural assumptions in high-dimension, but without any assumption, the curse of dimensionality is unavoidable. Once we have that, our method is applicable to obtain a valid prediction band with a small width.\n\n[1] Tripuraneni, N., et al., Covariate shift in high-dimensional random feature regression.\n\n[2] Tripuraneni, N., et al., Overparameterization improves robustness to covariate shift in high dimensions.\n\n3. **Similarity and dissimilarity with conformal prediction**: We agree with you that there is a similarity between our method and the conformal prediction in the second stage, i.e., the shrinkage (i.e., (3.3) and (4.2)), which is equivalent to (weighted) conformal prediction as both of these methods aim to derive weighted quantile of the sample score $s(x, y) = (y - \\hat{m}(x))/\\hat{f}(x)$. However, the key difference lies in the construction of $\\hat f$; by efficiently aggregating various predictors, $\\hat f$ accurately captures the shape of the interval, resulting in a smaller bandwidth. \n\n# Robustness check\nAnother advantage of our method is its robustness. Our method aggregates various predictor intervals, including an estimator for the conditional variance function (Estimator 5). In contrast, the weighted variance-adjusted conformal (WVAC) prediction interval [3] heavily relies on accurately estimating this conditional variance, while conditional quantile regression relies on accurate estimation of the conditional quantile function. We did a small robustness check using the following model: \n\\begin{align*}\n    & X \\sim{\\sf Unif}([-1, 1]), \\ \\ \\xi \\sim{\\sf Unif}([-1, 1]),\\ \\  X,\\xi \\text{ independent}, \\\\\n    & Y = \\sqrt{1 + 25 X^4} \\xi .\n\\end{align*}\n\nWe estimated the conditional variance using a random forest with varying depths \\{3, 5, 7, 15\\}. We generated n = 2500 samples, keeping $75\\\\%$ as source data and resampling the remaining 25\\% with weighted samples proportional to $w(x) \\propto (1 + \\exp{(-2x)})^{-1}$. As depth increases, overfitting leads to poor out-of-sample variance predictions. The following table is the result of 100 Monte Carlo experiments. The number inside the parenthesis is the median of coverage over these Monte Carlo iterations.\n\n| Max depth | Avg. width--Our | Avg. width--WVAC |\n|-----------|------------------|------------------|\n| 3         | 2.07 (0.975)    | 3.08 (0.9712)    |\n| 5         | 2.07 (0.95)      | 3.28 (0.9664)    |\n| 7         | 2.068 (0.94)    | 3.33 (0.97)      |\n| 15       | 2.08 (0.97)      | 5.00 (0.97)      |\n\nThe above table implies our method is more robust to the misspecification of some model components and remains stable.  \n\nIn a nutshell,  our method has a similar coverage guarantee as the conformal prediction, which does not require many assumptions; our coverage guarantee (Theorem 3.4) requires a reasonable estimation of $w(\\cdot)$, which is necessary for both our method and that of [3]; [3] even assumed $w(\\cdot)$ is known, whereas our bounds quantify the estimation error of $w(\\cdot)$. When we require a guarantee on the width, we require some assumptions on the predictor class.   \n\n[3] Tibshirani, R., et al., Conformal prediction under covariate shift.\n\n4. **Extensive empirical evaluation** Please see our global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4kqztJEy0", "reviewer": "Reviewer_88pD", "summary": "Building on the work of Fan et al. (2023), this paper addresses the challenge of computing prediction intervals in an unsupervised domain adaptation setting, where labeled samples are available from a related source domain, and unlabeled covariates are available for the target domain. The primary objective is to generate prediction intervals for the target domain that provide both an asymptotic coverage guarantee and minimal width. The authors also explore the aggregation of prediction intervals on the target domain under both covariate shift and domain shift scenarios. Theoretical guarantees, including finite sample concentration bounds on approximation errors and coverage guarantees, are presented. The proposed methods are illustrated and compared with the weighted split conformal prediction method using a real-world dataset.", "questions": "See weaknesses.", "limitations": "The limitations are not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles the important and challenging problem of uncertainty quantification in the unsupervised domain adaptation setting.\n\n- The authors propose algorithms to generate prediction intervals with both coverage and minimal width guarantees.\n\n- Experimental results show that the proposed method generates prediction intervals with comparable coverage but smaller average width compared to an existing method.", "weaknesses": "- The paper assumes familiarity with Fan et al. (2023), making it difficult for readers who have not read that work to understand some notations and concepts. Section 2.1 on problem formulation is particularly hard to read without prior knowledge.\n\t- There is a lack of flow in the paper, with some explanations and definitions missing or unclear. For instance, terms like \"valid prediction interval\", and expressions such as U(x) and l(x) are not clearly explained.\n\t- The authors write \"This translates into solving the following optimization problem:\" without any explanations. \n\t- Another example is Line 234, \"adding a slight delta to ensure coverage even when F is complex.\". This is not clear. It is discussed in Fan et al (2023) in (2.4). \n\t- Even if simple, I suggest adding important details about different expressions in the appendix to help the reader.\n\n\n- The title mentions \"optimal aggregation,\" but model aggregation is not the central focus of the paper. It is first discussed in Remark 4.2 and briefly mentioned in line 272.\n\n- The paper should highlight the similarities and differences with Fan et al. (2023) in terms of contributions, both theoretical and empirical.\n\nAlso, the authors use different quantile levels for estimators compared to Fan et al. (2023) without explaining the reason for these differences.  In Fan et al (2023), the following quantile levels are considered: 0.05, 0.35, 0.65, and 0.95, while the authors use 0.85, 0.95, 0.9, and 0.9.\n\n\n- The statement that \"the shape of the prediction band does not change much if we change the level of coverage\" lacks clarity on scenarios where this might not be true, such as multimodal distributions (?).\n\n\n- The discussion on lambda <=1 is confusing and needs improvement. The paper inconsistently explains the shrinkage factor and its implications. For example, the authors write \"Therefore, it is not immediate whether the shrinkage factor  is smaller than 1, i.e., whether we are indeed shrinking the confidence interval (lambda > 1 is undesirable, as it will widen finit , increasing the width of the prediction band).\" and \"Although ideally lambda <= 1, it is not immediately guaranteed as we use separate data  for shrinking.\". The authors also discuss it in Lemma 3.3. and Lemma 4.4. \n\n\n- The term \"shape of the prediction band\" with prediction intervals can be confusing as it can have other meanings (see functional data). I suggest the authors clarify this in line 147.\n\n\n- The paper ignores a significant amount of research on prediction interval estimation. I suggest discussing important related work.\n\n- The limitations of the paper are not discussed. For example, focusing solely on intervals might have implications under multimodal distributions, where High-Density Regions (HDR) would be the smallest regions.\n\n- The relationship between the proposed approach and conformal prediction is not clearly explained. The paper should discuss the guarantees provided by the proposed approach compared to conformal prediction, including the relevant theoretical guarantees and calibration aspects.\n\nFor example, the weighted conformal approach also splits the data and estimates a density ratio.  What about the (weaker?) guarantees you provide? The following paper could also be useful: \"On the expected size of conformal prediction sets\".\n\nAlso, in conformal prediction, in addition to finite-sample marginal guarantees, the authors often provide asymptotic conditional guarantees. What can you say about this with your approach?\n\n\n- The comparison with the conformal approach in the experiments might not be fair, as conformal prediction does not optimize for the smallest width. Additionally, reporting the interval score, which is a proper scoring rule for intervals, would be useful for evaluating the methods.\n\n\n- Typos and clarifications:\n\t- line 137: \"remains same.\"\n\t- line 154: \"via by solving\"\n\t- line 453 \"a.s. on source\". On the source domain?\n\t- line 470, \"we with have\"\n\t- It is not \"Leduox-Talagrand\" but Ledoux-Talagrand\".\n        - line 481, a minus sign is missing\n\t- Line 490 to 492. It is not clear why tilde tau <= alpha.\n\t- Line 182, what does \\mathcal{F} - f* mean?\n\t- About Theorem 3.4, we expect the performance (or convergence rates) to also depend on alpha. Where does this appear in the theoretical guarantees?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading our paper and for your insightful comments.\n\n1. **The paper assumes familiarity with Fan et al. (2023)** Thank you for raising this concern! We will carefully proofread the revised version of the manuscript according to your suggestions.\n\n2. **Mention optimal aggregation** Thank you for pointing this out. Indeed, our current approach presents the general problem and then casts prediction interval aggregation as a special case. We will highlight the aggregation perspective more in the revised version of the manuscript.\n\n\n3. **Similarities and dissimilarities to Fan et al. (2023)** Please see our global response. We selected the quantile levels of 0.85, 0.95, 0.9, and 0.9 because they are close to the desired coverage and we believe they effectively capture the correct shape of the interval. It is important to note that our method is not restricted to these specific quantiles; we can also apply the quantile levels used in Fan (2023) to our approach if needed.\n\n\n4. **Change of shape of prediction band** Thank you for your feedback. We will clarify this in line 147 in the revision. Specifically, when we refer to the \"the shape of the prediction band does not change much\", we mean that the form of the minimal average width prediction band covering the data at a certain coverage level should not change much if we slightly change the coverage level, i.e., the form is stable with respect to the perturbation of the coverage level. For instance, if the optimal prediction band that covers $98\\\\%$ of the data is quadratic in $X$, then the minimal-width interval that covers $95\\\\%$ of the data is also a quadratic function of $X$, rather than abruptly changing to a linear function. Under this consistency assumption, a constant shrinkage factor $\\lambda$ in the second step (i.e., (3.3) and (4.2)) is sufficient to adjust from $100\\\\%$ to $95\\\\%$ coverage. However, if this is not the case, we can modify our method to allow the shrinkage level $\\lambda$ to depend on the input $X$, ensuring more accurate adaptation to the data distribution. \nFor the multimodal distributions, we acknowledge that the optimal region is not necessarily a prediction band (i.e., $Y|X=x$ is not necessarily a single interval) and will include this in the limitation of the work.\n\n5. **Discussion on $\\lambda \\le 1$** Thank you for the comments. We will clarify this discussion in the revised version of the paper. In the first step, we typically choose $\\epsilon$ to be small; therefore, the average width of $\\hat f_{\\rm init}$ can be larger than necessary, and coverage is also typically larger than $95\\\\%$. Therefore, in the second step, we aim to shrink the interval (i.e., ensure $\\lambda \\leq 1$). However, it is not immediate that $\\hat \\lambda$, obtained by solving equation (3.3) is indeed $\\le 1$. To ensure this, we prove in Lemma 3.3 and Lemma 3.4 that with high probability, we indeed have $\\hat \\lambda(\\alpha) \\le 1$. \n\n\n6. **Research on prediction interval estimation** \nThank you for pointing this out. In our revision, we will add a detailed related work section that covers significant research on prediction interval estimation, including methods applicable to both shifted and non-shifted data. We will discuss approaches such as conformal prediction, quantile regression, and RKHS-based methods, highlighting their contributions and relevance to our work.\n\n\n7. **Limitations of the paper are not discussed** Thank you very much for highlighting this limitation! It is indeed true that if the conditional distribution of $Y$ given $X$ is multimodal, the prediction interval may not be the best thing to do. We will mention this in the limitation.\n\n8. **Comparison with conformal**\n\n(1) *Methodology relation and difference*: Step 2 of Algorithm 1, i.e., shrinkage (equation (3.3) and (4.2)) in our method is equivalent to (weighted) conformal prediction, using the score function $(y - \\hat{m}(x))/\\hat{f}(x)$, where $\\hat{f}$ is obtained from the first step of our method. By efficiently aggregating various predictors, our $\\hat{f}$ accurately captures the shape of the interval, resulting in a smaller bandwidth. In contrast, variance-adjusted conformal prediction selects $f$ as the variance function, which may not necessarily capture an optimal shape for a small bandwidth. Additionally, our method explicitly aims to minimize the bandwidth in the first step. In contrast, conformal prediction does not explicitly minimize bandwidth; it implicitly addresses this by setting the cutoff at the $1-\\alpha$ quantile of the score function. \n\n(2)*Robustness*: Our method is more robust compared to variance-adjusted conformal prediction. Please see our response to Reviewer 6ToN for the details of our experiment regarding robustness.\n\n(3)*Theoretical guarantees*: We provide finite sample guarantees for both the width and coverage of our prediction intervals. In contrast, while typical conformal literature offers finite-sample marginal or asymptotic conditional guarantees for coverage, they often do not include finite-sample guarantees for width, as our paper does. \nFor distribution shift [1] provides coverage guarantee assuming the density ratio $w(\\cdot)$ is known, whereas our Theorem 3.4 explicitly highlights the effect of the estimation error of $w(\\cdot)$.   \nWe acknowledge that our method is particularly focused on addressing marginal coverage, and therefore, its pointwise guarantee is not immediately clear. We will include this as a limitation in our discussion and consider it a topic for future research.\n\n[1] Tibshirani, R., et al., Conformal prediction under covariate shift.\n\n9. **Typo** Thank you very much for carefully reading our paper and pointing out the typos! We will take care of it in the revised version of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4TIcyQpnuX", "reviewer": "Reviewer_zRWP", "summary": "The authors study the problem of how to construct prediction intervals on a target domain under both covariate shift and domain shift assumptions (i.e., the source and target domains are related either via a bounded density ratio, or a measure-preserving transformation), designed to ensure adequate coverage while minimizing interval width.\n\nThey provide theoretical guarantees, with finite sample bounds, regarding the prediction interval coverage and width.\n\nThey apply their method on the airfoil dataset, comparing the performance with weighted split conformal prediction.", "questions": "- Could you extend the experimental evaluation with more datasets and baseline methods?\n\n- Why is the figure on page 9 missing a figure number and caption?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written overall, the authors definitely seem knowledgeable.\n\n- The paper studies an important and interesting problem.\n\n- The proposed method outperforms weighted split conformal prediction on one dataset.", "weaknesses": "- The experimental evaluation is limited to a single low-dimensional tabular dataset, with a synthetic/simulated distribution shift. It is not possible to determine if the proposed method actually would have significant real-world utility/impact.\n\n\n\n\nSummary:\n- Important problem and a well-written paper overall, but the experimental evaluation is too limited. I think this could be a solid paper, but I don't think it can be accepted in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading our paper and for your insightful comments.\n\n1. **Extension of empirical evaluation**: Thank you for your comment, please see our global response for the details of the additional experiments.\n\n(i). We now have two more real-data experiments on our first method, Algorithm 1(which relies on the estimation of the density ratio of the source and the target covariates), one using the real estate valuation data and the other using the energy efficiency data. Furthermore, we have an additional experiment on our second method, Algorithm 2 (which relies on estimating the optimal transport map), using the Airfoil data. \n\n(ii). We have now added one more baseline, namely the weighted quantile-adjusted conformal method, i.e. we now compare our against both variance-adjusted weighted conformal and weighted quantile-adjusted conformal method. \n\n(iii). We have also conducted a simulation experiment to showcase the robustness of our algorithm in comparison to weighted conformal prediction. Kindly see our response to Reviewer 6ToN for the details of this experiment. \n\n2. **Figure 9**: Thank you for pointing out this problem. We will take care of it in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bnoW9YuRxo", "reviewer": "Reviewer_EZSx", "summary": "It is crucial to assess and quantify uncertainties associated with distribution shifts in the context of unsupervised domain shift. The paper propose methodologies for aggregating prediction intervals, which capture the range of likely outcomes for a given prediction. The authors support these methodologies theoretically by considering a bounded density ratio and a measure-preserving transformation between source and target domains. This includes finite sample bounds for the coverage and width of their prediction intervals.", "questions": "1.What does $f \\in \\mathcal{F}$ signify in equation (2.1)? There is no $f$ explicitly used in the main part of this equation.\n\n2.What does $m_0$ represent in line 118? Is it the same as $m_0$ mentioned in line 137? What is the relationship between $m_0$ and $f_0$?\n\n3.In line 164, it states, \"However, any such estimator $ \\hat{\\omega}(x) $ can be non-zero for $ x $ where $ \\omega_0(x)=0 $ due to estimation error. Consequently, $ \\hat{\\omega} $ may not be efficient in selecting informative source samples.\" Given this observation, why not apply the hinge function not to $ \\hat{\\omega} $, but to $ (Y - m_0(x))^2 - f(x) $ in Equation (3.5)?\n\n4.The notation $B_{\\mathcal{F}}$ first appears in Theorem 3.2 (please correct me if I am mistaken), and there is no explanation provided for it.\n\n5.The paper introduces two algorithms, but Section 5 includes experiments for only one of them. Which algorithm is it, and why are there no experiments conducted on the other one?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method is innovative in aggregating various prediction intervals on the target domain under the domain-aligning assumption within an unsupervised domain adaptation framework.\n\n2.The paper includes sufficient theoretical discussions, presenting finite sample concentration bounds to demonstrate that their method achieves adequate coverage with a small width.", "weaknesses": "1.Apart from theoretical results, there is a lack of intuitive understanding as to why aggregation can achieve minimal-width prediction intervals.  \n\n2.The paper does not include comparisons with previous methods of aggregated prediction intervals (Hosen et al., 2014).\n\n3.Limited experiments and comparative methods make it difficult to affirm the effectiveness of the algorithm.\n\n[1] Hosen, A., Khosravi, A., Nahavandi, S., and Creighton, D. Improving the Quality of Prediction Intervals Through Optimal Aggregation. IEEE Transactions on Industrial Electronics. 2014", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for reading our paper and for your insightful comments.\n\n1. **Lack of intuitive understanding**: Thank you for your concern. To achieve a minimal-width prediction interval, it is crucial to accurately capture the shape of the interval. However, a single predictor may fail to do so due to its specific structural limitations. For example, the correct shape might be a polynomial function, but the predictor could output a linear function, resulting in model misspecification. By aggregating several predictors and considering a convex hull of these base predictors, we reduce model misspecification. This approach leads to an ensemble predictor that more accurately captures the interval's shape, resulting in a smaller width. The second step (shrinkage) is equivalent to conformal prediction, which maintains the coverage guarantee. \n\n2. **Comparison with Hosen et al. (2014)**: Thank you for pointing out this interesting paper. We will include a comparison with the previous methods from Hosen et al. (2014) in our revised version. \nHosen et al. (2014) focus on ensemble prediction intervals obtained using neural networks through ranking and weighted averaging. While they consider both coverage and width, they combine these metrics into a single optimization objective (coverage width criterion). In contrast, our method focuses on finding the interval with the smallest width among those that provide sufficient coverage.\nAdditionally, Hosen et al. (2014) do not provide theoretical guarantees for coverage and width, nor do they address domain shift scenarios. Our work, on the other hand, emphasizes providing a theoretically robust methodology specifically designed to handle domain shift challenges.\n\n\n3. **Limited experiments**: Thank you for raising this point. We have now conducted three additional real-data experiments to evaluate our method. We have added one more baseline, the weighted quantile conformal prediction method. Please see the global response for details on these experiments. Last but not least, we have also conducted a simulation experiment quantifying the robustness of our method against conformal prediction. Please see our response to Reviewer 6ToN for details of that experiment.\n   \n4. **What does $f \\in \\\\mathcal{F}$ signify?**: Sorry for the typo, it should be $\\min_{l, u}$. \n\n5. **What does $m_0$ represent?**: $m_0$ is the conditional mean function, as defined in line 137. We will clarify it in our revision. \n\n6. **Why not apply hinge to $\\hat w(x)$?**: The purpose of using a hinge function is to ensure the convexity of the problem. Specifically, we replace the indicator function for $(Y - m_0(x))^2 - f(x)$ with a convex surrogate (hinge function) to facilitate the use of standard convex optimization techniques. \n\n7. **Notation $\\\\mathcal{B}_{\\mathcal{F}}$**: $\\\\mathcal{B}\\_{\\mathcal{F}}$ is defined to be an upper bound of $\\sup_{f\\in\\mathcal{F}}\\|f\\|_{\\infty}$ (as shown in line 181 in Theorem 3.2). We will make it more clear in our revision.\n\n8. **Experiment on Algorithm 2**: We have conducted an experiment on the Airfoil data using our second algorithm, which requires estimating the optimal transport map. Please see our global response for the details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global response
We thank all the reviewers for their insightful comments. We address a few concerns that were raised by multiple reviewers. 
1. **Comparison with Fan et al. (2023):** The key contribution of our paper lies in adapting the methodology from Fan (2023) (which only addresses no-shift scenarios), to tackle domain shift challenges. 
We deal with unsupervised domain adaptation, i.e., i) the distribution of the covariates of the target domain is different from the source domain, and ii) we do not observe any label from the target domain. Therefore, we are required to change the methodology; 
to adapt to these challenges, the density ratio or the optimal transport map between the covariates must be estimated. Furthermore, as pointed out in Section 3, the shift may cause the optimization problem non-convex, for which we need to introduce a convex surrogate (e.g., the hinge function). As a consequence, we have various additional theoretical challenges to establish that our method indeed produces a prediction band on the target domain with adequate coverage and minimal width. For example, our theory highlights the precise effect of the estimation error of the density ratio or the optimal transport map in the bound on the coverage and the width. Furthermore, when we replace the indicator function with its convex surrogate hinge function (see (3.5)), we introduce two additional parameters $(\epsilon, \delta)$, effectively measuring the closeness between the original NP-hard problem (involving indicator function) and its convex surrogate. Our theoretical bounds also highlight the effect of these hyperparameters in our finite sample bounds. 

2. **Additional Experiments:** We conducted three more real-data experiments.
In the following, WVAC is weighted variance adjusted conformal prediction interval, where the score function is $s(x, y) = |y - \hat m(x)|/\hat \sigma(x)$ with $\hat m$ and $\sigma$ being the estimate of conditional mean and variance function respectively. WQC is the weighted quantile conformal prediction interval with $s(x, y) = \max \\{\hat q_{\alpha/2}(x) - y, y - \hat q_{(1-\alpha)/2}(x)\\}$, where $\hat q$ is the estimated quantile function. 

(a). **Real Estate Dataset** The Real Estate Valuation dataset consists of 414 instances. The goal is to predict the house price per unit area based on the 6 other features. The data is available at the UCI ML repository. The construction of shifted data (with $\beta = (-1, 0, -1, 0, 1, 1)$) and implementation procedure are the same as our paper. The following table and Figure 1 in PDF present the results over 200 Monte Carlo iterations. It is evident from the table that our method produced a small average width in comparison to the other methods while maintaining the coverage guarantee. 

| Outcome                                         | Our Method | WVAC    | WQC     |
|-------------------------------------------------|------------|---------|---------|
| **Coverage (Median)**                           | 0.98       | 0.962   | 0.971   |
| **Coverage (IQR)**                              | 0.058      | 0.048   | 0.048   |
| **Bandwidth (Median)**                          | 36.889     | 46.392  | 46.858  |
| **Bandwidth (IQR)**                             | 15.001     | 19.027  | 14.189  |
| **Bandwidth (Median for Coverage > 95%)**       | 40.774     | 50.703  | 51.031  |




(b). **Energy efficiency data** The goal of the Energy Efficiency dataset is to predict the heating load based on 8 other covariates. 
The construction of shifted data (with $\beta = (-1, 0, 1, 0, -1, 0, 0, -1)$) and implementation procedure are the same as our paper. This data is also available at the UCI ML repository. The following table and Figure 2 in PDF present the results over 200 Monte Carlo iterations. It shows that our method produced a smaller bandwidth than WVAC. While WQC has a smaller median bandwidth, it sacrifices coverage. The last row indicates that for experiments with coverage $\geq 95\\%$, WQC's median average width is significantly larger than ours. Thus, whenever WQC provides adequate coverage, its bandwidth is much larger than ours. 
| Outcome                                         | Our Method | WVAC    | WQC     |
|-------------------------------------------------|------------|---------|---------|
| **Coverage (Median)**                           | 0.995      | 0.969   | 0.973   |
| **Coverage (IQR)**                              | 0.047      | 0.036   | 0.05    |
| **Bandwidth (Median)**                          | 4.332      | 5.045   | 2.842   |
| **Bandwidth (IQR)**                             | 1.358      | 3.269   | 2.551   |
| **Bandwidth (Median for Coverage > 95%)**       | 4.373      | 5.681   | 4.94    |

(c). **Airfoil data with Algorithm 2** We implement our second method (based on OT) using the airfoil data. Here, we shift 25% of the data by a linear transformation: $x \mapsto Ax + b$, where $A$ = diagonal(1.5, 1.2, 1.6, 2, 1.8) and $b = (1, 0, 0, 1, 0)$. Our method produces a small width in comparison to other methods. See Figure 3 in PDF and the following table.

| Outcome                                         | Our Method | Our Method (without OT) | WVAC    | WQC     |
|-------------------------------------------------|------------|-------------------------|---------|---------|
| **Coverage (Median)**                           | 0.928      | 0.749                   | 0.984   | 0.952   |
| **Coverage (IQR)**                              | 0.035      | 0.22                    | 0.024   | 0.077   |
| **Bandwidth (Median)**                          | 15.075     | 18.512                  | 36.298  | 32.143  |
| **Bandwidth (IQR)**                             | 1.638      | 3.089                   | 10.619  | 8.364   |
| **Bandwidth (Median for Coverage > 95%)**       | 16.429     | 25.268                  | 37.783  | 36.433  |

(d).**For the robustness experiment, please see our response to Reviewer 6ToN.**

Meta-review: The authors propose a method to optimally combine prediction intervals under domain shift via convex optimization.

Reviewers had initial concerns but they were judiciously clarified by the authors.

I recommend accepting this paper.
Decision: Accept (poster)