Title: Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift
Venue: NeurIPS 2024
Accepted: yes
Keywords: prediction intervals aggregation

unsupervised domain shift

minimal width

adequate coverage

distribution shift
Abstract: As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts.
A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. 
The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. 
In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain.
Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation.
Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.

Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift

Abstract

As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts. A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.

1 Introduction

In the modern era of big data and complex machine learning models, extensive data collected from diverse sources are often used to build a predictive model. However, the assumption of independent and identically distributed (i.i.d.) data is frequently violated in practical scenarios. Take algorithmic fairness as an example: historical data often exhibit sampling biases towards certain groups, like females being underrepresented in credit card data. Over time, the differences in group proportions have diminished, leading to distribution shifts. Consequently, models trained on historical data may face shifted distributions during testing, and proper adjustments are needed. Distribution shift has garnered significant attention from statistical and machine learning communities under various names, i.e., transfer learning [PY09, WKW16], domain adaptation [FVRA21], domain generalization [ZLQ + 22, WLL + 22], continual learning [DLAM + 21, MLJ + 22], multitask learning [ZY21]

etc. While numerous methods are available in the literature for training predictive models under distribution shift, uncertainty quantification under distribution shift has received relatively scant attention despite its crucial importance. One notable exception is conformal prediction under distribution shift; [TFBCR19] proposed a variant of standard conformal inference methods to accommodate test data from a distinct distribution from the training data under the covariate shift. Recently, [GC21] introduced an adaptive conformal inference approach suitable for continuously changing distributions over time. Additionally, quantile regression under distribution shift offers another avenue for addressing uncertainty quantification under distribution shift [ERS + 22].

Although few methods exist for constructing prediction intervals under distribution shift, most focus primarily on ensuring coverage guarantee rather than minimizing interval width. This prompts the immediate question:

Can we generate prediction intervals in the target domain that provide both i) coverage guarantee and ii) minimal width?

This paper seeks to address this question by leveraging model aggregation techniques [NW15, MNW16, CEN14, Vov15, HKNC14]. Suppose we have K different methods for constructing prediction intervals in the source domain. Our proposed approach efficiently combines these methods to produce prediction intervals in the target domain with adequate coverage and minimal width. When individual methods are the elementary basis functions, such as the kernel basis, the resulting aggregation is indeed a construction of the prediction interval based on the basis functions. Our methodology draws inspiration primarily from recent work [FGM23] on prediction interval aggregation under the i.i.d. setting. However, a key distinction lies in our focus on unsupervised domain adaptation , where we can access labeled samples from the source and unlabeled samples from the target domain. Certain assumptions regarding the similarities between these domains are necessary to facilitate knowledge transfer from the source to the target domain. We explore two types of similarities in this paper: i) covariate shift , where we assume that the distribution of the response variable Y given X is consistent across both domains, albeit the distribution of X may differ, and ii) domain shift , where we assume that the conditional distribution of Y given X remains unchanged up to a measure-preserving transformation. Covariate shift is a well-explored concept in transfer learning and has also garnered attention in uncertainty quantification. It allows different distributions of X while maintaining identical conditional distributions Y | X across domains. For constructing conformal prediction intervals within this framework, see [TFBCR19, HL23, YKT22, LC21] and references therein. On the other hand, distribution shift is more general, allowing both the distribution of X and the conditional distribution of Y | X to differ across domains. Our methods in this context draw upon domain matching principles via transport map, as proposed in [CFT14] and further elaborated in subsequent works like [CFTR16, CFHR17, RHS17], among others. The key assumption is the existence of a measure-preserving/domain-aligning map T from the target to the source domain, such that the conditional distribution of Y | X on the target domain matches Y | T ( X ) on the source domain, i.e., conditional distributions matches upon domain alignment. The case where the domain-aligning map is the optimal transport map has received considerable attention in the literature, e.g., see [CFT14, CFTR16, CFHR17, XLW + 20]. Empirical evidence supports the efficacy of domain alignment through optimal transport maps across various datasets. For instance, in [XLW + 20], a variant of this method is applied for domain adaptation in image recognition tasks, such as recognizing similarities between USPS [Hul94], MNIST [LBBH98], and SVHN digit images [NWC + 11], as well as between different types of images in the Office-home dataset [VECP17], including artistic and product images. Additionally, in [CFT14], the authors explore the impact of domain alignment via optimal transport maps on the face recognition problem, where different poses give rise to distinct domains. However, most of these works concentrate on training predictors that perform well on the target domain without any guarantee regarding uncertainty quantification. To our knowledge, this is the first work to propose a method with rigorous theoretical guarantees for constructing prediction intervals on the target domain under the domain-aligning assumption within an unsupervised domain adaptation framework. We now summarize our contributions.

Our Contributions: This paper introduces a novel methodology for aggregating various prediction methods available on the source domain to construct a unified prediction interval on the target domain under both covariate shift and domain shift assumptions. Our approach is simple and easy to implement and requires solving a convex optimization problem, which can even be

simplified to a linear program problem in certain scenarios. We also establish rigorous theoretical guarantees, presenting finite sample concentration bounds to demonstrate that our method achieves adequate coverage with a small width. Furthermore, our methodology extends beyond model aggregation; it can be used to construct efficient prediction intervals from any convex collection of candidate functions. In the paper, we adopt this broader perspective, discussing how the aggregation of prediction intervals emerges as a particular case. Lastly, we validate the effectiveness of our approach by analyzing real-world datasets.

We also want to highlight the differences between our method and a related method proposed in [FGM23]. We deal with unsupervised domain adaptation, i.e., we do not observe any label from the target domain, in contrast to [FGM23], which only deals with i.i.d. data. Hence, significant changes in methodology are required to address the domain shift. Furthermore, as pointed out in Section 3, the shift may cause the optimization problem non-convex, for which we need to introduce a convex surrogate (e.g., the hinge function), leading to additional theoretical challenges.

2 Notations and preliminaries

Notation The covariates of the source and the target domains are denoted by X S and X T , respectively, and X := X S ∪ X T . The space of the label is denoted by Y . We use the notation E S (resp. E T ) to denote the expectation with respect to the source (resp. target) distribution. The expectation with respect to sample distribution is denoted by E n,S and E n,T . We use p S (resp. p T ) to denote the probability density function of X on the source and the target domain, respectively. Throughout the paper, we use c to denote universal constants, which may vary from line to line.

2.1 Problem formulation

Our setup aligns with the unsupervised domain adaption; we assume to have n S i.i.d. labeled samples { X S,i , Y S,i } n S i =1 ∼ P S ( X,Y ) from the source domain, and n T i.i.d. unlabeled samples { X T,i } n T i =1 ∼ P T ( X ) from the target domain. Given any α > 0 , ideally, we want to construct a valid prediction interval with minimal width on the target domain:

In many practical contexts, the preferred prediction interval takes the form of m ( X ) ± g ( X ) , where m ( X ) is a predictor for Y given X (an estimator of E T [ Y | X ] ), and g ( X ) gauges the uncertainty of the predictor m ( X ) . The optimizer of (2.1) takes this simplified form when the distribution of Y -E T [ Y | X ] is symmetric around 0 . Moreover, it offers a straightforward interpretation as the pair ( m,g ) is a predictor and a function quantifying its uncertainty. Within the framework of this simplified prediction interval, we need to estimate m and g . Estimating the conditional mean function m is relatively easy and has been extensively studied; one may use any suitable parametric/non-parametric method. Upon estimating m , we need to estimate g so that the prediction interval [ m ( X ) ± g ( X )] has both adequate coverage and minimal width. This translates into solving the following optimization problem:

Let f 0 be the solution of the above optimization problem. Then the optimal prediction interval is [ m 0 ( x ) ± √ f 0 ( x )] . However, the key challenge here is that we do not observe the response variable Y from the target, and consequently, solving (2.2) becomes infeasible. Hence, we must rely on transferring our knowledge acquired from labeled observations in the source domain, which necessitates making certain assumptions regarding the similarity between the two domains. Depending on the nature of these assumptions regarding domain similarity, our findings are presented in two sections: Section 3 addresses covariate shift under the bounded density ratio assumption, while Section 4 considers a more general distribution assumption under measure-preserving transformations. Furthermore, as will be shown later, this problem, though well-defined, is not easily implementable. Therefore, we propose a surrogate convex optimization problem in this paper and provide its theoretical guarantees.

2.2 Complexity measure

The complexity of the function class F is usually quantified through the Rademacher complexity, defined as follows.

Definition 2.1 (Rademacher complexity) . Let F be a function class and { X i } n i =1 be a set of samples drawn i.i.d. from a distribution D . The Rademacher complexity of F is defined as

where { ϵ i } n i =1 are i.i.d. Rademacher random variables that equals to ± 1 with probability 1 / 2 each.

3 Covariate shift with bounded density ratio

Setup and methodology In this section, we focus on the covariate shift problems, where the marginal densities p S ( X ) and p T ( X ) of the covariates may vary between the source and target domains, albeit the conditional distribution Y | X remains the same. Denote by m 0 ( x ) = E T [ Y | X = x ] = E S [ Y | X = x ] , the conditional mean function. For the ease of the presentation, we assume m 0 is known. If unknown, one may use the labeled source data to estimate it using a suitable parametric/non-parametric estimate (e.g., splines, local polynomial, or deep neural networks), subsequently substituting m 0 with ˆ m in our approach. The density ratio of the source and the target distribution of X is denoted by w 0 ( x ) := p T ( x ) /p S ( x ) . We henceforth assume that the density ratio is uniformly bounded:

Assumption 3.1. There exists W such that sup x ∈X S w 0 ( x ) ≤ W .

If w 0 is known, (2.2) has the following sample level counterpart:

which is NP-hard owing to the presence of the indicator function. However, in many practical scenarios, it is observed that the shape of the prediction band does not change much if we change the level of coverage (i.e., α ); only the bands shrink/expand. Indeed, the true shape determines the average width; if the shape is wrong, then the width of the prediction band is quite likely to be unnecessarily large. Therefore, to obtain a prediction interval with adequate coverage and minimal width, one should first identify the shape of the prediction band and then shrink/expand it appropriately to get the desired coverage. This motivates the following two steps procedure:

Step 1: (Shape estimation) Obtain an initial estimate ˆ f init by solving (3.1) for α = 0 (to capture the shape):

Step 2: (Shrinkage) Refine ˆ f init by scaling it down using ˆ λ ( α ) , defined as:

The final prediction interval is:

In Step 1, we relax (3.1) by effectively setting α = 0 . This relaxation aids in determining the optimal shape while also converting (3.1) into a convex optimization problem (equation (3.2)) as long as F is a convex collection of functions. Furthermore, in (3.2), we only consider those source observations for which w 0 ( x ) > 0 , as otherwise, the samples are not informative for the target domain. In practice, w 0 is typically unknown; one may use the source and target domain covariates to estimate w 0 . Various techniques are available for estimating the density ratio (e.g., [USS + 16, CMSE22, Qin98, GSH + 08] and references therein). However, any such estimator ˆ w ( x ) can be non-zero for x where w 0 ( x ) = 0 due to estimation error. Consequently, ˆ w may not be efficient in selecting informative source samples. To mitigate this issue, we propose below a modification of (3.2), utilizing a hinge function h δ ( t ) := max { 0 , ( t/δ ) + 1 } :

with δ and ϵ should be chosen based on sample size n S and the estimation accuracy of ˆ w . When ˆ w = w 0 (i.e., the density ratio is known), then by choosing ϵ = 0 and δ → 0 , (3.5) recovers (3.2). As h δ is convex, the optimization problem (3.5) is still a convex optimization problem. We summarize our algorithm in Algorithm 1.

Algorithm 1 Prediction intervals with bounded density ratio

1: Input: m 0 (or ˆ m if unknown), density ratio estimator ˆ w , function class F , sample D S = { ( X S,i , Y S,i ) } n S i =1 and D T = { X T,i } n T i =1 , parameters δ , ϵ , coverage level 1 -α .

2: Obtain ˆ f init by solving (3.5).

3: Obtain the shrink level ˆ λ ( α ) by solving (3.3) with w 0 replaced by ˆ w .

4: Output: ̂ PI 1 -α ( x ) defined in (3.4).

Theoretical results We next present theoretical guarantees of the prediction interval obtained via Algorithm 1. For technical convenience, we resort to data-splitting; we divide the source data into two equal parts ( D S, 1 and D S, 2 ), use D S, 1 and D T to solve (3.5), and D S, 2 to obtain the shrink level ˆ λ ( α ) . Without loss of generality, we assume m 0 ≡ 0 (otherwise, we set Y ← Y -m 0 ( X ) ). A careful inspection of Step 1 reveals that ˆ f init aims to approximate a function f ∗ defined as follows:

In other words, ˆ f init estimates f ∗ that has minimal width among all functions covering the response variable. This is motivated by the philosophy that the right shape leads to a smaller width . The following theorem provides a finite sample concentration bound on the approximation error of ˆ f init :

Theorem 3.2. Suppose Y 2 -f ∗ ( X ) ≤ B on the source domain and has a density bounded by L . Also assume ∥ f ∥ ∞ ≤ B F for all f ∈ F . Then for

we have with probability at least 1 -3 e -t :

where W ′ = ∥ ˆ w ∥ ∞ .

The bound in the above theorem depends on the Rademacher complexity of F (the smaller, the better), the estimation error of w 0 , and an interplay between the choice of ( ϵ, δ ) . The lower bound on ϵ in (3.7) depends on both δ and 1 /δ . Although it is not immediate from the above theorem why we need to choose ϵ to be as small as possible, it will be apparent in our subsequent analysis; indeed if ϵ is large in (3.5), then ˆ f init ≡ 0 will be a solution of (3.5). Consequently, the shape will not be captured. Therefore, one should first choose δ (say δ ∗ ), that minimizes the lower bound (3.7), and then set ϵ = ϵ ∗ equal to the value of the right-hand side of (3.7) with δ = δ ∗ , which ensures that ϵ ∗ is optimally defined to capture the shape accurately. Once the shape is identified, we shrink it properly in Step 2 to attain the desired coverage and reduce the width. Although ideally ˆ λ ( α ) ≤ 1 , it is not immediately guaranteed as we use separate data ( D S, 2 ) for shrinking. The following lemma shows that ˆ λ ( α ) ≤ 1 for any fixed α > 0 as long as the sample size is large enough. Recall that the data were split into exactly half with size n S = |D S | .

Lemma 3.3. Under the aforementioned choice of ( ϵ ∗ , δ ∗ ) , we have with high probability:

for all large n S , provided that ˆ w is a consistent estimator of w 0 . Hence, ˆ λ ( α ) ≤ 1 .

Our final theorem for this section provides a coverage guarantee for the prediction interval given by Algorithm 1.

Theorem 3.4. For the prediction interval obtained in (3.4) , with probability greater than 1 -2 e -t :

for some constant C > 0 and W ′ = ∥ ˆ w ∥ ∞ .

Theorem 3.4 validates the coverage of the prediction interval derived through Algorithm 1, achieving the desired coverage level as the estimate of w 0 improves and sample size expands. Theorems 3.2 and 3.4 collectively demonstrate the efficacy of our method in maintaining validity and accurately capturing the optimal shape of the prediction band, which in turn leads to small interval widths.

Remark 3.5. In our optimization problem, we've substituted the indicator loss with the hinge loss function to ensure convexity. However, it's worth noting that if we know the subset of X S where w 0 ( x ) > 0 beforehand, we could directly optimize (3.2) . This approach would be easy to implement and wouldn't involve tuning parameters ( δ, ϵ ) . A special case is when w 0 ( x ) > 0 for all x ∈ X S (as is true in our experiment), which simplifies the condition in (3.2) to f ( X i ) ≥ ( Y i -m 0 ( X i )) 2 for all 1 ≤ i ≤ n S . However, if this information is unavailable, one can still employ (3.2) by enforcing the constraint on all source observations. While this approach might result in wider prediction intervals, it is easy to implement and doesn't require tuning parameters.

4 Domain shift and transport map

Setup and methodology In the previous section, we assume a uniform bound on the density ratio. However, this may not be the case in reality; it is possible that there exists x ∈ supp ( X T ) ∩ supp ( X c S ) , which immediately implies that w 0 ( x ) = ∞ . In image recognition problems, if the source data are images taken during the day at some place, and the target data are images taken at night, then this directly results in an unbounded density ratio (due to the change in the background color). Yet a transport map could effectively model this shift by adapting features from the source to correspond with those of the target, maintaining the underlying patterns or object recognition capabilities across both domains. To perform transfer learning in this setup, we model the domain shift via a measure transport map T 0 that preserves the conditional distribution, as elaborated in the following assumption:

This assumption allows the extrapolation of source domain information to the target domain via T 0 , enabling the construction of prediction intervals at x ∈ X T by leveraging the analogous intervals at T 0 ( x ) ∈ X S . Inspired by this observation, we present our methodology in Algorithm 2 that essentially consists of two key steps: i) constructing a prediction interval in the source domain and ii) transporting this interval to the target domain using the estimated transport map T 0 . If T 0 (or its estimate) is not given, it must be estimated from the source and the target covariates. Various methods are available in the literature (e.g., [DNWP22, SDF + 17, MTOL20, DGS21]), and practitioners can pick a method at their convenience. Notably, the processes described in equations (4.1) and (4.2) follow the methodology (i.e., (3.2) and (3.3)) from Section 3 for scenarios without shift (i.e., w 0 ≡ 1 ), adding a slight δ to ensure coverage even when F is complex. In Algorithm 2, we assume

Algorithm 2 Transport map

1: Input: conditional mean function m 0 on the source domain, transport map estimator ˆ T 0 , function class F , sample D S = { ( X S,i , Y S,i ) } n S i =1 and D T = { X T,i } n T i =1 , parameter δ , coverage level 1 -α .

2: Obtain ˆ f init by solving:

3: Obtain the shrink level

the conditional mean function m 0 on the source domain is known. In cases where the conditional

mean function m 0 on the source domain is unknown, it can be estimated using standard regression methods from labeled source data, after which m 0 is replaced by this estimate, ˆ m .

Remark 4.2 (Model aggregation) . Suppose we have K different methods { f 1 , . . . , f K } for constructing prediction intervals in the source domain. In the context of model aggregation, (4.1) then reduces to:

In other words, the function class F is a linear combination of the candidate methods. The problem is then simplified to a linear program problem, which can be implemented efficiently using standard solvers.

Theoretical results We now present theoretical guarantees of our methodology to ensure that our method delivers what it promises: a prediction interval with adequate coverage and small width. For technical simplicity, we split data here: divide the labeled source observation with two equal parts (with n S / 2 observations in each), namely D S, 1 and D S, 2 . We use D S, 1 to solve (4.1) and obtain the initial estimator ˆ f init , and D S, 2 to solve (4.2), i.e. obtaining the shrinkage factor ˆ λ ( α ) . Henceforth, without loss of generality, we assume m 0 = 0 and present the theoretical guarantees of our estimator. We start with an analog of Theorem 3.2, which ensures that with high probability ˆ f init ◦ ˆ T 0 approximates the function that has minimal width among all the functions in F composed with T 0 that covers the labels on the target almost surely:

Theorem 4.3. Assume the function class F is B F -bounded and L F -Lipschitz. Define

Then we have with probability ≥ 1 -e -t :

The upper bound on the population width of ˆ f init ◦ ˆ T 0 ( x ) consists of four terms: the first term is the minimal possible width that can be achieved using the functions from F , the second term involves the Rademacher complexity of F , the third term encodes the estimation error of T 0 , and the last term is the deviation term that influences the probability. Hence, the margin between the width of the predicted interval and the minimum achievable width is small, with the convergence rate relying on the precision of estimating T 0 and the complexity of F , as expected.

We next establish the coverage guarantee of our estimator of Algorithm 2, obtained upon suitable truncation of ˆ f init . As mentioned, the shrinkage operation is performed on a separate dataset D S, 2 . Therefore, it is not immediate whether the shrinkage factor ˆ λ ( α ) is smaller than 1 , i.e., whether we are indeed shrinking the confidence interval ( ˆ λ ( α ) > 1 is undesirable, as it will widen ˆ f init , increasing the width of the prediction band). The following lemma shows that with high probability, ˆ λ ( α ) ≤ 1 .

Lemma 4.4. With probability greater than or equal to 1 -e -t , we have:

where

Here p n S is the conditional probability of a test observation Y falling outside [ -√ ˆ f init ( X ) + δ, √ ˆ f init ( X ) + δ ] , which is small as evident from the above lemma. In particular, for model aggregation, if F is the linear combination of K functions, then p n S is of the order √ K/n S . Hence, the final prediction interval is guaranteed to be a compressed form of ˆ f init with an overwhelmingly high probability. We present our last theorem of this section, confirming that the prediction interval derived from Algorithm 2 achieves the intended coverage level with a high probability:

Theorem 4.5. Under the same setup of Theorem 4.3, along with the assumption that f S ( y | x ) is uniformly bounded by G , we have with probability greater than 1 -cn -10 S that

As for Theorem 4.3, the bound obtained in Theorem 4.5 also depends on two crucial terms: Rademacher complexity of F and estimation error of T 0 . Therefore, the key takeaway of our theoretical analysis is that the prediction interval obtained from Algorithm 2 asymptotically achieves nominal coverage guarantee and minimal width. Furthermore, the approximation error intrinsically depends on the Rademacher complexity of the underlying function class and the precision in estimating T 0 .

Remark 4.6 (Measure preserving transformation) . In our approach, T 0 is employed to maintain measure transformation, although it may not necessarily be an optimal transport map. Yet, estimating T 0 can be challenging in many practical scenarios. In such cases, simpler transformations like linear or quadratic adjustments are often utilized to align the first few moments of the distributions. Various methods provide such simple solutions, including, but not limited to, CORAL [SFS17] and ADDA [THSD17].

5 Application

In this section, we illustrate the effectiveness of our method by applying it to five different datasets: i) airfoil dataset [DG19], ii) real estate data [Yeh18], iii) energy efficiency data [TX12b], iv) appliance energy prediction data [Can17], and v) ET Dataset (ETT-small) [ZZP + 21]. The first four datasets are freely available in the UCI repository, and the last dataset can be found in this GitHub link. Here, we illustrate the procedure using the airfoil dataset, and the details of our experiments using the other datasets can be found in Appendix C. The airfoil dataset includes 1503 observations, featuring a response variable Y (scaled sound pressure level) and a five-dimensional covariate X (log of frequency, angle of attack, chord length, free-stream velocity, log of suction side displacement thickness). We assess and compare the performance of our prediction intervals in terms of coverage and width with those generated by the weighted split conformal prediction method described in [TFBCR19]. We use the same data-generating process described in [TFBCR19] to facilitate a direct comparison. We run experiments 200 times; each time, we randomly partition the data into two parts D train and D test , where D train contains 75% of the data, and D test contains 25% of the data. Following [TFBCR19], we shift the distribution of the covariates of D test by weighted sampling with replacement, where the weights are proportional to

These reweighted observations in D test , which we call D shift , act as observations from the target domain. Clearly, by our data generation mechanism w 0 ( x ) = f T ( x ) /f S ( x ) = c exp ( x ⊤ β ) , where c is the normalizing constant. The source and target domains share the same support under this configuration. As our methodology is developed for unsupervised domain adaptation, we do not use the label information of D shift to develop the target domain's prediction interval.

Density ratio estimation We use the probabilistic classification technique to estimate the density based on the source and the target covariates. Let X 1 , . . . , X n 1 be the covariates in dataset D train and X n 1 +1 , . . . , X n 1 + n 2 be the covariates in dataset D shift. The density ratio estimation proceeds in two steps: (1) logistic regression is applied to the feature-class pairs { ( X i , C i ) } n i =1 , where C i = 0 for

i = 1 , . . . , n 1 and C i = 1 for i = n 1 +1 , . . . , n 1 + n 2 , yielding an estimate of P ( C = 1 | X = x ) , denoted as ˆ p ( x ) ; (2) the density ratio estimator is then defined as ˆ w ( x ) = n 1 n 2 · ˆ p ( x ) 1 -ˆ p ( x ) . Further explanations are provided in Appendix B.

Implementation of our method and results As the mean function m 0 ( x ) = E [ Y | X = x ] (which is the same on the source and the target domain) is unknown, we first estimate it via linear regression, which henceforth will be denoted by ˆ m ( x ) . To construct a prediction interval, we consider the model aggregation approach, i.e., the function class F is defined as the linear combination of the following six estimates:

Estimator 1 ( f 1 ) : Aneural network based estimator with depth=1, width=10 that estimates the 0.85 quantile function of ( Y -ˆ m ( X )) 2 | X = x .

Estimator 2 ( f 2 ) : A fully connected feed forward neural network with depth=2 and width=50 that estimates the 0.95 quantile function of ( Y -ˆ m ( X )) 2 | X = x .

Estimator 3 ( f 3 ) : Aquantile regression forest estimating the 0.9 quantile function of ( Y -ˆ m ( X )) 2 | X = x .

Estimator 4 ( f 4 ) : A gradient boosting model estimating the 0.9 quantile function of ( Y -ˆ m ( X )) 2 | X = x .

Estimator 5 ( f 5 ) : An estimate of E [( Y -ˆ m ( X )) 2 | X = x ] using random forest.

Estimator 6 ( f 6 ) : The constant function 1 .

Here, the quantile estimators are obtained by minimizing the corresponding check loss. The implementation of our method is summarized as follows: (1) We divide the training data D train into two halves D 1 ∪ D 2 . We utilize dataset D 1 to derive a mean estimator and six aforementioned estimates. We also employ the covariates from D 1 and D shift to compute a density ratio estimator. (2) We further split D 2 into two equal parts D 2 , 1 and D 2 , 2 . D 2 , 1 , along with covariates from D shift , is used to find the optimal aggregation of the six estimates to capture the shape, i.e., for obtaining ˆ f init . The second part D 2 , 2 is used to shrink the interval to achieve 1 -α = 0 . 95 coverage, i.e. to estimate ˆ λ ( α ) . (3) We evaluate the effectiveness of our approach in terms of the coverage and average bandwidth on the D shift dataset.

In Figure 1, we present the histograms of the coverage and the average bandwidth of our method, and a more general version of weighted conformal prediction in [TFBCR19] over 200 experiments (see Appendix B for details), which show that our method consistently yields a shorter prediction interval than the weighted conformal prediction while maintaining coverage. Over 200 experiments, the average coverage achieved by our method was 0.964029 (SD = 0.04), while the weighted conformal prediction method achieved an average coverage of 0.9535 (SD = 0.036). Additionally, the average width of the prediction intervals for our method was 13.654 (SD = 2.22), compared to 20.53 (SD = 4.13) for the weighted conformal prediction. Regarding the performance of intervals over 95% coverage, our method achieved this in 72 . 5% of cases with an average width of 14.35 (SD = 2.22). In contrast, the weighted conformal prediction method did so in 57% of cases with an average width of 21.4 (SD = 4.39). Boxplots are presented in Appendix B for further comparison.

5.1 Robustness of our method

One of the strengths of our approach lies in its resilience to the misspecification of certain components. The core idea behind our method is to combine multiple predictors to create a prediction interval that ensures sufficient coverage while maintaining a narrow average width. These individual prediction intervals may be based on estimators of conditional quantiles, means, variances, and other metrics. If some of the component prediction intervals exhibit poor performance, whether due to inadequate coverage or excessive width, our method typically assigns them lower weights, making it robust to their deficiencies. In contrast, other conformal methods that heavily depend on a single component tend to underperform, particularly with respect to average width. To illustrate this phenomenon, we evaluate our method under model misspecification using a simple one-dimensional simulation setup.

(a) Coverage of our method

(b) Bandwidth of our method

(c) Coverage of weighted conformal

(d) Bandwidth of weighted conformal

40

35

30

quno

25

20

15

10

5

0

0.80

0.85

0.90

0.95

1.00

Coverage--Our method

25

20

Count

15

10

5

0

0.80

0.85

0.90

0.95

1.00

Coverage--Weighted conformal

25

20

Count

15

10

5

0

10

12

14

16

18

20

22

24

Bandwidth--Our method

35 -

30

25

quno

20

15

10

5

0

15

20

25

30

35

40

45

Bandwidth--Weighted conformal

As mentioned in the previous subsection, our method aggregates six predictor intervals, including an estimator for the conditional variance function (Estimator 5). The weighted variance-adjusted conformal prediction interval (WVAC) relies on accurately estimating this conditional variance. We estimate the conditional variance using a random forest with varying depths ( { 3, 5, 7, 15 } ). For simulation purpose, we generate n = 2500 samples, keeping 75% as source data and resampling the remaining 25% with weighted samples proportional to w ( x ) ∝ (1 + exp ( -2 x )) -1 . As depth increases, overfitting leads to poor out-of-sample variance predictions. Table 1 summarizes our findings over 100 Monte Carlo iterations, showing that WVAC's average width increases with depth, while our method's average width remains stable.

6 Conclusion

This paper focuses on unsupervised domain shift problems, where we have labeled samples from the source domain and unlabeled samples from the target domain. We introduce methodologies for constructing prediction intervals on the target domain that are designed to ensure adequate coverage while minimizing width. Our analysis includes scenarios in which the source and target domains are related either through a bounded density ratio or a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. We further establish rigorous finite sample theoretical guarantees regarding the coverage and width of our prediction intervals. Finally, we demonstrate the practical effectiveness of our methodology through its application to the airfoil dataset.

References

[Can17] Luis Candanedo. Appliances Energy Prediction. UCI Machine Learning Repository, 2017. DOI: https://doi.org/10.24432/C5VC8G.

[CEN14] Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedings 10 , pages 231-240. Springer, 2014.

[CFHR17] Nicolas Courty, R´ emi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems , 30, 2017.

[CFT14] Nicolas Courty, R´ emi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14 , pages 274-289. Springer, 2014.

[CFTR16] Nicolas Courty, R´ emi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence , 39(9):1853-1865, 2016.

[CMSE22] Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In International Conference on Artificial Intelligence and Statistics , pages 2552-2573. PMLR, 2022.

[DG19] Dheeru Dua and Casey Graff. Uci machine learning repository. https://archive. ics.uci.edu , 2019.

[DGS21] Nabarun Deb, Promit Ghosal, and Bodhisattva Sen. Rates of estimation of optimal transport maps using plug-in estimators via barycentric projections. Advances in Neural Information Processing Systems , 34:29736-29753, 2021.

[DLAM + 21] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleˇ s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence , 44(7):3366-3385, 2021.

[DNWP22] Vincent Divol, Jonathan Niles-Weed, and Aram-Alexandre Pooladian. Optimal transport map estimation in general function spaces. arXiv preprint arXiv:2212.03722 , 2022.

[ERS + 22] Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von K¨ ugelgen, Hamed Hassani, George J Pappas, and Bernhard Sch¨ olkopf. Probable domain generalization via quantile risk minimization. Advances in Neural Information Processing Systems , 35:17340-17358, 2022.

[FGM23] Jianqing Fan, Jiawei Ge, and Debarghya Mukherjee. Utopia: Universally trainable optimal prediction intervals aggregation. arXiv preprint arXiv:2306.16549 , 2023.

[FVRA21] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020 , pages 877-894, 2021.

[GC21] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

[GSH + 08] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Sch¨ olkopf. Covariate shift by kernel mean matching. 2008.

[HKNC14] Mohammad Anwar Hosen, Abbas Khosravi, Saeid Nahavandi, and Douglas Creighton. Improving the quality of prediction intervals through optimal aggregation. IEEE Transactions on Industrial Electronics , 62(7):4420-4429, 2014.

[HL23] Xiaoyu Hu and Jing Lei. A two-sample conditional distribution test using conformal prediction and weighted rank sum. Journal of the American Statistical Association , pages 1-19, 2023.

[Hul94] Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence , 16(5):550-554, 1994.

[LBBH98] Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):22782324, 1998.

[LC21] Lihua Lei and Emmanuel J Cand` es. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(5):911-938, 2021.

[LGR + 18] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

[Mau16] Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27 , pages 3-17. Springer, 2016.

[MLJ + 22] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing , 469:28-51, 2022.

[MNW16] Katarzyna Maciejowska, Jakub Nowotarski, and Rafał Weron. Probabilistic forecasting of electricity spot prices using factor quantile regression averaging. International Journal of Forecasting , 32(3):957-965, 2016.

[MTOL20] Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning , pages 6672-6681. PMLR, 2020.

[NW15] Jakub Nowotarski and Rafał Weron. Computing electricity spot price prediction intervals using quantile regression and forecast averaging. Computational Statistics , 30:791-803, 2015.

[NWC + 11] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, page 7. Granada, Spain, 2011.

[PY09] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering , 22(10):1345-1359, 2009.

[Qin98] Jing Qin. Inferences for case-control and semiparametric two-sample density ratio models. Biometrika , 85(3):619-630, 1998.

[RHS17] Ievgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis of domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part II 10 , pages 737-753. Springer, 2017.

[RPC19] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

[SDF + 17] Vivien Seguy, Bharath Bhushan Damodaran, R´ emi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint arXiv:1711.02283 , 2017.

[SFS17] Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adaptation. Domain adaptation in computer vision applications , pages 153171, 2017.

[TFBCR19] Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems , 32, 2019.

[THSD17] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7167-7176, 2017.

[TX12a] Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and buildings , 49:560-567, 2012.

[TX12b] Athanasios Tsanas and Angeliki Xifara. Energy Efficiency. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C51307.

[USS + 16] Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from a density ratio estimation perspective. arXiv preprint arXiv:1610.02920 , 2016.

[VECP17] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 50185027, 2017.

[Vov15] Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence , 74:9-28, 2015.

[WKW16] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data , 3:1-40, 2016.

[WLL + 22] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing to unseen domains: A survey on domain generalization. IEEE transactions on knowledge and data engineering , 35(8):8052-8072, 2022.

[XLW + 20] Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 43944403, 2020.

[Yeh18] I-Cheng Yeh. Real Estate Valuation. UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C5J30W.

[YH18] I-Cheng Yeh and Tzu-Kuang Hsu. Building real estate valuation models with comparative approach through case-based reasoning. Applied Soft Computing , 65:260-271, 2018.

[YKT22] Yachong Yang, Arun Kumar Kuchibhotla, and Eric Tchetgen Tchetgen. Doubly robust calibration of prediction sets under covariate shift. arXiv preprint arXiv:2203.01761 , 2022.

[ZLQ + 22] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4396-4415, 2022.

[ZY21] Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering , 34(12):5586-5609, 2021.

[ZZP + 21] H Zhou, S Zhang, J Peng, S Zhang, J Li, H Xiong, and W Zhang Informer. Beyond efficient transformer for long sequence time-series forecasting., 2021, 35. DOI: https://doi. org/10.1609/aaai. v35i12 , 17325:11106-11115, 2021.

A Proofs

A.1 Proof of Theorem 3.2

First, we show that for our choice of ( ϵ, δ ) , as depicted in Theorem 3.2, f ∗ is a feasible solution of equation (3.5). Consider w 0 instead of ˆ w . By definition of f ∗ ,

This implies:

where the first equality follows from the fact that w 0 ( X ) 1 Y 2 >f ⋆ ( X ) = 0 a.s. on the source domain, the second equality follows from the fact that h δ ( t ) 1 t< -δ = 0 for all t , and the last inequality follows from the fact that h δ ( Y 2 i -f ⋆ ( X i )) ≤ 1 when Y 2 i -f ⋆ ( X i ) ≤ 0 . Since w 0 ( X ) 1 f ⋆ ( X ) -δ ≤ Y 2 ≤ f ⋆ ( X ) ≤ W , by Hoeffding's inequality, we have with probability at least 1 -e -t :

where L is upper bound on the density of Y 2 -f ∗ ( X ) . Call this event Ω 1 that the above bound holds. At this event we have:

where the last inequality follows from the fact that h δ ( t ) ≤ ( B + δ ) /δ if t ≤ B . Finally, to bound the last summand, we again apply Hoeffding's inequality. As ∥ ˆ w ∥ ∞ ≤ W ′ , we have with probability greater than or equal to 1 -e -t :

If we denote the event Ω 2 where the above inequality holds, then on the event Ω 1 ∩ Ω 2 , we have:

Furthermore,

Therefore, we conclude that with probability ≥ 1 -2 e -t , f ∗ is a feasible solution.

We now proof Theorem 2.2 on the event Ω 1 ∩ Ω 2 , when f ∗ is a feasible solution. Then we have, P n,T ( ˆ f init ( X )) ≤ P n,T ( f ∗ ( X )) on this event, by the optimality of ˆ f init in equation (3.5). Then we have:

Finally as f -f ∗ is upper bounded by F ′ = B F + ∥ f ∗ ∥ ∞ (as f is uniformly upper bounded by F). Therefore, by Mcdiarmid's inequality, we have with probability 1 -e -t :

Call this event Ω 3 . Furthermore, by standard symmetrization:

where R n T ( F f ∗ ) is the Rademacher complexity of F f ∗ . Therefore, on ∩ 3 i =1 Ω i , we have:

and P ( ∩ 3 i =1 Ω i ) ≥ 1 -3 e -t . This completes the proof.

A.2 Proof of Lemma 3.3

We prove the lemma into two steps; first we show that ˆ f init satisfies P T ( Y 2 > ˆ f init ( X )) ≤ τ with high probability for some small τ . Next we argue that, on D S, 2 , we have (2 /n S ) · ∑ i ∈D S, 2 ˆ w ( X i ) 1 ( Y 2 i ≥ ˆ f init ( X i )) ≤ ˇ τ with high probability for some small ˇ τ . Then as long as ˇ τ ≤ α , we conclude the proof of the lemma.

Step 1: Note that, by feasibility, ˆ f init satisfies:

This implies:

Now, as h δ ( Y 2 -f ( X )) ≤ ( B + δ ) /δ and w 0 ≤ W , we have by Mcdiarmid's inequality, with probability ≥ 1 -e -t :

Meanwhile, as in the proof of Theorem 3.2, with probability ≥ 1 -e -t :

Choosing t = 10log n S we obtain that with probability ≥ 1 -2 n -10 S :

We next bound the Rademacher complexity of R n S / 2 , F ( w 0 h δ ◦ f ) . By symmetrization, we have with ζ 1 , . . . ζ n S / 2 i.i.d. Rademacher (1 / 2) :

We first show that ϕ : R 2 → R is a Lipschitz function on its domain. The first argument of ϕ is w 0 ( x ) which lies within [ -W,W ] . The second argument of ϕ is Y 2 -f ( X ) (on the source domain), which is bounded by B . Therefore, h δ ( Y 2 -f ( X )) is bounded above by ( B + δ ) /δ . The derivative of h δ is 0 for x ≤ -δ and δ for x ≥ -δ . Hence, we have the following:

We next apply vector-valued Ledoux-Talagrand contraction inequality on the function ϕ (equation (1) of [Mau16]), to obtain the following bound on the Rademacher complexity:

Using this, we obtain the following:

E T ( h δ ( Y 2 -ˆ f init ( X ) )) ≤ ϵ + B + δ δ ( E S [ | ˆ w ( X ) -w 0 ( X ) | ] + (2 W + W ′ ) √ 5 log ( n S / 2) n S / 2 ) +4 √ 2 ( B + W + δ δ ) [ ∥ w 0 ∥ L 2 ( P XS ) + √ E S [ Y 4 ] √ n S + R n S / 2 ( F ) ] ≤ ϵ +4 √ 2 ( B + W + δ δ ) [ E [ | ˆ w ( X S ) -w ( X S ) | ] + (2 W + W ′ ) √ 5 log ( n S / 2) n S / 2 + ∥ w 0 ∥ L 2 ( P XS ) + √ E S [ Y 4 ] √ n S / 2 + R n S / 2 ( F ) ] ≤ ϵ +4 √ 2 ( B + W + δ δ ) [ E [ | ˆ w ( X S ) -w ( X S ) | ] + (2 W + W ′ ) √ 5 log ( n S / 2) n S / 2 + W + √ E S [ Y 4 ] √ n S / 2 + R n S / 2 ( F ) ] Choosing ϵ = Lδ + W √ 5 log ( n S / 2) n S / 2 + B + δ δ · ( E S [ | ˆ w ( X ) -w 0 ( X ) | ] + ( W + W ′ ) √ 5 log ( n S / 2) n S / 2 ) , we obtain E T ( h δ ( Y 2 -ˆ f init ( X ) )) ≲ Lδ + B + W + δ δ · ( ( E S [ | ˆ w ( X ) -w 0 ( X ) | ] + ( W + W ′ ) √ 5 log n S n S + R n S / 2 ( F ) ) ≲ √ √ √ √ L ( B + W ) ( ( E S [ | ˆ w ( X ) -w 0 ( X ) | ] + ( W + W ′ ) √ 5 log n S n S + R n S / 2 ( F ) ) +( E S [ | ˆ w ( X ) -w 0 ( X ) | ] + ( W + W ′ ) √ 5 log n S n S + R n S / 2 ( F ) (by choosing δ to balance the terms)

≜ τ

Call the above event Ω 1 . This completes the proof of Step 1.

Step 2: Coming back to D S, 2 , we have:

1 n S / 2 ∑ i ∈D S, 2 ˆ w ( X S,i ) 1 Y 2 i > ˆ f init ( X i ) ≤ 1 n S / 2 ∑ i ∈D S, 2 | ˆ w ( X i ) -w 0 ( X i ) | + 1 n S / 2 ∑ i ∈D S, 2 w 0 ( X i ) 1 Y 2 i > ˆ f init ( X i ) Furthermore, by Hoeffding's inequality, we have with probability ≥ 1 -e -t : 1 n S / 2 ∑ i ∈D 2 w 0 ( X i ) 1 Y 2 i > ˆ f init ( X i ) ≤ E S [ w 0 ( X ) 1 Y 2 > ˆ f init ( X ) ] + W √ t n S ≤ E S [ w 0 ( X ) h δ ( Y 2 -ˆ f init ( X ) )] + W √ t n S = E T ( h δ ( Y 2 -ˆ f init ( X ) )) + W √ t n S Meanwhile, with probability ≥ 1 -e -t : 1 n S / 2 ∑ i ∈D S, 2 | ˆ w ( X i ) -w 0 ( X i ) | ≤ E S [ | ˆ w ( X ) -w 0 ( X ) | ] + ( W + W ′ ) √ t n S .

Therefore, with t = 10log n S , we have with probability ≥ 1 -2 n -10 S :

Call this event Ω 2 . Therefore, on Ω 1 ∩ Ω 2 we have:

This completes the proof of Step 2. For any fixed α > 0 , we have ˜ τ ≤ α as long as n S is large enough and E S [ | ˆ w ( X ) -w 0 ( X ) | ] is small enough, and as a consequence ˆ λ ( α ) ≤ 1 . This completes the proof.

A.3 Proof of Theorem 3.4

Recall that we construct the prediction intervals using data splitting; from the first part of the data (namely D 1 ), we estimate ˆ f init and use the second part of the data (namely D 2 ) to estimate ˆ λ ( α ) . Conditional on D 1 , define a function class G ≡ G ( ˆ f ) as:

As G only depends on a scalar parameter λ (as w 0 and ˆ f init are fixed conditionally on D S, 1 , D T ), it is a VC class of function with VC-dim ≤ 2 .

Now, by the definition of ˆ λ ( α ) (see Step 2), we have:

We use a similar technique to control the second summand as in the proof of Theorem 3.2. By using the fact that the indicator function is less than one, we have:

Applying Hoeffding's inequality (with the fact that ∥ ˆ w ∥ ∞ ≤ W ′ and ∥ w 0 ∥ ∞ ≤ W ), we have with probability greater than or equal to 1 -e -t :

To control the third summand of (A.1), note that, conditional on D S, 1 and D T (i.e., assuming ˆ f init fixed), and using the fact that ∥ g ∥ ∞ ≤ ∥ w 0 ∥ ∞ ≤ W for all g ∈ G , we have by Mcdiarmid's

inequality with probability greater than or equal to 1 -e -t :

Now conditional on D S, 1 , D T , G is a VC class of function with VC dimension ≤ 2 . Therefore,

for some constant C > 0 . Thus, we have

Combining the bounds, we have, with probability ≥ 1 -2 e -t :

This completes the proof.

A.4 Proof of Theorem 4.3

We start with the following decomposition:

where the second equation follows from the fact that when X ∼ P T , then T 0 ( X ) ∼ P S , and the last line follows from the fact f ∈ F is L F Lipschitz. A similar argument as in the proof of Theorem 3.5 [FGM23] yields:

with probability ≥ 1 -e -t . We then finish the proofs.

A.5 Proof of Lemma 4.4

By the definition of ˆ λ ( α ) , we have

Now by an application of Chernoff bound for binomial distribution, we have:

Hence, we have the following:

We next establish the high probability bound on p n S . We define a function ℓ δ ( x ) which is 1 when x ≤ -δ , 0 when x ≥ 0 and -x/δ when -δ ≤ x ≤ 0 .

where the first inequality used ℓ δ ( x ) ≥ 1 ( x ≤ -δ ) , second inequality uses the fact that sample average of ℓ δ over D S, 1 is 0 by the definition of ˆ f init , third inequality uses Ledoux-Talagrand contraction inequality observing that ℓ δ is 1 /δ -Lipschitz. This completes the proof.

A.6 Proof of Theorem 4.5

We start with analyzing the first term:

Therefore, we need a high probability upper bound on P S ( Y 2 ≥ ˆ λ ( α )( ˆ f init ( X ) + δ ) | D S ∪ D T ) . Towards that end, we start with the following expansion:

Now, note that, by the definition of ˆ λ ( α ) , we have:

To bound the second term in (A.3), we use:

To bound the supremum we use standard techniques from the empirical process theory. Define a collection of functions G = { 1 Y 2 ≥ λ ( ˆ f init ( X )+ δ ) : λ ≥ 0 } . Note that, here we condition on D S, 1 , so we treat ˆ f init as a constant function. For notational simplicity, suppose

As the functions in G are uniformly bounded by 1 (and consequently, E [ g 2 ( X,Y )] ≤ 1 ), we have by Talagrand's concentration inequality of the suprema of the empirical process:

Therefore, we need an upper bound on Ψ n to obtain a high probability upper bound on Z n . Towards that end, observe that G is a VC class with VC-dim less than or equal to 2 (as it is an indicator function of a collection of functions with one parameter). Hence, we have, by symmetrization and Dudley's metric entropy bound:

Therefore, going back to (A.4), we have with probability ≥ 1 -e -t

Hence, we have:

with probability ≥ 1 -e -t . This completes the proof of T 1 . To obtain a bound on T 2 , note that:

Here, the penultimate inequality uses the fact that the conditional distribution of Y 2 T given X T is Lipschitz (as the density of Y 2 T given X T is bounded), and the last inequality uses the fact that ˆ f init is Lipschitz as we have assumed all functions in F are Lipschitz.

B Details of the experiment

B.1 Density ratio estimation via probabilistic classification

Suppose we observe { X 1 , . . . , X n 1 } from a distribution P (with density p ) and { X n 1 +1 , . . . , X n 1 + n 2 } from another distribution Q (with density q ). We are interested in estimating w 0 ( x ) = q ( x ) /p ( x ) , where we assume Q is absolutely continuous with respect to P

(otherwise, the density ratio can be unbounded with positive probability). Define, n 1 + n 2 mane binary random variables { C 1 , . . . , C n 1 + n 2 } such that C i = 0 for 1 ≤ i ≤ n 1 and C i = 1 for n 1 +1 ≤ i ≤ n 1 + n 2 . Consider the augmented dataset D = { ( X i , C i ) } 1 ≤ i ≤ n 1 + n 2 . We can think that this dataset is generated from a mixture distribution ρp ( X )+(1 -ρ ) q ( x ) where ρ = P ( C = 1) . For this mixture distribution, the posterior distribution of C given X is:

This implies:

Now, from the data, we can estimate ˆ ρ = n 2 / ( n 1 + n 2 ) and P ( C = 1 | X = x ) by any classification technique (e.g., using logistic regression, boosting, random forest, deep neural networks etc). Let ˆ g ( x ) be one such classifier. Then we can estimate w 0 ( x ) by ( n 1 /n 2 )(ˆ g ( x ) / (1 -ˆ g ( x ))) .

B.2 General weighted conformal prediction

The weighted conformal prediction method, as presented in [TFBCR19], consists of two main steps:

Split the source data into parts; estimate the conditional mean function E [ Y | X = x ] , say ˆ µ ( x ) using the first part of the source data.

Use the second part of the source data and the target data to construct weight w ( X i ) and the score function S ( x, y ) = | y -ˆ µ ( x ) | to construct the confidence interval.

In Section 5, we have implemented a generalized version of it, where we modify the score function as follows:

We estimate the conditional standard deviation function √ var ( Y | X = x ) along with the conditional mean function from the first part of the data. Call it ˆ σ ( x ) .

We use the modified score function s ( x, y ) = | y -ˆ µ ( x ) | / ˆ σ ( x ) .

The rest of the method is the same as [TFBCR19]. This additional estimated conditional variance function allows more expressivity and flexibility to the conformal prediction band, as observed in Section 5.2 of [LGR + 18], as this captures the local heterogeneity of the conditional distribution of Y given X .

B.3 Boxplots to compare coverage and bandwidth

In this subsection, we present two boxplots to compare the variation in coverage and average width of the prediction bands between our method and the generalized weighted conformal prediction (as described in the previous subsection).

40

35

30

25

20

15

10

sino

Weighted Conformal

(b) Coverage

1.00

0.95

0.90

0.85

0.80

Ours

Weighted Conformal

The boxplots immediately show that our methods yield similar coverage (even with lesser variability) with significantly lower average width than the generalized weighted conformal prediction method.

C Additional Experiments

In this section, we present the details of the additional experiments based on four other datasets, as mentioned in Section 5. We compare our methodology against two other widely used conformal methods, namely Weighted Variance-adjusted Conformal (WVAC) [TFBCR19], and Weighted Quantile-adjusted Conformal (WQC)-which is a variant of WVAC, where the score function is changed to the conditional quantile [RPC19].

C.1 Experiment 1: Real estate data

The Real Estate Valuation dataset available at the UCI Machine Learning Repository contains data used to estimate the value of real estate properties. Collected from Taipei, Taiwan, this dataset is useful for predictive modeling in the real estate sector. It consists of 414 instances, with each entry representing a different real estate transaction. This dataset was originally introduced in [TX12a]. In this dataset, the goal is to predict the house price per unit area based on the 6 other features, namely i) transaction date, ii) house age, iii) distance to the nearest MRT station, iv) number of convenience stores, v) latitude and vi) longitude. The construction of shifted data (with β = ( -1 , 0 , -1 , 0 , 1 , 1) ) and implementation procedure are the same as in Section 5. Table 2 and Figure 3 presented the results over 200 Monte Carlo iterations. It is evident from the table that our method produced a small average width in comparison to the other methods while maintaining the coverage guarantee.

C.2 Experiment 2: Energy efficiency data

The Energy Efficiency dataset available at the UCI Machine Learning Repository is designed to help predict the heating load and cooling load requirements of buildings. The dataset was originally introduced in [YH18]. The dataset includes various building parameters such as: i) wall area, ii) surface area, iii) roof area, iv) orientation, etc. The goal is to predict the heating load based on 8 other covariates. The construction of shifted data (with β = ( -1 , 0 , 1 , 0 , -1 , 0 , 0 , -1) ) and implementation procedure are the same as in Section 5. Our results over 200 Monte Carlo iterations are presented in Table 3 and Figure 4. The table shows that our method produced a smaller bandwidth than WVAC. While WQC has a smaller median bandwidth, it sacrifices coverage. The last row indicates that for experiments with coverage ≥ 95% , WQC's median average width is significantly larger than ours. Thus, whenever WQC provides adequate coverage, its bandwidth is much larger than ours.

C.3 Experiment 3: Appliances Energy Prediction Dataset

Appliances Energy Prediction Dataset is freely available from the UCI repository. This dataset is a time series data with 28 covariates and one response variable. We used data from 2016-01-11 to 2016-02-15 (5000 samples) as our training set and data from 2016-05-13 to 2016-05-27 (2000

samples) as our testing set. Since the source and target data are from different time periods, this experiment involves a non-synthetic real-world time shift. The results based on our Algorithm 2 are presented below:

C.4 Experiment 4: ETDataset

We applied our method to the ETDataset (ETT-small) from [ZZP + 21], which contains hourly-level data from two electricity transformers at two different stations, including load and oil temperature measurements. Each data point consists of 8 features, including the date of the point, the predictive value 'oil temperature', and 6 different types of external power load features. For our experiment, we used the data from one transformer during the period from July 1, 2016, to November 2, 2016, as our source data, and data from the same time period from the other transformer as our target data. As our source data and the target data are from different locations, we have a geographical covariate shift; see [ZZP + 21] for more details. Our results are as follows:

C.5 Experiment 5: Airfoil data

In Section 5, we have implemented our Algorithm 1 on the Airfoil data [TFBCR19] to showcase the efficacy of our method. Here, additionally, we implement Algorithm 2 on the same dataset to evaluate the performance of our second method based on optimal transport-based domain alignment. Here, the data shifting procedure (to create data from the target domain with a shifted distribution) is different from the rest of the experiments; we use the linear transformation x ↦→ Ax + b with A = diag (1 . 5 , 1 . 2 , 1 . 6 , 2 , 1 . 8) and b = (1 , 0 , 0 , 1 , 0) to generate covariates on the target domain. As before, we split the data into two parts; we keep 75% of the data as it is, and shift the rest 25% of the data. The results are given in Table 6 and Figure 5. The second column of Table 6, namely Our Method (Without OT) is the aggregation method proposed in this paper, without domain alignment

through optimal transport, i.e., we use source data to construct the prediction interval and use the same prediction interval on the target domain. As evident from Table 6, our method outperforms all other methods in terms of the average width of the prediction interval while maintaining a good coverage guarantee.

(a) Our-Width

(b) Our-Coverage

(c) WQC-Width

(d) WQC-Coverage

(f) WVAC-Coverage

Figure 3: Experiments on real estate data

(a) Our-Width

(b) Our-Coverage

(c) WQC-Width

(d) WQC-Coverage

(f) WVAC-Coverage

Figure 4: Experiments on Energy efficiency data

(a) Our-Width

(b) Our-Coverage

(c) Our(No OT)-Width

(d) Our(No OT)-Coverage

(e) WQC-Width

(f) WQC-Coverage

(g) WVAC-Width

(h) WVAC-Coverage

35

70

30

60

25

50

20

quno

40

15

30

10

20

10

20

40

Bandwidth-Our Method

60

80

100

120

总

0.60

0.65

0.70

Coverage-Our Method

0.75

0.80

0.85

0.90

0.95

1.00

20.0

120

17.5

100

15.0

12.5

80

10.0

60

7.5

40

5.0

2.5

20

0.0

BandwidthOur Method

6

0.5

0.6

Coverage--Our Method

0.7

0.8

0.9

1.0

30

20

25

15

20

10

15

10

30

40

Bandwidth-Weighted quantile conformal

50

60

70

80

90

100

0.80

Coverageweighted quantile conformal

0.85

0.90

0.95

1.00

50

40

20

30

15

10

20

10

Bandwidth-Weighted quantile conformal

4

5

0.80

Coverageweighted quantile conformal

0.85

0.90

0.95

1.00

25

20

20

quno

quno

15

10

10

20

Bandwidth-Weighited variance adjusted conformal

40

60

B0

100

120

0.825

Coverage-Wveighited variance adjusted conformall

1.000

30

30

S2

25

qunog

20

quno

20

15

15

10

10

2

BandwidthWeighted variance adjusted conformal

4

6

8

10

12

14

16

0.825

Coverage-Wweighted variance adjusted conformal

0.850

0.8750.9000.9250.9500.9751.000

20.0

16

20

17.5

20

14

15.0

12

qunog

15

12.5

qunog

SI

10

10

10.0

7.5

10

5.0

2.5

14

BandwidthOur method

16

18

20

0.0

0.86

0.88

Coverage--Our method

0.90

0.92

0.94

96°0

0.98

12

14

BandwidthOur method without OT

16

18

20

22

24

92

0.3

0.4

CoverageOur method without OT

0.5

0.6

0.7

0.8

0.9

1.0

25

25

40

20

30

20

35

30

quneg

15

15

10

15

10

20

15

10

10

5

15

20

BandwidthWeighted quantile conformal

30

35

40

45

50

0.60

0.65

0.70

CoverageWeighted quantile conformal

0.75

0.800.85

0.90

0.95

1.00

30

40

50

60

70

058'0

Coverage-Weighted variance adjusted conformal

0.875

0.900

0.925

0.950

0.975

1.000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 3, 4 and 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: We have proposed a method for aggregating prediction intervals along with theoretical guarantees and experiments in this paper. Our theory is valid under certain assumptions clearly mentioned in the main paper. One limitation of the theory is that we do not know the performance of our method if the assumptions are violated, as is true for any theory of methods. Furthermore, one needs to verify the methodology on various real datasets to under the applicability of our method, which is outside the scope of this paper. As these are quite standard limitations, we refrain from including them in the main draft. However, if the referees feel that this should be included in the main draft, we would be happy to oblige.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: Both methods in the experiment took approximately 10 minutes to run on a MacBook Pro laptop (with M2 Max CPU, 10 Cores, 32 GB RAM, and no GPU).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIt is crucial to assess and quantify uncertainties associated with distribution shifts in the context of unsupervised domain shift. The paper propose methodologies for aggregating prediction intervals, which capture the range of likely outcomes for a given prediction. The authors support these methodologies theoretically by considering a bounded density ratio and a measure-preserving transformation between source and target domains. This includes finite sample bounds for the coverage and width of their prediction intervals.\n\nSTRENGTHS:\n1.The method is innovative in aggregating various prediction intervals on the target domain under the domain-aligning assumption within an unsupervised domain adaptation framework.\n\n2.The paper includes sufficient theoretical discussions, presenting finite sample concentration bounds to demonstrate that their method achieves adequate coverage with a small width.\n\nWEAKNESSES:\n1.Apart from theoretical results, there is a lack of intuitive understanding as to why aggregation can achieve minimal-width prediction intervals.  \n\n2.The paper does not include comparisons with previous methods of aggregated prediction intervals (Hosen et al., 2014).\n\n3.Limited experiments and comparative methods make it difficult to affirm the effectiveness of the algorithm.\n\n[1] Hosen, A., Khosravi, A., Nahavandi, S., and Creighton, D. Improving the Quality of Prediction Intervals Through Optimal Aggregation. IEEE Transactions on Industrial Electronics. 2014\n\nQUESTIONS:\n1.What does $f \\in \\mathcal{F}$ signify in equation (2.1)? There is no $f$ explicitly used in the main part of this equation.\n\n2.What does $m_0$ represent in line 118? Is it the same as $m_0$ mentioned in line 137? What is the relationship between $m_0$ and $f_0$?\n\n3.In line 164, it states, \"However, any such estimator $ \\hat{\\omega}(x) $ can be non-zero for $ x $ where $ \\omega_0(x)=0 $ due to estimation error. Consequently, $ \\hat{\\omega} $ may not be efficient in selecting informative source samples.\" Given this observation, why not apply the hinge function not to $ \\hat{\\omega} $, but to $ (Y - m_0(x))^2 - f(x) $ in Equation (3.5)?\n\n4.The notation $B_{\\mathcal{F}}$ first appears in Theorem 3.2 (please correct me if I am mistaken), and there is no explanation provided for it.\n\n5.The paper introduces two algorithms, but Section 5 includes experiments for only one of them. Which algorithm is it, and why are there no experiments conducted on the other one?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
The authors study the problem of how to construct prediction intervals on a target domain under both covariate shift and domain shift assumptions (i.e., the source and target domains are related either via a bounded density ratio, or a measure-preserving transformation), designed to ensure adequate coverage while minimizing interval width.

They provide theoretical guarantees, with finite sample bounds, regarding the prediction interval coverage and width.

They apply their method on the airfoil dataset, comparing the performance with weighted split conformal prediction.

STRENGTHS:
- The paper is well-written overall, the authors definitely seem knowledgeable.

- The paper studies an important and interesting problem.

- The proposed method outperforms weighted split conformal prediction on one dataset.

WEAKNESSES:
- The experimental evaluation is limited to a single low-dimensional tabular dataset, with a synthetic/simulated distribution shift. It is not possible to determine if the proposed method actually would have significant real-world utility/impact.




Summary:
- Important problem and a well-written paper overall, but the experimental evaluation is too limited. I think this could be a solid paper, but I don't think it can be accepted in its current form.

QUESTIONS:
- Could you extend the experimental evaluation with more datasets and baseline methods?

- Why is the figure on page 9 missing a figure number and caption?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding on the work of Fan et al. (2023), this paper addresses the challenge of computing prediction intervals in an unsupervised domain adaptation setting, where labeled samples are available from a related source domain, and unlabeled covariates are available for the target domain. The primary objective is to generate prediction intervals for the target domain that provide both an asymptotic coverage guarantee and minimal width. The authors also explore the aggregation of prediction intervals on the target domain under both covariate shift and domain shift scenarios. Theoretical guarantees, including finite sample concentration bounds on approximation errors and coverage guarantees, are presented. The proposed methods are illustrated and compared with the weighted split conformal prediction method using a real-world dataset.\n\nSTRENGTHS:\n- The paper tackles the important and challenging problem of uncertainty quantification in the unsupervised domain adaptation setting.\n\n- The authors propose algorithms to generate prediction intervals with both coverage and minimal width guarantees.\n\n- Experimental results show that the proposed method generates prediction intervals with comparable coverage but smaller average width compared to an existing method.\n\nWEAKNESSES:\n- The paper assumes familiarity with Fan et al. (2023), making it difficult for readers who have not read that work to understand some notations and concepts. Section 2.1 on problem formulation is particularly hard to read without prior knowledge.\n\t- There is a lack of flow in the paper, with some explanations and definitions missing or unclear. For instance, terms like \"valid prediction interval\", and expressions such as U(x) and l(x) are not clearly explained.\n\t- The authors write \"This translates into solving the following optimization problem:\" without any explanations. \n\t- Another example is Line 234, \"adding a slight delta to ensure coverage even when F is complex.\". This is not clear. It is discussed in Fan et al (2023) in (2.4). \n\t- Even if simple, I suggest adding important details about different expressions in the appendix to help the reader.\n\n\n- The title mentions \"optimal aggregation,\" but model aggregation is not the central focus of the paper. It is first discussed in Remark 4.2 and briefly mentioned in line 272.\n\n- The paper should highlight the similarities and differences with Fan et al. (2023) in terms of contributions, both theoretical and empirical.\n\nAlso, the authors use different quantile levels for estimators compared to Fan et al. (2023) without explaining the reason for these differences.  In Fan et al (2023), the following quantile levels are considered: 0.05, 0.35, 0.65, and 0.95, while the authors use 0.85, 0.95, 0.9, and 0.9.\n\n\n- The statement that \"the shape of the prediction band does not change much if we change the level of coverage\" lacks clarity on scenarios where this might not be true, such as multimodal distributions (?).\n\n\n- The discussion on lambda <=1 is confusing and needs improvement. The paper inconsistently explains the shrinkage factor and its implications. For example, the authors write \"Therefore, it is not immediate whether the shrinkage factor  is smaller than 1, i.e., whether we are indeed shrinking the confidence interval (lambda > 1 is undesirable, as it will widen finit , increasing the width of the prediction band).\" and \"Although ideally lambda <= 1, it is not immediately guaranteed as we use separate data  for shrinking.\". The authors also discuss it in Lemma 3.3. and Lemma 4.4. \n\n\n- The term \"shape of the prediction band\" with prediction intervals can be confusing as it can have other meanings (see functional data). I suggest the authors clarify this in line 147.\n\n\n- The paper ignores a significant amount of research on prediction interval estimation. I suggest discussing important related work.\n\n- The limitations of the paper are not discussed. For example, focusing solely on intervals might have implications under multimodal distributions, where High-Density Regions (HDR) would be the smallest regions.\n\n- The relationship between the proposed approach and conformal prediction is not clearly explained. The paper should discuss the guarantees provided by the proposed approach compared to conformal prediction, including the relevant theoretical guarantees and calibration aspects.\n\nFor example, the weighted conformal approach also splits the data and estimates a density ratio.  What about the (weaker?) guarantees you provide? The following paper could also be useful: \"On the expected size of conformal prediction sets\".\n\nAlso, in conformal prediction, in addition to finite-sample marginal guarantees, the authors often provide asymptotic conditional guarantees. What can you say about this with your approach?\n\n\n- The comparison with the conformal approach in the experiments might not be fair, as conformal prediction does not optimize for the smallest width. Additionally, reporting the interval score, which is a proper scoring rule for intervals, would be useful for evaluating the methods.\n\n\n- Typos and clarifications:\n\t- line 137: \"remains same.\"\n\t- line 154: \"via by solving\"\n\t- line 453 \"a.s. on source\". On the source domain?\n\t- line 470, \"we with have\"\n\t- It is not \"Leduox-Talagrand\" but Ledoux-Talagrand\".\n        - line 481, a minus sign is missing\n\t- Line 490 to 492. It is not clear why tilde tau <= alpha.\n\t- Line 182, what does \\mathcal{F} - f* mean?\n\t- About Theorem 3.4, we expect the performance (or convergence rates) to also depend on alpha. Where does this appear in the theoretical guarantees?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe limitations are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a method to construct prediction intervals in distribution shift problems, where unlabeled data from the target domain is available. The method is inspired by Fan et al (2023) in the i.i.d. setting and assumes we can estimate a transform (defined by reweighting or a transport map) that maps the joint distribution from the source domain to the target.\n\nSTRENGTHS:\n- The problem studied is relevant.\n- The method appears reasonable and sound.\n\nWEAKNESSES:\n- My main concern is about the novelty of this work: given the assumed condition on domain shift (namely a good approximation to the reweighting or transport map is readily available) it is a natural application of the method in Fan et al (2023).  The proof will require some additional effort but given the assumptions I do not expect them to be technically challenging.\n\n- While the paper employed assumptions on domain shift are common in literature, I am uncertain about their relevance, especially in high-dimensional problems.  It is well-known that the bounded density ratio assumption in Section 3 is unrealistic in high dimensions. And while I have come across past works on the transport map assumption, I do not know of realistic examples where the assumption actually holds so that the coverage guarantees established in this work could be relevant: it is much easier to think of scenarios where the assumption does not necessarily hold, but heuristic methods based on this assumption achieve better predictive performance; the latter is more often the goal in previous works.\n\n- Finally, I have doubts about the novelty/utility of Fan et al (2023), which seems quite similar to conformal prediction.  It could be alternatively summarized as applying quantile estimation to the \"conformity score\" of $(y - \\hat m(x)) / f(x)$, and compared with conformal prediction the main difference is the lack of sample splitting (in split conformal) or leave-one-out operations (as in full conformal).  I wonder if the removal of sample splitting is really a wise choice: the coverage guarantees in Fan et al (2023) and this work relies on the Rademacher complexity of the function class for $f$.  If the function class is defined by general ML models the guarantees would be rather unreliable in high-dimensional problems, in sharp contrast to the distribution-free guarantees in (split/full) conformal prediction.  Otherwise, we would need further sample splitting to construct a finite set of candidates for $f$, in which case we could just do split conformal prediction.\n\nQUESTIONS:\nGiven the above questions on the theoretical motivation, novelty or significance, I believe there should be extensive empirical evaluation for the proposed method.\n\nIn light of question 3 above, I also wonder if the observed performance difference between the proposed method and conformal prediction could be attributable to the choice of different conformity scores; how does the method compare to an implementation of split conformal prediction using the same $(y - \\hat m(x)) / f(x)$ as the conformity socre?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}