Title: Robust Conformal Prediction under Joint Distribution Shift
Venue: NeurIPS 2024
Accepted: no
Keywords: conformal prediction

data distribution shift

coverage robustness
Abstract: Uncertainty prevails due to the lack of knowledge about data or model,
and conformal prediction (CP) predicts multiple potential targets,
hoping to cover the true target with a high probability. 
Regarding CP robustness,
importance weighting can address covariate shifts,
but CP under joint distribution shifts remains more challenging.
Prior attempts addressing joint shift via $f$-divergence
ignores the nuance of calibration and test distributions
that are critical for coverage guarantees.
More generally, with multiple test distributions shifted from the calibration distribution,
simultaneous coverage guarantees for all test domains requires a new paradigm.
We design Multi-domain Robust Conformal Prediction (mRCP) that first formulates the coverage difference that importance weighting fails to capture under any joint shift.
To squeeze such coverage difference and guarantee the $(1-\alpha)$ coverage in all test domains,
we propose Normalized Truncated Wasserstein distance (NTW) to comprehensively capture the nuance of any test and calibration conformal score distributions, and design an end-to-end training algorithm incorporating NTW to provide elasticity for simultaneous coverage guarantee over distinct test domains.
With diverse tasks (seven datasets) and architectures (black-box and physics-informed models), NTW strongly correlates (Pearson coefficient=0.905) with coverage differences beyond covariate shifts, while mRCP reduces coverage gap by 50% on average robustly over multiple distinct test domains.

Abstract

Uncertainty prevails due to the lack of knowledge about data or model, and conformal prediction (CP) predicts multiple potential targets, hoping to cover the true target with a high probability. Regarding CP robustness, importance weighting can address covariate shifts, but CP under joint distribution shifts remains more challenging. Prior attempts addressing joint shift via f -divergence ignores the nuance of calibration and test distributions that are critical for coverage guarantees. More generally, with multiple test distributions shifted from the calibration distribution, simultaneous coverage guarantees for all test domains requires a new paradigm. We design Multi-domain Robust Conformal Prediction (mRCP) that first formulates the coverage difference that importance weighting fails to capture under any joint shift. To squeeze such coverage difference and guarantee the (1 -α ) coverage in all test domains, we propose Normalized Truncated Wasserstein distance (NTW) to comprehensively capture the nuance of any test and calibration conformal score distributions, and design an end-to-end training algorithm incorporating NTW to provide elasticity for simultaneous coverage guarantee over distinct test domains. With diverse tasks (seven datasets) and architectures (black-box and physics-informed models), NTW strongly correlates (Pearson coefficient = 0.905) with coverage differences beyond covariate shifts, while mRCP reduces coverage gap by 50% on average robustly over multiple distinct test domains.

1 Introduction

The growing data volume, enhanced computation capability, and advanced models significantly 21 improve machine learning predictive accuracy. Nevertheless, noises, unobservable factors, and the 22 lack of knowledge lead to uncertainty that stakeholders should ponder along model predictions 23 when making decisions particularly in areas such as fintech [25], autonomous driving [2], traffic 24 forecasting [4], and epidemiology [32, 27]. Conformal Prediction (CP) addresses uncertainty by 25 predicting a set of possible target(s) rather than a single guess [31]. Specifically, CP computes 26 conformal scores (residuals between predicted and true targets for regression tasks) of a trained 27 model f on a calibration set, and calculates the 1 -α quantile q of these scores. For any input x , CP 28 produces the smallest prediction set C ( x ) consisting of target values whose conformal scores are less 29 than q . Assuming that the test and calibration data are exchangeable (including i.i.d.), the true target 30 y is guaranteed to be covered by C ( x ) with at least 1 -α probability. 31

̸

In practice, calibration distribution P XY and test distribution Q XY may differ thus P XY = Q XY , 32

termed as joint distribution shift and violate the exchangeability assumption. Joint shift can occur 33

̸

with either covariate shift ( P X = Q X ) or concept shift ( P Y | X = Q Y | X ), though what causes a 34

̸

joint shift is difficult to infer from the observed data only. With importance weighting, covariate 35

shift is shown not to affect the coverage confidence guarantee [29]. To address CP under joint 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

Weighted calibration conformal score CDF,

෠

𝐹𝑄/𝑃

Training Set 1

Training Set M

Calibration Set

Model

𝑓 𝜃

Normalization and Truncation

Robustness Penalty (NTW)

mRCP Loss

Prediction Residual

…

…

Optimization

𝑝(𝑋)

𝑝(𝑌|𝑋)

𝑋

𝑝(𝑋)

𝑌

𝑝(𝑌|𝑋)

…

𝑋

𝑝(𝑋)

𝑝(𝑌|𝑋)

𝑌

Test Domain 1

𝑋

𝑝(𝑋)

𝑝(𝑌|𝑋)

𝑌

Test Domain M

(a)

(b)

(c)

(d)

𝑋

𝑝(𝑋)

𝑝(𝑌|𝑋)

𝑌

Test Domain 2

Calibration conformal score CDF,

෠

𝐹𝑃

Test domain conformal score,

෠

𝐹𝑄

𝑣

Cumulative Probability

𝑞

𝑞 ∗

𝐷 joint

𝐷concept

𝐷 covariate

(1-𝛼)(𝑛+1)

𝑛

෠

𝐹𝑃

𝑞

෠

𝐹𝑄

𝑞

Weighted Calibration conformal score

Test domain 1 conformal score

Test domain 2 conformal score

𝑞2

(

highest

)

Cumulative Probability

confidence

𝑣

overestimated coverage

Wasserstein-1 distance between weighted calibration

and test domain 2

Wasserstein-1 distance between  weighted calibration

and test domain 1

Total variation

𝑞1

𝑣

𝑝(𝑣)

Calibration & Test Domain 1

𝑝(𝑣)

𝑣

Calibration & Test Domain 2

෠

𝐹𝑄/𝑃

𝑞 ∗

෠

𝐹𝑄

𝑞 ∗

Calibration Domain

̸

shift,

̸

-divergence is adopted in [33, 6] to measure the difference between

f

the corresponding conformal scores distributions. However,

XY

XY

̸

and or

P

Q

-divergence ignores where the two

f

distributions differ, which quantiles and coverage guarantees depend on (Figure 1, (c)). When test data are sampled from multiple distinct test distributions

to ensure simultaneous

(

e

)

XY

1

M

, it is desired

Q

, e

=

e

, ..., e

∈ E

{

}

coverage for all test distributions. Previous work selects the highest

1

α

-

1

quantile from all test distributions and constructs

α

-

producing excessively large set

(

e

X

1

M

for

,

C

(

x

)

x

Q

,

e

=

e

, ..., e

∈

∀

∈ E

{

}

. Selecting other quantiles may lead to smaller coverage on a

C

(

x

)

test domain than was expected during calibration, leading to prediction overconfidence. Without a new paradigm to guarantee coverage under multiple shifted test distributions, the dilemma between

CP coverage efficiency and confidence guarantee seems unavoidable.

̸

We first decompose the coverage difference under any joint distribution shift to a component due to covariate shift ( P X = Q ( e ) X , addressed by importance weighting [29]) and that due to concept shift ( P Y | X = Q ( e ) Y | X ). We propose Normalized Truncated Wasserstein distance (NTW) to robustly capture where the test and importance-weighted calibration conformal score cumulative density function (CDF) deviate (Figure 1, (b)). We design Multi-domain Robust Conformal Prediction (mRCP) by minimizing all NTW terms over E = { e 1 , ..., e M } during model training (Figure 1, (d)) to elastically guarantee coverage confidence for all test domains. Experiments on regression tasks on seven datasets demonstrate that: 1) NTWwell-correlates with the coverage difference after importance weighting (Pearson coefficient 0.905); 2) mRCP provides conformal predictions that reduce average coverage difference by 50% compared to baselines under multiple joint shifts; 3) mRCP is sufficiently general to address joint distribution shifts even after incorporating domain knowledge when available.

2 Background and related work 58

2.1 Conformal prediction 59

Let x ∈ X and y ∈ Y denote the input and output random variable, respectively, where X and 60 Y ⊆ ❘ is the input and output space, respectively. On X × Y , the calibration domain is defined by a 61

)

̸

̸

̸

joint distribution P XY , and we consider a calibration set S c = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } are drawn 62 i.i.d. from P XY . Similarly, a test set S t = { ( x 1 , y 1 ) , . . . , ( x m , y m ) } is drawn i.i.d. from test domain, 63 which is defined by a joint distribution Q XY . 64

With a trained regression model f , the conformal score v i = v ( x i , y i ) = | f ( x i ) -y i | is the residual 65 between the predicted target f ( x i ) and the true target y i . The set of calibration conformal scores is 66 denoted as V c = { v ( x i , y i ) | ( x i , y i ) ∈ S c } . Let q be the ⌈ (1 -α )( n +1) ⌉ /n quantile of V c : 67

where δ v i represents the point mass at v i (i.e., the distribution placing all mass at the value v i ). 68 Quantile (1 -α, F ) := inf { z | Pr ( Z < z ) ≥ 1 -α } and F is the CDF of Z . With the quantile q , the 69 CP prediction set of an input x from S t is 70

Most CP methods, such as[22, 23], rely on the assumption of exchangeability, which is relaxed from the i.i.d. assumption [31]. In our scenario, if the calibration and test samples are drawn from the identical joint probability distribution ( P XY = Q XY ), these calibration and test samples are i.i.d. Under this assumption, the probability that the true target y is included in C ( x ) is at least 1 -α , which is called coverage guanrantee , or more formally,

2.2 Conformal prediction under domain shift

̸

Covariate shift ( P X = Q X ) means marginal distributions between the calibration and test domains are different. CP under covariate shift is addressed using importance weighting [29]. Under a probabilistic view, [14] defined the covariate shift as a bounded perturbation on any test input and developed adaptive probabilistically robust CP. The condition of multiple test domains is discussed in [15], and similar topics include coverages under feature-stratification [7, 11].

̸

̸

Joint distribution shift ( P XY = Q XY ) indicates at least one of covariate shift ( P X = Q X ) and concept shift (different conditional distributions, P Y | X = Q Y | X ) will occur [17]. This shift is more general and the importance weighting method cannot address changes in conditional distribution. With M test domains E = { e 1 , ..., e M } , each e ∈ E is defined by a joint distribution Q ( e ) XY and holds a joint shift with calibration domain P XY (i.e., P XY = Q ( e ) XY ). Considering this condition, previous works, such as [6, 33], presume all test domains fall in a predefined f -divergence range, calculate confidence-specified quantile of each test domain, and apply the highest quantile to all domains. This method causes excessively high coverages and thus overlarge prediction sets, which reduces prediction efficiency because smaller prediction sets can help locate true targets better.

̸

3 Conformal prediction under joint distribution shift

3.1 Decomposition of coverage difference

We decompose the coverage difference between a calibration domain P XY and a test domain Q XY under joint distribution shift at a user-specified confidence (1 -α ) .

Similar to V c , we define the test conformal score set V t = { v ( x i , y i ) | ( x i , y i ) ∈ S t } . With the indicator function ✶ , empirical CDFs of calibration and test conformal scores are

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

With given 1 -α confidence, quantile q is calculated in Eq. (1), and the coverage difference under a 97 joint distribution shift can be quantified as 98

[29] employs importance weighting for CP under covariate shift. Specifically, if the ratio of test 99 to calibration covariate likelihoods, Q X /P X , is known, a calibration conformal score v i ∈ V c is 100

̸

weighted by p i = w ( x i ) / ∑ n j =1 w ( x j ) , where w ( x i ) = Q X ( x i ) /P X ( x i ) . Therefore, the empirical 101 CDF of weighted empirical calibration scores is given by 102

where the subscript Q/P indicates conformal scores of calibration domain P is weighted by conformal scores of test domain Q . The confidence-specified quantile of the weighted calibration conformal scores is

As importance weighting ensures the 1 -α coverage as though covariate shift were absent, coverage difference D covariate caused by covariate shift is the gap between the coverages under test conformal score CDF using quantiles on unweighted and weighted calibration conformal score distributions.

Importance weighting can not address CP under joint shift as it fails to capture changes in conditional probability distribution caused by concept shift, thus we present the coverage difference caused by concept shift is

which is remaining coverage difference after applying importance weighting. Here we assume ˆ F P ( q ) = ˆ F Q/P ( q ∗ ) , so we can rewrite D concept by

The error bound for the assumption is quite small especially when the calibration set size n is large. The detailed proof is provided in Appendix B. We denote D concept as D for simplification.

3.2 Normalized Truncated Wasserstein distance

To develop a metric that is independent of confidence level and can quantify the overall closeness between weight calibration and test conformal scores, we estimate the expected coverage difference under concept shift as

based on the approximation in Eq. (9), where ❊ indicates the expectation function.

Definition 1 (Wasserstein-1 Distance) . If F 1 and F 2 are two cumulative distribution functions (CDFs), the Wasserstein-1 distance, d W , is quantified by the area between F 1 and F 2 .

Applying Wasserstein-1 distance (W-distance) in Eq. (11) to ˆ F Q and ˆ F Q/P , we get 123

As we define conformal scores as the residuals between predicted and true targets, they are always 124 positive, so we only need to integral from 0 to ∞ in Eq. (12). 125

126

127

We assume V c is sorted. As both ˆ F Q and ˆ F Q/P are empirical CDFs, we can approximately represent d W ( ˆ F Q , ˆ F Q/P ) in a discrete form as

Eq. (13) shows d W ( ˆ F Q , ˆ F Q/P ) can be estimated as a weighted summation of | ˆ F Q ( v i ) -ˆ F Q/P ( v i ) | 128 for v i ∈ V c \{ v n } with the corresponding weight v i +1 -v i . Also, Eq. (10) indicates that ❊ [ D ] can 129 be regarded as the weighted summation of | ˆ F Q ( v i ) -ˆ F Q/P ( v i ) | for v i ∈ V c with weight 1 /n . The 130

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

131

132

133

134

135

136

137

138

139

140

141

similarity between Eq. (13) and Eq. (10) allows us to apply the W-distance between the test and weighted calibration conformal score to capture expected coverage difference under concept shift.

Care needs to be taken for Eq. (13) to make this metric more robust. At first, we expect the weights v i +1 -v i to be approximately equal, as weights in Eq. (10) are constants 1 /n . However, some outlier calibration conformal scores have large distances from their neighbors, causing involved weights much higher than 1 /n . These outlier scores are represented as a long tail of ˆ F Q/P when it converges to 1. Therefore, it is necessary to establish a partition threshold to truncate the long tail. We calculate the partition threshold

which is the smallest calibration conformal score whose coverage is greater or equal to a user-defined value 1 -σ . In contrast to the original d W ( ˆ F Q , ˆ F Q/P ) integrated on the set of real numbers, the truncated form is integrated from 0 to v σ as

Secondly, as the summation of weights in Eq. (10) is 1, we also need to divide each v i +1 -v i 142 by v σ -v 1 . When the calibration set is large enough, it is plausible to assume the existence of a 143 calibration sample fitting the trained model f very well, causing the smallest calibration conformal 144 score v 1 ≈ 0 . Therefore, this normalized can be formulated as 145

Alower d NTW indicates more similarity between ˆ F Q/P and ˆ F Q , thus leading to more robust conformal 146 prediction in the test domain. As a result, NTW enables us to assess the expected coverage difference 147 due to concept shift in Eq. (10). Experiment results in Section 5 and Appendix E show the necessity 148 of truncation and normalization. We also prove that the W-distance between the test and weighted 149 calibration conformal score population CDF can establish an upper bound for coverage difference 150 under concept shift in Appendix C. 151

152

153

154

155

156

157

158

159

160

161

162

163

164

4 Multi-domain robust conformal prediction

If a calibration set S c , and a test set S t are drawn from a domain P XY , the i.i.d. assumption is satisfied, and the coverage guarantee in Eq. (3) holds for ( x, y ) ∈ S t .

The domain P XY can be decomposed into M multiple domains, denoted as E = { e 1 , ..., e M } .

However, for e ∈ E , denote S ( e ) t a test set drawn from Q ( e ) XY , then the coverage guarantee may no longer hold for ( x, y ) ∈ S ( e ) t , because joint distribution shift may occur between P XY and Q ( e ) XY . It indicates CP can be overconfident and underconfident for samples from different Q ( e ) XY , resulting in prediction biases.

Inspired by the works of multi-domain generalization [26, 18, 19, 1], we propose Multi-domain Robust Conformal Prediction (mRCP) to make the coverage approach confidence in all domains, using a training set S ( e ) from the data distribution Q ( e ) XY for e ∈ E and a calibration set S c from P XY .

The objective function of mRCP includes two components. First, for the minimization of prediction residuals, denoting l a loss function, Empirical Risk Minimization (ERM) [30] is incorporated as

Secondly, we aim for robust conformal prediction on each domain during testing, seeking a low value 165 of ❊ [ D ] in Eq. (10) across test domains, so mRCP needs to address coverage differences due to 166 covariate and concept shifts simultaneously. To remove coverage differences due to covariate shifts, 167

it applies importance weighting to each domain e ∈ E during training and obtains ˆ F Q ( e ) /P , which is 168 the calibration conformal score CDF weighted by Q ( e ) XY . 169

170

171

172

173

174

175

176

177

178

179

Besides, as we have a training set S from domain Q XY , an empirical CDF of conformal scores in Q ( e ) XY can be computed, denoted as ˆ F tr Q ( e ) . NTWquantifies the expected coverage difference caused by concept shift between ˆ F Q ( e ) /P and training conformal score CDF ˆ F tr Q ( e ) . Combining these two components, the objective function of mRCP is

( e ) ( e )

where β is a hyperparameter balancing these two parts. mRCP algorithm is shown in Algorithm 1.

Algorithm 1 Multi-domain Robust Conformal Prediction

Require: M training sets S ( e ) , e ∈ E ; one calibration set S c ; N training epochs; model f θ ; partition value σ ; loss function l ; penalty hyperparameter β .

5 Experiment

In this section, we validate NTW in Eq. (16) as a good indicator of expected coverage difference due to concept shift and demonstrate the effectiveness of mRCP in obtaining coverage robustness across different test domains.

5.1 Datasets and models

We conducted experiments across various datasets: (a) the airfoil self-noise dataset [5]; (b) Seattle180 loop [9], PeMSD4, PeMSD8 [16] for traffic speed prediction; (c) US-Regions, US-States, and 181 Japan-Prefectures [10] for epidemic spread forecasting. The airfoil dataset was manually altered to 182 create three subsets demonstrating covariate and concept shifts. 24 domains for the traffic datasets 183 were designated based on data generation hours, while epidemic dataset instances were categorized 184 into four domains reflecting different pandemic stages. A multilayer perceptron (MLP) with a (input 185 dimension, 64, 64, 1) architecture was utilized for all datasets. Traffic and epidemic prediction tasks 186 were also trained on corresponding physics-informed partial differential equations (PDEs), which are 187

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

5.2 Experiments of NTW

For each of the experiment setups, a training set, a validation set, and a test set were sampled from each Q ( e ) XY for e ∈ E . One calibration set was sampled from P XY which is a mixture probability distribution of Q ( e ) XY for e ∈ E , as shown in Eq. (17). To validate NTW is a good indicator of ❊ [ D ] , we only need to use ERM in Eq. (18) to train the model f θ , which can be an MLP or a PDE. The loss function l is the ℓ 1 norm, as same as how we compute conformal scores.

After training, for e ∈ E , we first calculated the NTW between the calibration conformal score CDF weighted by Q ( e ) X /P X , and validation conformal score CDF of Q ( e ) X . Denote the NTW of domain e as d ( e ) NTW . Then, we estimated the expected coverage difference caused by concept shift on a test domain e , denoted as ❊ α [ D ( e ) ] , using the coverage difference expectation between the test and weighted calibration conformal score CDFs on a 1 -α confidence set { 0 . 1 , ..., 0 . 9 } .

❊ α [ D ( e ) ] and d ( e ) NTW should have a positive correlation for e ∈ E , proving NTW can capture the expected coverage difference caused by concept shift.

Baselines: We select six baseline metrics to validate the effectiveness of NTW. Total variation d TV [13], and Kullback-Leibler (KL) divergence d KL [21] are chosen as two typical f -divergence metrics. Expectation difference ∆ ❊ [19] is selected since it is a widely applied generalization metric. We also measure standard, normalized, and truncated W-distance, denoted as d W, d NW, and d TW respectively, to demonstrate applying normalization and truncation together is necessary.

Metric: We apply the Pearson coefficient to quantify the correlations between metrics and the coverage difference expectation. It measures the linear correlation between two values by giving a value between -1 and 1 inclusive. 1,0, and -1 indicate perfect positive linear, no linear, and negative linear correlations, respectively. Therefore, if the Pearson coefficient of a metric is higher , this metric can indicate the expected coverage difference better . We provide a detailed definition of the Pearson coefficient in Appendix E.

Results: Table 1 illustrates the Pearson coefficients between NTW and the coverage difference expectation among seven datasets and different models, compared with the other six baseline metrics. We highlight that NTW keeps holding the largest Pearson coefficient among all experiment setups, which means the proposed metric can keep indicating the coverage difference expectation. Specifically, the coefficients of total variation d TV and KL divergence d KL fluctuate along experiments, meaning that they can not truly indicate the coverage difference expectation. ∆ ❊ can not capture the coverage difference expectation either. Lastly, due to the lack of robustness to score scales and outliers, standard, normalized, and truncated W-distance, denoted as d W, d NW, and d TW respectively, can

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

not indicate the coverage difference expectation as well as d NTW. It also displays the average and standard deviation of the Pearson coefficient of the proposed NTW and six baselines. NTW not only has the highest average Pearson coefficient but also has the lowest standard deviation, which means the correlation between NTW and the coverage difference expectation caused by concept shift is very stable. In Figure 3 and Figure 4, we also visually show the correlation between the expected coverage difference under concept shift and each metric. We refer to Appendix E for detailed analysis. This observation suggests the potential of incorporating NTW in the training process, leading to the development of the mRCP approach. By applying the NTW metric, mRCP aims to enhance coverage robustness in test domains.

5.3 Experiments of mRCP

Since we prove NTW can assess expected coverage difference under concept shift effectively, mRCP is designed to minimize it during training. In this case, validation sets are unnecessary, and we only draw training, and test sets from Q ( e ) XY . Again, we draw one calibration set from P XY . The model f θ can also be an MLP or PDE based on different experiment setups. The loss function l is the ℓ 1 norm. We implement mRCP according to Algorithm 1.

Baselines: Two methods of optimization with out-of-distribution data are selected as baselines. DRO in Eq. (20) by [26] follows the minimax principle to reduce the highest L ( e ) to obtain fair prediction among test distributions. On the other hand, V-REx in Eq. (21), introduced by [18], focuses on reducing the variance of L ( e ) to obtain fairness. As we include importance weighting in mRCP, we do not take it as a baseline, and the effectiveness of importance weighting is discussed in Section 6.

Metric: Denote ❊ ′ e [ ❊ α [ D ( e ) ]] the expectation of coverage difference over confidence levels and test domains and ❊ ′ e [ L ( e ) ] the expectation of prediction residual over test domains. The two expectations become smaller means the algorithm's performance is better . Both values are normalized by the corresponding results from the same experiment setup trained by ERM. Changing the weight β in Eq. (19) will draw a Pareto front, thus we want the Pareto front closer to the origin. Since V-REX is also controlled by a hyperparameter, we draw Pareto fronts for it as well.

Result: Figure 2 displays the Pareto fronts for mRCP, DRO, and V-REx, highlighting the trade-offs between prediction residual and coverage difference expectation across different models and datasets. Figure 2, (a) shows the results for the airfoil self-noise dataset when trained with a Multilayer Perceptron (MLP) model. The mRCP method achieves a more favorable Pareto front compared to V-REx, indicating a better balance between prediction residual and coverage difference expectation. Additionally, mRCP attains a lower normalized coverage difference expectation than DRO at a comparable level of the prediction residual. In Figure 2, (b), we observe the experiment results on the epidemic spread prediction task using three epidemic datasets. With the same MLP architecture, mRCP delivers superior Pareto fronts relative to the baselines. When employing the epidemic PDE, the SIR model only has two trainable parameters, so their data points can not compose Pareto curves due to the model's limited flexibility. Thus, we show the average of these points. Despite this limitation, mRCP maintains its advantage over the baseline methods. Figure 2, (c) and (d) present results from the traffic prediction task on three different traffic datasets. Here, the Pareto curves for both the MLP and the reaction-diffusion (RD) PDE model are well-defined, because RD model with six parameters, offers greater adaptability, allowing for clearer Pareto fronts. Overall, Figure 2 collectively indicates that mRCP consistently achieves lower coverage difference expectations without compromising prediction residual as significantly as DRO and V-REx in different tasks and datasets.

6 Discussion

mRCP can distinguish coverage differences under concept shift and covariate shift. A notable feature of the mRCP Pareto curves depicted in Figure 2 is their results when β is small, which are not at ❊ ′ e [ ❊ α [ D ( e ) ]] = 1 , unlike the Pareto curves of V-REx. This is because, during training, mRCP

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

(a)

(b)

US-States

US-Regions

Japan-Prefectures

[(a)a]'a],a

0.8

0.6

1.0

byimportance weighting

by NTW

[()]]

60

1.0

0.8

0.7

1.05

1.00

1.00

1.05

60

0.7

0.6 -

0.8

1.0

0.7

0.9

0.8

1.02

1.00

0.98+

0.95

1.00

0.4

0.6

0.5

0.5

0.4 -

0.6

0.5

β

0.4

0.4

0.2

0.3

0.3

1.0

1.21.41.61.82.02.2

1.0

1.2

1.4

1.6

1.0

1.2

1.4

1.6

1

2

3

E'e[c(e)]

E'e[c(e)]

E'e[c(e)]

E'e[c(e)]

MLP REX

MLP mRCP

MLP DRO

MLP REX

MLP mRCP

MLP DRO

工

PDE REX

工

PDE DRO

(c)10

Seattle-loop

PeMSD4

PeMSD8

(d)0

Seattle-loop

PeMSD4

PeMSD8

1.4

工

1.1

1.0

60

1.1

[a]]a

0.8

1.2 -

0.9

[(a)]a]a

0.8

1.0

1.0

0.6

1.0

0.8

0.7

60

80

0.7

0.6-

0.8

0.4

0.8

0.6

0.5

0.7

0.6

0.2

0.6

0.5

0.4

0.6

1.0

1.1

1.2

1.3

1.0

1.2

1.4

1.6

1.0

1.2

1.4

1.6

1.00

1.05

1.10

1.0

1.1

1.2

1.00

1.05

1.10

1.15

E'[c(e)]

E'e[c(e)]

E'[c(e)]

E'e[c(e)]

E'e[c(e)]

E'[c(e)]

MLPREX

MLP mRCP

MLP DRO

PDE REX

PDE DRO

has considered the coverage difference under covariate shift by applying importance weighting to calibration conformal score CDF. Consequently, as β in Eq. (19) increases, the NTW term is only trained to mitigate the coverage difference under the concept shift, as shown in Figure 2,(a).

DRO and V-REx are defeated because of improper selection of optimization metrics. Examining Eq. (20) and Eq. (21), we can see both baselines aim to promote fairness by equalizing the expected losses across different domains. As the loss function is ℓ 1 norm, which is identical to how conformal scores are calculated, the experiment results of ∆ ❊ in the last row of Figure 3 show this metric is ineffective in capturing the coverage difference due to concept shift.

Nonetheless, mRCP's limitations arise from the inherent challenges associated with penaltybased optimization algorithms. Whether it is mRCP or V-REx, penalty-based optimization algorithms necessitate a model with a high capacity for fitting complex patterns. For instance, in Figure 2, (b), the Pareto curves are not discernible when predictions are derived from an epidemic PDE (SIR model) with only two adjustable parameters. In contrast, as shown in Figure 2, (d), the traffic PDE (RD model) demonstrates greater flexibility and adaptability with six tunable parameters, exhibiting distinct Pareto curves.

7 Conclusion

This study begins by decomposing the coverage difference caused by covariate and concept shifts. 286 We then introduce the Normalized Truncated Wasserstein distance (NTW) as a metric for capturing 287 coverage difference expectation under concept shift by comparing the test and weighted calibration 288 conformal score CDFs. This metric can indicate the discrepancy position in calibration and test score 289 distributions. Normalization and truncation make the metric score scales and outliers. Finally, we 290 develop an end-to-end algorithm called Multi-domain Robust Conformal Prediction (mRCP) that 291 incorporates NTW during training, allowing coverage to approach confidence in all test domains. 292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

References

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Mrinal R Bachute and Javed M Subhedar. Autonomous driving architectures: insights of machine learning and deep learning algorithms. Machine Learning with Applications , 6:100164, 2021.

Leonardo Bellocchi and Nikolas Geroliminis. Unraveling reaction-diffusion-like dynamics in urban congestion propagation: Insights from a large-scale road network. Scientific reports , 10(1):4876, 2020.

Azzedine Boukerche and Jiahao Wang. Machine learning-based traffic prediction models for intelligent transportation systems. Computer Networks , 181:107530, 2020.

Pope D. Brooks, Thomas and Michael Marcolini. Airfoil Self-Noise. UCI Machine Learning Repository, 2014. DOI: https://doi.org/10.24432/C5VW2C.

Maxime Cauchois, Suyash Gupta, Alnur Ali, and John C Duchi. Robust validation: Confident predictions even when distributions shift. Journal of the American Statistical Association , pages 1-66, 2024.

Maxime Cauchois, Suyash Gupta, and John C Duchi. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of machine learning research , 22(81):1-42, 2021.

Ian Cooper, Argha Mondal, and Chris G Antonopoulos. A sir model assumption for the spread of covid-19 in different communities. Chaos, Solitons & Fractals , 139:110057, 2020.

Zhiyong Cui, Kristian Henrickson, Ruimin Ke, and Yinhai Wang. Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting. IEEE Transactions on Intelligent Transportation Systems , 2019.

Songgaojun Deng, Shusen Wang, Huzefa Rangwala, Lijing Wang, and Yue Ning. Cola-gnn: Cross-location attention based graph neural networks for long-term ili prediction. In Proceedings of the 29th ACM international conference on information & knowledge management , pages 245-254, 2020.

Shai Feldman, Stephen Bates, and Yaniv Romano. Improving conditional coverage via orthogonal quantile regression. Advances in neural information processing systems , 34:2060-2071, 2021.

Robert E. Gaunt and Siqi Li. Bounding kolmogorov distances through wasserstein and related integral probability metrics, 2022.

Amanda Gentzel, Dan Garant, and David Jensen. The case for evaluating causal models using interventional measures and empirical data. Advances in Neural Information Processing Systems , 32, 2019.

Subhankar Ghosh, Yuanjie Shi, Taha Belkhouja, Yan Yan, Jana Doppa, and Brian Jones. Probabilistically robust conformal prediction. In Uncertainty in Artificial Intelligence , pages 681-690. PMLR, 2023.

Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616 , 2023.

Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatialtemporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 922-929, 2019.

Wouter M Kouw and Marco Loog. An introduction to domain adaptation and transfer learning. arXiv preprint arXiv:1812.11806 , 2018.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai 339 Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrap340 olation (rex). In International Conference on Machine Learning , pages 5815-5826. PMLR, 341 2021. 342

343

344

345

Sara Magliacane, Thijs Van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems , 31, 2018.

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

Pascal Massart. The tight constant in the dvoretzky-kiefer-wolfowitz inequality. The annals of Probability , pages 1269-1283, 1990.

Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen. Validating causal inference methods. In International conference on machine learning , pages 17346-17358. PMLR, 2022.

Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Neural Information Processing Systems , 2019.

Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. arXiv: Methodology , 2020.

Nathan Ross. Fundamentals of stein's method. 2011.

Hyun-Sun Ryu and Kwang Sun Ko. Sustainable development of fintech: Focused on uncertainty and perceived quality issues. Sustainability , 12(18):7669, 2020.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Silvia Seoni, Vicnesh Jahmunah, Massimo Salvi, Prabal Datta Barua, Filippo Molinari, and U Rajendra Acharya. Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013-2023). Computers in Biology and Medicine , page 107441, 2023.

Yue Sun, Chao Chen, Yuesheng Xu, Sihong Xie, Rick S Blum, and Parv Venkitasubramaniam. Reaction-diffusion graph ordinary differential equation networks: Traffic-law-informed speed prediction under mismatched data. The 12th International Workshop on Urban Computing, held in conjunction with . . . , 2023.

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems , 32, 2019.

Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems , 4, 1991.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world , volume 29. Springer, 2005.

Timothy L Wiemken and Robert R Kelley. Machine learning in epidemiology and health outcomes research. Annu Rev Public Health , 41(1):21-36, 2020.

Xin Zou and Weiwei Liu. Coverage-guaranteed prediction sets for out-of-distribution data. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17263-17270, 2024.

A Related work 380

381

382

383

384

385

386

B Error bound for the assumption of identical coverages

According to the computation of q and q ∗ in Eq. (1) and Eq. (6), respectively, we can define the coverages in unweighted and weighted calibration score distributions as

Denoting q + = inf { v i | v i ∈ V c , v i > q } and q ∗ + = inf { v i | v i ∈ V c , v i > q ∗ } , we can bound ˆ F P ( q ) and ˆ F Q/P ( q ∗ ) as

Therefore, the absolute difference between ˆ F ∗ ( q ∗ ) and ˆ F ( q ) is bounded by 387

388

389

390

391

392

393

394

395

396

397

Especially, when the calibration set size n is large enough (like having thousands of samples), ˆ F Q/P and ˆ F P will be quite smooth, the upper above will be even negligible, allowing us to assume ˆ F Q/P ( q ∗ ) = ˆ F P ( q ) .

C Upper bound of coverage difference under concept shift

In this section, we prove that the W-distance between a test and weighted calibration conformal score population CDF can establish an upper bound for coverage difference under concept shift.

As D quantifies the absolute difference between ˆ F Q/P and ˆ F Q at a calibration conformal score, it can be constrained by an upper bound given by the Kolmogorov distance [12] defined as follows.

Definition 2 (Kolmogorov Distance) . If F 1 and F 2 are two cumulative distribution functions (CDFs), the Kolmogorov distance, d K , is defined as the maximum absolute difference between the CDFs.

As ˆ F Q/P and ˆ F Q are empirical (not population) CDFs of weighted calibration and test conformal 398 scores, the bounding relationship can be reformulated as 399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

The upper bound d K ( ˆ F Q , ˆ F Q/P ) depends on the two conformal score sets V c and V t , indicating that the inclusion of samples in S c and S t is likely to introduce variability in d K ( ˆ F Q , ˆ F Q/P ) . Nevertheless, we aim for an upper bound that is not reliant on specific samples and relies on the calibration and test conformal score population CDFs, F P and F Q .

Firstly, we convert the upper limit in Eq. (22) into terms of F P and F Q . Denoting the joint probability density function (PDF) of features and score in the calibration and test domain as p XV and q XV respectively, the corresponding continuous CDFs of conformal scores are illustrated as

where X is the space of the feature variable X .

PDFs of features in calibration and test domains, denoted as p X and q X respectively, are defined as

To address the coverage difference due to covariate shift, importance weighting from [29] is rewritten as w = q X p X . Also, normalization is unnecessary, because w here is a correction function to transform the marginal distribution of p X into q X . The weighted version of p XV is denoted as p ′ XV = w p XV = q X p V | X , which can be applied to derive the weighted continuous CDF of calibration conformal score by

The Kolmogorov distance between F Q/P and F Q is d K ( F Q , F Q/P ) = sup v ∈ ❘ | F Q ( v ) -F Q/P ( v ) | .

Theorem 1 (Triangular Inequality for Kolmogorov Distance) . If F 1 , F 2 , and F 3 are three cumulative distribution functions (CDFs), their Kolmogorov distances follow this inequality:

Proof. Consider any point x ∈ R , then we have | F 1 ( x ) -F 3 ( x ) | ≤ | F 1 ( x ) -F 2 ( x ) | + | F 2 ( x ) -F 3 ( x ) | .This inequality holds due to the triangle inequality for absolute values. Now, taking the supremum over all x , we have sup x ∈ R | F 1 ( x ) -F 3 ( x ) | ≤ sup x ∈ R ( | F 1 ( x ) -F 2 ( x ) | + | F 2 ( x ) -F 3 ( x ) | ) . Note that the right-hand side is not necessarily equal to the sum of the suprema of the individual terms, because the points at which the suprema of | F 1 ( x ) -F 2 ( x ) | and | F 2 ( x ) -F 3 ( x ) | are attained may be different. However, we know that for any x , | F 1 ( x ) -F 2 ( x ) | is at most d K ( F 1 , F 2 ) and | F 2 ( x ) -F 3 ( x ) | is at most d K ( F 2 , F 3 ) . Therefore, sup x ∈ R | F 1 ( x ) -F 3 ( x ) | ≤ d K ( F 1 , F 2 ) + d K ( F 2 , F 3 ) . Since the left-hand side is the definition of d K ( F 1 , F 3 ) , we can demonstrate that d K ( F 1 , F 3 ) ≤ d K ( F 1 , F 2 ) + d K ( F 2 , F 3 ) .

As Kolmogorov distance satisfies the triangular inequality theorem, as shown and proved in Theorem 1, the triangular inequality relationship can be expanded to

Secondly, the Kolmogorov distance between an empirical CDF and its corresponding population CDF can be constrained by Dvoretzky-Kiefer-Wolfowitz (DKW) inequality [20], defined in Definition 3. Definition 3 (Dvoretzky-Kiefer-Wolfowitz (DKW) Inequality) . If F is a population cumulative distribution function (CDF), and ˆ F is an empirical CDF with n samples of a random variable X , then for any ϵ ≥ √ 1 2 n ln 2 , the following inequality holds.

Based on Definition 3, saying | V c | = n and | V t | = m , we can apply DKW inequality to d K ( ˆ F Q/P , F Q/P ) and d K ( ˆ F Q , F Q ) as follows, for ϵ ≥ √ 1 2 n ln 2 and ρ ≥ √ 1 2 m ln 2 .

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

If the two events d K ( ˆ F Q/P , F Q/P ) < ϵ and d K ( ˆ F Q , F Q ) < ρ are independent, the inequality in Eq. (26) can be expanded in Eq. (27), which holds with at least probability e -2( nϵ 2 + mρ 2 ) . By applying DKWinequality, we successfully quantify the variability of d K ( ˆ F Q , ˆ F Q/P ) in Eq. (22) as a form of a probable event, and use the population conformal score CDFs to limit the worst-case of coverage difference under concept shift.

Finally, having established in Eq. (13) that the W-distance can serve as an estimator for coverage difference expectation, we explore whether Eq. (27) may similarly be bounded by this metric. The W-distance of the two population conformal score CDFs are explicitly shown as

According to [24], if the weighted calibration conformal score probability density function (PDF) has Lebesgue density bounded by C , which means p ′ V does not exceed C , then for any test conformal score PDF q V , d K ( F Q , F Q/P ) can be bounded as

Finally, we can derive the upper limit of coverage difference under concept shift, sup v ∈ V c | D ( v ) | , in Eq. (30) at least probability e -2( nϵ 2 + mρ 2 ) .

This property is attractive in that the maximum difference in coverage due to concept shift can also be constrained in relation to the W-distance of population score CDFs, denoted as d W ( F Q , F Q/P ) . Despite the unobservability of d W ( F Q , F Q/P ) , we can still estimate it using its empirical form, d W ( ˆ F Q , ˆ F Q/P ) .

Even though coverage guarantee on an arbitrary joint shift is almost impossible, Eq. (28) demonstrates robust conformal prediction is attainable if we can train a function reducing the discrepancy between calibration and test conformal score distributions. To be specific, d W ( F Q , F Q/P ) can be reduced to zero as far as p V | X = q V | X . In other words, if we regard p XV and q XV as push-forward probability distribution of P XV and Q XV by the trained model f , making the concept shift between p V | X and q V | X smaller will reduce coverage difference expectation on test domain.

D Datasets, models, and experiment setups

Extensive experiments are conducted under 3 tasks with 7 datasets. Some tasks involve both black-box and physics-informed models to demonstrate the generalizability of NTW and mRCP.

D.1 Airfoil self-noise example

The airfoil dataset from the UCI Machine Learning Repository [5] consists of 1503 instances of 1-dimensional target Y and 5-dimensional feature X = ( X 1 , X 2 , X 3 , X 4 , X 5 ) . This dataset is manually separated and modified to create three different domains.

Domain separation:

Step 1. Covariate Shift by Data Separation. The original dataset is initially segmented into three primary subsets A,B,C based on the 33% and 66% quantiles of the first dimension X 1 . Subsequently, each of these subsets is further divided into three smaller portions at a 7:2:1 ratio, denoted like A 0 . 7 , A 0 . 2 , A 0 . 1 from A . Finally, we assemble three new datasets with covariate shift as S ( e 1 ) = A 0 . 7 ∪ B 0 . 2 ∪ C 0 . 1 , S ( e 2 ) = A 0 . 2 ∪ B 0 . 1 ∪ C 0 . 7 , S ( e 3 ) = A 0 . 2 ∪ B 0 . 1 ∪ C 0 . 2 .

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

Step 2. Concept Shift by Target Modification. Differently distributed random noises are added to target values to cause concept shifts. For y i from S ( e 1 ) , y i + = y i / 1000 ∗ τ ; for y i from S ( e 2 ) , y i + = y i /τ ; for y i from S ( e 3 ) , y i + = τ . τ follows a normal distribution N (0 , 10 2 ) . Since we obtain three subsets in the end, |E| = 3 .

Model selection:

We utilize a straightforward multilayer perceptron (MLP) as a trainable model, with an architecture of (input dimension, 64, 64, 1) tailored for the regression task.

D.2 Traffic speed prediction

The Seattle-loop [9], and PeMSD4, PeMSED8 datasets [16] contain sensor-observed traffic volume and speed data collected in Seattle, San Francisco, and San Bernardino. The snapshots from sensors are taken at 5-minute intervals. This task aims to predict the traffic speed of the local road segment in the next time step, using the traffic data from local and neighboring segments collected currently.

Domain separation:

Naturally, instances can be categorized into 24 subsets, |E = 24 | , based on the hour they are obtained. It is anticipated that there are joint shifts between the data distribution of every single hour (test domains) and the data distribution of the whole day (calibration domain), as traffic patterns vary over time, making it unnecessary to modify any data. We select the workday data from the three datasets.

Model selection:

(a) MLP with the same structure (input dimension, 64, 64, 1) is applied to the traffic prediction task.

(b) The Reaction-Diffusion (RD) model is selected as the physics-informed Partial differential equation (PDE) for traffic speed prediction. Reaction-diffusion mechanism, originally formulated for chemical systems to describe particle dynamics, has been adapted for traffic analysis by [3] to uncover traffic patterns on different road segments, offering an alternative to purely data-driven models like long-short-term memory. [28] further advanced this approach by integrating the RD model into graphical neural networks to capture traffic state interactions among adjacent road segments, with the reaction term accounting for influences against traffic flow and the diffusion term for influences along it. To be specific, for a given sensor i , with N d upstream and N r downstream neighboring sensors, the traffic states from these sensors impact sensor i after δt time through diffusion and reaction effects, respectively. We expand the original RD model in [28] to Eq. (31), where the traffic speed and volume at sensor i at time t is u i ( t ) and q i ( t ) , respectively. The parameters ρ ( i,j ) and σ ( i,j ) represent the diffusion and reaction strengths between sensor i and sensor j , while their superscripts indicate if they serve for speed or volume. Also, d i and r i are bias terms for the two components.

D.3 Epidemic spread prediction

Three epidemic datasets, US-Regions, US-States, and Japan-Prefectures [10] include the number of patients infected by influenza-like illness (ILI) recorded by U.S. Department of Health and Human Services, Center for Disease Control and Prevention (CDC), and Japan Infectious Diseases Weekly Report. We aim to use the local population, the rise in the number of infected patients observed this week, and the cumulative total of infections as predictive features of the increase in infections for the upcoming week.

Domain separation: According to the Pandemic Intervals Framework (PIF) by CDC, samples are divided by four pandemic intervals, Initiation, Acceleration, Declaration, and Subsidence, so |E| = 4 . We establish the interval endpoints based on specific percentages of the total infected patient count, specifically at the 15 % , 50 % , and 85 % thresholds.

Model selection:

(a) MLP with the same architecture is utilized for the epidemic spread forecasting task as well.

(b) PDE for this task is the SIR model that categorizes the population into three groups: those 516 susceptible to the disease S , those infectious I , and those who have recovered and gained immunity 517 R . It outlines the temporal changes in their populations, as described by [8]. The governing 518 differential equations can be expressed as Eq. 32, where N , λ , and γ represent the total population, 519 infection rate, and recovery rate, respectively. 520

We make the assumption that the location is isolated, hence N = S ( t ) + I ( t ) + R ( t ) . Additionally, 521 the population of recovered individuals is represented by R ( t ) = γ ∫ t 0 I ( t ) dt . Given this, if t o 522 signifies the initial time of the current epidemic and δt denotes the time step, which is a week in the 523 three datasets, we can express the dynamic change of infectious individuals discretely as Eq. (33). 524

D.4 Experiment setups for NTW and baseline metrics

As we only need to validate the positive correlation between NTW and coverage difference expectation, all models are trained by ERM. In the airfoil self-noise example, 100 trials are carried out. For the traffic task, 61 locations from the Seattle-loop, 59 locations from PeMSD4, and 33 locations from PeMSD8 are chosen, with 10 trials conducted at each location. For simplicity in the calculation, all selected locations have just one segment upstream and one segment downstream. For epidemic datasets, all locations from US-Regions, US-States, and Japan-Prefectures (49 locations in US-States, 10 locations in US-Regions, and 46 locations in Japan-Prefectures) are encompassed in the experiments, with 10 trials implemented on each location. The same experiment setups are operated on all baseline metrics and NTW. σ values for MLP and PDE are 0.8 and 0.95, respectively. The ratio of training, calibration, validation, and testing data on airfoil self-noise datasets, three traffic datasets, and three epidemic datasets are 1:1:1:1, 3:2:2:3, and 1:2:1:1, respectively. Data separation was conducted randomly. Adam optimizer with a learning rate of 0.001 was applied for all experiments. On average, one trial requires one hours on a workstation with double NVIDIA RTX 3090 GPU.

D.5 Experiment setups for mRCP, V-REx, and DRO

We define 1 trial as running a series of experiments of all predefined β values once, except for DRO. For the airfoil self-noise example, 100 trials with random data preprocessing are conducted. For the traffic speed prediction task, we randomly select 10 locations from each of the three traffic datasets and operate one trial on all selected locations. In the epidemic spread prediction task, all locations of the three datasets are included and we operate one trial on each of them. All combinations of models (MLP and PDE) and algorithms (mRCP, DRO, V-REx) share the same experiment setups mentioned above. σ values for MLP and PDE are 0.8 and 0.95, respectively. β values for mRCP and V-REx in different experiment setups are shown in Table 3. Each Pareto curve consists of at least 10 β values. For airfoil self-noise datasets and three traffic datasets, the original data is evenly and randomly split for training, calibration, and testing. For three epidemic datasets, we randomly split the original data for training, calibration, and testing with a ratio of 2:1:2. Adam optimizer with a learning rate of 0.001 was applied for all experiments. On average, one trial requires 12 hours on a workstation with double NVIDIA RTX 3090 GPU.

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

E Additional experiment results

E.1 Pearson coefficient definition

Here we provide a detailed definition of the Pearson coefficient as follows.

Definition 4 (Pearson coefficient) . The Pearson correlation coefficient, denoted as r , is calculated as the covariance of the two variables divided by the product of their standard deviations, as follows.

where x i and y i are the individual sample points of random variables X and Y indexed with i and x and y are the means of their samples, respectively.

The Pearson correlation coefficient measures the linear correlation between two variables. It gives a value between -1 and 1 inclusive, where 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear correlation.

E.2 Correlation visualization

Figure 3 shows the experimental results of the correlation between NTW and coverage difference

565 expectation, compared with three baselines: total variation, KL divergence, and expectation difference. 566 It is organized into a matrix of subplots, with each column corresponding to a specific dataset and 567 each row depicting the performance of a metric. Within these subplots, individual points represent 568

top row, showcasing NTW's strong correlation. In contrast, the other metrics exhibit inconsistent 571

correlations across the varied datasets and models, as seen in the lower three rows of subplots. 572

Figure 4 also illustrates the expected coverage difference's correlation to NTW, standard W-distance, 573

normalized W-distance, and truncated W-distance, proving that normalization and truncation are 574 equally important for robust correlations.

0.10

0.20

d

(

e

)

NTW

10

15

d

(

e

)

TV

100 200 300

d

(

e

)

KL

0.1

0.2

[

D

(

e

)

]

0.08

0.16

(

e

)

0.03

0.06

Seattle-loop

0.3

0.4

1

2

0.025

0.050

0.075

[

D

(

e

)

]

0.25

0.50

0.75

0.06

0.09

PeMSD4

3

4

5

40

60

0.04

0.06

0.08

[

D

(

e

)

]

0.15

0.30

0.04

0.06

PeMSD8

3

4

20

40

60

0.04

0.06

[

D

(

e

)

]

0.08

0.12

MLP

PDE

0.08

0.10

US-States

0.3

0.4

0.5

10

15

0.08

0.10

[

D

(

e

)

]

5

10

15

0.06

0.08

US-Regions

0.016

0.024

0.2

0.4

0.070

0.075

[

D

(

e

)

]

20

40

60

0.10

0.20

Japan-Prefectures

0.02

0.04

0.06

0.5

1.0

1.5

2.0

0.05

0.10

0.15

[

D

(

e

)

]

150

300

0.10

0.20

d

(

e

)

NTW

0.1

0.2

d

(

e

)

W

0.010

0.015

d

(

e

)

NW

0.1

0.2

[

D

(

e

)

]

0.02

0.04

d

(

e

)

TW

0.03

0.06

Seattle-loop

0.2

0.4

0.6

0.8

0.01

0.02

0.025

0.050

0.075

[

D

(

e

)

]

0.25

0.50

0.06

0.09

PeMSD4

0.2

0.3

0.4

0.02

0.03

0.04

0.06

0.08

[

D

(

e

)

]

0.10

0.15

0.04

0.06

PeMSD8

0.10

0.15

0.01

0.02

0.03

0.04

0.06

[

D

(

e

)

]

0.04

0.08

MLP

PDE

0.08

0.10

US-States

10

15

20

25

0.04

0.06

0.08

0.10

[

D

(

e

)

]

5

10

15

0.06

0.08

US-Regions

40

60

80

0.02

0.03

0.070

0.075

[

D

(

e

)

]

20

40

0.10

0.20

Japan-Prefectures

200

400

0.08

0.16

0.05

0.10

0.15

[

D

(

e

)

]

0

150

300

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We focus on robust conformal prediction under joint distribution shift. The abstract and introduction accurately state our contributions. We propose Normalized Wasserstein distance to quantify the coverage difference caused by concept shift and develop multi-domain robust conformal prediction to make coverage approach confidence when multiple test domains hold joint shifts with the calibration domain.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our proposed method in Section 6. The proposed method requires the training model's enough capacity to fit complex patterns. Also, as it needs to approximate the ratio of covariate likelihood between calibration and test domains, it requires enough training and calibration samples to conduct accurate kernel density estimation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We refer to Section 3, Appendix B, Appendix C for detailed theoretical work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show detailed experiment setups, including models, datasets, and algorithms, in Section 5 and Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We would like to provide open access to the data and code if this submission is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed information about data splits and preprocessing, hyperparameters, optimizer, black-box model architecture, and physics-informed equations in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Statistical measures of experiment results are shown in Figure 2 and Table ?? .

Guidelines: 732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information about our workstation and computation time for one trial in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research follows the NeurPIS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We mentioned the positive impact of robust conformal prediction in the introduction. We do not see negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our submission does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Datasets and models are properly cited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This research does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper adresses the issue of conformal prediction under distribution shift with multiple test domains. The goal is to reduce the deviation in coverage caused by different potential distribution shifts across these domains. The paper firstly proposes a way of disentangling a joint distribution shift (shift effects both in covariate and label distributions, which they term \"concept\") by employing weighted conformal prediction (Tibshirani et al, 2019) to address covariate shift, and then quantify the remaining shift in terms of a truncated normalized Wasserstein distance (D-NTW) between the original and weighted conformal score distributions (empirical CDFs). This D-NTW is then used as a regularizer term in a training algorithm to explicitly ensure that coverage deviations are minimized across test domains. Experiments include assessing the correlation between D-NTW and the actual expected coverage difference, which is shown to be high (vs. other distributional distance metrics), and comparing to two multi-test-domain optimization methods on a variety of datasets to show that the coverage difference is lower while not compromising on prediction accuracy.\n\nSTRENGTHS:\n- Addressing the issue of distribution shift in conformal prediction is a relevant problem, particularly for the less explored label shift setting\n- Attempts are made to disentangle covariate and label (concept) shifts, which can provide insights into the model's adaptation abilities\n- The suggested D-NTW distance metric is well motivated and benchmarked against multiple sensible alternative distance metrics\n- An interesting range of datasets from different domains is considered\n- The paper is clearly written and good to follow, including Fig I visualizing the procedure (albeit it is somewhat hard to read, especially part (c))\n\nWEAKNESSES:\nMy main concerns are w.r.t. practicality and evaluation. A fundamental requirement of the proposed algorithm and D-NTW is the availability of labelled samples from every test domain $Q_{XY}^{(e)}$ in order to obtain the conformal test score distributions $F_{Q^{(e)}}$, from which then both the likelihood ratios for covariate shifts and D-NTW can be explicitly computed. Beyond existing concerns about the practicality of estimating a likelihood $\\textit{ratio}$, we now require actually explicitly estimating every test domain distribution, which is prone to much error. Regardless, if we can now explicitly obtain $F_{Q^{(e)}}$ for every test domain, I am wondering why the direct solution is not to just compute a conformal quantile $q^{(e)}$ on the basis of this for every test domain and thus optimally adapt to the existing joint shifts per domain. Perhaps the loss of the disentanglement of shift contributions is the motivation? In general, I find this requirement to require knowing or estimating the test domain distributions on the basis of labelled test domain data and thus the use of extensive density estimation quite limiting, especially if we consider high-dimensional settings. Perhaps this is why the considered experiments are for 1-D regression data only. I was also wondering if the authors were able to make any connections between their proposals and obtainable coverage guarantees on the test domains. They propose a bound on the distance of D-NTW from the expected coverage difference, but perhaps more explicit connections to conformal coverage guarantees of the form in Eq. 3, e.g. by leveraging guarantees from (Tibshirani et al, 2019) and their linear decomposition of shift effects are worth investigating.\n\nIn regards to evaluation, I was missing a closer connection to existing conformal methods under shift, and the actual goals of these methods in terms of coverage guarantees on test data. In their comparison to test-domain optimization algorithms I was not surprised that their algorithm performs better on expected coverage difference, since it explicitly $\\textit{optimizes}$ for this goal, while the baselines target e.g. variance minimization. It would be more interesting to compare to conformal algorithms for shift such as the mentioned [2,3], showing e.g. that those are not able to fully capture the joint shift or are overly conservative, thus compromising on the metric. For example, I was surprised that [1] was not mentioned, since it explicitly targets label shifts. Similarly, while it is nice that the relative coverage difference is minimized or correlates well with D-NTW, this does not tell explicitly how my conformal methods will now perform on the test domains. It would be nice to also obtain an explicit assessment of the coverage $(1-\\alpha)$ on test domains, and the obtained prediction set sizes. Even if target coverage is not satisfied, it would already be a contribution to show that the proposed algorithm achieves better robustness by being closer to target coverage, or smaller set sizes at the same level. \n\nMinor: multiple typos e.g. L75, Fig I caption, L89, Eq. 2\n\nReferences\n- [1] Podkopaev, Aleksandr, and Aaditya Ramdas. \"Distribution-free uncertainty quantification for classification under label shift.\" Uncertainty in artificial intelligence. PMLR, 2021.\n- [2] Cauchois, Maxime, et al. \"Robust validation: Confident predictions even when distributions shift.\" Journal of the American Statistical Association (2024): 1-66.\n- [3] Zou, Xin, and Weiwei Liu. \"Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.\n\nQUESTIONS:\n- Could you comment on some of the concerns raised above, e.g. on the practicality of requiring multiple density estimations, simply computing optimal quantiles directly on each test domain, or evaluating the methods w.r.t. test coverage and conformal set size.\n- What methods are used to estimate the test domain distributions $Q_{XY}^{(e)}$?\n- Have you considered higher-dimensional experimental settings and/or using models that are more complex than an MLP? If not, could you explain if this is related to the limitations of your method in terms of higher dimensions?\n- In the experiments, all the generated test domain shifts are created manually. Have you considered datasets with unknown test shifts, e.g. the Yearbook dataset?\n- In sec 5.2. the correlation is assessed via Pearson coefficient. Could you provide a reasoning on why you believe a linear relationship exists, and/or if you have considered more robust measures such as rank correlation? Given that the estimated score distributions are empirical CDFs, this seems perhaps more intuitive.\n- Could you comment on the observed discrepancies in some of the results in Table 1? For example, For US-States and MLP, all distance metrics except D-NTW show negative correlation.\n- Could you comment on the limitations of the made assumption in Eq. 17 that the calibration domain $P_{XY}$ is considered a linear mixture of the \"unknown\" test domain distributions? This seems like a restriction that is not explicitly mentioned.\n\nLIMITATIONS:\nThe requirements of the algorithm and density estimations are mentioned, but the limitations of the approach are not explicitly discussed in terms of imposed assumptions on the problem setting. A more through discussion of the practical limitations would be helpful (some of which are inherited e.g. from (Tibshirani et al, 2019) simply by their use of likelihood ratio weights).  A small subsection in sec 6 mentions difficulties of optimization algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the coverage difference caused by covariate and concept shifts. Authors introduce the Normalized Truncated Wasserstein distance (NTW) as a metric for capturing coverage difference expectation under concept shift by comparing the test and weighted calibration conformal score CDFs. They also develop an end-to-end algorithm called Multi-domain Robust Conformal Prediction (mRCP) to incorporate NTW during training, allowing coverage to approach confidence in all test domains.\n\nSTRENGTHS:\n1. introduced NTW as a metric to capture the coverage gap\n\n2. high correlation between NTW and coverage difference expectation; mRCP can balance residual size and coverage gap\n\nWEAKNESSES:\n1. section 3.1 and 3.2 introduce some important definitions:  while authors provide some explanation, the theoretical understanding of them are very limited\n\n2. simulation:  authors mention the mRCP can achieve a balance between coverage gap and size of residual, further simulations need to be carried out ( I would be interested to see a plot including the avg. coverage vs avg. residual size\n\nQUESTIONS:\n3.1 the coverage difference works with the empirical distribution function, would it be possible to study the coverage gap in the form of P(...) - P(...)\n\n3.1 in the paper, a likelihood ratio dQ/dP is assumed to be known, how would you estimate it in practice? The definition of $q^{*}$ in the paper seems to be inconsistent with the conformal prediction literature. The factor (n+1)/n, I believe is typically added in the i.i.d case while  for covariate shift, the form is not like this if you take a look at the paper \"conformal prediction under covariate shift\".\n \n\n4  In practice, how would you select the tuning parameter beta? Better theoretical guarantees for section 4 need to be developed.\n\nLIMITATIONS:\nAuthors mentioned the limitation in the discussion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Robust Conformal Prediction under Joint Distribution Shift\" investigates the problem of predictive inference under the setting where we have both covariate shift and concept shift. The authors propose a conformal prediction-based procedure that accounts for such distribution shifts and illustrate the performance through experiments.\n\nSTRENGTHS:\nThis work provides extensive and thorough experimental results.\n\nWEAKNESSES:\nThe paper doesn't read very well as some notations appear without definitions and relevant assumptions are not written clearly. For example:\n\n- It is assumed that the likelihood ratio $dQ/dP$ is known, but this was not clearly stated in the problem setting.\n- Some assumptions are not written in advance but are rather introduced when they are needed.\n\nIt would be better if the authors could provide sufficient intuition and motivation for their methodology.\n\nQUESTIONS:\n1. First of all, as a theoretical statistician who is not very familiar with the standards in the machine learning community, I wonder what the motivation of this work is. Specifically, the main advantage of the conformal prediction framework is that it theoretically provides an exact finite sample guarantee without distributional assumptions. In this work, the proposed methods have no theoretical guarantee, and the focus is more on empirical illustrations. While I understand that machine learning emphasizes practical applications and performance on real data, why should one apply conformal prediction here if we cannot exploit the advantage of the conformal inference framework? In other words, If we are not aiming for a distribution-free guarantee (or any other theoretical guarantee), aren't there methods that work better for specific datasets?\n\n2. Evaluating the difference under distribution shift using (5) doesn't seem very reasonable to me. The prediction set is constructed using the calibration set, and the authors estimate the 'CDF of the score under no distribution shift' by $\\hat{F}_P$, which is also a function of the calibration set (though estimating the 'CDF under shift' with $\\hat{F}_Q$ sounds reasonable).\n\n3. In fact, $\\hat{F}_P(q)$\n\n  seems to be just $1-\\alpha$ (more precisely, $\\lceil(n+1)(1-\\alpha)\\rceil / n$)? According to the definition, $q$ is the quantile of the distribution $\\frac{1}{n}\\sum_{i=1}^n \\delta_{v_i}$, while $\\hat{F}_P$ \n\nis the CDF of the same distribution. So it seems to me that $\\hat{F}_P(q) = \\lceil(n+1)(1-\\alpha)\\rceil / n$ holds exactly.\n\n Then it is quite strange why they replace this known value with $\\hat{F}_{Q/P}(q^*)$, generating an additional error.\n\n4. Additionally, I'm a bit confused about equation (13), where again the exact equality seems to hold. The two empirical CDFs $\\hat{F}_Q$\n\n and $\\hat{F}_{Q/P}$ are step functions where the jumps occur at the same values $v_1, \\cdots, v_n$, so it seems to me that exact equality holds. In that sense, the remaining discussion in section 3.2 sounds a bit odd. For example, the empirical CDFs have values exactly 1 for $v \\geq \\max v_i$. What does the 'long tail' mean in this case?\n\n5. Could the authors clarify the motivation behind the need for 'multi-domain' inference? In the beginning of Section 4, it just says, \"The domain $P_{XY}$ can be decomposed into $M$ multiple domains,\" so it's a bit unclear what settings are under consideration. For example, are they considering a setting where there is a natural partition of the domain, and we aim for a good quality of inference conditional on each element of the partition? Or are they considering a setting where we artificially construct a partition (if so, what is the reason for that?), etc.? This might be provided in previous literature, but it would be better if relevant discussions are also provided in this work.\n\n6. The notation in equation (17) is hard to understand. The notation $P_{XY}$ was originally used to denote the distribution of $(X,Y)$, but the authors use it to denote the domain (in the line above equation (17)), and in equation (17) it is written as if the distribution is a mixture distribution of $M$ distributions $(Q_{XY}^{(e)})$ with mixture probability $(\\frac{1}{M}, \\cdots, \\frac{1}{M})$, rather than the domain being partitioned into $M$ bins. Does $Q_{XY}^{(e)}$ denote the conditional distribution of $(X,Y)$ under $P_{XY}$, given $(X,Y) \\in e$? If that's the case, how do we know that the probability $P((X,Y) \\in e)$ is $\\frac{1}{M}$ for every $e$ in the partition?\n\n7. I wish I could come up with more constructive questions and suggestions, but it was a bit hard for me to do so unless the above points (which are related to the technical foundation of the main results) are resolved. I understand that some of the points above might be due to my misunderstanding or confusion, and I'd be happy to have a further discussion or update the scores once I receive a reply from the authors.\n\n* Minor comments:\n\nIn equation (10), the notation `E' doesn't seem appropriate as it's a sample mean rather than the population mean.\n\nI believe It is more standard to denote $v_i$ as 'nonconformity score' rather than 'conformal score.'\n\n* Typos:\n\nabove equation (3): 'coverage guarantee'\n\nEquation (4): $v$ -> $v_i$\n\nEquation (11): $dx$ -> $dv$\n\nLIMITATIONS:\nI don't think this work has negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to train a predictive model (a regressor in their experiments) that minimizes an objective comprising the average performance loss across multiple domains (environments) along with a penalty term for the normalized truncated Wasserstein (NTW) distance between the non-conformity score CDFs of each environment and the importance-weighted one used to address covariate shift. Their experimental results demonstrate that the proposed NTW distance objective is correlated with coverage differences due to concept shift and can achieve different tradeoffs with the average prediction residuals, thereby reducing this gap.\n\nSTRENGTHS:\nOverall, the paper is easy to follow and the reasoning behind the proposed formulation is compelling. The problem that the authors address is relevant. The experimental results show that the proposed NTW distance is capturing the coverage difference due to concept shift.\n\nWEAKNESSES:\nWhile the motivation behind the regularization is compelling, it is not entirely clear, both empirically and theoretically, what specific benefits it offers over other state-of-the-art approaches that address differences in the non-conformity score distributions. To highlight the advantages of the proposed NTW regularization over simply minimizing the prediction residuals and then applying various post-hoc conformal prediction techniques, I suggest that the authors demonstrate the validity and efficiency of the prediction intervals obtained for different alphas (error levels) on test data. Additionally, they should provide empirical comparisons or some fundamental discussion/theoretical results in relation to the following approaches:\n\n* Split conformal prediction and group conditional split conformal prediction (where each environment is treated as a separate group). The latter can include standard SCP conditioned on each group or an approach such as the one in section 4.1 of [Barber et al. 2020, \"The Limits of Distribution-Free Conditional Predictive Inference\"] or BatchGCP/BatchMVP in [Jung et al. 2022, \"Batch Multivalid Conformal Prediction\"].\n\t\n* Performance of the covariate shift split conformal prediction, as already discussed in the paper. For example, if this is built on top of a model that minimizes ERM, DRO or V-REx does the proposed approach provide better prediction sets in terms of conditional coverage/validity on the domains.\n\n* An adaptive approach such as the one by [Amoukou and Brunel 2023, Adaptive Conformal Prediction by Reweighing Nonconformity Score].\n\nProviding such comparative results or analysis would significantly strengthen the paper's argument.\n\nI also think the authors should discuss how their work relates to [Barber, Rina Foygel, et al. \"Conformal prediction beyond exchangeability\"], where it is suggested that the non-conformity scores should be weighted based on the total variation distance between the source and target distributions. This approach could potentially serve as another baseline to consider, given the distance between the distributions of the non-conformity scores under P and Q(e).\n\nQUESTIONS:\nIn the experiments, how do the authors compute the importance weights for covariate shift?\n\nHow do you pick \\sigma?\n\nRevise typos such as \"Dcoviriate\" in Figure 1 caption.\n\nLIMITATIONS:\nyes, they mention some of the limitations of the proposed work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the challenge of obtaining conformal predictions that remain robust under distribution shifts. This is an important issue in many machine learning applications where the underlying data distribution may change across the training (or calibration) and test data sets. The authors propose an algorithm that appears promising based on empirical results. However, significant improvements are needed in terms of writing quality, clarity, citation of relevant literature, mathematical rigor, and explanation of the main ideas. Addressing these substantial weaknesses would make the paper more accessible and impactful for the research community.\n\nSTRENGTHS:\n- The problem of robust conformal predictions under distribution shifts is timely and relevant.\n\n- The paper introduces a concrete algorithm that demonstrates promising performance in some practical scenarios.\n\nWEAKNESSES:\n- Writing Quality: The paper is difficult to read and understand, even for experts. Key concepts are not clearly explained, and the text contains numerous awkward or unclear sentences, as well as pervasive grammatical errors and typos. Some sections seem poorly written, possibly by AI, while others could have been significantly improved with better editing.\n\n - Missing References: Important related works, such as \"Conformal prediction beyond exchangeability\" by Barber et al. (2023), are not discussed, which limits the paper's contextual grounding in existing literature.\n\n- Lack of Statistical/Mathematical Rigor: The mathematical details in the paper are imprecise. Assumptions and approximations are not clearly stated, and there is a frequent confusion between population and sample quantities in key sections.\n\n - Unclear Core Idea: The main idea of the proposed algorithm, particularly how it handles concept shift through covariate shift adjustments (as in Equation 10), is not clearly articulated and remains confusing.\n\nQUESTIONS:\n- Writing Quality: The current writing is difficult to understand, and professional editing may be an option if needed. Parts of the paper read like they were written by Chat-GPT, but not in a good way, with lots of very ackwards sentences and passages that don't make sense. Other parts could have been improved by Chat-GPT, due to the large number of grammatical errors and typos. \n\n- Clarity of Introduction: The introduction does not clearly explain the problem being addressed or the novelty of the proposed approach. Could you clarify these points to help readers understand the significance of your work?\n\n- Mathematical Precision: The paper seems to systematically confuse population and sample quantities, starting from Equations (7)-(9) and (10). For instance, why is an empirical distribution used in Equation (7) instead of a population distribution, in the definition of a quantity (the coverage gap) which should be a population quantity? Additionally, the \"expected value\" in Equation (10) does not seem correctly formulated, since the left-hand-side of the equation is a population quantity (hence fixed) but the right-hand-side is a sample quantity (hence random). I don't think this is correct math.\n\n- Assumptions and Justifications: Lines 112-115 introduce an assumption that is neither explained nor motivated, followed by a vague statement about a small error bound. Can you provide a clearer explanation and justification for this assumption? Is this an assumption or an approximation?\n\n- Core Concept Explanation: The explanation of handling concept shift via covariate shift, particularly in Equation (10), is confusing. Could you clarify this core idea, as it is central to your paper's contribution? It seems from Equation (10) that concept shift was reduced to a (much simpler) covariate shift problem. I don't understand how this happened.\n\n- Background on Conformal Prediction: Section 2.1 lacks precision. For example, Equation (2) describes a special case for regression, but the referenced works use different approaches. Can you clarify these points to make the section more accessible to a broad readership? Is this paper focusing on regression or classification? Is it limited to a specific type of conformity scores, such as that in Equation (2), or is it more general?\n\n- Connection to Related Work: Is there a connection between the coverage difference you considered and that studied in \"Adaptive conformal classification with noisy labels\" by Sesia et al. (2023)? It seems there might be a relationship, even though Sesia et al. consider a specific case of distribution shift. Can you discuss any similarities or differences?\n\nLIMITATIONS:\nThe paper's main limitations are its poor writing quality and lack of mathematical precision. These issues make it difficult to understand the main ideas and verify the soundness of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-23"}