Improving Conditional Coverage via Orthogonal Quantile Regression
Authors: Shai Feldman
Stephen Bates
Yaniv Romano
Venue: NeurIPS 2021 Poster (2021)
Keywords: Quantile Regression
Pinball Loss
Conditional Coverage
Conformal Prediction
Uncertainty Estimation

Abstract: We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, a property called conditional coverage. A typical approach to this task is to estimate the conditional quantiles with quantile regression---it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage. To remedy this, we modify the loss function to promote independence between the size of the intervals and the indicator of a miscoverage event. For the true conditional quantiles, these two quantities are independent (orthogonal), so the modified loss function continues to be valid. Moreover, we empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics. We also introduce two new metrics that check conditional coverage by looking at the strength of the dependence between the interval size and the indicator of miscoverage. 

Improving Conditional Coverage via Orthogonal Quantile Regression
Shai Feldman
Stephen Bates
Department of Computer Science Technion, Israel
Departments of Statistics and of EECS
UC Berkeley
shai. fe 1 dman@cs. technion.ac.il
stephenbaateslcs.beerre
edu
Yaniv Romano
Departments of Electrical and Computer Engineering and of Computer Science Technion, Israel
yromano@techiion.coo ac ac.
Abstract
We develop method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, property called conditional coverage. A typical approach this task estimate the conditional quantiles with quantile regression-it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage. To remedy this, modify the loss function to promote independence between the size of the intervals and the indicator of miscoverage event. For the true conditional quantiles, these two quantities are independent (orthogonal), so the modified loss function continues to be valid. Moreover, we empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics. We also introduce two new metrics that check conditional coverage by looking at the strength of the dependence between the interval size and the indicator of miscoverage.
Introduction
Learning algorithms are increasingly prevalent within consequential real-world systems, where reliability an essential consideration: confidently deploying learning algorithms requires more than high prediction accuracy in controlled testbeds 2[ Consider, for example, estimating the effects of drug for specific person given their demographic information and medical measurements. such high-stakes setting, giving point prediction for the drug's s effect is insufficient; the decision-maker must know what the plausible range of effects for this specific individual. Instance-wise uncertainty quantification in such settings is critical 315 One approach to this problem comes from the quantile regression and prediction interval literature 6 instead of a point prediction, we can return range of outcomes that represent the plausible response for given input. We would like these prediction intervals to achieve a pre-specified coverage level (e.g.. 90%) for all inputs- -tht is, across all regions of feature space. Training models to satisfy this validity guarantee is challenging, however, particularly with complex models like neural networks 10] In this work, we show how to generate prediction intervals that achieve coverage closer to the desired level evenly across all sub- uub-populations. Technically, we achieve this by augmenting the quantile regression loss function with an additional term that promotes appropriately balanced coverage across the feature space.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Formally, consider regression problem where we are given n training samples {(X,, Y;) where X IR is a feature vector, and Y R is a response variable. At test time, we observe feature vector X and our goal is to predict the unknown value of Y and and-importantly- on its uncertainty. In this work, we represent this uncertainty by constructing a prediction interval 1( S R that is likely to contain the response Yn+1. In particular, we seek to produce intervals that contain the response with a user-specified probability 1 - a that are valid across all regions of feature space:
report
P|Y++1 E (((x++1) | X,+1 = x] 2 11a,
(1)
a property known as conditional coverage. Notice that such prediction intervals are both correct in that they satisfy a coverage guarantee and also are adaptive, in that the size of the prediction intervals can change with the difficulty of the inputs: easy inputs give small intervals and hard inputs give large intervals. Returning to our medical example, consider predicting the outcome of drug from age, gender, blood pressure, and so on. The conditional coverage requirement in ① asks that intervals are correct for any age, gender, and health status combination. That is, no matter what an individual's value of the features x, the uncertainty quantification must be valid.
The conditional quantiles of Y X = I are the natural way to produce intervals satisfying ① Let Olo = a22 and Ohi 1 - az a/2. Given the true conditional quantiles, Ian (I), Ioa (2)), can build an oracle prediction interval satisfying in the following way:
(xx) = 19 (I), 9au (z)]
In practice, the conditional quantiles are unknown but can be estimated with quantile regression, yielding the interval C(x) 40.,, (2),
( (x)] This approach is attractive because quantile regression yields intervals that are adaptive to heteroscedasstccty without requiring parametric assumptions [T- [3]. but these intervals might not satisfy the conditional coverage statement (I), since qe,, (r), 4ay, (2)
are merely estimations of the true quantiles [14]. Indeed, we observe in experiments 5 that traditional quantile regression often gives intervals with poor conditional coverage, prompting the present investigation.
In this work, we propose a novel regularization scheme to push quantile regression algorithms towards solutions that better satisfies the conditional coverage requirement (T). The core idea is to force the coverage and interval length to be approximately independent, since this independence must hold for the optimal oracle intervals in ( A method that constructs intervals whose coverage and length are dependent is either sometimes too conservative (generating too wide intervals), sometimes too liberal (yeilding too short intervals), or both. In addition to improved training schemes, we propose two new tools to check the validity of the resulting predictions in meaningful way. Specifically, in 4 we present two new interpretable metrics to asses the violation of conditional coverage, taking advantage of the orthogonality property identified above. We use these (and other) metrics in Section to study our proposal on simulated data and nine real benchmark data sets. We find that our training scheme yields improvements when used together with both classic 66 and more recent [15] quantile regression method.
Section
synthetic two-group example
We begin with small synthetic experiment that demonstrates the challenges of constructing prediction intervals with accurate conditional coverage. We generate a dataset with two unbalanced sub- populations: 80% of the samples belong to majority group and the remaining 20% to minority group, where the conditional distribution Y X of the minority group is more dispersed than the majority group. In our experiments, group membership is included as one of the features. See Section S3.I of the Supplementary Material for a full description of the distribution.
As baseline method, we first fit a quantile neural network model (vanilla QR) by optimizing the pinball loss (see Section 2). attempting to estimate the low ag = 0.05 and high an 0.95 conditional quantiles. The left panel of Figure [ shows the coverage obtained by the vanilla QR model across training epochs. The coverage on the test data is far lower than suggested by the training data, failing to reach the desired 90% level and increasing as the training progresses. In particular, this gap remains large at the epoch in which the model achieves the minimal loss evaluated on an independent validation set. Here, the empirical test coverage measured over the majority and minority groups is equal to 80% and 68%, and the empirical average lengths evaluated over them are 1.55, 6.45, respectively.

---

Majority Group (vanilla QR)
Majority Group (orthogonal QR)
www.
0.8
0.6
Coverage
0.4
Train
Test
0.2
Nominal Coverage Level
Best Epoch
0.0
50
160
150
200
250
0
100
200
300
400
500
600
700
Epoch
Epoch
Minority Group (vanilla QR)
Minority Group (orthogonal QR)
0.8
Coverage
0.6
100
150
200
250
100 200
300
400
500
600
700
Epoch
Epoch
Figure I: Coverage as a function of the training epochs obtained by quantile natural network model; target coverage rate 90%. The vanilla QR model (left) is trained by optimizing the pinball loss whereas the orthogonal OR model (right) is fitted by combining the pinball loss with the proposed independence penalty. The purple vertical line marks the epoch that achieves the smallest empirical risk on validation set.
Next, we fit our proposed ort orthogonal QR model (which we will formally introduce in Section 3) on the same data. The coverage rate across epochs is illustrated in the right panel of Figure I In contrast to vanill 11a QR, the training and testing curves the majority group overlap for the entire training epochs and both approximately reach the desired 90% level. The minority group coverage has similar behavior, with significantly smaller gap between the two curves compared to van 1 QR. Here, the model that corresponds the best epoch achieves 86% coverage rate with 1.62 average length on the majority group, and 83% coverage rate with 9.33 average length on the minority group. Importantly, here we are able train for more epochs before overfitting, which leads better final model. To conclude, in this example or orthogonal QR prevents overfitting and leads to better conditional coverage.
2
Preliminaries and related work
2.1 Quantile regression
The task of estimating the low and high conditional quantiles, (x), Jong (1)), can be expressed as minimization problem taking the following form:
1 I CY.Y (X),
(Gow
Tcaa) argmin
(3)
foy EF
i=1
Above, la a loss function designed fit quantiles; we discuss two examples next.
The most common loss function for estimating a conditional quantile da is called the pinball loss or check function 6 [6,, expressed
Pa 9) [ayy
Ú O,
(1 a) y) otherwise.
Observe that for the choice a = 1/2 the above becomes the I., norm that is known estimate the conditional median. By setting * 1/2 get tilted version of the latter, with degree controlled

---

by the value of a. In the context of B we can set txt (y. (x), (2)) = Pan. (v. fan (z)) - Pon (y, for (x)) to simultaneously estimate the low and high quantiles. Throughout this paper, we will refer to this procedure as vani 11 la QR.
A recently-developee alternative the pinball loss is the interval score loss
defined as:
(v, fan (z), (x) =
2 (Son - Jap (x)) (Jon yy Lap (1)) 2 (y- ( (n,( (5)
Note that the left-most term encourages short intervals and the two remaining components promote intervals with the right coverage. To improve statistical efficiency, it is recommended by [TS] to simultaneously estimate all conditional quantiles by minimizing the following empirical risk function: E,--/o,11] (e (-) This is the approach we take in our experiments. Note that [[5] proposed additional learning schemes for improving efficiency, such as group batching and ensemble
learning; see also [8]. These ideas are complementary to our proposal and may further improve the
performance of our proposed orthogonal QR.
Quantile regression is large, active area of research, and we fimish by pointing out a few representa- tive strands of work in the literature. Estimates of conditional quantile functions using the pinball loss with specific models are proven to be asymptotically consistent under regularity conditions [[6]. The work reported in [3 offer a non- non-parametric version of quantile regression. This line of research was further developed by [19] that presented generalization to additive models that are non-paran non-parametric. The pinball loss and interval score loss can be also used estimate conditional probability distribution 17 20 Nevertheless, quantiles can be estimated in other ways rather than minimizing these two loss functions. These include quantile random forest { and the method proposed in [21] that iteratively estimates the conditional quantile through the Majorize-Mininne Minimize algorithm. Besides the regularization term we propose, there are other suggested penalties that are useful in different situations, such as using sparse modeling in high dimensional response [22 24
2.2 Conformal inference
Conformal inference (55 is a framework for building a prediction intervals that provably attain a weaker marginal coverage property:
P[Yn+1 E C(Xn+1)) 2 11-a a.
(6)
Importantly, one can guarantee that this holds for any joint distribution Pxy, sample size n, and
predictive algorithm. In contrast to ( the probability statement in (6) is marginal and taken over all
the training and test samples {(X,; - For example, in the context of the data from Figure 1
intervals that satisfy (6) would be allowed to undercover the minority group and overcover the majority group. Therefore, the statement (6) is much weaker than that in ①.. Yet, the former can be achieved for any distribution whereas the latter can not be achieved for badly-behaved distributions; see 14 26). While variants of the guarantee in (6) are possible 27-29 achieving coverage exactly balanced across a continuous feature cannot be done without further assumptions. Much recent work in conformal inference with quantile regression attempts to generate intervals that are adaptive to such heteroscedasticity so that they approximately achieve conditional coverage in ( while ensuring that 1 a marginal coverage in (6) is exactly achieved 7 35). The experiments we conduct show that our proposed Orthogona 1 QR method can be used in combination with conformal prediction to improve the conditional coverage property while attaining valid marginal coverage.
3 Proposed method:
orthogonal quantile regression
3.1 Formulating the learning scheme
This section presents modification of the pinball loss or interval score loss in order to fit models with improved conditional coverage. Denote by V = =1/Y C(X) the coverage identifier, and by [](x the interval's length. Our proposal motivated by the following observation.
'The code is hosted httpt/////tttaaaooo/o/ooooo//aaa

---

Proposition 1 (Independence of width and coverage), Let (X.Y) be sample drawn from Pxy. and let X be the support of X. If the distribution of Y fixed, deterministic interval-valuee
is continuous for all X. and the function C(X) satisfies P|Y E C(X)IX = =x] = 11- ar for all
X and some a E (0, 1). then the interval satisfies V 11 L.
In particular, the above implies that the length of an interval L qoe (X) qer. (X) constructed by the true low and high quantiles is independent of the coverage identifier V, as stated next.
Corollary 1. Under the assumptions Proposition quantiles satisfies V 1L L.
y interval constructed by the tree conditional
We note that an earlier, limited version of this observation appears in [36] in the context conditional coverage for classification problems. All proofs are presented in Section S2 of the Supplementary Material. Since intervals constructed by the conditional quantiles obey the independence property, forcing the fitted model to approximately satisfy this property during training can result in better conditional coverage for future test points. This leads us our proposed orthogonal QR objective:
(gom (cm) = argmin
la (o), (x.)) + +R V).
(7)
Joy EF
where la is either eeb or pint L IR" is vector that contains I,, San [X) as its elements, and V E R" is a vector with entries V = 1[,, E C(X))) (To facilitate training with gradient methods, in practice S1.1 of the Supplementary Material.) The function RL, V) E R returns a real-valued score that quantifies the strength of the dependence between I. and V, where large value indicates that the two are more dependent; we discuss specific choices in Section 3.2 The regularization strength is controlled by the hyperparameter In Supplementary Section [S4.1 explain how this parameter is determined, and in Supplementary Section 15.1.3 we demonstrate the effect this parameter on the performance of our method.
smooth approximation to the indicator function; see Section
Lastly, we point out that our proposal falls into the broader theme of fitting models while enforcing conditional independence properties, goal that is important for algorithmic faimesss [e.g..[37 39). This work aims to achieve uncertainty estimates that are equally good across all feature space, a prediction interval analog the goal of [40].
3.2 The orthogonality loss
We now turn to the question of choosing the specific dependence loss penalty, R in (7). In principle, could use any dependence measure from the many in the literature: chi-squared tests [41], Pearson's 's correlation, distance correlation [42), Kolmogorov-Snue Sominvv statistic [43], Randomized Dependence Coefficient [44), Hilbert-Schmiie independence criterion (HSIC) [5). and work we focus on Pearson's is correlation and HSIC which are described hereafter.
In this
The Pearson's 's correlation measures the linear dependency between two random variables. Here, the loss is defined as:
Cov(L, V)
Rever (L, V) =
(8)
L) / VVa((V)
The advantages of this choice are its simplicity and the minimal computational burden.
Next, HSIC is a more sophisticated, nonlinear complement to the Pearson's correlation measure, which can detect arbitrary complex relationships between the coverage identifier and the interval length. It is an analog of the well- Maximum Mean Discrepancy (MMD) distance [46]. but is measure of dependence. The idea is that while Rever (L, V) 0 does not necessarily imply that L and V are independent, having (R L), hvV)) = O for every continuous bounded functions g. guarantees the independence property [47]. While is impossible sweep over all possible continuous bounded functions, HSIC offers a tractable solution, guaranteeing that HSIC(L, 0 and only if 1L V [45). In our work, we utilize this measure, and define the orthogonality loss as RuSIc T., V)= = / HSIC(I., V) (taking the square root to magnify small values). This choice similar to the one advocated in [48]
5

---

With these choices of R. we now show that the true conditional quantiles are a solution for the orthogonal QR problem.
Theorem (Validity of orthogonal quantile regression). Suppose Y X follows continuous distribution for each & X, and suppose that gow (X), You (X) E F.
Consider the infinite-data version of the orthogonal QR optimization in
argmin Jaw
E fa) + O(X)))
where Č(X) = Von Sog la is either tht or Eor and R. is Reorr RuSIC. Then, true conditional quantiles are solutions to the above optimization problem. Moreover, if the solution is unique for then the solution is unique for all 0.
The uniqueness part of the theorem means that whenever vanilla QR is guaranteed to give the correct quantiles in the targe-sample limit, then orthogonal QR will give the same (correct) solution. The result continues to hold for any dependence measure R. that achieves its minimum value for any independent variables, basic property that all dependence measures that aware of satisfy. See the proof Section S2 of the Supplementary Material for further details.
Metrics for assessing conditional coverage
We next discuss several quantitative measures of conditional coverage. We will introduce two new metrics for conditional coverage, and then review one existing proposal from the literature. Lastly, we will discuss two ad-hoc metrics to help us compare orthogonal QR with vanilla QR in our upcoming simulation experiments.
4.1 Two new metries for conditional coverage
Pearson's s correlation: As previewed the previous section, the Pearson correlation be- tween the interval size and the indicator of coverage (i.e., R from (8) is simple, effective way to measure conditional coverage. However, to the best of our knowledge, we are the first to leverage for this purpose.
HSIC: Similarly, we consider the HSIC measure of dependence between the interval size and the indicator of coverage (i.e., RusIc above). We estimate this metric as described [40] As before, to our knowledge this has never been leveraged as a metric to asses conditional coverage.
4.2 Other metries for our empirical evaluations
AWSC:
As an additional measure of conditional coverage, we evaluate the coverage over the worst- slab as proposed in 49 To avoid case where an improvement in this quantity is obtained by naively enlarging all prediction intervals, we suggest a variant that we call AwsC. This metric is defined as the absolute difference between the worst-slab coverage and the marginal coverage, both evaluated on test data I:
Awsc WSC (((X)) Coverage (((X,, Yolrer
EiEI E Ō(x)) where ((x) is prediction inter- val method. Importantly, uniform increase of the length of all intervals will not deceive the AWSC measure as it will remain fixed.
Above, Coverage ((X,,, Y) Wher:
AILS-Coverage: We next consider measure that checks whether the intervals made larger by orthogonal OR compared to vanilla QR are necessary for improving the conditional coverage. In general, suppose given two algorithms A₁ and Ag for constructing prediction intervals. Let
AL, (x.) IC
The code is available at httpa////e///iitiooooo/oaaaaaaaa We used the implementation ftt.://ppp.///gggtta....//eeeeee

---

be the difference between the interval length (X,( obtained by A and Ag. evaluated on the same test point X, Next, let go. (oo(f&Li)eez) be the 90% empirical quantile of (ALi Then, let IISS be the 10% of samples whose length increased the most:
ILS = {i AL 2 goo(tAL4ee))
With this notation in place, we propose the AILS-Coverage metric:
AILS -ooverCee = Coverage (((X,, (i)) CA.) Coverage ((X,,))) :A,))
In words, the above is the absolute difference between the coverage over the ILS samples and the marginal coverage, evaluated for each algorithm Ak, k 1,2. A smaller value for k = indicates that the points with very different size under A₁ and Ag are handled better by Al.
As a variant of ÁILS-Coverage, we identify a cubp-olltttion character- ized by a small set of features such that the two algorithms A₁ and A2 produce very different intervals, and check the coverage on this region. To this end, we label the ILS samples as the positive class and fit binary classifier formulated as decision tree, aiming to predict whether a sample X belongs to the ILS set. Denote the set of tree nodes in depth at most three that contain at least 5% of the samples by [Node,],. where Node, C I. Next, let ND the set of indices of the samples that belong to the node that maximizes the following ratio: [Node; n ILS) Nodej ILS), Finally, given method for constructing prediction intervals C(-). compute the distance between the coverage over the ND samples and the marginal coverage, formulated as
ANode-Coverage
= Coverage ([(X;, KDDieND -Coverage (I(X, Kill (I)x)))))
Experiments
Armed with the performance metrics described in Section 4 now systematically quantify the effectiveness of the proposed independence penalty when combined with baseline quantile regression methods. In all experiments, prediction intervals with gives the details about the network architecture, training strategy, and details about this experimental setup. Software implementing the proposed method and reproducing our experiments can be found at https: / con/ Shaa122/111g
apply deep neural network as base model for constructing = 0.9 coverage level. Section [S4]of of the Supplementary Material
5.1 A synthetic two-group setting
We return to the synthetic two-group setting previewed Section 1 but first provide more details about the data. In this data set, the difference distribution between the majority and minority groups is controlled by modifying the noise level of the conditional distribution Y X of the minority group. Furthermore, X has 50 coordinates, the first of which indicates the group membership. Section SS.] of the Supplementary Material contains more details about the generation of this synthetic data. To analyze the performance of our method, we generate 7000 i.i.d. samples and repeat the following experiment for 30 random train/validation/test splits of the data. We fit quantile regression model with pinball loss on 5040 training samples, where we tune the number of epochs on an independent validation set that contains 560 samples. The remaining 1400 samples are used to test the model's s performance. We pre-processed the feature vector using z-score standardization, and normalized the features and response variables to have a zero mean and a unit variance.
In Table D we report the average coverage, length, and conditional coverage metries for vanilla QR and orthogonal QR for two minority-gron grroup noise levels. (The AILS-Coverae,, ANode-Coverage metrics are not reported in this case, since they both essentially correspond to the minority group's coverage level, which is given in the table.) The intervals constructed by the baseline vanilla QR undercover both the majority and minority group but; this tendency is more severe for the latter. By contrast, the regularized model achieves similar coverage rates for the two groups, with levels that are closer to the nominal 90%. This also reflected by the improvement in the Pearson's s correla lat i ion, IC, and AwSC metrics of conditional coverage. Overall, or orthogonal QR gives wider intervals for the minority group, which is anticipated since the cov- erage of the baseline model is far below the nominal rate. As for the majority group, proposed
7

---

Table I: Simulated data experiments. Performance of neural network quantile regression, using either vanilla OR (baseline) or orthogonal OR (OOR) with penalty term Reor The average coverage, length, and percent of improvement in the conditional coverage metries described in Section are evaluated over 30 independent trials. The standard errors for coverage and length are about 0.25, 0.06, respectively. The standard errors for the conditional coverage metrics are presented
in Supplementary Table S9
Minority
Majority Coverage (%)
Minority Coverage (%)
Majority Lengths
Minority Lengths
Improvement (%)
baseline DOR
baseline COOR
baseline / COR
baseline / OOR
cozE
#STC
AWSC
Low High
79.64 87.24
66.34 82.55
1.59 1.62
6.45 9.18
+64.07
+53.76
-13.88
86.97
83.74
1.88 1,71
+27.99
training gives intervals of about the same length compared to the baseline model, while achieving considerably higher coverage rate. fact, the regularized model constructs even shorter intervals in the high noise level case. We note that this performance continues to hold even when using interval score loss in place of the pinball loss; see Section S5.1.] the Supplementary Material. Lastly, also examine the effect of regularization wei gitee QR [50], and display the results in Section SS..1.2 of the Supplementary Material.
5.2
Real data
Next, we compare the performance of the proposed orthogonal QR to vanilla QR on nine benchmarks data sets as in [13] 30): Facebook comment volume variants and two (facebook_ facebook_2] 2), blog feedback bolo__dtta), physicochemical properties of protein tertiary structure (bio), forward kinematics of an 8 link robot arm (kin8nm), condition based maintenance of naval propulsion plants (naval), and medical expenditure panel survey number 19-21 (meps_19, meps_20, and meps_21 21). See Section S3.2 of the Supplementary Material for details about these data sets. We follow the experimental protocol and training strategy described in Section 5. We randomly split each data set into disjoint training (54%), validation (6%), and testing sets (40%). We normalized the features and response variables to have zero mean and unit variance each, except for facebook 1, facebook 2, blog data, and bio datasets in which we log transform Y before the standardization.
Table 2 summarizes the performance of vanilla QR and orthogonal QR. Our proposed method consistently improves the
conditional coverage, measured by the
Coverage
Pearson's s correlation
HSIC, and
ANSC metrics for conditional coverage, even though the latter two are not optimized directly
by orthogonal QR. In Figure we show the
vanilla orthogonal QR
coverage as function of the interval's length evaluated on the meps data set, and find that orthogonal QR (in orange) is closer to the nominal 90% level when compared to the base- line method. Our penalty also improves the in most data sets (see ble D), indicating that the baseline model tends to undercover the response of least one sub- population. Turning to the statistical efficiency, observe that the intervals produced by the regularized models tend to wider than the ones the baseline method, which is needed to better achieve conditional coverage. We further probe this phenomenon by checking the AILS- SLLL-Coeeragee which shows that the regions with wider intervals now have better coverage.
Length
Figure 2: Length versus coverage for vani lla QR and orthogonal QR over the meps_ 21 data. In Section $4.3 of the Supplementary Material we explain how the figure was constructed.
In Section S5.2 of the Supplementary Material we provide additional experiments by replacing the pinball loss with the inneoly-introuueedd interval score loss. The effect of the decorrelation penalty is similar to the one described above. Moreover, in Section SS.2 we also compare between the decorrelation and HSIC penalties for independence, and show that most data sets the decorrelation penalty achieves better performance over all metrics of conditional independence at the cost of

---

Table 2: Conditional coverage on real data. Performance neural network model for quantile regression, using either vanilla OR (baseline) or orthogonal QR (OQR) with penalty term Reorr. The average coverage, length, and percent improvement in the conditional coverage metrics described in Section 4 are evaluated over 30 independent trials. The standard errors for coverage and width are about 0.45 and 0.03, respectively. The standard errors for the conditional coverage metrics
presented in Supplementary Table SI1
Dataset Name
Coverage (%)
Length
Improvement (%)
baseline
OQR
haseline
OQR
CCTr
HSIC
AwsC
AILS
ANode
facebook
88.10
90.48
1.09
1.44
+81.08
+76.75
+33.42
+65.81
+47.30
facebook
87.38
91.13
+91.27
+96.36
433.86
+65.71
+55.63
blog bio 88.42
+77.49
89.08
1.88
+53.49
+75.55
-44.73
+43.20
+7.25
kinšnm
84.63
88.62
0.98
1.28
+27.54
+48.47
+14,97
457.38
-16.44
naval
89.89
89.50
0.56
1.49
+75.19
+33.98
-57.52
+72.36
-14.34
meps_ 19
82.44
85.16
0.84
1.00
+54.16
+22.25
+43.28
+88.05
+45,43
meps_ 20
82.81
84.27
0.86
1,03
+43.51
+1.99
+24.06
+84,71
+46.88
meps 21
82.50
84.07
0.86
0.99
+57.46
+13.62
+20.78
+85.02
+38.28
Table 3: Conditional coverage on real data, with conformalization. Performance of a neural network model for conformalized quantile regression, using either vanilla CQR (COR) or orthogonal COR (COQR) with penalty term Reor. Refer to the caption of Table 2 for further details. The standard errors for coverage and length are about 0.2 and 0.03, respectively. The standard errors for the conditional coverage metrics are presented in Supplementary Table S14
Dataset Name
Coverage (%)
Length
Improvement (%)
CQR
COQR
AwsC
AILS
ANode
facebook 1
90.08
90.11
+58.28
+37.61
+34.86
+67.79
+29.46
facebook
90.40
+76.25
+52.16
+29.51
+53.33
+52.46
blog data
89.93
90.15
1.41
L64
+13.77
-79.60
+24.38
+90.57
+39.38
bio 90.02
89.98
1.93
2.05
+51.19
+71.55
-30.80
+39.18
-40.88
kin8nm
90.11
90.11
1.12
1.33
+18.16
+13.06
-69.52
+58.14
-1.44
naval
89.84
90.06
0.55
1.51
+77.28
+36.78
-77.14
+68.24
-10.18
meps_ 19
89.96
89.84 84
0.92
1.05
+50.59
+39.94
+58.85
+83.77
+36.84
meps 20
90.08
90.19
0.95
1.09
+43.22
+39.66
+47.50
+84.13
+43.89
meps 21
89.93
89.89
0.94
1.05
+41.88
+29.69
+33.46
+79.47
+37,33
producing wider intervals. For completeness, we also present the results obtained by quantile regression forests
{ on the real data sets in Supplementary Table S15
Conformalized quantile regression results
In previous experiments the quantile regression methods tend to (marginally) undercover the response variables. This limitation is casily remedied by combining vanilla QR or ort orthogonal QR with conformalized quantile regression [30 that adjusts the estimated intervals to exactly achieve the marginal coverage property in (6) Table 3 summarizes the results, demonstrating that by conformalizing the intervals our proposed method precisely achieves the desired marginal coverage while improving the conditional coverage of the baseline model, measured by the Pearson! correlation ion, HSIC, and Awsc, ANode-Con Podde-Coverage metrics. The two independence metrics indicate that even after after adjusting the intervals to achieve marginal coverage, our method still results in improved independence between the coverage event and length. We note that the Awsc, and ANode ANode-Coveran Coveraage metries have more muted improvement compared the setting without conformalization, since the conformalization step smooths out the coverage to some extent. Further details regarding this conformalizing setting is given in Section S4.4 of the Supplementary Material.
Conclusion
In this work presented the orthogonal QR approach to achieve coverage closer to the desired level evenly across all sup-populations the setting of quantile regression algorithms. technical limitation of our method is the use of large batches during training, required to effectively detect the

---

dependencies between the intervals length and coverage events. In our experiments we focus on i.i data, but we believe that the orthogonal loss can be beneficial beyond this, such as in time-series data, which we hope to investigate in future work. A related future direction is to encourage independence between a coverage event and function of the feature vector, other than that of interval length- similar logic to that of our proposal means that this independence would hold for the true quanties. clever choice may better capture the relationships between X and the coverage obtained by the model, and further improve conditional coverage.
As a concluding remark, while empirical evidence shows that our orthogonal QR approach produces intervals that represent the uncertainty in subgroups more reliably than standard methods, it does not guarantee a valid coverage across all feature space with access to only a finite sample. This guarantee may be necessary for ensuring that predictions are unbiased against minority group of interest, indexed by an individual's 's gender or race, for example. To alleviate this, one can combine our methods with the equalized coverage framework (28) that builds upon conformal inference to achieve the desired coverage for pre-defined sub-populations, which is a weaker but achievable demand compared to conditional coverage.
Acknowledgments and Disclosure of Funding
S.F. and Y.R. were supported by the ISRAEL SCIENCE FOUNDATION (grant No. 729/21). Y.R. also thanks the Career Advancement Fellowship, Technion, for providing research support. We thank John Cherian for comments on an earlier version of this manuscript.
References
[1] Timothy John Sullivan. Introduction to uncertainty quantification, volume 63. Springer, 2015.
[2] Christian Soize. Uncertainty quantification. Springer, 2017.
[3] Lewis H. Mervin, Simon Johansson, Elizaveta Semenova, Kathryn A. Giblin, and Ola Engkvist. Uncertainty quantification in drug design. Drug Discovery Today, 26(2):474-489, 2021.
[4] Ryutaro Tanno, Daniel Worrall, Enrico Kaden, Aurobrata Ghosh, Francesco Grussu, Alberto Bizzi, Stamatios N. Sotiropoulos, Antonio Criminisi, and Daniel C. Alexander. Uncertainty modelling in deep learning for safer neuroimage enhancement: Demonstration in diffusion MRI. Neurolmage, 225:1 17366, 2021.
[5] Edmon Begoli, Tanmoy Bhattacharya, and Dimitri F. Kusnezov. The need for uncertainty quantification in machine-assisted medical decision making. Nature Machine Intelligence (Online), 1(1), 2019.
[6] Roger Koenker and Gilbert Bassett. Regression quantiles. Econometrica, 46(1)333 50, 1978.
[7] Rafael Izbicki, Gilson Shimizu. and Rafael Stern. Flexible distribution-free conditional pre- dictive bands using density estimators. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 3068-3077. PMLR. 26-28 Aug 2020.
[8] Nicolai Meinshausen. Quantile regression forests. J. Mach. Learn Res., 7:983-999, December 2006.
[9] Roger Koenker and Kevin F. Hallock. Quantile regression. Journal of Economic Perspectives, 15(4):143- 156, December 2001.
[10] Yichen Jia and Jong-Hyeon Jeong. Deep learning for quantile regression under right censoring: DeepQuantreg. arXiv preprint trXiv:2007.07055 2021.
[11] Ichiro Takeuchi, Quoc V. Le, Timothy D. Sears, and Alexander J. Smola. Nonparametric quantile estimation. Journal of Machine Learning Research, 7(45):1231-1266) 2006.
[12] Kenneth Q Zhou and Stephen L. Portnoy. Direct use of regression quantiles to construct confidence sets in linear models. The Annals of Statistics, 24(1):287 -306, 1996.
10

---

[13] Kenneth Q. Zhou and Stephen L. Portnoy. Statistical inference on heteroscedastic models based on regression quantiles. Journal of Nonparametric Statistics, 9(3):339 260, 1998.
[14] Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan Tibshirani. The limits of distribution-fies conditional predictive inference. Information and Inference: A Journal of the IMA, 2019.
[15] Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. arXiv preprint arXiv:2011.09558 2020.
[16] Ingo Steinwart and Andreas Christmann. Estimating conditional quantiles with the help of the pinball loss. Bernoulli, 17(1), Feb 2011.
[17] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378 2007.
[18] Taesup Kim, Rasool Fakoor, Jonas Mueller, Alexander Smola, and Ryan J Tibshirani. Deep quantile aggregation. arXiv preprint arXiv:2103.0000 2021.
[19] Roger Koenker. Additive models for quantile regression: Model selection and confidence bandaids. Brazilian Journal of Probability and Statistics, 25(3):239 - 262, 2011.
[20] James W. Taylor. A quantile regression neural network approach to estimating the conditional density of multiperiod returns. Journal of Forecasting, 1994)1299-311 2000.
[21] David R. Hunter and Kenneth Lange. Quantile regression via an MM algorithm. Journal of Computational and Graphical Statisties, 9(1):60-77, 2000.
[22] Alexandre Belloni, Victor Cheroozuuko, and Kengo Kato. Valid post-selection inference in nigh-dimensional approximately sparse quantile regression models. Journal of the American Statistical Association, (4(526):749-7588 2019.
[23] Vidyashankar Sivakumar and Arindam Banerjee. High-dimensional structured quantile regres- sion. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 3220-3229, 06-11 Aug 2017.
[24] Alexandre Belloni and Victor Chernozhukov. Ll-penalized quantile regression in high- dimensional sparse models. The Annals of Statistics, 39(1):82-130, 2011.
[25] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.
[26] Dhruv Medarametla and Emmanuel Candès. Distribution- conditional median inference. arXiv preprint arXiv:2102.07967, 2021.
[27] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pages 475-490. PMLR, 2012.
[28] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review, 2(2), 2020. https://hss..mipress. http://ww...iiccceu////eddeeeeee
[29] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free free, risk-con cisk-contoolling prediction sets. arXiv preprint. 2021. arXiv:2101.022 0100000.......
[30] Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Advances Neural Information Processing Systems, volume 32, 2019.
[31] Matteo Sesia and Emmanuel Candès. A comparison of some conformal quantile regression methods. Stat, 91), Jan 2020.
[32] Danijel Kivaranovic, Kory D Johnson, and Hannes Leeb. Adaptive, distribution- freebe prediction intervals for deep networks. In International Conference on Artificial Intelligence and Statistics, pages 4346-4356. PMLR, 2020.

---

[33] Victor Cheroohuuovv, Kaspar Wüthrich, and Yinchu Zhu. Distributional conformal prediction. arXiv preprint arXiv: wwt.i.99990899. 2021.
[34] Chirag Gupta, Arun K. Kuchibhotla, and Aaditya K. Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. arXiv preprint (iti::9910.05622 2021.
[35] Rafael Izhicki, Gilson Shimizu, and Rafael B. Stern, CD-split and HPD-split: efficient conformal regions in high dimensions. arXiv preprint arXiv:2007. 2021.
[36] Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint, 2020. arXiv:2009. 14193.
[37] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodrigus and Krishna P. Gummadi. Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research, 20(75): 4-2, 2019.
[38] Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 2796-2806, 2018.
[39] Yaniv Romano, Stephen Bates, and Emmanuel Candes. Achieving equalized odds by resampling sensitive attributes. In Advances in Neural Information Processing Systems, volume 33, pages 361-371, 2020.
[40] Daniel Greenfeld and Uri Shalit. Robust learning with the Hilbert-Schmidt independence criterion. International Conference on Machine Learning, pages
PMLR, 2020.
[41] Karl Pearson. On the Criterion that a Given System of Deviations from the Probable the Case of Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling, pages 11-28. Springer New York, 1992.
[42] Gábor J. Szckely, Maria L. Rizzo, and Nail K. Bakirov. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6), Dec 2007.
[43] Vladimir Ivanovich Smirnov. Kolmogorov-Sminon Test, pages 283-287. Springer New York, New York, NY, 2008
[44] David Lopez-Paz, Philipp Hennig, and Bernhard Schölkopf. The randomized dependence coef- ficient. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 1, page 1-9, Red Hook, NY, USA, 2013.
[45] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with Schbeett-Shhiit norms. In Algorithmic Learning Theory, pages 63-77, 2005.
[46] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012.
[47] Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungar- ica, 10(3):441-45, Sep 1959.
[48] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In Interna- tional Conference on Machine Learning, pages 1718-1727. PMLR, 2015.
[49] Maxime Cauchois, Suyash Gupta, and John C. Duchi. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of Machine Learning Research, 22(81):1-42 2021.
[50] Roger Koenker. Quantile Regression. Econometric Society Monographs. Cambridge University Press, 2005.
12


Reviews:
[{"review_id": "wEHQRRG1M6e", "reviewer": "Reviewer_CaJn", "summary": "The authors propose a novel method to construct prediction intervals with pre-specified conditional coverage probability. Starting point of the paper is the observation that the length $|\\hat{C}(X)|$ and conditional coverage indicator $1\\{Y \\in \\hat{C}(X)\\}$ are independent whenever the prediction interval $\\hat{C}(X)$ has exact coverage probability.  The authors therefore suggest to compute lower and upper bounds of the prediction interval by solving a quantile regression problem augmented by a penalty term that penalizes correlation between $|\\hat{C}(X)|$ and $1\\{Y \\in \\hat{C}(X)\\}$. The authors compare the performance of their procedure and competing methods on simulated data and nine benchmark data sets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper. It is very clearly written and makes an interesting contribution. Using quantile regression to construct prediction intervals with (asymptotically) correct conditional coverage probability is well-established. Augmenting this traditional approach with a penalty to encourage independence of length and conditional coverage indicator to improve finite sample performance is creative.\n\nMy main concern is that the key reason for the poor finite sample performance of the vanilla QR approach might not be due to the violation of the independence property, but simple because of the variability of the quantile regression estimates themselves. Instead of using classical QR, it would be interesting to see whether weighted/ efficient QR (Koenker 2005, Ch. 5.3) already leads to improved conditional coverage. By how much would enforcing the orthogonality condition further improve coverage based on weighted QR?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. They discuss limitations with regard to implementation of their method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "ZBRlILfnVvP", "reviewer": "Reviewer_dPMb", "summary": "This paper discusses a regularization scheme that aims to improve the conditional coverage of prediction intervals that are learned while performing quantile regression for a pair of quantiles. The empirical experiments show the effect of applying the regularization scheme by demonstrating improvement across a suite of metrics.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly written and well motivated. Learning accurate conditional quantiles and prediction intervals (PIs) is a relevant problem, especially to the general UQ community.\n\nThe proposed method is based on the findings presented in Proposition 1, which is applied via a regularization term as stated in equation (7). I believe these are relevant and interesting findings, and provide insights in the problem of learning accurate quantiles and PIs.\n\nI have some questions about the method, which is stated below in the Limitations section.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have a couple of concerns and questions regarding the method, and empirical evaluations. \n\nMethod:\n\n1. How does the regularization term impact the optimization procedure? I can see multiple ways in which the R(L,V)=0 can be satisfied, e.g. a very wide or narrow prediction interval which always has V=1 or V=0, or any homoscedastic prediction which always satisfies L=constant. Will the regularization term push the converged solution towards these degenerate cases? \n\n2. In relation to the first question, how should $\\gamma$ be tuned in (7)?\n\n3. Because of 1), I am dubious of the proposed metrics. E.g. if we only consider a class of homoscedastic models, all models in this class will have R_{corr} and R_{HSIC} = 0? \n\nEmpirical evaluation:\n\n4. The evaluation mainly displays how appending the regularization R(L,V) on top of other standard QR losses can improve performance. It's been observed in previous literature that vanilla QR (just optimizing the pinball loss) in the deep learning setting often results in miscalibration, and post-hoc recalibration methods can adjust the quantiles to achieve better calibration. What improvements does orthogonal QR provide over vanilla QR + recalibration? I believe conformalized QR (Section 5.3 and Table 3) is a form of post-hoc recalibration, and I'm not sure if there are significant improvements when comparing the OQR columns in Table 2 with CQR columns in Table 3.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KjrVCSj2vTE", "reviewer": "Reviewer_ZNkM", "summary": "The paper shows a new loss function that can be used for quantile regression. The key idea is to combine loss functions often used in quantile regression with a new penalty that enforces conditional coverage. Results show improvements according to several metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The language of the paper is very clear. The ideas and math are also technically correct. Quantile regression is relevant for uncertainty quantification, and thus the contributions are important.\n\nThe key to the development of the paper (in particular of the loss function) is Proposition 1, which is neat. Although similar results have been used in other contexts, I have never seen it being used for quantile regression. The empirical results are encouraging.\n\nMajor comments:\n\n- How is gamma (the penalty factor) chosen in practice? How sensitive are the empirical results to such a choice? I think this is a very important point that is not addressed in the paper.\n\n- It is natural that the method will outperform others in terms of Pearson and HSIC metrics (because they are explicitly used in the loss function); the improvements with respect to the other metrics are more appealing. However, if gamma is large, wouldn't it be the case that all metrics that were computed in the experiments would be better for OQR (because they basically measure conditional coverage), even though a large gamma does not guarantee that the quantiles would be well estimated? That is, a large gamma enforces good conditional coverage, but by itself, it would not enforce the quantiles to be correctly estimated. Other estimates also satisfy conditional coverage. If this is the case, I feel that other metrics would be necessary to complement the results (such as the pinball loss).\n\n- I also miss comparisons to other quantile estimators that don't directly target the pinball loss, such as quantile regression forests.\n\nMinor comments:\n\n- is the reference to interval score loss \"[17]\" correct? I think it should be \"[15]\".\n\n- it would be useful to explain what the worst-slab coverage means", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, limitations have been addressed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new type of regularization for quantile regression such that the accuracy of finite sample conditional coverage is improved. The proposed regularization is built on an interesting notion about independence between the size of the intervals and the indicator of a miscoverage event. All reviewers agreed that learning accurate conditional quantiles and prediction intervals (PIs) is an important problem in the uncertainty quantification community. The authors are encouraged to consider updating the paper based on the reviewers' comments and suggestions.
Decision: Accept (Poster)