Improving Conditional Coverage via Orthogonal Quantile Regression
Year: 2021

Abstract:
We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, a property called conditional coverage. A typical approach to this task is to estimate the conditional quantiles with quantile regression—it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage. To remedy this, we modify the loss function to promote independence between the size of the intervals and the indicator of a miscoverage event. For the true conditional quantiles, these two quantities are independent (orthogonal), so the modified loss function continues to be valid. Moreover, we empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics. We also introduce two new metrics that check conditional coverage by looking at the strength of the dependence between the interval size and the indicator of miscoverage.

heading: 1 Introduction
text: Learning algorithms are increasingly prevalent within consequential real-world systems, where reliability is an essential consideration: confidently deploying learning algorithms requires more than high prediction accuracy in controlled testbeds [1, 2]. Consider, for example, estimating the effects of a drug for a specific person given their demographic information and medical measurements. In such a high-stakes setting, giving a point prediction for the drug’s effect is insufficient; the decision-maker must know what the plausible range of effects for this specific individual. Instance-wise uncertainty quantification in such settings is critical [3–5]. One approach to this problem comes from the quantile regression and prediction interval literature [6–8]; instead of a point prediction, we can return a range of outcomes that represent the plausible response for a given input. We would like these prediction intervals to achieve a pre-specified coverage level (e.g., 90%) for all inputs—that is, across all regions of feature space. Training models to satisfy this validity guarantee is challenging, however, particularly with complex models like neural networks [9, 10]. In this work, we show how to generate prediction intervals that achieve coverage closer to the desired level evenly across all sub-populations. Technically, we achieve this by augmenting the quantile regression loss function with an additional term that promotes appropriately balanced coverage across the feature space.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Formally, consider a regression problem where we are given n training samples {(Xi, Yi)}ni=1, where X ∈ Rp is a feature vector , and Y ∈ R is a response variable. At test time, we observe a feature vector Xn+1 and our goal is to predict the unknown value of Yn+1 and—importantly—to report on its uncertainty. In this work, we represent this uncertainty by constructing a prediction interval Ĉ(Xn+1) ⊆ R that is likely to contain the response Yn+1. In particular, we seek to produce intervals that contain the response with a user-specified probability 1− α that are valid across all regions of feature space:
P[Yn+1 ∈ Ĉ(Xn+1) | Xn+1 = x] ≥ 1− α, (1) a property known as conditional coverage. Notice that such prediction intervals are both correct in that they satisfy a coverage guarantee and also are adaptive, in that the size of the prediction intervals can change with the difficulty of the inputs: easy inputs give small intervals and hard inputs give large intervals. Returning to our medical example, consider predicting the outcome of a drug from age, gender, blood pressure, and so on. The conditional coverage requirement in (1) asks that intervals are correct for any age, gender, and health status combination. That is, no matter what an individual’s value of the features x, the uncertainty quantification must be valid.
The conditional quantiles of Y | X = x are the natural way to produce intervals satisfying (1). Let αlo = α/2 and αhi = 1− α/2. Given the true conditional quantiles, qαlo(x), qαhi(x), we can build an oracle prediction interval satisfying (1) in the following way:
C(x) = [qαlo(x), qαhi(x)]. (2)
In practice, the conditional quantiles are unknown but can be estimated with quantile regression, yielding the interval Ĉ(x) = [q̂αlo(x), q̂αhi(x)]. This approach is attractive because quantile regression yields intervals that are adaptive to heteroscedasticity without requiring parametric assumptions [11– 13], but these intervals might not satisfy the conditional coverage statement (1), since q̂αlo(x), q̂αhi(x) are merely estimations of the true quantiles [14]. Indeed, we observe in experiments 5 that traditional quantile regression often gives intervals with poor conditional coverage, prompting the present investigation.
In this work, we propose a novel regularization scheme to push quantile regression algorithms towards solutions that better satisfies the conditional coverage requirement (1). The core idea is to force the coverage and interval length to be approximately independent, since this independence must hold for the optimal oracle intervals in (2). A method that constructs intervals whose coverage and length are dependent is either sometimes too conservative (generating too wide intervals), sometimes too liberal (yeilding too short intervals), or both. In addition to improved training schemes, we propose two new tools to check the validity of the resulting predictions in a meaningful way. Specifically, in Section 4, we present two new interpretable metrics to asses the violation of conditional coverage, taking advantage of the orthogonality property identified above. We use these (and other) metrics in Section 5 to study our proposal on simulated data and nine real benchmark data sets. We find that our training scheme yields improvements when used together with both a classic [6] and a more recent [15] quantile regression method.

heading: A synthetic two-group example
text: We begin with a small synthetic experiment that demonstrates the challenges of constructing prediction intervals with accurate conditional coverage. We generate a dataset with two unbalanced subpopulations: 80% of the samples belong to a majority group and the remaining 20% to a minority group, where the conditional distribution Y | X of the minority group is more dispersed than the majority group. In our experiments, group membership is included as one of the features. See Section S3.1 of the Supplementary Material for a full description of the distribution.
As a baseline method, we first fit a quantile neural network model (vanilla QR) by optimizing the pinball loss (see Section 2), attempting to estimate the low αlo = 0.05 and high αhi = 0.95 conditional quantiles. The left panel of Figure 1 shows the coverage obtained by the vanilla QR model across training epochs. The coverage on the test data is far lower than suggested by the training data, failing to reach the desired 90% level and increasing as the training progresses. In particular, this gap remains large at the epoch in which the model achieves the minimal loss evaluated on an independent validation set. Here, the empirical test coverage measured over the majority and minority groups is equal to 80% and 68%, and the empirical average lengths evaluated over them are 1.55, 6.45, respectively.
Next, we fit our proposed orthogonal QR model (which we will formally introduce in Section 3) on the same data. The coverage rate across epochs is illustrated in the right panel of Figure 1. In contrast to vanilla QR, the training and testing curves of the majority group overlap for the entire training epochs and both approximately reach the desired 90% level. The minority group coverage has similar behavior, with a significantly smaller gap between the two curves compared to vanilla QR. Here, the model that corresponds to the best epoch achieves 86% coverage rate with 1.62 average length on the majority group, and 83% coverage rate with 9.33 average length on the minority group. Importantly, here we are able to train for more epochs before overfitting, which leads to a better final model. To conclude, in this example orthogonal QR prevents overfitting and leads to better conditional coverage.

heading: 2 Preliminaries and related work

heading: 2.1 Quantile regression
text: The task of estimating the low and high conditional quantiles, q̂αlo(x), q̂αhi(x), can be expressed as a minimization problem taking the following form:
(q̂αlo , q̂αhi) = argmin fαlo ,fαhi∈F
1
n n∑ i=1 `α(Yi, fαlo(Xi), fαhi(Xi)). (3)
Above, `α is a loss function designed to fit quantiles; we discuss two examples next.
The most common loss function for estimating a conditional quantile q̂α is called the pinball loss or check function [6, 9, 16], expressed as
ρα(y, ŷ) = { α(y − ŷ) y − ŷ > 0, (1− α)(ŷ − y) otherwise. (4)
Observe that for the choice α = 1/2 the above becomes the L1 norm that is known to estimate the conditional median. By setting α 6= 1/2 we get a tilted version of the latter, with a degree controlled
by the value of α. In the context of (3), we can set `pbα (y, fαlo(x), fαhi(x)) = ραlo(y, fαlo(x)) + ραhi(y, fαhi(x)) to simultaneously estimate the low and high quantiles. Throughout this paper, we will refer to this procedure as vanilla QR.
A recently-developed alternative to the pinball loss is the interval score loss [17], defined as:
`intα (y, fαlo(x), fαhi(x)) =
(fαhi(x)− fαlo(x)) + 2
α (fαlo(x)− y)1[y < fαlo(x)] +
2 α (y − fαhi(x))1[y > fαhi(x)]. (5)
Note that the left-most term encourages short intervals and the two remaining components promote intervals with the right coverage. To improve statistical efficiency, it is recommended by [15] to simultaneously estimate all conditional quantiles by minimizing the following empirical risk function: Eα∼U [0,1][`intα (·)].1 This is the approach we take in our experiments. Note that [15] proposed additional learning schemes for improving efficiency, such as group batching and ensemble learning; see also [18]. These ideas are complementary to our proposal and may further improve the performance of our proposed orthogonal QR.
Quantile regression is a large, active area of research, and we finish by pointing out a few representative strands of work in the literature. Estimates of conditional quantile functions using the pinball loss with specific models are proven to be asymptotically consistent under regularity conditions [16]. The work reported in [11–13] offer a non-parametric version of quantile regression. This line of research was further developed by [19] that presented a generalization to additive models that are non-parametric. The pinball loss and interval score loss can be also used to estimate conditional probability distribution [17, 20]. Nevertheless, quantiles can be estimated in other ways rather than minimizing these two loss functions. These include quantile random forest [8] and the method proposed in [21] that iteratively estimates the conditional quantile through the Majorize-Minimize algorithm. Besides the regularization term we propose, there are other suggested penalties that are useful in different situations, such as using sparse modeling in high dimensional response [22–24].

heading: 2.2 Conformal inference
text: Conformal inference [25] is a framework for building a prediction intervals that provably attain a weaker marginal coverage property:
P[Yn+1 ∈ Ĉ(Xn+1)] ≥ 1− α. (6)
Importantly, one can guarantee that this holds for any joint distribution PXY , sample size n, and predictive algorithm. In contrast to (1), the probability statement in (6) is marginal and taken over all the training and test samples {(Xi, Yi)}n+1i=1 . For example, in the context of the data from Figure 1, intervals that satisfy (6) would be allowed to undercover the minority group and overcover the majority group. Therefore, the statement in (6) is much weaker than that in (1). Yet, the former can be achieved for any distribution whereas the latter can not be achieved for badly-behaved distributions; see [14, 26]. While variants of the guarantee in (6) are possible [27–29], achieving coverage exactly balanced across a continuous feature cannot be done without further assumptions. Much recent work in conformal inference with quantile regression attempts to generate intervals that are adaptive to such heteroscedasticity so that they approximately achieve conditional coverage in (1), while ensuring that 1− α marginal coverage in (6) is exactly achieved [7, 30–35]. The experiments we conduct show that our proposed Orthogonal QR method can be used in combination with conformal prediction to improve the conditional coverage property while attaining valid marginal coverage.

heading: 3 Proposed method: orthogonal quantile regression

heading: 3.1 Formulating the learning scheme
text: This section presents a modification of the pinball loss or interval score loss in order to fit models with improved conditional coverage. Denote by V = 1[Y ∈ Ĉ(X)] the coverage identifier, and by L = |Ĉ(X)| the interval’s length. Our proposal is motivated by the following observation.
1The code is hosted at https://github.com/YoungseogChung/calibrated-quantile-uq
Proposition 1 (Independence of width and coverage). Let (X,Y ) be a sample drawn from PXY , and let X be the support of X . If the distribution of Y | X = x is continuous for all x ∈ X , and the fixed, deterministic interval-valued function Ĉ(X) satisfies P[Y ∈ Ĉ(X)|X = x] = 1− α for all x ∈ X and some α ∈ (0, 1), then the interval satisfies V ⊥⊥ L.
In particular, the above implies that the length of an interval L = qαhi(X)− qαlo(X) constructed by the true low and high quantiles is independent of the coverage identifier V , as stated next. Corollary 1. Under the assumptions of Proposition 1, an interval constructed by the true conditional quantiles satisfies V ⊥⊥ L.
We note that an earlier, limited version of this observation appears in [36] in the context conditional coverage for classification problems. All proofs are presented in Section S2 of the Supplementary Material. Since intervals constructed by the true conditional quantiles obey the independence property, forcing the fitted model to approximately satisfy this property during training can result in better conditional coverage for future test points. This leads us to our proposed orthogonal QR objective:
(q̂αlo , q̂αhi) = argmin fαlo ,fαhi∈F
1
n n∑ i=1 `α(Yi, fαlo(Xi), fαhi(Xi)) + γR(L,V). (7)
where `α is either ` pb α or `intα , L ∈ Rn is a vector that contains Li = fαhi(Xi)− fαlo(Xi) = |Ĉ(Xi)| as its elements, and V ∈ Rn is a vector with entries Vi = 1[Yi ∈ Ĉ(Xi)]. (To facilitate training with gradient methods, in practice we use a smooth approximation to the indicator function; see Section S1.1 of the Supplementary Material.) The functionR(L,V) ∈ R+ returns a real-valued score that quantifies the strength of the dependence between L and V , where a large value indicates that the two are more dependent; we discuss specific choices in Section 3.2. The regularization strength is controlled by the hyperparameter γ. In Supplementary Section S4.1 we explain how this parameter is determined, and in Supplementary Section S5.1.3 we demonstrate the effect of this parameter on the performance of our method.
Lastly, we point out that our proposal falls into the broader theme of fitting models while enforcing conditional independence properties, a goal that is important for algorithmic fairness [e.g., 37–39]. This work aims to achieve uncertainty estimates that are equally good across all feature space, a prediction interval analog to the goal of [40].

heading: 3.2 The orthogonality loss
text: We now turn to the question of choosing the specific dependence loss penalty,R in (7). In principle, we could use any dependence measure from the many in the literature: chi-squared tests [41], Pearson’s correlation, distance correlation [42], Kolmogorov-Smirnov statistic [43], Randomized Dependence Coefficient [44], Hilbert-Schmidt independence criterion (HSIC) [45], and so on. In this work we focus on Pearson’s correlation and HSIC which are described hereafter.
The Pearson’s correlation measures the linear dependency between two random variables. Here, the loss is defined as:
Rcorr(L, V ) = ∣∣∣∣∣ Cov(L, V )√Var(L)√Var(V ) ∣∣∣∣∣ . (8)
The advantages of this choice are its simplicity and the minimal computational burden.
Next, HSIC is a more sophisticated, nonlinear complement to the Pearson’s correlation measure, which can detect arbitrary complex relationships between the coverage identifier and the interval length. It is an analog of the well-known Maximum Mean Discrepancy (MMD) distance [46], but is a measure of dependence. The idea is that while Rcorr(L, V ) = 0 does not necessarily imply that L and V are independent, having Rcorr(g(L), h(V )) = 0 for every continuous bounded functions g, h guarantees the independence property [47]. While it is impossible to sweep over all possible continuous bounded functions, HSIC offers a tractable solution, guaranteeing that HSIC(L, V ) = 0 if and only if L ⊥⊥ V [45]. In our work, we utilize this measure, and define the orthogonality loss as RHSIC(L, V ) = √ HSIC(L, V ) (taking the square root to magnify small values). This choice is similar to the one advocated in [48].
With these choices of R, we now show that the true conditional quantiles are a solution for the orthogonal QR problem. Theorem 1 (Validity of orthogonal quantile regression). Suppose Y | X = x follows a continuous distribution for each x ∈ X , and suppose that qαlo(X), qαhi(X) ∈ F . Consider the infinite-data version of the orthogonal QR optimization in (7):
argmin fαlo ,fαhi∈F
E [ `α(Y, fαlo(X), fαhi(X)) + γR ( |Ĉ(X)|, I[Y ∈ Ĉ(X)] )] ,
where Ĉ(X) = [fαlo(X), fαhi(X)], `α is either ` pb α or ` int α , and R is Rcorr or RHSIC. Then, true conditional quantiles are solutions to the above optimization problem. Moreover, if the solution is unique for γ = 0 then the solution is unique for all γ > 0.
The uniqueness part of the theorem means that whenever vanilla QR is guaranteed to give the correct quantiles in the large-sample limit, then orthogonal QR will give the same (correct) solution. The result continues to hold for any dependence measure R that achieves its minimum value for any two independent variables, a basic property that all dependence measures that we are aware of satisfy. See the proof in Section S2 of the Supplementary Material for further details.

heading: 4 Metrics for assessing conditional coverage
text: We next discuss several quantitative measures of conditional coverage. We will introduce two new metrics for conditional coverage, and then review one existing proposal from the literature. Lastly, we will discuss two ad-hoc metrics to help us compare orthogonal QR with vanilla QR in our upcoming simulation experiments.

heading: 4.1 Two new metrics for conditional coverage
text: Pearson’s correlation: As previewed in the previous section, the Pearson correlation between the interval size and the indicator of coverage (i.e.,Rcorr from (8)) is a simple, effective way to measure conditional coverage. However, to the best of our knowledge, we are the first to leverage it for this purpose.
HSIC: Similarly, we consider the HSIC measure of dependence between the interval size and the indicator of coverage (i.e.,RHSIC above). We estimate this metric as described in [40].2 As before, to our knowledge this has never been leveraged as a metric to asses conditional coverage.

heading: 4.2 Other metrics for our empirical evaluations
text: ∆WSC: As an additional measure of conditional coverage, we evaluate the coverage over the worstslab as proposed in [49].3 To avoid a case where an improvement in this quantity is obtained by naively enlarging all prediction intervals, we suggest a variant that we call ∆WSC. This metric is defined as the absolute difference between the worst-slab coverage and the marginal coverage, both evaluated on test data I:
∆WSC = ∣∣∣WSC({(Xi, Yi)}i∈I ; Ĉ)− Coverage({(Xi, Yi)}i∈I ; Ĉ)∣∣∣ .
Above, Coverage ( {(Xi, Yi)}i∈I ; Ĉ ) = 1|I| ∑ i∈I 1[Yi ∈ Ĉ(Xi)] where Ĉ(x) is a prediction interval method. Importantly, a uniform increase of the length of all intervals will not deceive the ∆WSC measure as it will remain fixed.
∆ILS-Coverage: We next consider a measure that checks whether the intervals made larger by orthogonal QR compared to vanilla QR are necessary for improving the conditional coverage. In general, suppose we are given two algorithms A1 and A2 for constructing prediction intervals. Let
∆Li = |ĈA1(Xi)| − |ĈA2(Xi)| 2The code is available at https://github.com/danielgreenfeld3/XIC 3We used the implementation from https://github.com/msesia/arc/
be the difference between the interval length |ĈA(Xi)| obtained by A1 and A2, evaluated on the same test point Xi. Next, let q0.9({∆Li}i∈I) be the 90% empirical quantile of {∆Li}i∈I . Then, let ILS be the 10% of samples whose length increased the most:
ILS = {i : ∆Li ≥ q0.9({∆Li}i∈I), i ∈ I}.
With this notation in place, we propose the ∆ILS-Coverage metric: ∆ILS-Coverage = ∣∣∣Coverage({(Xi, Yi)}i∈ILS ; ĈAk)− Coverage({(Xi, Yi)}i∈I ; ĈAk)∣∣∣.
In words, the above is the absolute difference between the coverage over the ILS samples and the marginal coverage, evaluated for each algorithm Ak, k = 1, 2. A smaller value for k = 1 indicates that the points with very different size under A1 and A2 are handled better by A1. ∆Node-Coverage: As a variant of ∆ILS-Coverage, we identify a sub-population characterized by a small set of features such that the two algorithmsA1 andA2 produce very different intervals, and check the coverage on this region. To this end, we label the ILS samples as the positive class and fit a binary classifier formulated as a decision tree, aiming to predict whether a sample X belongs to the ILS set. Denote the set of tree nodes in depth at most three that contain at least 5% of the samples by {Nodej}j , where Nodej ⊆ I. Next, let ND be the set of indices of the samples that belong to the node that maximizes the following ratio: |Nodej ∩ ILS| / |Nodej \ ILS|. Finally, given a method for constructing prediction intervals Ĉ(·), compute the distance between the coverage over the ND samples and the marginal coverage, formulated as
∆Node-Coverage = ∣∣∣Coverage({(Xi, Yi)}i∈ND ; Ĉ)− Coverage({(Xi, Yi)}i∈I ; Ĉ)∣∣∣ .

heading: 5 Experiments
text: Armed with the performance metrics described in Section 4, we now systematically quantify the effectiveness of the proposed independence penalty when combined with baseline quantile regression methods. In all experiments, we apply a deep neural network as a base model for constructing prediction intervals with 1 − α = 0.9 coverage level. Section S4 of the Supplementary Material gives the details about the network architecture, training strategy, and details about this experimental setup. Software implementing the proposed method and reproducing our experiments can be found at https://github.com/Shai128/oqr

heading: 5.1 A synthetic two-group setting
text: We return to the synthetic two-group setting previewed in Section 1, but first provide more details about the data. In this data set, the difference in distribution between the majority and minority groups is controlled by modifying the noise level of the conditional distribution Y | X of the minority group. Furthermore, X has 50 coordinates, the first of which indicates the group membership. Section S3.1 of the Supplementary Material contains more details about the generation of this synthetic data. To analyze the performance of our method, we generate 7000 i.i.d. samples and repeat the following experiment for 30 random train/validation/test splits of the data. We fit a quantile regression model with pinball loss on 5040 training samples, where we tune the number of epochs on an independent validation set that contains 560 samples. The remaining 1400 samples are used to test the model’s performance. We pre-processed the feature vector using z-score standardization, and normalized the features and response variables to have a zero mean and a unit variance.
In Table 1 we report the average coverage, length, and conditional coverage metrics for vanilla QR and orthogonal QR for two minority-group noise levels. (The ∆ILS-Coverage, ∆Node-Coverage metrics are not reported in this case, since they both essentially correspond to the minority group’s coverage level, which is given in the table.) The intervals constructed by the baseline vanilla QR undercover both the majority and minority group but; this tendency is more severe for the latter. By contrast, the regularized model achieves similar coverage rates for the two groups, with levels that are closer to the nominal 90%. This is also reflected by the improvement in the Pearson’s correlation, HSIC, and ∆WSC metrics of conditional coverage. Overall, orthogonal QR gives wider intervals for the minority group, which is anticipated since the coverage of the baseline model is far below the nominal rate. As for the majority group, our proposed
training gives intervals of about the same length compared to the baseline model, while achieving a considerably higher coverage rate. In fact, the regularized model constructs even shorter intervals in the high noise level case. We note that this performance continues to hold even when using interval score loss in place of the pinball loss; see Section S5.1.1 of the Supplementary Material. Lastly, we also examine the effect of our regularization on weighted QR [50], and display the results in Section S5.1.2 of the Supplementary Material.

heading: 5.2 Real data
text: Next, we compare the performance of the proposed orthogonal QR to vanilla QR on nine benchmarks data sets as in [15, 30]: Facebook comment volume variants one and two (facebook_1, facebook_2), blog feedback (blog_data), physicochemical properties of protein tertiary structure (bio), forward kinematics of an 8 link robot arm (kin8nm), condition based maintenance of naval propulsion plants (naval), and medical expenditure panel survey number 19-21 (meps_19, meps_20, and meps_21). See Section S3.2 of the Supplementary Material for details about these data sets. We follow the experimental protocol and training strategy described in Section 5.1. We randomly split each data set into disjoint training (54%), validation (6%), and testing sets (40%). We normalized the features and response variables to have a zero mean and a unit variance each, except for facebook_1, facebook_2, blog_data, and bio datasets in which we log transform Y before the standardization.
Table 2 summarizes the performance of vanilla QR and orthogonal QR. Our proposed method consistently improves the conditional coverage, as measured by the Pearson’s correlation, HSIC, and ∆WSC metrics for conditional coverage, even though the latter two are not optimized directly by orthogonal QR. In Figure 2, we show the coverage as a function of the interval’s length evaluated on the meps_21 data set, and find that orthogonal QR (in orange) is closer to the nominal 90% level when compared to the baseline method. Our penalty also improves the ∆Node-Coverage in most data sets (see Table 2), indicating that the baseline model tends to undercover the response of at least one sub-
population. Turning to the statistical efficiency, observe that the intervals produced by the regularized models tend to be wider than the ones of the baseline method, which is needed to better achieve conditional coverage. We further probe this phenomenon by checking the ∆ILS-Coverage which shows that the regions with wider intervals now have better coverage.
In Section S5.2 of the Supplementary Material we provide additional experiments by replacing the pinball loss with the recently-introduced interval score loss. The effect of the decorrelation penalty is similar to the one described above. Moreover, in Section S5.2 we also compare between the decorrelation and HSIC penalties for independence, and show that in most data sets the decorrelation penalty achieves better performance over all metrics of conditional independence at the cost of
producing wider intervals. For completeness, we also present the results obtained by quantile regression forests [8] on the real data sets in Supplementary Table S15.
Conformalized quantile regression results
In previous experiments the quantile regression methods tend to (marginally) undercover the response variables. This limitation is easily remedied by combining vanilla QR or orthogonal QR with conformalized quantile regression [30] that adjusts the estimated intervals to exactly achieve the marginal coverage property in (6). Table 3 summarizes the results, demonstrating that by conformalizing the intervals our proposed method precisely achieves the desired marginal coverage while improving the conditional coverage of the baseline model, as measured by the Pearson’s correlation, HSIC, and ∆WSC, ∆Node-Coverage metrics. The two independence metrics indicate that even after after adjusting the intervals to achieve marginal coverage, our method still results in improved independence between the coverage event and length. We note that the ∆WSC, and ∆Node-Coverage metrics have a more muted improvement compared to the setting without conformalization, since the conformalization step smooths out the coverage to some extent. Further details regarding this conformalizing setting is given in Section S4.4 of the Supplementary Material.

heading: 6 Conclusion
text: In this work we presented the orthogonal QR approach to achieve coverage closer to the desired level evenly across all sup-populations in the setting of quantile regression algorithms. A technical limitation of our method is the use of large batches during training, required to effectively detect the
dependencies between the intervals length and coverage events. In our experiments we focus on i.i.d data, but we believe that the orthogonal loss can be beneficial beyond this, such as in time-series data, which we hope to investigate in future work. A related future direction is to encourage independence between a coverage event and a function of the feature vector, other than that of interval length— similar logic to that of our proposal means that this independence would hold for the true quanties. A clever choice may better capture the relationships between X and the coverage obtained by the model, and further improve conditional coverage.
As a concluding remark, while empirical evidence shows that our orthogonal QR approach produces intervals that represent the uncertainty in subgroups more reliably than standard methods, it does not guarantee a valid coverage across all feature space with access to only a finite sample. This guarantee may be necessary for ensuring that predictions are unbiased against a minority group of interest, indexed by an individual’s gender or race, for example. To alleviate this, one can combine our methods with the equalized coverage framework [28] that builds upon conformal inference to achieve the desired coverage for pre-defined sub-populations, which is a weaker but achievable demand compared to conditional coverage.
Acknowledgments and Disclosure of Funding
S.F. and Y.R. were supported by the ISRAEL SCIENCE FOUNDATION (grant No. 729/21). Y.R. also thanks the Career Advancement Fellowship, Technion, for providing research support. We thank John Cherian for comments on an earlier version of this manuscript.

Review summaries:
The authors propose a novel method to construct prediction intervals with pre-specified conditional coverage probability. Starting point of the paper is the observation that the length $|\hat{C}(X)|$ and conditional coverage indicator $1\{Y \in \hat{C}(X)\}$ are independent whenever the prediction interval $\hat{C}(X)$ has exact coverage probability.  The authors therefore suggest to compute lower and upper bounds of the prediction interval by solving a quantile regression problem augmented by a penalty term that penalizes correlation between $|\hat{C}(X)|$ and $1\{Y \in \hat{C}(X)\}$. The authors compare the performance of their procedure and competing methods on simulated data and nine benchmark data sets.

The paper shows a new loss function that can be used for quantile regression. The key idea is to combine loss functions often used in quantile regression with a new penalty that enforces conditional coverage. Results show improvements according to several metrics.