Regression Conformal Prediction under Bias

## Abstract

Abstract Uncertainty quantification is crucial to account for the imperfect predictions of machine learning algorithms for high-impact applications. Conformal prediction (CP) is a powerful framework for uncertainty quantification that generates calibrated prediction intervals with valid coverage.
In this work, we study how CP intervals are affected by bias – the systematic deviation of a prediction from ground truth values – a phenomenon prevalent in many real-world applications.
We investigate the influence of bias on interval lengths of two different types of adjustments – symmetric adjustments, the conventional method where both sides of the interval are adjusted equally, and asymmetric adjustments, a more flexible method where the interval can be adjusted unequally in positive or negative directions.
We present theoretical and empirical analyses characterizing how symmetric and asymmetric adjustments impact the “tightness” of CP intervals for regression tasks.
Specifically for absolute residual and quantile-based non-conformity scores, we prove:
1) the upper bound of symmetrically adjusted interval lengths increases by 2 ⁢ | b | 2 𝑏 2|b| 2 | italic_b | where b 𝑏 b italic_b is a globally applied scalar value representing bias,
2) asymmetrically adjusted interval lengths are not affected by bias, and
3) conditions when asymmetrically adjusted interval lengths are guaranteed to be smaller than symmetric ones.
Our analyses suggest that even if predictions exhibit significant drift from ground truth values, asymmetrically adjusted intervals are still able to maintain the same tightness and validity of intervals as if the drift had never happened, while symmetric ones significantly inflate the lengths.
We demonstrate our theoretical results with two real-world prediction tasks: sparse-view computed tomography (CT) reconstruction and time-series weather forecasting. Our work paves the way for more bias-robust machine learning systems.

## 1 Introduction

With the growing application of deep learning algorithms to high-impact applications such as healthcare, finance, and climate science, it is equally crucial to develop methods that can robustly quantify their uncertainties. This is particularly important since deep learning algorithms are known to yield confident yet incorrect prediction values [, , ]. Given a prediction by a learning algorithm on a fresh test example, uncertainty quantification methods typically aim to return a *prediction set* with some guarantee that the true value lies within that set. In particular, a prediction set for a regression problem consists of an interval with lower and upper bounds [, ].

Conformal Prediction (CP) is a powerful family of uncertainty quantification methods that is *distribution-agnostic*, i.e., makes no assumptions about the underlying data distribution, and generates prediction sets with guarantees of containing ground truth values with some probability [, , , ].
For example, split CP is based on collecting a separate (from training) *calibration* dataset containing both ground truth and predicted values from the algorithm of interest. Then, given the algorithm’s prediction on a test example, split CP computes a *non-conformity* score quantifying how “unusual” new predictions will be with respect to the calibration dataset, and adjusts the prediction via the empirical quantile of the calibration scores to generate a prediction set.

Figure: Figure 1: Key Intuition. Conformal prediction interval lengths computed using symmetric adjustments significantly increase with increasing bias, where bias is defined as the systematic deviation of a prediction from ground truth. On the other hand, those computed using asymmetric adjustments are not affected by bias. We aim to understand how bias impacts symmetrically and asymmetrically adjusted prediction interval lengths.
Refer to caption: extracted/5896715/figs/fig1_intuition.png

In practice, to minimize prediction uncertainty, we aim to obtain the tightest possible intervals that maintain valid coverage. Interval tightness (or, inversely, its length) depends on various factors: non-conformity scores, data distributions, and underlying models []. There has been much theoretical work analyzing aspects of interval length, including optimal efficiency [, , , ], expected set sizes [], conditional and marginal set size differences [], and differences in set sizes between oracle and estimated prediction intervals [, ]. Existing empirical work includes investigations into lengths under covariate shift [], skewed distributions [], and heteroskedasticity [, ].

One important scenario, however, that has not been investigated is how CP fares for a learning algorithm that is *biased*, i.e., produces predictions that systematically deviate from ground truth values. For example, we can define the bias $b$ of an algorithm as the mean difference between the expected values of its predictions $\hat{Y}$ with respect to ground truth $Y$ over the calibration set:

$$ $b(Y,\hat{Y})=\frac{1}{n}\sum_{i=1}^{n}(\mathbb{E}[\hat{Y}_{i}]-Y_{i})$ (1) $$

Bias is a well-known issue plaguing machine learning models due to various factors such as skewed training distributions [], sensor drift [, , ], concept drift [, , ], attrition bias [], and noisy labels [].
We find that large bias inflates the conventional symmetrically adjusted interval lengths (Fig. 1-left) because the intervals must be adjusted equally in both positive and negative directions.

In this paper, we argue that the effects of bias on CP interval lengths can be mitigated by computing intervals with asymmetric adjustments [, ] instead of conventional symmetric adjustments. Asymmetric adjustments allow lower and upper endpoints to be adjusted independently to account for directional bias, maintaining the guarantee that the resulting interval contains the ground truth with high probability (Fig. 1-right). While asymmetric adjustments have been theorized to yield longer interval lengths as a consequence of stronger guarantees [], our work observes that with bias, that may not be the case. We expand the theoretical understanding of CP interval lengths for absolute residual ($L_{1}$) and quantile adjusted (i.e., Conformalized Quantile Regression or CQR []) non-conformity scores by analyzing their behavior for symmetrically and asymmetrically adjusted interval lengths under prediction bias. Specifically, we prove the following:

- 1.
The upper bound of symmetrically adjusted interval lengths increases by $2|b|$ (Thm. 2),
- 2.
Asymmetrically adjusted interval lengths are not affected by bias (Thm. 3), and
- 3.
Conditions when asymmetrically adjusted interval lengths are guaranteed to be smaller than those of symmetric adjustments (Cor. 3.1).

Our theoretical results are significant for several reasons.
First, existing theory shows that while asymmetric adjustments give stronger coverage guarantees, they also result in slightly longer intervals []. However, contrary to this theory, it has also been empirically observed that asymmetric adjustments yield tighter intervals than symmetric ones [, , ],
but the underlying reasons were unclear. Our work provides a theoretical answer to this phenomenon. Second, our theoretical results suggest that asymmetric adjustments are preferable in practice to symmetric ones when systematic biases are expected, e.g., with sensor drift. However, if symmetric adjustments are desired (e.g., when error distributions are assumed to be symmetric), we also offer a method to achieve tighter symmetric intervals during calibration.

We validate that our theoretical analyses align with synthetic and real prediction tasks.
Our synthetic experiments validate our theoretical analyses in the ideal setting when $n$ is large for no skew and skewed distributions.
Our real tasks validate our theoretical analyses in two settings: when $n$ is extremely low and time series.
Our real tasks deal with bias arising in different contexts: (1) when a medical imaging reconstruction algorithm systematically under- or over-estimates volumes of an anatomical region (computed tomography (CT) reconstruction), and (2) temporal “drift” of values over time (weather forecasting).
The contributions of this study give machine learning practitioners fundamental and practical insight into using CP under the common scenario of biased predictions.

## 2 Background: Split Conformal Prediction (CP)

We focus on a “split” CP setup [, ] in this work, but the same theoretical analysis can be applied to other CP forms and extensions [, ].
In split CP, we assume a calibration dataset $D_{C}=\{(\hat{Y}_{1},Y_{1}),...,(\hat{Y}_{n},Y_{n})\}$ and test point $\hat{Y}_{n+1}$, where $\hat{Y}_{i}$ and $Y_{i}$ represent the $i$-th prediction and ground truth values. The calibration data is separate from a training dataset used to train the ML algorithm of interest. The calibration dataset and test point are assumed to be exchangeable. The goal of CP is to construct a prediction interval $C(\hat{Y}_{n+1})=[L(\hat{Y}_{n+1}),U(\hat{Y}_{n+1})]$ for $\hat{Y}_{n+1}$, where $L(\hat{Y}_{n+1}),U(\hat{Y}_{n+1})\in\mathbb{R}$ are lower and upper bounds, such that $\mathbb{P}[Y_{n+1}\in C(\hat{Y}_{n+1})]\geq 1-\alpha$, for some user-specified mis-coverage rate $\alpha\in(0,1)$.
To compute symmetric intervals, we perform the following steps.
First, for each data point in the calibration set $D_{C}$, we compute non-conformity scores $S=\{s_{1},...,s_{n}\}$.
Next, we compute the $(1-\alpha)$-th empirical quantile of the non-conformity scores $q=Q_{1-\hat{\alpha}}(S)$, where $\hat{\alpha}=\frac{\lfloor\alpha(n+1)\rfloor}{n+1}$ denotes the finite-sample adjusted mis-coverage rate.
Finally, we adjust the predictions of the test data using $q$ to achieve valid prediction sets.
This algorithm provides marginal coverage: on average, the prediction sets contain ground truth $(1-\alpha)\%$ of the time. More rigorously, based on key CP results:

###### Lemma 1

Let $(\hat{Y}_{i},Y_{i})\in\mathbb{R}\times\mathbb{R},i=1,...,n+1$ be exchangeable random variables.
Assume that a predictor $f$ has been trained on a proper training set independent of and exchangeable with these $n+1$ points.
Consider a calibration set ${(\hat{Y}_{i},Y_{i})}_{i=1}^{n}$ and a fresh test point $\hat{Y}_{n+1}$.
Let $s_{i}$ be a non-conformity score computed using the predictor $f$ for $i=1,...,n+1$.
Let $q=Q_{1-\hat{\alpha}}(\{s_{i}\}_{i=1}^{n})$ be the $\lceil(1-\alpha)(n+1)\rceil$-th smallest value of $\{s_{i}\}_{i=1}^{n}$ and $C(\hat{Y}_{n+1})=\{y\in\mathbb{R}:s_{n+1}\leq q\}$ be the prediction set for the test point $\hat{Y}_{n+1}$.
Then, for any $\alpha\in(0,1)$:

- 1.
$\mathbb{P}[Y_{n+1}\in C(\hat{Y}_{n+1})]\geq 1-\alpha$ and
- 2.
$\mathbb{P}[Y_{n+1}\in C(\hat{Y}_{n+1})]\leq 1-\alpha+\frac{1}{n+1}$ if random variables $Y_{1},...,Y_{n+1}$ are almost surely distinct

Proof: See variations in [, , , ]

Many non-conformity scores exist [], including absolute residuals ($L_{1}$) [] and quantile-based (Conformalized Quantile Regression or CQR  []) scores.

This can be extend to asymmetric adjustments [, , ] by computing $(1-\alpha_{lo})$-th and $(1-\alpha_{hi})$-th empirical quantiles of the conformity scores, where $\alpha_{lo}$ and $\alpha_{hi}$ are lower and upper mis-coverage rates.
In the asymmetric case, the empirical quantiles for the lower and higher mis-coverage rates $\alpha_{lo}$ and $\alpha_{hi}$ are given by $\hat{\alpha}_{lo}=\frac{\lfloor\alpha_{lo}(n+1)\rfloor}{n+1}$ and $\hat{\alpha}_{hi}=\frac{\lfloor\alpha_{hi}(n+1)\rfloor}{n+1}$.
The proof of validity relies on applying Lem. 1 twice  [, ].
It is easy to see that when $\alpha_{lo}+\alpha_{hi}=\alpha$ the asymmetric case yields empirically larger coverage based on the finite sample adjustment.
The reason is due to the “rounding effect” of the ceiling function, which tends to push the empirical quantiles toward more extreme values (larger or smaller), especially when $n$ is small. Since the asymmetric case deals with two separate quantiles, this effect is compounded, leading to a prediction set that empirically offers larger coverage. However, as we will see, symmetrically adjusted interval lengths may not always be shorter than asymmetric ones in the presence of bias.

###### Lemma 1

## 3 Theoretical Analysis

We assume biased predictions $\hat{Y}_{i}^{b}=\hat{Y}_{i}^{0}+b$ where $b\in\mathbb{R}$ is a constant (positive or negative) applied between all unbiased predicted values ($\hat{Y}^{0}$) and ground truth values ($Y$) from the calibration set.
For example, for prediction and ground truth distributions that are symmetric and centered around their means, we can use Eq. 1. However, when the distributions are skewed, the mean may no longer be a good measure of central tendency [, ].
In Sec. 3.1, we will use results from our theoretical analyses to estimate bias more accurately in these cases.

We consider symmetric non-conformity scores with canonical expression:

$$ $s_{i}^{b}=\max(f_{lo}(\hat{Y}_{i}^{b})-Y_{i},Y_{i}-f_{hi}(\hat{Y}_{i}^{b})),$ (2) $$

where $f_{lo}$ and $f_{hi}$ are the lower adjustment and upper adjustment functions that have linear properties: $f_{lo}(\hat{Y}_{i}^{b})=f_{lo}(\hat{Y}_{i}^{0})+b$ and $f_{hi}(\hat{Y}_{i}^{b})=f_{hi}(\hat{Y}_{i}^{0})+b$.
Eq. 2 covers the conventional $L_{1}$ and CQR non-conformity scores.
For the $L_{1}$ non-conformity score given by $s_{i}^{b}=|Y_{i}-\hat{Y}_{i}^{b}|$, $\hat{Y}_{i}^{b}$ represents a point estimate, and the score can be rewritten as $s_{i}^{b}=\max(\hat{Y}_{i}^{b}-Y_{i},Y_{i}-\hat{Y}_{i}^{b})$.
For the CQR non-conformity score given by $s_{i}^{b}=\max(Q_{\alpha_{lo}}(\hat{Y}_{i}^{b})-Y_{i},Y_{i}-Q_{1-\alpha_{hi}}(
\hat{Y}_{i}^{b}))$, $\hat{Y}_{i}^{b}$ represents a set of samples $\hat{Y}_{i}^{b}=\{\hat{Y}_{ij}^{b}\}_{j=1}^{n_{s}}$.
The adjustment is given by $q^{b}=Q_{1-\hat{\alpha}}(\{s_{i}^{b}\}_{i=1}^{n})$, the prediction interval is given by $C(\hat{Y}^{b}_{n+1})=[f_{lo}(\hat{Y}^{b}_{n+1})-q^{b},f_{hi}(\hat{Y}^{b}_{n+1}
)+q^{b}]$, and the interval length is given by $L_{sym}(\hat{Y}^{b}_{n+1})=f_{hi}(\hat{Y}^{b}_{n+1})-f_{lo}(\hat{Y}^{b}_{n+1})
+2q^{b}$.
This setup does not cover locally adaptive non-conformity scores [, , ] and variations of CQR such as CQR-r and CQR-m non-conformity scores  [].
Using Eq. 2, we first derive an upper bound for symmetrically adjusted interval lengths under bias (Thm. 2):

###### Theorem 2

Given biased predictions for a fresh test point $\hat{Y}_{n+1}^{b}=\hat{Y}_{n+1}^{0}+b$, the upper bound on prediction interval lengths of non-conformity scores described in Eq. 2 is:

$$ $L_{sym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{0})+2|b|,$ (3) $$

where $L_{sym}(\hat{Y}_{n+1}^{b})$ and $L_{sym}(\hat{Y}_{n+1}^{0})$ are the interval lengths computed using symmetric adjustments for predictions with and without bias.

Proof: See App. A.1

We find the upper bounds of symmetrically adjusted interval lengths increase linearly with the magnitude of bias.
Next, we show 1) that asymmetric adjustments are not affected by bias and 2) conditions when using asymmetric adjustments produce shorter lengths than symmetric adjustments. To accomplish this, we introduce a similar canonical expression for asymmetric non-conformity scores:

$$ $(s_{i,lo}^{b},s_{i,hi}^{b})=(f_{lo}(\hat{Y}_{i}^{b})-Y_{i},Y_{i}-f_{hi}(\hat{Y }_{i}^{b})),$ (4) $$

where $s_{i,lo}^{b}$ and $s_{i,hi}^{b}$ represent lower and upper non-conformity score adjustments when predictions are biased with $b$. The lower and upper asymmetric adjustments are computed by taking the $(1-\alpha_{lo})$-th and $(1-\alpha_{hi})$-th empirical quantile of the sets of non-conformity scores
$q_{lo}^{b}=Q_{1-\hat{\alpha}_{lo}}(\{s_{i,lo}^{b}\}_{i=1}^{n})$ and $q_{hi}^{b}=Q_{1-\hat{\alpha}_{hi}}(\{s_{i,hi}^{b}\}_{i=1}^{n}$).
Thus, the prediction interval is $C(\hat{Y}_{n+1}^{b})=[f_{lo}(\hat{Y}_{n+1}^{b})-q_{lo}^{b},f_{hi}(\hat{Y}_{n+1
}^{b})+q_{hi}^{b}]$.

Using this setup, we prove the following relationship for the length of a CP prediction interval using asymmetric non-conformity scores, under bias $b$:

###### Theorem 3

Given biased predictions for a fresh test point $\hat{Y}_{n+1}^{b}=\hat{Y}_{n+1}^{0}+b$,
the lengths for $L_{1}$ and CQR non-conformity scores computed using asymmetric adjustments are bias-independent:

$$ $L_{asym}(\hat{Y}_{n+1}^{b})=L_{asym}(\hat{Y}_{n+1}^{0})$ (5) $$

where $L_{asym}(\hat{Y}_{n+1}^{b})$ and $L_{asym}(\hat{Y}_{n+1}^{0})$ are the interval lengths computed using asymmetric adjustments for predictions with and without bias.

Proof: See App. A.2

Figure: Algorithm 1 Estimating bias by minimizing symmetrically-adjusted interval lengths; Example using gradient descent.

We find that asymmetric adjustments are not affected at all by a constant bias $b$, which is a desirable property. However, recall that when predictions are unbiased, asymmetrically adjusted intervals tend to be longer than symmetric ones []. This raises the question: at what level of bias does this behavior reverse? We derive conditions under which, in the presence of bias, asymmetrically adjusted intervals become shorter than symmetric ones:

###### Corollary 3.1

For $L_{1}$ and CQR non-conformity scores, asymmetric adjustments produce smaller interval lengths than symmetric adjustments under the following condition:

$$ $2|b|\geq L_{asym}(\hat{Y}_{n+1}^{0})-L_{sym}(\hat{Y}_{n+1}^{0}).$ (6) $$

where $b$ is the bias, $L_{sym}(\hat{Y}_{n+1}^{0})$ and $L_{asym}(\hat{Y}_{n+1}^{0})$ are lengths computed using symmetric and asymmetric adjustments for predictions without bias.

Proof: The result is derived by using Thm. 3, setting $L_{asym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{b})$, substituting Eq. 3, and rearranging the inequality.

We find that when the difference in lengths for predictions without bias is greater than 2 times the magnitude of bias, the asymmetrically adjusted interval lengths will be guaranteed to be shorter than symmetric ones.

Our theoretical analyses (Thm. 2, Thm. 3 and Cor. 3.1) provide important insights into how bias affects length and the conditions which asymmetric adjustments yield shorter lengths than and symmetric adjustments. In practice, when $\alpha_{lo}+\alpha_{hi}=\alpha$ and $n$ is large, the interval lengths under no bias are approximately equal $L_{asym}(\hat{Y}_{n+1}^{0})\approx L_{sym}(\hat{Y}_{n+1}^{0})$, and the lengths for predictions with bias are shorter for asymmetric compared to symmetric adjustments $L_{asym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{b})$ when $|b|>0$.

###### Theorem 2

###### Theorem 3

###### Corollary 3.1

### 3.1 Estimating bias

One unanswered question is how to empirically determine bias $b$ given data. To do so, we leverage Thm. 2 and adjust the predictions by a scalar value $b_{eff}$ to minimize the maximum symmetrically adjusted interval lengths:

$$ $b_{eff}=\operatorname*{arg\,min}_{C}\bigg{[}\max(\{L_{sym}(\hat{Y}_{i}^{b}-C) \}_{i=1}^{n})\bigg{]}$ (7) $$

$L_{sym}(\hat{Y}_{i}^{b}-b_{eff})$ achieves the minimum length for symmetric adjustments (when $b=0$).
$b_{eff}$ can be thought of as the “debiasing” constant for biased predictions $\hat{Y}^{b}$. We prove that the objective function in Eq. 7 reduces to minimizing a vertically and horizontally translated absolute value function (App. A.2.1).
Therefore, the objective function is convex, and Alg. 1 converges using gradient descent and its variants. For our experiments, we implemented a PyTorch version for CQR-based and $L_{1}$ scores available at [https://github.com/matthewyccheung/conformal-metric](https://github.com/matthewyccheung/conformal-metric), and optimize using AutoGrad [].

## 4 Experiments

Figure: Figure 2: Synthetic experiments with skewed and noisy predictions align with theoretical analysis. We set $N(10,5)$ as the ground truth distribution and added $W(1,0,5)$ (right skew), $N(0,2)$ (no skew), and $-W(1,-2,5)$ (left skew) to simulate imperfect predictions. The parameter descriptions can be found in the scipy.stats documentation. We plot the bias versus the maximum length for symmetric CQR (red) and asymmetric CQR (green) . We also plot the theoretical upper bound (Thm. 2, dashed grey) and the value when lengths with asymmetric adjustments are smaller than those with symmetric adjustments (equality in Cor. 3.1, black).
Refer to caption: extracted/5896715/figs/fig2_synthetic.png

We next evaluate Thm. 2 and 3 and Cor. 3.1 for $L_{1}$ and CQR non-conformity scores with synthetic and real-life experiments.
For synthetic experiments, we assume normally distributed ground truth data, and simulate predictions by adding different types of noise. For real-life experiments, we consider two scenarios: when data is scarce (CT reconstructions for downstream radiotherapy planning using CQR) and when data is temporally varying (time series weather forecasting using $L_{1}$). The data distributions can be found in App. B.

### 4.1 Synthetic Data

We first demonstrate the validity of the theoretical analysis Sec. 3 for CQR using Gaussian $N$ and Weibull $W$ distributions to simulate estimate, ground truth, and noise distributions. We used $N(10,5)$ to simulate a ground truth distribution. We added noise characterized by $W(1,0,5)$, $N(0,2)$, and $-W(1,-2,5)$ to the ground truth samples to simulate left-, no-, and right-skewing predictions.
We used 1000 calibration data points, 1000 test data points, and 1000 samples per data point to estimate the quantiles. We set $\alpha=0.1$ for symmetric adjustments, and $\alpha_{lo}=\alpha_{hi}=0.05$ for asymmetric adjustments.
After determining $b_{eff}$ using Alg. 1, we added a constant bias term from $-2$ to $2$ to the debiased predictions to examine the effect of biased predictions on lengths (Fig. 2).

Results in Fig. 2 confirm our theoretical analyses. We find that symmetrically adjusted interval lengths (red) are always upper bounded by the sum of length at $b=0$ and $2|b|$ (dashed black) holds true (Thm 2).
We find that asymmetrically adjusted interval lengths (green) do not change under bias (Thm. 3) and that they are always smaller than symmetrically adjusted lengths (red) when Cor. 3.1 is true (dashed grey).

### 4.2 Real Data

Next, we validate our theoretical analyses in two different real data scenarios: where upstream image reconstruction tasks may not fully capture spatial dependencies in downstream metrics (sparse-view computed tomography (CT) reconstruction), and where predictions “drift” from the ground truth over time (time series weather forecasting).
Through these two examples, we aim to show the validity and usefulness of our theoretical analysis from different perspectives.

#### 4.2.1 Limited Data

Figure: Figure 3: Biases from using sparse-view CT reconstructions for a downstream segmentation task. We show slices of 4 different patient ground truth CT volumes. Each slice is overlaid with right lung segmentations from 10 probabilistically sampled reconstructions (red) and segmentations of the ground truth right lung (blue). The reconstructed right lung segmentations consistently overestimate organ volumes compared to the ground truth segmentations. See App. C for experiment details.
Refer to caption: extracted/5896715/figs/fig4_seg.png

In scenarios with limited imaging capabilities, such as low-resource clinics [, , ], reconstruction algorithms work with observations that do not contain complete information. For example, sparse cone-beam CT algorithms use limited ($<100$ instead of the standard $100$s) 2D X-ray observations to generate 3D CT scans [, , ]. The observed information is insufficient to recover the true image with complete certainty, leading to potential biases such as systematically over- or under-estimating organ volumes.

We simulate a medical imaging pipeline, where a patient is imaged using sparse-CT, an image reconstruction algorithm is applied to the projections, and the resulting volume is used for downstream radiotherapy planning (RT).
We use Neural Attenuation Fields (NAF) [], a self-supervised image reconstruction algorithm.
We synthetically injected noise to the projections, reconstructed the volumes using different initializations of the reconstruction algorithm, and generated plans using the Radiation Planning Assistant (RPA, FDA 510(k) cleared)
(^1^11RPA is a web-based tool that combines organ segmentation algorithms and physics simulations for RT planning.).
More details about our experimental setup can be found in App. C.

To validate our theoretical analysis in Sec. 3 holds true even for extremely low $n$, we use 19 patients for calibration and 1 patient for testing.
We generate 10 reconstructions per patient by perturbing acquisition angles, injecting noise into the projections, and using random initializations of NAF.
We perform leave-one-out cross-validation to examine each patient’s interval lengths when calibrated with the rest of the patients.
We use $\alpha=0.15$ for symmetric adjustments and $\alpha_{lo}=\alpha_{hi}=0.075$ for asymmetric adjustments, corresponding to $\hat{\alpha}=0.0567$ for the symmetric case and $\hat{\alpha}_{lo}=\hat{\alpha}_{hi}=0$ for the asymmetric case.
In the asymmetric case, this corresponds to an extreme case of taking the maximum and minimum non-conformity scores.

We show results in Tab. 1 for a variety of downstream RT planning metrics, including max dose to the heart (Heart $D_{0}$), heart volume, volume of right lung receiving 20Gy of dose (Right Lung $V_{20}$), dose to 35% relative volume of the right lung (Right Lung $D_{35}$), right lung volume, left lung volume, max dose to left lung (Left Lung $D_{0}$), and volume of the body.
These metrics have important implications for patient safety.
For example, in our setup, if a heart $D_{0}$ is $<5$Gy or right lung $V_{20}$ is $<35\%$, the plan is unsafe for the patient.
We look at the mean difference between asymmetrically and symmetrically adjusted interval lengths $\mu(L_{asym}(\hat{Y}_{n+1}^{b})-L_{sym}(\hat{Y}_{n+1}^{b}))$, effective bias $b_{eff}$ computed using Alg. 1, the probability that asymmetrically adjusted interval lengths are greater than that of symmetrically adjusted $P(L_{asym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{b}))$, and whether Cor. 3.1 is true ✓or false ✗(Tab. 1).

**Table 1: Real life application of sparse-view computed tomography for downstream radiotherapy planning reveals prevalent biases in predictions and validate bias conditions in Cor. 3.1. We use a variety of downstream RT planning metrics, including max dose to the heart (Heart $D_{0}$), heart volume, volume of right lung receiving 20Gy of dose (Right Lung $V_{20}$), dose to 35% relative volume of the right lung (Right Lung $D_{35}$), right lung volume, left lung volume, max dose to left lung (Left Lung $D_{0}$), and volume of the body. We show the mean difference in asymmetrically and symmetrically adjusted interval lengths $\mu(L_{asym}(\hat{Y}_{n+1}^{b})-L_{sym}(\hat{Y}_{n+1}^{b}))$, effective bias $b_{eff}$, the probability that asymmetrically adjusted interval lengths are greater than that for symmetric adjustments $P(L_{asym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{b}))$, and whether Cor. 3.1 is true (✓) or false (✗).**
| Metric | $b_{eff}$ | $\mu(L_{asym}(\hat{Y}_{n+1}^{b})$<br>$-L_{sym}(\hat{Y}_{n+1}^{b}))$ | $P(L_{asym}(\hat{Y}_{n+1}^{b})$<br>$\leq L_{sym}(\hat{Y}_{n+1}^{b}))$ | Cor. 3.1 |
| --- | --- | --- | --- | --- |
| Heart $D_{0}$ (Gy) | -0.31 | 334.41 | 0.05 | ✗ |
| Heart Volume ($cm^{3}$) | -19.74 | 365.40 | 0.05 | ✗ |
| Right Lung $V_{20}$ (%) | -9.47e-3 | 0.02 | 0.05 | ✗ |
| Right Lung $D_{35}$ (Gy) | -1.13 | 1.11 | 0.05 | ✗ |
| Right Lung Volume ($cm^{3}$) | 66.11 | -108.98 | 1.0 | ✓ |
| Left Lung Volume ($cm^{3}$) | 58.17 | -92.89 | 1.0 | ✓ |
| Left Lung $D_{0}$ (Gy) | -0.14 | -0.04 | 0.90 | ✓ |
| Body Volume ($cm^{3}$) | -287.48 | -564.53 | 1.0 | ✓ |

Figure: Figure 4: Real-life application of weather forecasting shows even if predictions drift “far away” from the ground truth values, asymmetrically adjusted intervals are still able to maintain the same tightness and validity of intervals as if the drift had never happened. We plot A) Temperature over time for biased predictions $\hat{Y}^{b}$ and ground truth $Y$, B) Coverage over time for weighted conformal prediction with asymmetric adjustments (blue) and symmetric adjustments (red), and naive (unweighted) conformal prediction (green). C) Intervals with symmetric adjustments (red), asymmetric adjustments (blue), and where they overlap (purple), and D) Bias versus lengths for symmetric adjustments (red), lengths for asymmetric adjustments (blue), and upper bound length for symmetric adjustments from Thm. 2 (purple).
Refer to caption: extracted/5896715/figs/fig3_weather.png

Results in Tab. 1 reveal that for many downstream tasks like segmentation (Fig. 3), predictions could be highly biased (column 2). Moreover, results in Cor. 3.1 can be reliably used to determine whether asymmetrically adjusted interval lengths are shorter than those of symmetric adjustments (columns 4 and 5).
When $P(L_{asym}(\hat{Y}_{n+1}^{b})\leq L_{sym}(\hat{Y}_{n+1}^{b}))$ tends to 1, the condition in Cor. 3.1 is met, and vice versa.
Our experimental results show that our theoretical analyses are robust to scenarios even with extremely low $n$.

#### 4.2.2 Times series

Weather forecasting is important for many aspects of daily life, from public safety to agriculture to disaster preparedness and response. We use the Yandex Weather Prediction dataset and the average pre-trained CatBoost model from  [] to predict temperature changes. The temporal dependencies between points violate the exchangeability assumption. Therefore, we use weighted conformal prediction where we use a different adjustment
for each new data point [].
We use the $L_{1}$ non-conformity score and weight the data points in the window of size $K=1000$ equally.
This setup effectively reduces to split CP applied each at time window and the theoretical analyses in Sec. 3 apply.
The symmetric non-conformity score for time $t$ is given by $s^{b}_{t}=|\hat{Y}^{b}_{t}-Y^{0}_{t}|$.
The asymmetric non-conformity scores for time $t$ are given by
$s^{b}_{t,lo}=\hat{Y}^{b}_{t}-Y^{0}_{t}$ and $s^{b}_{t,hi}=Y^{0}_{t}-\hat{Y}^{b}_{t}$.
We set $\alpha=0.1$ and inject an increasing negative bias to the unbiased predicted values $\hat{Y}^{b}_{t}=\hat{Y}^{0}_{t}-(2\times 10^{-4})t$.
We plot temperature over time for predictions $\hat{Y}^{b}$ and ground truth $Y$ (Fig. 4A),
coverage over time for weighted (symmetric and asymmetric adjustments) and naive (unweighted, symmetric adjustments) CP (Fig. 4B),
symmetrically (red) and asymmetrically (blue) adjusted intervals and where they overlap (purple) (Fig. 4C), and bias versus lengths for symmetric adjustments (red), lengths for asymmetric adjustments (blue), and upper bound lengths for symmetric adjustments from Thm. 2 (purple).

We observe that symmetric and asymmetric adjustments produce valid coverage while naive approaches do not, confirming prior work [, ] (Fig. 4B).
We observe that asymmetric adjustments are independent of bias (Fig. 4D) yet still produce valid prediction intervals.
We observe that symmetrically adjusted interval lengths increase linearly with increasing bias, bounded by Thm. 2 (Fig. 4D).
Our results suggest that even if predictions drift “far away” from the ground truth values, asymmetrically adjusted intervals are still able to maintain the same tightness and validity of intervals as if the drift had never happened.

## 5 Discussion and Conclusion

We will never collect perfect data in practice, or build perfect predictive models that are robust over time. Therefore, it is integral to account for these imperfections when designing practical systems.
In this work, we argue that the effects of bias on CP prediction interval lengths can be mitigated by computing asymmetric adjustments as opposed to the conventional symmetric adjustments.
We prove the following for $L_{1}$ and CQR non-conformity scores.
In Thm. 2 we showed that the upper bound of the prediction interval lengths with symmetric adjustments increases by $2|b|$.
In Thm. 3, we showed that prediction interval lengths with asymmetric adjustments are not affected by bias.
In Cor. 3.1, we showed the conditions when prediction interval lengths with asymmetric adjustments are guaranteed to be smaller than those of symmetric intervals.
We proposed an algorithm to empirically determine the bias and showed empirical evidence using synthetic and real-life data. Our results have have important implications on accounting for algorithmic bias, while also suggesting further areas of investigation:

Stability of bias estimation for low $n$.
Our work suggests that estimating bias based on symmetrically adjusted intervals is a straightforward, practical, and computationally efficient way to account for systematic errors in predictions.
However, it is crucial to consider the impact of sample size on the reliability of these estimates when the calibration set is small [].
Small calibration sets can lead to noisy estimates of bias [].
The challenges associated with limited data include under or overestimating the true bias, greater bias estimation variability, and being more susceptible to skewed, outliers, and random fluctuations.
We recommend considering the uncertainty in bias estimates when applying corrections and increasing sample size to improve the reliability of bias estimates where possible.

More complex scores.
Our analysis reveals that while simple non-conformity scores, as presented in Eq. 2 and 4, are tractable for theoretical guarantees, more complex non-conformity scores such as locally adaptive scores [, , ] and CQR variants [] present challenges.
These challenges arise due to modeling bias as a globally applied additive constant.
This simplification, although useful for theoretical and empirical analysis, may overlook that biases could exhibit more intricate patterns, possibly varying across the input space or depending on specific features.
For example, in time series data, we use a uniform weighting scheme that effectively reduces to split CP over each time window.
Our results suggest that incorporating these techniques in more complex settings could reveal interesting behaviors and could help design more bias-robust scores.

Covariate shift and Bias.
Our work suggests correcting for bias using a globally applied constant to the predictions can significantly reduce the interval lengths.
However, our approach does not account for situations where the predicted distribution changes between calibration and test datasets.
Prior work on CP under covariate shift [] weighted predictions by a probability proportional to their likelihood ratio.
However, when calibration predictions are “far away” from the expected test predictions, the likelihood ratio may be very small or zero.
Thus, it is impossible to perform a covariate shift without significant overlap between the calibration and expected test predictions when the expected bias is large.
Our work suggests exploring both bias correction and covariate shift together could lead to tighter and more reliable prediction intervals for these situations.

#### Acknowledgements

M.C. would like to acknowledge support from a fellowship from the Gulf Coast Consortia on the NLM Training Program in Biomedical Informatics and Data Science T15LM007093. T.N. would like to acknowledge the support of the NIH LRP award. The authors would like to thank the RPA team (Joy Zhang and Raphael Douglas) for their support. The authors would like to thank Lorenzo Luzi for their feedback.

## Appendix A Proofs

### A.1 Proof of Theorem 2

Using Eq. 2, we show the behavior of prediction interval lengths with symmetric adjustments when predictions are biased $\hat{Y}^{b}_{i}=\hat{Y}^{0}_{i}+b$ where $b\in\mathbb{R}$.
We show the behavior of symmetric adjustments under 1) no bias, 2) large negative bias, 3) large positive bias, 4) small negative bias and 5) small positive bias, and compare the resulting interval lengths.
We leverage a property of quantiles $Q_{\alpha}(\hat{Y}+b)=Q_{\alpha}(\hat{Y})+b$ where $b$ is a scalar value.

When $b=0$, we can write the adjustment (Eq. 9), the prediction interval (Eq. 10), and the prediction interval length (Eq. 11).

$$ $s^{0}_{i}=\max(f_{lo}(\hat{Y}_{i}^{0})-Y_{i},Y_{i}-f_{hi}(\hat{Y}^{0}_{i}))$ (8) $$

$$ $q^{0}=Q_{1-\hat{\alpha}}(\{s^{0}_{i}\}_{i=1}^{n})$ (9) $$

$$ $C_{sym}(\hat{Y}^{0}_{n+1})=[f_{lo}(\hat{Y}^{0}_{n+1})-q^{0},f_{hi}(\hat{Y}^{0} _{n+1})+q^{0}]$ (10) $$

$$ $L_{sym}(\hat{Y}^{0}_{n+1})=f_{hi}(\hat{Y}^{0}_{n+1})-f_{lo}(\hat{Y}^{0}_{n+1}) +2q^{0}$ (11) $$

For biased predictions, the 0 0 is replaced with a $b$.

Next, we examine when predictions are highly biased in the negative direction $\hat{Y}_{i}^{b^{--}}=\hat{Y}_{i}^{0}+b^{--}$ where $b^{--}<Y_{i}-f_{hi}(\hat{Y}^{b^{--}}_{i})<0$, so $Y_{i}>f_{hi}(\hat{Y}_{i}^{b^{--}})\geq f_{lo}(\hat{Y}_{i}^{b^{--}})$.
The non-conformity score reduces to the $Y_{i}-f_{hi}(\hat{Y}^{b^{--}}_{i})$ because $f_{lo}(\hat{Y}^{b^{--}}_{i})-Y_{i}<0$ and can be written as:

$$ $s_{i}^{b^{--}}=Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{--}})=Y_{i}-f_{hi}(\hat{Y}_{i}^{0} )-b^{--}\leq s_{i}^{0}-b^{--}$ (12) $$

The adjustment can be written as:

$$ $q^{b^{--}}=Q_{1-\hat{\alpha}}(\{s_{i}^{b^{--}}\}_{i=1}^{n})\leq Q_{1-\hat{ \alpha}}(\{s_{i}^{0}\}_{i=1}^{n})-b^{--}=q^{0}-b^{--}$ (13) $$

The length can be written as:

$$ $\displaystyle L(\hat{Y}_{n+1}^{b^{--}})$ $\displaystyle=f_{hi}(\hat{Y}_{n+1}^{b^{--}})-f_{lo}(\hat{Y}_{n+1}^{b^{--}})+2q ^{b^{--}}$ (14) $\displaystyle\leq f_{hi}(\hat{Y}^{0}_{n+1})-f_{lo}(\hat{Y}^{0}_{n+1})+2q^{0}-2 b^{--}$ (15) $\displaystyle=L(\hat{Y}^{0}_{n+1})-2b^{--}$ (16) $$

The inequalities in Eq.  12, 13 and  15 hold true because $\max(f_{lo}(\hat{Y}^{b^{--}}_{i})-Y_{i},Y_{i}-f_{hi}(\hat{Y}^{b^{--}}_{i}))
\geq Y_{i}-f_{hi}(\hat{Y}^{b^{--}}_{i})$.

Next, we examine when predictions are highly biased in the positive direction $\hat{Y}_{i}^{b^{++}}=\hat{Y}^{0}_{i}+b^{++}$ where $b^{++}>Y_{i}-f_{lo}(\hat{Y}_{i}^{b^{++}})>0$, so $Y_{i}<f_{lo}(\hat{Y}_{i}^{b^{++}})\leq f_{hi}(\hat{Y}_{i}^{b^{++}})$.
The non-conformity score reduces to the $f_{lo}(\hat{Y}_{i}^{b^{++}})-Y_{i}$ because $Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{++}})<0$ and can be written as:

$$ $s_{i}^{b^{++}}=f_{lo}(\hat{Y}_{i}^{b^{++}})-Y_{i}=f_{lo}(\hat{Y}^{0}_{i})-Y_{i }+b^{++}\leq s^{0}_{i}+b^{++}$ (17) $$

$$ $q^{b^{++}}=Q_{1-\hat{\alpha}}(\{s_{i}^{b^{++}}\}_{i=1}^{n})\leq Q_{1-\hat{ \alpha}}(\{s^{0}_{i}\}_{i=1}^{n})+b^{++}=q^{0}+b^{++}$ (18) $$

$$ $\displaystyle L(\hat{Y}_{n+1}^{b^{++}})$ $\displaystyle=f_{hi}(\hat{Y}_{n+1}^{b^{++}})-f_{lo}(\hat{Y}_{n+1}^{b^{++}})+2q ^{b^{++}}$ (19) $\displaystyle\leq f_{hi}(\hat{Y}^{0}_{n+1})-f_{lo}(\hat{Y}^{0}_{n+1})+2q^{0}+2 b^{++}$ (20) $\displaystyle=L(\hat{Y}^{0}_{n+1})+2b^{++}$ (21) $$

The inequalities in Eq. 17, 18 and 20 hold true because $\max(f_{lo}(\hat{Y}_{i}^{b^{++}})-Y_{i},Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{++}}))
\geq f_{lo}(\hat{Y}_{i}^{b^{++}})-Y_{i}$.

For the $L_{1}$ non-conformity score $f_{lo}(\hat{Y}_{i}^{b})=f_{hi}(\hat{Y}_{i}^{b})=\hat{Y}_{i}^{b}$, we can combine Eq. 16 and 21 to yield the desired result: $L(\hat{Y}_{n+1}^{b})\leq L(\hat{Y}^{0}_{n+1})+2|b|$

For the CQR non-conformity score $f_{lo}(\hat{Y}_{i}^{b})=Q_{\alpha_{lo}}(\hat{Y}_{i}^{b})$ and $f_{hi}(\hat{Y}_{i}^{b})=Q_{\alpha_{hi}}(\hat{Y}_{i}^{b})$, we need to analyze when prediction have small negative bias and small positive bias - in other words, when the ground truth is between the lower and upper adjustment points.

When predictions have small negative bias $\hat{Y}_{i}^{b^{-}}=\hat{Y}^{0}_{i}+b^{-}$ where $0>b^{-}>Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{-}})$ and $f_{lo}(\hat{Y}_{i}^{b^{-}})<Y_{i}<f_{hi}(\hat{Y}_{i}^{b^{-}})$.
The ground truth is closer to the upper bound than lower bound $f_{hi}(\hat{Y}_{i}^{b^{-}})-Y_{i}<Y_{i}-f_{lo}(\hat{Y}_{i}^{b^{-}})$. Taking negative on both sides gives $Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{-}})>f_{lo}(\hat{Y}_{i}^{b^{-}})-Y_{i}$. This is the same as large negative bias. The interval length reduces to Eq. 16.

When predictions have small positive bias $\hat{Y}_{i}^{b^{+}}=\hat{Y}^{0}_{i}+b^{+}$ where $0<b^{+}<Y_{i}-f_{lo}(\hat{Y}_{i}^{b^{+}})$ and $f_{lo}(\hat{Y}_{i}^{b^{+}})<Y_{i}<f_{hi}(\hat{Y}_{i}^{b^{+}})$. The ground truth is closer to the lower bound than upper bound $f_{hi}(\hat{Y}_{i}^{b^{+}})-Y_{i}>Y_{i}-f_{lo}(\hat{Y}_{i}^{b^{+}})$. Taking negative on both sides gives $Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{+}})<f_{lo}(\hat{Y}_{i}^{b^{+}})-Y_{i}$. This is the same as large positive bias. Thus, the interval length reduces to Eq. 21.

Combining inequalities gives the desired result: $L(\hat{Y}_{n+1}^{b})\leq L(\hat{Y}^{0}_{n+1})+2|b|$

### A.2 Proof of Theorem 3

We analyze the behavior of asymmetric adjustments under bias. We model the biased predictions as $\hat{Y}_{i}^{b}=\hat{Y}_{i}^{0}+b$ where $b$ is a global constant (can be both positive and negative) added to unbiased predictions $\hat{Y}^{0}_{i}$ .

First, the lower and upper scores can be written as:

$$ $s^{b}_{i,lo}=f_{lo}(\hat{Y}_{i}^{b})-Y_{i}=f_{lo}(\hat{Y}_{i}^{0})-Y_{i}+b=s^{ 0}_{i,lo}+b$ (22) $$

$$ $s^{b}_{i,hi}=Y_{i}-f_{hi}(\hat{Y}_{i}^{b})=Y_{i}-f_{hi}(\hat{Y}_{i}^{0})-b=s^{ 0}_{i,hi}-b$ (23) $$

Next, the lower and upper adjustments can be written as:

$$ $q^{b}_{lo}=Q_{1-\hat{\alpha}_{lo}}(\{s^{b}_{i,lo}\}_{i=1}^{n})=Q_{1-\hat{ \alpha}_{lo}}(\{s^{0}_{i,lo}\}_{i=1}^{n})+b=q^{0}_{lo}+b$ (24) $$

$$ $q^{b}_{hi}=Q_{1-\hat{\alpha}_{hi}}(\{s^{b}_{i,hi}\}_{i=1}^{n})=Q_{1-\hat{ \alpha}_{hi}}(\{s_{i,hi}\}_{i=1}^{n})-b=q^{0}_{hi}-b$ (25) $$

Finally, the lengths when predictions are biased can be simplified:

$$ $\displaystyle L_{asym}(\hat{Y}_{n+1}^{b})$ $\displaystyle=f_{hi}(\hat{Y}_{n+1}^{b})-f_{lo}(\hat{Y}_{n+1}^{b})+q^{b}_{hi}+q ^{b}_{lo}$ (26) $\displaystyle=f_{hi}(\hat{Y}^{0}_{n+1})+b-f_{lo}(\hat{Y}^{0}_{n+1})-b+q^{0}_{ hi}+b+q^{0}_{lo}-b$ (27) $\displaystyle=f_{hi}(\hat{Y}^{0}_{n+1})-f_{lo}(\hat{Y}^{0}_{n+1})+q^{0}_{hi}+q ^{0}_{lo}$ (28) $\displaystyle=L_{asym}(\hat{Y}^{0}_{n+1})$ (29) $$

The results in Eq. 26 indicate that asymmetric adjustments are not affected by bias $b$.

#### A.2.1 Proof of Alg. 1 Convergence

We seek to minimize the objective function:

$$ $\displaystyle f(b_{eff})$ $\displaystyle=\max(\{L_{sym}(\hat{Y}_{i}^{b}-b_{eff})\}_{i=1}^{n})$ (30) $\displaystyle=\max(\{L_{sym}(\hat{Y}^{0}_{i}+b-b_{eff})\}_{i=1}^{n})$ (31) $$

First, we can derive the symmetrically adjusted interval lengths with bias $b\in\mathbb{R}$ based on the techniques and results from App. A.1.
Initially, the predictions are assumed to be biased.
The non-conformity scores in canonical form (Eq. 2) when predictions are positively and negatively biased with $b^{+}>0$ and $b^{-}<0$ are given by:

$$ $\displaystyle s_{i}^{b^{+}}$ $\displaystyle=f_{lo}(\hat{Y}_{i}^{b^{+}})-Y_{i}=f_{lo}(\hat{Y}^{0}_{i})-Y_{i}+ b^{+}=s_{i}^{0}+b^{+}$ (32) $\displaystyle s_{i}^{b^{-}}$ $\displaystyle=Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{-}})=Y_{i}-f_{hi}(\hat{Y}^{0}_{i})- b^{-}=s^{0}_{i}-b^{-}$ (33) $$

where $s^{0}_{i}$ is the non-conformity score without bias.
The inequality is replaced with an equality because predictions are assumed to be biased during the optimization process.
Specifically, $\max(f_{lo}(\hat{Y}_{i}^{b^{+}})-Y_{i},Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{+}}))=f_{
lo}(\hat{Y}_{i}^{b^{+}})-Y_{i}$ under positive bias and $\max(f_{lo}(\hat{Y}_{i}^{b^{-}})-Y_{i},Y_{i}-f_{hi}(\hat{Y}_{i}^{b^{-}}))=Y_{i
}-f_{hi}(\hat{Y}_{i}^{b^{-}})$ under negative bias. Next, the adjustments can be written as:

$$ $\displaystyle q^{b^{+}}$ $\displaystyle=Q_{1-\hat{\alpha}}(\{s_{i}^{b^{+}}\}_{i=1}^{n})=Q_{1-\hat{\alpha }}(\{s^{0}_{i}\}_{i=1}^{n})+b^{+}=q^{0}+b^{+}$ (34) $\displaystyle q^{b^{-}}$ $\displaystyle=Q_{1-\hat{\alpha}}(\{s_{i}^{b^{-}}\}_{i=1}^{n})=Q_{1-\hat{\alpha }}(\{s_{i}^{0}\}_{i=1}^{n})-b^{-}=q^{0}-b^{-}$ (35) $$

where $q^{0}\in\mathbb{R}$ is the symmetric adjustment for predictions with no bias.
Combining the two equations gives a more general form of adjustments for positive and negative bias:

$$ $q^{b}=q^{0}+|b|$ (36) $$

Thus, symmetrically adjusted interval lengths with bias can be written as:

$$ $\displaystyle L_{sym}(\hat{Y}_{n+1}^{b})$ $\displaystyle=f_{hi}(\hat{Y}_{n+1}^{b})-f_{lo}(\hat{Y}_{n+1}^{b})+2q^{b}$ (37) $\displaystyle=f_{hi}(\hat{Y}_{n+1}^{b})-f_{lo}(\hat{Y}_{n+1}^{b})+2q^{0}+2|b|$ (38) $$

Using Eq. 38, we can recast the objective function as follows:

$$ $\displaystyle f(b_{eff})$ $\displaystyle=\max(\{L_{sym}(\hat{Y}_{i}+b_{eff})\}_{i=1}^{n})$ (39) $\displaystyle=\max(\{L_{sym}(\hat{Y}_{i}^{0}+b-b_{eff})\}_{i=1}^{n})$ (40) $\displaystyle=\max(\{f_{hi}(\hat{Y}_{i}^{0})-f_{lo}(\hat{Y}^{0}_{i})+2q^{0}\}_ {i=1}^{n})+2|b-b_{eff}|$ (41) $$

In Eq. 41, the terms inside the $\max$ function and $b$ are data-dependent constants and are not dependent on $b_{eff}$.
Thus, minimizing Eq. 39 results in minimizing a translated (horizontally and vertically) absolute value function (Eq. 41).
This problem is convex, not differentiable at $b_{eff}=b$, and has a global minimum at $b$.
When $b_{eff}\neq b$, the sub-gradient of $f(b_{eff})$ is $-2$ for $b_{eff}<b$ and $2$ for $b_{eff}>b$.
A standard convergence proof follows.

Let $b_{eff,k}$ be the $k$-th iteration of gradient descent.
The update rule is: $b_{eff,k+1}=b_{eff,k}-\gamma\nabla f(b_{eff,k})$.
For any step size $\gamma>0$, we have:

- 1.
If $b_{eff,k}>b$, $b_{eff,k+1}=b_{eff,k}-2\gamma$, moving towards $b$
- 2.
If $b_{eff,k}<b$, $b_{eff,k+1}=b_{eff,k}+2\gamma$, moving towards $b$

Thus, the distance to the optimum decreases in each iteration:

$$ $|b_{eff,k+1}-b|\leq|b_{eff,k}-b|-2\gamma$ (42) $$

After $k$ iterations, the distance to the optimum is at most:

$$ $|b_{eff,k}-b|\leq|b_{eff,0}-b|-2\gamma k$ (43) $$

Setting this to $\epsilon$ and solving for $k$ yields

$$ $k\geq\frac{|x_{0}-b|-\epsilon}{2\gamma}$ (44) $$

Thus gradient descent converges to the global minimum $b_{eff}=b$ with rate $O(1/k)$

## Appendix B Data Distributions

We show the data distributions for experiments in Sec. 4 in Fig. 5 and 6.

Figure: Figure 5: Data distribution for synthetic experiments with skewed and noisy predictions. We use $N(10,5)$ to simulate the ground truth (blue) distribution and added $N(0,2)$, $W(1,0,5)$, and $-W(1,-2,5)$ to the ground truth to simulate a un-, left- and right- skewed predictions (red). The parameter descriptions can be found in the [scipy.stats](scipy.stats) documentation. The dotted histograms indicate the “unbiased” predictions $\hat{Y}^{b}-b_{eff}$ where $b_{eff}$ is the empirical effective bias estimated through a simple optimization procedure.
Refer to caption: extracted/5896715/figs/app_synthdata.png

Figure: Figure 6: Data distribution for sparse view computed tomography (sparse CT) reconstruction applied to downstream radiotherapy planning. Including max dose to the heart (Heart $D_{0}$), heart volume, volume of right lung receiving 20Gy of dose (right lung $V_{20}$), dose to 35% relative volume of the right lung (right lung $D_{35}$), right lung volume, left lung volume, max dose to left lung (Left Lung $D_{0}$), and volume of the body. The predictions $\hat{Y}^{b}$ and ground truth $Y$ are shown in red and blue. The dotted histograms indicate the “unbiased” predictions $\hat{Y}^{b}-b_{eff}$ where $b_{eff}$ is the empirical effective bias estimated through a simple optimization procedure in Alg. 1.
Refer to caption: extracted/5896715/figs/app_rpa.png

## Appendix C Sparse CT for Radiotherapy Planning Details

We use a de-identified CT dataset of 20 patients retrospectively treated with radiotherapy at
[redacted].
This research was conducted using an approved institutional review board protocol.
For each patient, we generate 10 digitally reconstructed radiographs (DRR) from the ground truth CT scan using the TIGRE toolbox [].
The DRRs simulate image acquisition from a cone-beam geometry.
We simulate physical randomness (beam angle variability and sensor noise) by generating DRRs with 3% noise and 50 random projections between 0 and 360 degrees.
We use a self-supervised model, Neural Attenuation Fields (NAF), for reconstruction [].
We use the Radiation Planning Assistant (RPA, FDA 510(k) cleared), a web-based tool for radiotherapy planning. [, , ].
RPA automates treatment planning on CT images and provides dose and plan reports for clinics in low-and-middle-income countries [, , ].
The number of projections was increased from 2 to 50 until organ boundaries in the reconstructed volumes were perceptually discernible in the reconstruction by the RPA.
We use the default parameter setting in NAF [] and introduce computational randomness through random initializations of NAF [, ].

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: My main questions are in the weakness section. Below is a question regarding the proof: In the proof of Theorem 2 (line 712), I am confused since it is first stated that $Y_i - f_{\text{hi}}(\hat Y_i^{b^{--}}) <0$ and then 
$Y_i > f_{\text{hi}}(\hat Y_i^{b^{--}})$.
soundness: 2
strengths: The paper studies an important question and makes an interesting 
observation. Some theoretical attempts have also been made to understand the 
observation.
summary: This paper studies the efficiency of conformal prediction when the predicted value has a systematic bias, aiming to understand the effect of symmetric/asymmetric quantile adjustment on the corresponding 
conformal prediction length. Under a stylized model, theories are 
developed to understand the behavior of symmetric/asymmetric quantile adjustment.
The theory is then evaluated on synthetic and real data.
weaknesses: The paper's main theories (Theorem 2 and 3) are under highly stylized models and the statement lacks rigor (please correct me 
if I misunderstood anything). For example, Theorem 2 is stated for prediction intervals in the generic form of Eq. (2), but it appears that the proof only focuses on the case of residual errors and CQR. For another example, the proof of Theorem 2 seems to use the fact that 
$f_{\text{hi}}(\hat Y_i^{b--}) = f_{\text{hi}}(\hat Y_i^{0})+b^{--}$, which is not stated as an assumption. The theorem also appears to assume that the bias is sufficiently large in magnitude.


I am also not following the form of CQR used in this paper; in my understanding, 
CQR fits conditional quantiles of $Y$ given $X$ by minimizing the pin-ball loss, 
and the resulting fitted conditional quantile  function operates $X_i$ instead of the point prediction of $\hat Y_i$ 
(unless the quantile is obtained in a specific form centered around a point prediction).
I am also confused by the description of CQR in lines 194-196 --- what is $n_s$, what is the set of samples $\hat Y^b_i$ 
and how are they generated. If the prediction is deterministic, does this reduce to the residual error case? Please let me know if I misunderstood anything.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Is Romano et al. 2029 the only work where non-symmetric intervals are used and tested?
- What happens if an adaptive scheme is used to compute the interval? E.g. if the symmetric constant intervals are replaced by reweighted intervals as in Papadopoulous 2008? In particular, how would Theorem 2 change? 
- Would it be possible to compute an upper bound of the bias using the gap between symmetric and asymmetric intervals?
- Can the proposed approach help study the non-exchangeable situation where the bias is only present at test time? In other words, can one obtain an approximated version of Lemma 1 and Theorem 2?
- Have you run any experiments on Bias estimation?
soundness: 3
strengths: - As CP applies to any given model, the results may help establish a good trade-off between model flexibility and efficiency in applications where data are scarce.
- The algorithm for estimating the bias from the obtained interval is interesting.
summary: The proposed method aims to improve uncertainty quantification in the presence of model bias. The authors show that non-symmetric prediction intervals may be more robust to model bias than their symmetric counterparts. The claims are supported by theoretical and empirical results.
weaknesses: - Non-symmetric CP is not new. The paper's contribution would be better introduced by adding an intuitive explanation of why *asymmetric adjustments have been theorized to yield longer interval lengths as a consequence of stronger guarantees (Romano et al.,2019)* but *has also been empirically observed that asymmetric adjustments yield tighter intervals than symmetric ones.*
- The authors focus on the setup where comparable bias is present in the calibration and test samples. They should comment on whether it would be more efficient to 1) improve the underlying point-prediction model or 2) consider more flexible conformity scores (e.g. adding and training a free shifting term to the conformity scores).
- Asymmetric intervals can be defined by splitting the calibration samples according to the residual signs and then evaluating two sample quantiles. It would be helpful to see a theoretical comparison of the gap between the proposed method and such a naive approach as a function of the size of the calibration set.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: - How do the paper results get used in practice? Since any learning algorithm perfectly mitigates the bias that you present. So, in your case, the bias is not present in the training set but is in the calibration set (so that it can account for this). Can you provide a use case where this occurs?
- You need to correct the table reference; you see Tab. 4.2.1, while in the caption, it is Table 1. The same is true for the references to the figures, where you use Fig. X, and the caption states Figure X.  Change either the captions or the references.
soundness: 2
strengths: - **Clear and Comprehensive:** The CP mechanisms, the derivation of the bias effects, and the comparison between symmetric and asymmetric intervals are clearly explained.
- **Formalizing Good Message:** Recently, in the CP field/community, asymmetric adjustments or conformal prediction intervals with symmetric (the same) density are used beneath the lower bound and above the upper bound, which is appropriate for the end-user from an interpretation standpoint. Additionally, there is occasional observation of performance gains using asymmetric adjustment, as the paper rightfully mentions. Formalizing and theorizing these observations is undoubtedly beneficial for the field and practitioners.
summary: The paper studies how a systematic homoscedastic bias is present in the point prediction. It studies how it influences the efficiency of prediction intervals produced by inductive conformal prediction (ICP) and conformalized quantile regression (CQR), explicitly comparing the effect of symmetric and asymmetric adjustments. For ICP, the impact of using the absolute residual (symmetric adjustment) and the residual (asymmetric adjustment, controlling the coverage on the left and right) is compared. For CQR, similarly. 

They show that in the case of simple homoscedastic bias, $b$: 1) the upper bound for the size of the intervals of the biased predictions, which of the size of the intervals without prejudice and $2|b|$; 2) the asymmetric adjusted prediction interval is of the same size as there was no bias, and 3) they show under which condition the asymmetric or symmetric adjustment will result in smaller prediction intervals (however, this requires knowledge of the bias).
weaknesses: - **Elementary Theoretical Findings**: While the theoretical contributions are sound, the results may be considered elementary. The core findings, such as the impact of bias on symmetric intervals and the lack of bias influence on asymmetric intervals, are straightforward and follow from the basic principles of CP. Though helpful, these insights do not introduce particularly advanced or novel theoretical complexity, which may limit the paper's appeal to a more specialized audience seeking more profound theoretical innovations.
- **Simplified Bias Assumptions**: The assumption of bias as a constant noise term following the same global distribution across all predictions may not adequately capture more complex forms of bias present in real-world data, such as feature-specific or covariate-dependent biases. The paper also mentions on lines 089-094 different reasons for these biases. However, they result in way more complex biases.
- **Limited Discussion on More Complex Non-Conformity Scores**: While the paper acknowledges that more complex non-conformity scores may require different approaches, it does not explore these in-depth, potentially limiting the generality of the findings for more advanced CP methods.
- **Limited Synthetic Experiment Settings:** The paper only evaluates experiments where $n$ is large, and the data is generated from symmetrical distributions. However, this is problematic because the strong point of the symmetric adjustment is that it can better leverage asymmetries (skewness) of the aleatoric uncertainty distribution (noise distribution on the label).
- **Wrong Claims:** Your statement on line 266 is incorrect. When the number of samples is large and no bias is present, the length of prediction intervals generated by a symmetric adjustment is approximately equal to the ones generated from asymmetric adjustments. See the above point.
- **Section 3.1 is too bloated:** Given your bias, you could just take the mean of the calibration set's error, to retrieve it.

---

confidence: 3
contribution: 1
initial_rating: 1
presentation: 2
questions: Please have look at the weakness section.
soundness: 2
strengths: Practical Consideration of CP Efficiency: The authors address the critical issue of efficiency in conformal prediction (CP) sets, specifically focusing on the size of prediction intervals, which is relevant in real-world applications. By exploring the impact of bias on CP intervals, the authors attempt to improve the accuracy and reliability of predictions, which is essential for high-stakes tasks.

Experiments: The paper demonstrates multiple real-world experiments, such as sparse-view CT reconstruction and weather forecasting.
summary: This paper investigates how bias, defined as the systematic deviation from ground truth, affects Conformal Prediction (CP) intervals in regression tasks. CP is used for uncertainty quantification in machine learning models. The authors focus on two adjustment methods for CP intervals: symmetric (where both sides of the interval are adjusted equally) and asymmetric (where adjustments can be unequal). Through theoretical and empirical analyses, they show that:
1. Symmetrically adjusted intervals increase in length by 2|b|, where b is the bias.
2. Asymmetrically adjusted intervals are unaffected by bias.
3. Under certain conditions, asymmetric intervals are tighter than symmetric ones.
Their findings suggest that asymmetric intervals maintain their "tightness" even under biased predictions, unlike symmetric intervals that inflate in length. These conclusions are validated with real-world tasks in CT reconstruction and weather forecasting, highlighting the potential for more bias-robust machine learning systems.
weaknesses: Inconsistent Notations and presentation: The paper suffers from unclear and inconsistent notations. For instance, in equation (1), it's not explicit which bias is being addressed --- it is not clear how the randomness in the ground truth is being dealt with.  Additionally, \hat{Y_i^b} is ambiguously defined; it is not clear whether these quantities are scalars or they are allowed to be a set of values. For instance, in the first paragraph of Section 3 the Y_i is treated as scalars and in the second paragraph, they use Y_i = {Y_{ij}} as a vector. Besides, the score definition proposed by Romano et al.[1] is presented inaccurately, which raises concerns about the foundation of the theoretical analyses. The overall readability of the paper is not good. 

Limited Insight from Corollary: While Theorems 1 and 2 seems a bit straightforward and is a direct consequence of the linear assumptions. The utility of Corollary 3.1 is not clear, the RHS of eq. (6) is independent on b, and the LHS is dependent on b? It is not clear how the authors are setting L_{asymb} \leq L_{sym} in the proof. 

Missing Theoretical Justification for the Algorithm: The authors do not provide a theorem guaranteeing that the proposed algorithm retains the validity of the coverage guarantee for CP intervals. This omission is significant as it leaves a gap in understanding whether the method meets one of CP's fundamental requirements.

Lack of Coverage Probability in Simulations: The simulations fail to report the coverage probability, which is a critical metric for evaluating the reliability of proposed algorithm. This weakens the experimental validation of the proposed methods.

Comparison with related work: There are some related work which also aims to reduce the length of the conformal intervals [2,3]. It would be nice to compare the contribution of this work with these two related works. 

[1.] Romano, Yaniv, Evan Patterson, and Emmanuel Candes. "Conformalized quantile regression." Advances in neural information processing systems 32 (2019).

[2.] Xie, Ran, Rina Foygel Barber, and Emmanuel J. Candès. "Boosted Conformal Prediction Intervals." arXiv preprint arXiv:2406.07449 (2024).

[3.] Liang, Ruiting, Wanrong Zhu, and Rina Foygel Barber. "Conformal prediction after efficiency-oriented model selection." arXiv preprint arXiv:2408.07066 (2024).