Conformal Prediction as Bayesian Quadrature
Authors: Jake C. Snell
Thomas L. Griffiths
Venue: ICML 2025 oral (2025)
Keywords: bayesian quadrature
probabilistic numerics
conformal prediction
distribution-free uncertainty quantification
Primary area: probabilistic_methods->bayesian_models_and_methods
TL;DR: We propose an alternative to conformal prediction based on Bayesian quadrature that produces a distribution over test-time risk.

Abstract: As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive models will perform upon deployment. Distribution-free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from a Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We propose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers a richer representation of the likely range of losses to be observed at test time.

Conformal Prediction as Bayesian Quadrature
Jake C. Snell
Thomas L. Griffiths
Abstract
which aim to provide guarantees for model performance in distribution-free way. However, these techniques are based on ideas from frrqqueniist statistics, making it difficult to incorporate prior knowledge that might be available about specific models. For example, in particular setting we might have access some information about the distribution of the data that is likely to be encountered, and can construct tighter guarantees on the performance of models by making use of this information. Moreover, they focus on controlling the expected loss averaged over many unobserved datasets rather than focusing on the actual set observations.
As machine learning-based prediction systems are increasingly used in high-stakes situations, it is important to understand how such predictive mod- els will perform upon deployment. Distribution- free uncertainty quantification techniques such as conformal prediction provide guarantees about the loss black-box models will incur even when the details of the models are hidden. However, such methods are based on frequentist probability, which unduly limits their applicability. We revisit the central aspects of conformal prediction from Bayesian perspective and thereby illuminate the shortcomings of frequentist guarantees. We pro- pose a practical alternative based on Bayesian quadrature that provides interpretable guarantees and offers richer representation of the likely range of losses to be observed at test time.
In this paper, we show how methods for guaranteeing model performance be understood and extended by viewing them from Bayesian perspective. We develop framework in which we explicitly model uncertainty in the quantile values associated with particular observations, providing a nonparametric tool for characterizing possible distributions where the model might be deployed that is appropriately constrained by observed data. This framework allows us to draw upon methods from the fields of statistical prediction analysis (Aitchison & Dunsmore, 1975) and probabilistic numerics (Cockayne et al., 2019; Hennig et al., 2022) to develop guarantees that are interpretable and make adaptive use of available information.
1. Introduction
Machine learning systems based on deep learning are in- creasingly used in high-stakes settings, such as medical diagnosis or financial applications. These settings impose unique constraints on the performance of these systems: we want them to produce good outcomes in the aggregate, but also do fairly and with guarantee of a low probability of harm. However, predictive models based on deep learning be difficult interpret, and commercial models increas- ingly tend to offer little information about the techniques used in training. This creates a new challenge: How can we flexibly and reliably quantify the suitability of a model for deployment without making too many assumptions about how the model was trained or in which settings it will used?
We show that two popular uncertainty quantification meth- ods, split conformal prediction (Vovk al., 2005; Pa- padopoulos al., 2002) and conformal risk control (An- gelopoulos et al., 2024), can both be recovered as spe- cial cases of our framework. Our approach gives a more complete characterization of the performance of these ap- proaches, as we are able to determine the full distribution of possible outcomes rather than a single point estimate. Since our approach is grounded in Bayesian probability, we can easily incorporate knowledge relevant evaluating the performance of these models when it is present, such as monotonicity or distributional assumptions, while default- ing to existing methods when absent. Our results show that Bayesian probability, while is often discarded due to the apparent need to specify prior distributions, is actually well-suited for distribution- free uncertainty quantification.
Recent research on quantifying uncertainty has employed methods based on conformal prediction (Vovk et al., 2005),
'Department of Computer Science, Princeton University " DDepartment of Psychology, Princcton University. Correspondence to: Jake C. Snell <jsnenl@@rinnetoon.dd princcton.edu>..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Conformal Prediction as Bayesian Quadrature
2. Background
offered by Conformal Risk Control is of the form
Conformal prediction methods apply a wrapper on top of black-box predictive models to be able subject them to sta- tistical analysis. In order generate meaningful predictions about future performance, it is assumed that have access to small calibration dataset that is representative of the deployment conditions. Performance on this dataset then provides the foundation for generating predictions about future performance. We begin by reviewing existing current distribution- free uncertainty quantification techniques and Bayesian quadrature methods.
E (((C;(Xn+1)) (n+1)) S
(3)
where
2 = inf {x: n
Bn( n n
Sa
(4)
n n+1
and (A) = Ei= (,(A) is the empirical risk.
2.2. Bayesian Quadrature
2.1. Distribution-free Uncertainty Quantification Techniques
Bayesian quadrature (Diaconis, 1988; O'Hagan, 1991) is general technique for evaluating integrals that allows for uncertainty in the integrand. It estimates the value of an integral f fxx dr by the following four steps: (1) place prior p(f) on functions, (2) evaluate at æ1, #2...... En. (3) compute posterior given the observed values of by Bayes' rule, and (4) estimate 1 f(x) dr. Suppose that f(z,) yı for i 1,2,.... ........................... n. The posterior over f is
Uncertainty quantification techniques provide guarantees on the future performance of a black-box predictive model mapping inputs X to outputs Y based on a calibration set consisting of X1, Xi....; Xn, and , Yn- Different ap- proaches do so in different ways. For more information
on these techniques, refer to Shafer & Vovk (2008) or An- gelopoulos & Bates (2023).
n
p(f Fi:as Blmn x p(f) II [[o(v f(,)),
(5)
Split Conformal Prediction The goal of Split Conformal Prediction (Vovk et al., 2005; Papadopoulos et al., 2002) is to generate prediction set or interval that contains the ground-truth output with high probability. This is often expressed in terms of the coverage level - a. It relies on a score function sxx, y) which measures the disagreement between a predictor's 's output and the ground truth.
where d(-) is the Dirac delta function. The posterior mean then provides estimate for the integral:
1. [(x) dx 1. [n(()) dx, where
(6)
fn(t)= E((((t) Ti F1mssBin).
(7)
The conformal guarantee is
Pr (Yn+1 & C(Xn+1)) a.
(1) It has been demonstrated that many classical quadrature procedures such as the trapezoid rule can be recovered by placing a Gaussian process prior on functions (Karvonen Särkkä, 2017).
where
C(Xn+1) ty s(xn+1,9) S 9}
(2)
[[=+))(100]] quantile of 81
2.3. Summary and Prospectus
and 9 is the s(X,, ,,,, o(Nnn Here, C(X,+1) is
Bayesian quadrature provides an illustration of how a pri- marily numerical method be connected to Bayesian inference, and in doing potentially admit additional in- formation about the underlying function that can be incor- porated via prior distribution. In next section, we will see how a similar approach can be applied to conformal prediction, identifying Bayesian framework that repro- duces existing distribution-fii uncertainty quantification techniques. The challenge doing so is that we want guar- antees of the style obtained from Bayesian models, but want to make the approach as general as possible in its assumptions about the underlying distribution. We solve this problem via an approach inspired by probabilistic nu- meries to construct a nonparametric characterization the underlying distribution based on the calibration set.
prediction set or interval which aims include the ground-truth output.
Conformal Risk Control In Conformal Risk Control (An- gelopoulos et al., 2024), the goal is to generalize conformal prediction to more general loss functions that are mono- tonic functions of single parameter A. Conformal Risk Control (CRC) proceeds by viewing the coverage guaran- tee (1) the expected value of a 0-1 loss. It is assumed that the maximum possible value of the loss is B and that the problem is "achievable" by design in that there exists some setting Amax that satisfies the conformal guarantee. Additionally, each loss function (A) is assumed to be a monotonic non-increasing function of X. The guarantee
2

---

Conformal Prediction as Bayesian Quadrature
3. tecosion- Formulation
statistics. Let >eep be the following decision rule:
In this section we show how split conformal prediction and conformal risk control can be formulated as instances of general decision problem.
s((u+1)()-a))> 00,
if [(n + 1) - a)]
Aeep
otherwise.
(12)
Let (Z1..... 2n,) be set of calibration data where each observation = (Ii4, M/i consists of an input and ground truth label. Let 0 denote the true state of nature that defines a shared density [zv 0) for the data.' A new test point Enew assumed to have the same distribution. Let be control parameter (e.g. threshold) that must be chosen based on the calibration data. We assume the presence of loss function L(B, A) which quantifies the loss incurred by selecting À when the true state of nature is 0.
Then Asep is an a-acceptable decision rule for the miscov- erage loss Leep defined in (11).
Proof. Proofs for all theoretical results may be found in Appendix B.
Therefore the prediction set can be constructed as in (2):
Cecp (Frew) ) = {y E y ((Fnew: y) S Aspp
(13)
The theision-theoretic goal is to choose a decision rule X(z) that controls the risk:
and by Proposition 3.1, Cuep satisfies the conformal guaran- tee from (1).
R(O, A) = /
(8)
3.2. Recovering Conformal Risk Control
often desirable to choose so that it is robust to any possible state of nature 0. The maximum risk is defined
Conformal risk control generalizes split conformal predic- tion by considering losses that are monotonic non- non-inenenn increasing functions of single parameter A.
R(A) = sup A).
(f, X) = f [(Znew: (Zne d'zews
(14)
In distribution-free uncertainty quantification applications, it is often trivial to achieve arbitrarily low risk (for example by forming prediction sets covering the entire output space). We thus want to find decision rules whose risk is upper bounded by a constant a:
where (Znow, >) is an individual loss function that is mono- tonically non-increasing in X.
Proposition 3.2. Let Xero be the following decision rule:
R(A) S a.
(10)
Aero = inf { A:
(A:n+
1 {A: n +
Σ Elz A) + B a Ea.. (15)
and another criterion (such expected prediction set size) select among these. We call rule that satisfies (10)
Then Xere is an a-acceptable decision rule for Leve defined in (14).
an o-acceptable decision rule.
Note in particular that when 6(2, A) can be expressed in the form (In+1))1) 3n+1), this recovers the conformal risk control guarantee from (3).
3.1. Recovering Split Conformal Prediction
We now show how split conformal prediction is special case of this -heeoretcc problem. Let T (0, X) be the miscoverage loss:
4. Our Approach
Lup(0,A) A) Pr(s( 2}
(11)
We introduce our approach by reinterpreting split confor- mal prediction and conformal risk control as special cases of a more general Bayesian procedure. In order to do so, we borrow ideas from both Bayesian quadrature (Diaconis, 1988; O' O'Hagan, 1991) and distribution-fie tolerance re- gions (Guttman, 1970). Bayesian quadrature (Section 2.2) solves numerical integration problem by placing prior on functions and using Bayesian inference to compute a distribution over the value of the integral. Distribution- free tolerance regions provide a distribution over quantile spacings that holds regardless of the original underlying distribution. Putting these ideas together allows us to extend
= 11 - Prf Prfs((eecti ((oww) I A7
= =1- - / 1 1 (Znew) < A) (Znew 0) d.znew:
where s is an arbitrary nonconformity function.
Proposition 3.1. Define s; 2 s(z)) for = 1....,n n and let 5(1) S 8(2) .... S ((n) be the corresponding order
In the interest of notational convenience, assume densities and integrals over 2; but these may he replaced by probability mass functions and summations as appropriate.
3

---

Conformal Prediction as Bayesian Quadrature
conformal prediction by producing bounds on expected loss tailored to the actual losses observed in the calibration set.
expectation over individual losses:
= = I(0, x) =(0
f [((new: (Znww ()) d.zeww.
(20)
The remainder of this section is structured as follows. In Section 4.1, we discuss the relationship between risk control and Bayes risk. In Section 4.2, we describe a general ap- proach for using Bayesian quadrature to bound the posterior risk. In Section 4.3, we make the quadrature "distribution- free" by removing the dependence on prior over functions. In Section 4.4 we handle uncertainty in the evaluation loca- tions of the function by applying results that characterize the spacing between consecutive quantiles, In Section 4.5, we show how to use these results to produce an upper bound on the expected loss. Finally, in Section 4.6, we show how previous conformal prediction techniques can be viewed as a special case of our procedure that only considers the expectation of the posterior loss.
It is well-known that the expectation of a random variable is equal to the definite integral of its quantile function over its domain (Shorack, 2000, p. 1 16). Consider the distribution function of individual losses induced by À for particular value of 0:
F() & Fr(E(Znow: 2) 2)
(21)
The corresponding quantile function is:
K(t) F = inf(E: F(()) 2 t).
(22)
and the expected loss given K is simply So K(t) dt.
Instead of performing posterior inference over 6, we propose to take an approach inspired by Bayesian quadrature that places a corresponding prior over K. Figure I shows a schematic overview of Bayesian quadrature in this setting and how our proposed approach differs. The posterior risk given the observed individual losses l; - E(2), X) for i ,,..., becomes:
4.1. Bayes Risk
The risk R(O, A) measures the expected loss for one who already knows the true state of nature / but not the partieular data observed. However, in practical applications the situa- tion is reversed: we do know the observed data but there is uncertainty about the state of nature. Therefore, we want a decision rule that protects against high loss for a range of possible 0. This idea is expressed as the integrated risk:
E C1S1) = J[KTp(K) (in)
(23)
where J[K] A S6 ((t) dt and we have suppressed the dependence À for notational convenience. The posterior over quantile functions can be expressed as:
r( A) = / (O)
(16)
where the prior 7 (0) 2 0 measures the relative importance of the different possible states of nature. It is well-known that the minimizer of the integrated risk is the so-called Bayes decision rule:
p(K (im) = p(K timm lim)p(trra (1 dfim
(24)
X A argmin r(x | =),
(17)
p(K fi:ss (an) x T ((K II k(t.).
(25)
where r(A | 2) is the posterior risk
This resembles the Bayesian quadrature problem from Sec- tion 2.2, except the evaluation sites ti,..., fn are unknown. Fortunately, the distribution of f1,...; tu is independent of the true distribution of the losses, as we shall now show.
r(A =) FEL 2) / ((00)(=))<((1 =) (18)
and π (00 2) x T (0)f(( 0). Interestingly, the worst-case integrated risk of decision rule is identical to its maximum risk (9)
4.3. Elimination of the Prior Distribution
In order address the dependence of the posterior risk the prior π m(K), ), we derive upper bound on the posterior expected loss. The bound takes the form of a weighted sum of the observed losses, where the weights are determined by the spacing between consecutive quantiles.
F(A) A supr(m, A) sup A) = R(A). (19)
We can therefore focus on bounding the worst-case inte- grated risk F(A), since this will also bound the maximum risk R()).
Theorem 4.1. Let t(o) = ((++1) = 1, and C(n+1) B. Then
n+1
4.2. Reformulation as Bayesian Quadrature
sup E(L Ii:ns (1 S Σ u,l())
(26)
We now turn our attention to finding À minimizing the pos- terior risk (18). Consider risks that can be expressed as the
where Ue = 1()) - ((-1))

---

Conformal Prediction as Bayesian Quadrature
B
p(K Blmms)m))
p(L+ (1m)
(tra)
Density
Loss
Loss
(1/2) Fra))
(1)))
0
1
0
Quantile Level
Quantile Level
Loss
Figure 1. Overview of our approach. Left: Standard Bayesian quadrature places prior over the quantile function of the loss distribution. The posterior is formed via Bayes' rule after observing of loss values and quantile levels. However, in practice quantile levels directly observed. Middle: Our approach combines properties of quantile spacings with right rectangular integration rule to construct an upper bound the posterior distribution of the expected loss. Randomly sampled spacings and corresponding quantile functions are shown in blue along with 95% credible interval for each quantile level in black. Right: The posterior distribution for random variable L that upper bounds expected loss is constructed by integrating over the unknown quantile levels.
Theorem 4.1 is based on the definite integral of the "worst- case" quantile function that is consistent with the observa- tions. This strategy eliminates the need to specify a prior evaluate an integral over functions K. We now turn our attention to handling the uncertainty over the quantiles 1im..
the random variable defined as follows:
n+1
UI....,Un44 ~ 1), Σ
Udlo- (27)
Then for any b E (-o0, B).
4.4. Random Quantile Spacings
inf Pr(L < b| (1 2 Pr(L* $ ().
(28)
We now appeal to a result about distribution-free tolerance regions that characterizes the distribution of spacings be- tween consecutive ordered quantiles. Knowledge of this distribution will allow us to handle the input noise in the quadrature problem.
Theorem 4.3 states that L+ stochastically dominates the posterior risk?, which allows us directly form upper con- fidence bounds follows.
Corollary 4.4. For any desired confidence level 3 E (0, 1). define
Lemma 4.2 (Distribution of Quantile Spacings (Aitchison & Dunsmore, 1975, 140)). Suppose that fi....,6 ln are drawn i.i.d. with continuous? distribution function F. Let t; = F(6) and u, = t() - t(i-1), 1)- where by convention t()) 0 and t(n+1) 1. Then (uz, U2...... ... un+1) Dir(I......))
b9 = inf (b: Pr(I.t <b| (15)) 2 33.
(29)
Then inf, Pr(I. S b (i:) 2 for any b 2 bg.
The critical value by can be calculated by applying tech- niques for bounding linear combinations of Dirichlet ran- dom variables (Ng et al., 2011, p. 63). Alternatively, straight- forward Monte Carlo simulation of L+ is often sufficient, and the approach we take in our experiments. An illustra- tion shown in Figure
We are now ready to present our algorithm for bounding the expected loss E(L| (1s)).
4.5. Bound on Maximum Posterior Risk
Putting together Lemma 4.2 and Theorem 4.1 allows us to bound the maximum posterior risk.
4.6. Recovering Conformal Methods
Theorem 4.3. Define E(e to be the order statistics of l l,,..., ln for i = 1....,n and ((++1) B. Let L+ be
This perspective puts the previous distribution-fre uncer- tainty techniques in new light. Taking the expected value
"The correspondence to Dirichlet distribution holds exactly for continuous distributions. Weighted sums of Dirichlet random variates stochastically dominate weighted sums of discrete quan- tile spacings, and thus due space constraints only consider contimuous distributions here.
WWe note that the distribution of the U, conditions just on the order of the losses and not their specific values. This assumption is consistent with the frequentist treatment of conformal prediction, observation equally likely fall into
which assumes that each of the intervals formed by the loss values and the endpoints.
5

---

Conformal Prediction as Bayesian Quadrature
B
B
K(t)
Lo59
115) (4)
Lost
((5)
4(4
(3)
(3)
42)
42)
(1)
(1)
E[t[1]] E[t2)]] E[tp)] E[t(a)] E[t(s)] 1 Quantile Level t
((1) 422
((3)
(/4) Quantile Level
((S)
1
Figure Our Bayesian approach conformal prediction accounts for the variability in quantile levels better than previous approaches. Left: Conformal Risk Control (Angelopoulos et al., 2024) considers only the expectation over the unobserved quantile values ......ta ta. This can underestimate the true expected loss (shown here: estimated expected loss 0.45 vs. true expected loss 0.50). Right: Our approach makes use of the fact that the quantile spacings are drawn from Dirichlet distribution. By considering the full distribution over quantiles. we gain more complete view the expected loss. Shown here one sample drawn from this distribution, which estimates the expected loss as 0.58.
of Lt, we find
of controlling the risk such that R(0, A) S a for unknown O. We compare algorithms on the basis both the relative frequency of incurring risk greater than a and the prediction set size of the chosen A. The ideal algorithm would select À such that the relative frequency of exceeding the target risk is at most a target failure rate of 1 - B 0.05 while minimizing prediction set size.
- E(U E(UNO
n+1
1 n ++1
Σ 4 B). (30)
n
The Conformal Risk Control decision rule (15) then is sim- ply the infimum over À for which FE(T+)) a.
For, split conformal prediction, the individual loss is defined as 4 = 1 1[s; < A}. Therefore, suppose that À = S(k)-
As demonstrated in Section 4.6, our method recovers con- formal risk control by taking the expected value of L+, Therefore, in order demonstrate the effect of targeting a conditional guarantee (as opposed to marginal one in conformal risk control), we our Bayesian quadrature- based method to compute the decision rule based on the one-sided highest posterior density (HPD) interval:
The expected value L then becomes:
E(L*)
n + n Σ
1
n
1ss S {(N))
(31)
k
(32)
A inf Pr(L+ + S (1.)) 2 B),
(33)
Therefore, FTI.+) S a is satisfied whenever 2 (n + ()(1)),, and in particular by 1-' = [(x This recovers (12) when (nn 1)(1 a)1 n.
by finding the corresponding critical values b3 according to (29) via Monte Carlo simulation of Dirichlet random variates with 1000 samples. We include Risk-controlling Prediction Sets (RCPS) (Bates et al., 2021) with Hoeffding upper confidence bound as an additional baseline. Code for our experiments publicly available on Github.
Putting these results together, have recovered standard conformal prediction techniques but have the additional flexibility of considering the distribution of L rather than the expected value alone. Our experiments explore the value of this approach.
5.1. Synthetic Binomial Data
5. Experiments
We first sample directly from known loss distribution so that we can directly compute the frequency excessively large risk. Here the loss distribution is chosen be a scaled binomial distribution, normalized have maximum loss of B 1 and probability of failure set to 1 - X. This was
The primary goal of our experiments is demonstrate the utility of producing a posterior distribution over the expected loss. We conduct experiments on both synthetic data and calibration data collected from MS-COCO (Lin et al., 2014). For each data setting, we randomly generate M = 10,000 data splits. Each method is used select À with the goal
1nttpa:/ httpa:///gtthii com/ wwt.//////jaaaaaaaooaaa jakesnel1/ con fonformal 1 -as ses-bayes-gee quad
6

---

Conformal Prediction as Bayesian Quadrature
6. Discussion
Table 1. Relative frequency of trials (out of 10,000) for which the resulting decision rule X exceeded the target risk threshold
Our results Table 5.1 demonstrate that even though the Conformal Risk Control marginal guarantee holds, a sig- nificant number of individual trials (21.20%) may incur risk exceeding the target threshold. In contrast, by using the more conservative HPD criterion, very few of the trials (0.03%) exceeded the target risk. In Table 5.2, both RCPS and our method achieve failure rate below the target of 5% but our method achieves significantly smaller prediction intervals.
Decision Rule
Relative Freq.
95% CI
CRC
21.20%
[20.40%, 22.01 %]
RCPS
0.00%
[0.00%, 0.04%]
Ours (B = 0.95)
0.03%
[0.01 %. 0.09%]
Note: Error bars are computed as 95% Cooper-Pearsonn confidence intervals for binomial proportions.
These results point the qualitative difference in a marginal guarantee, which averages over many possible yet unob- served data sets vs. conditional guarantee which focuses on knowledge about the state of nature conditioned on the calibration data actually observed. Previous work on con- ditional guarantees (Barber et al., 2021; Gibbs et al., 2024) has focused on impu-conditional guarantees, where the guar- antee is conditioned on for all in the input domain. Guar- antees of this nature have been shown to be generally im- possible without stronger distribution assumptions. Our guarantees are perhaps better characterized by the term "data-conditional guarantee", where we condition on the set of observed loss values. Our experiments demonstrate the practical benefits of this by achieving decisions that produce smaller prediction sets and intervals while not vi- olating the constraint on maximum allowable failure rate. Our guarantees, in contrast, do not rely strong distribu- tion assumptions that would be necessary to produce an input-conditional guarantee.
simulated by computing
K 1[Vix > }}:
1 K
f(z X) =
(34)
kml
where Vi ~ Uniform(0,1) for i = 1,...., n and = I..... K. This loss is therefore monotonically non- increasing in À and achieves zero loss at Amax = 1. We set 10, K 4, and a 0.4.
Since the expectation of the loss (34) is 1 X, any trial for which A= < 0.6 constitutes risk exceeding the a threshold. The relative frequency of trials exceeding this risk threshold are tabulated in Table 5.1. A histogram of the chosen X for each of the methods across all 10,000 trials shown in Figure 3. For conformal risk control, the mean risk across all trials was 0.3363 = 0.0007 and for our approach X the mean risk was 0.1758 = 0.0006. In order to visualize the distribution of I,+, we plot histogram of I according to (27) estimated with 100,000 Dirichlet samples for three settings of À E {0.7, 0.8, 0.9). The results are shown in Figure 4.
The results are again confirmed in Table 5.3 on MS-COCO, which show that the marginal guarantees of Conformal Risk Control lead to an even greater percentage of trials exceed- ing the risk threshold. On the other hand, RCPS is able to control the risk but this comes at the cost of larger pre- diction sets. Our approach successfully balances these two concerns, producing prediction intervals that are shorter than baselines while not exceeding the maximum acceptable fail- ure rate. is also clear that the distribution of the expected loss upper bound L+ in Figure 4 provides a more complete view of the range of possible losses and its dependence on X, a perspective that is not offered by previous methods.
5.2. Synthetic Heteroskedastic Data
In this experiment we also use 10,000 random trials. We use n = 200 calibration samples each. To achieve heteroskedas- ticity, we let X ~ U[0, 4] and Y | X ~ (0, X²). The prediction intervals are then formed as 1- Xi where Å is selected by each method. The loss is the miscoverage loss and the target loss is set to a = 0.1 (i.e. 90% coverage). The maximum allowable risk failure rate is set to 5% (i.e. 3 0.95). The results are show in Table 5.2.
Our goal in this work is to show that the Bayesian viewpoint unlocks a richer interpretation compared to previous works, which focus on marginal guarantees that as we have shown in the paper correspond to the posterior mean. In order to draw an explicit correspondence between our work and previous approaches, the dependence the prior was re- moved in Section 4.3. The intuition is that that any rational decision maker operating according to the rules of proba- bility, regardless of prior (sufficiently expressive), would agree with the upper-bounding distribution of we derive. Naturally, commitment to a specific choice of prior would
5.3. False Negative Rate on MS-COCO
We also compare methods on controlling the false neg- ative rate of multilabel classification on the MS-COCO dataset (Lin et al., 2014). The experimental setup mirrors that used by Angelopoulos & Bates (2023, Section 5.1). Each random split contains 1000 calibration examples and 3952 test examples. The results of this experiment are sum- marized in Table 5.3.
7

---

Conformal Prediction as Bayesian Quadrature
500
600
Number of Trials
Number of Trials
400
400
300
200
200
100
0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Acre
0.95 hpd
Figure Comparison of risk incurred each procedure across multiple trials. Left: Histogram of the decision rule Aue chosen by Conformal Risk Control across M = 10,000 randomly sampled calibration sets. The region where per-trial risk exceeds a is highlighted in red. Right: Histogram of the 2 chosen according our 95% Bayesian posterior interval.
Table Relative frequency of trials (out of 10,000) for which the resulting decision rule exceeded the target risk threshold in the synthetic heteroskedastic experiment.
Decision Rule
Relative Freq.
95% CI
Mean Prediction Interval Length
Split Conformal Prediction/ CRC
46. 19%
[45.21% %, 47. 17%]
7.99
RCPS
0.0%
[0.0%, 0.04%]
14.29
Ours (B 0.95)
3.42%
[3.07%, 3.80%]
9.50
Note: Error bars are computed as 95% Clopper-Pearson confidence intervals for binomial proportions.
aforementioned assumptions do hold, the actual loss values may be significantly less than indicated by method.
Table 3. Results on MS-COCO comparing relative frequency of trials for which the resulting decision rule À exceeded the target risk threshold a and average prediction set size.
Overall, our approach demonstrates how conformal pre- diction techniques can be recovered and extended using Bayesian probability, all without having to specify prior distribution. This Bayesian formulation is highly flexible due its nonparametric nature, yet is amenable to incorpo- rating specific information about the distribution of losses likely to be encountered. In practical applications, maximiz- ing the risk with respect to all possible priors may be too conservative, and thus future work may explore the effect of specific priors on the risk estimate.
Method
Relative Freq.
Pred. Set Size
CRC
45.05%
2.92
RCPS
0.0%
3.57
Ours (8 0.95)
5.43%
3.04
lead to tighter distributions over the posterior risk, and in future work we seek to bridge these fields even further by exploring specific choices of priors over quantile functions.
The limitations of our method lie primarily in the two main assumptions it makes. First, assumes that the data at deployment time are independent and identically distributed the calibration data. Second, assumes an upper bound B the losses. If either of these assumptions do not hold, then the guarantees produced by our method are no longer valid. Additionally, the bounds produced by our method are conservative in the sense that they hold for any choice of prior for the loss distribution (provided that the prior is consistent with the calibration data). Therefore, the two
7. Related Work
Statistical Prediction Analysis. Statistical prediction analysis (Aitchison & Dunsmore, 1975) deals with the use of statistical inference to reason about the likely outcomes of future prediction tasks given past ones. Within statistical prediction analysis, the area of distribution-free prediction assumes that the parameters or the form of the distribu- tions involved cannot identified. This idea can traced back Wilks (1941), who constructed method to form
8

---

Conformal Prediction as Bayesian Quadrature
6
main grounded in frequentist probability. Another class of related work seeks to endow conformal prediction with prop- erties naturally enjoyed by Bayesian methods, including the ability to incorporate prior information (Hoff, 2023), pro- duce predictive distributions (Vovk et al., 2017), or adapt to time-varying data distributions (Prinster et al., 2024). Stan- ton et al. (2023) show how to apply conformal prediction to Bayesian optimization, another setting where maintaining a posterior over functions is important. Separately, Bayesian approaches to predictive uncertainty are popular (Hobb- hahn et al., 2022) but make extensive assumptions about the form of the underlying predictive model. To our knowledge. we are the first to apply statistical prediction analysis and Bayesian quadrature in order to analyze the performance of black-box predictive models in distribution-fiee way.
À = 0.7
Estimated Density
A = 0.8
À 0.9
4
2
0.0
0.2
0.4
0.6
0.8
1.0
L
Figure 4. Probability density for I with A E 40.7,0.8,0.9 0.93 esti- mated using 100,000 Dirichlet samples.
8. Conclusion
Safely deploying black-box predictive models, such as those based on deep neural networks, requires developing meth- ods that provide guarantees of their performance. Existing techniques for solving this problem are based on frequen- tist statistics, and are thus difficult to extend to incorporate knowledge about the situation in which models may be deployed. In this work we provided a Bayesian alterna- tive to distribution-fiee uncertainty quantificationn, show- ing that two popular existing methods are special cases of this approach. Our results show that Bayesian probability can be used to extend uncertainty quantification techniques, making their underlying assumptions more explicit, allow- ing incorporation of additional knowledge, and providing more intuitive foundation for constructing performance guarantees that avoid everly-optimiste guarantees that can be produced by existing methods.
distribution- frree tolerance regions. Tukey (1947; 1948) gen- eralized distribution-fres tolerance regions and introduced the concept of statistically equivalent blocks, which are anal- ogous the intervals between consecutive order statistics of the losses. Much of the relevant theory is summarized by Guttman (1970), and the Dirichlet distribution of quan- tile spacing is discussed by Aitchison & Dunsmore (1975). We build upon these works by connecting them to Bayesian quadrature and applying them in the more modern context of distribution-fret uncertainty quantification.
Bayesian Quadrature. The use of Bayesian probability to represent the outcome of a arbitrary computation is termed probabilistic numerics (Cockayne et al., 2019; Hennig et al., 2022). Since our approach is fundamentally based on in- tegration, we focus primarily on the relationship with the more narrow approach of Bayesian quadrature, which em- ploys Bayes rule to estimate the value of an integral. A lucid overview of this approach discussed under the term Bayesian numerical analysis by Diaconis (1988), who traces it back to the late nineteenth century (Poincaré, 1896). The use of Gaussian processes in performing Bayesian quadra- ture is discussed in detail by O' O'Hagan (1991). Our approach is formulated similarly but differs in two main ways: (a) we use conservative bound instead of an explicit prior, and (b) we have input noise induced by the random quantile spacings.
Impact Statement
This paper introduces a practical algorithm for computing posterior distribution for the expected loss based on the observed losses from a set of calibration data. The intended purpose of this algorithm is for the posterior distribution to inform deployment decisions of black-box predictive systems (e.g. deep neural networks) in safety-critical ap- plications. Our method makes use of certain assumptions, discussed in Section 6, which if violated will lead guaran- tees that may no longer hold. In particular, is important ensure proper monitoring detect distribution shift between calibration and deployment.
Distribution Uncertainty Quantification. Relevant background on distribution uncertainty quantification techniques is discussed in Section 2. A recent and com- prehensive introduction to conformal prediction and related techniques may be found in (Angelopoulos & Bates, 2023). Some recent works, like ours, also make use of quantile functions (Snell al., 2023; Farzaneh al., 2024) but re-
Acknowledgements
The authors would like to thank the anonymous reviewers for helpful comments and Hamed Hassani for helpful dis- cussions about the posterior quantile distribution. We also

---

Conformal Prediction as Bayesian Quadrature
would like thank Drew Prinster for bringing several re- lated works to attention. This work was supported by grant N00014-23-1-2510 from the Office of Naval Research.
Hobbhahn, M., Kristiadi, A., and Hennig, Fast predic- tive uncertainty for classification with Bayesian deep net- works. In Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, 2022.
References
Hoff, P. Bayes-optimal prediction with frequentist coverage control. Bernoulli, 29(2), May 2023.
Aitchison, J. and Dunsmore, I. R. Statistical Prediction Analysis. New York: Cambridge University Press. 1975.
Karvonen, T. and Särkkä, S. Classical quadrature rules via
Gaussian processes. In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), 2017.
Angelopoulos, A. N. and Bates, Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494-55 2023.
Kot, M. A First Course in the Calculus of Variations. Amer- ican Mathematical Society, 2014.
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. Conformal risk control. In The Twelfth International Conference on Leurning Representations, 2024.
Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. Distribution-free predictive inference for regression. Journal of the American Statistical Associa- tion, 113(523):1094-1111 July 2018.
Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, J. J. The limits of distribution- free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):55--482 June 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.. Microsoft COCO: Common objects in context. In 13th European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Bates, S., Angelopoulos, A., Lei, L.,, Malik, J., and Jor- dan, M. Distribution-free risk-controlling prediction sets. Journal of the ACM, 68(6), 2021.
Ng, K. W., Tian, G.-L., and Tang, M.-L. Dirichlet and Related Distributions: Theory, Methods and Applications. Wiley, 2011.
Cockayne, J., Oates, C. J., Sullivan, J., and Girolami, M. Bayesian probabilistic numerical methods. SIAM Review, 63))77556778, 2019.
O'Hagan, A. Bayes- quadrature. Journal of Statis- tical Planning and Inference, 29(3):245-261 November 1991.
Diaconis, P. Bayesian numerical analysis. In Berger, J. and Gupta, S. (eds.), Statistical Decision Theory and Related Topics IV, volume pp. 163-175. Springer-Verlle Verlag, 1988.
Papadopoulos, H., Proedrou, K., Vovk, V., and Gammer- man, A. Inductive confidence machines for regression. In Elomaa, T., Mannila, H., and Toivonen, H. (eds.), ECML 2002, volume 2430, pp. 345-356. Springer Berlin Heidel- berg, 2002.
Farzaneh, A., Park, S., and Simeone, O Quantile learn- then-test: Quantile-hased risk control for hyperparameter optimization. IEEE Signal Processing Letters, 2024.
Gibbs, I., Cherian, J., and Candès, Conformal Pre- diction With Conditional Guarantees, September 2024.
Poincaré, H. Calcul des Probabilités. Georges Carré, 1896.
Guttman, I. Statistical Tolerance Regions: Classical and Bayesian. Griffin's Statistical Monographs and Courses, No. 26. Griffin, 1970.
Prinster, D., Stanton, S., Liu, A., and Saria, S. Conformal va- lidity guarantees exist for any data distribution (and how to find them). In Proceedings of the 41st International Conference on Machine Learning, 2024.
Hennig, P., Osbore,, M. A., and Kersting, H. P. Proba- bilistic Numerics: Computation Machine Learning. Cambridge University Press, 2022.
Shafer, G. and Vovk, V. A tutorial on conformal prediction. Journal of Machine Learning Research, ((1)) 2008.
Hill, B. M. Posterior distribution of percentiles: Bayes' theorem for sampling from population. Journal of the American Statistical Association, 63(322):677-691, June
Shao, J. Mathematical Statistics. Springer, 2003.
Shorack, G. Probability for Statisticians. Springer-Veriie Verlag, 2000.
1968.
Hill, B. M. de Finetti's theorem, induction, and A,, or
Shorack, G. R. and Wellner, J. A. Empirical Processes with Applications to Statistics. Society for Industrial and Applied Mathematics, 2009.
Bayesian nonparametric predictive inference (with dis- cussion). Bayesian statistics, 3:211-241, 1988.
10

---

Conformal Prediction as Bayesian Quadrature
Snell, C., Zollo, T. P., Deng, Z., Pitassi, T., and Zemel, R. Quantile risk control: A flexible framework for bounding the probability of high-loss predictions. In The Eleventh International Conference on Learning Representations, 2023.
Stanton, S., Maddox, W., and Wilson, A. G. Bayesian Opti- mization with Conformal Prediction Sets. In Proceedings of The 26th International Conference on Artificial Intelli- gence and Statistics, pp. 959-986, April 2023.
Tukey, W. Nonparametric estimation II. Statistically equivalent blocks and tolerance regions-the the continuous case. The Annals of Mathematical Statistics, pp. 529-539, 1947.
Tukey, W. Nonparametric estimation, III. Statistically equivalent blocks and multivariate tolerance regions-thi the discontinuous case. The Annals of Mathematical Statis- tics, pp. 30-39, 1948.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic Learning Random World. Springer Science & Busi- ness Media, 2005.
Vovk, V., Shen, J., Manokhin, V., and Xie, M. Nonparamet- ric predictive distributions based on conformal prediction. In Proceedings of the Sixth Workshop Conformal and Probubilistic Prediction and Applications, pp. 82-102.
PMLR. May 2017.
Wilks, S. S. Determination of sample sizes for setting tolerance limits. The Annals of Mathematical Statistics, 12(1))91-96, 1941.

---

Conformal Prediction as Bayesian Quadrature
A. Theoretical Preliminaries
A.I. Review of Problem Setup
We first review some relevant aspects of our problem setup.
Loss Function. We assume an upper bound on the losses: I (-oc, B] for = I,..., We assume the same upper bound for feew-
Bayesian Quadrature of Quantile Functions. Recall Bayes rule for quantile functions:
p(K K fi (in) & n (K) II 6(4, ((t))),
(35)
where 8 is the Dirac delta function. The prior ((K) is assumed be sufficiently expressive have nonzero measure for the set K2, of quantile functions such that ((t,) E, for i I.....7 and K Kn, This is necessary to prevent the posterior distribution in (35) from becoming degenerate.
A.2. Background
We begin by recalling some basic properties of distribution functions and quantile funetions.
Proposition A.1 (Properties of Distribution Functions (Shao, 2003, p. 4)). Let F(x) = Pr(X S x) be a distribution
function. Then F (-oo)) lim F()) 0, ((o)) lim F(x) = F is nondecreasing (i.e., F(x) S Fly) if
v). and F is right continuous (i.e., limy-+x,y2 F(y) Fx:)).
Let F be distribution function and ((t) F (t) inf 1 F(x) 2 t be the corresponding quantile function.
Proposition A.2 (Quantile Functions are Nondecreasing). If S u, then K(1) S K(u).
Proof. Since u 2 t, it follows that fx : F(x)) R u} S fx: F(a) N t). Taking the infimum of both sides yields
inf(x F(x) w) inf F(x) t} K(u) 2 K(t).
(36)
also will make use the probability integral transformation, which we state here for convenience.
Proposition A.3 (Probability Integral Transformation (Shorack & Wellner, 2009, 5)), If X has distribution function F, then
Pr(F(X) st for all 0 < S1S1,
(37)
with equality failing if and only if is not in the closure of the range of F. Thus F is continuous, then T F(X) is Uniform(0, 1).
B. Proof of Results from the Main Paper
B.1. Proof Proposition 3.1
Recall that Leep(0 X) is the miscoverage loss:
Trag (0, A) = Pr(s (r(s((en knew)
> A}
(38)
= 1 - (r(w()new) 5 A}
= 1 - / (Knew)) ) S (zuw (8) džnew:
where s an arbitrary nonconformity function.
12

---

Conformal Prediction as Bayesian Quadrature
Proposition 3.1. Define 8; A s(z,) for i 1,...,. n and let 8(1) S 8(2) ... 8(m) be the corresponding order statistics. Let dspp be the following decision rule:
(((m+))()aa)))>> 0o,
if [m + 1)(1 - a:] S
rep
(12)
otherwise.
Then Aso is a-acceptable decision rule for the miscoverage loss Lecp defined in (11).
Proof. By Lei et al. (2018, Section 2),
Pr(Snew S SQ1-a) 2 a,
(39)
where
1((m++)(1-0]]
if [m + 1)(1 a)] S otherwise.
@ -1-0-
(40)
0c,
But Lscp(0. (0, = 1 Pr(Snow I A 0), so for À = â1-ai R(O, >sep) S a. This statement not depend on 6, and so R((Nup) S a..
B.2. Proof of Proposition 3.2
Recall that the Lev is defined as:
/ (Znew (Zne () dzzew:
Lure (0, A) =
(41)
where {(Znow, A) is an individual loss function that is monotonically non-increasing in A. Proposition 3.2. Let Pere be the following decision rule:
E A) + sag )
Acre
inf
Σ
(15)
12
Then Pere is an a-acceptable decision rule for Lere defined in (14).
Proof. Let Li...., Lnn Ln+1 be an exchangeable collection of non-increasing random functions L; A (-o0,
By Angelopoulos et al. (2024, Theorem 1),
BL S a,
(42)
where
A = inf { A: n (m)
+ B
<a
(43)
++11 n +
and R. (A) = (A) +... I,, (A))/n.
Interpreting these results using the notation from Section 3 of the main paper, we identify:
(A()) = E(2, for = 1.... n and (n+1(A)= = ((Zeew, A).
Aere identical Å from (43), and
(42) states that R(8, Acre) S a for any A.
Therefore, R())xe) sup, R(0, Aere) S a.
13

---

Conformal Prediction as Bayesian Quadrature
B.3. Proof of Theorem 4.1
In order prove Theorem 4.1, we will need to make use of two auxiliary propositions (Proposition B.1 and Proposition B.2). We state and prove these first, and then proceed prove Theorem 4.1.
Proposition B.I. Consider the following variational maximization problem:
T = [: f(x) dx
(44)
subject f(a) = fa. f(b) fb. and fo f(x) S for all (a, bl, where fa S fo. Then I[f] is maximized by
f*xx) { fa if Sb. otherwise,
(45)
and 1[ (b- a)fb-
Proof. We apply Euler's method (Kot, 2014, Section 2.2), which approximates the variational problem m-dimensional problem and takes the limit as m - oo. Let the interval [o, b] be divided into m++1 subintervals of equal width Ar The objective functional can then be approximated as
b
m + 1
Iff fm) = Σ fz
(46)
==
where fo fa and fm+11 fo due to the boundary conditions. In order handle the fo I f(z) fo constraint, first impose f(x) $ fo and check if the solution also satisfies f(r 2 fa- To that end, we substitute fj fb - -3.
I(E1,...,Em.) 5m)= - -
(47)
j=0
We then take partial derivatives with respect to Ex:
01
1 0I
26 Ar
26x
(48)
Ar
Taking the limit as m - oo and Ax + 0, the variational derivative becomes:
87 = -
(49)
85
87 yields {(x) 0, which recovers f(x) for except at æ a, where f(a) fa by the boundary conditions. Setting og This recovers (x) from (45), which indeed satisfies f(x: 2 fa. For f", it is evident that the value of the functional is [[f"] (b fb.
Proposition B.2. Let Kr, be the set of quantile functions for which K(t,) = l, for = 1,..., Then
12+
sup KER,,
J[K] = Σ (t() - t(--)))(41
(50)
where 40) 0, ((+1)) 1, (m+1)) B, and J[K] A S2 k(t) dt.
14

---

Conformal Prediction as Bayesian Quadrature
Proof. By Proposition A.2, quantile functions preserve orderings and therefore k((()) E()- We divide J[K] into
intervals with endpoints (0, ka)) (o)))) (1(m):
sup KERa
J[K] = sup KERn 0
((t)
dt
(51)
n+1
sup KEK.,
K(1)
(52)
n+1
sup KER,
K(t) dt
(53)
t6s 1
By Proposition A.2, K (1((-1)) S K(1)= & (((()) for any & (1)-1)): (()]. view each term as variational subproblem where J:[K.]
(t0 Ki(t) dt with boundary conditions Ki(((-1))) 1(3-1) and K.(l()) (o)- We therefore appeal to
1 Proposition B.1 conclude that
((-1)
if f1=44-11) otherwise.
(t)
(54)
and J[Kţ] (t(o) (1) We therefore have
n+1
sup KeR.,
J[K]
S
Σ (400 - 10-1)
(55)
By composing Ki from each subinterval, is straightforward see that the bound tight for
(()) if S (()
((2)
ff < <t $ t(2)
K2 Eim (t)
(56)
l(m) if (nn-)) < <ts t((m)
if [())-
din is therefore the "worst-case" quantile function that is consistent with the observations, and J JKK fim (im Ext (t() - 1()
We are now ready to prove Theorem 4.1.
Theorem 4.1. Let ((o)- 0, ((++1) and ((++1) B. Then
u+1
sup B(L bisn (1m)
Σ
uil
(26)
x
1=1
where u, = t(3) t(i-1)-
Proof. Let J[K] So' ((t) dt. The conditional expected loss can be expressed as:
E(L f1:m, (1:a) = / J[K]p(K Ii (1.1) dK
(57)
S sup J[K],
(58)
KEK,
where Krn, is the of quantile functions for which K(t,)) E, for 1,..., n. By Proposition B.2, it follows that
n+1
n+1
E(L| tims lim) <
Σ
(t() - t(: 1)) =
(59)
1=1
15

---

Conformal Prediction as Bayesian Quadrature
B.4. Proof of Lemma 4.2
Lemma 4.2 (Distribution of Quantile Spacings (Aitchison Dunsmore, 1975, p. 140)). Suppose that e in are drawn i.i.d. with continuous distribution function F. Let (4,) and u, (() (i- where by convention ((0) 0 and f(++1) = 1. Then (u1, u2...... Un+1) = Dir(I......)) 1).
Proof. By the probability integral transformation (Proposition A.3), T₁ is Uniform for .,..., Since the transformation from (f1...., (n) + (A(1)...., f(x)) is sorting operation where n! permutations map to the same vector of order statistics, the probability density for ((1), ...: [(n) is therefore
fa (t()) ..)............................. ((m) n!,
0 S (m) S t()) S... (m))
(()) S 1.
(60)
If M1:: = (()))))) where G' is differentiable and invertible, then by change of variables the density for umm can be expressed as
f (uim) = ftg (G-1 (21m)) det
(u1m))
(61)
Observe that the inverse transformation ((mm) = G- (m1:)) can be expressed as
t(1)
0
0
1/1
((2)
0
0
0
t(3)
1
1
0
0
U/3
(62)
".
:
(n-1))
1
1
0
Un-1
t(m)
1
Un
Hence the absolute Jacobian of inverse transformation ((mm) = GG is 1. The density of U1:n is therefore
fax. (u1m)) = Stq (G (G-'((In))) = n!, where 2 0 for i = 1,..., and
Us
S 1.
(63)
1=1
Recall that the Dirichlet density with parameter a1,..., On+11 is:
r 04)
n+1 i=
1
Dir(urm+1 (11+1))=
0--1 1=1
where u; 2 0 and Σ 1.
(64)
[(a)) I(ant1)
In particular, a1 = a2 = =... an+1 11
Dir(Uint1 1,..., I(n + 1) n!,
(65)
which is identical to (63) with un+11 = Ui Un Therefore, (u), u2..... un+1) = Dir(1....,,))
B.5. Proof Theorem 4.3
Theorem 4.3. Define (g) to be the order statistics of li,.... En....,,,,,e for 1...., variable defined as follows:
... n and ((+11) & B. Let L be the random
U U1,...,Un41 Dir(1...., 1), E+
(27)
Then for any b E (-o0, B).
inf S 5b| (1.) 2 Pr(L+ b).
(28)
'The correspondence to Dirichlet distribution holds exactly for continuous distributions. Weighted sums of Dirichlet random variates stochastically dominate weighted sums of discrete quantile spacings, and thus due space constraints only consider continuous distributions here.
16

---

Conformal Prediction as Bayesian Quadrature
Proof.
inf Pr(L =b (mm) = inf / = bb) I (i:)) dK
(66)
inf / [AIJIK] Sb] (DLK timm (ptnn (im) dt dK
(67)
inf // [[][K] 50
p(K tioms lim) dk p(tzsn (im) dtim b) [
(68)
/ inf / [J[[K] bb) tisms lim) dk
p(tin (im) dtin
(69)
inf 1 [J[K]] {b] p(tzm dlin KEKn.
(70)
2 [JKK
Ain s b) p(tim (in)
(71)
- / Σ (n 11-1))0000 b by plleml (is) dl
(72)
n+1
Σ
meleo) sb plurmaz (1x)) duin+i
(73)
= Pr(L** + <b) b)
(74)
Note that here the quantile distribution p((.: (mn takes into account the ordering of the losses but not the values themselves. This is consistent with the implicit assumption, made in the frequentist treatment conformal prediction, that a new observation is equally likely to fall into each of the intervals formed by the loss values and the endpoints. The results for characterizing this distribution (1975), and is also known as A,, in the literature (Hill, 1968; 1988). These results allow to make minimal assumptions about the form of the CDF.
be traced back to Tukey (1947), by way of Guttman (1970) and Aitchison & Dunsmore
B.6. Proof of Corollary 4.4
Corollary 4.4. For any desired confidence level (0. define
bg inf{b: Pr(L+ <bl (im) 2 3).
(29)
Then inf, Pr(L < 6 (1:n) Z B for any b z bjg
Proof. For any b 2 b3. Pr(L+ <b 61 ( 2 B. Substitution into (28) provides the desired result.
17


Reviews:
[{"review_id": "I1MHY8f6Kf", "reviewer": "Reviewer_K6B1", "summary": "This paper proposes a Bayesian reinterpretation of conformal prediction, framing it within a Bayesian quadrature framework. The authors shows that split conformal prediction and conformal risk control can be derived as special cases of Bayesian quadrature. By modeling uncertainty over quantile functions and leveraging Dirichlet-distributed quantile spacings, they derive a posterior distribution over expected losses, enabling more interpretable and adaptive risk guarantees.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to write a thoughtful and detailed review of our work. We appreciate that the review has recognized the \"very interesting and novel insights about UQ & decision making\" provided by our paper.\n\nWe appreciate the desire for additional experiments to provide additional insight about our method. To this end, we have implemented experiments on heteroskedastic data. The findings are largely in line with the results from Section 5 of our paper: our Bayesian interpretation produces prediction intervals that are shorter than baselines while not exceeding the maximum acceptable failure rate.\n\nIn each of the 10,000 random trials we use 200 calibration samples. To achieve heteroskedasticity, we let $X \\sim U[0, 4]$ and $Y|X \\sim \\mathcal{N}(0, X^2)$. Prediction intervals are formed as $[-\\hat{\\lambda}, \\hat{\\lambda}]$ where $\\hat{\\lambda}$ is to be selected by each method. The loss is the miscoverage loss and the target loss is set to 0.1 (i.e. 90% coverage). The maximum allowable risk failure rate is set to 5% (i.e. $\\beta = 0.95$). Please note that since the miscoverage loss is used here, Conformal Risk Control and Split Conformal Prediction coincide.\n\n| Method | Relative Freq. (Failure Rate) | 95% CI | Mean Prediction Interval Length |\n|---|---|---|---|\n| Split Conformal Prediction / CRC | 46.59% | [45.61%, 47.57%] | 7.67 |\n| RCPS | 0.0% | [0.0%, 0.04%] | 13.99 |\n| Ours ($\\beta = 0.95$) | 3.75% | [3.39%, 4.14%] | 9.14 |\n\nThe results indicate that even in the heteroskedastic setting, our method allow more precise control of $\\hat{\\lambda}$. Unlike RCPS, which is overly conservative, ours produces the shortest prediction intervals while not violating the maximum allowable risk failure rate.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "As far as I can tell, the theoretical claims are correct.", "experimental_designs_or_analyses": "The two datasets incorporated are well-designed and illuminating. I would love to see more experiments though, to provide some intuition on the strengths & drawbacks of the Bayesian quadrature method in practice. For example, would it be possible to test on synthetic data with non-monotonic loss functions or heteroscedastic noise?", "relation_to_broader_scientific_literature": "There have been attempts to formulate/use conformal prediction for Bayesian inference ([e.g.](https://arxiv.org/abs/2210.12496)), but non as unified as the proposed method.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Coming from some one who is mostly familiar with the frequentist side of things, this paper provides very interesting and novel insights about UQ & decision making. It shines light on the conservatism (the method needs to work for *any* prior), and failure modes of CRC because of the limitation of expectation. Although the empirical experiments are not comprehensive, the theoretical contributions and interesting enough to make up for it.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "See experiments section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ShitsGHboP", "reviewer": "Reviewer_vkyV", "summary": "The authors propose a Bayesian version of conformal prediction, which guarantees conditional coverage, rather than marginal coverage.  The technique is distribution free, since it considers the worst case risk by maximizing over all possible priors. This is made tractable by leveraging some prior results on distribution free analysis of quantile spacings, combined with Bayesian quadrature. Their method includes frequentist split conformal prediction and conformal risk control (CRC)  as special cases.They show significantly improved results (in terms of risk control and prediction set size) on synthetic data, and on controlling the false negative rate of multilabel classification on the MSCOCO, compared to  CRC and Risk-controlling Prediction Sets (RCPS). Overall a very impressive paper.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to write a thoughtful and detailed review. We are pleased that the review recognized the novelty of creatively combining Bayesian quadrature and distribution-free analysis of random quantile spacings.\n\nWe will update the paper to clarify that our guarantees are conditional on the observed calibration data. In essence, our guarantees are probabilistic statments about the risk conditioned on the observed losses (see e.g. the conditioning on $\\ell_{1:n}$ in Theorem 4.3 and Corollary 4.4). This stands in contrast to previous approaches which rely on marginalizing over many possible realizations of the calibration losses that were not observed. We will add a paragraph to the paper explicitly clarifying this point.", "claims_and_evidence": "Theoretical claims (see summary) are supported by proofs (not checked) and compelling experimental results.", "methods_and_evaluation_criteria": "Yes, good eval on synthetic data and a standard challenging real world image classification benchmark.", "theoretical_claims": "I did not check the correctness of the proofs.", "experimental_designs_or_analyses": "Experiments seem sound.", "relation_to_broader_scientific_literature": "Related work is very well explained. This particular combination of techniques (Bayesian quadrature and distribution-free analysis of random quantil spacings) seems entirely novel, and is very creative.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "As I said above, extremely strong paper.", "other_comments_or_suggestions": "Emphasize that your method is conditional on observed data, and is better than a marginal guarantee.\nThis part of Bayes is more important than using a prior (which you avoid).", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Szm1DVtR8i", "reviewer": "Reviewer_gGKC", "summary": "The paper proposes a Bayesian quadrature approach as a Bayesian alternative to conformal prediction, encompassing two widely used methods: split conformal prediction and conformal risk control. The equivalence between these approaches and Bayesian quadrature is clearly established through theoretical proofs. Empirically, the proposed Bayesian method demonstrates strong performance compared to frequentist alternatives, particularly in terms of lower risk violation frequency and more compact prediction sets.\n\n## Update after rebuttal\nI found the authors’ rebuttal satisfactory. I have maintained my score 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for writing a detailed review that recognizes the bridging nature of our work and its benefits for various decision-making domains.\n\n**Generalizing Conformal Prediction.** One of our main contributions is to show how both split conformal prediction and Conformal Risk Control can be recovered by taking the posterior mean of our upper bounding loss random variable $L^+$. Thus, our method illuminates a broader viewpoint and uses this insight to choose $\\lambda$ more effectively. We believe this is generally useful perspective that may have broader implications on conformal prediction in general. Nevertheless, we will update the text of the manuscript to clarify the nature of the generalization elaborated here (i.e. oriented towards split conformal prediction and Conformal Risk Control).\n\n**Role of the prior.** We thank the reviewer for raising this nuanced but important point. Our goal is to show that the Bayesian viewpoint unlocks a richer interpretation compared to previous works, which focus on marginal guarantees that as we have shown in the paper correspond to the posterior mean. For better or for worse, there is still a big gap in the literature between traditional approaches to distribution-free uncertainty quantification, which are predominantly frequentist in nature, and methods like Bayesian quadrature which are firmly Bayesian in nature. Therefore, to draw an explicit correspondence between the two, the dependence on the prior is removed in Section 4.3. The intuition is that that any rational decision maker operating according to the rules of probability, regardless of prior (sufficiently expressive), would agree with the upper-bounding distribution of $L^+$ we derive. Naturally, commitment to a specific choice of prior would lead to tighter distributions over the posterior risk, and in future work we seek to bridge these fields even further by exploring specific choices of priors over quantile functions. We will add a paragraph explicitly discussing these points to the manuscript.\n\n**Figure 1.** Thank you for raising this point. The interpretation stated in the review is correct: the smoothness does indeed result from the combination of a step function with the distribution over quantile spacings. Our original intention was to illustrate the general idea of Bayesian quadrature in our setting with Figure 1, and then move into the specifics of our method in Figure 2. However, we will update Figure 1 to more clearly signpost the use of the step function, both in the caption and by updating the figure itself.", "claims_and_evidence": "The paper claims that the proposed Bayesian quadrature approach generalizes conformal prediction. Specifically, Propositions 3.1 and 3.2 demonstrate that split conformal prediction and conformal risk control can be reinterpreted within a decision-theoretic framework, while equations (30)–(32) show that the expected loss of the quantile spacing approach recovers the corresponding conformal methods. These claims are supported by theoretical proofs. \nHowever, given my limited knowledge of conformal prediction, I am unsure whether these two methods alone are sufficient to establish that Bayesian quadrature is a generalization of conformal prediction. In this sense, the title may be somewhat strong. That said, I do agree that the proposed approach encompasses these two conformal methods.", "methods_and_evaluation_criteria": "They first establish the mathematical equivalence between their Bayesian quadrature approach and existing conformal prediction methods, specifically split conformal prediction and conformal risk control. They then present experimental results demonstrating that their proposed Bayesian posterior risk approach reduces the number of individual trials that exceed the target risk threshold while maintaining a smaller prediction set size compared to existing conformal methods.", "theoretical_claims": "I have not verified all the theoretical proofs in detail, but the claims appear reasonable to me.", "experimental_designs_or_analyses": "I reviewed the experimental results but did not examine all the details, such as the implementation code. However, the findings appear reasonable to me.", "relation_to_broader_scientific_literature": "Quantifying the uncertainty of black-box predictors is crucial for high-stakes applications and exploratory tasks. I expect this paper to benefit various decision-making domains, including autonomous driving, experimental design, and time-series forecasting.", "essential_references_not_discussed": "The literature review on Bayesian quadrature is relatively limited to fundamental works, which makes sense given that this paper’s approach differs significantly from recent advancements in Bayesian quadrature that rely on functional priors, such as Gaussian processes and kernel methods. Therefore, I have no concerns regarding the references.", "other_strengths_and_weaknesses": "Strengths:\n1. I like the novel attempt to leverage Bayesian quadrature to reformulate conformal prediction. This perspective is interesting and bridges two distinct fields.\n2. Applying a Bayesian approach to random quantile spacings is a clever idea, particularly in the context of integrating monotonic integrand functions like quantiles and cumulative density functions. Traditional kernel-based methods often struggle to enforce monotonicity from a functional prior, typically requiring crude approximations such as warped Gaussian processes. This method provides an interesting alternative.\n3. The results appear promising. The improved risk violation and sample efficiency suggest a compelling direction for further research.\n\nWeakness:\n1. The Bayesian interpretation is somewhat unclear. I suspect this is because the paper is primarily written for the conformal prediction community, but some parts give the impression of denying Bayesian principles. For instance, Section 4.3 mentions the \"elimination of the prior distribution,\" which, strictly speaking, contradicts the Bayesian viewpoint, where priors are valuable for incorporating contextual information beyond the observed data. I would appreciate a clearer explanation of the role of the prior in this approach. If the authors are referring to an uninformative prior, a prior that is robust against variation, or a weighted discrete distribution as the prior, it would be helpful to state this explicitly.\n2. Figure 1 could be misleading. It depicts the typical Bayesian quadrature procedure, where a functional prior is placed over the quantile function $F^{-1}$, assuming smooth monotonicity. However, if I understand correctly, the authors do not ultimately place a function space prior directly. Instead, they propose placing a Dirichlet prior on quantile spacings and using a Heaviside step function (akin to an empirical CDF) as a deterministic function, then this forms the non-parametric functional prior. The resulting posterior over varying spacings then induces a smooth posterior over the integral in the expected values of $L^{+}$. Is this interpretation correct? Clarifying this aspect would improve the paper’s clarity.", "other_comments_or_suggestions": "I did not find any typos.", "questions_for_authors": "See weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Not applicable"}}, {"review_id": "QktEnquZc4", "reviewer": "Reviewer_dgCV", "summary": "This paper proposes a new Bayesian interpretation of conformal prediction, which recovers standard conformal prediction as its mean and provides additional finite sample uncertainty estimate. Interestingly, as a Bayesian algorithm, the uncertainty comes from both 'lack of precise input locations' and finite (insufficient) number of observations. Another nice thing is that by taking the supremum of all possible priors, the practitioners do not need to worry about the choice of prior, which is usually a huge headache in practice. In the experiments, the advantage of the proposed method is clearly demonstrated.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for writing a thoughtful and detailed review. We are happy that the review has recognized that the choice of prior can be a headache in practical problems and how our method circumvents this. We also thank the reviewer for reading our proofs. We are glad that the reviewer found them to be enlightening.\n\n**Conformal Risk Control vs. Split Conformal Prediction.** We thank the reviewer for raising the point about the experiments being run on Conformal Risk Control rather than split conformal prediction. The main reason that the experiments focus on Conformal Risk Control is pragmatic in nature. CRC recovers split conformal prediction when the miscoverage loss is used (i.e. loss = 1 if prediction set/interval does not cover ground truth and loss = 0 otherwise), but allows more general loss functions to be considered. Therefore, in order to investigate more complex loss functions, the experiments focus on CRC. However, please see our next point below which includes results on Split Conformal Prediction.\n\n**Additional Experiments.** During the rebuttal period we ran additional experiments in a more traditional prediction interval setting with heteroskedastic data. In each of the 10,000 random trials we use 200 calibration samples. We let $X \\sim U[0, 4]$ and $Y|X \\sim \\mathcal{N}(0, X^2)$. Prediction intervals are formed as $[-\\hat{\\lambda}, \\hat{\\lambda}]$ where $\\hat{\\lambda}$ is to be selected by each method. The loss is the miscoverage loss and the target loss is set to 0.1 (i.e. 90% coverage). The maximum allowable risk failure rate is set to 5% (i.e. $\\beta = 0.95$). Please note that since the miscoverage loss is used here, Conformal Risk Control and Split Conformal Prediction coincide.\n\n| Method | Relative Freq. (Failure Rate) | 95% CI | Mean Prediction Interval Length |\n|---|---|---|---|\n| Split Conformal Prediction / CRC | 46.59% | [45.61%, 47.57%] | 7.67 |\n| RCPS | 0.0% | [0.0%, 0.04%] | 13.99 |\n| Ours ($\\beta = 0.95$) | 3.75% | [3.39%, 4.14%] | 9.14 |\n\nThe results indicate that even in the heteroskedastic setting, our method allow more precise control of $\\hat{\\lambda}$. Unlike RCPS, which is overly conservative, ours produces the shortest prediction intervals while not violating the maximum allowable risk failure rate.\n\n**On the nature of conditional coverage.** We thank the reviewer for raising this point. Previous work on conditional guarantees have focused on input-conditional guarantees, where the guarantee is conditioned on $X_{n+1} = x$ for all $x$ in the input domain. Guarantees of this nature have been shown to be generally impossible without stronger distribution assumptions. Our guarantees are perhaps better characterized by the term \"data-conditional guarantee\", where we condition on the set of observed loss values $\\ell_{1:n}$. Our experiments demonstrate the practical benefits of this by achieving decisions that produce smaller prediction sets and intervals while not violating the constraint on maximum allowable failure rate. Our guarantees do not rely on strong distribution assumptions that would be necessary to produce an input-conditional guarantee. The distinction between input-conditional and our data-conditional guarantees is an important one and we will be sure to clarify this in the paper by adding a paragraph discussing this point.\n\n**Notation.** We thank the reviewer for suggesting improvements to the notation. We would be happy to update the notation for clarity. However, unfortunately, the links in the review seem to be pointing to the same paper. If the reviewer would be so kind as to re-post the links in a comment, we would be glad to update our paper accordingly.", "claims_and_evidence": "Yes, the paper is well written.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I have checked all the proofs in the appendix. The proofs are well-written and easy to follow.", "experimental_designs_or_analyses": "Why the authors only include conformal risk control in the experiments and not do anything about split conformal prediction, given that the method works for both settings. I am quite familar with split conformal prediction, but I have to admit I know little about conformal risk control. I am not asking for more experiments, I am curious what make the authors only include experiments on conformal risk control.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Line 376-377. This paper claims that the Bayesian interpretation allows conditional coverage guarantees. I suggest the authors distingush their contributions from the real conditional coverage guarantee, find $\\hat{C}\\left(X_{n+1}\\right)$ such that $\\mathbb{P}\\left(Y_{n+1} \\in \\hat{C}\\left(X_{n+1}\\right) \\mid X_{n+1}=x\\right)=1-\\alpha$. See for example, https://arxiv.org/pdf/2305.12616, https://arxiv.org/abs/1903.04684. From my understanding, the authors' Bayesian interpretation can still only provide marginal coverage guarantees. I request the authors to make this distinction in the paper.", "other_strengths_and_weaknesses": "Strength:\n1. The section 4 makes serveal very interesting observations. 1) The standard expectation can be translated to an integral of the inverse CDF function. Then, even if the input locations are not known, one can deduce that they follow a Dirichlet distribution from a classical result presented as Lemma 4.2. As a result, the dirichilet distribution over the input location quantifies an upper bound on the CDF of the final integral. 2) The posterior mean can be upper bounded by the integral of the 'worst-case' quantile function that is consistent with the observations. I really enjoy reading this section and read the proofs in the appendix. It is very enlightening.\n\nWeakness:\n1. The way Bayesian quadrature is written is not standard in the BQ literature as I am aware of. The authors use the standard notation for regression setting, i.e Equation (15) in https://arxiv.org/pdf/1807.02582, while personally I prefer more the interpolation notation for Bayesian quadrature, see Equation (20) in https://arxiv.org/pdf/1807.02582, as the likelihood function is degenerate.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper develops a Bayesian perspective for conformal prediction (CP) using a Bayesian quadrature approach. It derives two widely used CP methods as special cases, namely, split CP and conformal risk control. One of the main benefits of this perspective is that it allows conditional coverage guarantees. The paper has strong theoretical and empirical results to demonstrate its benefits over frequentist baselines on both synthetic and real data.

All the reviewers' are positive about this paper and rightly so! This is a refreshingly novel paper and I recommend accepting this paper with an oral presentation at the conference.
Decision: Accept (oral)