CBMA: Improving Conformal Prediction through Bayesian Model Averaging
Authors: Pankaj Bhagwat
Linglong Kong
Bei Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: Bayesian framework
Conformal prediction
Model uncertainty
Uncertainty quantification
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We combine Bayesian model averaging with conformal prediction to construct more reliable and optimal prediction intervals.

Abstract: Conformal prediction has emerged as a popular technique for facilitating valid predictive inference across a spectrum of machine learning models, under minimal assumption of exchangeability. Recently, Hoff (2023) showed that full conformal Bayes provides the most efficient prediction sets (smallest by expected volume) among all prediction sets that are valid at the  $(1 - \alpha)$ level if the model is correctly specified. However, a critical issue arises when the Bayesian model itself may be mis-specified, resulting in prediction interval that might be suboptimal, even though it still enjoys the frequentist coverage guarantee. To address this limitation, we propose an innovative solution that combines Bayesian model averaging (BMA) with conformal prediction. This hybrid not only leverages the strengths of Bayesian conformal prediction but also introduces a layer of robustness through model averaging. Theoretically, we prove that the resulting prediction interval will converge to the optimal level of efficiency, if the true model is included among the candidate models. This assurance of optimality, even under potential model uncertainty, provides a significant improvement over existing methods, ensuring more reliable and precise uncertainty quantification.

Published as a conference paper at ICLR 2025
CBMA: IMPROVING CONFORMAL PREDICTION THROUGH BAYESIAN MODEL AVERAGING
Pankaj Bhagwat
Linglong Kong Department of Mathematical and Statistical Sciences, University of Alberta, AB, Canada Lkongeualberta.c
Department of Mathematical and Statistical Sciences, University of Alberta, AB, Canada
@bhagwat@@allerraa..
Bai Jiang
Department of Mathematical and Statistical Sciences, University of Alberta, AB, Canada
beil@ualberta.ca
ABSTRACT
Conformal prediction has emerged as a popular technique for facilitating valid predictive inference across spectrum of machine learning models, under minimal assumption of exchangeability. Recently, Hoff (2023) showed that full conformal Bayes provides the most efficient prediction sets (smallest by expected volume) among all prediction sets that are valid at the (1 a) level if the model is cor- rectly specified. However, a critical issue arises when the Bayesian model itself may be mis-specified, resulting in prediction set that might be suboptimal, even though it still enjoys the frequentist coverage guarantee. To address this limita- tion, propose innovative solution that combines Bayesian model averaging (BMA) with conformal prediction. This hybrid not only leverages the strengths Bayesian conformal prediction but also introduces layer of robustness through model averaging. Theoretically, we prove that the resulting prediction set will converge to the optimal level of efficiency, if the true model is included among the candidate models. This assurance of optimality, even under potential model uncertainty, provides a significant improvement over existing methods, ensuring more reliable and precise uncertainty quantification.
INTRODUCTION
Conformal prediction is a distribution-free uncertainty quantification method that generates predic- tion sets with valid coverage under minimal assumptions of exchangeability (Vovk et al.) 2005). By building on existing machine learning techniques, conformal prediction uses available data establish valid prediction set (X.+1) for new instances Xn+1 with coverage guarantee P(Y.+1 E C. (x(+++)) 2 1 - a. Prediction sets that include the ground truth with high proba- bility are essential for high-stakes applications, such as autonomous vehicles (Bojarski et al. 2016) and clinical settings (Esteva et al., (017).. However, it is also preferable for the prediction sets (x(xx+1) to be efficient i.e. as small as possible, as smaller sets are more informative. Recently, Hoff (2023) investigated the optimal efficiency of full conformal Bayes prediction sets under cor- rectly specified Bayesian model, showing that they yield prediction sets with the minimum expected volume at given target coverage level (1 - -a). In addition, Fong & Holmes (2021) introduced scal- able methods for generating full conformal Bayes prediction sets applicable to any Bayesian model, highlighting the advantages of conformal Bayesian predictions over traditional Bayesian posterior predictive sets, especially in the presence of model misspecification.
However, constructing selecting the correct Bayesian model obtain optimally efficient confor- mal prediction sets is challenging task. Although conformal prediction sets provide finite-sample coverage guarantees for any model, they may fail achieve volume optimality when the underlying

---

Published as a conference paper at ICLR 2025
model is misspecified. This challenge underscores broader limitation of traditional conformal pre- diction methods, which typically rely on a fixed machine learning model to generate prediction sets with predetermined marginal coverage level. Given the vast array of possible predictive models for a given problem, and the fact that conformal methods yield valid sets for all of them, identifying the most appropriate model becomes inherently difficult. This issue of model uncertainty has long been a persistent and underexplored problem in the conformal prediction literature.
Our Contributions: In this paper, we provide innovative solution in the form of Conformal Bayesian model averaging (CBMA) to the challenging issue of constructing efficient conformal prediction sets when there model uncertainty in the Bayesian framework. Our proposed CBMA prediction method seamlessly combines conformity scores of each Bayesian model in order to con- struct single conformal prediction set. This paves the way for incorporating model averaging procedures into conformal prediction framework which is currently lacking in the literature. To the best of our knowledge, our CBMA approach is the first method which combines conformity scores from diverse models to construct valid conformal prediction sets and requires no data splitting (data efficient). Our choice of conformity scores in the form of posterior predictive densities is also opti- mal and leads to the most efficient prediction set when the underlying Bayesian model is true (Hoff) 2023). Our CBMA method can construct full conformal prediction sets given samples for model parameters of each model from their posterior distributions and posterior model probabilities, which can be obtained as the output of BMA. Theoretically, we also prove the optimal efficiency achieved by conformal prediction sets constructed using CBMA method as the sample size increases. Such guarantee of optimal efficiency even under model uncertainty provides an improvement over exist- ing methods. Finally, our method incorporates both data and model uncertainty into the construction of prediction sets which enhances the reliability of the predictions.
2 PRELIMINARIES
We consider a collection of i.i.d. observations Z = ((x,, Y;) where each observation (X,, Y,) E Rdd IR: is a covariate-response pair. We begin by summarizing full conformal predic- tion framework (Vovk et al.] 2005). conformal Bayes (Fong & Holmes 2021) and Bayesian model averaging (BMA) (Raftery et al. 1997).
2.1 FULL CONFORMAL PREDICTION
In the full conformal prediction method, candidate labels yn+1 are included in (X,+1) if the resulting pair (Xa+1, 11+1) is sufficiently similar to the examples in Zim- A conformity score
0, = 0 (Z1:+113 2s),
takes input set of data points Z1m+11 = Z U Zn+1 where Zn+1 = {(Xn+1, Bn+1). and computes how similar the data point Z; is for I,.... n + 1. The key property of any conformity score is that it must be exchangeable in the first argument, meaning the conformity core for Z; remains unchanged under permutation of Z1m+1. We refer a conformity score with this property as valid conformity score. Assuming Zi.+1 is exchangeable, din+1 will also be exchangeable, and its rank will be uniformly distributed among {l...., + 1} (assuming continuous (1:++1). This implies that the rank of on+1 is valid p-value. For given value Yn+1 y (with X,+1 known), we can denote the rank of on+1 among 0im+1 as
(v))
Σ 1=1
1(0.5ont) S
n
For miscoverage level a, the full conformal predictive set, Co (X (y
r(v) > a).
ensures the desired frequentist coverage P (Yn+1 E Ca 2 a where P is over Z Zl A formal proof can be found in Vovk et al. (2005), Chapter 8.7 For continuous dim+1. Lei et al. (2018), Theorem I show that the conformal predictive set does not significantly over-cover.
2.2 FULL CONFORMAL BAYES
In the Bayesian prediction, given the model likelihood po (v((z) and prior on parameters, T()) for E TRP, the Bayesian posterior predictive distribution for the response at new instance
2

---

Published as a conference paper at ICLR 2025
X2+1 In+1 conditioned the observed data Zi. represents subjective and coherent beliefs. As noted by Draper (1995) and Fraser (2011). Bayesian predictive sets such as Highest posterior density predictive credible sets poorly calibrated in the frequentist sense. This has consequences for the robustness of such approaches and trust in using Bayesian models to aid decisions. Con- formal inference for calibrating Bayesian models
previously suggested in Vook al. (2005); Wasserman (201I). In Bayesian model, it is natural to consider the posterior predictive density as the conformity score:
p( X, Z1++1)= po (00 d0
This is valid conformity score, o; indeed invariant to the permutation Z1x+1 due to the form of π ( 1m++1) oc (0)
a+1
II (x,). This method is referred to Conformal Bayes in Fong & Holmes (2021).
2.3
BMА: BAYESIAN MODEL AVERAGING
In detail discussion on BMA can be found in the seminal work of Hoeting et al. (1999), Wasserman (2000). Raftery al. 1997). Let X IRd denote the vector consisting of all predictors under consideration and Y E R denote the response variable. In the supervised learning tasks, our goal is construct predictive model for based on via some model M. The Bayesian model M in such scenario can be represented as
~ (.(ul)) 0|M ~ <
where denotes the parameters of the model M, π (0) denotes the prior on the parameters and po(u/æ>) denotes the likelihood function under the model M.
Now suppose, we have several competitive models Mr with parameters 02 E O₂ leading to model space M "MM....,MM''" The Bayesian model for each model is described as:
Y|X, 0x, pe, (ulx): Oa Tx (0x).
If A is quantity of interest, say predicted values at covariate Rd. then the expected value of A given the data Zi. is obtained by first finding the posterior expectation of A under each model, and then weighting each expectation by the posterior probability of the model in the BMA framework:
E A
Σ p(M E(A Ma].
Similarly, the posterior distribution for A can be represented mixture distribution over all mod- els,
p Zi.) = Σ (1.a A
where 71,) Ma) rior probability of model M& is given by
the posterior distribution of A under model M In equation 1 the poste-
m Mc)p(MM)
p(Ma Z1.n)
K
Σ ml (2mm
where p(M&) is the prior probability that M₁ is the true model and MMg) is the marginal likelihood under model Mg.
ml 7 Mp) = I Pe, (Y()).)T+(O))) d0,
==1
obtained by integrating over the prior distributions for model specific parameters. Note that all conditional expressions in this section denotes condition Zt.n ((X,, Yo)
3

---

Published as a conference paper at ICLR 2025
2.4 PROBLEM SETUP
Our objective is to construct conformal prediction set C. (Xn+1) C I that provides range of possible responses for a test point Xn+1 based on a collection of i..d. observations Zim = f(X,, Y,) fi-1- We focus Bayesian prediction based on the training data Zi.-- Suppose we have K competing models for this data, denoted as M,....,My, forming a model space M {M, MMk J. For each model Md with k = ...,..... K, the data likelihood is specified as Po, (yi J.). where Og represents the parameters of model Mи. Let TA (0x) denote the prior distribution the parameters 0g of model Ma-
The key question is: with multiple candidate models in M. how can we combine their predictions create a single conformal prediction set? Additionally, we aim construct prediction sets using transductive conformal approach, rather than relying on data-splitting inductive methods, which are often less efficient in terms of information usage. We propose to average conformity scores from each model to construct combined conformity score. This task presents several challenges including the issue of the same dataset being utilized to train individual models, the conformity scores are not independent. Also, note that, we need to combine conformity scores in a fashion that assigns higher weights to the conformity scores of the true model, it is included in the model space. Thus, weights used for averaging should be data dependent.
2.5 RELATED WORKS
Conformal Bayes has been topic of research interest for long time. But some recent advances in terms of optimal efficiency of such prediction sets and scalable methods for the construction of full conformal sets are noteworthy. Hoff (202) demonstrated that the log predictive densities are the optimal conformity measure to construct conformal prediction sets (minimum expected volume) when the underlying Bayesian model is true. |Fong& & Holmes (202T) recently provided scalable Full conformal Bayes algorithm to construct Bayes conformal set efficiently. They also explored the benefits of conformal Bayes prediction over Bayesian predictive sets when the model is misspec- ified. But, the conformal prediction set they construct may be inefficient. We address this issue by incorporating model averaging technique of Bayesian model averaging (BMA) into the conformal framework. B.A has been proposed as natural Bayesian solution to account for model uncertainty (Hoeting et al. 1999; Wasserman] 2000; Raftery et al., 1997). Some recent works proposing model aggregation strategies to combine conformal prediction sets p-values include Gasparin & Ram- das (2024a), Gasparin & Ramdas (2024b), and Yang & Kuchibhotla (2024). Gasparin & Ramdas (2024b) proposed conformal set aggregation by majority vote strategy, and proved that a merged set suffers a loss in coverage i.e. merged conformal set will have at least 1 - 2a marginal coverage if the individual sets have a marginal coverage. Furthermore, size of such merged sets is shown be no larger than the maximum size among the given sets. However, they do not show if the merged set will achieve optimal efficiency under certain conditions. Yang & Kuchibhotla (2024) provide efficiency first conformal prediction method but this approach suffers from coverage loss in the aggregation step. Many existing works are available for aggregating inductive conformal predic- tors which require data splitting (Vovk 2015 Linusson et al. 2017| Carlsson et al. 2014 Toccaceli & Gammerman 2019) and are based on combining p-values which may not be well-calibrated. Linusson al. 2020) proposed the use of out-of-bag ensemble conformal predictors. But, their theoretical validity of coverage is only approximate and evaluated solely based on empirical studies. Few methods attempt to merge conformal scores from various models under the assumption that distinct data are employed to train each model (Gauraha & Spjuth 202T), thereby constructing con- formal sets accordingly. Overall, common limitations of existing methods include all or some of the following: the requirement data splitting or access to hold-out set for calibration, which may not be feasible for small sample sizes, the lack of coverage the aggregation step, not targeted to achieve efficient prediction sets (instead focus on efficient set aggregation which may not lead to narrower prediction sets), and no thcoretical guarantees of coverage and efficiency. To overcome these pitfalls of the existing approaches, we propose CBMA framework which (1) provide theoretical guarantee of efficiency of conformal prediction sets under model uncertainty, (2) allows the use of full data for all model fits and computing conformity scores for each model as well as aggregation weights are data and prior dependent, which are computed using the same data. Thus, there is no efficiency loss by data splitting at any step, and (3) no coverage loss in the aggregation process.
4

---

Published as a conference paper at ICLR 2025
3 CBMA: CONFORMAL BAYESIAN MODEL AVERAGING
In this section, we describe our CBMA approach to construct conformal predictive sets. For cach model Mr in the model space M [M1....,MM. ). consider Bayesian prediction using train- ing data Zi: for an outcome of interest Y; E R and covariates X; E R Given a model likelihood Pa, (y4 22;) and prior parameters, (0x) for 0g 6x the posterior predictive distribution for the response at new X In+1 takes on the form
PM₁ 111, Zi.)) = / Pox (y(Tn+1)) ) (0x) Z1m) d0x;
where TA (0%) Zi:n) is the Bayesian posterior and Tu (02 (1:)) x mh (0x) II Po, (vi m.))- We first fit
K candidate models using BMA. Following Holmes 2021 full conformal Bayes prediction sets are constructed using posterior predictive densities as conformity measures:
M.M Р.₁ (Y, Z1:+1)
This is a valid conformity score, oiMe is indeed invariant to the permutation of 1 due to the form of π x00 Zi.)). For each model Mr. we employ the Add-One-In In importance sampling strategy described by Fong & Holmes (2021) once obtain posterior samples 01-7 Tu (0x for large T using MCMC. Specifically, for Yn-1 =y and 0 (T 11:),,
compute:
T
a = PMM, XV 71
Σ
the
(1) Po ( X,),
(2)
t=1
where weights vig (4) are of the form
(1)
W = Poto) (yl xn41),
Wk
(t)
w/
(t)
wik
We can also compute posterior model probabilities (MMM) ( based on the observations Z1m so that we don't need refit the models.
Finally, we aggregate individual model conformity scores o; Mr to construct weighted combination of CBMA defined as:
K
CBMA
Σ (1: Zn+
(3)
where
p(M Zin))na (y/1n111 Zi.n)
qu (u 1n+1)
(4)
K Σ 11 (a (g/ Zi:n)
The aggregation weights are proportional to the posterior probabilities of the models and predictive posterior density of new test instance (y. xx)+ conditioned on the observed data Z Note that, we can easily obtain both the terms using posterior samples for parameters based on Z and BMA also provides posterior model probabilities the output. Now combined score a CBMA can be computed as
BlM 1Z...
T
M.
CBMA
(5)
Note that because of the data dependent weights gZ Zn+1) in the construction of a CBMA it is not clear is a valid conformity score in order to construct conformal prediction set. In Lemma we show that of CBMA is indeed conformity measure satisfying the exchangeability criterion. Then, we construct full conformal prediction sets with coverage guarantee. We call this method as Conformal Bayesian model averaging (CBMA). We summarize method in Algorithm

---

Published as a conference paper at ICLR 2025
Algorithm CBMA: Conformal Bayesian Model averaging
Input: Observed data Zimi test point Xx+1i Specity miscoverage level ; for every model M₁ E M, Model likelihood pe, (y]z) and prior (4.) Output: Return C.M.A. (Xx+1), Individual sets C. (Xn+1)
(Run the usual BMA)
for k E 1 K do
Obtain posterior samples O Th (z through MCMC Compute (0M (1m)
end for
for y E YGrid do
Compute a Me and Mr using
Store the rank, (3) of a Mr
R+11 among o 11+
Compute Iin CBMA and o n+1
CBMAA using (3) and (4)
Store the rank, (BMA(B) of CBMA among 1:n
BMA
n+1
end for
Return: Set Ca CCBMA(Xx+a) = Ey E Yoria TCBMA(U) > a); Individual Sets C. (Xn+1) {y E Yorid ry( a}
4 THEORETICAL PROPERTIES
In the Lemma we show that aggregated score of CCBMA in indeed valid conformity score.
Our proof of Lemma I relies alternative representation for oCBMA which can be interpreted as the posterior predictive density under the hierarchical Bayesian model:
Y| 8g ~ po. (ulx) Og|Ma ~ (#x)) MR ~ p(MM).
(6)
Lemma The aggregated score of CBMA can be rewritten as
K
K Σ D(MMllZi+tll
CBMA
Σ P(MulZin 1PM (Y]),,, Zi
0
This is the posterior predictive density under the hierarchical model and hence, is valid confor- mity measure. Here p(M (1:44)) 1) is the posterior model probability of My conditional on Z
Proof: We provide the details of obtaining alternative representation in 7 for aggregated scores given in (3) in Appendix 1. Hierarchical model (6) is fully Bayesian with two kinds of parameters: the parameters 04 of each of the model Mr and the models M2 's themselves. Thus, note that for i.i.d. observations Zi:+1 {(Y,, X,) n=+1 from the hierarchical model (6) are exchangeable. Also,
the conformity measure considered the posterior predictive density function, which is invariant
to the permutation of Z1.+t1. This follows by observing that the likelihood under each model is
n+1
II Pe, (Y.)X4) is invariant under the permutations of Zim+1- Thus, this is valid conformity measure.
Once have established of CCBMA valid conformity measure, define the conformal prediction set as
nil CBMA <
CBMA) nt+1
TCBMA(V)
C CBMA (X :ere:))) > a). (8)
n n+1
Theorem Assume that Zu = {(Y,, x+) are exchangeable, and the conformity measure a CBMA as in is invariant to the permutation of Z1 We then have
P (Yat1 E CCBM (TBMA(Xn+1)) (Xnt1))

---

Published as a conference paper at ICLR 2025
where, Ca(Xn+1) is defined in (8) and P is over Zmm+1- Moreover, the conformity measures {or (oCHMA A1 are almost surety distinct, then we have
Ji=1
P PYn+1 E Ca (CBMA(Xn-)) 1
1
n ++1
Our next result demonstrates the convergence of CBMA prediction sets the conformal prediction sets based on the true model as the sample size increases, if the true model is included in the model space M The intuition behind this result grounded in the established observation that BMA approaches the true model as the sample size n. becomes large, provided the true model is within the considered model space (Le & Clarke (022). This convergence ensures that the CBMA prediction sets become increasingly accurate and reliable, mirroring the properties of conformal prediction sets derived from the true model. The detailed proof of this result is provided in the Appendix A.2
Theorem Suppose true model Mirue is in the model space M = (MM Under the hypoth- esis of Theorem in Le & Clarke 20222, in probability when Mr = Merue and qI- (i(( (71m; Zm+1) + 0 in probability when MR & -Mruue-
+ 00, under Merues we have qu (Zi. Zu+1) -
Thus, for any i, the conformity score of CCBA converges in probability to conformity score o Mecuo under true model. Furthermore, let VCBMa(Xneve Zin) denote the volume of the CBMA prediction set and Vine [ Neww Zin1) denote the volume of the optimal conformal prediction set under the true model Minc Then, 1im
|EVeRma neww Zi Ine (innexXewww Zi =0.
Remark Theorem implies that, as the sample size n becomes large, our CBMA prediction set will be similar conformal Bayes prediction based true model. Thus, the expected volume of the CBMA prediction set converges to the expected volume of the conformal Bayes prediction set based on the true model. It is important to note that the expected volume the conformal prediction set under the true model is optimal in terms of minimum expected volume (Hoff. 2023). Therefore, grows, our CBMA prediction set becomes as efficient the optimal conformal prediction set derived from the true model. This convergence ensures that the CBMA prediction set not only aligns with the true model but also maintains efficiency, making it powerful tool for practical applications where model uncertainty is present.
Remark 2 In the case of small sample sizes, inferring the true model the data generation process challenging and often impossible. In such scenarios, our method can enhance the performance of predictions by leveraging the improved predictive power of BMA. Note that BMA has been shown to outperform individual models in terms of log scores (Raftery et al. 1997.
Remark When the true model not in the model space, we note that qo 11,, Zn+1) + 1 in probability as + for My- such that DKL (P.Merelllpp..
inf DEL (p (peeraa Ilp.)). ) k=l.....K
where (-1(- denotes the Kullback-Leible (KL) divergence. This follows from general posterior concentration results (Whitel 1982 Berk 1966 Ramamoorthi et al. (0115). Thus, if model space is large enough to contain a model in the KL divergence neighborhood of the true model, our CBMA approach will construct near-optimal conformal prediction sets.
SIMULATION STUDY
We evaluate the performance of our conformal Bayesian model averaging prediction method in two experimental settings: (i) when true model is a part of the model space, and (ii) when true model is not included in the model space. In the former case, our numerical results show that the performance of our methodology is similar to that of conformal Bayes and Bayes predictive sets based on true model. This accordance with the results obtained bynFong Holmes (202T) and (2023) but our method also incorporates model uncertainty adding an extra layer of robustness to the predictive inference and elevate reliability. In the second scenario, we note that our methodology leads to construction of valid prediction sets with smaller average size as compared to conformal Bayes prediction sets and Bayesian sets for individual models. This demonstrates the advantage offered by proposed method over conformal Bayes by Fong & Holmes 202II. The details of our simulation experiments are given in the Appendix A All codes and results are provided in the supplementary material.
7

---

Published as a conference paper at ICLR 2025
5.1 QUADRATIC MODEL
We draw independent and identically distributed (i.i.d.) samples I; ~ Unif(O, 1), for i = 1,..., The response y is then generated through following model:
y4 = 90 + 31 + 824 +
3o M (0, 0.25) 81 N (1, 0.25) 32 ~ (0.5,0.25)
(0,
We consider three models: Model 1 (M1) represents the true model, Model (M₂ y/ 3g + S1 + 6,) is linear model derived by omitting the quadratic term from Model 1, and Model 3 (M3 y/ 30 2207 + a) includes only an intercept and a quadratic term. In this experiment, set the miscoverage level a 0.20. We use 40% of the total sample size as the test set with the remaining 60% forming the training set. The summary the results of the different prediction sets is given in Table (for sample sizes 100 and 200), provided in A.4.1
We observe that, all conformal prediction sets (which are intervals here) achieves the target cover- age in all cases. Also, observed by Fong & Holmes (202I). conformal Bayes prediction sets have shorter lengths as compared to their corresponding Bayesian prediction sets, for both true and mis- specified models. Our proposed CBMA method provides prediction sets which have similar average lengths as that of conformal Bayes based on true model. Note that CBMA also incorporates model uncertainty opposed to conformal Bayes, hence, leads reliable predictive inference and pro- vides more precise uncertainty quantification. Furthermore, as shown Figure [ CBMA achieves optimal mean length, equal CB under model M₁, even at smaller sample sizes 100, 200, demonstrating practical utility of our CBMA approach.
Methods
CBMA
CBM1
0.64
CBM2
Mean Lengths Intervals
CBM3
0.62
0.60
0.58
0.54
0.52
0.50
50
100
200
Sample Size
Figure 1: Quadratic model: Comparison of CBMA and CBMI for different sample sizes.
5.2 APPROXIMATION USING HERMITE POLYNOMIALS
In this section, we consider data generating mechanisms similar to Lu & Su 2015). Let X, ~ Weibull(1, 1) i..d. The data is generated from the following model
exp(Xx}
Y = 0 1+ exp +
Here, consider e; (0.01 + X,)4 and &, -N(0, 1). This is scenario of heteroskedasticity: In this study, we employ Hermite polynomials to approximate the unknown function exp(X,) We build model space consisting of K models with the covariates X = 1,2,
"+exp[Xxa)'
(X; x)2
Xij = (X; Xy-1 exp
. = 1,2,
28x
8

---

Published as a conference paper at ICLR 2025
where X and x represent the sample mean and standard deviation of the set X,, respectively. In our simulation experiment, we set a = 0.8. We use 40% of the total sample size as the test set, with the remaining 60% forming the training set (Ntrain = 0.60 x n). The number of models considered = 11. We consider nested model sequence, i.e. for = 1,.... K, model Mr includes results based 100 in Figure lnd Figure 5 (in A.4.2). The results for other studies for n = 100, = 10 and 50, 0 method provides sets (here, prediction sets are in the form of intervals) for which average length is smaller than all other prediction sets. Also, note that, most of Bayes prediction sets overcover and hence have larger sizes. Conformal Bayes well as our CBMA prediction sets have valid target coverage. But, on average, CBMA prediction sets have shorter length than prediction sets. These results demonstrates the empirical efficiency of CBMA prediction sets over CB prediction sets.
covariates X,1 X,1,-.. Xik- We summarize
F. 50 experiments for samples size
reported Table and Table 6 in Appendix [A.4.2) We note that our CBMA
16-
Bayes
14-
CBMA
Mean lengeh of intervals
224
20.
A4
36
16
M6
BMA
CBMA
Models
Figure 2: Approximation using Hermite polynomials: Comparison of mean length for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (in red), and individual conformal Bayes (CB) prediction sets (in blue). Here, we report results based on E 50 number of experiments. We have set the target coverage (1-a) = 0.80, sample size n 100 and
5.3 REAL DATA EXAMPLE: CALIFORNIA HOUSING DATA
We demonstrate our method using the California Housing dataset. The dataset is available in sklearn, and consists of n 20640 subjects, where the continuous response variable denotes the median house value for California districts, expressed in hundreds of thousands of dollars (100, 000), and covariates, including median income and median house age in block group, average number of rooms per household, block group population. This dataset was derived from the 1990 U.S. census. We first draw random samples of sizes n 50, 100 or 150 from this dataset for our study. We then standardize all covariates and the response to have mean 0 and standard deviation 1. We consider four different models for these experiments by considering only two covariates in the model. Thus, the models we consider
Mr y= 81 + 81 +c, k 1,2,3,4.
The priors we consider on the parameters are 3; ~ N(O. 5), N4 10, half-normal distribution. Among all four models, Mj seems more appropriate as contains important covariates median income and median house age in block group, opposed to other covariates. We fit these four models to obtain MCMC samples from the posterior distribution and compute marginal likelihoods for each model. To check coverage, repeatedly divide our dataset into training and test dataset for 50 repeats, with 40% the dataset in the test split. We adopt the setup for simulations simi- lar to Fong & Holmes (2021) in their section 4.1, that we can compare the performance of

---

Published as a conference paper at ICLR 2025
CBMA method with individual conformal Bayes. We set the miscoverage level 0.2. To illustrate the time complexity, we note that, for n = 150, the average times for constructing con- formal Bayes prediction sets 0.806(0.004), 0.793 0.002), 0.786(0.002) and 0.794 for four models, Bayes credible sets took average times of 0.447 (0.003)))) 0.444(0.002), 0.441 (0.000)) and 0.449(0.000k Further, construct Bayes credible sets using BMA, average additional time was (0.90()0005) and our CBMA method, additional average time was 0.128 (0.007). We report average lengths and coverages of various prediction sets (prediction intervals this case) in Figure 3 and We note that, as suggested earlier, model MM fits data well as compared to other three models, we expect have conformal Bayes prediction sets based on this model will be the shortest among all four models. As indicated by our Theorem 2
and Remark our CBMA intervals achieves shortest lengths as the sample size becomes sufficiently large enough. In our results, we can see CBMA achieves shortest lengths even at finite sample size of n 150. All conformal Bayes ap- proaches as well as CBMA achieves, on average, the target coverage of a = 0.80. On the other hand, Bayes models over cover in almost all cases.
Samdl
Sarple
Sanale Size
20
CPH1
kom
May
Jea
E
Figure 3: California Housing data: comparison of mean lengths of intervals using Bayes prediction, conformal Bayes (CB) for all four models, CBMA and BMA for different sample sizes.
CONCLUSIONS
In this work, we identified a challenging issue with the traditional conformal prediction framework: selecting model advance in order construct conformal prediction sets may result in suboptimal prediction sets. In addressing this issue of model uncertainty within the conformal prediction frame- work, we offer natural Bayesian solution by integrating Bayesian model averaging in conformal prediction framework. Our proposed CBMA framework seamlessly amalgamates conformity scores from individual Bayesian models construct unified conformal prediction set, thus introducing the incorporation of model averaging procedures- a notable advancement currently overlooked from current literature on conformal prediction theory. Additionally, we present an efficient algorithm for constructing CBMA prediction sets, which essentially represent full conformal predictions. Theo- retically, we establish the optimal efficiency of our CBMA prediction sets as sample sizes increase, underpinning the efficacy of our approach. Overall, method holds particular value in real- weal-world scenarios where model uncertainty is a prevalent concern, providing a robust solution within the conformal prediction framework.
However, our approach does have limitations due to the inherent characteristics of MCMC algo- rithms. The error these sampling methods can affect prediction sets constructed using CBMA. As noted in Fong & Holmes (2021), we can also expect some robustness to conformal sets as they are based ranks and not actual values. Furthermore, since we expect that the errors in one of the models' training can be masked on averaging.
averaging over multiple models,
ACKNOWLEDGMENTS
Bei Jiang and Linglong Kong were partially supported by grants from the Canada CIFAR AI Chairs program, the Alberta Machine Intelligence Institute (AMII), and Natural Sciences and Engineering Council of Canada (NSERC), and Linglong Kong also partially supported by grants from the
10

---

Published as a conference paper at ICLR 2025
Canada Research Chair program from NSERC. The authors would also like to thank the anonymous reviewers for their constructive comments that improved the quality of this article.
REFERENCES
Robert H Berk. Limiting behavior of posterior distributions when the model is incorrect. The Annals of Mathematical Statistics, 37(1):51-58, 1966.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.07316 2016.
Lars Carlsson, Martin Eklund, and Ulf Norinder. Aggregated conformal prediction. In Artificial Intelligence Applications and Innovations: AIAI 2014 Workshops: CoPA, MHDW, IIVC, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedingy 10, pp. 231-240. Springer, 2014.
David Draper. Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society Series B: Statistical Methodology, 57(1):45-70, 1995.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-evvel classification of skin cancer with deep neural networks. nature, 542(7639): 2017.
Edwin Fong and Chris C Holmes. Conformal bayesian computation. Advances in Neural Informa- tion Processing Systems, 34: 1466-888-1,,79 2021.
D. A. S. Fraser. Is bayes posterior just quick and dirty confidence? Statistical Science, 26(3):299- 316, 2011, doi: 10.1214/1-STT3352.
Matteo Gasparin and Aaditya Ramdas. Conformal online model aggregation. arXiv preprint arXiv:2403. 15527, 2024a.
Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote. arXiv preprint MXXiv:2401.09339 2024b.
Niharika Gauraha and Ola Spjuth. Synergy conformal prediction. In Conformal and Probabilistic Prediction and Applications, pp. 91-110. PMLR, 2021.
Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris Volinsky. Bayesian model averaging: tutorial. Statistical science, 14(4):382 417, 1999.
Peter Hoff. Bayes-optimal prediction with frequentist coverage control. Bernoulli, 29(2):901-928, 2023.
Tri M Le and Bertrand S Clarke. Model averaging is asymptotically better than model selection for prediction. Journal of Machine Learning Research, 23(33): 2022.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 (523):1094-1111 2018.
H. Linusson, U. Johansson, and H. Boström. Efficient conformal predictor ensembles. Neu- rocomputing, 397:266-276 278, 2020. ISSN 0925-2312. doi: http://aoi.ogg///..006/.mmoommm 2019.07.113. URL
5092523121931611
Henrik Linusson, Ulf Norinder, Henrik Boström, Ulf Johansson, and Tuve Löfström. On the cali- bration of aggregated conformal predictors. In Proceedings of the Sixth Workshop on Conformal and Probabilistic Prediction and Applications, volume 60 of Proceedings of Machine Learning Research, pp. 154-173. PMLR, 13-16 Jun 2017.
Xun Lu and Liangjun Su. Jackknife model averaging for quantile regressions. Journal of Econo- metrics, (88(1)40-58 2015.
11

---

Published as a conference paper at ICLR 2025
Adrian E Raftery, David Madigan, and Jennifer A Hoeting. Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 92(457):179-19) 191, 1997.
RV Ramamoorthi, Karthik Sriram, and Ryan Martin. On posterior concentration in misspecified models. Bayesian Analysis, 10(4):759-789, 2015.
Paolo Toccaceli and Alexander Gammerman, Combination of inductive mondrian conformal pre- dictors. Machine Learning, 108:489-510, 2019.
Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74: 9-28, 2015.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
Larry Wasserman. Bayesian model selection and model averaging. Journal of mathematical psy- chology, 44(1):92-107, 2000.
Larry Wasserman. Frasian inference. Statistical Science, 26(3):322-3255 2011. ISSN 08834237.
Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pp. 1-25, 1982.
Yachong Yang and Arun Kumar Kuchibhotla. Selection and aggregation of conformal prediction sets. Journal of the American Statistical Association, pp. 1-13, 2024. doi: 10.1080/01621459, 2024.2344700.
12

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 PROOF OF LEMMA
Representation for the conformity measure oCBM. A in 7 Under hierarchical Bayesian model in 6 the posterior predictive density for new text point (y. In+1) can be obtained follows:
X Zim+1)
Zim+1lpm (Y] Z1:+1)
ml m(Zintil-MaDbe PM (Yil Z1m+1)
ml m(1m++|MM)D(MMs
K
C
ml MMe (Y: Zim+1)
m 11 M(Zmm+1lMa)p(MM
K
p pMM m( [(
m ( (Yi) Zim41)
21.m Mr
p(MR) ml m(Z
m(Z mo 24
Σ Maaaa (Yi Zimti)
Σ 2m.
K
Σ p(Ma 1,m, bnd Zim+1)
pMM 7nn)
A.2 PROOF OF THEOREM 2
Theorem Le & Clarke (022) establishes conditions under which the posterior model probabil- ities converge to their true values. Below, summarize key assumptions necessary for proving the theorem, which are also detailed in Le & Clarke 2002).
1. For each & = 1,..., K, let 0g O₁,, open subset of Rd
2. The densities po, (x)) are measurable y for all 0g. x and continuous 6g. All second partial derivatives of Pe, (yalæ;) w.r.t. On exist and are continuous for all ær, y, and maybe passed under the integral sign in po, (y: (r; Jdy
3. There exists a function K-)) such that Eg, K(Y) < oo and each element of the matrix 0² 00e,1001,5 logPo, Pos (3) 1z;)) Li==,...dd, is absolute value by K(y) uniformly for 0x in an open
neighbourhood of 0z.0.
4. Assume that, for any Zi.
82 [;(0v.l):;)= = (-EoL. 00
log po, (walz:))
15=1....d
is continuous open set around the true value 0R.0 and is positive definite at 0R.0-
5. For any Ōk.0 O and any Hi, po, (ye (x:)) Po.,o (ve():)) a.e. in y implies 0z- 0
13

---

Published as a conference paper at ICLR 2025
6. The average (1/n) Σ (0k,o (24) is invertible for any and the dy vector v (O (0,,0l (Ve, log Pes.o (3i (z:)) satisfies
I((0kolzz)) 11/2
Σ E n
Σ
1/3/22
7. Priors Tx (0x) continuous for 0y Ok.
Under these conditions, the posterior model probabilities satisfy
pMM (1:++) 1) 1 or 0 in probability as n oc,
when M₁ corresponds the true model Muse or an incorrect model, respectively. This result directly implies
0. CBMA + Muse in probability, as n + oc.
Consequently, the conformal Bayesian model averaging (CBMA) prediction set indicator satisfies
1yy E CCBMA (X + 1 E (Xn+1)) in probability as n + co.
Thus, the CBMA algorithm (Algorithm 1 produces prediction sets that asymptotically converge to the conformal Bayes prediction set C.Mtrie (Xn+1) based on the true model Mine.
A.3 EXPERIMENTS
A.4 COMPLEXITY OF CONSTRUCTING CBMA INTERVALS
Given the posterior samples and posterior model probabilities, obtaining individual models' confor- mity scores has a complexity of O(Egria x posterior samples X Mumin)) (Fong Holmes 2021). The aggregation step (requires dot product of K weights and K conformity scores) has linear overhead of OKK), where K is the number of models.
A.4.1 QUADRATIC MODEL
We now examine the experimental results for the quadratic model, discussed in Section Figure 4 presents the posterior model probabilities (PMP) for all three models under consideration across E 50 repetitions with sample size of n 100. The results indicate that Model frequently attains the highest posterior model probability. However, Model 2 also exhibits substantial number of instances where achieves higher PMP. This variability suggests that pre-selecting either Model or Model 2 for constructing conformal prediction sets may lead to suboptimal uncertainty quantifi- cation. contrast, our CBMA method leverages predictions from multiple models, ensuring a more comprehensive integration of model uncertainty. This leads to more efficient and reliable uncertainty quantification. Additionally, Table 2 presents descriptive statistics from these repeated experiments. Notably, CBMA prediction set lengths exhibit symmetric distribution. whereas CB.MI prediction sets display a negatively skewed length distribution. This suggests that CBMA results smaller prediction sets in many instances compared CB.M1, enhancing efficiency without compromising coverage.
EXECUTION TIMES:
Given the posterior samples obtained from MCMC, we evaluate the computational efficiency of constructing Conformal Bayes (CB) prediction sets and Bayes prediction sets. Table reports the mean execution times (with standard emrors parentheses) for constructing these intervals across different models and sample sizes. Additionally, Table the execution times for CBMA and B.A methods. Notably, CBMA incurs higher computational costs compared to BMA, reflecting the additional complexity introduced by model averaging in the conformal framework. However, these times remain negligible relative to the broader MCMC sampling process, making CBMA computationally feasible approach for uncertainty quantification.
14

---

Published as a conference paper at ICLR 2025
VEWW
Posterfor mode: probabilies
Series
Model
Model
Experiment No.
Figure 4: Posterior model probabilities (PMP) for all three models considered in our experiment with quadratic model with sample size 100.
Table 1: Quadratic model: Comparison coverage and length for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets BayesM1, BayesM2, BayesMM, individual conformal Bayes prediction sets CBM1, CBM2, CBM3. Here, we report mean and standard error (SE) based on F 50 number of experiments. We have set the target coverage as (1 - a) = 0.80.
Method
CBMA
BMA
BayesMI
BayesM2
BayesM3
CBMI
CBM2
CBM3
Coverage
n = 100
Mean SE
0.800
0.811
0.813
0.803
0.804
0.801
0.799
0.804
0.010
0.010
0.010
0.010
0.009
0.010
0.011
0.010
= 200
Mean
0.807
0.811
0.813
0.813
0.805
0.812
0.815
0.805
SE
0.007
0.007
0.007
0.007
0.007
0.007
0.007
0.008
Length
100 Mean SE
0.535
0.537
0.537
0.545
0.579
0.534
0.547
0.577
0.008
0.007
0.007
0.007
0.009
0.008
0.009
0.010
200
Mean
0,522
0.522
0.522
0.532
0.558
0.524
0,536
0,559
SE
0.006
0.004
0.004
0.005
0.006
0.006
0.007
0.007
A.4.2
APPROXIMATION USING HERMITE POLYNOMIAL.S
The results of studies for n = 100, 50, 10 and n = 50, 0 are reported in Table and Table We note that CBMA sets have shorter lengths average in all these experiments.
15

---

Published as a conference paper at ICLR 2025
Table 2: Descriptive statistics for average lengths of prediction obtained with different methods: CBMA, BMA, individual Bayes prediction sets Bayes MI, Bayes M2, Bayes M3, individual conformal Bayes prediction sets CB MI, CB M2, CB M3. Here, we report Median, Mean, Standard error (SE), Lower and upper quartiles (Q: and Qs), Inter- suartilae range (IQR) based on T 50 number of experiments. We have set a 0.1.
Prediction
CBMA
BMA
Bayes M1
Bayes M2
Bayes M3
CB M1
CB M2
CB M3
100 Median
0.684
0.697
0.695
0.715
0.739
0.702
0.688
0.736
Q1
0.64
0.651
0.652
0.666
0.706
0.643
0.64
0.666
Q3
0.736
0.738
0.74
0.756
0.788
0.735
0.755
0.782
IQR
0.096
0.087
0.088
0.09
0.082
0.092
0.115
0.116
Mean
0.686
0.697
0.696
0.709
0.745
0.687
0.696
0.732
SE
0.01
0.008
0.008
0.009
0.008
0.009
0.01
0.011
n ==
200
Median
0.676
0.676
0.675
0.69
0.706
0.671
0.681
0.709
Q1
0.643
0.644
0.647
0.664
0.683
0.64
0.646
0.681
Q3
0
0.699
0.699
0.711
0.745
0.711
0.71
0.74
IQR
0.068
0.055
0.052
0.047
0.062
0.071
0.064
0.059
Mean
0.673
0.676
0.675
0.691
0.714
0.672
0.682
0.708
SE
0.007
0.006
0.006
0.006
0.007
0.007
0.006
0.008
Model
100
200
CBMI
11.198 (0.071)
25,470 (0.067)
CBM2
11.631 (0.075)
28.040 (0.222)
CBM3
11.637 (0.140)
25.950 (0.222)
BayesM
4.425 (0.038)
8.380 (0.040)
BayesM2
4.641 (0.037)
9.496 (0. 113)
BayesM3
4.661 (0.061)
8.956 (0.110)
Table Mean times (SE) for CB and Bayes prediction intervals for models (M1, M1,M).
Model
n 100
n 200
CBMA
0.02397 (0.0014346)
0.09902 (0.0039644)) 0,00077 (0.0000435)
BMA
0,00123 (0.0000492)
Table 4: Additional mean times (SE) for CBMA and BMA prediction intervals.
A.5 REAL DATA EXAMPLE: CALIFORNIA HOUSING DATA
Figure presents the mean coverage probabilities for different methods, with a confidence level of a 0.8. The results indicate that Bayes prediction intervals overcover, providing overly con- servative intervals. In contrast, Conformal Bayes (CB) and CBMA consistently achieve the target coverage, ensuring both reliability and efficiency. Additionally, CBMA, by integrating model uncer- tainty, results in more adaptive and stable prediction intervals compared to individual CB models.
Furthermore, we compare our CBMA set aggregation step with majority vote conformal set aggrega- tion method proposed by Gasparin & Ramdas (2024b). While this method is designed to effectively aggregate sets while preserving the coverage guarantee, it does not necessarily aim minimize the size of the merged set. In contrast, our CBMA approach offers the additional advantage of allow- ing the weights the aggregation step to learned directly from the same data, addressing a key limitation of existing model aggregation methods. We conducted direct evaluation by comparing the lengths of PBMA-aggregale intervals to those obtained from the majority vote strategy using individual conformal Bayes prediction sets. For this comparison, we applied Corollary 4.1 from Gasparin Ramdas (2024b). accounting for the dependency among our individual sets. our example, the mean ratios of the lengths of intervals from the majority vote procedure relative
16

---

Published as a conference paper at ICLR 2025
1 1-a=00 0.8
1
Bayes
CB
1
BMA
Mcan coverage intervall
CBMA
2.2-
2.0
178-
2.8
34
H4
Ab
M4
14
341
stc
A
Abo
Afxa
BBA
CBMA
Models
Figure 5: Approximation using Hermite polynomials: Comparison mean coverage for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (in red), and individual conformal Bayes (CB) prediction sets (in blue). Here, we report results based on E 50 number of experiments. We have set the target coverage (1 a1) = 0.80, sample 100 and 1.
size n
1.3
Semo 5
1.00
Samore Size 100
3.90
Surrple
Chil
CMM
Roym
GIM
0.5
0.55
6.75
0.22
Figure 6: California Housing data: comparison of mean coverages of intervals using Bayes predic- tion, conformal Bayes (CB) for all four models, CBMA and BMA for different sample sizes. Here, we set 1 = 0.8.
those from CBMA were observed to be 1.0765 (0.0253), 1.1196 (0.0151), and 1. 1.1458 (0.0133) for sample sizes n 50, n 100, and n 150, respectively. These results shows that our CBMA approach gives shorter intervals compared to majority vote aggregation strategy.
17

---

Published as a conference paper at ICLR 2025
Table 5: Approximation using Hermite polynomials: Comparison of coverage and length for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (CBMI-CBMII). Here, we report mean and standard error (SE) based on E 50 number of experiments. We have set the target coverage as (1 ()) = 0.80 and 0 1.
and individual conformal Bayes prediction sets
Length
Coverage
Method
n=1(0)
n=50
n= 100
n=50
Mean
SE
Mean
SE
Mean
SE
Mean
SE
BayesM1
3.485
0.118
3.574
0.144
0.858
0.011
0.862
0.014
BayesM2
3.492
0.121
3.574
0.146
0.858
0.011
0.854
0.013
BayesM3
3.410
0.118
3.537
0.148
0.858
0.011
0.856
0.013
BayesM4
3.384
0.118
3,461
0.142
0.856
0.01
0.853
0,013
BAyesM5
3.348
0.116
3.389
0.141
0.853
0.01
0.846
0.013
BayesM6
3.333
0.116
3.344
0.139
0.848
0.01
0.845
0.013
BayesM7
3.297
0.115
3.290
0.137
0.844
0.01
0.838
0.014
BayesM8
3.208
0.108
3.240
0.143
0.837
0.011
0.827
0.015
BayesM9
3.138
0.103
3.182
0.141
0.837
0.011
0.828
0.016
BayesMI0
3.087
0,099
3.131
0.141
0.830
0.010
0.829
0,016
B1yesMIl
3.039
0.094
3.067
0.141
0.830
0.010
0.829
0.015
CBMI
2.689
0,097
2.864
0.128
0.801
0.011
0.819
0.017
CBM2
2.693
0.098
2.857
0.131
0.806
0.012
0.811
0.016
CBM3
2.689
0.101
2.883
0.132
0.799
0.012
0.808
0.016
CBM4
2.667
0.094
2.877
0.137
0.798
0.012
0.819
0.015
CBM5
2.679
0.095
2.884
0.136
0.795
0.012
0.816
0.015
CBM6
2.705
0.093
2.867
0.120
0.797
0.012
0.818
0.015
CBM7
2.703
0.095
2.873
0.120
0.796
0.013
0.818
0,015
CBM8
2.691
0.094
2.823
0.121
0.794
0.013
0.818
0.016
CBM9
2.694
0.093
2.763
0.115
0.798
0.012
0.821
0.016
CBM10
2.704
0.094
2.740
0.112
0.799
0.012
0.822
0.015
CBMTT
2.692
0.092
2.753
0.114
0.800
0.012
0.818
0.014
BMA
3.122
0.103
3.130
0.134
0.831
0.011
0.829
0.014
CBMA
2.634
0.091
2.683
0.118
0.792
0.013
0.808
0.015
18

---

Published as a conference paper at ICLR 2025
Table Approximation using Hermite polynomials: Comparison mean Coverage and Length for prediction sets obtained with different methods: CBMA (proposed method), BMA, individual Bayes prediction sets (BayesM1 BayesM individual conformal Bayes pre- diction sets CBMI, ..., CBM11. We have the target coverage a) = 0.8. Here,
10, = 11, 100.
Length
Coverage
Method
Mean
SE
Mean
SE
B1
4.958
0.101
0.861
0.01
B2
3.780
0.121
0.862
0.011
B3
3.634
0.121
0.856
0.011
B4
3.499
0.122
0.855
0.01
B5
3,465
0.121
0.852
0.01
B6
3,432
0.121
0.847
0.01
B7
3.406
0.120
0.843
0.01
B8
3.282
0.
0.835
0.011
B39
3.189
0.103
0.835
0.011
B10
3.141
0.099
0.834
0.011
B11
3.094
0.096
0.831
0.011
CB1
3.989
0.083
0.812
0.011
CB2
2.922
0.091
0.805
0.OTO
CB3
2.726
0.092
0.798
0.011
CB4
2.708
0.094
0.803
0.011
CB5
2.707
0.091
0.799
0.011
CB6
2.703
0.088
0.800
0.011
CB7
2.722
0.090
0.799
0.011
CB8
2.722
0.090
0.800
0.011
CB9
2.716
0.090
0.801
0.011
CB10
2.733
0.098
0.801
0.012
CBII
2.718
0.098
0.800
0.012
BMA
3.163
0.099
0.834
0.010
CBMA
2.683
0.089
0.799
0.011
19


Reviews:
[{"review_id": "aekiA82xsp", "reviewer": "Reviewer_Nttv", "summary": "The authors, working in the niche of bayesian conformal prediction, propose a bayesian model averaging procedure \"infused\" with conformal prediction, in order to recover coverage in the case of model misspecification", "questions": "-As shown in the \"weaknessess\", some proposals (usually in a frequentist setting) are present in the literature when it comes to model averaging/ensembling in a conformal sense. Can you comment on this, maybe providing a more thorough analysis of the outstanding literature?\n\n- I find the cases provided by the authors of limited applicative interest, I wonder if it could be possible to provide more challenging \"misspecification\" cases, like for instance cases where the model misspecification causes the conditional iid assumption to break (e.g. an AR2 model with only one lag).\n\n- The impact in terms of interval length does not seem statistically significant. Are the authors able to check this, using maybe the Scheffe' method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "I found the article quite well written.\nMoreover, the proposal is of clear theoretical and applied interest, and developed with care and mathematical soundness.", "weaknesses": "I fear that the authors have oversold their claim. In fact, in the literature proposals of conformal model averaging can be found  (https://www.sciencedirect.com/science/article/pii/S0925231219316108, https://arxiv.org/abs/2408.06642#:~:text=Unlike%20traditional%20methods%2C%20conformal%20ensembling,%2Dto%2Dinterpret%20uncertainty%20estimates. to name a few).\n\nI found the simulation quite lacking in terms of depth, as well and presentation of the results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnkDVJO1ho", "reviewer": "Reviewer_3y9y", "summary": "The authors highlight that the efficiency of traditional conformal prediction can degrade in the presence of model uncertainty. To address this, they propose Conformal Bayesian Model Averaging (CBMA), which combines multiple non-conformity scores from a Bayesian perspective. This approach integrates Bayesian model averaging with Bayesian conformal prediction (full conformal Bayes; Fong & Holmes, 2021), allowing it to leverage the optimality of Bayesian model averaging from a theoretical standpoint. From a computational perspective, CBMA utilizes the add-one-in importance sampling approach of Fong & Holmes (2021), enabling it to use all available data for model training while avoiding exhaustive computation. The work demonstrates CBMA’s high efficiency through numerical justification in both correctly specified and misspecified model scenarios.", "questions": "1. This question relates to the previously mentioned points. What do you consider the distinguishing features of your approach compared to existing conformal set aggregation methods? Were any experiments conducted to compare them? If so, what were the results, and if not, what are your thoughts on this?\n\n  2. In the simulations, the nominal value of $\\alpha$ was set to 0.1 in the linear model scenario, but 0.2 in the nonlinear case. Is there a particular reason for these distinct choices? Also, did you observe any systematic performance changes in the model based on the value of $\\alpha$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "While several works in the field of conformal prediction have applied a Bayesian perspective, this paper is meaningful since it is the first to attempt combining “multiple models” through Bayesian model averaging, marking a novel contribution. The proposed framework is straightforward, integrating traditional Bayesian model averaging directly, yet it is powerful as it incorporates model uncertainty more fully into the framework.", "weaknesses": "This work is limited by a lack of discussion comparing its approach to existing model aggregation methods in conformal prediction including frequentist’s perspective. Although Section 2.5 touches on related works, the primary advantage highlighted is the ability to use the full data, which may be better understood as a contribution from Fong & Holmes (2021) rather than an original novelty here. To justify the method empirically, it would be more reasonable to benchmark a few of these existing methods, comparing their coverage, interval length, and execution time with suggested method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fobi0D3NoB", "reviewer": "Reviewer_ctnD", "summary": "This paper proposes a solution in the form of Conformal\nBayesian model averaging (CBMA) that combines Bayesian Model Averaging (BMA) with conformal prediction to address the potential issue of suboptimal prediction intervals when the Bayesian model itself is misspecified. The theoretical and empirical results show the efficiency of CBMA.", "questions": "1.Is it a clear pattern in the paper for when the authors use the term \"prediction set\" versus \"prediction interval\"?\n\n2.Is there a comparison of the time costs between CBMA and other methods?\n\n3.Why are the values of $\\alpha$ set as well as the  ratio of the size of training set to test set differently in Table 1 and Table 2? It would be better to unify them or display both values or provide a rationale for why they differ, as this would improve the clarity and comparability of the results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The results presented in the paper confirm the effectiveness of the proposed method.", "weaknesses": "1.The novelty of the work is not very significant, as it simply combines several existing methods. \n\n2.The names of the subsections under the \"Simulation\" section are not entirely appropriate,  since quadratic model is a type of non-linear model. It may be more appropriate to title whether the model is polynomial.\n\n3.The paper seems not explain why the results for model 10 and model 11 are missing when $n=50$ in Table 2.\n\n4.The paper contains a few typographical errors. For example, the last sentence before Section 2 has a typo. Additionally, there is a missing equation reference on line 150.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IoEWiA1NBY", "reviewer": "Reviewer_bVQG", "summary": "The paper leverages Conformal Bayesian and Bayesian Model Averaging to reduce the size of Conformal Prediction sets. The Bayesian posterior of different models is used in a weighted average of Bayesian conformity scores, which produces optimal prediction sets when the average contains the exact model and the sample size is infinite. Unlike previous Conformal Bayesian methods, the proposed scheme allows possible misspecifications in the underlying Bayesian model.", "questions": "- Is accessing the model parameters required?\n- Is the independence of the noise from X a strict assumption for the method applicability?\n- Missing reference number below Eq 1.\n- In Section 2.4, you say \"We propose to average conformity scores from each model to construct combined conformity score.\" Is this equivalent to Eq.2? Why don't you condition on X in the equations on page 3?\n- What do you mean by 'valid conformity score'?  \n- How expansive is computing the prediction set? \n- Does BMA converge to the true model if $n \\to \\infty$ because its likelihood increases?\n- Is this the first work on Conformal Bayesian that does not assume the underlying Bayesian model is exact?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- Conformal Bayesian can combine the advantages of frequentist and probabilistic approaches.\n- Theoretical results on the optimal efficiency of the prediction sets are rare and challenging to obtain in the CP framework.   \n- Model averaging and ensemble methods perform well in various setups. Studying CP in that setup may produce powerful uncertainty quantification tools.", "weaknesses": "- The technical contribution is to combine existing techniques: the BMA and Conformal Bayes. \n- It is unclear whether using the Bayesian conformity score is needed or if the method applies to any conformity score.\n- The authors should clarify why averaging the (Bayesian) conformity scores associated with different models is better than computing the conformity score of a Bayesian average of the models. \n- The main theoretical result holds in the limit of an infinite sample size. In that limit, Bayesian confidence intervals are also correct. Why would one need CP? Including the exact model in the average may look unrealistic. Does the result generalize to models that are only approximately correct?\n- Combining the results of Theorem 3 with Remark 1 to have a direct result on the optimality of the prediction sets would align better with the paper's goals.\n- The model has not been tested on real data. For example, the authors could show that the scheme produces smaller intervals than standard CP or any other baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)