CBMA: Improving Conformal Prediction through Bayesian Model Averaging

Authors: Pankaj Bhagwat, Linglong Kong, Bei Jiang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Bayesian framework, Conformal prediction, Model uncertainty, Uncertainty quantification

Abstract:
Conformal prediction has emerged as a popular technique for facilitating valid predictive inference across a spectrum of machine learning models, under minimal assumption of exchangeability. Recently, Hoff (2023) showed that full conformal Bayes provides the most efficient prediction sets (smallest by expected volume) among all prediction sets that are valid at the  $(1 - \alpha)$ level if the model is correctly specified. However, a critical issue arises when the Bayesian model itself may be mis-specified, resulting in prediction interval that might be suboptimal, even though it still enjoys the frequentist coverage guarantee. To address this limitation, we propose an innovative solution that combines Bayesian model averaging (BMA) with conformal prediction. This hybrid not only leverages the strengths of Bayesian conformal prediction but also introduces a layer of robustness through model averaging. Theoretically, we prove that the resulting prediction interval will converge to the optimal level of efficiency, if the true model is included among the candidate models. This assurance of optimality, even under potential model uncertainty, provides a significant improvement over existing methods, ensuring more reliable and precise uncertainty quantification.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730396927830

thread_type:
official_review

<thread_object>
creator: reviewer_bvqg

<content>
summary:
The paper leverages Conformal Bayesian and Bayesian Model Averaging to reduce the size of Conformal Prediction sets. The Bayesian posterior of different models is used in a weighted average of Bayesian conformity scores, which produces optimal prediction sets when the average contains the exact model and the sample size is infinite. Unlike previous Conformal Bayesian methods, the proposed scheme allows possible misspecifications in the underlying Bayesian model.

strengths:
- Conformal Bayesian can combine the advantages of frequentist and probabilistic approaches.
- Theoretical results on the optimal efficiency of the prediction sets are rare and challenging to obtain in the CP framework.   
- Model averaging and ensemble methods perform well in various setups. Studying CP in that setup may produce powerful uncertainty quantification tools.

weaknesses:
- The technical contribution is to combine existing techniques: the BMA and Conformal Bayes. 
- It is unclear whether using the Bayesian conformity score is needed or if the method applies to any conformity score.
- The authors should clarify why averaging the (Bayesian) conformity scores associated with different models is better than computing the conformity score of a Bayesian average of the models. 
- The main theoretical result holds in the limit of an infinite sample size. In that limit, Bayesian confidence intervals are also correct. Why would one need CP? Including the exact model in the average may look unrealistic. Does the result generalize to models that are only approximately correct?
- Combining the results of Theorem 3 with Remark 1 to have a direct result on the optimality of the prediction sets would align better with the paper's goals.
- The model has not been tested on real data. For example, the authors could show that the scheme produces smaller intervals than standard CP or any other baseline.

questions:
- Is accessing the model parameters required?
- Is the independence of the noise from X a strict assumption for the method applicability?
- Missing reference number below Eq 1.
- In Section 2.4, you say "We propose to average conformity scores from each model to construct combined conformity score." Is this equivalent to Eq.2? Why don't you condition on X in the equations on page 3?
- What do you mean by 'valid conformity score'?  
- How expansive is computing the prediction set? 
- Does BMA converge to the true model if $n \to \infty$ because its likelihood increases?
- Is this the first work on Conformal Bayesian that does not assume the underlying Bayesian model is exact?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730529659224

thread_type:
official_review

<thread_object>
creator: reviewer_ctnd

<content>
summary:
This paper proposes a solution in the form of Conformal
Bayesian model averaging (CBMA) that combines Bayesian Model Averaging (BMA) with conformal prediction to address the potential issue of suboptimal prediction intervals when the Bayesian model itself is misspecified. The theoretical and empirical results show the efficiency of CBMA.

strengths:
The results presented in the paper confirm the effectiveness of the proposed method.

weaknesses:
1.The novelty of the work is not very significant, as it simply combines several existing methods. 

2.The names of the subsections under the "Simulation" section are not entirely appropriate,  since quadratic model is a type of non-linear model. It may be more appropriate to title whether the model is polynomial.

3.The paper seems not explain why the results for model 10 and model 11 are missing when $n=50$ in Table 2.

4.The paper contains a few typographical errors. For example, the last sentence before Section 2 has a typo. Additionally, there is a missing equation reference on line 150.

questions:
1.Is it a clear pattern in the paper for when the authors use the term "prediction set" versus "prediction interval"?

2.Is there a comparison of the time costs between CBMA and other methods?

3.Why are the values of $\alpha$ set as well as the  ratio of the size of training set to test set differently in Table 1 and Table 2? It would be better to unify them or display both values or provide a rationale for why they differ, as this would improve the clarity and comparability of the results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650146338

thread_type:
official_review

<thread_object>
creator: reviewer_3y9y

<content>
summary:
The authors highlight that the efficiency of traditional conformal prediction can degrade in the presence of model uncertainty. To address this, they propose Conformal Bayesian Model Averaging (CBMA), which combines multiple non-conformity scores from a Bayesian perspective. This approach integrates Bayesian model averaging with Bayesian conformal prediction (full conformal Bayes; Fong & Holmes, 2021), allowing it to leverage the optimality of Bayesian model averaging from a theoretical standpoint. From a computational perspective, CBMA utilizes the add-one-in importance sampling approach of Fong & Holmes (2021), enabling it to use all available data for model training while avoiding exhaustive computation. The work demonstrates CBMA’s high efficiency through numerical justification in both correctly specified and misspecified model scenarios.

strengths:
While several works in the field of conformal prediction have applied a Bayesian perspective, this paper is meaningful since it is the first to attempt combining “multiple models” through Bayesian model averaging, marking a novel contribution. The proposed framework is straightforward, integrating traditional Bayesian model averaging directly, yet it is powerful as it incorporates model uncertainty more fully into the framework.

weaknesses:
This work is limited by a lack of discussion comparing its approach to existing model aggregation methods in conformal prediction including frequentist’s perspective. Although Section 2.5 touches on related works, the primary advantage highlighted is the ability to use the full data, which may be better understood as a contribution from Fong & Holmes (2021) rather than an original novelty here. To justify the method empirically, it would be more reasonable to benchmark a few of these existing methods, comparing their coverage, interval length, and execution time with suggested method.

questions:
1. This question relates to the previously mentioned points. What do you consider the distinguishing features of your approach compared to existing conformal set aggregation methods? Were any experiments conducted to compare them? If so, what were the results, and if not, what are your thoughts on this?

  2. In the simulations, the nominal value of $\alpha$ was set to 0.1 in the linear model scenario, but 0.2 in the nonlinear case. Is there a particular reason for these distinct choices? Also, did you observe any systematic performance changes in the model based on the value of $\alpha$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659045260

thread_type:
official_review

<thread_object>
creator: reviewer_nttv

<content>
summary:
The authors, working in the niche of bayesian conformal prediction, propose a bayesian model averaging procedure "infused" with conformal prediction, in order to recover coverage in the case of model misspecification

strengths:
I found the article quite well written.
Moreover, the proposal is of clear theoretical and applied interest, and developed with care and mathematical soundness.

weaknesses:
I fear that the authors have oversold their claim. In fact, in the literature proposals of conformal model averaging can be found  (https://www.sciencedirect.com/science/article/pii/S0925231219316108, https://arxiv.org/abs/2408.06642#:~:text=Unlike%20traditional%20methods%2C%20conformal%20ensembling,%2Dto%2Dinterpret%20uncertainty%20estimates. to name a few).

I found the simulation quite lacking in terms of depth, as well and presentation of the results

questions:
-As shown in the "weaknessess", some proposals (usually in a frequentist setting) are present in the literature when it comes to model averaging/ensembling in a conformal sense. Can you comment on this, maybe providing a more thorough analysis of the outstanding literature?

- I find the cases provided by the authors of limited applicative interest, I wonder if it could be possible to provide more challenging "misspecification" cases, like for instance cases where the model misspecification causes the conditional iid assumption to break (e.g. an AR2 model with only one lag).

- The impact in terms of interval length does not seem statistically significant. Are the authors able to check this, using maybe the Scheffe' method?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The formal novelties of the paper lie in its integration of Bayesian Model Averaging  with conformal prediction to address model misspecification and improve prediction set efficiency. By aggregating conformity scores from multiple Bayesian models using posterior model probabilities, the proposed Conformal Bayesian Model Averaging ensures valid frequentist coverage while optimizing the expected volume of prediction sets. The method theoretically guarantees convergence to optimal prediction sets as sample size increases, when the true model is in the model space (which apparently holds even under potential model misspecification).

It is to be noted that Theorem 2 (consequence of  Le & Clarke (2022) is not distribution-free). I also did not understand the claim when the model is not well specified. The Bayesian Posterior Convergence is that the Posterior probabilities of the true model (if it exists in the candidate set) converge to 1 as the sample size (ok then the quantiles estimates converges to the one of the ground-truth and the convergence of the size). If the true model is not present, posterior probabilities concentrate on the candidate model closest to the true model in terms of (KL) divergence. In the later case, The size of the CBMA prediction set may be larger than the optimal size that would be achieved if the true model were included. Equation 9 in the proof is not established in this case (BTW, the overall proof should be improved). 

> In contrast, our work provides optimally efficient conformal prediction sets without requiring knowledge of the true model. Our efficiency guarantee holds even under potential model misspecification. 

I could not see any proof of such statement in the paper. When the model is not well specified, the consistency results does not hold and this claim is misleading. Furthermore, what the authors want to claim seems quite straightforward: *any consistent estimation of the ground-truth distribution would produce asymptotically efficient uncertainty set*. This has fairly much nothing to do with model averaging but only consistency. 

Along with lack of comparisons to existing methods, the proposed benchmark (e.g. the main Table 1) are quite unconvincing: all the methods achieve very similar size (one needs to look at 3rd digit to notice any difference between the lengths and this is even not significant given the standard deviation).

----
*After further discussions with the SAC, we suggest an Accept, leaving the final decision to the program chairs to potentially bump down. All reviewers raised their scores to 6 ("Marginally above the acceptance threshold") after acknowledging the authors' efforts to address concerns and improve the manuscript. While some issues remain, the paper has merits and could provide valuable insights to the community.*