ICML 2025

Conformal Prediction as Bayesian Quadrature
Jake C. Snell 1 Thomas L. Griffiths 1 2
Abstract
As machine learning-based prediction systems are
increasingly used in high-stakes situations, it is
important to understand how such predictive mod-
els will perform upon deployment. Distribution-
free uncertainty quantification techniques such as
conformal prediction provide guarantees about
the loss black-box models will incur even when
the details of the models are hidden. However,
such methods are based on frequentist probability,
which unduly limits their applicability. We revisit
the central aspects of conformal prediction from
a Bayesian perspective and thereby illuminate the
shortcomings of frequentist guarantees. We pro-
pose a practical alternative based on Bayesian
quadrature that provides interpretable guarantees
and offers a richer representation of the likely
range of losses to be observed at test time.
1. Introduction
Machine learning systems based on deep learning are in-
creasingly used in high-stakes settings, such as medical
diagnosis or financial applications. These settings impose
unique constraints on the performance of these systems: we
want them to produce good outcomes in the aggregate, but
also do so fairly and with a guarantee of a low probability of
harm. However, predictive models based on deep learning
can be difficult to interpret, and commercial models increas-
ingly tend to offer little information about the techniques
used in training. This creates a new challenge: How can we
flexibly and reliably quantify the suitability of a model for
deployment without making too many assumptions about
how the model was trained or in which settings it will be
used?
Recent research on quantifying uncertainty has employed
methods based on conformal prediction (Vovk et al., 2005),
1Department of Computer Science, Princeton University
2Department of Psychology, Princeton University. Correspondence
to: Jake C. Snell <jsnell@princeton.edu>.
Proceedings of the 4...