On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning
Authors: Paul Scemama
Ariel Kapusta
Venue: Accepted by TMLR ()

Abstract: Bayesian deep learning and conformal prediction are two methods that have been used to convey uncertainty and increase safety in machine learning systems. We focus on combining Bayesian deep learning with split conformal prediction and how the addition of conformal prediction affects out-of-distribution coverage that we would otherwise see; particularly in the case of multiclass image classification. We suggest that if the model is generally underconfident on the calibration set, then the resultant conformal sets may exhibit worse out-of-distribution coverage compared to simple predictive credible sets (i.e. not using conformal prediction). Conversely, if the model is overconfident on the calibration set, the use of conformal prediction may improve out-of-distribution coverage. In particular, we study the extent to which the addition of conformal prediction increases or decreases out-of-distribution coverage for a variety of inference techniques. In particular, (i) stochastic gradient descent, (ii) deep ensembles, (iii) mean-field variational inference, (iv) stochastic gradient Hamiltonian Monte Carlo, and (v) Laplace approximation. Our results suggest that the application of conformal prediction to different predictive deep learning methods can have significantly different consequences.

Published in Transactions on Machine Learning Research (02/2024)
On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning
Paul Scemama The MITRE Corporation
pscemama@miitre.ogg
Ariel Kapusta The MITRE Corporation
akapusta @imitre.ory
Reviewed OpenReview: https: opwnreviee.n. net/ formm? id= TySo8faSSU
Abstract
Bayesian deep learning and conformal prediction are two methods that have been used to convey uncertainty and increase safety in machine learning systems. We focus combining Bayesian doep learming with split conformal prediction and how the addition of conformal prediction affects out-of-distributiin coverage that we would otherwise see; particularly in the case of multiclass image classification. We suggest that if the model is generally underconfident the calibration dataset, then the resultant conformal sets may exhibit worse out-of-distribution coverage compared to simple predictive credible sets (i.e. not using conformal prediction). Conversely, if the model overconfident on the calibration dataset, the use of conformal prediction may improve aut-of-distributin coverage. In particular, study the extent to which the addition of conformal prediction increases decreases out-of-distribution coverage for variely of inference techniques. In particular, (i) stochastic gradient descent, (ii) deep ensembles, (iii) mean-field variational inference, (iv) stochastic gradient Hamiltonian Monte Carlo, and (v) Laplace approximation. Our results suggest that the application of conformal prediction to different predictive deep learning methods can have significantly different consequences.
1 Introduction
Bayesian deep learning and conformal prediction two paradigms that have been used to represent uncertainty and increase trust in machine learning systems. Bayesian deep learning attempts to endow deop learning models with the ability to represent predictive uncortainty. These models often provide more calibrated outputs on both in-distribution and out-of-distribution data by approximating epistemic and aleatoric uncertainty. Conformal prediction is a method that takes (possibly uncalibrated) predicted probabilities and produces prediction sets that follow attractive guarantees; namely marginal coverage for exchangeable data (Vovk et al., 2005). On ont-of-distribution data, however, this guarantee no longer holds unless knowledge of the distribution shift is known a priori (Tibshirani et al., 2019; Barber et al., 2023). It is natural then consider combining conformal prediction with Bayesian deep learning models to try and enjoy in-distribution guarantees and better calibration on out-of-distributtin data. In fact, combination of the two methods has been used to correct for misspecifications in the Bayesian modeling process (Dewolf el al., 2023; Stanlon el al., 2023), thereby improving coverage and thus trust in the broader machine learning system. However, we suggest that application of both methods in certain scenarios may be counterproductive and worsen performance on out-of-distribution examples (see Figure 1). To investigate this suggestion we evaluate the combination of Bayesian deep learning models and split conformal prediction methods on image classification lasks, where some of the test. inpuls are out-of-distribution Our primary contributions are follows:
Copyright @ 2023 The MITRE Corporation. ALL RIGHTS RESERVED. Approved for Public Release; Distribution Unlimited. Public Release Case Number 23-3738

---

Published in Transactions on Machine Learning Research (02/2024)
after training
after training
after conformal
after confernat
training points
training points
calibration points
calibration points
ood points
ood test points
Figure 1: A conceptual illustration of how conformal prediction can help or harm out-of-distribution coverage for error tolerance of 0.25. On the left is conceptual illustration of how conformal prediction make the overall machine learning system less confident after conformalizing a model that is overly confident on the calibration dataset. As a consequence, it gains coverage on out-of-distribution examples. The right conceptualizes the opposite direction and illustrates how conformal prediction can reduce coverage on out-of-distribution examples.
Offer explanation of how the under- over-confidence of model is tied to when conformal prediction may worsen or improve out-of distribution coverage.
(ii) Evidence to support the explanation in (i) and further demonstration of the extent to which conformal prediction affects out-of-distribution coverage in the form of two empirical evaluations focusing on the out-of-distribution coverage of Bayesian deep learning combined with split conformal prediction.
(iii Practical recommendations for those using Bayesian deep learning and conformal prediction to increase the safety of their machine learning systems.
2 Preliminaries
2.1
Calibration
Modeling decisions factor into the calibration of the resultant. predicted probabilities. Consider vector of predicted probabilities produced by our model for input x,
K
* *(x) = (+1 (x),x)),xxxxx)))
Σ +x) =1 VX
where the sample space for x and K the number of possible labels y can assume (i.e. the labels that. can be assigned to x). For model be well-calibrated (Zadrozny and Elkan, 2002; Vasilev and D'yakonov, 2023) the following must hold,
P = i)+;() p) =p Vi V. 00,
where the probability taken over the joint data distribution p(x, y) and y the sample space for labels y. This says that on average over p(x, y), the predicted probability assigned to each class (not just the one assigned highest probability) represents the true probability of that class being the true label. If model is well-calibrated, then we should be able to create prediction sets that achieve marginal coverage for an error tolerance a by creating a predictive credible set which we later abbreviate as cred. To do so, for each model output f(x), we order the probabilities therein from greatest to least and continue adding the corresponding labels until the cumulative probability mass just exceeds
2

---

Published in Transactions on Machine Learning Research (02/2024)
2.2 Bayesian Deep Learning
One important reason for miscalibration is the abstention of representing epistemic uncertainty (Wilson and Izmailov, 2020). In order to create more calibrated predicted probabilities for neural networks, a popular approach is Bayesian doep learning, where a major goal is to faithfully represent parameter uncertainty (a type of epistemic uncertainty). Parameter uncertainty* arises due to many different configurations of the model weights that can explain the training dala, which happens especially in deep learning where we have highly expressive models (Wilson, 2020). Hence, Bayesian deep learning is an attractive approach to helping achiove calibration in practice, even on out-of-distributton examples. However, in order for the Bayesian approach to perform well a few important assumptions are required:
Our observation model relating weights w and inputs x labels y, p(y|x, w), is well-spocified. This means that our observalion model has the ability produce the true data generating function.
Our prior over weights p(w) is well-specified. This means that when it is paired with the observation model, induce a distribution over functions p(f) that places sufficient probability on the true data generating function (MacKay, 2003).
We often need to approrimate the posterior distribution of the weights with respect to the training dataset, P(w)D), for many interesting observation models, including neural nelworks. It is required that this approximation to the true posterior acceptable in the sense that both yield similar results for the task at hand. For example, in the predictive modeling scenario, the approximated and true posterior predictive should be nearly identical at the inputs that we will realistically encounter.
In this study Gaussians) that have been shown to produce good inductive biases for image classification tasks (Wilson and Iamailov, 2020; Ixmailov et al., 2021); ); and focus on varying the method of approximating the posterior over parameters.
class of observation models (convolutional neural networks) and priors (zero-mean
2.3 Conformal Prediction
We restrict our attention to a subset of conformal prediction methods: split (or inductive) conformal prediction (Vovk, 2012). Split conformal prediction requires extra held-out calibration dataset (which denote Deat) to be used during a "calibration step". Importantly, split conformal prediction assumnss exchangeability of the calibration and test set data. By allowing our final output to be prediction set Y S (1,.., K), conformal prediction guarantees the true class y to be included on average with probability (confidence) 1 ac:
Pr(y E W) a
(3)
where a is user-chosen error tolerance and Pr reads "probability that". This guarantee is maryinal in the sense that it is guaranteed on average with respect to the data distribution p(x.y) y) well as the distribution over possible calibration datasets we could have selected. Not only does split conformal prediction guarantee the incquality in (3) but also upper bounds the coverage given that the scores s; have a continuous joint ribbttion In particular, let Neal be the number of data pairs in the calibration dataset, then
1
Pr(y Y) < 11--at a
(4)
neal +
which is proved in Lei et al. (2018). Split conformal prediction methods work by first defining a score function that measures the disagroement between output probabilities *xx) of model and label We denote a general score function as sxx y), but it should be noted that the outputted scores depend on the underlying fitted model through the (x) it produces. The conformal method computes the scores on the held-out calibration dataset and then takes the
1
(1-a)(1+
[Deal
Parameter is loaded term, and context needed precisely understand what it means. In this case, we mean the weights of the neural network; not the parameters that may govern the distribution over such weights.
3

---

Published in Transactions on Machine Learning Research (02/2024)
of those scores which we denote 7. Then, during test time, prediction sets are constructed by computing the score for each possible Yt, and including ys into the prediction set if its score is less than or equal to т. Informally, if the candidate label y/ produces a score that conforms to what we have seen on the calibration datasct, it is included.
3 Motivation
We focus on the setting, as is often the case, where we expect to encounter out-of-distribution examples when we deploy our predictive systems in the real world. We are faced with the following binary choice: to either apply split conformal prediction to our model or to instead use non-conformal predictive credible sets. This choice will of course depend on our preferences. For instance, consider the scenario in which we want guaranteed marginal coverage on in-distribution inputs and additionally we would like to maximize the chance that our prediction sets cover the true labels for out-of-distribution inputs. What, then, are the benefits and risks associated with applying split conformal prediction? In beginning to answer this question, we provide an empirical evaluation alongside an explanation as to when applying split conformal prediction is expected to increase or decrease out-of-distribution coverage. The empirical evidence suggests that conformal prediction can significantly impact the credible sets. And hence, understanding the interaction between certain predictive models and conformal prediction is important for the safe deployment of machine learning, especially in safety-critiaal scenarios.
coverage one would otherwise see with predictive
While conformal prediction guarantees marginal coverage when the calibration data and test data are exchangeable, it loses its guarantees when encountering out-of-listribution data at test time?. Bayesian deep learning, on the other hand, has no such guarantees but has been shown to improve calibration (and thus coverage) on out-of-distribution inputs; a symptom of trying to quantify epistemic uncertainty. A natural desire, then, is to combine conformal prediction with Bayesian deep learning models in order to enjoy in-distribution guarantees while reaping the benefits of out-of- distiiibbbiitio calibration. To be clear, the main question this study intends to address is not how particular conformal prediction methods perform on out-of-disttibbution data. Instead, we address a more specific question: what are the benefits and risks associated with either applying split conformal prediction to our predictive model or passing up on using conformal prediction and simply using predictive credible sets when the predictive model is designed to be umoertainty-aware? In answering this question, we first discuss when, for a fixed predictive model, conformal prediction is expected to increase or decrease out-of-distribution coverage based on only on the calibration dataset. Afterward, we examine the extent to which the addition of conformal prediction increases or decreases the out-of-distribution coverage real-world settings. As we will see, this depends on the behavior of the predictive uncertainty of the underlying model, particularly on out-of-distribution inputs.
The context is that we have a fixed predictive model and a split conformal prediction method. We would like to know whether this split conformal prediction method will increase (or decrease) marginal coverage on out-of-distribution data as compared to using credible sets. Conformal prediction does whatever it needs to guarantce the desired coverage on the calibration datasct; mcaning it provides a lower bound (3) and an upper-bound (4) (see Figure 1). It does this by creating a threshold t such that the routine "allow candidate label y; into the prediction set if the score s(x, y.) S r" produces sets that attain the desired marginal coverage on the calibration dataset. We say predictive model is overconfident if its credible sets do not reach the desired coverage on the calibration dataset. The conformal method will then, by definition, create 7 such that more labels are allowed into the prediction sets in order to achieve the lower bound (3). And so on the calibration dataset, the conformal prediction sets will exhibit larger average set size than the credible sets. This will have the affect that, on any given future data, we can expect the average set size of the conformal sets to be larger than that of the credible sets. And hence, the conformal prediction sets will have higher chance of achieving bet ter marginal coverage. Conversely, credible sets exceed the desired coverage on the calibration dataset. The conformal method will then, by definition, create a T such that less labels are allowed into the prediction sets in order to achieve the upper bound (4). And so on the calibration dataset, conformal prediction sets will exhibit smaller average set size than the credible sets. This will have the affect that, on any given future data, we can expect the average set.
say the predictive model is underconfident if its
This is true, unless, as mentioned before, there is priort knowledge about the ofitt--disttibuutio data (Tibshirani al., 2019; Barber et al., 2023)
4

---

Published in Transactions on Machine Learning Research (02/2024)
size of credible sets to be larger than conformal prediction sets. And thus, credible sets will have higher chance of achieving better marginal coverage.
It follows from this discussion that, for a fixed model, the prediction set method we expect to have the greatest out-of-distributiin coverage will have largest average set size on the calibration dataset. Put another way, increasing the chance of out-of-distribution coverage without any knowledge of the test distribution will be at the expense of larger average set size on in-distribution data. We have just proposed an explanation to when conformal prodiction will likely increase (or decrease) marginal coverage on out-of-distribution data in the context fixed predictive model. In Section we empirically examine the ealend to which conformal prediction can either help or harm out-o--distribution coverage as compared credible sets. Furthermore, examine this extent across many different predictive models that exhibit different behavior with respect to prodictive uncertainty on out-of-distribure data.
4 Related Work
The analysis of combining more traditional Bayesian models with conformal prediction has been studied in Wasserman (2011) and Hoff (2021). Combining full (not split) Bayesian deep learning with conformal regression has been studied in the context of efficient computation of conformal Bayesian sets and Bayesian optimization (Fong and Holmes, 2021; Stanton et al., 2023). Both of these works consider settings with non-oxchangeable data but assume a priori knowledge on the type of distribution shift.
Theoretical foundations of using conformal prediction with non-exchaggeable data can admit very powerful guarantees but have thus far assumed priori knowledge about the distribution shift (Tibshirani et al., 2019; Angelopoulos et al., 2022; Fannjiang et al., 2022; Barber et al., 2023). Additionally, other powerful conformal algorithms have been posed to deal with out-of-distribution examples but require an additional model to detect those out-of-iistributioo examples (Angelopoulos al., 2021). Dewolf et al. (2023) evaluate conformal prediction combined with Bayesian deep learning models but for regression tasks and with relatively low-dimensional inputs (no greater than 280 features). Additionally, Kompa et al. (2021) look the coverage of prediction sets from various Bayesian deep learning methods and mention conformal prediction but do not include conformal prediction in their empirical analysis. We are not aware of a study examining the interaction between conformal prediction and Bayesian deep learning methods as relates to the coverage of unknown, out-of-distribution examples for task that deep learning models have excelled-ing only evaluate the combination of Bayesian deep learning and conformal prediction on image classification but also provide an intuitive explanation as to why, in certain scenarios, conformal prediction can actually harm out-of-distribution coverage we would otherwise see with non-conformal predictive sets.
classification. We not
5
Evaluation & Method Details
5.1 Predictive Modeling Methods
Consider probabilistic conditional model p(y. w/x) p(y]x, w)p(w) where are the weights, p(w) is a prior distribution, and p(y|x, w) is the probability of y given our weights w and fixed inputs x. Denote the training dataset as D. Then the posterior predictive distribution can be written as
plylx, D) - f p(uix, w)p(w/D)dw.
f
(5)
We implement three methods for approximating (5), which is sometimes referred to the Bayesian model average (Wilson and Izmailov, 2020).
Stochastic Gradient Descent (SGD) typically involves finding the maximum a posteriori (MAP) estimate for w. In the context of (1), this means we approximate pDw(D) 5(w WMMP) where
WMAP = argmax (logp(w/D)) = argmax {(log (D[p) + log p () + constant)]
w
Neural networks trained via stochastic gradient descent have been found to often be uncalibrated by being overly confident. in their predictions, especially on out-of-distribution examples (Guo et al., 2017).
5

---

Published in Transactions on Machine Learning Research (02/2024)
Deep ensembles (ENS) works by combining the outputs of multiple neural networks with different initializations (Lalshminarayanan et al., 2017). The idea is that the variation in their respective outputs can represent epistemic uncertainty. In this case, we implicitly approximate the posterior and simply average the hypotheses generated by cach model the ensemble:
pluxx, D)
1 } J 1=1
Σ
plulx, WMAP,)
where WMAP, is the stochastic gradient solution for model j in the ensemble of J models. Deep ensembles can be viewed as an approximation to the Bayesian model average (5) (Wilson and Izmailov, 2020).
Mcan-field Variational Inference (MFV) seeks approximate the posterior with variational distribution (Blci et al., 2017):
m
p(w(D) = q( II (w,
(6)
which comes from the mean-field assumplion,, and q is usually a simple distribution (e.g. Gaussian) 3, To make the approximation in (6), maximize with respect to 0 a function equivalent to the KL divergence between p(w D) and q( g(w)0) up to constant. It is usually termed the evidence lower bound (ELBO). We can write the objective as
KT. between variational posterior and prior
0- argmax (ELBO D)} argmaxx 0
E () [log p (D|ww)]
KLig(w10) [I
Expeecedd log likelihood of data
The ELBO objective is a tradeoff between two terms. The expected log likelihood of the data prefers q()) place its mass the maximum likelihood estimate while the KI. divergence term prefers q(-) stay close to the prior (Blei et al., 2017). Bayesian neural networks for classification have been shown to be generally underconfident by overestimating aleatoric uncertainty (Kapoor et al., 2022). Indeed, we find this is the case for both our experiments (see Figure 5).
Stochastic Gradient Hamiltonian Monte Carlo (SGMHC) is Markov Chain Monte Carlo (MCMC) method. In the context of inferring the posterior over parameters w, the basic premise of MCMC is to construct Markov chain on the parameter space W whose stationary distribution is the posterior of interest P(w(D). Hamiltonian Monte Carlo (HMC) a popular MCMC technique that uses gradient information and auxiliary variables better perform in high-dimensional spaces (sce 12.4 and 12.5 of Murphy (2023)). SGHMC attempts to unify the efficient exploration of HMC and the computational fcasibility of stochastic gradients (Chen et al., 2014). To this end, it follows stochastic gradient HMC sampling with an added friction term that counters the effects of the (noisy) stochastic gradients. This results a second-order Langevin dynamical system, which closely related to stochastic gradient Langevin dynamics (SGLD) (Welling and Teh, 2011). SGLD uses first-order dynamics, and can be viewed as a limiting case of the second-order dynamics of SGHMC.
Laplace Approximation (LAPLACE) provides a Gaussian approximation to p(0 D) by casting the posterior in terms of an energy function E(w) = -log (i, D) and then Taylor approximating this energy function around the MAP solution WMAP- The result multivariate Gaussian approximation of the posterior with its mean WMAP and its covariance matrix as the inverse Hessian of the energy function E taken with respect to w and evaluated at WMAP. That is, we approximate the posterior as
p(w(D) = N(WW/WMP. H
See 7.4.3 in Murphy (2023) for a derivation of the Laplace approximation. There are many different variants of the Laplace approximation for deep learning. These variants usually stem from how one approximates
"Note that we considering only global variable models, not local Local variables, often denoted as 2,, each only affect corresponding data point. As such, the number 2; scales linearly with the size of the dataset. Meanwhile, global variables affect each data point in the same way.
6

---

Published in Transactions on Machine Learning Research (02/2024)
Credible Set Coverage On The Calibration Dataset
Method Accuracies (Test Sets)
SGD
ENS
MFV
SGHMC
LAPLACE
0.8
0.05 Error
88%
96%
99%
99%
99%
0.01 Error
92%
97%
100%
100%
100%
Avg. Accuracy
0.7
Average Set Size
The Calibration Dataset
SGD
ENS
LAPLACE
0.6
SGD
ared
1.38
1.76
2.97
2.82
ENS
1.72
0.5
MFV
ope
2.19
1.92
2.37
2.66
SGHMC
0.01 Error
SGD
ENS
MPV
SGHMC
LAPLACE
LAPLACE
1.69
2.20
0.4
1
3 Corruption Intensity
2
4
thr
4.10
3.32
3.30
3.02
4.41
ope
4.12
3.25
3.58
3.44
4.96
(a) CIFAR10 accuracy plot. The accuracy plot shows, for each corruption intensity, the average accuracy over all corrupted datasets at that intensity.
(b) CIFAR10: The first table displays the credible set coverage the calibration dataset to indicate over- and under-confident predictive methods. The second table displays the average set sizes the calibration dataset. For each predictive model method, prediction set. method with highest average set size is bolded.
Figure2 2: The accuracy plot and calibration dataset results for the CIFAR10 experiment.
H due to its infeasible computational requirements for large models as well as the form of the posterior predictive distribution (Daxberger al., 2021). A popular variant is determined by only considering the last layer weights, Kronecker factorizing the Hessian (Ritter et al., 2018), and using linearized predictive distribution (Immer et al., 2021). We implement this Laplace variant for our experiments in Section 6.1.
5.2 Conformal Prediction Methods
We implement two common split conformal prediction methods.
Threshold prediction sets (thr) uses the following score function (Sadinle et al., 2019):
s(x. y) 1 (x).
That is, the score for an input x with true label y is one minus the probability mass the model assigns to the true label y. This procedure only takes into account the probability mass assigned the correct label,
Adaptive prediction sets (aps) uses the following score function (Romano al., 2020):
s(x. = #1 (I) ................................ (x),
where (x: ... 2t, (x) and U is uniform random variable in [0, 1] to break ties. That is, order the probabilities in +(x) from greatest to least and continue adding the probabilities, stopping after reach the probability associated with the correct label y.
5.3 Evaluation Measures
A prediction set is any subset possible labels. We would like a collection of prediction sets to have certain minimal properties. We would first like the collection to be marginally covered: given a user-specified error tolerance a, the average probability that the true class y is contained in the given prediction sets is 1 a. We would also like each prediction set in the collection to be smell: the prediction sets should contain as few labels as possible without losing coverage. Hence, we evaluate on the basis of the marginal coverage and average size collection of prediction sets emitted by applying split conformal prediction to Bayesian deep learning model outputs.

---

Published in Transactions on Machine Learning Research (02/2024)
SGD
ENS
MFV
SGHMC
LAPLACE
Aug. Coverage (6)
AVe. Coveroge (%5
Aug. Coverage
cred
Avg.
cred
cred
Avg. Caverage Difference (Ka)
Avg. Coverage Difference (Ka
the
Difference
Difference
Avg. Coverage Difference
thr
AV1 500 51
Avg. Set Size
Avg. Set Size
AVg Set Siza
Avg. Set Size
Comupttin Intersity
Corruption Intensity
Corruption Intensity
Coriuption Intensity
Corruption Intensity
Figure 3: CIFARIO and CIFARI0-Corrppted results at the 0.05 error tolerance. Row 1 and Row 3 illustrate the average coverage and average set size (respectively) of each prediction set method (thr, ap, simple predictive (cred)) for each predictive modeling method. To explicitly indicate the extent to which conformal prediction effects simple predictive sets, Row shows the average coverage difference between conformal prediction methods (thr, ap) and simple predictive credible sets (cred).
Measuring marginal coverage: for given test set Drest {xi, Mer and prediction set function Y(x) that maps inputs to a prediction set, we measure marginal coverage as
1 Hyr E Y(x)))
Thest
where 1 denotes the indicator function. This is just the fraction (percent) of prediction sets that covered the corresponding true label ye-
Measuring set size: for a collection of prediction sets (Y(x:) Rion, we the set size simply the average size of the prediction sets:
[x(x)]]-
Ritsst
6 Experiments
6.1 STTAR10-Corrupter
Our first experiment loosely follows the setup of Izmailov al. (2021) and Ovadia et al. (2019). We train AlexNet inspired model on CIFARIO by means of stochastic gradient descent, deep ensembles, and mean-field variational inforence (Krizhevsky et al., 2009; 2012). We then take 1000 examples (without replacement) from the CIFAR10 test set for calibration dataset. We use this calibration dataset determine thresholds that for conformal methods thr and aps, respectively. We then evaluate the marginal coverage and
8

---

Published in Transactions on Machine Learning Research (02/2024)
SGD
ENS
MFV
SGHMC
LAPLACE
Avg. Coveragn (%0
82-
-2
Avg. Coverode (%)
Avg. Coverags
Aug. Coverage
Avg.
Avg. Caverage Difference (8a)
Avg. Coverage Differnnce (K)
Difference
Difference
Avg. Coverage Difference
the
thr
Avg. Sec Size
Ave. Set Size
Aog Set Size
Set
Avg.
Comupttin Intensity
Corruption Intensity
Corruption Intensity
Coriuption Intersity
Corruption Intensity
Figure 4: CIFAR10: CIFAR10 and CIFAR1O-Corrupter results at the 0.01 error tolerance. Row 1 and Row 3 illustrate the average coverage and average set size (respectively) of each prediction set method (thr, ap, simple predictive) for each predictive modeling method. To explicitly indicate the extent to which conformal prediction effects simple predictive sets, Row shows the average coverage difference between conformal prediction methods (thr, ap) and simple predictive credible sets.
average size of the predictive credible sets (cred), thr sets, and aps sets that were produced for every single QUFARIO-Corupper dataset (Heendrycks and Dietterich, 2019) at every intensity
We first look the credible set coverage of each predictive model method the calibration dalasel, which is shown in Figure 2b. Recall from Section 3 that we say a predictive model is overconfident if the coverage of its credible sets on the calibration dataset are less than the desired coverage, and underconfident if they exceed the desired coverage. In Figure 2b we see that SGD is overconfident at both error tolerances, ENS is overconfident at the 0.01 error tolerance and neither overconfident or underconfident the 0.05 error tolerance, and MFV, SGHMC, and LAPLACE are all underconfident.. We expect then that conformal prediction set methods will increase dustt--distriuttion coverage for SGD (and ENS at the 0.01 error tolerance), while decrease out-of-distribution coverage for MFV, SGHMC, and LAPLACE. However, to what extent this the case? To analyze this empirically, we show the average marginal coverage, the average difference marginal coverage between conformal prediction methods and simple predictive credible sets, and average set size across datasets at each intensity in Figures 3 and 4. The average accuracies for each predictive modeling method are shown in Figure 2a.
Remarks:
In the context of the methods evaluated, if a top priority is to capture true labels even unknown out-of-distributiin inputs, one generally better off using "uncertainty-awaree methods such as MFV, SGHMC, and LAPLACE without conformal prediction. However, these three methods vary widely in how large their average set size is in order to achieve good oul-of-distribution coverage.
We take out those semantically similar images from the corrupted test set that we used for the calibration dataset data leakage.
We take the average across different calibration and set splits as well as different datasets that comprise the CIFARIO- Corrupted dataset. To both specific results each of the different datasets (types of corruptions) and plots with error hars, the Appendix.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Credible Sei Cuverage On The Calibration Datazet
SGD
ENS
MFV
SGHMC
Method Accuracies (Test Sets)
0.05 Ervor
94%6
96%
9006
90%
0.9
SGD
0.01 Ervor
97%
98%
100%
100%
ENS
MFV
Avg. Accuracy
0.8
SGHMC
Average Set Sizes
On The Calibration Dataset
0.05 Ervor
SGD
ENS
MFV
SCIIMC
cred
1.20
1.34
2.97
1.87
0.7
1.18
1.15
1.59
1.58
1.95
1.57
0.6
0.01 Eroor
SGD
ENS
MFV
SCHMC
cred
1.65
1.94
5.08
2.96
0.5
1.88
4.96
5.32
4.85
2.87
organCmnist
organSmnist
Test Set
(a) MedMNIST accuracy plot. is evaluated for the in-distribution organCmnist dataset and the out- of-distribution organSmnist dataset.
(b) MedMNIST:
The first. table displays the credible set coverage the calibration dataset to indicate over- and under-confident predictive methods. The second table displays the average set sizes the calibration dalaset. For each predictive model method, the prediction set method with highest average set size is bolded.
Figure 5: The accuracy plot and calibration dataset results for the MedMNIST experiment.
For a fixed predictive model, the prediction set method with greatest out-of-disttibution coverage is that with the largest. average set size on those out-of-distribution inputs well as on the calibration dataset (see Figure 2b for average set size on the calibration dataset).
Across all predictive models, larger average set size of a prediction set method paired with a predictive model will usually mean better mut-of-distribution coverage, but there be exceptions arising from how different predictive models exhibit uncertainty on out-of-distribution inputs. For example, MFV credible sets are smaller on out-of-distribution inputs than SGHMC credible scts, but MFV credible sets achieve better coverage.
small change in set size on in-distribution data can have drastic effects on the out-of-distributiio coverage, especially on inpuls that are far from the training distribution. For example (see Figure 3) an x 1 decrease in average set size due to performing thr conformal prediction results in sy 20% drop in out-of-distribution coverage compared to SGHMC credible sets and an s 10% drop in out-of-distributtio coverage as compared MFV credible sets.
Being more robust to the corruptions in terms of accuracy (see Figure 2a) does not immediately translate to better coverage on out-of-distribbtiio examples (see Figures 3 and 4). For instance, test sets,
both MFV and SGHMC perform the worst in terms of accuracy on the however they perform better than SGD and ENS in terms of distriiiitiiitto coverage. And importantly, this is not due to being trivially underconfident,, i.c. producing sets of all possible labels for each inpul.
A majority of the time, thr and aps behave similarly in terms of average set size and out-of-distribution coverage. However, there are nolable exceptions where thr produces smaller sets on out-of-distribution inputs which in turn more negatively impacts out-of-distribution coverage (see, for example, Figure 3, SGHMC and MFV).
We do not argue that any method (or combination thereof) is always preferred over another. Indeed, what one docides upon will depend on many factors. Two important factors (that this study highlights) is both the frequency of out-of-distribution data encountered by a deployed model as well as how far these data may be from the training distribution. It is straightforward why the former is important, and the latter is important.
10

---

Published in Transactions on Machine Learning Research (02/2024)
SGD
ENS
MFV
SGHMC
SGD
ENS
MFV
5GHMC
Aug Coveroge (N)
Aep Coverage (5)
Coverge (N6
Aug Orvengg (N)
Aug. Coveroge (1)
Govenge
Coverage
Avg.
crd
Chd
Crrd
cldd
Crd
crd
Crid
ord
Nig. Cover rage Difference INl
Aug. Cover Offferance Hop
Ara. Cover rago Diffcrncce 186
Arg. Cover rago D Offfcrence 194
Cover enverage Difference MMg
Coue Difference
Coverage
Seg. Coverage
Coverige
ingaddenn
wgw
orper argan
Avg. Set Size
Aug. Sat Sat Stze
ANg Set Size
Avg. Siz0
Avg. S-ttttt Size
A00
Avg.
Test Set
Test Set
Test Set
Test Set
Test Set
Test Set
Test Set
Test Set
Error tolerance of 0.05 (i.e. 0.95 desired coverage)
(b) Error tolerance of 0.01 (i.e. 0.99 desired coverage)
which is denoted by the dashed horizontal red line.
which is denoted by the dashed horizontal red line.
Figure 6: MedMNIST: Average marginal coverage and average set size for the in-distribution organ Cmnnist dataset (denoted on the plot) and for the out-of-distribution organSmnist. dataset (denoted out on the plot.)
because we see large disparity between the type of coverage we would see with conformal prediction opposed to without it on data that is far from the training distribution (c.g. see Figure 3, SGHMC). We also illustrate that the error tolerance we choose does not represent our true desires if we believe out-of-distribution data will be encountered. While this is to be expected since our conformal prediction methods contain no guarantees for out-of-distribution data, we demonstrate the eztent to which this error tolerance is violated, and how in some cases it can be mitigated by foregoing the use of conformal prediction.
6.2 MedMNIST
Our second experiment provides more realistic safety-critical scenario: 11-class classification of radiology scans. We train a ResNet18 model on the organCmmist. dataset of MedMNIST with the same three modeling methods (He et al., 2016; Yang el al., 2023). Conformal calibration is done using the first. 500 examples from the test set of organCmnist. We then evaluate the coverage and size on the remaining examples from the test set of organCmnist Cmnistt for all three prediction set methods. The organSmnist dataset has the same classes but is the result of different views of the subject of interest in the radiology scan. Thus, it is from different distribution, and so we also evaluate the coverage and size on the test set of organSmnist to see how the combination of conformal methods and Bayesian deep learning affects out-of-distribbtiio coverage.
As in the previous experiment, we first note the credible set coverage of each predictive model method the calibration dalaset (shown in Figure 5b). Similar to the CIFAR10 experiment, SGD is overconfident and MFV and SGHMC are underconfident.. ENS is underconfident at the 0.05 error tolerance but overconfident at the 0.01 error tolerance. We expect then that, in terms out-of-distribution coverage, conformal prediction set methods will most likely help SGD, most likely harm MFV and SGHMC, and most likely harm ENS at the 0.05 error tolerance but help ENS at the 0.01 error tolerance. And again ask, to what extent is this the case? In answering this, the average marginal coverage and average set size for each dataset is shown in Figure 6a for the 0.05 error tolerance and Figure 6b for the 0.01 crror tolerance. The accuracies of the deep learning methods are shown in Figure 5a. The in-distribution dataset is organ Cmnist and the out-of-distribution dataset is SmninSm
11

---

Published in Transactions on Machine Learning Research (02/2024)
Remarks:
In the context of the methods evaluated, if a top priority is to capture true labels even on unknown out-of-disttibuution inpuls, one is generally better off using "uncerlainty-aware" methods such MFV and SGHMC without conformal prediction. An exception is at the 0.01 error tolerance where ENS aps sets and SGD aps sets attain slightly better coverage than SGHMC, although at the cost of significantly larger average set size.
For a fixed predictive model, the prediction set method with greatest out-of-distribution coverage is that with the largest average set size on those out-of-distribution inputs well as on the calibration dataset.
Across all predictive models, larger average set size of a prediction set method paired with a predictive model will usually mean better out-of-distribution coverage, but there can be exceptions arising from how different predictive models exhibit uncertainty on out-of of-distributtion inputs. For example at the 0.01 error tolerance, MFV aps sets are smaller on -iit-i--iittibutionn inputs than ENS aps sets but achieve better coverage.
small change in average set size on in-distribution data can have drastic effects on the out-of- distribution coverage. For example, an s 1 decrease in average set size due to thr with SGHMC results in 21% drop in in average set size due to aps with SGD results in z 10% increase in out-of-distributio coverage (Figure 6a).
coverage (Figure 6a). On the flip side, an = 3 increase
7 Discussion
Limitations and Future Work: We recognize that evaluated and compared only few of the many conformal and Bayesian methods. Furthermore, although the results presented here add an important dimension to the practical considerations of combining conformal and Bayesian deep learning methods, there are many other questions that remain to be answered (e.g. adaptivily gains from using Bayesian deep learning with conformal prediction). We developed experiments that provide evidence for the explanation offered in Section 3 (see also Figure 1) which consequently demonstrated that certain modeling and data scenarios can seriously impact the benefit of conformal prediction. Future work may include developing diagnostics or practical checks that suggest one is in a particular scenario in which the utility of conformal prediction can be predicted. Future work might also include further evaluations with additional measures such as size-stratified coverage (Angelopoulos et al., 2020), and further mathematical analysis. Such analysis might provide ional insights into when and how to combine certain conformal prediction and Bayesian deep learning methods.
Conclusion: We demonstrated important scenarios in which conformal prediction can decrease the out- of-distribution coverage one would otherwise see with simple predictive credible sets. Importantly, we also demonstrate the eztent to which conformal prediction does so. We also saw that in some cases it is not realistic to think that the error tolerance we select will be honored. We hope that this study motivates the need for better evaluation strategies for Bayesian deep learning models. Echoing some of the arguments made in Kompa et al. (2021), frequentist. coverage of both in-distribution and out-of- distribution examples for Bayesian deep learning models provides nuanced and practical representation of both the calibration of the models and the benefits of using conformal prediction in realistic settings. These are all of immediate practical importance for a wide range of application areas, particularly in those where unsafe mistakes can incur a large cost. If strong guarantees coverage are desired, then one may consider Bayesian deep learning, conformal prediction, or both, in an effort to provide those guarantees. Knowledge of the scope of application, assessment to identify breaking important assumptions (c.g. out-of-distribution data), and expected use may help decide the methods that should be applied. Being aware of these results and using the conclusions will better equip engineers in creating safer machine learning systems.
Disclosure of Funding: This research was funded by MITRE's Independent Research and Development Program
12

---

Published in Transactions on Machine Learning Research (02/2024)
References
TensorFlow Datasets, a collection of ready-to-use datasets. https://www. http::/twessrowoooo/ooe/oe...
A. Angelopoulos, S. Bates, J. Malik, and M. Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint ar arxi::2009.1440 14193, 2020.
A. N. Angelopoulos, S. Bates, J. Candès, M. I. Jordan, and L. Tei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint or arXiv:2110.011 01002, 2021.
A. N. Angelopoulos, Bates, A. Fisch, L. Lei, and T. Schuster. Conformal risk control. arXiv preprint Xiv: 022844 2022.
R. Barber, E. J. Candes, A. Ramdas, and R. J. Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, (2):)8816 845, 2023.
P. Bilic, P. Christ, Н. B. Li, E. Vorontsov, A. Ben-Cohen, G. Kaissis, A. Szeskin, C. Jacobs, G. E. H. Mamani, G. Chartrand, et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis, 84:102680, 2023.
D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 11 112(518):859 877, 2017.
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wandernan-Milne, and Q Zhang. JAX: composable transformations of Python+Numpp programs, 2018. URL http::/gg thu...uu...... com/google//aae
T. Chen, E. Fox, and C. Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pages 1683-1691. PMLR, 2014.
E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux-effortles Bayesian deep learning. In NewrIPS, 2021.
N. Dewolf, B. D. Bacts, and W. Waegeman. Valid prediction intervals for regression problems. Artificial Intelligence Review, 56(1)6577-613, 2023.
C. Fannjiang, S. Bates, A. N. Angelopoulos, Listgarten, and M. L. Jordan. Conformal prediction under feedback covariate shift for biomolecular design. Proceedings of the National Academy of Sciences, 119(43): c2204569119, 2022.
E. Fong and C. C. Holmes. Conformal bayesian computation. Advances in Neural Information Processing Systems, 34:1 1268 18279, 2021.
C. Guo, G. Pleiss, Y. Sun, and K. Q Weinberger. On calibration of modern neural networks. International conference machine learning, pages 1321-1330. PMLR, 2017.
C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, al. Array programming with numpy. Nature, 585(7825)3357 362, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and paltern recognition, pages 770-778, 2016.
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint wxXiv:1903.19960 2019.
T. Hemnigan, T. Cai, T. Norman, L. Martens, and I. Babuschkin. Haiku: Sonnet for JAX, 2020. URL attp://githut
P. Hoff. Opyes-optimal prediction with frequentist coverage control. arXiv preprint 3rXiw:2105.140.. 2021. J. D. Hunter. Matplotlib: A 2d graphics environment. Computing Science & Engineering, 9(3):90-955 95, 2007. doi: 10.1109/MCSF.2007.5
13

---

Published in Transactions on Machine Learning Research (02/2024)
A. Immer, M. Korzepa, and M. Bauer. Improving predictions of bayesian neural nets via local linearization. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 703-711. PMLR, htmml.
13-15 Apr 2021. URL ptttss:/prrccedinng mlr. aar.prenn
P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. G. Wilson. What are bayesian neural network posteriors really like? In International conference on machine learning, pages 4629-4640 PMLR, 2021.
S. Kapoor, W. J. Maddox, Izmailov, and A. G. Wilson. On uncertainty, tempering, and data augmentation in bayesian classification. Advances Neural Information Processing Systems, 35:18211-18225, 2022.
B. Kompa, Snoek, and A. L. Beam. Empirical frequentist coverage of deep learning uncertainty quantification procedures. Entropy, 23(12)11608, 2021.
Krizhevaky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neurol information processing systems, 25, 2012.
B. Lalshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman. Distribution free predictive inference for regression. Journal of the American Statistical Association, 113(523): 1094 1111, 2018
D. J. MacKay, Information theory, inference and learning algorithms. Cambridge university press, 2003.
K. P. Murphy, Probabilistic machine learning: Advanced topics. MIT press, 2023.
Y. Ovadia, F. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Takshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Н. Ritter, A. Botev, and D. Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations, 2018. URL https://openrevion/ http://pwrrrriioooooooooorrrrraa pettforrumiddSkdvdzza
Y. Romano, M. Sesia, and E. Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581 3591, 2020.
M. Sadinle, J. Lei, and L. Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statislical Association, 114(525):222 234, 2019.
S. Stanton, W. Maddox, and A. G. Wilson. Bayesian optimization with conformal prediction sets. In International Conference on Artificial Intelligence and Statistics, pages 959 986. PMLR, 2023.
D. Stutz, K. D. Dvijotham, A. T. Cemgil, and A. Doucet. Learning optimal conformal classifiers. In International Conference on Learning Representations, 2021.
R. J. Tibshirani, R. Foygel Barber, E. Candes, and A. Ramdas. Conformal prediction under covariate shift. Advances in newral information processing systems, 32, 2019.
G. Van Rossum and F. L. Drake. Python 3 Reference Manual. CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
R. Vasilev and A. D'yakonov. Calibration of neural networks. arXiv preprint arXiv: 2023.
V. Vovk. Conditional validity of inductive conformal predictors. In Asian conference machine learning, pages 475 490. PMLR, 2012.
V. Vork, A. Gammerman, and G. Shafer. Algorithmic learning in random world, volume 29. Springer, 2005.
M. L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021 3021, 2021. doi: 10.21 (0.21105/joss000 03021. URL https://doi: 22tp////.....//osss... 03021.
14

---

Published in Transactions on Machine Learning Research (02/2024)
L. Wasserman. Frasian inference. 2011.
M. Welling and Y. W. Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681 688, 2011.
G. Wilson. The case for bayesian deep learning. arXiv preprint arXiv:2001.10995, 2020.
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistie perspective of generalization. arXiv preprinl ar Xrx:222002 08791, 2020.
J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, Н. Pfister, and B. Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and dala mining, pages 694 699, 2002.
15

---

Published in Transactions on Machine Learning Research (02/2024)
APPENDIX
A TRAINING DETAILS
For both experiments we train SGD and MFV for 5 different scods. ENS is the result of combining 5 stochastic gradient descent model states from the different seeds. Due to the computational cost of running SGHMC, we only trained it for single seed. We measure the validation accuracy (or in the case of SGHMC collect a sample) every 10 epochs. For the CIFAR1O experiment we run SGHMC for 1000 burnin epochs and then 9000 epochs therafter resulting in 900 total samples. For the MedMNIST experiment we run SGHMC for 1000 burnin epochs and then 10000 epochs thereafter resulting in 1000 total samples.
For evaluation of SGD and MFV we select the model state that attains the best validation accuracy. If the best validation accuracy is shared between multiple checkpoints, use the model state from the carlicst checkpoint amongst those that are tied.
LAPLACE is a result starting with one of our pretrained SGD solutions and then conducting the laplace approximation with the package from Daxberger ct al. (2021). In particular, we compute a laplace approxi- mation for the last-layer with the kron approximation to the Hessian. Furthermore, use the glm alongside the probil approximation for the posterior predictive. See Daxberger et al. (2021) for more details. We only use the LAPLACE method in the CIFARIO experiment.
A.1 CIFAR10
Dataset: The CIFARIO dataset contains 60,000 32 32 X 3 RGB images in 10 classes, where each class contains 6,000 images each. There are 50,000 training images (5,000 images per class) and 10,000 test images (1,000 images per class). We take 5% of the original training dataset (0.05 x 50, 000 = 2, 500) examples as a validation set, and leave the remaining 95% (50, 000 2, 500 47, 500) examples for training. For preprocessing, normalize the images with mean (0.49, 0.48, 0.44) and standard deviation (0.2, 0.2, 0.2) for each of the 3 channels. This is taken from the code repository of (Izmailov et al., 2021). We performed no data augmentation.
Base Model: We use an AlexNet inspired convolutional neural network as a base model, which is taken from the code repository of (Izmailov et al., 2021).
Training Hyperparameters: The following tables illustrate the training hyperparameters for the CIFAR10 experiment.
Table 1: Hyperparameters for SGD and MFV. The initial o is the initial value of the standard deviation of the per-parameter Gaussians for mean-field variational inference.
(b) MFV Training Hyperparameters.
(a) SGD Training Hyperparameters
Name
Value
Name
Value
seeds
{l.....5) 5}
seeds
[I....,5] 5]
batch size
80
batch size
80
epochs
100
epochs
100
weight decay
5.0
weight decay
5.0
temperature
1.0
temperature
1.0
learning rate schedule
cosine
learning rate schedule
cosine
checkpoint frequency
10
checkpoint frequency
10
initial step size optimizer
4e-4
initial step size
8e-7
Adam
optimizer
sgd
initial o # samples
0.01
momentum decay
0.9
16

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Hyperparameters for SGHMC
Name
Value
seeds
batch size
80
epochs
10000
weight decay
5.0
temperature
1.0
learning rate schedule
cyclical
cycle epochs
75
checkpoint frequency
10
initial step size
3e-6
momentum decay
0.9
preconditioner
RMSprop
A.2 MedMNIST
Dataset: MedMNIST contains many standardized datasets of biomedical images (Yang et al., 2023). We train on one these datasets: organCmnist. This dataset is part larger cohort of three datasets which are based on the 3D CT images from the Liver Tumor Segmentation Benchmark (Bilic et al., 2023). The larger cohort is (organAmnist, organCmnist, organSmnist }, where A,C, and S are short for Axial, Coronal, and Sagittal. These describe different views of the CT scan (see Figure 7). We use the pre-specified training and validation sets provided by McdMNIST. These contain 13,000 training examples and 2,392 validation examples. Each image grayscale. For preprocessing, normalize the images with mean 0.49 and standard deviation 0.2 for the single channel. We performed no data augmentation.
Conoral
plara
plare
Adai plano
Figure An illustration describing the axial, coronal, and sagiltal views. htttp://anntoomytol.oor org/ andd-
content/lecturio- http://eccurioorrviin----- drwwwng-sagittennn ta -aee--lllae-eglll---aaal
Base Model: We use a ResNet18 neural network as a base model (He et al., 2016), which built-in model in the Haiku library (Hennigan al., 2020).
Training Hyperparameters: The following tables illustrate the training hyperparameters for the MedM- NIST experiment.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Hyperparameters for SGD and MFV. The initial a is the initial value of the standard deviation of the per-parameter Gaussians for mean-field variational inference.
(b) MFV Training Hyperparameters.
(a) SGD Training Hyperparameters
Name
Value
Name
Value
seeds {l,...,5 5)
seeds
(L.....5)
5}
batch size
80
batch size
80
epochs
100
epochs
100
weight decay
10.0
weight decay
10.0
temperature
1.0
temperature
1.0
learning rate schedule
cosine
learning rale schedule
cosine
checkpoint frequency initial step size optimizer
10
checkpoint frequency
10
le4
initial step size
6e-6
Adam
optimizer
sgd
initial o
0.01
momentum decay
0.9
# samples
Table 4: Hyperparameters for SGHMC
Name
Value
seeds
batch size
80
epochs
11000
weight decay
5.0
temperature
1.0
learning rate schedule
cyclical
cycle epochs
75
checkpoint frequency
10
initial step size
le-5
momentum decay
0.9
preconditioner
RMSprop
B EVALUATION DETAILS
We first note that we use 30 samples to approximate the posterior predictive density when using mean-field variational inference. For both experiments, we create prediction sets in the same way. Given an error tolerance a, for each method (stochastic gradient descent, deep ensembles, and mean-field variational inference) and cach evaluation datasct, we produce predicted probabilitics *xx) and then create...
We ordor the probabilities (x)x E (x) from greatest to least and continue adding the corresponding labels until the cunulative probability mass just exceeds 1 - a. We sometimes abbreviate this method as cred.
Predictive Credible Sets:
Using a calibration dataset Dent taken from the in-distribution test set, we compute scores for each example using the score function (Sadinle al., 2019):
Threshold Prediction Sets:
s(x,y) = (x)).)
Then we take the
1
a1 +
quaanilee
Deall
of these scores which we call 7. Then for each input we want to evaluate for, we create prediction sets as
18

---

Published in Transactions on Machine Learning Research (02/2024)
(x) = fu E s(x, v) S}
where y is the sample space for labels y. We sometimes abbreviate this method as thr.
Adaptive Prediction Sets: Using a calibration dataset Deal taken from the in-distribution test set, we compute scores for each example using the score function
(x, = #1 (x) +... (.)()))
where (x) (2) and U is uniform random variable in [0, 1] break ties (Romano et al., 2020). As in the case of threshold prediction, we take the
1 ququunneee
- (1 + Deall
of these scores which we call Then for each input we want to evaluate for, we create prediction sets
((x) = tu E 8 (x, y) ZT
where y is the sample space for labels y. We sometimes abbreviate this method as aps.
Remark on Adaptive Prediction Sets: It is useful to note that for coverage to be tight (i.e. having the upper bound in equation (4) of the paper), adaptive prediction sets requires distinct conformity scores. To handle this, an additional standard uniform random variable used. During the calibration phase, we take [Deat] random samples from this variable and subtract from the scores before computing the quantile т. And during the prediction phase, we take [Drett] random samples and subtract from the scores before checking if they are less than or equal to т, where Drest is the test set. We use the implementation from Stutz et al. (2021), which allows one to input a random seed to do the above procedure.
B.1 CIFAR10 and UFFAR10-Corrupted
In the CIFAR10 experiment evaluate the prediction sets on (i) the CIFAR10 test set (Krizhevsky et al., 2009), and (ii) all CIFAR10-Corupted test sets (Hendrycks and Dietterich, 2019). The CIFAR10-Corupted test sets contain 19 different corruptions, each with intensitics ranging from to 5.
The X 19 95 CIFAR10- CIFAR10-Corrupted test sets are all corrupted versions of the original CIFAR10 test set. Thus, when we take the 1, 000 examples from the original CIFAR10 test set to use calibration dataset for our conformal methods, we also take the 1, 000 corresponding (semantically similar) examples from all the CIFAR10-Corupted test sets. We evaluate the three prediction set methods on the remaining 9, 000 examples from the original CIFARIO test set, and then all the trimmed SHTARIO-Corrupee test sets (which each contain 000 examples). We run the procedure of taking a calibration dataset, finding т, and computing prediction sets all the test sets with three different seeds (1, Then take the average accuracy, marginal coverage, and set size across these three seeds. The accuracy results are presented in the main paper. The accuracy, marginal coverage, and set size results for each dataset are presented in section F.2. the main paper we go further and take the average marginal coverage and average set size across data sets each intensity and report those summarized results.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Table 5: Different type of corruptions CIFARIO- Corrupted.
Corruption Type
brightness
2
contrast
3
defocus blur
4
elastic fog frost
5
6
7
frosted glass blur
8
gaussian blur
gaussian noise
10
impulse noise
11
jpeg compression
12
motion blur
13
pixclate
14
saturate
15
shot noise
16
snow
Figure 8: An example of the spatter corruption at intensity level 4. https: http:://www.nnnsoofloo
17
spatter
tensorflow. org//aatasettsccaaalo//cctttaa httpataalllgcaaatoooorrrrroe
18
speckle noise
19
zoom blur
B.2 MedMNIST: organCmnist And organSmnist
In the MedMNIST experiment we only have two test sets. The organCmnist Cmnist test set (containing 8,268 examples) and the organSmnist test (containing 8,829 examples). The C in organCmnist standards for coronal and the S in organSmnist stands for sagittal (see Figure 7). We take 500 examples from the organCmnist test set to use as a calibration dataset for our conformal methods. We then evaluate the three prediction set methods on the remaining examples from the organCmnist test set as well as on the organSmnist test set, the latter serving as our out-of-distributton test set. We run the procedure of taking a calibration dalaset, finding т, and computing prediction sets all the test sets with three different, seeds {1, Then take the average accuracy, marginal coverage, and set size across these three seeds. The results are presented in the main paper. The class proportions for all splits of both organCmnis and organSmnist are shown in Figure 9.
20

---

Published in Transactions on Machine Learning Research (02/2024)
organC train
organC val
organC test
0.25
Class Proportion
0.00
Class
organS train
organ5 val
organS test
0.25
Class Propartion
0.00
Class
Figure 9: Class propotions for both the organ Cmnist datasets and the organSmnist datasets.
Class
Class 1
Class
Class
Class 4
Class Class
Class 7
Class
Class 9
Class 10
Figure 10: An example image from each of the 11 classes in the organ Cinnst dataset.
Class 0
Class 1
Class
Class 3
Class 4
Class 5
Class 6
Class 7
Class 8
Class 9
Class 10
Figure 11: An example image from each of the classes in the organSmnist dataset.
C SPACE & COMPLEXITY
Space: If is the number of learnable parameters for SGD, then MFV requires 2n. learnable parameters. This is due to treating each weight as random variable from Gaussian distribution, and instead having to fit the two parameters governing that distribution. ENS requires kn learnable parameters. We use à q is the number of samples collected, then SGHMC requires qn parameters be kept in order run inferences.
is the number of models in the ensemble, then
Runtime: If m is the number of forward passes needed to predict using SGD, then MFV inference requires pm forward passes where is the number of samples to construct a Monte Carlo approximation for the Bayesian model average. During training we have p = and during ovaluation we have p = 30. l: is the number of models in the ensemble, then deep ensembles requires kn forward passes. We use & 5. If the number of samples collected for SGHMC then inference requires gm forward passes.
21

---

Published in Transactions on Machine Learning Research (02/2024)
The Laplace approximation is post-hoc and needs to be "fit" the training data (i.e. the Hessian factors need to be computed which require the data). We defer to Appendix B of Daxberger et al. (2021) for space and runtime details.
D SOFTWARE PACKAGES
Python 3, PSF License Agreement (Van Rossum and Drake, 2009).
Matplotlib, Matplotlib License Agreement (Hunter, 2007).
Seaborn, BSD License (Waskom, 2021).
Numpy, BSD License (Harris et al., 2020).
JAX, Apache 2.0 License (Bradbury et al., 2018).
Haiku, Apache 2.0 License (Hennigan et al., 2020).
Tensorflow Datasets, Apache 2.0 License (TFD).
google-reeearc/ /bnn hmc, Apache 2.0 License (Izmailov al., 2021).
poogl=-depminn//oooornal training, Apache 2.0 License (Stutz et al., 2021).
aleximmer/Laplace, MIT License (Daxberger al., 2021).
E COMPUTE
We ran our experiments on an Ubuntu 18.04.6 system with a dual core 2.10GHz processor and 754 GiB of RAM. We also used a single Tesla V100-SXM2 GPU with 32 GiB of RAM.
CIFAR10 Experiment For training, SGD takes 3.6 minutes per seed, MFV takes 4.6 minutes per seed, SGHMC takes x 4 hours, and LAPLACE takes Rs 5 minutes. For evaluation, SGD takes se 3 minutes per seed, MFV takes x .3 minutes per seed, ENS takes z .4 minutes per seed, SGHMC takes z 20 minutes per seed, and LAPLACE takes ze minutes per seed. Assuming (i) you train SGD and MFV with 5 different soods, (ii) you evaluate using scods: the approximate time run the CTFAR10 experiment
(3.6 + 4.6) x + 240 + (0..3 + 0.3 + 0.4 + 20 + 5) x 33x
96
~ 129.5 hours
training
# datasets
McdMNIST Experiment For training, SGD takes As 2.6 minutes per soed, MFV takes minutes per seed, and SGHMC takes xs 5 hours. For evaluation, SGD takes .4 minutes per seed, MFV takes z 8 minutes per seed, ENS takes R 77 minutes per seed, SGHMC takes Rs 30 minutes per seed, and LAPLACE takes ss 5 minutes per seed. Assuming (i) you train SGD and MFV with 5 different seeds, (ii) you evaluate using 3 seeds: the approximate time to run the MedMNIST experiment is
(2.6 + 5) x 5 300) + (0..4 + 0.8 0.7 +0) x 3x
2
)= 7.25 hours
training
# datasets
22

---

Published in Transactions on Machine Learning Research (02/2024)
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 CIFARIO-Corrupted Box Plot Results
SGHMC
LAPLACE
1.0
1,0
2.0
1.0
1.0
0.9
0.9
0.9
0.9
0.9
0.8
Coveraye
Coverege
Coerage
0.7
0.6
0.6
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.5
cred
Commuition
Gomupion Intendly
Commption
Comuppion
Geruuiie
ciecd
cred the
cred
thr
thr
thr
505
3.25
rps
aps
2ps
3.00
2.75
Gre
Set 50z
2.25
2.00
1.75
Internalyy
Figure 12: Box plot results for the CIFAR10 experiment with an error tolerance of 0.05. Note that some outliers occur that are not shown for coverage for case of visualization.
23

---

Published in Transactions on Machine Learning Research (02/2024)
SGD
ENS
MFV
5GHMC
LAPLACE
0.90
0.90
0.90
0.90
0.90
0.85
0.85
0.85
0.85
0.05
ComM
Cooorape
Consaae
0.80
0.80
0.80
0.70
0.05
crel
0.65
crod
0.05
crrd
0.05
crad
0.65
crod
thr
thr
thr
thr
aps
403
206
aps
0.60
2
0.60
3
4
5
0.00
2
3
0.60
0.60
Comuption Intensity
Corruption Intersity
Cormution Intesity
Compption Incensty
Comppt Intersity
crmd
cred
10-
10-
the
the
thr
apa
ap9
ups
ups
7
S45
Set
Intersht
Figure 13: Box plot results for the CIFAR10 experiment with an error tolerance of 0.05. Note that some outliers occur that are not shown for coverage for ease of visualization.
24

---

Published in Transactions on Machine Learning Research (02/2024)
F.2 LFFAR1O-Corrppee Der-Dataset Results
0.05 Error Tolerance
Brightne64
Cantraat
SGD
ENS
MFV
SGHMC
LAPLACE
5GD
ENS
HFV
SGHMC
LAPLACE
Ang Soudery
AND ACCANT
OE TOTEE
Courg D1
Courgg
Decion
Hoallyy
SCHMC
Jog AOOURR
21
PYg
Frest
Ang COUPRR
Avy war.rngatt
WO CHAPERTN
Figure 14: CIFAR1O-Corupted per-dataset results the 0.05 Error Tolerance.
25

---

Published in Transactions on Machine Learning Research (02/2024)
0.05 Error Tolerance
2re Oroourr Po.
(eg 212a
MMC
Goussian Neive
Impulee Notor
540
ENS
MFV
SGHMC
54D
LAPLACE
Lig.
Pg Contrig Ph
Am Cropee thi
Pg Contigg Ba!
Are Donnaar
CONPUN Ta
Cangle
Jeeg Compression
Medion Blu
5GD
ENS
MPV
SGHMC
LAPLACE
SGD
ENS
HFV
SGHMC
LAPLACE
Aig
Pig Ginangg Fi
Ch
Arg Sortee
Figure 15: CIFAR1O-Coruupee per-dataset results at the 0.05 Error Tolerance.
26

---

Published in Transactions on Machine Learning Research (02/2024)
0.05 Error Tolerance
PPg CHePee Do
CCHPOC IS
MRC Tm
Shot Nobe
Snow
560
EN5
MFV
SGHMC
5G0
LAPLACE
Ar Crepee thi
Pg GoiPge Bh!
Pg CONPANN Eh!
Are
Agg.
Arg
Ang M113
Amg
Cosele
Spatter
Speckle Nolse
SGD
ENS
MFV
SGHMCC
LAPLACE
5GD
ENS
MFV
SGHMC
LAPLACE
Courgge
F
Ing AAII
Sorte
Figure 16: CIFAR1O-Coruupee per-dataset results at the 0.05 Error Tolerance.
27

---

Published in Transactions on Machine Learning Research (02/2024)
0.05 Error Tolerance
Dvg Cheerna PO
Pig Cremie To
>g Gherge
1
14C 141-
Figure 17: CIFARIO-Corupted per-dataset results the 0.05 Error Tolerance.
28

---

Published in Transactions on Machine Learning Research (02/2024)
0.01 Error Tolerance
29g Craerge
68
MMC
Defocua
Eleati::
MFV
SGNMC
LAPLACE
54D
LAPLACE
Are Crumae Bh
Agg Gorinng
Are Ornnap
Pgg
R
Caseli
Pag
Prost
SGD
ENS
MFV
SGHMCC
LAPLACE
SGD
ENS
MPV
SGHMC
LAPLACE
No Vo-Maare
CINERCT TA
481 11
F
2
Ing AEIM
Figure 18: CIFAR1O-Corupped per-dataset results at the 0.01 Error Tolerance.
29

---

Published in Transactions on Machine Learning Research (02/2024)
0.01 Error Tolerance
2re Oroourr Po.
(eg 212a
MMC
Goussian Neive
Impulee Notor
540
ENS
MFV
SGHMC
54D
LAPLACE
Lig.
Pg Contrig Ph
Am Cropee thi
Pg Contigg Ba!
Are Donnaar
CONPUN Ta
Cangle
Jeeg Compression
Medion Blu
5GD
ENS
MPV
SGHMC
LAPLACE
SGD
ENS
HFV
SGHMC
LAPLACE
Aig
Pig Ginangg Fi
Ch
Arg Sortee
Figure 19: CIFAR1O-Coruupee per-dataset results at the 0.01 Error Tolerance.
30

---

Published in Transactions on Machine Learning Research (02/2024)
0.01 Error Tolerance
PPg CHePee Do
CCNPOC IS
MRC Tm
Shot Nobe
Snow
560
EN5
MFV
SGHMC
5G0
LAPLACE
Ar Creeee thi
Pg GoiPge Bh!
Pg CONPANN Eh!
Are
Agg.
Arg
Ang M113
Amg
Cosele
Spatter
Speckle Nolse
SGD
ENS
MFV
SGHMCC
LAPLACE
5GD
ENS
MFV
SGHMC
LAPLACE
Courgge
F
Ing AAII
Sorte
Figure 20: CIFAR1O-Corupped per-dataset results at the 0.01 Error Tolerance.
31

---

Published in Transactions on Machine Learning Research (02/2024)
0.01 Error Tolerance
SUHNNC
LAFLACE
Figure 21: CIFAR1O-Corrupted per-dataset results the 0.01 Error Tolerance.
32

---

Published in Transactions on Machine Learning Research (02/2024)
F.3
MedMNIST Results With Variance Over Calibration / Test Splits
Train Method
Error
Dataset
Pred-
SGD
ENS
MFV
SGHMC
Set Method
Credible (cred)
94.98 - 0.0005
97.02 0.0005
99.21 0.0
98.92 0.0
Threshold (thr)
95.41 0.0006
94.64 0.0066
94.85 0.0
Adaptive (aps)
95.67
+ 9 0.0124
94.61 + 0.0038
95.00 0.0083
0.05
Credible (cred)
0.0
76.52
92.16
86.27 0.0
organSmnist
Threshold (thr)
69.65 0.052
64.86 = 0.035
66.46 + 0.024
64.70 0.0
Adaptive (aps)
77.63 = 0.026
80.10 = 0.0329
84.21 = 0.006
80.14 0.004
Credible (cred)
97.46 + 0.0004
98.85 + 0.0005
99.85 + 010003
99.59 + 0.0
organ Cmnist
Threshold (thr)
99.06 0.004
99.13 + 0.0073
98.93 + 0.0068
98.88 0.0
Adaptive (aps)
99.43 + 0.002
99.37 + 0.0052
99.41 + 0.0020
99.15 + 0.0024
0.01
Credible (cred)
81.64 0.0
88.48 + 0.0
98.61 + 0.0
94.00 + 0.0
organSmnist
Threshold (thr)
91.54 + 0.03
91.52 + 0.05
89.94 = 0.05
86.84 = 0.0
Adaptive (aps)
95.86 0.01
95.07 - 0.037
96.42 + 0.008
92.10 + 0.01
Table 6: MedMNIST Coverage with Variances
Train Method
Error
Dataset
Pred-
SGD
ENS
MFV
SGHMC
Set Method
Credible (cred)
1,27 = 0.002
1.41 + 0.002
2.90 + 0.0038
1.87 + 0.0
organ organCmnin Cmnnist
Threshold (Ihr)
1.31 + 0.132
1.18 + 0.066
1.21 + 0.0501
1.13
Adaptive (aps)
1.63 + 0.155
1.56
1.97 0.0354
1.56 0.031
0.05
Credible (cred)
0.0
0.0
4.79 0.0
0.0
organSmnist
Threshold (thr)
2.24 0.589
1.59
1.50 0.122
1.37
Adaptive (aps)
3.57 0.48
3.23
3.26 + 0.073
2.59 0.053
Credible (cred)
1.83 + 0.0052
5.44 0.0093
2.91 0.0
organ Cmnist
Threshold (thr)
3.05 + 0.6439
2.75 0.854
2.61 0.6606
1.91 0.0
Adaptive (aps)
4.38 0.9158
3.82 + 1.6073
4.26 0.3742
2.61 + 0.2080
0.01
Credible (cred)
4.24 0.0
5.06 0.0
8.07 + 0.0
4.83 0.0
organSmnist
Threshold (thr)
7.14 1.05
6.55 + 2.30
4.21 +
3.0
Adaptive (aps)
8.13 + 0.58
7.72 + 1.56
6.70 0.45
4.39 + 0.30
Table 7: MedMNIST Set Size with Variances
33


Reviews:
[{"review_id": "BZFUiuDJtZ", "reviewer": "Reviewer_JLk9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \nThe authors consider a very relevant question: does conformal prediction perform well in realistic scenarios? While they do not conclusively answer this question in the negative, they demonstrate some fairly simple scenarios where conformal prediction fails and where simple Bayesian credible sets obtained from variational Bayesian neural networks outperform it.\n\nThe paper is generally well-written, and the authors' arguments are easy to follow. In particular, the authors explain the core concepts well, such as calibration and conformal prediction; the problem they study is well-motivated, and the presentation and intuitive explanation of the observed phenomena is clear. Furthermore, the figures are very nicely done.\n\nI also looked at the appendix, which is very nicely laid out and seems to provide a lot of additional detail for the theory and the experiments, thus leaving little doubt about the replicability of the paper.\n\n## Weaknesses\nThough the authors acknowledge it, the paper's main weakness is its scope seems quite limited. \n\nWhile the authors diagnose and explain the issue well, they don't offer much of a remedy in the paper. In particular, I don't have much expertise in conformal prediction; my expertise is aligned a lot closer with Bayesian inference, and thus, my takeaway from the paper is essentially that conformal prediction methods are currently not worth the extra computational effort when the simple MFVI method with credible sets outperforms them. Hence, it would be good if the authors could at least discuss potential ways to fix the issues of conformal prediction when the necessary assumptions for its guarantees are broken. \n\nBesides this, there are some smaller issues in the main text:\n- The notation $\\hat{p}(x)$ is a bit confusing, as it conflicts with the probability notation later; perhaps consider using $\\pi(x)$ instead.\n- the index of $p_i$ in the first equation in section 2.2 should be $p_k$\n- the third point in the list in section 2.3 doesn't seem to be an assumption\n- eq 5 is missing a $dw$ term\n- $p(w | D) \\approx w_{MAP}$ should be $p(w | D) \\approx \\delta(w - w_{MAP})$\n- in the argmax, please use curly braces, as it is a function that selects the element of a set.\n- eq 7: \"kl between variational and prior\" should read \"variational posterior\"\n- footnote 2: \"knowledge is known\"\n- The color assignment between Figs 2a and 2b is inconsistent; please ensure the same method is assigned the same color in both plots.\n- In the Figures, what is the \"average\"? Is it a mean or median? Could the authors show error bars?\n\n## Questions\nI am quite surprised at how well MFVI performs; I would be interested in how well a Laplace-approximated BNN (see e.g., [1]), a more state-of-the-art approximate Bayesian model, would perform.\n\n## References\n[1] Immer, A., Korzepa, M., & Bauer, M. (2021, March). Improving predictions of Bayesian neural nets via local linearization. In International conference on artificial intelligence and statistics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors conduct an empirical study of the behavior of conformal prediction methods when the usual assumptions required for its theoretical guarantees are broken, such as when there is an unknown distribution shift between training and test sets. In the paper, the authors consider image classification on CIFAR-10 and medical MNIST with neural networks and apply conformal prediction techniques to:\n1. Neural networks optimized using a MAP objective with SGD.\n2. Deep ensembles.\n3. Mean-field (MF) variational Bayesian neural networks.\n\nThe authors compare the coverage of conformal predictive sets against Bayesian credible sets. Their experiments show that the credible set approach using the MF approximation can significantly outperform any conformal prediction approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should fix the errors I describe in the weaknesses section. I would be delighted if the authors added more discussion regarding potential fixes or added more experiments, but these are not critical.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hFtgbhRNr1", "reviewer": "Reviewer_NvAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Clarity of presentation: the paper is well written.\n2. Motivation: the interaction of conformal prediction with Bayesian techniques is relevant and important.\n3. Practicality: the results presented in the paper may guide practitioners to a more responsible use of conformal prediction in real-world scenarios.\n\nWeaknesses:\n1. Off-the-shelf CP is not designed for OOD calibration, so why should it to work?\n2. I am not sure I understand how the intuitive explanation provided in Sec. 3.1 applies to OOD calibration.\n\nI will expand on these questions and I am looking forward to discussing with the authors to better understand their intention and whether I am missing parts of the story.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission proposes an empirical evaluation of the effects of conformal prediction (CP) calibration techniques on out-of-distribution (OOD) coverage for predictors learned with stochastic gradient descent or Bayesian deep learning techniques. Experiments are carried on the CIFAR10 and CIFAR10-Corrupted as well as on the organCmnist and organSmnist datasets. Results show that depending on the choice of model, calibration technique, and desired miscoverage level, CP can reduce out-of-distribution coverage.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Off-the-shelf CP for OOD calibration\n\nI recognize the practical importance of showing the effects of CP on OOD calibration for Bayesian deep learning approaches. However, this is not the intended use of the CP methods used in the paper. Is the message here that practitioners should be careful when using off-the-shelf CP to provide claims on the reliability of their models? Using a calibration set drawn from a different distribution does not guarantee coverage, and that is okay.\n\nIn Sec. 3.1, the sentence \"However, this could [...] more confident than it *should* be.\" is confusing because CP yields a predictor that is as confident as a user asks it to be given a calibration set. If the calibration set does not represent the distribution the model is supposed to work on, and we do not know anything about this distribution, then there is little to hope for. Maybe a more appropriate approach would be to use online CP methods that adapt over time?\n\nSimilarly, in the Conclusion paragraph of Sec. 5, I am not sure the claim that \"conformal prediction can cause unintended consequences\" reflects that the CP methods used in the paper are not intended for OOD calibration.\n\n2. Intuitive explanation\n\nI agree that if I am overconfident (i.e., I undercover), CP will yield a larger prediction set. On the other hand, if I am underconfident (i.e., I overcover), CP will yield a smaller prediction set. But still, this applies to the calibration set.\n\nI am not sure I follow how the experimental results show this is the underlying mechanism that drives OOD coverage. The figures in the paper seem to show the average set size tends to increase with the level of corruption, which is not immediately obvious to me. There may be OOD distributions where the model is very confident in the wrong answer, and the average set size may not change? Is this phenomenon related to the specific models and datasets used in the experiments?\n\nThe intuitive explanation for why CP may hurt OOD calibration sounds a little like \"the trivial set predictor that includes all possible labels provides 100% OOD coverage\".\n\nAm I misunderstanding the story here? I am looking forward to hearing the authors' perspective.\n\n3. Miscellaneous comments:\n\n* Typo on page 5: \"overly confident in *its* predictions\" -> \"overly confident in *their* predictions\"\n* Eq. (6): what does the $\\overset{(i)}{=}$ symbol mean?\n* Fig. 3: Could the \"Corruption intensity\" labels be repeated across columns?\n* First paragraph of Sec. 4: \"that were produced for every single corruped CIFAR10 dataset\". Could this sentence be rephrased to \"that were produced for every single CIFAR10-Corruped dataset\" to clarify this is an existing dataset?\n* Claim at the bottom of page 7: \"For example, if one is looking for [...]\". Deep ensembles with *aps* at the 0.01 error tolerance do not always provide coverage greater than 96%? The line crosses 95% at corruption intensity 5. The average set size seems to be slightly larger than deep ensembles with *thr*? Both lines seems to overlap significantly and they are hard to distinguish.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PFVmvAFFfd", "reviewer": "Reviewer_dkAf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Combining Bayesian inference for accounting uncertainty and conformal predictions for ensuring in-distribution calibration is an intriguing possibility. However, the present work is a rather lightweight treatment of this intersection, since the authors simply apply the techniques in a sequential manner: Standard conformal prediction technique is used on top of predictions for Bayesian neural networks. The specific research question is how the combination works for out-of-distribution samples.\n\nMy first problem with this is that the approach is scientifically questionable. As the authors clearly say, the guarantees of conformal predictions do not hold for out-of-distribution samples and here the authors measure their quality specifically in this setting. Since conformal predictions are used as postprocessing there is no reason to believe the predictions would be good for out-of-distribution samples irrespective of how the initial predictions were formed. In other words, I do not see a clear theoretical justification for the approach. I understand the work is framed from the perspective of evaluating how this would work, rather than proposing it as a new approach, but I would still expect the approach being evaluated to be one that has some justification for why it could work. Why not consider conformal predictions methods that attempt to account for the out-of-distribution samples? The authors already cite works that present such extensions, such as Barber et al. (2023).\n\nThe main empirical result follows from known properties of conformal predictions for out-of-distribution samples. By definition, we need higher prediction uncertainty to model samples that do not follow the training distribution. Conformal prediction methods form sets that capture uncertainty well for in-distribution samples, and hence by definition will improve coverage of out-of-distribution samples when the initial predictions were overconfident and degrade it when the initial predictions were underconfident. The authors obverse this to be the case and provide the intuitive explanations, but the observations are as everyone would expect. For instance, SGD is obviously overconfident and conformal prediction then naturally helps also for out-of-distribution samples. I find it hard to believe this would be a new insight for someone familiar with the topic.\n\nAnother problem relates to scoping. Starting from the title, the paper talks about Bayesian deep learning. However, it is very shallow in terms of actual Bayesian DL. One of the so-called BDL methods is standard SGD, the other is an ensemble that was originally proposed specifically as a non-Bayesian alternative for quantifying uncertainty, and the final one is standard mean-field variational approximation that indeed is a BDL method but by no means state-of-the-art. In summary, the authors effectively compare one Bayesian method with two non-Bayesian methods and hence the paper is mispositioned in terms of title and story. In addition, the authors miss an opportunity of evaluating how alternative BDL methods (e.g. Laplace approximation, SGLD, Monte Carlo dropout, flow-based VI etc) would behave in this sense. There are well-known differences between the various approximations in terms of the predictive uncertainty calibration, and empirical quantification of those in conjunction with conformal predictions would provide results that would no longer be obvious and hence would be of interest for some readers.\n\nThe experiments are sound, but a bit shallow as the authors only inspect the problem from one perspective. I see no added value of explaining use of the exact same techniques on two different data sets as two separate experiments with detailed verbose explanations. Both experiments answer to the same general question and could well be combined into one that provides the numerical results for not just these two data sets but for a broader range of data sets, so that the observations could be stated in a more general form.\n\nFinally, I would like to commend the authors on a well-prepared Supplement. Explicit listing of software and their licenses is exemplary and I appreciate the detailed analysis of computing resources described in Appendix E.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyses the combined effect of using Bayesian inference for parameter estimation and conformal prediction for calibrating the predictions in the context of neural networks, specifically from the perspective of how well the predictions for out-of-distribution samples behave.\n\nThe two techniques are used in sequential manner and the main result of the work is observing that conformal predictions hurt when the original predictions were underconfident and help when they were overconfident. This happens because out-of-distribution samples require higher prediction uncertainty and conformal predictions ignore this but merely try to correct the confidence for the training distribution.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I believe the paper would require substantial changes to warrant publication. It should consider broader range of actual Bayesian DL techniques or alternatively it should not be presented from the perspective of Bayesian DL at all. More importantly, I feel that the overall approach of merely running standard conformal prediction as postprocessing for Bayesian predictions is not scientifically interesting, and to fix this the authors would need to go deeper into how out-of-distribution samples should be handled in this setting. The current results would serve as good baselines.\n\nMinor remarks for polishing:\n- The evaluation measures should be provided more formally, in form of equations\n- The Related work section would work better a bit later in the paper, when the reader knows the methods (Sections 2.2-2.4) better\n- Fig 4(a) and 5 are sightly confusing; these are not line plots and the linear visual style misleads the reader, even though I understand the attempt to make the visually similar to Fig 2 and 3", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After a lively discussion with the reviewers, it seems that the paper is generally interesting enough for the TMLR audience, but that the claims should be slightly modified to make sure that everyone feels confident that they are supported by evidence. Especially based on comments from reviewer NvAv, it is suggested that the claims of the paper be rephrased along the following lines:

- Clearly state and remark that the proposed explanation follows by definition of marginal coverage. For a fixed model, the predictive set with highest chances of increasing OOD coverage will always have largest average set size. According to the proposed explanation, for a fixed model, increasing the chances of OOD coverage without any knowledge on the test distribution will always be at the expense of average set size. This applies to Bayesian DL as well as CP, and not the latter only.

- Remark that across all experiments, the combination of model and prediction set that provides the highest OOD coverage is always the "least confident" (in terms of average set size), with average set sizes that are larger than half of the total possible classes. It is true that these choices will increase the chances of OOD coverage, at the same time, it needs to be acknowledged that a model that predicts more than half of the possible classes is of questionable utility.

- It is stated several times across the manuscript that the main motivation is to quantify the extent to which CP hurts OOD coverage compared to credible predictive sets. Besides including the bar plots in Fig. 4,6, these quantitative findings are not discussed in the text. In my discussion with the authors, it was mentioned "for example, for ~1.25 decrease in average set size, CP with THR causes SGHMC coverage to go down by around 30% (see Figure 3).". This analysis is missing from the current version of the manuscript, and it should be included, in particular with respects to the correlation between differences in OOD coverage and in average set size.

Since these all seem like minor modifications, I am happy to accept with minor revision and check the camera-ready version to make sure the claims have been adjusted.
Decision: Accept (poster)