Conformal Bayesian Computation

1 Introduction . We consider Bayesian prediction using training data Z1 : n = { Yi , Xi } i=1 : n for an outcome of interest Yi and covariates Xi ∈ Rd . Given a model likelihood fθ ( y | x ) and prior on parameters , π ( θ ) for θ ∈ Rp , the posterior predictive distribution for the response at a new Xn+1 = xn+1 takes on the form p ( y | xn+1 , Z1 : n ) = ∫ fθ ( y | xn+1 ) π ( θ | Z1 : n ) dθ , ( 1 ) where π ( θ | Z1 : n ) is the Bayesian posterior . Asymptotically exact samples from the posterior can be obtained through Markov chain Monte Carlo ( MCMC ) and the above density can be computed through Monte Carlo ( MC ) , or by direct sampling from an approximate model . Given a Bayesian predictive distribution , one can then construct the highest density 100× ( 1−α ) % posterior predictive credible intervals , which are the shortest intervals to contain ( 1 − α ) of the predictive probability . Alternatively , the central 100 × ( 1 − α ) % credible interval can be computed using the α/2 and 1− α/2 quantiles . Posterior predictive distributions condition on the observed Z1 : n and represent subjective and coherent beliefs . However , it is well known that model misspecification can lead Bayesian intervals to be poorly calibrated in the frequentist sense ( Dawid , 1982 ; Fraser et al. , 2011 ) , that is the long run proportion of the observed data lying in the ( 1− α ) Bayes predictive interval is not necessarily equal to ( 1− α ) . This has consequences for the robustness of such approaches and trust in using Bayesian models to aid decisions . Alternatively , one can seek intervals around a point prediction from the model , ŷ = µ̂ ( x ) , that have the correct frequentist coverage of ( 1 − α ) . This is precisely what is offered by the conformal prediction framework of Vovk et al . ( 2005 ) , which allows the construction of prediction bands with finite sample validity without assumptions on the generative model beyond exchangeability of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data . Formally , for Zi = { Yi , Xi } i=1 : n , Zi iid∼ P and miscoverage level α , conformal inference allows us to construct a confidence set Cα ( Xn+1 ) from Z1 : n and Xn+1 such that P ( Yn+1 ∈ Cα ( Xn+1 ) ) ≥ 1− α . ( 2 ) Note that P is over Z1 : n+1 , that is we also treat Xn+1 as random ; see Lei and Wasserman ( 2014 ) for a discussion on the difficulties of attaining coverage conditional on Xn+1 . In this paper we develop computationally efficient conformal inference methods for Bayesian models including extensions to hierarchical settings . A general theme of our work is that , somewhat counter-intuitively , Bayesian models are well suited for the conformal method . Conformal inference for calibrating Bayesian models was previously suggested in Melluish et al . ( 2001 ) , Vovk et al . ( 2005 ) , Wasserman ( 2011 ) and Burnaev and Vovk ( 2014 ) , where it is referred to as “ de-Bayesing ” , “ frequentizing ” and “ conformalizing ” , but only in the context of conjugate models . Here , we present a scalable MC method for conformal Bayes , implementing full conformal Bayesian prediction using an ‘ add-one-in ’ importance sampling algorithm . The automated method can construct conformal predictive intervals from any Bayesian model given only samples of model parameter values from the posterior θ ∼ π ( θ | Z1 : n ) , up to MC error . Such samples are readily available in most Bayesian analyses from probabilistic programming languages such as Stan ( Carpenter et al. , 2017 ) and PyMC3 ( Salvatier et al. , 2016 ) . We also extend conformal inference to partially exchangeable settings which utilize the important class of Bayesian hierarchical models , and note the connection to Mondrian conformal prediction ( Vovk et al. , 2005 , Chapter 4.5 ) . Previously , the extension of conformal prediction to random effects was introduced in Dunn et al . ( 2020 ) in a non-Bayesian setting , with a focus on prediction in new groups , as well as within-group predictions without covariates . We will see that the Bayesian hierarchical model allows for a natural sharing of information between groups for within-group predictions with covariates . We discuss the motivation behind using the Bayesian posterior predictive density as the conformity measure for both the Bayesian and the frequentist , and demonstrate the benefits in a number of examples . 1.1 Background . The conformal inference framework was first introduced by Gammerman et al . ( 1998 ) , followed by the thorough book of Vovk et al . ( 2005 ) . Full conformal prediction is computationally expensive , requiring the whole model to be retrained at each test covariate xn+1 and for each value in a reference grid of potential outcomes , e.g . y ∈ R for regression . This makes the task computationally infeasible beyond a few special cases where we can shortcut the evaluation along the outcome reference grid , e.g . ridge regression ( Vovk et al. , 2005 ; Burnaev and Vovk , 2014 ) and Lasso ( Lei , 2019 ) . Shrinking the search grid is possible , but still requires many refittings of the model ( Chen et al. , 2016 ) . The split conformal prediction method ( Lei et al. , 2018 ) is a useful alternative method which only requires a single model fit , but increases variability by dividing the data into a training and test set that includes randomness in the choice of the split , and has a tendency for wider intervals . Methods based on cross-validation such as cross-conformal prediction ( Vovk , 2015 ) and the jacknife+ ( Barber et al. , 2021 ) lie in between the split and full conformal method in terms of computation and average length of the intervals . A detailed comparison of various conformal methods are provided in Barber et al . ( 2021 , Section 4 ) . A review of recent advances in conformal prediction is given in Zeni et al . ( 2020 ) , and interesting extensions have been developed by works such as Tibshirani and Foygel ( 2019 ) ; Romano et al . ( 2019 ) ; Candès et al . ( 2021 ) . 2 Conformal Bayes . 2.1 Full Conformal Prediction . We begin by summarizing the full conformal prediction algorithm discussed in Vovk et al . ( 2005 ) ; Lei et al . ( 2018 ) . Firstly , a conformity ( goodness-of-fit ) measure , σi : = σ ( Z1 : n+1 ; Zi ) , takes as input a set of data points Z1 : n+1 , and computes how similar the data point Zi is for i = 1 , . . . , n + 1 . A typical conformity measure for regression would be the negative squared error arising from a point prediction − { yi − µ̂ ( xi ) } 2 , where µ̂ ( x ) is the point predictor fit to the augmented dataset Z1 : n+1 , assumed to be symmetric with respect to the permutation of the input dataset . The key property of any conformity measure is that it is exchangeable in the first argument , i.e . the conformity measure for Zi is invariant to the permutation of Z1 : n+1 . Under the assumption that Z1 : n+1 is exchangeable , we then have that σ1 : n+1 is also exchangeable , and its rank is uniform among { 1 , . . . , n+ 1 } ( assuming continuous σ1 : n+1 ) . From this , we have that the rank of σn+1 is a valid p-value . If we now consider a plug-in value Yn+1 = y ( where Xn+1 is known ) , we can define the rank of σn+1 among σ1 : n+1 as r ( y ) = 1 n+ 1 n+1∑ i=1 1 ( σi ≤ σn+1 ) . For miscoverage level α , the full conformal predictive set , Cα ( Xn+1 ) = { y ∈ R : r ( y ) > α } , ( 3 ) satisfies the desired frequentist coverage as in ( 2 ) under the exchangeability assumptions above . Intuitively , we are reporting the values of y which conform better than the fraction α of observed conformity scores in the augmented dataset . A formal proof can be found in Vovk et al . ( 2005 , Chapter 8.7 ) . For continuous σ1 : n+1 , we also have from Lei et al . ( 2018 , Theorem 1 ) that the conformal predictive set does not significantly over-cover . In practice , beyond a few exceptions , the function r ( y ) must be computed on a fine grid y ∈ Ygrid , for example of size 100 , in which case the model must be retrained 100 times to the augmented dataset to compute σ1 : n+1 , with plug-in values for yn+1 on the grid . This is illustrated in Algorithm 1 below . We note here that the grid method only provides approximate coverage , as y may be selected even if it lies between two grid points that are not selected . This is formalized in Chen et al . ( 2018 ) , but we do not discuss this further . In Appendix D.6 , we provide an empirical comparison of the grid effects . This is also valid for binary classification where we now have a finite Ygrid = { 0 , 1 } , and so the grid method for full conformal prediction is exact and feasible . Observed data is Z1 : n , Xn+1 ; Specify miscoverage level α for each y ∈ Ygrid do Fit model to augmented dataset { Z1 , . . . , Zn , { y , Xn+1 } } Compute σ1 : n and σn+1 Store the rank , r ( y ) , of σn+1 among σ1 : n+1 end Return the set Cα ( Xn+1 ) = { y ∈ Ygrid : r ( y ) > α } . Algorithm 1 : Full Conformal Prediction 2.2 Conformal Bayes and Add-One-In Importance Sampling . In a Bayesian model , a natural suggestion for the conformity score , as noted in Vovk et al . ( 2005 ) ; Wasserman ( 2011 ) , is the posterior predictive density ( 1 ) , that is σ ( Z1 : n+1 ; Zi ) = p ( Yi | Xi , Z1 : n+1 ) for i ∈ { 1 . . . , n+ 1 } , where p ( Yi | Xi , Z1 : n+1 ) = ∫ fθ ( Yi | Xi ) π ( θ | Z1 : n+1 ) dθ . ( 4 ) We highlight that the datum Zi = { Yi , Xi } also lies in the conditioning Z1 : n+1 . This is a valid conformity score , as we have π ( θ | Z1 : n+1 ) ∝ π ( θ ) ∏n+1 i=1 fθ ( Yi | Xi ) , and so σ is indeed invariant to the permutation of Z1 : n+1 . We refer to this method as conformal Bayes ( CB ) - this is a special case of full conformal prediction , and we will see shortly that the exchangeability structure of Bayesian models is key to constructing conformity scores in the partial exchangeability scenario . Beyond conjugate models , we are usually able to obtain ( asymptotically exact ) posterior samples θ ( 1 : T ) ∼ π ( θ | Z1 : n ) , e.g through MCMC , where T is a large integer . Such samples are typically available as standard output from Bayesian model fitting . The posterior predictive can then be computed up to Monte Carlo error through p̂ ( Yn+1 | Xn+1 , Z1 : n ) = 1 T T∑ t=1 fθ ( t ) ( Yn+1 | Xn+1 ) . The key insight is that refitting the Bayesian model with { Z1 , . . . , Zn , { y , Xn+1 } } is well approximated through importance sampling ( IS ) , as only { y , Xn+1 } changes between refits . This leads immediately to an IS approach to full conformal Bayes , where we just need to compute ‘ add-one-in ’ ( AOI ) predictive densities . Here AOI refers to the inclusion of { Yn+1 , Xn+1 } into the training set , named in relation to ‘ leave-one-out ’ ( LOO ) cross-validation . Specifically , for Yn+1 = y and θ ( 1 : T ) ∼ π ( θ | Z1 : n ) , we can compute p̂ ( Yi | Xi , Z1 : n+1 ) = T∑ t=1 w̃ ( t ) fθ ( t ) ( Yi | Xi ) ( 5 ) where w̃ ( t ) are our self-normalized importance weights of the form w ( t ) = fθ ( t ) ( y | Xn+1 ) , w̃ ( t ) = w ( t ) ∑T t′=1 w ( t′ ) · ( 6 ) We see that the unnormalized importance weights have the intuitive form of the predictive likelihood at the reference point { y , Xn+1 } given the model parameters θ ( t ) . The use of AOI importance sampling has similarities to the computation of Bayesian leave-one-out ( LOO ) predictive densities for cross-validation ( Vehtari et al. , 2017 ) , which is also used in accounting for model misspecification . An interesting aspect of AOI in comparison with LOO is that AOI predictive densities are less vulnerable to importance weight instability for the following reasons : • In LOO , the target π ( θ | Z−i ) generally has thicker tails than the proposal π ( θ | Z1 : n ) , leading to importance weight instability . In contrast , AOI uses the posterior π ( θ | Z1 : n ) as a proposal for the thinner-tailed π ( θ | Z1 : n+1 ) . For LOO the importance weights are proportional to 1/fθ ( y | x ) , in contrast to the typically bounded fθ ( y | x ) for AOI . • For AOI , we are predicting Zi given Z1 : n+1 which is always in-sample unlike in LOO where the datum is out-of-sample , so we can expect greater stability with AOI . • The IS weight stability is governed by Yn+1 = y , which is not random as we select it for the grid , unlike in LOO cross-validation . For sufficiently large α , we will not need to compute the AOI predictive density for extreme values of y . The above discussion is a natural analogy to the discussion provided in Vehtari et al . ( 2017 , Section 2.1.1 ) for computing LOO predictives . Note that the LOO predictive density is also a valid conformity score ; we provide more details and verify the above comparisons between LOO and AOI empirically in Appendix B . We also provide some IS weight diagnostics in Section 4.1.1 and find that they are stable . In difficult settings such as very high-dimensions , one can make use of the recommendations of Vehtari et al . ( 2015 ) for assessing and Pareto-smoothing the importance weights if necessary .

Summary: This paper proposes a scalable method for conformal Bayesian prediction, by estimating the modified posterior predictive density using "add-one-in" importance sampling.  On several tabular datasets with sparse and hierarchical models, estimation is shown to be efficient for non-extreme miscoverage levels, and the produced credible intervals have good coverage.  **Post-rebuttal update:** Thank you for the response.  I am keeping my score unchanged: on the plus side, the method is sensible and easy to implement; on the other hand, there are some (understandable) limitations when scaling to large datasets or applied on approximate posterior samples; and some reviewers raised questions about novelty, which I'm not most suited to judge, as I'm not an expert in this field.  In aggregate, the rating of "marginally above the acceptance threshold" still describes my opinions well.