Title: PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Conformal Prediction

PAC Bayes

Generalization Theory
Abstract: Inductive Conformal Prediction (ICP) provides a practical and effective approach for equipping deep learning models with uncertainty estimates in the form of set-valued predictions which are guaranteed to contain the ground truth with high probability.
Despite the appeal of this coverage guarantee, these sets may not be efficient: the size and contents of the prediction sets are not directly controlled, and instead depend on the underlying model and choice of score function.
To remedy this, recent work has proposed learning model and score function parameters using data to directly optimize the efficiency of the ICP prediction sets.
While appealing, the generalization theory for such an approach is lacking: direct optimization of empirical efficiency may yield prediction sets that are either no longer efficient on test data, or no longer obtain the required coverage on test data.
In this work, we use PAC-Bayes theory to obtain generalization bounds on both the coverage and the efficiency of set-valued predictors which can be directly optimized to maximize efficiency while satisfying a desired test coverage.
In contrast to prior work, our framework allows us to utilize the entire calibration dataset to learn the parameters of the model and score function, instead of requiring a separate hold-out set for obtaining test-time coverage guarantees.
We leverage these theoretical results to provide a practical algorithm for using calibration data to simultaneously fine-tune the parameters of a model and score function while guaranteeing test-time coverage and efficiency of the resulting prediction sets.
We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP, especially in the low-data regime.

PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction

Abstract

Inductive Conformal Prediction (ICP) provides a practical and effective approach for equipping deep learning models with uncertainty estimates in the form of set-valued predictions which are guaranteed to contain the ground truth with high probability. Despite the appeal of this coverage guarantee, these sets may not be efficient: the size and contents of the prediction sets are not directly controlled, and instead depend on the underlying model and choice of score function. To remedy this, recent work has proposed learning model and score function parameters using data to directly optimize the efficiency of the ICP prediction sets. While appealing, the generalization theory for such an approach is lacking: direct optimization of empirical efficiency may yield prediction sets that are either no longer efficient on test data, or no longer obtain the required coverage on test data. In this work, we use PAC-Bayes theory to obtain generalization bounds on both the coverage and the efficiency of set-valued predictors which can be directly optimized to maximize efficiency while satisfying a desired test coverage. In contrast to prior work, our framework allows us to utilize the entire calibration dataset to learn the parameters of the model and score function, instead of requiring a separate hold-out set for obtaining test-time coverage guarantees. We leverage these theoretical results to provide a practical algorithm for using calibration data to simultaneously finetune the parameters of a model and score function while guaranteeing test-time coverage and efficiency of the resulting prediction sets. We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP, especially in the low-data regime.

1 Introduction

Machine learning (ML) models have rapidly proliferated across numerous applications, including safety-critical ones such as autonomous vehicles [Schwarting et al., 2018, Waymo, 2021], medical diagnosis [Ahsan et al., 2022], and drug discovery [Vamathevan et al., 2019]. Due to the severity of outcomes in these applications, decision making cannot solely hinge on 'point' predictions from the ML model, but must also encapsulate a measure of the uncertainty in the predictions. As a result, accurate uncertainty estimation is a cornerstone of robust and trustworthy ML systems; however, overly-optimistic or overly-conservative estimates limit their usefulness. In this paper, we will present a method that builds upon inductive conformal prediction (ICP) [Vovk et al., 2005] and probably approximately correct (PAC) Bayes theory [McAllester, 1998] to furnish uncertainty estimates that provably control the uncertainty estimate's conservatism while achieving desired correctness rates.

Asher Hancock

Anirudha Majumdar

compute threshold

fine-tune score function

re-compute threshold

learn prior

optimize posterior with KL constraint

Used to optimize efficiency

Used to guarantee coverage

Guaranteed test-time coverage

Directly optimized test-time efficiency

Standard ICP

Learned ICP + Recal.

PAC-Bayes ICP

this work

Legend:

ICP has emerged as a practical approach for equipping deep learned models with uncertainty estimates in the form of set-valued predictions with high-probability coverage guarantees, i.e., with high probability, at test time, the ground truth labels will lie within the predicted set. However, post-hoc application of ICP can often lead to overly-conservative set sizes; the tightness of the predicted set (referred to as efficiency) highly depends on the underlying model as well as the choice of the score function used in ICP. Various approaches have worked towards alleviating this challenge either by developing new application-specific score functions [Romano et al., 2020, Yang and Pavone, 2023, Lindemann et al., 2023] or by optimizing the model and / or score function directly [Yang and Kuchibhotla, 2021, Cleaveland et al., 2023, Bai et al., 2022]. The direct optimization approach is appealing as a general application-agnostic method for obtaining tight uncertainty estimates; however, it requires re-calibration on held-out data in order to retain generalization guarantees on coverage. By coalescing ICP with PAC-Bayes, we are able to train using all available data, while retaining generalization guarantees for coverage and efficiency; see Fig. 1 for more details.

Contributions: In this work, we make the following core contributions:

We use PAC-Bayes theory to obtain generalization bounds on both the coverage and efficiency of set-valued predictors. In contrast to prior work (see Fig. 1 and Sec. 3), our framework allows us to utilize the entire calibration dataset to learn the model parameters and score function (instead of requiring a held-out set for obtaining a guarantee on coverage).

We leverage these theoretical results to provide a practical algorithm (see Alg. 1) for utilizing calibration data to fine-tune the parameters of the model and score function while guaranteeing test-time coverage and efficiency. This algorithm allows the user to specify a desired coverage level, which in turn specifies the degree to which efficiency can be optimized (since there is a trade-off between the desired coverage level and efficiency).

We evaluate our approach on regression and image classification problems with neural networkbased score functions. We show that our method can yield more efficient predictors than prior work on learned conformal prediction calibrated using a Hoeffding-bound based PAC guarantee on coverage [Vovk, 2012, Prop 2a].

2 Background: Inductive Conformal Prediction

We consider a supervised learning setup, wherein our goal is to predict labels y ∈ Y given inputs x ∈ X . In inductive conformal prediction (ICP) [Vovk et al., 2005], our goal is to develop a set-valued predictor which maps inputs x to a subset of the label space C ( x ) ⊆ Y . Specifically, we want this set-valued function to satisfy a coverage guarantee, ensuring that the chance that the prediction set fails to contain the ground truth label is bounded to a user-specified level α , i.e.

where D is the joint distribution over inputs and labels that test data are drawn from. In practice, D is not known, but we assume we have access to a calibration dataset D cal = { ( x i , y i ) } N i =1 where each example ( x i , y i ) is exchangeable with the test data. Given this set-up, ICP constructs a set-valued predictor of the form

i.e., the τ sub-level set of the nonconformity function s : X × Y → R evaluated at the input x . The nonconformity function assigns a scalar value quantifying how poorly an example x , y conforms to the training dataset. It is common to define a nonconformity function that depends on the learned model f , choosing s ( x , y ) = ℓ ( f ( x ) , y ) , measuring how poorly the prediction f ( x ) aligns with the label y [Angelopoulos and Bates, 2021]. Armed with the calibration dataset as well as the nonconformity function, all that remains to construct the set-valued predictor is to choose the proper threshold τ . ICP defines a calibration strategy to choose τ that guarantees a desired miscoverage rate α . Specifically, let S cal = { s ( x , y ) | x , y ∈ D cal } and choose τ ∗ ( D cal , α ) to be the q = ⌈ ( n +1)(1 -α ) ⌉ /n quantile of the set S . Then, so long as the score function s is independent from D cal , we have the following guarantee on the prediction sets defined by (2):

Importantly, this probabilistic guarantee is marginal over the sampling of the test data point as well as the calibration dataset. In practice, we are given a fixed calibration dataset, and would like to bound the miscoverage rate conditioned on observing D cal . As shown in Vovk [2012, Prop 2a], we can obtain a probably-approximately-correct (PAC) style guarantee on the ICP predictor of the form

In other words, with probability at least 1 -δ over the sampling of a fixed calibration dataset, we can ensure the test-time miscoverage rate is bounded by α by calibrating for a marginal coverage rate of ˆ α = α -√ -log δ/ 2 N . Alternatively, one can achieve the same PAC guarantee using the tighter bound in [Vovk, 2012, Prop 2b] by choosing the largest ˆ α ∈ (0 , α ) for which

where I x ( a, b ) is the regularized incomplete beta distribution. This lacks an analytic solution for the optimal ˆ α , but is less conservative.

The strength of ICP lies in its ability to guarantee test-time coverage for any base predictor and nonconformity score function. However, ICP does not directly control the efficiency of the resulting prediction sets; the size and make-up of the prediction set for any given input x is highly dependent on the score function s (and thus also the base prediction model f ). For this reason, naïve application of ICP can often yield prediction sets that are too 'loose' to be useful downstream. Often, both the model and the score function may depend on various parameters θ ∈ R d , e.g., the weights of a neural network base prediction model. The values of these parameters influence the ICP procedure, from the computation of the score function s ( x , y ; θ ) , the resulting threshold τ ∗ ( D cal , α ; θ ) , and finally, the prediction sets themselves C ( x ; τ ∗ , θ ) . Our goal in this work is to investigate how we can use calibration data to fine-tune these parameters to optimize an efficiency objective while retaining guarantees on test-time coverage.

3 Related Work

Handcrafted score functions. In classification ( Y = [ K ] ), suppose f ( x ) predicts the softmax class probabilities; typically the score function is chosen as either s ( x , y ) = 1 -f ( x ) y or s ( x , y ) = ∑ k ∈ [ K ] { f ( x ) k | f ( x ) k ≥ f ( x ) y } , where f ( x ) y denotes the softmax probability of the groundtruth label. While the former score function produces prediction sets with the smallest average size [Sadinle et al., 2019], the latter produces prediction sets whose size can adapt to the difficulty of the problem [Romano et al., 2020]. In 1-D regression ( Y = R ), [Romano et al., 2019] recommend training f ( x ) = [ f α/ 2 ( x ) , f 1 -α ( x )] using quantile regression to output the (heuristic) α/ 2 and 1 -α/ 2 quantiles, and then set the score function as s ( x , y ) = max { f α/ 2 ( x ) -y, y -f 1 -α ( x ) }

to compute the distance from the groundtruth y to the prediction interval f ( x ) . In n -D regression ( Y = R n ), it is common to design s ( f ( x ) , y ) = ∥ µ ( x ) -y ∥ u ( x ) , where f ( x ) = ( µ ( x ) , u ( x )) outputs both the mean of the label µ ( x ) and a heuristic notion of uncertainty u ( x ) [Yang and Pavone, 2023, Lindemann et al., 2023]. While this leads to a ball-shaped prediction set, designing s ( f ( x ) , y ) = max i ∈ [ n ] {| µ ( x ) i -y i |} can produce a box-shaped prediction set [Bai et al., 2022].

Learning score functions. [Yang and Kuchibhotla, 2021] propose selection algorithms to yield the smallest conformal prediction intervals given a family of learning algorithms. [Cleaveland et al., 2023] consider a time series prediction setup and leverage linear complementarity programming to optimize a score function parameterized over multiple time steps. [Stutz et al., 2021] develop conformal training to shrink the prediction set for classification, which learns the prediction function and the score function end-to-end by differentiating through and simulating the conformal prediction procedure during training. [Bai et al., 2022] considers optimizing the efficiency of a parameterized score function subject to coverage constraints on a calibration dataset, and derives generalization bounds depending on the complexity of the parametrization. [Einbinder et al., 2022] focus the classification-specific adaptive predictive set (APS) score function [Romano et al., 2020] score function, and propose an auxiliary loss term to train the model such that its predictions more closely meet the conditions under which the APS score function yields optimal efficiency. The drawback of [Cleaveland et al., 2023, Stutz et al., 2021, Bai et al., 2022, Einbinder et al., 2022] is that a separate held-out dataset is required to recalibrate the optimized score function to obtain test-time coverage guarantees. In this paper, we leverage PAC-Bayes theory to alleviate this drawback and allow learning the parameters of the model and score function using the entire calibration dataset, while offering efficiency and coverage guarantees.

Generalization theory and PAC-Bayes. Generalization theory seeks to quantify how well a given model will generalize to examples beyond the training set, and how to learn models that will generalize well. Early work includes VC theory [Vapnik and Chervonenkis, 1968], Rademacher complexity [Shalev-Shwartz and Ben-David, 2014], and the minimum description length principle [Rissanen, 1989, Blumer et al., 1987]. In this work, we make use of PAC-Bayes theory [McAllester, 1998]. In PAC-Bayes learning, one fixes a data-independent prior over models and then obtains a bound on the expected loss that holds for any (potentially data-dependent) choice of posterior distribution over models. One can then optimize the PAC-Bayes bound via the choice of posterior in order to obtain a certificate on generalization. In contrast to bounds based on VC theory and Rademacher complexity, PAC-Bayes provides numerically strong generalization bounds for deep neural networks for supervised learning [Dziugaite and Roy, 2017, Neyshabur et al., 2017a,b, Bartlett et al., 2017, Arora et al., 2018, Rivasplata et al., 2019, Pérez-Ortiz et al., 2021, Jiang et al., 2020, Lotfi et al., 2022] and reinforcement learning [Fard et al., 2012, Majumdar et al., 2021, Veer and Majumdar, 2020, Ren et al., 2021]. We highlight that the standard framework of generalization theory (including PAC-Bayes) provides bounds for point predictors (i.e., models that output a single prediction for a given input). Here, we utilize PAC-Bayes in the context of conformal prediction to learn set-valued predictors with guarantees on coverage and efficiency.

4 PAC-Bayes Generalization Bounds for Inductive Conformal Prediction

Optimizing efficiency with ICP requires splitting the dataset to train on one part and calibrate on the other. The reduced data for calibration can result in weaker miscoverage guarantees. Drawing from PAC-Bayes generalization theory, we will develop a theory that facilitates simultaneous calibration and efficiency optimization for ICP using the entire calibration dataset.

Our generalization bounds are derived by randomizing the parameters of the score function θ . Let Q ( θ ) be a distribution over parameters θ . Note that, for a fixed target miscoverage rate ˆ α , every sample from θ induces a different prediction set C ( x ; τ ∗ ( D cal , ˆ α ; θ ) , θ ) . Test-time coverage for such a randomized prediction set corresponds to the probability of miscoverage, marginalizing over the sampling of θ :

The efficiency ℓ eff of a predicted set C is a metric that can encode task-specific preferences (e.g., the volume of the predicted set) that we wish to minimize while satisfying the desired miscoverage rate.

Similar to (6), we can define the efficiency measure L eff on randomized prediction sets by taking an expectation of a single-set efficiency measure:

Our core theoretical contributions are to provide bounds on L cov ( Q ) and L eff ( Q ) that can be computed using D cal and, critically, hold uniformly for all choices of Q , even those that depend on D cal . This allows us to learn the distribution Q over score function parameters using D cal . First, we state our generalization bound for test-time coverage.

Theorem 1 (PAC-Bayes Bound on Coverage of ICP Prediction Sets) . Let P ( θ ) be a (dataindependent) prior distribution over the parameters of the score function. Let D N be a set of N samples from D . Choose empirical coverage parameter ˆ α ∈ (0 , 1) such that N > 1 / ˆ α -1 . Then, with probability greater than 1 -δ over the sampling of D N , the following holds simultaneously for all distributions Q ( θ ) :

where kl( p || q ) is the KL divergence between Bernoulli distributions with success probability p and q respectively, and B ( N ) := Beta( k -1 N -1 , k, N +1 -k = O ( √ N/ (ˆ α (1 -ˆ α )))) , where k = ⌊ ( N +1)ˆ α ⌋

The proof for this theorem is presented in App. A 1 . This theorem states that applying the calibration procedure will yield a test-time coverage rate L cov ( Q ) close to ⌊ ( N +1)ˆ α ⌋-1 N -1 , and the amount that it can differ shrinks as the cardinality N of the calibration set increases, but grows as Q deviates from P . Finally, we note that an upper bound on L cov ( Q ) can computed by inverting the KL bound [Dziugaite and Roy, 2017] using convex optimization [Majumdar et al., 2021, Sec. 3.1.1].

Next, we will provide a generalization bound on efficiency.

Theorem 2 (PAC-Bayes Bound on Efficiency of ICP Prediction Sets) . Let P ( θ ) be a (dataindependent) prior distribution over the parameters of the score function. Let D N be a set of N samples from D and γ > 0 . Let the efficiency ℓ eff of a predicted set C always lie within 2 [0 , 1] . Assume the score function is bounded, s ( x , y ; θ ) < β , and the efficiency loss τ → ℓ eff ( C ( x ; τ, θ ) is L τ -Lipschitz continuous in τ for all values of θ . Then, with probability greater than 1 -γ , we have for all Q ,

The proof for this theorem follows a similar approach to standard PAC-Bayes bounds, but is complicated by the dependence of τ ∗ on D N . To mitigate this, we instead consider the generalization error between empirical and test-time efficiency for the worst-case choice of τ , which requires assuming τ is bounded and the efficiency objective smooth w.r.t. τ . The full proof is presented in App. A.

5 Practical Algorithmic Implementation

The generalization bounds above suggest a practical algorithm for using calibration data to simultaneously conformalize a base predictor while also fine tuning model and score function parameters. Specifically, we leverage the theory to formulate a constrained optimization problem, where the objective aims to minimize an efficiency loss, while the constraint ensures that test-time coverage can still be guaranteed. The overall algorithm is summarized in Alg. 1.

5e+01

5e+02

5e+03

5e+04

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

Empirical Miscoverate Rate ˆ

α

Target

α

= 0

.

10,

δ

= 0

.

05

0

1

2

5

10

20

50

100

200

500

1000

2000

5000

Vovk Prop 2a

Vovk Prop 2b

Cannot provide guarantee

5e-05

5e-04

5e-03

5e-02

5e-01

Target

α

= 0

.

10,

N

= 5000

0

2 1

5

10

20

50

100

200

500

0

2

10

50

200

1000

5000

KL Budget

Calibration Set Size

Delta

5.1 Guaranteeing test-time coverage via a KL constraint

Using the bound in Theorem 1, we derive a constraint over data-dependent posteriors Q ( θ ) such that choosing a threshold using the conformal prediction procedure τ ( θ, D N , ˆ α ) will guarantee a test-time coverage greater than 1 -α , where the coverage we enforce over D N , (1ˆ α ), maybe different from the coverage we wish to achieve at test time, 1 -α .

Corollary 2.1 (Constraint on Data-Dependent Posterior) . Fix ˆ α ≤ α , and a prior distribution P ( θ ) . Given a calibration dataset D cal of N i.i.d. samples from D , we have that with probability greater than 1 -δ , then simultaneously for all distributions Q ( θ ) which satisfy

it holds that L cov ( Q ) ≤ α .

The proof for this corollary is provided in App. A. Importantly, this bound holds for all Q ( θ ) that satisfy the KL constraint, including Q that depend on the calibration data D N . Thus, we are free to use any optimization algorithm to search for a feasible Q that minimizes any auxiliary loss; with B ( α, ˆ α, δ, N ) serving as a 'budget' for optimization, limiting the degree to which Q can deviate from the data-independent prior P . Fig. 2 visualizes this budget as a function of N , and ˆ α for a fixed α and δ . As the plots show, by choosing the threshold τ to attain a more conservative ˆ α , we can afford more freedom to vary Q . The curve defined by B ( α, ˆ α, δ, N ) = 0 indicates the maximum ˆ α for which our theory provides a guarantee on 1 -α test time coverage with probability greater than 1 -δ for a randomized data-independent score function ( Q = P ). As visualized in the figure, this boundary aligns with maximum ˆ α implied by the PAC guarantee from Vovk [2012, Prop 2a], where differences are likely due to differences in bound derivation. However, there remains a gap between our analysis and the results from Vovk [2012, Prop 2b], suggesting room for future tighter bounds.

5.2 Optimizing efficiency via constrained stochastic gradient optimization

In order to select a Q from this set, we propose directly optimizing the efficiency objective 3 (7). However, direct optimization is difficult due to (i) the expectation over x and θ , and (ii) the nondifferentiability of the efficiency objective L eff ( C ) . To address (i), we use minibatches of inputs sampled from D cal and a finite set of K samples of θ from Q ( θ ) to construct a Monte-Carlo approximation of the expectation, providing a stochastic estimate of the objective that can be used in a stochastic gradient descent algorithm. To ensure differentiability, we follow the strategy proposed in [Stutz et al., 2021] and replace any non-differentiable operations in the computation of L eff with their 'soft' differentiable counterparts. For example, the quantile operation needed to compute the optimal threshold τ ( θ, D cal , ˆ α ) can be replaced with a soft quantile implementation leveraging a soft sorting

Require: Parameterized score function s ( x , y , θ ) , Calibration dataset D cal , Target coverage rate 1 -α , Probability of correctness δ , Differentiable efficiency objective ℓ eff ( C ) , Prior distribution P ( θ ) = N ( θ ; µ 0 , Σ 0 ) , Posterior distribution family Q ( θ ) = N ( θ ; µ, Σ) , Empirical coverage 1 -ˆ α Initialize posterior distribution parameters

algorithm [Cuturi et al., 2019, Grover et al., 2019]. As optimizing over the space of all probability distributions over Θ is intractable, we restrict Q to be within a parametric class of distributions for which obtaining samples and evaluating the KL divergence with respect to the prior is tractable. In this work, we follow prior work in the PAC-Bayes and Bayesian Neural Network literature and fix both P and Q to be a Gaussian with a diagonal covariance matrix, N ( µ , diag( σ 2 )) , where µ ∈ R d and σ 2 ∈ R d + . This allows analytic evaluation of the KL divergence, and differentiable sampling via the reparametrization trick, as shown in Alg. 1. In this way, both the optimization objective and the constraint can be evaluated in a differentiable manner, and fed into a gradient based constrained optimization algorithm. We choose the Q with the lowest loss that still satisfies Corollary 2.1. More details on the optimization procedure can be found in Appendix B.

5.3 Practical considerations

Choice of prior. A prior P ( θ ) may give high likelihood to values of θ that yield good prediction sets, but also to θ that perform poorly. To optimize efficiency, Q must shift probability mass away from poor θ towards good θ . However, due to the asymmetry of the KL divergence, KL( Q || P ) is larger if Q assigns probability mass where P does not than vice-versa. Thus, in order to effectively optimize efficiency under a tight KL budget, we must choose P that not only (i) assigns minimal probability density to bad values of θ , but more importantly, (ii) assigns significant probability density to good choices of θ . How do we choose an effective prior? In some cases, it suffices to choose P as an isotropic Gaussian distribution centered around a random initialization of θ . However, especially when θ correspond to parameters of a neural network, using a data-informed prior can lead to improved performance [Perez-Ortiz et al., 2021, Dziugaite and Roy, 2018]. Care is needed to ensure that a data-informed prior does not break the generalization guarantee; we find that a simple data-splitting approach similar to [Perez-Ortiz et al., 2021] is effective: We split D cal into two disjoint sets: D 0 , used to optimize the prior without any constraints, and D N , used to optimize the posterior according to Alg. 1. In our experiments, we consider two methods of optimizing a Gaussian prior P = N ( µ , diag( σ 2 )) : First, we can optimize only the mean µ to minimize L eff ( µ ) , keeping σ 2 fixed. This aims to shift P to assign more density to good values of θ , but the fixed σ 2 may still assign significant mass to bad choices of θ . Alternatively, we consider optimizing both µ and σ 2 to minimize L eff ( P ) . By also optimizing the variance, we can minimize probability mass assigned to bad choices of θ , but this comes with a risk of overfitting to D 0 .

Test-time evaluation. The guarantees on coverage and efficiency generalization hold in expectation over both data sampled from the data distribution D as well as model and score function parameters sampled from the learned posterior Q . Thus, in order to use such a predictor in practice and attain the desired coverage and efficiency, one would need to sample parameters θ from Q , and then compute the

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

2

-

1

0

1

2

N=500

ˆ

α

:0.04

L

cov

:0.043

L

eff

:-0.168

Standard

Cal. data

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ˆ

α

:0.02

L

cov

:0.019

L

eff

:-0.455

Learned

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

ˆ

α

:0.03

L

cov

:0.026

L

eff

:-0.568

PAC-Bayes

-

2

-

1

0

1

2

N=5000

ˆ

α

:0.08

L

cov

:0.081

L

eff

:-0.352

ˆ

α

:0.07

L

cov

:0.077

L

eff

:-0.793

ˆ

α

:0.07

L

cov

:0.069

L

eff

:-0.765

.

50

0

.

75

1

.

00

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

resulting prediction set C ( x ; τ ) . However, as the threshold for this prediction set τ = τ ∗ ( D cal , ˆ α ; θ ) depends on D cal , this approach would require holding on to the calibration dataset to find the optimal threshold of each sample of θ . In practice, once we optimize Q , we pre-sample K values of θ from Q , and pre-compute the optimal threshold for each. Then, at test-time, we randomly select one θ, τ pair from the set to evaluate each test input.

6 Experimental Results

We evaluate our approach on an illustrative regression problem as well as on an MNIST classification scenario with simulated distributional shift. 4 For each domain, we evaluate our PAC-Bayes approach against two baselines: (i) the standard ICP approach, where we use a fixed, data-independent score function s default ( x , y ) and use the entirety of the calibration data to compute the critical threshold τ ; and (ii) a learned ICP approach [Stutz et al., 2021] which uses a portion of D cal to optimize parameters θ of a parametric score function s optim ( x , y ; θ ) to minimize an efficiency loss L eff , and the remaining portion to re-estimate the threshold τ to guarantee test-time coverage. For the PACBayes method, we consider the same parametric form as the learned baseline, but instead randomize the score function by modeling a distribution over θ . Both the learned and the PAC-Bayes baseline require choosing a data split ratio, i.e. what fraction is used for optimization (or prior tuning for the PAC-Bayes approach), and what fraction is held out for recalibration (or constrained optimization). For all methods, we target the same test-time coverage guarantee of 1 -α coverage with probability greater 1 -δ . For the standard and learned methods, the amount of data in the held-out recalibration set determines the empirical coverage level ˆ α needed to attain this guarantee, using either Vovk Prop 2a (4), or Vovk Prop 2b (5). We compare against both bounds in our experiments. For the PAC-Bayes method, the choice of ˆ α is a hyperparameter: choosing a lower value yields a larger budget for optimization, but entails using a more extreme quantile for the threshold τ . As all methods provide guarantees on test-time coverage, our evaluation focuses on the efficiency of each method's predictions on held-out test data that is exchangeable with the calibration data. Additional results (including coverage results), as well as specific details for all experiments, can be found in App. B.

6.1 Illustrative demonstration: 1-D regression

As an illustrative example, we consider a 1-D regression problem where inputs x ∈ R and targets y ∈ R are drawn from a heteroskedastic distribution where noise increases with x , as shown in Figure 3. We train a two hidden layer fully connected neural network to minimize the mean squared error on this data, obtaining a base predictor f ( x ) that produces a single point estimate for y . We aim

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

Average Prediction Set Size

Standard (Vovk 2a)

Standard (Vovk 2b)

Learned (Vovk 2a)

Learned (Vovk 2b)

PAC-Bayes

0.25

0.5

0.75

Data Split

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

2

.

00

Relative Prediction Set Size

Standard (Vovk 2a)

Standard (Vovk 2b)

Learned (Vovk 2a)

Learned (Vovk 2b)

PAC-Bayes

to obtain a set-valued predictor using a held-out set of calibration data with α = 0 . 1 and δ = 0 . 05 . For the sake of this example, we assume that we no longer have access to the training data, and only have calibration data and a pre-trained base predictor which only outputs point estimates and no input-dependent estimate of variance. Therefore, for the standard application of ICP, we must use the typical nonconformity function for regression, s default ( x , y ) = ∥ f ( x ) -y ∥ . Note that this choice yields prediction sets with a fixed, input-independent size, which is suboptimal in this heteroskedastic setting. To address this, we consider using the calibration dataset not only to determine the threshold τ ∗ , but also to learn a parametric score function that can capture this heteroskedasticity. Specifically, we learn an input-dependent uncertainty score u ( x ; θ ) which is used to scale the nonconformity score s optim ( x , y ; θ ) = ∥ f ( x ) -y ∥ /u ( x ; θ ) [Angelopoulos and Bates, 2021]. We model u ( x , θ ) as a separate neural network with the same architecture as f , and optimize either θ or Q ( θ ) to minimize the volume of the resulting prediction sets. To do so, we minimize a loss proportional to the log of the radius of this set, which in this case can be written as ℓ eff ( C ( x , τ )) = log( u ( x ; θ ) · τ ) .

As can be seen in Figure 3, when N = 5000 , both the learned baseline and our PAC-Bayes approach are able to learn an appropriate input-dependent uncertainty function which yields prediction sets which capture the heteroskedasticity of the data, and thus yield tighter sets in expectation over x . However, if we reduce the amount of calibration data available to N = 500 , the learned approach starts to overfit on the split used to optimize efficiency. In contrast, the KL-constraint of the PACBayes approach mitigates this overfitting, and outperforms both baselines in terms of test set efficiency for smaller amount of calibration data.

6.2 Corrupted MNIST

Next, we consider a more challenging distribution shift scenario, where we aim to deploy a model trained on 'clean' data to a domain with distribution shift. As the base predictor, we use a LeNet convolutional neural network trained on a softmax objective to classify noise-free MNIST digits [LeCun et al., 1998]. Then, we simulate distribution shift by adding Gaussian noise and applying random rotations to a held-out set of the dataset: examples of the clean and corrupted digits are shown in Figure 4. Our goal is to use a calibration dataset of this corrupted data to design a set-valued predictor that achieves coverage on held-out test data with the same corruption distribution.

As is conventional, when applying ICP to probabilistic classifiers, we use the negative log probability as the nonconformity score s default ( x , y ) = [log softmax f ( x )] [ y ] [Stutz et al., 2021, Angelopoulos and Bates, 2021]. This score function leverages the implicit uncertainty information encoded in the base model's probabilistic predictions; but these probabilities are far from calibrated, especially in the presence of distribution shift [Ovadia et al., 2019, Sharma et al., 2021]. Thus, we consider using the calibration dataset, which is representative of the corrupted test distribution, to fine-tune the predicted probabilities, defining s optim ( x , y , θ ) = [log softmax f ( x ; θ )] [ y ] , where θ are the parameters of the fully connected layers of the base predictor. As before, we measure efficiency as the size of the prediction set, ℓ eff ( C ) = | C | . In contrast to the regression setting, here C is a discrete set over the set of class labels, and thus the size of this set is non-differentiable. To smooth this objective for training,

we follow the approach of [Stutz et al., 2021], and first use a sigmoid to compute soft set-membership assignment for each possible label, and then take the sum to estimate the size of the set,

where the temperature T is a hyperparameter which controls the smoothness of the approximation; in our experiments we use T = 0 . 1 .

We evaluate each method for different calibration set sizes N and data split ratios, repeating each experiment using 3 random seeds. For all approaches, we desire a coverage guarantee of α = 0 . 1 with δ = 0 . 05 . For the PAC-Bayes approach, we run the algorithm with δ = 0 . 01 , performing a grid search over 5 values of ˆ α evenly spaced between 0 . 2 and 0 . 8 , and choosing the approach with the best efficiency generalization certificate (9); using the union bound for probability ensures that the efficiency for the best of the five runs will hold with probability at least 1 -0 . 05 , i.e., an effective δ of 0 . 05 . For the PAC-Bayes approach, we optimize both the prior mean µ and variance σ 2 on D 0 , as we found that optimizing the mean alone yielded a poor prior; ablations are provided in App. B. The results are summarized in Figure 4. In the middle figure, we hold the data split fixed at 0 . 5 , and plot the average test set size as a function of N . First, we note that even for the standard ICP approach, increasing N leads to a smaller prediction set size, as we are able to calibrate to a larger ˆ α while still maintaining the desired test-time guarantee. In general, both the learned and PAC-Bayes approaches improve upon the standard baseline by fine-tuning on the corrupted calibration data. However, when data is limited ( N = 1000 ), the learned baseline overfits during optimization on the first half of the calibration data, and subsequent calibration yields set sizes that are even larger than the standard method. Our method mitigates this by using all the data to simultaneously fine-tune and calibrate while ensuring test-time generalization, thanks to the KL constraint. Empirically, the PAC-Bayes method yields prediction sets with efficiency comparable or higher than the learned baseline calibrated using Vovk 2a. However, there still remains a gap between our results and the learned baseline calibrated with tighter bound in V ovk 2b, as is evident in our analysis of our bound in Figure 2. In the right figure, we explore the sensitivity of both learned methods on the choice of data split ratio. We observe that the PAC-Bayes method is less sensitive to the choice of data split ratio.

7 Conclusion

In this paper, we introduced theory for optimizing conformal predictors while retaining PAC generalization guarantees on coverage and efficiency without the need to hold-out data for calibration. We achieved this by combining PAC-Bayes theory with ICP to furnish generalization bounds with the same data used for training. We translated our theory into a practical algorithm and demonstrated its efficacy on both regression and classification problems.

Limitations. As shown in Figure 2, our theory remains overconservative relative to the tight Vovk 2b PAC bound, which future work could aim to address. Practically, a key limitation of our approach is its dependence on the availability of a good prior. Although, we were able to mitigate this issue by training a prior on a portion of the calibration dataset, this strategy can struggle when datasets are small or the number of parameters to optimize are very large. Furthermore, while diagonal Gaussian distributions are convenient to optimize, they may not represent the optimal posterior. These limitations may explain why we obtained only modest improvements in efficiency with respect to the Vovk 2a baselines in our experiments; we are optimistic that future advances in prior selection and representation may lead to improvements. Finally, while PAC-Bayes theory requires sampling from the posterior Q , our set-up also requires computing the threshold τ ∗ for each parameter sample. Future work could explore applying disintegrated PAC-Bayes theory [Viallard et al., 2023] to yield (perhaps more conservative) generalization bounds that hold for a single sample from the posterior.

Broader Impact. This work is building towards safer and trustworthy ML systems that can reason about uncertainty in their predictions. In particular, this work is a step towards developing calibrated uncertainty set predictors that are efficient enough for practical applications.

Future work. This work opens up exciting new theoretical and practical questions to pursue. On the theoretical front, we are excited to explore the development of an online learning approach that can leverage this theory to provably adapt to distribution shifts on the fly. On the practical front, we look forward to using this framework for providing reasonable uncertainty estimates in robot autonomy stacks to facilitate decision making and safety certification.

Acknowledgments and Disclosure of Funding

Anirudha Majumdar was partially supported by the NSF CAREER Award [#2044149] and the Office of Naval Research [N00014-23-1-2148]. Asher Hancock was supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2146755. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

Md Manjurul Ahsan, Shahana Akter Luna, and Zahed Siddique. Machine-learning-based disease diagnosis: A comprehensive review. Healthcare , 10(3):541, 2022.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv:2107.07511 , 2021.

Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Int. Conf. on Machine Learning , pages 254-263, 2018.

Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, and Caiming Xiong. Efficient and differentiable conformal prediction with general function classes. In Int. Conf. on Learning Representations , 2022.

Peter L. Bartlett, Dylan J. Foster, and Matus J Telgarsky. Spectrally-Normalized Margin Bounds for Neural Networks. In Conf. on Neural Information Processing Systems , volume 30, 2017.

Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Occam's razor. Information Processing Letters , 24(6):377-380, 1987.

Matthew Cleaveland, Insup Lee, George J Pappas, and Lars Lindemann. Conformal prediction regions for time series using linear complementarity programming. arXiv:2304.01075 , 2023.

Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. In Conf. on Neural Information Processing Systems , volume 32, 2019.

Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36(2):183-212, 1983.

Gintare Karolina Dziugaite and Daniel M. Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. In Proc. Conf. on Uncertainty in Artificial Intelligence , 2017.

Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differential privacy. In Conf. on Neural Information Processing Systems , volume 31, 2018.

Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty-aware classifiers with conformalized deep learning. In Conf. on Neural Information Processing Systems , volume 35, 2022.

Mahdi Milani Fard, Joelle Pineau, and Csaba Szepesvári. Pac-bayesian policy evaluation for reinforcement learning. arXiv:1202.3717 , 2012.

Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization of sorting networks via continuous relaxations. In Int. Conf. on Learning Representations , 2019.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. In Int. Conf. on Learning Representations , 2020.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George J Pappas. Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters , 8, 2023.

Sanae Lotfi, Marc Finzi, Sanyam Kapoor, Andres Potapczynski, Micah Goldblum, and Andrew G Wilson. PAC-Bayes compression bounds so tight that they can explain generalization. In Conf. on Neural Information Processing Systems , volume 35, 2022.

Anirudha Majumdar, Alec Farid, and Anoopkumar Sonar. PAC-Bayes control: learning policies that provably generalize to novel environments. Int. Journal of Robotics Research , 40(2-3):574-593, 2021.

Andreas Maurer. A note on the pac bayesian theorem. arXiv:cs/0411099 , 2004.

David A McAllester. Some PAC-Bayesian theorems. In Proc. Computational Learning Theory , pages 230-234, 1998.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks. arXiv:1707.09564 , 2017a.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring Generalization in Deep Learning. In Conf. on Neural Information Processing Systems , volume 30, 2017b.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Conf. on Neural Information Processing Systems , 32, 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Conf. on Neural Information Processing Systems , volume 32, 2019.

Maria Perez-Ortiz, Omar Rivasplata, Benjamin Guedj, Matthew Gleeson, Jingyu Zhang, John ShaweTaylor, Miroslaw Bober, and Josef Kittler. Learning pac-bayes priors for probabilistic neural networks. arXiv:2109.10304 , 2021.

María Pérez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvári. Tighter risk certificates for neural networks. Journal of Machine Learning Research , 22:10326-10365, 2021.

Allen Ren, Sushant Veer, and Anirudha Majumdar. Generalization guarantees for imitation learning. In Conf. on Robot Learning , pages 1426-1442, 2021.

Jorma Rissanen. Stochastic Complexity in Statistical Inquiry . World Scientific Publishing, 1989.

Omar Rivasplata, Vikram M Tankasali, and Csaba Szepesvari. Pac-bayes with backprop. arXiv:1908.07380 , 2019.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. In Conf. on Neural Information Processing Systems , volume 32, 2019.

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. In Conf. on Neural Information Processing Systems , volume 33, 2020.

Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association , 114(525):223-234, 2019.

Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems , 1(1):187-210, 2018.

Matthias Seeger, John Langford, and Nimrod Megiddo. An improved predictive accuracy bound for averaging classifiers. In Int. Conf. on Machine Learning , pages 290-297, 2001.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge Univ. Press, 2014.

Apoorva Sharma, Navid Azizan, and Marco Pavone. Sketching curvature for efficient out-ofdistribution detection for deep neural networks. In Proc. Conf. on Uncertainty in Artificial Intelligence , pages 1958-1967, 2021.

David Stutz, Ali Taylan Cemgil, Arnaud Doucet, et al. Learning optimal conformal classifiers. In Int. Conf. on Learning Representations , 2021.

Jessica Vamathevan, Dominic Clark, Paul Czodrowski, Ian Dunham, Edgardo Ferran, George Lee, Bin Li, Anant Madabhushi, Parantu Shah, Michaela Spitzer, et al. Applications of machine learning in drug discovery and development. Nature Reviews Drug Discovery , 18(6):463-477, 2019.

Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Dokl. Akad. Nauk , 181(4), 1968.

Sushant Veer and Anirudha Majumdar. Probably approximately correct vision-based planning using motion primitives. In Conf. on Robot Learning , pages 1001-1014, 2020.

Paul Viallard, Pascal Germain, Amaury Habrard, and Emilie Morvant. A general framework for the practical disintegration of pac-bayesian bounds. Machine Learning , 2023.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conf. on Machine Learning , pages 475-490, 2012.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer, 2005.

Waymo. The Waymo Driver Handbook: Teaching an autonomous vehicle how to perceive and understand the world around it, 2021. Available at https://blog.waymo.com/2021/10/ the-waymo-driver-handbook-perception.html .

Omry Yadan. Hydra - a framework for elegantly configuring complex applications. Github, 2019. URL https://github.com/facebookresearch/hydra .

Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. In IEEE Conf. on Computer Vision and Pattern Recognition , 2023.

Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv:2104.13871 , 2021.

A Proofs

Our coverage and efficiency generalization bounds follow a similar form to many PAC-Bayes bounds. A fundamental building block of these proofs is the Donsker-Varadhan variational formula [Donsker and Varadhan, 1983], which we restate below.

Lemma 1 (Donsker-Varadhan variational formula) . For any measurable bounded function h : Θ → R , we have,

We then apply this lemma to obtain a core result relating to quantities depending on parameters θ and sampled data D N , which we derive first as a second lemma which is common to the proof of both generalization bounds.

Lemma 2. Let Φ( θ, D N ) : Θ × ( X × Y ) N → R be a measurable function mapping a parameter and a set of sampled data to a scalar. Let P be an arbitrary distribution over Θ , independent of D N . Let Q denote another arbitrary probability distribution over Θ . Suppose there exists a function B ( N ) > 0 such that, for any fixed θ ∈ Θ ,

Then, we have that

Proof of Lemma 2. We start with our assumed property of B ( N ) , and integrate it with respect to θ .

The integrand is non-negative and P is independent of D N ; so, by Tonelli's theorem, we can swap the order of the integration with respect to the parameters θ and the dataset D N , yielding

Now, we can apply the Donsker-Varadhan variational formula to get

Now, by the Chernoff bound, for any random variable X ,

so if we let δ = e -s , s = -log δ , we have

Now, applying this bound to X = sup Q ∈P ( θ ) [ E θ ∼ Q [Φ( θ, D N )] -KL( Q || P ) -log B ( N )] , we obtain

Now, since

the result from (14) implies that

Taking the complement, we have

Rearranging terms completes the proof.

Equipped with this lemma, we can now prove our core theorems. We start by proving the coverage bound.

Proof of Theorem 1. First, let L cov ( θ, D N , ˆ α ) represent the miscoverage rate of a conformal predictor calibrated to achieve 1 -ˆ α empirical coverage on D N :

Note that for a fixed θ , the distribution of coverage (and hence, mis-coverage) conditioned on a particular calibration dataset follows a Beta distribution [Angelopoulos and Bates, 2021, V ovk, 2012]. Specifically, L cov ( θ, D N , ˆ α ) ∼ Beta(( ⌊ ( N +1)ˆ α ⌋ , N +1 -⌊ ( N +1)ˆ α ⌋ )) .

Now, choose Φ( θ, D N ) to be

First, we derive the bound B ( N ) required to apply Lemma 2. Let k = ⌊ ( N +1)ˆ α ⌋ , a = k -1 N -1 , and b = L cov ( θ, D N , ˆ α ) . Then,

Thus, in this case we can analytically evaluate this expectation as

where Beta( x ; α, β ) is the pdf of the Beta distribution with parameters α, β .

To compare our bound with other PAC-Bayes bounds and to quantify the asymptotic behavior of B ( N ) , we can derive an upper bound. First, we plug in our definition of a , and group terms

By Stirling's approximation of a ! , we know

Therefore,

Plugging in these bounds, we can obtain a simplified expression which removes the factorials and high powers of N :

Notice that for 0 < k < N , the term in the exponent is negative. Given the definition of k , this is true as long as N ≥ 1 ˆ α -1 . In this case, due to monotonicity, the exponential term is less than 1. This yields

Having computed B ( N ) , we can now apply Lemma 2 to obtain

Since kl( · , · ) is convex in both arguments, we can use Jensen's inequality to bring the expectation over θ inside each argument, yielding,

Note that by definition in equation (6), L cov ( Q ) = E θ ∼ Q [ L cov ( θ, D N , ˆ α )] . Rearranging terms completes the proof.

Proof of Corollary 2.1. Let q = ⌊ ( n +1)ˆ α ⌋-1 n -1 , and for compactness, let L cov = L cov ( θ, τ ( θ, D N , ˆ α )) Let C be the event that our KL constraint holds, i.e.

Consider the event E that E θ ∼ Q [ L cov ] ≤ q . Since q ≤ α , we have that

Now, consider the complement, ¯ E . We know that kl( q || x ) is monotonically increasing in x for x ≥ q . Thus, conditioned on ¯ E , we have

Given these statements, we have

Now, due to the convexity of kl , we have through Jensen's inequality,

Plugging this in to the result of Theorem 1, we have

Now, since this holds for all Q , it must also hold for those Q satisfying the constraint. Thus,

Plugging this result into (17), we have

completing the proof.

Next, we prove our bound on efficiency.

Proof of Theorem 2. We wish to bound L eff ( Q ) as a function of the observed empirical efficiency. Define ˆ L eff ( θ, τ, D N ) as the empirical mean efficiency observed for a particular value of parameter θ and threshold τ , and L eff ( θ, τ ) as the expected efficiency of the prediction sets constructed with θ and τ on new data sampled from D :

In ICP, the threshold τ = τ ∗ ( D N , ˆ α, θ ) itself depends on D N . This means that even for a fixed θ , the observed efficiency for the prediction set on each sample in D N are not independent from one another. To avoid this complication, we define the quantity

which measures the worst-case difference between the empirical and true mean efficiencies over the worst-case choice of threshold. Note that

Next, we consider the expectation of this quantity over the sampling of D N :

Given that, by assumption, ℓ eff is L τ -Lipschitz in τ , and τ is bounded by β (due to s ( x , y , θ ) being bounded), we use the result shown in Proposition 4 of Bai et al. [2022] to show that R ( θ ) can be bounded for a fixed θ :

Furthermore, assuming ℓ eff is bounded by [0 , 1] , then, for a fixed θ , ˆ R ( θ, D N ) satisfies the bounded differences property with bound 1 /N . Therefore, applying McDairmid's inequality, we obtain

This bound only holds for a fixed θ independent of D N .

To obtain a bound which holds for θ ∼ Q for all Q , we can follow the standard PAC-Bayes bound sub-Gaussian random variables. Specifically, we turn to Lemma 2, this time defining

Note that given (20), applying Lemma 1.5 , we can bound

Now, we can apply Lemma 2 to obtain

Bringing the expectation into the quadratic using Jensen's inequality, and rearranging terms, we obtain

Now, substituting (19) and (18), we obtain that with probability greater than 1 -γ , for all Q , it holds that

Noting that by definition, L eff ( Q ) = E θ ∼ Q [ L eff ( θ, τ ∗ ( D N , ˆ α, θ ))] , which yields the theorem as stated, concluding the proof.

B Implementation Details

All experiments were performed on a workstation with a Intel Core i9-10980XE CPU with a NVIDIA GeForce RTX 3090 GPU.

B.1 Detailed Algorithm Overview

In this section, we give a detailed description of our general experimental method. Subsequent sections give specifics to each of the experimental domains we consider.

Model training. For each experiment, we first train a base neural network model on a training dataset. Next, for each calibration approach, we pair the neural network model with a score function which is used for calibration.

Model calibration. In the calibration phase, we first split the calibration data D cal into two random splits, a tuning dataset D 0 and true calibration dataset D N , where the fraction of data used for tuning (the data split ) is a hyperparameter. In the standard baseline, the no optimization is performed so D 0 is the empty set, and D N = D cal is used to calibrate the model. In the learned baseline, parameters are optimized on only D 0 , and D N is used to calibrate by computing the threshold τ ∗ . In our PAC-Bayes approach, we tune prior parameters on D 0 , and then optimize the posterior following Algorithm 1 on D N .

We aim for the same PAC guarantee of 1 -α test coverage rate with probability greater than 1 -δ . For the standard and learned baselines, N , i.e. the size of D N , determines the required empirical miscoverage rate ˆ α . We can either use ˆ α equal to the value given by (4) [Vovk, 2012, Prop 2a], or by finding the value (e.g. by grid search) satisfying (5) [Vovk, 2012, Prop 2b]. For our PAC-Bayes approach, ˆ α is a hyperparameter which trades off conservatism in choosing τ ∗ for the freedom to optimize score function parameters on D N while retaining the PAC generalization guarantees. To select this hyperparameter, we choose K values for ˆ α , and for each, run our algorithm with a tighter δ ' = δ/K . This allows us to use a union bound argument to achieve the same PAC guarantee as the other approaches uniformly over the K predictors we obtain. We select between these predictors by choosing the predictor with the best generalization bound on efficiency (9).

Optimization on D 0 . In both the learned and PAC-Bayes approaches, we optimize parameters on the tuning dataset to minimize the efficiency via gradient descent. Note that the efficiency of a conformal predictor depends on the score function evaluated at a given input x , as well as the threshold τ ∗ which corresponds to the empirical quantile of the score function evaluated on the calibration dataset. We follow the approach of Stutz et al. [2021], and for each minibatch, we first compute τ ∗ by evaluating the ˆ α quantile on the minibatch, and then use this threshold to evaluate the efficiency loss on the data. We use the softsort approach proposed by [Grover et al., 2019] to differentiate through the quantile function. The prior tuning in our PAC-Bayes approach follows the same protocol, with the addition of first sampling θ ∼ N ( µ 0 , Σ 0 ) (differentiably, via the reparameterization trick), before computing the efficiency loss for each sample and averaging.

Optimization on D N . In our PAC-Bayes approach, we further optimize the score function on the second split of data, this time holding the prior fixed and instead optimizing the posterior Q = N ( µ, Σ) to minimize the efficiency loss (7) subject to the KL constraint (10). The efficiency loss is evaluated in the same way as above, and the KL constraint can be evaluated analytically. To perform the constrained optimization, we implement an augmented Lagrangian method. Specifically, we construct an unconstrained optimization problem with the objective

where B ( α, ˆ α, δ, N ) is the KL budget defined in (10) and s ∈ R + is a positive slack variable translating the inequality constraint in (10) to an equality constraint c ( µ, Σ , s ) = 0 . In each outer iteration, we approximately solve this unconstrained optimization problem for µ, Σ and s using projected gradient descent (clamping s to 0 after any gradient step making it negative). After

optimization, we evaluate the constraint c ( µ, Σ , s ) , and update the penalty scale according to

before rerunning the inner optimization. We run 10 outer iterations, and return the µ, Σ which yielded the best efficiency loss while satisfying the constraint.

Calibration on D N . For all models, we calibrate using D N by choosing τ ∗ ( D N , ˆ alpha ) to be the q = ⌈ ( n +1)(1 -ˆ α ) ⌉ /n quantile of the set of score function values evaluated on D N . In the case of the PAC-Bayes approach, the score function (and hence the threshold) depends on the sampled value of θ . Therefore, after optimization, we first pre-sample M values of θ ∼ Q , and for each, compute τ ∗ as above. We store the sampled θ values along with their associated τ ∗ values for use at test time.

Evaluation. To evaluate we construct prediction sets according to the learned score functions on a held-out test set of N test points. For each point, the prediction set is constructed according to (2) using the θ, τ ∗ pair obtained from the calibration phase. In the case of the PAC-Bayes baseline, each test sample is randomly assigned to one of the pre-sampled parameter/threshold pairs. We measure the rate of coverage as well as the efficiency ℓ eff of each prediction set. We repeat the entire process from calibration and evaluation on multiple random seeds.

B.2 Illustrative example

Heteroskedastic data was generated by sampling x ∼ U ( -1 , 1) , and computing the target y = cos(5 x ) + 0 . 3 ∗ ϵ 1 +1 . 8 σ (5 x ) ϵ 2 , where ϵ 1 , ϵ 2 ∼ U ( -0 . 5 , 0 . 5) , and σ ( · ) is the sigmoid function.

The base predictor was a 2 hidden layer feedforward network with ReLU activations, with layer width of 64 neurons. We trained the network on 100 randomly sampled datapoints, and held out N new datapoints to use for calibration. In the results in the paper, for all methods we targeted α = 0 . 1 with δ = 0 . 05 . We held the base predictor fixed, and optimize the score function

where σ is the sigmoid, and θ g functions as a gating parameter, and u ( x ; θ ) is a learned inputdependent uncertainty function defined as

where FF is a two-hidden layer feedforward network with tanh activations and a layer width of 128 , and the softplus together with the offsets smoothly maps the output of the neural network to (approximately) R + .

For the learned model, we optimize the efficiency loss for 2000 steps with a learning rate of 1e-3 and a batch size of 100 samples. For the PAC-Bayes approach, we optimize using an augmented Lagrangian method, using 2000 steps of gradient descent with a learning rate of 1e-3 to solve the unconstrained penalized problem, running 7 outer iterations in which the penalty terms are updated and the unconstrained problem is re-solved. Each round of inner optimization requires around 1 minute of compute. In the results shown in the paper, we optimized only the prior mean µ , and held the prior variance fixed at σ 2 = 0 . 02 / √ F where F is the fan_in for that parameter.

B.3 Corrupted MNIST

For the corrupted MNIST experiments, we use 7000 images from the MNIST train split to train a base predictor with the LeNet-5 architecture. For the calibration and test data, we use the held-out test split from MNIST and apply random rotation, uniform between -30 ◦ and 30 ◦ and Gaussian noise with standard deviation 1 . 3 . We save these transformed data, and then split this data data into two disjoint subsets for the calibration and test sets respectively, using a different split for each choice of random seed. We evaluate each approach on 1000 test samples.

Here, the score function for all methods is fixed to be the negative log softmax output by the base predictor, but for the learned and PAC-Bayes methods, we consider optimizing the fully-connected portion of the network. The efficiency loss we use is a differentiable approximation of the output set

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

1

2

3

4

5

Average Prediction Set Size

Data split:

0.25:0.75

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

Data split:

0.50:0.50

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

Data split:

0.75:0.25

Standard (Vovk 2a)

Standard (Vovk 2b)

Learned (Vovk 2a)

Learned (Vovk 2b)

PAC-Bayes

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

0

.

86

0

.

88

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

1

.

02

Observed Coverage Rate on Test Set

Data split:

0.25:0.75

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

Data split:

0.50:0.50

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

Data split:

0.75:0.25

Target Coverage Rate: 1

-

α

95% CI on test-set coverage

Standard (Vovk 2a)

Standard (Vovk 2b)

Learned (Vovk 2a)

Learned (Vovk 2b)

PAC-Bayes

size. Note while the actual set-size of the resulting prediction sets is not Lipschitz continuous w.r.t. the threshold τ , the smoothed loss is, as it is the sum of K sigmoid functions which are themselves Lipschitz continuous, where K is the number of classes. Here, we optimize the efficiency using the same optimization parameters as in the illustrative example. For the PAC-Bayes method, we initialize the prior with mean equal to the base predictor's original trained weights, and a variance with scale σ 2 = 0 . 01 / √ F where once again, F corresponds to the is the fan_in for the layer that the parameter is in. In the body, we report results obtained by optimizing both the prior mean and variance to minimize the efficiency objective on D 0 before optimizing the posterior subject to the KL constraint on D N . We found that this led to better results, as selecting the variance using the simple initialization strategy we used in the illustrative example includes too many θ values which lead to inefficient sets, hindering the quality of the posterior that can be found within the KL constraint.

B.3.1 Additional Results

In Figure 5, we show the complete set of prediction set size vs calibration dataset size results, for all choices of dataset split. In general, the PAC-Bayes approach yields equal or better sets than baselines across the choice of data split. Figure 6 shows the achieved test-time coverage rates for all methods. All methods target a generalization coverage rate of at least 1 -α = 0 . 9 , but due to a finite test set size N test = 1000 , the empirical coverage rate may appear lower. We show 95% confidence intervals for the empirical mean of 1000 draws from a Bernoulli random variable with probability 0 . 9 . For our results to be consistent with the desired PAC guarantee, failures would only happen in the case of (a) our PAC guarantee failing to hold and the true coverage rate being lower than 1 -α , which should happen with probability less than δ , or (b) the coverage rate being higher than α but the empirical mean on the test set being lower than the 95% CI for α = 0 . 9 , which should occur with probability

0

.

88

0

.

90

0

.

92

0

.

94

0

.

96

Observed Coverage Rate on Test Set

1

2

3

4

5

Average Prediction Set Size

Data split:

0.25:0.75

0

.

88

0

.

90

0

.

92

0

.

94

0

.

96

Observed Coverage Rate on Test Set

Data split:

0.50:0.50

0

.

88

0

.

90

0

.

92

0

.

94

0

.

96

0

.

98

1

.

00

Observed Coverage Rate on Test Set

Data split:

0.75:0.25

Target Coverage Rate: 1

-

α

95% CI for

N

test

= 1000

Standard (Vovk 2a)

Standard (Vovk 2b)

Learned (Vovk 2a)

Learned (Vovk 2b)

PAC-Bayes

less than 0 . 05 . Indeed we see that all but one trial for the Learned (Vovk 2b) baseline yields an empirical coverage less than the 95% CI. We performed 30 total trials per method, so this observation is consistent with our PAC guarantee and finite test-set size. We find that the PAC-Bayes method is not significantly over-conservative compared to baselines using Vovk 2a to select the empirical coverage targets. However, Vovk 2b leads to much tighter results, indicating room for improvement in our PAC-Bayes results. The difference between the optimized methods and the standard ICP coverage rate is more pronounced for the 0.75:0.25 data split because the optimized method are using only a quarter of the data to guarantee test time coverage, while the standard method uses the whole dataset and thus can calibrate to a lower empirical miscoverage rate ˆ α following (4).

We also visualize efficiency (prediction set size) as a function of coverage in Figure 7. Indeed, these two quantities are linked: achieving higher coverage for a fixed score function requires increasing set size, coming at the cost of efficiency. The PAC-Bayes results lie on the Pareto frontier on this trade-off, highlighting that our method is able to learn good score functions that yield efficient test-sets on the test set. Future work tightening the generalization bound has the potential to improve practical utility by reducing over-conservatism and allowing calibrating to lower empirical coverage levels.

In Figure 8, we show the impact of different choices of prior optimization strategy, comparing no optimization (using the prior as initialized, using the whole calibration dataset to perform constrained optimization of the posterior) against both mean-only ( µ ), and mean-var ( µ , σ 2 ) . As can be seen, a crude, naive strategy of arbitrarily defining a prior by choosing a fixed variance around the initialization of the weights does not work well for this problem setting, highlighting the value of using some data to tune the prior. Furthermore, we find that optimizing both µ and σ 2 using a fraction of the data outperforms optimizing only µ , for all choices of data split.

1000

1500

2000

2500

5000

Size of Calibration Dataset (

N

)

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

4

.

0

4

.

5

5

.

0

Average Prediction Set Size

None 0.00/1.00

µ

0.25/0.75

µ, σ

2

0.25/0.75

µ

0.50/0.50

µ, σ

2

0.50/0.50

µ

0.75/0.25

µ, σ

2

0.75/0.25

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInductive Conformal Prediction (ICP) provides a coverage guarantee of constructed prediction sets, while does not provide any guarantee on the efficiency of prediction sets. The efficiency depends on a conformity score function, and direct approaches optimize the score function to minimize the efficiency of prediction sets in two steps: (1) given a hold-out set from a calibration set, optimize the score function, and (2) given another hold-out set from a calibration set, run an ICP algorithm. However, the direct approaches do not provide a generalization guarantee for efficiency. The proposed approach combines ICP and the PAC-Bayes to provide generalization guarantees for both coverage and efficiency. In particular, the paper provides novel generalization bounds for coverage and efficiency and proposes an algorithm from them, where the efficacy is evaluated on regression and image classification problems.\n\nSTRENGTHS:\n**Originality**: The paper proposes new generalization bounds on coverage and efficiency. \n\n**Quality**: The proposed algorithm is supported by theorems, making the algorithm rigorous. \n\n**Clarity**: I like the paper structure. It first provides the main theories, and then explains an algorithm with practical details (appreciate it). \n\n**Significance**: I believe the PAC-Bayes interpretation introduces an interesting, novel view on conformal prediction, which could trigger interesting CP algorithms.\n\nWEAKNESSES:\nCombining PAC-Bayes and ICP is very interesting, though I have a few concerns. \n\n1. It is not easy to find the benefit of the bound on efficiency (i.e., Thm2). This is novel, but in Algorithm 1, it also uses the first term of (8). What is the algorithmic benefit of the efficiency bound Thm2?\n2. I assume that this paper uses the PAC-style ICP (4) as the baselines (i.e., “the standard ICP approach” and “a  learned ICP approach”. However, (4) is known to be loose (as clearly mentioned in Vovk [2012]. In the same Vovk’s paper, it also has a tighter ICP (i.e., Proposition 2b in Vovk [2012]) and its tightness (and efficiency) is also well-demonstrated in deep learning applications in [R1]. If the authors use the tiger PAC-style ICP, would “the standard ICP” and “a learned ICP” baselines be possible to be better than the proposed approach in Figure 4? \n3. It is unclear what the benefit of PAC-Bayes is in achieving an efficient prediction set. Based on Line 50 (i.e., “our framework allows us to utilize the entire calibration dataset…”), I initially thought that the proposed approach fully utilizes the calibration set (while direct approaches require splitting calibration sets into two). However, the proposed approach also needs to split the calibration sets for prior learning (i.e., Line 238). At this point, I was unsure of the benefit of combining PAC-Bayes with ICP. My guess is simultaneous optimization of a score function distribution Q and threshold \\hat{tau} is one factor for achieving a better efficiency than the baselines, but I want to hear the author's clarification. \n4. The nonconformity score function of the standard ICP in regression is quite weak, which misleads readers. The score function with only the residual (i.e., Line 282) is not recommended as it does not encode per sample uncertainty (as also mentioned in the paper). Instead, a variance normalized score function (i.e., (8) in [R2], 3.6 in [R1], or 2.2.1 in [R3]) is a better choice, where the variance is simply trained via a training set. Given this better score function, I don’t think “the standard ICP” result in Figure 3 is as poor as shown in the paper. \n\n\n[R1] https://arxiv.org/abs/2001.00106\n\n[R2] http://proceedings.mlr.press/v91/vovk18a/vovk18a.pdf\n\n[R3] https://arxiv.org/abs/1612.01474\n\nQUESTIONS:\nEach question is associated with each weakness in Weaknesses.\n\n1. What is the algorithmic benefit of the efficiency bound in Thm2?\n2. If the authors use the tiger PAC-style ICP (i.e., Proposition 2b in Vovk [2012]), would “the standard ICP” and “a learned ICP” baselines be possible to be better than the proposed approach in Figure 4? \n3. Could you clarify on the benefit of PAC-Bayes in achieving an efficient prediction set?\n4. If the authors use the variance normalized score function, do we still observe the same limitation of “the standard ICP” in Figure 3?\n\nLIMITATIONS:\nThe limitations are clearly stated in Conclusion, and I agree with that.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work first derives coverage and efficiency generalization bound using PAC-Bayes for inductive conformal prediction. Furthermore, a practical algorithm, basing bayesian learning and conformal training, is proposed to learn efficient nonconformity score functions. In experiment section, the proposed algorithm is shown to outperform two baselines (original ICP and learned ICP) in some scenarios.\n\nSTRENGTHS:\n1, This work is mathematically solid, by combining PAC-Bayes theory with inductive conformal prediction. Particularly, it derives a efficient loss bound using PAC-Bayes for ICP. \n2, In the experiment section, the proposed method outperforms other two baselines (original ICP and learned ICP) in many scenarios, demonstrating high efficiency, especially when limited calibration data is available.\n3, The manuscript is well-written; and proposed method is mathematically sound.\n\nWEAKNESSES:\n1, In the abstract, it is claimed that \"the entire calibration dataset\" can be utilized. In the practical algorithm, splitting the calibration dataset is still needed (to derive a good prior). As a result, in my opinion, the mentioned claim is at least controversial.\n2, Though not mentioned, the proposed algorithm could be expensive in training and inference, as multiple models need to be sampled.\n3, The empirical evaluation is insufficient. Only one synthetic dataset and one practical dataset are used. It would be appreciated if experiments on more practical datasets. With more empirical results, it could be more convincing to claim that the proposed method produces more efficient prediction sets.\n\nQUESTIONS:\n1, How are the two derived bounds connected to the proposed algorithm?\n\nLIMITATIONS:\n1, The computation complexity might be so high that the proposed method cannot trivial extend to larger datasets and models.\n2, Without good prior, the derived bounds might be too loose.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe efficiency of set-valued predictor is crucial and less explored, this paper use the framework of PAC-Bayes to obtain the generalization bounds on both the coverage and the efficiency of set-valued predictors. The authors also propose a novel algorithm to optimize the efficiency without a separate hold-out dataset. They provide a theoretical guarantee for the proposed method and empirically verify the superiority of the proposed method.\n\nSTRENGTHS:\n- This paper uses the framework of PAC-Bayes to analyze the efficiency of inductive conformal prediction, which is novel and less explored.\n- The algorithm proposed by this paper does not require a separate hold-out dataset, while still having a theoretical guarantee of coverage.\n- This paper is well organized and the presentations are clear to me.\n\nWEAKNESSES:\n- To achieve the test-time coverage, the key constraint is shown in Corollary 2.1. However, the authors do not show whether it is satisfied in the experiments. \n- In the experiments, the authors show the average size of the prediction set, but I think the marginal coverage evaluated on the test data in the experiments is also important, and does a smaller average size of the prediction set empirically lead to worse coverage?\n- In the experiments, the description of the ***learned*** baseline is unclear. Do the authors train the model using the method proposed in [1]? Meanwhile, the authors miss a recent work [2] that aims to output a smaller conformal prediction set with higher conditional coverage. [1] and [2] should be chosen as baselines in the experiments.\n\n[1] David Stutz, Krishnamurthy Dvijotham, Ali Taylan Cemgil, Arnaud Doucet. Learning Optimal Conformal Classifiers. ICLR 2022.\n\n[2] Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, Yanfei Zhou. Training Uncertainty-Aware Classifiers with Conformalized Deep Learning. NeurIPS 2022.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nA limitation of this paper is that the theoretical results and proposed method can only be applied for a stochastic model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
Accurate uncertainty estimation is crucial in building robust and trustworthy machine learning systems. This paper utilizes PAC-Bayes theory and inductive conformal prediction (ICP) to develop a practical algorithm that fine-tunes parameters and score functions using calibration data, while ensuring inference coverage and efficiency. Empirical validation of this work shows that it is effective and outperforms prior methods.

STRENGTHS:
The paper presents a well-executed translation from theory to practical algorithm and provides a comprehensive analysis of related work. It also discusses the practical considerations in their proposed algorithm and effectively demonstrates its ability to make efficient predictions and achieve good generalization certificates in two experiments.

WEAKNESSES:
Here are some suggestions for improving the original content:

1. It would be beneficial to provide more information and explanation in section 5.2 to help readers better understand the algorithm.
2. Consider including an experiment that explores the tradeoff between desired coverage and prediction performance. This would provide valuable insights into the algorithm's effectiveness.

QUESTIONS:
I am interested in understanding the extent to which distribution shift can affect the performance of calibration and its coverage. Adding more noise to MNIST figures can increase the level of uncertainty, making it more challenging to ensure accuracy.

LIMITATIONS:
The paper identifies various limitations and proposes practical solutions to address them. The significance of this work lies in its contribution to achieving a more reliable and secure ML system. Particularly, it demonstrates its effectiveness in providing assurance to enhance the calibration of uncertainty.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the setting of inductive conformal prediction (ICP). In this setting, given a learned point-predictor, a calibration set and a scoring function is used to obtain a set-valued predictor that contains the correct label with high probability. A drawback of previous approaches is that, for ICP guarantees to be valid, the scoring function has to be fixed, or part of the calibration set needs to be used to learn the scoring function, while the remaining part is used to learn the coverage sets. In order to solve this, the paper proposes to use PAC-Bayesian bounds for self-certified learning, so that the scoring function can be learned from the calibration data, while using the same calibration data to obtain test-time bounds by using generalization guarantees. The approach relies on new bounds, which are similar to known PAC-Bayesian bounds but specialized to the ICP setting, and data-dependent Gaussian priors according to a standard procedure. Numerical results demonstrate the potential of the proposed approach to improve ICP in low-data settings.\n\n\n**Post-Rebuttal edit**: I read the author rebuttal, which addressed my questions, and consequently updated the rating (as stated below).\n\nSTRENGTHS:\n— The paper proposes a creative and relevant use of the PAC-Bayesian framework for a practical use-case\n\n— The presentation is pedagogic, providing an essentially self-contained summary of the necessary background from ICP and PAC-Bayes\n\n— In addition to just presenting bounds, the paper suggests a well-motivated practical algorithm that is shown to have potential to improve results for certain settings. The interpretation of the PAC-Bayesian bound as providing a \"KL budget\" is neat.\n\nWEAKNESSES:\n— The discussion of the PAC-Bayesian bounds does not make it entirely clear what the relation to prior work is (see questions below).\n\n— The improvements that are seen in the numerical experiments appear to be marginal, and mostly within the error bars. The non-monotonicity in the order between \"Learned\" and \"PAC-Bayes\" in Figure 4 (middle) appears to highlight that they are essentially indistinguishable. Also, it was not entirely clear to me how Figure 3 demonstrated an improvement for PAC-Bayes.\n\nQUESTIONS:\n— Theorem 1, in terms of the binary KL, is very similar to the Maurer-Langford-Seeger bound (see, e.g., Maurer (2004)). In Maurer's result, a smaller $B(N)$ is found. However, I don't think that the two arguments of your binary KL are related by an expectation as in Maurer's derivation, which necessitates the path that you take. Is this interpretation correct? I think the paper would benefit by clarifying the relation to these classical PAC-Bayesian bounds.\n\n— The derivation of Theorem 3, once the step has been taken from $\\mathcal L$ to $R$, appears to be the same as a standard PAC-Bayesian bound via \"sub-Gaussian\"-type concentration for bounded random variables. Is there any other difference in terms of the proof?\n\n— In the introduction, you refer to PAC-Bayesian priors as data-independent, which may cause some confusion since you use data-dependent priors in your experiments. Perhaps a note should be made that they can be data-dependent in certain circumstances.\n\n---\n\nMaurer, A Note on the PAC Bayesian Theorem, 2004\n\n___\n---\n\nMinor comments:\n\nEq (5): Should this be averaged over the posterior? This is typically the case for PAC-Bayes bounds.\n\nLine 196: “for all for all”\n\nLine 197: bounuded -> bounded in footnote\n\nLine 326: “we found that optimizing the yielded a poor prior” incomplete sentence\n\nDonkser -> Donsker\n\nOverfull margins in appendix\n\nLIMITATIONS:\nLimitations are adequately discussed. Perhaps it can be made clearer that the proposed approach does not yield particularly significant improvements over the \"Learned\" baseline.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}