PAC Confidence Predictions for Deep Neural Network Classifiers

1 INTRODUCTION . Due to the recent success of machine learning , there has been increasing interest in using predictive models such as deep neural networks ( DNNs ) in safety-critical settings , such as robotics ( e.g. , obstacle detection ( Ren et al. , 2015 ) and forecasting ( Kitani et al. , 2012 ) ) and healthcare ( e.g. , diagnosis ( Gulshan et al. , 2016 ; Esteva et al. , 2017 ) and patient care management ( Liao et al. , 2020 ) ) . One of the key challenges is the need to provide guarantees on the safety or performance of DNNs used in these settings . The potential for failure is inevitable when using DNNs , since they will inevitably make some mistakes in their predictions . Instead , our goal is to design tools for quantifying the uncertainty of these models ; then , the overall system can estimate and account for the risk inherent in using the predictions made by these models . For instance , a medical decision-making system may want to fall back on a doctor when its prediction is uncertain whether its diagnosis is correct , or a robot may want to stop moving and ask a human for help if it is uncertain to act safely . Uncertainty estimates can also be useful for human decision-makers—e.g. , for a doctor to decide whether to trust their intuition over the predicted diagnosis . While many DNNs provide confidences in their predictions , especially in the classification setting , these are often overconfident . This phenomenon is most likely because DNNs are designed to overfit the training data ( e.g. , to avoid local minima ( Safran & Shamir , 2018 ) ) , which results in the predicted probabilities on the training data being very close to one for the correct prediction . Recent work has demonstrated how to calibrate the confidences to significantly reduce overconfidence ( Guo et al. , 2017 ) . Intuitively , these techniques rescale the confidences on a held-out calibration set . Because they are only fitting a small number of parameters , they do not overfit the data as was the case in the original DNN training . However , these techniques do not provide theoretical guarantees on their correctness , which can be necessary in safety-critical settings to guarantee correctness . We propose a novel algorithm for calibrated prediction in the classification setting that provides theoretical guarantees on the predicted confidences . We focus on on-distribution guarantees— i.e. , where the test distribution is the same as the training distribution . In this setting , we can build on ideas from statistical learning theory to provide probably approximately correctness ( PAC ) guarantees ( Valiant , 1984 ) . Our approach is based on a calibrated prediction technique called histogram binning ( Zadrozny & Elkan , 2001 ) , which rescales the confidences by binning them and then rescaling each bin independently . We use Clopper-Pearson bounds on the tails of the binomial distribution to obtain PAC upper/lower bounds on the predicted confidences . Next , we study how it enables theoretical guarantees in two applications . First , we consider the problem of speeding up DNN inference by composing a fast but inaccurate model with a slow but accurate model—i.e. , by using the accurate model for inference only if the confidence of the inaccurate one is underconfident ( Teerapittayanon et al. , 2016 ) . We use our algorithm to obtain guarantees on accuracy of the composed model . Second , for safe planning , we consider using a DNN to predict whether or not a given action ( e.g. , move forward ) is safe ( e.g. , do not run into obstacles ) given an observation ( e.g. , a camera image ) . The robot only continues to act if the predicted confidence is above some threshold . We use our algorithm to ensure safety with high probability . Finally , we evaluate the efficacy of our approach in the context of these applications . Related work . Calibrated prediction ( Murphy , 1972 ; DeGroot & Fienberg , 1983 ; Platt , 1999 ) has recently gained attention as a way to improve DNN confidences ( Guo et al. , 2017 ) . Histogram binning is a non-parametric approach that sorts the data into finitely many bins and rescales the confidences per bin ( Zadrozny & Elkan , 2001 ; 2002 ; Naeini et al. , 2015 ) . However , traditional approaches do not provide theoretical guarantees on the predicted confidences . There has been work on predicting confidence sets ( i.e. , predict a set of labels instead of a single label ) with theoretical guarantees ( Park et al. , 2020a ) , but this approach does not provide the confidence of the most likely prediction , as is often desired . There has also been work providing guarantees on the overall calibration error ( Kumar et al. , 2019 ) , but this approach does not provide per-prediction guarantees . There has been work speeding up DNN inference ( Hinton et al. , 2015 ) . One approach is to allow intermediate layers to be dynamically skipped ( Teerapittayanon et al. , 2016 ; Figurnov et al. , 2017 ; Wang et al. , 2018 ) , which can be thought of as composing multiple models that share a backbone . Unlike our approach , they do not provide guarantees on the accuracy of the composed model . There has also been work on safe learning-based control ( Akametalu et al. , 2014 ; Fisac et al. , 2019 ; Bastani , 2019 ; Li & Bastani , 2020 ; Wabersich & Zeilinger , 2018 ; Alshiekh et al. , 2018 ) ; however , these approaches are not applicable to perception-based control . The most closely related work is Dean et al . ( 2019 ) , which handles perception , but they are restricted to known linear dynamics . 2 PAC CONFIDENCE PREDICTION . In this section , we begin by formalizing the PAC confidence coverage prediction problem ; then , we describe our algorithm for solving this problem based on histogram binning . Calibrated prediction . Let x ∈ X be an example and y ∈ Y be one of a finite label set , and let D be a distribution over X × Y . A confidence predictor is a model f̂ : X → PY , where PY is the space of probability distributions over labels . In particular , f̂ ( x ) y is the predicted confidence that the true label for x is y . We let ŷ : X → Y be the corresponding label predictor—i.e. , ŷ ( x ) : = arg maxy∈Y f̂ ( x ) y—and let p̂ : X → R≥0 be corresponding top-label confidence predictor— i.e. , p̂ ( x ) : = maxy∈Y f̂ ( x ) y . While traditional DNN classifiers are confidence predictors , a naively trained DNN is not reliable—i.e. , predicted confidence does not match to the true confidence ; recent work has studied heuristics for improving reliability ( Guo et al. , 2017 ) . In contrast , our goal is to construct a confidence predictor that comes with theoretical guarantees . We first introduce the definition of calibration ( DeGroot & Fienberg , 1983 ; Zadrozny & Elkan , 2002 ; Park et al. , 2020b ) —i.e. , what we mean for a predicted confidence to be “ correct ” . In many cases , the main quantity of interest is the confidence of the top prediction . Thus , we focus on ensuring that the top-label predicted confidence p̂ ( x ) is calibrated ( Guo et al. , 2017 ) ; our approach can easily be extended to providing guarantees on all confidences predicted using f̂ . Then , we say a confidence predictor f̂ is well-calibrated with respect to distribution D if P ( x , y ) ∼D [ y = ŷ ( x ) | p̂ ( x ) = t ] = t ( ∀t ∈ [ 0 , 1 ] ) . That is , among all examples x such that the label prediction ŷ ( x ) has predicted confidence t = p̂ ( x ) , ŷ ( x ) is the correct label for exactly a t fraction of these examples . Using a change of variables ( Park et al. , 2020b ) , f̂ being well-calibrated is equivalent to p̂ ( x ) = c∗ f̂ ( x ) : = P ( x′ , y′ ) ∼D [ y ′ = ŷ ( x′ ) | p̂ ( x′ ) = p̂ ( x ) ] ( ∀x ∈ X ) . ( 1 ) Then , the goal of well-calibration is to make p̂ equal to c∗ f̂ . Note that f̂ appears on both sides of the equation p̂ ( x ) = c∗ f̂ ( x ) —implicitly in p̂—which is what makes it challenging to satisfy . Indeed , in general , it is unlikely that ( 1 ) holds exactly for all x . Instead , based on the idea of histogram binning ( Zadrozny & Elkan , 2001 ) , we consider a variant where we partition the data into a fixed number of bins and then construct confidence coverages separately for each bin . In particular , consider K bins B1 , . . . , BK ⊆ [ 0 , 1 ] , where B1 = [ 0 , b1 ] and Bk = ( bk−1 , bk ] for k > 1 . Here , K and 0 ≤ b1 ≤ · · · ≤ bK−1 ≤ bK = 1 are hyperparameters . Given f̂ , let κf̂ : X → { 1 , . . . , K } to denote the index of the bin that contains p̂ ( x ) —i.e. , p̂ ( x ) ∈ Bκf̂ ( x ) . Definition 1 We say f̂ is well-calibrated for a distribution D and bins B1 , . . . , BK if p̂ ( x ) = cf̂ ( x ) : = P ( x′ , y′ ) ∼D [ y′ = ŷ ( x′ ) ∣∣∣ p̂ ( x′ ) ∈ Bκf̂ ( x ) ] ( ∀x ∈ X ) , ( 2 ) where we refer to cf̂ ( x ) as the true confidence . Intuitively , this definition “ coarsens ” the calibration problem across the bins—i.e. , rather than sorting the inputs x into a continuum of “ bins ” p̂ ( x ) = t for each t ∈ [ 0 , 1 ] as in ( 1 ) , we sort them into a finite number of bins p̂ ( x ) ∈ Bk ; intuitively , we have c∗ f̂ ≈ cf̂ if the bin sizes are close to zero . It may not be obvious what downstream guarantees can be obtained based on this definition ; we provide examples in Sections 3 & 4 . Problem formulation . We formalize the problem of PAC calibration . We focus on the setting where the training and test distributions are identical—e.g. , we can not handle adversarial examples or changes in covariate distribution ( e.g. , common in reinforcement learning ) . Importantly , while we assume a pre-trained confidence predictor f̂ is given , we make no assumptions about f̂—e.g. , it can be uncalibrated or heuristically calibrated . If f̂ performs poorly , then the predicted confidences will be close to 1/|Y|—i.e. , express no confidence in the predictions . Thus , it is fine if f̂ is poorly calibrated ; the important property is that the confidence predictor f̂ have similar true confidences . The challenge in formalizing PAC calibration is that quantifying over all x in ( 2 ) . One approach is to provide guarantees in expectation over x ( Kumar et al. , 2019 ) ; however , this approach does not provide guarantees for individual predictions . Instead , our goal is to find a set of predicted confidences that includes the true confidence with high probability . Of course , we could simply predict the interval [ 0 , 1 ] , which always contains the true confidence ; thus , simultaneously want to make the size of the interval small . To this end , we consider a confidence coverage predictor Ĉ : X → 2R , where cf̂ ( x ) ∈ Ĉ ( x ) with high probability . In particular , Ĉ ( x ) outputs an interval [ c , c ] ⊆ R , where c ≤ c , instead of a set . We only consider a single interval ( rather than disjoint intervals ) since one suffices to localize the true confidence cf̂ . We are interested in providing theoretical guarantees for an algorithm used to construct confidence coverage predictor Ĉ given a held-out calibration set Z ⊆ X × Y . In addition , we assume the algorithm is given a pretrained confidence predictor f̂ . Thus , we consider Ĉ as depending on Z and f̂ , which we denote by Ĉ ( · ; f̂ , Z ) . Then , we want Ĉ to satisfy the following guarantee : Definition 2 Given δ ∈ R > 0 and n ∈ N , Ĉ is probably approximately correct ( PAC ) if for any D , PZ∼Dn [ ∧ x∈X cf̂ ( x ) ∈ Ĉ ( x ; f̂ , Z ) ] ≥ 1− δ . ( 3 ) Note that cf̂ depends on D. Here , “ approximately correct ” technically refers to the mean of Ĉ ( x ; f̂ , Z ) , which is an estimate of cf̂ ( x ) ; the interval captures the bound on the error of this estimate ; see Appendix A for details . Furthermore , the conjunction over all x ∈ X may seem strong . We can obtain such a guarantee due to our binning strategy : the property cf̂ ( x ) ∈ Ĉ ( x ; f̂ , Z ) only depends on the bin Bκf̂ ( x ) , so the conjunction is really only over bins k ∈ { 1 , ... , K } . Algorithm . We propose a confidence coverage predictor that satisfies the PAC property . The problem of estimating the confidence interval Ĉ ( x ) of the binned true confidence cf̂ ( x ) is closely related to the binomial proportion confidence interval estimation ; consider a Bernoulli random variable b ∼ B : = Bernoulli ( θ ) for any θ ∈ [ 0 , 1 ] , where b = 1 denotes a success and b = 0 denotes a failure , and θ is unknown . Given a sequence of observations b1 : n : = ( b1 , . . . , bn ) ∼ Bn , the goal is to construct an interval Θ̂ ( b1 : n ) ⊆ R that includes θ with high probability—i.e. , Pb1 : n∼Bn [ θ ∈ Θ̂ ( b1 : n ) ] ≥ 1− α , ( 4 ) where α ∈ R > 0 is a given confidence level . In particular , the Clopper-Pearson interval Θ̂CP ( b1 : n ; α ) : = [ inf θ { θ ∣∣∣ Pθ [ S ≥ s ] ≥ α 2 } , sup θ { θ ∣∣∣ Pθ [ S ≤ s ] ≥ α 2 } ] , guarantees ( 4 ) ( Clopper & Pearson , 1934 ; Brown et al. , 2001 ) , where s = ∑n i=1 bi is the number of observed successes , n is the number of observations , and S is a Binomial random variable S ∼ Binomial ( n , θ ) . Intuitively , the interval is constructed such that the number of observed success falls in the region with high-probability for any θ in the interval . The following expression is equivalent due to the relationship between the Binomial and Beta distributions ( Hartley & Fitch , 1951 ; Brown et al. , 2001 ) —i.e. , Pθ [ S ≥ s ] = Iθ ( s , n− s+ 1 ) , where Iθ is the CDF of Beta ( s , n− s+ 1 ) : Θ̂CP ( b1 : n ; α ) = [ α 2 quantile of Beta ( s , n− s+ 1 ) , ( 1− α 2 ) quantile of Beta ( s+ 1 , n− s ) ] . Now , for each of the K bins , we apply Θ̂CP with confidence level α = δK—i.e. , Ĉ ( x ; f̂ , Z , δ ) : = Θ̂CP ( Wκf̂ ( x ) ; δ K ) where Wk : = { 1 ( ŷ ( x ) = y ) ∣∣∣ ( x , y ) ∈ Z s.t . κf̂ ( x ) = k } . Here , Wk is the set of observations of successes vs. failures corresponding to the subset of labeled examples ( x , y ) ∈ Z such that p̂ ( x ) falls in the bin Bk , where a success is defined to be a correct prediction ŷ ( x ) = y . We note that for efficiency , the confidence interval for each of the K bins can be precomputed . Our construction of Ĉ satisfies the following ; see Appendix B for a proof : Theorem 1 Our confidence coverage predictor Ĉ is PAC for any δ ∈ R > 0 and n ∈ N. Note that Clopper-Pearson intervals are exact , ensuring the size of Ĉ for each bin is small in practice . Finally , an important special case is when there is a single bin B = [ 0 , 1 ] —i.e. , Ĉ0 ( x ; f̂ , Z ′ , δ ) : = Θ̂CP ( W ; δ ) where W : = { 1 ( ŷ ( x′ ) = y′ ) | ( x′ , y′ ) ∈ Z ′ } . Note that Ĉ0 does not depend on x , so we drop it—i.e. , Ĉ0 ( f̂ , Z ′ , δ ) : = Θ̂CP ( W ; δ ) —i.e. , Ĉ0 computes the Clopper-Pearson interval over Z ′ , which is a subset of the original set Z .

Summary: This paper proposes a method for obtaining probably-approximately correct (PAC) predictions given a pre-trained classifier. The PAC intervals are connected to calibration, and take the form of confidence intervals given the bin a prediction falls in. They demonstrate and explore two use cases: applying this technique to get faster inference in deep neural networks, and using the PAC predictor to do safe planning. Experiments in both of these cases show improvements in speed-accuracy or safety-accuracy tradeoffs, as compared to baselines.