Top-label calibration and multiclass-to-binary reductions

1 INTRODUCTION . Machine learning models often make probabilistic predictions over the output value , with the ideal goal being to learn the true probability distribution of the output given the input . However , nature never reveals true probability distributions , making it infeasible to achieve this ideal goal in most practical situations . Instead , there is significant interest in the ML community on designing models that achieve the simpler goal of being calibrated . Consider the problem of predicting the probability of rain . When a meteorologist claims that the probability of rain on a particular day is 0.7 , we can not actually test if 0.7 was the ‘ true probability of rain ’ without making unverifiable assumptions ( Dawid , 1982 ) . However , we can test if the meteorologist is calibrated by requiring that on the D days when 0.7 is predicted , it indeed rained on roughly 0.7D of those days ( and the same is true for other probabilities ) . Arguably , a calibrated meteorologist is more interpretable than an uncalibrated one — in the same sense , an ML classification model is arguably more interpretable if the probabilistic predictions it makes for the classes are calibrated ( Platt , 1999 ; Zadrozny and Elkan , 2001 ; Niculescu-Mizil and Caruana , 2005 ) . The rain-prediction example is readily converted into a formal notion of calibration for binary classification . Denote a random ( feature , label ) -pair as pX , Y q P X ˆt0 , 1u . A predictor h : X Ñ r0 , 1s is said to be calibrated if PrpY “ 1 | hpXq “ qq “ q for every q P r0 , 1s ( almost surely ) . If the output is multiclass , Y P rLs : “ t1 , 2 , . . . , L ě 3u , we consider the calibration of a multi-dimensional predictor h that makes predictions in the pL´1q-dimensional probability simplex : h : X Ñ ∆L´1 . The natural extension of binary calibration yields that h is calibrated if PrpY “ l | hpXq “ qq “ ql for every l P rLs , where ql denotes the l ’ th component of q . However , this notion of calibration becomes infeasible to achieve or verify once L is even 4 or 5 ( Vaicenavicius et al. , 2019 ) . Thus a number of relaxations of this notion have become popular in the ML literature : confidence calibration ( Guo et al. , 2017 ) , class-wise calibration ( Kull et al. , 2017 ) , and extensions of these such as thresholded-class-wise calibration ( Nixon et al. , 2020 ) and top-K confidence calibration ( Gupta et al. , 2021 ) . Each of these relaxations assess predictors by focusing on one or more binary calibration claims . In our work , we dig deeper into what these notions of multiclass calibration mean in terms of the associated probabilistic claim they are verifying . Our study leads us to a new notion of calibration — top-label calibration — and novel calibration algorithms ( henceforth called ‘ calibrators ’ ) for achieving multiclass calibration . A summary of our contributions is below : ( a ) In Section 2 , we demonstrate that while confidence calibration is a simple and popular onedimensional view of calibration , it does not lead to a practically meaningful claim . Our proposed notion of top-label calibration corresponds to a subtle but critical change to confidence calibration that rectifies this issue . In contrast to all previous works that have proposed to fix confidence calibration by optimizing the full calibration vector ( Kull et al. , 2019 ; Nixon et al. , 2020 ; Kumar et al. , 2019 ; Zadrozny and Elkan , 2002 ; Vaicenavicius et al. , 2019 ; Widmann et al. , 2019 ) , our work continues to advocate for calibrating probabilities only for the top predicted class . ( b ) In Section 3 , we show that most popular notions of multiclass calibration internally verify binary calibration claims . We characterize these as ‘ multiclass-to-binary ’ or ‘ M2B ’ notions . The benefit of this characterization is that for any M2B notion of calibration , a unique customized M2B calibrator can be designed that is specifically tuned to the given notion . ( c ) In Section 4 , we instantiate the M2B approach using the binary calibration algorithm of histogram binning ( Zadrozny and Elkan , 2001 ; Gupta and Ramdas , 2021 ) . We show that histogram binning , when used with the right M2B wrapper , ( i ) allows for strong theoretical guarantees without distributional assumptions , and ( ii ) achieves state-of-the-art results with respect to certain standard notions of calibration error . We briefly note that identifying the right notion of calibration ( i.e . practically useful , philosophically meaningful , and statistically tractable ) is also an important problem for structured output spaces ( Kuleshov et al. , 2018 ; Gneiting et al. , 2007 ) . While this paper is entirely in the multiclass setting , our ideas have implications for identifying the right notion of calibration for other output spaces . 2 MODIFYING CONFIDENCE CALIBRATION TO TOP-LABEL CALIBRATION . A commonly used notion of calibration in multiclass classification settings is confidence calibration , a simple ‘ one-dimensional ’ reduction to binary calibration proposed by Guo et al . ( 2017 ) . For the rest of this paper , we use P for the data-generating distribution over X ˆ rLs , and we denote a random point from this distribution as pX , Y q „ P . Let c : X Ñ rLs denote a classifier or ‘ toplabel predictor ’ and h : X Ñ r0 , 1s a function that provides a confidence or probability score for the top-label cpXq . The predictor pc , hq is said to be confidence calibrated ( for P ) if ( confidence calibration ) P pY “ cpXq | hpXqq “ hpXq . ( 1 ) In other words , when the reported confidence hpXq equals p P r0 , 1s , then the fraction of instances where the predicted label is correct also approximately equals p. Note that for an L-dimensional predictor h : X Ñ ∆L´1 , one would use cp¨q “ arg maxlPrLs hlp¨q and hp¨q “ hcp¨qp¨q ; ties are broken arbitrarily . Then h is confidence calibrated if the corresponding pc , hq satisfies ( 1 ) . Confidence calibration is most applicable in high-accuracy settings where we trust the label prediction cpxq . For instance , if a high-accuracy cancer-grade-prediction model predicts a patient as having “ 95 % grade III , 3 % grade II , and 2 % grade I ” , we would suggest the patient to undergo an intrusive treatment . However , we may want to know ( and control ) the number of non-grade-III patients that were given this suggestion incorrectly . In other words , is Prpcancer is not grade III | cancer is predicted to be of grade III with confidence 95 % q equal to 5 % ? Confidence calibration attempts to capture this intuition by focusing on the the probability of the predicted label . However , as we illustrate next , confidence calibration fails at this goal by providing a guarantee that is neither practically interpretable , nor actionable . Translating the probabilistic statement ( 1 ) into words , we ascertain that confidence calibration leads to guarantees of the form : “ if the confidence hpXq in the top-label is 0.6 , then the accuracy ( frequency with which Y equals cpXq ) is 0.6 ” . How useful is such a guarantee ? Not very useful . Suppose a patient P is informed ( based on their symptoms X ) , they are most likely to have a certain disease D with probability 0.6 . Further patient P is told that this score is confidence calibrated . Based on the previous paragraph , P can now infer the following : “ among all patients who have probability 0.6 of having some unspecified disease , the fraction who have that unspecified disease is also 0.6. ” However , P is concerned only about disease D , and not about other diseases . That is , P wants to know the probability of having D among patients who were predicted to have disease D with confidence 0.6 , not among patients who were predicted to have some disease with confidence 0.6 . In other words , P cares about the occurrence of D among patients who were told the same thing that P has been told . It is tempting to wish that the confidence calibrated probability 0.6 has any bearing on what P cares about . However , this faith is misguided , as the above reasoning suggests , and further illustrated through the following example . Example 1 . Suppose the instance space is pX , Y q P ta , bu ˆ t1 , 2 , . . .u . ( X can be seen as the random patient , and Y as the disease they are suffering from . ) Consider a predictor pc , hq and let the values taken by pX , Y , c , hq be as follows : Feature x P pX “ xq Class prediction cpxq Confidence hpxq P pY “ cpXq | X “ xq a 0.5 1 0.6 0.2 b 0.5 2 0.6 1.0 The table specifies only the probabilities P pY “ cpXq | X “ xq ; the probabilities P pY “ l | X “ xq for l ‰ cpxq can be set arbitrarily . We verify that pc , hq is perfectly confidence calibrated : P pY “ cpXq | hpXq “ 0.6q “ 0.5pP pY “ 1 | X “ aq ` P pY “ 2 | X “ bqq “ 0.5p0.2 ` 1q “ 0.6 . However , whether the actual practical instance is X “ a or X “ b , the probabilistic claim of 0.6 bears no correspondence with reality . If X “ a , hpXq “ 0.6 is extremely overconfident since P pY “ 1 | X “ aq “ 0.2 . On the other hand , if X “ b , hpXq “ 0.6 is extremely underconfident since P pY “ 1 | X “ bq “ 1.0 . The reason for the strange behavior above is that the probability P pY “ cpXq | hpXqq is not interpretable from a decision-making perspective . In practice , we never report just the confidence hpXq , but also the class prediction cpXq ( obviously ! ) . Thus it is more reasonable to talk about the conditional probability of Y “ cpXq , given what we know , that is both cpXq and hpXq . Thus , we make a small but critical change to ( 1 ) ; we say that pc , hq is top-label calibrated if ( top-label calibration ) P pY “ cpXq | hpXq , cpXqq “ hpXq . ( 2 ) ( See the disambiguating Remark 2 on terminology . ) Going back to the patient-disease example , top-label calibration would tell patient P the following : “ among all patients , who ( just like you ) are predicted to have disease D with probability 0.6 , the fraction who actually have disease D is also 0.6. ” Philosophically , it makes perfect sense to condition on what is reported — both the top label and its confidence — because that is what is known to the recipient of the information ; and there is no apparent justification for not conditioning on both . A commonly used metric for quantifying the miscalibration of a model is the expected-calibrationerror ( ECE ) metric . The ECE associated with confidence calibration is defined as conf-ECEpc , hq : “ EX |P pY “ cpXq | hpXqq ´ hpXq| . ( 3 ) We define top-label-ECE ( TL-ECE ) in an analogous fashion , but also condition on cpXq : TL-ECEpc , hq : “ EX |P pY “ cpXq | cpXq , hpXqq ´ hpXq| . ( 4 ) Higher values of ECE indicate worse calibration performance . The predictor in Example 1 has conf-ECEpc , hq “ 0 . However , it has TL-ECEpc , hq “ 0.4 , revealing its miscalibration . More generally , it can be deduced as a straightforward consequence of Jensen ’ s inequality that conf-ECEpc , hq is always smaller than the TL-ECEpc , hq ( see Proposition 4 in Appendix G ) . As illustrated by Example 1 , the difference can be significant . In the following subsection we illustrate that the difference can be significant on a real dataset as well . First , we make a couple of remarks . Remark 1 ( ECE estimation using binning ) . Estimating the ECE requires estimating probabilities conditional on some prediction such as hpxq . A common strategy to do this is to ‘ bin ’ together nearby values of hpxq using binning schemes ( Nixon et al. , 2020 , Section 2.1 ) , and compute a single estimate for the predicted and true probability using all the points in a bin . The calibration method we espouse in this work is histogram binning ( HB ) , a method whose output is already discrete and does not require further binning . Based on this , we use the following experimental protocol : we report unbinned ECE estimates while assessing HB , and binned ECE estimates for all other compared methods , which are continuous output methods ( deep-nets , temperature scaling , etc ) . It is commonly understood that binning leads to underestimation of the effective ECE ( Vaicenavicius et al. , 2019 ; Kumar et al. , 2019 ) . Thus , using unbinned ECE estimates for HB gives HB a disadvantage compared to the binned ECE estimates we use for other methods . ( This further strengthens our positive results for HB . ) The binning scheme we use is equal-width binning , where the interval r0 , 1s is divided into B equal-width intervals . Equal-width binning typically leads to lower ECE estimates compared to adaptive-width binning ( Nixon et al. , 2020 ) . Remark 2 ( Terminology ) . The term conf-ECE was introduced by Kull et al . ( 2019 ) . Most works refer to conf-ECE as just ECE ( Guo et al. , 2017 ; Nixon et al. , 2020 ; Mukhoti et al. , 2020 ; Kumar et al. , 2018 ) . However , some papers refer to conf-ECE as ‘ top-label-ECE ’ ( Kumar et al. , 2019 ; Zhang et al. , 2020 ) , resulting in two different terms for the same concept . We call the older notion as conf-ECE , and our definition of top-label calibration/ECE ( 4 ) is different from previous ones . ( a ) Confidence reliability diagram ( points marked ‹ ) and top-label reliability diagram ( points marked ` ) for a ResNet-50 model on the CIFAR-10 dataset ; see further details in points ( a ) and ( b ) below . The gray bars denote the fraction of predictions in each bin . The confidence reliability diagram ( mistakenly ) suggests better calibration than the top-label reliability diagram . ( b ) Class-wise and zoomed-in version of Figure 1a for bin 6 ( top ) and bin 10 ( bottom ) ; see further details in point ( c ) below . The ‹ markers are in the same position as Figure 1a , and denote the average predicted and true probabilities . The colored points denote the predicted and true probabilities when seen class-wise . The histograms on the right show the number of test points per class within bins 6 and 10 . Figure 1 : Confidence reliability diagrams misrepresent the effective miscalibration .

Summary: The paper proposes a new method to fix confidence calibration by conditioning on the predicted label when formulating the calibration task (as well as on the class probability estimate for that label, as done in confidence calibration). This new approach is called "top-label" calibration. The paper shows how to modify the calibration process to achieve this by building a separate calibration model for each predicted class value. Experiments compare this approach, using binning as the calibration method, to confidence calibration as well as standard 1-vs-rest calibration (both with and without normalization in the latter method) based on calibration of pre-trained CIFAR-10/100 CNNs. Temperature scaling, vector scaling, and Dirichlet scaling are also included. Evaluation is performed using 1-vs-rest ECE, a new top-label ECE, and a maximum calibration error (MCE) variant of the latter. Surprisingly, the proposed approach does not perform as well as 1-vs-rest calibration when using the top-label variant of ECE; it does perform a lot better wrt the MCE variant on the CIFAR-10 data though. When considering 1-vs-rest ECE, the paper reports that 1-vs-rest *without* normalization performs better than the same approach with normalization, and also better than all scaling-based methods. The paper also explains how to adapt top-K-confidence calibration to obtain top-K-label calibration, but this is not evaluated.