Top-label calibration and multiclass-to-binary reductions

1 INTRODUCTION . Machine learning models often make probabilistic predictions over the output value , with the ideal goal being to learn the true probability distribution of the output given the input . However , nature never reveals true probability distributions , making it infeasible to achieve this ideal goal in most practical situations . Instead , there is significant interest in the ML community on designing models that achieve the simpler goal of being calibrated . Consider the problem of predicting the probability of rain . When a meteorologist claims that the probability of rain on a particular day is 0.7 , we can not actually test if 0.7 was the ‘ true probability of rain ’ without making unverifiable assumptions ( Dawid , 1982 ) . However , we can test if the meteorologist is calibrated by requiring that on the D days when 0.7 is predicted , it indeed rained on roughly 0.7D of those days ( and the same is true for other probabilities ) . Arguably , a calibrated meteorologist is more interpretable than an uncalibrated one — in the same sense , an ML classification model is arguably more interpretable if the probabilistic predictions it makes for the classes are calibrated ( Platt , 1999 ; Zadrozny and Elkan , 2001 ; Niculescu-Mizil and Caruana , 2005 ) . The rain-prediction example is readily converted into a formal notion of calibration for binary classification . Denote a random ( feature , label ) -pair as pX , Y q P X ˆt0 , 1u . A predictor h : X Ñ r0 , 1s is said to be calibrated if PrpY “ 1 | hpXq “ qq “ q for every q P r0 , 1s ( almost surely ) . If the output is multiclass , Y P rLs : “ t1 , 2 , . . . , L ě 3u , we consider the calibration of a multi-dimensional predictor h that makes predictions in the pL´1q-dimensional probability simplex : h : X Ñ ∆L´1 . The natural extension of binary calibration yields that h is calibrated if PrpY “ l | hpXq “ qq “ ql for every l P rLs , where ql denotes the l ’ th component of q . However , this notion of calibration becomes infeasible to achieve or verify once L is even 4 or 5 ( Vaicenavicius et al. , 2019 ) . Thus a number of relaxations of this notion have become popular in the ML literature : confidence calibration ( Guo et al. , 2017 ) , class-wise calibration ( Kull et al. , 2017 ) , and extensions of these such as thresholded-class-wise calibration ( Nixon et al. , 2020 ) and top-K confidence calibration ( Gupta et al. , 2021 ) . Each of these relaxations assess predictors by focusing on one or more binary calibration claims . In our work , we dig deeper into what these notions of multiclass calibration mean in terms of the associated probabilistic claim they are verifying . Our study leads us to a new notion of calibration — top-label calibration — and novel calibration algorithms ( henceforth called ‘ calibrators ’ ) for achieving multiclass calibration . A summary of our contributions is below : ( a ) In Section 2 , we demonstrate that while confidence calibration is a simple and popular onedimensional view of calibration , it does not lead to a practically meaningful claim . Our proposed notion of top-label calibration corresponds to a subtle but critical change to confidence calibration that rectifies this issue . In contrast to all previous works that have proposed to fix confidence calibration by optimizing the full calibration vector ( Kull et al. , 2019 ; Nixon et al. , 2020 ; Kumar et al. , 2019 ; Zadrozny and Elkan , 2002 ; Vaicenavicius et al. , 2019 ; Widmann et al. , 2019 ) , our work continues to advocate for calibrating probabilities only for the top predicted class . ( b ) In Section 3 , we show that most popular notions of multiclass calibration internally verify binary calibration claims . We characterize these as ‘ multiclass-to-binary ’ or ‘ M2B ’ notions . The benefit of this characterization is that for any M2B notion of calibration , a unique customized M2B calibrator can be designed that is specifically tuned to the given notion . ( c ) In Section 4 , we instantiate the M2B approach using the binary calibration algorithm of histogram binning ( Zadrozny and Elkan , 2001 ; Gupta and Ramdas , 2021 ) . We show that histogram binning , when used with the right M2B wrapper , ( i ) allows for strong theoretical guarantees without distributional assumptions , and ( ii ) achieves state-of-the-art results with respect to certain standard notions of calibration error . We briefly note that identifying the right notion of calibration ( i.e . practically useful , philosophically meaningful , and statistically tractable ) is also an important problem for structured output spaces ( Kuleshov et al. , 2018 ; Gneiting et al. , 2007 ) . While this paper is entirely in the multiclass setting , our ideas have implications for identifying the right notion of calibration for other output spaces . 2 MODIFYING CONFIDENCE CALIBRATION TO TOP-LABEL CALIBRATION . A commonly used notion of calibration in multiclass classification settings is confidence calibration , a simple ‘ one-dimensional ’ reduction to binary calibration proposed by Guo et al . ( 2017 ) . For the rest of this paper , we use P for the data-generating distribution over X ˆ rLs , and we denote a random point from this distribution as pX , Y q „ P . Let c : X Ñ rLs denote a classifier or ‘ toplabel predictor ’ and h : X Ñ r0 , 1s a function that provides a confidence or probability score for the top-label cpXq . The predictor pc , hq is said to be confidence calibrated ( for P ) if ( confidence calibration ) P pY “ cpXq | hpXqq “ hpXq . ( 1 ) In other words , when the reported confidence hpXq equals p P r0 , 1s , then the fraction of instances where the predicted label is correct also approximately equals p. Note that for an L-dimensional predictor h : X Ñ ∆L´1 , one would use cp¨q “ arg maxlPrLs hlp¨q and hp¨q “ hcp¨qp¨q ; ties are broken arbitrarily . Then h is confidence calibrated if the corresponding pc , hq satisfies ( 1 ) . Confidence calibration is most applicable in high-accuracy settings where we trust the label prediction cpxq . For instance , if a high-accuracy cancer-grade-prediction model predicts a patient as having “ 95 % grade III , 3 % grade II , and 2 % grade I ” , we would suggest the patient to undergo an intrusive treatment . However , we may want to know ( and control ) the number of non-grade-III patients that were given this suggestion incorrectly . In other words , is Prpcancer is not grade III | cancer is predicted to be of grade III with confidence 95 % q equal to 5 % ? Confidence calibration attempts to capture this intuition by focusing on the the probability of the predicted label . However , as we illustrate next , confidence calibration fails at this goal by providing a guarantee that is neither practically interpretable , nor actionable . Translating the probabilistic statement ( 1 ) into words , we ascertain that confidence calibration leads to guarantees of the form : “ if the confidence hpXq in the top-label is 0.6 , then the accuracy ( frequency with which Y equals cpXq ) is 0.6 ” . How useful is such a guarantee ? Not very useful . Suppose a patient P is informed ( based on their symptoms X ) , they are most likely to have a certain disease D with probability 0.6 . Further patient P is told that this score is confidence calibrated . Based on the previous paragraph , P can now infer the following : “ among all patients who have probability 0.6 of having some unspecified disease , the fraction who have that unspecified disease is also 0.6. ” However , P is concerned only about disease D , and not about other diseases . That is , P wants to know the probability of having D among patients who were predicted to have disease D with confidence 0.6 , not among patients who were predicted to have some disease with confidence 0.6 . In other words , P cares about the occurrence of D among patients who were told the same thing that P has been told . It is tempting to wish that the confidence calibrated probability 0.6 has any bearing on what P cares about . However , this faith is misguided , as the above reasoning suggests , and further illustrated through the following example . Example 1 . Suppose the instance space is pX , Y q P ta , bu ˆ t1 , 2 , . . .u . ( X can be seen as the random patient , and Y as the disease they are suffering from . ) Consider a predictor pc , hq and let the values taken by pX , Y , c , hq be as follows : Feature x P pX “ xq Class prediction cpxq Confidence hpxq P pY “ cpXq | X “ xq a 0.5 1 0.6 0.2 b 0.5 2 0.6 1.0 The table specifies only the probabilities P pY “ cpXq | X “ xq ; the probabilities P pY “ l | X “ xq for l ‰ cpxq can be set arbitrarily . We verify that pc , hq is perfectly confidence calibrated : P pY “ cpXq | hpXq “ 0.6q “ 0.5pP pY “ 1 | X “ aq ` P pY “ 2 | X “ bqq “ 0.5p0.2 ` 1q “ 0.6 . However , whether the actual practical instance is X “ a or X “ b , the probabilistic claim of 0.6 bears no correspondence with reality . If X “ a , hpXq “ 0.6 is extremely overconfident since P pY “ 1 | X “ aq “ 0.2 . On the other hand , if X “ b , hpXq “ 0.6 is extremely underconfident since P pY “ 1 | X “ bq “ 1.0 . The reason for the strange behavior above is that the probability P pY “ cpXq | hpXqq is not interpretable from a decision-making perspective . In practice , we never report just the confidence hpXq , but also the class prediction cpXq ( obviously ! ) . Thus it is more reasonable to talk about the conditional probability of Y “ cpXq , given what we know , that is both cpXq and hpXq . Thus , we make a small but critical change to ( 1 ) ; we say that pc , hq is top-label calibrated if ( top-label calibration ) P pY “ cpXq | hpXq , cpXqq “ hpXq . ( 2 ) ( See the disambiguating Remark 2 on terminology . ) Going back to the patient-disease example , top-label calibration would tell patient P the following : “ among all patients , who ( just like you ) are predicted to have disease D with probability 0.6 , the fraction who actually have disease D is also 0.6. ” Philosophically , it makes perfect sense to condition on what is reported — both the top label and its confidence — because that is what is known to the recipient of the information ; and there is no apparent justification for not conditioning on both . A commonly used metric for quantifying the miscalibration of a model is the expected-calibrationerror ( ECE ) metric . The ECE associated with confidence calibration is defined as conf-ECEpc , hq : “ EX |P pY “ cpXq | hpXqq ´ hpXq| . ( 3 ) We define top-label-ECE ( TL-ECE ) in an analogous fashion , but also condition on cpXq : TL-ECEpc , hq : “ EX |P pY “ cpXq | cpXq , hpXqq ´ hpXq| . ( 4 ) Higher values of ECE indicate worse calibration performance . The predictor in Example 1 has conf-ECEpc , hq “ 0 . However , it has TL-ECEpc , hq “ 0.4 , revealing its miscalibration . More generally , it can be deduced as a straightforward consequence of Jensen ’ s inequality that conf-ECEpc , hq is always smaller than the TL-ECEpc , hq ( see Proposition 4 in Appendix G ) . As illustrated by Example 1 , the difference can be significant . In the following subsection we illustrate that the difference can be significant on a real dataset as well . First , we make a couple of remarks . Remark 1 ( ECE estimation using binning ) . Estimating the ECE requires estimating probabilities conditional on some prediction such as hpxq . A common strategy to do this is to ‘ bin ’ together nearby values of hpxq using binning schemes ( Nixon et al. , 2020 , Section 2.1 ) , and compute a single estimate for the predicted and true probability using all the points in a bin . The calibration method we espouse in this work is histogram binning ( HB ) , a method whose output is already discrete and does not require further binning . Based on this , we use the following experimental protocol : we report unbinned ECE estimates while assessing HB , and binned ECE estimates for all other compared methods , which are continuous output methods ( deep-nets , temperature scaling , etc ) . It is commonly understood that binning leads to underestimation of the effective ECE ( Vaicenavicius et al. , 2019 ; Kumar et al. , 2019 ) . Thus , using unbinned ECE estimates for HB gives HB a disadvantage compared to the binned ECE estimates we use for other methods . ( This further strengthens our positive results for HB . ) The binning scheme we use is equal-width binning , where the interval r0 , 1s is divided into B equal-width intervals . Equal-width binning typically leads to lower ECE estimates compared to adaptive-width binning ( Nixon et al. , 2020 ) . Remark 2 ( Terminology ) . The term conf-ECE was introduced by Kull et al . ( 2019 ) . Most works refer to conf-ECE as just ECE ( Guo et al. , 2017 ; Nixon et al. , 2020 ; Mukhoti et al. , 2020 ; Kumar et al. , 2018 ) . However , some papers refer to conf-ECE as ‘ top-label-ECE ’ ( Kumar et al. , 2019 ; Zhang et al. , 2020 ) , resulting in two different terms for the same concept . We call the older notion as conf-ECE , and our definition of top-label calibration/ECE ( 4 ) is different from previous ones . ( a ) Confidence reliability diagram ( points marked ‹ ) and top-label reliability diagram ( points marked ` ) for a ResNet-50 model on the CIFAR-10 dataset ; see further details in points ( a ) and ( b ) below . The gray bars denote the fraction of predictions in each bin . The confidence reliability diagram ( mistakenly ) suggests better calibration than the top-label reliability diagram . ( b ) Class-wise and zoomed-in version of Figure 1a for bin 6 ( top ) and bin 10 ( bottom ) ; see further details in point ( c ) below . The ‹ markers are in the same position as Figure 1a , and denote the average predicted and true probabilities . The colored points denote the predicted and true probabilities when seen class-wise . The histograms on the right show the number of test points per class within bins 6 and 10 . Figure 1 : Confidence reliability diagrams misrepresent the effective miscalibration .

Summary: This paper discusses the topic of classifier calibration in multi-class classification. The focus is on obtaining well-calibrated probabilities for the top-predicted classes. The authors give arguments, examples and experimental results to show that the commonly-used confidence calibration method suffers a number of shortcomings that makes it less useful in practice. As an alternative, top-label calibration is proposed, where calibration is analyzed on a per-class basis, when a specific class is the top class.   In addition, the authors discuss shortcomings of confidence reliability diagrams, and they propose multi-class to binary reductions for achieving top-label calibration. In the experiments several methods are compared on two classical image classification benchmarks.