Local Calibration: Metrics and Recalibration 

Probabilistic classifiers output confidence scores along with their predictions , and these confidence scores should be calibrated , i.e. , they should reflect the reliability of the prediction . Confidence scores that minimize standard metrics such as the expected calibration error ( ECE ) accurately measure the reliability on average across the entire population . However , it is in general impossible to measure the reliability of an individual prediction . In this work , we propose the local calibration error ( LCE ) to span the gap between average and individual reliability . For each individual prediction , the LCE measures the average reliability of a set of similar predictions , where similarity is quantified by a kernel function on a pretrained feature space and by a binning scheme over predicted model confidences . We show theoretically that the LCE can be estimated sample-efficiently from data , and empirically find that it reveals miscalibration modes that are more fine-grained than the ECE can detect . Our key result is a novel local recalibration method LoRe , to improve confidence scores for individual predictions and decrease the LCE . Experimentally , we show that our recalibration method produces more accurate confidence scores , which improves downstream fairness and decision making on classification tasks with both image and tabular data . 1 INTRODUCTION . Uncertainty estimation is extremely important in high stakes decision-making tasks . For example , a patient wants to know the probability that a medical diagnosis is correct ; an autonomous driving system wants to know the probability that a pedestrian is correctly identified . Uncertainty estimates are usually achieved by predicting a probability along with each classification . Ideally , we want to achieve individual calibration , i.e. , we want to predict the probability that each sample is misclassified . However , each sample is observed only once for most datasets ( e.g. , image classification datasets do not contain identical images ) , making it impossible to estimate , or even define , the probability of incorrect classification for individual samples . Because of this , commonly used metrics such as the expected calibration error ( ECE ) measure the gap between a classifier ’ s confidence and accuracy averaged across the entire dataset . Consequently , ECE can be accurately estimated but does not measure the reliability of individual predictions . In this work , we propose the local calibration error ( LCE ) , a calibration metric that spans the gap between fully global ( e.g. , ECE ) and fully individual calibration . Motivated by the success of kernel-based locality in other fields such as fairness ( where similar individuals should be treated similarly ) ( Dwork et al. , 2012 ; Pleiss et al. , 2017 ) and causal inference ( where matching techniques are used to find similar neighboring samples ) ( Stuart , 2010 ) , we approximate the probability of misclassification for an individual sample by computing the average classification error over similar samples , where similarity is measured by a kernel function in a pre-trained feature space and a binning scheme over predicted confidences . Intuitively , two samples are similar if they are close in a pretrained feature space and have similar predicted confidence scores . By choosing the bandwidth of the kernel function , we can trade off estimation accuracy and individuality : when the bandwidth is very large , we recover existing global calibration metrics ; when the bandwidth is small , we approximate individual calibration . We choose an intermediate bandwidth , so our metric can be accurately estimated , and provides some measurement on the reliability of individual predictions . Theoretically , we show that the LCE can be estimated with polynomially many samples if the kernel function is bounded . Empirically , we also show that for intermediate values of the bandwidth , the LCE can be accurately estimated and reveals modes of miscalibration that global metrics ( such as ECE ) fail to uncover . In addition , we introduce a non-parametric , post-hoc localized recalibration method ( LoRe ) , for lowering the LCE . Empirically , LoRe improves fairness by achieving low calibration error on all potentially sensitive subsets of the data , such as racial groups . Notably , it can do so without any prior knowledge of those groups , and is more effective than global methods at this task . In addition , our recalibration method improves decision making when there is a “ safe ” action that is selected whenever the predicted confidence is low . For example , an automated system which classifies tissue samples as cancerous should request a human expert opinion whenever it is unsure about a classification . In a simulation on an image classification dataset , we show that recalibrated prediction models more accurately choose whether to use the “ safe ” action , which improves the overall utility . In summary , the contributions of our paper are as follows . ( 1 ) We introduce a local calibration metric , the LCE , that is both easy to compute and can estimate the reliability of individual predictions . ( 2 ) We introduce a post-hoc localized recalibration method LoRe , that transforms a model ’ s confidence predictions to improve the local calibration . ( 3 ) We empirically evaluate LoRe on several downstream tasks and observe that LoRe improves fairness and decision-making more than existing baselines . 2 BACKGROUND AND RELATED WORK . 2.1 GLOBAL CALIBRATION METRICS . Consider a classification task that maps from some input domain ( e.g. , images ) X to a finite set of labels Y = { 1 , · · · , m } . A classifier is a pair ( f , p̂ ) where f : X → Y maps each input x ∈ X to a label y ∈ Y and p̂ : X → [ 0 , 1 ] maps each input x to a confidence value c. Let Pr be a joint distribution on X × Y ( e.g. , from which training or test data pairs ( x , y ) are drawn ) . The classifier ( f , p̂ ) is perfectly calibrated ( Guo et al. , 2017 ) with respect to Pr if for all c ∈ [ 0 , 1 ] Pr [ f ( X ) = Y | p̂ ( X ) = c ] = c. ( 1 ) To numerically measure how well a classifier is calibrated , the most commonly used metric is the expected calibration error ( ECE ) ( Naeini et al. , 2015 ; Guo et al. , 2017 ) , which measures the average absolute deviation from Eq . 1 over the domain . In practice , given a finite dataset , the ECE is approximated by binning . The predicted confidences p̂ are partitioned into bins B1 , . . . , Bk , and then a weighted average is taken of the absolute difference between the average confidence conf ( Bi ) and average accuracy acc ( Bi ) for each bin Bi : ECE ( f , p̂ ) : = k∑ i=1 |Bi| N |conf ( Bi ) − acc ( Bi ) | . ( 2 ) Similarly , the maximum calibration error ( MCE ) ( Naeini et al. , 2015 ; Guo et al. , 2017 ) measures the average deviation from Eq . 1 in the bin with the highest calibration error , and is defined as MCE ( f , p̂ ) : = max i |conf ( Bi ) − acc ( Bi ) | . ( 3 ) 2.2 EXISTING GLOBAL RECALIBRATION METHODS . Many existing methods apply a post-hoc adjustment that changes a model ’ s confidence predictions to improve global calibration , including Platt scaling ( Platt , 1999 ) , temperature scaling ( Guo et al. , 2017 ) , isotonic regression ( Zadrozny & Elkan , 2002 ) , and histogram binning ( Zadrozny & Elkan , 2001 ) . These methods all learn a simple transformation from the original confidence predictions to new confidence predictions , and aim to decrease the expected calibration error ( ECE ) . Platt scaling fits a logistic regression model ; temperature scaling learns a single temperature parameter to rescale confidence scores for all samples simultaneously ; isotonic regression learns a piece-wise constant monotonic function ; histogram binning partitions confidence scores into bins { [ 0 , ) , [ , 2 ) , · · · , [ 1− , 1 ] } and sorts each validation sample into a bin based on its confidence p̂ ( x ) ; it then resets the confidence level of all samples in the bin to match the classification accuracy of that bin . 2.3 LOCAL CALIBRATION . Two notions of calibration that address some of the deficits of global calibration are class-wise calibration and group-wise calibration . Class-wise calibration groups samples by their true class label ( Kull et al. , 2019 ; Nixon et al. , 2019 ) and measures the average class ECE , while group-wise calibration uses pre-specified groupings ( e.g. , race or gender ) ( Kleinberg et al. , 2016 ; Pleiss et al. , 2017 ) and measures the average group-wise ECE or maximum group-wise MCE . A few recalibration methods have been proposed for these notions of calibration as well . Dirichlet calibration ( Kull et al. , 2019 ) achieves calibration for groups defined by class labels , but does not generalize well to settings with many classes ( Zhao et al. , 2021 ) . Multicalibration ( Hébert-Johnson et al. , 2017 ) achieves calibration for any group that can be represented by a polynomial sized circuit , but lacks a tractable algorithm . If the groups are known a priori , one can also apply global calibration methods within each group ; however , this is impractical in many situations where the groups are not known for new examples at inference time . At an even more local level , Zhao et al . ( 2020 ) look at individual calibration in the regression setting and conclude that individual calibration is impossible to verify with a deterministic forecaster , and thus there is no general method to achieve individual calibration . 2.4 KERNEL-BASED CALIBRATION METRICS . Kumar et al . ( 2018 ) introduce the maximum mean calibration error ( MMCE ) , a kernel-based quantity that replaces the hard binning of the standard ECE estimator with a kernel similarity k ( p̂ ( x ) , p̂ ( x′ ) ) between the confidence of two examples . They further propose to optimize the MMCE directly in order to achieve better model calibration globally . Widmann et al . ( 2019 ) extend their work and propose the more general kernel calibration error . Zhang et al . ( 2020 ) and Gupta et al . ( 2020 ) also consider kernel-based calibration . However , these methods only consider the similarity between model confidences p̂ ( x ) , p̂ ( x′ ) , rather than the inputs x , x′ themselves . 3 THE LOCAL CALIBRATION ERROR . Recall that commonly used metrics for calibration , such as the ECE or the MCE , are global in nature and thus only measure an aggregate reliability over the entire dataset , making them insufficient for many applications . An ideal calibration metric would instead measure calibration at an individual level ; however , doing so is impossible without making assumptions about the ground truth distribution ( Zhao et al. , 2020 ) . A localized calibration metric represents an adjustable balance between these two extremes . Ideally , such a metric should measure calibration at a local level ( where the extent of the local neighborhood can be chosen by the user ) and group similar data points together . In this section , we introduce the local calibration error ( LCE ) , a kernel-based metric that allows us to measure the calibration locally around a prediction . Our metric leverages learned features to automatically group similar samples into a soft neighborhood , and allows the neighborhood size to be set with a hyperparameter γ . We also consider only points with a similar model confidence as the prediction , so that similarity is defined in terms of distance both in the feature space and in model confidence . Thus , the LCE effectively creates soft groupings that depend on the feature space ; with a semantically meaningful feature space , these groupings correspond to useful subsets of the data . We then mention a few design choices and visualize LCE maps over a 2D feature space to show that we can use our metric to diagnose regions of local miscalibration .

Summary: This work proposes a new metric for calibration in classification where calibration is measured over localities in the input space, and the localities are determined with a kernel over the feature space. A recalibration algorithm (LoRe) is additionally proposed, which aims to recalibrate the class predictions w.r.t. local calibration error (LCE). The experiments show that 1) LoRe achieves better local calibration (measured by MLCE, i.e. maximum LCE) than baseline recalibration methods on ImageNet, 2) LoRe achieves better group-wise calibration (measured by worst group-wise maximum calibration error) across 3 datasets with identifiable groups, and 3) achieves competitive performance in a simple decision making task where confident incorrect predictions incur high costs.