Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration

1 Introduction . Machine learning predictions are increasingly employed by downstream decision makers who have little or no visibility on how the models were designed and trained . In high-stakes settings , such as healthcare applications , decision makers want predictions they can trust . For example in healthcare , suppose a machine learning service offers a supervised learning model to healthcare providers that claims to predict the probability of various skin diseases , given an image of a lesion . Each healthcare provider want assurance that the model ’ s predictions lead to beneficial decisions , according to their own loss functions . As a result , the healthcare providers may reasonably worry that the model was trained using a loss function different than their own . This mismatch is often inevitable because the ML service may provide the same prediction model to many healthcare providers , which may have different treatment options available and loss functions . Even the same healthcare provider could have different loss functions throughout time , due to changes in treatment availability . If predicted probabilities perfectly equal the true probability of the event , this issue of trust would not arise because they would lead to optimal decision making regardless of the loss function or task considered by downstream decision makers . In practice , however , predicted probabilities are never perfect . To address this , the healthcare providers may insist that the prediction function be 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . distribution calibrated , requiring that amongst the inputs that receive predicted class probability vectors q , the actual distribution over classes is q . This solves the trust issue because among the patients who receive prediction q , the healthcare providers knows that the true label distribution is q , and hence knows the true expected loss of a treatment on these patients . Unfortunately , to achieve distribution calibration , we need to reason about the set of individuals x who receive prediction q , for every possible predicted q . As the number of distinct predictions may naturally grow exponentially in the number of classes C , the amount of data needed to accurately certify distribution calibration tends to be prohibitive . Due to this statistical barrier , most work on calibrated multi-class predictions focuses on obtaining relaxed variants of calibration . These include confidence calibration ( Guo et al. , 2017 ) , which calibrates predictions only over the most likely class , and classwise calibration ( Kull et al. , 2019 ) , which calibrates predictions for each class marginally . While feasible , these notions are significantly weaker than distribution calibration and do not address the trust issue highlighted above . Is there a calibration notion that addresses the issue of trust , but can also be verified and achieved efficiently ? Our paper answers this question affirmatively . Our Contributions . We introduce a new notion of calibration—decision calibration—where we take the perspective of potential decision-makers : the only differences in predictions that matter are those that could lead to different decisions . Inspired by Dwork et al . ( 2021 ) , we formalize this intuition by requiring that predictions are “ indistinguishable ” from the true outcomes , according to a collection of decision-makers . First , we show that prior notions of calibration can be characterized as special cases of decision calibration under different collections of decision-makers . This framing explains the strengths and weakness of existing notions of calibration , and clarifies the guarantees they offer to decision makers . For example , we show that a predictor is distribution calibrated if and only if it is decision calibrated with respect to all loss functions and decision rules . This characterization demonstrates why distribution calibration is so challenging : achieving distribution calibration requires simultaneously reasoning about all possible decision tasks . The set of all decision rules include those that choose between exponentially ( in number of classes C ) many actions . In practice , however , decision-makers typically choose from a bounded ( or slowlygrowing as a function of C ) set of actions . Our main contribution is an algorithm that guarantees decision calibration for such more realistic decision-makers . In particular , we give a sample-efficient algorithm that takes a pre-trained predictor and post-processes it to achieve decision calibration with respect to all decision-makers choosing from a bounded set of actions . Our recalibration procedure does not harm other common performance metrics , and actually improves accuracy and likelihood of the predictions . In fact , we argue formally that , in the setting of bounded actions , optimizing for decision calibration recovers many of the benefits of distribution calibration , while drastically improving the sample complexity . Empirically , we use our algorithm to recalibrate deep network predictors on two large scale datasets : skin lesion classification ( HAM10000 ) and Imagenet . Our recalibration algorithm improves decision making , and allow for more accurate decision loss estimation compared to existing recalibration methods even under distribution shift . 2 Background . 2.1 Setup and Notation . We consider the prediction problem with random variables X and Y , where X ∈ X denotes the input features , and Y ∈ Y denotes the label . We focus on classification where Y = { ( 1 , 0 , · · · , 0 ) , ( 0 , 1 , · · · , 0 ) , · · · , ( 0 , 0 , · · · , 1 ) } where each y ∈ Y is a one-hot vector with C ∈ N classes . 1 A probability prediction function is a map p̂ : X → ∆C where ∆C is the C-dimensional simplex . We define the support of p̂ as the set of distributions it could predict , i.e . { p̂ ( x ) |x ∈ X } . We use p∗ : X → ∆C to denote the true conditional probability vector , i.e . p∗ ( x ) = E [ Y | X = x ] ∈ ∆C , and for all c ∈ [ C ] , each coordinate gives the probability of the class p∗ ( x ) c = Pr [ Yc | X = x ] . 1We can also equivalently define Y = { 1 , 2 , · · · , C } , here we denote y by a one-hot vector for notation convenience when taking expectations . 2.2 Decision-Making Tasks and Loss Functions . We formalize a decision-making task as a loss minimization problem . The decision-maker has some set of available actions A and a loss function ` : Y × A → R. In this paper we assume the loss function does not directly depend on the input features X . For notational simplicity we often refer to a ( action set , loss function ) pair ( A , ` ) only by the loss function ` : the set of actions A is implicitly defined by the domain of ` . We denote the set of all possible loss functions as Lall = { ` |∀ set A and function ` : Y ×A → R } . We treat all action sets A with the same cardinality as the same set — they are equivalent up to renaming the actions . A convenient way to think about this is that we only consider actions sets A ∈ { [ 1 ] , [ 2 ] , · · · , [ K ] , · · · , N , R } where [ K ] = { 1 , · · · , K } . 2.3 Bayes Decision-Making . Given some predicted probability p̂ ( X ) on Y , a decision-maker selects an action inA . We assume that the decision-maker selects the action based on the predicted probability . That is , we define a decision function as any map from the predicted probability to an action δ : ∆C → A. and denote by all as the set of all decision functions with any set of actions all = { δ | ∀ set A and function δ : ∆C → A } . Typically a decision-maker selects the action that minimizes the expected loss ( under the predicted probability ) . This strategy is formalized by the following definition of Bayes decision-making . Definition 1 ( Bayes Decision ) . Choose any ` ∈ Lall with corresponding action set A and prediction p̂ , define the Bayes decision function as δ ` ( p̂ ( x ) ) = arg infa∈A EŶ∼p̂ ( x ) [ ` ( Ŷ , a ) ] . For any subset L ⊂ Lall denote the set of all Bayes decision functions as L : = { δ ` | ∀ ` ∈ L } . 3 Calibration : A Decision-Making Perspective . In our setup , the decision-maker outsources the prediction task to a third-party forecaster , which returns a prediction function p̂ ( e.g. , an ML Prediction API ) . The decision maker will use p̂ and a loss function ` to make decisions . However , the forecaster does not necessarily know the loss function ` in advance , and in more challenging cases , needs to serve multiple decision-makers with different loss functions . For example , the prediction function may be trained to optimize a standard objective such as L2 error or log likelihood , then sold to decision-makers as an off-the-shelf solution . In such a setting , the decision makers may be concerned that the off-the-shelf solution may not perform well according to their loss function . If the forecaster could predict optimally ( i.e . p̂ ( X ) = p∗ ( X ) almost surely ) , then there would be no issue of trust ; of course , perfect predictions are usually impossible , so the forecaster needs feasible ways of conveying trust to the decision makers . To mitigate concerns about the performance of the prediction function , the forecaster might aim to offer performance guarantees applicable to decision makers whose loss functions come from class of losses L ⊂ Lall . 3.1 Decision Calibration . First and foremost , a decision maker wants assurance that the Bayes decision rule δ ` with p̂ gives low expected loss . Second , the decision maker wants to know how much loss is going to be incurred ( before the actions are deployed and outcomes are revealed ) ; the decision maker does not want to incur any additional loss in surprise that she has not prepared for . To capture these desiderata we formalize a definition based on the following intuition : suppose a decision maker with some loss function ` considers a decision rule δ ∈ all ( that may or may not be the Bayes decision rule ) , the decision maker should be able to correctly compute the expected loss of using δ to make decisions , as a function of the predictions p̂ . Definition 2 ( Decision Calibration ) . For a set of loss functions L ⊂ Lall and a set of decision rules ⊂ all , we say that a prediction p̂ is ( L ; ) -decision calibrated ( with respect to p∗ ) if ∀ ` ∈ L and δ ∈ with the same action space A 2 , the computed loss ( based on p̂ ) of δ is the same as the actual 2We require ` and δ to have the same action space A for type check reasons . Eq . ( 2 ) only has meaning if the loss ` and decision rule δ are associated with the same action space A. loss , i.e . EXEŶ∼p̂ ( X ) [ ` ( Ŷ , δ ( p̂ ( X ) ) ) ] = EXEY∼p∗ ( X ) [ ` ( Y , δ ( p̂ ( X ) ) ) ] ( 1 ) In particular , we say p̂ is L-decision calibrated if it is ( L ; L ) -decision calibrated , where L is the set of all Bayes decision rules for loss functions in L. The left hand side of Eq . ( 2 ) is the “ simulated ” loss where the outcome Ŷ is hypothetically drawn from the predicted distribution . The decision maker can compute this just by knowing the input features X and without knowing the outcome Y . The right hand side of Eq . ( 2 ) is the true loss that the decision maker incurs in reality if she uses the decision rule δ . Intuitively , the definition captures the idea that the losses in L and decision rules in do not distinguish between outcomes sampled according to the predicted probability and the true probability ; specifically , the definition can be viewed as an instantiation of the framework of Outcome Indistinguishability ( Dwork et al. , 2021 ) . As a cautionary remark , Eq . ( 2 ) should not be mis-interpreted as guarantees about individual decisions ; Eq . ( 2 ) only looks at the average loss when X , Y is a random draw from the population . Individual guarantees are usually impossible without tools beyond machine learning ( Zhao & Ermon , 2021 ) . In addition , Definition 2 does not consider decision rules that can directly depend on X , as δ ∈ all only depends on X via the predicted probability p̂ ( X ) . Studying decision rules that can directly depend on X require tools such as multicalibration ( Hébert-Johnson et al. , 2018 ) which are beyond the scope of this paper ( see related work ) . In Definition 2 we also define the special notion of L-decision calibrated because given a set of loss functions L , we are often only interested in the associated Bayes decision rules L , i.e . the set of decision rules that are optimal under some loss function . For the rest of the paper we focus on L-decision calibration for simplicity . L-decision calibration can capture the desiderata we discussed above formalized in the following proposition . Proposition 1 . If a prediction function p̂ is L-decision calibrated , then it satisfies ∀δ′ ∈ L EXEY∼p∗ ( X ) [ ` ( Y , δ ` ( p̂ ( X ) ) ) ] ≤ EXEY∼p∗ ( X ) [ ` ( Y , δ′ ( p̂ ( X ) ) ) ] ( Bayes Decision Optimality ) EXEŶ∼p̂ ( X ) [ ` ( Ŷ , δ ` ( p̂ ( X ) ) ) ] = EXEY∼p∗ ( X ) [ ` ( Y , δ ` ( p̂ ( X ) ) ) ] ( Accurate loss estimation ) Bayes Decision Optimality states that the Bayes decision rule δ ` is not worse than any alternative decision rule δ′ ∈ L. In other words , the decision maker is incentivized to take optimal actions according to their true loss function . That is , using the predictions given by p̂ , the decision maker can not improve their actions by using a decision rule δ′ that arises from a different loss function ` ′ ∈ L. Accurate loss estimation states that for the Bayes decision rule δ ` , the simulated loss on the left hand side ( which the decision maker can compute before the outcomes are revealed ) equals the true loss on the right hand side . This ensures that the decision maker knows the expected loss that will be incurred over the distribution of individuals and can prepare for it . This is important because in most prediction tasks , the labels Y are revealed with a significant delay or never revealed . For example , the hospital might be unable to follow up on the true outcome of all of its patients . In practice , the forecaster chooses some set L to achieve L-decision calibration , and advertise it to decision makers . A decision makers can then check whether their loss function ` belongs to the advertised set L. If it does , the decision maker should be confident that the Bayes decision rule δ ` has low loss compared to alternatives in L , and they can know in advance the loss that will be incurred .

Summary: The paper defines a new notion of decision calibration and demonstrates that approximate $\mathcal{L}^K$ decision calibration is verifiable and achievable in practice, providing a practical algorithm for it. This is a result with practical calibration guarantees in multi-class calibration, which to my knowledge is the first achievement of this kind. The proposed method outperforms the existing calibration methods of temperature scaling and Dirichlet calibration in experiments on HAM10000 and ImageNet datasets and multiple architectures.