Reliable Decisions with Threshold Calibration

1 Introduction . Decision makers need to understand the consequences of their decisions prior to making them . When decisions are based on predictions from a machine learning model , the decision loss – the loss incurred under a decision rule based on the predictions – summarizes the consequences of these decisions . As an example , suppose a machine learning practitioner develops a model to predict patient length-of-stay in the hospital [ 17 , 3 ] . A hospital decides whether they have capacity to admit new patients based on the model ’ s predictions of current patients ’ length-of-stay ( e.g . for each current patient who is predicted to have a length-of-stay that is less than k days , the hospital schedules a new patient ) . Incorrect decisions due to the model ’ s predictions cause the hospital to accrue costs from under-utilizing resources or overbooking procedures . The decision loss is an aggregation of the costs incurred from incorrect decisions . To determine whether a decision rule is safe to use , the hospital would like to have an accurate estimate of the decision loss under different choices of k and different costs associated with errors . This type of decision-making scenario occurs in many high-stakes settings such as designing interventions for adverse weather events [ 33 , 9 ] and resource allocation decisions using economic estimates [ 15 , 32 ] . Probabilistic predictions ( probabilistic forecasts ) can be used to estimate decision loss prior to deployment . In this work , we consider the regression setup , where a forecast is represented by a cumulative distribution function over the possible outcomes . If the forecasted probabilities of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . incorrect decisions match the true frequencies of these events , then the average decision loss can be accurately predicted from the forecasts . However , forecasted probabilities of incorrect decisions do not typically match the true ones , yielding inaccurate decision loss predictions . We refer to the absolute difference between the average loss predicted by forecaster and the true average loss as the reliability gap . Many previous works in calibration and uncertainty quantification are motivated by the assumption that calibrated uncertainty estimates will yield safer or more reliable downstream decisions [ 31 , 2 , 22 , 24 , 25 ] . However , we find that the standard notion of calibration , average calibration , does not guarantee zero reliability gap for even a simple class of decision rules : threshold decision rules ( left , Figure 1 ) . In a threshold decision , a decision maker takes one of two possible actions depending on whether an outcome falls above or below a cutoff ( e.g . the hospital schedules a new patient if a current patient ’ s length-of-stay is less than 3 days , otherwise the hospital does not schedule a new patient ) . Stronger calibration properties , such as distribution calibration [ 31 ] , are theoretically guaranteed to yield zero reliability gap but are difficult to achieve in practice . In particular , flexible distribution families can better approximate the true distribution than simple ones and yield lower decision loss , but applying distribution calibration to such forecasters can increase the decision loss and enlarge the reliability gap compared to the uncalibrated forecaster . Thus , existing calibration definitions are either insufficient or impractical for minimizing the reliability gap under threshold decision rules . To address these shortcomings , we propose a new notion of calibration called threshold calibration . Threshold calibration strikes a balance between average and distribution calibration ; it is exactly the condition required to guarantee zero reliability gap under threshold decisions and is practical to enforce ( Figure 1 , Right ) . First , we establish that threshold calibration is the necessary and sufficient condition to guarantee zero reliability gap for any threshold decision under any threshold loss . Second , we design an efficient algorithm that takes an uncalibrated forecaster as input and provably outputs a threshold-calibrated forecaster . Third , we show that empirically , threshold calibration is a practical solution ; in two real-world settings and a suite of benchmark regression tasks , we find that threshold calibration minimizes the reliability gap across decision makers with different threshold loss functions while achieving similar or improved decision loss compared to the baselines . 2 Preliminaries . 2.1 Notation and Forecasting Setup . We consider the regression setup with a feature space X and a label space Y ⊂ R. The input is a random variable X ∈ X and the label is a random variable Y ∈ Y . We use capital letters to denote random variables X , Y and lower case letters to denote their values x , y . Let F ( Y ) be the space of cumulative distribution functions ( CDFs ) over Y . A forecaster h : X → F ( Y ) is a function that maps an input from the feature space to a CDF on Y . In other words , given a fixed input x ∈ X , the forecaster outputs the predicted CDF h [ x ] ∈ F ( Y ) . Ideally , the forecaster aims to predict the CDF of Y given X . To further clarify the notation , for a fixed input-label pair ( x , y ) ∈ X × Y , h [ x ] is a CDF over the predicted label values and h [ x ] ( y ) ∈ [ 0 , 1 ] is the value of the CDF h [ x ] at the point y . We note that h [ X ] is a random variable that takes values in F ( Y ) and h [ X ] ( Y ) is a random variable that takes values in [ 0 , 1 ] . Let h∗ [ X ] be the true conditional CDF of Y given X . We use ∼ to denote the distribution of a random variable . We have that Y ∼ h∗ [ X ] . We introduce a new random variable Ỹ to represent a label distributed according to the h [ X ] , the forecasted conditional distribution , so Ỹ ∼ h [ X ] . 2.2 Decision-Making . Let A be a countable action space . A decision rule δ : X → A is any map from an input x ( e.g . a current patient ’ s attributes ) to an action a ( e.g . admit a new patient ) . We assume that a decision maker has a loss function ` : X ×Y ×A → R , describing the loss incurred when choosing an action a on an input-label pair ( x , y ) . Because the labels y are unobserved , the decision maker often wants to minimize their expected loss assuming that the labels are distributed according to the forecasted distribution . As a result , they use the Bayes decision rule with respect to h. Definition 1 ( Bayes Decision Rule ) . Given a space of decision rules ∆ , the Bayes decision rule with respect to the forecaster h is the decision rule in ∆ that minimizes the expected loss under the forecasted distribution δ∗h = arg inf δ∈∆ EXEỸ∼h [ X ] [ ` ( X , Ỹ , δ ( X ) ) ] ] 2.3 Threshold Decisions . We focus on the setting where the decision maker aims to minimize a threshold loss function . The action space A consists of two actions so A = { 0 , 1 } . A threshold loss function ` is defined as follows ` ( x , y , a ) = ∑ i∈ { 0,1 } c1 , iI ( y ≤ y0 , a = i ) + ∑ i∈ { 0,1 } c0 , iI ( y > y0 , a = i ) , where ci , j ∈ R. The ci , j ’ s denote decision costs , costs associated with different outcome-action pairs , and y0 is a decision threshold . Let L be the space of threshold loss functions , which are all losses of this form with any ci , j ∈ R and y0 ∈ R. Given a threshold loss function ` , the decision maker can use the Bayes decision rule δ∗h in Definition 1 to select which action to take . We show that the resulting decision rules always take the form of δ∗h ( x ) = I ( h [ x ] ( y0 ) ≥ α ) or δ∗h ( x ) = I ( h [ x ] ( y0 ) ≤ α ) for some parameters α ∈ [ 0 , 1 ] and y0 ∈ Y that depends on the loss function ( proved in Appendix B ) . We call such decision rules threshold decision rules because intuitively , they choose the action based on whether h [ x ] ( y0 ) is greater ( or less ) than a threshold α . We denote the space of such decision rules as ∆h . Since the decision maker ’ s loss function is a threshold loss function , the decision maker can restrict the space of decision rules they consider to threshold decision rules on the forecasted CDFs . 3 Reliable Decision-Making with Threshold Calibration . 3.1 Problem Setup . Forecasts are often produced by one group , such as machine learning practitioners or scientists , and consumed by another , such as policy makers or private agents [ 14 ] . Motivated by this paradigm , we model these two entities separately : 1 . A forecaster h takes inputs x ∈ X and produces CDFs h [ x ] over the possible outcomes in Y . The provider of h does not know the specific downstream tasks for which h is used . 2 . A decision maker has a dataset of unlabeled inputs D = { xi } ni=1 , binary action space A = { 0 , 1 } , and a threshold loss function ` ∈ L of interest . The decision maker must take an action ai ∈ A for each unlabeled input xi . The decision maker uses the forecaster h to select { ai } ni=1 because ( 1 ) the decision maker does not have enough labeled data to build their own model locally or ( 2 ) building the model requires a domain expert . Multiple decision makers may rely on the same forecaster but have different loss functions . Further , a decision maker ’ s loss function can change if their decision costs or decision threshold change . In this setting , we identify the conditions on h that the provider can enforce to ensure reliable decision-making under threshold decisions . 3.2 Reliability Gap . Decision makers often need to accurately estimate the average decision loss incurred under a decision rule prior to deployment . To quantify the accuracy of these decision loss predictions , we define the reliability gap . Definition 2 ( Reliability Gap ) . Given a forecaster h , we define the the reliability gap γ ( δ , ` ) of a particular decision rule δ under a loss function ` as γ ( δ , ` ) = |EXEỸ∼h [ X ] [ ` ( X , Ỹ , δ ( X ) ) ] − EXEY∼h∗ [ X ] [ ` ( X , Y , δ ( X ) ) ] | . The first term in the equation is the average decision loss predicted by the forecaster . Under the forecasted distribution , the labels Ỹ are distributed according to h [ X ] . As a result , the first term does not depend on the true labels and can be computed by the decision maker using the unlabeled data prior to deployment . The second term is the true average decision loss . Under the true conditional distribution , the labels Y are distributed according to h∗ [ X ] . So , the second term can be thought of as the loss that is incurred at test-time . One caveat is that the reliability gap quantifies the reliability of average decision loss prediction and obtaining zero reliability gap does not imply any instance-based guarantees for individual decisions . When the forecaster perfectly matches the true distribution ( i.e . h = h∗ ) , we have γ ( δ , ` ) = 0 for any decision rule δ and any loss function ` . However , in practice , we can not assume that the forecaster predicts the true distribution . In addition , we would like the forecaster to be applicable for different downstream decision makers . As a result , we study the necessary and sufficient conditions on the forecaster that guarantee zero reliability gap for any threshold decision on the forecasted CDFs and any threshold loss function .

Summary: This paper considers the problem of calibrating the conditional CDFs from a regression model.    The authors propose a new definition of threshold calibration, which requires the conditional CDFs to have calibrated quantiles when these CDFs have a $\alpha$-quantile that are all on the same side regarding a given $y_0$ (either above or below).    The proposed definition is a stronger notion of quantile calibration in regression (called averaged calibration in this paper), and a weaker notion of distribution calibration in regression. To further justify the proposed definition, the authors consider measuring the so-called reliability gap, which is based on a given loss function and the corresponding decision rule. The authors then show that having threshold-calibrated results can ensure a 0 reliability gap. The necessity and sufficiency among different calibration definitions are later demonstrated.  On the method level, the proposed approach is hence to iteratively search for quantile levels and thresholds with large calibration errors and applies isotonic regression to calibrate the corresponding quantiles. The final calibrated model will include the uncalibrated models and a sequence of calibration maps.  Experiments are performed with real datasets with a cost-sensitive setup. The proposed method is compared with uncalibrated model, quantile calibrated model and distribution calibrated model.