Reliable Decisions with Threshold Calibration
Authors: Roshni Sahoo
Shengjia Zhao
Alyssa Chen
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: uncertainty quantification
calibration

Abstract: Decision makers rely on probabilistic forecasts to predict the loss of different decision rules before deployment. When the forecasted probabilities match the true frequencies, predicted losses will be accurate. Although perfect forecasts are typically impossible, probabilities can be calibrated to match the true frequencies on average. However, we find that this \textit{average} notion of calibration, which is typically used in practice, does not necessarily guarantee accurate decision loss prediction. Specifically in the regression setting, the loss of threshold decisions, which are decisions based on whether the forecasted outcome falls above or below a cutoff, might not be predicted accurately. We propose a stronger notion of calibration called threshold calibration, which is exactly the condition required to ensure that decision loss is predicted accurately for threshold decisions. We provide an efficient algorithm which takes an uncalibrated forecaster as input and provably outputs a threshold-calibrated forecaster. Our procedure allows downstream decision makers to confidently estimate the loss of any threshold decision under any threshold loss function. Empirically, threshold calibration improves decision loss prediction without compromising on the quality of the decisions in two real-world settings: hospital scheduling decisions and resource allocation decisions.

Reliable Decisions with Threshold Calibration
Roshni Sahoo Stanford University rsahoo@stanford edu
Shengjia Zhao Stanford University sj zhao@stanf ord. edu
Alyssa Chen
Stefano Ermon Stanford University ermon@stanford. edu
UTSW Medical Center alyssa. chen@utsw... edu
Abstract
Decision makers rely on probabilistic forecasts to predict the loss of different decision rules before deployment. When the forecasted probabilities match the true frequencies, predicted losses will be accurate. Although perfect forecasts are typically impossible, probabilities can calibrated to match the true frequencies average. However, we find that this average notion of calibration, which is typically used practice, does not necessarily guarantee accurate decision loss prediction. Specifically in the regression setting, the loss of threshold decisions, which are decisions based on whether the forecasted outcome falls above or below cutoff, might not be predicted accurately. We propose stronger notion of calibration called threshold calibration, which is exactly the condition required to ensure that decision loss is predicted accurately for threshold decisions. We provide an efficient algorithm which takes an uncalibrated forecaster as input and provably outputs threshold-calibraee decision makers to confidently estimate the loss of any threshold decision under any threshold loss function. Empirically, threshold calibration improves decision loss prediction without compromising on the quality of the decisions in two real-world settings: hospital scheduling decisions and resource allocation decisions.
forecaster. Our procedure allows downstream
Introduction
Decision makers need understand the consequences of their decisions prior making them. When decisions are based on predictions from a machine learning model, the decision loss the loss incurred under decision rule based on the predictions summarizes the consequences of these decisions. As an example, suppose machine learning practitioner develops model to predict patient length-of-stay in the hospital [17, 3). A hospital decides whether they have capacity admit new patients based the model's predictions of current patients' length-of-stay (e.g. for each current patient who is predicted to have a length-of-stay that is less than days, the hospital schedules new patient). Incorrect decisions due to the model's is predictions cause the hospital to accrue costs from under-utilizing resources or overbooking procedures. The decision loss an aggregation of the costs incurred from incorrect decisions. To determine whether decision rule is safe use, the hospital would like to have an accurate estimate the decision loss under different choices of and different costs associated with errors. This type of decision-maie maaking scenario occurs in many high-stakes settings such as designing interventions for adverse weather events [33, 9] and resource allocation decisions using economic estimates [15, 32).
Probabilistic predictions (probabilistic forecasts) can be used to estimate decision loss prior to deployment. In this work, we consider the regression setup, where forecast is represented by a cumulative distribution function over the possible outcomes. the forecasted probabilities of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Average-Calirre Forecaster
Forecaster Reliabilty Gap
Threshold Callbrated Forecaster
True Decision Loss
1.4
Average Threshald
True Decision Loss
Predicted Decision
1.2
Predicted Decision Loss
Decision Loss
3.0
1.0
Decision Loss
3,0
2.5
Bellability
2.5
0.6
2.0
0.4
2.0
0.2
1.5
1.5
0.0
0.2 0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.4
0.6
Acceptance Threshold
Acceptance nce Threshold
Acceptance Threshold
Figure/ 1: We evaluate wwerage-calibaate and threshold-calibraee patient length-of-star forecasters across range of threshold decision rules. Left: The average-calibrate forecaster underestimates the true decision loss for some decision rules and overestimates it on others, resulting in nonzero reliability gap. Middle: The reliability gap minimized under the threshold-calibrated forecaster but not under the average-calibrated forecaster. Right: The threshold-calibrated forecaster accurately predicts the true decision loss across a range of decision rules.
incorrect decisions match the true frequencies of these events, then the average decision loss can be accurately predicted from the forecasts. However, forecasted probabilities of incorrect decisions do not typically match the true ones, yielding inaccurate decision loss predictions. We refer to the absolute difference between the average loss predicted by forecaster and the true average loss as the reliability gap.
Many previous works in calibration and uncertainty quantification motivated by the assumption that calibrated uncertainty estimates will yield safer or more reliable downstream decisions [31, 2, 22, 24, 25]. However, find that the standard notion of calibration, average calibration, does not guarantee zero reliability gap for even simple class of decision rules: threshold decision rules (left, Figure 1). In threshold decision, decision maker takes of possible actions depending on whether an outcome falls above or below a cutoff (e.g. the hospital schedules a new patient if a current patient's length-of-stay
less than 3 days, otherwise the hospital does not schedule new patient). Stronger calibration properties, such as distribution calibration [31], are theoretically guaranteed to yield zero reliability gap but are difficult to achieve in practice. In particular, flexible distribution families can better approximate the true distribution than simple ones and yield lower decision loss, but applying distribution calibration to such forecasters can increase the decision loss and enlarge the reliability gap compared to the uncalibrated forecaster. Thus, existing calibration definitions are either insufficient or impractical for minimizing the reliability gap under threshold decision rules.
To address these shortcomings. we propose a new notion of calibration called threshold calibration. Threshold calibration strikes balance between average and distribution calibration; it is exactly the condition required to guarantee zero reliability gap under threshold decisions and is practical to enforce (Figure 1, Right). First, we establish that threshold calibration the necessary and sufficient condition to guarantee zero reliability gap for any threshold decision under any threshold loss. Second, we design an efficient algorithm that takes an uncalibrated forecaster as input and provably outputs a wreshold-calibrann forecaster. Third, we show that empirically, threshold calibration is practical solution; in two real-world settings and a suite of benchmark regression tasks, we find that threshold calibration minimizes the reliability gap across decision makers with different threshold loss functions while achieving similar or improved decision loss compared to the baselines.
2 Preliminaries
2.1 Notation and Forecasting Setup
We consider the regression setup with feature space X' and a label space y C R. The input is random variable X E X' and the label is a random variable Y E y. We use capital letters to denote random variables X, Y and lower case letters denote their values v, y-
Let F()) be the space of cumulative distribution functions (CDFs) over V. forecaster h: X F()) is function that maps input from the feature space to CDF on Y. In other

---

words, given fixed input x' E X. the forecaster outputs the predicted CDF h[z] E F()). Ideally, the forecaster aims to predict the CDF of Y given X.
To further clarify the notation, for fixed input-label pair (z, v) E x V. M/a] is a CDF over the predicted label values and (v) [0,1] is the value of the CDF M(ar] at the point y. We note that h[X] is random variable that takes values in F()) and h([](() is a random variable that takes values in [0, 1].
Let h [X] be the true conditional CDF of given X. We use ~ to denote the distribution random variable. We have that Y ~ ['" We introduce a new random variable y to represent a label distributed according to the h[X], the forecasted conditional distribution, so Y ~ h[X].
2.2 Decision-Making
Let A be a countable action space. A decision rule 5 & - A is any map from an input x (e.g. current patient's attributes) an action a (e.g. admit a new patient). We assume that a decision maker has a loss function É X x y x A + R, describing the loss incurred when choosing an action input-label pair (Ir, y). Because the labels y are unobserved, the decision maker often wants to minimize their expected loss assuming that the labels are distributed according to the forecasted distribution. As a result, they use the Bayes decision rule with respect to h.
Definition 1 (Bayes Decision Rule). Given a space of decision rules A, the Bayes decision rule with respect to the forecaster h is the decision rule in A that minimizes the expected loss under the forecasted distribution
%% = arg inf Ex ( BEA
2.3 Threshold Decisions
We focus on the setting where the decision maker aims to minimize a threshold loss function. The action space A consists of two actions so A {0, 1}. A threshold loss function é is defined as follows
((, = E
Cyy I yo, a = i) + Σ 1E/0,1}
Co,IIly > yo. a =
iE(U,1}
where C.j E R. The Ca. 's denote decision costs, costs associated with different outcome-action pairs, and yo is decision threshold. Let be the space of threshold loss functions, which are all losses of this form with any Cij E IR. and yo E IR.
Given threshold loss function É, the decision maker can use the Bayes decision rule 8 in Definition 1 to select which action to take. We show that the resulting decision rules always take the form of
5% (x) = (vo) 2 a) 0% (x) = T(h (())) S o:)
for some parameters E [0. 1] and yo y that depends on the loss function (proved in Appendix B). We call such decision rules threshold decision rules because intuitively, they choose the action based on whether h (1x))))))))))))))))))))))))))( is greater (or less) than a threshold a. We denote the space of such decision rules as An. Since the decision maker's loss function is a threshold loss function, the decision maker restrict the space of decision rules they consider threshold decision rules the forecasted CDFs.
3 Reliable Decision-Making with Threshold Calibration
3.1 Problem Sctup
Forecasts are often produced by one group, such as machine learning practitioners or scientists, and consumed by another, such policy makers or private agents [14]. Motivated by this paradigm, model these two entities separately:
1. A forecaster h takes inputs x X' and produces CDFs h(x) over the possible outcomes in y. The provider of h does not know the specific downstream tasks for which h is used.
3

---

2. A decision maker has a dataset of unlabeled inputs D = {z, 1i- binary action space A {0, 1), and threshold loss function & E C of interest. The decision maker must take an action a; E A for each unlabeled input zę. The decision maker uses the forecaster h to select {a,};= because (1) the decision maker does not have enough labeled data to build their own model locally or (2) building the model requires domain expert.
Multiple decision makers may rely on the same forecaster but have different loss functions. Further, decision maker's loss function change if their decision costs decision threshold change. In this setting, we identify the conditions on h that the provider can enforce to ensure reliable decision-making under threshold decisions.
3.2 Reliability Gap
Decision makers often need to accurately estimate the average decision loss incurred under decision rule prior to deployment. To quantify the accuracy of these decision loss predictions, we define the reliability gap.
Definition 2 (Reliability Gap). Given a forecaster h, we define the the reliability gap 7 E) of particular decision rule 8 under loss function & as
766 e) -E ((X,,),))
((x()) -
The first term in the equation is the average decision loss predicted by the forecaster. Under the forecasted distribution, the labels are distributed according to h[X]. As result, the first term does not depend on the true labels and can be computed by the decision maker using the unlabeled data prior to deployment. The second term is the true average decision loss. Under the true conditional distribution, the labels are distributed according to h h=[X]. So, the second term can be thought of as the loss that is incurred at test-time. One caveat is that the reliability gap quantifies the reliability of average decision loss prediction and obtaining zero reliability gap does not imply any instance-based guarantees for individual decisions.
When the forecaster perfectly matches the true distribution (i.e. h h²), we have (8, = 0 for any decision rule 5 and any loss function f. However, in practice, we cannot assume that the forecaster predicts the true distribution. In addition, we would like the forecaster to applicable for different downstream decision makers. As result, we study the necessary and sufficient conditions on the forecaster that guarantee zero reliability gap for any threshold decision on the forecasted CDFs and any threshold loss function.
3.3 Threshold Calibration
We define the property of threshold calibration and show that it necessary and sufficient to ensure zero reliability gap under any threshold decision on the forecasted CDFs and any threshold loss function. The lemma and theorem in this section are proven Appendix
We define the property of threshold calibration below.
Definition (Threshold Calibration). A forecaster satisfies threshold calibration if
Pr[b[[](Y) s h[x](1)) S a] Vyo E VV,a E [0, 10,1), [0.
(1)
A threshold-calibrated forecaster is cerage-calibrated on subsets of the predicted CDFs that sat- isfy h[X](1)) I a. We make the following observation about conditioning on the complementary predicted CDFs.
Lemma 1. Given forecaster h that satisfies Definition 3, then we have that Vyo E V,a E [o, 1], Vc [0, 1], Pr[h([](Y)) S b[[][())) =
In threshold decision task, decision maker will take action given inputs with predicted CDFs satisfying (1)) S a (and take a complementary action given inputs with predicted CDFs
satisfying (36) > a). Intuitively, threshold calibration ensures that the forecaster satisfies
average calibration on the subsets of predicted CDFs where the decision maker chooses a = 0 and =

---

Threshold calibration is specific type of group calibration [28], where calibration across the collection of groups (X x [1X] S oee is desired. Since threshold calibration requires achieving calibration on intersecting groups, it is also related to the notion of multicalibration [18]. In Section 4, we give an efficient algorithm for achieving threshold calibration that is inspired by previous work on multicalibration.
Using Definition 3 and Lemma I, we define the threshold calibration error (TCE) measure deviation from threshold calibration at a threshold E y and quantile a E [0, 1].
Definition (Threshold Calibration Error).
TCE(h, ao = [ (P(MXI()) - a) -cl de
L Pr[MX](()) S (X) al de.
Threshold calibration is desirable property due to its connection to achieving zero reliability gap.
Theorem 1. Let be the space of threshold loss functions. Given a forecaster h, let D, be the space of threshold decision rules the forecasted CDFs of h. A forecasrrr h satisfies threshold calibration if and only (6, E) = 0 45 E An, 4E L.
We obtain this result by observing that the expected decision loss under the true distribution can be decomposed into two terms. The first term corresponds to the cost incurred from "false positive" errors and the second term corresponds to the cost incurred from "false negative" errors. Under threshold calibration, the forecaster's ' predicted eгrоr rates match the true eπror rates. Since the decision loss (with any choice of costs) is linear combination of these error rates, the expected decision loss predicted by the forecaster matches the expected decision loss under the true distribution. Thus, under threshold-calibratee forecaster, we achieve zero reliability gap under any threshold decision on the forecasted CDFs and any threshold loss function.
3.4 Comparison to Existing Calibration Definitions
We compare threshold calibration to other methods for calibrating probabilistic forecasts. Average calibration is the standard definition of calibration for regression [23, 12].
Definition 5 (Average Calibration). A forecaster h satisfies average calibration if
Pr[M[X](() s = Vc [0, 1].
In other words, forecaster averago-calibrated if the true label is below the e-th quantile of the forecasted CDF (([x]] exactly percent of the time.
In contrast, distribution calibration is much stronger definition of calibration [31]. Intuitively, distribution calibration requires a forecaster to be calibrated for every distribution in the forecaster's model family.
Definition 6 (Distribution Calibration). A forecaster h satisfies distribution calibration if
Pr[M[x](Y) S c M(x] g] = Vg E F()),,
where F is space of CDFs corresponding to the forecaster'ss 's model family.
We outline the relationship between average, threshold, and distribution calibration in the following proposition.
Proposition 1. If a forecaster satisfies distribution calibration, then it satisfies threshold calibration. a forecaster satisfies threshold calibration, then it satisfies average calibration.
We note that the converses of the statements in Proposition 1 are not necessarily true. A threshold- calibrated forecaster does not necessarily satisfy distribution calibration. An average-calibrated forecaster does not necessarily satisfy threshold calibration or distribution calibration (see Appendix C). This implies that an average-calibrated forecaster does not satisfy the necessary condition of Theorem I, meaning that the reliability gap under threshold decisions may not be zero. So, decision
5

---

makers who rely on forecaster that only satisfies average calibration (but not threshold calibration) are not guaranteed accurately estimate their decision loss under threshold decisions.
From Proposition 1, we have that a distribution-calibaaeed forecaster satisfies the necessary condition of Theorem 1. However, distribution calibration be challenging to achieve in practice because the same CDF is rarely predicted more than one time on the training samples, making it difficult to guarantee calibration without compromising the sharpness of the forecasts. Sharpness corresponds to the width of the prediction intervals generated from the forecasts, and sharp forecasts yield short prediction intervals. Although distribution calibration theoretically guaranteed yield zero reliability gap. we observe that achieving distribution calibration is challenging when the model family complex (Section 5).
Finally, we emphasize the threshold calibration exactly the condition needed to guarantee the reliability gap is zero in Theorem 1.
4 Achieving Threshold Calibration
We design a recalibration algorithm that takes an uncalibrated forecaster as input and provably outputs careshold-calibrated forecaster. Our algorithm is iterative procedure that terminates when the maximum TCE is less than a user specified threshold e. Our key result is that the algorithm must terminate after (L))²²) iterations.
"seudo-code for the algorithm is shown in Algorithm 1. Intuitively, at each iteration of the algorithm, find the y and a' where the TCE in Definition 4 is maximized. This partitions the input X' into two parts: those where h (16) < a' and those where [=x](16) > a'. For each partition, we use a standard recalibration algorithm (Isotonic regression [23]) achieve average calibration. Intuitively, after the recalibration step, the forecaster should satisfy average calibration for each partition, and hence the TCE in Definition 4 must be (close to) for vó and a.. We repeat this procedure until the TCE is less than e for every possible and a.
Algorithm 1: Threshold Recalibration
Input: Forecaster h X' - F()",, maximum error 0
2 Output: threshold-calibratee forecaster
3 Set h +
4 for t 1, until maximum threshold calibration error sup
TCE(h'-1, ,30,, a) S do
Find the yo and a that maximize threshold calibration error.
yú, a² + argsup, (ng-n)eyx(o.1 TCB(h'-1, yo, a)
Partition input features X' into Xn + far E X' h² [x][6]] S Saa} and Xi = X Xo.
Use Isotonic regression to learn recalibration maps ob. of F()) + F())) on X, and X; respectively.
Apply the recalibration map to obtain new prediction functions. (h' [x]]) if Xa (1+-(()))) otherwise
end
10 return AT where the final iteration count.
The following theorem shows that our iterative threshold calibration procedure converges in small number iterations. The intuition of the proof that after each iteration, the L2 distance between the prediction functions h and the true CDF ht must decrease by least e². Therefore, the algorithm must terminate before the 1.2 distance decreases below 0 (which is impossible). A full proof is provided in Appendix B.
Theorem 2. Algorithm converges after at most O(1/e) iterations and outputs forecaster with threshold calibration error at most e.
For simplicity, we do not consider finite sample approximation of the TCE in line 5 of Algorithm 1. Line 5 can interpreted in two ways: line 5 estimates the TCE on the true distribution (which we can only do with infinite samples), the empirical distribution (i.c. the uniform distribution

---

the recalibration data). Under the former interpretation, Theorem 2 holds assuming that line 5 can estimate the true TCE (which is the ideal scenario with infinite data). Under the latter interpretation. Theorem 2 holds for the empirical distribution, i.e. it guarantees that Algorithm will output forecaster with threshold calibration error at most on the empirical distribution rather than the true distribution. We will instead use experiments to show that Algorithm can generalize to the true distribution. Note that under both interpretations, Algorithm will converge after at most ((1/²²) iterations. For completeness, we describe the finite sample version of the algorithm in Appendix A.
5 Experiments
In the following experiments, we demonstrate that threshold calibration minimize the reliability gap (1) across a range of decision costs, (2) across a range of decision thresholds, and (3) in simple and complex model families. Across all datasets and forecaster model families that we consider, we find that threshold calibration outperforms the baselines in reducing the size of the reliability gap while attaining similar improved decision loss compared to the baselines.
5.1 Datasets
We consider datasets that relate to real-world decision decision-making tasks and standard benchmarks. In the main paper, we show results on the UCI Protein and the MIMIC-III datasets. All remaining results can be found in Appendix A.
MIMIC-IIL Patient length-of-stay predictions are used for hospital scheduling and resource man- agement [17]. We consider a patient length-of-stay forecaster trained on patient admission laboratory values from the MIMIC-III dataset [20]. In our decision task, the hospital decides to schedule a new patient for an elective procedure if a current patient is predicted to have a short length of stay.
Demographic and Health Survey (DHS). Local wealth measurements allocation decisions. We use the DHS data from Sheehan et al. [30] to predict asset wealth from satellite images as done in Yeh et al. [32] and Sheehan et al. [30]. Our experimental setup is motivated by the decision task defined in Yeh et al. [32], where aid is allocated to regions where the predicted asset wealth falls below particular threshold.
used to inform resource
UCI Regression Datasets. We evaluate on a suite of UCI regression datasets (Naval, Protein, Energy, Crime) [11]. They are common benchmarks in the uncertainty quantification literature [31, 2, 8, 23].
5.2 Experimental Setup and Baselines
Experimental Setup. We consider a forecaster that outputs Gaussian distributions and a forecaster that outputs Gaussian-Laplace mixture distributions. We train/validationntest split. The un- calibrated forecaster is a neural network trained on the training set with the validation set used for early stopping. For large datasets (Protein, Energy, Naval, MIMIC-II), the recalibration transform is trained on the validation set. For small datasets (Crime, DHS), the recalibration transform is trained on the training and validation set. On the test set, we evaluate our method and the baselines using decision decision-making making metrics (Section 5.3). Calibration metrics are also measured and results are provided in Appendix A.
Baselines. We compare the uncalibrated forecaster to the forecaster after enforcing average, threshold, or distribution calibration through posthoc recalibration procedure. Methods for achieving these properties are described in Appendix A.
5.3 Decision-Making Metrics
We simulate decision makers enumerated 1. who use a probabilistic forecaster h for their threshold decision tasks. We assume that there is no cost associated with true positives or true negatives, and the total cost of a false positive plus a false nega- tive is equal to 10 for all decision makers. As a result, decision maker i's task is deter- mined by a decision threshold y and decision cost ratio G.. Each decision maker has a
loss function (1x y. a) 10c,1l(a 1, 2 vd) + 10(1 c.)l((a 0, y < vo) and decision rule
7

---

Patient Length-af-SSae
Protein
Length-offSSe
UCI
Reliabiltyy Cap
Trreshole
Averoge
Nare
Reliabiiity Gap
True Decision Loss
0.8
1.0
True Decigion Loss
0.6
0.8
0.6
airibut on
0.2
0.6
0.4
0.4
0.4
0.22
0.0
0.0
Decision Threshold
Decision Threshald
Decision Threshold
Decision Threshald
Figure 2: Under the Gaussian forecaster and across different decision thresholds, threshold calibration reduces the reliability gap on both datasets while average calibration does not reduce the reliability gap on the Protein dataset (Left, Middle Left), and all calibration methods yield improved or comparable decision loss compared to the uncalibrated forecaster (Middle Right, Right). Error bars represent 95% confidence intervals and are generated over random trials.
(x) ((][x] 2 a.. We consider decision makers with (36- c,) Vo x C. where Vo and C each consist of 50 uniformly spaced points that span the label space and [0.05, 0.95], respectively.
For each decision maker i, we compute the decision loss (the loss incurred by the Bayes decision rule 5% (X)) and the reliability gap (averaged over the possible threshold decision rules).
1 [C] Σ h18h ())
Decision Loss Ex (XX, (X,Y,
Reliability Gap
Aggregate statisties can obtained by averaging over all M decision makers, all decision makers who share the same threshold yo. or all decision makers who share the same cost ratio c.
5.4 Results
Using the MIMIC-III and UCI Protein datasets, we study the effect of recalibration the reliability gap and the decision loss achieved by decision makers with different decision thresholds and cost ratios. Furthermore, we examine the effect of recalibration forecasters that output CDFs from simple (Gaussian) and complex (Gaussian-Laplace mixture) model families.
Threshold Calibration Minimizes Reliability Gap Across Decision Thresholds. We evaluate the effect of recalibrating the Gaussian forecaster on decision makers with different decision thresh- olds. On both datasets, threshold calibration yields the largest decrease in the reliability gap (left plots, Figure 2). Distribution calibration also decreases the reliability gap across decision thresholds, relative to the uncalibrated Gaussian forecaster. Average calibration does not consistently reduce the reliability gap: on the UCI Protein dataset, the reliability gap of the werage-callibaate forecaster enjoys slight decrease at some decision thresholds but is increased at others, relative to the un- calibrated Gaussian forecaster (middle left, Figure 2). Lastly, these calibration methods achieve similar decision loss the uncalibrated forecaster (right plots, Figure 2). These trends are consistent with the results obtained on the other datasets under the Gaussian forecaster. Threshold calibration outperforms baselines across different decision thresholds under the Laussian-Laplace forecaster, as well (Appendix A).
Threshold Calibration Minimizes Reliability Gap Across Decision Cost Ratios. Across deci- sion makers with different cost ratios, distribution and threshold calibration reduce the reliability gap relative to the uncalibrated forecaster, with threshold calibration yielding the largest decreases in the reliability gap (left plots, Figure 3). Mcanwhile, average calibration does not consistently reduce the reliability gap; the UCI Protein dataset, it achieves similar reliability gap the uncalibrated forecaster (middle left, Figure 3). As before, these calibration methods achieve similar decision loss to the uncalibrated forecaster (right plots, Figure 3). These trends are consistent with results obtained on the other datasets under the Gaussian forecaster. Threshold calibration outperforms baselines across different decision cost ratios under the Caussian-Laplace forecaster, as well (Figure 4).
Distribution Calibration Degrades Performance under Complex Model Families. Forecasters that can output CDFs from more flexible model families (c.g. -aussian-Laplace mixture distributions)
8

---

0.18/MIC--II Length-of-Stay
Length-of-Stay
0.125
0.25
True Decision Less
0.3
True Decision Loss
0.20
0.100
Reliabilty
Deliabiiity
0.2
0.075
Thresho
0.15
0.4
bution
0.050
0.10
0.025
0.05
0.75 Decision Cost Ratio
0.25
0.75 Cost Ratio
0.25
0.25 0.50 Decision
Figure 3: Under the Gaussian forecaster and across different decision cost ratios, threshold calibration reduces the reliability gap on both datasets while average calibration does not reduce the reliability gap on the Protein dataset (Left, Middle Left), and calibration methods yield improved or comparable decision loss compared to the uncalibrated forecaster (Middle Right, Right). Error bars represent 95% confidence intervals and are generated over random trials.
MIMIC-III Patient Longth-af-Stay
UCI Protein
MIMIC-III Patient Length.of-Stay
UCI Protein
0.125
Trreshold
Average
Rellabiiity C89
True Decision Loss
0.3
True Decision Loss
Rejabiity
0.100
Dist butlon
0.2
0.075
0.050
0.025
0.05
0.2
0,75 Decision Ratto
0.25 0,50 Dacision Cost Ratio
Dec Cost Ratio
Decision Cost Ratio
Figure 4: We consider the effect of recalibrating the Laussian-Laplace forecaster under a range of decision cost ratios. Threshold calibration reduces the reliability gap while distribution calibration can enlarge the reliability gap (Left, Middle Left), Average and threshold calibration achieve comparable or lower decision loss as the baseline forecaster, while distribution calibration increases the decision loss. Error bars represent 95% confidence intervals and generated over 6 random trials.
may be able to better capture the true conditional distribution of given x compared to Gaussian forecasters. As result, examine the effect of the recalibration procedures when the uncalibrated forecasts follow a more flexible distribution. The uncalibrated Gaussian-l .aplace forecaster (Figure 4) yields smaller reliability gap and smaller decision loss compared the uncalibrated Gaussian forecaster (Figure 3). Applying threshold calibration to the Gaussian-Laplace forecaster further reduces the reliability gap. However, under the Gaussian-Laplace forecaster, distribution calibration enlarges the size of the reliability gap and increases the decision loss. Although distribution calibration theoretically guaranteed minimize the reliability gap. is challenging achieve in finite samples without compromising the sharpness of the forecasts (in our case, decision loss). So, we find that decision loss and reliability gap increase. We hypothesize that the recalibration dataset may not contain many instances that yield similar distribution parameters, so the recalibration transform does not generalize well unseen data. We also observe these trends the UCI Crime, UCI Energy, and DHS datasets.
6 Related Work
Forecasting and Decision Making. The connection between forecasts and decision making was first studied in economies [1, 29]. The development of Bayesian decision analysis connected topics of forecasts and decision-based loss functions [10, 4]. Decision-makinn under uncertainty with probabilistic forecasts was then studied in econometrics
[7]. [19] also considers learning regression functions that minimize decision loss. While [19] focuses on transforming the predicted CDF to a point prediction, our method focuses transforming the predicted CDF into new CDF [19] also requires knowing the loss function to learn the transformation, while our method assumes that the loss function belongs to commonly used function family (threshold loss functions).
Calibration. Calibration definitions have been studied in the statisties literature [5, 6, 26]. For the regression setting, methods for ensuring that machine learning models satisfy average calibration

---

have been studied in [23, 8]. In addition, methods for achieving stronger calibration notions have also been introduced such distribution calibration [3 and individual calibration [34]. Calibration and trustworthy predictions in the medical domain are also studied in [16]. [16] introduces the notion of D-calibration, which is related to our average calibration baseline, but is tailored to the survival analysis task. A perfectly average calibrated prediction function is also D-calibrated, and vice versa.
Multicalibration. Our definition of threshold calibration is most related to the line of work on multicalibration [18, 21J. Given a large collection & of potentially intersecting groups of the data, predictor is multicalibrated on it is simultaneously calibrated on every sufficiently large group in G [18]. Previous works give methods for achieving mean and moment multicalibration for predictor functions. Our iterative procedure for achieving threshold calibration inspired by methods for achieving multicalibration.
7 Limitations and Societal Impact
Our work demonstrates that certain types of calibration enable decision makers to estimate decision loss before deployment, which should not be confused with enabling decision makers to make optimal decisions. For example, forecaster that always outputs the marginal distribution of Y is threshold-calibrated but likely incurs high decision loss. Furthermore, posthoc recalibration is limited by the quality of the baseline model. If the baseline model outputs the marginal distribution of Y. then it is already preeshld-ccaibratee but likely is not useful for decision making. Applying our threshold calibration method will not offer any benefit in this case.
Also, our work assumes that predictions of do not affect the true label Y. However, when predictions used to make decisions, they can often influence the outcome they aim to predict [27]. Our work does not account for these performative effects, so the decision loss may not be accurately estimated in these settings. Future work could focus developing calibration procedures that enable forecasters to be robust to such distribution shifts. In addition, we specifically focus on binary-action threshold decisions. Future work may generalize our results the setting where decision makers have loss functions involving multiple thresholds and multiple actions.
There is potential for negative societal impact if threshold calibration is incompatible with fairness criteria. Nevertheless, we note that the perfect predictor (that predicts the true conditional probability) satisfies calibration definition. Consequently, if the perfect predictor satisfies some faimesss notion (such as group calibration), then our calibration definition is also compatible with that fairness notion. Note that the perfect predictor does not satisfy faimess notion called demographic parity, hence calibration definition is not compatible with demographic parity either.
Conclusion
We show that a threshold-calibrated forecaster theoretically guarantees accurate decision loss estima- tion under threshold decision losses and threshold decision rules. We provide an iterative procedure for achieving threshold calibration and show that in practice it minimizes the reliability gap relative to baselines without compromising the forecaster's s decision loss. Such estimates permit decision makers to reason about the consequences of their decisions prior deployment.
Acknowledgements
RS supported in part by a NSF GRFP under grant number DGE-1656518. SZ is supported in part by a JP Morgan fellowship and a Qualcomm innovation fellowship. SE is supported in part by NSF #1522054, #1733686). ONR (N000141912145) AFOSR (FA95501910024) ARO (W911NF-21-1-0123 and Sloan Fellowship. We are grateful for Rishi Bommasani, Kristy Choi, Matthew Jörke, Judy Shen, Rui Shu, Fan-Yur Sun, Rohan Taori, Ke Alexander Wang, Rose Wang, and Henry Zhu for insightful discussions.
References
[1] H. theil. economic forecasts and policy. assisted by j.s. cramer, h. moerman, a. russchen. contributions to economic analysis, amsterdam, north-holland publishing company, 1958,
10

---

xxxi 562 p.. fl. 50- Bulletin de I'Instiuut de recherches économiques et sociales, 25(2): 169-169, 1959. doi: 10.1017/5133397190007777
[2] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14927-14937. Cur- ran Associates, Inc., 2020. URL https: //ppr//eeingo/gep/eeeeeee///e neurips cctccc//pppa wwwpppppr///2///a//// aab085461de 2tt.B//dd...00000000777777777110 Paperr
[3] Barnes, Eric Hamrock, Matthew F. Toerper, S. Siddiqui, and S. Levin. Real-time prediction of inpatient length of stay for discharge prioritization. Journal of the American Medical Informaties Association JAMIA, 23 e1:e2-ell 2016.
[4] James O. Berger and James O. Berger. Statistical Bayesian analysis. Springer- Verlag, New York, 1985. ISBN 0387960988
decision theory and
9780387960982 3540960988 9783540960980.
URL http:/ /wwp://wwwww/c/// amazon. com/
[5] GLENN W. BRIER. Verification of forecasts expressed terms of probability Monthly Weather Review, 78(1);1 - 1950. doi: 101.7711122000005500000000001110
URL https:/ ametsoc.....................o... www.aaa..rg/tiieeeeee...eee..eee /tt.://www.........ooooooooaaaaa
xml.
[6] Nicolo Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games. Cambridge University Press, USA. 2006. ISBN 0521841089.
[7] Gary Chamberlain. Econometrics and decision theory. Journal of Econometries, 95:255-283, 2000.
[8] Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17164-17175. Cur- ran Associates, Inc., 2020. URL ntttps://proceedingg ieapeeedinnnnnuuiiipeeeee hwt.rpp..cc/ppppe///20000 cctppppee/222//f12ee. c7A4c aa50a Papee Pffer.pdf..
[9] Murray Dale, Jon Wicks, Ken Mylne, Florian Pappenberger, Stefan Laeger, and Steve Taylor. Probabilistic flood forecasting and decision-making: An innovative risk-based approach. Natural Hazards, 70, 11 2014. doi: 10.10077/11009--02-04444
[10] Morris H. DeGroot. Optimal statistical decisions. McGraw-Hill, New York, NY [uaa], 1970. ISBN 0070162425. URL http: bibsooooobbbbbo...oo
[11] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. ics uci. edu/ml.
[12] Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B, 69(2):243-268, 2007. URL https://EconPaperts repec. org Rtt//RePPP::lle bla:jorss v 69: 2007 : :::::::::::::::::::::: 243-268..
[13) Ary I. Goldherger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Bhung--ang Peng. and Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of research resource for complex physiologic signals. Circulation, 101(2))e215-=222,, 2000.
[14) Clive W.J. Granger and Mark Machina. Forecasting and decision theory. In G. Elliott,
C. Granger, and A. Timmermann, editors, Handbook of Economic Forecasting, volume I of Handbook of Economic Forecasting, chapter 2, pages 81-98. Elsevier, 2006. URL https: //ideas repec. org/o.///// eeee// ecof ch/ofch/11 1ethh//--a..1 html.
[15] Margaret Grosh, Carlo del Ninno, Emil Tesliuc, and Azedine Ouerghi. For Protection and Promotion: The Design and Implementation of Effective Safety Nets. The World Bank, 2008 URL https:/ /Ew////peeeeeeeeeeeeeeeeeeee repec. org/ org/RePEc wbb: wbpubs: 6582.

---

[16] Humza Haider, Bret Hoehn, Sarah Davis, and Russell Greiner. Effective ways to build and evaluate individual survival distributions. Journal of Machine Learning Research, 21(85):1-63, 2020. URL http: / jmlr. .rgg http:/pappss//11/////ee
html.
[17] H. Harutyunyan, Hrant Khachatrian, David C. Kale, and A. Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6, 2019.
[18] Ursula Hebert-Johnne Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1939-1948. PMLR, 10-15 Jul 2018. URL http: / /ttp:////cceee...../..e mlr. press/ vate heep///eeerttteee http//ebbbrrrrooneeee.. johnson18a hattoonn.....m........
pnppuutatiioalll--dnnffaaal masses. In Jennifer Dy and Andreas Krause,
[19] José Hernandez-Orallo. Probabilistic reframing for cost-sensitive regression. ACM Trans. Knowl. Discov. Data, 8(4), August 2014. ISSN 1556-4681. doi: 10.1145/2641758, URL
https://doi.. orgeiiorrg/00/ 10. 1145/2641758.
[20] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035, 2016.
[21] Christopher Jung, Changhwa Lee, Mallesh M. Pai, Aaron Roth, and Rakesh Vohra. Moment multicalibration for uncertainty estimation, 2020.
[22] Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18237-18248. Curran Associates, Inc., 2020. URL attps://proceett file/ (tllll/2aa4444022222555555538882 -aeeeeddd6dee82--e--0e--
www.nneu...... wwtpwcceppppppcccccceeeeee /020/
[23] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learn ing using calibrated regression. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüs- san, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2801-2809, PMLR, 2018. URL http::/prroceedngg mlr, www.-pressseeeeee greas/v80/ kuleshore eshov18a...... html....
[24] Meelis Kull, Telmo de Menezes Silva Filho, and Peter Flach. Beta calibration: well- founded and easily implemented improvement on logistic calibration for binary classifiers. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL. USA, volume 54 Proceedings of Machine Learning Research, pages 623-631. PMLR, 2017. URL http: http:////oeeeeenn.eeeee mlr, pwwss/eeeveeeeeee h/////54////// /wll/alllll../ html.
[25]
Ali Malik, Volodymyr Kuleshov, Jiaming Song. Danny Nemer, Harlan Seymour, and Stefano Ermon. Calibrated model- deep reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4323. PMLR, 2019. URL http: pttp///eecccee...../ mlr. ppwss/v97 http://99/mmakkkkaa malik191llaaaaammmmmmmmmmmm1mm1m html.
[26] Allan H. Murphy. A new vector partition of the probability score. Journal of Applied Mete- orology and Climatology, 12(4):595 600, 1973. doi: 10. 10.1175/122004450997330110 12/ 2NV URL https: ametsoc. org/ amwtsso...rrg/ii///ooraaaalll 4/ 4/1520- 0172000055011110 11w1_1111111__11111111 4/2.220......11302222222........
xml.
[27] Juan Perdomo, Tijana Zmnic, Celestine Mendler- Dünner,, and Moritz Hardt. Performative prediction. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7599-7609. PMLR, 13-18 Jul 2020. URI. http: http:// httta::://pooceiingss./. .ml.......................... mlr. htt..prres/vv11
perdomo20a html.
12

---

[28] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On faimess and calibration. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http:////p/o/oeed/e////ee www.ssnnerrs.... heerripp.ccppaper/20777 file/
pdf.
[29] Robert W. Rudd. Theil, henri, applied economic forecasting, chicago, rand menally ... company, 1966. xxv + 474 pp. ($14.00). American Journal of Agricultural Economics, 49(l1_aar____ 1):241- 243, 1967. doi:
URL https://onllieeiibrree wiley.
com/ com/doi/abs/ /tcabb//1... 2307 12337096.
[30] Evan Shechan, Chenlin Meng. Matthew Tan, Burak Uzkent, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon. Predicting economic development using geolocated wikipedia articles. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD '19, page 2698-2708 New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 1450362016. 3330784.
doi: 10.1145/3292500.3333733.. URL
http:://ddo.orrgg
/ 3292500. 01144///22500
[31] Hao Song. Tom Diethe, Meelis Kull, and Peter A. Flach. Distribution calibration for regression. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5897-5906. PMLR, 2019. URL http: http://oooeeednn...ee/ mll watirrppess/vve /99 song19a. html.
[32] Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 11(1), 5 2020. ISSN 2041-1723. doi: 10. 1038/s41467-0D55 16185-w. articles/s41455 020 16185-w.
nature. com/ URL https: http:://wwwwwwww..at/eaaaaaaeae.
[33] Weiran Yuchi, Jiayun Yao, Kathleen E. McLean, Roland Stull, Radenko Pavlovic, Didier Davi- gnon, Michael D. Moran, and Sarah B. Henderson. Blending forest fire smoke forecasts with observed data can improve their utility for public health applications Atmospheric Environment, 145:308-317, 2016. ISSN 1352-2310. doi: cttp://ccscc//rrrc//////////22/2
URL https: / /www///wwww///oo sciencedirect. wtteeiirrrttttcceeeeeeeeeeeeeeee wtt.cco///c/eeeeeeeee
[34] Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11387-11397. PMLR, 13-18 Jul 2020. URL http:/ httt:://rroceedngs. mlr press/ v119/z 2219/zhao2ee. html.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Appendix A.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/]]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
13

---

(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section and Appendix A
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix A.
(b) Did you mention the license of the assets? [Yes] See Appendix A.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See Appendix A.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A] We did not directly obtain data from anyone but used publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The creators the MIMIC-III dataset. which we use. deanonymized the data to remove any personally identifiable information. To the best of our knowledge, there no other potential source of personally identifiable or offensive content in the data we use.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not do human subject research.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not do human subject research.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did do human subject research.
14


Reviews:
[{"review_id": "gfdqCahtnte", "reviewer": "Reviewer_VsY9", "summary": "This paper considers the problem of calibrating the conditional CDFs from a regression model.  \n\nThe authors propose a new definition of threshold calibration, which requires the conditional CDFs to have calibrated quantiles when these CDFs have a $\\alpha$-quantile that are all on the same side regarding a given $y_0$ (either above or below).  \n\nThe proposed definition is a stronger notion of quantile calibration in regression (called averaged calibration in this paper), and a weaker notion of distribution calibration in regression. To further justify the proposed definition, the authors consider measuring the so-called reliability gap, which is based on a given loss function and the corresponding decision rule. The authors then show that having threshold-calibrated results can ensure a 0 reliability gap. The necessity and sufficiency among different calibration definitions are later demonstrated.\n\nOn the method level, the proposed approach is hence to iteratively search for quantile levels and thresholds with large calibration errors and applies isotonic regression to calibrate the corresponding quantiles. The final calibrated model will include the uncalibrated models and a sequence of calibration maps.\n\nExperiments are performed with real datasets with a cost-sensitive setup. The proposed method is compared with uncalibrated model, quantile calibrated model and distribution calibrated model. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the top level, the problem of calibration has been recently revisited a lot among the classification context, and it is interesting to see follow-ups on the regression scenario. As indicated by the authors, most existing works focus on the definition of quantile calibration (df.5 average calibration), so it is good that the authors propose a stronger notion of calibration.\n\nRegarding the definition of threshold calibration, while it is less intuitive at the beginning (due to the nested notation of CDFs), it can be told that the authors propose to group the conditional CDFs that have the $\\alpha$-quantile below a given threshold of $y_0$. (or above, as it is exchangeable given Lemma 1). The authors use Theorem 1 to justify that the proposed calibration can ensure a minimal reliability gap as proposed. Later the authors discuss the necessity and sufficiency between threshold calibration and quantile calibration/distribution calibration. It is generally sounding to me as threshold calibration requires a stronger grouping on the CDFs than quantile calibration, and is weaker than distribution calibration.\n\nOn the method level, the proposed method seems straightforward. The idea is to iteratively search the quantile level $\\alpha$ and threshold $y_0$ with large calibration errors. And the calibration method is similar to the ones in [22] and uses isotonic regression. However, one thing I notice here is that the calibration map is essential to a chained sequence of maps following the learning iterations. This concerns me as isotonic regression can be expensive on larger datasets. Although the authors show that there is some guarantee on the number of iterations (hence calibration maps), it still worries me in terms of the scalability.\n\nRegarding the experiments, current results should provide some basic intuition over the benefits of threshold calibration, and I appreciate the comparison against quantile calibration and distribution calibration. \nIn general, I agree with the intuition that threshold calibration provides a middle ground between the other two definitions. While distribution calibration is the strongest requirement, it might be hard to perform for certain datasets. However, later I notice that in the appendix the authors suggest that they perform distribution calibration through a grid level isotonic calibration, which is different from the method proposed in [30]. Since distribution calibration is quite hard to achieve in general, I have concerns that the grid isotonic approach in this paper can easily lead to sub-optimal results. I am not saying the authors should run the method from [30], but it would be nice to point out the current results are all on the variants of isotonic regression. So it is still interesting to ask if a strong distribution calibration approach can achieve similar performances.\n\n==============After author feedback=========================================\n\nThanks for the feedback. \n\nI will keep current score of 7. I agree with some points from other reviewers, so there is still some room for improvement before I can give anything higher.\n\nRegarding the details, I can see why (vtF8) have issues with the notations. The current notations and definition of calibration were also a bit confusing to me during the reviewing time (e.g. it uses a nested CDF in the expression). \n\nI have the same question regarding the computational costs as the reviewer (eKGZ). Although the authors did answer with some empirical running time, they didn't argue the fact that isotonic regression is O(N) for each iteration (O(MN) in total with M being the number of iterations). While I don't think this cost could be a deal-breaker in general, but it might require some minor modification with a really large dataset.\n\nRegarding the method in [30], I agree it requires some modification to change the Gaussian mean embedding (kernel matrix) to any other general distributions, but it shouldn't be very hard to instead use Monte Carlo samples to approximate the kernel matrix. Disregarding the outcome of this paper, it would be nice if the authors can include some future experiments to show whether the calibration aim or the calibration method contributes more to the final results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitation: The authors discuss the limitations from a decision point of view.  It would be nice to further mention that post-hoc calibration is limited as it depends on a good uncalibrated model. That is, if a model simply outputs the marginal distribution of the target, this isn't much a calibration method can do.\n\nSocietal Impact: It is not discussed in section 7 as suggested. But have calibrated results for decision making should generally have a positive impact, such as seen in the ICU experiment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 4 hours"}, "extra_text": {}}, {"review_id": "bAo_HNL5vYk", "reviewer": "Reviewer_vtF8", "summary": "This paper discusses calibrated regression from a decision-making perspective, by introducing the concept of 'threshold calibration'. The popular notion of calibrated regression, average calibration, is insufficient for decision-making. On the other hand, the notion of distribution calibration is sufficient for decision-making, but may be difficult to achieve. In practice, decisions based on regression predictions are often made on the basis of whether the prediction is smaller or larger than some threshold. Thus, a decision-oriented goal is to ensure calibration with respect to such a threshold, called threshold calibration. The paper proposes an algorithm to achieve threshold calibration. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "An important question that has not been considered in the ML literature is how a calibrated regressor can be used meaningfully to make practical decisions. Thus, I believe this paper asks a question that is both highly relevant to the field and has not been studied before. In lines 41–45, the authors summarize these motivations well.\n\nWhile I was excited about the motivation, the paper uses fundamentally flawed notation that is not in keeping with standard measure-theoretic conventions (more details in the following paragraphs). Due to this, I was unable to understand the technical content after Section 3.3 despite putting more effort than should be necessary to read a paper in a top ML conference. For these reasons, I recommend that the authors revise their paper to follow the standard conventions. (The authors themselves summarize these conventions correctly in lines 70-77, but do not follow them.)\n\nThe key Definition 3 does not parse. h\\[x\\](y_0) is not a random variable; it is just some fixed value as mentioned on line 75. Thus conditioning on h\\[x\\](y_0) \\leq \\alpha makes no sense. In particular, the definition does not capture “condition on predicted CDFs which have α-quantile equal to y0”.  At this point, as a reader, I am not sure what is random and what is not, and what is being conditioned on, and I find it quite difficult to follow the rest of the paper. Since I don’t understand Def 3, I do not understand threshold calibration error (TCE). In the same sense, Def 6 does not parse since h(x) = g is a statement that is either true or false; there is nothing random. While proposing Def 6, the present authors cite previous work that proposed it [30, Definition 3]. The mathematical exposition in [30] is precise as per standard measure theory conventions, and I suggest that the authors follow the same conventions to convey their ideas.\n\nI tried skipping forward to Algorithm 1, but could not understand the main idea without understanding TCE. However I have some concerns regarding the presentation of Alg 1 and Thm 2, summarized in the following paragraph.\n\nAlgorithm 1 requires access to the true TCE and recalibration maps. In my assessment, the authors are not very transparent about this. This reliance on the true TCE is only briefly alluded to in lines 211-212, and is mentioned nowhere else in the main paper, as far as I can tell. One suggestion to improve transparency is to specify these oracles in the “input” line of the algorithm. Crucially, this lack of transparency makes Theorem 2 appear to be much stronger than it is. The claim of Theorem 2 is regarding the TCE of an algorithm that has access to the true TCE. In practice, when this TCE is estimated on a split of the training data (let us call it calibration data), it only shows that the calibration TCE will be smaller than \\epsilon in 1/\\epsilon^2 steps. Theorem 2 makes no claims regarding the generalization from calibration TCE to the true TCE, and no other comments are provided to this end either (in the main paper or the proof of Theorem 2). Line 201: “Our algorithm is an iterative procedure that terminates when the maximum TCE is less than a user specified threshold” implicitly suggests that the algorithm is reducing the true TCE, which is incorrect.\n\n**Minor comments:**\n- Lines 90, 127, etc: What is the little 'x' in the subscript? There is no specific feature x \\in \\Xcal mentioned anywhere else in Definition 1. \n- Line 172: “c percent of the times”: this is incorrect since c percent suggests 0.01c. \n- Line 373: \"[10] A. Dawid. Present position and potential developments: some personal views. 1984.\" This is not a proper citation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned in the previous section, the reliance of Algorithm 1 (and its guarantee) on the TCE oracle is a key limitation that was not discussed transparently. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XnOa9NFawN", "reviewer": "Reviewer_eKGZ", "summary": "This paper focuses on the task of \"calibrating\" forecasts used in downstream decision-making tasks to ensure accurate estimation of the loss or \"cost\" of using each decision rule (i.e. the decision loss). The paper is specifically focused on forecasts of real-valued quantities (regression) and binary decisions (or actions) made by thresholding the forecast (for example, deciding whether to accept or reject a new patient based on the forecasted length of stay of a current patient being above or below some threshold).\n\nIn this setting, the paper shows that average calibration -- a common type of calibration -- does not ensure accurate estimation of the decision loss. The paper then introduces threshold calibration, proves that threshold-calibrated forecasts maximize the accuracy of decision loss estimates (and, moreover, that threshold-calibration is necessary and sufficient to maximize this accuracy), and provides an efficient algorithm to generate threshold-calibrated forecasts from a finite data sample. The paper validates the performance of their algorithm on real-world data (although with a synthetic downstream decision-making setup), and demonstrates more accurate decision loss estimates than those obtained by average and distribution calibration. The paper also derives the relationship between distribution, threshold, and average calibration in Proposition 1, which may be of independent interest.", "questions": "", "limitations": "", "rating": 9, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Key Strengths\n\n**Novelty and significance.** The paper proposes novel notion of forecast calibration for the common setting of regression forecasts and binary threshold decisions. Several real-world decision-making tasks conform to this setting, such as evaluating loan applicants based on their risk/credit score (action = accept or deny loan based on the forecasted risk), planning hardware replacements in datacenters (action =. retain or replace the server based on the forecasted time to failure), and the medical and aid allocation settings described in the paper.\n\nIn each of these settings, decision-makers would benefit from accurate estimates of their costs/loss for each possible decision rule (i.e. the chosen threshold). The paper introduces \"threshold calibration\" and proves that this maximizes the accuracy of these decision loss estimates (in addition to being necessary and sufficient for this accuracy), provides a practical algorithm to threshold-calibrate forecasts, and empirically demonstrates more accurate decision loss estimates (compared to other types of calibration) on real-world data. The provided algorithm is inspired by work in multicalibration, but is novel as applied to this setting.\n\nThe paper also develops links between distribution, threshold, and average calibration in Proposition 1, which may be of independent interest to future research in this space.\n\n**Links to related work.** The paper concisely and adequately surveys and connects with related work in this space.\n\n**Clarity.** The paper provides intuition and examples for each theoretical result, and the paper overall is written clearly with well-organized sections and notation.\n\n# Key Weaknesses\n\n**Less-than-satisfactory discussion of the algorithm's running time in-practice.** While algorithm 1 involves $O(1/\\epsilon^2)$ iterations for an error-rate of $\\epsilon$, I believe each of these iterations involves significant computational work.\n\nFinding $y_0^t$ and $\\alpha^t$ in each iteration involves searching over a grid and evaluating the TCE integral for each pair of values. This grid search would involve testing each value of  $y_0^t$ in the label space, which could be continuous and unbounded, requiring some kind of ad-hoc discretization and restriction to an interval. The input space partitioning would scale linearly with the number of features, which can be large in many real world settings. Isotonic regression adds additional computational overhead.\n\nWhile a consequence of this computational cost is more accurate decision loss estimates (as the paper empirically shows), I think the authors should make a note of the computational cost of achieving this accuracy relative to average calibration, so readers are aware of this tradeoff (actual running times of Algorithm 1 vs. average calibration in the experimental evaluation would be illuminating).\n\n# Minor Concerns\n\n**Section 5.3:** What is the value of $M$ used in the experiments? Should \"all $k$ decision makers\" at the end of this section read \"all $M$ decision makers\" instead?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have noted important issues when forecasts could influence the target of future predictions, and note that their proposed method helps decision makers evaluate pre-deployment decision losses (and not make optimal decisions). They also not that their scope is limited to binary thresholds and actions.\n\nI believe this section is sufficiently comprehensive, but could be improved by noting potential [links between calibration and fairness](https://papers.nips.cc/paper/2017/file/b8b9c74ac526fffbeb2d39ab038d1cd7-Paper.pdf). Specifically, could there be a potential negative societal impact if threshold calibration and some common notion of fairness is incompatible (eg. the linked paper discusses this for average calibration and equalized odds)?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "KWM2oTbYJ4T", "reviewer": "Reviewer_APiD", "summary": "This paper proposes a method to include threshold calibration in decision making.  This calibration is required to ensure that decision loss is predicted accurately for threshold decisions.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a problem where decision makers need to set thresholds in regression instances provided from forecasters. Optimising the values for the thresholds is important to reduce costs associated to bad decisions or bad forecasts.\n\nThe paper is well written, although the Figure 1 is hard to understand just reading the introduction. The authors motivate properly the problem, and formalise the framework based on two components, forecasters and decision makers. Following these two steps, they introduce the notion of Threshold Calibration to improve decisions of the makers based on the estimation of forecasters. Some theoretical properties of the framework are introduced.\n\nInspired by these properties, the authors propose a recalibration algorithm based on an iterative procedure that reduces the threshold calibration error (TCE).\n\nExperiments over a set of real examples are valid to show  that their proposal can obtain good performance in different scenarios:  across a range of decision costs, across a range of decision thresholds, and considering simple and complex model families. A summary of the results are included in the paper, however, I missed a complete table with all the results considering the studied datasets. \n\nIn general, I find the paper interesting since the propose a novel problem and introduce methods that can improve the cost reduction based on threshold configuration. Experimental results are useful to support the claims of the authors.\n\nAs points to be improved, I would extend the section of related works. For instance, the problem seems to be related to survival analysis where calibration has been studied:\n\nHaider, Humza, et al. \"Effective Ways to Build and Evaluate Individual Survival Distributions.\" J. Mach. Learn. Res. 21 (2020): 85-1.\nGoldstein, Mark, et al. \"X-CAL: Explicit Calibration for Survival Analysis.\" Advances in neural information processing systems 33 (2020): 18296.\n\nAlso, cost-sensitive regression is related:\n\nHernandez-Orallo, José. \"Probabilistic reframing for cost-sensitive regression.\" ACM Transactions on Knowledge Discovery from Data (TKDD) 8.4 (2014): 1-55.\nSong, Hao, et al. \"Distribution calibration for regression.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, correct.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"], "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a general agreement among the reviewers on the novelty and significance of the work. However, it is still clear based on the reviews that the presentation of the submitted paper could be further improved. That said, given the the satisfactory work that the authors have made during the discussion period, I am happy to suggest the acceptance of this paper and strongly encourage the authors to improve their paper based on the discussion with the reviewers. The major points to revisit are: i) polishing of the notation (as suggested by vtF8); ii) extension of the related work section (as suggested by  reviewer APiD); iii) clarifications on the scalability of the proposed algorithm and on the implementation of baselines (as suggested by reviewer VsY9). 
Decision: Accept (Poster)