Reliable Decisions with Threshold Calibration

Abstract: Decision makers rely on probabilistic forecasts to predict the loss of different decision rules before deployment. When the forecasted probabilities match the true frequencies, predicted losses will be accurate. Although perfect forecasts are typically impossible, probabilities can be calibrated to match the true frequencies on average. However, we find that this \textit{average} notion of calibration, which is typically used in practice, does not necessarily guarantee accurate decision loss prediction. Specifically in the regression setting, the loss of threshold decisions, which are decisions based on whether the forecasted outcome falls above or below a cutoff, might not be predicted accurately. We propose a stronger notion of calibration called threshold calibration, which is exactly the condition required to ensure that decision loss is predicted accurately for threshold decisions. We provide an efficient algorithm which takes an uncalibrated forecaster as input and provably outputs a threshold-calibrated forecaster. Our procedure allows downstream decision makers to confidently estimate the loss of any threshold decision under any threshold loss function. Empirically, threshold calibration improves decision loss prediction without compromising on the quality of the decisions in two real-world settings: hospital scheduling decisions and resource allocation decisions.


Reviews and author responses:

This paper considers the problem of calibrating the conditional CDFs from a regression model.  

The authors propose a new definition of threshold calibration, which requires the conditional CDFs to have calibrated quantiles when these CDFs have a $\alpha$-quantile that are all on the same side regarding a given $y_0$ (either above or below).  

The proposed definition is a stronger notion of quantile calibration in regression (called averaged calibration in this paper), and a weaker notion of distribution calibration in regression. To further justify the proposed definition, the authors consider measuring the so-called reliability gap, which is based on a given loss function and the corresponding decision rule. The authors then show that having threshold-calibrated results can ensure a 0 reliability gap. The necessity and sufficiency among different calibration definitions are later demonstrated.

On the method level, the proposed approach is hence to iteratively search for quantile levels and thresholds with large calibration errors and applies isotonic regression to calibrate the corresponding quantiles. The final calibrated model will include the uncalibrated models and a sequence of calibration maps.

Experiments are performed with real datasets with a cost-sensitive setup. The proposed method is compared with uncalibrated model, quantile calibrated model and distribution calibrated model. 
  On the top level, the problem of calibration has been recently revisited a lot among the classification context, and it is interesting to see follow-ups on the regression scenario. As indicated by the authors, most existing works focus on the definition of quantile calibration (df.5 average calibration), so it is good that the authors propose a stronger notion of calibration.

Regarding the definition of threshold calibration, while it is less intuitive at the beginning (due to the nested notation of CDFs), it can be told that the authors propose to group the conditional CDFs that have the $\alpha$-quantile below a given threshold of $y_0$. (or above, as it is exchangeable given Lemma 1). The authors use Theorem 1 to justify that the proposed calibration can ensure a minimal reliability gap as proposed. Later the authors discuss the necessity and sufficiency between threshold calibration and quantile calibration/distribution calibration. It is generally sounding to me as threshold calibration requires a stronger grouping on the CDFs than quantile calibration, and is weaker than distribution calibration.

On the method level, the proposed method seems straightforward. The idea is to iteratively search the quantile level $\alpha$ and threshold $y_0$ with large calibration errors. And the calibration method is similar to the ones in [22] and uses isotonic regression. However, one thing I notice here is that the calibration map is essential to a chained sequence of maps following the learning iterations. This concerns me as isotonic regression can be expensive on larger datasets. Although the authors show that there is some guarantee on the number of iterations (hence calibration maps), it still worries me in terms of the scalability.

Regarding the experiments, current results should provide some basic intuition over the benefits of threshold calibration, and I appreciate the comparison against quantile calibration and distribution calibration. 
In general, I agree with the intuition that threshold calibration provides a middle ground between the other two definitions. While distribution calibration is the strongest requirement, it might be hard to perform for certain datasets. However, later I notice that in the appendix the authors suggest that they perform distribution calibration through a grid level isotonic calibration, which is different from the method proposed in [30]. Since distribution calibration is quite hard to achieve in general, I have concerns that the grid isotonic approach in this paper can easily lead to sub-optimal results. I am not saying the authors should run the method from [30], but it would be nice to point out the current results are all on the variants of isotonic regression. So it is still interesting to ask if a strong distribution calibration approach can achieve similar performances.

==============After author feedback=========================================

Thanks for the feedback. 

I will keep current score of 7. I agree with some points from other reviewers, so there is still some room for improvement before I can give anything higher.

Regarding the details, I can see why (vtF8) have issues with the notations. The current notations and definition of calibration were also a bit confusing to me during the reviewing time (e.g. it uses a nested CDF in the expression). 

I have the same question regarding the computational costs as the reviewer (eKGZ). Although the authors did answer with some empirical running time, they didn't argue the fact that isotonic regression is O(N) for each iteration (O(MN) in total with M being the number of iterations). While I don't think this cost could be a deal-breaker in general, but it might require some minor modification with a really large dataset.

Regarding the method in [30], I agree it requires some modification to change the Gaussian mean embedding (kernel matrix) to any other general distributions, but it shouldn't be very hard to instead use Monte Carlo samples to approximate the kernel matrix. Disregarding the outcome of this paper, it would be nice if the authors can include some future experiments to show whether the calibration aim or the calibration method contributes more to the final results.
 Limitation: The authors discuss the limitations from a decision point of view.  It would be nice to further mention that post-hoc calibration is limited as it depends on a good uncalibrated model. That is, if a model simply outputs the marginal distribution of the target, this isn't much a calibration method can do.

Societal Impact: It is not discussed in section 7 as suggested. But have calibrated results for decision making should generally have a positive impact, such as seen in the ICU experiment.

 I am excited about reading the paper once the notation is updated. 

 Thank you for your response, they address my concerns.

 Thank you for your feedback and time reviewing the paper. 

$Q$: The notation is confusing.

We acknowledge an unfortunate typo in Definition 3-6 and Lemma 1 where we miswrote the capital $X$ (which traditionally denotes a random variable) as lower case $x$ (which traditionally denotes a value). For example, Eq.(1) in definition 3 should instead read

$$
P[h [X] (Y) \leq c \mid h [X] (y_{0}) \leq \alpha ] = c \quad \forall y_{0} \in \mathcal{Y}, \alpha \in [0, 1], \forall c \in [0, 1]. 
$$

In the proofs and the Appendix, our notation follows the correct convention. 
 
$Q$: Connection between true TCE and estimated TCE in algorithm

We will make the suggested edits. We would like to highlight that the convergence results (Theorem 2) also hold for the finite sample version of Algorithm 1 (which we described in Appendix A). Because Theorem 2 holds for any true distribution, it also holds for the empirical distribution (defined by uniform random sampling from the recalibration set). Therefore, the finite sample version of Algorithm 1 will at least terminate after $O(1/\epsilon^2)$ iterations and output a forecaster with threshold calibration error at most $\epsilon$ *on the empirical distribution*. 

Generalization is a separate question. Instead of proving theoretical results, we investigated generalization empirically because theoretical concentration bounds are usually loose and validation on held-out data is often the best way to evaluate test performance. 


 Thank you for your comments and suggestions on our paper.

$Q$: Full experimental results.

Due to space constraints, we moved the table with complete results to the Appendix (Tables 2-6 and Figures 5-8). 

$Q$: Additional related works.

We will add the following discussions in the related work section: (Haider et al, Goldstein et al, 2020) also study calibration and trust-worthy predictions in the medical domain, and the D-calibration they enforce is similar to average calibration (which is our baseline). A perfectly average calibrated prediction function is also D-calibrated, and vice versa. The major difference is that D-calibration measures calibration error in a way that’s tailored for the survival analysis task. 

(Hernandez-Orallo et al) also considers learning regression functions that minimize a decision loss. The main difference is that (Hernandez-Orallo et al) focuses on transforming the predicted CDF to a point prediction, while our method focuses on transforming the predicted CDF into a new CDF. (Hernandez-Orallo et al) also requires knowing the loss function to learn the transformation, while our method assumes that the loss function belongs to a commonly used function family (threshold loss functions). 


 Thank you for your comments and suggestions on our paper.

$Q$: Scalability of algorithm due to isotonic regression.

In our most expensive experiment, we find that each isotonic regression model takes 1 ms to fit on a single CPU. Since we fit 2 isotonic regression models per iteration and run our algorithm for 40 iterations, the total time cost of isotonic regression is 80 ms. We believe the computational burden is a minor part of the training-recalibration pipeline. For example, training the neural network predictor on the DHS Satellite Image dataset took 3 hours.


$Q$: Implementation details of distribution calibration baseline

Thank you for this suggestion. We will add the additional discussion on the choice of distribution calibration algorithms. Our implementation uses binning, while [30] fits a Gaussian process (which can be thought of as soft binning). 

The GP-Beta method from [30] is designed for Gaussian distribution predictions. In some of our experiments, our predictor can output more flexible Gaussian-Laplace distributions. The more flexible distribution family improves performance compared to using Gaussian distributions alone, but would require major modifications to the original methods in [30]. 


 Thank you for the detailed comments and suggestions. We appreciate your positive comments on the novelty and significance of our work.

$Q$: Detailed discussion of the running time.

Computation has not been a significant practical issue in our experiments. In our most expensive experiment, each iteration of our algorithm takes 17 seconds on a single CPU. We ran 40 iterations, taking a total of ~11 minutes . We believe the computational burden is a minor part of the training-recalibration pipeline. For example, training the neural network predictor on the DHS Satellite Image dataset took 3 hours. 
The reason for the low computation burden is because we only do grid search over a 2-dimensional space:  we only have to discretize $y_0^t$ and $\alpha^t$, and both are 1 dimensional real numbers. 

$Q$: Number of decision makers in the experiments.

You are correct that line 265 should read all $M$ decision makers. We use $M=2500$ in our experiments. We updated the draft accordingly.

$Q$: Discuss relationship with fairness.

Thank you for this suggestion. We will add the following discussion: 
The perfect predictor (that predicts the true conditional probability) satisfies our calibration definition. Consequently, if the perfect predictor satisfies some fairness notion (such as group calibration), then our calibration definition is also compatible with that fairness notion. Note that the perfect predictor does not satisfy a fairness notion called demographic parity, hence our calibration definition is not compatible with demographic parity either. 


This paper discusses calibrated regression from a decision-making perspective, by introducing the concept of 'threshold calibration'. The popular notion of calibrated regression, average calibration, is insufficient for decision-making. On the other hand, the notion of distribution calibration is sufficient for decision-making, but may be difficult to achieve. In practice, decisions based on regression predictions are often made on the basis of whether the prediction is smaller or larger than some threshold. Thus, a decision-oriented goal is to ensure calibration with respect to such a threshold, called threshold calibration. The paper proposes an algorithm to achieve threshold calibration.   An important question that has not been considered in the ML literature is how a calibrated regressor can be used meaningfully to make practical decisions. Thus, I believe this paper asks a question that is both highly relevant to the field and has not been studied before. In lines 41–45, the authors summarize these motivations well.

While I was excited about the motivation, the paper uses fundamentally flawed notation that is not in keeping with standard measure-theoretic conventions (more details in the following paragraphs). Due to this, I was unable to understand the technical content after Section 3.3 despite putting more effort than should be necessary to read a paper in a top ML conference. For these reasons, I recommend that the authors revise their paper to follow the standard conventions. (The authors themselves summarize these conventions correctly in lines 70-77, but do not follow them.)

The key Definition 3 does not parse. h\[x\](y_0) is not a random variable; it is just some fixed value as mentioned on line 75. Thus conditioning on h\[x\](y_0) \leq \alpha makes no sense. In particular, the definition does not capture “condition on predicted CDFs which have α-quantile equal to y0”.  At this point, as a reader, I am not sure what is random and what is not, and what is being conditioned on, and I find it quite difficult to follow the rest of the paper. Since I don’t understand Def 3, I do not understand threshold calibration error (TCE). In the same sense, Def 6 does not parse since h(x) = g is a statement that is either true or false; there is nothing random. While proposing Def 6, the present authors cite previous work that proposed it [30, Definition 3]. The mathematical exposition in [30] is precise as per standard measure theory conventions, and I suggest that the authors follow the same conventions to convey their ideas.

I tried skipping forward to Algorithm 1, but could not understand the main idea without understanding TCE. However I have some concerns regarding the presentation of Alg 1 and Thm 2, summarized in the following paragraph.

Algorithm 1 requires access to the true TCE and recalibration maps. In my assessment, the authors are not very transparent about this. This reliance on the true TCE is only briefly alluded to in lines 211-212, and is mentioned nowhere else in the main paper, as far as I can tell. One suggestion to improve transparency is to specify these oracles in the “input” line of the algorithm. Crucially, this lack of transparency makes Theorem 2 appear to be much stronger than it is. The claim of Theorem 2 is regarding the TCE of an algorithm that has access to the true TCE. In practice, when this TCE is estimated on a split of the training data (let us call it calibration data), it only shows that the calibration TCE will be smaller than \epsilon in 1/\epsilon^2 steps. Theorem 2 makes no claims regarding the generalization from calibration TCE to the true TCE, and no other comments are provided to this end either (in the main paper or the proof of Theorem 2). Line 201: “Our algorithm is an iterative procedure that terminates when the maximum TCE is less than a user specified threshold” implicitly suggests that the algorithm is reducing the true TCE, which is incorrect.

**Minor comments:**
- Lines 90, 127, etc: What is the little 'x' in the subscript? There is no specific feature x \in \Xcal mentioned anywhere else in Definition 1. 
- Line 172: “c percent of the times”: this is incorrect since c percent suggests 0.01c. 
- Line 373: "[10] A. Dawid. Present position and potential developments: some personal views. 1984." This is not a proper citation.  As mentioned in the previous section, the reliance of Algorithm 1 (and its guarantee) on the TCE oracle is a key limitation that was not discussed transparently. 

This paper focuses on the task of "calibrating" forecasts used in downstream decision-making tasks to ensure accurate estimation of the loss or "cost" of using each decision rule (i.e. the decision loss). The paper is specifically focused on forecasts of real-valued quantities (regression) and binary decisions (or actions) made by thresholding the forecast (for example, deciding whether to accept or reject a new patient based on the forecasted length of stay of a current patient being above or below some threshold).

In this setting, the paper shows that average calibration -- a common type of calibration -- does not ensure accurate estimation of the decision loss. The paper then introduces threshold calibration, proves that threshold-calibrated forecasts maximize the accuracy of decision loss estimates (and, moreover, that threshold-calibration is necessary and sufficient to maximize this accuracy), and provides an efficient algorithm to generate threshold-calibrated forecasts from a finite data sample. The paper validates the performance of their algorithm on real-world data (although with a synthetic downstream decision-making setup), and demonstrates more accurate decision loss estimates than those obtained by average and distribution calibration. The paper also derives the relationship between distribution, threshold, and average calibration in Proposition 1, which may be of independent interest.  # Key Strengths

**Novelty and significance.** The paper proposes novel notion of forecast calibration for the common setting of regression forecasts and binary threshold decisions. Several real-world decision-making tasks conform to this setting, such as evaluating loan applicants based on their risk/credit score (action = accept or deny loan based on the forecasted risk), planning hardware replacements in datacenters (action =. retain or replace the server based on the forecasted time to failure), and the medical and aid allocation settings described in the paper.

In each of these settings, decision-makers would benefit from accurate estimates of their costs/loss for each possible decision rule (i.e. the chosen threshold). The paper introduces "threshold calibration" and proves that this maximizes the accuracy of these decision loss estimates (in addition to being necessary and sufficient for this accuracy), provides a practical algorithm to threshold-calibrate forecasts, and empirically demonstrates more accurate decision loss estimates (compared to other types of calibration) on real-world data. The provided algorithm is inspired by work in multicalibration, but is novel as applied to this setting.

The paper also develops links between distribution, threshold, and average calibration in Proposition 1, which may be of independent interest to future research in this space.

**Links to related work.** The paper concisely and adequately surveys and connects with related work in this space.

**Clarity.** The paper provides intuition and examples for each theoretical result, and the paper overall is written clearly with well-organized sections and notation.

# Key Weaknesses

**Less-than-satisfactory discussion of the algorithm's running time in-practice.** While algorithm 1 involves $O(1/\epsilon^2)$ iterations for an error-rate of $\epsilon$, I believe each of these iterations involves significant computational work.

Finding $y_0^t$ and $\alpha^t$ in each iteration involves searching over a grid and evaluating the TCE integral for each pair of values. This grid search would involve testing each value of  $y_0^t$ in the label space, which could be continuous and unbounded, requiring some kind of ad-hoc discretization and restriction to an interval. The input space partitioning would scale linearly with the number of features, which can be large in many real world settings. Isotonic regression adds additional computational overhead.

While a consequence of this computational cost is more accurate decision loss estimates (as the paper empirically shows), I think the authors should make a note of the computational cost of achieving this accuracy relative to average calibration, so readers are aware of this tradeoff (actual running times of Algorithm 1 vs. average calibration in the experimental evaluation would be illuminating).

# Minor Concerns

**Section 5.3:** What is the value of $M$ used in the experiments? Should "all $k$ decision makers" at the end of this section read "all $M$ decision makers" instead? The authors have noted important issues when forecasts could influence the target of future predictions, and note that their proposed method helps decision makers evaluate pre-deployment decision losses (and not make optimal decisions). They also not that their scope is limited to binary thresholds and actions.

I believe this section is sufficiently comprehensive, but could be improved by noting potential [links between calibration and fairness](https://papers.nips.cc/paper/2017/file/b8b9c74ac526fffbeb2d39ab038d1cd7-Paper.pdf). Specifically, could there be a potential negative societal impact if threshold calibration and some common notion of fairness is incompatible (eg. the linked paper discusses this for average calibration and equalized odds)?

This paper proposes a method to include threshold calibration in decision making.  This calibration is required to ensure that decision loss is predicted accurately for threshold decisions.  The paper presents a problem where decision makers need to set thresholds in regression instances provided from forecasters. Optimising the values for the thresholds is important to reduce costs associated to bad decisions or bad forecasts.

The paper is well written, although the Figure 1 is hard to understand just reading the introduction. The authors motivate properly the problem, and formalise the framework based on two components, forecasters and decision makers. Following these two steps, they introduce the notion of Threshold Calibration to improve decisions of the makers based on the estimation of forecasters. Some theoretical properties of the framework are introduced.

Inspired by these properties, the authors propose a recalibration algorithm based on an iterative procedure that reduces the threshold calibration error (TCE).

Experiments over a set of real examples are valid to show  that their proposal can obtain good performance in different scenarios:  across a range of decision costs, across a range of decision thresholds, and considering simple and complex model families. A summary of the results are included in the paper, however, I missed a complete table with all the results considering the studied datasets. 

In general, I find the paper interesting since the propose a novel problem and introduce methods that can improve the cost reduction based on threshold configuration. Experimental results are useful to support the claims of the authors.

As points to be improved, I would extend the section of related works. For instance, the problem seems to be related to survival analysis where calibration has been studied:

Haider, Humza, et al. "Effective Ways to Build and Evaluate Individual Survival Distributions." J. Mach. Learn. Res. 21 (2020): 85-1.
Goldstein, Mark, et al. "X-CAL: Explicit Calibration for Survival Analysis." Advances in neural information processing systems 33 (2020): 18296.

Also, cost-sensitive regression is related:

Hernandez-Orallo, José. "Probabilistic reframing for cost-sensitive regression." ACM Transactions on Knowledge Discovery from Data (TKDD) 8.4 (2014): 1-55.
Song, Hao, et al. "Distribution calibration for regression." International Conference on Machine Learning. PMLR, 2019. Yes, correct.

Meta-review: There is a general agreement among the reviewers on the novelty and significance of the work. However, it is still clear based on the reviews that the presentation of the submitted paper could be further improved. That said, given the the satisfactory work that the authors have made during the discussion period, I am happy to suggest the acceptance of this paper and strongly encourage the authors to improve their paper based on the discussion with the reviewers. The major points to revisit are: i) polishing of the notation (as suggested by vtF8); ii) extension of the related work section (as suggested by  reviewer APiD); iii) clarifications on the scalability of the proposed algorithm and on the implementation of baselines (as suggested by reviewer VsY9). 

Decision: accept