Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
Authors: Shengjia Zhao
Michael P. Kim
Roshni Sahoo
Tengyu Ma
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: Calibration
decision making
uncertainty quantification
indistinguishability

Abstract: When facing uncertainty, decision-makers want predictions they can trust. A machine learning provider can convey confidence to decision-makers by guaranteeing their predictions are distribution calibrated--- amongst the inputs that receive a predicted vector of class probabilities q, the actual distribution over classes is given by q. For multi-class prediction problems, however, directly optimizing predictions under distribution calibration tends to be infeasible, requiring sample complexity that grows exponentially in the number of classes C. In this work, we introduce a new notion---decision calibration---that requires the predicted distribution and true distribution over classes to be ``indistinguishable'' to downstream decision-makers. This perspective gives a new characterization of distribution calibration: a predictor is distribution calibrated if and only if it is decision calibrated with respect to all decision-makers. Our main result shows that under a mild restriction, unlike distribution calibration, decision calibration is actually feasible. We design a recalibration algorithm that provably achieves decision calibration efficiently, provided that the decision-makers have a bounded number of actions (e.g., polynomial in C). We validate our recalibration algorithm empirically: compared to existing methods, decision calibration improves decision-making on skin lesion and ImageNet classification with modern neural network predictors.


Calibrating Predictions to Decisions: A Novel
Approach to Multi-Class Calibration
Shengjia Zhao Stanford University sj sjzhao@stanford.co edu
Michael P. Kim UC Berkeley
Roshni Sahoo Stanford University
mpkim@berkeley.com edu
rsahoo@stanf ord. edu
Tengyu Ma Stanford University
Stefano Ermon
Stanford University ermon@stanford. edu
tengyuma@s taat canfordd.eoooooooo edu
Abstract
When facing uncertainty, decision-makers want predictions they can trust. A ma- chine learning provider can convey confidence to decision- by guaranteeing their predictions distribution calibrated amongst the inputs that receive a predicted class probabilities vector q, the actual distribution over classes is For multi-class prediction problems, however, achieving distribution calibration tends to be infeasible, requiring sample complexity exponential in the number of classes this work, we introduce a new notion ddciisinn calibration that requires the predicted distribution and true distribution to be "indistinguishable" to set downstream decision-makers. When all possible decision makers are under consideration, decision calibration is the same distribution calibration. However, when we only consider decision makers choosing between a bounded number actions (e.g. polynomial in C), main result shows that decisions calibration becomes feasible we design a recalibration algorithm that requires sample com- plexity polynomial in the number of actions and the number of classes, We validate our recalibration algorithm empirically: compared to existing methods, decision calibration improves decision-male making skin lesion and ImageNet classification with modern neural network predictors.
Introduction
Machine learning predictions are increasingly employed by downstream decision makers who have little visibility how the models were designed and trained. In high-stakes settings, such as healthcare applications, decision makers want predictions they can trust. For example in healthcare, suppose machine learning service offers supervised learning model to healthcare providers that claims to predict the probability of various skin diseases, given an image lesion. Each healthcare provider want assurance that the model's predictions lead beneficial decisions, according their own loss functions. As a result, the healthcare providers may reasonably worry that the model was trained using loss function different than their This mismatch is often inevitable because the ML. service may provide the same prediction model to many healthcare providers, which may have different treatment options available and loss functions. Even the same healtheare provider could have different loss functions throughout time, due to changes in treatment availability.
If predicted probabilities perfectly equal the true probability of the event, this issue of trust would not arise because they would lead to optimal decision making regardless of the loss function or task considered by downstream decision makers. In practice, however, predicted probabilities never perfect. To address this, the healthcare providers may insist that the prediction function be
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

distribution calibrated, requiring that amongst the inputs that receive predicted class probability vectors q, the actual distribution over classes is q. This solves the trust issue because among the patients who receive prediction q. the healthcare providers knows that the true label distribution is q- and hence knows the true expected loss of a treatment on these patients. Unfortunately, to achieve distribution calibration, we need to reason about the set of individuals z who receive prediction g, for every possible predicted q. As the number of distinct predictions may naturally grow exponentially in the number of classes C, the amount of data needed to accurately certify distribution calibration tends to be prohibitive. Due to this statistical barrier, most work on calibrated multi-class predictions focuses obtaining relaxed variants of calibration. These include confidence calibration (Guo et al. 2017). which calibrates predictions only over the most likely class, and classwise calibration (Kull al. 2019), which calibrates predictions for each class marginally. While feasible, these notions are significantly weaker than distribution calibration and do not address the trust issue highlighted above. Is there calibration notion that addresses the issue of trust, but can also be verified and achieved efficiently? Our paper answers this question affirmatively.
Our Contributions. We introduce a new notion of calibration-ddecsiion calibration-wheer we take the perspective of potential decision-makers: the only differences in predictions that matter are those that could lead to different decisions. Inspired by Dwork et al. (202T), we formalize this intuition by requiring that predictions are "indistinguishable" from the true outcomes, according to a collection of decision-maken.
First, we show that prior notions of calibration can be characterized special cases of decision calibration under different collections of decision-makers. This framing explains the strengths and weakness of existing notions of calibration, and clarifies the guarantees they offer decision makers. For example, we show that a predictor is distribution calibrated and only if it is decision calibrated with respect all loss functions and decision rules. This characterization demonstrates why distribution calibration is so challenging: achieving distribution calibration requires simultaneously reasoning about all possible decision tasks.
The set of all decision rules include those that choose between exponentially (in number of classes C') many actions. In practice, however, decision-make typically choose from bounded (or slowly- growing as a function of C) set of actions. Our main contribution is an algorithm that guarantees decision calibration for such more realistic decision- In particular, we give sample-efficies algorithm that takes a pre-trate pre-trained predictor and post-processes it to achieve decision calibration with respect to all meaksion-makers choosing from bounded set of actions. Our recalibration procedure does not harm other common performance metrics, and actually improves accuracy and likelihood of the predictions. In fact, argue formally that, in the setting of bounded actions, optimizing for decision calibration recovers many of the benefits of distribution calibration, while drastically improving the sample complexity. Empirically, we use our algorithm to recalibrate deep network predictors on two large scale datasets: skin lesion classification (HAM10000) and Imagenet. Our recalibration algorithm improves decision making, and allow for more accurate decision loss estimation compared to existing recalibration methods even under distribution shift.
Background
2.1 Setup and Notation
We consider the prediction problem with random variables X and Y, where X E X' de- notes the input features, and Y E y denotes the label. We focus on classification where y (11,0,.. Q), 10, 0), ... (0, ..... 1)} where each y E y is one-hot vector with Ce N classes. D' A probability prediction function is a map p : X' + A where A is the C-dimensional simplex. We define the support of P the set of distributions it could predict, i.e. {p(x){ E X3. We us X + A to denote the true conditional probability vector, i.e. p" = E|y X x| AC and for all E [C], each coordinate gives the probability of the class p=()) Pr(Ya X =x]
'We can also equivalently define y' = 21, CC.. here we denote y by a one-hor vector for notation convenience when taking expectations.

---

2.2 Decision-Making Tasks and Loss Functions
We formalize a decision-maiee making task as a loss minimization problem. The decision- maker has some set of available actions A and loss function É y x A + R. In this paper we assume the loss function does not directly depend on the input features X. For notational simplicity we often refer to (action set, loss function) pair (A, E) only by the loss function É: the set of actions A is implicitly defined by the domain of We denote the set of possible loss functions as fall [LV set A and function & y A R).
We treat all action sets A with the same cardinality as the same set they are equivalent up to renaming the actions. A convenient way to think about this is that we only consider actions sets [K],... .. N, R} where [K] - [1,-.. K}.
A E [1], [2], ...
2.3 Bayes Decision-Making
Given some predicted probability x(X) on Y, a decision- selects an action in A. We assume that the decision-maker selects the action based the predicted probability. That is, define decision function as any map from the predicted probability to an action 5 : A + A.. and denote by A as the set of all decision functions with any of actions الله {6 V set A and function A A}.
Typically a decision meeker selects the action that minimizes the expected loss (under the predicted probability). This strategy is formalized by the following definition of Bayes decision-making. making.
Definition (Bayes Decision). Choose any i E Lall with corresponding action set A and prediction define the Bayes decision function as de(((x)) arg infeE. E Ex C C Can denote the set of all Bayes decision functions as & = 16r 14 E C3.
For any subset
3 Calibration: A Decision-Making Perspective
In our setup, the decision- maker outsources the prediction task to a chird-party forecaster, which returns a prediction function P (e.g., an ML Prediction API). The decision maker will use ý and a loss function E to make decisions. However, the forecaster does not necessarily know the loss function & in advance, and in more challenging cases, needs to serve multiple decision-makers with different loss functions. For example, the prediction function may be trained to optimize standard objective such as L2 error or log likelihood, then sold to -ocision-makers as an off-the-ste shelf solution.
In such a setting, the decision makers may be concerned that the off-the-shelf solution may not perform well according to their loss function. If the forecaster could predict optimally (i.e. p(X) = p (X) almost surely), then there would be no issue of trust; of course, perfect predictions are usually impossible, so the forecaster needs feasible ways of conveying trust the decision makers. To mitigate concerns about the performance of the prediction function, the forecaster might aim to offer performance guarantees applicable to decision makers whose loss functions come from class of losses L C Call-
3.1 Decision Calibration
First and foremost, a decision maker wants assurance that the Bayes decision rule dg with á gives low expected loss. Second, the decision maker wants to know how much loss is going to be incurred (before the actions are deployed and outcomes are revealed); the decision maker does not want to incur any additional loss in surprise that she has not prepared for.
To capture these desiderata we formalize a definition based on the following intuition: suppose decision maker with some loss function & considers a decision rule å E A (that may or may not be the Bayes decision rule), the decision maker should be able correctly compute the expected loss of using å to make decisions, as function of the predictions p.
Definition (Decision Calibration). For a set of loss functions C Lall and a set of decision rules
Aall we say that prediction p is (L: A---ecision calibrated (with respect to p² )") if L and 5 E A with the same action space A the computed loss (based on p) of 5 is the same as the actual
We require l and ó to have the same action space A for type check reasons, Eq. only has meaning if the loss and decision rule d are associated with the same action space A.
3

---

loss. i.e.
E ExE xx)xx))( ((((X))) (x
In particular, we say p is C-decision calibrated if it is (C; -c)-iiiiionnn calibrated, where De the set of all Bayes decision rules for loss functions in L.
The left hand side of Eq. is the "simulated" loss where the outcome Y is hypothetically drawn from the predicted distribution. The decision maker can compute this just by knowing the input features X and without knowing the outcome Y. The right hand side of Eq.(2) is the true loss that the decision maker incurs in reality if she uses the decision rule 6. Intuitively, the definition captures the idea that the losses in L and decision rules in & do not distinguish between outcomes sampled according to the predicted probability and the true probability; specifically, the definition can be viewed as an instantiation of the framework of Outcome Indistinguishaabiiiy (Dwork et al.) 2021).
As a cautionary remark, (2 should not be mis- interpreted as guarantees about individual decisions; (2) only looks at the average loss when X, Y is a random draw from the population. Individual guarantees are usually impossible without tools beyond machine learning (Zhao & Ermon 202T). In addition, Definition does not consider decision rules that can directly depend on X, as d E Aal only depends on X via the predicted probability p(X). Studying decision rules that can directly depend on X require tools such as multicalibration (Hébert-Johnson et al. 2018) which are beyond the scope of this paper (see related work).
In Definition 2 we also define the special notion of L-decision calibrated because given a set of loss functions 2, we are often only interested in the associated Bayes decision rules ág, i.e. the set of decision rules that are optimal under some loss function. For the rest of the paper we focus on C-decision calibration for simplicity. C-decision calibration can capture the desiderata we discussed above formalized in the following proposition.
Proposition If prediction function P is C-decision calibrated, then it satisfies 48 E bb
Ex (x)x((())) (x((()))) < Ex (KY, (((()))) (Bayes Decision Optimality) (Accurate loss estimation)
Ex Pxxe)-ex))0 (e(p(x)))) = ExlEy (x) (YY,
Bayes Decision Optimality states that the Bayes decision rule 8; is not worse than any alternative decision rule E &g- In other words, the decision maker is incentivized to take optimal actions according their true loss function. That is, using the predictions given by p. the decision maker cannot improve their actions by using a decision rule d' that arises from a different loss function l' E.
Accurate loss estimation states that for the Bayes decision rule òG, the simulated loss on the left hand side (which the decision maker can compute before the outcomes are revealed) equals the true loss on the right hand side. This ensures that the decision maker knows the expected loss that will be incurred over the distribution of individuals and can prepare for it. This is important because in most prediction tasks, the labels Y are revealed with a significant delay or never revealed. For example, the hospital might be unable to follow up on the
outcome of all of its patients.
In practice, the forecaster chooses some set L to achieve L-decision calibration, and advertise it to decision makers. A decision makers can then check whether their loss function & belongs to the advertised set L. If it does, the decision maker should be confident that the Bayes decision rule de has low loss compared to alternatives in &c. and they can know in advance the loss that will be incurred.
3.2 Decision Calibration Generalizes Existing Notions of Calibration
We show that by varying the choice of loss class C, decision calibration can actually express prior notions of calibration. For example, consider confidence calibration, where among the samples whose the top probability is B, the top accuracy indeed B. Formally, confidence calibration requires that
Pr|Y = arg max ((X) max p(X) B] = -8.
We show that prediction function is confidence calibrated if and only it C, decision calibration, where E, is defined by
+ 8 1(a y1 [0. C, (e(y,a) 1(y + a a # 1)+

---

Existing Calibration Definitions
Associated Loss Functions
Confidence Calibration (Guo et al.
2017)
:= [, a: 3] A [0
Pr(Y arg max ((X) max P X 3
where fly, a; B) == 1ly #aaa # +)+ 1a0---aaa1 1(==))
v3 E [0,
a [E)u(L]
Lor : (lely, 31, 32 v91 32 E R [
Classwise Calibration Kull et al. 2019)
where fly, a: B1 82,
I(a = -) 61 1(y = T)
E[Ye| px( = 3] = 5, Vc E VB E [0, 1]
+52 1(y fca== F), a a (T.F.1)
Distribution Calibration (Kull & Flach 2015)
{ set A and function e: x A - R) Call
E[y| ((X) q] = q. Vq
A
Table prediction function P satisfies the calibration definitions on the left if and only if it satisfies f-decision calibration for the loss function families on the right (Theorem T
Intuitively, loss functions in C, corresponds the refrained prediction task: a decision maker chooses between reporting a class label, or reporting "I don't 't know." denoted J.. She incurs a loss of 0 for correctly predicting the label y, loss of 1 for reporting an incorrect class label, and a loss of 3 < 1 for reporting "I don't it know". If a decision maker's 's loss function belong to this simple class of losses f., he or she can use confidence calibrated prediction function p. because the two desiderata (Bayes decision optimality and accurate loss estimation) in Proposition [ are true for the decision maker. However, such "refrained prediction" decision tasks only account for a tiny subset of all possible tasks that are interesting to decision makers. Similarly, classwise calibration can be characterized through decision calibration using a class of loss functions that penalizes class-specific false positives and negatives.
In this way, decision calibration clarifies the implications of existing notions of calibration on decision making: relaxed notions of calibration correspond to decision calibration over restricted classes of losses. In general, decision calibration provides a unified view of most existing notions of calibration as the following theorem shows.
Theorem 1. [Decision Calibration Generalizes Existing Notions] For any true distribution p', and for the loss function sets Ler defined Table [
funnciooo p is
confidence calibrated iff it is de-desision calibrated.
classwise calibrated iff is dec-dicision calibrated.
distribution calibrated iff it is Lau-decision calibrated.
For proof of this theorem see Appendix C In Table [ confidence and classwise calibration are weak notions of calibration; correspondingly the loss function families E, and Lar are also very restricted.
On the other hand, distribution calibration (i.e. E[Y p(X) = <l q. Vg E A ) is equivalent to dan-decision calibration. This means that distribution calibrated predictor guarantees the Bayes decision optimality and accurate loss estimation properties as in Proposition to a decision maker holding any loss functions. Unfortunately, distribution calibration is very challenging to verify or achieve. To understand the challenges, consider certifying whether a given predictor ş is distribution calibrated. Because
need to reason about the conditional distribution FlY p(() y) for every that P can predict (i.e. the support of p). the necessary sample complexity grows linearly in the support of p. Of course, for a trivial predictors that map all inputs the same prediction go (i.e. p(x) = Vr E X)) distribution calibration is easy to certify (Widmann et al. 2019). but such predictors have no practical use.
Our characterization of distribution calibration further sheds light on why it is so difficult to achieve.
Lan consists of all loss function, including all loss functions é ) x A R whose action space A contains exponentially many elements (e.g. |A| 2c). The corresponding decision rules 8 E &C... A + A may also map AC exponentially many possible values. Enforcing Definition for such complex loss functions and decision rules is naturally difficult.
2
3.3 Decision Calibration over Bounded Action Space
In many contexts, directly optimizing for distribution calibration may be overkill. In particular, in most realistic settings, decision makers tend to have bounded number of possible actions, so the relevant losses come from C for reasonable K E N. Thus, we consider obtaining decision calibration for all loss functions defined over bounded number of actions K. In the remainder of the
5

---

paper, we focus on this restriction of decision calibration to the class of losses with bounded action space; we reiterate the definition of decision calibration for the special case of C*
Definition 3 (C -Decision Calibration). Let LK be the set of all loss functions with K actions _ {e) VA, |A| = K, ve: y A + R), we say that a prediction p L -decision calibrated (with respect to p" if E LK and ö E A
F Fx y I0 ((0(x))) =Ex (
The key result (that we show in Section 4) that C -decision calibration can be efficiently verified and achieved for all predictors. Specifically, we show that the sample complexity necessary to learn f -eccssion calibrated predictors depends on the number of actions K, not on the support of p. This is in contrast to the standard approach for establishing distribution calibration, where the sample complexity scales with the support of ý (which typically exponentially in the number of classes).
Achieving Decision Calibration with PAC Guarantees
4.1 Approximate CK Decision Calibration is Verifiable and Achievable
Decision calibration in Definition ussually cannot be achieved perfectly. The definition has to be relaxed to allow for statistical and numerical errors. To meaningfully define approximate calibration we assume that the loss functions are bounded, i.e. no outcome y E y and action a A can incur an infinite loss. In particular, we bound E by its 2-norm maxa]]((;; a))l2 == maxa v /Vvey {yy a²²
Now we can proceed to define approximate decision calibration. In particular, we compare the difference between the two sides in Eq. (2) of Definition with the maximum magnitude of the loss function.
Definition (Approximate decision calibration). prediction P is ( e-ddccssion calibrated (with respect to p²) if E L and ō E d
(((())((())) E[((Y) S supple(( a2
(3)
.EAA
Definition is relaxation of Definition 2 if prediction function is (C, -)deiiion calibrated, then
is L-decision calibrated (Definition) 2
The main observation in paper is that for the set of loss functions with K actions, (C* -)-decision calibration can be verified and achieved with polynomially many samples. addition, we achieve decision calibration without deteriorating the L2 error E[lp(XX- YI
Theorem (Main Theorem, informal). There is an algorithm, such that for any predictor p and given polynomially (in K, C/1/c many samples, can with high probability
correctly decide if p satisfies (LK, ,e)-dcissoon calibration
2. output a new predictor ÿ' that satisfies (LK, -e-decision calibration without degrading the original predictions (in terms of L2 error).
Note that trivial predictors p(x) = E[Y], Va satisfy ())ddddciiiiinn calibration. To maintain meaningful prediction function we also require "sharpness" which we measure by Lg error. We guarantee that the L2 етror of g' can only improve under post-processing: that is, (x] Y < E[llØ(X) - Y|]]] In fact, the algorithm works by iteratively updating the predictions to make progress in L2. In addition to Lg error, empirically our recalibration algorithm also slightly improves the likelihood and accuracy. For rest of this section, we propose concrete algorithms that satisfy
Theorem 2
4.2 Verification of Decision Calibration
This section focuses on the first part of Theorem where we certify (CK c)-decision calibration. naive approach use samples to directly estimate (by replacing all expectations with empirical sample
'The choice of 2-norm is for convenience. All p-norms are equivalent up to multiplicative factor polynomial in C. main theorem (Theorem 2 still hold for any p-norms up the polynomial factor.

---

averages)
sup sup EL(((, -140X / (supll((. BEACK RECK
AaEA (
(4)
and compare it with c. Even though might be possible upper bound the estimation error for Eq.44) directly, the analysis would be complex because of the multiple sup in the equation. To gain deeper insight into the problem and simplify analysis, we will make several observations to transform this complex optimization problem to simpler problem that resembles linear classification.
Observation
The first observation that we do have to take the supremum over C because for any choice of & E & by simple calculations we have
E ((088)(6())) Ele(Y, - (((())
E (((0)-))(66())))
sup
sup&cA ((-,0)12 ((),))l((
EECK
This statement is formally proved part of the proof for Proposition 3 Appendix C.2 Intuitively on the right hand side, å partitions the probabilities A based on the optimal decision A := {q e
A A (()(g) 1))..... "AK {q E A ((((g) = K)}. For each partition AR measure the difference between predicted label and true label average, i.e. E[(Y ())(((X) As)].
We observe that the partitionns of are defined by linear classification boundaries. Formally, introduce new notation for the linear multi-class classification functions
Observation 2.
Bk Veu RKXC]
{
where b (g. a) I(a arg sup (q. ww))
(6)
d'E(k]
Note that this new classification task is tool to aid in understanding decision calibration, and bears no relationship with the original prediction task (predicting Y from X).. Intuitively ws defines the weights of a linear classifier; given input features E A and candidate class a, bw outputs an and otherwise.
indicator: (g, a) if the optimal decision of q is equal
The following equality draws the connection between (5) and linear classification. The proof is simply translation from the original notations to the new notations.
K
[[( Y))(((((X))) = =a)] sup BEBK
sup
SEAg
The final outcome of our derivations is the following proposition (Proof in Appendix C.2
Proposition 2. A predictor P satisfies (C ,))-dciision calibration if and only if
K
sup BEBK
Σ
E(( Y)6((()..)) Se
(8)
In words, á satisfies decision calibration and only if there no linear classification function that can partition AC into parts, such that the average difference Ý (or equivalently ((X) - p" (X)) in
each partition is large. Theorem 2 follows naturally because BK has low Radamacher complexity, so the left hand side of Eq. can be accurately estimated with polynomially many samples. For formal statement and proof Appendix C.2
The remaining problem is computation. With unlimited compute, we can upper bound Eq. (8) by brute force search over =K; in practice, we surrogate objective optimizable with gradient descent.
This is the topic of Section 4.4
4.3 Recalibration Algorithm
This section discusses the second part of Theorem where we design post-processing recalibration algorithm. The algorithm based on the following intuition, inspired by (Héhert-Johnson al.) 2018): given a predictor we find the worst b BK that violates (8. (line of Algorithm I then,

---

we make an update to f to minimize the violation of Eq.8) for the worst b (line 4,5 of Algorithm 1 This process is be repeated until we get a (B* -e-diiiio calibrated prediction (line 2). The sketch of the algorithm is shown in Algorithm 1 and the detailed algorithm is in the Appendix.
Algorithm 1: Recalibration algorithm to achieve L decision calibration.
Input current prediction function p. tolerance c. Initialize p(0) p;
2 for 1, 1,2, . until output g(T) when it satisftes Egg. 8) do
Find b E B that maximizes Ea=1 ||[(Y - p(r (bb (x)))]]
Compute the adjustments d, =(( (X)) (X),
Set p()) x + (x) + >m= ((( a) da (projecting onto [0, 1] if necessary);
end
Given dataset with N samples, the expectations in Algorithm 1 are replaced with empirical averages. The following theorem demonstrates that Algorithm Satisfies the conditions stated in Theorem Theorem 2.2. Given any input P and tolerance 6, Algorithum |
terminates in O(K/E²) iterations. For any X given (poly((K, C, log(1/8), A)) samples, with ó probability Algorithm outputs (CK ----dciision calibrated prediction function p' that satisfies E E|ll Y ElllØ(X)
|
The theorem demonstrates that if we can solve the inner search problem over BK then we can obtain decision calibrated predictions efficiently in samples and computation.
4.4 Relaxation of Decision Calibration for Computational Efficiency
We complete the discussion by addressing the open computational question. Directly optimizing over B is difficult, so we instead define the softmax relaxation.
ela.wa)
B &bw Veu E RRXC)
where bu a) Ene e(g,w.r)
The key motivation behind this relaxation that by E BK now differentiable in w, so we can optimize over BK using gradient descent. Correspondingly some technical details in Algorithm change to accommodate soft partitions; we address these modifications in Appendix A and show that after these modifications Theorem 2.2 still holds. Intuitively, the main reason that can still achieve decision calibration with softmax relaxation is because B* a subset of the closure of BK Therefore, compared to Eq. (8) we enforce slightly stronger condition with the softmax relaxation. This can be formalized in the following proposition.
Proposition 3. A prediction function p is (C .) -)-deissoo calibrated if it satisfies
K
EI( - ())(-)))(())))))/ Se:
sup BE
(9)
We remark that unlike Proposition 2 which is "if and only if" statement), Proposition is "if" statement. This is because Eq. implies (q. but not vice versa.
Empirical Evaluation
5.1 Skin Legion Classification
This experiment materializes motivating example in the introduction. We aim show real medical prediction dataset, our recalibration algorithm improves both the decision loss and reduces the decision loss estimation error. For the estimation error, as in Definition for any loss function č and corresponding Bayes decision rule &, we measure
loss gap = E[e(Ý, 8e(((X))) E[A(Y, ( supll((;,a)ls aEA
(10)
In addition to the loss function in Figure (which is motivated by medical domain knowledge), we also consider a set of 500 random loss functions where for each E )/aa E A, f(y, a) ~ Normal 1),
and report both the average loss gap and the maximum loss gap across the loss functions.

---

Predicted Loss True Loss
After recalibration
Function
10
predicted loss true_los
12.5
predicted loss
10.0
treatment
7.5-
treatment
3.0
akiec
bcc
bid
df
TV
VOSC
mel
2.5
-2.5
0.0
2.5
5.0
7.5 10.0 0.30
0.15
0.20
0.25
0.30
0:35
0:40
0.30
0.15
0.20
0.25
0.30
0:35
0.40
Figure 1: An illustrative example of accurate loss estimation on the HAM10000 dataset. The predictor predicts the probability of 7 skin illness categories (akiec, bcc, ..... mel); the hospital decides between two actions (treatment vs. no treatment). Left An example loss function (details not important). Blue indicates low loss and red indicates high loss. For the malignant conditions such as 'akiec' and 'mel', no treatment has high loss (red); for the benign conditions such as "nv', (unnecessary) treatment has moderate loss. Middle Histogram of predicted loss vs. the true loss. The predicted loss is the loss the hospital expected to incur assuming the predicted probabilities are correct under the Bayes decision rule.The true loss is the loss the hospital actually incurs (after ground-truth outcomes are revealed). We plot the histogram from random train/test splits of the dataset, and observe that the true loss is generally much greater than the predicted loss. Because the hospital might incur loss it didn't 't expect or prepare for, the hospital cannot trust the prediction function. Right After applying our recalibration algorithm, the true loss almost matches the predicted loss.
0:15
worst-decision-tett
decision-val decision-test
torst-dirfhhletttest
0.73
0.10
worst-dacisioonvall
decossion
035
0.05
averoge decision test
-0.74
average dirichlet-test
-0.75
0.00
average deciisioonl
15
20
recelibrelee etion Steps
ret recallir ation steps
Figure 2: Calibration improves decision loss and its estimation on the HAM10000 skin legion classification dataset. Left The gap between the predicted decision loss and the true decision loss in (T)) on a set of randomly sampled loss functions. We plot both the average gap (orange) and the worst gap (blue) out of all the loss functions. The dotted lines are the validation set performance, and solid lines are the test set performance. With more recalibration steps in Algorithm[] [ both the average gap and the worst gap improves. The improvements are greater than Dirichlet recalibration (dashed line) Right The decision loss (averaged across all random loss functions). With more recalibration steps the decision loss also improves slightly. Dirichlet calibration (dash-dot line) also improves the loss gap for this dataset, but not as much as decision calibration. Note that Dirichlet calibration only has be applied once, so its performance is constant instead of a function of the recalibration steps.
Setup We use the HAM10000 dataset (Tschandl et al. 2018). We partition the dataset into orain/validationntesst sets, where approximately 15% of the data used for validation, while 10% are used for the test set. We use the train set to learn the baseline classifier f., validation set to recalibrate, and the test set to measure final performance. For modeling we use the densenet-121 architecture (Huang et al. 2017). which achieves around 90% accuracy.
For our method we use Algorithm 2 in Appendix A (which is a small extension of Algorithm I explained in Section 4.4) We compare with temperature scaling (Guo et al.) (017)) and Dirichlet calibration (Kull al. 2019). In this experiment, we first apply temperature scaling then apply other methods (decision calibration or Dirichlet calibration). The temperature scaling baseline corresponds to 0 decision recalibration steps in Figure
Results The results are shown in Figure 2 For these experiments we set the number of actions K = 3. For other choices we obtain qualitatively similar results in Appendix [ The main observation is that decision recalibration improves the loss gap in Eq.(TO) and slightly improves the decision loss. We also observe that our recalibration algorithm slightly improves top-1 I accuracy (the average improvement is 0.40 + 0.08%) and L2 loss (the average decrease is 0.010 + 0.001) on the test set.
5.2 Imagenet Classification
We stress test our algorithm on Imagenet. The aim is to show that even with deeply optimized classifiers (such as inception-v3 and resnet) that are tailor made for the Imagenet benchmark and a large number of classes, recalibration algorithm can improve the loss gap in (I.

---

ciasse5-10
classcs-50
dlassos-200
classes-1000
850t1i 1o0s 060
2.66
decision loon gho
0.00
0.00
800
beseline: Is-only
bascline: original
loos
0.00
deciiion loos
bazaline: bs-drichle:
000-
bascline: dirchlet
curs: ts-Decision
ours: decision
0.00
200
040
150 159
123
250
30
130
150
100
155
number recel www.iiaaiiooo steps
number of recalfbration Steps
number of recelibration steps
number
lecalite alion steps
Figure 3: Calibration improves decision loss estimation gap on Imagenet- 1000 with inception- v3 on the test set. The meaning of the plot is identical to Figure From left to right vary the number of classification categories C' from 10 to 1000. Our algorithm reduces the gap between predicted loss and true loss in Eq. T0 even up to 1000 classes, although more classes require more iterations and are more prone to over-fitting. Dirichlet calibration is less scalable than our method, and has worse loss estimation with more classes. Using temperature scaling (ts) in additional decision calibration drastically reduces the number of iterations required for decision calibration to converge.
Setup The setup and baselines are identical to the HAM10000 experiment with two differences: use pretrained models provided by pytorch, and among the 50000 standard validation samples, we use 40000 samples for recalibration and 10000 samples for testing. Similar the previous experiment, we randomly generate set of 500 loss functions from normal distributions.
We additional test the performance of decision calibration under distribution drift the Imagenet- benchmark (Hendrycks & Dietterich 2019). We partition the Imagenet validation data into recalibration set and a test set. We train decision calibration on the recalibration set (with no corruption), and measure performance on the test set (with corruption).
The results are shown in Figure with additional plots in Appendix B Decision calibration can generalize larger number of classes, and still provides some (albeit smaller) benefits with 1000 classes. Recalibration does not hurt accuracy and 1.2 error, as we observe that both improve by modest amount (on average by 0.30% and 00307 respectively). We contrast decision calibration with Dirichlet calibration (Kull et al. 2019). Dirichlet calibration also reduces the loss gap when the number of classes is small (e.g. 10 classes), but is less scaleble than decision recalibration. With more classes its performance degrades much more than decision calibration. In the appendix Figure 6 and Figure [7] we further show that even under distribution drift (on the Imagenet-C) benchmark, decision calibration can improve both decision loss and decision loss estimation.
Results
6 Related Work
Calibration
Early calibration research foeus binary classification BBrier 1950
Dawid 1984).
For multiclass classification, the strongest definition is distribution (strong) calibration Kull & Flach 2015 (Sog al.) 2019) but hindered by sample complexity. Weaker notions such as confidence (weak) calibration (Platt et al. 1999; Guo et al., 2017). class-wise calibration (Kull et al. 2019) or average calibration average calibration (Kuleshov al. 2018) are more commonly used in practice.
To unify these notions, [Widmann et al., 2019) proposes F-calibration but lacks detailed guidance on which notions use.
Individual calibration Our paper discusses the average decision loss over the population X. A stronger requirement is to guarantee the loss for each individual decision. Usually individual guarantees are near-impossible (Barber et al. 2019 and are only achievable with hedging (Zhao & Ermon 202T) or randomization (Zhao et al.) 2020).
Multi-calibration and Outcome Indistinguishabiliy. Calibration have been the focus of many works on fairness, starting with (Kleinberg et al. 2016 Pleiss et al. 2017). Multi-calibrant emerged as a noteworthy notion of faimess (Hébert-
has et al. 2018 Kim et al. 2019 Dwork et al.) 2019 Shabat et al., 2020 Jung et al., 2020 Dwork et al.) 2021 because goes beyond "protected" groups, and guarantees cal for any group that is identifialle within some computational bound. Recently, (Dwork et al. 2021) generalizes multicalibration to outcome indistinguishability (OI).
Decision calibration is special form of OI.
10

---

7 Acknowledgements
SZ is supported in part by a JP Morgan fellowship and a Qualcomm innovation fellowship. MPK is supported by the Miller Institute for Basic Research in Science and the Simons Collaboration on the Theory of Algorithmic Faimess. TM acknowledges support of Google Faculty Award and NSF IIS 2045685. SE acknowledges support by NSF(#1651 #1522054, #1 #1733686), ONR (N000141912145), AFOSR (FA95501910024), ARO
and Sloan Fellowship.
References
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-frr free conditional predictive inference. arXiv preprint arXiv: 09903,46844, 2019.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach. Journal of the Royal Statistical Society: Series (General), 147(2): 278-290, 1984.
Cynthia Dwork, Michael Kim, Omer Reingold, Guy N Rothblum, and Gal Yona. Learning from outcomes: Evidence-baut rankings. In 2019 IEEE 60th Annual Symposium Foundations of Computer Science (FOCS), pp. 106-125. IEEE, 2019.
Cynthia Dwork, Michael P Kim, Omer Reingold, Guy N Rothblum, and Gal Yona. Outcome indistinguishability. STOC, 2021.
Yuguang Fang, Kenneth A Loparo, and Xiangbo Feng. Inequalities for the trace of matrix product. IEEE Transactions on Automatic Control, 39(12):2499-2490 1994.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cali- bration for the coomputationall-identiiaan masses. In International Conference on Machine Learning, pp. 1939 1948. PMLR, 2018
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 122611 2019.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4700-4708, 2017.
Christopher Jung, Changhwa Lee, Mallesh M Pai, Aaron Roth, and Rakesh Vohra. Moment multical- ibration for uncertainty estimation. arXiv preprint arXiv:2008.08057 2020.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for faimnss in classification. In Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pp. 247-254, 2019.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determi- nation of risk scores. arXiv preprint arXiv:1609.0580) 2016.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pp. 2796-2268888 -804. PMLR, 2018.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor calibration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 68-85. Springer, 2015.

---

Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Hao Song, Peter Flach, et al. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with dirichlet calibration. arXiv preprint arXiv: /2256,,,,0,, 2019.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems, pp. 5680-5689, 2017.
Eliran Shabat, Lee Cohen, and Yishay Mansour. Sample complexity of uniform convergence for multicalibration. NeurIPS, 2020.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference Machine Learning, pp- 5897-5906. PMLR, 2019.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. arXiv preprint 1910v 11385, 2019.
Shengjia Zhao and Stefano Ermon. Right decisions from wrong predictions: A mechanism design alternative individual calibration. In International Conference on Artificial Intelligence and Statistics, pp. 2683-2699 PMLR, 2021.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. arXiv preprint arXiv:2006. 2020.
12


Reviews:
[{"review_id": "zXqRPX57C2a", "reviewer": "Reviewer_JJNJ", "summary": "This paper operates within the context of uncertainty quantification in classification. Naive generalization of the notion of a calibrated predictor from the binary to the multiclass setting is extremely prohibitive in terms of sample complexity (with respect to number of classes for the problem at hand). The authors introduce a novel notion --- decision calibration --- and analyze it theoretically and empirically on a set of prediction tasks. ", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is hard to follow, and insufficient clarity makes it hard to evaluate its significance. Findings seem to be questionable. Writing can also be improved in several directions:\n1. Equations are parts of sentences, so punctuation is needed (and is missing currently).\n2. Centering, commas etc. (e.g., see equation 5).\n3. Line 94 --> missing fullstop.\nUnfortunately, it seems to me that the paper is not well-suited for this conference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Connections to related works in the field should be discussed in more detail.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "mg-4-b9wXET", "reviewer": "Reviewer_TRQr", "summary": "The paper defines a new notion of decision calibration and demonstrates that approximate $\\mathcal{L}^K$ decision calibration is verifiable and achievable in practice, providing a practical algorithm for it. This is a result with practical calibration guarantees in multi-class calibration, which to my knowledge is the first achievement of this kind. The proposed method outperforms the existing calibration methods of temperature scaling and Dirichlet calibration in experiments on HAM10000 and ImageNet datasets and multiple architectures.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and with very high significance for the field of classifier calibration. The paper demonstrates that taking the perspective of all decision-makers is a very useful one. I have the following comments, suggestions and questions.\n\nMajor:\n\nNo evidence provided for the claim 'We observe that all recalibration methods (including ours) work better if we first apply temperature scaling, hence we first apply it in all experiments.' (p.8,l.290). I would like to see results with and without temperature scaling. Given that the theoretical guarantees apply for the proposed method both with and without temperature scaling, how come there is a significant difference? Is it that the loss is comparable on validation (recalibration) data but different on test data?\n\nIt is not clear from the paper whether the experiments have been performed with a single train/validation/test split or there are more replicates with different random seeds. If there is a single split, then how are the error bars obtained in Figure 2 and lines 297-298 (p.8)?\n\nThe checklist answers '[Yes] See Appendix' for the question 'Did you include the total amount of compute and the type of resources used', but I did not find any information on this in the Appendix.\n\nThe main intuition behind Algorithm 1 has not been explained in sufficient detail: find the worst-case decision task, group by train-optimal action, shift the predictions within each group to be calibrated for that group on average, iterate. Currently it involves a lot of mathematical work to read it out from the notation.\n\nMinor:\n\nI suggest that the authors could offer a name for the notion of $\\mathcal{L}^K$ decision calibration that would not require mathematical fonts, to be used in future research.\n\nAlthough perhaps obvious, the paper does not discuss the relationship of $\\mathcal{L}^K$ decision calibration with the existing notions of confidence calibration and classwise ca\nlibration.\n\nSome earlier works have used cost matrices as a way to define multi-class loss functions for finite action sets. I suggest to at least mention the term 'cost matrix' because it might provide useful insights for some readers.\n\np.3, l.92: notation of $\\Delta$ with an extra line on the left side (as if it were from mathbb font) is quite a rare choice of symbol, is this sufficiently justified?\n\np.4, Table 1: the associated loss functions are not unique, are they? In the sense that for each notion of calibration there exist infinitely many associated loss functions, e.g. with a different choice of a multiplicative factor but other transformations exist? I suggest to consider mentioning non-uniqueness somewhere in the paper. Does there exist a loss function associated to confidence calibration but not involving the action \"I don't know\"?\n\np.5, l.176: 'for a trivial predictors' -> 'for trivial predictors'\n\np.6, l.216: The L2 error is actually the Brier score, isn't it? I suggest to mention this. Could one use some other divergence measures in place of the L2 error here?\n\nThroughout the paper: 'Radamacher' -> 'Rademacher'\n\np.8, l.284: 'approximately 15% of the data are used for validation' - why 'approximately'? Is it just rounding to the closest integer or something else that makes it approximate?\n\np.8, l.296: 'in about 5 steps' - why 'about', are there more than 1 replicate experiments? Or is this 'about' referring to the results in Appendix with different numbers of actions?\n\np.9, l.319: 'average calibration average calibration'\n\np.10, l.342: 'A Philip Dawid' -> A=?\n\np.14, l.476: 'Prosition' -> 'Proposition'\n\np.15,16, l.509,510: I suggest adding $\\cdot$ in front of the identity function to separate it from the preceding symbol. Otherwise it might confuse some readers as the indicator function symbol looks a bit like a rectangle potentially modifying the previous symbol.\n\np.17, Eq.(12): one of the $\\hat{p}$ should actually be $p^*$.\n\n---\n\nAfter discussion: I am glad to keep to grade 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and social impact have been covered adequately.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ZS1OP373y7Q", "reviewer": "Reviewer_WEMe", "summary": "The paper looks at the problem of calibration for multi-class classification, with respect to downstream decision making.  The primary contribution of the paper is theoretical, providing a framework, algorithm, and analysis for decision calibration.  The decision calibration framework is intended to get around complexity issues for distribution calibration.  The authors provide some illustrative empirical evaluations to demonstrate the theoretical findings, using use cases in healthcare and image recognition.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For me, the main strength of the paper is the theoretical framing and analysis.  I appreciate the value added to the community through the discussion of weaknesses of distribution calibration.  The reframing of decision calibration is elegant and novel, and the resulting algorithm and theoretical analysis are a strong contribution.\n\nIn my view, the paper has some serious weaknesses in the empirical evaluation, and also in the overall framing in terms of being a practical approach.\n\nIn the theoretical evaluation, the method is applied and results are reported, but there are no baselines or ablations performed.  Indeed, the authors report that temperature scaling is used because it improves results, but there is no attempt to use temperature scaling on its own as a recalibration method or an ablation showing what happens when it is not applied.  Thus, the reader cannot judge whether the benefits shown are due to the recalibration method itself, or whether temperature scaling was responsible for the benefits, or whether the two together offer a unique value.  Similar comments can be made to other calibration methods, of which many appear in the literature, and at least some of which should have been tested as part of the empirical evaluation.\n\nThe focus on per-healthcare provider loss functions is maybe obscuring the point slightly – there are absolutely going to be different per-healthcare provider distributions.  While loss functions formally are defined with respect to a distribution, colloquially ML folks often focus more on the mathematical definition (such as squared loss vs. cross entropy loss) and less on the data distribution on which it is defined (such as patients admitted to Hospital X vs all people living within 5 miles of Hospital X).  This point becomes important on line 31, because the idea of matching empirical frequencies exactly only makes sense when talking about specific distributions.  Indeed, it is possible (and all too common) to have a predictor that matches empirical frequencies for training distribution D but does not match empirical frequencies on the deployment distribution D’.  \n\nPersonally, I feel that it is important to evaluate recalibration methods in the presence of distributional shift, especially when healthcare is a motivating example.  Some methods such as temperature scaling have been shown to be actively detrimental under realistic conditions of distributional shift.  I'm willing to accept that this may be out of scope for this short paper, but the paper would surely be improved if this were able to be addressed in some way.\n\nI believe that there should be at least some discussion of what happens when instead we treat k-class multiclass calibration as k-independent binary class problems, and apply binary-class calibration methods?  Why is this insufficient?\n\nThere’s a shift in notation in 2.1 compared to the intro.  Before, we we were talking about decision vectors q.  This q notation went away in the formal setup, which is a little confusing.\n\nIf I am reading correctly, this paper only focuses on multi-class single label vs. multi-class multi-label.  This should probably be called out explicitly (as the simplex would no longer be usable for multi-class multi-label).\n\nI believe that Skin Lesions is correct (rather than Skin Legions)\n\nIn the empirical evaluation, is it really a choice of 2 actions (treatment vs. no treatment), or rather a choice of at least 14 actions (treatment, no treatment) X (7 possible conditions)?\n\nI’m not sure what the reader is intended to gain from Figure 1 left, since there are no details given.\n\nFig 1 histograms – vertical axis scaling should be equal on both charts for accurate visual comparison\n\nFor the empirical evaluation, no other methods are tested.  How do we know that these improvements would not also be seen using other simple methods?\n\nThe related work section is extremely terse.\n\nSection 7 can just be cut to make space.  There’s no need for a conclusion section of this form in a short conference paper, unless it adds new value or insights.\n\n\n======\n\nUpdate after reading the author responses:\n\nI very much appreciate the clarifications, and in particular the additional empirical results exploring the question of robustness and showing strong performance there.  I will be increasing my score accordingly.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For me, the primary limitations of the paper are that there has not been an empirical evaluation of sufficient depth to uncover any potential lack of robustness.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was not complete consensus on this paper, two reviewers found the work to be valuable -- especially after the author responses, that included clarifications and the additional empirical results to address key concerns raised around ablations and robustness.  Additionally, I found that the most negative review was also one that provided almost no detail or reasoning, which leads me to discount this review significantly.

I will quote reviewer TRQr's private response within discussion amongst reviewers as a useful summary of the paper's strengths:

"I think that the authors have done a very good job answering all concerns raised by all reviewers and have even run very relevant additional experiments. I am definitely supporting acceptance of this paper because it fills the conceptual gap between confidence calibration and classwise calibration on one side and distribution calibration on the other side. It is not just some arbitrary mathematical generalization, it is exactly the right one in my opinion - taking the perspective of the decision maker is the best thing one can do, because that is what calibration is usually meant to serve."

I do expect that the authors will include the additional results in the final paper, and that they will use the reviewer comments to further strengthen the paper.
Decision: Accept (Poster)