Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper looks at the problem of calibration for multi-class classification, with respect to downstream decision making.  The primary contribution of the paper is theoretical, providing a framework, algorithm, and analysis for decision calibration.  The decision calibration framework is intended to get around complexity issues for distribution calibration.  The authors provide some illustrative empirical evaluations to demonstrate the theoretical findings, using use cases in healthcare and image recognition.
main_review: For me, the main strength of the paper is the theoretical framing and analysis.  I appreciate the value added to the community through the discussion of weaknesses of distribution calibration.  The reframing of decision calibration is elegant and novel, and the resulting algorithm and theoretical analysis are a strong contribution.

In my view, the paper has some serious weaknesses in the empirical evaluation, and also in the overall framing in terms of being a practical approach.

In the theoretical evaluation, the method is applied and results are reported, but there are no baselines or ablations performed.  Indeed, the authors report that temperature scaling is used because it improves results, but there is no attempt to use temperature scaling on its own as a recalibration method or an ablation showing what happens when it is not applied.  Thus, the reader cannot judge whether the benefits shown are due to the recalibration method itself, or whether temperature scaling was responsible for the benefits, or whether the two together offer a unique value.  Similar comments can be made to other calibration methods, of which many appear in the literature, and at least some of which should have been tested as part of the empirical evaluation.

The focus on per-healthcare provider loss functions is maybe obscuring the point slightly – there are absolutely going to be different per-healthcare provider distributions.  While loss functions formally are defined with respect to a distribution, colloquially ML folks often focus more on the mathematical definition (such as squared loss vs. cross entropy loss) and less on the data distribution on which it is defined (such as patients admitted to Hospital X vs all people living within 5 miles of Hospital X).  This point becomes important on line 31, because the idea of matching empirical frequencies exactly only makes sense when talking about specific distributions.  Indeed, it is possible (and all too common) to have a predictor that matches empirical frequencies for training distribution D but does not match empirical frequencies on the deployment distribution D’.  

Personally, I feel that it is important to evaluate recalibration methods in the presence of distributional shift, especially when healthcare is a motivating example.  Some methods such as temperature scaling have been shown to be actively detrimental under realistic conditions of distributional shift.  I'm willing to accept that this may be out of scope for this short paper, but the paper would surely be improved if this were able to be addressed in some way.

I believe that there should be at least some discussion of what happens when instead we treat k-class multiclass calibration as k-independent binary class problems, and apply binary-class calibration methods?  Why is this insufficient?

There’s a shift in notation in 2.1 compared to the intro.  Before, we we were talking about decision vectors q.  This q notation went away in the formal setup, which is a little confusing.

If I am reading correctly, this paper only focuses on multi-class single label vs. multi-class multi-label.  This should probably be called out explicitly (as the simplex would no longer be usable for multi-class multi-label).

I believe that Skin Lesions is correct (rather than Skin Legions)

In the empirical evaluation, is it really a choice of 2 actions (treatment vs. no treatment), or rather a choice of at least 14 actions (treatment, no treatment) X (7 possible conditions)?

I’m not sure what the reader is intended to gain from Figure 1 left, since there are no details given.

Fig 1 histograms – vertical axis scaling should be equal on both charts for accurate visual comparison

For the empirical evaluation, no other methods are tested.  How do we know that these improvements would not also be seen using other simple methods?

The related work section is extremely terse.

Section 7 can just be cut to make space.  There’s no need for a conclusion section of this form in a short conference paper, unless it adds new value or insights.


======

Update after reading the author responses:

I very much appreciate the clarifications, and in particular the additional empirical results exploring the question of robustness and showing strong performance there.  I will be increasing my score accordingly.



limitations_and_societal_impact: For me, the primary limitations of the paper are that there has not been an empirical evaluation of sufficient depth to uncover any potential lack of robustness.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper operates within the context of uncertainty quantification in classification. Naive generalization of the notion of a calibrated predictor from the binary to the multiclass setting is extremely prohibitive in terms of sample complexity (with respect to number of classes for the problem at hand). The authors introduce a novel notion --- decision calibration --- and analyze it theoretically and empirically on a set of prediction tasks. 
main_review: The paper is hard to follow, and insufficient clarity makes it hard to evaluate its significance. Findings seem to be questionable. Writing can also be improved in several directions:
1. Equations are parts of sentences, so punctuation is needed (and is missing currently).
2. Centering, commas etc. (e.g., see equation 5).
3. Line 94 --> missing fullstop.
Unfortunately, it seems to me that the paper is not well-suited for this conference. 
limitations_and_societal_impact: Connections to related works in the field should be discussed in more detail.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper defines a new notion of decision calibration and demonstrates that approximate $\mathcal{L}^K$ decision calibration is verifiable and achievable in practice, providing a practical algorithm for it. This is a result with practical calibration guarantees in multi-class calibration, which to my knowledge is the first achievement of this kind. The proposed method outperforms the existing calibration methods of temperature scaling and Dirichlet calibration in experiments on HAM10000 and ImageNet datasets and multiple architectures.
main_review: The paper is very well written and with very high significance for the field of classifier calibration. The paper demonstrates that taking the perspective of all decision-makers is a very useful one. I have the following comments, suggestions and questions.

Major:

No evidence provided for the claim 'We observe that all recalibration methods (including ours) work better if we first apply temperature scaling, hence we first apply it in all experiments.' (p.8,l.290). I would like to see results with and without temperature scaling. Given that the theoretical guarantees apply for the proposed method both with and without temperature scaling, how come there is a significant difference? Is it that the loss is comparable on validation (recalibration) data but different on test data?

It is not clear from the paper whether the experiments have been performed with a single train/validation/test split or there are more replicates with different random seeds. If there is a single split, then how are the error bars obtained in Figure 2 and lines 297-298 (p.8)?

The checklist answers '[Yes] See Appendix' for the question 'Did you include the total amount of compute and the type of resources used', but I did not find any information on this in the Appendix.

The main intuition behind Algorithm 1 has not been explained in sufficient detail: find the worst-case decision task, group by train-optimal action, shift the predictions within each group to be calibrated for that group on average, iterate. Currently it involves a lot of mathematical work to read it out from the notation.

Minor:

I suggest that the authors could offer a name for the notion of $\mathcal{L}^K$ decision calibration that would not require mathematical fonts, to be used in future research.

Although perhaps obvious, the paper does not discuss the relationship of $\mathcal{L}^K$ decision calibration with the existing notions of confidence calibration and classwise ca
libration.

Some earlier works have used cost matrices as a way to define multi-class loss functions for finite action sets. I suggest to at least mention the term 'cost matrix' because it might provide useful insights for some readers.

p.3, l.92: notation of $\Delta$ with an extra line on the left side (as if it were from mathbb font) is quite a rare choice of symbol, is this sufficiently justified?

p.4, Table 1: the associated loss functions are not unique, are they? In the sense that for each notion of calibration there exist infinitely many associated loss functions, e.g. with a different choice of a multiplicative factor but other transformations exist? I suggest to consider mentioning non-uniqueness somewhere in the paper. Does there exist a loss function associated to confidence calibration but not involving the action "I don't know"?

p.5, l.176: 'for a trivial predictors' -> 'for trivial predictors'

p.6, l.216: The L2 error is actually the Brier score, isn't it? I suggest to mention this. Could one use some other divergence measures in place of the L2 error here?

Throughout the paper: 'Radamacher' -> 'Rademacher'

p.8, l.284: 'approximately 15% of the data are used for validation' - why 'approximately'? Is it just rounding to the closest integer or something else that makes it approximate?

p.8, l.296: 'in about 5 steps' - why 'about', are there more than 1 replicate experiments? Or is this 'about' referring to the results in Appendix with different numbers of actions?

p.9, l.319: 'average calibration average calibration'

p.10, l.342: 'A Philip Dawid' -> A=?

p.14, l.476: 'Prosition' -> 'Proposition'

p.15,16, l.509,510: I suggest adding $\cdot$ in front of the identity function to separate it from the preceding symbol. Otherwise it might confuse some readers as the indicator function symbol looks a bit like a rectangle potentially modifying the previous symbol.

p.17, Eq.(12): one of the $\hat{p}$ should actually be $p^*$.

---

After discussion: I am glad to keep to grade 8.
limitations_and_societal_impact: The limitations and social impact have been covered adequately.
review_title: null

Meta-review: While there was not complete consensus on this paper, two reviewers found the work to be valuable -- especially after the author responses, that included clarifications and the additional empirical results to address key concerns raised around ablations and robustness.  Additionally, I found that the most negative review was also one that provided almost no detail or reasoning, which leads me to discount this review significantly.

I will quote reviewer TRQr's private response within discussion amongst reviewers as a useful summary of the paper's strengths:

"I think that the authors have done a very good job answering all concerns raised by all reviewers and have even run very relevant additional experiments. I am definitely supporting acceptance of this paper because it fills the conceptual gap between confidence calibration and classwise calibration on one side and distribution calibration on the other side. It is not just some arbitrary mathematical generalization, it is exactly the right one in my opinion - taking the perspective of the decision maker is the best thing one can do, because that is what calibration is usually meant to serve."

I do expect that the authors will include the additional results in the final paper, and that they will use the reviewer comments to further strengthen the paper.
Decision: Accept (Poster)