Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a soft-binning scheme to compute calibration metric so that the models can be trained using SGD to minimize the calibration error together with the classification error. Two metrics have been tried (ECE and AvUC) and the results show improved performance on image classification datasets.
main_review: ## Post-rebuttal update
The authors have clarified both of my concerns about the paper. As mentioned in the discussion below, I would strongly encourage the authors to include the discussion on train ECE overfitting and the role of interleaved training. I'm increasing the score.

## Strengths
- The idea of soft-binning to make the calibration metric trainable is interesting and may have some value.
- Overall the paper is well written.

## Weaknesses
- The main issue of miscalibration is due to overfitting in the training set (wrt to calibration) [Guo et al]. So it is not clear to me that minimize calibration error on the training set would solve the miscalibration issue on the test set. Note that it has been observed that the modern networks have perfect calibration on the training set (if a proper loss such as CE is minimized) but miscalibrated on an unseen test set [Mukhoti et al]. This is a major concern and the marginal improvements in performance may be due to some artifacts of the training scheme. Please clarify this.

- Since the method is optimized for ECE it would make sense to measure calibration error using a different metric such as KS-error [Gupta et al]. This is important to show that the method is not overfitted to ECE in some way. Also, KS-error does not have any issues related to binning. 

- In Table 1 latest methods need to be compared such as [Kull et al, 2019, Gupta et al]. 
limitations_and_societal_impact: Adequate
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses neural networks miscalibration by introducing a soft version of two different calibration measures, namely ECE and AvUC. 

To make ECE differentiable, the authors propose to soften the bin membership in ECE calculation and define differentiable bin size, bin confidence, and bin accuracy based on the soft membership function. The membership function is characterized by a categorical distribution (implemented by softmax) over the temperature-scaled Euclidean distances between the confidence score of a sample and the centers of the bins. The centers of the bins are pre-determined as the center of equal-width binning with some pre-known number of bins. The authors use the differentiable ECE as an additional term added to traditional loss functions for training classifiers. They also use this loss alone for post-hoc calibration. In addition to ECE, they soften the newly introduced AvUC loss with similar tricks.

Experiments on benchmark datasets are conducted to demonstrate the effectiveness of the proposed method regarding ECE performance metrics.
main_review: Overall, the formulation that leads to soft bin membership is exciting and novel.
While I appreciate the rigor that the authors aim at providing to the problem, I have several concerns/questions that need to be addressed:

- It is not mentioned in the paper how the quantities in equations 8-10 are evaluated for each sample during training. Is it required to compute the bin size, accuracy, and confidence after each training iteration on all training data?

- Post-hoc calibration using SB-ECE: I believe there is no need to optimize the soft-binned ECE to find a good temperature value. One can perform a search (multi-resolution) to find the optimal temperature using the hard-binned ECE. It is easy to evaluate the ECE for an input value of t, and it won't take too much time (compared to training the original model) to search for a good temperature value. It also keeps the objective exact without introducing the inaccuracies added by soft-binned ECE.

- In figure 1, For CIFAR datasets, SB-ECE and S-AvUC, when added to NLL, seem not to make much improvement to calibration while making accuracy worse. Adding focal loss seems to improve them. However, this seems like a mystery in the paper. It is the addition of focal loss that makes them work. Can the authors elaborate more on that and explain why this is happening? 

- When training for NLL, we usually have the accuracy on the training set is 100%, and the confidence of the training samples are high, meaning that training on the NLL will result in calibrated confidence scores on training data. I am wondering how the SC-ECE and S-AvUC avoid that? Mainly, is S-AvUC a proper loss such that one can theoretically guarantee upon training one can get calibrated confidences on the training data itself? What happens if one uses these loss functions to continue training on a pre-trained model?

- The numbers in Table 1 show different accuracies/ECEs compared to a similar model in [1] (Table F.1 and Table 2). 
Why are there so many differences between these measures and the numbers in Table 1 of the paper? Some details might have been missed in training (e.g., training for 350 epochs rather than 200 epochs).

- Only top-label ECE is computed. Also, only the TS method is compared. The authors have cited many strong post-hoc calibration methods in the related work but have not compared them with their method. 
Furthermore, there are many cases (especially in the applications mentioned in the introduction, such as medical diagnosis)  the calibration of other classes is necessary (That is why measures like classwise calibration or marginal calibration errors have been introduced[2,3]). These are not evaluated in the paper. Also, it would be nice if the extension of soft-calibration losses is discussed for these cases. ECE itself has many flaws as one has to select the number of bins beforehand. There also biases due to small number of bins, etc., [3]. The method introduced in the paper suffers from the same issues.

- Figure S1 in the supplementary shows that the uncalibrated model is more robust under distribution shift than the proposed post-hoc method in most of the cases. These results are hidden in the main paper and have not been discussed. I hope the authors can explain the reason.

Minor:
- In abstract line 10. 82% reduction refers to which experiment/table in the main paper?
- In eq. 6 and 11, "i" in the sum should be replaced with "j."
- In eq. 14, $S_{\rm AU}$, etc., are not defined.


References:

[1] Mukhoti et al., "Calibrating Deep Neural Networks using Focal Loss," NeurIPS 2020 (arXiv:2002.09437).

[2] Kull et al., "Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichletcalibration," NeurIPS 2019.

[3] Kumar et al., "Verified Uncertainty Calibration," NeurIPS 2019.
limitations_and_societal_impact: YES
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents differentiable loss functions to improve the calibration of deep nets class probability estimates. To achieve this, the authors propose a soft approximation of binning operation, which is at the core of popular measures for model calibration error. Empirical evidence shows that the new approach achieves improved calibration performance compared to existing methods while maintaining accuracy to the extent possible.
main_review: Originality

The proposal of a loss that improves calibration is not new. For instance, R. Krishnan and O. Tickoo took this approach in their work titled “Improving model calibration with accuracy versus uncertainty optimization.” With that said, the novelty of new resides in the explicit minimization of metrics for calibration errors (based on binning), which seems to improve performance. This approach is closer to that of “Calibration of neural networks using splines” by Gupta et al. The authors also soften the AvUC loss offered by Krishnan and Tickoo, allowing the training of non-Bayesian neural networks.

Quality

I believe the writing of the paper can be improved. There is extensive use of acronyms and notations, which make the paper hard to read and follow. I find the proposed idea straightforward; I was hoping to see a theoretical analysis for the overall approximation error obtained by the soft operations. Also, suppose we are given a perfectly calibrated model---would the SB-ECE loss will be equal to zero?

Main concern

In the experiments, the best performing combination of “accuracy loss” + “calibration loss” is not consistent across the benchmark data sets. As a result, it is not clear what should be the default approach. I also do not understand why the calibration loss reduces predictive performance; it seems to be a reasonable regularization. Would the ideal model satisfy this property? In other words, does the optimal classifier minimize the proposed losses (with infinite data)?

Significance

The problem of calibrating the probability estimates obtained by machine learning algorithms is important. The paper presents a new way to encourage calibrated predictions during training. The experiments demonstrate the effectiveness of this approach; however, it is not clear which loss works best.

limitations_and_societal_impact: The limitations are partially described. See "main concern" above.
review_title: null

Meta-review: The authors' response has addressed all the reviewers' concerns, and all the reviewers' final score was "weak accept", which is borderline. It seems that main issue was that they were not completely sure how the different design choices are related to the final performance, which would make this method somewhat less useful to practitioners. For example, Reviewer CL78 summarized:

It seems incorporating the soft-calibration objectives helps to improve the calibration during training. I am not getting a higher score because most of the paper is devoted to SB-ECE, while the most successful secondary loss is S-AvUC. However, there is not much theoretical support (except the previous work under the SVI setting) for different design choices made by the authors (I am not much familiar with SVI). Furthermore, there are some combinations of primary losses that have a great impact on the final results. Although the soft-calibration objectives can improve the calibration in most cases, it might not be easy for a practitioner to summarize which combinations of primary and secondary loss should be selected in advance.
Decision: Accept (Poster)