Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary:  The paper proposes a differentiable (and therefore trainable using gradient descent), low-bias calibration error estimator based on Dirichlet kernel density estimates, which is consistent w.r.t $L_{p}$ calibration error. 

- The computational complexity of the estimator in the paper reported is $\mathcal{O}\left(n^{2}\right)$, while the convergence rate is $\mathcal{O}\left(n^{-1 / 2}\right)$, and it is unbiased up to $\mathcal{O}\left(n^{-2}\right)$, which the authors achieve by debiasing the estimate.

- This estimator shows strong performance in estimating the exact $L_1$, $L_p$ calibration error in a synthetic dataset with known calibrations. Since it is also a differentiable function, it can be used in conjunction with crossentropy loss to train neural nets, and it shows improved accuracy and calibration properties over other differentiable proxies.
strengths_and_weaknesses: 1. Originality

The field of uncertainty calibration has seen significant research in recent years in trying to find a differentiable approximation to calibration error, in order to be able to train calibration error using gradient descent. To my knowledge, previous methods have attempted to (1) perform soft binning, obtaining a differentiable approximation to the binned ECE (Bohdal et al 2021), (2) Mix-n-Match (Zhang et al 2020), which uses numerical integration and doesn't extend to multiclass calibration as far as I am aware. This paper addresses both these concerns quite well, as their metric is independent of binning, and doesn't require an expensive integration step.  I think the approach is quite original, and the authors have done a good job performing a literature survey. To my limited knowledge in this field, I didn't notice any similarities to previous work, and it seems like the current approach mitigates many concerns other methods have had.

2. Quality

I believe this paper is well written, has a clear motivation, strong experiments, and well-derived proofs and theorems. Not only does this paper propose a differentiable proxy to Lp calibration error with good guarantees, this proxy seems to provide good regularisation to empirical risk minimization, resulting in better classification performance on real-world datasets.

3. Clarity

I found the paper mostly easy to follow. I will admit that I found it a bit difficult to follow Appendix C, though this is due to my limited knowledge in proving such quantities. I feel like there wasn't much discussion about the bandwidth parameter $b$, and maybe explaining a little bit more how the leave-one-out MLE selection was performed would make this a bit clearer.

4. Significance

Model calibration is a very important field, and there has been much work on trying to find a differentiable proxy to ECE. To this extent, this paper accomplishes a significant milestone, as they propose a consistent, low-bias, differentiable approximator to Lp calibration error.
questions: I mostly have minor nits to the paper, and I think it is a well-rounded paper in general, that covers most of my doubts.

1. "Interestingly, we observe that temperature scaling does not improve canonical calibration error, contrary to its reported improvements on top-label calibration." I think this line warrants much more study. Temperature scaling is a very commonly used technique to acheive quick calibration, and if it is not showing any improvements on the ECE metric proposed by the paper, that poses some interesting questions. Temperature scaling clearly improves model calibration, results in better performance across benchmarks on ECE (binned), Brier score, and OOD performance. However, this improvement in performance is not being captured by the new ECE metric? Do the authors have any intuition as to why? Is the KDE independent to division by a scalar term?

Minor nits:
- Line 81: "Maximum"
limitations: I believe the authors have addressed limitations well, and have a good section talking about potential societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a novel strategy to estimate distributional or canonical calibration error based on Dirichlet kernel density estimates which asymptotically converges to the true Lp calibration error.  Such a calibration penalty is differentiable and can be incorporated while training a classifier.  The proposed approach improves the L1 ECE across multiple classification benchmarks and model architectures.
strengths_and_weaknesses: Strengths

The idea of using Dirichlet kernel density estimates as the means to estimate canonical calibration is novel. The paper clearly illustrates how the conventional metrics such as ECE^BIN do not clearly convey the notion of canonical calibration and why it is important to estimate such a calibration error.  The work is mathematically well grounded and demonstrates a number of synthetic as well as real-data experiments. Overall the paper is well written with minor typos. 

Weaknesses

Kindly refer to the questions below


questions: How does adding the KDE calibration into the training pipeline explicitly control over-confidence in NN ? It would be beneficial to plot the accuracy vs confidence reliability plots along with comparisons with existing methods. 

Can such a strategy be directly adopted for regression tasks ? It would be beneficial to demonstrate empirical results for regression. 

Can this training strategy lead to models that can detect distribution shifts much better than other training strategies that include different calibration objectives ?  How does canonical calibration improve robustness to distribution shifts ? 

Fig. 4 can be better expanded and scaled.

These are some of the typos I had spotted
Line 106 - It would be beneficial for readers if you can expand the notations used in the definition of the probability space (omega, A, P). 
Line 149 - Need to specify that lambda and gamma are used interchangeably
Line 111: Left paranthesis can be removed before 'Calibration'
Eq. 12, the index j needs to be defined

Suggestions
It would be great if you can change some of the arxiv references to their actual conference/journal references (if published)

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Classifiers with deep neural networks tend to output overconfident class-posterior probabilities.
This paper proposes a novel method for calibrating the confidence (i.e., the predicted class-posterior probability) during training a classifier with deep neural networks. The proposed method adds a term to the loss for penalizing miscalibrated confidence scores.
The added term is an estimate of the $L_p$ distance between the predicted class-posterior probability and the true one, which is differentiable so that we can use the gradient descent method for training.
They show that the estimator of the discrepancy is statistically consistent, and calculation of it takes $O(B^2)$ time for mini-batch size $B$.
The paper presents experiments with a synthetic dataset and a three benchmark datasets whose results support the usefulness of the proposed method.
strengths_and_weaknesses: # Strength
- The paper tackles the important task of confidence calibration for classification with deep neural networks.
- The authors provide some guarantees for their estimator of the miscalibration penalty term. Specifically, the estimator has $O(1/n)$ bias, and the rate of convergence is $O(1/n)$. The bias can be further improved to $O(1/n^2)$.
- The overhead in computation time added by the proposed method is $O(B^2)$ for mini-batch size $B$, which may be small when $B$ is small. In fact, they empirically showed that adding the proposed term only increases the training time by a few percentage points in their experiments.

# Weaknesses
- Their convergence analysis is done for a fixed function and is not guaranteed to hold for all functions of the feasible space of their optimization problem. Analysis of uniform convergence would be preferable.
- For a mini-batch size $B$, the estimator will have a bias of $O(1/B^2)$. For example, if $B$ is constant determined by the GPU memory size, we will have constant bias.
- Proofs in the paper are too brief and it is difficult to confirm the statements are correct.
questions: - Denoting the mini-batch size as $B$, the overhead in the training time is indeed $O(B^2)$, but we only need to evaluate the function $B$ points in each iteration (which is nice). If this is correct, I suggest the authors mention this in the paper.
- I am not very sure if "scalable" is the right word for describing the overhead of $O(B^2)$. I suggest simply removing this from the title or using milder words.
- I request that the authors clearly present the mini-batch sizes used in the experiments rather than just referring to other papers.
- I do not think the statements about consistency in the paper are precise. The convergence has to be stated in terms of convergence in probability. I suggest making them more formal. The proofs are also very rough should be improved.
- Also, the paper only provides point-wise convergence analyses although the proposed method performs optimization over a function class. I suggest providing convergence analysis that holds uniformly over the function class. 
- Are the color scales of Figure 1(b) and Figure 1(c) the same?
- "realistic scenarios where the probability scores are concentrated along lower dimensional faces of the simplex": What makes the authors believe it is "realistic"?
- How did the authors choose the bandwidth (0.5) and the location of KDE (0.17) for the synthetic data experiments?
- "We found that for KDE-XE, values of $\lambda \in [0.001, 0.2]$ provide a good trade-off in terms of accuracy and calibration error": How do we define a "good" trade-off?
- l.190: What does "high-cost settings" mean?
- What is the "partially" debiased estimator?
limitations: As I mentioned in "Questions" section, the training time overhead $O(B^2)$ can limit the mini-batch size and thus create bias. It would be nice if the authors could provide the maximum mini-batch size they can handle.
review_title: null

Meta-review: metareview: This meta review is based on the reviews, the authors rebuttal and the discussion with the reviewers, and ultimately my own judgement on the paper. There was a consensus that the paper contributes an interesting and novel strategy to model calibration based on Dirichelt kernel density estimates, and the reviewers praised several of its aspects. I feel this work deserves to be featured at NeurIPS and will attract interest from the community. I would like to personally invite the authors to carefully revise their manuscript to take into account the remarks and suggestions made by reviewers. Congratulations!
confidence: Certain
Decision: Accept