Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a trainable objective less ESD as the extra calibration loss jointly optimized with the NLL loss during training. It is claimed as a binning-free objective lease without need to tune any additional parameters. The experiments were extensively conducted across three architectures (MLPs, CNNs, and Transformers) and both vision and NLP datasets. The experimental results seem to strongly validate the efficacy of the proposed method.
strength_and_weaknesses: #### Strengths
- Overall the paper is well written and easy to follow.
- The idea of ESD is very interesting and should be able to bring a positive impact to the AI fields not limited to both vision and NLP domain.
- Strong theoretic explanation with solid mathematical functions. 

#### Weaknesses
- The accuracy has no obvious advantages when compared to other competing baselines, according to Table 1.
- As the paper proposes a general calibration loss, I would like to suggest the authors to conduct the experiments on the more general large-scale vision datasets (e.g., MS-COCO and ImageNet).
clarity,_quality,_novelty_and_reproducibility: - Clarity: overall this article is well-organized and easy to follow. 
- Quality and novelty: the paper is solid and the novelty is intermediate.
- Reproducibility: The authors promise to release  the code.
summary_of_the_review: Based on the above statements, I would like to weakly accept the paper at the initial stage. After carefully reading the other reviewers' comments and the authors' response, I would like to keep the initial rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a hyperparameter free training based calibration method using proposed calibration metric, namely expected squared difference (ESD). The proposed calibration metric is inspired from the recently introduced KS metric that compares cumulative distribution.
strength_and_weaknesses: Strengths:
- The paper is easy to follow and well-written.
- There has been a lot of research focused on post-hoc calibration methods in the literature but this paper looks upon the solving the mis-calibration at the training time without significant additional computational costs. This problem is particularly challenging as the overfitting is a significant challenge to solve this task. 
- The paper achieves impressive results on multiple benchmarks of calibration while proposing a theoretically concrete approach to calibration.

Weaknesses:
- Since this is a training based calibration method, I would suggest the authors add calibration results on large scale datasets such as ImageNet. 
- An important addition to the paper, should be along the lines of calibration performance under distribution shifts [a] as this has been known to be difficult for post-hoc calibration methods but I would assume the proposed method should be able to perform better under such a scenario.

References:
- Ovadia et al. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS 2019.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written.
Quality: Empirical and theoretically solid.
Novelty: The proposed method of calibration is novel especially in terms of its application at train time.
Reproducibility: The authors have promised to release code publicly.
summary_of_the_review: I am in favor of acceptance of the paper as its a novel way of calibration and proposed method looks clean. I suggest some few additions to the paper which will make it stronger. If my concerns are addressed, I will increase my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper present Expected Squared Difference (ESD), a tuning-free (i.e., hyperparameter-free) trainable calibration objective loss, which views the calibration error fromthe perspective of the squared difference between two expectations.
strength_and_weaknesses: Strengths:
1. Build a well-established  trainable calibration objective loss ESD, which is binning-free and does not need hyperparameters. 
2. Extensive experiments to prove that the esd produces more accurate ECE without tuning the hyperparameters.
Weaknesses:
1. This calibration method itself does not need any hyper-parameter but still incorporates some other hyperparameter, such as the regularizer hyperparameter λ, monte carlo sample number N, etc. 
clarity,_quality,_novelty_and_reproducibility: Clarity&Quality: this paper is well-written.
Novelty: notable novelty
reproducibility: Settings and parameters are listed and the authors committed to release the code publicly
summary_of_the_review: This paper proposes a hyperparameter-free calibration method, ESD and produces accurate predictions and calibration results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new method to improve calibration of multi-class classifiers. It is similar to the existing methods of MMCE and SB-ECE in the sense that it adds a trainable auxiliary loss function (Expected Squared Difference, ESD) to the cross-entropy loss (NLL) during training of the classifier. Similarly to SB-ECE, the paper uses interleaved training where the new auxiliary loss and NLL are trained on separate parts of the training set. ESD is parameter-free and its estimator is unbiased. The experiments on 5 datasets demonstrate that compared to MMCE, SB-ECE and the baseline NLL it results in better-calibrated predictions (lower ECE) both before and after post-hoc calibration with temperature scaling or vector scaling. About 1% of accuracy is sacrificed compared to the baseline NLL.
strength_and_weaknesses: Strengths:
* The paper is well-written and clear.
* The proposed method ESD has been justified theoretically and the ESD estimator has been proved to be unbiased and consistent.
* The experiments on 5 datasets demonstrate that ESD results in better calibrated probabilities than earlier methods, while not sacrificing more accuracy.


Weaknesses:
* The intuition behind the proposed method could have been explained more, perhaps on a toy example.

* More information about the experimental results could have been reported, for example as an appendix. It would be good to know whether the range of values for the hyperparameter lambda is sufficient and which values were eventually chosen by the method on different datasets. Also, it would have been good to report NLL as an evaluation measure, in addition to accuracy and ECE. E.g., Table 1 could have had extra rows for NLL, NLL after TS, and NLL after VS, or alternatively, this information could have been provided in the appendix. The paper does not mention why accuracy after vector scaling is so low for CIFAR100 and ANLI. Can there really be so much overfitting even though the number of classes is quite high?

* The conclusion fails to mention that some accuracy has been sacrificed to achieve better calibration. On the positive side, this fact has been mentioned in Section 6 about experimental results.

* Notation $k$ is overloaded in Section 3 - first it is the total number of classes, then just any class in Eq.(1), and then the class with maximum output probability in Eq.(2). This can create confusion, in particular because Eq.(1) is about being classwise-calibrated whereas later the paper only studies confidence calibration.

Minor weaknesses:
* Table 3 and Section 7.3 fails to mention which dataset it is about. I assume CIFAR100?

* The last sentence of page 7 (first paragraph of Section 6) states the decrease of accuracy by more than 2% - is it not about 1% in this case (47.08 vs 48.0) because the comparison is all between ESD and ESD+VS?
clarity,_quality,_novelty_and_reproducibility: All is novel and reproducible and clear, except for the weaknesses highlighted above.
summary_of_the_review: The paper proposes a parameterless method and improves the state-of-the-art. The proposed method ESD is simpler and better than the existing methods MMCE and SB-ECE. There are some weaknesses highlighted above but these can be fixed for the camera-ready v
ersion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Uncertainty can be challenging in deep learning, and this work aims to improve their calibration. In particular, the authors propose a calibration objective Expected Squared Difference that is differentiable and tuning-free, able to be used during training. It is similar to recent works in this space such as MMCE and SB-ECE. The argument for the new proposal is that it's simpler and shows better results in practice. They validate their proposal on MNIST, CIFAR-10, and SNLI.

Reviewers found the work well-written and easy to follow. The theoretical justification was also highlighted a positive. I think the problem is quite challenging as it's easy to overfit when training for calibration specifically (see also the paper's Section 3.2.2). They build on Karandikar et al.'s approach of interleaved training and the experiments show that among the datasets analyzed, ESD is quite better than the 2 previous methods.

All reviewers leaned toward accept and I agree with their consensus.
justification_for_why_not_higher_score: Trainable calibration objectives is a somewhat incremental space since works have already been proposed in this area, especially SB-ECE. The improvements are statistically significant on the limited set of experiments shown, but haven't been checked on even ImageNet scale which seems standard practice particularly for these models. The baseline accuracy #s across the datasets aren't strong enough to be convincing enough that even in the space of trainable calibration methods, that this is _the_ best method. A true proposal for the best calibration would further compare to the broader space of methods improving calibration (e.g., efficient ensembles).
justification_for_why_not_lower_score: Reviewers have uniform agreement of accept, and the method is simple and interesting to consider.
Decision: Accept: poster