Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is insufficient and wasn't able to support its main claims. Needs more work.
main_review: ## Review
This paper made three main points.
1. learning calibrated predictions during training via regularization (e.g., focal loss) improves calibration, but if we are allowed to use post-hoc techniques on top, this approach performs worse than standard cross-entropy training in terms of final calibration performance.
2. those regularization methods align the average confidence to accuracy without achieving fine-grained calibration, which the authors argue is the reason of the underperformance.
3. based on the point 2, they propose inverse focal loss which *de-regularize* that should improve calibration performance when adding post-hoc techniques.

##  Questions and Comments
- First of all, I don't see very strong signal in supporting point 1 especially when looking at the Appendix tables. Pick table C as an example, focal loss with gamma = 1 and L1 norm with alpha=0.001 are actually better than CE with TS.
- I suspected the search space for each regularization method could also be flawed. For instance in table C, L1 Norm would probably have better calibration performance if the alpha is allowed to be smaller than the current range, according to the trend.
- The reasoning behind point 2 is not  very clear - the authors showed the regularization methods compress to much the histogram of logits but it seems not clear that the observation leads to the conclusion that the regularization methods are simply aligning the average confidence of whole dataset to accuracy (after all, the corresponding reliability diagrams didn't seem to suggest that is the case)
- suppose the point 2 is valid, the proposed inverse focal loss should open more room for post-hoc to improve - however as the authors also admitted, that wasn't the case. Would be interesting to see if the new loss actually widen the histogram of logits.
limitations_and_societal_impact: yes in the appendix
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper analyzes the effect of regularizers that improve calibration and find that these regularizers diminish the effects of post-training calibration, such that models trained with simple cross-entropy and after-training temperature scaling exhibit better (lower) ECE.

Moreover, the paper shows that inverse focal loss during training can improve post-training calibration's effect by separating high-confidence samples more strongly from low-confidence ones.
main_review: ### Originality

The examination of calibration through regularization and post-training calibration is novel as is the formulation of the inverse focal loss which is examined in the second half of the paper.

I think the examinations and observations are of high quality and the paper is a very interesting insightful read. Especially the histogram of logits that shows that regularizers that improve calibration lead to "squared together" logits which makes post-training calibration more difficult and the concept of "learned epoch" is beautiful (even if inspired by [29]).

PS: There already exists an "inverse focal loss" in the literature it seems, which was introduced in a totally different context and using a different mathematical expression, but similar motivation:
["Conditional Adversarial Domain Adaptation",Long et al, 2017 (https://arxiv.org/abs/1705.10667)](https://arxiv.org/abs/1705.10667).

### Quality

Overall, the paper's arguments seem sound and are supported by empirical evidence and thoughtful observations, see above. 

The work is honest about strengths and weaknesses within its reported results.

Concerns:
1. The paper mentions it has been calibrating using the test set. This leads to overfitting and reduces my confidence in the reported results.
2. The shared code used LBFGS to find the best temperature. Anecdotal evidence suggests that LBFGS is not the right choice and a grid search gives different & better results usually. This also reduces my confidence in the reported results. Sadly, the temperature scaling code that is most commonly used online does not do this (https://github.com/gpleiss/temperature_scaling/blob/master/temperature_scaling.py, which seems to be the code that was used by the authors of this paper---the source should be marked btw if the code was to be shared widely!)
3. ["Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift", Ovadia et al, 2019 (https://arxiv.org/abs/1906.02530)](https://arxiv.org/abs/1906.02530) shows that post-training calibration does not necessarily work well under distribution shifts. I am uncertain about whether the regularized objectives cope better or worse with distribution shifts.

#3 in particular is important because the behaviour under distribution shift might be most important for practioners.

Re 2.: See ["Calibrating Deep Neural Networks using Focal Loss", Mukhoti, 2020 (https://arxiv.org/abs/2002.09437)](https://arxiv.org/abs/2002.09437) for details: validation NLL is not a convex function of the temperature necessarily and grid searches provide better results.

### Clarity

The submission is clearly written and well organized. The empirical section (together with the appendix) clearly states everything that is needed for reproducing the results.

### Significance

The observations and results are significant. Researchers and practitioners will find them insightful and it will help guide future research. Personally, I would prefer training with regularizers over post-training calibration, and thus paper was surprising and might influence how I train models.

### Summary

An insightful and well-written paper. Given that the temperatures were computed using LBFGS, I have minor doubts about the reliability of the results, however.

### Questions

1. Could you update results with a separate validation set? Or explain why calibration and evaluation on the test set is alright?
2. Could you update results using a grid search to verify that the observations are not artefacts of using LBFGS?
3. Could you comment on distribution shift?

I will adjust my score depending on how any or all of the above questions are answered.

I really like the paper very much, but I'm unsure given my concerns (see above & questions). 

### Typos

1. l. 57: "from worse calibratable" -> "from not being as calibratable" (eg)
2. l. 145: "degrade the ECE" -> "decrease the ECE"
3. l. 148: "would be harmed" -> "are harmed"
4. l. 180: "Figure 2(d)" -> "Figure 2"? not sure about the (d)
5. Figure 2.: export plots as pdf and not rasterized images
6. l. 243: "with the standard CE try to" -> "with the standard CE manage to"
7. Figure 6: "Tempture" -> "Temperature" (y axis label)

limitations_and_societal_impact: In the appendix.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The work studies methods that improve calibration during training e.g., label smoothing, logits regularization. Work shows that while these methods improve calibration compared with conventional training, they are less liable to improve from post-hoc calibration, such as temperature scaling e.g., TS >> label smoothing/etc. + TS. The work suggests some explanations of this effect and proposes an inverse focal loss that based on the proposed intuition might promote the quality of a post-hoc calibration.  
main_review: More detailed discussion:

1. The main effect (Tables 1,2). The work noticed that techniques like label smoothing benefit less from post-hoc calibration. In the end, it is likely more beneficial, to use conventional training with post-hok calibration instead.

Comment 1: No question asked, the effect is interesting, and it needs to see the light of the community.

2.  Explanations (lines 228-244). To the best of my understanding, the hypothesis is the following: conventional training preserves more information about the relative complexity of examples compared to label smoothing and other regularization-based methods. This might be the key factor behind the main effect. Fig 5 presents histograms of max logits(x_i) to confirm that.

Comment 2: There is no clear evidence that "preserve information" leads to better calibration. We do observe both, but there is no indication of a casual relationship in any direction. Thus, I find the finding of this part unjustified e.g., "try to preserve this information, ... hence achieve better results after post-hoc calibration". They are fine as hypotheses, but hypotheses should be clearly marked in the text. 

Comment 3: I want to suggest an alternative hypothesis. From Table 2, we can notice that difference in ECE is much less pronounced with more expressive histogram binning (HB) post hock calibration (TS: 0.72 vs 1.35, HB: 0.68 vs 0.70). This makes me hypothesize that after conventional training all predictions are "equally uncalibrated" meaning that, they are all overconfident and can be easily corrected by TS. After label smoothing it might be the case that the "equally uncalibrated" property fails, and more complex object-dependent calibration is needed. 

3. Inverse focal loss (lines 245-284). The "inverse focal loss assigns larger weights to the samples with larger output confidences" (line 258). To the best of my understanding the intuition is the following: if overconfident predictions fit better for post-hoc calibration, can we push this even further?

Comment 4: The results loos inconsistent to me. My conclusion is that CE is a good default choice of loss, and using IFL is likely (unsure) to make results worse. It is worth adding global statistics to see the whole picture.

Technical drawbacks:
1. Classification models are significantly undertrained. ResNet-32 on CIFAR-10 has 90.5-ish accuracy, but it should be 92.5-ish. ResNet-110 on CIFAR-10 has 89.10-ish accuracy, but it should be 93.7-ish. The same for CIFAR-100 ResNet-110 (Table G, Appendix) has 62.6-ish but it should be 77.3-ish. This unfortunately might be important to findings as ECE is heavily used accuracy, and should be corrected. Or is it due to a 45000/5000/10000 data split? It looks like in this case a drop is too intense at least for CIFAR-100. Maybe that be a consequence of fewer updates due to fewer objects?

2. ECE is a biased measure of calibration, with a model-specific basis [https://arxiv.org/abs/1902.06977]. Thus the results presented in the paper might benefit from using additional metrics. It is unclear, how the bias influences ECE in practice, however, I'm suggesting verifying the findings with other metrics. There is the squared kernel calibration error (SKCE) proposed in [https://arxiv.org/abs/1910.11385] that, also it might worth to report calibrated log-likelihood [https://arxiv.org/abs/2002.06470]. 

3. Table 1. "The values with underline in first row represent the chosen coefficients of each
regularization method on four datasets according to the ECE on test data." Is there any specific reason to choose hyperparameters on test data? 

Overall: 

The paper is most clearly written. The main effect is interesting and deserves to be published. However, these results are obtained on undertrained networks, which can influence results. The explanation part has a weak justification, and the new loss function performs not consistent.  My score at the moment is "6: Marginally above the acceptance threshold". However, it is possible that I will change my score after the discussion.
limitations_and_societal_impact: No.
review_title: null

Meta-review: This paper makes the claim that although deep neural networks tend to be overconfident and poorly calibrated after training, they are “calibratable” in that post-hoc calibration methods like Platt or temperature scaling can recover good calibration.  The authors argue that many regularization methods that have been proposed to improve calibration during training actually result in less calibratable classifiers.  Experiments are run using a ResNet-32 on SVHN, CIFAR10/100 and a FCNN on 20 newsgroups.  

Weaknesses
- All evaluation is done using the standard in-distribution validation and test sets.  
- The only metric that is evaluated is expected calibration error, which is not a proper scoring rule.  It’s well established that ECE can be gamed and should not be used as the sole metric for comparison of models.  
- This paper would be much stronger if it included more difficult / modern datasets and models.  A start would be imagenet with corresponding shifted / OOD variants like imagenet A, C, V2.  In cases where the data distribution changes we know, TS doesn’t perform well.  Thus is this going to provide the right recommendation under those circumstances?  The claims are quite broad given the scope of the experiments.  
- Can we expect to make general claims over all of deep learning given results of a specific ResNet architecture on 32x32 images and a FCNN on 20-newsgroups?
- There are some concerns about showing results that were calibrated using the test data

Strengths
- The reviewers found that the paper was interesting to read and well written.
- They found some of the analysis especially insightful (e.g. "Especially the histogram of logits that shows that regularizers that improve calibration lead to "squared together" logits which makes post-training calibration more difficult and the concept of "learned epoch" is beautiful").
- The arguments were found to be sound and validated by the empirical results.

Overall, this paper presents an interesting observation, but the claims seem somewhat too broad and sweeping given the content of the empirical evaluation.  One might expect to see some evaluation of these models on cases with changes to the data distribution, more rigorous metrics (e.g. NLL),  more varied architectures, some analysis of epistemic vs aleatoric uncertainty. Nevertheless, the reviewers consensus is that the strengths outweigh the weaknesses of the paper and that these results will be useful and interesting to the community.  Thus the recommendation is to accept.

Nit: Bayesian DNNs “which indirectly infer prediction uncertainty through weight uncertainties” is a mischaracterization of BNNs.  They directly infer prediction uncertainty by marginalizing over models, producing a distribution of predictions.

Decision: Accept (Poster)