Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes AdaFocal, an adaptive focal loss that modifies the focal loss hyperparameter $\gamma$ for different sample bins based on historic values of $\gamma$ and the model's per-bin calibration properties. The authors evaluate AdaFocal on several image classification benchmarks (CIFAR-10, CIFAR-100, TinyImageNet, ImageNet) and one text classification dataset (20 Newsgroup).

**Implementation details**
The $\gamma$ update rule adjusts $\gamma_{t+1}$ based on the value of $\gamma_{t}$ and the difference between the model's accuracy and confidence in a particular validation bin. Training examples are assigned to particular validation bin based on validation bin boundaries that are re-computed once per epoch. The adjustment for $\gamma_{t+1}$ is then computed based on the validation bin miscalibration that the training example is assigned to.
strengths_and_weaknesses: Strengths:
- The update rule for gamma and the algorithm for AdaFocal is fairly straightforward. The overall simplicity will ease implementation and increase adoption of the method.
- The authors give strong empirical evidence that the confidence of the neural network on the training data roughly matches the confidence of the validation data for the image classification datasets, which helps motivate the AdaFocal method.
- The empirical results for AdaFocal are strong across all experimental setups for pre-temperature scaling and strong for CIFAR-10/CIFAR-100 for post-temperature scaling.


Weaknesses:
- The method still involves some heuristics which may not generalize to other datasets/tasks and the paper uses empirical evidence collected only on a limited selection of datasets/tasks to justify the method.
  - For example, one such heuristic is thresholding $\gamma_{t}$ based on a pre-set hyperparameter $S_{th}$ to switch between the inverse-focal and focal loss. Another heuristic is the number of bins. Both of these hyperparameters may vary in optimal value based on the dataset, model, and task.
- Only one model architecture is tested for TinyImageNet, ImageNet, and Newsgroup 20. 
- Post temperature scaling, other methods outperform AdaFocal on both ImageNet and Newsgroup 20 tasks.
- Error bars should be included in Figure 4 if the plots are showing an average over 5 runs.
- The plots comparing confidence of training and validation samples should be included for the Newsgroup 20 classification task.
- Missing baseline comparison to NeurIPS '21 calibration-during-training method "Soft Calibration Objectives for Neural Networks" (https://arxiv.org/abs/2108.00106)

My main reason for leaning towards a borderline reject rather than accept is that the method is justified purely based on empirical evaluation but the empirical evaluation has several weaknesses (mentioned above).
questions: - The validation bin boundaries are only updated once per training epoch. What happens if they are updated more frequently (i.e. every minibatch or at least a few times per training epoch)?

limitations: No, the authors should address limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a focal-loss-based method called AdaFocal for uncertainty calibration problem. When training models using AdaFocal, the hyper-parameter $\gamma$ used in focal loss can be dynamically varied during epochs and different samples.

The experimental results show the effectiveness of AdaFocal for calibration in both the cases of with and without post-hoc calibration. The experiments also shown AdaFocal can easily align mean confidence with accuracy of each bin in validation set.

strengths_and_weaknesses: Strengths:

1. The proposed AdaFocal method exploits an adaptive strategy to determine the important hyperparameter $\gamma$ used in focal loss. AdaFocal can be considered as a generalized version of naive Focal loss and Focal-53.

2. The illustrative experiments before introducing the proposed method give clear motivation.

3. The datasets used in experiment part are from different domains, while have different difficulty degree. The results show the effectiveness of the proposed method.

Weaknesses:

1. The proposed method is used to adaptively determine the hyperparameter $\gamma$ used in Focal loss. However, other hyperparameters are induced in the proposed method like $\lambda$ and $S_{th}$. Would these hyperparameters significantly impact influence the final performance? Given this more complicated method, one naturally expects that the choosing of new hyperparameters like $\lambda$ and $S_{th}$ is much easier than that of $\gamma$.

2. Figure 2 shows the good correspondence between the average confidence of training set and validation set. In this case, the training samples are grouped using **validation-bin boundaries**. I think a fairer setting is that the training samples and validation samples are grouped independently into their respective training-bins and validation-bins. But in this case (as is shown in Figure 9) the correspondence between them is not good.
questions: See weaknesses.
limitations: The authors have addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present an adaptation of Focal Loss that adapts the focal hyperparameter over the course of training to the data. This produces on average better calibrated models than traditional cross entropy or focal loss, especially when combined with post-hoc recalibration.


strengths_and_weaknesses: **Strengths**

- Incidentally, it also removes the focal hyperparameter from consideration.
- The authors introduce problem and related concepts well.
- The results section is comprehensive within their chosen focus and concise.
- The work overall is novel and relevant as model calibration is important for reliability.

**Weaknesses**

- The authors mostly focus on image classification. Some of their observations may not hold for other scenarios. e.g. their hyperparameter $\lambda$ might become relevant.
- Section 3 and following would benefit from some editing.
- There are a few naming inconsistencies e.g. calibration error and calibration gap. Consistency here makes it easier to read.
- The explanation of how the validation binning is applied to the training samples could be expanded.
- There are some typos (FLDS, parametera) and I think the qualities you describe as "innate" are more inherent.
questions: - l. 76 what is $\mathbf{H}$ again?
- Can you reason why for ImageNet Post Temperature Scaling CE produces the best calibrated model?
limitations: not applicable
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes AdaFocal, a scheme to dynamically pick the $\gamma$ weight of the focal loss for the different groups of samples when training so that the expected calibration error is minimized w.r.t. validation set, during training a classifier. The weight $\gamma$ were adjusted so that the focal loss can suggest a classifier to be overconfident or underconfident in a certain region to calibrate the confidence to be closed to the mean accuracy. Experiments showed that AdaFocal outperformed many existing methods for calibrating the classifier, including the recent focal loss-based calibration method (FLSD-53) by Mukhoti et al. 
strengths_and_weaknesses: Significant and originality
- To my knowledge, the proposed method is novel, and its performance of the proposed method is impressive. 

Strengths
- Proposed method is relatively simple to implement although there are several hyperparameters. It seems the standard choice of hyperparameters is applicable to many datasets at the same time (at least for image datasets in the experiments).
- Experiments on OOD detection have also been conducted, where AdaFocal is superior to other methods.

Weaknesses
- No theoretical aspect of the loss is discussed. It is uncleared what is happening to the minimizer of this loss function. In my understanding, the method is quite heuristic, the choice of weighting function in Algorithm is a little bit arbitrary (using exp(\lambda). Nevertheless, I think the intuition of the proposed method makes a lot of sense and quite interesting to try. 
- AdaFocal introduces several hyperparameters to the model training process, e.g., number of bins to use, max/min of $\gamma$
- AdaFocal also introduces more training time since it has to evaluate the validation performance on each bin in every epoch (maybe not a big drawback in normal case) 
- The accuracy of using AdaFocal is not guaranteed to be the same compared with standard training with CE + postprocessing methods like temperature scaling. Although it is unclear when AdaFocal can have better performance than CE (and vice versa: see CIFAR-10 ACC and CIFAR-100 Acc). For anyone who simply wants to use the same loss function (because the model has been using CE for a long time and many promising hyperparameters for CE may not be the one for AdaFocal) , using AdaFocal may not be the way to go.
- (minor) Reliability diagram result is not provided. I think looking at the reliability diagram can also be informative (at least in the appendix for future reference would be nice).

Clarity:
- The paper is well-written overall and not difficult to follow. However, I found that figure is a bit difficult to read and takes some time to understand. But the information provided in the paper is enough to understand the figure.

Overall:
I think the paper has sufficient evidence to suggest that using different $\gamma$ for different bins can be effective when using the Focal-loss style technique for improving the calibration of the classifier. The idea of using an over-confident version of Focal loss for calibration problems is not that straightforward in my opinion, and could potentially be used for a different problem in the future. Thus, I feel the paper has merits that outweigh its drawbacks, which are a lack of theoretical evidence and the increasing number of hyperparameters for training a model.
questions: 1. It is unclear to me what the weakness of AdaFocal is. Is it possible that training with AdaFocal may require a longer training time than using CE, etc.?
2. There is also a recent theoretical work [A], where they conducted a rigorous theoretical analysis of the behavior of $\gamma$ of focal loss and its relationship to calibration, which suggested that focal loss is not estimating class posterior probability. Also, it has been shown in their work that in a non-deep setting (Table 2 of [A] in the appendix: linear model or simple multilayered perception), using focal loss is worse than vanilla cross-entropy. I was wondering how is the performance of AdaFocal in such a simple setting. 
3. SVHN is also a dataset that standard CE seems to be better than Focal loss in terms of calibration even in the deep learning setting (according to [A]). In AdaFocal paper and Mukhoti et al. papers, SVHN is not used. In AdaFocal paper, it is used in OOD detection but the ECE score is not presented in the main body of the paper. I was wondering if AdaFocal can mitigate this drawback of the focal loss (I feel AdaFocal may work well in SVHN but Mukhoti's method might not), because this loss use validation set to enforce the confidence is always calibrated w.r.t. calibration error. 
4. Regarding the weakness of AdaFocal from the theoretical perspective highlighted in the "Weakness" section, have the authors considered the theoretical perspective of AdaFocal, perhaps something similar to the analysis in [Mukhoti et al.] or [A]? 
5. Did the authors apply early stopping to the learner, since each method may have a different time to achieve best performance? For example, in Tiny Imagenet according to Figure 4, CE achieves the best performance around epoch 40 (where other methods also performed pretty well there). On the other hand, CE performance degraded as the epoch increases and temperature scaling may not be able to a reliable confidence score.

Minor
1: Table 8 in Appendix is an empty table. Please remove it or the authors may have forgotten to put the table information.


[A] Charoenphakdee, N., Vongkulbhisal, J., Chairatanakul, N., & Sugiyama, M. (2021). On focal loss for class-posterior probability estimation: A theoretical perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5202-5211).
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: Prior work has shown that training with focal loss improves the uncertainly calibration of the models. In this work, authors show that the adoptive choice of the hyper-parameter used in the focal loss would further improve the calibration. Extensive empirical results showed in the paper suggest that the proposed method, AdaFocal, improves over the baselines in several domains/datasets and architectures in terms of calibration error and OOD detection.

The paper is well-written and easy to follow. The proposed method is novel and its relationship with prior work is properly discussed. Furthermore, empirical results show that this method outperforms existing baselines. Therefore, I am recommending acceptance. Given lack of any theoretical insights or motivation for the proposed method, it would certainly help to improve the empirical results particularly by adding results for a few other architectures trained on ImageNet. Furthermore, please add discussions and ablations on the sensitivity of hyper-parameter choices for the camera-ready version.
confidence: Less certain
Decision: Accept