Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a new sets of algorithms to modify \gamma in the focal loss, where a tunable \gamma is applied at different region of model predications. Empirically study on Cifar-10 and Cifar-100 data with various of benchmark networks show better ECE as compared to the other methods. 
main_review: The idea of the paper is to optimize \gamma tuning in focal loss to improve the calibration of neural networks model. The motivation makes a lot of sense. Although the novelty of the paper seems limited and idea seems adhoc, while practically the proposed approach could be helpful to the loss function and calibration study.  Some concerns to this paper are 

[1] The paper seems written in a rush and the paper organization and writing is weak. 
- In abstract, FLSD-53 is ambiguous to in its abbreviation when first used
- The plots in the paper mostly have tiny legends

[2] In table2, it seems AdaFocal  improved calibration but not accuracy in Cifar-10 data, while in the opposite way on Cifar-100 data, as compared to FLSD-53. Are there any interpretations about the such calibration and accuracy trade-offs?
summary_of_the_review: Overall the paper proposes a new strategy to optimize calibration via tuning \gamma in focal loss. Writing in this paper is a little concerning. The idea is a little incremental but could be potential beneficial to the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the calibration of deep learning, which aims to make confidence store accurately describe predictions' correctness probabilities. The authors improve focal loss and propose a calibration-aware focal loss for better calibration.  The proposed approach adaptively adjusts the coefficient of focal loss according to the momentums and current predictions' confidence. The authors conduct experiments on SVNH, CIFAR10/100 datasets to verify the approach's efficacy. 
main_review: Strength:
1. This paper gives an in-depth analysis of the calibration problem in deep learning. The authors investigate the calibration properties of focal loss, observe that fixed $\gamma$ is not optimal for better calibration,  and find a correspondence to adjust $\gamma$ in each iteration adaptively. 
2. The authors present many figures and investigation experiments for analysis and verification of effectiveness. The proposed method is effective in CIFAR10/100 and SVNH datasets. 

Weakness:
1. The generalization of the proposed method is not promised. The observed problem of fixed $\gamma$ are empirical and may vary between datasets.  On the other hand, the optimal hyperparameters for the proposed method may also vary between datasets.  Besides,  it would be better to report the large-scale datasets' results, such as ImageNet. The current experimental results are not convincing. 
2. The figures in this paper are not easy to understand. (a)There are too many figures, and hard to understand the representations of each line. Some legends even block important information. The authors may present less but important figures with concise conclusions in captions. (b) It would be better to smooth some lines in some figures. For example, figure 1 will be better if the lines are smoothed. (c)Some figures are too small to read after printing. And some figures are not readable without color printing. 
3. How will the proposed method affect the test accuracies? Will the proposed method lead to an accuracy drop as the expense for improved calibration? Figure 4 reports the accuracies but is very hard to tell the accuracies difference. 
summary_of_the_review: The calibration problem is interesting and practical in real-world applications.  This paper presents an in-depth analysis of the calibration problem and proposes an effective method to improve it. However, the proposed method has not been proved to generalize well in different datasets. The authors may answer this question by a theoretical analysis or experiments in large-scale datasets. Another suggestion is to improve the figures of this paper. If the authors can address my questions, I am willing to raise my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of model calibration. Existing works calibrates the model by post-hoc approaches or objective function tailored for calibration. The authors of the paper propose an adaptive version based on Focal loss, which regularizes the overconfidence of neural networks. They observe that although focal loss improves the calibration, it leaves out the under-confident samples. To mitigate the issue, they propose adjusting the hyper-parameter $\gamma$ in focal loss according to the model's under/over-confidence. Experiments on vision and NLP classification tasks showcase the effectiveness of the adaptive version.
main_review: # Strong points

- The paper proposes an adaptive version of focal loss for model calibration. The hyper-parameters in focal loss can dynamically adjust for the calibration criterion.

- A similar validation set is used for updating the hyper-parameters. The updating rule is simple and can be easily plugged in existing methods.

- The proposed method outperforms other baselines on extensive datasets


# Weak points

- One glaring issue of the current version is the readability of figures. Fig (1),(2),(4),(5) are difficult to parse. There are even lines with duplicate colors in the second-row figures of Fig (2).b.

- "And from a calibration point of view, our strategy going forward would be to exploit this behavior to keep C_{train,true,i} (which we have control over during training) closer to A_{val,i} so that, in turn, C_{val,top,i} also stays closer to A_{val,i} to overall achieve low calibration error C_{val,top,i} − A_{val,i}". It seems to me there are two pitfalls in the sentence: (1) since the validation set is the actual target (claimed by the author on page 2), how can people make use of the accuracies on the validation set to tune models? (2) The correspondence in average confidence $C$ can not be directly translated into the correspondence in the calibration error $|C-A|$. 

- It seems that the update rule in Eq. (4) is unstable. For example, if $C-A > k$ for $m$ steps, then $\gamma \ge e^{mk}$, which in turn makes the focal loss rather small. The authors propose to use a threshold to rein in the explosion. However, it seems to me that the explosion will commonly happen, and at most of the time $\gamma=\gamma_{max}$. Could the authors provide the dynamics of $\gamma$ during the training process?

# Minors

- In the second paragraph from the bottom: $\mathcal{L}_f$ should be the Focal loss L_Focal?

- It seems there is an extra "top" in the subtitle of Fig (1).b?

- Bottom paragraph on page 4: the $n$-th sample should be in $i$-th bin?

- Sentence above Eq (1): A_{val,b} instead of A_{val,i}

- What's the y-axis in Fig .(3)? Classification error on validation set?



summary_of_the_review: All in all, the idea is interesting but I find the paper in its current form to be somewhat below the standard for ICLR acceptance, especially the write-up itself.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new focal loss method of calibrating a model via regularization during training. They propose an adaptive method of adjusting the focal loss parameter by empirical validation performance which adjusts the parameter based on specific bin over/under confidence. 
main_review: # Pros 

- The authors consider a wide variety of recent calibration literature.
- The approach is intuitive and is verified by empirical results showing that the patterns between validation and training ECE seem to hold. 
- The authors motivate the need by empirically showing that a single focal loss parameter is not optimal among different bins.
- The experiments are extensive and cover important calibration and OOD metrics.

# Cons

- The axes and legends in figure 1 are too small and very hard to read
- The axes are unlabeled in figure 2a
- Section 5: “this mostly holds true” $\rightarrow$ It is not clear what “this” refers to. From the context it appears that “this” refers to “focal loss not leading to smaller gradients.” It appears the next sentence leads to the opposite conclusion which is confusing. 
- Figure 5 is too cluttered and extremely hard to read
- Table 1 and the preceding paragraph: In both places, the text states that AdaFocal’s performance is averaged over 5 runs, which implies that the baselines are not. How many runs were the baselines trained for?
- The authors highlight that AdaFocal gives the best performance among the “Pre T” results, but isn’t it also the case that AdaFocal is the only algorithm which has made use of the validation data at this point? This would explain why there is not much improvement with temperature scaling and AdaFocal.
- The method of updating $\gamma$ is based on summing the exponents together which may lead to large values which are mitigated by clamping to a maximum value. Have the authors considered using an exponential moving average of the exponents such that the argument to the exponential is of the form $\alpha E_{val,t} + (1 - \alpha) E_{val,t+1}$? I believe this would prevent the instability of the sum and not require any clamping to a maximum value.
 


# Minor

- Page 3, section 4: The sentence which begins with “Nonetheless, for completeness…” is missing a prenthesis and probably also an unnecessary comma which makes the sentence confusing to read.
- Section 4: approaches towards $\rightarrow$ approaches.
- Section 5: During majority $\rightarrow$ during the majority
- Section 5.1: hyperparameter$\gamma$ $\rightarrow$ hyperparameter $\gamma$
- Section 5.2: this update rule do $\rightarrow$ this updates rules does
- Section 5.2: reign in $\rightarrow$ rein in

summary_of_the_review: The authors do a good job in motivating their method and empirically showing why it is needed and why it should work. In mu opinion, the results are very hard to read and the paper has a problem with formatting/visibility or tables and figures. Additionally, If I have understood everything correctly, I think the formulation of equation 4 may be improved, eliminating a hyperparameter (see last bullet point in Cons section). Overall, these are the main points influencing my score.
review_title: null

Meta-review: The work AdaFocal proposes an approach to tune Focal Loss' $\gamma$ hyperparameter to improve the model's calibration, particularly to avoid the occasional underconfidence when using focal loss. This tuning is done not as a learned constant hyperparameter across training but as one that evolves over training.

The work is both well-motivated and well-written. However, multiple reviewers share the concern (which I agree with) that the method fails on ImageNet experiments, and the method often fails to beat even temperature scaling. The experimental comparison arguing that the approach improves upon many methods pre-temperature scaling is unfair as no other method leverages the validation set. This makes for a fairly deceiving slight of hand if not read carefully. When compared to temperature scaling, which does use the validation set, the performance improvement gap is diminished altogether.

I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue. In particular, improve the clarity around the experimental validation.
Decision: Reject