Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper describes how the use of the now-standard focal loss can lead to improved calibration results when used to fit deep-models. When fitting a large capacity model with NLL, the model can often try to drive its predictions close to 1 (i.e. infinity pre-softmax) on the training set, ultimately leading to poorly calibrated models and overfitting behaviour. The focal loss appears to mitigate this issue.

The approach is extremely simple to implement, the theoretical justifications are believable, and the calibration/accuracy performances seem to be good -- for this reasons, I think that the paper should be accepted.

(1) it would be interesting to compare the approach to using the standard cross-entropy applied to smoothed labels (i.e. (1-eps,eps) instead of (1,0) in binary classification and obvious generalisation in multi-class setting).

(2) data-augmentation often greatly helps with calibration -- the paper did not describe in details what has been done on that front for the numerical investigations.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper studies the effect of the focal loss, proposed by Lin et al. in 2017 on network miscalibration, which appears when the network's confidence in its prediction does not match its correctness. The authors provide a theoretical explanation to the superior results of the focal loss for calibration. The temperature scaling technique of Guo et al. 2017 is applied (dividing the network's logits by a scalar learnt on a val set prior to softmax) to networks trained using the focal loss, with different options for the focal parameter, as well as the standard multi-class cross entropy and a few others. The experiments on CIFAR10/100 as well as two text dataset (20 Newsgroups, Stanford Sentiment Treebank) reach lower expected calibration error compared to the cross entropy (75% of relative improvement on cifar100 for instance).

The importance of the contribution will probably be discussed here. At first glance, it seems that the works build mainly on advances from Lin et al & Guo et al, but the authors do a promising job in combining the two.  

Positive aspects: 
- The paper is well written. 
- Experiments on both image and text dataset demonstrate the superiority of the focal loss on several calibration metrics. 
- The theoretical explanation is convincing.   

Negative points:
- The importance of the problem is motivated by future assessments by downstream tasks but do not address this aspect in the experiments. In particular, as the images experiments are conducted on tiny images, an experiment on a real size image dataset would strengthen the paper. 
- The policy that works best for defining the sample wise tuning of the focal parameter was hand-made but ultimately uses only 3 parameters so finally it is not so bad.  

Minor: 
- It'd be nice to illustrate the confidence improvements on a few qualitative examples, maybe in appendix.
- 10 pages is too much (given that were were given instructions to be more severe with long paper) table 6 and 3 could be merged for instance. 
- The focal loss column results of table 1 should be the same as Table 5 (sample wise)?
- could specify what MMCE means
- clean the bibliography

I've read the other reviews and authors' responses. Experiences on Tiny ImageNet are better than CIFAR but still a little far from what I'd call real images but I understand it can be difficult to run experiments on ImageNet. Since the choice of gamma seems to be leading consistent results also on tinyIN, I find it less concerning.    
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper explores how focal loss can be used to improve calibration for classifiers. Focal loss extends the cross-entropy loss, which is -log(p_label), with a multiplicative factor equal to (1 - p_label)^gamma. Intuitively, this downweights the loss for elements where the probability of the correct label p_label is close to 1, relatively increasing the weight of the misclassified examples.

Somewhat surprisingly, this tends to improve the calibration of the model. I say surprisingly because the focal loss is not a bregman divergence for all values of alpha so in general the expected minimizer of the focal loss for a fractional label is not the fractional label (i.e. the minimizer wrt x of - p (1-x)^gamma log(x) - (1-p) x^gamma log (1 -x) is not in general p).

The paper shows somewhat thorough experiments on many datasets justifying this observation, but the theoretical part is rather weak since it doesn't seem to address this issue with the focal loss.

It's also not very clear from reading the paper what the p0 should be when using the rule to automatically select the gamma of the focal loss.

I'd support accepting the paper if the calibration properties of the focal loss itself was better analyzed on a simpler setup (linear models, or single parameter models) so it's easier to understand how it's helping calibration in the deep network setup and if the algorithm for choosing per-example gammas was more clearly stated out.
review_title: Official Blind Review #1

Meta-review: The paper investigates the effect of focal loss on calibration of neural nets.

On one hand, the reviewers agree that this paper is well-written and the empirical results are interesting. On the other hand, the reviewers felt that there could be better evaluation of the effect of calibration on downstream tasks, and better justification for the choice of optimal gamma (e.g. on a simpler problem setup).

I encourage the others to revise the draft and resubmit to a different venue.  

Decision: Reject